Distributional Reinforcement Learning for Risk-Sensitive Policies

1 INTRODUCTION . In standard reinforcement learning ( RL ) ( Sutton & Barto , 2018 ) , one seeks to learn a policy that maximizes an objective , usually the expected total discounted rewards or the long-term average rewards . In stochastic domains , especially when the level of uncertainty involved is high , maximizing the expectation may not be the most desirable since the solution may have high variance and occasionally performs badly . In such scenarios one may choose to learn a policy that is more risk-averse and avoids bad outcomes , even though the long-term average performance is slightly lower than the optimal . In this work we consider optimizing the conditional value-at-risk ( CVaR ) ( Rockafellar & Uryasev , 2000 ) , a popular risk measure , widely used in financial applications , and is increasingly being used in RL . The CVaR objective focuses on the lower tail of the return and is therefore more sensitive to rare but catastrophic outcomes . Various settings and RL approaches have been proposed to solve this problem ( Petrik & Subramanian , 2012 ; Chow & Ghavamzadeh , 2014 ; Chow & Pavone , 2014 ; Tamar et al. , 2015 ; Tamar et al. , 2017 ; Huang & Haskell , 2020 ) . Most of the proposed approaches , however , involve more complicated algorithms than standard RL algorithms such as Q-learning ( Watkins & Dayan , 1992 ) and its deep variants , e.g . DQN ( Mnih et al. , 2015 ) . Recently , the distributional approach to RL ( Bellemare et al. , 2017 ; Morimura et al. , 2010 ) has received increased attention due to its ability to learn better policies than the standard approaches in many challenging tasks ( Dabney et al. , 2018a ; b ; Yang et al. , 2019 ) . Instead of learning a value function that provides the expected return of each state-action pair , the distributional approach learns the entire return distribution of each state-action pair . While this is computationally more costly , the approach itself is a simple extension to standard RL and is therefore easy to implement and able to leverage many of the advances in deep RL . Since the entire distribution is available , one naturally considers exploiting this information to optimize for an objective other than the expectation . Dabney et al . ( 2018a ) presented a simple way to do so for a family of risk measures including the CVaR . The theoretical properties of such approach , however , are not clear . In particular , it is not clear whether the algorithm converges to any particular variant of CVaR-optimal policy . We address this issue in this work . Our main contribution is to first show that the proposed algorithm in ( Dabney et al. , 2018a ) overestimates the dynamic , Markovian CVaR but empirically can be as conservative . It has been demon- strated that this variant of CVaR can be overly conservative in many scenarios ( Tamar et al. , 2017 ; Yu et al. , 2017 ) , and one may prefer the static CVaR instead as the objective . Our second contribution is to propose a modified algorithm that can help achieve this . Empirically , we show that the proposed approach learns policies that perform better in terms of the overall CVaR objective on both synthetic and real-world problems . We close the introduction section with some references to related works . We formally present our problem setup as well as our main analytical results in Section 2 . Section 3 describes our proposed algorithm and finally , Section 4 presents our empirical results . 1.1 RELATED WORKS . The literature on distributional RL has been greatly expanded recently ( Morimura et al. , 2010 ; Bellemare et al. , 2017 ; Barth-Maron et al. , 2018 ; Dabney et al. , 2018a ; b ; Yang et al. , 2019 ) . Most of these works focus on the modeling aspects , such as the choice of representations for the value distributions . The approach has been used to enhance exploration in RL ( Mavrin et al. , 2019 ; Zhang & Yao , 2019 ) and in risk-sensitive applications ( Wang et al. , 2019 ; Bernhard et al. , 2019 ) . Solving Markov decision processes ( MDP ) with risk-sensitive objectives have been addressed in many works ( Howard & Matheson , 1972 ; Ruszczynski , 2010 ; Bäuerle & Ott , 2011 ) , including RL approaches ( Borkar , 2001 ; Tamar et al. , 2012 ; L.A. & Ghavamzadeh , 2013 ) . In particular , Chow & Ghavamzadeh ( 2014 ) ; Tamar et al . ( 2015 ) deal with the static CVaR objectives while Petrik & Subramanian ( 2012 ) ; Chow & Pavone ( 2014 ) deal with the dynamic CVaR objectives . Tamar et al . ( 2017 ) proposed a policy-gradient approach that deals with both the static and the dynamic CVaR objectives . Closest to ours is the work by Stanko & Macek ( 2019 ) . Their proposed approach also makes use of distributional RL but it is not clear whether their action selection strategy properly optimizes either the static or the dynamic CVaR . 2 PROBLEM SETUP AND MAIN RESULTS . We consider a discrete-time Markov decision process with state space X and action space A . For simplicity we assume that X and A are finite , although our results and algorithm can be readily extended to more general state-action spaces . We assume that the rewards are bounded and drawn from a countable set R ⊂ R. Given states xt , xt+1 ∈ X for any t ∈ { 0 , 1 , . . . } , the probability of receiving reward rt ∈ R and transitioning to xt+1 after executing at ∈ A in xt is given by p ( rt , xt+1|xt , at ) . Without loss of generality we assume a fixed initial state x0 , unless stated otherwise . Given a policy π : H → P ( A ) , where H is the set of all histories so far ht : = ( x0 , a0 , r0 , x1 , a1 , r1 , . . . , xt ) ∈ H , and P ( A ) the space of distributions over A , its expected total discounted reward over time is given by V π : = Eπp [ ∞∑ t=0 γtrt ] where γ ∈ ( 0 , 1 ) is a discount factor . The superscript π in the expectation indicates that the actions at are drawn from π ( ht ) . The subscript p indicates that the rewards and state transitions are induced by p. In standard RL , we aim to find a policy that maximizes V π . It is well-known that there exists a deterministic stationary policy π : X → A whose decisions depend only on the current state , that gives optimal V π , and therefore one typically works in the space of stationary deterministic policies . Key to a dynamic-programming solution to the above problem is the use of a value function Qπ ( x , a ) : = Eπp [ ∑∞ t=0 γ trt|x0 = x , a0 = a ] , which satisfies the Bellman equation ∀x , a , Qπ ( x , a ) = ∑ r , x′ p ( r , x′|x , a ) [ r + γQπ ( x′ , π ( x′ ) ) ] . ( 1 ) The optimal valueQ∗ ( x , a ) : = Qπ ∗ ( x , a ) for any optimal policy π∗ satisfies the Bellman optimality equation ∀x , a , Q∗ ( x , a ) = ∑ r , x′ p ( r , x′|x , a ) [ r + γmax a′ Q∗ ( x′ , a′ ) ] . ( 2 ) Furthermore , for any Q-function Q ∈ Q : = { q : X × A → R | q ( x , a ) < ∞ , ∀x , a } , one can show that the operator T π defined by T πQ ( x , a ) : = ∑ r , x′ p ( r , x ′|x , a ) [ r + γQ ( x′ , π ( x′ ) ) ] is a γ-contraction in the sup-norm ‖Q‖∞ : = maxx , a |Q ( x , a ) | with fixed-point satisfying ( 1 ) . One can therefore start with an arbitraryQ-function and repeatedly apply T π , or its stochastic approximation , to learn Qπ . An analogous operator T can also be shown to be a γ-contraction with fixed-point satisfying ( 2 ) . 2.1 STATIC AND DYNAMIC CVAR . The expected return V π is risk-neutral in the sense that it does not take into account the inherent variability of the return . In many application scenarios , one may prefer a policy that is more riskaverse , with better sensitivity to bad outcomes . In this work , we focus on the conditional value-atrisk ( CVaR ) , which is a popular risk measure that satisfies the properties of being coherent ( Artzner et al. , 1999 ) . The α-level CVaR for a random real-valued variable Z , for α ∈ ( 0 , 1 ] , is given by ( Rockafellar & Uryasev , 2000 ) Cα ( Z ) : = max s∈R s− 1 α E [ ( s− Z ) + ] where ( x ) + = max { x , 0 } . Note that we are concerned with Z that represents returns ( the higher , the better ) , so this particular version of CVaR focuses on the lower tail of the distribution . In particular , the function s 7→ s − 1αE [ ( s − Z ) + ] is concave in s and the maximum is always attained at the α-level quantile , defined as qα ( Z ) : = inf { s : Pr ( Z ≤ s ) ≥ α } . For α = 1 , Cα reduces to the standard expectation . In the case Z is absolutely continuous , we have the intuitive Cα ( Z ) = E [ Z|Z < qα ] . Our target random variable is the total discounted return Zπ : = ∑∞ t=0 γ trt of a policy π , and our objective is to find a policy that maximizes Cα ( Zπ ) , where the optimal CVaR is given by max π max s s− 1 α Eπp [ ( s− Zπ ) + ] . ( 3 ) In the context where Z is accumulated over multiple time steps , the objective ( 3 ) corresponds to maximizing the so-called static CVaR . This objective is time-inconsistent in the sense that the optimal policy may be history-dependent and therefore non-Markov . This is , however , perfectly expected since the optimal behavior in the later time steps may depend on how much rewards have been accumulated thus far – more risky actions can be taken if one has already collected sufficiently large total rewards , and vice versa . From the point of view of dynamic programming , an alternative , time-consistent or Markovian version of CVaR may be more convenient . A class of such risk measures was proposed by Ruszczynski ( 2010 ) , and we shall refer to this version of CVaR as the dynamic CVaR , defined recursively as1 ∀π , x , a , Dπα,0 ( x , a ) : = Cα [ rt|xt = x , at = a ] , ∀π , x , a , T > 0 , Dπα , T ( x , a ) : = Cα [ rt + γDπα , T−1 ( xt+1 , π ( xt+1 ) ) |xt = x , at = a ] , and ∀π , x , a , Dπα ( x , a ) : = lim T→∞ Dπα , T ( x , a ) . It can be shown ( Ruszczynski , 2010 ) that there exists a stationary deterministic optimal policy π∗ , maximizing Dπα ( x , a ) for all x , a , whose dynamic CVaR is given by D ∗ α : = D π∗ α . In particular , the operator TD , α defined by TD , αD ( x , a ) : = Cα [ rt + γmax a′ D ( xt+1 , a ′ ) |xt = x , at = a ] ( 4 ) for D ∈ Q is a γ-contraction in sup-norm with fixed-point satisfying ∀x , a , D∗α ( x , a ) = Cα [ rt + γmax a′ D∗α ( xt+1 , a ′ ) |xt = x , at = a ] . ( 5 ) 1We use a slightly different definition from that in ( Ruszczynski , 2010 ) , but conceptually they are essentially the same . The dynamic CVaR , however , can be overly conservative in many cases . We illustrate this with some empirical results in Section 4 . In such cases it may be favorable to use the static CVaR . Bäuerle & Ott ( 2011 ) suggest an iterative process that can be used to solve for the optimal static CVaR policy . The approach is based on ( 3 ) : 1 . For a fixed s , one can solve for the optimal policy with respect to maxπ E [ − ( s− Zπ ) + ] . 2 . For a fixed π , the optimal s is given by the α-level quantile of Zπ . 3 . Repeat until convergence . Step one above can be done by solving an augmented MDP with states x̃ = ( x , s ) ∈ X × R , where s is a moving threshold keeping track of the accumulated rewards so far . In particular , this MDP has no rewards and state transition is given by p ( 0 , ( x′ , s−rγ ) | ( x , s ) , a ) : = p ( r , x ′|x , a ) . Solving this augmented MDP directly using RL , however , can result in poor sample efficiency since each example ( x , a , r , x′ ) may need to be experienced many times under different threshold s. In this work , we propose an alternative solution using the approach of distributional RL .

Summary: This paper is about risk-sensitive RL based on the CVaR risk measure. This paper is mainly based on the work presented in Dabney et al. in 2018 which is about distributional RL for a family of risk measures which includes CVaR as well. The main motivation for this work was the point that the method presented in Dabney et al. 2018 overestimates the dynamics and could be excessively conservative in certain scenarios. Authors have proposed to use static CVaR instead and have developed algorithms to do that.