Entropic Risk-Sensitive Reinforcement Learning: A Meta Regret Framework with Function Approximation

1 INTRODUCTION . Risk is one of the most important considerations in decision making , so should it be in reinforcement learning ( RL ) . As a prominent paradigm in RL that performs learning while accounting for risk , risksensitive RL explicitly models risk of decisions via certain risk measures and optimizes for rewards simultaneously . It is poised to play an essential role in application domains where accounting for risk in decision making is crucial . A partial list of such domains includes autonomous driving ( Buehler et al. , 2009 ; Thrun , 2010 ) , behavior modeling ( Niv et al. , 2012 ; Shen et al. , 2014 ) , realtime strategy games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ) and robotic surgery ( Fagogenis et al. , 2019 ; Shademan et al. , 2016 ) . In this paper , we study risk-sensitive RL through the lens of function approximation , which is an important apparatus for scaling up and accelerating RL algorithms in applications of high dimension . We focus on risk-sensitive RL with the entropic risk measure , a classical framework established by the seminal work of Howard & Matheson ( 1972 ) . Informally , for a fixed risk parameter β 6= 0 , our goal is to maximize the objective Vβ = 1 β log { EeβR } . ( 1 ) The definition of Vβ will be made formal later in ( 2 ) . The objective ( 1 ) admits a Taylor expansion Vβ = E [ R ] + β2 Var ( R ) + O ( β 2 ) . Comparing ( 1 ) with the risk-neutral objective V = E [ R ] studied in the standard RL setting , we see that β > 0 induces a risk-seeking objective and β < 0 induces a risk-averse one . Therefore , the formulation with the entropic risk measure in ( 1 ) accounts for both risk-seeking and risk-averse modes of decision making , whereas most others are restricted to the risk-averse setting ( Fu et al. , 2018 ) . It can also be seen that Vβ tends to the risk-neutral V as β → 0 . Existing works on function approximation for RL have mostly focused on the risk-neutral setting and heavily exploits the linearity of risk-neutral objective V in both transition dynamics ( implicitly captured by the expectation ) and the reward R , which is clearly not available in the risk-sensitive objective ( 1 ) . It is also well known that even in the risk-neutral setting , improperly implemented function approximation could result in errors that scale exponentially in the size of the state space . Combined with nonlinearity of the risk-sensitive objective ( 1 ) , it compounds the difficulties of implementing function approximation in risk-sensitive RL with provable guarantees . This work provides a principled solution to function approximation in risk-sensitive RL by overcoming the above difficulties . Under the finite-horizon MDP setting , we propose a meta algorithm based on value iteration , and from that we derive two concrete algorithms for linear and general function approximation , which we name RSVI.L and RSVI.G , respectively . By modeling a shifted exponential transformation of estimated value functions , RSVI.L and RSVI.G cater to the nonlinearity of the risk-sensitive objective ( 1 ) and adapt to both risk-seeking and risk-averse settings . Moreover , both RSVI.L and RSVI.G maintain risk-sensitive optimism in the face of uncertainty for effective exploration . In particular , RSVI.L exploits a synergistic relationship between feature mapping and regularization in a risk-sensitive fashion . The resulting structure of RSVI.L makes it more efficient in runtime and memory than RSVI.G under linear function approximation , while RSVI.G is more general and allows for function approximation beyond the linear setting . Furthermore , we develop a meta regret analytic framework and identify a risk-sensitive optimism condition that serves as the core component of the framework . Under the optimism condition , we prove a meta regret bound incurred by any instance of the meta algorithm , regardless of function approximation settings . Furthermore , we show that both RSVI.L and RSVI.G satisfy the optimism condition under the respective function approximation and achieve regret that scales sublinearly in the number of episodes . The meta framework therefore helps us disentangle the analysis associated with function approximation from the generic analysis , shedding light on the role of function approximation in regret guarantees . We hope that our meta framework will motivate and benefit future studies of function approximation in risk-sensitive RL . Our contributions . We may summarize the contributions of the present paper as follows : • we study function approximation in risk-sensitive RL with the entropic risk measure ; we provide a meta algorithm , from which we derive two concrete algorithms for linear and general function approximation , respectively ; the concrete algorithms are both shown to adapt to all levels of risk sensitivity and maintain risk-sensitive optimism over the learning process ; • we develop a meta regret analytic framework and identify a risk-sensitive optimism condition , under which we prove a meta regret bound for the meta algorithm ; furthermore , by showing that the optimism condition holds for both concrete algorithms , we establish regret bounds for them under linear and general function approximation , respectively . Notations . For a positive integer n , we let [ n ] : = { 1 , 2 , . . . , n } . For a number u 6= 0 , we define sign ( u ) = 1 if u > 0 and −1 if u < 0 . For two non-negative sequences { ai } and { bi } , we write ai . bi if there exists a universal constant C > 0 such that ai ≤ Cbi for all i , and write ai bi if ai . bi and bi . ai . We use Õ ( · ) to denote O ( · ) while hiding logarithmic factors . For any ε > 0 and set X , we letNε ( X , ‖ · ‖ ) be the ε-net of the set X with respect to the norm ‖ · ‖ . We let ∆ ( X ) be the set of probability distributions supported on X . For any vector u ∈ Rn and symmetric and positive definite matrix Γ ∈ Rn×n , we let ‖u‖Γ : = √ u > Γu . We denote by In the n × n identity matrix . 2 RELATED WORK . Initiated by the seminal work of Howard & Matheson ( 1972 ) , risk-sensitive control/RL with the entropic risk measure has been studied in a vast body of literature ( Bäuerle & Rieder , 2014 ; Borkar , 2001 ; 2002 ; 2010 ; Borkar & Meyn , 2002 ; Cavazos-Cadena & Hernández-Hernández , 2011 ; Coraluppi & Marcus , 1999 ; Di Masi & Stettner , 1999 ; 2000 ; 2007 ; Fleming & McEneaney , 1995 ; Hernández-Hernández & Marcus , 1996 ; Jaśkiewicz , 2007 ; Marcus et al. , 1997 ; Mihatsch & Neuneier , 2002 ; Osogami , 2012 ; Patek , 2001 ; Shen et al. , 2013 ; 2014 ; Whittle , 1990 ) . Yet , this line of works either assumes known transition kernels or focuses on asymptotic behaviors of the problem/algorithms , and finite-sample/time results with unknown transitions have rarely been investigated . The most relevant work to ours is perhaps Fei et al . ( 2020 ) , who consider the same problem as ours under the tabular setting . They propose two algorithms based on value iteration and Q-learning . They prove regret bounds for their algorithms , which are then certified to be nearly optimal by a lower bound . However , their algorithms and analysis are restricted to the tabular setting . Compared to Fei et al . ( 2020 ) , our paper provides a novel and unified framework of algorithms and analysis for function approximation . We study linear and general function approximation as two instances of the framework , both of which subsume the tabular setting . We also briefly discuss existing works on function approximation with regret analysis , which so far have focused on the risk-neutral setting . The works of Cai et al . ( 2019 ) ; Jin et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Yang & Wang ( 2019 ) ; Zhou et al . ( 2020 ) study linear function approximation , while Ayoub et al . ( 2020 ) ; Wang et al . ( 2020 ) investigate general function approximation . In addition , all these works prove Õ ( K1/2 ) -regret for their algorithms , although dependence on other parameters varies in settings . As we have argued in the previous section , the nonlinear objective ( 1 ) makes algorithm design and regret analysis for function approximation much more challenging in risksensitive settings than in the standard risk-neutral one . 3 PROBLEM FORMULATION . 3.1 EPISODIC MDP . An episodic MDP is parameterized by a tuple ( K , H , S , A , { Ph } h∈ [ H ] , { rh } h∈ [ H ] ) , where K is the number of episodes , H is the number of steps in each episode , S is the state space , A is the action space , Ph : S ×A → ∆ ( S ) is the transition kernel at step h , and rh : S ×A → [ 0 , 1 ] is the reward function at step h. We assume that { Ph } are unknown . For simplicity we also assume that { rh } are known and deterministic , as is done in existing works such as Yang & Wang ( 2019 ) ; Zhou et al . ( 2020 ) . We interact with the episodic MDP as follows . In the beginning of each episode k ∈ [ K ] , the environment chooses an arbitrary initial state sk1 ∈ S . Then in each step h ∈ [ H ] , we take an action akh ∈ A , receives a reward rh ( skh , akh ) and transitions to the next state skh+1 ∈ S sampled from Ph ( · | skh , akh ) . Once we reach skH+1 , the current episode terminates and we advance to the next episode unless k = K . 3.2 VALUE FUNCTIONS , BELLMAN EQUATIONS AND REGRET . We assume that β is fixed prior to the learning process , and for notational simplicity we omit it from quantities to be introduced subsequently . In risk-sensitive RL with the entropic risk measure , we aim to find a policy π = { πh : S → A } so as to maximize the value function given by V πh ( s ) : = 1 β log { E [ exp ( β H∑ h′=h rh′ ( sh′ , πh′ ( sh′ ) ) ) ] ∣∣∣∣∣ sh = s } , ( 2 ) for all ( h , s ) ∈ [ H ] × S. Under some mild regularity conditions , there exists a greedy policy π∗ = { π∗h } which gives the optimal value V π ∗ h ( s ) = supπ V π h ( s ) for all ( h , s ) ∈ [ H ] × S ( Bäuerle & Rieder , 2014 ) . In addition to the value function , another key notion is the action-value function defined as Qπh ( s , a ) : = 1 β log { E [ exp ( β H∑ h′=h rh′ ( sh′ , ah′ ) ) ∣∣∣∣∣ sh = s , ah = a ] } , ( 3 ) for all ( h , s , a ) ∈ [ H ] × S ×A . The action-value function Qπh is closely associated with the value function V π h via the so-called Bellman equation : Qπh ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ exp ( β · V πh+1 ( s′ ) ) ] } , ( 4 ) V πh ( s ) = Q π h ( s , πh ( s ) ) , V π H+1 ( s ) = 0 , which holds for all ( h , s , a ) ∈ [ H ] × S ×A . Note that the identity of Qπh in ( 4 ) is a result of simple calculation based on ( 2 ) and ( 3 ) . Similarly , the Bellman optimality equation is given by Q∗h ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ exp ( β · V ∗h+1 ( s′ ) ) ] } , ( 5 ) V ∗h ( s ) = max a∈A Q∗h ( s , a ) , V ∗ H+1 ( s ) = 0 , again for all ( h , s , a ) ∈ [ H ] × S × A . In the above , we use the shorthand Q∗h ( · , · ) : = Qπ ∗ h ( · , · ) for all h ∈ [ H ] and V ∗h ( · ) is similarly defined . The identity V ∗h ( · ) = maxa∈AQ∗h ( · , a ) implies that the optimal π∗ is the greedy policy with respect to the optimal action-value function { Q∗h } h∈ [ H ] . During the learning process , the policy πk in each episode k may be different from the optimal π∗ . We quantify this difference over all K episodes through the notion of regret , defined as Regret ( K ) : = ∑ k∈ [ K ] [ V ∗1 ( s k 1 ) − V π k 1 ( s k 1 ) ] . ( 6 ) Since V ∗1 ( s ) ≥ V π1 ( s ) for any π and s ∈ S by definition , regret also characterizes the suboptimality of { πk } relative to the optimal π∗ .

Summary: This paper proposes a risk-sensitive algorithm with function approximation in reinforcement learning. To handle the uncertainty, the proposed algorithms consider an entropic risk value function controlled by a risk parameter, which provides a unified framework for both risk-sensitive and risk-averse settings. The main contribution of this paper is to provide theoretical guarantees for the proposed algorithms.