Entropic Risk-Sensitive Reinforcement Learning: A Meta Regret Framework with Function Approximation

1 INTRODUCTION . Risk is one of the most important considerations in decision making , so should it be in reinforcement learning ( RL ) . As a prominent paradigm in RL that performs learning while accounting for risk , risksensitive RL explicitly models risk of decisions via certain risk measures and optimizes for rewards simultaneously . It is poised to play an essential role in application domains where accounting for risk in decision making is crucial . A partial list of such domains includes autonomous driving ( Buehler et al. , 2009 ; Thrun , 2010 ) , behavior modeling ( Niv et al. , 2012 ; Shen et al. , 2014 ) , realtime strategy games ( Berner et al. , 2019 ; Vinyals et al. , 2019 ) and robotic surgery ( Fagogenis et al. , 2019 ; Shademan et al. , 2016 ) . In this paper , we study risk-sensitive RL through the lens of function approximation , which is an important apparatus for scaling up and accelerating RL algorithms in applications of high dimension . We focus on risk-sensitive RL with the entropic risk measure , a classical framework established by the seminal work of Howard & Matheson ( 1972 ) . Informally , for a fixed risk parameter β 6= 0 , our goal is to maximize the objective Vβ = 1 β log { EeβR } . ( 1 ) The definition of Vβ will be made formal later in ( 2 ) . The objective ( 1 ) admits a Taylor expansion Vβ = E [ R ] + β2 Var ( R ) + O ( β 2 ) . Comparing ( 1 ) with the risk-neutral objective V = E [ R ] studied in the standard RL setting , we see that β > 0 induces a risk-seeking objective and β < 0 induces a risk-averse one . Therefore , the formulation with the entropic risk measure in ( 1 ) accounts for both risk-seeking and risk-averse modes of decision making , whereas most others are restricted to the risk-averse setting ( Fu et al. , 2018 ) . It can also be seen that Vβ tends to the risk-neutral V as β → 0 . Existing works on function approximation for RL have mostly focused on the risk-neutral setting and heavily exploits the linearity of risk-neutral objective V in both transition dynamics ( implicitly captured by the expectation ) and the reward R , which is clearly not available in the risk-sensitive objective ( 1 ) . It is also well known that even in the risk-neutral setting , improperly implemented function approximation could result in errors that scale exponentially in the size of the state space . Combined with nonlinearity of the risk-sensitive objective ( 1 ) , it compounds the difficulties of implementing function approximation in risk-sensitive RL with provable guarantees . This work provides a principled solution to function approximation in risk-sensitive RL by overcoming the above difficulties . Under the finite-horizon MDP setting , we propose a meta algorithm based on value iteration , and from that we derive two concrete algorithms for linear and general function approximation , which we name RSVI.L and RSVI.G , respectively . By modeling a shifted exponential transformation of estimated value functions , RSVI.L and RSVI.G cater to the nonlinearity of the risk-sensitive objective ( 1 ) and adapt to both risk-seeking and risk-averse settings . Moreover , both RSVI.L and RSVI.G maintain risk-sensitive optimism in the face of uncertainty for effective exploration . In particular , RSVI.L exploits a synergistic relationship between feature mapping and regularization in a risk-sensitive fashion . The resulting structure of RSVI.L makes it more efficient in runtime and memory than RSVI.G under linear function approximation , while RSVI.G is more general and allows for function approximation beyond the linear setting . Furthermore , we develop a meta regret analytic framework and identify a risk-sensitive optimism condition that serves as the core component of the framework . Under the optimism condition , we prove a meta regret bound incurred by any instance of the meta algorithm , regardless of function approximation settings . Furthermore , we show that both RSVI.L and RSVI.G satisfy the optimism condition under the respective function approximation and achieve regret that scales sublinearly in the number of episodes . The meta framework therefore helps us disentangle the analysis associated with function approximation from the generic analysis , shedding light on the role of function approximation in regret guarantees . We hope that our meta framework will motivate and benefit future studies of function approximation in risk-sensitive RL . Our contributions . We may summarize the contributions of the present paper as follows : • we study function approximation in risk-sensitive RL with the entropic risk measure ; we provide a meta algorithm , from which we derive two concrete algorithms for linear and general function approximation , respectively ; the concrete algorithms are both shown to adapt to all levels of risk sensitivity and maintain risk-sensitive optimism over the learning process ; • we develop a meta regret analytic framework and identify a risk-sensitive optimism condition , under which we prove a meta regret bound for the meta algorithm ; furthermore , by showing that the optimism condition holds for both concrete algorithms , we establish regret bounds for them under linear and general function approximation , respectively . Notations . For a positive integer n , we let [ n ] : = { 1 , 2 , . . . , n } . For a number u 6= 0 , we define sign ( u ) = 1 if u > 0 and −1 if u < 0 . For two non-negative sequences { ai } and { bi } , we write ai . bi if there exists a universal constant C > 0 such that ai ≤ Cbi for all i , and write ai bi if ai . bi and bi . ai . We use Õ ( · ) to denote O ( · ) while hiding logarithmic factors . For any ε > 0 and set X , we letNε ( X , ‖ · ‖ ) be the ε-net of the set X with respect to the norm ‖ · ‖ . We let ∆ ( X ) be the set of probability distributions supported on X . For any vector u ∈ Rn and symmetric and positive definite matrix Γ ∈ Rn×n , we let ‖u‖Γ : = √ u > Γu . We denote by In the n × n identity matrix . 2 RELATED WORK . Initiated by the seminal work of Howard & Matheson ( 1972 ) , risk-sensitive control/RL with the entropic risk measure has been studied in a vast body of literature ( Bäuerle & Rieder , 2014 ; Borkar , 2001 ; 2002 ; 2010 ; Borkar & Meyn , 2002 ; Cavazos-Cadena & Hernández-Hernández , 2011 ; Coraluppi & Marcus , 1999 ; Di Masi & Stettner , 1999 ; 2000 ; 2007 ; Fleming & McEneaney , 1995 ; Hernández-Hernández & Marcus , 1996 ; Jaśkiewicz , 2007 ; Marcus et al. , 1997 ; Mihatsch & Neuneier , 2002 ; Osogami , 2012 ; Patek , 2001 ; Shen et al. , 2013 ; 2014 ; Whittle , 1990 ) . Yet , this line of works either assumes known transition kernels or focuses on asymptotic behaviors of the problem/algorithms , and finite-sample/time results with unknown transitions have rarely been investigated . The most relevant work to ours is perhaps Fei et al . ( 2020 ) , who consider the same problem as ours under the tabular setting . They propose two algorithms based on value iteration and Q-learning . They prove regret bounds for their algorithms , which are then certified to be nearly optimal by a lower bound . However , their algorithms and analysis are restricted to the tabular setting . Compared to Fei et al . ( 2020 ) , our paper provides a novel and unified framework of algorithms and analysis for function approximation . We study linear and general function approximation as two instances of the framework , both of which subsume the tabular setting . We also briefly discuss existing works on function approximation with regret analysis , which so far have focused on the risk-neutral setting . The works of Cai et al . ( 2019 ) ; Jin et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Yang & Wang ( 2019 ) ; Zhou et al . ( 2020 ) study linear function approximation , while Ayoub et al . ( 2020 ) ; Wang et al . ( 2020 ) investigate general function approximation . In addition , all these works prove Õ ( K1/2 ) -regret for their algorithms , although dependence on other parameters varies in settings . As we have argued in the previous section , the nonlinear objective ( 1 ) makes algorithm design and regret analysis for function approximation much more challenging in risksensitive settings than in the standard risk-neutral one . 3 PROBLEM FORMULATION . 3.1 EPISODIC MDP . An episodic MDP is parameterized by a tuple ( K , H , S , A , { Ph } h∈ [ H ] , { rh } h∈ [ H ] ) , where K is the number of episodes , H is the number of steps in each episode , S is the state space , A is the action space , Ph : S ×A → ∆ ( S ) is the transition kernel at step h , and rh : S ×A → [ 0 , 1 ] is the reward function at step h. We assume that { Ph } are unknown . For simplicity we also assume that { rh } are known and deterministic , as is done in existing works such as Yang & Wang ( 2019 ) ; Zhou et al . ( 2020 ) . We interact with the episodic MDP as follows . In the beginning of each episode k ∈ [ K ] , the environment chooses an arbitrary initial state sk1 ∈ S . Then in each step h ∈ [ H ] , we take an action akh ∈ A , receives a reward rh ( skh , akh ) and transitions to the next state skh+1 ∈ S sampled from Ph ( · | skh , akh ) . Once we reach skH+1 , the current episode terminates and we advance to the next episode unless k = K . 3.2 VALUE FUNCTIONS , BELLMAN EQUATIONS AND REGRET . We assume that β is fixed prior to the learning process , and for notational simplicity we omit it from quantities to be introduced subsequently . In risk-sensitive RL with the entropic risk measure , we aim to find a policy π = { πh : S → A } so as to maximize the value function given by V πh ( s ) : = 1 β log { E [ exp ( β H∑ h′=h rh′ ( sh′ , πh′ ( sh′ ) ) ) ] ∣∣∣∣∣ sh = s } , ( 2 ) for all ( h , s ) ∈ [ H ] × S. Under some mild regularity conditions , there exists a greedy policy π∗ = { π∗h } which gives the optimal value V π ∗ h ( s ) = supπ V π h ( s ) for all ( h , s ) ∈ [ H ] × S ( Bäuerle & Rieder , 2014 ) . In addition to the value function , another key notion is the action-value function defined as Qπh ( s , a ) : = 1 β log { E [ exp ( β H∑ h′=h rh′ ( sh′ , ah′ ) ) ∣∣∣∣∣ sh = s , ah = a ] } , ( 3 ) for all ( h , s , a ) ∈ [ H ] × S ×A . The action-value function Qπh is closely associated with the value function V π h via the so-called Bellman equation : Qπh ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ exp ( β · V πh+1 ( s′ ) ) ] } , ( 4 ) V πh ( s ) = Q π h ( s , πh ( s ) ) , V π H+1 ( s ) = 0 , which holds for all ( h , s , a ) ∈ [ H ] × S ×A . Note that the identity of Qπh in ( 4 ) is a result of simple calculation based on ( 2 ) and ( 3 ) . Similarly , the Bellman optimality equation is given by Q∗h ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ exp ( β · V ∗h+1 ( s′ ) ) ] } , ( 5 ) V ∗h ( s ) = max a∈A Q∗h ( s , a ) , V ∗ H+1 ( s ) = 0 , again for all ( h , s , a ) ∈ [ H ] × S × A . In the above , we use the shorthand Q∗h ( · , · ) : = Qπ ∗ h ( · , · ) for all h ∈ [ H ] and V ∗h ( · ) is similarly defined . The identity V ∗h ( · ) = maxa∈AQ∗h ( · , a ) implies that the optimal π∗ is the greedy policy with respect to the optimal action-value function { Q∗h } h∈ [ H ] . During the learning process , the policy πk in each episode k may be different from the optimal π∗ . We quantify this difference over all K episodes through the notion of regret , defined as Regret ( K ) : = ∑ k∈ [ K ] [ V ∗1 ( s k 1 ) − V π k 1 ( s k 1 ) ] . ( 6 ) Since V ∗1 ( s ) ≥ V π1 ( s ) for any π and s ∈ S by definition , regret also characterizes the suboptimality of { πk } relative to the optimal π∗ .

Summary: The paper studies risk-sensitive reinforcement learning with the entropic risk measure and function approximation. A meta algorithm based on value iteration is first proposed, then the paper proposes two concrete instantiations, one for linear function approximation and one for general function approximation. Regret bound for both algorithms depend sub-linearly in the number of episodes, and the linear one depends polynomially on the ambient dimension, whereas the general one depends polynomially on the Eluder dimension.