Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning

1 Introduction . Risk-sensitive reinforcement learning ( RL ) is important for practical and high-stake applications , such as self-driving and robotic surgery . In contrast with standard and risk-neutral RL , it optimizes some risk measure of cumulative rewards instead of their expectation . One foundational framework for risk-sensitive RL maximizes the entropic risk measure of the reward , which takes the form of V π = 1 β log { Eπ [ eβR ] } , with respect to the policy π , where β 6= 0 is a given risk parameter and R denotes the cumulative rewards . Recently , the works of [ 20 , 21 ] investigate the online setting of the above risk-sensitive RL problem . Under K-episode MDPs with horizon length of H , they propose two model-free algorithms , namely RSVI and RSQ , and prove that their algorithms achieve the regret upper bound ( with its informal form given by ) Regret ( K ) . e|β|H 2 · e |β|H − 1 |β|H √ poly ( H ) ·K without assuming knowledge of the transition distribution or access to a simulator . They also provide a lower bound ( informally presented as ) Regret ( K ) & e|β|H ′ − 1 |β|H √ poly ( H ) ·K that any algorithm has to incur , where H ′ is a linear function of H . Despite the non-asymptotic nature of their results , it is not hard to see that a wide gap exists between the two bounds . Specifically , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the upper bound has an additional e|β|H 2 factor compared to the lower bound , and even worse , this factor is dominating in the upper bound since the quadratic exponent in e|β|H 2 makes it exponentially larger than e |β|H−1 |β|H even for moderate values of |β| and H . It is unclear whether the factor of e |β|H2 is intrinsic in the upper bound . In this paper , we show that the additional factor in the upper bound is not intrinsic for the upper bound and can be eliminated by a refined algorithmic design and analysis . We identify two deficiencies in the existing algorithms and their analysis : ( 1 ) the main element of the analysis follows existing analysis of risk-neutral RL algorithms , which fails to exploit the special structure of the Bellman equations of risk-sensitive RL ; ( 2 ) the existing algorithms use an excessively large bonus that results in the exponential blow-up in the regret upper bound . To address the above shortcomings , we consider a simple transformation of the Bellman equations analyzed so far in the literature , which we call the exponential Bellman equation . A distinctive feature of the exponential Bellman equation is that they associate the instantaneous reward and value function of the next step in a multiplicative way , rather than in an additive way as in the standard Bellman equations . From the exponential Bellman equation , we develop a novel analysis of the Bellman backup procedure for risk-sensitive RL algorithms that are based on the principle of optimism . The analysis further motivates a novel exploration mechanism called doubly decaying bonus , which helps the algorithms adapt to their estimation error over each horizon step while at the same time exploring efficiently . These discoveries enable us to propose two model-free algorithms for RL with the entropic risk measure based on the novel bonus . By combining the new analysis and bonus design , we prove that the preceding algorithms attain nearly optimal regret bounds under episodic and finite-horizon MDPs . Compared to prior results , our regret bounds feature an exponential improvement with respect to the horizon length and risk parameter , removing the factor of e|β|H 2 from existing upper bounds . This significantly narrows the gap between upper bounds and the existing lower bound of regret . In summary , we make the following theoretical contributions in this paper . 1 . We investigate the gap between existing upper and lower regret bounds in the context of risk-sensitive RL , and identify deficiencies of the existing algorithms and analysis ; 2 . We consider the exponential Bellman equation , which inspires us to propose a novel analysis of the Bellman backup procedure for RL algorithms based on the entropic risk measure . It further motivates a novel bonus design called doubly decaying bonus . We then design two model-free risk-sensitive RL algorithms equipped with the novel bonus . 3 . The novel analytic framework and bonus design together enable us to prove that the preceding algorithms achieve nearly optimal regret bounds , which improve upon existing ones by an exponential factor in terms of the horizon length and risk sensitivity . 2 Related works . The problem of RL with respect to the entropic risk measure is first proposed by the classical work of [ 24 ] , and has since inspired a large body of studies [ 2 , 4–8 , 13 , 16–18 , 22 , 23 , 25 , 26 , 31 , 33 , 37 , 38 , 40 , 41 , 43 ] . However , the algorithms from this line of works require knowledge of the transition kernel or assume access to a simulator of the underlying environment . Theoretical properties of these algorithms are investigated based on these assumptions , but the results are mostly of asymptotic nature , which do not shed light on their dependency on key parameters of the environment and agent . The work of [ 20 ] represents the first effort to investigate the setting where transitions are unknown and simulators of the environment are unavailable . It establishes the first non-asymptotic regret or sample complexity guarantees under the tabular setting . Building upon [ 20 ] , the authors of [ 21 ] extend the results to the function approximation setting , by considering linear and general function approximations of the underlying MDPs . Nevertheless , as discussed in Section 1 , both works leave open an exponential gap between the regret upper and lower bounds , which the present work aims to address via novel algorithms and analysis motivated by the exponential Bellman equation . We remark that although the exponential Bellman equation has been previously investigated in the literature of risk-sensitive RL [ 2 , 5 ] , this is the first time that it is explored for deriving regret and sample complexity guarantees of risk-sensitive RL algorithms . In Appendix A , we also make connections between risk-sensitive RL and distributional RL through the exponential Bellman equation . Notations . For a positive integer n , we let [ n ] : = { 1 , 2 , . . . , n } . For two non-negative sequences { ai } and { bi } , we write ai . bi if there exists a universal constant C > 0 such that ai ≤ Cbi for all i , and write ai bi if ai . bi and bi . ai . We use Õ ( · ) to denote O ( · ) while hiding logarithmic factors . For functions f , g : U → R , where U denotes their domain , we write f ≥ g if f ( u ) ≥ g ( u ) for any u ∈ U . We denote by I { · } the indicator function . 3 Problem background . 3.1 Episodic and finite-horizon MDP . The setting of episodic Markov decision processes can be denoted by MDP ( S , A , H , P , R ) , where S is the set of states , A is the set of actions , H ∈ Z > 0 is the length of each episode , and P = { Ph } h∈ [ H ] and R = { rh } h∈ [ H ] are the sets of transition kernels and reward functions , respectively . We let S : = |S| and A : = |A| , and we assume S , A < ∞ . We let Ph ( · | s , a ) denote the probability distribution over successor states of step h+ 1 if action a is executed in state s at step h. We assume that the reward function rh : S ×A → [ 0 , 1 ] is deterministic . We also assume that both P andR are unknown to learning agents . Under the setting of an episodic MDP , the agent aims to learn the optimal policy by interacting with the environment throughout K > 0 episodes , described as follows . At the beginning of episode k , an initial state sk1 is selected by the environment and we assume s k 1 stays the same for all k ∈ [ K ] . In each step h ∈ [ H ] of episode k , the agent observes state skh ∈ S , executes an action akh ∈ A , and receives a reward equal to rh ( skh , a k h ) from the environment . The MDP then transitions into state skh+1 randomly drawn from the transition kernel Ph ( · | skh , akh ) . The episode terminates at step H + 1 , in which the agent does not take actions or receive rewards . We define a policy π = { πh } h∈ [ H ] as a collection of functions πh : S → A , where πh ( s ) is the action that the agent takes in state s at step h of the episode . 3.2 Risk-sensitive RL . For each h ∈ [ H ] , we define the value function V πh : S → R of a policy π as the cumulative utility of the agent at state s of step h under the entropic risk measure , assuming that the agent commits to policy π in later steps . Specifically , we define ∀ ( h , s ) ∈ [ H ] × S , V πh ( s ) : = 1 β log { E [ eβ ∑H i=h ri ( si , πi ( si ) ) ∣∣∣ sh = s ] } , ( 1 ) where β 6= 0 is a given risk parameter . The agent aims to maximize his cumulative utility in step 1 , that is , to find a policy π such that V π1 ( s ) is maximized for all state s ∈ S. Under this setting , if β > 0 , the agent is risk-seeking and if β < 0 , the agent is risk-averse . Furthermore , as β → 0 the agent tends to be risk-neutral and V πh ( s ) tends to the classical value function . We may also define the action-value function Qπh : S ×A → R , which is the cumulative utility of the agent who follows policy π , conditional on a particular state-action pair ; formally , this is given by ∀ ( h , s , a ) ∈ [ H ] × S ×A , Qπh ( s , a ) : = 1 β log { E [ eβ ∑H i=h ri ( si , ai ) ∣∣∣ sh = s , ah = a ] } , ( 2 ) Under some mild regularity conditions [ 2 ] , there always exists an optimal policy , which we denote as π∗ , that yields the optimal value V ∗h ( s ) : = supπ V π h ( s ) for all ( h , s ) ∈ [ H ] × S . Bellman equations . For all ( s , a ) ∈ S × A , the Bellman equation associated with a policy π is given by Qπh ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ eβ·V π h+1 ( s ′ ) ] } , ( 3 ) V πh ( s ) = Q π h ( s , π ( s ) ) , V π H+1 ( s ) = 0 for h ∈ [ H ] . In Equation ( 3 ) , it can be seen that the action value Qπh of step h is a non-linear function of the value function V πh+1 of the later step . This is in contrast with the linear Bellman equations in the risk-neutral setting ( β → 0 ) , where Qπh ( s , a ) = rh ( s , a ) +Es′ [ V πh+1 ( s′ ) ] . Based on Equation ( 3 ) , for h ∈ [ H ] , the Bellman optimality equation is given by Q∗h ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ eβ·V ∗ h+1 ( s ′ ) ] } , ( 4 ) V ∗h ( s ) = max a∈A Q∗h ( s , a ) , V ∗ H+1 ( s ) = 0 . Exponential Bellman equation . We introduce the exponential Bellman equation , which is an exponential transformation of Equations ( 3 ) and ( 4 ) ( by taking exponential on both sides ) : for any policy π and tuple ( h , s , a ) , we have eβ·Q π h ( s , a ) = Es′∼Ph ( · | s , a ) [ eβ ( rh ( s , a ) +V π h+1 ( s ′ ) ) ] . ( 5 ) When π = π∗ , we obtain the corresponding optimality equation eβ·Q ∗ h ( s , a ) = Es′∼Ph ( · | s , a ) [ eβ ( rh ( s , a ) +V ∗ h+1 ( s ′ ) ) ] . ( 6 ) Note that Equation ( 5 ) associates the current and future cumulative utilities ( Qπh and V π h+1 ) in a multiplicative way . An implication of Equation ( 5 ) is that one may estimate eβ·Q π h ( s , a ) by a quantity of the form wh ( s , a ) = SampAvg ( { eβ ( rh ( sh , ah ) +Vh+1 ( sh+1 ) ) : ( sh , ah ) = ( s , a ) } ) ( 7 ) given some estimate of the value function Vh+1 . Here , we denote by SampAvg ( X ) the sample average computed over elements in the set X throughout past episodes , and it can be seen as an empirical MGF of cumulative rewards from step h + 1 . Equation ( 5 ) also suggests the following policy improvement procedure for a risk-sensitive policy π : πh ( s ) ← argmax a′∈A Qh ( s , a ′ ) = { argmaxa′∈A e β·Qh ( s , a′ ) , if β > 0 argmina′∈A e β·Qh ( s , a′ ) , if β < 0 , ( 8 ) where Qh denotes some estimated action-value function , possibly obtained from the quantity wh . In the next section , we will discuss how the exponential Bellman equation ( 5 ) inspires the development of a novel analytic framework for risk-sensitive RL . Before proceeding , we introduce a performance metric for the agent . For each episode k , recall that sk1 is the initial state chosen by the environment and let πk be the policy of the agent at the beginning of episode k. Then the difference V ∗1 ( s k 1 ) −V π k 1 ( s k 1 ) is called the regret of the agent in episode k. Therefore , after K episodes , the total regret for the agent is given by Regret ( K ) : = ∑ k∈ [ K ] [ V ∗1 ( s k 1 ) − V π k 1 ( s k 1 ) ] , ( 9 ) which serves as the key performance metric studied in this paper .

Summary: In this work a theoretical analysis is conducted in the context of risk-sensitive reinforcement learning with entropic risk measures. This allows connecting distributional RL and risk-sensitive RL. Two algorithms are proposed with a novel exploration scheme based on two bonuses. Regrets bounds are provided that improve previous analyses by an exponential term.