Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning

1 Introduction . Risk-sensitive reinforcement learning ( RL ) is important for practical and high-stake applications , such as self-driving and robotic surgery . In contrast with standard and risk-neutral RL , it optimizes some risk measure of cumulative rewards instead of their expectation . One foundational framework for risk-sensitive RL maximizes the entropic risk measure of the reward , which takes the form of V π = 1 β log { Eπ [ eβR ] } , with respect to the policy π , where β 6= 0 is a given risk parameter and R denotes the cumulative rewards . Recently , the works of [ 20 , 21 ] investigate the online setting of the above risk-sensitive RL problem . Under K-episode MDPs with horizon length of H , they propose two model-free algorithms , namely RSVI and RSQ , and prove that their algorithms achieve the regret upper bound ( with its informal form given by ) Regret ( K ) . e|β|H 2 · e |β|H − 1 |β|H √ poly ( H ) ·K without assuming knowledge of the transition distribution or access to a simulator . They also provide a lower bound ( informally presented as ) Regret ( K ) & e|β|H ′ − 1 |β|H √ poly ( H ) ·K that any algorithm has to incur , where H ′ is a linear function of H . Despite the non-asymptotic nature of their results , it is not hard to see that a wide gap exists between the two bounds . Specifically , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the upper bound has an additional e|β|H 2 factor compared to the lower bound , and even worse , this factor is dominating in the upper bound since the quadratic exponent in e|β|H 2 makes it exponentially larger than e |β|H−1 |β|H even for moderate values of |β| and H . It is unclear whether the factor of e |β|H2 is intrinsic in the upper bound . In this paper , we show that the additional factor in the upper bound is not intrinsic for the upper bound and can be eliminated by a refined algorithmic design and analysis . We identify two deficiencies in the existing algorithms and their analysis : ( 1 ) the main element of the analysis follows existing analysis of risk-neutral RL algorithms , which fails to exploit the special structure of the Bellman equations of risk-sensitive RL ; ( 2 ) the existing algorithms use an excessively large bonus that results in the exponential blow-up in the regret upper bound . To address the above shortcomings , we consider a simple transformation of the Bellman equations analyzed so far in the literature , which we call the exponential Bellman equation . A distinctive feature of the exponential Bellman equation is that they associate the instantaneous reward and value function of the next step in a multiplicative way , rather than in an additive way as in the standard Bellman equations . From the exponential Bellman equation , we develop a novel analysis of the Bellman backup procedure for risk-sensitive RL algorithms that are based on the principle of optimism . The analysis further motivates a novel exploration mechanism called doubly decaying bonus , which helps the algorithms adapt to their estimation error over each horizon step while at the same time exploring efficiently . These discoveries enable us to propose two model-free algorithms for RL with the entropic risk measure based on the novel bonus . By combining the new analysis and bonus design , we prove that the preceding algorithms attain nearly optimal regret bounds under episodic and finite-horizon MDPs . Compared to prior results , our regret bounds feature an exponential improvement with respect to the horizon length and risk parameter , removing the factor of e|β|H 2 from existing upper bounds . This significantly narrows the gap between upper bounds and the existing lower bound of regret . In summary , we make the following theoretical contributions in this paper . 1 . We investigate the gap between existing upper and lower regret bounds in the context of risk-sensitive RL , and identify deficiencies of the existing algorithms and analysis ; 2 . We consider the exponential Bellman equation , which inspires us to propose a novel analysis of the Bellman backup procedure for RL algorithms based on the entropic risk measure . It further motivates a novel bonus design called doubly decaying bonus . We then design two model-free risk-sensitive RL algorithms equipped with the novel bonus . 3 . The novel analytic framework and bonus design together enable us to prove that the preceding algorithms achieve nearly optimal regret bounds , which improve upon existing ones by an exponential factor in terms of the horizon length and risk sensitivity . 2 Related works . The problem of RL with respect to the entropic risk measure is first proposed by the classical work of [ 24 ] , and has since inspired a large body of studies [ 2 , 4–8 , 13 , 16–18 , 22 , 23 , 25 , 26 , 31 , 33 , 37 , 38 , 40 , 41 , 43 ] . However , the algorithms from this line of works require knowledge of the transition kernel or assume access to a simulator of the underlying environment . Theoretical properties of these algorithms are investigated based on these assumptions , but the results are mostly of asymptotic nature , which do not shed light on their dependency on key parameters of the environment and agent . The work of [ 20 ] represents the first effort to investigate the setting where transitions are unknown and simulators of the environment are unavailable . It establishes the first non-asymptotic regret or sample complexity guarantees under the tabular setting . Building upon [ 20 ] , the authors of [ 21 ] extend the results to the function approximation setting , by considering linear and general function approximations of the underlying MDPs . Nevertheless , as discussed in Section 1 , both works leave open an exponential gap between the regret upper and lower bounds , which the present work aims to address via novel algorithms and analysis motivated by the exponential Bellman equation . We remark that although the exponential Bellman equation has been previously investigated in the literature of risk-sensitive RL [ 2 , 5 ] , this is the first time that it is explored for deriving regret and sample complexity guarantees of risk-sensitive RL algorithms . In Appendix A , we also make connections between risk-sensitive RL and distributional RL through the exponential Bellman equation . Notations . For a positive integer n , we let [ n ] : = { 1 , 2 , . . . , n } . For two non-negative sequences { ai } and { bi } , we write ai . bi if there exists a universal constant C > 0 such that ai ≤ Cbi for all i , and write ai bi if ai . bi and bi . ai . We use Õ ( · ) to denote O ( · ) while hiding logarithmic factors . For functions f , g : U → R , where U denotes their domain , we write f ≥ g if f ( u ) ≥ g ( u ) for any u ∈ U . We denote by I { · } the indicator function . 3 Problem background . 3.1 Episodic and finite-horizon MDP . The setting of episodic Markov decision processes can be denoted by MDP ( S , A , H , P , R ) , where S is the set of states , A is the set of actions , H ∈ Z > 0 is the length of each episode , and P = { Ph } h∈ [ H ] and R = { rh } h∈ [ H ] are the sets of transition kernels and reward functions , respectively . We let S : = |S| and A : = |A| , and we assume S , A < ∞ . We let Ph ( · | s , a ) denote the probability distribution over successor states of step h+ 1 if action a is executed in state s at step h. We assume that the reward function rh : S ×A → [ 0 , 1 ] is deterministic . We also assume that both P andR are unknown to learning agents . Under the setting of an episodic MDP , the agent aims to learn the optimal policy by interacting with the environment throughout K > 0 episodes , described as follows . At the beginning of episode k , an initial state sk1 is selected by the environment and we assume s k 1 stays the same for all k ∈ [ K ] . In each step h ∈ [ H ] of episode k , the agent observes state skh ∈ S , executes an action akh ∈ A , and receives a reward equal to rh ( skh , a k h ) from the environment . The MDP then transitions into state skh+1 randomly drawn from the transition kernel Ph ( · | skh , akh ) . The episode terminates at step H + 1 , in which the agent does not take actions or receive rewards . We define a policy π = { πh } h∈ [ H ] as a collection of functions πh : S → A , where πh ( s ) is the action that the agent takes in state s at step h of the episode . 3.2 Risk-sensitive RL . For each h ∈ [ H ] , we define the value function V πh : S → R of a policy π as the cumulative utility of the agent at state s of step h under the entropic risk measure , assuming that the agent commits to policy π in later steps . Specifically , we define ∀ ( h , s ) ∈ [ H ] × S , V πh ( s ) : = 1 β log { E [ eβ ∑H i=h ri ( si , πi ( si ) ) ∣∣∣ sh = s ] } , ( 1 ) where β 6= 0 is a given risk parameter . The agent aims to maximize his cumulative utility in step 1 , that is , to find a policy π such that V π1 ( s ) is maximized for all state s ∈ S. Under this setting , if β > 0 , the agent is risk-seeking and if β < 0 , the agent is risk-averse . Furthermore , as β → 0 the agent tends to be risk-neutral and V πh ( s ) tends to the classical value function . We may also define the action-value function Qπh : S ×A → R , which is the cumulative utility of the agent who follows policy π , conditional on a particular state-action pair ; formally , this is given by ∀ ( h , s , a ) ∈ [ H ] × S ×A , Qπh ( s , a ) : = 1 β log { E [ eβ ∑H i=h ri ( si , ai ) ∣∣∣ sh = s , ah = a ] } , ( 2 ) Under some mild regularity conditions [ 2 ] , there always exists an optimal policy , which we denote as π∗ , that yields the optimal value V ∗h ( s ) : = supπ V π h ( s ) for all ( h , s ) ∈ [ H ] × S . Bellman equations . For all ( s , a ) ∈ S × A , the Bellman equation associated with a policy π is given by Qπh ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ eβ·V π h+1 ( s ′ ) ] } , ( 3 ) V πh ( s ) = Q π h ( s , π ( s ) ) , V π H+1 ( s ) = 0 for h ∈ [ H ] . In Equation ( 3 ) , it can be seen that the action value Qπh of step h is a non-linear function of the value function V πh+1 of the later step . This is in contrast with the linear Bellman equations in the risk-neutral setting ( β → 0 ) , where Qπh ( s , a ) = rh ( s , a ) +Es′ [ V πh+1 ( s′ ) ] . Based on Equation ( 3 ) , for h ∈ [ H ] , the Bellman optimality equation is given by Q∗h ( s , a ) = rh ( s , a ) + 1 β log { Es′∼Ph ( · | s , a ) [ eβ·V ∗ h+1 ( s ′ ) ] } , ( 4 ) V ∗h ( s ) = max a∈A Q∗h ( s , a ) , V ∗ H+1 ( s ) = 0 . Exponential Bellman equation . We introduce the exponential Bellman equation , which is an exponential transformation of Equations ( 3 ) and ( 4 ) ( by taking exponential on both sides ) : for any policy π and tuple ( h , s , a ) , we have eβ·Q π h ( s , a ) = Es′∼Ph ( · | s , a ) [ eβ ( rh ( s , a ) +V π h+1 ( s ′ ) ) ] . ( 5 ) When π = π∗ , we obtain the corresponding optimality equation eβ·Q ∗ h ( s , a ) = Es′∼Ph ( · | s , a ) [ eβ ( rh ( s , a ) +V ∗ h+1 ( s ′ ) ) ] . ( 6 ) Note that Equation ( 5 ) associates the current and future cumulative utilities ( Qπh and V π h+1 ) in a multiplicative way . An implication of Equation ( 5 ) is that one may estimate eβ·Q π h ( s , a ) by a quantity of the form wh ( s , a ) = SampAvg ( { eβ ( rh ( sh , ah ) +Vh+1 ( sh+1 ) ) : ( sh , ah ) = ( s , a ) } ) ( 7 ) given some estimate of the value function Vh+1 . Here , we denote by SampAvg ( X ) the sample average computed over elements in the set X throughout past episodes , and it can be seen as an empirical MGF of cumulative rewards from step h + 1 . Equation ( 5 ) also suggests the following policy improvement procedure for a risk-sensitive policy π : πh ( s ) ← argmax a′∈A Qh ( s , a ′ ) = { argmaxa′∈A e β·Qh ( s , a′ ) , if β > 0 argmina′∈A e β·Qh ( s , a′ ) , if β < 0 , ( 8 ) where Qh denotes some estimated action-value function , possibly obtained from the quantity wh . In the next section , we will discuss how the exponential Bellman equation ( 5 ) inspires the development of a novel analytic framework for risk-sensitive RL . Before proceeding , we introduce a performance metric for the agent . For each episode k , recall that sk1 is the initial state chosen by the environment and let πk be the policy of the agent at the beginning of episode k. Then the difference V ∗1 ( s k 1 ) −V π k 1 ( s k 1 ) is called the regret of the agent in episode k. Therefore , after K episodes , the total regret for the agent is given by Regret ( K ) : = ∑ k∈ [ K ] [ V ∗1 ( s k 1 ) − V π k 1 ( s k 1 ) ] , ( 9 ) which serves as the key performance metric studied in this paper .

Summary: The paper studies the exponential Bellman equation that arises in risk sensitive reinforcement learning when using entropic risk measure as the risk measure. It claims to "offer a route for connecting distributional RL and risk sensitive RL", although this question is never seriously addressed. The main result of their analysis consists in design and analyse two novel RL algorithms that employ a novel exploration mechanism. The regret bounds on this mechanism improves by an exponential factor.