Delayed Geometric Discounts: An alternative criterion for Reinforcement Learning

1 INTRODUCTION . In the infinite horizon setting , and without further assumptions on the underlying Markov Decision Process ( MDP ) , available RL algorithms learn optimal policies only in the sense of the discounted cumulative rewards ( Puterman , 2014 ) . While the geometric discounting is well suited to model a termination probability or an exponentially decaying interest in the future , it is not flexible enough to model alternative weighting of the returns . Consider for example settings where the agent is willing to sacrifice short term rewards in favor of the long term outcome . Clearly , for such situations , a discounted optimality criterion is limited and does not describe the actual objective function . This is particularly true in hard exploration tasks , where rewards can be sparse , deceptive or adversarial . In such scenarios , performing random exploration can rarely lead to successful states and thus rarely obtain meaningful feedback . Geometric discounts tend to fail in such scenarios . Consider for example the U-maze environment in Figure 1a , where the reinforcement signal provides a high reward ( +1 ) when reaching the green dot in the bottom arm , a deceptive reward ( +0.9 ) when reaching the blue dot in the upper arm , and a negative reward ( −1 ) for crossing the red corridor . If the agent is only interested in the long term returns , then the optimal control should always lead to the green dot . However , depending on the initial state , optimal policies in the sense of the discounted RL problem are likely to prefer the deceptive reward due to the exponentially decaying interest in the future ( Figure 1b ) . Naturally , higher discount factors are associated with optimal policies that also optimize the average returns ( Blackwell , 1962 ) , which can solve in principle the described hard exploration problem . However , in practice , such discount values can be arbitrarily close to 1 which entails severe computational instabilities . In addition , and particularly in continuous settings or when tasks span over extremely long episodes , discount-based RL approaches are sample-inefficient and are slow at propagating interesting feed-backs to early states . In this paper , we generalize the geometric discount to derive a variety of alternative time weighting distributions and we investigate the underlying implications of solving the associated RL problem both theoretically and practically . Our contributions are twofold . First , we introduce a novel family that generalizes the geometrically discounted criteria , which we call the delayed discounted criteria . Second , we derive tractable solutions for optimal control for both stationary and non-stationary policies using these novel criteria . Finally , we evaluate our methods on both hard exploration tabular environments and continuous long-episodic robotics tasks where we show that : 1 . Our agents can solve the hard exploration problem in a proof of concept setup . 2 . Our methods improve sample-efficiency on continuous robotics tasks compared to Soft- Actor-Critic . Figure 1c showcases how non-geometrically discounted criteria impacts the profile of optimal value function in the U-maze example . 2 REINFORCEMENT LEARNING WITH NON GEOMETRIC DISCOUNT . Consider an infinite horizon Markov Decision Process ( MDP ) M = { S , A , P , r , γ , p0 } , where S and A are either finite or compact subsets of respectively Rd and Rd′ , ( d , d′ ) ∈ N2 , P : S × A → ∆ ( S ) is a state transition kernel1 , r : S × A → R is a continuous reward function , p0 ∈ ∆ ( S ) an initial state distribution and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a mapping indicating , at each time step t ∈ N , the action at to be chosen at the current state st . The goal of geometrically discounted reinforcement learning algorithms is to optimize the discounted returns : L ( π , r ) : = Eπ , p0 [ ∞∑ t=0 γtrt ] ; rt : = r ( st , at ) where Eπ , p0 denotes the expectation over trajectories generated in M using the policy π and initialised according to p0 . In this section , we present our methods . First , Section 2.1 introduces our delayed discounted family of optimality criteria . Then , Section 2.2 investigates the optimization of the linear combination of this new family in the context of stationary policies . Finally , Section 2.3 generalizes the optimal control to non-stationary policies . 2.1 BEYOND THE GEOMETRIC DISCOUNT : A DELAYED DISCOUNTED CRITERION . We propose to investigate a particular parametric family of optimality criteria that is defined by a sequence of discount factors . For a given delay parameter D ∈ N , we define the discount factors γd ∈ ( 0 , 1 ) for any integer d ∈ [ 0 , D ] and we consider the following loss function : LD ( π , r ) : = Eπ , p0 [ ∞∑ t=0 ΦD ( t ) rt ] where ΦD ( t ) : = ∑ { ad∈N } Dd=0 such that ∑ d ad=t N∏ d=0 γadd ( 1 ) This class of optimality criteria can be seen as a generalization of the classical geometric discount . In fact , we highlight that for D = 0 , Φ0 ( t ) = γt0 which implies that L0 ( π , r ) = L ( π , r ) for any policy π and any reward function r. In Figure 2 , we report the normalized distribution of the weights ΦD ( t ) ( i.e . y ( t ) = ΦD ( t ) ∑ i ΦD ( i ) ) over time as we vary the parameter D ∈ { 0 , . . . , 9 } . Notice how the mode of the probability distribution is shifted towards the right as we increased the delay , thus putting more weights on future time steps . 1∆ ( S ) denotes the set of probability measures over S Intuitively , the proposed criterion ( LD ) describes the goal of an agent that discards short term gains in favor of long term discounted returns . In the following , we consider yet a more diverse problem formulation by using a linear combination of the delayed losses . Let Lη be the following objective function defined as : Lη ( π , r ) : = Eπ , p0 [ ∑ t η ( t ) rt ] such that η ( t ) = D∑ d=0 wdΦd ( t ) ( 2 ) where the depth D ∈ N and the coefficients wd ∈ R for any d ∈ [ 0 , D ] are known . In general , the optimal control in the sense of Lη is not stationary . However , learning solutions that admit compact representations is crucial for obvious computational reasons . In the following we propose to learn either the optimal stationary solution , or to approximate the optimal control with a non-stationary policy over a finite horizon followed by a stationary one . 2.2 OPTIMAL STATIONARY POLICIES . In this section , we propose to learn stationary solution using a policy-iteration like algorithmic scheme . As in the classical setting , the goal is to learn a control π∗ that maximizes the state-action value function Qπη : Qπη ( s , a ) = D∑ d=0 wdQ π d ( s , a ) where Q π d ( s , a ) : = Eπ [ ∞∑ t=0 Φd ( t ) rt|s0 , a0 = s , a ] ( 3 ) This is done by iteratively evaluatingQπη and then updating the policy to maximize the learned value function . Due to the linearity illustrated in Equation 3 , the policy evaluation step is reduced to the estimation of Qπd . In geometrically discounted setting , the value function is the fixed point of the Bellman optimality operator . Luckily , this property is also valid for the quantities Qπd : Proposition 1 For any discount parameters ( γd ) Dd=0 , the value functions QπD is the unique fixed point of the following γD-contraction : [ TDπ ( q ) ] ( s , a ) = Es′∼P ( s , a ) a′∼π ( s′ ) [ r ( s , a ) + D−1∑ d=0 γdQ π d ( s ′ , a′ ) ] ︸ ︷︷ ︸ : = rπD ( s , a ) +γDEs′∼P ( s , a ) a′∼π ( s′ ) [ q ( s′ , a′ ) ] . ( 4 ) The valueQπD of a policy π with respect to the delayed criterionLD is the state-action value function of the same policy using an augmented policy dependent reward rπD w.r.t . the γD-discounted returns . Intuitively , the instantaneous worth of an action ( rπD ) is the sum of the environments ’ myopic returns ( r ( s , a ) ) and the long term evaluations ( with lower delay parameters ( Qπd ) d < D ) . This has the beneficial side-effect of enhancing sample efficiency as it helps the agent to rapidly back-propagate long-term feed-backs to early states . This reduces the time needed to distinguish good from bad behaviors , particularly in continuous settings where function approximation are typically used to learn policies . This is discussed in details in Section 4.2 . Similarly to standard value based RL algorithms , given a data set of trajectories D = { s , a , s′ } , the value function Qπd can be approximated with parametric approximator Qθd by optimising J Q d ( θ ) : JQD ( θ ) = E s , a , s′∼D a′∼πφ ( s′ ) [ 1 2 ( Qθ − ( r ( s , a ) + D∑ d=0 γdQθ̄d ( s ′ , a′ ) ) 2 ) ] ( 5 ) As for the policy update step , inspired from the Soft-Actor-Critic ( SAC ) algorithm , we propose to optimize an entropy regularized soft Q-value using the following loss where α is a parameter : Jπη ( φ ) = −Es∼D , a∼πφ [ D∑ d=0 wdQθ̄d ( s , a ) − α log ( πφ ( a|s ) ) ] ( 6 ) We use Equations 5 and 6 to construct Algorithm 1 , a generalization of the SAC algorithm that approximates optimal stationary policies in the sense of Lη . In practice , this can be further improved using the double Q-network trick and the automatic tuning of the regularization parameter α . This is discussed in Appendix A.1 . Unfortunately , unlike the geometrically discounted setting , the policy improvement theorem is no longer guaranteed in the sense of Lη . This means that depending on the initialization parameters , Algorithm 1 can either converge to the optimal stationary control or get stuck in a loop of sub-optimal policies . This is discussed in detail in Section 4.1 . Algorithm 1 Generalized Soft Actor Critic 1 : Input : initial parameters ( θd ) Dd=0 , φ , learning rates ( λd ) Dd=0 , λπ , pollyak parameter τ 2 : initialise target network θ̄d ← θd and initialise replay buffer D ← ∅ 3 : for each iteration do 4 : for each environment step do 5 : at ∼ πφ ( st ) , st+1 ∼ P ( st , at ) , D ← D ∪ { ( st , at , st+1 , ct ) } 6 : for d ∈ [ 0 , D ] do 7 : for each Qd gradient step do 8 : update parameter θd ← θd − λd∇̂θdJ Q d ( θd ) and update target θ̄d ← τ θ̄d + ( 1− τ ) θd 9 : for each policy update do 10 : update policy φ← φ− λπ∇̂φJπη ( φ ) 11 : Return : πφ , ( Qθ̄d ) D d=0

Summary: **Summary.** This paper proposes a generalization to the typical geometric discounting scheme in reinforcement learning. In particular, a new _delayed_ discount criterion is proposed (Equation 1) that captures traditional geometric discounting when $D=0$, but also allows for other kinds of temporal preferences. Figure 2 provides a clean visual illustrating the different kinds of discounting afforded for different settings of $D$. Following this, several natural questions emerge: What happens to the $\gamma$-contraction in repeated application of the Bellman Operator? Proposition 1 addresses this question by stating that a $\gamma_D$-contraction is still guaranteed. How might typical reinforcement learning algorithms handle this new discounting scheme? The paper addresses this question by introducing two new algorithms. The first (Algorithm 1) is a generalization of Soft Actor Critic, while the second (Algorithm 2) is a policy-based algorithm that approximates the finite horizon problem. The first set of experiments focus on three maze problems. Estimates of the expected returns of learned stationary policies are reported for Algorithm 1 under four conditions as the depth $D$ is changed from 0 up to 8. The main finding is that as $D$, returns do tend to increase up to a certain point, at which point it drops (around when $D = 6$ or $7$). Further experiments inspect the performance of Algorithm 2 in a similar way (Figure 5), and contrast the performance of Algorithm 1 with typical SAC on continuous control games like Ant-v2. On the chosen four games (Figure 6), the average reward of GSAC improves over SAC slightly, though the variance is arguably slightly higher.