Delayed Geometric Discounts: An alternative criterion for Reinforcement Learning

1 INTRODUCTION . In the infinite horizon setting , and without further assumptions on the underlying Markov Decision Process ( MDP ) , available RL algorithms learn optimal policies only in the sense of the discounted cumulative rewards ( Puterman , 2014 ) . While the geometric discounting is well suited to model a termination probability or an exponentially decaying interest in the future , it is not flexible enough to model alternative weighting of the returns . Consider for example settings where the agent is willing to sacrifice short term rewards in favor of the long term outcome . Clearly , for such situations , a discounted optimality criterion is limited and does not describe the actual objective function . This is particularly true in hard exploration tasks , where rewards can be sparse , deceptive or adversarial . In such scenarios , performing random exploration can rarely lead to successful states and thus rarely obtain meaningful feedback . Geometric discounts tend to fail in such scenarios . Consider for example the U-maze environment in Figure 1a , where the reinforcement signal provides a high reward ( +1 ) when reaching the green dot in the bottom arm , a deceptive reward ( +0.9 ) when reaching the blue dot in the upper arm , and a negative reward ( −1 ) for crossing the red corridor . If the agent is only interested in the long term returns , then the optimal control should always lead to the green dot . However , depending on the initial state , optimal policies in the sense of the discounted RL problem are likely to prefer the deceptive reward due to the exponentially decaying interest in the future ( Figure 1b ) . Naturally , higher discount factors are associated with optimal policies that also optimize the average returns ( Blackwell , 1962 ) , which can solve in principle the described hard exploration problem . However , in practice , such discount values can be arbitrarily close to 1 which entails severe computational instabilities . In addition , and particularly in continuous settings or when tasks span over extremely long episodes , discount-based RL approaches are sample-inefficient and are slow at propagating interesting feed-backs to early states . In this paper , we generalize the geometric discount to derive a variety of alternative time weighting distributions and we investigate the underlying implications of solving the associated RL problem both theoretically and practically . Our contributions are twofold . First , we introduce a novel family that generalizes the geometrically discounted criteria , which we call the delayed discounted criteria . Second , we derive tractable solutions for optimal control for both stationary and non-stationary policies using these novel criteria . Finally , we evaluate our methods on both hard exploration tabular environments and continuous long-episodic robotics tasks where we show that : 1 . Our agents can solve the hard exploration problem in a proof of concept setup . 2 . Our methods improve sample-efficiency on continuous robotics tasks compared to Soft- Actor-Critic . Figure 1c showcases how non-geometrically discounted criteria impacts the profile of optimal value function in the U-maze example . 2 REINFORCEMENT LEARNING WITH NON GEOMETRIC DISCOUNT . Consider an infinite horizon Markov Decision Process ( MDP ) M = { S , A , P , r , γ , p0 } , where S and A are either finite or compact subsets of respectively Rd and Rd′ , ( d , d′ ) ∈ N2 , P : S × A → ∆ ( S ) is a state transition kernel1 , r : S × A → R is a continuous reward function , p0 ∈ ∆ ( S ) an initial state distribution and γ ∈ ( 0 , 1 ) is the discount factor . A policy π is a mapping indicating , at each time step t ∈ N , the action at to be chosen at the current state st . The goal of geometrically discounted reinforcement learning algorithms is to optimize the discounted returns : L ( π , r ) : = Eπ , p0 [ ∞∑ t=0 γtrt ] ; rt : = r ( st , at ) where Eπ , p0 denotes the expectation over trajectories generated in M using the policy π and initialised according to p0 . In this section , we present our methods . First , Section 2.1 introduces our delayed discounted family of optimality criteria . Then , Section 2.2 investigates the optimization of the linear combination of this new family in the context of stationary policies . Finally , Section 2.3 generalizes the optimal control to non-stationary policies . 2.1 BEYOND THE GEOMETRIC DISCOUNT : A DELAYED DISCOUNTED CRITERION . We propose to investigate a particular parametric family of optimality criteria that is defined by a sequence of discount factors . For a given delay parameter D ∈ N , we define the discount factors γd ∈ ( 0 , 1 ) for any integer d ∈ [ 0 , D ] and we consider the following loss function : LD ( π , r ) : = Eπ , p0 [ ∞∑ t=0 ΦD ( t ) rt ] where ΦD ( t ) : = ∑ { ad∈N } Dd=0 such that ∑ d ad=t N∏ d=0 γadd ( 1 ) This class of optimality criteria can be seen as a generalization of the classical geometric discount . In fact , we highlight that for D = 0 , Φ0 ( t ) = γt0 which implies that L0 ( π , r ) = L ( π , r ) for any policy π and any reward function r. In Figure 2 , we report the normalized distribution of the weights ΦD ( t ) ( i.e . y ( t ) = ΦD ( t ) ∑ i ΦD ( i ) ) over time as we vary the parameter D ∈ { 0 , . . . , 9 } . Notice how the mode of the probability distribution is shifted towards the right as we increased the delay , thus putting more weights on future time steps . 1∆ ( S ) denotes the set of probability measures over S Intuitively , the proposed criterion ( LD ) describes the goal of an agent that discards short term gains in favor of long term discounted returns . In the following , we consider yet a more diverse problem formulation by using a linear combination of the delayed losses . Let Lη be the following objective function defined as : Lη ( π , r ) : = Eπ , p0 [ ∑ t η ( t ) rt ] such that η ( t ) = D∑ d=0 wdΦd ( t ) ( 2 ) where the depth D ∈ N and the coefficients wd ∈ R for any d ∈ [ 0 , D ] are known . In general , the optimal control in the sense of Lη is not stationary . However , learning solutions that admit compact representations is crucial for obvious computational reasons . In the following we propose to learn either the optimal stationary solution , or to approximate the optimal control with a non-stationary policy over a finite horizon followed by a stationary one . 2.2 OPTIMAL STATIONARY POLICIES . In this section , we propose to learn stationary solution using a policy-iteration like algorithmic scheme . As in the classical setting , the goal is to learn a control π∗ that maximizes the state-action value function Qπη : Qπη ( s , a ) = D∑ d=0 wdQ π d ( s , a ) where Q π d ( s , a ) : = Eπ [ ∞∑ t=0 Φd ( t ) rt|s0 , a0 = s , a ] ( 3 ) This is done by iteratively evaluatingQπη and then updating the policy to maximize the learned value function . Due to the linearity illustrated in Equation 3 , the policy evaluation step is reduced to the estimation of Qπd . In geometrically discounted setting , the value function is the fixed point of the Bellman optimality operator . Luckily , this property is also valid for the quantities Qπd : Proposition 1 For any discount parameters ( γd ) Dd=0 , the value functions QπD is the unique fixed point of the following γD-contraction : [ TDπ ( q ) ] ( s , a ) = Es′∼P ( s , a ) a′∼π ( s′ ) [ r ( s , a ) + D−1∑ d=0 γdQ π d ( s ′ , a′ ) ] ︸ ︷︷ ︸ : = rπD ( s , a ) +γDEs′∼P ( s , a ) a′∼π ( s′ ) [ q ( s′ , a′ ) ] . ( 4 ) The valueQπD of a policy π with respect to the delayed criterionLD is the state-action value function of the same policy using an augmented policy dependent reward rπD w.r.t . the γD-discounted returns . Intuitively , the instantaneous worth of an action ( rπD ) is the sum of the environments ’ myopic returns ( r ( s , a ) ) and the long term evaluations ( with lower delay parameters ( Qπd ) d < D ) . This has the beneficial side-effect of enhancing sample efficiency as it helps the agent to rapidly back-propagate long-term feed-backs to early states . This reduces the time needed to distinguish good from bad behaviors , particularly in continuous settings where function approximation are typically used to learn policies . This is discussed in details in Section 4.2 . Similarly to standard value based RL algorithms , given a data set of trajectories D = { s , a , s′ } , the value function Qπd can be approximated with parametric approximator Qθd by optimising J Q d ( θ ) : JQD ( θ ) = E s , a , s′∼D a′∼πφ ( s′ ) [ 1 2 ( Qθ − ( r ( s , a ) + D∑ d=0 γdQθ̄d ( s ′ , a′ ) ) 2 ) ] ( 5 ) As for the policy update step , inspired from the Soft-Actor-Critic ( SAC ) algorithm , we propose to optimize an entropy regularized soft Q-value using the following loss where α is a parameter : Jπη ( φ ) = −Es∼D , a∼πφ [ D∑ d=0 wdQθ̄d ( s , a ) − α log ( πφ ( a|s ) ) ] ( 6 ) We use Equations 5 and 6 to construct Algorithm 1 , a generalization of the SAC algorithm that approximates optimal stationary policies in the sense of Lη . In practice , this can be further improved using the double Q-network trick and the automatic tuning of the regularization parameter α . This is discussed in Appendix A.1 . Unfortunately , unlike the geometrically discounted setting , the policy improvement theorem is no longer guaranteed in the sense of Lη . This means that depending on the initialization parameters , Algorithm 1 can either converge to the optimal stationary control or get stuck in a loop of sub-optimal policies . This is discussed in detail in Section 4.1 . Algorithm 1 Generalized Soft Actor Critic 1 : Input : initial parameters ( θd ) Dd=0 , φ , learning rates ( λd ) Dd=0 , λπ , pollyak parameter τ 2 : initialise target network θ̄d ← θd and initialise replay buffer D ← ∅ 3 : for each iteration do 4 : for each environment step do 5 : at ∼ πφ ( st ) , st+1 ∼ P ( st , at ) , D ← D ∪ { ( st , at , st+1 , ct ) } 6 : for d ∈ [ 0 , D ] do 7 : for each Qd gradient step do 8 : update parameter θd ← θd − λd∇̂θdJ Q d ( θd ) and update target θ̄d ← τ θ̄d + ( 1− τ ) θd 9 : for each policy update do 10 : update policy φ← φ− λπ∇̂φJπη ( φ ) 11 : Return : πφ , ( Qθ̄d ) D d=0

Summary: The paper proposes an optimality criterion for RL that is generalizes the standard exponential discount commonly used in a large volume of RL research. In particular, the paper proposes substituting the standard discount term $\gamma^t$ by a generalized discount term $\Phi_D(t)$ built from a set of discounts $\gamma_0,\ldots,\gamma_D$. The discount terms $\Phi_D$ can be defined recursively as   - $\Phi_0(t)=\gamma_0^t$ (corresponding to the standard exponential discounting); - $\Phi_D(t)=\sum_{k=0}^t\gamma_D^k\Phi_{D-1}(t-k)$  The consideration of non-uniform discounting implies that the resulting optimal policies may, in general, be non-stationary. Nevertheless, the paper introduces an equivalent to the $Q$-functions in the standard RL formulation and shows that these functions can be computed using standard dynamic programming. Letting  $Q_d^\pi(s,a)=E\left[\sum_{t=0}^\infty\Phi_d(t)r_t\mid s_0=s,a_0=a\right]$,  it follows that $Q_0^\pi$ corresponds to the standard $Q^\pi$ with discount factor $\gamma_0$, and each $Q_d^\pi$ can be computed from $Q_{d-1}^\pi$ using a dynamic programming operator $T_\pi^d$ that has $Q_d^\pi$ as its unique fixed point. These functions can also be computed using existing RL algorithms as part of a generalized policy iteration cycle that---upon convergence---may yield an "optimal stationary policy" (convergence is not guaranteed, though).  The paper also proposes an approximate policy that breaks down the value function for the problem into two terms---one that accounts for the rewards in the first $H$ steps and the other for the remaining steps. Under mild conditions on the discounts $\gamma_0,\ldots,\gamma_D$, the latter term can be approximated by the standard optimal value function (with standard exponential discounting) for large $H$, allowing for the computation of an approximate (non-stationary) policy.