Gradient descent temporal difference-difference learning

1 INTRODUCTION . Off-policy algorithms for value function learning enable an agent to use a behavior policy that differs from the target policy in order to gain experience for learning . However , because off-policy methods learn a value function for a target policy given data due to a different behavior policy , they often exhibit greater variance in parameter updates . When applied to problems involving function approximation , off-policy methods are slower to converge than on-policy methods and may even diverge ( Baird , 1995 ; Sutton & Barto , 2018 ) . Two general approaches have been investigated to address the challenge of developing stable and effective off-policy temporal-difference algorithms . One approach is to use importance sampling methods to warp the update distribution back to the on-policy distribution ( Precup et al. , 2000 ; Mahmood et al. , 2014 ) . This approach is useful for decreasing the variance of parameter updates , but it does not address stability issues . The second main approach to addressing the challenge of off-policy learning is to develop true gradient descent-based methods that are guaranteed to be stable regardless of the update distribution . Sutton et al . ( 2009a ; b ) proposed the first off-policy gradientdescent-based temporal difference ( GTD and GTD2 , respectively ) algorithms . These algorithms are guaranteed to be stable , with computational complexity scaling linearly with the size of the function approximator . Empirically , however , their convergence is much slower than conventional temporal difference ( TD ) learning , limiting their practical utility ( Ghiassian et al. , 2020 ; White & White , 2016 ) . Building on this work , extensions to the GTD family of algorithms ( see Ghiassian et al . ( 2018 ) for a review ) have allowed for incorporating eligibility traces ( Maei & Sutton , 2010 ; Geist & Scherrer , 2014 ) , non-linear function approximation such as with a neural network ( Maei , 2011 ) , and reformulation of the optimization as a saddle point problem ( Liu et al. , 2015 ; Du et al. , 2017 ) . However , due to their slow convergence , none of these stable off-policy methods are commonly used in practice . In this work , we introduce a new gradient descent algorithm for temporal difference learning with linear value function approximation . This algorithm , which we call gradient descent temporal difference-difference ( Gradient-DD ) learning , is an acceleration technique that employs second- order differences in successive parameter updates . The basic idea of Gradient-DD is to modify the error objective function by additionally considering the prediction error obtained in last time step , then to derive a gradient-descent algorithm based on this modified objective function . In addition to exploiting the Bellman equation to get the solution , this modified error objective function avoids drastic changes in the value function estimate by encouraging local search around the current estimate . Algorithmically , the Gradient-DD approach only adds an additional term to the update rule of the GTD2 method , and the extra computational cost is negligible . We show mathematically that applying this method significantly improves the convergence rate relative to the GTD2 method for linear function approximation . This result is supported by numerical experiments , which also show that Gradient-DD obtains better convergence in many cases than conventional TD learning . 1.1 RELATED WORK . In related approaches to ours , some previous studies have attempted to improve Gradient-TD algorithms by adding regularization terms to the objective function . Liu et al . ( 2012 ) have used l1 regularization on weights to learn sparse representations of value functions , and Ghiassian et al . ( 2020 ) has used l2 regularization on weights . Unlike these references , our approach modifies the error objective function by regularizing the evaluation error obtained in the most recent time step . With this modification , our method provides a learning rule that contains second-order differences in successive parameter updates . Our approach is similar to trust region policy optimization ( Peters & Schaal , 2008 ; Schulman et al. , 2015 ) or relative entropy policy search ( Peters et al. , 2010 ) , which penalize large changes being learned in policy learning . In these methods , constrained optimization is used to update the policy by considering the constraint on some measure between the new policy and the old policy . Here , however , our aim here is to look for the optimal value function , and the regularization term uses the previous value function estimate to avoid drastic changes in the updating process . 2 GRADIENT DESCENT METHOD FOR OFF-POLICY TEMPORAL DIFFERENCE LEARNING . 2.1 PROBLEM DEFINITION AND BACKGROUND . In this section , we formalize the problem of learning the value function for a given policy under the Markov Decision Process ( MDP ) framework . In this framework , the agent interacts with the environment over a sequence of discrete time steps , t = 1 , 2 , . . .. At each time step the agent observes a partial summary of the state st ∈ S and selects an action at ∈ A . In response , the environment emits a reward rt ∈ R and transitions the agent to its next state st+1 ∈ S. The state and action sets are finite . State transitions are stochastic and dependent on the immediately preceding state and action . Rewards are stochastic and dependent on the preceding state and action , as well as on the next state . The process generating the agent ’ s actions is termed the behavior policy . In off-policy learning , this behavior policy is in general different from the target policy π : S → A . The objective is to learn an approximation to the state-value function under the target policy in a particular environment : V ( s ) = Eπ [ ∞∑ t=1 γt−1rt|s1 = s ] , ( 1 ) where γ ∈ [ 0 , 1 ) is the discount rate . In problems for which the state space is large , it is practical to approximate the value function . In this paper we consider linear function approximation , where states are mapped to feature vectors with fewer components than the number of states . Specifically , for each state s ∈ S there is a corresponding feature vector x ( s ) ∈ Rp , with p ≤ |S| , such that the approximate value function is given by Vw ( s ) : = w > x ( s ) . ( 2 ) The goal is then to learn the parameters w such that Vw ( s ) ≈ V ( s ) . 2.2 GRADIENT TEMPORAL DIFFERENCE LEARNING . A major breakthrough for the study of the convergence properties of MDP systems came with the introduction of the GTD and GTD2 learning algorithms ( Sutton et al. , 2009a ; b ) . We begin by briefly recapitulating the GTD algorithms , which we will then extend in the following sections . To begin , we introduce the Bellman operator B such that the true value function V ∈ R|S| satisfies the Bellman equation : V = R + γPV = : BV , where R is the reward vector with components E ( rn+1|sn = s ) , and P is a matrix of state transition probabilities . In temporal difference methods , an appropriate objective function should minimize the difference between the approximate value function and the solution to the Bellman equation . Having defined the Bellman operator , we next introduce the projection operator Π , which takes any value function V and projects it to the nearest value function within the space of approximate value functions of the form ( 2 ) . Letting X be the matrix whose rows are x ( s ) , the approximate value function can be expressed as Vw = Xw . We will also assume that there exists a limiting probability distribution such that ds = limn→∞ p ( sn = s ) ( or , in the episodic case , ds is the proportion of time steps spent in state s ) . The projection operator is then given by Π = X ( X > DX ) −1X > D , where the matrix D is diagonal , with diagonal elements ds . The natural measure of how closely the approximation Vw satisfies the Bellman equation is the mean-squared Bellman error : MSBE ( w ) = ‖Vw −BVw‖2D , ( 3 ) where the norm is weighted by D , such that ‖V‖2D = V > DV . However , because the Bellman operator follows the underlying state dynamics of the Markov chain , irrespective of the structure of the linear function approximator , BVw will typically not be representable as Vw for any w. An alternative objective function , therefore , is the mean squared projected Bellman error ( MSPBE ) , which we define as J ( w ) = ‖Vw −ΠBVw‖2D . ( 4 ) Following ( Sutton et al. , 2009b ) , our objective is to minimize this error measure . As usual in stochastic gradient descent , the weights at each time step are then updated by ∆w = −α∇wJ ( w ) , where α > 0 , and −1 2 ∇wJ ( w ) =− E [ ( γxn+1 − xn ) x > n ] [ E ( xnx > n ) ] −1E ( δnxn ) ≈− E [ ( γxn+1 − xn ) x > n ] η . ( 5 ) For notational simplicity , we have denoted the feature vector associated with sn as xn = x ( sn ) . We have also introduced the temporal difference error δn = rn + ( γxn+1 − xn ) > wn , as well as η , a linear predictor to approximate [ E ( xnx > n ) ] −1E ( δnxn ) . Because the factors in Eqn . ( 5 ) can be directly sampled , the resulting updates in each step are δn =rn + ( γxn+1 − xn ) > wn ηn+1 =ηn + βn ( δn − x > n ηn ) xn wn+1 =wn − αn ( γxn+1 − xn ) ( x > n ηn ) . ( 6 ) These updates define the GTD2 learning algorithm , which we will build upon in the following section . 3 GRADIENT DESCENT TEMPORAL DIFFERENCE-DIFFERENCE LEARNING . In order to improve the GTD2 algorithm described above , in this section we modify the objective function via additionally considering the approximation error Vw−Vwn−1 given the previous time step n− 1 . Specifically , we modify Eqn . ( 4 ) as follows : JGDD ( w|wn−1 ) = J ( w ) + κ‖Vw −Vwn−1‖2D , ( 7 ) Figure 1 : Schematic diagram of Gradient-DD learning with w ∈ R2 . Rather than updating w directly along the gradient of the MSPBE ( arrow ) , the update rule selects wn that minimizes the MSPBE while satisfying the constraint ‖Vw −Vwn−1‖2D ≤ µ ( shaded ellipse ) . where κ ≥ 0 is a parameter of the regularization . Minimizing Eqn . ( 7 ) is equivalent to the following optimization arg min w J ( w ) s.t . ‖Vw −Vwn−1‖2D ≤ µ ( 8 ) where µ > 0 is a parameter which becomes large when κ is small , so that the MSPBE objective is recovered as µ→∞ , equivalent to κ→ 0 in Eqn . ( 7 ) . We show in the Appendix that for any µ > 0 , there exist κ ≥ 0 such that the solution of Eqn . ( 7 ) and that of Eqn . ( 8 ) are the same . Eqns . ( 7 ) and ( 8 ) represent a tradeoff between minimizing the MSPBE error and preventing the estimated value function from changing too drastically . Rather than simply minimizing the optimal prediction from the projected Bellman equation , the agent makes use of the most recent update to look for the solution . Figure 1 gives a schematic view of the effect of the regularization . Rather than directly following the direction of the MSPBE gradient , the update chooses a w that minimizes the MSPBE while following the constraint that the estimated value function should not change too greatly . In effect , the regularization term encourages searching around the estimate at previous time step , especially when the state space is large . With these considerations in mind , the negative gradient of JGDD ( w|wn−1 ) is − 1 2 ∇wJGDD ( w|wn−1 ) =− E [ ( γxn+1 − xn ) x > n ] [ E ( xnx > n ) ] −1E ( δnxn ) − κE [ ( x > nwn − x > nwn−1 ) xn ] ≈− E [ ( γxn+1 − xn ) x > n ] ηn − κE [ ( x > nwn − x > nwn−1 ) xn ] . ( 9 ) Because the terms in Eqn . ( 9 ) can be directly sampled , the stochastic gradient descent updates are given by δn =rn + ( γxn+1 − xn ) > wn ηn+1 =ηn + βn ( δn − x > n ηn ) xn wn+1 =wn − κn ( x > nwn − x > nwn−1 ) xn − αn ( γxn+1 − xn ) ( x > n ηn ) . ( 10 ) These update equations define the Gradient-DD method , in which the GTD2 update equations ( 6 ) are generalized by including a second-order update term in the third update equation , where this term originates from the squared bias term in the objective ( 7 ) . In the following sections , we shall analytically and numerically investigate the convergence and performance of Gradient-DD learning .

Summary: This paper proposes a variant of the GTD2 algorithm by adding an additional regularization term to the objective function, and the new algorithm is named as Gradient-DD (GDD). The regularization ensures that the value function does not change drastically between consecutive iterations. The authors show that the update rule of GDD can be written as a difference equation and aim to further show the convergence via Lyapunov based analysis. An simulation study is provided to compare the proposed GDD algorithm with TD, ETD, and GTD.