HR-TD: A Regularized TD Method to Avoid Over-Generalization

1 INTRODUCTION . Temporal Difference ( TD ) learning is one of the most important paradigms in Reinforcement Learning ( Sutton and Barto , 1998 ) . Techniques based on combining TD learning with nonlinear function approximators and stochastic gradient descent , such as deep networks , have led to significant breakthroughs in large-scale problems to which these methods can be applied ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Schulman et al. , 2015 ) . At its heart , the TD learning update is straightforward . v ( s ) estimates the value of being in a state s. After an action a that transitions the agent from s to next state s′ , v ( s ) is altered to be closer to the ( discounted ) estimated value of s′ , v ( s′ ) ( plus any received reward , r ) . The difference between these estimated values is called the temporal difference error ( TD error ) and is typically denoted as δ . Formally , δ = r + γv ( s′ ) − v ( s ) , where γ is the discount factor , and r + γv ( s′ ) is known as the TD target . When states are represented individually ( the tabular case ) , v ( s ) can be altered independently from v ( s′ ) using the update rule v ( s ) ← v ( s ) + αδ , where α is the learning rate . In fully deterministic environments , α can be set to 1 , thus causing v ( s ) to change all the way to the TD target . Otherwise , in a stochastic environment , α is set less than 1 so that v ( s ) only moves part of the way towards the TD target , thus avoiding over-generalization from a single example . When , on the other hand , states are represented with a function approximator , as is necessary in large or continuous environments , v ( s ) can no longer be updated independently from v ( s′ ) . That is because s and s′ are likely to be similar ( assuming actions have local effects ) , any change to v ( s ) is likely to also alter v ( s′ ) . While such generalization is desirable in principle , it also has the unintended consequence of changing the TD target , which in turn can cause the TD update to lead to an increase in the TD error between s and s′ . This unintended consequence can be seen as a second form of over-generalization : one that can be much more difficult to avoid . Past work has identified this form of over-generalization in RL , has observed that it is particularly relevant in methods that use neural network function approximators such as DQN ( Mnih et al. , 2015 ) , and has proposed initial solutions ( Durugkar and Stone , 2017 ; Pohlen et al. , 2018 ) . In this paper , we present a deeper analysis of the reasons for this form of over-generalization and introduce a novel learning algorithm termed HR-TD , based on the recursive proximal mapping formulation of TD learning ( Bertsekas , 2011 ) , which offers a mathematical framework for parameter regularization that allows one to control for this form of over-generalization . Empirical results across multiple domains demonstrate that our novel algorithm learns more efficiently ( from fewer samples ) than prior approaches . The rest of the paper is organized as follows . Section 2 offers a brief background on TD learning , the over-generalization problem , and optimization techniques used in the derivation of our algorithm . In Section 3 , we discuss the state-of-the-art research in this direction . The motivation and the design of our algorithm are presented in Section 4 . Finally , the experimental results of Section 5 validate the effectiveness of the proposed algorithm . 2 BACKGROUND . This section builds on the notation introduced in Section 1 to specify the problem of interest in full detail . We introduce the background for TD learning , over-generalization , and proximal mapping , which are instrumental in the problem formulation and algorithm design . 2.1 REINFORCEMENT LEARNING AND OVER-GENERALIZATION . Reinforcement Learning problems are generally defined as Markov Decision Processes ( MDPs ) . We use the definition and notation as used in Sutton and Barto ( 2017 ) , unless otherwise specified . In this paper , we focus on domains with large or continuous state spaces such that function approximation is needed . We define the value estimate of state s with parameter θ when following policy π as , vπ ( s|θ ) = Eπ [ Rt + γRt+1 + γ 2Rt+2 + . . . |St = s ] . Here Rt is the random variable associated with a reward at time t , and rt is used as an instantiation of this random variable . The optimal ( true ) value function v∗π satisfies the Bellman equation given as v ∗ π ( s|θ ) = Eπ [ Rt + γv∗π ( s′|θ ) ] . During TD learning , the estimated value function is altered to try to make this property hold . In effect , state values are updated by bootstrapping off of the estimated value of the predicted next states . We focus on 1-step TD methods , i.e. , TD ( 0 ) , that bootstrap from the value of the immediate next state or states in the MDP to learn the value of the current state . The TD error δt ( st , st+1|θ ) to be minimized is as follows : δt ( st , st+1|θ ) = ( rt + γvπ ( st+1|θ ) ) − vπ ( st|θ ) In the following , δt ( st , st+1|θ ) is written as δt for short . When using function approximation and gradient descent to optimize the parameters , the loss to be minimized is the squared TD error . At the t-th time-step , the objective function used in TD learning is LTD = ‖rt+γvπ ( st+1|θ ) −vπ ( st|θ ) ‖2 . Similarly , the optimal action value function Q satisfies the Bellman optimality equation Q∗ ( st , at|θ ) = Rt + γmax a Q∗ ( st+1 , a|θ ) . The objective used in Q-Learning is thus LQ = ‖rt + γmaxaQ ( st+1 , a|θ ) −Q ( st , at|θ ) ‖2 . The partial derivative of v ( st|θ ) orQ ( st , at|θ ) with respect to θ is the direction in which TD learning methods update the parameters . We use gt ( st|θ ) and gt ( st , at|θ ) to refer to these vectors . In the linear case , v ( st|θ ) = θ > t φ ( st ) , where φ ( st ) are the features of state st . In this case , gt ( st , at|θ ) is the feature vector φ ( st , at ) , and in general , gt ( st , at|θ ) = ∂θQ ( st , at|θ ) . It is computed as : gt ( st , at|θ ) = ∂Q ( st , at|θ ) ∂θ θ ← θ + αδtgt ( st , at|θ ) . We have already briefly alluded to the issue of over-generalization in Section 1 . One of the reasons we use function approximation is that we want the values we learn to generalize to similar states . But one of these similar states is likely to be the target of our Bellman equation v ( st+1|θ ) . If the weights that correspond to large or important features in φ ( st+1 ) are strengthened , then the TD error might not decrease as much as it should , or it might even increase . We refer to parameter updates that work against the objective of reducing the TD error as over-generalization . 2.2 PROXIMAL MAPPING FORMULATION OF TD LEARNING . In this section , we introduce the basics of proximal mapping , which provide the mathematical formulation of our algorithm design . A proximal mapping ( Parikh and Boyd , 2013 ) proxf ( w ) associated with a convex function f is defined as proxf ( w ) = arg minx ( f ( x ) + 1 2 ‖w − x‖22 ) ( 1 ) Such a proximal mapping is typically used after a parameter update step to incorporate constraints on the parameters . Intuitively , the first term f ( x ) provides incentive to move x in the direction that minimizes f , whereas the second term 12‖w − x‖ 2 2 provides pressure to keep x close to w. If f ( x ) = 0 , then proxf ( w ) = w , the identity function . f can often be a regularization term to help incorporate prior knowledge . For example , for learning sparse representations , the case of f ( x ) = β‖x‖1 is particularly important . In this case , the entry-wise proximal operator is : proxf ( w ) i = sign ( wi ) max ( |wi|−β , 0 ) Proximal methods have been shown to be useful for various reinforcement learning problems , e.g. , proximal gradient TD learning ( Liu et al. , 2015 ) integrates the proximal method with gradient TD learning ( Sutton et al. , 2009 ) using the Legendre-Fenchel convex conjugate function ( Boyd and Vandenberghe , 2004 ) , and projected natural actor-critic ( Thomas et al. , 2013 ) interprets natural gradient as a special case of proximal mapping . We now introduce the recursive proximal mapping formulation of TD learning algorithm ( Bertsekas , 2011 ) . At the t-th iteration , the TD update law solves a recursive proximal mapping , i.e. , θt+1 = θt + αtδtgt ( st ) , which is equivalent to θt+1 = arg min x { 〈x , −δtgt ( st ) 〉+ 1 2αt ||x− θt||22 } ( 2 ) It should be noted that Eq . ( 2 ) is different from Eq . ( 1 ) in that Eq . ( 1 ) has an explicit objective function f to optimize . Eq . ( 2 ) does not have an explicit objective function , but rather corresponds to a fixed-point equation . In fact , it has been proven that the TD update term δtgt ( st ) does not optimize any objective function ( Maei , 2011 ) . Discussing this in details goes beyond the scope of the paper , and we refer interested readers to ( Maei , 2011 ; Bertsekas , 2011 ) for a comprehensive discussion of this topic . 3 RELATED WORK . To the best of our knowledge , the closest work to ours to address the over-generalization problem is the Temporal Consistency loss ( TC-loss ) method ( Pohlen et al. , 2018 ) and the constrained TD approach ( Durugkar and Stone , 2017 ) . The TC-loss ( Pohlen et al. , 2018 ) aims to minimize the change to the target state by minimizing explicitly a separate loss that measures change in the value of s′ , i.e. , L ( Vθ ( s ′ , a′ ) − Vθt−1 ( s′ , a′ ) ) . When used in conjunction with a TD loss , it guarantees that the updated estimates adhere to the Bellman operator and thus are temporally consistent . However , there are some drawbacks to this method . Firstly , the asymptotic solution of the TC-loss method is different from the TD solution due to the two separate losses , and the solution property remains unclear . Secondly , each parameter component plays a different role in changing v ( s′ ) . For instance , if the component of θ is or close to 0 , then this component does not have any impact on changing v ( s′ ) . Different parameter components , therefore , should be treated differently according to their impact on the value function changes ( or action-value change in case of DQN ) . Another recent work in this direction is the constrained TD ( CTD ) algorithm ( Durugkar and Stone , 2017 ) . To avoid the over-generalization among similar sates , CTD tends to alleviate overgeneralization by using the vector rejection technique to diminish the update along the direction of the gradient of the action-value function of the successive state . In other words , the real update is made to be orthogonal to the gradient of the next state . However , the CTD method suffers from the double-sampling problem , which is explained in detail in Appendix A . Moreover , since it mainly uses vector rejection , this method is not straightforward to extend to nonlinear function approximation , such as the DQN network , where over-generalization can be severe . Lastly , if the state representation of st and st+1 are highly similar , as in case of visual environments like Atari games , then the vector rejection causes the update to be almost orthogonal to the computed gradient .

Summary: This paper introduces a variation on temporal difference learning for the function approximation case that attempts to resolve the issue of over-generalization across temporally-successive states. The new approach is applied to both linear and non-linear function approximation, and for prediction and control problems. The algorithmic contribution is demonstrated with a suite of experiments in classic benchmark control domains (Mountain Car and Acrobot), and in Pong.