Assessing Generalization in TD methods for Deep Reinforcement Learning

1 INTRODUCTION . Deep neural networks ( DNNs ) trained on supervised learning tasks using i.i.d . data have shown the capacity to learn quickly even from a small amount of samples ( Hardt et al. , 2016 ) . Intuitively , this is due to each sample also providing information about the estimate corresponding to other samples ; research suggests that DNNs first extract structures that are informative of the modes of the data ( even if later on they can also memorize see Zhang et al . ( 2016 ) ; Arpit et al . ( 2017 ) ) , and that they can transfer well ( Yosinski et al. , 2014 ; Li et al. , 2015 ) , even from relatively few samples . In contrast , in Deep Reinforcement Learning ( DRL ) , the number of samples required for an agent to learn successfully is often very high ; many modern algorithms struggle to perform well until they acquire tens of millions of samples ( Mirowski et al. , 2016 ; Vinyals et al. , 2017 ; Hessel et al. , 2018 ) , and some even diverge to bad solutions ( Anschel et al. , 2017 ) . While there are many facets to sample complexity and brittleness , we posit that a contributing factor is a lack of what we call gradient update generalization , i.e. , whether performing updates at one state provides useful information about the value/policy at other states . Generalization in RL is of two types : ( a ) generalization to unseen states–will an agent trained on a single MDP pick the optimal action for a state it has never seen before ? ( b ) generalization to unseen tasks–will an agent trained on a distribution of MDPs know how to act in an MDP it has never seen before ? Both of these facets are actively studied . For example , Farebrother et al . ( 2018 ) expose some generalization failures on the Atari domain ( Bellemare et al. , 2013 ) and study the impact of regularization , Zhang et al . ( 2018 ) study the generalization capabilities of DRL agents on randomized mazes , Packer et al . ( 2018 ) study the extrapolation capabilities of DRL agents trained on a distribution of environment parameters ( e.g . pole mass in CartPole ) outside of the training distribution , Cobbe et al . ( 2018 ) find that even on procedurally generated environments , DRL agents can easily overfit on their training set unless regularized , Oh et al . ( 2017 ) study the embedding regularizations necessary for agents to generalize to new instruction sequences on navigation tasks . In this study , we are not interested in measuring state generalization ( i.e . predictions for unseen states ) , nor task generalization ( i.e . in terms of the quality of the behaviour ) , but rather generalization within the process of stochastic gradient learning . In other words , since any kind of generalization must arise through the accumulation of parameter updates , it seems useful to measure whether these parameter updates are themselves general . To this end , we propose the measure of gradient update generalization , best understood as a side-effect of neural networks sharing parameters over their entire input space . That is , updating parameters after seeing one state will change the prediction for virtually all other states ; we are interested in measuring that change . TD methods are a broad class of RL algorithms that form a target for an update by utilizing the current estimate of the value function . They include TD ( 0 ) and TD ( λ ) methods for estimating the value of a fixed policy , as well as Sarsa and Q-learning algorithms for control . TD methods have achieved success in some challenging tasks ( Tesauro , 1995 ; Mnih et al. , 2013 ; Hessel et al. , 2018 ) , but they are also known to have problems when coupled with function approximation ( Sutton , 1995 ; Baird , 1995 ; Tsitsiklis & Van Roy , 1997 ; Chung et al. , 2018 ) . Previous studies explicitly addressed problems such as leakage propagation in TD ( Penedones et al. , 2018 ) , while others aimed to provide sampling improvements ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Fu et al. , 2019 ) , explicit temporal regularization ( Thodoroff et al. , 2018 ) , or auxiliary tasks which push the agent to learn more about the temporal structure in the data ( Jaderberg et al. , 2016 ) . To our knowledge , no study to date has focused on the dynamics of the generalization process itself , within TD-based DRL methods1 such as deep Q-Learning ( Riedmiller , 2005 ; Mnih et al. , 2013 ) , Sarsa ( Rummery & Niranjan , 1994 ) , and TD ( λ ) ( Sutton , 1988 ; Schulman et al. , 2015 ) . For this study , we introduce the aforementioned measure of gradient update generalization , which enables us to differentiate the learning behaviours of different methods . Overall , we find that : 1. when doing a TD ( 0 ) update for a single state , parameters change in such a way that the value prediction of other states is generally not affected , surprisingly even for states that are close either temporally or in an annotated “ ground truth ” state space ; 2 . DNNs trained with TD ( 0 ) , in contrast with DNNs trained on a memorization task or using a supervised objective , do not entirely memorize their state space , yet also do not generalize in the way we would expect ; 3. both the choice of optimizer and the nature of the objective impact the generalization behaviours of models ; in particular , when increasing the λ parameter in TD ( λ ) , DNNs appear to capture more temporal structure . 2 TECHNICAL BACKGROUND . A Markov Decision Process ( MDP ) ( Bellman , 1957 ; Sutton & Barto , 2018 ) M = 〈S , A , R , P , γ〉 consists of a state space S , an action space A , a reward function R : S → R and a transition probability distribution P ( s′|s , a ) . RL agents aim to optimize the expectation of the long-term return : G ( St ) = ∞∑ k=t γk−tR ( Sk ) . ( 1 ) where γ ∈ [ 0 , 1 ) is called the discount factor . Policies π ( a|s ) map states to action distributions . Value functions V π and Qπ map states/states-action pairs to expected returns , and can be expressed recursively : V π ( St ) = Eπ [ G ( St ) ] = Eπ [ R ( St ) + γV ( St+1 ) |At ∼ π ( St ) , St+1 ∼ P ( St , At ) ] ( 2 ) Qπ ( St , At ) = Eπ [ R ( St ) + γ ∑ a π ( a|St+1 ) Q ( St+1 , a ) |St+1 ∼ P ( St , At ) ] ( 3 ) While V π could also be learned via regression to observed values of G , these recursive equations give rise to the Temporal Difference ( TD ) update rules for policy evaluation , relying on current estimates of V to bootstrap , e.g . : V ( St ) ← V ( St ) − α ( V ( St ) − ( R ( St ) + γV ( St+1 ) ) ) , ( 4 ) where α ∈ [ 0 , 1 ) is the step-size . Bootstrapping leads also to algorithms such as Q-Learning ( Watkins & Dayan , 1992 ) and fitted-Q ( Ernst et al. , 2005 ; Riedmiller , 2005 ) : LQL ( St , At , Rt , St+1 ) = [ Qθ ( St , At ) − ( Rt + γmax a Qθ ( St+1 , a ) ) ] 2 , ( 5 ) Sarsa ( Rummery & Niranjan , 1994 ) : LSarsa ( St , At , Rt , St+1 , At+1 ) = [ Qθ ( St , At ) − ( Rt+γQθ ( St+1 , At+1 ) ) ] 2 with At ∼ π ( St ) ( 6 ) 1In contrast , policy-gradient algorithms such as PPO ( Schulman et al. , 2017 ) A3C ( Mnih et al. , 2016 ) and SAC ( Haarnoja et al. , 2018 ) are capable of learning good policies without necessarily having learned a good value function , and although interesting results have emerged to understand learning behaviours in policy-gradient methods ( Ilyas et al. , 2018 ) , these methods build upon TD and analyzing them would add undesired confounders . and TD ( λ ) , which trades off between the unbiased target G ( St ) and the biased TD ( 0 ) target ( biased due to relying on the estimated V ( St+1 ) ) , using a weighted averaging of future targets called a λ-return ( Sutton , 1988 ; Munos et al. , 2016 ) : Gλ ( St ) = ( 1− λ ) ∞∑ n=1 λn−1 γnV ( St+n ) + n−1∑ j=0 γjR ( St+j ) ( 7 ) LTD ( λ ) ( St ) = ( Vθ ( St ) −Gλ ( St ) ) 2 ( 8 ) ( note that the return depends implicitly on the trajectory followed from St ) . When λ = 0 , the loss is simply ( Vθ ( St ) − ( Rt + γVθ ( St+1 ) ) ) 2 , leading to the algorithm called TD ( 0 ) ( Sutton , 1988 ) . 3 UPDATE GENERALIZATION IN DEEP RL . We will now define the measure we propose in order to quantify the speed at which generalization to unseen states occurs , and to characterize the structure under which this generalization occurs . We define gradient update generalization as the expected improvement in the loss function L : Θ×X → R after updating parameters θ ∈ Θ , on sample XU ∈ X , using update function UL : Θ× X → Θ ( e.g . SGD or a semi-gradient methods like TD ( 0 ) ) : YL ( XU ; θ , U ) = EX [ L ( X ; θ ) − L ( X ; UL ( θ , XU ) ) ] . ( 9 ) If generalization from the samples in XU to X is good , this measure of gain should be large , and intuitively fewer other samples should be needed to achieve a desired level of performance . On the other hand , if on average the loss only decreases for the samples XU used in training , then more data in X −XU will have to be visited before the model can learn . Hence , this measure is related to both sample complexity and the speed of learning ( see Fig . 15 for empirical confirmation of this phenomenon ) . As computing the exact expectation is usually intractable , we empirically measure gains on different subsets X ⊂ X . In particular , when X is chosen to be a slice around XU in the replay buffer , we write Y near . We also subscript Y with the corresponding loss ’ subscript , e.g . for ( 5 ) , LQL , we write YQL . In this study , we are interested in TD-based methods that rely heavily on bootstrapping , Q-Learning , Sarsa , and TD ( λ ) , and measure Y using their respective losses , ( 5 ) , ( 6 ) , and ( 8 ) . Structure in DNNs A common intuition in deep learning ( Zhang et al. , 2016 ; Arpit et al. , 2017 ; Zhang et al. , 2018 ) is that DNNs first learn about the structure of their data , meaning the underlying ( usually linear ) factors of variation of the data being mapped into the hidden units ’ space via parameter sharing . These factors of variation are usually conceptualized as a low-dimensional space where each dimension explains part of the data ( Bengio et al. , 2013 ) . It is commonly assumed that a model which generalizes well will naturally capture these factors in the configuration of its parameters , in which case the gradient of the prediction w.r.t . all examples sharing the same latent factors of variation will be very close ; updating with only one sample will change the prediction for all the related examples . Hence , a DNN which captures structure correctly should show high gradient update generalization . Temporal structure in RL Data used in RL algorithms usually exhibits two additional types of structure : coherence of the inputs in a trajectory over time ( e.g . pixel values in adjacent frames are often similar ) , and smoothness of the value function in time ( in the sparse-reward case with γ close to 1 , V ( St ) ≈ γV ( St+1 ) , which is smooth in time , aside from rare discontinuities upon seeing rewards ) . Since RL data consists of trajectories which often have strong temporal structure of both types , we hypothesize that the gain Y near of temporally correlated examples should increase closer in time to the sample used in the update . Parameter sharing Another indirect measure of update generalization related to parameter sharing is the difference since last visit , which we denote as ∆ . At each update iteration k , we compute the difference between the value Vθk ( s ) or Qθk ( s , a ) predicted from the current parameters , θk , and Vθlast ( s ) ( s ) or Qθlast ( s ) ( s , a ) , i.e . the prediction made the last time state s was used for a gradient update.2 To illustrate , if Vθ was a lookup table , ∆ would always be 0 , while for a DNN , when states 2In practice , we simply cache the value prediction for all states in a replay buffer ( as states in a continuous state space are unlikely to be encountered many times ) , and update the cache after a minibatch update ( for those states only ) . are aliased together , ∆ should accurately reflect the effect of parameter sharing after performing sequences of updates ( in contrast , ( 9 ) uses only a single update ) .

Summary: This paper performs an empirical evaluation of generalization by TD methods with neural nets as function approximators. To quantify generalization, the paper considers the change in the loss function at similar states to the one where the update rule is being applied (where “similar” is usually defined as nearby in time). It comes to a variety of conclusions including that TD(0) does not induce much generalization, that TD(0) does not induce as much generalization as supervised learning, and that the choice of optimizer and objective changes the behavior according to their generalization criteria in various ways.