Regularizing Trajectories to Mitigate Catastrophic Forgetting

1 INTRODUCTION It is good to have an end to journey toward ; but it is the journey that matters , in the end . Ursula K. Le Guin Endowing machine learning models with the capability to learn a variety of tasks in a sequential manner is critical to obtain agents that are both versatile and persistent . However , continual learning of multiple tasks is hampered by catastrophic forgetting ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) , the tendency of previously acquired knowledge to be overwritten when learning a new task . Techniques to mitigate catastrophic forgetting can be roughly categorized into 3 lines of work ( see Parisi et al . ( 2019 ) for a comprehensive overview ) : 1. regularization-based approaches , where forgetting is mitigated by the addition of a penalty term in the learning objective ( Kirkpatrick et al . ( 2017 ) ; Chaudhry et al . ( 2018a ) , inter alia ) , 2. dynamic architectures approaches , which incrementally increase the model ’ s capacity to accomodate the new tasks ( Rusu et al. , 2016 ) , and 3. memorybased approaches , which retain data from learned tasks for later reuse ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018b ; 2019 ) . Among these , regularization-based approaches are particularly appealing because they do not increase the model size and do not require access to past data . This is particularly relevant to real-world scenarios where keeping data from previous training tasks may be impractical because of infrastructural or privacy-related reasons . Moreover , they are of independent intellectual interest because of their biological inspiration rooted in the idea of synaptic consolidation ( Kirkpatrick et al. , 2017 ) . A good regularizer ensures that , when learning a new task , gradient descent will ultimately converge to parameters that yield good results on the new task while preserving performance on previously learned tasks . Critically , this is predicated upon successful optimization of the regularized objective , a fact that has been largely taken for granted in previous work . Non-convexity of the loss function , along with noise in the data ( due to small or biased datasets ) or in the gradients ( due to stochastic gradient descent ) , can yield optimization trajectories — and ultimately convergence points — that are highly non-deterministic , even for the same starting parameters . As we demonstrate in this paper , this can cause unintended catastrophic forgetting along the optimization path . This is illustrated in a toy setting in Figure 1 : a two parameter model is trained to perform task T2 ( an arbitrary bi-modal loss function ) after having learned task T1 ( a logistic regression task ) . Standard finetuning , even in 1We commit to releasing the code to implement our method and reproduce our experiments upon acceptance . the presence of a regularized objective ( EWC ; Kirkpatrick et al . ( 2017 ) ) , quickly changes the loss of T1 and tends converge to a solution with high T1 loss . We propose to remedy this issue by regularizing the optimization trajectory itself , specifically by preconditioning gradient descent with the empirical Fisher information of previously learned tasks ( §3 ) . This yields what we refer to as a co-natural gradient , an update rule inspired by the natural gradient ( Amari , 1997 ) , but taking the Fisher information of previous tasks as a natural Riemannian metric2 of the parameter space , instead of the Fisher information of the task being optimized for . When we introduce our proposed co-natural gradient for the toy example of Figure 1 , the learning trajectory follows a path that changes the loss on T1 much more slowly , and tends to converges to the optimum that incurs the lowest performance degradation on T1 . We test the validity of our approach in a continual learning scenario ( §4 ) . We show that the co-natural gradient consistently reduces forgetting in a variety of existing continual learning approaches by a factor of≈ 1.5 to 9 , and greatly improves performance over simple finetuning , without modification to the training objective . We further investigate the special case of transfer learning in a two-task , low-resource scenario . In this specific case , control over the optimization trajectory is particularly useful because the optimizer has to rely on early stopping to prevent overfitting to the meager amount of training data in the target task . We show that the co-natural gradient yields the best trade-offs between source and target domain performance over a variety of hyper-parameters ( §5 ) . 2 BACKGROUND AND NOTATIONS . We first give a brief overview of the continual learning paradigm and existing approaches for overcoming catastrophic forgetting . 2.1 NOTATION . Let us define a task as a triplet containing an input space X and an output space Y , both measurable spaces , as well as a distribution D over X × Y . In general , learning a task will consist of training a model to approximate the conditional distribution p ( y | x ) induced by D. Consider a probabilistic model pθ parametrized by θ ∈ Rd where d is the size of the model , trained to perform a source task S = 〈XS , YS , DS〉 to some level of performance , yielding parameters θS . In the most simple instance of continual learning , we are tasked with learning a second target task T = 〈XT , YT , DT 〉 . In general in a multitask setting , it is not the case that the input or output spaces are the same . The discrepancy between input/output space can be addressed in various ways , e.g . by adding a minimal number of task-specific parameters ( for example , different softmax layers for different label sets ) . To simplify exposition , we set these more specific considerations aside for now , and assume that XS = XT and YS = YT . 2Informally , the reader can think of a Riemannian metric as a function that assigns an inner product u , v 7→ gx ( u , v ) to each point x in the space , thus inducing a localized notion of distance and curvature . At any given point during training for task T , our objective will be to minimize the loss function LT ( θ ) – generally the expected log-likelihood Ex , y∼DT [ − log pθ ( y | x ) ] . Typically , this will be performed by iteratively adding incremental update vectors δ ∈ Rd to the parameters θ ←− θ + δ . 2.2 EXISTING APPROACHES FOR CONTINUAL LEARNING . In this paper , we focus on those models that have a fixed architecture over the course of continual learning . The study of continual learning for models of fixed capacity can be split into two distinct ( but often overlapping ) streams of work : Regularization-based approaches introduce a penalty in the loss function LT , typically quadratic , pushing the weights θ back towards θS : LT ( θ ) = Ex , y∼DT − log pθ ( y | x ) ︸ ︷︷ ︸ NLL on task T +λ ( θ − θS ) TΩS ( θ − θS ) ︸ ︷︷ ︸ Regularization term ( 1 ) where ΩS is a matrix , typically diagonal , that encodes the respective importance of each parameter with respect to task S , and λ is a regularization strength hyper-parameter . Various choices have been proposed for ΩS ; the diagonal empirical Fisher information matrix ( Kirkpatrick et al. , 2017 ) , or pathintegral based importance measures ( Zenke et al. , 2017 ; Chaudhry et al. , 2018a ) . More elaborate regularizers have been proposed based on e.g . a Bayesian formulation of continual learning ( Nguyen et al. , 2017 ; Ahn et al. , 2019 ) or a distillation term ( Li & Hoiem , 2016 ; Dhar et al. , 2019 ) . The main advantage of these approaches is that they do not rely on having access to training data of previous tasks . Memory-based approaches store data from previously seen tasks for re-use in continued learning , either as a form of constraint , by e.g . ensuring that training on the new task doesn ’ t increase the loss on previous tasks ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018b ) , or for replay i.e . by retraining on instances from previous tasks ( Rebuffi et al. , 2017 ; Chaudhry et al. , 2019 ; Aljundi et al. , 2019b ; a ) . Various techniques have been proposed for the selection of samples to store in the memory ( Chaudhry et al. , 2019 ; Aljundi et al. , 2019b ) or for retrieval of the samples to be used for replay Aljundi et al . ( 2019a ) . All of these methods rely on stochastic gradient descent to optimize their regularized objective or to perform experience replay , with the notable exception of GEM ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018b ) , where the gradients are projected onto the orthogonal complement of previous task ’ s gradients . However , this method has been shown to perform poorly in comparison with simple replay ( Chaudhry et al. , 2019 ) , and it still necessitates access to data from previous tasks . 3 REGULARIZING THE TRAJECTORY . After briefly recalling how the usual update is obtained in gradient descent , we derive a new , conatural update designed to better preserve the distribution induced by the model over previous tasks . 3.1 WARM UP : THE STANDARD GRADIENT DESCENT UPDATE . At point θ in the parameter space , gradient descent finds the optimal update δ that is ( 1 ) small and ( 2 ) locally minimizes the decrease in loss L ( θ + δ ) − L ( θ ) ( ≈ δᵀ∇θL at the first order ) . Traditionally this can be formulated as minimizing the Lagrangian : L ( δ ) = δᵀ∇θLT︸ ︷︷ ︸ first order loss minimization term + µ‖δ‖2︸ ︷︷ ︸ “ small update ” term ( 2 ) with Lagrangian multiplier µ > 0 . Minimizing L for δ yields the well-known optimal update δ∗ : δ∗ = − 1 2µ ∇θLT ( 3 ) where 12µ corresponds to the learning rate ( see Appendix A.1 for the full derivation ) . 3.2 KL REGULARIZATION OF TRAJECTORIES . The ‖δ‖2 term in L implicitly expresses the underlying assumption that the best measure of distance between parameters θ and θ + δ is the Euclidean distance . In a continual learning setting however , the quantity we are most interested in preserving is the probability distribution that θ models on the source task S : pSθ ( x , y ) = pθ ( y | x ) pS ( x ) ( 4 ) Therefore , a more natural distance between θ and θ + δ is the Kullback-Leibler divergence KL ( pSθ ‖pSθ+δ ) ( Kullback & Leibler , 1951 ) . For preventing catastrophic forgetting along the optimization path , we incorporate incorporate this KL term into the Lagrangian L itself : L ( δ ) = δᵀ∇θLT + µ‖δ‖2 + νKL ( pSθ ‖pSθ+δ ) ( 5 ) Doing so means that the optimization trajectory will tend to follow the direction that changes the distribution of the model the least . Notably , this is not a function of the previous objective LS , so knowledge of the original training objective is not necessary during continual learning ( which is typically the case in path-integral based regularization methods ( Zenke et al. , 2017 ) or experience replay ( Chaudhry et al. , 2019 ) ) .

Summary: This paper amends the gradient update rule for continual learning using a natural-gradient-style formulation in order to regularise the trajectory during learning to forget previous task(s) less. They show experiments where this 'co-natural gradient' update rule improves some baselines. They also provide experiments showing the benefits of this update rule for low-resource finetuning settings.