Overcoming Catastrophic Forgetting via Hessian-free Curvature Estimates

Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks . This leads to a poor performance on old tasks – a phenomenon framed as catastrophic forgetting . While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization . Based on the Fisher information matrix ( FIM ) changes to parameters that are relevant to old tasks are penalized , which forces the task to be mapped into the available remaining capacity of the network . This requires to calculate the Hessian around a mode , which makes learning tractable . In this paper , we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian . In contrast to previous work , we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task . Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work . 1 INTRODUCTION . The main goal of machine learning is the ability to generalize from the given training data to unseen examples . However , in practice the achievable degree of generalization is limited . While in the ideal case an end-to-end system learns complex functions from minimum input , it is often necessary to introduce a certain amount of prior knowledge . Such prior knowledge operates as an inductive bias and therefore has a constraining effect on the hypothesis space , i.e. , the set of all possible functions that can be learned by the learning algorithm ( Mitchell , 1980 ) . While this sounds counter-intuitive such a reduction of the hypothesis space may lead to better generalization properties in practice ( Mitchell , 1980 ) . Hence , instead of eliminating the bias to increase generalization ( as suggested by Hessel et al . ( 2019 ) ) , a promising direction of research tries to identify and introduce the right form of it . We can achieve this by limiting the functions that can be expressed by the learning algorithm or by introducing bias to the learning algorithm itself . Simple examples include the choice for linear activations to only allow approximations of linear functions or to add a regularization term to the objective function . Similar to this , we can also improve generalization by training on different tasks ( Baxter , 2000 ) from a task family at the same time or by introducing auxiliary tasks ( Jaderberg et al. , 2017 ) . This is commonly known as multitask learning and has shown to not only improve generalization properties but also to be more sample-efficient ( Baxter , 2000 ) . Due to the limited availability of data for training we need a well-tuned inductive bias . Hence , such choices are crucial for the final real-world performance of any machine learning algorithm . While multitask learning is a great tool to improve generalization and to reduce the amount of samples that are necessary to learn a family of tasks it is still limited in its scalability . Both the amount of tasks that can be learned and the amount of data required to learn them are strongly limiting factors . Consider , for instance , a reinforcement learning setup where an agent learns different tasks from interacting with in an environment . In practice we are limited in storing the data for all relevant tasks required to train a model on all tasks jointly . However , learning those tasks sequentially is also not an option as gradient descent and its variants ( which are the dominant learning approaches for neural networks ) do not consider the importance of individual parameters for early tasks . This destructive learning is commonly termed as catastrophic forgetting ( McCloskey & Cohen , 1989 ) . While in the context of fine-tuning and pre-training ( Erhan et al. , 2009 ) this does not bear a problem ( as the goal is not to reuse the previous parameter state , but rather to optimize the learning process for some target task ) it becomes important in multitask problems where we wish to maximize generalization and sample-efficiency . It is also critical in the continual learning framework , where the parameters of a neural network are optimized over multiple datasets ( representing different tasks ) provided sequentially , which are not available at later time . The goal is hence to retain all ( or most ) of the important parameters for previous tasks and to be able to build-up on this knowledge for an arbitrary number of future tasks . Thus , the scalability of learning would only be limited by the capacity of the neural network but not by the properties of the training method . The Bayesian framework ( Kirkpatrick et al. , 2017 ; Ritter et al. , 2018 ) is a promising approach to address catastrophic forgetting . The information about former tasks is condensed in a prior , which not only preserves the knowledge about tasks but also introduces an inductive bias based on the learned tasks . Elastic Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) is a simple yet efficient way to reduce catastrophic forgetting . EWC approximates the prior with a Gaussian centered around the optimized network parameters for previous tasks , where the diagonal precision is given by the diagonal approximation of the Fisher Information Matrix ( FIM ) . This approach has two significant downsides : i ) each new task adds a new regularization term that penalizes changes of parameters that are relevant to previous tasks ; and ii ) the diagonal approximation of the FIM assumes independent network parameters , which leads to information loss with a growing number of tasks . Ritter et al . ( 2018 ) extend EWC but still approximate the prior from previous tasks using a Gaussian . They devise a block-diagonal approximation for the prior from the older tasks by defining a quadratic approximation whose solution requires to calculate the Hessian . The Hessian is in turn approximated by the block-diagonal Kronecker-factored approximation . In this work we propose an alternative way of calculating the Hessian , based on well established Hessian-free ( Schraudolph , 2002 ; Pearlmutter , 1994 ) methods to estimate curvature information of the network parameters . In contrast to Ritter et al . ( 2018 ) , we exploit the fact that most regions in the loss surface are flat ( Ghorbani et al. , 2019 ) . This allows us to use only a small subset of the Hessian as it holds enough relevant information . We then use a Hessian-vector-product to sample from this subset . This way , we can incorporate the importance of individual weights and include dependencies between the network parameters when we train the network over a long sequence of tasks . We evaluate our algorithm on permuted MNIST ( Kirkpatrick et al. , 2017 ) , disjoint MNIST ( Ritter et al. , 2018 ) and single-headed disjoint MNIST ( Farquhar & Gal , 2019 ) , and compare with state of the art approaches . Our results show that we consistently outperform EWC across all tasks and that we are en par with Ritter et al . ( 2018 ) on the disjoint tasks , while our method has significantly lower space complexity compared to both EWC and Kronecker-factored approximation . The remainder of this paper is structured as follows . Section 2 provides background on continual learning , EWC , and Kronecker-factored Laplace approximation . Section 3 describes our method in detail . Section 4 shows the efficiency of our approach and compares it against state of the art on a variety of well-known task sequences . Section 5 discusses related work . Section 6 concludes . 2 BACKGROUND . 2.1 CONTINUAL LEARNING AND CATASTROPHIC FORGETTING . In the continual learning framework the parameters θ ∈ Rn of a neural network are optimized over multiple datasets D1 , . . . , Dt , . . . , DT . These individual datasets become available to the training algorithm one after another and usually can not be revisited at a later time . The goal is to achieve a high accuracy/performance on the current task ( represented by the current dataset Dt ) while still preserving ( high or most of ) the performance for all the previously visited tasks . However , this is usually challenging for neural network models as commonly used gradient-based optimization methods can not distinguish between important and unimportant parameters for previous tasks . As a consequence parameters that are relevant for previous tasks are modified ( heavily ) , which leads to performance degradation when the network is used on any of those previous tasks ( Rusu et al. , 2016 ) . Hence , to address catastrophic forgetting in neural networks we need to retain the parameters that are important for previous tasks while still allowing the network to learn new tasks . However , at the same time we also want the space complexity of the network to be independent of the amount of tasks that were observed so far ( and that are about to come ) . This means that learning a new task while retaining high performance on all prior tasks should be possible without adding new parameters or regularization terms for each new task , at least as long sufficient capacity is available . As a plus we want to foster some degree of parameter sharing to enable positive transfer effects , e.g. , improved sample-efficiency due to the fact that past experience can be reused . 2.2 ELASTIC WEIGHT CONSOLIDATION ( EWC ) . EWC ( Kirkpatrick et al. , 2017 ) is a simple yet efficient approach that meets most of the above mentioned requirements . The key idea is to add a penalty when parameters that are important for previous tasks are about to be changed while parameters that are less relevant for previous tasks do not receive a penalty . EWC uses a quadratic penalty term that is derived from a Bayesian formulation of the problem ( where all the information of all previous tasks is condensed in the prior ) as follows : p ( θ|D1 : t+1 ) = p ( Dt+1|θ ) p ( θ|D1 : t ) p ( Dt+1 ) , ( 1 ) where p ( θ|D1 : t+1 ) and p ( θ|D1 : t ) are the posterior and prior distributions over the parameters θ of the network and D1 , . . . , Dt , Dt+1 are the datasets corresponding to the respective tasks . If we want to learn a new task we update the posterior by conditioning it on the newly available data Dt+1 . However , we have to address two problems that stem from Equation 1 . First , maintaining the full posterior over all previous datasets is usually intractable ( Ritter et al. , 2018 ; Opper & Winther , 1998 ) and we instead need to approximate it . Second , without storing the information from all previous tasks there is no easy solution to update the posterior . The first problem can be addressed by approximating the posterior with a Gaussian ( MacKay , 1992 ) : p ( θ|D1 : t ) ∼ N ( µt , Σt ) . ( 2 ) With two tasks A and B and their datasets DA and DB , for the posterior p ( θ|DA ) the mean µA is given by the solution for the previous task θ∗A , and the precision Σ −1 A , i.e. , the inverse of the covariance , by the diagonal of the Fisher information matrix ( FIM ) F . Learning tasks A and B consecutively then results in the following objective function : L ( θ ) = LB ( θ ) + λ 2 ( θ − θ∗A ) TF ( θ − θ∗A ) , ( 3 ) where LB ( θ ) is the loss depending on the current data DB , and λ is a hyperparameter that controls the influence of the regularization term . At this point we only need to store the previous weights and the diagonal approximation of the FIM for the previous task . For another task C we store a separate FIM for that new task together with the solution for task B θ∗B , and add another regularization term : L ( θ ) = LC ( θ ) + λ 2 ( θ − θ∗A ) TFA ( θ − θ∗A ) + λ 2 ( θ − θ∗B ) TFB ( θ − θ∗B ) . ( 4 )

Summary: The paper focuses on alleviating the problem of "catastrophic forgetting", exhibited by neural networks learned with gradient-based algorithms over long sequence of tasks. In such learning scenarios, tuning of parameters over the new tasks lead to degradation of performance over the old tasks as the parameters important for the latter are overwritten. The gradient-based algorithms are unable to distinguish between the important and  the not-so-important parameters of the old tasks. Hence, one direction of works, including the proposed one, aim at identifying the most important parameters for all the old tasks and discourage modifications on those parameters during the training of the new tasks.