Continual Deep Learning by Functional Regularisation of Memorable Past

1 INTRODUCTION . The ability to quickly adapt to changing environments is an important quality of intelligent systems . For such quick adaptation , it is important to be able to identify , memorise , and recall useful past experiences when acquiring new ones . Unfortunately , standard deep-learning methods are not good at maintaining previously acquired skills , and can quickly forget them when learning new skills ( Kirkpatrick et al. , 2017 ) . Such catastrophic forgetting presents a big challenge when deploying deep-learning methods for applications , such as robotics , where new tasks can appear during the training , and data from the previous tasks might be unavailable for retraining . In recent years , many methods have been proposed to address catastrophic forgetting in deep learning . One of the most popular approaches is to regularise the network weights to keep them close to the weights obtained for the previous tasks/data ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2018 ; Zenke et al. , 2017 ; Ebrahimi et al. , 2019 ; Serra et al. , 2018 ) . This is challenging due to the difficulty in identifying the weights that are relevant to past tasks . The exact values of the weights in fact do not matter directly , but rather the network output ( Benjamin et al. , 2018 ) . Figuring out which weight affects the output is therefore usually difficult . Typically , the Fisher information matrix or covariance matrices over weights are used ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2018 ) , but they only partially address the issue . A better approach is to directly regularise the network outputs , also referred to as functionalregularisation , requiring a memory of past examples ( Benjamin et al. , 2018 ; Lopez-Paz & Ranzato , 2017 ; Rebuffi et al. , 2017 ) . However , such methods still lack a mechanism to automatically weight more relevant past memory in the context of the new task , and also do not take uncertainty of the output into account . Methods based on Gaussian processes do this automatically ( Titsias et al. , 2019 ) , but require optimisation over inducing points and specification of a good kernel , both of which are difficult tasks . In summary , existing methods fall short in building scalable functional-regularisation methods for continual learning . In this paper , we propose a new functional-regularisation method where we regularise over a few memorable past examples ( see Fig . 1 ) . Our approach builds upon a recent method of Khan et al . ( 2019 ) that expresses deep networks as Gaussian processes ( GPs ) . We show that the GP formulation not only enables the identification of examples crucial to avoid forgetting , but also computes uncertainty over the network output to appropriately weight the past examples in the light of new After Task 1 After Task 2 After Task 5 ones . Motivated by the GP perspective , we propose a new loss function for function-regularisation with deep networks . A Laplace approximation of this objective enables scalable training . Our work in this paper focusses on avoiding forgetting , but it opens a new direction for life-long learning methods where regularisation methods are naturally combined with memory-based methods . Other related works . Broadly , existing work on continual learning can be split into three approaches : inference based , memory/rehearsal based , and model based . Inference based approaches have mostly focused on weight regularisation , with some recent efforts on functional regularisation . Our work falls in the latter category . Memory based approaches either maintain a memory of past data examples ( Rebuffi et al. , 2017 ) or train generative models on previous tasks to rehearse pseudo-inputs ( Shin et al. , 2017 ) . An advantage of our method compared to previous ones is that building memory does not require solving an optimisation problem : the computation simply involves a forward-pass through the network followed by sorting ( see Section 3.2 ) . Similarly to our work , there have also been some efforts in combining the different flavours of approaching continual learning , e.g. , VCL plus coresets ( Nguyen et al. , 2018 ) and Gradient-Episodic Memory ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018 ) . Benjamin et al . ( 2018 ) have proposed a similar combination for functional regularisation . In these approaches , two separate methods are usually used for regularisation and memory-building . Recent follow-up work ( Aljundi et al. , 2019 ; Chaudhry et al. , 2019 ) has focussed on improving memory-building methods , but this separation has remained the case . In contrast , in our approach , both of these are done within the same GP framework by using the method of Khan et al . ( 2019 ) . Finally , model based approaches change the model architecture during training ( Rusu et al. , 2016 ) , and this can be combined with other approaches ( Schwarz et al. , 2018 ) . It is possible to use similar features in our GP based framework , which is an interesting future direction to be pursued . 2 CONTINUAL LEARNING WITH WEIGHT/FUNCTIONAL REGULARISATION . In deep learning , we minimise loss functions to estimate network weights . For example , in supervised multi-class classification problems , we are given a dataset D of N input-output pairs with outputs yi , a vector of K classes , and inputs xi , a vector of length D , and our goal is to minimise a loss which takes the following form : ¯̀ ( w ) + δR ( w ) , where ¯̀ ( w ) : = 1N ∑N i=1 ` ( yi , fw ( xi ) ) with deep neural network fw ( x ) ∈ RK and its weights w. ` ( y , ŷ ) denotes a differentiable loss function between an output y and its prediction ŷ , R ( w ) is a regularisation function ( usually an L2-regulariser R ( w ) = w > w ) and δ > 0 controls the regularisation strength . Standard deep-learning approaches rely on an unbiased stochastic-gradient of the loss ¯̀ , which usually requires access to all of the data examples for all classes ( Bottou , 2010 ) . It is this unbiased , minibatch setting where deep-learning excels and achieves state-of-the-art performance on many benchmark datasets . In reality , we do not always have access to all the data at once , and it is not possible to obtain unbiased stochastic gradients . New classes may appear during training and old classes may never be seen again . For such settings , vanilla mini-batch stochastic-gradient methods lead to catastrophic forgetting of past information ( Kirkpatrick et al. , 2017 ) . Our goal in this paper is to design methods that can avoid such catastrophic forgetting . We focus on a particular setting where the classification task is divided into several tasks , e.g. , a task may consist of a classification problem over a subset of classes . We assume that the tasks arrive sequentially one after the other . Once the learning is over , we may never see that task again . Such continual-learning settings have been considered in previous works ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2018 ; Zenke et al. , 2017 ) , and our goal is to avoid forgetting of old tasks in this setting . Recent methods have proposed weight-regularisation as a way to combat catastrophic forgetting . The main idea is to keep the new network weights close to the old ones , e.g. , when training a task t while given network weights wt−1 trained on task t − 1 , we can minimise the following loss : ¯̀ t ( w ) + δ ( w −wt−1 ) > Ft ( w −wt−1 ) , where ¯̀t ( w ) is the loss defined over all data examples from task t and Ft is a preconditioning matrix that favors the weights relevant to the past tasks more than the rest . The Elastic-Weight Consolidation ( EWC ) method ( Kirkpatrick et al. , 2017 ) , for example , uses the Fisher information matrix as the pre-conditioner , while Ritter et al . ( 2018 ) use the Hessian of the loss , and VCL ( Nguyen et al. , 2018 ) essentially employs the precision matrix of the variational approximation to do the same . Such weight-space methods reduce forgetting but do not produce satisfactory results . The challenge in using weight-regularisation lies in the fact that the exact values of the weights do not really matter due to parametric symmetries ( Benjamin et al. , 2018 ; Bishop , 2006 ) . Since only the network outputs matter , an alternative approach is to directly regularise these . Benjamin et al . ( 2018 ) propose to use an L2-regulariser over the function values on data examples from past tasks : min w ¯̀ t ( w ) + δ t−1∑ s=1 ∑ i∈Ms ‖ fw ( xi ) − fwt−1 ( xi ) ‖22 , ( 1 ) where Ms is the set of small examples for task s stored in the working memory ( Lopez-Paz & Ranzato , 2017 ; Rebuffi et al. , 2017 ) . One major issue with this approach is that the L2-regulariser weights all the data points equally . In reality , some examples in the memory are more important than others to learn a given task . In addition , the uncertainty of the prediction is also ignored . As we will show , working with distributions over functions allows us to address these two issues . Gaussian processes ( GPs ) , for example , enable automatic reweighting of old tasks in light of a new one , which happens by using the posterior covariance . Unfortunately , both scalability of GPs as well as the necessity to save many instances of the past makes them impractical . A recent approach by Titsias et al . ( 2019 ) attempts to address these issue by employing sparse GP methods with inducing points and using a neural network feature map . However , their approach has some difficulties . Their approach heavily depends on a proper choice of inducing points , which are normally obtained via an ad-hoc procedure . Additionally , they propose using the last layer of the neural network as kernel features , which is limiting as it does not use the whole network ’ s weights in the kernel . Ideally , we would want to perform functional regularisation with a deep learning optimiser while borrowing ideas from the GP methods . Our work in this paper takes a step in this direction . Our proposal is to use a GP formulation of neural networks to perform functional regularisation that still allows training with standard deep-learning methods . We also show that the GP view helps in selecting an informative set of memory points . 3 FUNCTIONAL-REGULARISATION OF MEMORABLE PAST ( FROMP ) . We will now describe our proposed method . The first step is to use the GP formulation of Khan et al . ( 2019 ) to compute GP-like posteriors over functions . Then , we propose a method to identify a set of memorable past examples for a task . Finally , we describe our functional-regularisation and discuss approximations used to build a scalable method . 3.1 FROM DEEP NETWORKS TO GAUSSIAN PROCESS POSTERIORS . Khan et al . ( 2019 ) propose an approach to convert deep networks into Gaussian processes . Their main result ( see Theorem 1 in their paper ) states that , at a local minimiser w∗ of ¯̀ ( w ) + δ2 w > w , a Laplace approximation of the posterior over w is equivalent to the posterior distribution of a linear model . Specifically , they use the following scalable variant of the Laplace approximation : p ( w|D ) ≈ q ( w ) : = N ( w|µ , Σ ) , where µ = w∗ and Σ−1 = N∑ i=1 J ( xi ) > Λ ( xi , yi ) J ( xi ) + δIP , ( 2 ) where J ( xi ) : = ∇wfw ( xi ) > is a K × P Jacobian matrix ( P being the number of parameters ) , and Λ ( x , y ) : = ∇2ff ` ( y , f ) is the K × K Hessian of the loss with f = fw ( x ) , all evaluated at w = w∗ . They show that q ( w ) is equal to the posterior of the following linear model : ỹ = J ( x ) w + , with ∼ N ( 0 , ( Λ ( x , y ) ) −1 ) and w ∼ N ( 0 , δ−1IP ) , ( 3 ) where the observations are defined as ỹi : = J ( xi ) w∗ − ( Λ ( xi , yi ) ) −1 r ( xi , yi ) with the residual r ( x , y ) : = ∇f ` ( y , f ) . They also show that the predictive distribution of this linear model is equivalent to that of a GP regression model defined with a K × K neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) : ỹ = fGP ( x ) + , with ∼ N ( 0 , ( Λ ( x , y ) ) −1 ) and fGP ( x ) ∼ GP ( 0 , δ−1J ( x ) J ( x′ ) > ) . ( 4 ) The above result gives a posterior in the defined observation space of ỹ . In the case where we can find a map from ỹ to y , the above inference problem allows for reparameterisation into the original data space . Therefore , we will obtain a GP that serves as an approximation to the neural network posterior . Below , we demonstrate this for binary classification with sigmoid link function and Bernoulli likelihood ( see Appendix A for details and extension to multi-class classification ) . We parameterise the Bernoulli likelihood by p ( x ) : = σ ( fw∗ ( x ) ) where σ is the sigmoid function ( note that we now have scalars { f , y , ... } as we are dealing with a single output ) . The residual and loss Hessian for this model are given by r ( x , y ) = p ( x ) − y and Λ ( x , y ) = p ( x ) ( 1− p ( x ) ) , respectively , where y ∈ { 0 , 1 } . Because residuals are linear in y and the Hessian is independent of y , we can substitute the definitions of ỹ and r ( x , y ) into Eq . 3 , and rearrange for y , y = p ( x ) + Λ ( x , y ) J ( x ) ( w − w∗ ) ︸ ︷︷ ︸ : =flin ( x ) +τ , with τ ∼ N ( 0 , Λ ( x , y ) ) and w ∼ N ( 0 , δ−1IP ) . ( 5 ) The Laplace approximation in Eq . 2 is the posterior of this model after observing data D. We can equivalently write the posterior predictive as a GP ( Rasmussen , 2003 ) with mean and covariance m ( x ) : = Eq ( w ) [ flin ( x ) ] = p ( x , y ) + Λ ( x , y ) J ( x ) ( µ − w∗ ) = p ( x ) , ( 6 ) k ( x , x′ ) : = Eq ( w ) [ ( flin ( x ) −m ( x ) ) ( flin ( x′ ) −m ( x′ ) ) ] = Λ ( x , y ) J ( x ) Eq ( w ) [ ( w − w∗ ) ( w − w∗ ) > ] J ( x′ ) > Λ ( x′ , y′ ) = Λ ( x , y ) J ( x ) ΣJ ( x′ ) > Λ ( x′ , y′ ) . ( 7 ) A GP defined with the above mean and covariance function can be viewed as an approximation to the posterior process over network outputs . Throughout the paper , we will denote the process obtained at a weight w∗ by a distribution qw∗ ( f ) where f is a vector of f ( x ) evaluated at many different inputs x . We will use this GP posterior predictive for functional regularisation .

Summary: The paper proposes a method for continuous learning called Functional Regularization of Memorable Past (FROMP) which maintains the output distribution of models on memory samples. FROMP uses the Laplace approximation and Gaussian process with neural tangent kernel (NTK) to approximate the output distribution. According to the leverage score strategy, the sample to be stored is selected. The leverage score strategy tends to select the sample of highest variances.