Functional Regularisation for  Continual Learning with Gaussian Processes

1 INTRODUCTION . Recent years have seen a resurgence of interest in continual learning , which refers to systems that learn in an online fashion from data associated with possibly an ever-increasing number of tasks ( Ring , 1994 ; Robins , 1995 ; Schmidhuber , 2013 ; Goodfellow et al. , 2013 ) . A continual learning system must adapt to perform well on all earlier tasks without requiring extensive re-training on previous data . There are two main challenges for continual learning ( i ) avoiding catastrophic forgetting , i.e . remembering how to solve earlier tasks , and ( ii ) scalability over the number of tasks . Other possible desiderata may include forward and backward transfer , i.e . learning new tasks faster and retrospectively improving on previously tasks . Similarly to many recent works on continual learning ( Kirkpatrick et al. , 2017 ; Nguyen et al. , 2017 ; Rusu et al. , 2016 ; Li & Hoiem , 2017 ; Farquhar & Gal , 2018 ) , we focus on the scenario where a sequence of supervised learning tasks are presented to a continual learning system based on a deep neural network . While most methods assume known task boundaries , our approach will be also extended to deal with unknown task boundaries . Among the different techniques proposed to address this problem , we have methods which constrain or regularise the parameters of the network to not deviate significantly from those learnt on previous tasks . This includes methods that frame continual learning as sequential approximate Bayesian inference , including EWC ( Kirkpatrick et al. , 2017 ) and VCL ( Nguyen et al. , 2017 ) . Such approaches suffer from brittleness due to representation drift . That is , as parameters adapt to new tasks the values that other parameters are constrained/regularised towards become obsolete ( see Section 2.5 for further discussion on this ) . On the other hand , we have rehearsal/replay buffer methods , which use a memory store of past observations to remember previous ∗Equal contribution tasks ( Robins , 1995 ; Robins & McCallum , 1998 ; Lopez-Paz et al. , 2017 ; Rebuffi et al. , 2017 ) . While these methods tend to not suffer from brittleness , uncertainty about the unknown functions is not expressed . Furthermore , they rely on various heuristics to decide which data to store ( Rolnick et al. , 2018 ) , often requiring large quantities of stored observations to achieving good performance . In this paper we will address the open problem of deriving an optimisation objective to select the best observations for storage . In this paper , we develop a new approach to continual learning which addresses the shortcomings of both categories . It is based on approximate Bayesian inference , but on the space of functions instead of neural network parameters , so does not suffer from the aforementioned brittleness . Intuitively , while previous approaches constrain the parameters of a neural network to limit deviations from previously learnt parameters , our approach instead constrains the neural network predictions from deviating too far from those that solve previous tasks . Effectively , our approach avoids forgetting an earlier task by memorising an approximate posterior belief over the underlying task-specific function . To implement this , we consider Gaussian processes ( GPs ) ( Rasmussen & Williams , 2005 ) , and make use of inducing point sparse GP methods ( Csato & Opper , 2002 ; Titsias , 2009 ; Hensman et al. , 2013 ; Bui et al. , 2017b ) , which summarise posterior distributions over functions using small numbers of so-called inducing points . These inducing points are selected from the training set by optimising a variational objective , providing a principled mechanism to compress the dataset to a meaningful subset of fixed size . They are kept around when moving to the next task and , together with their posterior distributions , are used to regularise the continual learning of future tasks , through Kullback-Leibler regularisation terms within a variational inference framework , thus avoiding catastrophic forgetting of earlier tasks . Our approach bears similarities to replay-based approaches , with inducing points playing the role of the rehersal/replay buffer , but has two important advantages . First the approximate posterior distributions at the inducing points captures the uncertainty of the unknown function as well , rather than providing merely target values . Second , inducing points can be optimised using specialised criteria from the GP literature , achieving better performance than a random selection of observations . An intuitive depiction of our approach is given in Figure 1 . To enable our functional regularisation approach to deal with high-dimensional and complex datasets , we use a linear kernel with features parameterised by neural networks ( Wilson et al. , 2016 ) . Such GPs can be understood as Bayesian neural networks , where only the weights of the last layer are treated in a Bayesian fashion , while those in earlier layers are optimised . This view allows for a more computationally efficient and accurate training procedure to be carried out in weight space , before the approximation is translated into function space where the inducing points are constructed and then used for regularising learning of future tasks . Finally , note that inducing points are also used to regularize the deep network , even though they were selected to best represent functions given by the GP . See Section 2.3 for further details . 2 FUNCTIONAL REGULARISATION FOR CONTINUAL LEARNING . We consider supervised learning of multiple tasks , with known task boundaries , that are processed sequentially one at a time . At each step we receive a set of examples ( Xi , yi ) where Xi = { xi , j } Nij=1 are input vectors and yi = { yi , j } Nij=1 are output targets so that each yi , j is assigned to the input xi , j ∈ RD . We assume the most extreme case ( and challenging in terms of avoiding forgetting ) where each dataset ( Xi , yi ) introduces a new task , while less extreme cases can be treated similarly . We wish to sequentially train a shared model or representation from all tasks so that catastrophic forgetting is avoided , i.e . when the model is trained on the i-th task it should still provide accurate predictions for all tasks j < i seen in the past . As a model we consider a deep neural network with its final hidden layer providing the feature vector φ ( x ; θ ) ∈ RK where x is the input vector and θ are the model parameters . This representation is shared across tasks and θ is a task-shared parameter . To solve a specific task i we additionally construct an output layer fi ( x ; wi ) ≡ fi ( x ; wi , θ ) = w > i φ ( x ; θ ) , ( 1 ) where for simplicity we assume that fi ( x ; wi ) is a scalar function and wi is the vector of task-specific weights . Dealing with vector-valued functions is straightforward and is discussed in the Appendix . By placing a Gaussian prior on the output weights , wi ∼ N ( wi|0 , σ2wI ) , we obtain a distribution over functions . While each task has its own independent/private weight vector wi the whole distribution refers to the full infinite set of tasks that can be tackled by the same feature vector φ ( x ; θ ) . We can marginalise out wi and obtain the equivalent function space view of the model , where each task-specific function is an independent draw from a GP ( Rasmussen & Williams , 2005 ) , i.e . fi ( x ) ∼ GP ( 0 , k ( x , x′ ) ) , k ( x , x′ ) = σ2wφ ( x ; θ ) > φ ( x′ ; θ ) , where the kernel function is defined by the dot product of the neural network feature vector . By assuming for now that all possible tasks are simultaneously present similarly to multi-task GPs ( Bonilla et al. , 2008 ; Álvarez et al. , 2012 ) , the joint distribution over function values and output data for all tasks is written as ∏ i p ( yi|fi ) p ( fi ) = ∏ i p ( yi|fi ) N ( fi|0 , KXi ) , where the vector fi stores all function values for the input dataset Xi , i.e . fi , j = f ( xi , j ) , j = 1 , . . . , Ni . Also the kernel matrix KXi is obtained by evaluating the kernel function on Xi , i.e . each element [ KXi ] j , k = σ 2 wφ ( xi , j ; θ ) > φ ( xi , k ; θ ) where xi , j , xi , k ∈ Xi . The form of each likelihood function p ( yi|fi ) depends on the task , for example if the i-th task involves binary classification then p ( yi|fi ) = ∏Ni j=1 p ( yi , j |fi , j ) = ∏Ni j=1 1 1+e−yi , jfi , j where yi , j ∈ { −1 , 1 } indicates the binary class label . Inference in this model requires estimating each posterior distribution p ( fi|yi , Xi ) , which can be approximated by a multivariate Gaussian N ( fi|µi , Σi ) . Given this Gaussian we can express our posterior belief over any function value fi , ∗ at some test input xi , ∗ using the posterior GP ( Rasmussen & Williams , 2005 ) , p ( fi , ∗|Xi , yi ) = ∫ pθ ( fi , ∗|fi ) N ( fi|µi , Σi ) dfi . Given that the tasks arrive one at a time , the above suggests that one way to avoid forgetting the i-th task is to memorise the corresponding posterior belief N ( fi|µi , Σi ) . While this can regularise continual learning of subsequent tasks ( similarly to the more general variational framework in the next section ) , it can be prohibitively expensive since the non-parametric nature of the model means that for each N ( fi|µi , Σi ) we need to store O ( N2i ) parameters and additionally we need to keep in memory the full set of input vectors Xi . Therefore , in order to reduce the time and memory requirements we would like to apply data distillation and approximate each full posterior by applying sparse GP methods . As shown next , by applying variational sparse GP inference ( Titsias , 2009 ) in a sequential fashion we obtain a new algorithm for function space regularisation in continual learning . 2.1 LEARNING THE FIRST TASK . Suppose we encounter the first task with data ( X1 , y1 ) . We introduce a small set Z1 = { z1 , j } M1j=1 of inducing inputs where each z1 , j lives in the same space as each training input x1 , j . The inducing set Z1 can be a subset of X1 or it can contain pseudo inputs ( Snelson & Ghahramani , 2006 ) , i.e . points lying between the training inputs . For simplicity next we consider Z1 as pseudo points , although in practice for continual learning it can be more suitable to select them from the training inputs ( see Section 2.4 ) . By evaluating the function output at each z1 , j we obtain a vector of auxiliary function values u1 = { u1 , j } M1j=1 , where each u1 , j = f ( z1 , j ) . Hence , we obtain the joint distribution p ( y1 , f1 , u1 ) = p ( y1|f1 ) pθ ( f1|u1 ) pθ ( u1 ) . ( 2 ) The exact posterior distribution pθ ( f1|u1 , y1 ) pθ ( u1|y1 ) is approximated by a distribution of the form , q ( f1 , u1 ) = pθ ( f1|u1 ) q ( u1 ) , where q ( ui ) is a variational distribution and pθ ( f1|u1 ) is the GP prior conditional , pθ ( f1|u1 ) = N ( f1|KX1Z1K−1Z1 u1 , KX1 − KX1Z1K −1 Z1 KZ1X1 ) . Here , KX1Z1 is the cross kernel matrix between the sets X1 and Z1 , KZ1X1 = K > X1Z1 and KZ1 is the kernel matrix on Z1 . The method learns ( q ( u1 ) , Z1 ) by minimising the KL divergence KL ( pθ ( f1|u1 ) q ( u1 ) ||pθ ( f1|u1 , y1 ) pθ ( u1|y1 ) ) . The ELBO is also maximised over the neural network feature vector parameters θ that determine the kernel matrices . This ELBO is generally written in the form ( Hensman et al. , 2013 ; Lloyd et al. , 2015 ; Dezfouli & Bonilla , 2015 ; Hensman et al. , 2015 ; Sheth et al. , 2015 ) , F ( θ , q ( u1 ) ) = N1∑ j=1 Eq ( f1 , j ) [ log p ( y1 , j |f1 , j ) ] − KL ( q ( u1 ) ||pθ ( u1 ) ) , ( 3 ) where q ( f1 , j ) = ∫ p ( f1 , j |u1 ) q ( u1 ) du1 is an univariate Gaussian distribution with analytic mean and variance that depend on ( θ , Z1 , q ( u1 ) , x1 , j ) . Each expectation Eq ( f1 , j ) [ log p ( y1 , j |f1 , j ) ] is a one-dimensional integral and can be estimated by Gaussian quadrature . The variational distribution q ( u1 ) is chosen to be a Gaussian , parameterised as q ( u1 ) = N ( u1|µu1 , Lu1L > u1 ) , where Lu1 is a square root matrix such as a lower triangular Cholesky factor . Then , based on the above we can jointly apply stochastic variational inference ( Hensman et al. , 2013 ) to maximise the ELBO over ( θ , µu1 , Lu1 ) and optionally over the inducing inputs Z1 .

Summary: The paper develops a continual learning method based on Gaussian Processes (GPs) applied in the way introduced by prior work as Deep Kernel Learning (DKL). The proposed method summarizes tasks as sparse GPs and use them as regularizers for the subsequent tasks in order to avoid catastrophic forgetting. Salleviating the instability resulting from the representation drift.