Functional Regularisation for  Continual Learning with Gaussian Processes
Authors: Michalis K. Titsias
Jonathan Schwarz
Alexander G. de G. Matthews
Razvan Pascanu
Yee Whye Teh
Venue:  (2020)
Keywords: Continual Learning
Gaussian Processes
Lifelong learning
Incremental Learning

Abstract: We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred to as functional regularisation for Continual Learning, avoids forgetting a previous task by constructing and memorising an approximate posterior belief over the underlying task-specific function. To achieve this we rely on a Gaussian process obtained by treating the weights of the last layer of a neural network as random and Gaussian distributed. Then, the training algorithm sequentially encounters tasks and constructs posterior beliefs over the task-specific functions by using inducing point sparse Gaussian process methods. At each step a new task is first learnt and then a summary is constructed consisting of (i) inducing inputs – a fixed-size subset of the task inputs selected such that it optimally represents the task – and (ii) a posterior distribution over the function values at these inputs. This summary then regularises learning of future tasks, through Kullback-Leibler regularisation terms. Our method thus unites approaches focused on (pseudo-)rehearsal with those derived from a sequential Bayesian inference perspective in a principled way, leading to strong results on accepted benchmarks.

Published as a conference paper at ICLR 2020
FUNCTIONAL REGULARISATION FOR CONTINUAL LEARNING WITH GAUSSIAN PROCESSES
Michalis K. Titsias"
Jonathan Schwarz' DeepMind & University College London schwarzjn@googl1 com
DeepMind
mtitsi as@goooleeee com
Alexander G. de G. Matthews
Razvan Pascanu
Yee Whye Teh DeepMind ywtet?ggoogle.com
DeepMind al alexmatthees@@oooll.e com
DeepMind
razpegoogle.com
ABSTRACT
We introduce a framework for Continual Learning (CL) based on Bayesian infer- ence over the function space rather than the parameters of deep neural network. This method, referred to functional regularisation for Continual Learning, avoids forgetting previous task by constructing and memorising an approximate posterior belief over the underlying -ask-specific function. To achieve this we rely on a Gaus- sian process obtained by treating the weights of the last layer of a neural network as random and Gaussian distributed. Then, the training algorithm sequentially encounters tasks and constructs posterior beliefs over the task-specith functions by using inducing point sparse Gaussian process methods. At each step a new task is first learnt and then summary is constructed consisting of (i) inducing inputs fixed-size subset of the task inputs selected such that optimally represents the task and (ii) a posterior distribution over the function values these inputs. This summary then regularises learning of future tasks, through Kullback-Leible regu- larisation terms. Our method thus unites approaches focused on (pseudo-Jrehearsal with those derived from sequential Bayesian inference perspective in principled way, leading to strong results on accepted benchmarks.
INTRODUCTION
Recent years have seen resurgence of interest in continual learning, which refers to systems that learn in an online fashion from data associated with possibly an ever-increasing number of tasks (Ring) 1994 Robins 1995] Schmidhuber] 2013] Goodfellow et al.) 2013). A continual learning system must adapt to perform well on all earlier tasks without requiring extensive re-training on previous data. There two main challenges for continual learning (i) avoiding catastrophic forgetting, i.e. remembering how to solve earlier tasks, and (ii) scalability over the number of tasks. Other possible desiderata may include forward and backward transfer, i.e. learning new tasks faster and retrospectively improving on previously tasks.
Similarly many recent works on continual learning (Kirkpatrick al.) 2017; Nguyen al. 2017] Rusu et al. 2016 Li & Hoiem, 2017, Farquhar & Gal (018)), we focus on the scenario where sequence supervised learning tasks are presented a continual learning system based on a deep neural network. While most methods assume known task boundaries, our approach will be also extended to deal with unknown task boundaries. Among the different techniques proposed to address this problem, have methods which constrain or regularise the parameters of the network to not deviate significantly from those learnt on previous tasks. This includes methods that frame continual learning sequential approximate Bayesian inference, including EWC (Kirkpatrick et al.) 2017) and VCL. (Nguyen et al. 2017). Such approaches suffer from brittleness due to representation drift. tasks the values that other parameters
That is, as parameters adapt towards become obsolete (see Section 2.5 for further discussion on this), On the other hand, we have rehearsal/replay buffer methods, which memory store of past observations remember previous
constrained/regulriise
* Eqqual contribution

---

Figure 1: Depiction of the proposed approach. See also the provided pseudocode. When learning task 1, first, parameters of the network and oulput layer are fitted (Panel A). Afterwards, the learned GP is sparsified and inducing points u1,.. are found (Panel B). When moving to the next task the same steps are repeated. The only difference is that now the previously found summaries (in this case points u1, Us) are used to regularise the Published as a conference paper at ICLR 2020
TASK 1
TASK 2 C2(x, y - [L(g(u))lp(u))
C yl))
((x)
A.
function (via KL-divergence term), such that the first task is not forgotten.
tasks (Robins 1995
Robins& & McCallum 1998 Lopez-Paz et al. 2017 Rebuffi et al. 2017). While
these methods tend to not suffer from brittleness, uncertainty about the unknown functions is not expressed. Furthermore, they rely various heuristics to decide which data to store (Rolnick et al., 2018). often requiring large quantities of stored observations to achieving good performance. In this paper we will address the open problem of deriving an optimisation objective to select the best observations for storage.
In this paper, we develop a new approach continual learning which addresses the shortcomings of both categories. It is based on approximate Bayesian inference, but on the space of functions instead of neural network parameters, so does not suffer from the aforementioned brittleness. Intuitively, while previous approaches constrain the parameters of neural network to limit deviations from previously learnt parameters, our approach instead constrains the neural network predictions from deviating too far from those that solve previous tasks.
Effectively, our approach avoids forgetting an earlier task by memorising an approximate posterior belief over the underlying task-specific function. To implement this, we consider Gaussian processes (GPs) (Rasmussen & Williams (2005), and make use of inducing point sparse GP methods (Csato & Opper [0022] Titsias 2009 Hensman et al. 2013 Bui et al. (017b), which summarise posterior distributions over functions using small numbers of so-called inducing points. These inducing points are selected from the training set by optimising a variational objective, providing a principled mechanism to compress the dataset to a meaningful subset of fixed size. They are kept around when moving the next task and, together with their posterior distributions, are used regularise the continual learning of future tasks, through Kullback-LLeiller regularisation terms within a variational inference framework, thus avoiding catastrophic forgetting of earlier tasks. Our approach bears similarities to replay-based approaches, with inducing points playing the role of the rehersalrrelaay buffer, but has two important advantages. First the approximate posterior distributions at the inducing points captures the uncertainty of the unknown function as well, rather than providing merely target values. Second, inducing points can be optimised using specialised criteria from the GP literature, achieving better performance than a random selection of observations. An intuitive depiction of our approach is given in Figure T
To enable our functional regularisation approach to deal with high-dimensional and complex datasets, we use a linear kernel with features parameterised by neural networks (Wilson et al., 2016). Such GPs can be understood as Bayesian neural networks, where only the weights of the last layer are treated in a Bayesian fashion, while those in earlier layers are optimised. This view allows for a more computationally efficient and accurate training procedure be carried out in weight space, before the approximation is translated into function space where the inducing points are constructed and then used for regularising learning of future tasks. Finally, note that inducing points also used to regularize the deep network, even though they were selected to best represent functions given by the GP. See Section 2.3 for further details.
FUNCTIONAL REGULARISATION FOR CONTINUAL LEARNING
We consider supervised learning of multiple tasks, with known task boundaries, that are processed sequentially one at time. At each step we receive set of examples (x) where X; (2 No;
2

---

Published as a conference paper at ICLR 2020
imput vectors and y, {vk, )را are output targets that each Ut,j is assigned to the input Fi, E RRD. We assume the most extreme case (and challenging in terms of avoiding forgetting) where each dataset (X,,):) introduces a new task, while less extreme cases can be treated similarly. We wish to sequentially train a shared model or representation from all tasks so that catastrophic forgetting is avoided, i.e. when the model is trained on the i-th task it should still provide accurate predictions for all tasks j < ? seen in the past. As a model we consider a deep neural network with its final hidden layer providing the feature vector (22; 0) R where is the input vector and 0 are the model parameters. This representation is shared across tasks and 0 is a task-shared parameter. To solve specific task we additionally construct output layer
(1) where for simplicity assume that fx;; wy) is scalar function and w; is the vector of task-specific weights. Dealing with vector-valued functions is straightforward and is discussed in the Appendix. By placing Gaussian prior on the output weights, w; (wx/0, 1) we obtain distribution over functions. While each task has its own independent/private weight vector wi the whole distribution refers to the full infinite set of tasks that can be tackled by the same feature vector o(x; 0). We can marginalise out wi and obtain the equivalent function space view of the model, where each task-specific function is an independent draw from a GP (Rasmussen & Williams 2005), i.e.
fil2; wx) = fa w;, 0) wo ((z;
[((x)) ~ GP(O, k(x,)))) k(xx,x²) oa 0)* (T(())((())) O)x
where the kernel function is defined by the dot product of the neural network feature vector. By assum- ing for now that all possible tasks are simultancously present similarly to multi-task GPs (Bonilla et al. 2008; Álvarez et al. 2012). the joint distribution over function values and output data for all tasks is written as II, p(y: Jp((.) II. (C., Kx,), where the vector f; stores all function values for the input dataset X,, i.e. faj = f = ...... N₁. Also the kernel matrix Kx, is obtained by evaluating the kernel function on X,, i.e. each element [Kx,] = o (= 0) 6(2,,x: f) where F F E X, The form of each likelihood function p(y, [f.) depends on the task, for example if the
i-th task involves binary classification then p(y:(()) = II 1=1 p(u. (1,z) 11 Ite 25.3 where
Vi.j E 1-1, indicates the binary class label.
Inference in this model requires estimating each posterior distribution p(f, X²), which can be approximated by multivariate Gaussian A/ (f. E.). Given this Gaussian we can express our posterior belief over any function value Si, at some test input Zi,- using the posterior GP (Rasmussen & Williams 2005),
p(Jr, X., vi) = ro
E.)df,
Given that the tasks arrive one at a time, the above suggests that one way to avoid forgetting the i-th task is to memorise the corresponding posterior belief.N(f.jpu.. (fflp Σ,). While this can regularise continual learning of subsequent tasks (similarly to the more general variational framework in the next section), it can be prohibitively expensive since the non-pan -orramametric nature of the model means that for each (f, ju, Σx) we need to store O(N?) parameters and additionally we need to keep in memory the full set of input vectors X,-
Therefore, in order to reduce the time and memory requirements we would like to apply data distillation and approximate each full posterior by applying sparse GP methods. As shown next, by applying variational sparse GP inference (Titsias) 2009) in sequential fashion obtain new algorithm for function space regularisation in continual learning.
2.1 LEARNING THE FIRST TASK
Suppose we encounter the first task with data (1). We introduce a small set Z, = ( M of inducing inputs where each ži.j lives in the same space as each training input æi,j. The inducing set Z₁ can be a subset of X1 or it can contain pseudo inputs (Snelson & Ghahramani] 2006). i.e. points lying between the training inputs. For simplicity next we consider 21 as pseudo points, although in practice for continual learning it can be more suitable to select them from the training inputs (see Section 2.4 By evaluating the function output at each 21,j we obtain vector of auxiliary function
values U1 = {u1,5 1==1
M,
where each U1.j = f(1,)) Hence, we obtain the joint distribution
p(y1, fi, un) = p(y: [f) )o [u, ( (poo))oo)) (u:).
3

---

Hensman al.. Published as a conference paper at ICLR 2020
The exact posterior distribution pe (f( /n y1 po)po(uullu))) (u)l() is approximated by distribution of the form, (( u1) = (f [1)q (u1), where qu,)) is a variational distribution and (o(filu) is |1) = (K 2, 22, u1, - Kx 2x, K2,x,) Here, Kx,2, is the cross kernel matrix between the sets X1 and Z1, K2,x, = K x, and Kz, is the kemne matrix on Z1. The method learns 1,(uz).24 Z1) by minimising the KL divergence (L(po [n ( (lpoo) [n Y1 yi (p [y))). The ELBO is also maximised over the neural net- work feature vector parameters that determine the kernel matrices. This ELBO is generally written in the form (Hensman et al., 2013: Lloyd et al. 2015 Dezfouli & Bonilla 2015
the GP prior conditional,
2015 Sheth et al. 2015).
N1 F(0,g(uz)) = Σ Dogttw (L(((u)))
where ((fi.) = p(fi.lu (u) )du, is an univariate Gaussian distribution with analytic mean and variance that depend on (0, Z1, (( x1.). Each expectation Egth, ,) [log fi.s)] is one-dimensional integral and can be estimated by Gaussian quadrature. The variational distribution q(uz) is chosen be a Gaussian, parameterised as q(u1) ((ul/aa Lu, L where La, is a square root matrix such as a lower triangular Cholesky factor. Then, based on the above we can jointly apply stochastic variational inference (Hensman al. 2013) to maximise the ELBO over (0, pmg; I and optionally over the inducing inputs Z1.
2.2 LEARNING THE SECOND AND SUBSEQUENT TASKS
The functional regularisation framework for continual learning arises from the variational sparse GP inference method as we encounter the second and subsequent tasks.
Once we have learned the first task we throw away the dataset (11,y1) and we keep in memory only task summary consisting of the inducing inputs Zi and the variational Gaussian distribution q(uz) (i.e. its parameters fou and L.,). Note also that 0 (that determines the neural network feature vector ((x; f)) has a current value obtained by learning the first task. When the dataset (X2. V2) for the second task arrives, a suitable ELBO to continue learning 0 and also estimate the second task summary (Zz, ((u2)) is
Σ Ba(sa.)) Dogr(e222g
Σ KL(g(u;)) (u:)),
which is just the sum of the corresponding ELBOs for the two tasks. We need to approximate this ideal objective by making use of the fixed summary (Zi, q(u)) )) that we have kept in memory for the first task. By considering Z1 X1 our replay buffer with outputs ý1 y and ũj f, the above can be approximated by
N1 Mi
Eg(u llg.l (1 +
gg(s, [log plo2. Σ KL(q(u:)lpo(uu)
where each ((u1,)) ) is univariate marginal of q(u)). However, since quuz) is kept fixed the whole Nj
expected log-likelihood term M. Σ Flg(u..)) [log ((rr,((1..)) is just a constant that does not depend on the parameters 0 any more. Thus, the objective function when learning the second task reduces maximising.
F(0.g(u2)) = Raus Dog
Σ (L.((u)))loouu.)) 1,2
The only term associated with the first task is (L(((uu)]lPo
While quu)) is fixed (i.e. its parameters are constant), the GP prior (o( = O( K2,) is still a function of the feature vector parameters 0, since K2, depends on 0. Thus, this KL term regularises the parameters 0 so that, while learning the second task, the feature vector still needs to explain the posterior distribution over the function values uj at input locations Z1. Notice that (L, (ppo))))))))))) is further simplified as J (qu)) log po(u, (u, )du, + const, which shows that the regularisation is such that po (u) needs to be consistent with all infinite draws from q(u)) in moment-matching maximum likelihood sense.
4

---

Published as a conference paper at ICLR 2020
Similarly for the subsequent tasks conclude that for any new task & the objective will be
F(0,g(uk)) Balses log ((vsssl) (I=(( (ppo())))
(L(g(m)))loo(;))) (4)
objectiive for lask
regularisation from previous tasks
Thus, functional regularisation when learning a new task is achieved through the sum of the KL divergences Σx=1 (L( (e()))) of all previous tasks, where each q(u.) ) is the fixed posterior distribution which encodes our previously obtained knowledge about task i k. Furthermore, in order to keep the optimisation scalable over tasks, we can form unbiased approximations of this latter sum by sub-sampling the KL terms, i.e. by performing minibatch-baa based stochastic approximation over the regularisation terms associated with these previous tasks.
2.3 ACCURATE WEIGITT SPACE INFERENCE FOR THE CURRENT TASK
While the above framework arises by applying sparse GP inference, it can still be limited. When the budget of inducing variables is small, the sparse GP approximation may lead inaccurate estimates of the posterior belief gf ((u)), which will degrade the quality of regularisation when learning new tasks. This is worrisome in continual learning it desirable to keep the size of the inducing set as small as possible.
One way to deal with this issue is to use much larger set of inducing points for the current task even maximise the full GP ELBO Σj= By(se.s) loggolpss ((ssss/Es)) (L po (f.) (i.e. by using as many inducing points training examples), and once training is completed to distill the small subset Zz. U₁ C Xx. fa. and the corresponding marginal distribution q( ((uR) from ((fx). for subsequently regularising continual learning. However, carrying out this maximisation in the function space be extremely slow since it scales O(N2) per optimisation step. To our rescue, there is an alternative computationally efficient way to achieve this, by relying on the linear form of the kernel, that performs inference over the current task in the weight space. While this inference does not immediately provides us with the summary (induced points) for building the functional regularisation term, we can distill this term afterwards as discussed next. This allows to address the continual learning aspect of the problem. Given that the current k-th task is represented the weight space as fo (z: wx) W. (xx; 0), wy ~ (0, o² [). we introduce full Gaussian variational approximation ((w)) = N (WillPa Ew,, where f4 is a dimensional mean vector and Ev,, is the corresponding K X K full covariance matrix parameterised as Ewe = Lug LTg. Learning the k-th task is carried
by maximising the objective in equation 4 with the only difference that the ELBO for the current task now in the weight space. The objective becomes
No.
((w)) Σ
Ba(we (log p(ye /un ((rns 0))] (L(((w))ll(wxz)
(l
where By(ws) [logg)]oog/ p(u.. i /w o(Ik.j 0)) can be re-written as one-dimensional integral and estimated using Gaussian quadrature. Once the variational distribution g(wx) has been optimised, together with the constantly updated feature parameters 0, we can rely on this solution to select inducing points Z See Section 2.4 for more detail. We also compute the posterior distribution over their function values u, according to ((u)) = A -uuullpu : Lug L..). where
(5) and the matrix Z, stores as rows the feature vectors evaluated Z4- Subsequently, we store the k-th task summary (Z,, Pag- Tena) ) and use it for regularising continual learning of subsequent tasks, by always maximising the objective F(0, q(w)). Pseudo-code of the procedure is given Algorithm 1.
Mwg. = 17 Mwy, Los = 17 Lwy
2.4 SELECTION OF THE INDUCING POINTS
After having seen the k-th task, and given that it is straightforward to compute the posterior distribution ((u)) for any set of function values, the only issue remaining is to select the inducing inputs Z A simple choice that works well in practice is to select Zg as a random subset of the training inputs X₃-- The question is whether we can do better with some more structured criterion.
5

---

Published as a conference paper at ICLR 2020
Algorithm 1 Functional Regularised Continual Learning (FRCL) with task boundary detection
Feature vector o(2; 8) with initial value of A, task k 0, starting time(k) 10. Construct output weights wp and initialise variational parameters Mung (e.g. around zero) and Lune = I. for / 1,2..... do
Input:
Receive next data minibatch Yc)-
Compute KL values & { J. for any zt, E X,, if' t starting_ting (ime min min_time_ (aime_in(k) and StatisticaTess((e. lota) is significant then # new task has been detected.
Select inducing inputs Zµ for current task.
Compute the parameters of g(us) from equation 5 and store them.
Construct new output weights We+1 and variational parameters (Nrwx (12 Lonk 12)
k k starting (iiee())m =
else
fold = L. end if
Gradient step to update (0, M Lwe) by maximising F(0, g(w))). end for
In our experiments we will investigate several criteria where the most effective one will be an unsupervised criterion that only depends on the training inputs, while the other supervised criteria are described in the Appendix. This unsupervised criterion quantifies how well we reconstruct the full kernel matrix Kx, from the inducing set Z₁ and it can be expressed as the trace of the covariance matrix of the prior GP conditional p(fs [u)).
p J.....i.. i.e.
No. (Z)g)
T(Z,) = tr (Kx. Kx, KZ K Kz.x,)
Σ 1=1
K2 ana.s K
KZ k
(6)
where each k( Fk..) - k2 .... =.., Kzz K kze 2 is reconstruction error for an individual
training point. The above quantity appears the ELBO in Titsias 2009), is also used in (Csato & Opper 2002) and it has deep connections with the Nystróm approximation (Williams & Seeger (001
and principal component analysis. The criterion equation promotes finding inducing points Z
that are repulsive with another and spread evenly in the input space under similarity/distanee
implied by the dot product of the feature vector p(;; 02) (with O2 being the current parameter values after having trained with the k-th task). An illustration of this repulsive property given in Section 4
To select Zc. we minimise T(Z)) by applying discrete optimisation where we select points from the training inputs Xę. The specific optimisation strategy we use in the experiments is to start with an initial random set Za C X and then further refine it by doing local moves where random points in Z, are proposed be changed with random points of X
2.5 PREDICTION AND DIFFERENCES WITH WEIGHT SPACE METHODS
Prediction at any i-th task that has been encountered in the past follows the standard sparse GP predictive equations. Given a test data point T,,- the predictive density of its output value yr takes the form p(1i,))
(.f.,, (f uu (u.)(u)) go (J,,,) -N((,, o is an univariate posterior Gaussian with mean and variance,
p(v,,~ (fi.. (oo.lfff))))(ff.. (fo(f..)))) )df.. where
Hi, = -uu ",T,K% _. - 22 (Zz,, 0). o². ((Ii,,, kT KZ [L,, LT K2,] K kz....
where in Mi.. we have explicitly written the cross kernel vector kz,,,,, b( 0) (assuming o 1 for simplicity) to reveal crucial property of this prediction. Specifically, given that f.. w. p( f) the vector pn Kー D acts as mean prediction for the task parameter row vector w... As we learn future tasks and the parameter 9 changes, this mean parameter prediction automatically adapts (recall that Kz,- Φ T 2; and Φ2, vary with and only Mh, is constant) in order to counteract changes in the feature vector o(x;,.; f), so that the overall prediction for the function value, i.c. fi.. = El does not become obsolete. For instance, the prediction of the function values at the inducing inputs Z, always remains constant fixed/stored mean belief pu

---

Published as a conference paper at ICLR 2020
KL >> 0
KL Rz
atseenacions
Prior
Task Observat lnn
Figure Detecting task boundaries using the predictive uncertainty of a Gaussian Process. As GP predictions revert to the prior (shaded blue) when queried from observed data (shown as black dots), we can test a distribution shift by comparing the GP posterior over functions (in green) to the prior. Small distance between predictive distributions at test points (red dots) suggest a task switch.
since by setting vH,-- Zy the formula gives Hu, K2, $2 12 p...- Similar observations can be made for the predictive variances.
The above analysis reveals important difference between continual learning in function space and in weight space, where in the latter framework task-specific parameters such as uy might not automatically adapt to counteract changes in the feature vector dbx;; 0), and 0 changes. For instance, if as a summary of the task, instead of the function space posterior distribution q(u)). we had kept in memory the weight space posterior (ww;) (see Section then the corresponding mean prediction on the function value, [[fi,.] 1 (0 (Zi,=; 0), can become obsolete as Φ (Ti..; () changes and fw,, remains constant.
learn new tasks
DETECTING TASK BOUNDARIES USING BAYESIAN UNCERTAINTIES
So far have made the assumption that task switches are known, which may not always be a realistic setting. Instead, we now introduce novel approach for detecting task boundaries in continual learning arising naturally from our method by a simple observation: The GP predictive uncertainty grows as the model is queried far away from observed data, eventually falling back to the prior. When minibatch of data {zi, yi 3 from new task arrives, we thus expect the distance between prior and posterior to be small (see Figure 2}. Thus, simple way detect change in the input distribution is to compare the GP univariate posterior density
a(f(z)) = N((f(=))) v ((f(()))) X))(()))(((((ff((ee. X)d((z)).
where (mi, of) are predictive mean and variance, with the prior GP density p(f(x:) = (x((); (((r,,))) This can be achieved by using divergence measure between distributions such as the symmetrised KL divergence,
p(f(x))) (f()))) i..,..,, - ( (((()))) log ((f(z)) df(x;)+ (D(F(E.)) log ((f(x,)) p(f(x:))
computed separately for any in the minibatch. Given that all distributions are univariate Gaussians the above can be obtained analytically. When each score & is close to zero this indicate that the input distribution has changed so that a task switch can be detected. Each l 0 can be thought of as expressing degree of surprise about æ;, i.c the smaller is & the more surprising is z4. Thus our idea has close links to Bayesian surprise (Itti & Baldi 2006).
In order to use this intuition to detect task switches, we can perform a statistical test between the values { 1i_ for the current batch and those from the previous batch {gald) Ji=1 before making any updates to the parameters using the current batch. A suitable choice is Welch's 's t-test (due to unequal variances), demanding that with high statistical significance the mean of (&, 1b is smaller than the mean of (gold) 1--1-
The ability to detect changes based on the above procedure arises from our framework as we construct posterior distributions over function values f(,) that depend on inputs æ; (while in contrast posterior over weights alone does not depend on any input). Subsequently. these predictive densities contain information about the distribution of these inputs in the sense that when 2; is close the
7

---

Published as a conference paper at ICLR 2020
training inputs from the same task expect reduced uncertainty, while for inputs of a different task we expect high uncertainty that falls back to the prior uncertainty.
EXPERIMENTS
We now test the scalability and competitiveness of our method on various continual learning problems, referring to the proposed approach as Functional Regularised Continual Learning (FRCL). Throughout this section, we will aim to answer three key questions:
(i) How does FRCL compare state-of-the-art algorithms for Continual Learning?
(ii) How important is principled criterion for inducing point selection? How do varying numbers of inducing points/task affect overall performance?
(iii) If ground truth task boundaries are not given, does the detection method outlined in Section succeed in detecting task changes?
In order to answer these questions, we consider experiments on three established Continual Learning classification problems: Split-MNIST, PerMuted-MNIST and sequential Omniglot (Goodfellww et al. 2013 Zenke et al. 2017 Schwarz et al. 2018), described the Appendix. FRCL methods have been implemented using GPñoow (Matthews et al. 2017).
In addition to comparing our method with other approaches in the literature by quoting published results, we also show results for an additional baseline (BASELINE) corresponding simple replay-buffer method for Continual Learning (explained in the Appendix).
4.1 IS FRCT. A COMPETITIVE MODEL FOR CONTINUAL LEARNING?
Addressing first question (i), we show results on the MNIST-variations in Table[[and on the more challenging Omniglot benchmark in Table 2 Note that we also specify the inducing point optimisation criterion in brackets, i.c. FRCL (TRACE-TERM) corresponds to the loss in equa- 0.6-
Spic-MNIST
1--
0.9-
tion We observe strong results on all bench- marks. setting new state-of-the-an results Permuted-MNIST & Omniglot while coming close existing results on Split-MNIST. The improvement on the BASELINE shows that ap- proximate posterior distributions over functions values can lead to more effective regularisation for CL. compared to just having a replay buffer of input-output pairs. Furthermore, despite its simplicity, the simple BASELINE strategy per- forms competitively. In line with other results, we conclude that rehearsal-based ideas continue to provide a strong baseline. This also gives justification to a main motivation of this work: To unite the previously separate lines of CL work rehearsal-based and Bayesian methods. Nevertheless, other methods may be more suitable when storing data is not feasible.
(9:00)
Porm MNIST
1.0.
0.-
0.8-
0.5
60
Figure Comparing optimisation criteria for varying number of inducing points.
4.2 INDUCING POINT OPTIMISATION AND TASK SWITCH DETECTION
An appealing theoretical property of our method is the principled selection of inducing points through optimisation. Answering question (ii), we now proceed to investigate the importance of the criterion used as well as the dependence on the number of inducing points. These results are shown in Figure 3 Note that the definition of objectives Class. Error; ELBO & Log pred. density are given in the Appendix. In accordance with our intuition, we observe that optimisation becomes increasingly important the number of inducing points is reduced. The results also give strong statistical motivation to use the trace-term motivated before. Further, as can be seen looking at the results for Log pred. density, a poorly chosen criterion may behave worse than random.
8

---

Published as a conference paper at ICLR 2020
Table 1: Results Permuted- and Split-MNIST. Baseline results taken from Nguyen et al. (2017). For the experiments conducted this work we show the mean standard deviation over 10 random repetitions. Where applicable, we also report the number of inducing points/replay buffer size per task in parentheses.
Algorithm
Permuted-MNIST
Split-MNIST
DLP (Eskin al.
2004
82%
61.2%
EWC (Kirkpatrick
2017
84%
63.1%
SI (Zenke ct al. 2017)
86%
98.9%
VCL (gguynn cf al.
2017
90%
97.0%
random Coreset
93% (200 points/task)
k-center Coreset
93% (200 points/task)
+ unspecified Coreset selection
98.4% (40 points/task)
BASELINE
48.6% 1: points/task)
FRCI. (RANDOM)
48.2% + 4.0 (10 points/task)
59.8% = 8.0 (2 points/task)
FRCL (TRACE)
61.7% + 1.5 (10 points/task)
82.0% 5.0 (2 points/task)
BASELINE
82.3% + 0.3 (200 points/task)
95.8% 1.1 (40 points/task)
FRCL. (RANDOM)
94.2% = 0.1 (200 points/task)
97.1% 0.7 (40 points/task)
FRCL (TRACE)
94.3% 0.2 (200 points/task)
97.8% 0.7 (40 points/task)
Table 2: Results on sequential Omniglot. Baseline results are taken from Schwarz et al. (2018). Shown are mean and standard deviation over 5 random task permutations. Note that methods 'Single model per Task' and Progressive Nets' are directly comparable due unrealistic assumptions, but serve upper bound on the performance for the remaining continual learning methods.
Algorithm
Test Accuracy
Single model per Task Progressive Nets
88.34
86.50 0.
Finetuning
26.20 4.6
Learning Without Forgetting
62.06 2.0
Elastic Weight Consolidation (EWC)
67.32 4: 4.7
Online EWC
69.99 +: 3.2
Progress& & Compress
70.32 + 3.3
Methods evaluated in this paper
1 point/char
points/char 57.17 + 12
points/char 65.32 + 1.1
BASELINE
42.73 = 1.2 69.74
FRCL (RANDOM)
80.32 225
81.42
FRCL (TRACE)
72.02 =
81.47 + 1.6
81.01 +
ong Bapppins
Fisss indceed poinss
* Finsl induned poins
0 X 3 3
5
6
6
6
2 Z 2. 3
3
5
8
9
O O | 2 5
5
5
6
8 9 6 9 9
0 0 3 S
6
0 d 6
8
(b) Random inducing points
0 3
4 5
7 8 9 9 9 9 8 9
0
1
2
3
4
5
7
8
0
2
3
4
5
6
7
8 8
0
/
2
4
5
6
7 7
0
2
3
4
5
6
(a) A TSNE projection of inducing points in feature space. (c) Optimised inducing points
benchmark. The number of inducing points was limited to 10. Left: A example optimisation shown in the feature space a trained network. Points are coloured by class label. Data shown corresponds to the first row in the images on the right. Right: Optimised inducing points consistently cover examples of all classes. Each row corresponds to different with random initialisation. Best viewed in colour.
Figure Inducing point optimisation for first task on the Permuted-MNIn
9

---

Published as a conference paper at ICLR 2020
To provide an insight into the solutions obtained by the trace-term criterion, we provide a visualisation of inducing points in Figure Remarkably, even though the objective is unsupervised, it results in a consistent allocation of one example per class. Furthermore, the optimised inducing points spread across the input space as shown by the TNSE (Maaten & Hinton 2008) visualisation, which is in line with the intuition that the objective encourages repulsive inducing points.
Finally, we answer question (iii) by first showing both the mean of the terms {4)]==1 (top) as well
as the result of Welch's 's t-test (bottom) between terms {4,}== {er'd) in Figure 5, using only a small number Omniglot alphabets and 1000 training iterations per task for illustrative purposes. We
note that the intuition built up in Section 3 holds, with clear spikes being shown whenever the t-test returns a positive result.
Furthermore, we provide a quantitative comparison in Table On the positive side, we note very strong results for Split- & Perm-MNIST and further observe that we find similar t-test threshold value to apply to all dataset, making this an easy hyper-parameter to set. While the task boundary detection results for Omniglot are less strong, which may due to the smaller batch size (32 for Omniglot, 2 100 for the MNIST-verse versions,,,,,,,,, resulting in noisier test result. Note that this could be easily mitigated by using a larger set (Lald), J. c.g. the last 10 minibatches, which would make this test more robust.
Table 3: Task boundary detection evaluated as binary classification task (Positive labels corresponds to task switches).
200
Hean symmetiic, Es divergence bettreen OF prior and posteriar,
6.5
Predsion//Recll// Threshold
statistre
Split-MNIST
Penn-MNIST
Omniglot
20-
4.0
0.94/1 000.97
0.43/ 98/0.60
0. 13/0 90/0
5.0
1.0/100/10
0.95/0.94/0.94 94
0.71/0.75/0.7
6.0
1.0/1.0/100
1.0/0.9100.95
0.89/0.59/0.7
1ax Seroan
3000
Task 2: De
1858
Task Drya
3900
Teok Tiactar
7.0
1.0/0.1/00955
0.8990.46/.66
8.0
1.0/0.86/0.92
0.88/0.32/0.
Figure 5: Visualising KL terms and test statistics on multiple Omniglot tasks.
DISCUSSION
We introduced a functional regularisation approach for supervised continual learning that combines inducing point GP inference with deep neural networks. Our method constructs task-specific posterior beliefs or summaries on inducing inputs. Subsequently, the task-specific summaries allow us to regularise continual learning and avoid catastrophic forgetting. Our approach unifies the two extant approaches to continual learning, of parameter regularisation and replay/rehersal. Viewed from the regularisation perspective, our approach regularises the functional outputs of the neural network, thus avoid the brittleness due to representation drift. Viewed from a rehearsal method perspective, we provide a principled way of compressing data from previous task, by means of optimizing the selection of inducing points. By investigating the behaviour of the posterior beliefs, we also proposed a method for detecting task boundaries. All these improvements lead to strong empirical gains compared to state-of-the-ar continual learning methods.
Regarding related work on online learning using GPs, notice that previous algorithms (Bui et al., 2017a Csato & Opper| 2002 Csato (002) learn in an online fashion a single task where data from this task arrive sequentially. In contrast in this paper we developed a continual learning method for dealing with a sequence of different tasks.
A direction for future research is to enforce fixed memory buffer (or a buffer that grows sub-linearly w.r.t. the number of tasks), in which case one would need to compress the summaries of all previous seen tasks into a single summary. Finally, while in this paper we applied the method to supervised classification tasks, it will be interesting to consider also applications in other domains such as reinforcement learning.
ACKNOWLEDGEMENTS
We would like to thank Hyunjik Kim and Raia Hadsell for useful discussions and feedback.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kemells for vector-valued functions:
A review. Found. Trends Mach. Learn., 4(3):155-266, March 2012. ISSN 1935-8237. doi:
10.1561/22000000066 CRLLhttp:/// doi www.wooo.......ooooooo wtt.o/////...................... 1561 1ttp://10..6///200033330 /200000036
Edwin V Bonilla, Kian M. Chai, and Christopher Williams. Multi-task gaussian process prediction. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis (eds.), Advances in Neural Information
Processing Systems 20, pp. 160. Curran Associates, Inc., 2008. URL http::/ppaper nips. httccc//pppppo httpc/papp/331111 multi task pasesian-ppoossee-eeppp-------s- E. pdf
Thang D Bui, Cuong Nguyen, and Richard E Turner. Streaming Sparse Gaussian Process Approx- imations. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3299-3307. Curran Associates, Inc., 2017a.
Thang D. Bui, Josiah Yan, and Richard E. Turner. Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation. Journal of Machine Learning Research, 18(104):1-72,
Csato. Gaussian processes: iterative sparse approximations. PhD thesis, Aston University, 2002.
L. Csato and M. Opper. Sparse online Gaussian processes. Neural Computation, 14:641-668, 2002.
Amir Dezfouli and Edwin V Bonilla. Scalable Inference for Gaussian Process Models with Black- Box Likelihoods. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1414-143 2015.
Eleazar Eskin, Alex Smola, and SVN Vishwanathan. Laplace propagation. In Advances in neural information processing systems, pp. 441-448, 2004.
Sebastian Farquhar and Yarin Gal. Towards Robust Evaluations of Continual Learning. arXiv preprint arXiv: 1805.0973. 2018.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in muudient-based neural networks. arXiv preprint 1312::1312.622 2013.
James Hensman, Nicolo Fusi, and Neil Lawrence. Gaussian processes for Big Data. In Conference on Uncertainty in Artificial Intellegence, pp. 282-290, auai.org, 2013.
James Hensman, Alexander G de G Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, 2015.
Laurent Itti and Pierre F Baldi. Bayesian surprise attracts human attention. In Advances in neural information processing systems, pp. 547-554, 2006.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska-Barwinske. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, pp. 201611834 2017.
Brenden I Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2017.
Chris Lloyd, Tom Gunter, Michael A. Osbore,, and Stephen J. Roberts. Variational inference for gaussian process modulated poisson processes. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML'IS. 15, pp. 1814 1822, 2015.
11

---

Published as a conference paper at ICLR 2020
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6470 6470-6479, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, (NNvv)2257922603 2008.
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke, Fujii, Alexis Boukouvalas, Pablo Le'on- Villagr'a a, Zoubin Ghahramani, and James Hensman. GPflow: A Gaussian process library using l'owsorFlow. Journal of Machine Learning Research, 18(40):1-6, April 2017.
Cuong V Nguyen, Yingzhen Li, Thang Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv: /710. 10628, 2017.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Sylvestre-Alvy -Aliis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5533-5542 IEEE, 2017.
Mark Bishop Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin Austin, Texas 78712, 1994.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
Anthony Robins and Simon McCallum, Catastrophic forgetting and the pseudorehearsal solution in hopfield-type networks. Connection Science, 10(2):121-135, 1998.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lilllccra,, and Greg Wayne. Experience replay for continual learning. arXiv preprint arXiv: I. 1682, 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 2016.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4:313, 2013.
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki, Agnieszka Trabska-Barwinske Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning. arXiv preprint arXiv: 1005.063700 2018.
Rishit Sheth, Yuyang Wang, and Roni Khardon. Sparse variational inference for generalized gp models. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1302-1311, Lille, France, 07-09 Jul 2015. PMLR.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Y. Weiss, B. Schölkopf, and J. C. Platt (eds.), Advances in Neural Information Processing Systems 18, pp. 1257-1264 2006.
Michalis K Titsias. Variational learning of inducing variables sparse Gaussian processes. In International Conference Artificial Intelligence and Statisties, pp. 567-574, 2009.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, pp. 3630-3638, 2016.
Christopher K. I. Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. T. K. Leen, G. Dietterich, and Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 682-688. MIT Press, 2001.
12

---

Published as a conference paper at ICLR 2020
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 May 2016. PMLR.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. arXiv preprint arXiv: 2017.
A TASK DESCRIPTIONS
Split-MNIST and Bermuted-MNIST Among a large number of diverse experiments in continual learning publications, two versions of the popular MNIST dataset have recently started to become in- creasingly popular benchmarks: Permuted- and Split-MNIST. In Permuted-MNIST (e.g. Goodfellow al. 2013 Kirkpatrick et al. 2017 Zenke et al. 2017), each task is variant of the initial 10-class MNIST classification task where all imput pixels have undergone a fixed (random) permutation. The Split-MNIST experiment was introduced by Zenke et al. (2017): Five binary classification tasks constructed from the classes in the following order: 0/1, 2/3, 4/5, 6/7, and 8/9.
Omniglot. To assess method under more challenging conditions, we consider the sequential Omniglot task proposed for continual learning in (Schhara et al. (2018). Omniglot Lake et al. (2011) is dataset of 50 alphabets, each with varying number of classes/characters which we treat of as sequence of distinct classification problems. As suggested in Schwarz et al. (2018), for Omniglot, we apply aaeaaugmenttiionn and use the same vaann/vlidation/tes/ split. Following the same experimental setup proposed, we used an identical convolutional network to construct the feature vector o(x; f). Results reported are obtained by training on the union of training and validation set after choosing any
based on the validation set. Note that all experiments were run with data processing and neural network construction code kindly provided by the authors in Schwarz et al. (2018), ensuring directly comparable results.
Given that Permuted-MNIST and Omniglot are multi-class classification problems, where each k-th task involves classification over Cj classes, need to generalise the model and the variational method to deal with multiple GP functions per task. This is outlined in the next section.
B EXTENSION TO MULTI-CLASS (OR MULTIPLE-OUTPUTS) TASKS
For multi-class classification problems, such as permuted MNIST and Omniglot considered in our experiments, where in general each k-th task involves classification over varying number of classes, we need to extend the method to deal with multiple functions per task. For instance, assume that the k-th task is a multi-class classification problem that inolves Ck classes. To model this we need Cj independent draws from the GP, such that fx( GP ((xxz2')) with = 1,..., Ck. which are combined based on a multi-class likelihood such as softmax. While in the main text presented the method assuming a single GP function per task, the generalisation to multiple functions is straightforward by assuming that all variational approximations factorise across different functions/classes. For example, the variational distribution over all inducing variables UA = (ug)cz Cx. C2
takes the form q(Ux)) = IT - q(u)) and similarly the variational approximation over the task weights W = {w needed in the ELBO in Section 3.2, also factorises across classes. Notice also that all inducing variables Us are evaluated the same inputs Zu. Furthermroe, the KL regularization term for each task takes the form of a sum over the different functions, i.e. KL(g(U&)) (Uw))
C.
C BASELINE MODEL
The BASELINE model (see main text) is based on storing an explicit replay buffer (V., X,), i.e. subset of the training data where y, C y, and X, C X,, for each past task. Then, at each step when encounter the k-th task training is performed by optimising unbiased estimate of the full loss
13

---

Published as a conference paper at ICLR 2020
(i.e. if we had all k tasks at once), given by
N
(yv,, X,; w;; 0); T,0, w1:k) = (y NRy 0) + M₁
where each (.) is a task-specific loss function, such as cross entropy for classification, and each scalar N, corrects for the bias on the loss value caused by approximating the initial full loss by M.
a random replay buffer of size M, All output weights Wick of the current and old tasks, in the multi-head architecture, are constantly updated together with the feature parameter vector f. Also each step a fresh set of output weights is constructed in order to deal with the current task.
D SELECTION OF THE INDUCING POINTS AND OPTIMISATION CRITERIA
After having seen the k-th task, and given that it is straightforward to compute the posterior distribution ((uz) for any set of function values, the only issue remaining is to select the inducing inputs Zg- A simple choice that works well in practice is to select 2 as a random subset of the training imputs X₃- do better with some more structured criterion. In experiments we will investigate three supervised criteria, that make use of class labels, and one unsupervised that only depends on inputs and the neural network feature vector. All criteria below optimise over Zk using discrete search.
The question is whether
The first supervised criterion is minimize the negative average log predictive density,
Log pred. density(Zx)
Σ
logp(yk.j (z),
X
FL.EXeiZ.
computed at all remaining training inputs by excluding the selected inducing inputs Zu- Each predictive density p (xx) is obtained through the inducing inputs Zy and it takes the form
p(yk.) [z,) f p(yk.s ( Zx)du; = p(y..,
Muee Lug L,, ddu,
(8)
which for the classification problems, where the likelihood p(y..s fk.j) ) is not Gaussian, is computed numerically by one-dimensional Gaussian quadrature. Notice that for the multi-class case the predictive density is obtained by integrating over (U*)) = q(Ux|Za) (see previous Appendix B). Modern GP packages, such GPflow, provide efficient implementation of the above predictive densities.
The second supervised criterion is similar, but log p(k.. 7xk) is replaced by classification error score T(yk.j # vi
Class. Error (kx)
Σ
I(yk.s # vE.)).
(9)
Xx
Z
EX
where vi.j the predicted label, which is obtained by first computing the predictive density p(yk.) [xx) conditional given set of inducing imputs Z₁ and then choosing vi., by taking argmax (i.e. selecting the class with the largest predictive probability).
Both criteria above essentially capture how well predict the remaining points from the inducing points Zμ. thus good choices for Z₃ are presumably those that lead to good prediction for the remaining points.
The third supevised criterion is the standard sparse GP variational lower bound,
No.
ELBO(Za) = Σ Egtt,,) [log p(y. ( (c((gu//Z))llpdd(( (z)))))
(10)
which is viewed purely as a function of Z₁ while all remaining quantities are constant. This criterion tries to optimise over Z₁ in order approximate as best possible the marginal likelihood on the training data and it the one used by all variational sparse GP training methods (although there the ELBO is maximized jointly over both Zz with the remaining parameters).
14

---

Published as a conference paper at ICLR 2020
The last unsupervised criterion (also discussed in the main paper) expresses how well reconstruct the full kernel matrix Xx2 from the inducing set Zy. which can be described by
Trace term - (Kx, Kx,z, K Kz,x,)
Σ ((RR) k2x K Kin
(11)
where each k(Zagj Tk.5) kz Zx. mi,t training point. The above quantity appears in the ELBO in TTissas 2009), is also used in (Csato & Opper] 2002) and it has deep connections with the Nystróm approximation (Williams & Seeger 22001)
K2- K.. 2 0 is a reconstruction error for an individual
and principal component analysis. The criterion in equation 6 promotes finding inducing points Za that are repulsive with one another and are spread evenly the input space under similarity/distance
implied by the dot product of the feature vector Φ(2; 0%) (with 22 being the current parameter values
after having trained with the k-th task). An illustration of this is given the Figure 4
Figure iin the Experiments Section illustrates the optimisation of the inducing inputs in Permuted- MNIST. Also, Figure 6 a) shows randomly chosen examples per class) for the Greek alphabet/task in the Omniglot experiments, while Figure (b(b) shows the corresponding points after have been optimised by using the trace criterion above.
Figure 3 shows the evolution of the performance for all selection criteria on individual tasks for Split-MNIST and Permuted-MNIIST.
R
3
GL
FL
5
S
3
E
C
T
5
5
3
3
E
M
n
e
L
9
T
6
6
K
Λ
X
M
A-
K
K
7
2
A
M
1+
V
5
5
Y
V
E
TC
77
P
م
O
6
O
T
TC
P
O
T
Up
CP
J
1p
EQ
4p
U
Y
W
3
X
X
X
Y
LO
W
w
(a) Randomly selected inducing points
(b) Optimised inducing points
Figure Inducing points for Greek alphabet of the Omniglot benchmark. The number of inducing points limited 3 per character.
E TASK BOUNDARY DETECTION
In order to apply the task-bounder detection for multiple classes, we perform Welch's t-test separately for each function. We consider mean, median and the maximum of those t-test results to make decision. Furthermore, we also found that conducting the test in log-space significantly improved robustness of the test results and thus find that tests using the max over functions in log-space allow for higher thresholds and more robust results. In order to give justification to those conclusions we show experiments Bermuted-MNIST in Figure 7 once again using 10 random task permutations.
Note also that after a task boundary was detected, we do not consider a new test for the next 10 iterations.
At this point it should said that while the boundary detection method simple, cheap to compute, unsupervised and general, it is not applicable when the Continual Learning faces a continuum of tasks without clear switches or when the input distribution is constant and different tasks merely correspond to varying labels.
15

---

Published as a conference paper at ICLR 2020
T-test in log space
score
0.6
Aspregater mean
0.2
0.0
median
T-test in original space
score
0.8
0.2
8.0
Test threshold
Ablation study comparing aggregation methods and whether tests should be conducted in log-space. Results are shown Permuted MNIST, 10 random task permutations.
Figure 7:
Table 4: Hyperparameters for the experiments on Split MNIST. Optimal values (in bold) were chosen on the validation set. Test set results were obtained by training on the union of vaaidiin&vllidatiinn set using those values.
Parameter
Considered range
Comment
Network size (#Layers Units)
[2) x (256)
Based Zenke al. 2017
Activation function
(f(x) max (0, x) (ReLu))
Learning rate
(5 10 10-4 10-3)
Batch size
{32, 64, 100, 128)
#Training steps
(2500, 3500)
#Inducing points optimisation steps
[1000, 2000)
No significant difference.
F EXPERIMENTAL DETAILS
Experimental details for all experiments are shown in Tables 4 5 and 6. Note that for the MNIST results, we obtain final results after optimising hyperparameters on the validation set and using those values to train on the union of training & validation set.
For Omniglot on the other hand, we report final test-set results only after training on the training set in order to remain consistent with the results in Schwarz et al. (2018)
Table 5: Hyperparameters for the experiments on Permuted MNIST. Optimal values (in bold) were chosen on the validation set. Test set results were obtained by training on the union of training&valliattion set using those values.
Parameter
Considered range
Comment Based on Zenke al. 2017).
Network size (HLayers Units)
(2) (100) (f()) max(0, (ReLw)
Activation function
Learning rate
[10 3, 5. 10-4 10-1,01 10-5)
Batch size
(32, 64, 128)
#Training steps
(2000, 2500]
No significant difference.
#inducing points optimisation steps
{1000, 2000
No significant difference.
16

---

Published as a conference paper at ICLR 2020
Table 6: Hyperparameters for the experiments Omniglot. Optimal values (in bold) chosen the validation set. Test results were obtained by training on the union of training&vallidttinn set using those values.
Parameter
Considered range
Comment
Conv. filters
64, 64, 64]6
Based on Vinyals et (016))
Conv. Kernel size
{3 x 3}
Conv. Padding
[SAME]
Max Pool. Kernel size
3)
Pool. stride
(2 x 2}
Max Pool. Padding
{VALID)
Activation function
{f(x) max(0, x) (ReLu))
Learning rate
[10 (10-3, 10 ",1""
Batch size
{32)
#Training steps
[2500}
#Inducing points optimisation steps
(1000, 2000)
significant difference.
G COMPARISON TO VCL ON OMNIGLOT
To provide a further comparison to VCL Nguyen et al. 2017) ) we show results on Omniglot using Multi-Layer Perceptrons (MLP). A comparison with MLPs is due the fact that reliable variational inference methods for CNNs (which are usually used for Omniglot) are yet to be developed. All our results for VCL are obtained using code provided by the authors.
For all experiments, we used an MI.P with 4 hidden layers of 256 units each and ReLU activations, batch size of 100 and the Adam Optimiser (Step size of 0.001 for VCL and 0.0001 for FRCL). We optimised both types of algorithms independently and found the following parameters:
1. VCL: 100 training epochs per task, 50 adaptation epochs to coreset, Multi-Head
2. FRCL (TRACE): 1500 training steps per task, 2000 discrete optimization steps, inducing points initialised uniform distribution over classes
Table 7: Results on sequential Omniglot using MLP.
Algorithm
Test Accuracy 48.4 0.
VCL (NO CORESET)
point/char
points/char 50,5 12
points/char 51.64 # L.0
VCL (RANDOM CORESET)
49.18 2:21
VCL (K-CENTER CORESET)
48.89 + 1.1
49.58 + 1.4
49.61
FRCL (TRACE)
48.84 +
52.10 +
53.86 2.3
17


Reviews:
[{"review_id": "HkeRF7LnKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops a continual learning method based on Gaussian Processes (GPs) applied in the way introduced by prior work as Deep Kernel Learning (DKL). The proposed method summarizes tasks as sparse GPs and use them as regularizers for the subsequent tasks in order to avoid catastrophic forgetting. Salleviating the instability resulting from the representation drift.\n\nEmploying inducing point training for task memorization is a novel and interesting idea, which could be useful for the continual learning community. The fact that this approach also captures the uncertainty of the replays contributes fairly to robustness. Lastly, performing knowledge transfer by inheriting the KL term of the ELBO is also interesting, however, its theoretical implications deserve a close look. It would be enlightening to analyze which true posterior the learned model then corresponds to. Would not it be a slightly more principled Bayesian approach (i.e. one that has stronger grounds at first principles) to perform the knowledge transfer to assign the posterior of one task as the prior of the other, alternatively to keeping the entire KL term intact which employs the q(u_i) as the surrogate for q(u_j), i.e. the way introduced by Nguyen et al., 2017?\n\nThe presentation clarity of the paper is open for improvement. For instance, the abstract is written in a sort of convoluted way. I do not get how the KL divergence suddenly kicks in and for what exact purpose. Is it variational inference or a hand-designed regularization term?\n\nI find the argumentation from Eq. 1 downwards until the end of Sec 2.1 on BNNs with stochastic weights and their relation to GPs a bit unnecessary complication. These are very well known facts. It would suffice to state briefly that the task learner is a vanilla DKL used within a vanilla sparse GP.\n\nFigure 1 is also not so descriptive. I do not get what the GP here is exactly doing. What is input to and for which output modelity does it find a mapping? What is the calligraphic L in the figure? Is it a neural net loss or an ELBO? \n\nIn general I could not grasp why it makes sense to treat the the output layer params of a neural net treated for continual learning? They will not be sufficient to encode a task anyway, as an expressive enough neural net will leave only a linear mapping to the final layer. What happens if the intermediate representations of the input observations require a domain shift as the tasks evolve?\n\nOverall, the presented ideas are fairly interesting and the experimental results are good enough for proof-of-concept, though not groundbreaking (behind the close relative VCL on MNIST and no comparison against VCL on Omniglot). Hence, this is a decent piece of work that lies somewhere around the borderline. My major concern is that the proposed method is conceptually not novel enough compared to Nguyen et al., 2017. My secondary concern is that the presentation is very much open to improvement in points hinted above.\n\n--\nPost-rebuttal: Thanks to authors for their tremendous effort to alleviate my concerns. The fact is that the conceptual novelty of the paper is too slim compared to VCL. As mentioned above, I even find the VCL approach more principled. I could have viewed the outcome of the paper as a slightly bigger news for the community if there was something unexpectedly positive on the reported results. However, as it appears from the comment below, the authors propose a super close variant of VCL that combines a few well-known techniques in a rather straightforward way and achieves in the end a model that performs on par with it. Under these conditions, I have hard time to find a reason to champion this paper for acceptance. That being said, I view this paper a tight borderline case due to its technical depth, hence I will not object to a reject decision either.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeWwjLpqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a method to perform continual learning with neural networks by incorporating variational Gaussian Processes as a top layer (also called Deep Kernel Learning) and constructing an objective utilizing the inducing inputs and outputs to memorize across tasks.\nThey further study ways to approximate this behavior with weight space models and use their model for task boundary detection by utilizing statistical tests and Bayesian model selection.\nExperiments show good performance of their method.\n\nComments:\n1. The mathematical formulation of the basic model is very elegant. However, it is not immediately clear to me that the joint ELBO across successive tasks is still lower-bounding the actual objective.\n2. The paper is well written overall.\n3. To the best of my knowledge using such a model for task boundary detection is novel and quite interesting. There are obvious links to Bayesian changepoint detection in the timeseries setting. Possibly these links would be made more clear by a citation to a recent paper such as Spatio-temporal Bayesian On-line Changepoint Detection with Model Selection by Knoblauch and Damoulas, or any other paper of similar content. The link is quite fascinating.\n4. Sections 2.3 and 2.4 of the paper are the weakest points and quite unsatisfactory as they forgo the elegance of the proposed approach to do \"something else\" that Sec. D explains how to salvage with \"tricks\". Especially with regards to Sec. 2.4, why can't we just do inference on Z_i and have to pick datapoints via discrete optimization? That comparison would be useful in the experiments. Furthermore, recent papers utilizing GPytorch by Gardner et al have dramatically sped up GP inference. Could we aim to make the original idea fast enough to be used instead of resorting to an approximate model with weight spaces and corrections to extract Z and u per task?\n5. The experiments are good, but very focused on MNIST tasks. I would appreciate tasks of different structure given how well the method appears to work.\n\n\nDecision:\nI find the basic idea of the paper quite appealing as it leverages the elegance of the deep kernel learning formulation to yield an attempt at a principled Bayesian version of continual learning and demonstrates empirical value. \nSome discussion on the objective might be warranted to demonstrate that it actually lower bounds the true LLK.\nI am quite happy with the task boundary detection section and would encourage the authors to strengthen the link to changepoint detection.\nMy biggest qualms with the paper are that it departs from that strategy and performs weight space inference for training per task and then \"corrects\" to move back to the GP representation. A more convincing discussion would be welcome here.\nThe experiments are functional and show good results, but I would appreciate more diversity in the tasks.\nAs the paper stands I learn towards recommending acceptance and would strongly encourage the authors to iron out the weaknesses of the paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1xoID52tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a function-space based approach to continual learning problems (CL), wherein a learned embedding\n\n    $\\hat{\\mathbf{x}} = \\text{NN}(\\mathbf{x}; \\theta)$\n\nis shared between task-specific GPs s.t.\n\n    $f_{i}(\\mathbf{X}) \\sim \\mathcal{N}(\\mathbf{0}, k_{i}(\\hat{\\mathbf{X}},\\hat{\\mathbf{X}}))$, \n\nwhere the $i$-th task's covariance $k_{i}$ is a defined via standard variational inducing points methods. CL manifests as KL divergences between tasks' variational posteriors $q_{i}$ and their respective priors $p_{i}$. Since the embedding helps define $p_{i}$, its parameters $\\theta$ are regularized to promote sharing.\n\nThe work investigates both practical and theoretical implications of this setup. On the practical side, the authors discuss enhanced 'on-task' inference via hybridization of function- and weight-space based approaches and, subsequently, strategies for optimizing inducing points. Additionally, a novel approach for automatically detect task switching is introduced that exploits the Bayesian aspects of the proposed framework.\n\nOn the theoretical side, points of (personal) interest revolved around differences between weight- and function-space approaches to CL. Here, I think that streamlining the presented argument would go a long ways. Paraphrasing, one of the authors' key insights is that:\n\n  1) CL in weight-spaces is hard, since weights' semantics are moving target that change along with shared parameters.\n  2) CL in function-space is easy, since the functions (i.e. tasks) themselves remain the same.\n\nThis information is provided in the introduction, but (as a relative newcomer to CL) I failed to connect regularization and rehearsal/replay based methods with the aforementioned spaces. It was only upon reading Sect 2.5 that this intuition 'clicked' for me. Hence, I suggest making this observation as obvious and intuitive as possible.\n\nThe provided experiments seem reasonable and do a good job highlighting different facets of the paper. Two additional results would be appreciated:\n\n  a) How well calibrated are FRCL-based classifiers?\n  b) How impactful is the hybrid representation (Sect 2.3) for test-time performance?\n\nGP approximations formulated solely in terms of weighted sums of (finitely many) basis functions typically suffer from degradation of predictive uncertainties. Since one often motivates use of GPs via a desire for well-calibrated uncertainty, (a) seem quite pertinent.\n\n\nNitpicks, Spelling, & Grammar:\n  - Lots of run-on sentences; consider breaking these up.\n  - Introductory modifying phrases are missing commas.\n  - Consider citing other recent works that use NN basis functions in conjunction with Bayesian Linear Regression.\n  - Various missing or superfluous words resulting in some garbled sentences, e.g.:\n      - \"... our approach looks constraining.\"\n      - \"The ability to detect changes based on the above procedure comes from that in\"\n      - \"While the task boundary detection results for Omniglot are less strong, which may due to the smaller batch size (32 for Omniglot, ≥ 100 for the MNIST-versions), resulting a noisier test result.\"\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a framework for continual learning in neural networks based on sparse Gaussian process methods. The reviewers had a number of questions and concerns, that were adequately addressed during the discussion phase. This is an interesting addition to the continual learning literature. Please be sure to update the paper based on the discussion.
Decision: Accept (Poster)