Continual Deep Learning by Functional Regularisation of Memorable Past
Authors: Pingbo Pan
Alexander Immer
Siddharth Swaroop
Runa Eschenhagen
Richard E Turner
Mohammad Emtiyaz Khan
Venue:  (2020)
Keywords: Continual learning
deep learning
functional regularisation

Abstract: Continually learning new skills without forgetting old ones is an important quality for an intelligent system, yet most deep learning methods suffer from catastrophic forgetting of the past. Recent works have addressed this by regularising the network weights, but it is challenging to identify weights crucial to avoid forgetting. A better approach is to directly regularise the network outputs at past inputs, e.g., by using Gaussian processes (GPs), but this is usually computationally challenging. In this paper, we propose a scalable functional-regularisation approach where we regularise only over a few memorable past examples that are crucial to avoid forgetting. Our key idea is to use a GP formulation of deep networks, enabling us to both identify the memorable past and regularise over them. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation methods are naturally combined with memory-based methods.

Under review as a conference paper at ICLR 2020
CONTINUAL DEEP LEARNING BY FUNCTIONAL REGULARISATION OF MEMORABLE PAST
Anonymous authors Paper under double-blind review
ABSTRACT
Continually learning new skills without forgetting old ones is important quality for an intelligent system, yet most deep learning methods suffer from catastrophic forgetting of the past. Recent works have addressed this by regularising the net- work weights, but it is challenging to identify weights crucial to avoid forgetting. A better approach is to directly regularise the network outputs at past imputs, e.g.. by using Gaussian processes (GPs), but this is usually computationally challeng- ing. In this paper, we propose a scalable practional-regulariinn we regularise only over a few memorable past examples that are crucial to avoid forgetting. Our key idea is to use a GP formulation of deep networks, enabling to both identify the memorable past and regularise over them. Our method achieves state-of-the-ane performance on standard benchmarks and opens a new direction for life-long learning where regularisation methods are naturally com- bined with memory-based methods.
approach where
INTRODUCTION
The ability quickly adapt to changing environments an important quality of intelligent systems. For such quick adaptation, it is important to be able to identify, memorise, and recall useful past experiences when acquiring new ones. Unfortunately, standard deep-learning methods are not good at maintaining previously acquired skills, and can quickly forget them when learning new skills (Kirkpatrick al., (017). Such catastrophic forgetting presents a big challenge when deploying deep-learning methods for applications, such as robotics, where new tasks can appear during the training, and data from the previous tasks might be unavailable for retraining.
In recent years, many methods have been proposed to address catastrophic forgetting in deep learn- ing. One of the most popular approaches is to regularise the network weights to keep them close to the weights obtained for the previous tasks/data (Kirkpatrick et al. 2017: Nguyen et al. 2018 Zenke et al. 2017 Ebrahimi et al. 2019 Serra et al., 2018). This is challenging due to the difficulty relevant to past tasks. The exact values of the weights in fact
in identifying the weights that do not matter directly, but rather the network output (Benjamin et al., 2018). Figuring out which weight affects the output is therefore usually difficult. Typically, the Fisher information matrix or covariance matrices over weights are used (Kirkpatrick et al. 2017: Nguyen et al., 2018). but they only partially address the issue.
A better approach is to directly regularise the network outputs, also referred to as functional- regularisation, requiring memory of past examples (Benjamin al. 2018 Lopez-Paz & Ranzato, 2017 Rebuffi et al., 2017). However, such methods still lack a mechanism to automatically weight more relevant past memory in the context of the new task, and also do not take uncertainty of the out- put into account. Methods based on Gaussian processes do this automatically (Titsias et al.) 2019). require optimisation over inducing points and specification of a good kernel, both which are difficult tasks. In summary, existing methods fall short in building scalable finetionnl-regullarstttooo methods for continual learning.
In this paper, we propose new functional-reguulristtti method where we regularise over few memorable past examples (see Fig. [ Our approach builds upon a recent method of Khan et al. 2019) that expresses deep networks Gaussian processes (GPs). We show that the GP formula- tion not only enables the identification of examples crucial to avoid forgetting, but also computes uncertainty over the network output appropriately weight the past examples in the light of new

---

Under review as a conference paper at ICLR 2020
After Task 1
After Task 2
After Task 5
Figure I: This figure illustrates our method. Leftmost figure shows the result of training on task 1. Examples corresponding memorable-past, shown with big markers, chosen using GP formulation of the neural network. These points usually are the ones that support the decision boundary. Middle figure shows the result after task where new network functions are regularised at memorable-pat examples to give the same prediction as the previous ones. The resulting green decision boundary classifies both task and 2 well. The rightmost figure shows the result along with memorable-per past of each task where the performance over the past tasks is maintained.
ones. Motivated by the GP perspective, propose a new loss function for cuncciio-reeulaaiiste with deep networks. A Laplace approximation this objective enables scalable training. Our work in this paper focusses on avoiding forgetting, but it opens new direction for life-long learning methods where regularisation methods are naturally combined with memory-based methods.
Other related works. Broadly, existing work continual learning can be split into three ap- proaches: inference based, memory/rchearsal based, and model based. Inference based approaches have mostly focused on weight regularisation, with some recent efforts on functional regularisa- tion. Our work falls in the latter category. Memory based approaches either maintain memory of past data examples (Rebuffi et al., 2017) or train generative models on previous tasks to rehearse pseudo-inputs (Shin et al.) (017). An advantage of our method compared to previous ones is that building memory does not require solving an optimisation problem: the computation simply involves a forward-pas through the network followed by sorting (see Section 3.2).
Similarly to our work, there have also been some efforts in combining the different flavours of approaching continual learning, c.g., VCL plus coresets (Nguyen et al. (2018) and Pradient-Episodee Memory (Lopez-Pat Paz & Ranzato 2017 Chaudhry et al., 2018). [Bnjaamin et al. (018)) have proposed similar combination for functional regularisation. In these approaches, two separate methods usually used for regularisation and eemorry-bildiing Chaudhry al. 2019) has focussed improving memory- building methods, but this separation has remained the case. In contrast, in our approach, both of these are done within the same GP framework by using the method of Khan al. (2019).
Recent follow-up work (Aljundi et al.) 2019,
Finally, model based approaches change the model architecture during training (Rusu et al., 2016), and this can be combined with other approaches (Schwarz al. 20188). It possible use similar features in our GP based framework, which is an interesting future direction to be pursued.
2 CONTINUAL LEARNING WITH WEIGHT/FUNCTIONAL REGULARISATION
In deep learning. minimise loss functions estimate network weights. For example, in super- vised multi-class classification problems, we are given dataset D of N input-output pairs with outputs y,. a vector of K classes, and inputs x,, vector of length D, and our goal is to minimise loss which takes the following form: Z(w) + 5R(w), where [(w) := + ΣN ((y,, (x,)) with deep neural network f,. (x) E RK and its weights w. ((y.9) denotes a differentiable loss function be- tween output y and its prediction J. R(w) is a regularisation function (usually an La-regulariser R(w) = w w) and 8 > 0 controls the regularisation strength. Standard deep-learning approaches rely unbiased stochastic gradiintgr of the loss ε, which usually requires access to all of the data examples for all classes (Bottou) 2010). It is this unbiased, minibatch setting where deep-learnin excels and achieves state-of-the-an performance on many benchmark datasets.
2

---

and Under review as a conference paper at ICLR 2020
In reality, do not always have access to all the data at once, and it is not possible to obtain unbiased stochastic gradients. New classes may appear during training and old classes may never be seen again. For such settings, vanilla mini-batch stochastic-gradee
methods lead to catastrophic forgetting of past information (Kirkpatrick et al. 2017). Our goal in this paper is to design methods that can avoid such catastrophic forgetting. We focus on a particular setting where the classification task is divided into several tasks, c.g., a task may consist of a classification problem over subset of classes. We assume that the tasks arrive sequentially one after the other. Once the learning is over, we may never see that task again. Such continual-eearing settings have been considered in previous works (Kirkpatrick et al. 2017] Nguyen et al. 2018 Zenke et al., 2017), and our goal is to avoid forgetting of old tasks in this setting.
Recent methods have proposed weight-regulariiation as a way to combat catastrophic forgetting. The main idea is to keep the new network weights close to the old ones, e.g., when training task r while given network weights W,-1 trained on task -- 1, we can minimise the following loss: (w) + 5(w w,-1) (w w,-1), where (ww) is the loss defined over all data examples from task t and F, is a preconditioning matrix that favors the weights relevant to the past tasks more than the rest. The Elastic Weight Consolidation (EWC) method (Kirkpatrick et al. 2017). for example, uses the Fisher information matrix as the pre-conditioner, while Ritter et al. (2018) use the Hessian of the loss, and VCL (Nguyen et al. 2018) essentially employs the precision matrix of the variational approximation to do the same. Such weight-space methods reduce forgetting but do not produce satisfactory results.
The challenge in using weight-regullaiistion lies in the fact that the exact values of the weights do not really matter due to parametric symmetries (Benjamin al. 2018 Bishop 2006). Since only the network outputs matter, an alternative approach is to directly regularise these. [Bnnjmmin et al. (0188) propose to use an L2-regulariser over the function values on data examples from past tasks:
min (ww) Σ
Σ (lx(x))mm..110 ieM,
w
where M₁ is the set of small examples for task stored in the working memory (Lopez-Paz & Ranzato 2017, Rebuffi et al. (017]. One major issue with this approach is that the La-regulariser weights all the data points equally. In reality, some examples in the memory are more important than others to learn given task. In addition, the uncertainty of the prediction is also ignored. As we will show, working with distributions over functions allows us to address these two issues.
Gaussian processes (GPs), for example, enable automatic reweighting of old tasks in light of new one, which happens by using the posterior covariance. Unfortunately, both scalability of GPs as well as the necessity to save many instances of the past makes them impractical. A recent approach by Titsias et al. (2019) attempts to address these issue by employing sparse GP methods with inducing points and using a neural network feature map. However, their approach has some difficulties. Their approach heavily depends on a proper choice of inducing points, which are normally obtained via an ad-hoc procedure. Additionally, they propose using the last layer of the neural network as kernel features, which is limiting as it does not use the whole network's 's weights in the kernel.
Ideally, we would want to perform functional regularisation with deep learning optimiser while borrowing ideas from the GP methods. Our work in this paper takes step in this direction. Our proposal is to use a GP formulation of neural networks to perform functional regularisation that still allows training with standard deep-learning methods. We also show that the GP view helps in selecting an informative set of memory points.
3 DAACTIONAL-REGULAIISATIIII OF MEMORABLE PAST (FROMP)
We will now describe our proposed method. The first step is to use the GP formulation of Khan et al. (2019) compute Gi-like posteriors over functions. Then, we propose method to identify a set of memorable past examples for a task. Finally, we describe our discuss approximations used to build scalable method.
3

---

Under review as a conference paper at ICLR 2020
3.1
FROM DEEP NETWORKS TO GAUSSIAN PROCESS POSTERIORS
Khan et al. (019 propose an approach to convert deep networks into Gaussian processes. Their main result (see Theorem in their paper) states that, local minimiser of ((w) gw"w,
Laplace approximation of the posterior over w is equívalent to the posterior distribution linear model. Specifically, they use the following scalable variant of the Laplace upproximation:
N p(w(D) & q(w) NW Σ), where μ W. and E Σ J(xi) A(x, 6lp,
where J(x,) == ( is a K x Jacobian matrix (P being the number of parameters), and y)x,y: == (yy, f) is the K x K Hessian of the loss with f' = f,, (x), all evaluated at w w,, They show that (() equal the posterior of the following linear model:
y = J(x)w te, with e (A(x) y)) and w N(O,0
(3)
where the observations are defined as Y; J(x,w (A(x, r(x,,;) with the residual r(x,y) := V,f(y,f). f). They also show that the predictive distribution of this linear model is equiv- alent to that of a GP regression model defined with K neural tangent kernel (NTK) (Jacot ct al. 2018):
y = [GP(X)+ +e, with E NOO (A(x.))))) and (xp(x) GP (o. 10.5-10 ((5-J(())(x)))
(4)
The above result gives posterior in the defined observation space of y. In the case where we find map from ý to y, the above inference problem allows for reparameterisation into the original data space. Therefore, we will obtain a GP that serves as an approximation to the neural network posterior. Below, we demonstrate this for binary classification with sigmoid link function and Bernoulli likelihood (see Appendix A for details and extension to multi-class classification).
We parameterise the Bernoulli likelihood by p(x) (r(f., (x)) where is the sigmoid function (note that we now have scalars (f.y...) as we are dealing with a single output). The residual and loss Hessian for this model are given by r(x, y) = p(x) -y and A(x y) p(x)(1-p(x)) respectively, where y E {0, 1). Because residuals are linear in y and the Hessian is independent of y, we can substitute the definitions of ý and r(x, y) into Eq. B and rearrange for y,
y = p(x) + A(x ()Jx))( w.) +T, with r ~ N(O, A(x, and w 1I).
(5)
(X)
The Laplace approximation in Eq. 2 the posterior of this model after observing data D. We can equivalently write the posterior predictive as a GP (Rasmussen) 2003) with mean and covariance
mxx= == Bg(iw) [x]] p(x. + A(x, (x)( - w.) p(x).
k(x, x := Bgiw) [(fin -
- m(x) -
Axxx y(x) [(w w.)(w J(x))xxx))
A(X EJE A(x..y)))
A GP defined with the above mean and covariance function can be viewed as an approximation to the posterior process over network outputs. Throughout the paper, we will denote the process obtained at weight w, by distribution 4w. (f) where f is vector of f(xx evaluated at many different inputs We will use this GP posterior predictive for functional regularisation.
3.2 MEMORABLE ABLE PAST
In the previous section, derived GP posterior approximation over the network outputs. Now, we propose a method to obtain a small set of examples that are crucial to avoid forgetting. We first note that the posterior GP mean in Eq. corresponds computation of (6-1K- K A -->- where A is block-diagona. matrix containing all A, A(x, y;) and K is the NTK defined in Eq. 4 The predictions therefore strongly depend on the eigenvalues of the preconditioning matrix. Selection of important data examples therefore boils down to selecting
kernel Ridge regression that requires the
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Functional Regularisation of Memorable Past (FROMP)
1: Initialise M < Ø, < 0, W Wo-
2: for task / 1,2,3........ .. do
13:
function f fr.ggrad(, mi:r.
3: for previous task = 1,2. ....! do Precompute m,, K2 using Eq. 5: while not converged do
14:
- 0
15:
for task 1,2, !.., do
16:
f, vector of (x)( Vxj E Mr
6:
Stochastic gradients
17:
J, - v.rfs
7: R**+* ++ fz.grad(w, :::-: KT
18:
g J.K, (f,-
8:
Adam update for using gradients
19:
return
9:
s <- s (ieg( J(x;)' A(X, A;)x); (x,))
10:
Σ < diagonal matrix with diagonal 1/(s 6)
20:
function memorable memorable.past w)
21: Calculate A(x, y2) for all x; in Dr-
11:
M, - memorable.par (2,
22:
return M examples with highest (xx;,1))
12:
M MUM,
Figure 2: pseudo-code for our FROMP algorithm. The additional computations on top of Adam are in lines 4 and 7, where we add the contribution from the functional- regiliaaarrelllariaattioon term. After every task, we update the memorable past Eq. 9-11 which involves a matrix inversion size M, and forward pass through the network to compute Axx
important columns of this matrix, such that the predictions remain unchanged. The theory of lever- age score sampling (Alaoui Mahoney 2015 [ach 2013) suggests picking the points proportional the leverage score defined be the diagonal of the following matrix: K(K 5A-1)-1 Typically, this matrix is difficult to compute and methods are employed to approximately obtain the leverage score (Alaoui & M
For deep-lcarning applications too, exact computation of leverage score is very difficult. We instead propose a simple solution. Since the eigenvalues of the previous matrix heavily depend on 5A,1, we can pick the data examples by simply sorting A; For the inverse of K + 6A to have high eigenvalues, we should favour examples with smaller values of A, Therefore, we simply sort the A; and pick the top examples as the most relevant ones. This simple solution very effective for particular problem because A; noise variances for the data examples, obtained by using an already trained network. These are second derivatives of the loss for data examples, and so also reflect the sensitivity of the decision boundaries if particular data point is perturbed. Therefore, they tend to reflect the relevance of data examples. An example is shown in Figure [ where we clearly see that our solution picks the examples lying close to decision boundary. Computation of A; requires us to run the forward pass to get the ((v,.);) and then compute its second derivative with respect to 9,- both of which cheap operations. Throughout the paper, examples chosen by this method are referred to as the memorable past examples, and denoted M, for task
3.3
REGULARISATION
So far, we described the construction of the GP posterior predictive over the function space, as well the construction of set of memorable- cerorabbe-ppsttt examples. We now ready describe our objective function where we employ these two to perform functional regularisation.
Suppose that we given network weights w,-1 that are obtained by training over data examples from task - 1. Our goal then is to train network with weights w such that its performance on memorable past Miz-1 unchanged. We denote the vector of function outputs over these examples by aj:--I- A straightforward idea is to directly optimise the weights w such that the predictions using (f,) are good on current tasks while the predictive distribution (at:-1) is close qw (a):-1). Since the number of tasks can be very large, we choose to regularise each task separately, i.c., we will match q (a,) with qw,-, (a,) separately for all tasks < 1, in line with Titsias al. (0199) These choices give us the following objective function with trade-off parameter T:
min
TB BB)
Σ av.. f(x,)) icD,
Σ (xx qw, (a,)).
(8)
5

---

Under review as a conference paper at ICLR 2020
Table 1: Train accuracy of FROMP and batch-trained Adam (upper bound on performance) on variations of a toy 2D binary classification dataset, with mean and standard deviations over 10 runs (3 runs for Adam). FROMP performs well across variations. See Appendix D.2 visualisations.
Dataset variation
FROMP
Batch Adam
10x less data (400 per task)
99.9% = 0.0
99.7% = 0.2
10x more data (40000 per task) Introduced 6th task Increased std dev of each class distribution
96.9% # 3.0
99.7% + 0.0
97.8% 3.3
99.6% #.1
96.0% = 2.4
96.9% 2
2 tasks have overlapping data
90.1% = 0.8
91.1% 0.3
A major issue with this objective is that computing gradients wrt. the posterior predictive will be computationally heavy as it involves gradients of the Jacobian and A,, which involve higher order derivatives. During training on a task. we have no distribution on parameters (cf. Laplace approximation) and can treat as fixed. Therefore, taking the gradient wrt. the parameter of the above objective will only involve differentiating the mean function m(x;) formed by the neural network, and we have equivalence to the following objective:
T Σ Elv, (x))) + Σ m ms, iED,
min W
Wy-155 (Ms,w
(9)
where - and -.,- are the vector and matrices containing the corresponding mean and co- variance function of the network weights w,-1 evaluated at the memorable past for task s. Eqs and are used to calculate m and Kw,.L. respectively. Essentially, this gives us something similar to T instead of the L2 distance, kernel is used to improve weighting of examples. The final FROMP algorithm is summarised in Algorithm Please see App. B for details on scaling to the multi-class setting.
The additional time complexity of this algorithm above vanilla Adam is O(MPKr) per optimisation step, and O( 1r ]r+ NP) once per task. O(NPK) is for updating the Laplace approxi- mation Σ at the end of training on a task, as the Jacobian for each input training point needs to be calculated (as in Eq. ( This is similar to EWC (Kirkpatrick al., 2017). Note that, we diagonal approximation to Σ, inverting it is fast. O(PPKt) is for calculating each of the r matrices Kwy.1.s. and (M2)) is for inverting them. This is done once before training on each task (Eq, 7 Finally, during optimisation of the objective function, we need O OMMPK) ) for differentiating the mean function ). All of these additional costs are small for small M, except for O(N PK), which can be reduced by randomly sampling subset of the task's data. Note that it is possible to be stochastic in sampling a subset of previous tasks' memory during every update, removing the time complexity dependency on f. Doing this gives the same overall complexity per step as in Adam (as M is usually of the same order as the mini-batch size).
4 EXPERIMENTS
We run the proposed method FROMP on toy datasets, permuted MNIST and split MNIST LeCCnn ct al. 1998 Goodfellow et al. 2013), and split version of CIFAR-10 and CIFAR- 100 (Krizhessky
et al. 2009). We optimise the objective in Eq. Juinng Adam (Kingma & Ba 2015) with parameter speed up training. To identify the individual benefits of the kernel in the loss (Eq. 9 as well as the selection of data points, we compare the following four methods: FROMP uses kernel and the proposed example selection, while FRORP instead uses randomly chosen examples. Further, we replace the kernel by an identity matrix leading to func- tional L2 regularisation and call the corresponding methods with and without our example selection technique FROMP-/2 and FRORP-/2.
Bi 0.99 and further gradient clipping
4.1 TOY DATASET
We first test FROMP on many variations of a toy 2D binary classification dataset like that in Figure We want to test its performance when exposed to different datasets of varying difficulty. We use

---

Under review as a conference paper at ICLR 2020
6
Validaton accuraty
K3
10
44
6
0.7
20
FROMPLL
0.5
FROMP
70
12
15
77
76
15
16
10
40
100
200
Number examples
(a) random example kernel
(b) memorable past kernel
(c) accuracy vs. memory size
Figure 3: Permuted MNIST: added kernels across classes (with subtracted diagonal for visualisation purposes), and performance as function of memory size (average accuracy after 10 tasks). Mem- orable past examples lead to more uniform kernel structure that prevents weighting previously overfit examples too highly, e.g. task in the random selection exhibits strong correlation and low variance. As we reduce the number of examples in memory. FROMP gracefully reduces accuracy.
2-hidden layer MLP (with 20 hidden units in each layer) for all experiments. Appendix D.3 details hyperparameter selection. In Appendix [DI] we show the brittleness and inconsistent behaviour of weight-space regularisation methods. In contrast, FROMP performs extremely well across many variations, showing consistently good results (see Table and Appendix D.2 for visualisations).
The average validation accuracy on Permuted-MNIST (10 tasks) and Split-MNIST. "200p/t" denotes that 200 examples are selected for each task. We report mean and standard devia- tions over runs, and use results from Nguyen et al. (2018) for baselines. FROMP is state-of-the-art, Additionally, as we reduce the number points (Figure 3). FROMP gracefully reduces accuracy, due to the clever choice of memorable past and the use of kernels in the functional regularisation.
Table 2:
Method
Permuted MNIST
Split MNIST
DLP (Smola et al.
2003
82%
61.2%
EWC (Kirkpatrick
2017
84%
63.1%
SI (Zenke et al.
2017
86%
98.9%
Improved VCL Swaroop et al.
2019
93% +
98.4% 0.4
+ random Coreset
94.6% # 0.3 (200 p/t)
98.2% 0.4 (40 p/t)
FRCL-RND (Titsias al.
2019
94.2% # 0.1 (200 p/t)
96.7% 1.0 (40 p/t)
FRCL-TR Titsias et al.
2019
94.3% + 0.1 (200 p/t)
97.4% + 0.6 (40 p/t)
FRORP-L2
87.9% = 0.7 (200 p/t)
98.5% (40 p/t)
FROMP-L2
94.6% 0.1 (200 p/t)
98.7% 0.1 (40
FRORP
94.6% 0.1 (200 p/t)
99.0% + 0.1 (40 p/t)
FROMP
94.9% 0.1 (200 p/t)
99.0% 0.1 (40 p/t)
4.2 PERMUTED AND SPLIT MNIST
Permuted MNIST consists of series of tasks where each task is fixed permutation of pixels the entire labelled MNIST dataset. Like in previous work (Nguyen et al.) [2018) Kirkpatrick et al. 2017: Zenke al. 2017 Titsias al., 2019), we implement fully connected single-head network with two hidden layers, and report performance after 10 tasks. Each hidden layer consists of 100 hidden units and ReLU activation functions. We set the learning rate to 0.001, batch size to 128, and learn each task for 10 epochs. We use 1 for both FROMP and FROMP-L2 (see Eq. 9p.
The Split MNIST experiment was introduced by Zenke et al. (2177) and consists of five binary clas- sification tasks built from MNIST: 0/1, 2/3, 4/5, 6/7. and 8/9. We use fully connected multi-head neural network with two hidden layers, each with 256 hidden units and ReLU activation functions. Following the settings of previous work, select 40 memorable points per task. The learning rate set to 0.0001, batch size to 128, and we learn each task for 15 epochs. We find optimal parameters 1 and 10 for FROMP and FROMP-/2 respectively.
7

---

Under review as a conference paper at ICLR 2020
O
0
0
0
0
/
/
/
/
1
/:
1
1
/
1
1
4
2
2
2
2
2
3
2
2
2
2
>
2
2
2
à
2
Z
2
a
3
3
3
3
3
3
3
3
3
3
3
3
8
3
3
3
3
3
3
3
4
/
4
4
4
4
4
4
4
4
4
4
4
4
2
4
4
4
5
5
5
5
5
5
5
3
5
5
5
5
5
$
6
4
5
6
6
6
6
6
6
6
6
6
4
6
6
0
6
6
6
6
7
7
7
7
7
7
7
7
2
7
1
+
*
7
7
<
+
-
h
7
&
8
8
8
8
8
8
1
K
4
4
4
8
8
p
8
8
9
9
9
9
9
9
9
9
C
9
9
4
9
(a) Random Past
(b) Memorable Past
randomly selected samples and memorable past in comparison. 10 samples per class and method. In line with the toy example in Fig. I memorable past examples appear to be closer to the decision boundary of classifier: many samples of the memorable past are possibly hard to distinguish from other classes.
Figure 4: Split MNIST:
We report the final average accuracy for both benchmarks across all the tasks in Table 2 posed method achieves better performance than the weight-space methods EWC and VCI., as well as compared to the function-space method FRCL. that is based on GP formulation. Further, the benchmarks show superior performance of both the approach to select important examples (Sec.3.2) and the functional regularisation using the kernel (Sec. 3.3 Memorable examples improve perfor- mance of the naive and efficient FRORP-Le method by more than 6% on permuted MNIST and by 0.2% the split MNIST. Standard deviation is also reduced in both cases. FROMP does not profit much from memorable examples compared to FRORP, probably because the performance is already close the maximum achievable. Furthermore, Fig. 3c shows that our selection method greatly reduces the number of memorable points required: the L2 algorithm with random points requires over 100 points to match the performance when using 20 carefully selected points, and 200 points to match performance with 40, respectively. When combined with kernel-basee functional regulari- sation, obtain the best-performing method, particularly when memory size is small. In line with the toy example in Fig. [ lected examples. The memorable past examples appear be more special cases, and may therefore lie closer to the decision boundary.
The pro-
Fig. 4 illustrates the memorable past examples compared to randomly se-
Figs. 3a and 3h show the summed kernels across all the classes in permuted MNIST for a random and memorable set of points. For visualisation purposes, the diagonal is suppressed. Note that ran- dom points lead to less uniform weighting in the kernel, making even more important in functional regularisation, leading to better performance (FRORP FROMP). All memorable points are im- portant and the kernel more uniform. In Fig. 3a the kernel's weighting of task is very different from other tasks, leading different magnitudes in the functional regularisation among tasks and therefore to eventual forgetting. The kernel further tells us that the tasks are correlated, as expected.
4.3 SPLIT CIFAR
Split CIFAR is far more complex than the MNIST experiments and consists of tasks. The first task is the full CIFAR-10 dataset, followed by tasks, each consisting 10 consecutive classes from CIFAR-100. We follow the SI paper (Zenke et al. 2017) for our model architecture, using a multi-head CNN with 4 convolutional layers, followed by dense layers with dropout. We use learning rate 0.0001 and batch size 256. All tasks are learned for 80 epochs and of 10-200 examples. We use 0.1 and 0.05 for FROMP and FROMP-L2 respectively. In addition to continual learning baselines, report the performance of networks trained from scratch
memory
8

---

Under review as a conference paper at ICLR 2020
Validation acuracy
FROMP FROMP-4 ?
Voldbtion sccrary
0.7
$7
FWVC
VCL-CGrnest
From Scratch
FRORP L2
Join
0.6
FROMPAL
66
20
J0
70
100
Number examples
(a) final performance on individual tasks
(b) accuracy vs, memory size
Figure 5: Split CIFAR: performance for each task, and performance variation as function of memory size, after training on the final task. We run all methods 5 times and report the mean and standard error. For baselines, we train from scratch on each task and jointly on all tasks achieving 73.6% 5 +..4 and 78. 1 1% 0.3, respectively. The left figure reports results for 200 memory examples per task. The final average validation accuracy of FROMP is 76.2% + 0.4, FROMP-L2 is 74.6% 0.4, SI is 73.5% + 0.5 (result from Zenke et al. 2017)) EWC is 71.6% = 0.9, VCL random coreset is 67.4% + 1.4. FROMP outperforms all other methods, and is close to the performance of the jointly trained model, particularly in tasks 4-6. Additionally, performs well, even with only 20 examples per task.
reduce the memory size, FROMP still
on each task. These cannot profit from forward/backward transfer. We also report the performance of network jointly trained on all tasks. This is upper bound performance.
The experimental results in Fig. Sasshow that FROMP outperforms other continual learning methods by notable margin. The weight-space methods employed baselines either cannot learn later tasks to a high accuracy (EWC, SI), or forget previous tasks (VCL). Interestingly, averaged over all tasks, FROMP also out-performs 'from scratch' training and achieves performance close a jointly training on all tasks by margin of less than 2%. Note that on tasks 4-6, FROMP achieves the same accuracy as the jointly trained model, showing no forgetting. We also calculate the backward transfer metric BWT from Lopez-Paz & Ranzato (2017). We find that FROMP has score of -2.6 0.9, which is joint best with EWC's score of -2.3 + 1.4 (VCL+coresets has -9.2 + 1.8). Although the cost of forward transfer. We calculate
EWC performs well in backward transfer, it does a forward transfer metric as the average of the improvement independently trained model on that task (see App. App more precise definitions of these metries). FROMP achieves a forward transfer 6.1 0.7, whereas EWC has 0.17 + 0.9 and VCL+coresets has 1.8 3.1. Although we only focussed preventing catastrophic forgetting, of forward transfer, a key requirement in continual learning. Overall, given final average accuracy, backward transfer and forward transfer. FROMP clearly outperforms the other baselines.
accuracy on a new task over an
find evidence
In contrast to the rather simple MNIST benchmarks, both the benefit of selecting memorable points as well as using the kernel are clearly visible in Fig. 5b If we only memorise few examples, the performance gap due to using the kemnel is around 4%. The selection of memorable points according our metric leads to an increase performance of around 7%. Applying both kernel and memorable point selection increases the performance by up to 1 11%. Additionally, standard deviation reduced when using memorable points or the kernel. proposed algorithm are vital in achieving state-of-the-ard performance on this benchmark.
clear that both parts of the
DISCUSSION
We propose FROMP, a scalable funetion-regulaaisation approach for continual learning. FROMP uses a GP formulation of neural networks select memorable past examples, regularising them using a kernel, and achieving sfate-of-the-an performance across benchmarks. This work enables way of combining regularisation methods and memory-based methods in continual learning. Future research could investigate other ways of selecting a memorable past (e.g. fixed memory size), more efficient ways of calculating kernel matrices, and the case where data does arrive in tasks.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ahmed Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In Advances Neural Information Processing Systems, pp. 775-783, 2015.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. ArXiv, abs/1903.08671. 2019.
Francis Bach. Sharp analysis of low-rank kernel matrix approximations. In Conference on Learning Theory, pp. 185-209, 2013.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arXiv: arXiv:1805.0888 180..08289, 2018.
Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT pp. 177-186. Springer, 2010.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. ArXiv, abs/18 8 812.004200 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. ArXiv, abs/1902. 10486, 2019.
Sayna Ebrahimi, Mohamed Elhoseiny, and Mareus Rohrbach Trevor Darrell. Uncertainty-guided continual learning with bayesian neural networks. ArXiv, abs/1906.02425, 2019.
Sebastian Farquhar and Yarin Gal. A unifying bayesian view of continual learning. ArXiv, abs/1902.06494, 2019.
lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- investigation of catastrophic forgetting in gradient-base neural networks. arXiv preprint arXiv: 131266131. 2013.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.
Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate inference turns deep networks into gaussian processes. NeurlPS, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bal Barkkks--aikiaa,,,aa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):321-3356, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009,
Yann LeCun, Corinna Cortes, and Christopher JC. Burges. The mnist database of handwritten digits. 1998.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Cuong V Nguyen, Yingzhen Li. Thang D Bui, and Richard E Turner. Variational continual learning. ICLR, 2018.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pp. 63-71. Springer, 2003.
10

---

Under review as a conference paper at ICLR 2020
Seivestre-Alvise Rebufii, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp. 3738-37 3748, 2018.
Andrei A. Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, 1ess1606.0461,, 2016.
Jonathan Schwarz, Jelena Luketina, Wojciech Marian Czarnecki, Agnieszka -aaaska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning. ArXiv, abs/1805.06370, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In ICML, pp. 4548 4557. PMLR. org, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, 2017.
Alexander Smola, Vishy Vishwanathan, and Eleazar Eskin. Laplace propagation. In NIPS, pp- 441-448, 2003.
Siddharth Swaroop, Cuong V Nguyen, Thang D Bui, and Richard E Turner. Improving and under- standing variational continual learning. arXiv preprint arXiv: 1005.00990.0000000 1905.02099, 2019.
Michalis Titsias, Jomathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning using gaussian processes. arXiv preprint arXiv: 11356, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 9997-3995. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2020
A DETAILS ON DEEP NETWORKS TO GAUSSIAN PROCESSES
In this section, we detail the map that is presented in Sec. 3.1 go from the posterior of deep neural network to Gaussian process. The map for linear regression and logistic regression can be found in the appendix of Khan al.) (2019). We recap the map for logistic regression and, in addition, walk through the case of softmax regression here. Recall the linear model from Eq. 3
Model: y = J(x)w with ~ N(O. (A(x,))) and w N(0,0-1Pp)
(10)
Data: y; = J(x) - (A)x,);)) y:).
(11)
Inference in this model yields the deep neural network posterior g(W) in Eq. 2
Logistic regression: as mentioned in Sec. 3.1 the parameter of the Bernoulli likelihood is given by applying the sigmoid link function f(x) and defined p(x) := (fu, (x)). Further, the derivative
and Hessian of loss were particularised as r(x. y) = p(x) and A(x, y) = p(x)(1 p(x)). We write
g N(O, 1) for standard noise and plug the quantities model (Eq. 10p:
J(x)w. p(x) -Y J(x)w (A(x, 6t. y(x
(12)
which can be re-arranged to a new model that has an equivalent posterior, but where inference is done by observing original data points D:
y p(x) y)())) w.) with N(O. A(x y)) and NOO,
(13)
Softmax regression is a simple extension to the logistic regression case: we have a categorical likelihood with parameter p(x)= == softmax(f,., (x)) and labels y; that are standard one-hot encoded basis vectors, i.e. have a single 1 at some position and otherwise zeros. Then we have r(x.y) = p(x) y and A(x. y) = diag(p(x)) - p(x)p(x)" Unfortunately. A(x, y) is of rank K I and therefore we cannot uniquely invert it. The theory Khan et al. (2019) therefore does not allow this because it requires A(x y) 0. However, we can still write model in line with Eq. 13 that we expect to have a posterior close to that of the deep neural network:
y = p(x) + A(x. --w.) +T with r N/O, and w (10.0-1p)). (14)
Conversion to GP can be achieved by taking the expectation and covariance the linear model. To obtain the GP prior, take the expectation of the linear model (10) over the prior p(w). To obtain the posterior, we take the expectation over q(w). Since we are interested in the posterior process, the latter is of interest and calculated for the logistic case in Sec. 3.1 The softmax regression case works equivalently but yields a K x K kernel, which can be represented as a symmetric tensor, or a NK x K matrix over data set D with N examples. To avoid growing complexity with more classes, we model the GPs for each class as being independent, as discussed in Appendix B
B REDUCING COMPLEXITY IN THE MULTICLASS SETTING
For the softmax regression with K classes, we build individual GP for each class, in order to avoid growing complexity with more classes. We utilise y(8) to denote the k-th item of y in Eq. (14). Then y(k) will be:
(0)) = ( + +A(x ((x - w.) + (R)
(15)
Here A(x, ((k) denotes the k-th row of the Hessian matrix. We transform the K models to function space and treat them independent. This allows us to split the KL-divergence in Eq. g into K individual divergences, and we obtain the simplified final objective,
min Σ My. fx) + ieD
m w
m S,
[ (K) Wy-n.a
(m
(R) k.w m
(16)
s=1
12

---

Under review as a conference paper at ICLR 2020
C FURTHER DETAIL ON CONTINUAL LEARNING METRICS REPORTED
We report a backward transfer metric and a forward transfer metric on split CIFAR. The backward transfer metric is exactly as defined in Lopez-Paz & Ranzato 2017). The forward transfer metric is measure of how well the method uses previously seen knowledge to improve classification accuracy on newly seen tasks. Let there be total of 7 tasks. Let Ri be the classification accuracy of the model on task 1j after observing the last sample from task f;. Let Rinnd be the classification accuracy of an independent model trained only on task i. Then,
T-1
Backward Transfer, BWT
RI. - -R,i-
T-1 1
i=1
Forward Transfer
Σ F=2
REi R
D FURTHER TOY DATA EXPERIMENTS
This section provides further information and visualisations of toy 2D datasets, as well as hyperpa- rameter settings for VCL.
D.1 WEIGHT-SPACE REGULARISATION'S INCONSISTENT BEHAVIOUR
Table 3: Train accuracy of FROMP, VCL (no coresets), VCL+coresets and batch-trained Adam (an upper bound on performance) on toy 2D binary classification dataset, with mean and standard deviations over runs for VCL and batch Adam, and 10 runs for FROMP. 'VCL' is without coresets. VCL-RP and FRORP have the same (random) coreset selections. VCL-MP is provided with 'ideal' coreset points as chosen by an independent run of FROMP. VCL (no coreset) does very poorly, forgetting previous tasks. VCL+coresets brittle with high standard deviations, while FROMP is stable.
FROMP
FRORP
VCL-RP
VCL-MP
VCL
Batch Adam 99.70% + 0.03
99.6% 0.2 98.5% 0.6 92% + 10 85% 14 68% =
Table 3 summarises the performance (measured by train accuracy) of FROMP and VCL+coresets on toy dataset similar to that in Figure [ FROMP very consistent, while VCL (with coresets) is extremely brittle: it can perform well sometimes (1 run out of 5), but usually does not (4 runs out of 5). This is regardless of coreset points chosen for VCL. Without coresets, VCL forgets many past tasks, with very low performance. Previous work (Farguhar & Gal 2019) has argued that this is inevitable for woight-regullarisationnoool methods such as VCL and EWC.
We now 3 runs with different random seeds of VCL-MP from Table 3 the coreset is chosen from an independent run of FROMP, with datapoints all on the task boundary. This selection of coreset intuitively better than random coreset selection. Please note that the results show here not specific to coreset selection. Any coreset selection (whether random or otherwise) all show the same inconsistency when VCL. is run them. This behaviour is not specific coreset choice.
13

---

Under review as a conference paper at ICLR 2020
After Task 5
After Task 5
After Task 5
nLA
Figure 6: Three of VCL-MP toy 2D data. These the middle performing 3 runs out of runs with different random seeds. VCL's inconsistent behaviour clear.
D.2 DATASET VARIATIONS
This section visualises the different dataset variations presented in Table We pick the middle performing FROMP run (out of 5) and batch Adam run to show.
Figure 7: FROMP (middle performing of 5 runs) and batch Adam on a dataset 10x smaller (400 points per task).
14

---

Under review as a conference paper at ICLR 2020
للدية
لل
Figure 8: FROMP (middle performing of runs), left, and batch Adam, right, on a dataset 10x larger (40,000 points per task).
Figure 9: FROMP (middle performing of 5 runs), left, and batch Adam, right, on a dataset with a new, easy, 6th task.
Figure 10: FROMP (middle performing of 5 runs), left, and batch Adam, right, on a dataset with increased standard deviations of each class' points, making classification tougher.
15

---

Under review as a conference paper at ICLR 2020
Figure 11: FROMP (middle performing of 5 runs), left, and batch Adam, right, on a dataset with tasks having overlapping data, which is not separable.
D.3 VCL AND FROMP HYPERPARAMETES SETTINGS FOR TOY DATASETS
FROMP. We optimised the number of epochs, Adam learning rate, and batch size. We optimised by running various settings for 5 runs and picking the settings with largest mean train accuracy on the toy dataset in Figure T We found the best settings were: number of epochs=50, batch size=20, learning rate=0.01. The hyperparameters were then fixed across all toy data experimental runs, including across dataset variations (number epochs was appropriately scaled by 10 if dataset size was scaled by 10).
CCL+corsssts. We optimised the number of epochs, the number of coreset epochs (because VCL+coresets trains non-coreset data first, then coreset data just before test-time: see Seuyen ct al. )2018), learning rate (we use Adam to optimise the means and standard deviations of each parameter), batch size, and prior variance. We optimised by running various settings for runs and picking the settings with largest mean train accuracy. We found the best settings were: number of epochs=200, number of coreset epochs=200, standard normal prior (variance=l) l), batch size=40, learning rate=0.01. VCL is slow to run (an order of magnitude longer) compared to all other methods (FROMP and batch Adam).
16


Reviews:
[{"review_id": "SkxrQ_HCKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper uses a Gaussian Processes framework previously introduced in [1] to identify the most important samples from the past for functional regularization. For evaluation authors report their average accuracy on Permuted MNIST, Split-MNIST, and CIFAR10-100 and achieve superior performance over EWC, DLP, SI, VCL-Coreset, and FRCL.\n\nPros:\n(+): The paper is well-written, addressed the prior work quite well despite missing a few important work from the past (more on this later)\n(+): The paper is well motivated\n\nCons that significantly affected my score and resulted in rejecting the paper are as follows:\n\n1- lack of support for “scalability”:\nAuthors claim their method is scalable in several parts of the paper (abstract in line 7, Section 3 in the 1st paragraph, and Section 5 in Discussion). However, this claim is not supported in the experimental setting as the benchmark used are only toy datasets (Permuted MNIST, Split MNIST, and CIFAR10 followed by CIFAR100) where the maximum # of task considered is 10 and the maximum size of the datasets is 60K which is not convincing for ability to scale. There is also no time complexity provided. \n\n2- Incremental novelty over the prior work (FRCL by Titsias et al 2019):\nThis baseline is the closest prior work to this work which according to the experiments shown in Table 2 are slightly outperformed by the proposed method. (for example for P-MNIST the gain is 0.6%+-0.1) where there is a lack of complete discussion on how the two methods are different. Particularly I suggest that the authors elaborate more on their claimed differences stated on page 4, paragraph 5 such as “tractability of the objective function only when we assume independence across tasks”. Do authors mean assuming clear task boundary between tasks? If so, have they considered a “no-task” or an \"overlapping” task boundary in their experiment? Isn't it necessary to back up this if it is stated as a shortcoming of FRCL? Also, how are these methods differ in their computational expenses?\n\n3- Lack of measuring forgetting: \nThis is the most important drawback in the experimental setting. Authors indicate on page 3 “Our goal in this paper is to design methods that can avoid such catastrophic forgetting.” and reiterate on this on other parts of the paper yet there is no forgetting evaluation to support this claim. Authors can simply report the initial performance of the model on each task so that readers can compare it with the reported accuracy after being done with all tasks. Having a method with high average accuracy does not necessarily mean it has minimum forgetting. You can use forgetting measurements such as Backward Transfer (BWT) introduced in [1] or forgetting ratio defined in [4] for this assessment. \n\n4- Ambiguous claims about prior work:\n(a) On page 1, paragraph 3, when authors mention that methods such as GEM or iCaRL use random selection to pick previous samples, I think the line of follow-up work on these methods should be mentioned as well that have explored different techniques for sample selection and have provided benchmark comparisons (ex. [2,3]). In fact it would be beneficial if authors could compare the samples selected by their method versus other sampling techniques. \n(b) On page 1, paragraph 3, they mention some prior work such as GEM and iCaRL “do not take uncertainty of the output into account”. While it is true, there have been methods proposed that use uncertainty of the output for parameter regularization [5]. It appears to be a parallel work to this but it’s worth mentioning to prevent false claims.\n\n5- Claim on the state of the art should be double-checked:\n\tAlthough the results shown for the experiments are superior to the provided baselines, there is an important baseline missing which has achieved higher performance than the reported ones. Also missed to be cited in the prior work list. Serra et al [4] proposed a method at ICML 2018 called HAT, which is a regularization technique with no memory usage that learns an attention mask over parameters and was shown to be very effective on small and long sequence of significantly different tasks. They do not use samples from previous task but yet achieved good average ACC as well as minimum forgetting ratio. Note that 5-Split MNIST is not reported in [4], but a recent work has reported HAT’s performance on this dataset (https://openreview.net/forum?id=HklUCCVKDB) that achieves 99.59%. I recommend authors provide comparison of their own on the given benchmarks with the original HAT’s implementation (https://github.com/joansj/hat) before claiming to be SoTA. In my opinion, it is not an issue if a novel method achieves a slightly lower performance to the sota because I think it still adds value and proposes a new direction. However, a false claim should not be stated.\n\nLess major (only to help, and not necessarily part of my decision assessment):\n\n1- Providing upper bound?\nIt is common to show an upper bound for any continual learning algorithm by showing joint training performance which is considered to be the maximum achievable performance. I also recommend showing the naive baseline of fine-tuning for the proposed method  which often can give insight to maximum forgetting ratio.\n\n2- Forward transfer?\nRegularization techniques combined with memory might have an ability to perform zero-shot transfer or so called FWT. I recommend authors provide such metric to further support their method.\n\n3- Hyper parameter tuning?\nIt is also worth mentioning how the tuning process was performed. In continual learning we cannot assume that we have access to all tasks' data, hence authors might want to shed some light on this.\n\nReferences: \n[1] Khan, Mohammad Emtiyaz, et al. \"Approximate Inference Turns Deep Networks into Gaussian Processes.\" arXiv preprint arXiv:1906.01930 (2019).\n\n[2] Chaudhry, Arslan, et al. \"Continual Learning with Tiny Episodic Memories.\" arXiv preprint arXiv:1902.10486 (2019). (https://arxiv.org/abs/1902.10486)\n\n[3] Aljundi, Rahaf, et al. \"Gradient based sample selection for online continual learning.\" arXiv preprint arXiv:1903.08671 (2019). (https://arxiv.org/abs/1903.08671)\n\n[4] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557\n\n[5] Ebrahimi, Sayna, et al. \"Uncertainty-guided Continual Learning with Bayesian Neural Networks.\" arXiv preprint arXiv:1906.02425 (2019).\n\n\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------\n----------------------------------------------------------------------------------------------------------------------------------------------------------------------\nPOST-REBUTTAL Response from R1:\n\nThank you for taking the time and replying to comments. Here are my responses to authors' replies:\n\n[Authors' response:] 1. Scalability: Our algorithm only adds a small computational overhead on top of Adam on a standard neural network. This is what we mean by scalable. The additional complexity scales cubically in M, the coreset size. This is due to the inversion of the kernel in fr_grad. Another overhead is the computation of Jacobian which is order PKM, where K is the dimensionality of the output and P is the number of network parameters. Both of these additional costs are small for small coreset sizes M. We will add these details to make these points clear in the paper.\n\n[Reviewer's response:] I still insist on the fact that simply explaining the overhead of a method is not a support for scalability claim versus showing the performance on a large scale dataset and comparing it with other CL methods that also have high scalability given the fact that authors only use MNIST and CIFAR datasets.\n\n[Authors' response:] 4. Prior work: We discuss other works in Section 1 (“two separate methods are usually used for regularisation and memory-building”), and we will expand upon this sentence, going into more detail, and also referencing iCaRL and other works (including [3]). Note that our method of choosing a memorable past follows directly from the theory in Section 3.1, and is achieved with a single forward-pass through the trained network (as mentioned in the paper). Other techniques for sample selection do not integrate so naturally with the framework, and are not as straightforward to understand or implement either.\n\n[Reviewer's response:] I disagree with authors on this because GEM, its faster version (A-GEM (Chaudhry et al. 2018)), and all other methods explored in the recent study which I mentioned in my review (Ref#2) use the single epoch protocol and are perfect match to be compared with this method but there is no memory-based baseline except for VCL with coreset and FRCL (only for MNIST variations) which makes it difficult to measure this method's capabilities (performance, memory size, and computational time) against methods which only require one epoch to be trained.\n\nAuthors have provided FWT for their method as 6% which is unbelievably large for this metric (see GEM paper) and hence does not make sense to me. Please double check whether you computed this value right. \n\nWhile I accept the response for the remaining questions from authors but I am still concerned about the weak experiments and an issue brought up by R3 regarding lack of enough comparisons with FRCL on any other datasets besides split MNIST and P-MNIST. Also in  CIFAR experiment, what is the architecture used across the baselines? More importantly in results reported for VCL on CIFAR, it is not clear to me how authors obtained this results. Did they use a conv net? VCL was originally shown on MLPs only and it is one of the downside of this method that was never shown to be working in convolutional networks. Therefor, it is important to mention how they are obtained. This might explain the reason for the huge forgetting reported for VCL with coreset (−9.2 ± 1.8) as opposed to −2.3 ± 1.4 for EWC which is really strange as VCL even without coreset (on permuted mnist for example) is reported superior to EWC by a large margin (6%) in the original VCL paper. Overall I am concerned about the experimental setup and some of the reported results and hence intend to keep my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skgef0KaYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a new functional regularization method with gaussian process which has similar direction with recent two works (khan et al, titsias et al).\nTo perform functional regularization, they introduce small coreset which are selected from previous dataset instances, called memorable past. They select most memorable samples depends on eigenvalue. The model FROMP outperforms baselines and their ablations. However, the experiments are only performed on shallow networks, it is required to apply on much deeper networks, such as ResNet. Also, in the experiment results, I feel the performance of the FROMP largely depends on the number of the coreset, while 'important' selection just shows marginal effects even on split CIFAR. \nFROMP show higher performance than FRORP with only a few of examples, but it isn't meaningful results that anyway the performances are too poor that are even worse than old baseline, EWC. \n\nI have several wonderings on the paper.\n\n- How about of training time on FROMP? I wonder if utilizing or selecting memorable pasts requires much time for training.\n\n- Is there an analysis like figure 1 on real dataset, such as MNIST or CIFAR?\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJezZcn6FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper proposes a method for continuous learning called Functional Regularization of Memorable Past (FROMP) which maintains the output distribution of models on memory samples. FROMP uses the Laplace approximation and Gaussian process with neural tangent kernel (NTK) to approximate the output distribution. According to the leverage score strategy, the sample to be stored is selected. The leverage score strategy tends to select the sample of highest variances. \n \nStrengths\nTo some extent, I think the proposed method is novel, although there is a similar work named as Functional Regularisation for Continual Learning (FRCL). FROMP first uses NTK in Gaussian process for continual learning and proposes a new strategy of selecting memory samples.\nThe strategy of selecting samples to be stored is simple and effective.\nThe method achieves a good performance.\nThe paper is clearly written and easy to follow.\n \nWeaknesses\nIt needs more experimental comparisons between FROMP and FRCL, like adding comparison results of FROMP and FRCL for Split-Cifar. Currently, this paper only shows the performance on Permuted MNIST and Split MNIST but those two benchmark are quite simple and also the improvement is limited.  \nThe experimental section needs more detailed analysis. At least, in current version, it is not clear how many tasks in Permuted MNIST. The setting of hype-parameters for dropout are not provided.\n \nOther comments \nIn this paper, for Split MNIST experiment with multi-head, it shows that the method of EWC achieves worse results than SI. However, in my experiment, the precision of EWC is at least larger than 97%. In theory, I think they should have the similar performance and at least the discrepancy of accuracy between them is not as big as shown in this paper. I expect authors could explain this point.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work tackles the problem of catastrophic forgetting by using Gaussian processes to identify "memory samples" to regularize learning.

Although the approach seems promising and well-motivated, the reviewers ultimately felt that some claims, such as scalability, need stronger justifications. These justifications could come, for example, from further experiments, including ablation studies to gain insights. Making the paper more convincing in this way is particularly desirable since the directions taken by this paper largely overlap with recent literature (as argued by reviewers).

Decision: Reject