Improving Continual Learning by Accurate Gradient Reconstructions of the Past
Authors: Erik Daxberger
Siddharth Swaroop
Kazuki Osawa
Rio Yokota
Richard E Turner
José Miguel Hernández-Lobato
Mohammad Emtiyaz Khan
Venue: Accepted by TMLR ()

Abstract: Weight-regularization and experience replay are two popular continual-learning strategies with complementary strengths: while weight-regularization requires less memory, replay can more accurately mimic batch training. How can we combine them to get better methods? Despite the simplicity of the question, little is known or done to optimally combine these approaches. In this paper, we present such a method by using a recently proposed principle of adaptation that relies on a faithful reconstruction of the gradients of the past data. Using this principle, we design a prior which combines two types of replay methods with a quadratic weight-regularizer and achieves better gradient reconstructions. The combination improves performance on standard task-incremental continual learning benchmarks such as Split-CIFAR, SplitTinyImageNet, and ImageNet-1000, achieving $>\!80\%$ of the batch performance by simply utilizing a memory of $<\!10\%$ of the past data. Our work shows that a good combination of the two strategies can be very effective in reducing forgetting.

Published in Transactions on Machine Learning Research (11/2023)
Improving Continual Learning by Accurate Gradient Reconstruction of the Past
Erik Daxberger
ead5 @com.ac uk
University of Cambridge & MPI for Intelligent Systems, Tübingen
Siddharth Swaroop
siddhartthhlsenssavvrrr
Harvard University
Kazuki Osawa
Incuuiooscunn@ggoooe..com
Google Deep
Rio Yokota
rioyokota@gsic.tieee wwtpsssiittttttha...
Tokyo Institute of Technology
Richard E. Turner University of Cambridge
ret260ccom.cc uk
José Miguel Hernández-Lobato University of Cambridge
jmh233@com.ccc uk
Mohammad Emtiyaz Khan RIKEN Center for AT Project
emtiyaz. khon @riken..jp
Reviewed OpenReview: https: openrvve/p. net/ forum? id= jifpf
Abstract
Weight-reeuulaiization and experience replay two popular continual-learingg strategies with complementary strengths: while weight-regglaarzztioo requires less memory, replay can more accurately mimic batch training. How Despite the simplicity of the question, little is known or done to optimally combine these approaches. In this paper, we present such a method by using a recently proposed principle of adaptation that relies on a faithful reconstruction
combine them to get better methods?
the gradients of the past data. Using this principle, we design prior which combines two types of replay methods with quadratic weight-reeullrrzer and achieves better gradient reconstructions. The combination improves performance standard task-incremental continual learning benchmarks such as Split-CIFAR, SplitTinnlmagelt performance by simply utilizing a memory of < 10% of the past data. Our work shows that a good combination of the two strategies can be very effective in reducing forgetting.
and ImageNet-11000 achieving 80% of the batch
1 Introduction
Continual learning Parisi et al. 2019 aims for accurate incremental training over large number of in- dividual tasks/ /cxs//as........ This can potentially reduce the frequency of retraining in deep learning, making algorithrns easier to use and deploy, while also reducing their environmental impact (Diethe et. al.) 2019 Paleyes et al.] 2020 The main challenge in continual learning to remember past knowledge and reuse continue adapt new data. This can be difficult bocause the future is unknown and can interfere with past knowledge (Sutton 1986 Mermillod al. 2013 Kirkpatrick et al. 2017). Performance, therefore, heavily depends the strategies used represent and reuse past knowledge.

---

Published in Transactions on Machine Learning Research (11/2023)
Previous Methods
Our Method
EWC
Wold )'
F old ala( (w--w-- Wold)
EWC+FR+Replay
Wold
old
Wold
F FoldyM
(w-Wold) Fold (ald)M((WWool
Function Regularizer (FR)
Σ
(2i), San (2:))
(( (2:), (x:))
iM
M
Wold
M
Wold
M
Σ 1EM₂
Elyss fwo
Experience Replay-
Σ iEM
Elys, (wox))
M
y
M2
y
Figure 1: Using the principle of gradient reconstructions, we design new prior (right) to combine different types of regularization and memory- based methods (left). EWC uses quadratic regularizer based the old weight-vector wold and its importance Fold, while experience replay uses memory M of past examples along with their labels. Function regularization (FR) is similar but often does not use labels y- Our method combines EWC, FR, and replay by using two memory sets, M₁ and M2. The notable differences are that our method's importance matrix excludes the examples in M = UM, and that experience replay is applied only the socond memory M2. The examples in Mi do not require labels, and can be compressod only keep a small set of representative inputs that may not be part of the old data; Section for details.
Two popular strategies of knowledge reuse are based on weight-reeullariztion and experience replay. The two strategies have complementary strengths. For example, the well-known Elastic- Weight Consolidation (EWC) (Kirkpatrick al. 2017), which regularizes the new weight- vector keep it close the old one, requires a fixed memory size to store the weight vector and its importance. A variety of other such regularizers have been proposed (Schwarz et al. 2018 Zenke al. 2017 Li & Hoiem 2017; Nguyen et al. 2018). Experience replay [Robins] 1995 Shin et al. 2017), on the other hand, often increases memory as learning progresses. Memory cost here can be substantial, but it can boost accuracy if the memory represents the past well. Clearly, combining the two approaches can strike good balance between performance and memory size. At present, little has been done to find principled ways to combine them. Some works have used knowledge distiolation (Rebulfi et al.] 2017 Buzzega et al., (2200) or functional regularization (Titsias et al., 2020 Pan et al. 2020), where predictions evaluated the examples memory are regularized. Such attempts are promising, but it is not clear why the specific choices of regularizers and memory work well, and whether there are better choices that lead to further improvements.
In this paper, we provide a new approach to combine and improve regultt-egullriizatio and experience replay. The approach is based on a recently proposed principle of adaptation, where a prior called the Knowledge-adaptation prior (K-prior) is used to reconstruct the gradients of the past training objective (Khan & Swaroop 2021p. We use this as a guideline Swaroop [2021) h consider only one-task adaptation, and relied on a simple quadratic regularizer without any experience replay. We extend their method to multiple tasks and use it to combine weightt-regularizer,, functional regularizer, and experience replay (Fig. D. Each piece contributes in complementary way to the reduction of different type of error, and the combination overall gives lower gradient- reddient-rconnstractioo error than each individual component. This leads to consistent improvements on standard benchmarks for multi- task image classification task-incremennal continual learning, such as Split-CIFAR, Split- TinylmageNet,, and ImageNet- 1000, across various memory budgets from small large sizes. On these benchmarks, our method can achieve 80% of the batch performance by simply utilizing a memory of < 10% of the past data. Our approach yields better strategies than the current heuristics used in the literature.
design better priors for continual learning. Khan
2

---

Published in Transactions on Machine Learning Research (11/2023)
2
Related work
Continual lcarning paradigms. Existing CL methods can be classified into three complementary paradigms or combinations thereof: 1) weighl-regulaiiztion methods regularize parameter updates by pe- nalizing changes in 'important' parameters for previous tasks, 2) memory-replay methods rehearse some past data or pseudo-data, and 3) architecfure-basel methods change the model architecture or mask model weights (Parisi et al., 2019; Swaroop 2022). Here, we focus on the first two CL paradigms. Weiggh-reguularization
approaches include Elastic Weight Consolication (EWC) (Kirkpatrick et al., 2017), Online EWC (Schwarz
et al. 2018), Synaptic Intelligence (SI) (Zenke et al. 2017], Learning without Forgetting (LwF) Li &
Hoiem 2017). and Variational Continual Learning (VCL) (Nguyen et al. 2018). Memory-replay approaches
include Experience Replay (Robins) 1995) and Deep Generative Replay (DGR) (Shin et al.) 2017). as well
as reguctioonl or distillaton methods such as Functional Regularization for CL (FRCL) (Tit-
sias et al.] 2020), Functional Regularisation of Memorable Past (FROMP) (Pan et al. 2020), Incremental
Classifier and Representation Learning (iCaRL) (Rebuffi et al. 2017 and Dark Experience Replay (DER)
Buzzega et al. 2020
Continual learning settings. There are three distinct continual learning settings (Van de Ven & Tolias 2019 1) insk-incrrmentll learning, where the task identity is known, allowing us to have task-specific components such as a multiple output heads; 2) domaiin-nnrrmmntal learning, where the task identity is not known and the model is not required to infer the task identity, which is typically solved by single-headed architectures, and 3) dlass-incremenlll learning, where the task identily is not known and is required to be inferred by the model, which includes common real-world problems such as sequentially learning new object classes (Van de Ven & Tolias, 2019). In this work, we exclusivcly focus inskeenncremnntal lcarning, but it is also possible to exlend our approach to other settings.
priors (K-priors) Khan & Swaroop (0021) is a general approach for general, quick and accurate model adaptation that unifics and genoralizes many previous methods. In addition to our main technical contribution of improving K-priors by correcting for two major sources of error, we substantially expand upon upon[Khan & Swaroop (2021) two aspects: 1) They only consider one-step adaptation settings (i.e. with tasks), and do not handle the accumulation of error as the number of tasks increases. It is especially important to assess more realistic CL scenarios with longer task sequences, which is exactly the focus of this work. 2) Their experiments are limited to small-scale models and datasets, the largest only involving small CNN (1.2M parameters) on CIFAR-10 (10 classes, 60K data points). In contrast, we demonstrate scalability up to a ResNet-18 (11M parameters) on ImageNet (1000 classes, 1.2M data points).
Knowledge-adaptateo
3 Continual learning methods
We focus on a continual learning (CL) problem to incrementally learn from a sequence of data sets ....... Dr corresponding to a total of tasks. This is different from the commonly used batch-training in deep learning, where data from all the tasks is assumod to be available at all times during training. CL is challenging as knowledge. Our
the model needs to repeatedly adapt to new lasks, while not forgetting previously- goal is to get performance as close as possible to the model that is batch-trained on data from all tasks.
Formally, consider a supervised learning problem with Dj containing N input-output pairs (æ;, y:), with a; E R and yi y, and we wish to train a model with output (x,) (also denoted as fw), and a P-length parameter w in a space W C IR.. Then, at any given task t, the best possible model parameters w, can be obtained by training on all the data examples Dis = Di, for example, by solving the optimization problem shown below,
w. arg min (wa)) where ebatch (w)) WEW
Σ ((y,, + R(w).
(1)
SEDI.
We assume the loss Eyy,, (f+)) is defined through the log-likelihood of an exponential family distribution (e.g., cross-cntropy). with of) being a transformation of the model outputs (c.g., softmax) defined using the (inverse) link-function of the distribution. We denote the regularizer by R(w), and in what follows, we will use an Ta regularizer R(w) = fl with 8 ' 0; other regularizers can also be used.
3

---

Published in Transactions on Machine Learning Research (11/2023)
The main challenge in CL is to remember the useful past knowledge extracted from previous tasks D1:--1: and reuse it while training over the new task Dr to get close as possible to wy. We would like compact summary of the past knowledge, because we are not allowed to store all past data. We will now describe two strategies used in the literature, based on weight-regulaaization and experience replay, and discuss the challenges in combining them.
Weiglttregullarzationn approaches aim to keep the new weight-vector close the old one, hoping that this will help avoid forgetting and facilitate knowledge reuse. The most common is Elastic- Weight Consolidation (EWC) (Krrkptrrick et al. 2017), which minimizes the following objective using the previous weight wt-1,
A (w wi-1) Fi-1(w wi-1), Σ Ey (fw)) ID.
Cwright
()
(2)
where the second term is a quadratic regularizer with a weight-importane matrix Fi-1, and À 2 0 is a trade-off hyperparameter. The simplest choice of Fr-1 is to use the diagonal of a generalized Gauss-Newlad (GGN) matrix (Martens 2020). The GGN (over previous tasks) is defined as,
G--1(Paa-))== = Σ
of( [[
(3)
1EDI
where fi denotes the Jacobian, that is, derivative (x,) with respect to w at w = wt-1, and (ff)) denotes the derivative of (J) with respect to / at f- f Often, the regularizer d is also added to the GGN matrix to reduce ill-conditioning Computation of FT-1 can be done in an online fashion as the training proceeds over tasks (Schwarz et al. 2018). This method uses a compact representation of the past knowledge as we need to only store wi-1 and the diagonal matrix Fr-1, requiring ((P) memory. The method also simple to implement within deep-learning codebases, requiring relatively little overhead. Other choices are possible for the importance matrix ÜZenke al. 2017 Aljundi et al.) 2018 Benzing 2022).
An alternative to nright-regularizttion methods is based experience replay (Robins 1995
Shin et al.
2017 Such hods store subset of past data in memory Mr and add it to the new data during training:
2 (w)
Σ
( + R(ws).
RED.UM,-1
This method can be accurate when the memory represents previous tasks' data well, because as M + Dict-1, arg min Gr + we, but this often requires a large memory that grows with the number of tasks.
A popular approach to combine these two previous approaches is to use functional regularization (Titsias) et al. 2020 Pan et al. 2020) where, instead of regularizing the weights, regularize the function outputs at few past input locations stored in memory:
punnc (wr) = Σ C +2
Σ ((((R))))) JEMc-1
(4)
RED
Some methods implement this via knowledgg-diisillatinn (Rebuffi et al. 2017 Buzzega et al. 2020 Some simply use the original loss (as written in Eq. (4)) while others use the squared loss (Benjamin et al. 2019 An advantage of functional regularization is that it does not require the labels associated with the inputs in Mi-1, which enables the use an arbitrary input æ which is not restricted to be from D1a-1- For example, we can use deep gencrative model to generate pscudo-inputs (Shin al. 2017 or learn them as in sparse Gaussian processes ( 2009).
Overall, see that all these strategies have their complementary strengths: reigltt-regularization is compact, experience replay can be more accurate, and functional regularization can use arbitrary memory inputs. Combining these approaches can strike a good balance between performance and memory size, but al present, little has been done to find principled ways to combine them. One could simply add them together, but there are many choices one needs to make. For example, how should we choose the importance matrix and the specific forms of the regularizers? A simple re-weighting is likely not the best choice. Our goal in this approach to answer such questions.
paper is to provide theoretically-mottn
4

---

Published in Transactions on Machine Learning Research (11/2023)
4 A principle of adaptation: gradient reconstruction of the past
We will use the recently proposed principle of adaptation by Khan & Swaroop 2021 to combine and improve the CL strategies discussed in Section 3 The principle suggests to reconstruct the gradient of the past objective by using a combination of weight- and function-space regularizers. Specifically, at task t, we consider minimizing (for some T > 0)
K-prior (w) = Σ
P(vs + ( (.
(5)
iED:
where (w; wi-1. M,-1) is regularizer that combines weight-space regularizer using w:-1 and a function- space regularizer over the memory Mr-1- The regularizer is called the Knowleige-adaptation prior (K-prior), and is designed with the goal to minimize the gradient reconstruction error of the past training objective. We will use == 1 unless noted otherwise.
Specifically, at task i, the past training objective is e (w), and we want to design the prior to minimize the magnitude of the gradient error for all w (more details are Appendix
e((w)= vehatch(ww) (w) Ve
vek-prior (w) = Vel
baatch (w) - (1-
(6)
The loss ebatch depends all the past data. P1c-1, and goal is to reconstruct its gradient by using the weight vector wt-1 and memory M Khan & Swaroop (202)) showed that many existing adaptive strategies in machine learning for one-step adaplation tasks can be recovered from this principle. For example, the following K-prior with M = D1.-11 gives zero error ea (w) for generalized-lineal models für = 0. w with feature vectors Φ; = ((x:):
+ E e(o(Fer.))
5 (w wi-1) wi-1).
(www; 11-1,
(7)
2
iEMi-1
where 8 0 is the L2 regularization constant. The K-prior does not use the labels, just like functional- regularization discussed earlier, yet the gradient can be reconstructed by using the predictions at the inputs locations 2; D1a-1- To see that e, vanishes, consider the following derivation from Khan & Swaroop 20021 Eq. 10):
e:(w)= = V
Ve batch (w)
Σ iED
Φi (((( o(fur ) + ō(w w,-1)
Σ JEDMI
Φi ( vi)] + dw
Σ di [o(te ;-==0)00 y/ + +8(20-1)
iCD
-Vehatchaao..
-eeluthh (w0))))(0
=0, 0,
where the second line is obtained by using Eq. (66) and noting that for a GLM, V = Φi, and the third line is obtained by adding and subtracting outputs yi the first term of VKG In the third line, the second term is equal similar results are shown by Khan & Swaroop (2021) for other models, such support vector machines, Gaussian processes, knowledge distillation, functional reguuarizin,,,l and the memory- based methods used in continual learning.
because w;-1 minimizer of ghatch and therefore
(ppatc((w,,))
Many other
K-prior is promising, but there are still multiple issues with the work of Khan & Swaroop 2021 which we address in this paper. First, they did not consider the sequential setup such as continual learning. Second, the error incurred by their K-prior, of form Eq. 7 is non-zero for neural networks: see Khan Swaroop (2021] Sec. 4.2). For the continual learning problem, this can be disastrous because errors can accumulate quickly over tasks, deteriorating performance. Third, the weight-regularizer in Fq. (7 ignores the weight
5

---

Published in Transactions on Machine Learning Research (11/2023)
importance, which is commonly used in other works Kirkpatrick et al. 2017 Schwarz et al. 2018 Ritter et al. 2018) and can improve performance.
In this paper, we will address these issues, combining and improving regularization and memory- methods. We will start with functionalreeuuaazzationn and then add a weight regularizer and experience replay to decrease error. This will yield prior with lower error than each individual method on its own.
5 A new improved K-prior
Using the principle described in the previous section, we will now design a prior that combines a weight regularizer, a functional regularizer, and experience replay (Fig, [ As described in the previous section, our goal is to minimize the gradient reconstruction error of the past training objective, ie., the difference in the gradient of the loss for the full batch method and the K-priors loss with limited memory, The functional regularizer is based on knowledge distillation, and combined with an EWC-style quadratic regularizer which uses specific importance vector to minimize the error in the K-prior of Eq. 7 An additional experienc--replay term further reduces the error by storing the labels for a subset of the memory set. We will sec that each piece contributes to the reduction of a different type of crror, and the combination overall gives lower gradient reconstruction error than each individual component.
Eq. (6)
5.1 The error in the K-prior Eq. 7 when using limited memory
We start by analyzing the gradient error in the K-prior of Eq. when it is defined with a limited memory Mr-1 instead the full data D12-1- As shown in Appendix
the error given as follows,
(<(w))
Σ
Viw ((ii)) - o of
Virr w._1 + Sw
(8)
iEDi
:=e) (oNNN(WwwwW)) (ur,a
where ri., (Je.. - yr is the residual The first error term et emm arises due to the use of limited memory Mr-1, and can be reduced to zero by increasing the memory size to include all past input examples. In contrast, the second error term ea arises due to the use of neural networks, and reduces only when the network gets better in predicting the past data D1a-1, that is, when the residuals go to zero. We will now show that emmem can be reduced by adding an EWC-style weight regularizer (Section 5.2) while e; can be
reduced by adding an experience replay term with specific memory (Section 5.3
5.2 Reducing mem using an EWC-style regularizer
The error e.eem can be reduced by using first-order Taylor approximation of of at wi-1,
(fw R (ff...))) + o' (V J(w wi-1). Plugging this into the definition of the error, get
(9)
mem
Σ
V ] d'(
[V _1 (w w,-1),
jEDIA I/M
-G-iPPc--MMM-
where we use the definition of the GGN matrix given in Eq. (3) and approximate V 2 Vfg. The right hand side equal to the gradient an F.WC-style regularizer,
mem 2 (w
wt-1),
(10)
Note that the combined error Eq. unifies the limited memory term and neural network error stated respectively, Rq. (11) and Eq. (16) & Swarnop 2021).

---

Published in Transactions on Machine Learning Research (11/2023)
which uses the GGN over the past data but, importantly, excludes the memory M from it We can now define a new K-prior by adding the correction term to Eq. (5), giving,
K w 11,1, F Fr-1;
E iEMe-1
e(olfin, 1) o + ( - w;-1) ([w-w,)) FF-I(( wr-1),
(11)
where define FŞ-1 Gi-I(Pra- + SI. By correcting for the error term, this K-prior reduces the error introduced in the K-prior of Eq. 7 due to limited memory. Similarly to Online EWC (Schwarz) et al. 2018). we can use diagonal approximation to the GGN, and update it online.
The new K-prior not only reduces the gradient error, but also is more general because other rogularizers are obtained as special cases by changing the memory. For an emply memory M = 0, reduces the EWC regularizer of Eq. (2): the first term in Eq. (11) disappears and the importance F,-1 1 the GGN defined over all the past data, plus øI. On the other hand, when the memory includes all past data, it reduces to the original K-prior in Eq. ( When using limited memory, the new K-prior combines the functional and weight regularizers in way to reduce the error in both EWC and the original K-prior.
5.3 Reducing NN using experience replay
The et NN term depends on the mistakes made the past data, and can be corrected by including an additional memory, denoted by M2,c-1, of the past data where mistakes are significant. We first note that the error is equivalent to the gradient of the following,
NN
Σ iEM2,1-1
fur dow wi-1:
(12)
when M2,111 is set to all the past data. Therefore, by adding KNN Kr + coN with subset of past data M2,1-1, we reduce the error simply to EiED E
V r
1\Ma
5.4 K-prior with EWC-style regularizer and experience replay
Our new improved K-prior is obtained by simply correcting the K-prior from Eq. terms from Eq. 10 and Fq. 12 that is,
7 by adding the correction
NN + cuem K.new = IC₃ + cct
Σ iEMe-1
e (C(Sii )io((eo)) +
Σ
f ni we
iEM2.11
+ fow T wi-1 + (ww wi-1) Fr-1(w wi-1).
(13)
We make a specific choice of the memory: we choose M2,1-1 to be a subset of M This, as we show now, simplifies the computation and brings experience replay into our new K-prior.
Consider the first two terms from improved K-prior in Eq. 13
Σ
( ((
+
Σ
fi "
iEML
sEMa.
Note that the derivation of Eq. 10 follows the derivation Eq. (12) in Khan Swaroop

---

Published in Transactions on Machine Learning Research (11/2023)
The gradient of these two terms can be simplified when the second memory is subset of the first one,
Σ
e
Σ
Purwi
we-1
iEML-1
iEM2,e-1
E
(((())
-o((m-)))
Σ
wi-1
1
iEM2,
Σ
V ((fw) -o -f(iro
Σ
((mrr.)) + VSo V (f()))
iEM2.-1
E
f ((fu) o offer
Σ
v (o( yi)]
iEMa,
where in the second line we used the expression in Eq. (16) for the gradient of the (exponential-aamily) loss, and where in the last line we used the definition of the residual r a (f yi to simplify. This gradient equal the gradient of a sum of a runctional-roguuarieee term and an experience replay term,
E (olfer +
E ((,,)(f
(14)
iEM2,c11
again using the expression for the loss gradient in Eq. 16 the following combination,
Therefore, we can rewrite the new K- K-prior
(w; w; Fr F.-1
E
(olFen +
Σ
iEMi.e1
iEM2,1-1
Functioona-regulariattooo
Experience replay
1 [w - w. 1) Ww- - w:-1) + fow wi-1
(15)
Weight-regulaaiiation
where M1,1-1 Mr-1\Ma1-11 and M2,111 two disjoint subsets of the memory, and where define Fr = (P= + Å.. The new prior combines weight regularizer, functional regularizer, and experience replay. The functional regularizer is based on knowledge distillation, and combined with an EWC-style quadratic regularizer which uses an importance vector obtained over all the past data but excluding the memory M = M1.-IUMM2.s Using this combination, the new prior gives lower gradient roconstruction error than each method alone. We will refer to our method as EWC+FR+Replay (Elastic Weight Consolidation Function Regularization + Experience Replay).
In our experiments, select the memories as follows. We first randomly (uniformly) sample the memory set Mi- from all data Dı.-1, and then randomly split it into the two memory subsets M1,1-1 and M22-11; for simplicity, we use equally sized subsets, i.e., [MI,--ll = |M22-11. We found this simple selection strategy to work surprisingly well, and leave the study of more sophisticated approaches (as well as of settings where [MM # (22 for future work.
6 Empirical evaluation
We first describe the general experimental setup used throughout evaluation. We then present empirical results showing the practical benefits of our proposed EWC+FR+Replay method. We focus on multi-class classification in the cask-incrementse learning setting. In particular, we evaluate on three continual learning benchmarks with increasing size and thus difficulty: 1) Split-CIFAR (medium-scale), 2) Aplit-Tinyll Imageett poodiumm-t--lrge-saaaa and 3) ImageNet-1000 (large-scale). See Appendix for more details on the exper- iments (e.g. hyperparameters used).
6.1 Experimental setup
Continual learning setup. We mostly follow previous works on CL. We consider the common multi-head fask-incremental (Van de Ven & Tolias 2019) setting with known task identities: each method is trained
8

---

Published in Transactions on Machine Learning Research (11/2023)
Batch Joint-
Test acc. (avg. over tasks)
EWCaay+Repla
Forgetting (avg. over tasks)
0%
Batch Joint-
80%
Batch Separate
-2%
Online EWC
EWC+FR
Online EWC
EWC+FR+
75%
FR
-4%
FR
-6%
70%
3%
10%
33%
4
5
6
2
3
4
5
Memory size (% data)
Number of tasks
Number of tasks
Figure 2: Results on Split-CIFAR, ENC+FR+Replay is superior across memory sizes, closely approaching Batch Joint for memory size 33.3% (left; x-axis log-scaled). It also suffers less from FORCETTING (relative Batch Joint) with a growing number of tasks, for memory sizes 0.7% (middle) and 33.3% (right).
sequentially on all tasks with a separate classification head per task, and is told which task an input belongs to both at train and test time. We report test accuracy of the final model trained on the entire task sequence (averaged over the Lest sets of all observed tasks). We also compute average PORGETTING (aka backward- transfer) as defined in Lopez-Paz & Ranzato (017), which captures the (average) difference in accuracy between when a task first trained and after the final task. We plot mcan = standard error over three seeds, and assess performance across wide range of memory sizes.
Methods. In addition to our proposed EWC+FR+Replay method, we evaluate five relevant baselines for comparison. In summary, we consider the following methods:
EWC+FR+Replay. Our proposed regularizer which combines the K-prior function regularizer over Mj with the FWC-style weight regularizer and the experience replay term M2-
2. FR. The original K-prior function regularizer over M₁ as proposed by Khan & Swaroop (2021) without the L'WC-style weight regularization term and without the experience replay term over M2.
EWC+FR. regularizer combining the K-prior function regularizer over M1 with only the EWC-style weight regularization, i.e. without the experience replay term over M2.
4. Batch Joint. Joint batch training of a single multi-head model across the data of all tasks, i.c the optimal CL solution which serves as an upper bound we wish to approach.
Batch Separate. Independent batch training of seporale model for each task.
6. Online EWC (Schwarz al. 2018 which has the same weight regularizer EMC+FR+Replay, but without the function regularizer over Mi or the experience replay term over M2.
6.2 Results on Split-CIFAR
Sctup. Split-CIFAR (Zenke et al. 2017) has tasks with 10 classes cach. The first task is CIFAR-10 (Krizhevsky et al. 2009) with 50,000 training and 10,000 test data points across 10 classes. The subsequent
5 tasks are taken sequentially from CIFAR-100 (Krizhevsky al., 2009), each with 5,000 training and 1,000 test data points across 10 classes. In total, we thus have 90,000 data points. We use the same CifarNet model as Zenke et al. (2017): ; Pan et al. (2020): a multi-head CNN with convolutional layers, followed by 2 dense layers with dropout, with ~~ 1.2M model parameters in total. On each task, we train for 80 epochs using Adam with learning rate 10 and batch size 256.
Results. Fig. 2 shows our results on Split-CIFAR. We consider memory sizes between 100 and 5,000 per fask; 5,000, we thus store 10% of the data for task 1, and all data for tasks 2-5 (which in total thus

---

Published in Transactions on Machine Learning Research (11/2023)
55%
Batch Joint-
Ours
Test acc. (avg. over tasks)
50%
PackNet
EWC+FR
iCaRL
50%
MAS
Batch Separate
45%
HAT
EBLL
45%
Online EWC
EWC
LwF
-R-FM
GEM
R-PM
40%
mode-IMM
FR
35%
1%
3%
10%
45%
5.6%
11.2%
Memory size (% of data)
Memory size (% of data)
Figure 3: Results on Spit--TinllmaeeNe.. EWC+FR performs well across memory sizes (left; x-axis log- scaled) and compared to further strong baselines from Delange et al. (2021) (right).
corresponds to third of all data). We see that FR performs poorly at small memory sizes and is much worse than Online EWC. While performance improves noticeably with growing memory size, FR remains far below Batch Joint even at memory size 33.3%. This confirms that when using a NN instead theory behind FR (see Section 4 indeed ceases to hold. Adding the EWC-style weight regularizer (EWC+FR) substantially improves performance at small memory sizes, empirically confirming the property that EWC+FR converges to Online EWC for small memories (see Section 5.2 However, we also confirm that for large memories, EWC+FR converges to FR, resulting in performance drop.
GLM, the
Finally, when we also add the experience replay term to obtain EWC+FR+Replay, performance is substantially boosted also at large memory sizes, actually enabling us to reach Batch Joint performance for memory size 33.3%. We see that ENC+PR+Replay obtains ~97% of the batch performance with just ~00% of the data, but does not degrade as memory size decreases (unlike FR). EWC+FR+Replay thus combines the complementary benefits of both error correction terms, i.c. of both EWC-style woight-regullrtizttinn and experience replay, to significantly improve upon FR in both the small and large memory regime. Fig. 2 (mid & right) further shows that our method can leverage prior knowledge more effectively than other methods, thereby suffering less from FORCETTING with growing number of tasks. This confirms that the two error correction terms in EWC+Fa+R are particularly important for mitigating error accumulation across longer task sequences.
6.3 Results Split- TinylmageNet
Setup. Following Delange el al. 2021), we construct Split-Tiny Net by dividing TinyImageNet Le &
Yang 2015 into a sequence of 10 tasks with 20 classes each (using the same random division as in Delange et al. 2021)). ). Each class has 500 data points split into training (80%) and validation (20%). and 50 test points (totalling to 110,000 points). We use the VGG-like BASE model from Delange et al. (2021) with 6 convolutional layers, max pooling layers, and 2 dense layers, with a total of ~3.5M parameters. On each task, we train for 70 epochs (with early stopping and exponential learning rate decay, without regularization) using SGD with momentum 0.9 and batch size 200. This replicates the setup of Delange al. 20221 to make results directly comparable
We found that the experience replay error correction term does not help on this benchmark, so we representatively plot just. FR and EWC+FR We again see that EWC+FR can substantially improve over FR (especially at small memory sizes) and Online EWC, achieving ~90% of the batch performance with ~00% of the data (Fig. 3 left). It also compares favourably against a diverse range of other strong CL methods across all three CL paradigms: 1) memory/re memory/rehearsal
Results. Fig. 3 shows our results on
iCaRL (Rebulii et al. 2017), GEM (Lopez-Paz & Ranzato, 2017). R-FM & R-PM (Delange et. al.) 2021
TThe only difference lies in the hyperparameter tuning procedure: while Delange et al.
2002) their proposed online
tuning algorithm, resort to standard grid search for simplicity; Appendix B for details.
4This is likely because almost perfect train accuracy attained on all tasks (see e.g. Table 14 Thus, e
Delange al.
2021
NN Fq. & is close zero, such that NN error correction cannot boost performance.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Test acc, (avg. over tasks)
80%
Batch Batch Separate ................................
Forgetting (avg. over tasks)
0%
Batch Joint
FWC+FRR+Rephe
EWC+FRG RR-Rplaay
-5%
60%
Online EWC
-10%
40%
FR
FR
-15%
20%
Online EWC
0,1%
0.5%
2.0%
7.5%
2
5
10
Memory size (% of data)
Number of tasks
Results on ImageNct-1000. EWC+FR+Replay performs favorably across a range of memory sizes (left; x-axis log-scaled), and suffers less from FORCETTING (relative to Batch Joint) with an increasing number of tasks, here exemplary shown at the largest memory size of 7.5% (right).
Figure 4:
2) weighhtreeuularization LwF (Li & Hoiem 2017), EBLL (Rannen et al. 2017
EWC (Kirkpatrick et al.
2017), SI (Zenke et al. 2017). MAS (Aljundi et al. 2018). mode- LMM [Lee et al. 2017 and 3) architectural PackNet (Mallya & Lazebnik
2018). HAT (Serra et al. 2018 (Fig. right
6.4 Results on ImageNet-1000
Setup. We consider the ImageNee-1000 benchmark proposed by Rebulii et al. (2017). which randomly splits the full ImageNet dataset (Deng et al. 2009) of ~1.2M data points into a sequence of 10 tasks with 100 classes and ~22K data points each. Following Rebuffi et al. (2017). we use a ResNet-18 with ~11M model parameters. For training on cach task, we use the ImageNet reference training pipclinc (with 40 epoch configuration) of the FFCV library (Leclerc et al., 2022
Results. Fig. shows results ImageNeet11000 We consider memory sizes between 200 and 10K per task, where the latter amounts to 7.5% of the entire data. The observed trends qualitatively match those from previous experiments. In particular, FR underperforms for small memory sizes, and while it improves with increasing memory, it peaks at a 3.8% memory and then even starts declining. We hypothesize that this is again due to accumulation of the NN error, which might become more severe with larger memory as more data points can contribute to the error. EWC+FR again improves accuracy for small memories, but does not help for large memories. Finally, correcting for the NN error by addit including the experience replay term (EWC+FR+Replay) substantially boosts performance also at the large 7.5% memory. EWC+FR+Replay thus combines the benefits of both error correction terms to perform well across all memory sizes, achieving > 80% of the batch performance with a memory of < 10% of the past data. It also again suffers less from FORgeTTING along the task sequence, demonstrating that it better mitigates crror accumulation.
7 Conclusion
In this work, we proposed address the continual learning problem in a theoretically-grouudee way by explicitly approximating the optimal model obtained via batch-training on all tasks jointly. To this end, we developed EMC+FR+Replay, new continual learning method which efficiently re-uses prior knowledge to reconstruct the gradients of the past training objective faithfully as possible. To achieve this, method combines principles from function-rggularizaton reduce the gradient-reconsttucciio error. Empirically, we demonstrated the effectiveness and scalability of EWC+FR+Replay across different. memory sizes on common task-incremental continual learning benchmarks. particular, we showed that our proposed EWC+ER+Replay approach can be less susceptible to catastrophic
weight-regnlarization. and experience replay to
Results are from Delange et (202)); their total memory sizes [4500. 9000] correspond (5.6% 11.2%] of the data. Far all details of The training procedure, see http:////ttthooommm///ffffftftrr
11

---

Published in Transactions on Machine Learning Research (11/2023)
forgetting and thus achieve better performance compared to various basclines. Notably, EWC+FR-Replay can achieve > 80% of the batch performance by utilizing a memory of < 10% of the past data on the task- incremental continual learning benchmarks considered. For future work, we aim to investigate how more sophisticated strategies for selecting the two memory sets can further boost performance of our method.
Acknowledgements
We would like thank Runa Eschenhagen for many helpful discussions throughout the project. ED acknowl- edges funding from the EPSRC and Qualcomm. MEK and RY are supported by the Bayes duality project, JST CREST Grant Number JPMJCR2112. JMHL acknowledges support from a Turing Al Fellowship under grant EP/ EP/V023756/1.
Author Contributions Statement
List of Authors: Erik Daxberger (ED), Siddharth Swaroop (SS), Kazuki Osawa (KO), Rio Yokota (RY), Richard E. Turner (RET), José Miguel Hernández-Lobato (JMHL), Mohammad Emtiyaz Khan (MEK).
SS and MEK conceived the original idea. This was then discussed and further refined with ED, KO, RY, RET, JMHL. ED led the project. SS derived the combination of K-prior and EWC, and together with ED and MEK the method was exlended to include replay. SS implemented the original K-prior algorithm. KO designed and implemented the distributed training pipeline. ED designed and conducted all the experiments, with continuous support from SS and KO and feedback from all other authors. MEK and RY provided the computational resources for running the experiments. ED wrote the first draft of the paper, which was then substantially revised by SS and MEK. RY, RET, JMHT, gave regular feedback throughout the project.
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhossiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV), pp. 139 154, 2018.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. International Conference on Learning Representation, 2019.
Frederik Benzing. Unifying importance based regularisation methods for continual learning, In International Conference on Artificial Intelligence and Statistics, pp. 2372-2396. PMLR, 2022.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advonces in neural information processing systems, 33:15920-15930, 2020.
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. Continual learning in practice. In NewrIPS 2018 workshop on Continual Learning, 2019.
Mohammad Emtiyaz Khan and Siddharth Swaroop. Knowledgo-adaptation priors. Advances in neural information processing syslems, 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -3526, 2017.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. Technical report (CS 231N), 2015.
Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. https://github. com com/Libffcr/ffec/ 2022. commit 113865572213711cc660001 Le311333350000333300000500522230
Sang- Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tae Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pottern analysis and machine intelligence, 40(12):2935-22) 2947, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765 7773, 2018.
James Martens. New insights and perspectives on the natural gradient method. The Journal of Machine Learning Research, 21())55776-5851, 2020.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticii dilemma: Investigating the continuum from catastrophic forgetting to ago-limited learning effects. Frontiers in psychology, 4:504, 2013.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. International Conference on Learning Representation, 2018.
Andrei Paleyes, Raoul-Gabricl Urma, and Neil D Lawrence. Challenges in deploying machine learning: a survey of case studies. arXiv preprint Xii:20011009926, 2020.
Pingho Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E. Turner, and Moham- mad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. Advances in neural information processing systems, 2020.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54 71, 2019.
Amal Ramnen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning, In Proceedings of the IEEE International Conference on Computer Vision, pp. 1320-1328, 2017.
Sylvestre-Alvies Alvise Rebuffi, Alexander Kolesníkov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pallern Recognilion, pp. 2001 2010, 2017.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured Laplace approximations for over- coming catastrophic forgetting. In Advances én Neural Information Processing Systems, pp. 3738-3748, 2018.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123 146, 1995.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of Machine Learning Research. PMLR, 2018.
Joan Scrra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In International Conference on Machine Learning, pp. 1548-4557 PMLR, 2018.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in neural information processing systems, 2017.
Richard Sutton. Two problems with back propagation and other steepest descent learning procedures for networks. In Proceedings of the Fighth Annual Conference of the Cognitive Science Sociely, 1986, pp- 823-832, 1986.
Siddharth Swaroop. Probubilistic Continual Learning using Neural Networks. PhD thesis, University of Cambridge, 2022.
Michalis Titsias.
Variational learning of inducing variables in sparse Gaussian processes. In Artificial intel- ligence and statislics, pp. 567-50 574. PMLR, 2009.
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Tch. Functional regularisation for continual learning using Gaussian processes. International Conference Learning Representation, 2020.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.0779, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence.
International Conference on Machine Learning, pp. 3987 3995. PMLR, 2017.
14

---

Published in Transactions on Machine Learning Research (11/2023)
A Derivation of error for K-priors with limited memory Eq. (8)
Recall that ft = (w)) is shorthand for the model outputs. For casc-of-notation, we will also use the shorthand ow - (fw) (x,) for the model predictions. Consider the following expression for the gradient of the (exponential-ammil)) loss (Khan & Swaroop 2021
o(ly),(((w) = fiv [o(f)) -
(16)
We start by deriving the gradient error in Eq. 6
V Vel (wb)tch(w) Vek-prior (wr)
(pacch (w) ((rprro((()
Σ (utsow) + claich (wo)
Σ
E((uroww K(w;; w)
iED
V (baach (w) K(w; ((
Vebaach (w) VIC(E: (0-1,
We then have,
(bbatc (w) K((u;; (0)
Σ
e (Wi -
Ee l ( (oler-1: + = 112
iEDrit1
iEM
Vl Ve owm) - - Ve (ot
Te 11 ow) -1
iEM
16
m vi]
Sio J0
omm
+ dw.-1
40
iEDIa
iEM
Σ
V/ + +o
pnv + to
o
V pá
+ Sw
1
iEM
- y/
V
Vin
+ :o
iEDa-11
iEDIe-1
iEM
V
o ww:1
Σ
0 -w.- Vi
+ 8w;
iED
iEDie-1
Σ
V
Tiu wi-1
+ +,w
w
W1-1
IED IED
iEDi.-1
where pi -1
of -
ys is the residual of the i'th imput using the past model parameters w;-1
B Experiment details
For all methods in all experiments, tuned hyperparameterss in the common way by conducting (exponentially-spacee) grid search and evaluating performance
held-out validation set. In particu-
lar, tune the following hyperparameters: a temperature parameter T to scale the logits in the experience replay term (as commonly-done in knowledge distillation, see Khan & Swaroop (2021) for a discussion), a trade-off parameter т in front of the K-prior-style function regularization term, and a trade-off parameter X in front of the EWC-style weight regularization term.
Note that this derivation follows the derivation in Appendix in Khan Swaroop 2021
15

---

Published in Transactions on Machine Learning Research (11/2023)
We used the following tuned values for those hyperparameters: for Split- we have T 2.0 for all methods, = 0.1 for FR, 0.25 and À = 2.0 for EWC+FR and EWC+FR+Replay, and X = 10.0 for Online EWC. For
we have T = 1.0 and t = 16.0 for all methods. We found that a fixed A across tasks does not work well for this benchmark, so we tuned a separate per task, resulting in the sequence [330, 85, 45, 30, 20, 15, 10, 10] for all methods. In contrast to our grid search procedure, Delange cct al. (2021) use a dedicated iterative hyperparameter tuning strategy that trades off plasticity vs. stability. For ImageNet, we used T 1.0, x 1.0 and 0.16 for all methods.
16


Reviews:
[{"review_id": "KAGctEit9S", "reviewer": "Reviewer_Njsb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, easy to read, and most ideas can be grasped quickly (I especially commend the authors on their use of very precise notations throughout).\n\nOn the negative side, The authors claim to study how to \"optimally combine [CL] approaches\". However, the algorithm they propose (Eq. (17)) is very simple, as a sum of all the possible regularization terms. The only novelty is how the memory is structured (two sets of past data, with the EWC matrix computed on the remaining subset of data). Unless I am mistaken, this is not benchmarked in the experiments, eg, what happens if I ignore this and use double the memory for both terms?\n\nAlso note that the major difficulty in combining regularization terms is weighting them appropriately. Here the authors are simply assuming a weight of 1 for all of them, which turns out to be suboptimal in some scenarios (as they are forced to remove certain terms in some experiments). I think that saying they \"solved\" the issue is a bit of an overstatement.\n\nThe novelty compared to (Khan and Swaroop, 2021) is also a bit overstated. K-priors with limited memory are explored in Section 3.2  of (Khan and Swaroop, 2021) using precisely a first-order approximation (which is the same that is done in this paper), while the connection to experience replay and NNs is explored in Section 4.2 of (Khan and Swaroop, 2021). \n\nExperiments are only done on TIL scenarios, which is a bit limiting nowadays. I would have expected at least 1 comparison in a CIL scenario. In addition, I am unsure why baselines (and not ablations) are only considered in a small subset of experiments (Fig. 3, right). Why not comparing on all datasets?\n\nThe paper should also discuss a bit more in-depth related works, such as GEM, functional regularization on the embeddings, FROMP, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the combination of functional regularization and weight regularization strategies in continual learning, under the lens of K-priors introduced in (Khan and Swaroop, 2021). A K-prior is a combination of a weight and functional regulariser used to adapt one model from one scenario to another (eg, adding data, changing regularizer, etc.), and it is defined to faithfully approximate the gradient on the past scenario. \n\nThe authors extend (Khan and Swaroop, 2021) by focusing on neural networks and on CL, where the network is adapted repeatedly to multiple tasks. In this case, the K-prior has two sources of errors, the limited memory of the past and the NN error. They improve on this by considering an EWC-style term for the former (stemming from a 1st order approximation of the K-prior) and an experience replay term on a second memory for the latter. Their final algorithm (Eq. (17)) is a sum of EWC, experience replay, functional regularization, and L2 regularization on the weights.\n\nThey evaluate their algorithm on several task-incremental learning (TIL) datasets with a few ablations and comparisons, showing that the method performs well.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please clarify more carefully which derivations in the paper are novel and which are straightforward applications of the material from (Khan and Swaroop, 2021).\n2. Ablate the use of separate memories in the algorithm (comparing to a single memory with double the size used in all terms).\n3. Include benchmarks on CIL and extend the comparisons to state-of-the-art models.\n4. Focus more the related works section.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DDGWVFqNBH", "reviewer": "Reviewer_ioCd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper is written clearly and easy to understand.\n\nWeakness: The major concern is about the novelty of this paper. While replay is a specific method in continual learning, it is well known that regularization can improve the generalization for most machine learning algorithms. As a result, without more in-depth theoretical insights on why regularization particularly fits continual learning, the novelty of the proposed method is insufficient. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to combine regularization with replay to improve continual learning. Numerical experiments demonstrate the improvement of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors are encouraged to provide toy examples with theoretical justifications on why regularization is important to address their novelty of the paper.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cDuozFYDhW", "reviewer": "Reviewer_GymF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The authors propose to combine two continual learning techniques using theoretically grounded arguments. \n* Most derivations are clear and easy to follow.\n\nWeaknesses:\n* I do not agree that in its current state, the paper describes a principled way to combine regularization and replay-based continual learning methods as claimed by the authors (see the requested change section).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method that combines three existing continual learning (CL) approaches: elastic weight consolidation (EWC), functional regularization and experience replay. The choice of these three approaches is motivated by the so-called principle of adaptation. This previously proposed principle adds a regularizing term (K-prior) to the training objective. The prior is chosen to minimize the magnitude of the gradient error. The authors demonstrate how the combination of EWC, experience replay and functional regularization can be used to reduce different parts of the gradient error. Empirical evaluation demonstrates that this combination of continual learning techniques improves the performance in task-incremental continual learning on various benchmarks.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**I have one major concern that is critical for my final evaluation of the paper:**\n\nThe authors make the following claim in the introduction: “...we provide a principled approach to combine and improve the two strategies” [regularization and experience replay]. Further, at the end of section 3, the authors write: “...how should we choose the importance matrix, the memory set, and the specific forms of the regularizers? Our goal in this paper is to provide a principled approach to answer such questions”.\n\nI do not agree that the paper fulfils these claims. The paper demonstrates that the EWC form of regularization reduces a certain part of the gradient error, while experience replay reduces another. This demonstrates the benefit of combining the two. However, the paper does not study other regularization-based CL methods (e.g.  Synaptic Intelligence or Memory Aware Synapses, etc.) from the same point of view. Various ways to select a replay buffer are not considered as well. That being said, I still have a question posed by the authors: how do we choose the importance matrix and the memory set?\n Furthermore, I would expect a principled approach to provide certain guidelines on how the two approaches could be weighted in the loss. For example, a larger memory buffer places more emphasis on the experience replay. Does one need to counteract that by reweighting the regularization part of the loss?\n\nI would gladly hear the author's clarification on how these questions can be potentially answered using their framework. In the current state of the work, I do not think this can be called a principled approach and formulations in the paper should be softened.\n\n**Minor concern**:\n* In section 5.2 first-order Taylor approximation is used for the term $\\sigma(f_{\\omega}^i)$ and constant approximation is chosen for $\\nabla f_{\\omega}^i$. Why don’t we consider 1st order approximation for the whole $e^{mem}_t$ term? How would that influence the conclusions of the section?\n\n**Recommended minor adjustments:**\n* Introduction: Paragraphs 3 and 4 both start with the sentence that mentions “two strategies”. It should be known to the reader from the beginning of paragraph 2 that it refers to regularization and experience replay. However, I think, this cross-paragraph reference harms the “flow” of the paper. \n* Equation (7) provides the definition of the gradient error. However, the derivation of the gradient error in Appendix A begins from a different expression. After a few transformations, we do arrive at the same thing as equation (7), but it is quite confusing at the beginning.  \n* Equation (17): the same letter $\\delta$ is used in the definition of $F_{t-1}$ and in front of the $L_2$ regularization. Please, consider renaming one of them.\n\n**Typos:**\n* Page 3 (continual learning settings): “...task-incremental learning, where the task identity is known, allowing us **TO** have task-specific model”\n* Page 3 (Knowledge-adaptation priors): “Khan & Swaroop (2021) **IS** a principled, foundational approach…”\n* Equation (11): $\\tfrac{1}{2}$ is missing\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the paper is well-written and provides a theoretical foundation for methods in continual learning. There is some doubt about the novelty of the paper, however, it provides enough theoretical and empirical evidence for its claims.
The authors did a good job in the rebuttal, answering the concerns raised by the reviewers in a satisfactory manner.
Decision: Accept (poster)