Natural continual learning: success is a journey, not (just) a destination
Authors: Ta-Chu Kao
Kristopher T Jensen
Gido Martijn van de Ven
Alberto Bernacchia
Guillaume Hennequin
Venue: NeurIPS 2021 Poster (2021)
Keywords: continual learning
recurrent neural networks
deep learning
neuroscience
Bayesian inference
trust region optimization
neural dynamics

Abstract: Biological agents are known to learn many different tasks over the course of their lives, and to be able to revisit previous tasks and behaviors with little to no loss in performance. In contrast, artificial agents are prone to ‘catastrophic forgetting’ whereby performance on previous tasks deteriorates rapidly as new ones are acquired. This shortcoming has recently been addressed using methods that encourage parameters to stay close to those used for previous tasks. This can be done by (i) using specific parameter regularizers that map out suitable destinations in parameter space, or (ii) guiding the optimization journey by projecting gradients into subspaces that do not interfere with previous tasks. However, these methods often exhibit subpar performance in both feedforward and recurrent neural networks, with recurrent networks being of interest to the study of neural dynamics supporting biological continual learning. In this work, we propose Natural Continual Learning (NCL), a new method that unifies weight regularization and projected gradient descent. NCL uses Bayesian weight regularization to encourage good performance on all tasks at convergence and combines this with gradient projection using the prior precision, which prevents catastrophic forgetting during optimization. Our method outperforms both standard weight regularization techniques and projection based approaches when applied to continual learning problems in feedforward and recurrent networks. Finally, the trained networks evolve task-specific dynamics that are strongly preserved as new tasks are learned, similar to experimental findings in biological circuits.

Natural continual learning: success is a journey, not (just) a destination
Ta-Chu Kao
Kristopher T. Jensen 1*
Gido M. van de Ven
Alberto Bernacchia
Guillaume Hennequin'
Department of Engineering, University of Cambridge 2. Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine
3. MediaTek Research, Cambridge
ftck29, ktj 2tajm ac. ac. uk ven@bcm.edu
alberto. heaptt/bbbbracchcamtmtteeeeeeeee com gj gjeh20cam. ac.
Abstract
Biological agents are known to learn many different tasks over the course of their lives, and to be able revisit previous tasks and behaviors with little loss in performance.
contrast, artificial agents are prone to 'catastrophic forget- ting' whereby performance on previous tasks deteriorates rapidly new ones are acquired. This shortcoming has recently been addressed using methods that encourage parameters stay close to those used for previous tasks. This can be done by (i) using specific parameter regularizers that map out suitable destinations in parameter space, or (ii) guiding the optimization journey by projecting gradients into subspaces that do not interfere with previous tasks. However, these methods of- ten exhibit subpar performance both feedforward and recurrent neural networks, with recurrent networks being of interest to the study of neural dynamics supporting biological continual learning. this work, we propose Natural Continual Learning (NCL), new method that unifies weight regularization and projected gradient descent. NCL uses Bayesian weight regularization to encourage good performance all tasks at convergence and combines this with gradient projection using the prior precision, which prevents catastrophic forgetting during optimization. Our method outperforms both standard weight regularization techniques and projection based approaches when applied to continual learning problems in feedforward and recurrent networks. Finally, the trained networks evolve task-specific dynamics that are strongly preserved as new tasks are learned, similar to experimental findings biological circuits.
Introduction
Catastrophic forgetting common feature of machine learning algorithms where training on new task often leads to poor performance on previously learned tasks. This is in contrast to biological agents which are capable of learning many different behaviors over the course of their lives with little to no interference across tasks. The study of continual learning in biological networks may therefore help inspire novel approaches machine learning, while the development and study of continual learning algorithms in artificial agents can help us better understand how this challenge is overcome in the biological domain. This is particularly true more challenging continual learning settings where task identity is not provided at test time, and for continual learning in recurrent neural
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

networks (RNNs), which is important due to the practical and biological relevance of RNNs. However, continual learning these settings has recently proven challenging for many existing algorithms, particularly those that rely on parameter regularization to mitigate forgetting [12, 13, 47]. In this work, we address these shortcomings by developing a continual learning algorithm that not only encourages good performance across tasks at convergence but also regularizes the optimization path itself using trust region optimization. This leads to improved performance compared to existing methods.
Previous work has addressed the challenge of continual learning in artificial agents using weight regularization, where parameters important for previous tasks are regularized to stay close to their previous values [1, 17, 24, 32, 37, 54]. This approach can be motivated by findings the neuroscience literature of increased stability for subset of synapses after learning [49, 50]. More recently, approaches based on projecting gradients into subspaces orthogonal to those that are important for previous tasks have been developed in both feedforward [40, 53] and recurrent [12] neural networks. This is consistent with experimental findings that neural dynamics often occupy orthogonal subspaces across contexts in biological circuits [3, 14, 20, 22]. While these methods have been found to perform well in many continual learning settings, they also suffer from several shortcomings. In particular, while Bayesian weight regularization provides a natural way to weigh previous and current task information, this approach can fail in practice due to its approximate nature and often requires additional tuning of the importance of the prior beyond what would be expected in rigorous Bayesian treatment [46]. In contrast, while projection
-aaeed based methods have been found empirically to mitigate catastrophic forgetting, it is unclear how the 'important subspaces' should be selected and how such methods behave when task demands begin to saturate the network capacity.
In this work, we develop natural continual learning (NCL), new method that combines (i) Bayesian continual learning using weight regularization with (ii) an optimization procedure that relies on a trust region constructed from approximate posterior distribution over the parameters given previous tasks. This encourages parameter updates predominantly tasks while maintaining convergence to a maximum of the Bayesian approximate posterior. We show that NCL outperforms previous continual learning algorithms in both feedforward and recurrent networks. We also show that the projection- based methods introduced by Duncker et al. [12] and Zeng et al. [53] can be viewed as approximations to such trust region optimization using the posterior from previous tasks. Finally, we tools from the neuroscience literature to investigate how the learned networks overcome the challenge of continual learning. Here, we find that the networks learn latent task representations that stable over time after initial task learning, consistent with results from biological circuits.
the null-space of previously acquired
Method
Notations We use XT, X-1, Tr(X) and vec(X) to denote the transpose. inverse, trace, and column-wise vectorization of matrix X. We use X @ Y to represent the Kronecker product between matrices X E R²*= and Y E mmmx such that (X @ Y m)i++kmf++ = YiyY Yu. We 'dataset' corresponding to task k which in this work generally consists of a set of input-output pairs {x y (2) such that & (0) log
use bold lower-case letters to denote column vectors. D2 refers
(Dx () = E, log togh ark is the task-related performance task k for model
(4) (2)
with parameters 0. Finally, we use D, to refer to a dataset generated by inputs from the Atth task
where (0( (1) po (3)) are drawn from the model distribution M.
2.1 Bayesian continual learning
Problem statement In continual learning, we train a model on a set of K tasks (DI,... .., Dx} ) that arrive sequentially, where the data distribution Dr for task k in general differs from Dy4t- The aim is to learn a probabilistic model p(D(0) that performs well on all tasks. The challenge in the continual learning setting stems from the sequential nature of learning, and in particular from the common assumption that the learner does not have access to "past" tasks (i.e., D, for j < k) when learning task k. While we enforce this stringent condition in this paper, our approach may be easily combined with memory-based techniques such as coresets or generative replay [8, 10, 13, 32, 34, 36, 38, 42, 43, 45, 48].
2

---

Bayesian approach The continual learning problem is naturally formalized in a Bayesian frame- work whereby the posterior after i 1 tasks used as a prior for task k. More specifically, we choose prior p(0) on the model parameters and compute the posterior after observing & tasks according to Bayes' rule:
k II p (Dx
p())) x p()) 1' x p 1 1
where Di.k concatenation of the first tasks (Di..... DD)) In theory, it is thus possible to compute the exact posterior p( (00 Di.)) after l tasks, while only observing Dr. by using the posterior p(0)(1..-1) after difficulty here is that the posterior is typically intractable. To address this challenge, it is common to perform approximate online Bayesian inference. That is, the posterior p(0) 1: 1) is approximated by parametric distribution with parameters Φh-1. The approximate posterior g(O; Φx-1) is then used as a prior for task k.
tasks as a prior. However, as is often the case Bayesian inference, the
Online Laplace approximation A common approach is use the Laplace approximation whereby the posterior p (D1.k-1) is approximated as a multivariate Gaussian q using local gradient infor- mation [17, 24, 37J. This involves (i) finding mode μ4 of the posterior during task k, and (ii) performing a second-order Taylor expansion around μ4 to construct an approximate Gaussian poste- rior q(;; Φx) (0; Us; " where A₃ is the precision matrix and Φe (My: Ag). In this case, gradient-based optimization is used to find the posterior mode on task & (c.f. Equation I):
μ = arg max log p(0 DR, x
(2)
= arg max log (D& (0) + log g(;; x
(3)
1 (0 1k-1) A6-1 - 14-11 arg max (0)
(4)
(0)
The precision matrix A₃ is given by the Hessian of the negative log posterior at pay
A v (0og((012) Dr. @x-1) 0=p H HD pa)
Ak-1,
(5)
where H(Ds, = Va log o p- is the Hessian of the negative log likelihood of Dr.
Continual learning with the online Laplace approximation thus involves two steps for each new task Dr. First, given Dr and the previous posterior (0: p4-1, A (i.e. the new prior), s4 is found using gradient-based optimization (Equation 4). This step can be interpreted as optimizing the likelihood of D& while penalizing changes in the parameters according their importance for previous tasks, as determined by the prior precision matrix A Second, the new posterior precision matrix A is computed according to Equation 5.
Approximating the Hessian In practice, computing A presents two major difficulties. First, because (0; Φb)) a Gaussian distribution, A₁ has to positive semi-definite (PSD), which is not guaranteed for the Hessian H (Dx, pt)). Second, if the number of model parameters no is large, it may be prohibitive to compute full (no x no) matrix. To address the first issue, it common to approximate the Hessian with the Fisher information matrix (FIM; 17, 30, 37):
F E P(DL10)
Vo Logp((,, (0) log log (D(()))
~ HDD,, m1)
(6)
0=1a
The FIM is PSD, which ensures that A = Σ FR. is also PSD. Computing Fş may still be impractical if there are many model parameters, and it is therefore common to further approximate the FIM using structured approximations with fewer parameters. In particular, diagonal approximation to Fg recovers Elastic Weight Consolidation (EWC; 24), while Kronecker-factored approxima- tion [31] recovers the method proposed by Ritter et al. [37]. We denote this method 'KFAC' and use it in Scction 3 as comparison for our own Kronecker- factored method.
3

---

convex loss
non-convex loss
Laplace
Projected
NCL (ours)
Task
Task 2
Global
Continual learning in a toy problem. (A) Loss landscapes of task (61; left), task (f2; middle) and the combined loss l1+2 {i + &y (right). Stars indicate the global optima for ε (red), &g (blue), and f1+2 (purple). We assume that has been optimized for L and consider how learning proceeds on task 2 using either the Laplace posterior ('Laplace', green), projected gradient descent on &2 with preconditioning according to task ('Projected', pink), or NCL. (black dashed). Laplace follows the steepest gradient of 21+2 and transiently forgets task 1. NCL follows a flat direction of & and converges the global optimum of f++ with good performance on task throughout. Projected gradient descent follows similar optimization path to NCL but eventually diverges towards the optimum of 22. (B) As in (A), now with non-convex l (center), leading to a second local optimum of 21+2 (right) while & is unchanged (left). In this case, Laplace can converge to local optimum which has 'catastrophically' forgotten task 1. Projected gradient descent moves only slowly in 'steep' directions of l but eventually converges to a minimum of fz. Finally, NCL finds a local optimum of l1t2 which retains good performance on task 1. See Appendix K for further mathematical details.
Figure 1:
2.2 Natural continual learning
While the online Laplace approximation has been applied successfully in several continual learning settings [24, 37], has also been found to perform sub-optimally on a range of problems [12, 46). Additionally, its Bayesian interpretation in theory prescribes unique way of weighting the contributions of previous and current tasks to the loss. However, to perform well in practice, weight regularization approaches have been found to require ad-hoc re-weighting of the prior term by several orders of magnitude [24, 37, 46). These shortcomings could be due an inadequacy of the approximations used to construct the posterior (Section 2.1). However, we show in Figure that standard gradient descent on the Laplace posterior has important drawbacks even in the exact case. First, we show that exact Bayesian inference indirect optimization paths along which previous tasks are transiently forgotten as a new task being learned (Figure 1A; green). Second, when the loss is non-convex, show that exact Bayesian inference can still lead to catastrophic forgetting (Figure IB; green).
simple continual regression problem can produce
An alternative approach that has found recent success in a continual learning setting involves pro- jection based methods which restrict parameter updates to subspace that does not interfere with previous tasks [12, 53). However, is not immediately obvious how this projected subspace should be selected way that appropriately balances learning on previous and current tasks. Additionally, such projection-baas based algorithms have fixed points that are minima of the current task, but not necessarily minima of the (negative) Bayesian posterior. This can lead to catastrophic forgetting in the limit of long training times (Figure I; pink), unless the learning rate is exactly zero in directions that interfere with previous tasks.
To combine the desirable features of both classes of methods, we introduce "Natural Continual Learning" (NCL) - an extension of the online Laplace approximation that also restricts parameter updates directions which do not interfere strongly with previous tasks. In a Bayesian setting, we can conveniently express what is meant by such directions terms of the prior precision matrix A. In particular, 'flat" directions of the prior (low precision) correspond to directions that will not significantly affect the performance on previous tasks. Formally, we derive NCL as the solution of a trust region optimization problem. This involves minimizing the posterior loss (x( within region of radius centered around with distance metric of the form d(0, 0 5) = /8T A 1 /2 that takes into account the curvature of the prior via its precision matrix AR-1:
= arg min (x) + (oLx subject to T
TT A < 1²
where (0 + 8) (0) VoCx is first-order approximation to the updated Laplace objective. The solution to this subproblem is given by d A Vol+(0) (0 -4k-1) (see

---

Appendix A for a derivation), which gives rise to the NCL update rule
0 0 + [ (o (0 Ms-11]
for a learning rate parameter y (which is implicitly function of in Equation 7). To get some intuition for this learning rule, we note that A acts preconditioner for the first (likelihood) term, which drives learning on the current task while encouraging parameter changes predominantly in directions that do not interfere with previous tasks. Meanwhile, the second term encourages 0 to stay close to pk-1- the optimal parameters for the previous task. As we illustrate in Figure 1, this combines the desirable features of both Bayesian weight regularization and projection- -aaccdd methods. In particular, NCL shares the fixed points of the Bayesian posterior while also mitigating intermediate or complete forgetting of previous tasks by preconditioning with the prior covariance. Notably, if the loss landscape is non-convex (as it generally will be), NCL. can converge to a different local optimum from standard weight regularization despite having the same fixed points (Figure IB).
Implementation The general NCL framework can be applied with different approximations to the Fisher matrix F3 in Equation 6 (see Section 2.1). In this work, we use a fronecker-factored Kronecker-factoree approximation to F;
approximation [31, 37]. However, even after making for each task k. it remains difficult to compute the inverse of sum of & Kronecker products (c.f. Equation 5). To address this challenge, we derived an efficient algorithm for making Kronecker- factored approximation to A₁ = F5 + A RJ A₁ & G. when Ag-1 = As- @ G and F& are also Kronecker products. This approximation minimizes the KL-divergence between M(,, (Ax @ and N (Hu,, (Ax-1 + Fz)-) (see Appendix for details). Before training on the first task, we assume spherical Gaussian prior 0 ~ (N, p I). The scale parameter Pw can either be set to fixed value (c.g. 1) or treated as a hyperparameter, and we optimize Pw explicitly for our experiments in feedforward networks. NCL also has parameter which is used to stabilize the matrix inversion Ar- ~ (A @ G + a 1- (Appendix E). This is equivalent to a hyperparameter used for such matrix inversions in OWM [53] and DOWM [12], and it is important for good performance with these methods. The Pw and a are largely redundant for NCL, and we generally prefer to fix to small value (10 (10-19) and optimize the Pa, only. However, for our experiments in RNNs, we instead fix Pw = 1 and perform hyperparameter optimization over for a more direct comparison with OWM and DOWM. The NCL algorithm is described in pscudocode in Appendix E together with additional implementation and computational details. Finally, while Laplace approximation in this section for simplicity, can similarly be applied in the variational continual learning framework of Nguyen et al. [32] (Appendix J). Our code is available online!
Gx) })
have derived NCL with
2.3 Related work
As discussed in Section 1. our method is derived from prior work that relies on Bayesian inference
to perform weight regularization for continual learning [17, 24, 32, 37]. However, we also take inspiration from the literature on natural gradient descent [2. 25] to introduce a preconditioner that encourages parameter updates primarily in flat directions of previously learned tasks (Appendix H).
Recent projection- based methods [12, 40, 53] have addressed the continual learning problem using an update rule of the form
0 レ 0 7P, oo PR)
where Pt and PR are projection matrices constructed from previous tasks which encourage parameter updates that do not interfere with performance on these tasks. Using Kronecker identities, we can rewrite Equation 9
0 - 0 + 7 (Pn & PL)
(10)
This resembles the NCL update rule in Equation 8 where we identify PR & Pt with the approximate inverse prior precision matrix used for gradient preconditioning in NCL, k-1 A & - Indeed, we note that for a Sronecker-structurr the empirical covariance matrix of the network activations experienced during all tasks up to k: - (4, 31, Appendix D), which is exactly the inverse of the projection matrix Pn. used in previous work [12, 53]. We thus see that NCL. takes the form of recent projection-based continual learning algorithms with two notable differences:
approximation to FE, the matrix Aş-1 approximates
5

---

(i) NCL uses a left projection matrix PL designed to approximate the posterior covariance of previous tasks ATT R Pn @ Pt. (i.e., the prior covariance on task k; Appendix D), while Zeng et al. [53] use the identity matrix I and Duncker al. [12] use the covariance of recurrent inputs (Appendix F). Notably, both of these choices of PL still provide reasonable approximations to A and thus the parameter updates of OWM and DOWM can also be viewed as projecting out steep directions of the prior on task & (Appendix F).
(ii) NCL includes an additional regularization term (0 (4k-1) derived from the Bayesian posterior objective, while Duncker et al. [12] and Zeng et al. [53] do not use such regularization. Importantly, this means that while NCL. has a similar preconditioner and optimization path to these projection based methods, NCL has stationary points at the modes of the approximate Bayesian posterior while the stationary points of OWM and DOWM do not incorporate prior information from previous tasks (c.f. Figure 1).
It is also interesting to note that previous Bayesian continual learning algorithms include a hyperpa- rameter À that scales the prior compared to the likelihood term for the current task [27]:
c Al (0) = log 0) - X(e - 1k-1) A (0 (k-1)).
(11)
To minimize this loss and thus find a mode of the approximate posterior, it is common to employ sodend stochastic gradient-based optimization algorithms such as Adam [23] that use their own gradient preconditioner based on an approximation to the Hessian of Equation 11. Interestingly, this Hessian is given by H2 -H (Px, 0) - AAA-1, which in the limit of large becomes increasingly similar to preconditioning with the prior precision as in NCL. Consistent with this, previous work using the online Laplace approximation has found that large values of are generally required for good performance [24, 37, 46]. Recent work has also combined Bayesian continual learning with natural gradient descent [33, 44], and in this case a relatively high value of A 100 was similarly found to maximize performance [33].
3 Experiments and results
3.1 NCL in feedforward networks
To verify the utility of NCL for continual learning, we first compared our algorithm to standard methods in feedforward networks across two continual learning benchmarks: split MNIST and split CIFAR- 100 (see Appendix B for task details). For each benchmark, we considered three continual learning settings [47]. the "task-incremental" setting, task identity is available to the network at test time, in our case via multi-head output layer [5]. In the 'domain-incrementta" setting, task identity is unavailable at test time, and the output layer shared between all tasks. Finally, in the 'class-inent intremenncem setting, the network has to both infer task identity and solve the task, in our case by performing classification over all possible classes irrespective of which task the input in question is drawn from.
van de Ven and Tolias previously showed that parameter regularization methods such as EWC perform poorly in the domain- and class- caassinnrrmmntal settings [47]. We therefore applied NCL as well as synaptic intelligence [SI; 54], online EWC [41], Kronecker factored EWC [KFAC; 37], and orthogonal weight modification [OWM; 53] to split MNIST and split CIFAR-100 in the task-, domain- and class-incremental learning settings. For these continual learning problems, we found that NCL outperformed all the baseline methods in the task- and nomain-incremental learning settings (Figure 2). In the class-increma incremental settings, we found that NCL performed comparably to but slightly worse than OWM. However, both OWM and NCL comfortably outperformed the other compared methods in this setting. These results suggest that the subpar performance of parameter regularization methods can be alleviated by regularizing their optimization paths, particularly in the domain- and class-ineremental learning settings.
For the split MNIST and split CIFAR- 100 experiments, each baseline method had a single hyper- parameter (c for SI, À for EWC and KFAC, a for OWM, and pm for NCL; Appendix E) that was optimized on a held-out seed (see Appendix 1.2). However, by setting the NCL. prior to unit Gaussian, we were also able to achieve good performance across task sets in hyperparameter- setting, further highlighting the robustness of the method (see "NCL (no opt)" in Figure 2).

---

task
domain
class
1.0
1.0
1.0
accuracy
MNIST
0.8
0.6
0.9
0.6
0.2
0.8
0.5
CIFAR accuracy
0.4
0.7
0.3
0.3
0.2
0.1
0.6
SI EWC KFAC OWM NCL (no opt)
NCL
SI
EWC
KFAC OWM
NCL (no opt)
NCL
Si
EWC KFAC OWM
NCL (no opt)
NCL
Figure 2: NCL performance in feedforward networks. Average test accuracy after learning all tasks on split MNIST (top row) and split CIFAR- 100 (bottom-row) the task-, domain- and class- incremental learning setting. Dashed horizontal lines denote average performance when networks trained simultaneously on all tasks. Solid horizontal lines denote average performance when networks trained sequentially each task without applying any continual learning methods. Error bars denote standard error across 20 (MNIST) or 10 (CIFAR) random seeds. 'NCI. indicates natural continual learning where the initial prior has been optimized on held-out random seed, and "NCL (no-prior)' indicates NCL with a simple unit Gaussian prior and no hyperparameter optimization. Numerical results for these experiments provided in Table in Appendix I.3.
3.2 NCL in recurrent neural networks
We then proceeded consider how NCL compares previous methods in recurrent neural networks (RNNs), a setting that has recently proven challenging for continual learning [12, 13] and which is of interest the study of continual learning in biological circuits [12, 51J. In these experiments, the task identity is available to the RNN (i.e., we consider the task-incremental learning setting).
Stimulus- response tasks In this section, we consider set of neuroscience inspired "stimulus- response' (SR) tasks (51; details Appendix B). We first compared the performance and behavior of NCL to OWM, the top performing method in the feedforward setting (Figure 2), and to the projection-ban based DOWM method designed explicitly for RNNS [12]. For a more direct comparison with OWM and DOWM, we fixed the NCL prior to unit Gaussian for all RNN experiments and instead performed a hyperparameter optimization over 'a' used to regularize the matrix inversions for all three methods (Section 2.2, Appendix E, Appendix I.2, 12, 53). Following previous work, we trained RNNs with 256 recurrent units to sequentially solve six stimulus-response tasks [12, 50]. While NCL, OWM and DOWM all managed to learn the six tasks without catastrophic forgetting, found that NCL achieved superior average performance across tasks after training (Figure 3A).
We then compared NCL, OWM, and DOWM to KFAC, the top performing parameter regularization method in our feedforward experiments (Figure 2) which uses Adam [23] to optimize the objective in Equation 4 with Kronecker-factoree approximation to the posterior precision matrix (Section 2.1: 37). Consistent with the results shown Duncker et al. [12]. we found that NCL, OWM, and DOWM outperformed KFAC with À (Figure 3A; see also 12 for comparison of DOWM and EWC). We note that NCL and KFAC optimize the same objective function (Equation 4) and approximate the posterior precision matrix in the same way, but they differ in the way they precondition the gradient of the objective. These results thus demonstrate empirically that the choice of optimization algorithm is important to prevent forgetting, consistent with the intuition provided by Figure 1.
In feedforward networks, poor performance with weight regularization approaches such as EWC and KFAC has been mitigated by optimizing the hyperparameter X, which increases the importance of the prior term compared standard Bayesian treatment (Equation 11; Section 3.1, 24, 27, 37). We confirmed this here by performing grid search over X, which showed that KFAC with À

---

A
SR
B
SMNIST
C
SMNIST
0.04
0.56
classification error
0.15
0.24
0.15
0.10
A error
loss
0.03
0.10
0.05
0.00
0.02
0.05
-0.05
NCL DOWM OWM KFAC (Aope) KFAC (A= 1)
NCL DOWM OWM KFAC (Aopt) KFAC (1=1 1)
5
10
15
task number
Figure 3: Performance on SR and SMNIST tasks. (A) Mean loss of NCL, DOWM, OWM, KFAC
(optimal A), and KFAC (A = 1) across stimulus-response tasks after sequential training on all tasks. Error bars indicate standard error across 5 random seeds. Here and in (B), KFAC with =1 failed catastrophically. and its performance is indicated in text as does not fit on the axes. (B) Mean classification error across SMNIST tasks after sequential training. (C) Difference between the mean classification error of Saplace-DOWN and NCL as a function of task number. Error bars in (B) and (C) indicate standard error across 100 random task permutations.
[100, 1000] could perform comparably the projection- baased methods (Appendix L1; Figure 3A). We hypothesize that the good performance provided by high is partly due to the approximate second order nature of Adam which, together with the relative increase in the prior term compared to the data term, leads to preconditioning with a matrix resembling the prior Ag- (Section 2.3). In support of this hypothesis, we found that the KL divergence between the Adam preconditioner and the approximate prior precision Ag-1 decreased with increasing X, and that the performance of KFAC with Adam could also be rescued by increasing only when computing the preconditioner while retaining À = when computing the gradients (Appendix I.1).
Stroke MNIST One way to challenge the continual learning algorithms further is to increase the number tasks. We thus considered an augmented version of the stroke MNIST dataset [SMNIST; 91. The original dataset consists of the MNIST digits transformed into pen strokes with the direction of the stroke at each time point provided as an input to the network. Similar to Ehret et al. [13], constructed continual learning problem by considering consecutive binary classification tasks inspired by the split MNIST task set. We further increased the number of tasks by including a set of extra digits where the x and y dimensions have been swapped in the input stroke data, and another set where both the and dimensions have changed sign. We also added high- variance noise to the inputs increase the task difficulty. This gave rise total of 15 binary classification tasks, each with unique digits not used in other tasks, which we sought to learn in a continual fashion using an RNN with 30 recurrent units (see Appendix for details).
As for the SR task set in Section 3.2, we found that NCL outperformed previous projection- based methods (Figure 3B). We again found that weight regularization with a KFAC approximation per- formed poorly with À = 1, and that this poor performance could be partially rescued by optimizing over À (Figure 3B). To investigate how the difference in performance between NCL and DOWM was affected by their different approximations to the Fisher matrix (Appendix F), we implemented NCL. using the DOWM projection matrices as an alternative approximation the inverse Fisher matrix. We refer to this method as Laplace-Dor -DWMM. We then considered how the performance on each task at the end of training depended on task number, averaged over different task permutations (Figure 3C). DDWMace-DOWM We found that while Laplace- DOWM outperformed NCL on the first task, this method generally performed worse on subsequent tasks. Notably, Laplace- -DMMM exhibited near-min monotonic decrease in relative performance with task number, which is consistent with the intuition that DOWM overesti- mates the dimensionality of the parameter subspace that matters for previous tasks (Appendix F). In contrast, although neural circuits are known to use orthogonal subspaces in different contexts, there no general sense that learning more tasks in the past should systematically hinder learning in future contexts for biological agents.
8

---

task 4/5 learned
task 1/7 learned
k=
k=2
k=3
k= 6
k=10
k= 15
-'10.44
P-10
P-10 1.0
- 0.98
A' 0.95
-'00.55
SSsh 5
7=000
-.00
P=10
P=10
A= 0.99
VS 7
Figure 4: Latent dynamics during SMNIST. We considered two example tasks, 4 vs 5 (top) and 1 (bottom). For each task, we simulated the response of network trained by NCL to 100 digits drawn from that task distribution at different times during learning. We then fitted a factor analysis model for each example task to the response of the network right after the correponding task had been learned (squares; k and k = 3 respectively). We used this model to project the responses at different times during learning into a common latent space for each example task. For both example tasks, the network initially exhibited variable dynamics with clear separation of inputs and subsequently acquired stable dynamics after learning to solve the task. The y² values above each plot indicate the similarity of neural population activity with that collected immediately after learning the corresponding task, quantified across all neurons (not just the 2D projection).
3.3 Dissecting the dynamics of networks trained on the SMNIST task set
To further investigate how the trained RNNs solve the continual learning problems and how this relates to the neuroscience literature, we dissected the dynamics of networks trained on the SMNIST task set using the NCL algorithm. To do this, analyzed latent representations of the RNN activity trajectories, as is commonly done to study the collective dynamics of artificial and biological networks [15, 18, 20, 29, 52]. We considered two consecutive classification tasks, namely classifying 4's vs 5's (k = 2) and classifying I's vs 7's (k- 3). For each of these tasks, we trained a factor analysis model right after the task learned, using network activity collected while presenting 50 examples of each of the two input digits associated with the task. We then tracked the network responses to the same set of stimuli at various stages learning, both before and after the task in question was acquired. using the trained factor analysis model to visualize low-dimensional summaries of the dynamics (Figure
Consistent with the network having successfully learned to solve these two tasks, we found that latent trajectories diverged over time for the types of inputs in each task. Critically, these diverging dynamics only emerged after the task was learned, and remained highly stable thereafter (Figure 4). The stability of the task-associated representations is consistent with recent work in the neuroscience literature showing that, in a primate reaching task, latent neural trajectories remain stable after learning [15). Since here we have access to the activity of all neurons throughout the task, we proceeded to quantify the source of this stability at the level of single units. The stability of such single-neuron dynamics after learning has recently been topic of much interest in biological circuits 28, 39]. In the RNNs, we found that the single-unit representations of given digit changed during learning of the task involving that digit but stabilized after learning, consistent with work in several distinet biological circuits [6, 11, 16, 19, 21, 35]. Similar results were found using the DOWM algorithm, which was explicitly designed preserve network dynamics on previously learned tasks [12]. Interestingly, the stable task representations learned by NCL and DOWM differed markedly from network trained with replay for continual learning, which instead led to task representations that continued to change after initial task acquisition (Appendix 1.4). This illustrates how different approaches to continual learning can lead qualitatively different circuit dynamics, and it suggests the use of continual learning in artificial networks model system for biological continual learning.

---

4 Discussion
In summary, we have developed new framework for continual learning based on approximate Bayesian inference combined with trust-region optimization. We showed that this framework en- compasses recent projection-bane based methods and found that it performs better than naive weight regularization. This was particularly evident when task identity was not provided at test time and in recurrent neural networks, settings which have previously been challenging for many continual learning algorithms [12, 13, 47J. Furthermore, we showed that our principled probabilistic approach outperforms previous projection-basee
methods [12, 53], in particular when the number of tasks and their complexity challenges the network's 's capacity. Finally, we analyzed the dynamics of the learned RNNs in a sequential binary classification problem, where we found that the latent dynamics adapt to each new task. We also found that the task associasd dynamics were subsequently con- served during further learning, consistent with experimental reports of stable neural representations [11, 15, 19]. Importantly, our results suggest that preconditioning with the prior covariance can lead to improved performance over existing continual learning algorithms. In future work, will therefore be interesting to apply this idea other weight regularization approaches such as EWC with a diagonal approximate posterior [24]. Finally, a separate branch of continual learning utilizes replay-like mechanisms to reduce catastrophic forgetting [8, 26, 34, 42, 43, 46]. While our work has focused on weight regularization, such regularization and replay are not mutually exclusive. Instead, these two approaches have been found further improve robustness to catastrophic forgetting when combined [32, 45].
Impact and limitations While we have shown that NCL represents an important conceptual and methodological advance for continual learning, it also comes with several limitations. One such limitation arises from the relative difficulty of computing the prior Fisher matrix which is needed for our projection step. Indeed the success of methods such as Adam [23] and EWC [24] is due in part to their ease of implementation which facilitates broad applicability. It will therefore be interesting to investigate how approximations such as running average of diagonal approximation to the empirical Fisher matrix as used in Adam could facilitate the development of simple yet powerful variants of NCL.
Furthermore, while NCL mitigates the need to overcount the prior from previous tasks via À in KFAC, it does introduce two other (largely redundant) hyperparameters in the form of (i) the scale of the prior before the first task, and (ii) the parameter a used to regularize the inversion of the prior Fisher matrix, similar to OWM and DOWM [12, 53J. While is an important hyperparameter for OWM and DOWM and we also optimize it in the RNN setting for more direct comparison (Section 3.2), we find it more natural to set this parameter to a constant small value present only for numerical stability (Appendix E). This leaves the prior scale which we optimize explicitly in the feedforward setting (Section 3.1). However, in future work it would be interesting to consider whether good prior can be determined in a data free manner to make NCL hyperparameter free method. Finally, computing the Fisher matrix used for pre-conditioning requires explicit knowledge of task boundaries. In future work, it will therefore be interesting to develop an algorithm similar to NCL which also works for online learning problems with continually changing task distributions.
Addressing these challenges important since machine learning algorithms increasingly need to be robust to changing data distributions and dynamic task specifications as they become more prominent in our everyday lives. Much work has therefore gone into the development of methods for continual learning in the machine learning community. However, with the increasing prevalence of practical algorithms for continual learning, also becomes increasingly important that we understand how and why these algorithms work insights that can also help us understand when they might fail. In this work, we have therefore attempted to shed light on the relationship between recent methods for continual learning as well as developing new algorithm with a principled probabilistic interpretation that makes its underlying assumptions more explicit. Taken together, we hope that this work will help improve our understanding of methods for continual learning while also providing an avenue for further research to increase the reliability and robustness of future continual lcarning algorithms.
10

---

Acknowledgements
We are grateful Siddharth Swaroop, Lea Duncker, Laura Driscoll, Naama Kadmon Harpaz, and Yashar Ahmadian for insightful discussions. We thank Siddharth Swaroop and Robert Pinsler for useful comments on the manuscript.
Funding disclosure
KTJ funded by a Gates Cambridge scholarship. GMV was supported by the Lifelong Learning Machines (L2M) program of the Defence Advanced Research Projects Agency (DARPA) via contract number HR0011-18-2-0025,
References
[1] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. (2018). Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV), pages 139-154.
[2] Amari, S.-I. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2):251-276.
[3] Ames, K. C. and Churchland, M. M. (2019). Motor cortex signals for each arm are mixed across hemispheres and neurons yet partitioned within the population response. eLife, 8:c46159.
[4] Bernacchia, A., Lengyel, M., and Hennequin, G. (2018). Exact natural gradient in deep linear networks and its application to the nonlinear case. Advances in Neural Information Processing Systems,
[5] Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. (2018). Riemannian walk for incre- mental learning: Understanding forgetting and intransigence. arXiv preprint arXiv:1801. 10112.
[6] Chestek, C. A., Batista, A. P., Santhanam, G., Byron, M. Y., Afshar, A., Cunningham, J. P., Gilja, V., Ryu, S. I., Churchland, M. M., and Shenoy, K. V. (2007). Single-neuron stability during repeated reaching in macaque premotor cortex. Journal of Neuroscience, 27(40):10742-10755
[7] Clopath, C., Bonhoeffeer T., Hübener, M., and Rose, T. (2017). Variance and invariance of neuronal long-term representations. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(17155:2110161.
[8] Cong, Y., Zhao, M., Li, J., Wang, S., and Carin, L. (2020). GAN memory with no forgetting. Advances in Neural Information Processing Systems, 33.
[9] de Jong, E. (2016). Incremental sequence learning. arXiv preprint arXiv:1611.0306
[10] De Lange, M. and Tuytelaars, T. (2020). Continual prototype evolution: Learning online from non-stationary data streams. arXiv preprint vrXiv:2009.009) 00919.
[11] Dhawale, A. K., Poddar, R., Wolff, Normand, V. A., Kopelowitz, E., and Ölveczky, (2017). Automated long-term recording and analysis of neural activity in behaving animals. elife,, 6:e27702.
[12] Duncker, L., Driscoll, I.., Shenoy, K. V., Sahani, M., and Sussillo, D. (2020). Organizing recurrent network dynamies by aaskcomputatiinn to enable continual learning. Advances in Neural Information Processing Systems, 33.
[13] Ehret, B., Henning, C., Cervera, M. R., Meulemans, A., von Oswald, J., and Grewe, B. F. (2020). Continual learning in recurrent neural networks with hypernetworks. arXiv preprins arXiv:2006.1 12109.
[14] Failor, S. W., Carandini, M., and Harris, K. D. (2021). Learning orthogonalizes visual cortical population codes. bioRxiv.

---

[15] Gallego, J. A., Perich, M. G., Chowdhury, R. H., Solla, S. A., and Miller, L. E. (2020). Long- term stability of cortical population dynamics underlying consistent behavior. Nature Neuroscience, 23(2)260-270.
[16] Ganguly, K. and Carmena, J. M. (2009). Emergence of stable cortical map for neuroprosthetic control. PLoS Biology, 1(7)x=0001153
[17] Huszár, F. (2017). On quadratic penalties elastic weight consolidation. arXiv preprint arXiv:1712.03847.
[18] Jensen, K., Kao, T.-C., Tripodi, M., and Hennequin, G. (2020). Manifold GPLVMs for discovering non-Euclidean latent structure in neural data. Advances in Neural Information Processing Systems, 33.
[19] Jensen, K. T., Kadmon Harpaz, N., Dhawale, A. K., Wolff, S. B. E., and Ölveczky, B. (2021a). Long-term stability of neural activity in the motor system. bioRxiv.
[20] Jensen, K. T., Kao, T.-C., Stone, J. T., and Hennequin, G. (2021b). Scalable Bayesian GPFA with automatic relevance determination and discrete noise models. bioRxiv.
[21] Katlowitz, K. A., Picardo, M. A., and Long, M. A. (2018). Stable sequential activity underlying the maintenance of precisely executed skilled behavior. Neuron, 98(6): 1133-1100.
[22] Kaufman, M. T., Churchland, M. M., Ryu, S. I., and Shenoy, K. V. (2014). Cortical activity in the null space: permitting preparation without movement. Nature Neuroscience, 17(3):440 448.
[23] Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[24] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A., Milan, K., Quan, J.. Ramalho, T., Grabska-Barwinske Barwinska, A., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13)35511335
[25] Kunstner, F., Balles, L., and Hennig, P. (2019), Limitations of the empirical fisher approximation for natural gradient descent. arXiv preprint arXiv: 1905. 12558.
[26] Li, Z. and Hoiem, D. (2017). Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40( (0(12):2935-29) -947..
[27] Loo, N., Swaroop, S., and Turner, R. E. (2020). Generalized variational continual learning. arXiv preprint arXiv:2011.12328
[28] Lütcke, H., Margolis, J., and Helmchen, F. (2013). Steady changing? long-term monitoring of neuronal population activity. Trends in Neurosciences, 36(7):375-33)
[29] Mante, V., Sussillo, D., Shenoy, K. V., and Newsome, W. T. (2013). Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78 84.
[30] Martens, J. (2014). New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193.
[31] Martens, J. and Grosse, R. (2015). Optimizing neural networks with Kronecker-factore approximate curvature. ICML, pages 2408-2417.
[32] Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E. (2017). Variational continual learning. arXiv preprint 1rXiv: 106..
[33] Osawa, K., Swaroop. S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. (2019). Practical deep learning with Bayesian principles. arXiv preprint arXiv:1906.02500
[34] Pan, P. Swaroop, S., Immer, A., Eschenhagen, R., Turner, R. E., and Khan, M. E. (2020). Contin- ual deep learning by functional regularisation of memorable past. arXiv preprint arXiv:2004. 1000020004.44000 /4070.
[35] Peters, A. J., Chen, X., and Komiyama, T. (2014). Emergence of reproducible spatiotemporal activity during motor learning. Nature, 5107750))2635352265
12

---

[36] Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. (2017). icarl: Incremental classifier and representation learning. In CVPR.
[37] Ritter, H., Botev, A., and Barber, (2018). Online structured laplace approximations for overcoming catastrophic forgetting. arXiv preprint 1r0iv
[38] Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap. T., and Wayne, G. (2019). Experience replay for continual learning. In Advances in Neural Information Processing Systems, pages 350-360.
[39] Rule, M. E., O'Leery, T., and Harvey, C. D. (2019). Causes and consequences of representational drift. Current Opinion in Neurobiology, 58:141-147.
[40] Saha, G., Garg, I., and Roy, K. (2021). Gradient projection memory for continual learning. arXiv preprint arXiv:2103.0976.
[41] Schwarz, J., Czarnecki, W., Luketina, J., Barsskaa-ariissa,, A., Teh, Y. W., Pascanu, R., and Hadsell, R. (2018). Progress compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4528 4537. PMLR.
[42] Shin, H., Lee, J. K., Kim, J., and Kim, J. (2017). Continual learning with deep generative replay. arXiv preprint arXiv:
[43] Titsias, M. K., Schwarz, J., Matthews, A. G. d. G., Pascanu, R., and Teh, Y. W. (2020). Functional regularisation for continual learning with gaussian processes. International Conference on Learning Representations.
[44] Tseran, H., Khan, M. E., Harada, T., and Bui, T. D. (2018). Natural variational continual learning. In Continual Learning Workshop NeurlPS, volume
[45] van de Ven, G. M., Siegelmann, H. T., and Tolias, A. S. (2020). Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11(1):1-14.
[46] van de Ven, G. M. and Tolias, A. S. (2018). Generative replay with feedback connections general strategy for continual learning. arXiv preprint arXiv: 10635.
[47] van de Ven, G. M. and Tolias, A. S. (2019). Three scenarios for continual learning. arXiv preprint 1xXiv:1904.0772.
[48] von Oswald, J., Henning, C., Sacramento, J., and Grewe, B. (2019). Continual learning with hypernetworss. arXiv preprint arXiv:
[49] Xu, T., Yu, X., Perlik, A. J., Tobin, W. F., Zweig, J. A., Tennant, K., Jones, T., and Zuo, Y. (2009). Rapid formation and selective stabilization of synapses for enduring motor memories. Nature, 462(7275):915-919
[50] Yang, G., Pan, F., and Gan, W.-B. (2009). Stably maintained dendritic spines are associated with lifelong memories. Nature, 462(7275):920-924
[51] Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T., and Wang, X.-J (2019). Task representations in neural networks trained perform many cognitive tasks. Nature Neuroscience, 22(2):297-306
[52] Yu, B. M., Cunningham, J. P., Santhanam, G., Ryu, S. I., Shenoy, K. V., and Sahani, M. (2009). Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Journal of Neurophysiology, 102(1):614-635.
[53] Zeng, G., Chen, Y., Cui, B., and Yu, S. (2019). Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364-37) 372.
[54] Zenke, F., Poole, B., and Ganguli, S. (2017). Continual learning through synaptic intelligence.
In International Conference Machine Learning, pages 3987-3995. PMLR.
13


Reviews:
[{"review_id": "zGLLMDW_953", "reviewer": "Reviewer_L9uv", "summary": "This paper proposes a model for continual learning. Specifically, it intends to merge parameter regularization methods and gradient projection methods. The regularization is achieved by taking a bayesian perspective over the distribution over parameters, that is approximated by a Laplace distribution in a variational fashion, with online updates. The gradient projection is addressed by trust-region optimization, resulting in an update rule that uses the precision matrix to precondition the gradient of the current task objective, plus a term that keeps the parameters close to the previous mean. Experiments are carried out for sequential problems on a stimulus-response dataset and on stroke MNIST.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall clear and well written, and the approach seems technically sound. All the mathematical notation is correct and the authors provide intuition for the most important equations, which helps readibility.\n\nIn terms of technical contribution, the novelty of this work is somewhat limited. As acknowledged by the authors, the bayesian perspective is very common in continual learning and the variational approximations involved here are quite popular. The use of the Fisher information matrix in lieu of the hessian is also standard. To my understanding, the overall model is closely related to Online-EWC [1], where however in the overall update the precision matrix weight the gradient of the current task objective rather than the regularization term. Also, it is not very clear to me the benefit of this scheme, besides getting rid of the need to tune an hyperparameter for the regularization loss.\n\nMy main concerns lie however in the experimental section. Specifically:\n  - it is peculiar that the authors limit the application of their model to RNN problems only. Indeed, there is nothing specific in the formulation that suggests that the model would behave especially well for sequential problems. Given the fact that those are not so popular in continual learning, I think the paper would benefit from the comparison on standard experiments such as the ones derived from MNIST (split, rotated, permuted) or CIFAR-10. \n  - the datasets tested seem to be quite simple, and more complex learning problems may be included to make the conclusions more convincing. \n  - Some other standard models such as EWC and its online version should be added to the comparisons, given they are very popular and close in spirit to the model proposed here.\n  - for the main comparisons, the authors report the loss (not clear whether it is on training or test sets) as a main metric. Other metrics would be more suitable, such as accuracy or forgetting measures.\n\n---\nOverall, I think the paper is slighly below the standard for NeurIPS. The approach presented here is clear and solid, but not fundamentally novel to my understanding. However, as mentioned above, the main weaknesses lie specifically in the experimental section. I think with a revision of the experiments this submission could meet the standard for acceptance. \n\nReferences:\n[1] Schwarz, Jonathan, et al. \"Progress & compress: A scalable framework for continual learning.\" International Conference on Machine Learning. PMLR, 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qg6A6wisGV", "reviewer": "Reviewer_tQo4", "summary": "This paper proposes a novel regularization based continual learning approach, NCL, which is mainly used for recurrent neural network (RNN). Since NCL is based on Bayesian continual learning, NCL utilizes the approximate posterior of previous tasks as a prior for current task, and the Laplace approximation is used to compute the approximate posterior. Through simple toy analyses on convex and non-convex loss landscapes for NCL and other baseline methods, authors show that previous methods have a problem on the trajectories of convergence when learning a new task. Based on the results on simple toy analyses, authors proposed a novel trust region based regularization method that modulate the gradient by considering the curvature of prior distribution via its precision matrix. The proposed technique is highly similar to projection based methods, but its origin is quite different from previous methods. Experiment results show that NCL outperforms other baselines in stimulus-response and MNIST tasks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n- This paper proposed a novel trust-region based regularization methods that can outperform other baselines. The methods modifying the gradients used to update the parameters are quite novel and impressive, and the authors well analyzed the similarity between NCL and other projection based methods in related work.\n\n- By showing simple toy analyses with other baselines, it is easy to see the problems of previous methods and exact trajectories when learning a new task. \n\nCons\n\n- It would be great for carrying out some large-scale experiments on language or time-series dataset. Based on the experiment results that considered, it is hard to decide NCL is effective on general RNN architecture.\n\n- I think the overall performance of NCL could be highly sensitive to the hyper parameter $r$ in Eq.(7). Since the $\\gamma$ in Eq.(8) depends on the $r$, I think it is not easy to choose the proper value of $r$,\n\n\nQuestion\n\n- The mathematical form of updating weights in NCL is highly similar to gradient projection based methods except the term $(\\theta - \\mu_{k-1})$ in Eq.(8). Does the excellence of NCL over gradient projection based methods is from considering Hessian in Laplace approximation or the term $(\\theta - \\mu_{k-1})$ which tries to reduce the deviation from $\\mu_{k-1}$?\n\n\nMinor comments\n\n- In my humble opinion, in line 146, there is a typo: involves \"maximizing\" the posterior --> involves \"minimizing\" the posterior.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48 hours"}, "extra_text": {}}, {"review_id": "jxMzsHd5RLb", "reviewer": "Reviewer_wyVR", "summary": "The authors introduce an extension to Bayesian continual learning whereby updates in successive tasks are constrained to be close to a “trust region” of the parameters found in preceding tasks. Exploiting the Bayesian nature of the model, the trust region in question can be defined in terms of the prior precision matrix. The authors show that the method, when applied to RNNs, is effective against baselines in a) a suite of neuroscience related tasks, and b) stroke MNIST datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper offers an elegant contribution to the continual learning literature, and I can find very little to criticise, so this review is pretty short. The formal presentation is rigorous and clear and (as far as I was able to check) appears sound. Figure 1 presents the intuition behind the work very clearly. The evaluation is thorough. The method’s relationship to the rest of the literature is well described (although I’m not sufficiently familiar with this particular corner of the continual learning field to vouch that nothing is missing). The paper is logically organised and well written, and I find myself in the unusual position of not having found a single typo. Overall I’m happy to recommend acceptance.\n\nI have one small recommendation. It wasn’t obvious to me why the authors chose an RNN model (and only an RNN model) to evaluate their method. Relatedly, it would have been interesting to see the method applied to other standard (non dynamic) continual learning benchmarks. I assume that these are the sorts of tasks where previous methods don’t do so well. (Apologies if the authors said that somewhere, but I can’t find it looking back over the paper.) Perhaps some intuition about why the method does well in this particular setting could also be given.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is at the level of basic research, and - in my opinion - is far enough from deployment not to have to societal impact explicitly", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "UC5frIov8t", "reviewer": "Reviewer_v6dD", "summary": "The paper addresses the problem of catastrophic forgetting during sequential task learning with recurrent neural networks, a notoriously difficult continual learning setting. \n\nThe main theoretical contribution is a formalism, Natural Continual Learning (NCL), which generalizes several related continual learning approaches in the regularization-based family, including several recent SOTA techniques. This formalism applies to the probabilistic learning setting using neural networks, so it is not restricted to supervised learning or recurrent models.\n\nExperimental evidence is provided to support the claim that NCL outperforms previous approaches.\n\nFinally, interesting analogies are made to models of biological learning, which are also supported by accompanying experiments; similar analysis tools are applied to practical instantiations of the NCL framework in controlled experimental settings.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\n## Context and Related work\nThe paper build on a well developed line of work in continual learning research, namely the regularization-based family of approaches. Such methods rely on relatively \"coarse\" approximations of second-order quantities, and hence have to come up with practical workaround which mitigate the issues arising from errors in such approximation and their systematic interactions throughout learning.\n\n***\n\n# Quality\n\n## Technical Soundness\n* I believe the technical framework proposed for NCL provides clarity and unifies several related approaches.\n* However, this it is not itself novel, since all such approaches start from a Bayesian perspective and diverge in practical implementations of diverse approximations of similar quantities.\n* While I believe there is merit to a unified approach, the primary limitations of this family of approaches are efficient and effective approximations; I believe the conceptual goals of this family of continual learning approaches were made clear in the literature for some time now, even as early as the EWC paper.\n \n\n## Assumptions and Limitations\n* While the text correctly identifies limitations of related approaches, it does not characterize the significance of the difference between related work and practical implementations of the proposed method. Could you please rectify this or point me to the relevant paragraphs?\n\n\n## Completeness of Evaluation\n* I am not convinced that the presented experiments sufficiently support the claims of superiority to related methods or indeed of relevance to the wider set of continual learning approaches.\n* Could you please compare like-for-like with existing approaches on standard settings beyond MNIST classification? While RNNs are of particular importance, it is not clear that the tasks considered are themselves sufficient to demonstrate the superiority of proposed approach, at least not in my view. This is because we already have many solutions to toy continual learning problems; in my view, sufficient progress means solving continual learning in more realistic problem settings.\n\n***\n\n# Significance\n\n## What's unique about this work?\n* I appreciate the in-depth study of continual learning with RNNs presented in this paper, the detailed experiments and carefully controlled problem setups considered. This is a valuable paper! \n* However, I do not believe it presents enough evidence for the proposed practical algorithms to support claims of superior performance to related or alternative methods. Please let me know whether there are other reasons why no other continual learning approach could be successfully applied to this niche and please argue for the niche itself rather than claiming general applicability. If you mean general applicability in practice, I believe much more evidence is required. If you mean general applicability in principle, please quantify exactly what \"general\" means and provide an exhaustive list limitations, of both conceptual and practical nature.\n\n\n# Summary\n* This paper is interesting to a number of researchers working on regularization-based continual learning solutions and perhaps modelers of biological learning systems with recurrent models. However, I cannot argue for wider applicability and relevance, which is reflected in my score.\n* Furthermore, while progress is being made and well reported in this paper, the significance of this progress to date since to be comparable to that of previously published papers, since the capabilities of continual learning algorithms proposed in this paper are virtually identical to previous work. I could not identify a strong argument for superior capabilities, but I am willing to be convinced otherwise.\n* While the perspective is interesting and the analyses revealing, I am not convinced that they contribute enough to recommend acceptance.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of impact is not particular to this paper, and does apply to the entire field of machine learning, but is well thought out, despite not being specific to this paper.\n\nI would suggest including a discussion of the impact of the work itself and its context rather than making too general remarks.\n\nAs it is the case with most papers and discouraged by prevailing reviewing practices, the paper expends little effort to characterize the limitations of the proposed practical algorithm, the diminished relevance of experiments presented within to setups of interest, to related approaches or the wider subfield. Truisms and 'catchphrases of the day' are offered in place of a honest discussion of what current continual learning methods have to offer today, and what is left for future work. I know this is difficult, but please make an effort along these lines!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "RQa_Kwobd0V", "reviewer": "Reviewer_fhnq", "summary": "The authors propose a method called Natural Continual Learning (NCL). NCL unifies the concepts of weight regularization and gradient projection into non-interfering subspaces. Gradient projection is implemented as a trust region optimization using the Fisher information matrix. The authors also introduce a Kronecker-factor approximation to improve the scalability properties of their method. The comparison against other continual learning approaches is conducted using recurrent neural network (RNN) architectures. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper to be original. \n\nThe paper is mostly clearly written (see Minor Issues later on) and well structured. \n\nA major advantage of the paper is that it is quite easy to follow the authors' motivation and thought process that led to the development of the proposed method.\n\nThe paper's significance is reduced by the fact that the experimental evaluation was somewhat unconventional and it was conducted only using RNNs. I consider this a significant downside, given that the method is not designed exclusively with RNNs in mind. The paper would be considerably strengthened were the authors to show strong results in more standard CL benchmarks, and when using more challenging datasets than MNIST.\n\nSpecifically for subsection 3.2, I found the conclusion reached (see lines 264-266) very interesting and I think it should be validated further using other datasets.\n\nMinor Issues:\n- Lines 59-60: The authors write \"or network size decreases.\" At first, I thought that the authors meant a decreasing network size over time, but I think they just mean a limited network capacity relative to the learning task at hand. If the latter is correct, you should probably rephrase.\n- In the problem statement paragraph, the authors should probably emphasize that each task has its own distinct data distribution.\n- I would also recommend, instead of using the words task and dataset interchangeably (e.g., in subsection 2.1), to instead use only the word task as is customary in CL papers.\n- For the vector Δ, introduced in line 147, I would use a bold lowercase character to emphasize the fact that it is indeed a vector and not a matrix.\n- Line 334: I think \"is not\" should be replaced with \"are not.\"\n- I recommend using the same vertical axis for Figures 3A and 3B so that they are directly comparable.\n\nQuestions:\n- Why did you choose to focus only on RNNs instead of experimenting with multiple types of architectures?\n- In the non-convex case of the toy problem, why did you set the Task 1 loss to be convex?\n- What is the motivation for expanding the number of tasks in the Stroke MNIST benchmark, and why did you make the specific choices detailed in lines 273-278?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework that unifies regularization based techniques in CL and projection techniques. As other reviewers pointed out (e.g. v6dD) this is maybe one of the better formally motivated and better studied direction for addressing CL. And hence at a high level the final result might not seem surprising (e.g. that regularizion methods can be seen as a trust region method, and then there is a deep connection between projection methods and these regularization methods). 

However I want to stress that the final algorithm and its derivation is by no means trivial. And I think the improvement (at least in the theoretical understanding) of these methods is of great values for those focusing on such techniques. So from my perspective, in terms of novelty and significance I think the manuscript has provided enough of both. And while there might be some scalability worries, I think there is a significant subset of the CL community that will find this result very useful. 

The main weakness of the work is the empirical exploration. While I believe the experiments are done carefully, and provide evidence for the efficacy of the algorithm (plus I welcome the new experiments on feedforward models), the scale of the empirical section is a bit weaker compared to an average paper on this topic. However, particularly considering the new results mentioned by the authors that should be integrated in the paper, I think they might just be sufficient to reach the requirement of the conference. 


Decision: Accept (Poster)