Regularizing Trajectories to Mitigate Catastrophic Forgetting
Authors: Paul Michel
Elisabeth Salesky
Graham Neubig
Venue:  (2020)
Keywords: Continual Learning
Regularization
Adaptation
Natural Gradient

Abstract: Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in most practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters.
In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the co-natural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low resource scenario.

Under review as a conference paper at ICLR 2020
REGULARIZING TRAJECTORIES TO MITIGATE CATASTROPHIC FORGETTING
Anonymous authors
Paper under double-blind review
ABSTRACT
Regularization- based continual learning approaches generally prevent catas- trophic forgetting by augmenting the training loss with an auxiliary objective. However in practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical pa- rameters. In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for con- tinual learning whereby the new task gradients are preconditioned with the em- pirical Fisher information of previously learnt tasks. We show that using the co- natural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning new task in a low-resource scenario
INTRODUCTION
It is good to have end journey toward; but is the journey that matters, the end.
Ursula K. Le Guin
Endowing machine learning models with the capability learn a variety of tasks in sequential manner is critical to obtain agents that are both versatile and persistent. However, continual learning of multiple tasks is hampered by catastrophic forgetting (McCloskey & Cohen 1989 Ratcliff 1990), the tendency of previously acquired knowledge to be overwritten when learning new task.
Techniques to mitigate catastrophic forgetting can be roughly categorized into lines of work (see Parisi et al. (2019) for a comprehensive overview): 1. regularization-bass approaches, where for- getting is mitigated by the addition of penalty term in the learning objective (Kirkpatrick et al.) (2177) ; [Chaudhry al. (2018a). inter alia), 2. dynamic architectures approaches, which incremen- tally increase the model's ' capacity to accomodate the new tasks (Rusu et al., 2016), and 3. memory- based approaches, which retain data from Icarned tasks for later reuse (Lopez-Paz Paz & Ranzato 2017]
Chaudhry et al.] 2018b) 2019). Among these, regularization- based approaches are particularly ap- pealing because they do not increase the model size and do not require access to past data. This is particularly relevant to real-world scenarios where keeping data from previous training tasks may be impractical because of infrastructural or privacy-related reasons. Moreover, they are of inde- pendent intellectual interest because of their biological inspiration rooted in the idea of synaptic consolidation (Kirkpatrick al., (017).
A good regularizer ensures that, when learning new task, gradient descent will ultimately converge to parameters that yield good results on the new task while preserving performance on previously learned tasks. Critically, this is predicated upon successful optimization of the regularized objective, a fact that has been largely taken for granted in previous work. Non-convexity of the loss function, along with noise in the data (due to small or biased datasets) or in the gradients (due to stochastic gradient descent), can yield optimization trajectories - and ultimately convergence points that highly non-deterministic, even for the same starting parameters. As we demonstrate in this paper, this can cause unintended catastrophic forgetting along the optimization path. This is illustrated in toy setting in Figure 1 two parameter model is trained to perform task T2 (an arbitrary bi-modal loss function) after having learned task Tj (a logistic regression task). Standard finetuning, even in
We commit releasing the code implement our method and reproduce experiments upon acceptance.

---

Under review as a conference paper at ICLR 2020
Finetuning
EWC
Co-natural finetuning
On the importance trajectories: an example with 2-dimensional logistic regres- sion. Having learned task Ti, the model is trained on T2 with two different objectives: minimizing the loss on Tg (Finetuning) and regularized objective (EWC: Kirkpatrick et al. (2017). We add small amount of Gaussian noise to gradients in order to simulate the stochasticity of the trajectory. Plain finetuning and EWC often converge to a solution with high loss for Ti, but the co-natural optimization trajectory consistently converges towards the optimum with lowest loss for Ti.
Figure I:
the presence of a regularized objective (EWC; Kirkpatrick et al. (017)). quickly changes the loss of Ti and tends converge to solution with high 7 loss.
We propose to remedy this issue by regularizing the optimization trajectory itself. specifically by preconditioning gradient descent with the empirical Fisher information of previously learned tasks (33). This yields what we refer to co-natural gradient, an update rule inspired by the natural gradient (Amari) 1997), but taking the Fisher information of previous tasks as natural Riemannian metrid of the parameter space, instead of the Fisher information of the task being optimized for. When we introduce our proposed co-natural gradient for the toy example of Figure [ the learning trajectory follows a path that changes the loss on I' much more slowly, and tends to converges to the optimum that incurs the lowest performance degradation on Ti.
We test the validity of our approach in continual learning scenario (14). We show that the co-natural gradient consistently reduces forgetting in a variety of existing continual learning approaches by a factor of s 1.5 9, and greatly improves performance over simple finetuning, without modification to the training objective. We further investigate the special case of transfer learning in a two-task, low-resource scenario. In this specific case, control over the optimization trajectory particularly useful because the optimizer has to rely on early stopping to prevent overfitting to the meager amount of training data in the target task. We show that the co-natural gradient yields the best trade-offs between source and target domain performance over variety of aaper-parameters (5))
2 BACKGROUND AND NOTATIONS
We first give brief overview of the continual learning paradigm and existing approaches for over- coming catastrophic forgetting.
2.1 NOTATION
Let us define task as a triplet containing an input space X' and an output space ), both measurable spaces, well distribution D over X' y. In general, learning task will consist of training model to approximate the conditional distribution p(y x) induced by D.
Consider probabilistic model pe parametrized by 0 E Rd where d is the size of the model, trained to perform a source task S = (X's. Ds) to some level of performance, yielding parameters 0g. In the most simple instance of continual learning, T= (Xr, Rr, Dr). In general in a multitask setting, it not the case that the imput or output spaces the same. The discrepancy between input/output space be addressed in various ways, e.g. by adding a minimal number of task-specific parameters (for example, different softmax layers for different label sets). To simplify exposition, we these more specific considerations aside for now, and assume that X = X7 and Ys- = yT.
tasked with learning second target task
Inforrally the reader can think of Riemannian metric as function that assigns inner product u, + Ose (t, v)) each point the space, thus inducing localized notion of distance and curvature.
2

---

Under review as a conference paper at ICLR 2020
At any given point during training for task T, our objective will be to minimize the loss function (r) generally the expected log-likelihood Ex,g--r 1-cpprlllogg.. po po(y x)}. Typically, this will be performed by itcratively adding incremental update vectors E R to the parameters + 0 0 8.
2.2 EXISTING APPROACHES FOR CONTINUAL, LEARNING
In this paper, we focus on those models that have fixed architecture over the course of continual learning. The study of continual learning for models of fixed capacity can be split into two distinct (but often overlapping) streams of work:
Regularization-base approaches introduce penalty in the loss function Lr. typically quadratic, pushing the weights back towards 0g:
(0) Ez,,yDDr - log Pe 1))+ X(O 0s) 0g)
NLL task T
Regularization term
where Ggg is a matrix, typically diagonal, that encodes the respective importance of each parameter with respect to task S, and À is regularization strength typer-parameter. Various choices have been proposed for Q2y; the diagonal empirical Fisher information matrix (Kirkpatrick et al. 2017). or path-
integral based importance measures (Zenke et al. 2017 Chaudhry et al.) 20)8 More elaborate regularizers have been proposed based on egg. a Bayesian formulation of continual learning (Nguyen
et al. 2017]] Ahn et al. 2019) or distillation term (Li & Hoiem 2016 Dhar et al.) (019).. The main
advantage of these approaches is that they do not rely on having access to training data of previous tasks.
Memory-basee approaches store data from previously seen tasks for re-use in continued learning,
either as form of constraint, by e.g. ensuring that training on the new task doesn't 't increase the loss on previous tasks (Lopez-Paz & Ranzato 2017 Chaudhry et al. 2018b). or for replay i.e. by al., [019 Aljundi et al. 2019b a). Various techniques have been proposed for the selection of samples to store in the memory (Chaudhry et al. 22199; Aljundi al. 2019b) for retrieval of the samples to be used for replay Aljundi et al. (2019a).
retraining instances from previous tasks (Rebuffi al., [017]] Chaudhry
All of these methods rely on stochastic gradient descent to optimize their regularized objective or to perform experience replay, with the notable exception of GEM (Lopez-Paz & Ranzato [2017] Chaudhry et al., 2018b), where the gradients are projected onto the orthogonal complement of pre- vious task's s gradients. However, this method has been shown to perform poorly in comparison with simple replay (Chaudhry et al. 2019), and still necessitates access to data from previous tasks.
REGULARIZING THE TRAJECTORY
After briefly recalling how the usual update is obtained gradient descent, we derive new, co- natural update designed to better preserve the distribution induced by the model over previous tasks.
3.1 WARM UP: THE STANDARD GRADIENT DESCENT UPDATE
At point in the parameter space, gradient descent finds the optimal update that is (1) small and (2) locally minimizes the decrease in loss L(0 + 6) - C(0) (=e át Ve at the first order). Traditionally
this can be formulated as minimizing the Lagrangian:
L(()=
8T V
11
first order loss minimization term
"smlll update" term
with Lagrangian multiplier y > 0. Minimizing IL for 5 yields the well-known optimal update 8":
1
8*
VoLT
(3)
24
3

---

Under review as a conference paper at ICLR 2020
where 1. corresponds to the learning rate (see Appendix A.1 for the full derivation). 2p
3.2 KL REGULARRRZZAION OF TRAJECTORIES
The ||6]|2 term in LL implicitly expresses the underlying assumption that the best measure of distance between parameters 9 and 0 + is the Euclidean distance. In continual learning setting however, the quantity are most interested preserving is the probability distribution that 0 models on the source task S:
(r, po(y xx)²(x²)
(4)
Therefore, a more natural distance between 0 and + 5 is the Kullback-Leibler divergence (L(p)) Il (Kullback & Leibler 1951). For preventing catastrophic forgetting along the opti- mization path, we incorporate incorporate this KL term into the Lagrangian L itself:
(K L(o) = VVV p|00] / IPp5.5
(5)
Doing so means that the optimization trajectory will tend to follow the direction that changes the distribution of the model the least. Notably, this is not function of the previous objective Ls. so knowledge of the original training objective is not necessary during continual learning (which is typically the case in path-integral based regularization methods (Zenke al. 2017) or experience replay (Chaudhry et al. 2019).
3.3 CO-NATURAL GRADIENT OPTIMIZATION
Presuming that is small, we can perform a second order Taylor approximation the function + KL(yållp (1p)+)) around O. Considering that both the zeroeth and first order terms are null because 0 is global minimizer of + (L this reduces the Lagrangian quadratic optimization problem (we refer the reader to Pascanu & Bengio (2013) for a more detailed derivation.):
1 L(6) = volr u|1011 + P F/
(6)
2
where PS is the Hessian of the KL divergence around A crucial, well-known property of this matrix is that it coincides with the Fisher information matrix Ez.y--oll [(V log (V log p ) (the expectation being taken over the model's distribution po: see Appendix A.1 for details). This is appealing from a computational perspective because the Fisher can be computed by means of first order derivatives only.
Minimizing for ō yields the following optimal update:
8" = - + aI] al] VoLr
[FS
where coefficients μ and folded into two hyper-parameee coefficient a (the step-by-step derivation can be found in Appendix [. In practice, especially with low damping coefficients, it is common to obtain updates that too large (typically when some parameters have no effect on the KL divergence). To address this, we re-normalize 8* to have the same norm the original gradient, |[[]-
the learning rate À and damping
For computational reasons, we will make 3 key practical approximations to the Fisher:
1. F ~ .g. we maintain the Fisher computed at 0s, instead of recomputing Fs at every step of training. This relieves us of the computational burden of updating the Fisher for ev- ery new value of 0. This approximation (shared by previous work, e.g. [Kirkptrrick et al. (haudhyyy et al. (2018a)) is only valid insofar as 0g and 0 are close. Empirically we observe that this still leads to good results.
'ennce of the letter F designate the Hessian
4

---

Under review as a conference paper at ICLR 2020
2. F$ is diagonal: this is common approximation in practice with two appealing properties. First, this makes it possible to store the d diagonal Fisher coefficients in memory. Second, this trivializes the inverse operation (simply invert the diagonal elements).
3. Empirical Fisher: this common approximation replaces the expectation under the model's distribution by the expected log-likelihood of the true distribution: E-pp-p [(V log Pá (V (V log p J (mind the subscript) This particularly useful in tasks with large unbounded number of classes (e.g. structured prediction), where summing over all possible outputs intractable. We can then compute the diagonal of the empirical Fisher using Monte Carlo sampling: N E (V log xz]]² with (I), y4) sampled from Dy (we use N 1000 for all experiments).
This formulation bears many similarities with the natural gradient from Amari 1997), which also uses the KL divergence a metric for choosing the optimal update 8*. There however crucial difference, both in execution and purpose: where the natural gradient uses knowledge of the curvature the KI. divergence 'Dr to speed-up convergence, our proposed method leverages the curvature of the KI. divergence on Ds to slow-down divergence from Pd. To highlight the resemblance and complementarity between these two concepts, we refer to the new update as the co-natural gradient.
3.4 BEYOND TWO TASKS
In a continual learning scenario, we are confronted with large number of tasks Tq ... Tn presented in sequential order. When learning Tn, we can change the Lagrangian L from 5 to incorporate the constraints for all previous tasks Ti...Tn-1.
=(8)= 8 2
Σ
n. KL (1p84.)
This in turn changes the Fisher in Eq. 8 to Fn := Σm F The choice the coefficients i is crucial. Setting all v, to the same value, i.e. assigning the same importance to all tasks is suboptimal for a few reasons. First and foremost, it is unreasonable to expect of model with finite capacity to remember an unbounded number of tasks (as tasks "fill-up" the model capacity, Fa-1 is likely to become more "homogemcous"). Second, as training progresses and changes, approximation that F For; is less and less likely to hold.
We address this issue in the same fashion as Schwarz et al. 20188. by keeping rolling exponential average of the Fisher matrices:
F7 = + +(1 F.
In this case, previous tasks are gracefully forgotten an exponential rate controlled by 7. We account for the damping term in Eq. 7 by setting Fg T. In preliminary experiments, have found y 0.9 to yield consistently good results, and this value in all presented experiments.
4 CONTINUAL LEARNING EXPERIMENTS
4.1 EXPERIMENTAL SETTING
To corroborate hypothesis that controlling the optimization trajectory with the co-natural gradi- ent reduces catastrophic forgetting, we perform experiments on two continual learning testbeds:
Split CIFAR: The CIFAR100 dataset, split into 20 independent 5-way classification tasks. Similarly to Chaudhry et al. (2018)). we use smaller version of the ResNet architecture (He et al. 2016).
5

---

Under review as a conference paper at ICLR 2020
Table 1: Average accuracies and forgetting after all tasks have been learnt, with and without the co-natural gradient. Results are reported in percentages (#: the standard deviation over 5 re-runs). Bold print indicates statistically significant difference between standard and co- co-natural (p 0.05).
Split CIFAR EWC
Omniglot EWC
Finetuning
ER
Finetuning
ER
Average accuracy t
Standard
35.92 1.19
44.86
+2.01
61.08
+0.94
16.31 +1.05
70.31
+3,46
70.90
=0.80
Co-natural
56.82 11.47
56.50
+1.28
59.34
+2.02 71.25 +4.90
69.11
+4.70
75.48
11.2
Forgetting
Standard Co-natural
34.05 +0.99
17.50
+2.09
10.66
+0.70
77.26
+1.11
14.16
+2.35
22.43 0..8
5.53 +1.12 4.91
+0.92
5.25 +1.20
8.49 +2.75
8.73
+1.81
5,21 +0.30
Table 2: continual-leamingg results for Split MinilmageNet (see Figure | for details).
Split MinilmageNeet Finetuning
EWC
ER
Average accuracy 1
Standard Co-natural
36.86 12.10
58.43 11.73
63.10 14.04
63.14 12.57
62.90 11.61
70.59
10.25
Forgetting
Standard
40.51 11.9
12.47 42.2
15.29 +3,97
Co-natural
11.06 +:3.2
8.90 1.91
8.12 +1.2
the Omniglot dataset (Lake al. 2015) consists of 50 independent character recog- nition datasets different alphabet. We adopt the setting Schwarz et al. (2018) 8) and consider each alphabet as a separate task 4 On this dataset we use the same small CNN architecture as
Omniglot:
Schwarz et al. (2018).
Split MinilmageNet: The MinilmageNet dataset (a subset of the popular ImageNet (Deng et al., 2009) datase Vinyals et al. (2016)). Split the dataset into 20 independent 5-way classi- fication tasks, similarly to Split CIFAR, and use the same smaller ResNet.
We adopt the experimental setup from Chaudhry et al. (2019): in each dataset we create a "validation set" of tasks, used to select the best hyper- paper-parameters, and keep the remaining tasks for evaluation. This split is chosen at random and kept the same across all experiments. and possibly the number of classes for each task changes. We account for this by training separate softmax layer for each task, and apply continual learning only to the remaining, "feature-extraation" part of the model.
these datasets, the nature
We report results along two common metrics for continual learning: average accuracy, the accuracy the end of training averaged over all tasks, and forgetting. Forgetting is defined in Chaudbry et al. (2018a) as the difference in performance on a task between the current model and the best performing model on this task. Formally AT represents the accuracy on task T step of training. the forgetting F? at step is defined as FT = max, <t AT - A!" 'Low forgetting" means that the model tend to keep the same level of performance on task it has learned.
We implement the co-natural update rule on top of 3 baselines:
Finetuning: Simply train the model on the task at hand, without any form of regularization.
EWC: Proposed by Kirkpatrick al. (017). it simple but effective quadratic regularization approach. While neither the most recent nor sophisticate regularization technique. it is a natural bascline for us compare in that it also consists in Fisher-basee penalty although in the
NNte that this is different setting than the usual meta-learnnng scenario that Omniglot used for. Similarly to Omniglot, MinilmageNet was originally intended as a meta-learning benchmark and therefore its standard train/validationntest split consists of disjoint classes. We perform custom transversal split so that the dataset can he used standard 100-way classification task. The accuracies reported here are not to be compared with the meta-learning literature.

---

Under review as a conference paper at ICLR 2020
0
10
20
30
40
0
10
20
30
40
0
10
20
30
40
0 10
20
30
40
(a) Finetuning
(b) EWC
(c) ER
(d) Co-natural Finetuning
Figure 2: Evolution of task performance over the course of continual learning on one ordering of Omniglot. For visibility we only show accuracies for every fifth task. The rectangular shaded regions delineate the period during which each task is being trained upon; with the exception of ER, this is the only period the model has access to the data for this task.
loss function instead the optimization dynamics. We also use the rolling Fisher described in Section 3.4 making our EWC baseline equivalent to the superior online EWC introduced by Schwarz et (2018).
ER: Experience replay with a fixed sized episodic memory proposed by Chaudhry et al. 2019). While not directly comparable to EWC in that it presupposes access to data from previous tasks, ER is a simple approach that boasts the best performances on a variety of benchmarks memory size 1,000 with reservoir sampling.
(Chaudhry al. 2019). In all experiments,
Training proceeds as follows: we perform exhaustive search on all the syper-parameter combina- tions using the validation tasks. Every combination is reran 3 times (the order of tasks, model initialization and order of training examples changes with each restart), and rated by accuracy aver- aged over tasks and restarts. We then evaluate the best ayper-parameters by continual training the evaluation tasks. Results are reported over 5 random restarts (3 for MinilmageNet, and we control for statistical significance using a paired t-test (we pair together runs with the task ordering). We refer to Appendix A.2 for more details regarding fine-graa fine-grained design choices.
4.2 RESULTS
The upper half of Table reports the average accuracy of all the tasks the end of training (higher better). We observe that the co-natural gradient always improves greatly over simple finetuning, and occasionally over EWC and ER. We note that on both datasets, bare-bone co-natural finetuning matches or exceeds the performance of EWC and ER even though it requires strictly fewer resources (no need to store the previous parameters in EWC, data in ER).
Even more appreciable is the effect the co-natural trajectories on forgetting, as shown in the lower half of Table [ As evidenced by the results in the lowest row, using the co-natural gradient systematically results in large drops in forgetting across all approaches and both datasets, even when the average accuracy is not increased.
To get qualitative assessment of the learning trajectories that yield such results, we visualize the accuracy curves of 10 out of the 47 evaluation tasks of Omniglot in Figure [ We observe that previous approaches do poorly at keeping stable levels of performance over long period of time (especially for tasks learned early training), problem that largely resolved by the co-natural preconditioning. This seems come at the cost of more intransigence (Chaudhry et al. 201 8a), i.e. some of the later tasks are not being learnt properly. models of fixed capacity, there natural trade-off between intransigence and forgetting (see also the Sttability-plastiiiyy dilemma in neuroscience [Grossberg (1982)). Our results position the co-natural gradient as strong low- forgetting/moderate intransigence basis for future work.
LOW-RESOURCE ADAPTATION EXPERIMENTS
In this section we take a closer look at the specific case of adapting a model from a single task to another, when we only have access minimal amount of data in the target task. In this case,

---

Under review as a conference paper at ICLR 2020
39.2
Finatuning
39.0
EWC
Co-natural finetuning Co-natural EWC
90.0
38.8
Minimagenee accuracy
87.5
11 DLEU
38.6
85.0
38.4
82.5
38.2
80.0
38.0
Base model score
77.5
37.8
Co-natural finetuning
*k
37.6
Coinatural EWC
75.0
10
14
16
30.0 31.0 31.2 31.4 31.6 31.8 32.0 32.2 32.4
accuracy
(a) MinilmageNet to CUB adaptation
(b) WMT to MTNT fine-tuning
Figure 3: Low-resource adaptation results. The source (resp. target) task performance is represented on the vertical (resp. horizontal) axis. Pareto optimal configurations for each method are highlighted and the frontier represented with dashed lines. The solid gray lines indicate the score of the original model trained the source task.
controlling the learning trajectory particularly important because the model is being trained on an unreliable sample of the true distribution of the target task, and we have to rely on carly-stopping to prevent overfitting. We show that using the co-natural gradient during adaptation helps both at preserving source task performance and reach higher overall target task performance.
5.1 EXPERIMENTAL SETTING
We perform experiments different scenarios:
Image classification We take Minilmagenet as a source task and CUB (a 200-way birds species classification dataset; Welinder al. (2010) as target task. To guarantee strong base model despite the small size of MinilmageNet, we start off from a ResNet18 model (He et al. 2016) pretrained on the full ImageNet, which retrofit to MinilmageNet by replacing the last fully con- nected layer with a separate linear layer regressed over the MinilmageNet training data. To simulate a fow-resource setting, we sub-sami sample the CUB training set to 200 images (s= per class). Scores for these tasks are reported in terms of accuracy.
Machine translation We consider adaptation of an English French model trained on WMT15 (a dataset of parallel sentences crawled from parliamentary proceedings, news commentary and web page crawls: Bojar et (2055)) to MTNT (a dataset of Reddit comments: Michel & Neubig (2018)) Our model Transformer (Vaswani et al.) 2017)) pretrained WMT15, Similarly to CUB, simulate a low-resource setting by taking a sub-sample
1000 sentence pairs as a training set. Scores for these two datasets reported in terms of BLEU score Papineni et al. 2002
Here we do not allow any access to data the source task when training on the target task. We compare four methods Finetuning (our baseline), Co-natural finetuning. EWC (which has been proven effective for domain adaptation, see Thoomson et al. (019)) and Co-natural EWC.
Given that different methods might lead to different trade-off' between source and target task per- formance, with some variation depending on the hyper-parametery parameters (e-g. learning rate, regularization strength...), take inspiration from Thompson et al. (2019) and graphically report results for all hyper-parameter configuration of each method on the dimensional space defined by the score on
"We sacrebleu (Post 2018) with int1 recommended by Michel Neubig 2018
8

---

Under review as a conference paper at ICLR 2020
source and target tasks Additionally, we highlight the Pareto frontier of each method i.e. the set of configurations that are not strictly worse than any other configuration for the same model.
5.2 RESULTS
The adaptation results for both scenarios are reported in Figure We find that in both cases, the co-natural gradient not only helps preserving the source task performance, but to some extent also allows the model to reach better performance on the target task as well. We take this to corrobo- rate our starting hypothesis: while introducing a regularizer does help, controlling the optimization dynamics actively helps counteract overfitting to the very small amount of training data, because the co-natural pre-conditioning makes it harder for stochastic gradient descent to push the model towards directions that would also hurt the source task.
CONCLUSION
have presented the co-natural gradient, a technique that regularizes the optimization trajectory of models trained in continual setting. We have shown that the co- co-natural gradient stands its own as an efficient approach for overcoming catastrophic forgetting, and that it effectively complements and stabilizes other existing techniques minimal cost. We believe that the co-natural gradient and more generally, trajectory regularization - can serve as a solid bedrock for building agents that learn without forgetting.
Forr CUB in particular report the average accuracy of every configuration over 5 runs, each with a different 200-sized random subset of the data.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hongjoon Ahn, Donggyu Lee, Sungmin Cha, and Taesup Moon. Uncertainty- based continual learn- ing with adaptive regularization. In Proceedings of the 32nd Annual Conference on Neural Infor- mation Processing Systems (NIPS), 2019.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NIPS), 2019:
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NIPS), 2019b.
Shun-ichi Amari. Neural learning in structured parameter spaces-natural riemannian gradient. In Proceedings of the 9th Annual Conference on Neural Information Processing Systems (NIPS), pp- 127-133, 1997.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the 10th Workshop on Statistical Machine Translation (WMT), pp- 1-46, 2015.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the 16th European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. Proceedings of the International Conference Learning Repre- sentations (ICLR), 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic mem- ories. arXiv preprint arXiv: 10486, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng. Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Stephen T Grossberg. Studies Mind and Brain: Neural Principles of Learning, Perception, Devel- opment, Cognition, and Motor Control, volume 70. Springer Science & Business Media, 1982.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the 29th IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 770-778, 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnkkakaaaikkaaa..aa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, (14(13):351-3356 2017.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):99-86, 1951.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350: 1332-1338,, 2015.
Zhizhong Li and Derek Hoiem. Learning without forgetting. Proceedings of the 14th European Conference on Computer Vision (ECCV), 2016.
10

---

Under review as a conference paper at ICLR 2020
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS), pp. 6467-6476, 2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Paul Michel and Graham Neubig. MTNT: testbed for machine translation of noisy text. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1443-53,777 2018.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311-318, 2002.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2013.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the 3rd Conference Machine Translation (WMT), pp. 186-191, 2018.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 2016.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Saabska-Barwinska. Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning. In (P Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 4535 4544, 2018.
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Kochn. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLLT), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS), pp. 5098-5999009 6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Proceedings of the 30th Annual Conference Neural Information Processing Systems (NIPS), pp. 3630-3633 2016.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical Report ENS-TR-2010-00 California Institute of Technology, 2010.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference Machine Learning (ICML), pp. 3987- 3995, 2017.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 DERIVATIONS
A.1.1 THE STANDARD GRADIENT UPDATE (EQUATION
We derive the standard update in Eq. by solving the Lagrangian in Eq. for 6, Given that its first and second derivatives are:
VL VLr + 2pó
V 2p1/
the problem is trivially strictly convex and its global minimizer 8 satisfies:
1 VLT
VL = -
21
A.1.2 EQUIVALENCE OF THE HESSIAN OF THE KL DIVERGENCE AND THE FISHER INFORMATION MATRIX
To simplify notation, let perform the change of variables 0 + 5 - We show that the Hessian of the KI. coincides with the Fisher on in other words, (pel(ppll Epo (V PT) Under mild regularity assumption we can write:
A VLKL(pellzz> = 2 Ens llog pa V Epp [log
E 2 log px
V Pz.
Now note that log can rewritten via standard derivatives manipulations as (Vp.)(V).) This leads to:
P2
P2p
Vpz
Vp
V'KL(pollpa) -Bp
Ep
Pz
Pr.
When taken at 0, the first term evaluates to
V2pa
Pa
f po(=)
(ppo(z) dz (=(2)
E
Px
am00
v2 pz()d
2
Po (=(2)d 0
By using the identity
Vps Ps
log Par and evaluating at 9. the second term gives us:
V (KL(ppolppal Epa [(V pog (V ]
"Essentially allowing us to interchange derivatives and integrals. We abuse notation and write Vpz |p=|znp as Vps
12

---

Under review as a conference paper at ICLR 2020
A.1.3 OBTAINING THE CO-NATURAL UPDATE (BQUATION)
7
We solve the Lagrangian from Eq. i similar fashion in A.I.1 and Hessian:
First we compute its gradient
VL = VLr + 2nō 2
VLr 2(UF5 uT)
L 2(vF) ul)
While not as straightforwardly as the one in A.I.1. this problem is also strongly convex: indeed F is positive semi-definite (as an expectation of PSD matrices) and the addition of μI ensures that VL is positive definite. We find the unique solution by solving:
vL13- =0
+ 2(vFF pI)0 =0
12 + +-21|-
Set == 2μ and a = to get Eq.
A.2 ADDITIONAL EXPERIMENTAL SETTINGS FOR CONTINUAL. LEARNING
This section is intended to facilitate the reproduction of our results. The full details can be found with our code at anonymi zed url.
A.2.1 SPLIT CIFAR
We split the dataset into 20 disjoint sub-tasks with each classes, 2500 training examples and 500 test examples. This split, performed at random, is kept the same across all experiments, only the order of these tasks is changed. During continual training. we train the model for one epoch on each task with batch size 10, following the setup in Chaudhry et al. (2018b).
A.2.2 OMNIGLOT
We consider each alphabet as a separate task, and split each task such that every character is present 12, 4 and 4 times in the training, validation and test set respectively (out of the 20 images for each character). During continual training, we train for 2500 steps with batch size 32 (in keeping with Schwarz al. (2018). We ignore the validation data and simply evaluate the test set at the end of training.
A.2.3 GRID-SEARCH PARAMETERS
For each method, we perform grid-search over the following parameter values:
Learning rate (all methods): 0.1, 0.03, 0.01
Regularization strength (EWC, Co-natural EWC): 0.5, 1,5
Fisher damping coefficient (Co-natural finetuning, Co-natural EWC): 0,1.0,0.1 for Split CIFAR and 0,0. 10.01 for Omniglot
For ER, we simply set the batch size to the same value standard training (10 and 32 for Split CIFAR and Omniglot respectively). Note that whenever applicable, we re-normalize the diagonal Fisher that the of its weights is equal to the number of parameters in the model. This is so that the hyper-parameter choice is less dependent on the size of the model. In particular this means that the magnitude of each diagonal element is much bigger, which is why do grid-search over smaller regularization parameters for EWC.
13


Reviews:
[{"review_id": "rkeTDi2TYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel regularization methods for continual learning. The authors introduce co-natural gradients, which is an incremental development of natural gradient methods, note that co-natural gradients use Fisher information to regularize the trajectory of the gradients which will be optimal on both tasks.\n\nThe method can be applied on existing regularization-based continual learning approach such as EWC, or ER. And with co-natural gradients, the model can perform better on continual learning problem.\n\nI think that the performance of finetuning + co-gradient is too natural, and not much meaningful. And the tradeoff between accuracy and forgetting for baselines is hard to compare on table 1 since the tradeoff depends on the hyperparameters. \n\nAnd it is required to apply on heterogeneous datasets to evaluate the performance when problems are really different. (like MNIST->CIFAR100->Omniglot->Imagenet, and so on). In the paper, the experiments are performed on very similar problems (split a single dataset).\n\nAlso, it would be great to show an illustration like figure 1, on real dataset, like split CIFAR.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyxd4ffhFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper puts forward a new regularization based continual learning method that explicitly regularizes the optimization trajectory by constraining in the distribution space. The paper is well written, and preliminary empirical results are promising. \n \n[Model]\n After surveying previous work, I am not sure if the paper is really novel: \n\nFirst of all, adding KL-based constrains to alleviate model forgetting has already been widely explored in prior arts, as the authors also acknowledged in the paper. The proposed regularizer has a close relationship with the EWC, especially with the EWC++. EWC++ encourages the KL-divergence between two distributions learned at successive tasks to be minimized, while the authors proposed to regularize the KL-divergence between two nearby updates, which leads to the well-known natural gradient descent.\n\nNatural gradient updates require to calculate the Fisher based on the current curvature, which is computationally expensive in practice. The authors further proposed to use a static Fisher estimated at the previous task for fast approximation. However, the approximation makes the algorithm not a natural gradient descent approach, nor a valid KL-regularized optimization problem. The theoretical implications of the static Fisher approximation are not discussed in the paper.\n \n[Experiments]\nThe authors may consider comparing with EWC++, which is a closely related baseline. \nThe experiment results have shown that the co-nature gradient method help from time to time. Understandably, the co-nature gradient-based optimization has the add-on benefit for any continual learning tasks. Still, it would be much better to see how it can help more state-of-the-art methods like LwM, LwF, etc, and especially on a few more challenging datasets.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJx1bKaTFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper amends the gradient update rule for continual learning using a natural-gradient-style formulation in order to regularise the trajectory during learning to forget previous task(s) less. They show experiments where this 'co-natural gradient' update rule improves some baselines. They also provide experiments showing the benefits of this update rule for low-resource finetuning settings.\n\nAlthough the idea seems reasonable and interesting, I feel like this paper needs work both in the theory and experiments. Figure 1 is a nice visualisation of the key take-away point of the paper.\n\nTheory: the authors take the natural-gradient updates from batch learning and just modify it so that the KL term is now for the previous task(s) instead of the current one. Although this may seem reasonable, I would appreciate some analysis as to what this implies or means.\n\nExperiments: \n- Split CIFAR from Chaudhry et al. (2018b) uses 10 tasks, why does this paper use 20 tasks? \n- Previous works usually find that for EWC, large values of the \\lambda hyperparameter provide best results. This corresponds to lower forgetting of previous tasks. The hyperparameter range in Appendix A.2.3 is only over small values of \\lambda (by orders of magnitude). \n- Why do the authors only allow 1 epoch per task for Split CIFAR? This probably results in early stopping: the new tasks are not able to reach their new optimal points (with or without regularised trajectories). This seems to go against the intuition provided by Figure 1, where the authors are showing that changing the trajectory results in a better local minimum being found. \nIn fact, by adding another regularisation term, it is unsurprising that co-natural gradient updates have less forgetting, as the extra regularisation term probably means the trained parameters are even closer to the previous parameters.\n\n-------------------\nEDIT: Score changed to 'Weak Accept' following discussion with the authors.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a 'co-natural' gradient update rule to precondition the optimization trajectory using a Fisher information estimate acquired from previous experience. This results in reduced sensitivity and forgetting when new tasks are learned. 

The reviews were mixed on this paper, and unfortunately not all reviewers had enough expertise in the field. After reading the paper carefully, I believe that the paper has significance and relevance to the field of continual learning, however it will benefit from more careful positioning with respect to other work as well as more empirical support. The application to the low-data-regime is interesting and could be expanded and refined in a future submission. 

The recommendation is for rejection.
Decision: Reject