Title: REGULARIZING TRAJECTORIES TO MITIGATE CATASTROPHIC FORGETTING

Abstract: Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters. In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the conatural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low-resource scenario. 1

1. INTRODUCTION:

It is good to have an end to journey toward; but it is the journey that matters, in the end.

Ursula K. Le Guin:

Endowing machine learning models with the capability to learn a variety of tasks in a sequential manner is critical to obtain agents that are both versatile and persistent. However, continual learning of multiple tasks is hampered by catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990) , the tendency of previously acquired knowledge to be overwritten when learning a new task.

Techniques to mitigate catastrophic forgetting can be roughly categorized into 3 lines of work (see Parisi et al. (2019) for a comprehensive overview): 1. regularization-based approaches, where forgetting is mitigated by the addition of a penalty term in the learning objective (Kirkpatrick et al. (2017) ; Chaudhry et al. (2018a) , inter alia), 2. dynamic architectures approaches, which incrementally increase the model's capacity to accomodate the new tasks (Rusu et al., 2016) , and 3. memorybased approaches, which retain data from learned tasks for later reuse (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b; 2019) . Among these, regularization-based approaches are particularly appealing because they do not increase the model size and do not require access to past data. This is particularly relevant to real-world scenarios where keeping data from previous training tasks may be impractical because of infrastructural or privacy-related reasons. Moreover, they are of independent intellectual interest because of their biological inspiration rooted in the idea of synaptic consolidation (Kirkpatrick et al., 2017) .

A good regularizer ensures that, when learning a new task, gradient descent will ultimately converge to parameters that yield good results on the new task while preserving performance on previously learned tasks. Critically, this is predicated upon successful optimization of the regularized objective, a fact that has been largely taken for granted in previous work. Non-convexity of the loss function, along with noise in the data (due to small or biased datasets) or in the gradients (due to stochastic gradient descent), can yield optimization trajectories -and ultimately convergence points -that are highly non-deterministic, even for the same starting parameters. As we demonstrate in this paper, this can cause unintended catastrophic forgetting along the optimization path. This is illustrated in a toy setting in Figure 1 : a two parameter model is trained to perform task T 2 (an arbitrary bi-modal loss function) after having learned task T 1 (a logistic regression task). Standard finetuning, even in Having learned task T 1 , the model is trained on T 2 with two different objectives: minimizing the loss on T 2 (Finetuning) and a regularized objective (EWC; Kirkpatrick et al. (2017) ). We add a small amount of Gaussian noise to gradients in order to simulate the stochasticity of the trajectory. Plain finetuning and EWC often converge to a solution with high loss for T 1 , but the co-natural optimization trajectory consistently converges towards the optimum with lowest loss for T 1 .

T 1 T 2 Finetuning EWC Co-natural finetuning

the presence of a regularized objective (EWC; Kirkpatrick et al. (2017) ), quickly changes the loss of T 1 and tends converge to a solution with high T 1 loss.

We propose to remedy this issue by regularizing the optimization trajectory itself, specifically by preconditioning gradient descent with the empirical Fisher information of previously learned tasks ( §3). This yields what we refer to as a co-natural gradient, an update rule inspired by the natural gradient (Amari, 1997) , but taking the Fisher information of previous tasks as a natural Riemannian metric2 of the parameter space, instead of the Fisher information of the task being optimized for. When we introduce our proposed co-natural gradient for the toy example of Figure 1 , the learning trajectory follows a path that changes the loss on T 1 much more slowly, and tends to converges to the optimum that incurs the lowest performance degradation on T 1 .

We test the validity of our approach in a continual learning scenario ( §4). We show that the co-natural gradient consistently reduces forgetting in a variety of existing continual learning approaches by a factor of ≈ 1.5 to 9, and greatly improves performance over simple finetuning, without modification to the training objective. We further investigate the special case of transfer learning in a two-task, low-resource scenario. In this specific case, control over the optimization trajectory is particularly useful because the optimizer has to rely on early stopping to prevent overfitting to the meager amount of training data in the target task. We show that the co-natural gradient yields the best trade-offs between source and target domain performance over a variety of hyper-parameters ( §5).

2. BACKGROUND AND NOTATIONS:

We first give a brief overview of the continual learning paradigm and existing approaches for overcoming catastrophic forgetting.

2.1. NOTATION:

Let us define a task as a triplet containing an input space X and an output space Y, both measurable spaces, as well as a distribution D over X × Y. In general, learning a task will consist of training a model to approximate the conditional distribution p(y | x) induced by D.

Consider a probabilistic model p θ parametrized by θ ∈ R d where d is the size of the model, trained to perform a source task S = X S , Y S , D S to some level of performance, yielding parameters θ S .

In the most simple instance of continual learning, we are tasked with learning a second target task T = X T , Y T , D T . In general in a multitask setting, it is not the case that the input or output spaces are the same. The discrepancy between input/output space can be addressed in various ways, e.g. by adding a minimal number of task-specific parameters (for example, different softmax layers for different label sets). To simplify exposition, we set these more specific considerations aside for now, and assume that X S = X T and Y S = Y T .

At any given point during training for task T , our objective will be to minimize the loss function L T (θ) -generally the expected log-likelihood E x,y∼D T [-log p θ (y | x)]. Typically, this will be performed by iteratively adding incremental update vectors δ ∈ R d to the parameters θ ←-θ + δ.

2.2. EXISTING APPROACHES FOR CONTINUAL LEARNING:

In this paper, we focus on those models that have a fixed architecture over the course of continual learning. The study of continual learning for models of fixed capacity can be split into two distinct (but often overlapping) streams of work:

Regularization-based approaches introduce a penalty in the loss function L T , typically quadratic, pushing the weights θ back towards θ S :

L T (θ) = E x,y∼D T -log p θ (y | x) NLL on task T + λ(θ -θ S ) T Ω S (θ -θ S )

Regularization term

(1)

where Ω S is a matrix, typically diagonal, that encodes the respective importance of each parameter with respect to task S, and λ is a regularization strength hyper-parameter. Various choices have been proposed for Ω S ; the diagonal empirical Fisher information matrix (Kirkpatrick et al., 2017) , or pathintegral based importance measures (Zenke et al., 2017; Chaudhry et al., 2018a) . More elaborate regularizers have been proposed based on e.g. a Bayesian formulation of continual learning (Nguyen et al., 2017; Ahn et al., 2019) or a distillation term (Li & Hoiem, 2016; Dhar et al., 2019) . The main advantage of these approaches is that they do not rely on having access to training data of previous tasks.

Memory-based approaches store data from previously seen tasks for re-use in continued learning, either as a form of constraint, by e.g. ensuring that training on the new task doesn't increase the loss on previous tasks (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b) , or for replay i.e. by retraining on instances from previous tasks (Rebuffi et al., 2017; Chaudhry et al., 2019; Aljundi et al., 2019b; a) . Various techniques have been proposed for the selection of samples to store in the memory (Chaudhry et al., 2019; Aljundi et al., 2019b) or for retrieval of the samples to be used for replay Aljundi et al. (2019a) .

All of these methods rely on stochastic gradient descent to optimize their regularized objective or to perform experience replay, with the notable exception of GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b) , where the gradients are projected onto the orthogonal complement of previous task's gradients. However, this method has been shown to perform poorly in comparison with simple replay (Chaudhry et al., 2019) , and it still necessitates access to data from previous tasks.

3. REGULARIZING THE TRAJECTORY:

After briefly recalling how the usual update is obtained in gradient descent, we derive a new, conatural update designed to better preserve the distribution induced by the model over previous tasks.

3.1. WARM UP: THE STANDARD GRADIENT DESCENT UPDATE:

At point θ in the parameter space, gradient descent finds the optimal update δ that is (1) small and (2) locally minimizes the decrease in loss L(θ + δ) -L(θ) (≈ δ ∇ θ L at the first order). Traditionally this can be formulated as minimizing the Lagrangian:

EQUATION

with Lagrangian multiplier µ > 0. Minimizing L for δ yields the well-known optimal update δ * :

δ * = - 1 2µ ∇ θ L T (3)

where 1 2µ corresponds to the learning rate (see Appendix A.1 for the full derivation).

3.2. KL REGULARIZATION OF TRAJECTORIES:

The δ 2 term in L implicitly expresses the underlying assumption that the best measure of distance between parameters θ and θ + δ is the Euclidean distance. In a continual learning setting however, the quantity we are most interested in preserving is the probability distribution that θ models on the source task S:

EQUATION

Therefore, a more natural distance between θ and θ + δ is the Kullback-Leibler divergence KL(p S θ p S θ+δ ) (Kullback & Leibler, 1951) . For preventing catastrophic forgetting along the optimization path, we incorporate incorporate this KL term into the Lagrangian L itself:

EQUATION

Doing so means that the optimization trajectory will tend to follow the direction that changes the distribution of the model the least. Notably, this is not a function of the previous objective L S , so knowledge of the original training objective is not necessary during continual learning (which is typically the case in path-integral based regularization methods (Zenke et al., 2017) or experience replay (Chaudhry et al., 2019) ).

3.3. CO-NATURAL GRADIENT OPTIMIZATION:

Presuming that δ is small, we can perform a second order Taylor approximation of the function δ → KL(p S θ p S θ+δ ) around 0. Considering that both the zeroeth and first order terms are null because 0 is a global minimizer of δ → KL(p S θ p S θ+δ ), this reduces the Lagrangian to a quadratic optimization problem (we refer the reader to Pascanu & Bengio (2013) for a more detailed derivation.):

L(δ) = δ ∇ θ L T + µ δ 2 + 1 2 νδ F S θ δ (6)

where F S θ is the Hessian of the KL divergence around θ. A crucial, well-known property of this matrix is that it coincides with the Fisher information matrix3 E x,y∼p θ [(∇ log p S θ )(∇ log p S θ ) T ] (the expectation being taken over the model's distribution p θ ; see Appendix A.1 for details). This is appealing from a computational perspective because the Fisher can be computed by means of first order derivatives only.

Minimizing for δ yields the following optimal update:

δ * = -λ F S θ + αI -1 ∇ θ L T (7)

where coefficients µ and ν are folded into two hyper-parameters: the learning rate λ and a damping coefficient α (the step-by-step derivation can be found in Appendix A.1). In practice, especially with low damping coefficients, it is common to obtain updates that are too large (typically when some parameters have no effect on the KL divergence). To address this, we re-normalize δ * to have the same norm as the original gradient, ∇L T .

For computational reasons, we will make 3 key practical approximations to the Fisher:

1. F S θ ≈ F S θ S : we maintain the Fisher computed at θ S , instead of recomputing F S at every step of training. This relieves us of the computational burden of updating the Fisher for every new value of θ. This approximation (shared by previous work, e.g. Kirkpatrick et al. (2017) ; Chaudhry et al. (2018a) ) is only valid insofar as θ S and θ are close. Empirically we observe that this still leads to good results.

2. F S is diagonal: this is a common approximation in practice with two appealing properties.

First, this makes it possible to store the d diagonal Fisher coefficients in memory. Second, this trivializes the inverse operation (simply invert the diagonal elements).

3. Empirical Fisher: this common approximation replaces the expectation under the model's distribution by the expected log-likelihood of the true distribution: E x,y∼p S [(∇ log p S θ )(∇ log p S θ ) T ] (mind the subscript). This is particularly useful in tasks with a large or unbounded number of classes (e.g. structured prediction), where summing over all possible outputs is intractable. We can then compute the diagonal of the empirical Fisher using Monte Carlo sampling:

1 N N i=1 [∇ log p S θ (y i | x i )]

2 with (x i , y i ) sampled from D S (we use N = 1000 for all experiments). This formulation bears many similarities with the natural gradient from Amari (1997) , which also uses the KL divergence as a metric for choosing the optimal update δ * . There is a however a crucial difference, both in execution and purpose: where the natural gradient uses knowledge of the curvature of the KL divergence of D T to speed-up convergence, our proposed method leverages the curvature of the KL divergence on D S to slow-down divergence from p S θ S . To highlight the resemblance and complementarity between these two concepts, we refer to the new update as the co-natural gradient.

3.4. BEYOND TWO TASKS:

In a continual learning scenario, we are confronted with a large number of tasks T 1 . . . T n presented in sequential order. When learning T n , we can change the Lagrangian L from 5 to incorporate the constraints for all previous tasks T 1 . . . T n-1 :

EQUATION

This in turn changes the Fisher in Eq. 8 to Fn-1 := 1 2 n-1 i=1 ν i F Ti . The choice of the coefficients ν i is crucial. Setting all ν i to the same value, i.e. assigning the same importance to all tasks is suboptimal for a few reasons. First and foremost, it is unreasonable to expect of a model with finite capacity to remember an unbounded number of tasks (as tasks "fill-up" the model capacity, Fn-1 is likely to become more "homogeneous"). Second, as training progresses and θ changes, our

approximation that F Ti θ ≈ F Ti θ T i

is less and less likely to hold.

We address this issue in the same fashion as Schwarz et al. (2018) , by keeping a rolling exponential average of the Fisher matrices:

EQUATION

In this case, previous tasks are gracefully forgotten at an exponential rate controlled by γ. We account for the damping α term in Eq. 7 by setting F0 := α γ I. In preliminary experiments, we have found γ = 0.9 to yield consistently good results, and use this value in all presented experiments.

4. CONTINUAL LEARNING EXPERIMENTS:



4.1. EXPERIMENTAL SETTING:

To corroborate our hypothesis that controlling the optimization trajectory with the co-natural gradient reduces catastrophic forgetting, we perform experiments on two continual learning testbeds:

• Split CIFAR: The CIFAR100 dataset, split into 20 independent 5-way classification tasks.

Similarly to Chaudhry et al. (2018b) , we use a smaller version of the ResNet architecture (He et al., 2016) .

Table 1 : Average accuracies and forgetting after all tasks have been learnt, with and without the co-natural gradient. Results are reported in percentages (± the standard deviation over 5 re-runs). Bold print indicates statistically significant difference between standard and co-natural (p < 0.05).

Split CIFAR Omniglot:

Finetuning • Omniglot: the Omniglot dataset (Lake et al., 2015) consists of 50 independent character recognition datasets on different alphabet. We adopt the setting of Schwarz et al. (2018) and consider each alphabet as a separate task. 4 On this dataset we use the same small CNN architecture as Schwarz et al. (2018) .

• Split MiniImageNet: The MiniImageNet dataset (a subset of the popular ImageNet (Deng et al., 2009) dataset5 ; Vinyals et al. (2016) ). Split the dataset into 20 independent 5-way classification tasks, similarly to Split CIFAR, and use the same smaller ResNet.

We adopt the experimental setup from Chaudhry et al. (2019) : in each dataset we create a "validation set" of 3 tasks, used to select the best hyper-parameters, and keep the remaining tasks for evaluation. This split is chosen at random and kept the same across all experiments. In these datasets, the nature and possibly the number of classes for each task changes. We account for this by training a separate softmax layer for each task, and apply continual learning only to the remaining, "feature-extraction" part of the model.

We report results along two common metrics for continual learning: average accuracy, the accuracy at the end of training averaged over all tasks, and forgetting. Forgetting is defined in Chaudhry et al. (2018a) as the difference in performance on a task between the current model and the best performing model on this task. Formally if A T t represents the accuracy on task T at step t of training, the forgetting F T t at step t is defined as

F T t = max τ <t A T τ -A T t .

'Low forgetting' means that the model tend to keep the same level of performance on a task it has learned.

We implement the co-natural update rule on top of 3 baselines:

• Finetuning: Simply train the model on the task at hand, without any form of regularization.

• EWC: Proposed by Kirkpatrick et al. (2017) , it is a simple but effective quadratic regularization approach. While neither the most recent nor sophisticate regularization technique, it is a natural baseline for us to compare to in that it also consists in a Fisher-based penalty -although in the 0 10 20 30 40 • ER: Experience replay with a fixed sized episodic memory proposed by Chaudhry et al. (2019) .

While not directly comparable to EWC in that it presupposes access to data from previous tasks, ER is a simple approach that boasts the best performances on a variety of benchmarks (Chaudhry et al., 2019) . In all experiments, we use memory size 1,000 with reservoir sampling.

Training proceeds as follows: we perform exhaustive search on all the hyper-parameter combinations using the validation tasks. Every combination is reran 3 times (the order of tasks, model initialization and order of training examples changes with each restart), and rated by accuracy averaged over tasks and restarts. We then evaluate the best hyper-parameters by continual training on the evaluation tasks. Results are reported over 5 random restarts (3 for MiniImageNet), and we control for statistical significance using a paired t-test (we pair together runs with the same task ordering). We refer to Appendix A.2 for more details regarding fine-grained design choices.

4.2. RESULTS:

The upper half of Table 1 reports the average accuracy of all the tasks at the end of training (higher is better). We observe that the co-natural gradient always improves greatly over simple finetuning, and occasionally over EWC and ER. We note that on both datasets, bare-bone co-natural finetuning matches or exceeds the performance of EWC and ER even though it requires strictly fewer resources (no need to store the previous parameters as in EWC, or data in ER).

Even more appreciable is the effect of the co-natural trajectories on forgetting, as shown in the lower half of Table 1 . As evidenced by the results in the lowest row, using the co-natural gradient systematically results in large drops in forgetting across all approaches and both datasets, even when the average accuracy is not increased.

To get a qualitative assessment of the learning trajectories that yield such results, we visualize the accuracy curves of 10 out of the 47 evaluation tasks of Omniglot in Figure 2 . We observe that previous approaches do poorly at keeping stable levels of performance over a long period of time (especially for tasks learned early in training), a problem that is largely resolved by the co-natural preconditioning. This seems to come at the cost of more intransigence (Chaudhry et al., 2018a) , i.e. some of the later tasks are not being learnt properly. In models of fixed capacity, there is a natural trade-off between intransigence and forgetting (see also the "stability-plasticity" dilemma in neuroscience Grossberg (1982)). Our results position the co-natural gradient as a strong lowforgetting/moderate intransigence basis for future work.

5. LOW-RESOURCE ADAPTATION EXPERIMENTS:

In this section we take a closer look at the specific case of adapting a model from a single task to another, when we only have access to a minimal amount of data in the target task. In this case, controlling the learning trajectory is particularly important because the model is being trained on an unreliable sample of the true distribution of the target task, and we have to rely on early-stopping to prevent overfitting. We show that using the co-natural gradient during adaptation helps both at preserving source task performance and reach higher overall target task performance.

We perform experiments on two different scenarios: (Papineni et al., 2002) Here we do not allow any access to data in the source task when training on the target task. We compare four methods Finetuning (our baseline), Co-natural finetuning, EWC (which has been proven effective for domain adaptation, see Thompson et al. (2019) ) and Co-natural EWC.

Given that different methods might lead to different trade-offs between source and target task performance, with some variation depending on the hyper-parameters (e.g. learning rate, regularization strength. . . ), we take inspiration from Thompson et al. (2019) and graphically report results for all hyper-parameter configuration of each method on the 2 dimensional space defined by the score on source and target tasks7 . Additionally, we highlight the Pareto frontier of each method i.e. the set of configurations that are not strictly worse than any other configuration for the same model.

The adaptation results for both scenarios are reported in Figure 3 . We find that in both cases, the co-natural gradient not only helps preserving the source task performance, but to some extent it also allows the model to reach better performance on the target task as well. We take this to corroborate our starting hypothesis: while introducing a regularizer does help, controlling the optimization dynamics actively helps counteract overfitting to the very small amount of training data, because the co-natural pre-conditioning makes it harder for stochastic gradient descent to push the model towards directions that would also hurt the source task.

6. CONCLUSION:

We have presented the co-natural gradient, a technique that regularizes the optimization trajectory of models trained in a continual setting. We have shown that the co-natural gradient stands on its own as an efficient approach for overcoming catastrophic forgetting, and that it effectively complements and stabilizes other existing techniques at a minimal cost. We believe that the co-natural gradientand more generally, trajectory regularization -can serve as a solid bedrock for building agents that learn without forgetting.

A APPENDIX:

A.1 DERIVATIONS A.1.1 THE STANDARD GRADIENT UPDATE (EQUATION 3)

We derive the standard update in Eq. 3 by solving the Lagrangian L in Eq. 2 for δ. Given that its first and second derivatives are:

∇L = ∇L T + 2µδ ∇ 2 L = 2µI

the problem is trivially strictly convex and its global minimizer δ * satisfies: 

∇L δ * =

∇ 2 KL(p θ p x ) = ∇ 2 E p θ [log p θ ] =0 -∇ 2 E p θ [log p x ] = -E p θ [∇ 2 log p x ]

Now note that ∇ 2 log p x can be rewritten via standard derivatives manipulations as ∇ 2 px px -(∇px)(∇px) p 2

x . This leads to:

∇ 2 KL(p θ p x ) = -E p θ ∇ 2 p x p x + E p θ ∇p x p x ∇p x p 2

x When taken at θ, the first term evaluates to 9 :

E p θ ∇ 2 p x p x x=θ = p θ (z) ∇ 2 p θ (z) p θ (z) dz = ∇ 2 p θ (z)dz = ∇ 2 p θ (z)dz =1 = 0

By using the identity ∇px px = ∇ log p x and evaluating at x = θ, the second term gives us:

∇ 2 KL(p θ p x ) x=θ = E p θ [(∇ log p θ )(∇ log p θ ) ]

8 Essentially allowing us to interchange derivatives and integrals. 9 We abuse notation and write ∇px x=θ as ∇p θ

:

We commit to releasing the code to implement our method and reproduce our experiments upon acceptance.

Informally, the reader can think of a Riemannian metric as a function that assigns an inner product u, v → gx(u, v) to each point x in the space, thus inducing a localized notion of distance and curvature.

Hence our use of the letter F to designate the Hessian

Note that this is a different setting than the usual meta-learning scenario that Omniglot is used for.

Similarly to Omniglot, MiniImageNet was originally intended as a meta-learning benchmark and therefore its standard train/validation/test split consists of disjoint classes. We perform a custom transversal split so that the dataset can be used as a standard 100-way classification task. The accuracies reported here are not to be compared with the meta-learning literature.

We use sacrebleu(Post, 2018) with -tok intl as recommended byMichel & Neubig (2018).

For CUB in particular we report the average accuracy of every configuration over 5 runs, each with a different 200-sized random subset of the data.

Figures and Tables:

Figure1: On the importance of trajectories: an example with 2-dimensional logistic regression. Having learned task T 1 , the model is trained on T 2 with two different objectives: minimizing the loss on T 2 (Finetuning) and a regularized objective (EWC;Kirkpatrick et al. (2017)). We add a small amount of Gaussian noise to gradients in order to simulate the stochasticity of the trajectory. Plain finetuning and EWC often converge to a solution with high loss for T 1 , but the co-natural optimization trajectory consistently converges towards the optimum with lowest loss for T 1 .

Figure2: Evolution of task performance over the course of continual learning on one ordering of Omniglot. For visibility we only show accuracies for every fifth task. The rectangular shaded regions delineate the period during which each task is being trained upon; with the exception of ER, this is the only period the model has access to the data for this task.

0 ⇐⇒ δ * = -1 2µ ∇L T A.1.2 EQUIVALENCE OF THE HESSIAN OF THE KL DIVERGENCE AND THE FISHER INFORMATION MATRIX To simplify notation, let us perform the change of variables θ + δ → x. We show that the Hessian of the KL coincides with the Fisher on θ: in other words, ∇ 2 KL(p θ p x ) x=θ = E p θ [(∇ log p θ )(∇ log p θ ) ]. Under mild regularity assumptions 8 , we can write:

Standard 35.92 ±1.19 44.86 ±2.01 61.08 ±0.94 16.31 ±1.05 70.31 ±3.46 70.90 ±0.80 Co-natural 56.82 ±1.47 56.50 ±1.28 59.34 ±2.02 71.25 ±4.90 69.11 ±4.70 75.48 ±1.92 Forgetting ↓ Standard 34.05 ±0.99 17.50 ±2.09 10.66 ±0.70 77.26 ±1.11 14.16 ±2.35 22.43 ±0.80 Co-natural 5.53 ±1.12 4.91 ±0.92 5.25 ±1.20 8.49 ±2.75 8.73 ±1.81 5.21 ±0.30 Continual-learning results for Split MiniImageNet (see Figure 1 for details). Standard 36.86 ±2.10 58.43 ±1.73 63.10 ±4.04 Co-natural 63.14 ±2.57 62.90 ±1.61 70.59 ±0.25 Forgetting ↓ Standard 40.51 ±1.99 12.47 ±2.42 15.29 ±3.97 Co-natural 11.06 ±3.32 8.90 ±1.91 8.12 ±1.25

Table content: <table><tr><td>EWC</td><td>ER</td><td>Finetuning</td><td>EWC</td><td>ER</td></tr><tr><td colspan="3">Average accuracy ↑</td><td/><td/></tr><tr><td colspan="3">Split MiniImageNet</td><td/><td/></tr><tr><td>Finetuning</td><td/><td>EWC</td><td>ER</td><td/></tr><tr><td colspan="3">Average accuracy ↑</td><td/><td/></tr></table>

Low-resource adaptation results. The source (resp. target) task performance is represented on the vertical (resp. horizontal) axis. Pareto optimal configurations for each method are highlighted and the frontier is represented with dashed lines. The solid gray lines indicate the score of the original model trained on the source task.

Table content: <table><tr><td/><td>90.0 92.5</td><td>Finetuning Co-natural EWC Co-natural finetuning EWC Base model score</td><td/><td>38.8 39.0 39.2</td></tr><tr><td>MiniImagenet accuracy</td><td>75.0 77.5 80.0 82.5 85.0 87.5</td><td>0 2 4 6 8 10 12 14 16 18 CUB accuracy</td><td>newstest2014 BLEU</td><td>MTNT BLEU 30.8 31.0 31.2 31.4 31.6 31.8 32.0 32.2 32.4 37.6 37.8 38.2 38.4 38.6 38.0 Base model score Finetuning Co-natural finetuning EWC Co-natural EWC</td></tr><tr><td/><td colspan="2">(a) MiniImageNet to CUB adaptation</td><td/><td>(b) WMT to MTNT fine-tuning</td></tr><tr><td colspan="3">Figure 3:</td><td/><td/></tr></table>

Image classificationWe take MiniImagenet as a source task and CUB (a 200-way birds species classification dataset;Welinder et al. (2010)) as a target task. To guarantee a strong base model despite the small size of MiniImageNet, we start off from a ResNet18 model(He et al., 2016) pretrained on the full ImageNet, which we retrofit to MiniImageNet by replacing the last fully connected layer with a separate linear layer regressed over the MiniImageNet training data. To simulate a low-resource setting, we sub-sample the CUB training set to 200 images (≈ 1 per class). Scores for these tasks are reported in terms of accuracy.Machine translationWe consider adaptation of an English to French model trained on WMT15 (a dataset of parallel sentences crawled from parliamentary proceedings, news commentary and web page crawls;Bojar et al. (2015)) to MTNT (a dataset of Reddit comments;Michel & Neubig (2018)). Our model is a Transformer(Vaswani et al., 2017) pretrained on WMT15. Similarly to CUB, we simulate a low-resource setting by taking a sub-sample of 1000 sentence pairs as a training set. Scores for these two datasets are reported in terms of BLEU score.6

Table content: <table/>

Peer review (iclr2020):
The paper puts forward a new regularization based continual learning method that explicitly regularizes the optimization trajectory by constraining in the distribution space. The paper is well written, and preliminary empirical results are promising. [Model] After surveying previous work, I am not sure if the paper is really novel: First of all, adding KL-based constrains to alleviate model forgetting has already been widely explored in prior arts, as the authors also acknowledged in the paper. The proposed regularizer has a close relationship with the EWC, especially with the EWC++. EWC++ encourages the KL-divergence between two distributions learned at successive tasks to be minimized, while the authors proposed to regularize the KL-divergence between two nearby updates, which leads to the well-known natural gradient descent. Natural gradient updates require to calculate the Fisher based on the current curvature, which is computationally expensive in practice. The authors further proposed to use a static Fisher estimated at the previous task for fast approximation. However, the approximation makes the algorithm not a natural gradient descent approach, nor a valid KL-regularized optimization problem. The theoretical implications of the static Fisher approximation are not discussed in the paper. [Experiments] The authors may consider comparing with EWC++, which is a closely related baseline. The experiment results have shown that the co-nature gradient method help from time to time. Understandably, the co-nature gradient-based optimization has the add-on benefit for any continual learning tasks. Still, it would be much better to see how it can help more state-of-the-art methods like LwM, LwF, etc, and especially on a few more challenging datasets.