Year: 2019

Abstract:
Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters. In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the conatural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low-resource scenario.1

text: 1 INTRODUCTION
It is good to have an end to journey toward; but it is the journey that matters, in the end.
Ursula K. Le Guin
Endowing machine learning models with the capability to learn a variety of tasks in a sequential manner is critical to obtain agents that are both versatile and persistent. However, continual learning of multiple tasks is hampered by catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990), the tendency of previously acquired knowledge to be overwritten when learning a new task.
Techniques to mitigate catastrophic forgetting can be roughly categorized into 3 lines of work (see Parisi et al. (2019) for a comprehensive overview): 1. regularization-based approaches, where forgetting is mitigated by the addition of a penalty term in the learning objective (Kirkpatrick et al. (2017); Chaudhry et al. (2018a), inter alia), 2. dynamic architectures approaches, which incrementally increase the model’s capacity to accomodate the new tasks (Rusu et al., 2016), and 3. memorybased approaches, which retain data from learned tasks for later reuse (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b; 2019). Among these, regularization-based approaches are particularly appealing because they do not increase the model size and do not require access to past data. This is particularly relevant to real-world scenarios where keeping data from previous training tasks may be impractical because of infrastructural or privacy-related reasons. Moreover, they are of independent intellectual interest because of their biological inspiration rooted in the idea of synaptic consolidation (Kirkpatrick et al., 2017).
A good regularizer ensures that, when learning a new task, gradient descent will ultimately converge to parameters that yield good results on the new task while preserving performance on previously learned tasks. Critically, this is predicated upon successful optimization of the regularized objective, a fact that has been largely taken for granted in previous work. Non-convexity of the loss function, along with noise in the data (due to small or biased datasets) or in the gradients (due to stochastic gradient descent), can yield optimization trajectories — and ultimately convergence points — that are highly non-deterministic, even for the same starting parameters. As we demonstrate in this paper, this can cause unintended catastrophic forgetting along the optimization path. This is illustrated in a toy setting in Figure 1: a two parameter model is trained to perform task T2 (an arbitrary bi-modal loss function) after having learned task T1 (a logistic regression task). Standard finetuning, even in
1We commit to releasing the code to implement our method and reproduce our experiments upon acceptance.
the presence of a regularized objective (EWC; Kirkpatrick et al. (2017)), quickly changes the loss of T1 and tends converge to a solution with high T1 loss.
We propose to remedy this issue by regularizing the optimization trajectory itself, specifically by preconditioning gradient descent with the empirical Fisher information of previously learned tasks (§3). This yields what we refer to as a co-natural gradient, an update rule inspired by the natural gradient (Amari, 1997), but taking the Fisher information of previous tasks as a natural Riemannian metric2 of the parameter space, instead of the Fisher information of the task being optimized for. When we introduce our proposed co-natural gradient for the toy example of Figure 1, the learning trajectory follows a path that changes the loss on T1 much more slowly, and tends to converges to the optimum that incurs the lowest performance degradation on T1.
We test the validity of our approach in a continual learning scenario (§4). We show that the co-natural gradient consistently reduces forgetting in a variety of existing continual learning approaches by a factor of≈ 1.5 to 9, and greatly improves performance over simple finetuning, without modification to the training objective. We further investigate the special case of transfer learning in a two-task, low-resource scenario. In this specific case, control over the optimization trajectory is particularly useful because the optimizer has to rely on early stopping to prevent overfitting to the meager amount of training data in the target task. We show that the co-natural gradient yields the best trade-offs between source and target domain performance over a variety of hyper-parameters (§5).

heading: 2 BACKGROUND AND NOTATIONS
text: We first give a brief overview of the continual learning paradigm and existing approaches for overcoming catastrophic forgetting.

heading: 2.1 NOTATION
text: Let us define a task as a triplet containing an input space X and an output space Y , both measurable spaces, as well as a distribution D over X × Y . In general, learning a task will consist of training a model to approximate the conditional distribution p(y | x) induced by D. Consider a probabilistic model pθ parametrized by θ ∈ Rd where d is the size of the model, trained to perform a source task S = 〈XS ,YS ,DS〉 to some level of performance, yielding parameters θS . In the most simple instance of continual learning, we are tasked with learning a second target task T = 〈XT ,YT ,DT 〉. In general in a multitask setting, it is not the case that the input or output spaces are the same. The discrepancy between input/output space can be addressed in various ways, e.g. by adding a minimal number of task-specific parameters (for example, different softmax layers for different label sets). To simplify exposition, we set these more specific considerations aside for now, and assume that XS = XT and YS = YT . 2Informally, the reader can think of a Riemannian metric as a function that assigns an inner product u, v 7→ gx(u, v) to each point x in the space, thus inducing a localized notion of distance and curvature.
At any given point during training for task T , our objective will be to minimize the loss function LT (θ) – generally the expected log-likelihood Ex,y∼DT [− log pθ(y | x)]. Typically, this will be performed by iteratively adding incremental update vectors δ ∈ Rd to the parameters θ ←− θ + δ.

heading: 2.2 EXISTING APPROACHES FOR CONTINUAL LEARNING
text: In this paper, we focus on those models that have a fixed architecture over the course of continual learning. The study of continual learning for models of fixed capacity can be split into two distinct (but often overlapping) streams of work:
Regularization-based approaches introduce a penalty in the loss function LT , typically quadratic, pushing the weights θ back towards θS :
LT (θ) = Ex,y∼DT − log pθ(y | x)︸ ︷︷ ︸ NLL on task T +λ(θ − θS)TΩS(θ − θS)︸ ︷︷ ︸ Regularization term
(1)
where ΩS is a matrix, typically diagonal, that encodes the respective importance of each parameter with respect to task S, and λ is a regularization strength hyper-parameter. Various choices have been proposed for ΩS ; the diagonal empirical Fisher information matrix (Kirkpatrick et al., 2017), or pathintegral based importance measures (Zenke et al., 2017; Chaudhry et al., 2018a). More elaborate regularizers have been proposed based on e.g. a Bayesian formulation of continual learning (Nguyen et al., 2017; Ahn et al., 2019) or a distillation term (Li & Hoiem, 2016; Dhar et al., 2019). The main advantage of these approaches is that they do not rely on having access to training data of previous tasks.
Memory-based approaches store data from previously seen tasks for re-use in continued learning, either as a form of constraint, by e.g. ensuring that training on the new task doesn’t increase the loss on previous tasks (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b), or for replay i.e. by retraining on instances from previous tasks (Rebuffi et al., 2017; Chaudhry et al., 2019; Aljundi et al., 2019b;a). Various techniques have been proposed for the selection of samples to store in the memory (Chaudhry et al., 2019; Aljundi et al., 2019b) or for retrieval of the samples to be used for replay Aljundi et al. (2019a).
All of these methods rely on stochastic gradient descent to optimize their regularized objective or to perform experience replay, with the notable exception of GEM (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b), where the gradients are projected onto the orthogonal complement of previous task’s gradients. However, this method has been shown to perform poorly in comparison with simple replay (Chaudhry et al., 2019), and it still necessitates access to data from previous tasks.

heading: 3 REGULARIZING THE TRAJECTORY
text: After briefly recalling how the usual update is obtained in gradient descent, we derive a new, conatural update designed to better preserve the distribution induced by the model over previous tasks.

heading: 3.1 WARM UP: THE STANDARD GRADIENT DESCENT UPDATE
text: At point θ in the parameter space, gradient descent finds the optimal update δ that is (1) small and (2) locally minimizes the decrease in loss L(θ + δ)− L(θ) (≈ δᵀ∇θL at the first order). Traditionally this can be formulated as minimizing the Lagrangian:
L(δ) = δᵀ∇θLT︸ ︷︷ ︸ first order
loss minimization term
+ µ‖δ‖2︸ ︷︷ ︸ “small update” term
(2)
with Lagrangian multiplier µ > 0. Minimizing L for δ yields the well-known optimal update δ∗:
δ∗ = − 1 2µ ∇θLT (3)
where 12µ corresponds to the learning rate (see Appendix A.1 for the full derivation).

heading: 3.2 KL REGULARIZATION OF TRAJECTORIES
text: The ‖δ‖2 term in L implicitly expresses the underlying assumption that the best measure of distance between parameters θ and θ + δ is the Euclidean distance. In a continual learning setting however, the quantity we are most interested in preserving is the probability distribution that θ models on the source task S:
pSθ (x, y) = pθ(y | x)pS(x) (4)
Therefore, a more natural distance between θ and θ + δ is the Kullback-Leibler divergence KL(pSθ ‖pSθ+δ) (Kullback & Leibler, 1951). For preventing catastrophic forgetting along the optimization path, we incorporate incorporate this KL term into the Lagrangian L itself:
L(δ) = δᵀ∇θLT + µ‖δ‖2 + νKL(pSθ ‖pSθ+δ) (5)
Doing so means that the optimization trajectory will tend to follow the direction that changes the distribution of the model the least. Notably, this is not a function of the previous objective LS , so knowledge of the original training objective is not necessary during continual learning (which is typically the case in path-integral based regularization methods (Zenke et al., 2017) or experience replay (Chaudhry et al., 2019)).

heading: 3.3 CO-NATURAL GRADIENT OPTIMIZATION
text: Presuming that δ is small, we can perform a second order Taylor approximation of the function δ 7→ KL(pSθ ‖pSθ+δ) around 0. Considering that both the zeroeth and first order terms are null because 0 is a global minimizer of δ 7→ KL(pSθ ‖pSθ+δ), this reduces the Lagrangian to a quadratic optimization problem (we refer the reader to Pascanu & Bengio (2013) for a more detailed derivation.):
L(δ) = δᵀ∇θLT + µ‖δ‖2 + 1
2 νδᵀFSθ δ (6)
where FSθ is the Hessian of the KL divergence around θ. A crucial, well-known property of this matrix is that it coincides with the Fisher information matrix3 Ex,y∼pθ [(∇ log pSθ )(∇ log pSθ )T ] (the expectation being taken over the model’s distribution pθ; see Appendix A.1 for details). This is appealing from a computational perspective because the Fisher can be computed by means of first order derivatives only.
Minimizing for δ yields the following optimal update:
δ∗ = −λ [ FSθ + αI ]−1∇θLT (7) where coefficients µ and ν are folded into two hyper-parameters: the learning rate λ and a damping coefficient α (the step-by-step derivation can be found in Appendix A.1). In practice, especially with low damping coefficients, it is common to obtain updates that are too large (typically when some parameters have no effect on the KL divergence). To address this, we re-normalize δ∗ to have the same norm as the original gradient, ‖∇LT ‖. For computational reasons, we will make 3 key practical approximations to the Fisher:
1. FSθ ≈ FSθS : we maintain the Fisher computed at θS , instead of recomputing FS at every step of training. This relieves us of the computational burden of updating the Fisher for every new value of θ. This approximation (shared by previous work, e.g. Kirkpatrick et al. (2017);Chaudhry et al. (2018a)) is only valid insofar as θS and θ are close. Empirically we observe that this still leads to good results.
3Hence our use of the letter F to designate the Hessian
2. FS is diagonal: this is a common approximation in practice with two appealing properties. First, this makes it possible to store the d diagonal Fisher coefficients in memory. Second, this trivializes the inverse operation (simply invert the diagonal elements).
3. Empirical Fisher: this common approximation replaces the expectation under the model’s distribution by the expected log-likelihood of the true distribution: Ex,y∼pS [(∇ log pSθ )(∇ log pSθ )T ] (mind the subscript). This is particularly useful in tasks with a large or unbounded number of classes (e.g. structured prediction), where summing over all possible outputs is intractable. We can then compute the diagonal of the empirical Fisher using Monte Carlo sampling: 1N ∑N i=1[∇ log pSθ (yi | xi)]2 with (xi, yi) sampled from
DS (we use N = 1000 for all experiments).
This formulation bears many similarities with the natural gradient from Amari (1997), which also uses the KL divergence as a metric for choosing the optimal update δ∗. There is a however a crucial difference, both in execution and purpose: where the natural gradient uses knowledge of the curvature of the KL divergence of DT to speed-up convergence, our proposed method leverages the curvature of the KL divergence on DS to slow-down divergence from pSθS . To highlight the resemblance and complementarity between these two concepts, we refer to the new update as the co-natural gradient.

heading: 3.4 BEYOND TWO TASKS
text: In a continual learning scenario, we are confronted with a large number of tasks T1 . . . Tn presented in sequential order. When learning Tn, we can change the Lagrangian L from 5 to incorporate the constraints for all previous tasks T1 . . . Tn−1:
L(δ) = δᵀ∇θLTn + µ‖δ‖2 + n−1∑ i=1 νiKL(pTiθ ‖p Ti θ+δ) (8)
This in turn changes the Fisher in Eq. 8 to F̃n−1 := 12 ∑n−1 i=1 νiF
Ti . The choice of the coefficients νi is crucial. Setting all νi to the same value, i.e. assigning the same importance to all tasks is suboptimal for a few reasons. First and foremost, it is unreasonable to expect of a model with finite capacity to remember an unbounded number of tasks (as tasks “fill-up” the model capacity, F̃n−1 is likely to become more “homogeneous”). Second, as training progresses and θ changes, our approximation that FTiθ ≈ F Ti θTi is less and less likely to hold.
We address this issue in the same fashion as Schwarz et al. (2018), by keeping a rolling exponential average of the Fisher matrices:
F̃ γn = γFTn + (1− γ)F̃ γ n−1 (9)
In this case, previous tasks are gracefully forgotten at an exponential rate controlled by γ. We account for the damping α term in Eq. 7 by setting F̃0 := αγ I . In preliminary experiments, we have found γ = 0.9 to yield consistently good results, and use this value in all presented experiments.

heading: 4 CONTINUAL LEARNING EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETTING
text: To corroborate our hypothesis that controlling the optimization trajectory with the co-natural gradient reduces catastrophic forgetting, we perform experiments on two continual learning testbeds:
• Split CIFAR: The CIFAR100 dataset, split into 20 independent 5-way classification tasks. Similarly to Chaudhry et al. (2018b), we use a smaller version of the ResNet architecture (He et al., 2016).
We adopt the experimental setup from Chaudhry et al. (2019): in each dataset we create a “validation set” of 3 tasks, used to select the best hyper-parameters, and keep the remaining tasks for evaluation. This split is chosen at random and kept the same across all experiments. In these datasets, the nature and possibly the number of classes for each task changes. We account for this by training a separate softmax layer for each task, and apply continual learning only to the remaining, “feature-extraction” part of the model.
We report results along two common metrics for continual learning: average accuracy, the accuracy at the end of training averaged over all tasks, and forgetting. Forgetting is defined in Chaudhry et al. (2018a) as the difference in performance on a task between the current model and the best performing model on this task. Formally if ATt represents the accuracy on task T at step t of training, the forgetting FTt at step t is defined as F T t = maxτ<tA T τ −ATt . ‘Low forgetting’ means that the model tend to keep the same level of performance on a task it has learned.
We implement the co-natural update rule on top of 3 baselines:
• Finetuning: Simply train the model on the task at hand, without any form of regularization. • EWC: Proposed by Kirkpatrick et al. (2017), it is a simple but effective quadratic regularization
approach. While neither the most recent nor sophisticate regularization technique, it is a natural baseline for us to compare to in that it also consists in a Fisher-based penalty — although in the
4Note that this is a different setting than the usual meta-learning scenario that Omniglot is used for. 5Similarly to Omniglot, MiniImageNet was originally intended as a meta-learning benchmark and therefore its standard train/validation/test split consists of disjoint classes. We perform a custom transversal split so that the dataset can be used as a standard 100-way classification task. The accuracies reported here are not to be compared with the meta-learning literature.
loss function instead of the optimization dynamics. We also use the rolling Fisher described in Section 3.4, making our EWC baseline equivalent to the superior online EWC introduced by Schwarz et al. (2018).
• ER: Experience replay with a fixed sized episodic memory proposed by Chaudhry et al. (2019). While not directly comparable to EWC in that it presupposes access to data from previous tasks, ER is a simple approach that boasts the best performances on a variety of benchmarks (Chaudhry et al., 2019). In all experiments, we use memory size 1,000 with reservoir sampling.
Training proceeds as follows: we perform exhaustive search on all the hyper-parameter combinations using the validation tasks. Every combination is reran 3 times (the order of tasks, model initialization and order of training examples changes with each restart), and rated by accuracy averaged over tasks and restarts. We then evaluate the best hyper-parameters by continual training on the evaluation tasks. Results are reported over 5 random restarts (3 for MiniImageNet), and we control for statistical significance using a paired t-test (we pair together runs with the same task ordering). We refer to Appendix A.2 for more details regarding fine-grained design choices.

heading: 4.2 RESULTS
text: The upper half of Table 1 reports the average accuracy of all the tasks at the end of training (higher is better). We observe that the co-natural gradient always improves greatly over simple finetuning, and occasionally over EWC and ER. We note that on both datasets, bare-bone co-natural finetuning matches or exceeds the performance of EWC and ER even though it requires strictly fewer resources (no need to store the previous parameters as in EWC, or data in ER).
Even more appreciable is the effect of the co-natural trajectories on forgetting, as shown in the lower half of Table 1. As evidenced by the results in the lowest row, using the co-natural gradient systematically results in large drops in forgetting across all approaches and both datasets, even when the average accuracy is not increased.
To get a qualitative assessment of the learning trajectories that yield such results, we visualize the accuracy curves of 10 out of the 47 evaluation tasks of Omniglot in Figure 2. We observe that previous approaches do poorly at keeping stable levels of performance over a long period of time (especially for tasks learned early in training), a problem that is largely resolved by the co-natural preconditioning. This seems to come at the cost of more intransigence (Chaudhry et al., 2018a), i.e. some of the later tasks are not being learnt properly. In models of fixed capacity, there is a natural trade-off between intransigence and forgetting (see also the “stability-plasticity” dilemma in neuroscience Grossberg (1982)). Our results position the co-natural gradient as a strong lowforgetting/moderate intransigence basis for future work.

heading: 5 LOW-RESOURCE ADAPTATION EXPERIMENTS
text: In this section we take a closer look at the specific case of adapting a model from a single task to another, when we only have access to a minimal amount of data in the target task. In this case,
controlling the learning trajectory is particularly important because the model is being trained on an unreliable sample of the true distribution of the target task, and we have to rely on early-stopping to prevent overfitting. We show that using the co-natural gradient during adaptation helps both at preserving source task performance and reach higher overall target task performance.

heading: 5.1 EXPERIMENTAL SETTING
text: We perform experiments on two different scenarios:
Image classification We take MiniImagenet as a source task and CUB (a 200-way birds species classification dataset; Welinder et al. (2010)) as a target task. To guarantee a strong base model despite the small size of MiniImageNet, we start off from a ResNet18 model (He et al., 2016) pretrained on the full ImageNet, which we retrofit to MiniImageNet by replacing the last fully connected layer with a separate linear layer regressed over the MiniImageNet training data. To simulate a low-resource setting, we sub-sample the CUB training set to 200 images (≈ 1 per class). Scores for these tasks are reported in terms of accuracy.
Machine translation We consider adaptation of an English to French model trained on WMT15 (a dataset of parallel sentences crawled from parliamentary proceedings, news commentary and web page crawls; Bojar et al. (2015)) to MTNT (a dataset of Reddit comments; Michel & Neubig (2018)). Our model is a Transformer (Vaswani et al., 2017) pretrained on WMT15. Similarly to CUB, we simulate a low-resource setting by taking a sub-sample of 1000 sentence pairs as a training set. Scores for these two datasets are reported in terms of BLEU score.6 (Papineni et al., 2002)
Here we do not allow any access to data in the source task when training on the target task. We compare four methods Finetuning (our baseline), Co-natural finetuning, EWC (which has been proven effective for domain adaptation, see Thompson et al. (2019)) and Co-natural EWC.
Given that different methods might lead to different trade-offs between source and target task performance, with some variation depending on the hyper-parameters (e.g. learning rate, regularization strength. . . ), we take inspiration from Thompson et al. (2019) and graphically report results for all hyper-parameter configuration of each method on the 2 dimensional space defined by the score on
6We use sacrebleu (Post, 2018) with -tok intl as recommended by Michel & Neubig (2018).
source and target tasks7. Additionally, we highlight the Pareto frontier of each method i.e. the set of configurations that are not strictly worse than any other configuration for the same model.

heading: 5.2 RESULTS
text: The adaptation results for both scenarios are reported in Figure 3. We find that in both cases, the co-natural gradient not only helps preserving the source task performance, but to some extent it also allows the model to reach better performance on the target task as well. We take this to corroborate our starting hypothesis: while introducing a regularizer does help, controlling the optimization dynamics actively helps counteract overfitting to the very small amount of training data, because the co-natural pre-conditioning makes it harder for stochastic gradient descent to push the model towards directions that would also hurt the source task.

heading: 6 CONCLUSION
text: We have presented the co-natural gradient, a technique that regularizes the optimization trajectory of models trained in a continual setting. We have shown that the co-natural gradient stands on its own as an efficient approach for overcoming catastrophic forgetting, and that it effectively complements and stabilizes other existing techniques at a minimal cost. We believe that the co-natural gradient — and more generally, trajectory regularization — can serve as a solid bedrock for building agents that learn without forgetting.
7For CUB in particular we report the average accuracy of every configuration over 5 runs, each with a different 200-sized random subset of the data.

heading: A APPENDIX

heading: A.1 DERIVATIONS

heading: A.1.1 THE STANDARD GRADIENT UPDATE (EQUATION 3)
text: We derive the standard update in Eq. 3 by solving the Lagrangian L in Eq. 2 for δ. Given that its first and second derivatives are:
∇L = ∇LT + 2µδ ∇2L = 2µI
the problem is trivially strictly convex and its global minimizer δ∗ satisfies:
∇L ∣∣ δ∗ = 0⇐⇒ δ∗ = − 1 2µ ∇LT

heading: A.1.2 EQUIVALENCE OF THE HESSIAN OF THE KL DIVERGENCE AND THE FISHER INFORMATION MATRIX
text: To simplify notation, let us perform the change of variables θ + δ → x. We show that the Hessian of the KL coincides with the Fisher on θ: in other words, ∇2KL(pθ‖px) ∣∣ x=θ = Epθ [(∇ log pθ)(∇ log pθ)ᵀ]. Under mild regularity assumptions8, we can write:
∇2KL(pθ‖px) = ∇2Epθ [log pθ]︸ ︷︷ ︸ =0 −∇2Epθ [log px]
= −Epθ [∇2 log px]
Now note that ∇2 log px can be rewritten via standard derivatives manipulations as ∇ 2px px − (∇px)(∇px)ᵀ p2x . This leads to:
∇2KL(pθ‖px) = −Epθ [ ∇2px px ] + Epθ [( ∇px px )( ∇px p2x )ᵀ] When taken at θ, the first term evaluates to9 :
Epθ [ ∇2px px ]∣∣∣∣ x=θ = ∫ pθ(z) ∇2pθ(z) pθ(z) dz
= ∫ ∇2pθ(z)dz
= ∇2 ∫ pθ(z)dz︸ ︷︷ ︸ =1 = 0
By using the identity ∇pxpx = ∇ log px and evaluating at x = θ, the second term gives us:
∇2KL(pθ‖px) ∣∣ x=θ = Epθ [(∇ log pθ)(∇ log pθ)ᵀ]
8Essentially allowing us to interchange derivatives and integrals. 9We abuse notation and write∇px ∣∣ x=θ as∇pθ

heading: A.1.3 OBTAINING THE CO-NATURAL UPDATE (EQUATION 7)
text: We solve the Lagrangian from Eq. 6 in a similar fashion as in A.1.1. First we compute its gradient and Hessian:
∇L = ∇LT + 2µδ + 2νFSθ δ = ∇LT + 2(νFSθ + µI)δ
∇2L = 2(νFSθ + µI)
While not as straightforwardly as the one in A.1.1, this problem is also strongly convex: indeed FSθ is positive semi-definite (as an expectation of PSD matrices) and the addition of µI ensures that ∇2L is positive definite. We find the unique solution by solving:
∇L ∣∣ δ∗
= 0⇐⇒ ∇LT + 2(νFSθ + µI)δ∗ = 0 ⇐⇒ δ∗ = −[2µFSθ + 2νI]−1∇LT
Set λ := 12µ and α := ν µ to get Eq. 7

heading: A.2 ADDITIONAL EXPERIMENTAL SETTINGS FOR CONTINUAL LEARNING
text: This section is intended to facilitate the reproduction of our results. The full details can be found with our code at anonymized url.

heading: A.2.1 SPLIT CIFAR
text: We split the dataset into 20 disjoint sub-tasks with each 5 classes, 2500 training examples and 500 test examples. This split, performed at random, is kept the same across all experiments, only the order of these tasks is changed. During continual training, we train the model for one epoch on each task with batch size 10, following the setup in Chaudhry et al. (2018b).

heading: A.2.2 OMNIGLOT
text: We consider each alphabet as a separate task, and split each task such that every character is present 12, 4 and 4 times in the training, validation and test set respectively (out of the 20 images for each character). During continual training, we train for 2500 steps with batch size 32 (in keeping with Schwarz et al. (2018)). We ignore the validation data and simply evaluate on the test set at the end of training.

heading: A.2.3 GRID-SEARCH PARAMETERS
text: For each method, we perform grid-search over the following parameter values:
• Learning rate (all methods): 0.1, 0.03, 0.01 • Regularization strength (EWC, Co-natural EWC): 0.5, 1, 5 • Fisher damping coefficient (Co-natural finetuning, Co-natural EWC): 0,1.0,0.1 for Split CIFAR
and 0,0.1,0.01 for Omniglot
For ER, we simply set the batch size to the same value as standard training (10 and 32 for Split CIFAR and Omniglot respectively). Note that whenever applicable, we re-normalize the diagonal Fisher so that the sum of its weights is equal to the number of parameters in the model. This is so that the hyper-parameter choice is less dependent on the size of the model. In particular this means that the magnitude of each diagonal element is much bigger, which is why we do grid-search over smaller regularization parameters for EWC.

Review summaries:
This paper amends the gradient update rule for continual learning using a natural-gradient-style formulation in order to regularise the trajectory during learning to forget previous task(s) less. They show experiments where this 'co-natural gradient' update rule improves some baselines. They also provide experiments showing the benefits of this update rule for low-resource finetuning settings.