Regularizing Trajectories to Mitigate Catastrophic Forgetting

Abstract: Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in most practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters.
      In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the co-natural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low resource scenario.

Reviews and author responses:

This paper amends the gradient update rule for continual learning using a natural-gradient-style formulation in order to regularise the trajectory during learning to forget previous task(s) less. They show experiments where this 'co-natural gradient' update rule improves some baselines. They also provide experiments showing the benefits of this update rule for low-resource finetuning settings.

Although the idea seems reasonable and interesting, I feel like this paper needs work both in the theory and experiments. Figure 1 is a nice visualisation of the key take-away point of the paper.

Theory: the authors take the natural-gradient updates from batch learning and just modify it so that the KL term is now for the previous task(s) instead of the current one. Although this may seem reasonable, I would appreciate some analysis as to what this implies or means.

Experiments: 
- Split CIFAR from Chaudhry et al. (2018b) uses 10 tasks, why does this paper use 20 tasks? 
- Previous works usually find that for EWC, large values of the \lambda hyperparameter provide best results. This corresponds to lower forgetting of previous tasks. The hyperparameter range in Appendix A.2.3 is only over small values of \lambda (by orders of magnitude). 
- Why do the authors only allow 1 epoch per task for Split CIFAR? This probably results in early stopping: the new tasks are not able to reach their new optimal points (with or without regularised trajectories). This seems to go against the intuition provided by Figure 1, where the authors are showing that changing the trajectory results in a better local minimum being found. 
In fact, by adding another regularisation term, it is unsurprising that co-natural gradient updates have less forgetting, as the extra regularisation term probably means the trained parameters are even closer to the previous parameters.

-------------------
EDIT: Score changed to 'Weak Accept' following discussion with the authors.

Many thanks to the reviewer for taking the time to read and respond to our rebuttal. Here is our response to the concerns listed above.

> Re theory: I would have liked to see some analysis, if possible, about adding this specific KL term to the Lagrangian. I only say this because the idea seems very similar to previous natural-gradient literature, and therefore there might be something you can say / derive theoretically. I understand the intuitive explanation for adding this term. Of course, it may not be possible to say anything more theoretically.

As far as we understand it, the reviewer is asking for some sort of convergence guarantees or Fisher efficiency results in the vein of Amari et al. (1998) ("Natural Gradient Works Efficiently in Learning"). We have no such result as of now. The strongest guarantee we can give is that in the case of a fixed, positive definite Fisher, the co-natural gradient is guaranteed to converge in the convex case (since we are merely preconditioning with a fixed matrix).

> Re experiments: Thank you for clarifying about EWC, and apologies for missing the 20 task definition of split CIFAR in Chaudhry et al. (2018b). My concerns about 1 epoch still remain, however. A-GEM is designed for the setting where data is only visited once (1 epoch), whereas EWC (as well as many other methods) are designed for the case where there can be multiple epochs per task. See Appendix F in their paper where they show EWC performing better as the number of epochs is increased.
>
> I do not actually see any reason why the co-natural gradient update should do worse if trained for many more epochs. My intuition tells me that it could/should reach a better optimal solution ('better' = for continual learning). But my concern is that 1 epoch is not enough to let it reach a local minimum, therefore reducing the conclusions that can be drawn from the experiment. That being said, I am aware this does not impact your other results. How many epochs do you use for the new results on Split MiniImageNet?

We understand the reviewer's point with respect to convergence and the number of epochs (and we agree that the co-natural gradient is not expected to be "worse" if trained for longer --- barring the problem of the validity of our assumption on a static Fisher --- we simply meant to emphasize that regularizing the trajectory directly is likely to help at any stage of the optimization process).

Indeed, both our Omniglot and MiniImageNet experiments are run for more epochs (Omniglot for 2500 steps: ~=150-250 epochs depending on the alphabet and MiniImageNet for 500 steps, ~= 5.5 epochs per tasks). We take this as evidence that the co-natural gradient helps both when the model has converged (MiniImageNet, Omniglot) or not (CIFAR).

> Re re-normalizing the co-natural gradient norm: I am struggling to understand why this is important. Does the method essentially fail without this, even if you increase alpha? I would have hoped that the form of the update (without re-normalizing) was good enough itself. I would have expected that if some parameter updates were too large (because they have little to no effect on the KL divergence), then increasing alpha would ensure the update size is still of reasonable magnitude.

Yes, we found this to be important for the co-natural gradient to succeed. The main reason is as follow: the reviewer is correct that increasing the damping coefficient should in principle keep "almost-zero" parameters of the Fisher under control. However, higher alpha is also confounded with a "smoothing" effect on the Fisher: the ratio between any two diagonal elements becomes closer to 1 as alpha increases. This smooths over the relative importance of different parameters. Thus, higher alpha leads to better behaved update but to lesser regularization. On the other hand, re-normalizing decouples damping (control of the amount of regularization <=> the relative learning rate of any two parameter) and update size (now entirely controlled by the learning rate and the norm of the original gradient).

Perhaps a more elegant solution would have been to solve the Lagrangian on a hypersphere of fixed radius (=the desired norm $\Vert \nabla L_T \Vert$). This can be shown to yield an update of the form $\delta^*\propto -[F+\eta(\nabla L_T)I]^{-1}\nabla L_T$ where importantly the damping coefficient $\eta(\nabla L)$ is a function of the gradient $\nabla L_T$. As far as we know there is no closed form solution for $\eta$, and finding it iteratively would add a non-negligible computational overhead, hence our decision to use the renormalization approach.


Many thanks for your response. You have cleared up some questions I had. I still wonder about a few things.

Re theory: I would have liked to see some analysis, if possible, about adding this specific KL term to the Lagrangian. I only say this because the idea seems very similar to previous natural-gradient literature, and therefore there might be something you can say / derive theoretically. I understand the intuitive explanation for adding this term. Of course, it may not be possible to say anything more theoretically.

Re experiments: Thank you for clarifying about EWC, and apologies for missing the 20 task definition of split CIFAR in Chaudhry et al. (2018b). My concerns about 1 epoch still remain, however. A-GEM is designed for the setting where data is only visited once (1 epoch), whereas EWC (as well as many other methods) are designed for the case where there can be multiple epochs per task. See Appendix F in their paper where they show EWC performing better as the number of epochs is increased.

I do not actually see any reason why the co-natural gradient update should do worse if trained for many more epochs. My intuition tells me that it could/should reach a better optimal solution ('better' = for continual learning). But my concern is that 1 epoch is not enough to let it reach a local minimum, therefore reducing the conclusions that can be drawn from the experiment. That being said, I am aware this does not impact your other results. How many epochs do you use for the new results on Split MiniImageNet?

Re re-normalising the co-natural gradient norm: I am struggling to understand why this is important. Does the method essentially fail without this, even if you increase alpha? I would have hoped that the form of the update (without renormalising) was good enough itself. I would have expected that if some parameter updates were too large (because they have little to no effect on the KL divergence), then increasing alpha would ensure the update size is still of reasonable magnitude.

We thank the reviewer for their comments. See our top-level comment for a summary of the main points and updates to the paper. We respond to specific points in the review below:

> [Model] After surveying previous work, I am not sure if the paper is really novel: 
> First of all, adding KL-based constrains to alleviate model forgetting has already been widely explored in prior arts, as the authors also acknowledged in the paper. The proposed regularizer has a close relationship with the EWC, especially with the EWC++. 

First, let us state that the relationship and differences with EWC are clearly stated in the paper (section 4.1): “it is a natural baseline for us to compare to in that it also consists in a Fisher-based penalty --- although in the loss function instead of the optimization dynamics”.

Regarding EWC+++, we would like to apologize for an omission in the paper that likely led to confusion. In our EWC experiments, we use the same rolling Fisher described in section 3.4. This makes our EWC implementation equivalent to the online-EWC introduced by Schwartz et al. (2018). The paper was updated to reflect this. Now as we understand it from Chaudhry et al. (2018a), EWC++ differs from EWC in two ways:
- Use of only one, rolling Fisher for computing the l2 penalty (which is what we, and online-EWC also do)
- Compute the Fisher for a task in an online. This is justified by the setting in Chaudhry et al. (2018a) where the model can only see the data once. However, we place no such restriction in our experiments, and as such it is strictly better to compute the Fisher for a given task after training on this task has finished, which we and online-EWC do.

>EWC++ encourages the KL-divergence between two distributions learned at successive tasks to be minimized, while the authors proposed to regularize the KL-divergence between two nearby updates, which leads to the well-known natural gradient descent.

Our method does not just regularize the KL between two nearby update like the natural gradient. We quote the end of section 3.3 to highlight the difference:

“There is a however a crucial difference, both in execution and purpose: where the natural gradient uses knowledge of the curvature of the KL divergence of $\mathcal{D}_T$ to \emph{speed-up} convergence, our proposed method leverages the curvature of the KL divergence on $\mathcal{D}_S$ to \emph{slow-down} divergence from $p^S_{\theta_S}$.”

In particular the KL is taken between distributions over the past tasks which is not the case for the natural gradient.

> The authors [...] proposed to use a static Fisher estimated at the previous task for fast approximation. However, the approximation makes the algorithm not a natural gradient descent approach, nor a valid KL-regularized optimization problem.

While this is certainly true, this approximation is shared with the large majority of previous work using the Fisher for continual learning, including EWC (Kirkpatrick et al., 2017) EWC++ (Chaudhry et al, 2018a) and online-EWC (Schwartz et al., 2018).

> The theoretical implications of the static Fisher approximation are not discussed in the paper.

We clarify this in our revised version: this approximation is only valid insofar as we are staying close to the original parameters. As per our experimental results, this approximation still leads to good results empirically. Finally we would like to reiterate that this approximation is found in related work as well (EWC, EWC++, online-EWC...).

> [Experiments] The authors may consider comparing with EWC++, which is a closely related baseline. 

See our comment above, our EWC baseline actually uses the rolling Fisher described in sec. 3.4, thus making it equivalent to the “online-EWC” described in Schwartz et al. (2018). Apologies for the omission, this is clarified in the text now.

> Still, it would be much better to see how it can help more state-of-the-art methods like LwM, LwF, etc, and especially on a few more challenging datasets.

We would argue that LwF (2016) is not a state of the art approach, in fact Schwartz et al. (2018) show that it is inferior to online-EWC (one of our baselines). We thank the reviewer for bringing up LwM to our attention. While it seems like this approach was mostly intended for class incremental learning, a slightly different problem that the one considered in our experiments, we have updated our submission to cite it. Last but not least we emphasize that our ER baseline from Chaudhry et al. (2019) is a very recent approach, shown to be SOTA compared to a variety of other methods.

Regarding more challenging datasets, we have added results for MiniImageNet to the paper, with similar results. While we are limited by the time alloted to author discussion, we are open to suggestions from the reviewer. We also refer to section 5, where we perform experiments on more realistic settings (MiniImageNet -> CUB and machine translation experiments).

EDIT: fixed title to "Reviewer #2"

We thank all reviewers for their valuable feedback. We have responded to most comments individually. The paper has been updated to include:

- A clarification that our EWC baseline coincides with online EWC (Schwarz et al. 2018): We keep only one rolling Fisher for EWC, which we update at the end of each task's training. We believe that this addresses most comments regarding a stronger EWC baseline.
- Clarify that we renormalize the Fisher so that the average diagonal element has magnitude ~= 1. This explains our choice of hyper-parameters for EWC.
- Additional continual learning experiments on MiniImageNet: this is to provide experiments on a more challenging dataset. We observe similar conclusions as on Split CIFAR and Omniglot
- Clarification on when the static Fisher approximation is expected to hold, and a note on its ubiquity in the related literature


We thank the reviewer for their comments. See our top-level comment for a summary of the main points and updates to the paper. We respond to specific points in the review below:

> I think that the performance of finetuning + co-gradient is too natural, and not much meaningful.

We are not entirely sure of what the reviewer means with this statement. Surely it is natural that a method designed to reduce catastrophic forgetting indeed improves performance in continual learning. We respectfully but firmly disagree with the reviewer that the fact that the proposed approach is simple or natural means that it is “not much meaningful”. In our opinion the simplicity of the proposed approach is a strength, not a weakness.


> And the tradeoff between accuracy and forgetting for baselines is hard to compare on table 1 since the tradeoff depends on the hyperparameters.

We argue (we think, in line with Chaudhry et. al (2018b) and follow-up work) that comparing these results with different sets of hyper-parameters chosen with grid-search makes for a *better* comparison. Indeed, we are comparing the best set of hyper-parameters (as chosen by our validation procedure) for each method. For a more exhaustive exploration of the trade-off for EWC and co-natural fine-tuning (albeit in a slightly different setting), we refer to Section 5

> And it is required to apply on heterogeneous datasets to evaluate the performance when problems are really different. (like MNIST->CIFAR100->Omniglot->Imagenet, and so on). In the paper, the experiments are performed on very similar problems (split a single dataset).

First and foremost, most recent continual learning work uses the datasets that we used or even less realistic datasets (permuted MNIST...). The reason we do this is that this is an easy way to get 1. Many tasks (so we can test for long term forgetting) and 2. Tasks that are of similar difficulty (the impact of task ordering is limited).
With only 4 tasks (MNIST->CIFAR100->Omniglot->Imagenet) we encounter other issues: too few tasks to perform grid search on a subset of tasks, curriculum effect (learning Imagenet and then MNIST vs MNIST then imagenet), different data size, etc… overall we believe that this is a valid criticism but it is more addressed to the current literature on CL than this paper specifically.

Finally note that, albeit in a different setting, Section 5 provides examples of adapting to different, more heterogeneous datasets.

>Also, it would be great to show an illustration like figure 1, on real dataset, like split CIFAR.

We are not entirely sure what the reviewer has in mind. Given that in that case the dimension of the parameter space is of the order of the millions, such a representation (projected in 2d) is unlikely to be meaningful. We are open to suggestions.


We thank the reviewer for their comments. See our top-level comment for a summary of the main points and updates to the paper. We respond to specific points in the review below:

> Theory: the authors take the natural-gradient updates from batch learning and just modify it so that the KL term is now for the previous task(s) instead of the current one. Although this may seem reasonable, I would appreciate some analysis as to what this implies or means.

As made explicit in Section 3.2 of the paper:

“ In a continual learning setting [...], the quantity we are most interested in preserving is the probability distribution that θ models on the source task S [...]. Therefore, a  more natural distance between θ and θ+δ is the Kullback-Leibler divergence”

Applying a KL penalty in the Lagrangian from equation (2) will encourage the updates to be selected to as to preserve the distribution induced by the model on past tasks (and as a proxy, its performance). Let us know if there is anything more we can add to clarify what this implies or mean.

>Experiments:
>  - Split CIFAR from Chaudhry et al. (2018b) uses 10 tasks, why does this paper use 20 tasks?

Quoting Chaudhry et al. (2018b): “Split CIFAR (Zenke et al., 2017) consists of splitting the original CIFAR-100 dataset (Krizhevsky & Hinton, 2009) into 20 disjoint subsets [...]”. See https://arxiv.org/pdf/1812.00420.pdf

>  - Previous works usually find that for EWC, large values of the \lambda hyperparameter provide best results. This corresponds to lower forgetting of previous tasks. The hyperparameter range in Appendix A.2.3 is only over small values of \lambda (by orders of magnitude).

In all our experiments, we renormalize the fisher so that it sums up to #params. This is to make the average value of the diagonal elements independent of the size of the model, easing hyper-parameter selection across models. In particular this means that the average magnitude of our fisher weights is much higher. As a consequence, we can get away with ranges of EWC regularization that are much smaller than what is usual in the literature. This was omitted in the original version, and we have amended the paper to make it clear (Appendix A.2.2). That being said, we have run additional experiments with EWC regularization values 10 and 50 for split CIFAR and found that grid search never prefers these values over the ones in {5, 1, 0.5}.

> - Why do the authors only allow 1 epoch per task for Split CIFAR? This probably results in early stopping: the new tasks are not able to reach their new optimal points (with or without regularised trajectories). This seems to go against the intuition provided by Figure 1, where the authors are showing that changing the trajectory results in a better local minimum being found. 

We follow Chaudhry et al (2018b). Note that the batch size is relatively small (10), so we are performing as many updates than as if we were training for 3 epochs with batch size 32. Moreover is not at odds with the intuition in figure 1: especially if optimization does not converge, following the trajectory that hurts previously learnt tasks the least is advantageous. This can be seen in Figure 1 by the fact that the co-natural trajectory tends to follow a path where the loss on T1 increases much more slowly.

> In fact, by adding another regularisation term, it is unsurprising that co-natural gradient updates have less forgetting, as the extra regularisation term probably means the trained parameters are even closer to the previous parameters.

While the reviewer is correct that the positive effect of adding the co-natural gradient is not unexpected, we feel like their statement is dismissive of the three following points:
 - The co-natural gradient leads to lower forgetting than the two other approaches on its own. Thus, it is not just any regularization term, rather it is the one that leads to the least forgetting.
 - The co-natural gradient alone is still competitive or even better (see Omniglot) than the other methods, in terms of average accuracy.
 - The co-natural gradient requires strictly less resources than EWC (need the Fisher and the previous parameters) or ER (need access to data from previous tasks). We only require the Fisher

Finally, let us note that the co-natural gradient does not explicitly encourage parameters to stay close to the original parameters (like EWC does). In fact, as mentioned in section 3.3, we re-normalize the co-natural gradient to the same norm as the standard gradient: the co-natural gradient does not change the magnitude of the updates, rather their direction.


The paper puts forward a new regularization based continual learning method that explicitly regularizes the optimization trajectory by constraining in the distribution space. The paper is well written, and preliminary empirical results are promising. 
 
[Model]
 After surveying previous work, I am not sure if the paper is really novel: 

First of all, adding KL-based constrains to alleviate model forgetting has already been widely explored in prior arts, as the authors also acknowledged in the paper. The proposed regularizer has a close relationship with the EWC, especially with the EWC++. EWC++ encourages the KL-divergence between two distributions learned at successive tasks to be minimized, while the authors proposed to regularize the KL-divergence between two nearby updates, which leads to the well-known natural gradient descent.

Natural gradient updates require to calculate the Fisher based on the current curvature, which is computationally expensive in practice. The authors further proposed to use a static Fisher estimated at the previous task for fast approximation. However, the approximation makes the algorithm not a natural gradient descent approach, nor a valid KL-regularized optimization problem. The theoretical implications of the static Fisher approximation are not discussed in the paper.
 
[Experiments]
The authors may consider comparing with EWC++, which is a closely related baseline. 
The experiment results have shown that the co-nature gradient method help from time to time. Understandably, the co-nature gradient-based optimization has the add-on benefit for any continual learning tasks. Still, it would be much better to see how it can help more state-of-the-art methods like LwM, LwF, etc, and especially on a few more challenging datasets.



The paper proposed a novel regularization methods for continual learning. The authors introduce co-natural gradients, which is an incremental development of natural gradient methods, note that co-natural gradients use Fisher information to regularize the trajectory of the gradients which will be optimal on both tasks.

The method can be applied on existing regularization-based continual learning approach such as EWC, or ER. And with co-natural gradients, the model can perform better on continual learning problem.

I think that the performance of finetuning + co-gradient is too natural, and not much meaningful. And the tradeoff between accuracy and forgetting for baselines is hard to compare on table 1 since the tradeoff depends on the hyperparameters. 

And it is required to apply on heterogeneous datasets to evaluate the performance when problems are really different. (like MNIST->CIFAR100->Omniglot->Imagenet, and so on). In the paper, the experiments are performed on very similar problems (split a single dataset).

Also, it would be great to show an illustration like figure 1, on real dataset, like split CIFAR.



Meta-review: The submission proposes a 'co-natural' gradient update rule to precondition the optimization trajectory using a Fisher information estimate acquired from previous experience. This results in reduced sensitivity and forgetting when new tasks are learned. 

The reviews were mixed on this paper, and unfortunately not all reviewers had enough expertise in the field. After reading the paper carefully, I believe that the paper has significance and relevance to the field of continual learning, however it will benefit from more careful positioning with respect to other work as well as more empirical support. The application to the low-data-regime is interesting and could be expanded and refined in a future submission. 

The recommendation is for rejection.

Decision: reject