Overcoming Catastrophic Forgetting via Hessian-free Curvature Estimates
Authors: Leonid Butyrev
Georgios Kontes
Christoffer Löffler
Christopher Mutschler
Venue:  (2020)
Keywords: catastrophic forgetting
multi-task learning
continual learning

Abstract: Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks. This leads to a poor performance on old tasks – a phenomenon framed as catastrophic forgetting.  While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization.  Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network. This requires to calculate the Hessian around a mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.  In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task. Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work.

Under review as a conference paper at ICLR 2020
OVERCOMING CATASTROPHIC FORGETTING HESSIAN-FREE CURVATURE ESTIMATES
VIA
Anonymous authors Paper under double-blind review
ABSTRACT
Learning neural networks with gradient descent over long sequence of tasks is problematic their fine-tuning to tasks overwrites the network weights that are important for previous tasks. This leads to a poor performance on old tasks -a phenomenon framed as catastrophic forgetting. While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization. Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network. This requires to calculate the Hessian around mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as alternative method to actually calculating the Hessian. In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate bessiian-vectorprrotee current task. Our experiments show that on a variety of well-known task sequences we either significantly outperform or en par with previous work.
around the surface that is relevant for the
INTRODUCTION
The main goal of machine learning is the ability to generalize from the given training data to unseen examples. However, practice the achievable degree of generalization is limited. While in the ideal case an end-to-end system learns complex functions from minimum input, it is often necessary to in- troduce certain amount of prior knowledge. Such prior knowledge operates as an inductive bias and therefore has constraining effect on the hypothesis space, i.e., the set of all possible functions that can be learned by the learning algorithm (Mitchell 1980). While this sounds counter-intuitiv such a reduction of the hypothesis space may lead to better generalization properties in practice (Mitchell, 1980 Hence, instead of eliminating the bias to increase generalization (as suggested by Hessel al. 2019)), promising direction of research tries to identify and introduce the right form of it.
We can achieve this by limiting the functions that can be expressed by the learning algorithm or by introducing bias to the learning algorithm itself. Simple examples include the choice for linear activations to only allow approximations of linear functions or to add a regularization term to the objective function. Similar
this, we can also improve generalization by training on different lasks [axtter 2000) from a task family at the same time or by introducing auxiliary tasks (Jaderberg al. 2017). This is commonly known as multitask learning and has shown not only improve generalization properties but also to be more sample-efficien (Baxter 2000). Due to the limited availability of data for training we need well-tuned inductive bias. Hence, such choices are crucial for the final real-world performance of any machine learning algorithm.
While multitask learning a great tool to improve generalization and to reduce the amount of sam- ples that are necessary to learn family of tasks it is still limited in its scalability. Both the amount of tasks that can be learned and the amount data required to learn them are strongly limiting factors. Consider, for instance, reinforcement learning setup where an agent learns different tasks from interacting with in an environment. In practice we are limited in storing the data for all relevant tasks required to train model all tasks jointly. However, learning those tasks sequentially is also not an option as gradient descent and its variants (which are the dominant learning approaches for neural networks) do not consider the importance of individual parameters for early tasks.

---

Under review as a conference paper at ICLR 2020
This destructive learning is commonly termed catastrophic forgetting (McCloskey & Cohen 1989). While in the context of fine-tuning and pre-training (Erhan et al.) 2009) this does not bear a problem (as the goal is not to reuse the previous parameter state, but rather to optimize the learning process for some target task) it becomes important in multitask problems where we wish to maximize generalization and sample-efficincyy It is also critical in the continual learning framework, where the parameters of a neural network are optimized over multiple datasets (representing different tasks) provided sequentially, which are not available later time. The goal is hence to retain all (or most) of the important parameters for previous tasks and to be able to build-up on this knowledge for an arbitrary number of future tasks. Thus, the scalability of learning would only be limited by the capacity of the neural network but not by the properties of the training method.
The Bayesian framework (Kirkpatrick et al., 2017 Ritter et al.) 2018) a promising approach to address catastrophic forgetting. The information about former tasks is condensed in a prior, which not only preserves the knowledge about tasks but also introduces an inductive bias based on the learned tasks. Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017) is simple yet efficient way to reduce catastrophic forgetting. EWC approximates the prior with a Gaussian centered around the optimized network parameters for previous tasks, where the diagonal precision is given by the diagonal approximation of the Fisher Information Matrix (FIM). This approach has two significant downsides: i) each new task adds a new regularization term that penalizes changes of parameters that are relevant to previous tasks; and ii) the diagonal approximation of the FIM assumes independent network parameters, which leads to information loss with a growing number of tasks. Ritter et al. (2018) extend EWC but still approximate the prior from previous tasks using a Gaussian. They devise block-diagonal approximation for the prior from the older tasks by defining a quadratic approximation whose solution requires to calculate the Hessian. The Hessian is in turn approximated by the block diagonna Kronecker factored- approximation.
In this work we propose an alternative way of calculating the Hessian, based on well established Hessian-free (Schraudolph) 2002] Pearlmutter] 1994) methods to estimate curvature information of the network parameters. In contrast to Ritter et (2018), we exploit the fact that most regions in the loss surface are flat (Ghorbani et al. 2019). This allows us to use only a small subset of the Hessian as it holds enough relevant information. We then use Hessian-vector vector-productt sample from this subset. This way, we can incorporate the importance of individual weights and include dependencies between the network parameters when train the network over long sequence of tasks. We evaluate our algorithm on permuted MNIST (Kirkpatrick et al. [211)), disjoint MNIST (Ritter et al.] 2018) and single- headed disjoint MNIST (Farquhar & Gall 2019), and compare with state of the art approaches. Our results show that we consistently outperform EWC across all tasks and that we are en par with Ritter al. (2018) on the disjoint tasks, while our method has significantly lower space complexity compared to both EWC and fronecker-factoree approximation.
The remainder of this paper is structured as follows. Section 2 provides background on continual learning, EWC, and Kronecker-facee factored Laplace approximation. Section 3 describes our method in detail. Section [ shows the efficiency of our approach and compares against state of the art on a variety of well-known task sequences. Section discusses related work. Seccioono]concuuud
BACKGROUND
2.1 CONTINUAL LEARNING AND CATASTROPHIC FORGETTING
In the continual learning framework the parameters e R" of neural network are optimized over multiple datasets Di...., D,,.,,,,,.... D.,..., Dr. These individual datasets become available to the training algorithm one after another and usually cannot be revisited at a later time. The goal is to achieve a high accuracy/performance on the current task (represented by the current dataset DD) while still preserving (high or most of) the performance for all the previously visited tasks. However, this is usually challenging for neural network models as commonly used gradient-based optimization methods cannot distinguish between important and unimportant parameters for previous tasks. As consequence parameters that are relevant for previous tasks are modified (heavily), which leads to performance degradation when the network is used on any of those previous tasks (Rusu et al.
2016).
2

---

Under review as a conference paper at ICLR 2020
Hence, to address catastrophic forgetting in neural networks we need to retain the parameters that are important for previous tasks while still allowing the network to learn new tasks. However, at the same time also want the space complexity of the network to be independent of the amount of tasks that were observed so far (and that are about to come). This means that learning a new task while retaining high performance on all prior tasks should be possible without adding new parameters or regularization terms for each new task, at least as long sufficient capacity is available. As plus we want foster some degree of parameter sharing enable positive transfer effects, e.g., improved
due to the fact that past experience can be reused.
2.2 ELASTIC WEIGHT CONSOLIDATION (EWC)
EWC (Kirkpatrick et al., 2017) simple yet efficient approach that meets most of the above mentioned requirements. The key idea is to add penalty when parameters that are important for previous tasks are about to be changed while parameters that are less relevant for previous tasks do not receive penalty. EWC uses quadratic penalty term that is derived from Bayesian formulation of the problem (where all the information of all previous tasks is condensed the prior) as follows:
p(D++1 10)p(0
p(0 11
(1)
p(D++1)
where 1(+4)) and p(:) are the posterior and prior distributions over the parameters 0 of the network and DI...., Dr, D++1 the datasets corresponding to the respective tasks. If we want to learn a new task we update the posterior by conditioning it on the newly available data D++1-
However, we have to address two problems that stem from Equation 1 First, maintaining the full posterior over all previous datasets is usually intractable (Ritter et al. 2018 Opper & Winther 1998) and we instead need to approximate it. Second, without storing the information from all previous tasks there is no easy solution to update the posterior.
The first problem can be addressed by approximating the posterior with Gaussian (MacKay 1992
p(0)Di:)) ~ ~N(, Ex).
With two tasks A and B and their datasets DA and DB. for the posterior p(0)DA) the mean PA given by the solution for the previous task 07 and the precision Σd1 i.e., the inverse of the covariance, by the diagonal of the Fisher information matrix (FIM) F. Learning tasks and consecutively then results in the following objective function:
[()) = (0)
Cn()) +
02) (0-
02),
(3)
where (0) is the loss depending on the current data Dg. and À is a hyperparameter that controls the influence of the regularization term. At this point we only need store the previous weights and the diagonal approximation of the FIM for the previous task. For another task C we store separate FIM for that new task together with the solution for task g. and add another regularization term:
[(0) - -c(0) - 2 (0 FA 02)
2 (0 10-0%) FB 0z))
2.3 KRONECKER- FACTORED LAPLACE APPROXIMATION
The diagonal approximation of the FIM assumes the parameters to be independent, which is rarely the case in practice. Rittrr et al. (2018) address this shortcoming by adopting the Bayesian online learning approach (Opper & Winther 1998). As the prior p(0 Dıa) preserves all the information about the previous tasks recursively using the previous posterior as the next prior makes it possi- ble to find a MAP-estimate 8 arg max, p(f|DI..... D(41)) sequentially. Due to the fact that the posterior conditioned on all previous tasks is intractable, a parameterization of the posterior p(0(De+1, w()) with parameters w(() is introduced. To update this parametric approximate poste- rior requires two steps:
1. Update Step: in update step the old approximative posterior p(0) w())) is used to perform an update using the Bayesian rule (see Ritter et al. (2018) for a detailed analysis):
((0 = [ d0 00''(D2 10 (w(())
(5)
3

---

Under review as a conference paper at ICLR 2020
2. Projection Step: In a projection step the new posterior p(0 w(t)) is projected onto the same parametric family as p(0 mx))) (as they are usually not from the same parametric family):
q (( + 1)) = p(0)De11,
(6)
Similar to EWC the update step be approximated by Gaussian approximate posterior:
(()) 1 | -0- 14))
As before, the mean H4 is given by the solution for the previous task Or- Accordingly, the parameters w(() are given by w()) = {14,, Στ-} The core improvement that this framework offers is encapsu- lated in the projection step: instead of adding a new regularization term for each new task, Σ, is instead projected to Σy-1 which then maintains information about all tasks up to task t + 1. Ritter et al. (2018) realize this by computing the Hessian around the most recent solution 07+1 1+ and adding It to the Hessians from all previous solutions:
E+1 14++(00+11 + ET where
H (0 8 PPp(De+ll
0000
0=0
This way information about previous tasks can be preserved while still limiting the storage require- ments to a constant number of parameters. However, in practice this approach needs to store a set of parameters per task.
3 HESSIAN-FREE CURVATURE ESTIMATION
Previous approaches identify the most important parameters for each previous task and then prevent the modification of those parameters during the training of a new task. EWC uses the diagonal of the FIM while Ritter et al. (208)) use a Hessian approximated using the block-diagonal Kronecker- factored approximation.
We address the same problem but approach it differently. We build upon the intuition of meta- learning in general and from the model-agnostic meta learning (MAML) algorithm (Finn et al.) 2017) in particular. MAML identifies model parameters that (upon modification) lead to faster learning for all tasks in given task distribution. By defining a meta-learning objective and using available data for all tasks in the task distribution it leams network weights that will lead to faster learning and generalization in new tasks, if being used as a starting point for the optimization.
In our case, apart from the fact that assume access to samples from previous tasks, invert the intuition behind MAML: we identify model parameters that are sensitive to changes in each task but instead of tuning these parameters be a good starting point for the fine-tuning of all tasks, we penalize large changes to them, as this will deteriorate the performance of previous tasks.
In order identify the important network parameters, i.e., parameters that upon being changed lead to big change in the loss, we also use the Hessian matrix, but in contrast to the Kronecker-factore Laplace approximation we exploit the fact that most regions of the loss surface are flat (Ghorbani et al. 2019). This allows us to use only a small subset of the Hessian as this subset already holds enough relevant information. We then use a Hessian- wector-eeeerpppe vector-producc product to sample from this subset.
In essence, we need to estimate directions with high curvature as at those points we find the im- portant weights of the network. However, any computation involving the exact Hessian for larger networks is infeasible in practice. Hence, it is key to find a good approximation of the Hessian while still preserving enough curvature information to determine which parameters are crucial for the previous tasks. Fortunately, extract information about the few regions that exhibit a high curvature. Thus, instead of computing the full Hessian we compute Hessian-vector protuct, which is similar sampling the curvature in the direction of a given vector. There are two important questions to answer here: (i) how to efficiently calculate the
most regions in the loss surface are flat it is sufficient only
vector product, and (ii) how to chose suitable vector/direction.
An efficient Hessian-vector product calculation was initially presented in Pearlmutter (1994) and has subsequently been used for several Hessian-free (also called truncated-Newton) optimization
4

---

the momentum Under review as a conference paper at ICLR 2020
methods (Schraudolph) 2002 Martens 2010). The key idea is that the Hessian is not calculated explicitly. Instead, for a given vector v the Hessian-veet vector- cettoor-prouctt Hu is directly computed using finite differences (Martens 2010) at the cost of forward- and backward- pass through the network (e.g., using algorithms such as back proppappppggt.on) The Hessian-vector-pnonn product is then calculated by (see [Pearlmutter] (194) for the implementation details):
Vf(0 + Ev)
v 0 V
He lim c-0
e
de
=
Given that the Hessian-vector-pe vector-product can be computed as described above, the second question is how to choose the vector that defines the direction in which sample the curvature. Inspired by Stochastic Meta-Descent (Bray et al.] [2004a]b) which uses the combination and a Hessian-vector-produu to select the vector v is to use the momentum. In our case the momentum is calculated using the exponentially weighted moving average of the past gradients:
estimate gradient directions with low curvature, our first choice
V4+1 at, + v())
(10)
where a controls the discount older observations. The momentum is a sensible choice for the vector as it holds information about the parameters that have been changed the most during the training. The assumption is then that exactly these parameters will be among the most important ones for the most recent task. As such, if the parameters for the previous task 8 an optimum, any change to important parameters results in a performance drop.
An alternative to the momentum is the eigenvector corresponding the largest eigenvalue. This eigenvector represents the direction highest curvature, and therefore by definition includes the most important parameters for the most recent task. A simple way to compute this eigenvector is to use the power method (Wilkinson] 1965), which entails computing a dresiaan-vectorprroddee
Both versions result in vector which maintains critical information about second-order interactions. From this vector we construct a positive semidefinite matrix by placing its absolute values as the entries of diagonal matrix. Let he be the resulting vector of the dessian-veetor-prouus H for task t, then our curvature estimate C2 is given as:
h4,2
C.
(11)
with n the number of network parameters. The projection step then defined as:
Σ-1= C. E
D-1
(12)
and the final objective function for new task + 1 as:
> 00) 0%) [()) = (0)
(13)
Similar to KKikpptrick et al. (2018) and Ritter et al. (2018) we add a hyperparameter À to control the influence of the regularization term the overall loss, i.e., that controls how to weigh the importance of the previous tasks over the most recent task.
One of the main advantages of our approach is the low storage requirements. Following the analysis in Ritter et al. (0118), Kronecker-factor approximation approach requires that all Hessians for pre- vious tasks FIM for all previous tasks are required to learn each new task. Instead, our approach only needs to store two vectors with the same size as the network parameters independently of the size of the task sequence.
kept in memory and the same holds for EWC, as the diagonal approximation of the
4 EXPERIMENTS
In our experiments, we compare both of our Hessian-free curvature estimations (eigenvector and momentum) to closely related methods, i.e.. EWC (Kirkpatrick el. 2017) and Kronecker-factored
5

---

Under review as a conference paper at ICLR 2020
1.0
0.9
Test accuracy in %
0.8
0.7
EWC
0.6
Momentum
Hv Eigen
Kronecker
0.5
20
40
50
Number of observed tasks
Figure 1: Experimental results using permutedMNIST for the best hyperparameters in compari- son, and details of EWC, Kronecker factored approximation and Hessian-free with momentum and eigenvector.
approximation (Ritter al. 2018 For both EWC and Kromecker-factored approximation we adapt the implementation from ht ht /ttpt::///ghhbb../// http:///gg.tttoooooo/////aaa/aa/ We release the source code of our methods upon publication.
4.1 PERMUTED MNIST
For our first evaluation, we utilize the widely-used pe MermutedMNTS dataset as presented in Good- fellow al. (2013) and used Kirkpatrick et (201)) and aitter al. (208)). The dataset contains 28 28 grey-scale images, that permuted randomly in order to generate new tasks. Each permu- tation a truly new task, since it unrecognizable from its original.
For the evaluation, we perform hyperparameter search with the following range of parameters: i) network structure: either 1 layer with 200 hidden units or layers with 100 hidden units each; úi X [1, 3, 10, 20, 30, 100, 300]. We use the ADAM optimizer with learning rate of 0.001, momentum of 0.5, and a batch size of 64 over 10 epochs.
average accuracy over all 50 tasks with the best hyperparameters discov- ered for each method. While Kronecker-facte approximation achieves 83. 82%, Hessian-free curva- ture estimation achieves 62.58% and Hessian-free curvature estimation with the largest eigenvector achieves 61. .63%, leading to better results compared to EWC (51.62%) for the last 15 tasks.
Figure shows the
Even though Kronecker-factored approximation achieves better performance compared our ap- proach, according to Farguhar & Gal (2019) in order to evaluate continual learning approaches other tasks can be more representative. In fact, Farquhar & Gal (2019) suggest use specific version of dis joisttiI ST which we evaluate below.
4.2 DISIOINTMNIST
For an evaluation according the jistointMNIST (Ritter al. 2018) split MNIST into two tasks: (1) letters '0' to '4' and (2) letters '5' to '9'. For this experiment we use network with a ten-way classifier which makes the problem considerably more challenging than in the previous experiment where we used a five-way classifier. Hence, here the classifier learns strong (bad) prior for the (respective) unseen classes the datasets. more difficult as training on the second split can easily overwrite the parameters of the ten-way classifiers for the classes of the first split. We use a simple dense need-forward network architecture with 2 layers and 100 hidden units in each layer well as batch size of 250 reported in Ritter et al. (2018). We use 10 epochs and the same Adam parameters as in the Pe Permutedo MdMueddMN IST experiment. This allows a comparison of our results against Kronecker-factonee
approximation and EWC.
Following the same evaluation procedure from Ritter et al. 2018) Figure 2a illustrates the result of hyperparameter search over À E [10", 10a... ...,, 10 for EWC. Kronecker-factored approximation,

---

Under review as a conference paper at ICLR 2020
0.9
1.0
Test accuracy in
0.8
Test acouracy
0.8
EWC
0.4
EWC
Kronecker Elgen
Kronecker
Hv Eigen
0.5
Hv Momentum
Hv Momentum
0.0
10"
10"
10"
10"
10
10"
10
0
2
3
Hyperparameter lambda
Number observed tasks
(a) dí jjjintMNIS
(b) S ngl HeHeeeddd-Sppp tN IST
Figure Influence of À on different approaches.
and ours (i.c., Hessian-free curvature estimation using either the largest eigenvector the momen- tum to estimate v). The results show the balancing of retaining information on old tasks over the learning accuracy
tasks. Note that the different scales in À between our results and that from Ritter et al. (2018) only stem from different implementation details (but the results are still comparable). Similar to the Pe Permut edMNI ST experiment, we see that our approach (using the momentum) outperforms EWC with 91.01% (at X = 10%) vs 86.11% (which is what we expected as EWC disregards parameter dependencies that are not reflected by the diagonal of the FIM). Sur- prisingly, our approach is even comparable to the 94.93%) although our method uses considerably less storage memory to store information on the importance of parameters. The use of the largest eigenvector on the other hand performs poorly compared to the other methods with 72.69% for À = 10
mamecker-factoree approximation (which reaches
4.3 SINGLE-HEADED SPLIT MNIST
For the Si Single- Heaglee-Haddd-SSe it -MMNIST task (Farquhar & Gal 2019) the available digits split into five groups (i.e., tasks) of two classes each. The classifier (as for the Permuted PermutedMNIS uses ten outputs, i.e., for each digit, and the network is trained on each task one after another. In contrast to some other work (Zenke et al. 2017) all the tasks share the classifier head instead of having multiple task-specific outputs. Hence, the predictions are made for all possible outputs, only for the outputs of classes that belong the most recent task.
We use the same network as in the previous experiments (i.e., layers of 100 hidden units each) and a batch of 64. Figure 2b shows the results after a hyperparameter search over X. As in the previous experiments we can observe that both of our Hessian-free eurvature estimations consistently outper- form EWC (Hessian-free with momentum achieves 57.54% and the eigenvector approach 55.36% while EWC reaches 46.73% 73%) and that the momentum-baas Kronecker-factoree approximation (which is at 57. 2%% at the end).
variant even comes again close to the
5 RELATED WORK
Related work around the field of catastrophic forgetring is mainly driven by regularization methods, rehearsal methods, and dynamic architecture methods.
Regularization Methods. Elastic Weight Consolidation (Kirkpatrick et al. 2017) measures the dis- tance between the network weights for the current task and the weight state of previous tasks, and applies a quadratic penalty weighted by diagonal approximation of the Fisher information matrix to ensure that the new weights are not too far from the old weights. EWC only penalizes important parameters while the parameters that have no influence on the performance of previous tasks allowed to change freely. Similar approaches have been proposed by Alijndi et al. (2018) and Lee et al. (2017). The main difference is how the importance of parameters for previous tasks approx- imated. However, all these approaches have limited performance as they do not consider interactions between the parameters. Instead of using the diagonal of the Fisher information matrix (Ritter al.

---

Under review as a conference paper at ICLR 2020
2018) apply sonnecker-factored approximation of the Hessian. This leads to strong improvements over EWC. This approach is most similar to ours, as it attempts to capture second-ordes parameter interactions to regularize parameter change. The main difference to our method is the usage of the Kronecker factorization to store the Hessian in a compact way while we exploit the fact that most regions in the loss surface are flat (Ghorbani et al. (019). This allows us use only a small subset of the Hessian as it holds enough relevant information. We then use a Hessian- vectorr-eetor-eeee vector--root to sample from this subset.
Rehearsal Methods. Rehearsal methods attempt to reduce catastrophic forgetting by replaying examples of previous tasks when learning a new task. A first approach here is not only learn the actual task at hand but also the distribution of the training data. When new task is learned, artificial samples from this learned distribution are added to the current set of training data. Typically this is done by adding Variational Autoencoder (Kamra et al., 2017). Recent approaches (Shin et al.) 2017] also employ generative adversarial networks with promising results. second, more direct approach preserves subset of the training data for each task in an episodic memory and reuses to constrain the learning process of future tasks (Lopez-Paz Papez-Paz&oee et al. (017)). However, while being effective in reducing catastrophic forgetting in general, both approaches have shortcomings as the inherent problem of catastrophic forgetting is simply shifted to a scalability problem. In generative approaches samples for all previous tasks must be replayed each time to preserve old parameter states and as the number of tasks increases this becomes problematic. Similarly for the direct approach, even if only a small subset of examples for each task is preserved, still we can end up with large dataset as the number of tasks increases.
Dynamic Architecture Methods. Another way address catastrophic forgetting is to incremen- tally increase the capacity of the architecture. Approaches vary mainly in whether new capacity is added for each new task by default, or whether this is determined by a metric. Progressive Neural Networks (Rusu et al., (016) add a new network for each new task and each new network is con- nected via lateral connections to the old ones to allow for transfer from previous tasks to the current one. This avoids catastrophic forgetting by design but as each new task requires new network this approach does not scale well with the number of tasks. In contrast to Progressive Nets other approaches only add capacity when it is necessary. Part & Lemon (2016) present an approach based on Self- Self-Orranizing Map, which employs a similarity metric to determine whether a new node should be added to the network. Similar to this, Xiao al.) (2014) start out with classifier with one super class and add new parameters, based on an error signal. Depending on the error made by the current model, only the final layer is extended by another output dimension, or a whole new sub-network is to
added as a subelass. Yooon et al. (2018) use the combination of sparsity and breadth-first-seat determine which parameters should be retrained for the current task. If the features learned so far are not able to represent the new task, more capacity is added dynamically (as in Xiao et al. (2014)) While these methods suffer significantly less from scalability issues, their main disadvantage lies in the fact that they have very stringent architectural constraints, which cannot be easily transferred to any arbitrary existing model.
CONCLUSION
This paper addressed catastrophic forgetting within a continual learning framework where the ulti- mate goal lies in the identification of the network weights that are important to previously learned tasks. While previous work this direction is either limited in the achievable accuracy (as it only considers the diagonal of the Fisher Information Matrix) or limited in number of tasks (as they need to store information that grows linearly with the number of tasks) we set out to provide a first ap- proach that uses second-order parameter dependencies with constant space complexity. We exploit the fact that most regions in the loss surface are flat, which allows us to use only a small subset the Hessian as it holds enough relevant information. We then use a Hessian- cietiaa-veeoo-peee vector-product to sample from this subset. This way, we can incorporate the importance of individual weights and include dependencies between the parameters when we train the network over a long task sequence.
We evaluated our algorithm on three widely used benchmarks and compared it with state of the art Our results show that we consistently outperform EWC across all benchmarks and that we are better or at least en par with Kronecker- factor approximation, while our method at the same time requires significantly less memory.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proc. European Conf. Computer Vision (ECCV), pp. 139-154, 2018.
Jonathan Baxter. A model of inductive bias learning. J. Artifiicia Intelligence Research, 12:149-198, 2000.
Matthieu Bray, Esther Koller-Meict Pascal Müller, Luc Van Gool, and Nicol N Schraudolph. 3d hand tracking by rapid stochastic gradient descent using a skinning model. Proc. 1st European Conf. Visual Media Production (CVMP), 2004a.
Matthieu Bray, Esther Koller-Meier Nicol N Schraudolph, and Lue Van Gool. Stochastic meta- descent for tracking articulated structures. In Proc. Intl. Conf. Computer Vision and Pattern Recognition (CVPR), 2004b.
Dumitru Erhan, Pitore-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect unsupervised pre-training. Artificial Intelligence and Statistics, pp. 153-160, 2009.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv:/805.09933, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proc. 34th Intl. Conf. Machine Learning, pp. 1126-1135, 2017.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proc. 36th Intl. Conf. Machine Learning, pp.
2019.
Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. Proc. 6th Intl. Conf. Learning Representations (ICLR), 2013.
Matteo Hessel, Hado van Hasselt, Joseph Modayil, and David Silver. On inductive biases in deep reinforcement learning. arXiv preprint arXiv:1907.02000 02908, 2019.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czamecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In Proc. 5th Intl. Conf. Learning Representations, ICLR, 2017.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: 17.0. 10368, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barvinska. et al. Over- coming catastrophic forgetting in neural networks. Proc. National Academy of Sciences, 114(13): 3521-3526, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka huszár: The clastic weight consolidation penalty is empirically valid. Proc. National Academy of Sciences, E2241)
et al. Reply to
2018.
Sang-Wor Woo Lee, Jin-He HinHwwa Kim, Jachyun Jun, Jung-Wer Woo Ha, and Byoung- Tak Zhang, Overcoming catastrophic forgetting by incremental moment matching. In Proc. Advances in Neural Informa- tion Processing Systems (NeurIPS), pp. 4652-4662, 2017.
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 6467-6476, 2017.
David JC MacKay, practical bayesian framework for backpropagation networks. Neural compu- fation, 4(3):488 472, 1992.

---

Under review as a conference paper at ICLR 2020
James Martens. Deep learning via hessian bessian-free optimization. In Proc. Intl. Conf. Machine Learning (ICML), pp. 335-742, 2010.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
TM Mitchell. The need for biases in learning generalizations (rutgers computer science tech. rept. cbm-tr-117). Rutgers University, 1980.
Manfred Opper and Ole Winther. A bayesian approach to on-line learning. On-line learning in neural networks, pp. 363-378, 1998.
Jose I. Part and Oliver Lemon. Incremental on-line learning of object classes using combination of self-organizing incremental neural networks and deep convolutional neural networks. In Proc. Workshop Bio-inspired Social Robot Learning in Home Scenarios, 2016.
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 3738 -748,, 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 140,60466,. 2016.
Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1723-1766 2002.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 2990-299099 2017.
James Hardy Wilkinson. The algebraic eigenvalue problem, volume 662. Oxford Clarendon, 1965.
Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, Yuxin Peng, and Zheng Zhang. Error-driven incre- mental learning in deep convolutional neural network for large-scale image classification. In Proc. 22nd ACM Inil, Conf. Multimedia, pp. 177-186, 2014.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Proc. 6th Intl. Conf. Learning Representations (ICLR), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proc. 34th Intl. Conf. Machine Learning (ICML), 2017.
10


Reviews:
[{"review_id": "rkelpoAaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on alleviating the problem of \"catastrophic forgetting\", exhibited by neural networks learned with gradient-based algorithms over long sequence of tasks. In such learning scenarios, tuning of parameters over the new tasks lead to degradation of performance over the old tasks as the parameters important for the latter are overwritten. The gradient-based algorithms are unable to distinguish between the important and  the not-so-important parameters of the old tasks. Hence, one direction of works, including the proposed one, aim at identifying the most important parameters for all the old tasks and discourage modifications on those parameters during the training of the new tasks. \n\nExisting works like Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) have proposed a Bayesian framework to lessen such forgetfulness by condensing the information of the previous tasks and supplying it as a prior for the new task. In such a framework, Ritter et al. (2018) propose a quadratic approximation of the prior which requires computing (an approximate block-diagonal Kronecker-factored) Hessian. \n\nThe paper employs a recent result (Ghorbani et al., 2019) to argue that most regions of the loss surface are flat. Hence, computing the Hessian in only a few regions (which exhibit high curvature) should suffice. However, computing the exact Hessian for large networks is infeasible in practice. The paper, therefore, uses Hessian-vector-product (Schraudolph, 2002; Pearlmutter, 1994), which is similar to sampling the curvature in the direction of a given vector. The key advantage of the proposed approach is the low storage requirements. Regarding how to chose a suitable direction/vector, the paper suggests two choices: the momentum vector or the eigenvector corresponding to the largest eigenvalue (of the Hessian). The  motivation behind the above choices, especially the former option, is unsatisfactory. Empirically, we observe that the momentum vector is a better option than the eigenvector. However, a (theoretical/empirical) deep-dive into why momentum vector is a good candidate should be done. \n\nEmpirically, the proposed approach with momentum vector performs better than EWC but worse than Ritter et al. (2018). More discussion into the results (esp. Hv-momentum vs Hv-eigenvector) would have shed more light on the proposed approach. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklI585CtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for tackling catastrophic forgetting. Similar to previous methods such as EWC (Kirkpatrick et al., 2017), they penalize parameter updates that align with the Fisher information matrix of the previous tasks. This will prevent the model from changing the previously useful parameters. They try to match the result of previous fisher-based methods but at a lower computational cost. They propose using a low-rank approximation to the Hessian using Hessian-vector-product with two types of vectors: the momentum velocity vector and the largest eigen-vector of the hessian. Then they build a diagonal approximation to the Hessian.\n\nCons:\n- Eq 11, there is no justification for forming a curvature matrix by putting the absolute value of the hessian-vector-product with the proposed vectors on the diagonal. Particularly considering the largest eigen-value, Hv will be a vector of zeros with exactly one 1. This does not seem to be a good estimate of the hessian.\n- Fig 1, the proposed method seem to perform poorly compared to the kfac-based method on permuted mnist.\n- Figure 2 mainly compares to EWC as a baseline. In Farquhar & Gal (2019), other methods such as VGR perform significantly better. The proposed method is not competitive with state-of-the-art.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlV6dostB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary:\nThe paper considers neural network training in the continual learning setting -- data arrive sequentially and we can not revisit past data. The paper proposes an approximate Laplace’s method, in which the Hessian the log likelihood of the data is approximated by some form of Hessian-vector project (? - I will get to this question mark below). The paper considers some benchmark continual learning datasets and compares the proposed approach to EWC and Kronecker-factored online Laplace. The performance of the proposed approach is similar to that of EWC and worse than Kronecker-factored Laplace in most cases. Another sales pitch that the paper brings up a lot is the low space complexity, however this benefit has not been fully demonstrated, given the small-scale network/experiments.\n\n2. Opinion and rationales\n\nI’m leaning towards “strong reject” as I think the presentation needs another round of polishing and that the technical contributions need to be clarified / unpacked. I explain my thinking below.\n\na. The presentation/explanation/flow are not clear.\nThe abstract does not read well. For example: “This requires to calculate the Hessian around a mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.” This sentence makes it sound like current approaches are tractable, so what this paper is trying to address? The technical summary is also not precise, the Hessian-free methods used in the paper is to compute Hessian-vector products, not the actual Hessian.\n\nThe introduction motivates the continual learning problem using generalisation of neural networks leading to the need for multi-task learning; however multi-task learning is not scalable given the large number of tasks and thus we need to learn sequentially. However, I find this motivation not clear: if multi-task learning and its scalability issue are the reasons why we need continual learning, with the scale of the experiments considered in the paper, wouldn’t it always more beneficial to use multi-task learning instead of continual learning?\n\nThe prior work section is also not clear, in my opinion. The paper starts out by describing EWC as Bayesian updates and cites MacKay (1992), then talks about the Kronecker-factored Laplace approximation as “address this shortcoming by adopting the Bayesian online learning approach”, as if these methods are very different while in fact, these methods are some variants of the Laplace approximation, with different ways to approximate the Hessian. The issues described in section 2.2 “two problems that stem from eq 1” are not very clear, for example, “without storing the information from all previous tasks there is no easy solution to update the posterior” (?). I would follow the presentation/explanation in Ritter et al (2018), Huszar (2018) [a note on the quadratic penalty of EWC] and section 5 of the variational continual learning paper (Nguyen et al 2018) to provide a more succinct connection between these methods.\nThe connections between this work and MAML in section 3 is not clear to me. The continual learning and meta learning settings are also quite different.\n\nb. The technical contribution is not clear and if correct, if of limited novelty.\n\nWhat is not clear from reading section 3 is what quantity is being approximated, at what point a Hessian-vector product appears and thus we can use Hessian-free methods to approximate it. The paper talks about flat loss surface and sampling a small subset of the Hessian -- I’m not sure I understand these connections. In eq 11, the paper replaces the Hessian values with results of the Hessian-vector-product approximations -- this seems very odd to me, especially in terms of semantics and units, Hessian and hessian-vector-products are two very different things. Again, it is perhaps just me not understanding what is being approximated in the first place. The technical contribution of this paper is thus limited: using Hessian-free methods to approximate Hessian-vector products in the continual learning context.\n\nc. The performance of the proposed method is not super exciting. Pragmatically speaking, it is not clear why practitioners should be using this in the near future given Kronecker-factored Laplace works and scales well in practice and there are a plethora of other recent methods (e.g. VCL) that are also developed from the Bayesian principle and work much better than EWC.\n\n\n3. Minor details:\n\na. In eq 1, the denominator should be p(D_{t+1} | D_{1:t}).\n\nb. Figs 1 and 2, I would use the same colour scheme throughout to be consistent.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have provided thorough reviews of your work. I encourage you to read them carefully should you decide to resubmit it to a later conference.
Decision: Reject