Unifying Regularisation Methods for Continual Learning
Authors: Frederik Benzing
Venue:  (2021)
Keywords: Continual Learning
Regularisation
Fisher Information

Abstract: Continual Learning addresses the challenge of learning a number of different distributions sequentially. The goal of maintaining knowledge of earlier distributions without re-accessing them starkly conflicts with standard SGD training for artificial neural networks. An influential method to tackle this are so-called regularisation approaches. They measure the importance of each parameter for modelling a given distribution and subsequently protect important parameters from large changes. In the literature, three ways to measure parameter importance have been put forward and they have inspired a large body of follow-up work. Here, we present strong theoretical and empirical evidence that these three methods, Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI) and Memory Aware Synapses (MAS), all approximate the Fisher Information. Only EWC intentionally relies on the Fisher, while the other two methods stem from rather different motivations. We find that for SI the relation to the Fisher -- and in fact its performance -- is due to a previously unknown bias. Altogether, this unifies a large body of regularisation approaches. It also provides the first theoretical explanation for  the effectiveness of SI- and MAS-based algorithms and offers theoretically justified versions of these algorithms. From a practical viewpoint, our insights offer computational speed-ups and uncover conditions needed for different algorithms to work.

Under review as a conference paper at ICLR 2020
UNIFYING REGULARISATION METHODS FOR CONTIN- UAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual Learning addresses the challenge of learning a number of different distri- butions sequentially. The goal of maintaining knowledge of earlier distributions without re- re-accessing them starkly conflicts with standard SGD training for artificial neural networks. An influential method to tackle this are so-called regularisation approaches. They measure the importance of each parameter for modelling given distribution and subsequently protect important parameters from large changes. In the literature, three ways to measure parameter importance have been put for- ward and they have inspired a large body of follow-up work. Here, we present strong theoretical and empirical evidence that these three methods, Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI) and Memory Aware Synapses (MAS), are all related to the Fisher Information. Only EWC intentionally relies on the Fisher, while the other two methods stem from rather different motivations. We find that for SI the relation the Fisher and in fact its performance is due to a previously unknown bias. Altogether, this unifies a body of regularisation approaches. provides better explanations for the effectiveness of SI- and MAS- based algorithms and offers more justified versions of these algorithms. From a practical viewpoint, our insights uncover conditions needed for different algorithms to work and allow predicting their relative performance in some situations as well as offering improvements.
INTRODUCTION
Despite considerable progress, many gaps between biological and machine intelligence remain. Animals, for example, flexibly learn new tasks throughout their lives, while at the same time maintaining robust memories of previous knowledge. This ability conflicts with traditional training procedures for artificial neural networks, which overwrite previous skills when optimizing new tasks (MeCloskey & Cohen 1989, Goodfellow et al., 2013). The field of continual learning is dedicated to mitigate this crucial shortcoming of machine learning. It exposes neural networks to sequence of distinct tasks. During training new tasks, the algorithm is not allowed to revisit old data, but should retain previous skills while remaining flexible enough to also acquire new knowledge.
An influential line of work to approach this challenge was introduced by Kirkpatrick et al. (017)) 7). who proposed the first egguarisation-aagoriih a task, EWC measures the importance of each parameter and introduces an auxiliary loss penalising large changes in important parameters. Naturally, this raises the question of how measure this importance'. While EWC uses the diagonal Fisher Information, two alternatives have been proposed: Synaptic Intelligence (SI, Zenke al. (2017)) aims to attribute the decrease in loss during training to individual parameters and Memory Aware Synapses (MAS, Aljundi et al. (2018) ()) introduces heuristic measure of output sensitivity. Together, these three approaches have inspired many further regularisation- based approaches, including combinations of them (Chaudhry al. 2018), refinements (Huszár, 2018 Ritter et al.) 2018) Chaudhry et al.| 2018 Yin et al. (2020), extensions (Schwarz et al.
Elastic Weight Consolidation (EWC). After training
2018 Liu al. 2018 Park al. 2019 Lee et al. 2020) and applications in different continual learning settings (Aljundi et al. (019)) as well as different domains of machine learning (Lan et al.
2019). Almost every continual learning method compares to at least one of EWC, SI and MAS.
Despite their popularity and influence, basic practical and theoretical questions regarding these algorithms had previously been unanswered. Notably, it was unkown how similar these importance

---

Under review as a conference paper at ICLR 2020
measures are. Additionally, for SI and MAS (and their follow-ups) there solid theoretical understanding of their effectiveness. Here, we close both these gaps through a theoretical analysis confirmed by a series of carefully designed experiments standard continual learning benchmarks. Our main findings can be summarised as follows:
(a) We show that MAS is almost identical to the 'Absolute Fisher', quantity similar to the Fisher Information.
(b) We show that SI's importance approximation is biased, that the bias responsible for SI's performance and that the bias, like MAS, is tightly linked to the Absolute Fisher.
(c) Our fine-grained analysis leads to new baselines, including more justified versions of MAS and SI.
(d) Together, (a) and (b) show that all three regularisation approaches (and their follow-ups) are closely linked to the Fisher Information. This unifies a large body of regularisation literature. It also gives more solid theoretical justification for the effectiveness of SI- and MAS-basel algorithms.
(c) Our precise understanding of SI allows predicting and improving its performance in different situations and offers better performing alternative.
2 RELATED WORK
The problem of catastrophic forgetting in neural networks has been studied for many decades (McCloskey & Cohen 1989 Ratclifi 1990 French 1999). In the context of deep learning, received more attention again Goodfellow et al. [0133] Srivastava et al. 2013). All previous work we are aware of proposes new or modified algorithms tackle continual learning. No attention has been directed towards understanding or unifying existing methods and we hope that our work will not remain the only effort of this kind. We now review the broad body of continual learning algorithms. Following (Parisi et al. 2019), they are often categorised into regularisation-, replay- and architectural approaches. Most regularisation methods will be reviewed more closely the next section. There we cover regularisation for standard neural nets, but the same idea has also been applied to bayesian neural networks (Nguyen et al., 2017| Swwrropp et al., 2019). An alternative approach (Mirzadehi let al.) 2020) does not penalize deviating from important parameters, but rather modifies training so that the network is robust to such deviations. Replay methods refer to algorithms which either store small sample generate data of old distributions and use this data while training new methods (Rebuffi et al. 2017 Lopez-Paz & Ranzato 2017 7, Shin et al. 2017: Kemker & Kanan 2017). These approaches can be seen investigating how far standard I.i.d.-trinnnng can be relaxed towards the (highly non-i.i.d.) continual learning setting without losing too much performance. They are interesting, but usually circumvent the original motivation of continual learning maintain knowledge without accessing old distributions. Intriguingly, the most effective way to use old data appears be simply replaying i.e. mimicking training with i.i.d. batches sampled from all tasks simultaneously (Chaudhry al.) 2019). Architectural methods extend the network as new tasks arrive (Fernando et al., 2017: Li et al. 2019; Schwarz et al., 2018 Golkar et al., 2019 von Oswald et al. 2019). This can be seen study of how old parts of the network can be effectively used to solve new tasks and touches upon transfer learning. Typically, it avoids the challenge of integrating new knowledge into existing networks. Finally, yaan de Ven & Tolias (2019): (Hsu et al. (2018); Farquhar & Gal (2018) point out that different continual learning scenarios and assumptions with varying difficulty were used across the literature
REVIEW OF EXISTING REGULARISATION METHODS
3.1 FORMAL DESCRIPTION OF CONTINUAL LEARNING
In continual learning we are given K datasets Di...., Dx sequentially. When training a neural net with N parameters 0 N on dataset DE. have access to the previously seen datasets Di:k-1- However, at test time the algorithms is tested on all K tasks, usually the average accuracy is taken as measure of the algorithms performance, but see also Chaudhry et (0018)
In the appendix, we critically review and question some of their experimental results.
2

---

Under review as a conference paper at ICLR 2020
3.2 COMMON FRAMEWORK FOR REGULARISATION METHODS
Regularisation based approaches introduced in Kirkpatrick et al. (017): Huszár (018) protect previous memories by modifying the loss function Lx related dataset Dj. Let us denote the parameters obtained after finishing training on task by ((k) and let w(k) IR be the parameters' importances. When training on task k, regularisation methods use the loss
N La La + Σ Ww!-ii
where e is hyperparameter. The first term L& the standard (e.g. cross entropy) loss of task The second term ensures that the parameters do not move away too far from their previous values. Usually, (1) = w (k-1) + Wi, where w; is the importance for the most recently learned task k.
3.3 REVIEW OF EWC, MAS, SI AND FOLLOW-UPS
Elastic Weight Consolidation (Kirkpatrick et al. 2017) uses the diagonal of the Fisher Information importance measure. It is evaluated the end of training given task. To define the Fisher Information Matrix F, we omit the task index k and simply consider a dataset X. For each datapoint X X the network predicts a probability distribution qx over the set of labels where we suppress dependence of qx on for simplicity of notation. We denote the predicted probability of class by 4x (v) for each y E L. Let assume that we minimize the negative log- likelihood log qx (v) and write g (, v) = 9 for its gradient. Then the Fisher Information F is given by 20
F Ex~xERBB ax ()x,y)
Ex~x Σ Exx(v) (ec()))(X,,)eee g(X,u)9(Xxu)
Taking only the diagonal Fisher means wi w; (FWC) = Ex [X, The Empirical Fisher Information replaces the expectation over 9x by the (deterministic) label y given by the labels of the dataset. We also define the Predicted Fisher Information as taking the maximum-likelihove predicted label, i.e. the argmax of 9x (-). When the network classifies images correctly and confidently, these three versions clearly become increasingly similar.
Under the assumption that the learned label distribution 9.x is the real label distribution the Fisher equals the Hessian of the loss (Martens) 2014 Pascanu & Bengio 2013). Its use for continual learning has bayesian, theoretical interpretation (Kirkpatrick et al. 2017
Huszár 2018).
Memory Aware Synapses [Aljundi et al. 2018 heuristically argues that the sensitivity of the func- tion output with respect a parameter should be used as the parameter's importance. This sensitivity is evaluated after training given task. The formulation in the paper suggests using the predicted probability distribution to measure parameter sensitivity. The probability distribution the output of the neural network and the quantity that one aims to conserve during continual learning. Below we describe the precise importance this leads to. was however brought our attention that the MAS codebase seems to interpret the logits rather then the probability distribution as output of the neural network. In Appendix [ we show that the resulting version of MAS has the same performance as the one described here, and, erucially, is also related to the Fisher Information. Denoting, as before, the final layer learned probabilitics by qx, the MAS importance is imple- mented as (using euclidean norm)
a 9x a0;
(MAS) = Ex-x
Synaptic Intelligence (Zenke et al., 2017) approximates the contribution of each parameter to
the decrease in loss and uses this contribution importance. To formalise the 'contribution of parameter', let us denote the parameters at time t by f(t) and the loss by L(t). If the parameters follow smooth trajectory in parameter space, can write the decrease loss from time 0 7 as
((T) @L(t)
N
P.(T)) al(t)
L(0)- L(T) = 0(0))
((()) = Σ
(.(0) 00 (t())dt.
(2)
00
3

---

Under review as a conference paper at ICLR 2020
The i-th summand in can be interpreted as the contribution of parameter 8; the decrease in loss. While we cannot evaluate the integral precisely, we can use a first order approximation to obtain the importances. To do so, write (t( (0,(t + 1) (t))) for approximation of (t) and get
(,(ST) = aL() (t)).
00,
Thus, we have (:(S)) x L()) - L(T). In addition, SI rescales its importances as follows
max f0, (SI)
wx(SI)= (0,(T) - (0))²
Zenke et al. (2017) use additional assumptions to justify this importance. One the them using Tull batch gradient descent - is violated in practice and we will show that this has important effect.
Follow up work. We presented a version EWC due to Huszár (2018) and tested in (Chaudhry et al. 2018 Schwarz et al. 2018). It is theoretically more sound and was shown perform better. [Chaudhry ct al. 2018) combine EWC with a 'KL-rescaled' -SI. Ritter et al. (2018) use a block-diagonal (rather than diagonal) approximation of the Fisher; Yin et al. (2020) use the full Hessian matrix for small networks. Liu et al. (2018) rotate the network to diagonalise the most recent Fisher Matrix, Park et al. (2019) modify the loss of SI and and Lee et al. (2020) aim to account for batch-normalisation..
MEMORY AWARE SYNAPSES (MAS) APPROXIMATE ABSOLUTE FISHER
Here, we explain why despite its different motivation - MAS is almost linearly dependent of the diagonal Absolute Fisher, a variant the diagonal Fisher taking absolute values of gradients rather than squares. We first present the theory and then carry out careful empirical confirmations of the underlying assumptions. For a summary of the definitions of algorithms and baselines see Table B.I In Appendix D we present analogous results for MAS based logits rather than the predicted probability distribution, showing that it performs equally and, crucially, is also related to the Fisher.
4.1 THEORETICAL RELATION OF MAS AND ABSOLUTE FISHER
We take closer look at the definition of the importance of MAS. Recall that we use the predicted probability distribution of the network to measure sensitivity rather than logits. With linearity of derivatives, the chain rule and writing y/ arg max qx. we see (omitting expectation over X X)
0l|l| 30
dq(v)
Eq(v))
log ((v))
MAS:
2
2 Σ WEY
2J 2 (30)
= 2yo
(5)
30
00
00
Here we made the assumption that the sum is dominated by its maximun-likelihood label yo. which should be the case if the network classifies its images confidently, i.c. q(30) >> q(v) for y & yo- Recall that the importance is measured the end of training, so that our assumption is justified if the task has been learned successfully. Using the same assumption for the Absolute Fisher we obtain
Abs. Fisher: Ey-ax [| (X v) (1)))19( (X, vo)] = q(20) Ooo q(8o)
00
This shows the similarity between MAS and the Absolute Fisher. The only difference is a factor of 2qx (vo), which we can assume to be approximately constant as the model learned to classify training images confidently. Even with a pessimistic guess that qx (yo) in range of 0.5 (rather inconfident) and 1.0 (absolutely confident), the two measures would be highly correlated.
4.2 EMPIRICAL RELATION OF MAS AND ABSOLUTE FISHER
Our theoretical derivation used two approximations, both which we now explicitly check empiri- cally on standard benchmarks; see Section 7 for summary of our experimental setup.
Note that the max(0,)) is not part of the description in Zenke et al. 2017). However, we needed to include it to reproduce their results. A similar mechanism can he found in the official SI code.
4

---

Under review as a conference paper at ICLR 2020
A
CIFAR Task r=0.99
CIFAR Task r=0.99
Task (Split CIFAR)
MASM
MAS MASK
Task (P-MNIST)
MAS
MAS
Figure 1: Empirical Relation between MAS and baselines MASX, Absolute Fisher. Left: Sum-
mary of Pearson Correlations (top: CIFAR, bottom: MNIST), error bars represent std over 4 runs. Mid & Right: Example scatter plots of importance measures, Each point in the scatter plot corre- sponds to one weight of the net. A straight line through the origin corresponds to two importance measures being multiples of each other as was predicted by our theoretical analysis.
Likeliidld Approximation First, in equation we assumed that sum over the labels is dom- inated by its maximum likelihood label yoo To assess the validity of this, directly compare the entire sum (MAS) with the part only using the max-likelihood label (MASX). The results in Figure ] show scatter plots and summary of the pearson correlations between MAS and MASX on each task. On P-MNIST the correlations range from 0.95 to 0.99, on CIFAR they are always above 0.99. This shows that MAS and MASX are nearly equal as claimed.
Absolute Fisher Approximation Second, we assumed that the factor 2qx (vo) distinguishing MASX (RHS of equation 5) and Absolute Fisher roughly constant. Recall that even 4xx (30) varied between 0.5 and 1.0, the two measures would have large correlations. Figure shows that we are far above this pessimistic regime with correlations between MASX and AF always above 0.98, showing that MASX approximates AF. The data also shows that the resulting correlation between MAS and AF is large (between 0.9-0.97 on MNIST, >0.99 on CIFAR), confirming that MAS approximates AF
Continual Learning Performance Experiment As additional check of our hypothesis, we implemented continual learning baselines based on MAS, MAS-MAX, AF and compared their performances. The results in Table show that all baselines have the same performance, further supporting the claim that MAS is effective because it approximates the Absolute Fisher.
SYNAPTIC INTELLIGENCE APPROXIMATES (ONLINE) ABSOLUTE FISHER
Here, we explain why the importance of SI related the Absolute Fisher, despite the apparent contrast between SI's path integral and the Fisher Information. The theoretical part first identifies the bias of SI when approximating the path integral. then relates the bias to the Absolute Fisher. As before, we carefully design baselines to validate each assumption of our analysis empirically.
5.1 BIAS OF SYNAPTIC INTELLIGENCE
To calculate wS1) (equation we need to calculate the product p = aL(t) 09
A(t for each t.
is too expensive, so SI uses a stochastic minibatch gradient. The estimate of p biased since the same minibatch used for the update A and the estimate of aL/00. We now give the calculations detailing this argument. For ease of exposition, let us assume vanilla SGD with learning rate is used. Given a minibatch, denote its gradient estimate by g + a, where g àL/00 denotes the full gradient and a the noise. The update is A g a. Thus, p should be p = g. (g + a). However, using g + a, which was used for the parameter update, estimate g results in Pbins (g + o² Thus, the gradient noise introduces bias of IB[o² + ag] B[
87 Evaluating the full gradient 20
Unbiased SI. Having understood the bias, we can design an unbiased estimate by using two independent minbatches to calculate A and estimate g. We get A = g + and an estimate + o' for g with independent noise a'. We obtain Pro_bias = (g + o') (g + a) which in expectation equals p g. (g + o). Based this we define an unbiased importance measure
T-1
5. (SIU) = Σ
(g: + + - A())-
1=0
5

---

Under review as a conference paper at ICLR 2020
Bias-Only version of SI. To isolate the bias, can take the difference between biased and unbiased estimate. Concretely, this gives an importance which only measures the bias of SI
T-1 Σ (lo - (g + of)) ) A(t).
3 (SBB)
Observe that this estimate multiplies the parameter-update (t) with nothing but stochastic gradient noise. From the perspective of the SI path-integral, this should be meaningless and perform poorly, Our theory, detailed below, predicts differently.
5.2 RELATION OF BIAS OF SI TO ABSOLUTE FISHER
The bias of SI depends on the optimizer used. The original SI-paper (and we) uses Adam (Kingma & Ba 2014) and we now analyse the influence of this choice in detail. We discuss the effects other optimizers, including SGD++moenntum in Appendix F the theoretical and empirical findings are analogous to the ones presented here. Recall that (SI) is sum over terms áL(!) At)), where
(tt) (( + 1) 0(t) is the parameter update at time t. Both terms, aL(t) 09
80 as well as A(t), are computed using the same mini-batch. Given a stochastic gradient 9t + Jj, Adam keeps an exponential average of the gradient m, (1-B)) +,,) ) + m, as well as the squared gradient e = (1 - -3: + 22 12 Ignoring minor normalisations and the learning rate, the parameter update is (()) = m²/ ma/(vet + c), with 31 = 0.9 and 822 = 0.999. Thus,
aL(t) (t)) (1-A) (1) + 0.) + B1 + a.)mm-1 (1 - B1) (1- (9c + o,)² 00
Voz+ +
Vur +E
o +
Here, we made Assumption 1 that the gradient noise is larger than the gradient, or more precisely: (1- )o 31m4-19: (we ignore 0, since E0 = 0 and since we average SI over many time steps). Assumption is equivalent to the bias of SI being larger than its unbiased part as detailed in Appendix G.I Experiments in Section provide strong support for this assumption.
Next, consider Ut. It is slowly moving exponential average of (g: + ox)² and is thus approximately [[(ge + )² is reasonable to expect that voi is roughly equal to [gt + ozl. This clearly does not hold for all possible distributions of 9: + dt, but is unlikely to be violated by non adversarially
chosen, realistic noise distribution. With equation 7 and Assumption 2 that vvv- RJ (gt + ozl, we get
((SI) = Σ aI((t) A (t)
(ge o.)2 = (1 31) Σ 19: + tzl.
20 31)
Σ
(8)
a0
Ve
An alternative way to arrive at equation 8 is following the analysis of Balles & Hennig (2018), who show that the update of Adam often depends on the sign of the moment m, rather than its magnitude.
Summary. equation g shows that SI approximates the Online Absolute Fisher, which averages the (empirical) Absolute Fisher 'online" along the training trajectory of the parameters rather than evaluating it only the final parameter-point. We point out one subtle difference between equation g and AF, namely that equation 8 takes absolute values of mini-batch gradients rather than single-image gradients (like AF). In Appendix G.2 we show why this is negligible under Assumption 1 of large gradient noise. Empirically, we also find it to be negligible, as we show below that version of EWC based on the batched empirical Fisher has the same performance EWC itself.
Relation of OnAF to AF. To see if SI, like MAS, is related to the Absolute Fisher (AF), it remains to explore the relation of OnAF and AF, If the latter two are similar, is clear from the above that SI is similar to AF. Theoretically, in the limit of small learning rates or small Hessian (of the loss), OnAF and AF become equal. While neither of these assumption strictly hold in practice, we empirically observed to a possibly surprising extend strong correlations between AF and OnAF (Figure (.2). This observation still raises the question whether OnAFs relation to AF is not decreased by gradients early in training. Indeed, present two experiments in line with this intuition in Appendix E showing that putting more weight on later gradients can improve performance.
Influence of Regularisation The derivation above assumes that the parameter update A(t) is given by the gradients of the current task, ignoring the auxiliary regularisation loss. The latter will change update direction and momentum of the optimizer. Thus, strong regularisation will make the relation between SI and OnAF noisy, but not abolish it. This will be confirmed empirically in Section 5.3

---

Under review as a conference paper at ICLR 2020
Table 1: Test accuracies on Permuted-MINIST and Split CIFAR 10/100. (Mean and std of average accuracy over 3 resp. 10 runs for MNIST resp. CIFAR) Each part compares known algorithms to baselines, explaining algorithms' performances (1-3), or showing speed-ups at equal performance (4).
No.
Algo.
P-MNIST
CTFAR
No.
Algo.
P-MNIST
CIFAR
MAS
97.3=0.1
73.7=0.5
SI
97.2-0.11
74.4::0.5
(1)
MAS-MAX
97.4=0.2
73.7=0.2
(3)
OnAF
97.3+0.1
74.4=0066
AF
97.4+0.1
73.400.4
AF
97.4=0.1
73.4+0,4
SI
97.2.1.11
74,410.5
Fisher (EWC)
97.110.3
73.1
(2)
SI-Bias
97.2+0.1
75.1=044
(4)
Emp. Fisher (EF)
97.1=0.1
73.0+0.5
SI-Unbiased
96.3+0.1
72.5=0.8
Batch-EF
97.1=0.1
73.0
5.3 EMPRICAL INVESTIGATION OF SI, ITS BIAS AND ABSOLUTE FISHER
Bias dominates SI & checking Assumption 1. According to the motivation of SI the sum of impor-
tances (over parameters) Σ, We (SI) should track the decrease in loss L(0) - L(T), see equation 2 Therefore, we investigated the summed importances for SI, its unbiased version SIU and for an approximation of the path integral based on the full training set gradient (rather than approximating it). The results, Figure 2 (left) show: (I) the bias is 5-6 times larger than the unbiased part; (2) using an unbiased gradient estimate and the full gradient gives almost identical sums, supporting the validity of the unbiased estimator. Note that even the unbiased first order approximation of the path integral overestimates the decrease in loss. This is consistent with previous studies showing that the loss has positive curvature (Jastrzebski et al. 2018: Lan et al., 2019). We emphasise that the bias (difference between SI and SIU) in Figure 2 (left) is due to the term (1 - 3i)o² and that this bias is considerably larger than the unbiased part. This is direct, strong evidence of Assumption (see Appendix G.I for full calculation). Additionally, a detailed analysis of the gradient noise in Appendix H gives further support for Assumption Results on CIFAR are analogous (Appendix Figure J.6
Bias explains SI's performance. We saw that the bias is much larger than the unbiased part. But how does it influence SI's performance? To quantify this, we compared SI to its unbiased version SIU and the bias-only version SIB. Note that SIB is completely independent of the path integral motivating SI, only measures gradient noise and therefore should perform poorly according to SI's original motivation. However, the results in Table 2 reveal the opposite: Removing the bias reduces performance of (SIU worse), whereas isolating the bias does not affect or slightly improve performance (difference between SIB and SI on CIFAR is statistically significant, p < 0.005 two-sided (-test). This demonstrates that SI relies on its bias for its continual learning performance.
Bias and Fisher & Checking Assumption 2. As additional experiment, we checked how much the relation of SI to the Fisher depends on the bias. Our theory predicts that the relation between SI and OnAF is mostly based on the bias. Figure 2 (mid) confirms this, showing strong correlation between SI, SIB and OnAF and a weaker correlation between SIU and OnAF. Note that the correlation of OnAF and SI clearly supports Assumption 2; results on CIFAR are similar (c.f. Figure J.1 but with a stronger influence of regularisation as seen in Figure (right) and full text below.
Relation of SI, OnAF AF. Next, we investigated the relation between OnAF and AF, finding strong correlations: between 0.73-0.87 on MNIST and 0.86-0.95 on CIFAR, see Figure J.2 Figure (mid panel for MNIST, right panel for CIFAR, blue) shows that this carries over to relating SI and AT An additional control in Appendix J.2.1 shows that this observation is not an artefact of long training times. These results confirm that SI is closely related to AF.
We clarify that here we used SI importances before the division in equation 4 We show in Appendix that the division has close to no effect on the correlations of with AF. We also show in Appendix[Edte that putting more weight on latter summands of SI, limiting the influence of carly gradients, slightly improves performance, indicating that making SI more similar to AF at the final parameter point is beneficial.
Effect of Regularisation on SL. Next, we assessed our derivation that large regularisation will diminish the correlation between SI and OnAF. To this end, compared standard SI to two variants of it which have less strong regularisation. The first control simply sets regularisation strength to = 0. The second control refrains from re-initialising the network weights after each task (exactly as
7

---

Under review as a conference paper at ICLR 2020
Summed Importances
SI Ewith biast SIU (unblesed) (ull gradient
Correlation lereeiation
8
Pearson Correlation
8 1 8
Pearson Carre
On4F 51.OnAF
S10
Skin Init)GGAA
Siin inti-Ar
SU--F
SRi O
S((=O))AF
108
2016
4000
Training Iteratioo
Task (P:MNIST)
Task (Split CIFAR)
Figure 2: Effect of Bias and Regularisation on SI. Left: Summed Importances for SI and its unbiased version, showing that the bias dominates SI and that Assumption holds. Mid: Pearson Correlations of SI, its bias (SIB), and unbiased version (SIU) on MNIST, showing that relation between SI and (On)AF due bias. Right: Relation between standard-SI (blue) and OnAF and two Sl-controls: 'no init' (turquoise) does not re- re-initialise network weights after each task; "c = O' (purple) has regularisation strength O. This shows that strong regularisation weakens the tie between SI and AOn)AF as explained by our theory.
in original SI, albeit with slightly worse validation performance than the version with re-initialisation). In the second setting the current parameters 0 never move too far from their old value ((k-1), implying smaller gradients from the quadratic regularisation loss, and also meaning that smaller value of e = 0.5 is optimal. We see that for both controls with weak regularisation the relation to OnAF is larger than for standard-SI with strong regularisation (Figure (right)), consistent with our theory.
Empirical, batched Fisher We demonstrated that SI is related to the OnAF and pointed out that OnAF uses the Empirical Fisher and averages across batches before taking absolute values. Here, investigate replacing the Fisher Information in EWC by the Empirical Fisher and averaging it across a mini-batch. We refer to this algorithm Batch-EF. A theoretical analysis (Appendix G.2 suggests that with moderate batch size, Batch-EF is close to EF. Empirically, we find that EF, Batch-FF and EWC have the same performance, see Table (4), confirming the theoretical prediction. this context, we note that on P-MNIST the number of samples used to approximate the Fisher and analogous quantities has some surprising effects on performance. See C.I.1 for details.
RELATION BETWEEN ABSOLUTE FISHER AND REAL FISHER
It remains to investigate the relation of Absolute and Real Fisher. Recall that (taking expectations over inputs and predicted labels) the diagonal Fisher is defined as = Eilg + o while the diagonal Absolute Fisher is defined as AF Bllg + orI,, where
is the (stochastic) gradient.
Empirically, we see that AF and F are highly correlated, see Figure left.
Theoretically, the exact relationship of Fisher and Absolute Fisher depends on the gradient distribution. If, for example, gradients are distributed normally (g o)) ~ N(u, ,, Σ) with Σj.; >> 14; (corresponding to the observation that the noise is much bigger than the gradients), we obtain F, ~ Σj,; and (since lg + a follows a folded normal distribution) we also have AF, R ev E, with c = /2/m. n. Thus, in the case of normal distribution with large noise, we have F X (AF)², The relationship ox (AF)² is in fact what we empirically observe on Split CIFAR, where correlations between (AF)² and F are 0.98 for all tasks and repetitions, and partly on P-MNIST, where correlations between (AF)² and F are between 0.86 and 0.96.
10a
Fober (EWC
So-- yn
Aciua you
1.0 CITAR)
20a
Fister
Figure 3: Empirical Relation between Absolute Fisher and Real Fisher. Left: Note y-axis range. MNIST), error bars represent std over 4 runs. Mid Right: Example scatter plots. Relation on Split CIFAR nearly quadratic across tasks and repetitions. There is more variation on P-MNIST (two extremes shown). Yellow lines show least-squares fit of the model fx) = az² with parameter a.
Summary of Pearson Correlations (top: CIFAR, bottom:
8

---

Under review as a conference paper at ICLR 2020
7 EXPERIMENTAL SETUP
We outline the experimental setup, see Appendix C for details. We closely follow SI's setting Zenke et al. 2017): In domain--nccremenal Permuted MNIST (Goodfellow al. 2013) each of 10 tasks consists of a random (but fixed) pixcl-permutation of MNIST and a fully connected ReLU network used. The task-incremental Split CIFAR 10/100 ZZekee et al.) 2017) consists of six 10-way classification tasks, the first is CIFAR 10, and the other ones are extracted from CIFAR 100. The keras default CIFAR 10 convolutional architecture (with dropout and max-pon poollnng is used keras The only difference to Zenke et al. (2017) is that like (Swaroop et al. (2019), we usually re- re-initialise network weights after each task, observing better performance. Code available on github. For brevity, we partly showed plots for only one benchmark, see appenidx for analogous plots.
8 DISCUSSION
We have investigated regularisation approaches for continual learning, which are the method of choice for continual learning without replaying old data or expanding the model. We have provided strong theoretical and experimental evidence that the three most influential methods in this domain are surprisingly similar. This does not only unify these three methods, but also their many follow ups. While the Absolute Fisher has no clear theoretical interpretation itself, its similarity to the Fisher provides a more plausible explanation of the effectiveness of MAS- and SI based algorithms.
Moreover, our algorithm SIU to approximate SI's path integral can used for the (non continual learn- ing) algorithm LCA (Lan et al., 2019). which relies on a computationally expensive approximation of the same integral. This opens up new opportunities to apply LCA to larger models and datascts.
Practical Relevance and Experimental Predictions. Some may wonder what the use of under- standing algorithms is. On top of the intrinsic theoretical benefit of understanding similarities between methods, our framework allows to predict in which situations algorithms will perform well and how they can be improved. Let us give one concrete example, that is relevant as distributed large batch training becomes more widespread: Our analysis reveals that SI is related to the Fisher through its bias, which is caused by gradient noise. Thus, increasing the batchsize, which decreases noise, will harm SI's performance according to our theory. We tested this specific prediction and found it to be true: With a batchsize of 2048 on P-MNIST, SI's performance degrades to 95.6 = 0.1, while OnAF (with the same, large batchsize) retains 97. 0 = 0.1. On Split CIFAR, SI degrades to 70.0 + 0.9, while OnAF retains 74.5 = 0.6. We emphasise that this phenomenon is not simply explained by the change in 'training regime" (see e.g. Mirzadeh et al. 2020)), since SI, as predicted by our theory, is affected much more than OnAF by the increase in batch size. To summarise, our theory not only predicts degradation of SI correctly but also offers well-performing alternative.
We now describe other predictions to exemplify the use of our framework and leave testing these and discovering further ones to future work. A commonly used optimisation technique is learning rate decay. Its effect on SI is that it puts more weights on those summands of SI's importance (see equation 3.) which correspond to larger learning rates. Large learning rates occur early in training, thus making SI's sum less similar to the AF at the end of training 5 Thus, we predict that learning rate decay has negative effect on SI. In fact, we present two experiments in Appendix B showing that putting more weight on the latter summands of SI is beneficial, in line with our prediction. Interestingly, learning rate decay may be the reason that a recent empirical large-scale comparison of regularisation methods (De Lange et al. (2199) tended to observe worse performance of SI than MAS. Another somewha hat puzzling observation made by De Lange et al. (2019) is that SI is more sensitive than other methods to the order in which datasetsets are presented, in cases where dataset sizes vary. In these experiments, smaller datasets are trained for fewer iterations (due to a fixed number of epochs), meaning that the sum of SI (think of OnAF) contains few summands, and is systematically smaller than for larger datasets. Thus, when first learning small datasets, SI will likely forget these since they have small importances. Starting with large datasets will not have this effect and we would expect Si to perform better. This is precisely the empirical finding in (De Lange et al.] 2019).
So far, studies providing better understanding of regularisation algorithms and their similarities had been neglected. Our contribution fills this gap and, as exemplified above, proves to be a useful tool to predict, understand and improve these algorithms.
IIn the original framework of ST this intended, while our framework predicts that is harmful.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Implementation ((OSD/) 16), pp. 265-283, 2016.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139-154, 2018
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp- 11254-11263, 2019.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pp. 404-413, 2018.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European Conference on Computer Vision (ECCV), pp. 332-547, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv: 10086 2019.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv: arXiv:1909.0880 2019.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv:/805.09733. 2018
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: [td::700 1701.0834, 2017.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 (4):128-135, 1999.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv: 044776 2019.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv: 1312.6211, 2013.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: categorization and case for strong baselines. arXiv preprint arXiv: 12488, 2018.
Ferenc Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, pp. 201717042, 2018
Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint 171iv:1711.0000 2017.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. arXiv preprint arXiv: arXiv: 1007,0533, 2018
10

---

Under review as a conference paper at ICLR 2020
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv preprint arXiv: 2017.
keras. Official keras cnn example. https:///githii coneerasscco//eaas---aaaaaa httm:///rrrssttt//aaaaa///ool htt
master//xxmplee//cii
cnn. py Accessed: 2020-09-25.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint urXiv:/4/2.6980 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnkk---Biiiik,a..r. et al. Overcoming catastrophic forgetting in neural networks. P Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical fisher approxima- tion, arXiv preprint arXiv: /905. /2558, 2019.
Janice Lan, Rosanne Liu, Hattie Zhou, and Jason Yosinski. Lea: Loss change allocation for neural network training. In Advances in Neural Information Processing Systems, pp. 3614-3624 2019.
Janghyeon Lee, Hyeong Gwon Hong. Donggyu Joo, and Junmo Kim. Continual learning with extended krotecker-factored approximate curvature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9001-9010, 2020.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. arXiv preprint 2019.
Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2262-2268. IEBE, 2018.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
James Martens. New insights and perspectives on the natural gradient method. arXiv preprint orXiv:/412.114 2014.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation Advances in Research and Theory, 24(C):109-165, January 1989. ISSN 0079-7421. doi: 10011///000777777700086666055550
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understand- ing the role of training regimes in continual learning, 2020.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv: 10628, 2017.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Dongmin Park, Seokil Hong, Bohyung Han, and Kyoung Mu Lee. Continual learning by asymmetric loss approximation with single-side overestimation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3335-3344 2019.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint 100.:41301.3584, 2013.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
11

---

Under review as a conference paper at ICLR 2020
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp. 3738-3748, 2018.
Nicolas L Roux, Pierre-Antoine Manzagol, and Yoshua Bengio. Topmoumoute online natural gradient algorithm. In Advances in neural information processing systems, pp. 849-856, 2008.
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki, Agnieszka Btrabsk-BBawwinss Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning, arXiv preprint
2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Jürgen Schmid- huber. Compete to compute. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 2310-2318. Curran Associates, Inc., 2013. URL 5059 httes---ommee-------mueeeeeeepp
www..
Siddharth Swaroop, Cuong V Nguyen, Thang D Bui, and Richard E Turner. Improving and under- standing variational continual learning. arXiv preprint arXiv: 19Xiv:1905.00 1905.02099, 2019.
Valentin Thomas, Fabian Pedregosa, Bart Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics, pp. 3503-3513. PMLR, 2020.
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning, arXiv preprint 07734, 2019.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv:1906.0000 2019.
Dong Yin, Mehrdad Farajtabar, and Ang Li. Sola: Continual learning with second-order loss approximation, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
This appendix has several components:
1. A tabular overview over algorithms and baselines described in the paper, B
2. Full experimental details including hyperparameter seraches and values, as well as full details about Batch-EF, Appendix C
3. Performance and Relation to Fisher of MAS based on logits, Appendix
4. Two improvements of SI, Apprendix E
5. A more detailed account of the influence of the optimizer on SI, Apprendix
6. Details for two calculations omitted in the main paper, Appendix G
7. We empirically investigate the gradient noise in Appendix This experiment is not specific to continual learning.
8. We critically review experimental claims from previous work about regularisation methods in Appendix!! !
9. We include additional experiments and plots in Appendix J rescaling SI affects its relation to AF.
including investigations how
B OVERVIEW OVER ALGORITHMS AND BASELINES
Table B.I: Summary of Regularisation Methods and Related Baselines. Details: Algorithms on the left calculate importance 'online' along the parameter trajectory during training. Algorithms on the right calculate importance at the end of training a task by going through (part of) the training set again. Thus, the sum is over timesteps (left) datapoints X (right). N is the number of images over which is summed. All the algorithms on the left rescale their final importances as in equation equation 4 for fair comparison. ( O( 1) 0(t) refers the parameter update at time t, which depends on both the current task's s loss and the auxiliary regularisation loss. Moreover, (9t + ox) refers to the stochastic gradient estimate of the current task's loss (where 9: the full gradient and a the noise) given to the optimizer to update parameters. In contrast, (ge + o') refers to an independent stochastic gradient estimate. For a datapoint X, 4x denotes the predicted label distribution and g(X, y) refers to the gradient of the negative likeliool of (X,
Paramater importance w(+)
Paramater importance w(-)
Name
Name
SI
Σ ge (t))((t)
Fisher (EWC)
Ey-ax [o(X,))
SIU (SI-Unbiascd)
Σ (a. +t))((t)
AF (Absolute Fisher)
By (lx(X,)))
N
0 q
12
SIB (SI Bias-only)
Σ (ar -1))(())
MAS
Σ
N
00
a(max 9x
OnAF (Online AF)
Σ Ie de
MASX (MAS-Max)
Σ
N
00
C EXPERIMENTAL DETAILS
C.1 DETAILS, VARIANTS EWC, MAS AND AF.
For EWC we calculate the 'real' Fisher Information as defined in the main article For P-MNIST, we randomly sample 2000 training images (rather than iterating through the entire training set, which is prohibitively expensive). For Split CIFAR we use 500 random training images.
"We use a 'nalve' implementation to calculate the Fisher, which simply iterates over all potential labels. Theoretically, this implementation could be improved, but in practice this is infeasible since the necessary computations are currently not well-supported by standard deep-learning libraries; see Kunstner et al. (2019) Appendix for details.
13

---

Under review as a conference paper at ICLR 2020
Similarly, the Empirical Fisher is based on 2000 (500) samples.
For Batch-EF, we also use 2000 (resp 500) samples and split them into 20 (resp 5) mini-batches of size 100. We compute the average gradient for each mini-batch, square and average the squares over the mini-batches. We also multiply the result by the mini-batch size (100) for normalisation. For theoretical justification see G.2
MAS, MAS-X and AF algorithms are based on 1000 resp 500 random samples. When comparing these algorithms, we use the same set of samples for each algorithm.
When calculating the correlation between SI and AF, we use 30000 (MNIST) resp 500 (CIFAR) samples to evaluate AF. This is because observed higher variance MNIST for AF, see Figure 1.2 and subsection below.
C.1.1 EFFECT OF SAMPLE SIZES USED TO CALCULATTE FISHER
On P-MNIST, we observed for EWC that using more samples to approximate the Fisher could make performance more unstable. We believe that this is due to the following: When passing one image through a fully connected ReLU network, there will be some 'dead' neurons with zero activations. All weights connected to these neurons will have gradients equal to O, and thus Fisher Information (for this single image) equal to O. When more samples are used, less neurons will be 'dead' and less weights will have Fisher Information equal to O. While this leads to a better approximation of the Fisher, changing importances to non-zero values takes away flexibility of the network for latter tasks. Empirically, this flexibility seems to help performance on P-MNIST. We explicitly checked how the number of weights with O importance depends on the number of samples used: With 1000, 2000, 4000 resp 8000 samples the number of zero importances after the first task was approximately 2M, 1. 1.8M, 1.5M resp 1.3M (out of total 5.6M weights). We confirmed that this result consistent across random seeds.
We note that with convolutional nets, this phenomenon of zero-importance is much less likely to appear since each weight is applied to many different neurons, which are unlikely all be 'dead', In line with this, we found that AF varies much less with sample size on the CIFAR task (with convolutional net) than MNIST (with fully connected net), see green line J.2
The fact that for EWC, MAS(X), AF there are many weights with 0 importance is likely the reason that for these methods the optimal hyperparametrrsation do not re-initialise network weights after each task (see Table C.2) unlike for all other settings
the importance a weight is zero, re-initialising it, will move it far from its original value with no reason to return. If instead, it is not re-initialised it is more likely to not move far away during training consecutive tasks (even if there's 's no direct incentive from the loss function to avoid this).
C.2 BENCHMARKS
In Permuted MNIST (Goodfellow al. 2013) each task consists of predicting the label of random (but tixed) pexelppermutation of MNIST. We use total of 10 tasks. As noted, we use a domain incremental setting, i.c. a single output head shared by all tasks.
In Split CIFAR10/100 the network first has to classify CIFARIO and then is successively given groups of 10 (consecutive) classes of CIFAR 100. As in (Zenke et al., 2017). we use tasks in total. Preliminary experiments on the maximum of 11 tasks showed very little difference. We use task-incremental setting, i.c. each task has its output head and task-identity is known during training and testing.
C.3 PRE- PROCESSINNI
Following Zenke et al. 2017). we normalise all pixel values to be in the interval [0, 1] (i.e. we divide by 255) for MNIST and CIFAR datasets and use data augmentation. We point out that pre- pre-processing can affect performance, e.g. differences in pre- preccescessing are part of the reason why results for SI on Hommuted-MNIST in (Hsu et al.) 2018) are considerably worse than reported in (Zenke et al. 2017) (the other two reasons being an unusually small learning rate of 10 for Adam in (Hsu et al. 2018) and slightly smaller architecture).
14

---

Under review as a conference paper at ICLR 2020
C.4 ARCHITECTURES AND INITIALIZATION
For P-MNIST we use a fully connected net with ReLu activations and two hidden layers of 2000 units each. For Split CIFAR CIFAR10/100 we use the default Keras CNN for CIFAR 10 with convolutional layers and two fully connected layers, dropout (Srivastava et al. 2014) and maxpooling, see Table We use Glorot-uniform Glloot & Bengio 2010) initialization for both architectures.
C.5 OPTIMIZATION
We use the Adam Optimizer (Kingma Ba 2014) with tensorflow (Abadi et al. 2016) default settings (Ir = 0.001, 11 0.9, 132 0.999, < T0-s and batchsize 256 for both tasks like Zenke ctt al. (0177). On Permuted-MNIST we train each task for 20 epochs, on Split CIFAR we train each task for 60 epochs, also following Zenke al. 2017). We reset the optimizer variables (momentum and such) after each task.
For the two experiments confirming prediction regarding the effect of batchsize on SI, made two changes on top of using a batch size Of 2048 (and, naturally, re-tuned hyperparameterss) On P-MNIST increased the learning rate by factor of 8, since the ratio of batchsize and learning rate thought to influence the flatness of minima (e.g. Jastrzbstz et al. (2017); Mirzadeh et al. (2020)) We did not tune try other learning rates. On CIFAR, we tried the same learning rate adjustment but found that it led to divergence and resorted back to the default. We observed that with a batch size of 2048, already on task 2, the network did converge during 60 epochs (for tasks 2-6, one epoch corresponds to only 5 parameter updates with this batch size). We therefore increased the number of epochs 600 on tasks 2-6, match the number of parameter updates of Task (CIFAR10).
C.6 SI & ONAF DETAILS
Recall that we applied the operation max(0,)) .) (i.e. a ReLU activation) the importance measure of each individual task (equation 4 of main paper), before adding it to the overall importance. In the original SI implementation, this seems to be replaced by applying the same operation to the overall importances (after adding potentially negative values from a given task). No description of either of these operations is given in the SI paper. light of our findings, our version seems more justified.
Somewhat naturally, the gradient aL not the total loss including regularisation. For CIFAR we evaluated this gradient without dropout (but the parameter update was of course evaluated with dropout) F
80 usually refers to the cross-entropy loss of the current task and
For OnAF, similarly to SI, we used the gradient evaluated without dropout on the cross-entropy loss of the current task for importance measure.
When evaluated on benchmarks, SI, SIU, SIB and OnAF are all rescaled according to equation 4 to make the comparison fair as possible. Scatter plots and correlations use values before this diviston, unless noted otherwise.
C.7 HYPERPARAMETERS
For all methods and benchmarks, we performed grid searches for the hyperparameter e and over the choice whether or not re-initialise model variables after each task.
The grid of e included values a 10 where E (1,2, 5, and i was chosen in a suitable range (if hyperparameter close to a boundary of our grid performed best, we extended the grid).
For CIFAR, we measured validation set performance based on at least three repetitions for good hyperparameters. We then picked the best HP and ran 10 repetitions on the test set. For MNIST, we measured HP-quulity on the test set based on at least 3 runs for good HPs.
Additionally, SI and consequently SIU, SIB as well as OnAF have rescaled importance (c.f. equation equation ffrom main paper). The damping term & in this rescaling was set to 0.1 for MNIST and to 0.001 for CIFAR following Zenke al. (2017) without further HP search.
We did not check the original SI code for what's done there and the paper does not describe which version used.
15

---

Under review as a conference paper at ICLR 2020
Table C.I: CIFAR 10/100 architecture. Following Zenke al. (2017) we use the keras default architecture for CIFAR 10. Below, 'Filt." refers to the number of filters of convolutional layer, or respectively the number of neurons
fully connected layer. 'Drop.' refers to the dropout rate.
'Non-Lin." refers to the type of non-linearity used.
Table reproduced and slightly adapted from (enke et al. 2017
Layer
Kernel
Stride
Filt.
Drop.
Non-Lin.
3x 3x33x32 input
Convolution
3x3
1x1
32
ReLU ReLU
Convolution
3x3
1xl
32
MaxPool
2x2
2x2
0.25
Convolution
1x1
ReLU ReLU
Convolution
3x3
1x1
64
MaxPool
2x2
2x2
0.25
FC
512
0.5
ReLU
Task 1: FC
10
softmax softmax
...:FC
10
Task 6: FC
softmax
Table C.2: Hyperparameter values for experiments. See also maintext.
MNIST
CIFAR
Algorithm
re-init
re-init
SI
0.2
5.0
SIU
2.0
2.0
SIB
0.5
V
5.0
OnAF
5e-5
V
5e-4
MAS
X
200
MASX
200
X
1e3
EWC
1e3
5e4
AF
200
le3
Emp. Fisher
le6
X
2e7
Batch-EF
5c5
X
2c7
All results shown based on the same hyperparameterss obtained individually for each method as described above. They can be found in Table We note that the difference between the HPs for MAS and MASX might seem to contradict our claims that the measures are almost identical (they should require the same in this case), but this is most likely due to similar performance for different HPs and random fluctuations. For example on CIFAR, MAS had almost identical validation performance for 200 (best for MAS) and 1000 (best for MASX) (74.2 + 0.7 vs 74.1 = 0.5).
Also the other methods, we observed that usually there were two or more HP-configuration which performed very similarly. The precise "best' values as found by the HP search and reported in Table C.2 are therefore subject to random fluctuations the grid search.
C.8 SCATTER PLOT DATA COLLECTION AND INTENSITY PLOTS
The scatter plots are based on run of SI, where we in parallel to evaluating the SI importance measure also evaluated all other importance measures. Figures and show data based on repetitions of this experiment and confirm that the correlations observed in the scatter plots are representative, also shows the effect of sample sizes on AF.
Intensity Plots (as used for some figures in the appendix) Show same kind of data as scatter plots, but each weight is binned into one of 80x50 equispaced bins. The number of weights per bin is divided by the maximum number of weights per bin in that column and the resulting value is shown on gray scale. Normalisation per column by total number of weights) only weights in the bottom left are visible (as most weights have small importances); the number of bins chosen match aspect ratio of plots.
performed since for 'global normalisation' (divide
16

---

Under review as a conference paper at ICLR 2020
C.9 COMPUTING INFRASTRUCTURE
We ran experiments on one NVIDIA GeForce GTX 980 and on one NVIDIA GeForce GTX 1080 Ti GPU. We used tensorflow 1.4 for all experiments.
D MAS BASED ON LOGITS
As pointed out in 3.3 there are two versions of MAS. Here, we describe results related the version based on logits. We note that this version has the potentially undesirable feature of not being invariant to reparametrisations: If, for example, we add constant e to all the logits, this does not change predictions (or the training process), but it does change the (logit-based) MAS importance.
We found identical performance as for the other version of MAS (of course after re-tuning HPs): 97.2 + 0.1 P-MNIST and 73.9 + 0.7 on CIFAR for logit based MAS, as compared to 97.3 + 0.1 and 73.7 0.5 for the version based on the difference based on 10 repetitions
output. For both benchmarks, the not statistically significant (p > 0.4, t-test).
Next, we investigated the relation between logit based MAS, MASX and the Absolute Fisher. On CIFAR we found correlations very close for all pairs. On MNIST, the correlations between MAS and MASX (based on logits) to AF are weaker than the ones for the version of MAS based on the predicted distribution. Nevertheless, they are similar the correlation of SI and AF we observed on MNIST and above the correlations between SI and AF we observed on CIFAR. The results are shown in Figure D.I
Task (Split CIFARI
Pearson Corr.
MAS--F
Task (P-MNIST)
Figure D.1: Empirical Relation between MAS and baselines MASX, Absolute Fisher. Same as Figure left, but using MAS based on logits rather than output distribution.
E IMPROVEMENTS OF SI
As we have argued in the discussion, we expect SI to be a more faithful approximation of AF if more weight is put on the summands of SI (see equation 3 which occur later in training. As consequence, putting more weight on latter summands could lead better performance. We tested this prediction by running two variants of SI. For the first one, only considered summands occuring in the second half of training (epochs 10 to 20, of 20). For the second one, we used an exponential moving average with deacy of 0.999. Both variants achieved average accuracy on P-MNIST of 97.5% which is better than SI's 97. 2%. This confirms our prediction.
F INFLUENCE OF OPTIMIZER ON SI
As pointed out in the main paper, the bias of SI depends on the optimizer used. For SGD+momentum, one can follow precisely the same steps as for Adam (simply ignoring the division by its second moment estimate) to see that in this case SI will resemble the Online Fisher, i.c. (assuming learning rate of 1.0)
((SI) R (1 -1) Σ (gt + 0,)²
17

---

Under review as a conference paper at ICLR 2020
Analogously Adam, in this case the relation of SI to the Fisher is due to the bias of SI. Thus, checking whether in this situation, too, the unbiased version SIU performs worse than SI, is another validation of our theory. Indeed, on P-MINST, an average of 97.4%, while the best run for SIU was at 96.1 1%. We did not test SGD further, due to instabilities described now: For both SI, SIU using SGDD+ resulted in unstable hehaviour, with runs occasionally diverging. This is likely due to the fact the the regularisation loss is ill- conditioned, since the importances span several orders of magnitudes. This suggests that, without further tricks and efforts, adaptive optimisers are more suitable choice for regularisation methods.
found that SI with SGD and momentum achieves
For other ootmisers, the bias of SI will take different shapes and how this affects performance will have to be seen, Our theory suggests. that biasing SI to (a version of) the Fisher will help its performance, while we predict that fundamentally different biases lead SI having worse performance than e-g. OnAF. An instructive example in this case is considering approximation of natural gradient descent, which preconditions with the inverse diagonal of the empirical fisher (rather than the inverse fisher, which is expensive to compute). In this case, SI would have constant importances (across parameters) and would likely perform badly.
G DETAILED CALCULATIONS FOR THE BIAS OF SI, AND FOR CALCULATING FISHER IN BATCHES
Here give the details for two calculations omitted in the main text.
G.1 FORMULA BIAS OF SI WITH ADAM
Here, we show that the bias, i.e. the expected difference between SI and SIU, is equal to (1 31 )a? ([Vo e) and that Assumption equivalent to the bias being larger than the unbiased part of SI.
Recall that for SI, we approximate Adam. So we get
0L(t)
by 9t + oi, which is the same gradient estimate given to 00
Ol(1) 00
A()) 31- + 0,)²
Bi + o1)m--1
SI:
Voz +
vt +
For SIU, we use an independent mini-batch estimate ge of for aLL)) 80
and therefore obtain
AT(1)
(t)) = (1 - (1) + o + o1) + 31 + m1)m- 00
SIU:
v +
Vor +
Taking the difference between these two and ignoring all terms which have expectation zero (note that [[or]] [[o']] and that ae, of are independent of m and gi) gives
SI SIU:
(1-B1) B1) Vee +E
as claimed.
To see that Assumption I is equivalent to the bias being larger than the unbiased part, note that in expectation SIU equals
(1 (1-1)) B1)
o
+ B1
g:m
2 B1
m
Ve, +
Vu +c
lur +
The last approximation here is valid because 31m-11 (1 B1 B1)9: which holds since (1) 31 > [[]ge]]- (1) follows from 31 = 0.9, and (2) holds since E1m;-1] gr
(1 - 31) and (2) E[[m,-11 I|] 2 and since m4-1 has noise component (due to gradient noise), while gi is deterministic quantity, implying Ei[m,~1ll 2 Elgg:]] by elementary considerations.
In particular, the two caleulations for the bias and the expectation of SIU above imply that Assumption 1, which (1 - o) > 3 m,m 9tt. is equivalent to the bias of SI being larger than its unbiased part.
18

---

Under review as a conference paper at ICLR 2020
G.2 EFFECT OF CALCULATING FISHER IN MINI-BATCHES; ANALYSIS OF BATCH-EF
For this subsection, let us slightly change notation and denote the images by XI,..., X and the gradients (with respect their labels and the cross entropy loss) by g + 01,-.. ... + a Here, again g is the overall training set gradient and o; is the noise (i.e. Σ 0, = 0). Then the Empirical Fisher is given by
D E
Ee
EF = D 1=1
We want to compare this to evaluating the squared gradient over a batch. Let f1.-.., in denote uniformly random, independent indices from {1, ...., D)), so that Xi,..., is random mini- batch size Let + a be the gradient on this mini-batch. We then have, taking expectations over the random indices,
b
Bi(g + +o)
E
Σ
(o
b²
r,a=1
b(b
1)
Blig oi (g o1a)
Bl(g o;
122
12
1
9"
EF
5
b
EF
This conclusion may seem surprising, but recall that it is only based assumption, namely that the gradient noise is considerably bigger than the gradient itself an assumption for which we have collected ample evidence.
This conclusion may seem surprising, but only relies on only two assumptions: (1) We draw a minibatch with i.i.d. elements (i.e. with replacement) and (2) the noise considerably bigger than the gradient.
(1) If we draw minibatch elements without replacement (on the extreme end would be drawing the full batch), then our derivation ignores the contribution of E[o,, ] < O. Note that as long as the batchsize is small (square root of the size of the dataset or smaller) drawing a batch with without replacement has almost the same distribution.
(2) Concretely, we assume [g + o;) > 1)g2 Note that unlike in other sections of this manuscript, here o, refers to the gradient noise of individual images (rather than mini-batch noise). We have seen that even with a batch-size of 256 (i.e. when we reduce then noise by a factor of 256) the noise is still much bigger than the gradient (recall Figures J.5
H.2). so that our assumption
true with room to spare.
We also point out that the relation between the (diagonal of the) covariance matrix of minibatch gradients and the Fisher Information has also been described several other places the literature, e.g. Roux et al. (2008) TTooas et al. (2020); Jastrzębski et al. (2017).
H GRADIENT NOISE
Here, we quantitatively assess the noise magnitude outside the continual learning context. Recall that Figure 2 (left) from the main paper, as well as Figures and alreddy show that the noise dominates the SI importance measure, which indicates that the noise is considerably larger than the gradient itself.
To obtain an assessment independent of the SI continual learning importance measure, we trained our network on MNIST as described before, i.c. ReLu network with 2 hidden layers of 2000 units each, trained for 20 epochs with batch size 256 and default Adam settings. At each training iteration, top of calculating the stochastic mini-batch gradient used for optimization, we also computed the full gradient the entire training set and computed the noise which refers to the squared lz distance between the stochastic mini-batch. gradient and the full gradient as well as the ratio between noise and gradient, measured the ratio of squared l norms. The results shown in Figure H.] In
19

---

Under review as a conference paper at ICLR 2020
rato woongradeers
oode
0.4
00
0.2
1030
2000
3002
4000
1000
2000
2000
Traininng
Figure H.1: Gradient noise, measured as squared lg distance between full training set gradient and the stochastic mini-batch gradient with a batch size 256. 'Full gradient' magnitude is also measured as squared & norm.
Data obtained by training a ReLu network with hidden layers of 2000 hidden units for 20 epochs with default Adam settings. Only every 20-th datapoint shown for better visualisation.
1.-
0.-
0.6
10
30
Figure H.2: y-value shows fraction of training iterations in which the ratio between mini-batch noise and full training set gradient was at least z-value. Data obtained described in main text / Figure HH.] particular the batch-size was 256. 'Ratio" refers to the ratio of squared & norms of the respective values.
addition, we computed the fraction of iterations in which the ratio between noise and squared gradient norm is above a certain threshold, see Figure
I RELATIVE PERFORMANCE OF REGULARISATION METHODS
Several papers (van de Ven & Tolias 2019 Hsu et al., 2018 Farquhar & Gal 2018) played an important role in recognising that different continual learning algorithms had been evaluated in settings with varying difficulties. These papers also greatly clarified these setting, making an important contribution for future work.
Here, we want to point out that some of the results (and thus conclusions) of the experiments in (Hsu al., 2018) (these were the only ones we investigated more closely) are specific to suboptimal settings and hyperparameters used there.
For example, it found there that EWC, MAS, SI perform worse than an importance measure which assigns the same importance to all parameters (called '1.2 regularisation" ) This is in contrast to results reported in the EWC paper (Kirkpatrick et al., 2017). 7). who found the L2 baseline to perform worse than EWC. We also attempted to reproduce the L2 results and tried two versions of this approach, one with constant importance for all tasks and one with importance increasing linearly with the number of tasks. The latter approach worked better, but still considerably underperformed EWC, MAS, SI on Permuted- -NISST (88% vs 97%). We didn't it run these experiments on CIFAR. An alternative importance measure based the weights' magnitudes after training was better, but still could not
20

---

Under review as a conference paper at ICLR 2020
compete with SI, EWC, MAS (<95% vs >97%) showing that these importance measure use- and meaningful than naive baselines.
Additionally, in other work found that for domain incremental settings (again, these were the only ones we investigated), results of basclines EWC and SI, but also fine-tune (a baseline which takes measures prevent catastrophic forgetting) can be improved, often by more than 10% compared to previously mentioned reports (Hsu et al. 2018 van de Ven & Tolias 2019). This also implies that the difference between well set-up regularisation approaches and for example repaly-methods is not nearly as big as previously thought. The differences between these results are explained by different factors (we explicitly tested eeach factor): hyperparameters (for example using Adam with the standard learning rate 0.001 improves performance over the learning rate 0.0001 used previously). initialisation (Glorot Uniform Initialisation works considerably better for SI than pytorch standard initialisation for linear layers, which reveals unexpected defaults upon close investigation), training duration (regularisation approaches usually notably benefit from longer training, this is reported in Swaroop et al. (2019), and we also found that performance on P-MNIST can be improved to 98% by training for 200 rather than 20 epochs; Hsu et al. (2018) use very short training times), data-preph prepooeppio (depending the setting different normalisations have different, non-negligible impacts).
J MORE EXPERIMENTS AND PLOTS
J.1 PLOTS, DATA ANALOGOUS TO FIGURES/TABLES IN MAIN PAPER
Here, we show data for experiments already partly reported in the main paper, including both benchmarks and each task of each benchmark.
We start with the magnitude of the bias of SI (Figure 2 left panel); The observation that most of the SI importance is due to its bias is consistent across datasets and tasks, see Figures p.5 and [.6] Intriguingly, on CIFAR we find that the unbiased approximation of SI slightly underestimates the decrease loss in the last task, suggesting that strong regularisation pushes the parameters in places, where the cross-entropy of the current task has negative curvature.
We show correlations between SI, SIB, SIU and AF, OnAF on CIFAR (analogous to Figure middle) in Figure J.1 The findings are in accordance with the ones on MNIST reported in the main paper. Note that only the first task of CIFAR (i.e. CIFARI0) shows slightly different behaviour, as SIU and OnAF are more similar than
other tasks: correlation with OnAF is 0.994 + 0.001 for SI; 0.991 0.001 for SIU; 0.979 = 0.003 for SIB. The similarity between SIU and OnAF in this situation is explained by the weights with largest OnAF importance: If we remove the 5% of weights which have highest OnAF importance, correlation between OnAF, SIU drops to 0.599 + 0.027, but for SI resp. SIB the same procedure yields 0.977 0.002 resp. 0.965 + 0.003. This indicates that for the first task of CIFAR there is a small fraction of weights with large OnAF importances, which are dominated by the gradient rather than the noise. The remainder of weights is in accordance with our previous observations and dominated by noise, recall also Figure J.6
Pearson Correlation
SI-AF
SI-OnAF SIB-OnAF
6.3-
SIB-AF
0.2-
SIU-AF
SIU-OnAF
0.7
0.5-
05-
8.4
0.3
Task (Split CIFAR)
Figure J.1: Effeet of Bias on SI. Analogous to Figure 2 (middle) but on CIFAR rather than MNIST.
Drop of correlations for tasks 2-6 is due to strong correlation as predicted in Section 5.2 and confirmed in Figure 2 (right).
21

---

Under review as a conference paper at ICLR 2020
We did not conduct the analogue of experiment corresponding to Figure 2 right panel, on MNIST since there the influence of regularisation was already weak there.
J.2 RELATION OF ONAF AND AF
We show summaries of the correlations between OnAF and AF in Figure This includes compar- isons of AF based on different sample sizes. We also investigated the influence training duration on AF, OnAF, next section.
Pearson Correlation
20
3.-
MNIST
34-
CIFAR
OnAF-AF
AF-AF small
Task
Figure J.2: Correlations between AF and OnAF, Solid lines show correlations on MNIST, dotted lines show correlations on CIFAR. "AF" based on 30000 (MNIST) resp. 5000 (CIFAR) samples, while AF_ AFF_mmall' is based on 1000 resp. 500 samples.
J.2.1 INFLUENCE OF TRAINING DURATION ON AF, ONAF
seems plausible that the correlation between AF and OnAF we observed due to training on tasks for long time the model might converge quickly so that the sum of OnAF is dominated by summands which approximate AF close to the final point in training. To test this, we trained the first task of P-MNIST and Split CIFAR10/100
networks for varying numbers of epochs and measured the correlation between OnAF and AF. The results in Tables J.1 J.2 show that the correlation between AF, OnAF does not rely on long training duration. On MNIST, decreasing correlation with longer training time, this may be due to either an actual decrease in correlation due higher variance when estimating AF using 10000 samples (or both).
even see
Table J.1: Correlation between AF and OnAF on CIFAR10 depending on training time. We show mean and standard deviation of Pearson correlation over runs. AF is based 500 samples. If the standard deviation below 0.005 it is shown as 0.00.
NUMBER OF EPOCHS
CORRELATION
0.941 0.01
2
0.95 0.00
4
0.96+ 0.01
0.96+ 0.01
0.97 0.00
30
0.97 0.00
60
0.95 0.01
J.3 ADDITIONAL SCATTER PLOTS AND INTENSITY PLOTS
We show additional scatter and intensity plots in Figure J.7 Note that for both architectures there are few million weights. The scatter plots overerowded and show the whole range of dependencies between two measure that can possibly occur. Intensity plots show the dependencies that the majority of weights adheres to.
22

---

Under review as a conference paper at ICLR 2020
Table J.2: Correlation between AF and OnAF on MNIST depending on training time. We show mean and standard deviation of Pearson correlation over 3 runs. AF is based on 10000 samples. Recall that for 30000 samples and 20 epochs correlation larger.
NUMBER OF EPOCHS
CORRELATION
0.90+ 0.01
0.90= 0.01
0.88 0.01
10
0.78 0.04
0.68 0.05
J.4 ALTERNATIVE COMPARISON OF SI, SIU, SIB TO AF, ONAF
Finally, we point out that our comparisons between SI, SIU, SIB and AF, OnAF are usually obtained
after applying max(0, -) (c.f. equation equation of main paper) to the SI (and SIU, SIB) impor- tances (the algorithms diverge without this operation) and before the division in equation (our theory doesn't make predictions about this division).
Here, we first show that the division in equation 4 does not affect correlations between SI and AF, see Figure J.3
Next, we present results without the max operation (and without division) in L.4 further confirming our analysis that (1) correlation between OnAF and SI weakened by strong regularisation and (2) the correlation is due to the bias of SI.
Pearson Correlation
Pearson Correlation 6
SWAF SIK-AF
SIK-AF
SIAF
(P-MNIST)
Task (Split CIFAR)
Figure J.3: Analogous to Figure 2 (middle)) from main paper, comparing SI and SIN to AF, SI refers to the importance after applying the mar(. 0) operation, but before the division. SIN also includes the division (or Normalisation), see equation
+ 34.0035
- 98.4
Figure J.4: Analogous to Figure 2 (middle) from main paper, comparing SI, SIU, SIB but before the operation m 0) is applied to importances of SI, SIU, SIB. See fulltext. Data from CIFAR tasks. On P-MNIST the operation max(-, 0) has close no effect.
23

---

Under review as a conference paper at ICLR 2020
Task
Task 2
(blased)
SIU (unbia sed)
50
loss
50
1000
2000
3000
4000
1000
2000
3000
4000
Task
Task 4
100
100
50
50
0-
1000
2000
3000
4000
1000
2000
3000
4000
Task 5
Task
100
50
50
0
1000
2000
3000
4000
1000
2000
3000
4000
Task
Task
100
1000
2000
3000
4000
0
1000
2000
3000
4000
Task 9
Task 10
100
100
1000
3000
4000
1000
3000
4000
Figure J.5: Summed importances for all P-MNIST tasks for SI and its unbiased version. Analogous Figure (left) from main paper.
24

---

Under review as a conference paper at ICLR 2020
Task 1
Task
140
SU (untlased)
125
80
60-
20
2000
4000
6000
8000
10000
12500
202
600
860
1000
Task 3
Task 4
10
200
400
60b
A00
1002
700
400
600
800
1000
Task 5
Task 6
100
200
400
60D
800
1000
700
400
600
8C0
1000
Figure J.6: Summed importances for all Split CIFAR 10/100 tasks for SI and its unbiased version. Analogous to Figure 2 (left) from main paper.
25

---

Under review as a conference paper at ICLR 2020
A
CIFAR Task 0w0.99
B
CIFRR Task rw0.99
CIFAR Task re0.95
CIFAR Tosk r-0.95
MAS
CITAR Tesk 0.9.9
CIEAR Tesk 1:0.00
GTAR Tast & r:0.6
GFAR Tass r=
MAS
OnAF
KNST 120.80
MNIST 1:0.9
MNIT Tack
MNIST Task re0
MAS
OnAF
CIFAR Task ru0 99
B
CFAR Task ru 99
CIFAR Taok
CIFAR Teck
rap
MASK
Tank 1=0.69
OHAF
MNS1 Task 5, r-0 98
MUIST Tack r=c
MNIST Task (w0.81
MUST Task rep
COAF
M65
OnAF
Figure J.7: More scatter and intensity plots. Refer to the title of each panel for details.
26


Reviews:
[{"review_id": "uv6pc3EfOy7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Disclaimer: I am not an expert in continual learning even if I have already experimented with EWC, but rather my main expertise is related to FIM in other contexts.\n\nThis works aims at unifying 3 popular regularisation type continual learning methods, namely EWC, SI and MAS, by showing that under some assumptions they all relate to the Fisher Information Matrix. These assumptions are shown to hold on 2 different tasks trained using Adam.\n\nI however found several limitations while reading your paper:\n\n### Limitations regarding MAS\n\nIn the github repo of MAS (it's not entirely clear from their paper), they seem to consider the function $F$ to be the score for each class (i.e. the output of the last linear transformation), while in your paper your $\\mathbf q_X$ is the class probability (i.e. the softmax). So their $\\omega\\left(\\text{MAS}\\right)$ is different from yours.\n\n### Limitations regarding SI\n\nYour argument regarding SI (sec 5.2) seems to be valid only when using Adam, while SGD+momentum is the standard for image classification nowadays. In my opinion you can come up with a more general argument even using standard stochastic gradient algorithm, e.g. the relationship between the FIM and the covariance of the minibatch gradients as already been studied e.g. in [1] and [2]. If your empirical analysis and conclusion that it is the bias that explains SI's performance still holds with SG (without Adam), then the argument is straightforward.\n\nMoreover the denominator $\\theta\\left(T\\right) - \\theta\\left(0\\right)$ of eq. 4 is not discussed later in the text, or in other words, if $\\tilde\\omega\\left(\\text{SI}\\right)$ is similar to $\\omega\\left(\\text{EWC}\\right)$, what about $\\omega\\left(\\text{SI}\\right)$?\n\n### Conclusion\n\nIn conclusion, I really appreciate the effort of trying to unify which is certainly more useful than inventing countless slightly different variants of the very same technique. I would however like to see these points addressed before publication.\n\n\n[1] Le Roux, N., Manzagol, P. A., & Bengio, Y., Topmoumoute online natural gradient algorithm, NeurIPS 2008.\n\n[2] Thomas, V., Pedregosa, F., Merriënboer, B., Manzagol, P. A., Bengio, Y., & Le Roux, N., On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics, AISTATS 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "jyJ_UNtI1U", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThis paper attempts to unify the three most prominent regularization-based continual learning methods: EWC, MAS, and SI.\nWhile EWC has a solid theoretical justification under certain assumptions, the other two are based on intuition and heuristics.\nThe authors show that the importance weights of MAS and SI are similar to the diagonal absolute Fisher.\n\n---\n\n## Pros\n\n- Connecting the path integral of SI to the Absolute Fisher is interesting.\n- The paper is well-organized and easy to follow.\n\n---\n\n## Cons\n\n### Lack of justification for the Absolute Fisher\n\nThe Absolute Fisher seemingly appears from nowhere. In contrast to the diagonal Fisher, it does not seem to have any other interpretation. The authors merely show that MAS and SI are similar to using Absolute Fisher as importance weights. The similarity alone is not enough to be a theoretical explanation for effectiveness. I think the authors should justify that the Absolute Fisher is an optimal regularization under certain assumptions. \n\n### Weak connection between the diagonal Fisher Information and the Absolute Fisher\n\nDespite searching the web, I could not find any proper material on the Absolute Fisher that explains its connection to the diagonal Fisher.\nThe only similarity that I find between the Absolute Fisher and the original Fisher is that they can be computed with the gradient. I do not see any reason to call it the Absolute *Fisher*.\nTherefore, I cannot agree with the claim that this paper presents a unified framework, just because it fits several methods into distinct concepts that have \"Fisher\" common in their names.\n\n### Doubts about the value of a unified framework\n\nI ask the authors a more fundamental question: do we really need a unified framework? Although the derivation of EWC is mathematically grounded, it heavily relies on certain assumptions about the shape of the loss surface. The assumptions make the use of the diagonal Fisher information an optimal choice. However, considering its poor CL performance, those assumptions seemingly do not hold in deep neural networks.\n\nSimilarly, I think other methods, such as MAS and SI, can also be optimal under different assumptions. Therefore, I argue that it is more meaningful to investigate which condition/assumption makes a certain algorithm optimal, rather than framing all algorithms into one unified framework. I suspect that the latter case is not even possible.\n\nThe experiments also support my claim: AF does not necessarily outperform MAS or SI. Instead, the focus of this paper is the correlation among methods. I think the best regularization method varies depending on the specific model architecture and task design.\n\n### Minor issues\n- Assumption 1 in Section 5.2 depends heavily on the batch size. The gradient noise will quickly diminish as the batch size grows.\n\n---\n\n## Overall evaluation\n\nIt is hard for me to agree with this paper's fundamental motivation: a unified framework for regularization methods. Also, the overall logic of this paper is too weak. Since I could not see other utility in this paper, I recommend rejection.\n\n---\n\n## Post rebuttal\n\nIn response to my doubts about a unified framework, the authors claimed that their *theory* could *predict* an algorithm or hyperparameter's performance.\nSince there was no description of the theory, I assumed that the theory is:\n> If an algorithm is different from AF, its performance is expected to be poor.\n\nAnd the authors refuted my interpretation:\n> We claim that SI and MAS work because they are similar to AF.\n\nThe authors claim that the theory somehow applies to SI and MAS but not others. However, I could not find any description of why the applicability is restricted and to what extent it is applicable. I think these are vital parts of a proper theory. Without them, a theory is useless since we cannot decide whether it applies to a new CL algorithm until we actually run some tests.\n\nAlso, I want to emphasize that association is not causation. The authors should have claimed, \"SI and MAS work, **and** they are similar to AF,\" instead of \"SI and MAS work **because** they are similar to AF.\"\n\nEven after the discussion with other reviewers, my concerns are not resolved.\nTherefore, I retain my initial rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WdBTNtD5vym", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper**\n\nThis paper draws links between three common regularisation methods for continual learning: EWC, MAS, and SI. It shows that MAS and SI approximate the Absolute Fisher matrix. The authors provide many experiments to test their claims and assumptions. Finally, the authors also propose a cheaper way to run EWC.\n\n**Review summary**\n\nI really like the majority of this paper. Unifying these regularisation methods is great, and not obvious (particularly in the case of SI). The accompanying experiments are crucial and well-conducted. The paper is also written well, with an emphasis on good research practices. However, I have an issue with the proposed quicker/cheaper update for calculating the (diagonal empirical) Fisher Information Matrix for Online EWC (\"Batch-EF\"), as I detail later. If it were not for this, this paper would be a clear accept for me. I hope to resolve this issue with the authors during the discussion period, depending on which I can raise (or lower) my score.\n\n**Pros of paper** (mostly already written in the \"Review summary\")\n\n1. The paper is written well, with good detail and very good experiments.\n2. The work is of significance for continual learning, with interesting conclusions.\n\n**Cons of paper**\n\n3. I am not convinced that the minibatching that the authors suggest (both for SI as an approximation to the AF and for EWC in the last paragraph of Section 5) is correct (\"Batch-EF\"). It appears to me that by minibatching instead of squaring each gradient element, one should obtain much worse approximations to the empirical Fisher information matrix (\"EF\"). \nIntuition: By calculating the gradient over minibatches and then squaring, one is reducing the noise that is being squared. Intuitively, this must affect the EF calculation in a bad way. For example, consider a full-batch calculation. In this case, Batch-EF will just be $g^2$. At the end of training, when we have converged to a low loss, this will be very small. In Appendix D.2, the authors argue that when gradient noise >> gradient, then Batch-EF $\\approx$ EF, and derive that Batch-EF has larger values than EF. However, I expect Batch-EF to have smaller values than EF because of the reduced noise on average.\nAdditionally, I have myself experimented in the past with Batch-EF. I did not find that Batch-EF gave the same results as EF for EWC on similar benchmarks. I do not know why, in this paper, the authors found that the two gave same results (Table 1); perhaps it only works for specific hyperparameters.\nFinally, a small note that may be of interest to the authors: the HAT codebase (github.com/joansj/hat) implements EWC as a baseline, however, my collaborators and I found that they implement EWC differently/incorrectly. One of the ways they are different is to do Batch-EF (along with other differences).\n4. I am also not convinced that OnAF (\"Online Absolute Fisher\") and AF are / should be the same. After training, individual gradients should be relatively small (as we have converged to a solution), meaning that I would expect the AF to have small values in general. However, during training (especially near the beginning), gradients can be large, meaning that OnAF can end up having large values. Empirically, the Pearson correlations in Figure 2 (mid) show differences between the OnAF and AF versions.\n\n**Additional suggestions to authors**\n\n- The authors could consider adding a reference for the Absolute Fisher (Section 4.1). Can they say anything about the links between the Absolute Fisher and the empirical Fisher?\n- Typo Section 4.2 second para: \"Max-likeilhood\"\n- I felt that Section 5 got complicated, with many algorithms that need to be compared. I strongly recommend splitting the experiments part (Section 5.3) into experiments relevant for the two preceding sections (5.1 and 5.2) to reduce the complexity of writing.\n\n**Update to review**\n\nI am increasing my score from 5 to 6. I believe this paper is a good paper. However, an extremely extensive discussion with other reviewers has left some questions / concerns. Although I disagree with some of these, I agree with others:\n- Some claims are overstated in the paper. The authors already changed these claims somewhat in the updated paper. Some reviewers are arguing for further changes. I think some claims can still be reduced, particularly, the link between AF and EF (and hence the link to EWC).\n- One of the biggest reason I find this paper is interesting is not mentioned (enough) by the authors. In my opinion, this is a big reason why the work is significant, and if I were writing the paper, I would put it as one of the biggest motivations:\n    -  There have been works recently looking at the Generalised Gauss-Newton approximation (= EF for classification), and trying to view optimisation algorithms as approximating the Hessian matrix. For example, see Khan et al., 2018 (\"vAdam\"), Kessler et al., 2020 (\"BAdam\"), Zhang et al., 2018 (\"Noisy Adam\"), Osawa et al., 2019 (\"VOGN\"). Such works provide evidence that different approximations of the EF can work well. Although I am not aware of previous works using the absolute value of gradients (as in AF), this paper provides evidence that such an approximation might be worth considering. Should we try and approximate the Fisher matrix in more ways in CL? \n- Finally, it is my personal opinion (although others disagree) that the current paper is significant enough / provides enough insight already to be a good paper. However, performing a further state-of-the-art experiment or similar would undoubtedly improve the quality significantly.\n\nI very much look forward to an updated version of this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CfXFr3JNcGz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "******************************************************************\n********************** POST DISCUSSION UPDATE **********************\n******************************************************************\nThank you to the authors for the discussion. Given that the relationship between AF and F has now been addressed, I will increase my score. However, since the connection mainly hinges on the empirical correlation between the two, I still don't think that the paper quite establishes the theoretical connection between the three continual learning methods that it aims for. Finally, I'm not quite convinced by the potential impact of the insights, which seem fairly specific to choosing the batch size of SI, so overall I think the paper still is below the acceptance threshold. \n******************************************************************\n************************** END OF UPDATE***************************\n******************************************************************\n\nThe paper unifies two regularisation-based continual learning methods from the literature (Synaptic Intelligence and Memory Aware Synapses) by arguing that they both approximate the \"Absolute Fisher\", a variant of the Fisher Information that averages absolute instead of squared gradients, to determine the regularisation weights. By this the authors claim to establish a relationship between these two approaches and Elastic Weight Consolidation, which approximates the diagonal of the Fisher.\n\nWhile the paper is well-written and -structured, and SI and MAS are popular methods in the literature, I'm not convinced by the link through the \"Absolute Fisher\". The Fisher Information is a well-studied quantity with many appealing statistical properties and even a subtle change to its definition such as replacing the expectation over the labels with the empirical data breaks many of these (see the referenced Kunstner et al. paper). So simply taking absolute instead of squared gradients and stating that this is \"a natural variant\" is not a sufficient basis for a paper, especially considering that the term has not appeared anywhere in the literature before as far as I could tell. I think the authors really need to either provide some references to existing work or establish themselves that this variant makes sense theoretically. So overall my recommendation is to **reject** the paper.\n\nFurther, I'm not really sure what the takeaway from the proposed unification of these methods is even assuming that it can be put on a more solid foundation. Is it that through the relationship to EWC they are all approximately Bayesian? How would this inform future work? I feel like the paper pokes in that direction through empirically comparing some variants of SI and MAS, but most variants perform almost identically and are highly correlated, so again I am not sure what exactly to take away from this.  The part on SI relying on its 'bias' seems potentially interesting, but since -- if I understand things correctly -- this is an unexpected empirical result from the theoretical point the paper is trying to establish, it would be necessary to go a bit more in depth. To summarise, I think the authors need to more clearly explain and establish the impact of their work.\n\nAs a minor final point, I do not find the proposed efficient version of EWC convincing at all. First, estimating the Fisher as the square of the averaged gradients has been done before, for example it is implemented in the [tensorflow KFAC codebase](https://github.com/tensorflow/kfac/blob/3ee1bec8dcd851d50618cd542a8d1aff92512f7c/kfac/python/ops/fisher_factors.py#L945-L950). Second, the empirical comparison does not make much sense, since Jacobians in Pytorch can be calculated efficiently for linear and convolutional layers without too much effort through backward hooks (see e.g. [the backpack library](https://backpack.pt/) for an implementation). In more recent versions, Pytorch also provides autograd functions for computing jacobians natively -- I'm not sure how efficiently they are implemented, but in any case the empirical comparison here needs to use an efficient, batched implementation for calculating the Fisher in order for it to be meaningful. Calculating gradients for individual data points is neither sensible nor necessary.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a unification of three popular baseline regularizers in continual learning. The unification is realized through a claim that they all regularize (surprisingly) related objectives.

The key strengths of the paper highlighted by the reviewers were:
1. The established connection is valuable and interesting, even if weaker than suggested originally
2. Good motivation (unifying different regularization methods is useful for the community)
3. Clear writing

The key weakness of the paper is a weak empirical validation of the claim that these three regularizers work *because* they regularize the norm of the gradient (as mentioned in the discussion by R3). Rather, the key claims are correlational. The authors correctly say that (1) the three regularizers all regularize related objects (namely different norms of the gradient) and (2) they reduce forgetting. However, it is not sufficiently well demonstrated that (1) => (2). Relatedly, given that the paper does not have a very clear theoretical contribution, it would be really helpful to demonstrate utility. It would be useful to extend experiments that apply these insights to developing novel regularizers or improving/simplifying hyperparameter tuning.

Additionally, in the review process, the link was discovered to be weaker than originally suggested. The paper casts the relation in terms of the Fisher Information Matrix, which suggests it is theoretical and sound. After the discussion, it seems that viewing this relationship in terms of the Fisher Information Matrix is somewhat misleading. The three different regularization methods all regularize different norms of the gradient (L1 or L2), which are empirically, and under some assumption theoretically, related. More precisely, EWC regularizes the trace of the *Empirical* Fisher, which is equivalent to the L2 norm of the gradient of the loss function. SI regularizes a term similar to the L1 norm of the gradient. These effects were seen by the reviewers to be somewhat loosely related to the Fisher Information Matrix.

Based on the above, I have to recommend rejection. I would like to thank the Authors for submitting the work for consideration to ICLR. I hope the feedback will be useful for improving the work.
Decision: Reject