Continual learning with neural activation importance
Authors: Sohee Kim
Seungkyu Lee
Venue:  (2021)

Abstract: Continual learning is a concept of online learning along with multiple sequential tasks. One of the critical barriers of continual learning is that a network should learn a new task keeping the knowledge of old tasks without access to any data of the old tasks. In this paper, we propose a neuron importance based regularization method for stable continual learning.
We propose a comprehensive experimental evaluation framework on existing benchmark data sets to evaluate not just the accuracy of a certain order of continual learning performance also the robustness of the accuracy along with the changes in the order of tasks.

Under review as a conference paper at ICLR 2021
CONTINUAL LEARNING WITH NEURAL ACTIVATION IMPORTANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning is a concept of online learning along with multiple sequential tasks. One of the critical barriers of continual learning is that a network should learn a new task keeping the knowledge of old tasks without access any data of the old tasks. this paper, we propose a neuron importance based regularization method for stable continual learning. We propose comprehensive experimental evaluation framework on existing benchmark data sets to evaluate not just the accuracy of certain order of continual learning performance also the robustness of the accuracy along with the changes in the order of tasks.
INTRODUCTION
Continual learning, or sequential learning, is a concept of online learning along multiple sequential tasks. The aim of continual learning is learning set of related tasks that observed irregularly or separately online. Therefore each task does not necessarily contain overlapped classes with oth- ers and, in worst case, different tasks consist of mutually disjoint classes. Therefore, in continual learning, one of the main challenges is training new tasks with new classes without catastrophic for- getting existing knowledge of prior tasks (and their classes). A model adapts to new task without access to some or entire classes of past tasks but keeping and maintaining acquired knowledge from new tasks (Thrun 1996). Since the training of a neural network is influenced more by recently and frequently observed data, a machine learning model forgets what it has learned in prior tasks without continuing access to them in the current task. On the other hand, rigorous methods that maintain the knowledge of entire previous tasks are impractical in adapting new tasks. Thus, researchers have de- veloped diverse methods achieve both stability (remembering past tasks) and plasticity (adapting new tasks) in continual learning.
There are three major types in previous continual learning methods: 1) architectural approaches modifying the architecture of neural networks (Yoon et al. 2017 Sharif Razavian et al. [014 Rusu et al., 20166) 2) rehearsal approaches using sampled data from previous tasks (Riemer et al,
2018 Aljundi et al. 2019 Gupta et al., (220)). and 3) regularization approaches freezing significant weights of a model calculating the importance of weights or neurons (Li & Hoiem| 2017 Kirk-
patrick al., 2017 Zenke al., 2017 Nguyen et al. 2017, Aljundi al., 20188 b Zeno et al. 2018][Ahn et al. 201 [019]Javed & White 2019) Jung et al. 2020). Most recent regularizztion methods have tackled the problem more fundamental way with regularization approaches that utilize the
weights of a given network to the hit. The basic idea of regularization approaches is to constrain essential weights of old tasks not to change. In general, they alleviate catastrophic interference by imposing a penalty on the difference of weights between the past tasks and the current task. The degree of the penalty follows the importance of weights or neurons with respective measurements. Significance of an weight stands for how important the weight is in solving certain task.
EWC (Kirkpatrick et al., 2017) introduces elastic weight consolidation which estimates parameter importance using the diagonal of the Fisher information matrix equivalent to the second derivative of the loss. However, they compute weights' importance after network training. SI (Zenke et al., 2017) measures the importance of weights in an online manner by calculating each parameter's 's sensitivity to the loss change while training a task. To be specific, when a certain parameter changes slightly during training batches but its contribution to the loss is high (i.e., rapid change of its gradient), the parameter is considered to be crucial and restricted to be updated while learning future tasks. However, the accuracy of their method shows limited stability even with the same order of tasks of

---

Under review as a conference paper at ICLR 2021
test data. Unlike Zenke et al. (2017). MAS (Aljundi et al. 2018a) assesses the contribution of each weight to the change of learned function. In other words, it considers the gradient of outputs of a model with mean square error loss. Gradient itself represents a change of outputs with respect to the weights. The strength of the method lies in the scheme of data utilization. It considers only the degree of change of output values of the network, that any data (even unlabeled one) is valid to compute the gradient of the learned function with regard to weights. VCL (Nguyen) ct al., 2017) is a Bayesian neural network based method. It decides weight importance through variational inference. BGD (Zeno et al. 2018) is another Bayesian neural network based apporoach. In this method, finds posterior parameters(e.g. mean and variance) assuming that posterior and the prior distribution are Gaussian. These methods (Kirkpatrick et al. 2017 Zenke et al., 2017] Aljundi et al. 201 Nguyen et al.] 2017 Zeno et al.] 2018)) calculate and assign importance to weights directly as describe in Figure Ta In order to alleviate interference across multiple tasks,
weight importance based approaches such as EWC, SI, and MAS assign importance Q2 to each weight w; in the network. However, in the case of convolutional neural networks, weights in the same convolutional filter map should have the same importance. Furthermore, since those methods basically consider the amount of change of weight, it is impossible to reinitialize weights each training of a new task, which decreases the plasticity of the network. (Additional explanation of weight reinitialization is discussed in section 2.2
a(w))
D(M))
(N
D(ys)
9(%)
((N))
(a) EM-EI
(b) EM-NI
(c) NM-NI
Figure 1: Three different types of Measurementt Importance (a) EM-EI (Edge Measurement, Edge Importance): Weight importance is calculated based on weight measurement. (b) EM-NI (Edge Measurement, Node Importance): Neuron importance is calculated based on weight measurement. Weight importance is redefined as the importance its connected neuron. (c) NM-NI (Node
Mcasurement, Node Importance): Neuron importance is calculated based on neuron measurement. Weight importance is defined as the importance of its connected neuron.
UCL4Ahn et al., 2019) propose Bayesian neural network based method to mitigate catastrophic forgetting by incorporating weight uncertainty measurement indicating the variance of weight distri- bution. They claim that the distribution of essential weights for past tasks has low variance and such stable weights during training a task are regarded as important weight not to forget. As illustrated Figure 16 they suggest a node based importance neural network. First, the smallest variance among the weights incoming to and outgoing from a corresponding neuron decides the importance of the neuron, and then all those weights take the same neurons importance as their weight impor- tance. Ahn AAh hyper-parameters. Furthermore, it is computationally expensive to train compared to Zenke et al.) (2017) and our proposed method. Jung et al. (2020) another recent algorithm based on neuron importance. Its node importance depends on the average activation value. This idea is simple but powerful. Activation value itself is a measurement of neuron importance, and weights connected to the neuron get identical weight importance. This corresponds to the type in Figure Ic
al. (219)) is applicable only Bayesian neural network and highly dependent upon
One of our key observations in prior experimental evaluations is that the accuracy of each task significantly changes when we change the order of tasks, as it is also claimed and discussed in (Yoon) al.) 2019) proposing order robust method of continual learning in their architectural approach. Evaluating with fixed task order does not coincide with the fundamental aim of continual learning where no dedicated order of tasks is given. Figure 2 shows example test results of state of the art continual learning methods compared to our proposed method. Classification accuracy values of prior methods fluctuate as the order of tasks changes(from Figure 2a to Figure 2b). Based on the observation, we propose to evaluate the robustness to the order of tasks in comprehensive manner in which we evaluate the average and standard deviation of classification accuracy with multiple sets of randomly shuffled orders.
2

---

Under review as a conference paper at ICLR 2021
100
100
95
95
Accuracy (%)
90
85
85
80
75
75
70
T
Task
T
Task
Task
Task
Task
Task
Task 4
Task
Task Order
Task Order
-B SI MAS UCL OURRS
SI MAS
UCL -UOURS
(a) Task order: T5-
(b) Task order: T3-
Figure 2: Classification accuracy continual learning on Split C1far10. SI (Zenke et al. 2017), MAS (Aljundi et al. 2018a) and UCL (Ahn et al. 2019) show critical changes their performance as the order of tasks changes.
The key contributions of our work follows:
We propose a simple but intuitive and effective continual learning method introducing activation based neuron importance.
We propose a comprehensive experimental evaluation framework on existing benchmark data sets evaluate not just the final accuracy of continual learning also the robustness of the accuracy along the changes of the order of tasks. Based on the evaluation framework, existing state-of-the- methods well as our proposed method are extensively evaluated.
2 PROPOSED METHOD
2.1 NEURON IMPORTANCE WITH AVERAGE OF ACTIVATION VALUE
As French [1991] and French (1999) suggested, since activation overlap causes catastrophic forget- ting. a successful network is required to build representations with as little overlap of activation as possible across continual tasks, called sparse representations, mitigate mutual interference. If network builds dense representation, small amount of changes in the representation affects the sta- bility of all trained classes since nodes and weights a network are in casual relations. On the other hand, if network produces distributed representations for all classes, alteration of representation for certain class does not affect others.
In our method, we adopt NM-NI (Node Measurement, Node Importance) type in Figure Ic based in the measurement of activation value at each neuron. Similar to Jung et al. (2020), we measure the average activation value of 3thh neuron as its neuron importance 14.- Incoming weights connected to the same neurons have the same importance. In the case of convolutional neural networks, the average activation value corresponds the average activation value of a feature map (i.c., global average pooling value).
Average activation value itself, however, not able fully represent the characteristics of sential neuron. Encoded features at each layer describe different aspects of an input image and, as result, activation at each layer cannot be evaluated together. Due the difference in absolute aver- age activation values across the layers, weights of carlier layers tend to be considered more essential as Figure 3 shows. If average activation value is used as neuron importance, method will prefer to keep the weights of earlier layers. Instead, we propose to use layer-wise average activation divided by respective standard deviation for importance measurement. Even though a simple extension of (Jung et al. 2020), it prevents particular layer from getting excessive importance compared to other layers, that, in turn, prevents our network from overfitting to a particular task keeping its plas- ticity regardless of task order. As Figure 19a shows, for each method, normalized average weight importance of each total 6 layers) is calculated. Prior average activation based regularization term assigns around 57% of total importance layer 1(57%, 12%, 10%, 6%, 8%, 8%, respectively

---

Under review as a conference paper at ICLR 2021
Layer
Layer 2
Layer
Layer 4
Layer S
Layer
10
14
12
Frequency
3
5
6
8
8
0.250
0.250
0.1
0.25
0
0.1
0.25
0
0.1
0.25
0
0.1
0.25
Weight Importance
(a) Average Activation Value based weight importance distribution
Layer
Layer 2
Layer
Layer
Layer
Layer 6
Frequency
3
10
16
16
6
3
8
0
8
0
8
0
3
8
Weight Importance
(b) Average Activation STD based weight importance distribution
Figure 3:
Distribution of Weight Importance. Note that range of weight importance value is normal- ized by dividing the standard deviation value into the average activation value. This is based on the first task of split CIFAR 10.(task order: 2-0-1-3-4)
for the layers). On the other hand, our proposed regularization loss term assigns 26% of total importance to layer 1. Furthermore, our method avoids assigning excessive importance to certain layer(26% 16%, 16%, 15%, 15%, 12%).
Then, why this improves the continual learning performance regardless of task order? In prior works, more weights of lower layers tend to be frozen earlier tasks that eliminate the chance of upcoming tasks to build new low-level feature sets. Only new task that is fortunately able to rebuild higher layer features based on the frozen lower layer weights from previous tasks could survive. the other hand, ours keeps the balance of frozen weights in all layers securing feature descriptions for new tasks both lower and higher layers. Indeed, lower layer features such edges not class(task) dependent features. Therefore, freezing lower layer features actually not preferable in continual learning. Even though lower level features are mostly changed over tasks, task may find alternative lower level features keeping its plasticity. On the other hand, the network fails to freeze some higher level features that are class(task) dependent, it may lose robustness. As have seen, prior works spend most of their importance lower layers losing its robustness.
freedom of
In order to encode such relation, we propose use average and std. of activation values of each layer rather than using weighs directly. Our loss function is described as follows.
L = f, + Σ Eoe (mt f wf)?
where F, is loss current task, f task index, is weight index, and Ω indicates 1 neuron importance. is strength parameter control the amount of weights consolidation. Neuron importance is defined as follows.
Σ (fa()(") +. Σ (a)
(2)
N
where N₂ is the number of instances, &o is neuron index. fa(-) is activation value, and instance in- dex. We introduce to prevent the numerator from being zero when the standard deviation becomes
4

---

Under review as a conference paper at ICLR 2021
2.2 WEIGHT RE-INITIALIZATION FOR BETTER PLASTICITY
For continual learning, networks have to not only avoid catastrophic forgetting but also learn new tasks to their fullest. According to the degree of difference optimal classification feature space of different tasks, optimized feature space in previous task might be significantly changed in new task. Thus, with new task, we can let the model start either from random weights or from optimized weights with previous task. Even though the optimized weights with previous task can be considered as one set of random weights for a new task, we hope to avoid any chance of optimized weights for one task working as a local optimal for another similar task that may hinder the training from learning a new global or better optimal feature space. The situation can be explained with Q2 (2 - w²)² term in the loss function of our network. During a learning of next task, the network informed of past tasks by 92 (wo fo--- - w term which lets the network maintain essential weights of past tasks assigning high Q4 values. In other words. Qu(wk - w - delivers the knowledge of previous tasks. Whatever the magnitude of S2 is, however, (2 1- - = term is ignored if w almost equals wat already in the initial epoch of the training of a new task, which prevents the network from learning a new task. This situation is alleviated by weight re- re-initialization that allows wt) be high enough regardless of the magnitude of S2 in the training of new task. In this case, still the knowledge of previous tasks will be delivered by Or and affect the training of a new task. We have verified the effect of reinitialization on Split CIFAR10 10 data set. We achieve 4% higher average accuracy when we apply weight reinitialization to our algorithm. Moreover, the performance of UCL(Ahn et al. 2019) increases as well with reinitialization. The results are presented in Appendix.
the value of S2g (uf
EXPERIMENTAL EVALUATIONS
We perform experimental evaluations of our method compared to existing state-of-the-art methods for continual learning on several benchmark data sets; split and permuted MNIST (LeCun et al.
1998 Goodfellow et al. 2013), and incrementally learning classes of CIFAR-10 10 and CIFAR 100 Krizhevsky et al. 2009). Our algorithm is trained based on the description in Ahn et al. (2019) which has tested existing approaches with different hyper-panne parameters to find their best performance.
We train all different tasks with batch size of 256 and Adam using the same learning rate; 0.001.
(Learning rate adaptively changes during learning based on the validation loss.) For the split CIFAR tasks, aforementioned,
perform the evaluation multiple times shuffling the order of tasks randomly to evaluate the robustness to task orders. "Absolute task order" indicates the sequence of tasks that model learns. For instance, task stands for the first task that a model learned no matter what classes comprise the task. We test with all 120 combinations of task sequences for split CIFARIO and 200 random orders for split CIFARIO- 1000 To minimize statistical fluctuations in accuracy, each combination of task sequences is repeated three times.
We define several evaluation metrics. Task-wise average accuracy (TA accuracy) means the accu- racy of each assigned task averaged through the whole continual learning steps. We estimate the degree of forgetting per task via TA accuracy. Learning step-wise average accuracy (LA accuracy) represents the accuracy of each learning step averaged through the whole tasks involved. The degree of interference per learning step is measured by LA accuracy.
3.1 MNIST
We first evaluate our algorithm on a split MNIST benchmark. In this experiment, 2 sequential classes compose each task (total 5 tasks). We use multi-headed and multi-layer perceptrons with two hidden layers with 400 ReLU activations. Each task has its own output layer with two outputs and Softmax. We train our network for 40 epochs with 0.0045.
In Figure 4 we compare task accuracy and LA accuracy. MAS outperforms all other basclines reaching 99.81 81% % while ours achieves 99.7%. However, the accuracy is almost saturated due the low complexity of data so that the difference of accuracy is not significant. Note that ours greatly outperforms EWC, achieving 99.73% and 98.2% respectively.
We also evaluate methods on Permuted MNIST dataset. Our model used in this evaluation is MLP which consists of two hidden layers with 400 ReLUS each and one output layer with Softmax.
5

---

Under review as a conference paper at ICLR 2021
Task 1 (0 or I)
Task (2 or 3)
Task 3 (4 5)
Avcuracy (%5)
99
97
97
95
95
95
Task Task Task Task 4
Task 5
Task
Task
Task 3
Task
Task
Task
Task
Task 3
Task
Task
Task Order
Task Order
Task Order
MA8 UCL -WW
UCL -WEW
MAS
UCL -WWW
-VVCL
BGD +UORR
VCL
BGD +DDURRS
VCL
BGD
-UURS
Task 4 (6 or 7)
Task (8 9)
LA occuracy
99
D
Aaxoung (%)
97
97
95
95
Task Task
Task
Task
Task 5
Task
Task
Task 3
Task
Task
Task
Task Task
Task 4
Task
Task Order
Task Orker
Task Order
MA8
UCL -WE
MAS
- UCL EWC
St
MAS
-UUL -WWW
-VVCL
+ BGD -OURS
VCL
+ BGD -OURS
-CLL
BGD
-OURS
Figure 4: Results on Split MNIST benchmark. Here, VCL indicates VCL(without (oreset)(Nguyen et al. 2017).
The network trained for 20 epochs with À 0.005. Also, to normalize the range of activation value, ReLU applied to the output layer additionally when computing neuron importance Ω2. Our algorithm(95.22 (95.21 ()) outperforms MAS(94.70% EWC(82.45% and VCL(without c8975et))89.76%) and on the other hand, UCL((6.722) ), S1(96.3%% and BGD(96.1 16%%) show better results. However most results this data set achieve almost saturated accuracy.
3.2 SPLIT CIFARIO
Evaluation on Split CIFARIO dataset is based on the heate- network with six convolution layers and two fully connected layers where the output layer is different for each task. We train our network for 100 epochs with a 0.7.
6.5
5.5
65
5,5
80
Sundard
1.5
15
0.5
Task Absolute Task Order
Task
Absolute Task Order
SI MAS UCLL -OOURS
SI MAS LCUL +-OURS
(a) Average Accuracy
(b) Standard Deviation of Average Accuracy
Figure 5: LA accuracy on Split CIFAR10.
Standard Deviation (N6)
6.5
3,5
4.5
70
0.5
Task
T
Task
Fask Task
Tisk
Task 2
Task
Task 5
Absolute Task Order
Absolute Task Order UCL -OOURS
SI MAS - UCL -OURS
Si - MAS
(a) Average Accuracy
(b) Standard Deviation of Average Accuracy
Figure 6: TA accuracy on Split CIFAR10
As Figure shows, method outperforms all other methods with large margins. Also, shows that our algorithm more robust to the order of tasks. In Figure [ we achieve overall highest accuracy than others. Accuracy drop in TA accuracy measurement of our method is natural because

---

Under review as a conference paper at ICLR 2021
weight importance based methods inevitably increase the number of high importance weights loos- ing their plasticity gradually. On the contrary, other compared methods show increasing accuracy. relatively high, however, it implicitly also indicates that their sta- bilities are quite limited possibly because weight importance based continual learning is not working appropriately. Note that TA accuracy of task1 in Figure [6a] for other methods relative quite low compared to TA accuracy of task5. Proposed method shows better stability in the order of tasks and also has low degree of forgetting. In our method, average degraded degree of performance is lowest as 1.23%, whereas SI is 18.06%, UCL is 7.35%, and MAS is 22.89%
looks like that their plasticities
Average Degree of Forgeting (%k
STD of Dagres
10
5
10
Task
Tisk 1
Task 2
Task
Task
Task
Absulute Task Orler
ST MAS
UCL --OURS
MAS HUOURS
(b) Standard Deviation of Average Degree of Forget-
(a) Average Degree Forgetting
ting
Figure 7: Degree of Interference on Split CIFARIO. It is calculated by each task's ' first learned accuracy accuracy after learning the last task
3.3 SPLIT CIFARIO-100
We evaluate our method Split CIFAR10/100 benchmark where each task has 10 consecutive classes. We use the same multi-headed setup as the case of Split CIFARIO. We train our network for 100 epochs with 0.5. We fix task I as CIFARIO 10 due to the difference in the size of dataset between CIFARIO and CIFAR100. The order of task to task 11 that consists CIFAR100 is randomly shuffled. Our algorithm shows better stability showing best accuracy values in old tasks.
Sapdard Deviation (56)
1.3
1.1
0.9
77
0.7
0.3
0.1
Tell
Tube Taakd Traks Twi9 Takit/haki
Taske Taaks TakliTskki
Absolute Task Order
Absolute Task Order
UCL ++OURS
MAS UCL OURR
(a) Average Accuracy
Standard Deviation of Average Accuracy
Figure 8: LA accuracy CIFARIO-100. 10-
Standard Deviatoon (66)
10
2.7
Average of Dogree of
2.2
Forgetting (%6
1.7
1.2
0.7
Task
Taki
Toaks
Tasks
Tosk?
Toals
Tado
Todito
Teaki
0.2
Tast 1282 Tadls Tadel 1od0 Tosks Thaky Teebs Tary 1hel1/
Absolute Task Order St MAMAS UCL -UUURS
Absolute Task Order
S1 - MAS -LUL OUURS
(a) Average Accuracy
(b) Standard Deviation of Average Accuracy
Figure 9: TA accuracy on CIFARIO-100.
On the other hand, previous methods seems prefer be better with recent new tasks proving that
7

---

Under review as a conference paper at ICLR 2021
10
Avenae Degree of
STD of Dagree of Forgelling (%)
Froegeting minng (46)
1.5
2
Teaky
Taye
Talls
Taue?
Task
Talon
Tadif
Taett Tark Tada Trak4
Tuks Tadd Task? Taddi Theee Tack(Tuakk)
Absolutc Tisk Order
Absolute Order
ST MAS --UCL --OURS
MAN -UOURS
(b) Standard Deviation of Average Degree of Forget- ting
(a) Average Degree Forgetting
Figure 10: Degree of Interference on Split CIFARIO-100. is calculated by each task's first learned accuracy accuracy after learning the last task.
100
100
LA Accuracy (%)
95
Accuracy
85
85
Tusk
Task
Task
Task
Task
Task
Task
Task 1
Task 2
Task 5
Task Order
Task Order
- OUURS org.
- OURS double
- OURRS org.
OURS double
(a) LA accuracy of split Cifarl0
(b) TA accuracy of split Cifar10
80
Aveuracy
75
75
66
60
Tikil Takid
Tad8 Tiakio
Task Order
Task Order
org.
org.
double
(c) LA accuracy split Cifar10-100
(d) accuracy of split Cifar10-100
The performance CIFAR10 and CIFARIO-100 with doubled channel. Accuracy in- creases when we use a doubled channel network.
Figure 11:
our importance based continual learning is working appropriately. Indeed, SI seems that it learns new tasks very well but it forgets what has learned. practical, decrease of plasticity in our method can be addressed by using larger network. Figure II show that network with doubled number channels has improved accuracy keeping its stability and better plasticity.
Figure shows that our method obtains lowest average degraded degree of performance among SI, MAS, UCL and ours, achieving 5.02%, 6.3%, 1.06%, and 0.98% respectively. One observation on the stability with CIFAR10- 100 data set despite the larger size of it compared to CIFAR10 is that training CIFARIOS 100 after CIFARIO has the effect of pre-training a network with CIFARIO.
4 CONCLUSION
In this work, we have proposed an activation importance based continual learning method. Com- prehensive evaluation have proved that the proposed method has implemented importance based continual learning achieving fundamental aim of continual learning tasks balancing between stabil- ity and plasticity.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Aincertainty-basae continual learn- ing with adaptive regularization. In Advances in Neural Information Processing Systems, pp. 4392-4402, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 139-154, 2018a.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning, arXiv preprint vrXiv:1806.0542.. 2018b.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pp. 11816 11825, 2019.
Robert M French. Using semiidistributed representations overcome catastrophic forgetting in connectionist networks. In Proceedings of the 13th annual cognitive science society conference, volume 1, pp. 173-178, 1991.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 1rXiv:1312.6211,20 2013.
Pankaj Gupta, Yatin Chaudhary, Thomas Runkler, and Hinrich Schütze. Neural topic modeling with continual lifelong learning. arXiv preprint arXiv:2006. 10909, 2020.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Ad- vances in Neural Information Processing Systems, pp. 1820-1830, 2019.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Adaptive group sparse regular- ization for continual learning. arXiv preprint arXiv:2003. 13726, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 86(11):2278-2324 1998.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-22 22455,2997,222 2017.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard Turner. Variational continual learning. arXiv preprint arXiv: 10628, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv: / 1910, 2018.
Andrei A Rusu, Neil C Rabinowitz. Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1606v 1606:06666.46 2016.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cun features off- the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition workshops, pp. 806-813, 2014.

---

Under review as a conference paper at ICLR 2021
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In Advances in neural information processing systems, pp. 640-646, 1996.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 170000117877 2017.
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. arXiv preprint arXiv: 1902.09432 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv: /0123, 2018.
A APPENDIX
A.1
EVALUATION METHOD
Task1
Task2
Task3
Task4
Task5
Learning stepl
(TA accuracy)
Learning step2
Task-wise
Learning step3
Acc.
Learning step4
Learning step5
Learning step-wise (I.A accuracy)
Figure 12: Concept of Evaluation.
A.2 HYPERPARAMETERS FOR EXPERIMENTS
In MNIST experimnets, we use 3 0.0001 for UCL,, c 1.0 for SI, À = 4000 for EWC, À = 1.0 for MAS and std 022 and 7) for BGD(Zeno et al., 2018). In Cifar experiments, we use 0.7 for SI, À 1.0 for MAS and 3 0.0002 for UCL.
A.3 ADDITIONAL ABLATION STUDY OF OUR ALGORITHM
Figure 15 visualizes weight importance values accumulated over tasks. Each elements represents one weight importance and the intensity of color indicates the magnitude of weight importance value. All values are normalized to I. Each difference map between two consecutive importance maps shows respective weight groups received relatively higher importance values.
10

---

Under review as a conference paper at ICLR 2021
100
LAAccurcyy (%)
90
80
75
70
65
Task
4
Task
Task
Task
Task Order
sid
Figure 13: Comparison Performance on Split CIFARIO. Taks Order: T5 - Note that the performance increases when regard neuron importance avergae activation value / STD.
-4
98
100
96
Accurapy (%)
94
Accuracy (%)
95
92
90
88
86
84
80
80
Task
Task
Task 3
Task 2
Task
Task
Task
Task 3
Task 2
Task I
Task Order
Task Order
OURS w/o Weight Reinitialization
OURS
UCL Weight Reinitialization UCL
(a) LA accuracy of Ours
(b) LA accuracy of UCL
Figure 14: Comparison Performance on Split CIFARIO with Ours and UCL. Taks Order: T5 -> T4 - T3 T2 TI. Note that both performance of Ours and UCL inerease when weight initialization applied.
Task
Task2
Task3
Task4
Task:
Ditti 1.2)
Dithi 2.3)
Dif1334)
D
(a) Weight Importance Map
Figure 15: Weight Importance Maps of Continual Tasks
11

---

Under review as a conference paper at ICLR 2021
98
Accuracy (%)
88
78
68
Task
Task 2
Task 3
Task 4
Task
Absolute Task Order
SI
MAS
UCL
OURS
Figure 16: Another representation of split cifar 10 LA accuracy. To describe both accuracy and stan- dard deviation at once, we change the form of graph. However, it is difficult to indicate difference of accuracy and standard deviation.
08
0.8
6.6
06
04
01
0.0
0.0
Figure 17: Representation on each task's data with the model learned the last task.
0.
0.5
0.
0.5
0.4
0A
-03
02
03
20
62
20
Figure 18: Representation Task I data as learning step progresses.
12

---

Under review as a conference paper at ICLR 2021
0.4
0.6
0.35
Normalizel Avergge
Weight WeightTImporinn Importance
0.5
Normalized Avenge STD
0.3
0.4
Weight importunce
0.3
0.25
0.2
0.2
0.15
0.1
Layer Layer 2 Layer3 3 Layer4 Layer
Layer
Layer
Layer Layer
Layer 4 Layer Layer 6
Convolution Layers
Convolution Layers
saca-aaattttttttooon * S1- UCL -eaving activation / std (OURS) + avrg activation
S1 UCL -ravg_ectivation sid (OURS) activatiin
(a) Normalized average of weight importance distri- bution
(b) Normalized Average STD of weight importance distribution
Figure 19: Normalized Weight importance distribution each convolution layer. 19a represents
normalized average of weight importance of each convolution layer. [9h indicates normalized average/std of weight importance of each convolution layer. Our method relaxes the tendency to consolidate weights of earlier layers. This based the first task of split CIFAR 10.(task order:
2-0-1-3-4)
99
Accuracy(%)
97
95
Toskl
Task2
Task3
Task4
Task5
Task6
Task7
Task8
Task9
Task10
Task Order
ST - MAS
UCL
EWC
VCL
BGD
OURS
Figure 20: Permuted MNIST Average Accuracy. Due to the lack of memory capacity, we evaluate
VCL using multi-layer peceptrons with two hidden layers with 100 ReLU activations.
Ageuracy (%g
43
Standard Deviation (%))
3.5
38
2.5
33
1.5
28
Tedi Taskd Thak3 Tonkd Tad5
Tadé Taak? Thaks:
Tidk Task1
Taakd Takl:
Taskk Taak TaakS Thals TiokT Tiak8 Tokk Takio
Absolute
- ST -*-OURS
ST -0-OURS
(a) Average Accuracy
(b) Standard Deviation of Average Accuracy
Figure 21: LA accuracy on Tiny ImageNet.
50
Assuracy (%0)
45
Standard Devistion
4.5
40
35
3.5
30
25
2.5
20
Tald
Tisk3
Toskd
Tisk5
Takl Thok7
Tists
Tosk
T>sk10
Tastl
Tisk2
Tosk Tost Tick5 Tisid Tesk Tists Tesk DaRL
Absolute
4 SI ++OURS
a Si -0OOURS
(a) Average Accuracy
(b) Standard Deviation of Average Accuracy
Figure 22: TA accuracy on Tiny ImageNet.
13

---

Under review as a conference paper at ICLR 2021
of Degres of
STD of Degnne of
Forgetting (%6
Forgetting (h)
30
Thakl Take Take Tiokk Thiks Todo Tik? Taakk Tado Takk0
Telt Trst2 Tosks Todel Taild Tisss Tosk? Tade Thie Toskt
Absolute Order
Absolltee Task Order
ST -0-OURS
SI -+-OURS
(b) Standard Deviation of Average Degree of Forget- ting
(a) Average Degree of Forgetting
Figure 23: Degree of Interference on split Tiny ImageNet. It is calculated by each task's first learned accuracy accuracy after learning the last task.
14


Reviews:
[{"review_id": "dcaCiUmXM9m", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method that tackles the problem of catastrophic forgetting in continual neural networks by assigning importance to neuron activations while tasks are executed in sequence. Similar to previous research (Jung et al., 2020), the proposed method measures neuron importance using average activation values divided by corresponding standard deviation. This strategy is accompanied by weight re-initialization to guarantee that new tasks are fully learned. The method is tested in benchmark datasets for continual learning. \n\nAs positive aspects of the paper I would remark: \n-\tThe paper is clearly placed within the existing literature in continual learning, as part of regularization-based approaches. \n-\tThe experimental evaluation tests several options of tasks sequences to prevent results to be dependent on the task order. This is an important aspect to demonstrate the validity of the results. \n\nWeaknesses of the paper are: \n-\tThe main weakness in my opinion is that the proposed method represents a simple extension of previous work (Jung et al., 2020). This compromises the novelty of the proposed approach. Furthermore, the experimental results do not denote a remarkable advantage of the proposed method compared to existing approaches. \n-\tI am missing some insights on how the experimental results are connected to the motivation of the proposed method. For example, how adding a sense of the standard deviation of neurons importance help to prevent keeping weights of earlier layers in the experimental datasets?  \n-\tThe experimental setup is limited to a few tasks, which undermines the results presented in the sense that continual learning systems may be composed of several tasks (hundreds, thousands) received in sequence. \n\nSome remaining comments/questions: \n-\tThere are some minor typos/formatting problems along the paper. E.g. caption in Figure 3 does not mention from which dataset these results come from. The presentation of plots of standard deviations alongside mean accuracies is unnatural – why not to use error bars so the differences are clearer?\n-\tHow adding a sense of the standard deviation of neurons importance help to prevent keeping weights of earlier layers in the experimental datasets? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZXn1x1LpIuP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work focuses on improving continual learning framework by introducing neural importance determined by activation value. In doing so, the authors introduce neuron importance as weight factor in minimizing catastrophic forgetting via regularization term. They also investigate continual learning by changing the order of the tasks. \n\n\nStrengths:\n- introduces the neuron importance in regularization terms for minimizing catastrophic forgetting  \n- the paper is well written \n- sound quantitative evaluation and performance analysis  using several data sets\n\nWeaknesses:\n- lack novelty and limited methodological contributions \n- it is unclear the need for introducing neuron importance in regularization techniques for catastrophic forgetting, since the activation value of neurons is also derived from the network weights\n- the gains are marginal for some data sets\n\n\nQuestions:\n- Since neural networks (neuron activity) is too sensitive with change in representation (or input) and therefore, does not guarantee stable results on a particular historical task. This work focuses on determining neuron importance by activation value, however does this activation value correlates to task performance?\n- Since neural activity/importance is calculated based on the network weights, introducing neural importance as well as weight regularization is redundant? Please clarify. \n\nAdditional comments:\n- Unclear, what is f_n_k in equation 2?\n\nAdditional references (On Page 1, para 2 or 3):\n1. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences  2017. \n2. Gupta, P., Chaudhary, Y., Runkler, T., Schütze, H. Neural Topic Modeling with Continual Lifelong Learning. In ICML 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UW-KAjjqNDL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a regularization approach for stable continual learning of sequential tasks. The proposed method computes neuron importance based on the activation values of nodes with their respective standard deviation. It further suggests a weight re-initialization scheme to achieve better performance. Experimental results on several continual learning scenarios show that the presented approach is comparable to other existing competitors.\n\nThe paper first gives analyses by categorizing existing continual learning strategies based on regularization and tackles an important issue that arises when the order of incoming tasks changes. From the analyses, the paper proposes an approach to defining new neuron importance, as shown in equation (2). However, the presented strategy to find neuron importance is simple and incremental. Moreover, no rigorous analyses of the proposed scheme exist on how it is derived, what are benefits, etc. Thus, I am not convinced that it is particularly useful compared to its strong competitors sharing a similar motivation and strategy. For example, I am not sure why the proposed method can be robust to the order of tasks.\n\nSince the proposed method deals with an order-robust approach, it is compared with other approaches sharing the same goal (e.g., Yoon et al., Scalable and Order-robust Continual Learning with Additive Parameter Decomposition, ICLR 2020). A comparison with existing similar approaches would make this work stronger.\n\nExperimental results do not show the proposed method is promising. The proposed method does not outperform existing works and even performs poorer than some competitors for some experiments. From the results, I am not convinced about the benefits of the proposal. Figure 13 seems not notable output.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GR5lHl1iCkc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to regularization-based continual learning that looks to preserve parameters based on node importance rather than weight importance. The paper categorises existing techniques into these groups (most prior work is based on weight importance), argues that node importance is a better measure, and then performs evaluation on numerous benchmarks and with different variations (importance calculation, task ordering, etc).\n\nThis is an important and promising research direction, but unfortunately I think the paper has a number of problems in its current form which preclude publication at this stage.\n\nFirst, the approach seems to derive extensively from Jung et al (2020), and it is unclear whether there is any algorithmic or technical innovation that is added here. From Section 2.1, the only potential improvement I could see is that the average activation is scaled by the standard deviation, which is quite a trivial change. Reinitialization of weights is also discussed, but this is present in previous work (e.g. Ahn et al (2019)) as well. Figure 3 doesn’t show much of a difference in the distribution of weight importances (beyond the obvious scaling), and most crucially, the experiments do not compare to Jung et al, which makes it very difficult to gauge whether there is any novelty at all.\n\nSecond, the approach compares against SI/MAS/EWC as the canonical weight-importance-based approaches, but these are quite old, and a number of more recent techniques also fall into this category, such as VCL [1] (without coreset) and BGD [2] - which estimate the posterior distribution over weights at each timestep, hence determining which ones are important. The experiments would be more convincing if they demonstrated improvement over these more recent approaches.\n\nLast, the writing is quite unclear in parts, and I’d recommend a thorough proofread for spelling/grammar and clarity. A few things I spotted are listed below, but there are many others.\n\nOther questions and comments:\n- Clarification for page 1: EWC computes weight importance after each task, not “after network training”\n- Not clear why Fig 1b and 1c are necessarily different categories - the only way they are different is how the node importance is calculated. Is this fundamentally a different case, and why?\n- One claim in page 3: “If average activation value is used as neuron importance, method will prefer to keep the weights of earlier layers” is stated as a problem - why is this so? This seems reasonable, as lower layers (edge filters, etc) are more likely to be relevant to many tasks, and the higher layers that model more complex features are more likely to need greater plasticity to adapt to new tasks.\n- At the start of page 2, it says that with weight-importance based methods, “it is impossible to reinitialize weights at each training of a new task, which decreases the plasticity of the network.” We usually don’t want to reinit weights for each task, so it’d be good to provide more context upfront for why we might want to do this (this comes later in section 2.1.1).\n\nSome typos and writing issues:\n- Page 1, not sure what “utilize the weights of a given network to the hit” means.\n- Page 2, “One of our key observation…” should be observations; “We propose comprehensive evaluation …” should be “... a comprehensive…”.\n- Page 3: “As exampled in …”; “not only networks have to…”; “we can let the model starts…”, etc.\n\n[1] Nguyen, Cuong V., et al. \"Variational continual learning.\" arXiv preprint arXiv:1710.10628 (2017).\n[2] Zeno, Chen, et al. \"Task agnostic continual learning using online variational bayes.\" arXiv preprint arXiv:1803.10123 (2018).\n\n\n=================\nPost-discussion:\n\nAfter reading the authors' response and the changes to the paper, I must unfortunately stick with my current score. I applaud the authors for taking my feedback on board, and certainly many changes have been made to improve the paper, but my main concern of novelty regrettably still remains. The paper offers a very simple improvement over Jung et al (effectively scaling the importance measure), which I believe is quite incremental in this setting. While I believe simple advances can often have broad impact (eg. dropout, batchnorm, etc), in this case it is not clear that the proposed change offers any benefits outside of the very specific area of importance-based continual learning.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although this paper proposes an intriguing method for using neuron-importance-based regularization to reduce catastrophic forgetting in continual learning, the method is substantially based upon Jung et al (2020), reducing its novelty. Additionally, the experimental evaluation was unconvincing that the proposed method is an improvement over current methods and precisely how the proposed method differs from Jung et al.  The authors are encouraged to revise the paper to incorporate the reviewers suggestions and many of the points the authors raised in their rebuttals, which the reviewers felt were not adequately addressed in the current version of the paper (as mentioned in private discussions among the reviewers).
Decision: Reject