Continual Learning via Neural Pruning
Authors: Siavash Golkar
Micheal Kagan
Kyunghyun Cho
Venue:  (2020)
Keywords: continual learning
lifelong learning
catastrophic forgetting
sparsification

Abstract: We introduce Continual Learning via Neural Pruning~(CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of \emph{graceful forgetting}: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular,  we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable.   We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods. CLNP can also be applied in single-head architectures providing the first viable such algorithm for continual learning. 

Under review as conference paper at ICLR 2020
CONTINUAL LEARNING VIA NEURAL PRUNING
A Anonymous authors
Paper under double-bline review
ABSTRACT
We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer small amount of forgetting in controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks well the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods. CLNP can also be applied in single-head architectures for completely fixed capacity continual learning.
INTRODUCTION
Continual learning, the ability of models to learn to solve new tasks beyond what has previously been trained, has garnered much attention from the machine learning community in recent years. This driven in part by the practical advantages promised by continual learning schemes such as improved performance on subsequent tasks as well as a more efficient use of resources in machines with memory constraints. There is also great interest in continual learning from longer-term perspective, in that any approach towards artificial general intelligence needs to be able to continuously build on top of prior experiences.
As stands today, the main obstacle in the path of effective continual learning is the problem of catas- trophic forgetting: machines trained new problems forget about the tasks that they were previously trained on. There are multiple approaches which seek to alleviate this problem, ranging from employing networks with many submodules Rusu et al. (2016) (Femaando et al. (2017) ( et. (017)) to methods which penalize changing the weights of the network that are deemed important for previous tasks Kirk patrick al. (2016): Zenke et al. (017)) Hou & Kwok (2018). These approaches either require specialized genetic algorithm training schemes or still suffer catastrophic forgetting, albeit at a smaller rate. In particular, the best of our knowledge, there no form of guarantee regarding the performance of previous tasks among the fixed capacity models which use standard SGD training schemes.
In this work we introduce a simple continual learning algorithm for fixed capacity networks which can be trained using standard gradient descent methods and by construction suffers zero deterioration on previ- ously learned problems during the training of new tasks. We take advantage of the over-paran pererarraattiziioonnn of neural networks by using activation based neural pruning scheme to train models which only use a fraction of their width. We then train subsequent tasks utilizing the unused capacity of the model. By cutting off certain connection in the network, make that new tasks can take advantage of previously learned features but cause no interference in the pathways of the previously learned tasks.
MAIN CONTRIBUTIONS
We introduce Continual Learning via Neural Pruning (CLNP) with the following properties:

---

Under review as conference paper at ICLR 2020
Given a network with activation based sparsity, CLNP trains tasks utilizing the unused weights of the network in a manner which prevents any catastrophic forgetting of the previous tasks while taking advantage of the already learned features.
CLNP provides simple diagnostics in the form of the number of remaining and reused neurons and filters. In particular, in experiments we see that CLNP automatically takes advantage of the fact that the features of earlier layers are more transferable.
We formalize the idea of graceful forgetting, the notion that it is preferable to suffer from forgetting in a controlled manner it helps regain network capacity and prevents uncontrolled loss of performance. We use this idea to balance network sparsity and model accuracy.
We show empirically that using an activation based neural pruning sparsification scheme, we outperform previous approaches based on weight elasticity. We also demonstrate in one exam- ple that using a slightly more advanced variation of our sparsification method, the network suf- fers virtually no loss of performance, either from catastrophic forgetting or from sparsification.
In its single-hews head incarnation, CLNP provides one of the first viable single- head continual learning algorithms, and the first to avoid catastrophic forgetting by construction as far as we aware.
2 RELATED WORK
Lifelong learning. Prior work addressing catastrophic forgetting generally fall under two categories. In the first category, the model is comprised of many individual modules at each layer and forgetting is prevented either by routing the data through different modules Fernando al. (2017) by successively modules/Fernando
adding new modules for each new task Rusu et al. (2016); Lee et al. (2017). This approach often has the advantage of suffering zero forgetting, however, the structure of these networks is specialized. In the case of Rusu et al. (2016); Lee et al. (011)) the model architecture is not fixed, and in the case of Fernando et al. (2017) training is done using a tournament selection genetic algorithm. In contrast, algorithms in the second category do not require any specific network structures or training schemes. Here, forgetting is minimized by penalizing changes of weights which are deemed important Kirkpatrick et al. (2016); Zenke et al. (2017); Ritter et al. (2018); Hou & K Kwok (201 8). These approaches. generally referred to as weight clasticity methods, have the advantage of simpler training schemes but still suffer catastrophic forgetting, albeit at smaller rate than unconstrained training. Some approaches combine aspects from both categories, for instance Schwarz et al.) (2018) utilizes weight elasticity methods to aid knowledge distillation into fixed capacity knowledge base.
The approach we take in this paper falls under the second category with standard network structure and training scheme, and forgetting prevented by constraining certain weights from changing. However, it shares the main advantage of the first category approaches in that we suffer zero catastrophic forgetting during the training of subsequent tasks. In particular, our method can be thought of as following the spirit of the path based approach Fernando et al. (2017) using activation based sparsification and the idea of graceful forgetting. Since our method directly comparable other second category approaches, we will provide quantitative comparisons with other methods in this category.
Network superposition. Our algorithm can be thought of as non- non-destructive way of superposing multiple instances of the same architecture using sparsification. Prior work in this direction (c.g. Saxena & Verbeek (2016); Shazeer et al. (2017)) are primarily concerned with neural architecture search and are not aimed at preventing forgetting. The most relevant work in this area is Cheung et al. (2019). which can be thought of as an approximate Fourier space implementation of our approach.
Sparsification. While sparsification is a crucial tool that we use, it is not in itself a focus of this work. For accessibility, we use a simple neuron/filter based sparsification scheme which can be thought of as single iteration variation of Hu et al. (016))
3 METHODOLOGY
The core idea of our method is to take advantage of the fact that neural networks are generally werr-parametrized (Neyshabur et al. 2018). A manifestation of this over-ovrannne wer-parametrization is through the practice of sparsification, i.c. the compression of neural network with relatively little loss of
performance (LeCun al. 1989 Sun et al. 2015 Gale et al. 2019). As example, [uo al. (2017)
2

---

Under review as conference paper at ICLR 2020
show that VGG-16 16 (Simonyan & Zisserman 2014) can be compressed by more than 16 times without significant loss of performance. In this section we first show that given an activation based sparse network, we can leverage the unused capacity of the model to develop continual learning scheme which suffers no catastrophic forgetting. We then discuss the idea of graceful forgetting to address the tension between sparsification and model performance in the context of lifelong learning. In the next section, we will see empirically that even fairly small networks are sufficiently over-parametrized for our method provide competitive results.
It is important to differentiate activation based neuronal sparsity from parameter based weight sparsity. The former implies only a subset of the neurons or filters of each layer assume non-zero values on given dataset whereas the latter means that many of the weights are zero but does not necessarily imply the existence of permanently inactive neurons. the remainder of the paper, when we mention sparsity, we are referring to activation based neuronal sparsity.
In what follows we will discuss sparsity for fully connected layers by looking the individual neurons. The same argument goes through identically for individual channels of convolutional layers.
3.1 GENERALITIES
Let us assume that we have a trained network which sparse the sense that only subset of the neurons of the network are active. Networks with this form of sparsity can be thought of as narrower networks embedded inside the original structure. There are many approaches that aim to train such sparse networks with little loss of performance (e.g. Luo et al. (2017); Hu et al. (016)). We will discuss our sparsification method in detail in $3.2
Fig. D shows cartoon of our approach, where have network with activation based neuronal sparsity, where the active and inactive neurons are respectively denoted blue and grey nodes. Based on the connectivity structure, the weights of the network can also be split into three classes. First, denoted in blue in Fig. 2 we have the active weights W which connect active nodes to active nodes. Next we have the weights which connect any node to inactive nodes, we call these the free weights Where denoted in grey. Finally we have the weights which connect the inactive nodes to the active nodes, we call these the interference weights Wim denoted in red dashed lines. A more precise definition of the active and inactive neurons and weights is given in $3.2
Output
Input
The crux of our approach is the simple observation that if all the interference weights Wi are set to zero, the free weights Wheee can be changed arbitrarily without causing any change whatsoever to the output of the network. We can therefore utilize these weights train new tasks without causing any harm to the performance of the previous tasks.
Figure 1:
The partition of network with neuronal sparsity into active (blue). inactive (grey) and interference (red) parts.
We can further split the free weights into two groups First, the weights which connect active nodes to inactive nodes. These are the weights that take advantage of previously learned features and are therefore responsible for transfer learning throughout the network. We also have the weights that connect inactive nodes to inactive nodes. These weights can form completely new pathways to the input and train new features. A complimentary diagnostic for the amount of transfer learning taking place is the number of new active neurons at each layer after the training of subsequent tasks. Given that an efficient sparse training scheme would not need to relearn the features that are already present in the network, the number of new neurons grown at each stage of training is indicator of the sufficiency of the already learned features for the purposes of the new task. We will see more of this point in $4
Output architecture. In order to fully flesh out a continual learning scheme, we need to specify the connectivity structure of the output nodes. Starting with a network trained on the first task with the interference weights severed (Fig. middle). there are two intuitive routes that we take. In order to train a new task, one option is to use a new output layer (i.c. a new head) while saving the previous output layer. This option, demonstrated in Fig. on the left, is known as the multi-head approach and
3

---

Under review as conference paper at ICLR 2020
Output
Head
Head 2
Output
Input
Input
Input
Figure 2: Middle: network with neuronal spansity. Left, right: the multi-hear and single-head expansions.
standard in continual learning. Because each new output layer comes with its own sets of weights which connect the final hidden layer neurons, this method is fully fixed capacity method. In approach to continual learning, training a multi-head network with fully depleted core structure, i.e. a network where are no more free neurons left, is equivalent to final layer transfer learning.
In scenarios where the output layer of the different tasks are structurally compatible, for example when all tasks are classification with the same number of classes, we can use single-head approach. Demonstrated in Fig.Zon the right, in this approach we use the same output layer for all tasks, but for each task, we mask out the neurons of the final hidden layer that were trained on other tasks. In the case of Fig. 2 only green nodes in the final hidden layer are connected to the output for the second task and only blue nodes for the first task. This is equivalent hidden layer into multiple unequal sized parts, one part for each task. In practice this is done using a multiplicative masking operation with a task dependent mask, denoted Fig. by dashed lines after the final hidden layer. This structure is truly fixed capacity, but is more restrictive to train than its multi-head counterpart. CLNP is among the first methods which provides a viable algorithm for meagleheedd continual learning, and the first to avoid catastrophic forgetting by construction far as we are aware.
dynamic partitioning of the final
3.2 METHODOLOGG DETAILS
Sparsification. So far in this section we have shown that given sparse network trained on a number of tasks, we can train the network on new tasks without suffering any catastrophic forgetting. We now discuss the specific sparsification scheme that we use throughout this paper, which similar in spirit to the network trimming approach put forward in Hu et al. (2016).
Our sparsification method is comprised of three parts. First, during the training of each task, we add an L' weight regularizer to promote sparsity in the network and to regulate the magnitude of the weights of the network. The coefficient a of this regulator a hyperparameter our approach. Also, since different layers have different weight distributions, sparsity in each layer by choosing a different a for each layer. The second part of our sparsification scheme is post-training neuron pruning based on the average activity of each neuron. The third part of the scheme is referred to as fine-tuning and involves adjusting the surviving weights of the network after pruning and is done by retraining the network for a few epochs while only updating the weights which survive sparsification. This causes the model to regain some of the performance loss suffered because pruning. To achieve yet higher level of sparsity, one can iterate the pruning and fine-tuning steps multiple times. In our method, we use one or two iterations of training, with or without fine tuning depending on the task.
can gain
control over the amount of
In $3.1 partitions is as follows. Given network N, comprised L layers, we denote the neurons of each layer as N, with / - 1, L. Let us also assume that the network N has been trained dataset S. In order to find the active and inactive neurons of the network, we compute the average activity over the entire dataset S for each individual neuron. We identify the active neurons Nat i.e. the blue nodes in Fig. 2 as those whose average activation magnitude exceeds some threshold parameter Na (Ni (N, E5 (INi)) > 0). The inactive neurons taken the complement Ninat= N₁ \ The
we partitioned the network into active and inactive parts. A precise definition of these different
4

---

Under review as conference paper at ICLR 2020
threshold value 0 is a post-training hyperparameter of our approach. Similar to the L' weightregulatte hyperparameter a, 8 can take different values for the different layers.. Furthermore, i 0, Ni would be given by the neurons in the network which are completely dead and the function being computed by the network is entirely captured in Nya. We can therefore view Ng as compression of the network into netton of smaller width. Based on their connectivity structure, the weights of each layer are again divided into active, free and interference parts, respectively corresponding to the blue, grey
and red lines in Fig. 2
Summary. The overall algorithm of our approach in its multi-head incarnation given in Alg. During the training of each task, all interference weights are first set to zero and the free weights are re-initialized. We then place the new head for the i' i'th task and train while updat- ing only the free weights Wffiee, We then update the set of active neurons to include the newly activated neurons during the training of the i'th task and accordingly update the set of interference and free weights. We then repeat the process for task 1+
Algorithm 1: Continual Learning via Neural Pruning (CLNP)- Multi-head
Data: datasets S= ( network with I'th layer neurons N;
NT Wii < 03 Witre - W
- Ni+1);
for S do
Wii <0::
Re-initialize Wffee:
Place new head N/.
Train on S, updating only Wieee
3.3 GRACEFUL FORGETTING
Choose highest @ such that Val. Acc
N
N1 UUE Nr Ess (Nii) > 0};
While sparsity is erucial in our approach for the training of later tasks, care needs to be taken so as not to overly sparsify and thereby reduce the model's performance. In practice, model sparsity has the same relationship with generalization as other regularization schemes. As spar- sity increases, initially the generalization performance of the model improves. However, as we push our sparsity knobs (i.e. the I, regulator and activity threshold) higher and make the network sparser, eventually both training and validation accuracy will suffer and the network fails to fit the data properly. This means that in choosing these hyperparameters. we have make a compromise between model performance and remaining network capacity for future tasks.
Nimas
No N
Wilh
W (Ni
- N22)
Winee
(Ni - NN411;
This brings us to a subject which is often overlooked in lifelong learning literature generally referred to as graceful forgetting (Weng & Luciw 2009 Weng & Lu- ciw 2010). This is the general notion that it would be preferable to sacrifice small amount of accuracy in a controlled manner, if it reduces catastrophic forgetting of this task and helps in the training of future tasks. We believe any successful fixed capacity continual learning algorithm needs to implement some form of graceful forgetting scheme. In our approach, graceful forgetting is implemented through the sparsity vs. performance compromise. In other words, after the training of each task, we sparsify the model up to some acceptable level of performance loss in a controlled manner. We then move on to subsequent tasks knowing that the model no longer suffers any further deterioration from training fu- ture tasks. This has to be contrasted with other weight elasticity approaches which use soft constraints on the weights of the network and cannot guarantee future per- formance of previously trained tasks. We will see in the next section that using the exact same network structure, our approach leads to noticeably improved results over
graceful forgetting margin
Sparsity
Validation Accuracy
Figure 3: Cartoon of sparsification hyper- parameter search. Each circular node corre- sponds to model trained with a different pre- training hyperparameter. The green circles are the models within m% of the best validation accuracy. The dashed lines and green squares denote the trajectory and final end- point the Iilddation/spasiii of the model as the pruning threshold 0 is increased. The model with the highest sparsity is then chosen.
existing methods.
The cartoon of the choice of sparsity hyperparameters depicted in Fig. 3 of hyperparameters (i.e. a, the I weight regulator and &, the learning rate) using grid search and
We first scan over a range
5

---

Under review as conference paper at ICLR 2020
note the value of the best validation accuracy across all hyperparameters We then pick the models which achieve validation accuracy within margin of m% of this best validation accuracy. The margin parameter controls how much are willing to compromise on accuracy regain capacity and in experiments we take it to be generally in the range of O. 0.05% to 2% depending on the task. We sparsify the picked models using the highest activation threshold 0 such that the model remains within this margin of the best validation accuracy We finally pick the hyperparameters which give the highest sparsity among these models. In this way, we efficiently find the hyperparameters which afford the highest sparsity model with validation accuracy within m% of its highest value.
After pruning away the unused weights and neurons of the model with the hyperparameters chosen as above. we report the test accuracy of the sparsified network. This algorithm for training and hyperparameter grid search does not incur any significant additional computational burden over standard practice. The hyperparameter search is performed in standard fashion, and the additional steps of selecting networks within the acceptable margin, scanning the threshold, and selecting the highest sparsity network only require evaluation and do not include any additional network training.
4 EXPERIMENTS
We evaluate our approach for continual learning on the permuted MNIST (LeCun & Cortes 2010) and split versions of CIFAR-10 and CIFAR-100) (Krizhevsky et al. 2009), and compare previous results. In all experiments we use Adam optimizer with hyperparameters as in Kingma & Ba 2014))
For network initialization, well as reinitializing the free weights, we employ He-normalization (He ct al. 2015), using all neurons (not just free ones) to compute the fan in.
As mentioned in the previous section, by construction, the performance of previous tasks does not degrade whatsoever when training subsequent tasks. We therefore do not include performance degrada- tion as a function of number of subsequent task plots. In exchange, we provide the final performance plots after all tasks have been trained.
4.1 PERMUTED MNIST
In this experiment, we look at the performance of our approach on ten tasks derived from the MNIST dataset via ten random permutations of the pixels. To make direct comparison with prior work, we employ the most restrictive network reported in a recent survey on this benchmark by Swaaroopp et al. (2019): 2 layer network with 100 neurons in each layer, ReLU activation (Glorot et al. 2011 and softmax multi-class cross-entropy loss trained using batches of 256 samples.
We use a learning rate of 0.002 and L weight regularization a= 10-7, 7,10 -5 ,110- respectively for the first, second and final layers. Finally, we sparsify using 2 iterations of pruning and retraining with a graceful forgetting margin of m m= 3%. This is the smallest margin that allows for the training of all 10 tasks without running out of capacity in the body of the network, after which the performance of the model on newly learned tasks (e.g. task 11) would be lower by around 10%. After training the final task, the average test accuracy of the network across all tasks is 95.8 + 0. 15% (mean = standard deviation across 5 trials). The exhaustive comparison table from Swaroop al. (2019) updated with our results, is reproduced in Tab. ii the supplementary materials for convenience. The best prior performance on this task using the same narrow network architecture is provided by Ritter et al. (2018) who achieve training accuracy of ~ 96%. However, this method requires the computation of the (block-diagone) Kronecker factored) Hessian and therefore much higher complexity than CLNP. The fact that CLNP on this narrow network matches state-of- the-art performance is an indication that excessively wide networks are not necessary for CLNP to be competitive.
To verify the viability of CLNP in a single- head architecture, we also perform the experiment with a wider 2- l-yeyer network of width 2000. This network (or rather its multi-head variant) was used earlier
The choice of both a and & affects the sparsity of the trained network. In practice, however, this dependence is casily captured via the same grid search used to determine hest model performance and does not invoke any extra compatational cost.
2 More specifically, we ramp up one layer at a time starting from the deepest layer. practice, we find that the choice of direction does not make a large difference in the final sparsity of the model.
Value inferred from graph in the source paper.
6

---

Under review as conference paper at ICLR 2020
76-
60
Valcodon
40
Updd
Zocke
CLMF-m- m--imm-ie
CUMP ment%
From stracch
30
Tase
Taak
Conv1
Conv
Conu
Coni
(a) Validation accuracy comparison
(b) Average network capacity usage per task
Figure 4: CIFAR-10 and split CIFAR- 100 results on multi-head network. (a) Final validation accuracy after all tasks are trained. The red dashed bars task and denote that the network ran out of capacity after training task 4. At this point the performance of the network suffers be seen comparison to other methods. Percentage of capacity each layer used after the training of each task.
experiments on this benchmark (e.g. Zenke et al. (2017) and provides an interesting comparison with current and previous multi-heat head algorithms. Here, we achieve an average test accuracy across 10 tasks of 98.42- 112-00 0.04%, which compares favorably to the 97% average performance achieved by Zenke et al.) (0117). While the single-head algorithm requires a wide final layer, as can be seen in Tab. it provides very competitive results on such networks.
4.2 SPLIT CIFAR-10/CIFARS
In this experiment, train image classifier sequentially, first on CIFAR-10 (task 1) and then CIFAR- 100 split into 10 different tasks, each with 10 classes (tasks 1). This experiment is a better benchmark of continual learning since it tests how well an algorithm implements transfer learning across different tasks. However, because of the higher computation requirements of this task, there are not many comparison points for this benchmark. To facilitate comparison with previous algorithms, we use the same multi-head network as in Zenke et al. (017)). We also use wider single-head network for comparison. The structure of these two networks are given respectively given in Tab. and Tab. 2 of the supplementary materials. We train sequentially on the first 6 tasks of this problem using Adam optimizer with learning rate 0.001 and an T,7 weight regularizer coefficient a 10-5, We perform the experiment under three different CLNP schemes: first using single iteration of pruning with graceful forgetting margins m = 1% and m 2% without fine-tuning. We then repeated the experiment under a third scheme defined by a larger margin of m 4% but this time followed by fine-tuning. In each case, we perform the experiment 5 times and report the validation accuracy and its standard deviation on the 6 tasks.
The results of the experiment are shown in Fig. Here, the most ambitious scheme with a graceful forgetting of less than 1%, runs of capacity after the fourth task is trained. We notice that after the model capacity is depleted, the performance of this scheme plummets, showing the necessity for the presence of free neurons in the core of the network during training. The more moderate forgetting scheme m 2%, however, maintains high performance throughout all tasks and does not run out of capacity until the final task learned. The best performance, however, comes from m 4% graceful forgetting scheme followed by one iteration of fine-tuning We see that here there is virtually no catastrophic forgetting on the first task (if anything the model performs even better after pruning and retraining as has been reported in previous sparsity literature LeCun et al. (1989); Hu et al. (2016)) The remaining tasks also get significant boost from this improved sparsification method. In this case, CLNP beats prior results by Zenke et al. (0177) by more than 4% on average.
Although we outperform previous methods, the narrow structure of this network ill-suited for our algorithm. An ideal network for our method would have evenly werrpaaamettrized capacity at all layers of the structure but with this network, 95% of the parameters of the entire network are concentrated in single layer (Krizhevsky) 2014). Fig, 4b shows the network capacity usage per task for the moderate m = 1% forgetting scheme averaged over the 5 runs. The first task alone almost fills up the entirety of the first and second convolutional layer capacities, leaving little room for new
7

---

Under review as conference paper at ICLR 2020
100
Uoed faver cappotyy (5))
40
Santtan ring (Depinted)
20
Todk 2 Tasks 4ase4 Tass
Tasks Task? Tases Task9 Task 10
Con
Cony
(a) Test accuracy comparison
(b) Sample network capacity usage per task
Figure 5: CIFAR-10 10 and split CIFAR- R-10 results on wide single-head network. (a) Final test accuracy after all tasks are trained. The orange dashed bar on task 10 denotes that the network ran out of capacity after training task 9. At this point the performance of the network suffers as can be seen in comparison to other methods. (b) Percentage of the capacity of each layer used after the training of each lask. Lower layers stop training new channels after training few tasks. Deeper layers keep training new channels for all tasks.
convolutional channels to be learned from future tasks. The fact that even with this structure we still outperform previous methods is an indication that a large amount of transfer learning is taking place.
In what follows, we look at two variations on our approach this problem: first with a slightly more advanced sparsification scheme and second with a single-head network of much wider width.
Wide single-head network. To gain further insight into the interplay of transfer learning and network width in networks with activation based neuronal sparsity, we train a much wider fully convolutional network on the same task (Tab. 2 of supplementary materials). The use of this very wide network is made possible only by the computational simplicity of our algorithm and would prove prohibitive for competing methods which require, for example, the computation of the Hessian. This time we use single-head structure making the task more challenging because of the fixed total capacity of the network. In this case, we use a slightly different training scheme, employing heldout validation set comprised of 10% of the training samples for hyperparameter selection and early stopping. We run the experiment 5 times and report the mean and standard deviation of the test accuracy after all tasks have been trained.
The wider network in this experiment has the capacity to train 10 tasks (i.e. CIFAR- 10 plus 9 out of the 10 CIFAR- tasks) before depleting its final hidden layer. Unlike multi- hhad structures where depleted final hidden layer leads to naive transfer learning on new tasks, for single-head structures, a depleted final hidden layer simply has no free connections to the output and the network is entirely fixed. Fig. 5 shows the results of this experiment For comparison we have provided the results of multi-task training and training each task individually from scratch. On average, CLNP outperforms training from scratch we outperform by margin of 2-5% and trails simultancous training by 1 -%. This trend is reversed only on the first task (CIFAR- 10) which has an abundance of training samples and does not benefit from being trained concurrentyy with the other tasks. Details of the multi-task training is given in the supplementary materials section.
il
The average neuron capacity usage per task is given in Fig. 5b There are number of interesting features in this graph. First, tasks number 1-7 each roughly takes up the same number of neurons of the final hidden layer. This is again as expected since the neurons of this layer are connected to the output layer and are not reused in order to avoid interference. By the time we get to task 8, more than 85% of the capacity of this layer is depleted. This directly affects the performance of the network which can be seen in terms of reduced test accuracy both in comparison to the results from multi-task training and individual training from scratch.
The only comparable single-head experiment on this dataset that we are aware of was done by Cheung al. 2019) who use a single-head 6-layer convolutional network. Using superposition techniques they find that the first task suffers about 10% catastrophic forgetting (from 72% down to ~ 63% after training 4 new tasks, accuracy on other tasks not reported). In comparison our method suffers less than 2% drop via graceful forgetting and remains unchanged during the subsequent tasks.
8

---

Under review as conference paper at ICLR 2020
The most interesting observation in the training of the wide network is in the number of new channels learned at each layer for each consecutive tasks. The first convolutional layer trains new channels only for task and 2. The second and third convolutional layers grow new channels up to task 3 and task respectively. The fourth layer keeps training new channels up to the last task. The fact that the first layer grows new channels after the second task implies that the features learned during the training of the first two tasks are fully utilized and deemed sufficient for the training of the subsequent tasks. The fact that this sufficiency happens after training more tasks for layers and 3 a verification of the fact that features learned in lower layers are more general and thus more transferable in comparison with the features of the higher layers which are known to specialize Yosinski al. (2014). This observation implies that models which hope to be effective at continual learning need to be wider in the higher layers to accommodate this lack of transferability of the features at these scales.
5 CONCLUSION
In this work we have introduced an intuitive lifelong learning method which leverages the over- parametrization of neural networks train new tasks in the inactive neurons/filters of the network without suffering any catastrophic forgetting in the previously trained tasks, We implemented controlled way of graceful forgetting by sacrificing some accuracy the end of the training of each task in order to regain network capacity for training new tasks. We showed empirically that this method leads to results which exceed match the current state-of-the of-the-art while being less computationally intensive. Because of this, we can employ larger models than otherwise possible, given fixed computational resources.
Our methodology comes with simple diagnostics based on the number of free neurons left for the training of new tasks. Model capacity usage graphs are informative regarding the transferability and sufficiency of the features of different layers. Using such graphs, we have verified the notion that the features learned in earlier layers are more transferable. We can leverage these diagnostic tools to pinpoint any layers that run out of capacity prematurely, and resolve these bottlenecks in the network by increasing the number of neurons these layers when moving on to the next task. In this way, method can expand to accommodate more tasks and compensate for sub-optimal network width choices.
Our algorithm is crucially dependent on the sparsification method used. In this work we employed neuron activity sparsification scheme based on a single pruning iteration. We demonstrated that using a slightly more advanced sparsification scheme, i.e. adding fine tuning step after pruning can lead to even better results. This shows that our results, can only improve with more effective sparsification methods. This observation opens the path for a new class of neuronal sparsity focused lifelong learning methods as an alternative to current weight elasticity based approaches.
REFERENCES
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. Feb 2019.
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. Rusu, A. Pritzel, and D. Wierstra. PathNet: Evolution Channels Gradient Descent in Super Neural Networks. January 2017.
Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks. Feb 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourleenth international conference on artificial intelligence and statistics, pp. 315-323, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv e-prints, art. arX1v:1502.01882,, Feb 2015.
Lu Hou and James T. Kwok. Power law in sparsified deep neural networks. CoRR, 2018. URL http: //wrxiarxii// http::///awww..oo//aa/o///////// www./b/////aae. wttpa//www.ogg/a/////..0.111. 0188

---

Under review as conference paper at ICLR 2020
Hengyuan Hu, Rui Peng, Yu- Yu-Wing Tai, and Khi-Keeng Tang. Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures. Jul 2016.
Diederik Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. arXiv e-prints, art. arXiv:1412.6980, Dec 2014.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Bariiskka, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Over- coming catastrophic forgetting in neural networks. December 2016.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks, arXiv e-prints, art. arXiv:1404.3997.Ap Apr 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- (canadiaan institute for advanced researcchh) 2009. URLhhtt:://ww. http://wwwww...oooooottttettto/s Scrrotto.edu//Kiiii/iiffeee html
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann lecun. ceon//xdb/mmnn
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In NIPS, 1989.
Jeongtae Lee, Jachong Yoon, Eunho Yang. and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. CoRR, 2017. UtLhttt:://axxtt wrg. http://bbs1777...0 01547
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. Jul 2017.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parameetiizatoon in generalization of neural networks. CoRR, abs/1805. 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting. arXiv e-prints, art. 1cXiv:18050078110 May 2018.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and Hadsell. Progressive Neural Networks. June 2016.
S. Saxena and J. Verbeek. Convolutional Neural Fabrics. June 2016.
Jonathan Schwarz, Jelena Luketina, Wojeiech M. Czarnecki, Agnieszka Baansska-Barwinshe Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress Compress: A scalable framework for continual learning. In Proceedings of the Thirty-Fifthe International Conference on Machine Learning, ICML. '18 2018.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Expert Layer. January 2017.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv e-prints, art. arXiv:1409.1555: Sep 2014.
Yi Sun, Xiaogang Wang, and Xiaoou CoRR, 1bs/1512.01111 2015. URL BRRL:ttpp///aaaa...rr///////2/22
for face recognition.
Siddharth Swaroop, Cuong V. Nguyen, Thang D. Bui, and Richard E. Turner. Improving and Under- standing Variational Continual Learning. arXiv e-prints, art. arXiv:1905.0200 .2009, May 2019.
Weng and M. Luciw. Dually optimal neuronal layers: Lobe component analysis. IEEE Transactions on Autonomous Mental Development, 1(1):68 85, May 2009, doi: 10. 109/TAMDD2009922216660
Jianqing Weng and Matthew D. Luciw. Cortex contex-inspired goal-directed recurrent networks for develop- mental visual attention and recognition with complex backgrounds. 2010.
10

---

Under review as conference paper at ICLR 2020
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3320-3328, Curran Associates, Inc., 2014. URL http::ttpppppssssssssssss ptapperr..ipss... -tg.raaare-rr---tttee------ee-ee
Zenke, B. Poole, and Ganguli. Continual learning with intelligent synapses. Proceedings of International Conference on Machine Learning (ICML),2017

---

Under review conference paper at ICLR 2020
A PERMUTATION MNIST RESULT COMPARISON
For convenience, here we reproduce Tab. 2 from Swaroop et al. (2019) for comparison with prior work. We update this table with new result which was not available at the time this comparison was published. With the exception of our single-head experiment, all other methods use a multi-head architecture.
Method/Paper
Hidden
Number tasks
Final average
layer size
test accuracy
CLNP (ours)
(100, 100)
10
CLNP single-head (ours)
(2000, 2000)
10
98.42++0.04%
VCL
{100, 1001
10
93= 1%
*VLL+ 200 random coreset
(100,100)
10
94.6-10.3%
Previous VCL
1100,100)
10
90%
"Previous VCL 200 random coreset
(100,
10
93%
Kronecker-factoree Laplace
[100,
50
90%
Fronecker-factored Laplace
(100,100
96% **
EWC
(2000, 2000)
10
97%
EWC
{100, 100)
10
84%
SI
{2000, 2000}
10
97%
SI
(100, 100)
10
86%
A-GEM
(256, 256)
20
89. 14.4%
A-GEM
(256, 256)
10
92.3%
GEM
(100, 100)
20
80%
BLLL-REG
(100, 100)
10
92.2%
*GEM
[256, 256)
20
89.5+0048%
*GEM
[256, 256]
10
93. %
Riemannian Walk
[256, 256)
20
85.7++0.56%
Riemannian Walk
(256, 256)
10
91.6% *.
Progressive NNs
{256, 256)
20
93.510007%%
Progressive NNs
(256, 256)
10
94.6%**
Table 1: Final average test accuracy on permuted MNIST for various methods (results taken from respective papers). A hidden layer size of (21,22) indicates two hidden layers, the lower hidden layer having ni hidden units, the upper hidden layer having nz (followed by softmax over the 10 MNIST classes). Methods with asterisk (*)) use some sort of episodic memory. Results marked with double asterisk (**) were read from graph in the source paper.
B MULTI-TASK TRAINING ON CIFAR-10/000
Multi-task learning, the simultaneous training of multiple tasks at the same time, generally a complicated problem. Extra care has be taken especially when the different tasks have different difficulties or different dataset sizes. In the mixed CIFAR-10/100 problem, the dataset for the first task ten times larger than the other tasks. An important practical choice which affects the relative performance of the tasks is how to perform minibatch training. In particular during one epoch of training, if we draw from all tasks indiscriminately, the samples of the smaller datasets run when only 10% of the larger dataset is seen. In this case we can either start new epoch i.e. cut the larger dataset short or we can keep training on the larger dataset until it runs out. In practice, this choice amounts either sacrificing the performance of the first task in favor of the other tasks or vice versa.
For the purposes of comparison to lifelong learning, we take compromise approach. When training all tasks at the same time, we take each batch to comprise of 50 samples from task I and 10 samples from each of the other tasks, the total batch size adding to 140. We then start a new epoch when the the smaller datasets
out. In this way, 1 seen. Note that the relative number of samples in each task biases the network towards one task another, therefore picking even higher number of samples of the first task would again lead to
epoch is started when only 50% of the large dataset of task
12

---

Under review as conference paper at ICLR 2020
sacrificing the accuracy the other tasks. We made the choice of the 5 to ratio purely middle ground. is possible that other choices can lead to better overall performance.
In order to adapt the single-head segde model of Tab. for multi-task training, partition the neurons of the final hidden layer into 10 equal parts which form the "heads" the 10 different tasks. We chose to train only 10 out of the 11 tasks in order to provide fair comparison since continuous learning algorithm depletes the network after 10 tasks. We train using 120 epochs using Adam optimizer with learning rate 0.001 and learning rate schedule with milestones 50 and 90 epochs and 7 We use the same heldout validation set as continuous learning and individual learning for early stopping. We run the training 5 times and report the mean and standard deviation of the test accuracy Fig. 5a
C EXPERIMENT DETAILS
The networks used for the split CIFAR- 10 and CIFAR-100 are given in tables and
Layer
Chan.
Ker.
Str.
Pad.
Dropout
32 x 32 input
3
Conv
32
3x3
Cony 2
32
3x3
1
1
MaxPool
3x3
2
1
0.25
Conv
64
3x3
Conv 4
64
3x3
1
1
MaxPool
3x3
2
0.25
Donse
512
0,5
Task I: Dense
10
Dense
10
Task 6: Dense
10
Table 2: Split CIFAR narrow multi-head model. Convolutional layers and Dense 1 are followed by ReLU activation.
Layer
Chan.
Ker.
Str.
Pad.
32 x 32 input
3
Conv 1
128
3x3
Conv 2
256
3x3
2
Conv 3
512
3x3
0
Conv
1024
3x3
Conv 5
2048
3x3
1
0
Conv 6
10
3x3
AvgPool
Table 3: Split CIFAR wide single-head model. Convolutions 1-5 followed by BatchNorm then ReLU activation.
13


Reviews:
[{"review_id": "rJlNAOYzqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General:\nThe paper proposed neural pruning method to overcome the catastrophic forgetting. Neural pruning identifies important nodes after learning each task, and sets the values of the incoming weights to 0 to preserve the node activation values. Also, the paper propose a method for gracefully forgetting, which is certainly needed for fixed capacity network. \n\nPros:\n1. When the network capacity does not get depleted, they showed the accuracy does not get dropped. \n2. Good results of outperforming several SOTA algorithms. \n3. Proposed a new method for gracefully forgetting, and achieved a good result. \n\nCon & Questions:\n1. Too many hyperparameters which will only dramatically increase with depths. \n2. The method for gracefully forgetting is not very practical. \n3. Depletion often happens - accuracy not dropping seems to be an overclaim. \n4. I think it is almost impossible to reproduce the results of this paper. \n5. What happens if all the layers have the same hyperparameters?\n6. When a depletion happens for a single-headed network, then how can you learn a new task?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyx_oR5nKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for mitigating catastrophic forgetting in neural networks, which is an important and unsolved problem in the sequential training of multiple tasks.  The method works by (i) identifying an active subnetwork after training on one task that can perform the task to a sufficient degree of accuracy, (ii) freezing the subnetwork and pruning interfering connections to it from the rest of the network, and (iii) training the rest of the network on the next task and iterating. The contribution is a method that allows for training of sequential tasks in a fixed capacity architecture with zero catastrophic forgetting guaranteed, while still allowing for transfer between tasks via feature sharing. Additionally, it allows for controlled fine-tuning of the trade-off between accuracy and network capacity, which the authors refer to as ‘graceful forgetting’, by adjusting two hyperparameters that adjust the level of sparsity during the training of a task. The method is shown to outperform several other continual learning methods on permuted MNIST and one other method on split CIFAR-100, both standard evaluations in continual learning.\nWhile the model demonstrates the enviable properties for a continual learning algorithm of zero forgetting and ability to transfer in a fixed network, it suffers from a significant limitation (acknowledged by the authors) that leads me to recommend it for rejection in its current incarnation: \n* Due to the progressive freezing of the parameters, the network eventually reaches full capacity, making it theoretically difficult (in the multi-head setting, depending on the degree of transfer between tasks) or impossible (in the single-head setting) to learn how to perform new tasks.\n* This is particularly problematic because the mechanism for 'graceful forgetting’ can only be applied on the subnetwork for a task that has just been trained on; once N tasks have been learnt, the subnetwork for task k<N cannot be altered without potentially affecting the performance on tasks k to N.\n* In the paper, subnetworks are sparsified to within a fixed margin of the maximum accuracy of each task - with this procedure, you cannot predict how much capacity will be taken up by each task, and so you cannot know in advance how many tasks will fit into the network. If instead you chose to sparsify to a given subnetwork capacity per task in order to guarantee being able to fit a certain number in, then you can not control the accuracy level for each task. \n* Another consequence of the method is that it is not clear how one can easily resume training on a previous task (perhaps to take advantage of transfer from subsequent tasks). You could reinstate connections from the features of subsequent tasks to the earlier task and train, but presumably these would initially interfere with the original subnetwork. Have the authors considered any ways of achieving this?\nIn fairness to the authors, they (a) show that the growth in utilised capacity slows as more tasks are added and (b) propose that the layers of the network could be dynamically extended as a way of overcoming a network at full capacity. For the reasons given above, however, it is not a practical fixed capacity algorithm for a lifelong learning setting, which constitutes an indefinite stream of sequential tasks, since at capacity it can suffer from sudden 'catastrophic remembering’, as opposed to the ‘graceful forgetting’ advertised in the paper. I do think that the idea has potential and that the algorithm would be *significantly* strengthened if there were a mechanism for graceful forgetting of all or selected previous tasks when the network is at capacity.\n\nFurther Comments / questions:\n* The method is claimed to be the “first viable algorithm for single-head continual learning”. This statement is unspecific - what does ‘viable’ mean in this context? Online EWC [1] is a continual learning algorithm that can theoretically applied in a single-head setting - what makes it non-viable?\nMinor comments not affecting review:\n* Better to label the weight types in the caption of Figure 1 rather than just in the main text\n* In order to see the effect of depletion more clearly, it would also be useful to see the performance of the current task in isolation rather than just the average of all previous tasks.\n* Section 2, line 8: “speci[c]fic\"\n* The main text reference to Figure 5b compares it to the single-head MNIST usage graph - where is this?\n\n[1] Schwarz, Jonathan, et al. \"Progress & Compress: A scalable framework for continual learning.\" International Conference on Machine Learning. 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylhcAATtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Continual lifelong learning is very interesting and has attracted increasing attention in recent years. Generally, there are two group of methods. And, the proposed method called CLNP leverages the merits from both of them. The experimental results clearly shows the effectiveness of this method.\n\nMy main concern is that this paper actually was accepted by a Neuro AI workshop of NeurIPS and thus this submission does not comply the rules of double-blind review process. Anyway, overall, this paper is well-written and interesting.\n\nI have more following concerns\n1) As the review given by Neuro AI, the submission just compare their results to Zenke et al. 2017 for SPLIT CIFAR-10/CIFAR-100. In this version, authors does not add new experiments to compare some new methods. Actually, most concerns raised by the reviewers in Neuro AI have not been considered.\n2) It is not clear how to set the hyperparameter such as threshold \\theta in different layers. Most settings and criteria are given empirically and this paper seriously lacks theoretical analysis.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There are several concerns with the brittleness and reproducibility of the proposed approach and experiments.
Decision: Reject