SparCL: Sparse Continual Learning on the Edge
Authors: Zifeng Wang
Zheng Zhan
Yifan Gong
Geng Yuan
Wei Niu
Tong Jian
Bin Ren
Stratis Ioannidis
Yanzhi Wang
Jennifer Dy
Venue: NeurIPS 2022 Accept (2022)
Keywords: Continual Learning
Sparse Training

Abstract: Existing work in continual learning (CL) focuses on mitigating catastrophic forgetting, i.e., model performance deterioration on past tasks when learning a new task. However, the training efficiency of a CL system is under-investigated, which limits the real-world application of CL systems under resource-limited scenarios. In this work, we propose a novel framework called Sparse Continual Learning (SparCL), which is the first study that leverages sparsity to enable cost-effective continual learning on edge devices. SparCL achieves both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity, data efficiency, and gradient sparsity. Specifically, we propose task-aware dynamic masking (TDM) to learn a sparse network throughout the entire CL process, dynamic data removal (DDR) to remove less informative training data, and dynamic gradient masking (DGM) to sparsify the gradient updates. Each of them not only improves efficiency, but also further mitigates catastrophic forgetting.  SparCL consistently improves the training efficiency of existing state-of-the-art (SOTA) CL methods by at most 23X less training FLOPs, and, surprisingly, further improves the SOTA accuracy by at most 1.7%. SparCL also outperforms competitive baselines obtained from adapting SOTA sparse training methods to the CL setting in both efficiency and accuracy. We also evaluate the effectiveness of SparCL on a real mobile phone, further indicating the practical potential of our method.

SparCL: Sparse Continual Learning on the Edge
Zifeng Wang Zheng Zhan" Yifan Gong', Geng Yuan', Wei Niu?, Tong Jian', Bin Ren", Stratis Ioannidis' Yanzhi Wang', Jennifer Dy
Northeastern University. College of William and Mary gong-yifa, yifa, geng. yuan, yanz. wang] Pmoorteasteers edu, (zifengwang, jian, ioannidis, @dye neu, edu, wniućemail. wm. edu, bren@cs wm. edu
[zhan, zhe, gong
Abstract
Existing work in continual learning (CL) focuses on mitigating catastrophic for- getting, Le., model performance deterioration on past tasks when learning a new task. However, the training efficiency of a CL. system is under-investigated, which limits the real-world application of CL systems under resource-limited scenarios. In this work, we propose novel framework called Sparse Continual Learning (SparCL), which is the first study that leverages sparsity to enable cost-effective continual learning edge devices. SparCL achieves both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity, propose task-aware dynamic
data efficiency, and gradient sparsity. Specifically, masking (TDM) to learn a sparse network throughout the entire CL process, dy- namic data removal (DDR) to remove less informative training data, and dynamic gradient masking (DGM) to sparsify the gradient updates. Each of them not only improves efficiency, but also further mitigates catastrophic forgetting. SparCI. consistently improves the training efficiency of existing state-of-the-ard (SOTA) CL methods by at most 23x less training FLOPs, and, surprisingly. further improves the SOTA accuracy by at most 1.7%. SparCL also outperforms competitive base- lines obtained from adapting SOTA sparse training methods the CL setting both efficiency and accuracy. We also evaluate the effectiveness of SparCL on a real mobile phone, further indicating the practical potential of our method.
Introduction
The objective of Continual Learning (CL) is enable intelligent system accumulate knowledge from a sequence tasks, such that exhibits satisfying performance on both old and new tasks [32]. Recent methods mostly focus addressing the catastrophic forgetting [43] problem learning model tends to suffer performance deterioration on previously seen tasks. However, in the real world, when CL applications are deployed in resource-limited platforms [48] such edge devices, the learning efficiency, w.r.t. both training speed and memory footprint, are also crucial metrics of interest, yet they are rarely explored in prior CL works.
Existing CL methods can be categorized into regularization-be based [2, 32, 37, 68], rehearsal-based [8, 12, 50, 61], and architecture-bact [31, 42, 52, 58, 59, 70]. Both regularization- and rehearsal-basg based methods directly train a dense model, which might be over-parameterizzd even for the union of all tasks [19, 39]. Though several architecture-bart based methods [51, 57, 64) start with sparse sub-network from the dense model, they still grow the model size progressively to learn emerging tasks. The aforementioned methods, although striving for greater performance with less forgetting, still introduce significant memory and computation overhead during the whole CL process.
Both authors contributed equally to this work
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

23
Weight Sparsity TDM
22
36
Triming FLOPs (100))
21
20
Sparsty
19-
10
18
Effilenny- DDR
17
16
Figure 1: Left: Overview of SparCL. SparCL consists of three complementary components: task-awarr dynamic masking (TDM) for weight sparsity. dynamic data removal (DDR) for data efficiency, and dynamic gradient masking (DGM) for gradient sparsity. Right: SparCL successfully preserves the accuracy and significantly improves efficiency over DER++ [8], one of the SOTA CL methods, with diffcrent sparsity ratios on the Split Image [16] dataset.
Recently, another stream of work, sparse training [4, 20, 35], has emerged training trend to achieve training acceleration, which embraces the promising sparse training, each iteration takes less time with the reduction computation achieved by sparsity. Inspired by these sparse training methods, under the traditional i.i.d. learning setting, we naturally think about introducing sparse training to the field of CL. straightforward idea is directly combine existing sparse training methods, such as SNIP [35]. RigL. [20], with a rehearsal buffer under the CL setting. However, these methods fail to consider key challenges CL to mitigate catastrophic forgetting, for example, properly handling transition between tasks. As result, these sparse training methods, though enhancing training efficiency, cause significant accuracy drop (see Section 5.2). Thus, we would like to explore a general strategy, orthogonal to existing CL methods, that not only leverages the idea of sparse training for efficiency, but also addresses key challenges in CI. to preserve (or even improve) accuracy.
paradigm. With
In this work, we propose Sparse Continual Learning (SparCL), a general framework for cost-effective continual learning, aiming at enabling practical CL on edge devices. As shown in Figure I (left), SparCL achieves both learning acceleration and accuracy preservation through the synergy of three aspects: weight sparsity, data efficiency, and gradient sparsity. Specifically, to maintain small dynamic sparse network during the whole CL process, we develop a novel task aask- dynamic masking (TDM) strategy to keep only important weights for both the current and past tasks, with special consideration during task transitions. Moreover, we propose a dynamic data removal (DDR) scheme, which progressively removes "easy-to-learn" examples from training iterations, which further accelerates the training process and also improves accuracy of CL. by balancing current and past data and keeping more informative samples in the buffer. Finally, provide additional dynamic gradient masking (DGM) strategy to leverage gradient sparsity for even better efficiency and knowledge preservation of learned tasks, such that only subset of sparse weights are updated. Figure 1 (right) demonstrates that SparCL successfully preserves the accuracy and significantly improves efficiency DER++ [8], of the SOTA CL methods, under different sparsity ratios.
SparCL simple in concept, compatible with various existing rehearsal-based CL methods, and efficient under practical scenarios. We conduct comprehensive experiments on multiple CL bench- marks to evaluate the effectiveness of our method. We show that SparCI. works collaboratively with existing CL. methods, greatly accelerates the learning process under different sparsity ratios, and even sometimes improves upon the state-of-the-an accuracy. We also establish competing baselines by combining representative sparse training methods with advanced rehearsal -based CL methods. SparCL again outperforms these baselines terms of both efficiency and accuracy. Most importantly, evaluate our SparCL framework real edge devices to demonstrate the practical potential of our method. We are not aware any prior CL works that explored this area and considered the constraints of limited resources during training.
In summary, our work makes the following contributions:
We propose Sparse Continual Learning (SparCI.), a general framework for cost-effective continual learning, which achieves learning acceleration through the synergy of weight sparsity, data effi-

---

ciency, and gradient sparsity. To the best of our knowledge, our work is the first to introduce the idea of sparse training to enable efficient CL on edge devices. Our code is publicly available".
SparCL shows superior performance compared both conventional CL methods and CL-adapted sparse training methods on all benchmark datasets, leading to at most 23x less training FLOPs and, surprisingly, an 1.7% improvement over SOTA accuracy.
We evaluate SparCL. on real mobile edge device, demonstrating the practical potential of method and also encouraging future research on CL on-the-edge. The results indicate that our framework can achieve at most 3.1x training acceleration.
2
Related work
2.1 Continual Learning
The main focus in continual learning (CL) has been mitigating catastrophic forgetting. Existing methods can be classified into three major categories. Regularization-bun -based methods [2, 32, 37, 68] limit updates of important parameters for the prior tasks by adding corresponding regularization terms. While these methods reduce catastrophic forgetting some extent, their performance deteriorates under challenging settings [40], and on more complex benchmarks [50, 61]. Rehearsal- based methods [13, 14, 25] save examples from previous tasks into a small-sized buffer to train the model jointly with the current task. Though simple in concept, the idea of rehearsal is very effective in practice and has been adopted by many state-of-the-art methods [8, 11, 49]. Architecture-baser methods [42, 51, 57, 59, 63] isolate existing model parameters or assign additional parameters for each task to reduce interference among tasks. As mentioned in Section 1, most of these methods use a dense model without consideration of efficiency and memory footprint, thus are not applicable to resource-limited settings. Our work, orthogonal these methods, serves general framework for making these existing methods efficient and enabling a broader deployment, e.g., CL on edge devices.
A limited number of works explore sparsity in CL, however, for different purposes. Several methods [41, 42, 53, 57] incorporate the idea of weight pruning [24] to allocate a sparse sub-network for each task reduce inter-task interference. Nevertheless, these methods still reduce the full model sparsity progressively for every task and finally end up with a much denser model. On the contrary, SparCI. maintains sparse network throughout the whole CL. process, introducing great efficiency and memory benefits both during training and at the output model. A recent work [15] aims at discovering lottery tickets [21] under CL, but still does not address efficiency. However, the existence of lottery tickets in CL serves as a strong justification for the outstanding performance of our SparCL.
2.2 Sparse Training
There are two main approaches for sparse training: fixed-mask sparse training and dynamic sparse training. Fixed-mask sparse training methods [35, 54, 56, 60] first apply pruning, then execute traditional training on the sparse model with the obtained fixed mask. The pre-fixed structure limits the accuracy performance, and the first stage still causes huge computation and memory consumption. To overcome these drawbacks, dynamic mask methods [4, 17, 20, 45, 46) adjust the sparsity topology during training while maintaining low memory footprint. These methods start with a sparse model structure from an untrained dense model, then combine sparse topology exploration at the given sparsity ratio with the sparse model training. Recent work [67] further considers incorporating data efficiency into sparse training for better training accelerations. However, all prior sparse training works are focused on the traditional training setting, while CL is more complicated and difficult scenario with inherent characteristics not explored by these works. In contrast to prior sparse training methods, work explores new learning paradigm that introduces sparse training into CL for efficiency and also addresses key challenges in CL, mitigating catastrophic forgetting.
3 Continual Learning Problem Setup
In supervised CL, a model fo learns from a sequence of tasks D (DI..... Dr). where each task D. = {( 11, consists of input-label pairs, and each task has a disjoint set of classes. Tasks
'https://github http::////ttuub..co//ee--e----e wwwp///eeweeaaaaa/ee.e..
3

---

Dynamic Masking
Dynamic
Dynamic Gradient Masking (DGM)
(TDM)
(DDR)
Epodss
Tasks
(V Stutee--add--yaan
Remove "masier" samplex
Important gradients
Figure 2:
Illustration of the SparCL workflow. Three components work synergistically to improve training efficiency and further mitigate catastrophic forgetting for preserving accuracy.
arrive sequentially, and the model must adapt to them. At the t-th step, the model gains access to data from the t-th task. However, a small fix-sized rehearsal buffer M is allowed to save data from prior tasks. At test time, the easiest setting is to assume task identity is known for each coming test example, named task-incremental learning (Task-IL.). If this assumption does not hold, we have the more difficult class-incremental learning (Class-IL) setting. In this work, we mainly focus on the more challenging Class-IL setting, and only report Task-IL performance for reference.
The goal of conventional CL is to train a model sequentially that performs well on all tasks at test time. The main evaluation metric is average test accuracy all tasks. In real-world resource- limited scenarios, we should further consider training efficiency of the model. Thus, we measure the performance of the model more comprehensively by including training FLOPs and memory footprint.
4 Sparse Continual Learning (SparCL)
Our method, Sparse Continual Learning, is unified framework composed of three complementary components: task-aware dynamic masking for weight sparsity, dynamic data removal for data efficiency, and dynamic gradient masking for gradient sparsity. The entire framework is shown in Figure 2. We will illustrate each component in detail in this section.
4.1 Task-aware Dynamic Masking
To enable cost-effective CL in resource limited scenarios, SparCL. is designed to maintain a dynamic structure when learning a sequence of tasks, such that it not only achieves high efficiency, but also intelligently adapts to the data stream for better performance. Specifically, we propose strategy named task-aware dynamic masking (TDM), which dynamically removes less important weights and grows back unused weights for stronger representation power periodically by maintaining a single binary weight mask throughout the CL process. Different from typical sparse training work, which only leverages the weight magnitude [45] or the gradient w.r.t. data from a single training task [20, 671, TDM considers also the importance of weights w.r.t. data saved in the rehearsal buffer, as well as the switch between CL tasks.
Specifically, TDM strategy starts from randomly initialized binary mask Ma Mo. with given sparsity constraint || |M |Meloo/||0lo = 8, where E [0. 1] is the sparsity ratio. Moreover, it makes different intra- and inter-task adjustments keep dynamic sparse set of weights based on their continual weight importance (CWI). We summarize the process of task-aware dynamic masking in Algorithm and elaborate its key components in detail below.
Continual weight importance (CWI). For a model fo parameterized by 6, the CWI of weight w C defined as follows:
CWI(w) = |wl + al aā(D;:0)
3L(M:0) Iw +1
(1)
Sw
where D, denotes the training data from the t-th task, M is the current rehearsal buffer, and 8 are coefficients to control the influence of current and buffered data, respectively. Moreover, L represents the cross-entropy loss for classification, while & is the single-head [1] version of the cross-entropy loss, which only considers classes from the current task by masking out the logits of other classes.

---

Algorithm 1: Task-aware Dynamic Masking (TDM)
Input: Model weight 0, number of tasks T, training epochs of the I-th task K, binary sparse mask Ma. sparsity ratio 8, intra-task adjustment ratio Pintra: inter-task adjustment ratio
Pinter. update interval 8k
Initialize: 0, MM. s.t. ||Mello/ | = 1
for 1, 1...., T do for e = 1,.... Kr do
if > then
/* Inter-task adjustment */
Expand Me by randomly adding unused weights,
s.t. (.o.|ooolo/|l00p = (s Pintar)
if -==d0 5k then
Shrink Ma by removing the least important weights according to equation s.t. || / -
end
end
if mod 8k = 0 then
Intra-task adjustment
Shrink Mo by removing the least important weights according to equation 1,
s.t. | |
(s + Pintra)
Expand Ma by randomly adding unused weights,
s.t. Moloo |
end
Update 0 @ Mg via backpropagation
end end
Intuitively, CWI ensures we keep (1) weights of larger magnitude for output stability, (2) weights important for the current task for learning capacity, and (3) weights important for past data to mitigate catastrophic forgetting. Moreover, inspired by the classification bias in CL [1]. we use the single- head cross-entropy loss when calculating the importance score w.r.t the current task to make the importance estimation more accurate.
Intra-task adjustment. When training the 1-th task, natural assumption is that the data distribution is consistent inside the task. As a result, we would like to update the sparse model in relatively stable way while keeping its flexibility. Thus, in Algorithm way every åk epochs. We first remove Pintra of the weights of least CWI retain learned knowledge so far. Then we randomly select unused weights to recover the learning capacity for the model and keep the sparsity ratio unchanged.
choose to update the sparsity mask
Me in a shrink--nd-exppnn
Inter-task adjustment. When tasks switch, the contrary, we assume that the data distribution shifts immediately. Ideally, we would like the model to keep the knowledge learned from old tasks much as possible, and to have enough learning capacity accommodate the new task. Thus, instead of the shrink-and-expand strategy for intra-task adjustment, we follow an expand-and-shan scheme. Specifically, at the beginning of the (t + 1)-th task, we expand the sparse model by randomly adding a proportion of Pinter unused weights. Intuitively, the additional learning capacity facilitates fast adoption of new knowledge and reduces interference with learned knowledge. We allow our model to have smaller sparsity (i.e., larger learning capacity) temporarily for the first 5k epochs as warm-up up period, and then remove the Pinter weights with least CWI, following the same process in the intra-task case, to satisfy the sparsity constraint.
4.2 Dynamic Data Removal
In addition to weight sparsity, decreasing the amount of training data can be directly translated into training time savings. Thus, we would also like explore data efficiency to reduce the training workload. Some prior CL works select informative examples to construct the rehearsal buffer [3, 6, 65]. However, their main purpose is not training acceleration; thus, they either introduce excessive computational cost consider different problem settings. By considering the features of CL,
5

---

present simple yet effective strategy, dynamic data removal (DDR), to reduce training data for further acceleration.
We measure the importance of each training example by the occurrence of misclassification [55, 67] during CL. In TDM, the sparse structure of model updates periodically every ok epochs, so align our data removal process with weight mask updates for further efficiency and training stability, In Section 4. we have partitioned the training process for the t-th task into N₂- K stages based on the dynamic mask update. Therefore, we gradually remove training data at the end of i-th stage, based on the following policy: 1) Calculate the total number of misclassifications fz(zy) for each training example during the i-th stage. 2) Remove a proportion of Ps training samples with the least number of misclassifications Although our main purpose is to keep the "harder" examples to learn to consolidate the sparse model, we can get further benefits for better CL results. First, the removal of "easier" examples increases the probability that "harder" examples to be saved to the rehearsal buffer, given the common strategy, e.g. reservoir sampling [14], to buffer examples. Thus, we construct a more informative buffer in a implicit way without heavy computation. Moreover, since the buffer size is much smaller than the training set size of each task, the data from the buffer and the new task is highly imbalanced; dynamic data removal also relieves this data imbalance issue.
Formally, we set the data removal proportion for each task as E [0, 1]. and a cutoff' stage, such that:
cutoff
PP=PP.
Σ
Pi
c
The cutoff stage controls the trade-off between efficiency and accuracy: when we the cutoff stage earlier, we reduce the training time for all the following stages; however, when the cutoff stage is set too early, the model might underfit the removed training data. Note that when we set P: = 0 for all i i= 1.2..... N₂ and cutoff = Nr, we simply recover the vanilla setting without any data efficiency considerations.
our experiments, we assume P = p/cutoff, i.e., removing equal proportion of data at the end of every stage, for simplicity. We also conduct a comprehensive exploration study of p and the selection of the cutoff stage in Section 5.3 and Appendix B.3.
4.3 Dynamic Gradient Masking
With TDM and DDR, we can already achieve weight efficiency and data efficiency during training. To further boost training efficiency, we explore gradient sparsity and propose dynamic gradient masking (DGM) for CL. Our method focuses on reducing computational costs by only applying the most important gradients onto the corresponding unpruned model parameters via gradient mask. The gradient mask is also dynamically updated along with the weight mask defined in Section 4.1. Intuitively, while targeting for better training efficiency, DGM also promotes the preservation of past knowledge by preventing fraction of weights from updating.
Formally, our goal here is find subset of unpruned parameters (or, equivalently, gradient mask Mc) to update over multiple training iterations. For a model fe parameterized by 8, we have the corresponding gradient matrix G caleulated during each iteration. To prevent the pruned weights from updating, the weight mask Md will be applied onto the gradient matrix G as G @ M, during backpropagation. Besides the gradients of pruned weights, we in addition consider to remove less important gradient coefficients for faster training. To achieve this, we introduce the continual gradient importance (CGI) based on the CWI measure the importance of weight gradients:
(w) = al 2c(D=:0) + 33 OC(M:0)
(3)
Ow
dr
We remove a proportion of non-zero gradients from G with less importance measured by CGI and we have M |00||o - (s g). The gradient mask Mc is then applied onto the gradient matrix G. During the entire training process, the gradient mask Mg is updated with a fixed interval.
Experiment
5.1 Experiment Setting
Datasets. We evaluate our SparCI. on two representative CI. benchmarks, Split CIFAR-10 [33] and Split Tmny-Im [16] to verify the efficacy of SparCL. In particular, follow [8, 68] by

---

Table I: Comparison with CL methods. SparCI. consistently improves training efficiency of the corresponding
CL. methods while preserves (or even improves) accuracy on both class- and task-incremental settings.
Split CIFAR-10
Split Tiny-EmageNet
Method
Sparsity
Buffer size
FLOPs Tmin
Fl FLOPs Train
Class-IL (?)
Task-1L (t)
x10' ())
Class-IL (t)
Tisk-L
x10"
[32]
19.49:m11
68 2932
8.3
7.58
19.20
LWF [37]
0.00
19.61=005
63.29:225
8.3
8.46
15. 15.85
13.3
[42]
0.501
93.73,00s
5.0
61.88.100
7.3
LPS [57]
94,50.0
5.0
63.37.001
7.3
A-GEM [13]
20
83.88
22. 20
iCaRL [50]
49.02=330
88, 18.99=2.1
11.1
7.53 79
28. 19=1.47
17.8
FDR [5]
0.00
g.74
13.9
8.70
22.2
ER [14]
44.79.135
91.19 iuee
11.1
8,49
38. 17.1000
17.8
DER+
64.88
91.92s00
10.96
40.87
SparCI
0.75
46.89
92.02
2.0
8,
39. 14goss
3.2
66.3
94.06.144
2.5
12.73
12706
SparCL DER+Hoo
SparCT -ERoo
0.90
200
45.81
65.79.113
93.73.014
91 2
0.9
8.67.
38. 19.0.
1.4
SparCL-EBa ERas
44.599n22
9107 20.4
0.5
8.43+am
38. 20:044
0.8
SparCL-DER+tus
0.95
18
=037
0.6
10.7640
40.54
A-GEM [13]
22.67
89 195
11.1
8,06
25.33
17.8
iCaRL [50]
47.55=395
88 22.22+262
11.1
9,38=1
FDR [5]
0.00
500
28.71 1:3
93.29 29-9
13.9
10.54
49.88
22.2
[14]
57.74=021
9,90+029
48 48.6428046
DER+ [8]
72.70=136
93.88
13.9
19.38
51.91=008
22.2
SparCL HRr
60.80
12:822
2.0
10.4810.2
50.83
SparCL-DERrae DER++
0.75
74.09=004
35.19=024
2.5
20.75
52. $2.199004
4.0
SparCL-ERw
39.34=007
93.33-a010
0.9
10. 12:.:3
49.46=1121
SparCL-DER+ty
0.90
500
73.42=095
94.82=013
1.1
19.62
51,93e035
1.8
SparCL-ERvo
92.73-034
0.5
9.91=a
48.57=050
0.8
SparCT SparCI-DER+* ++gg
72. 14g
94.39an 15
0.6
19.0
51.26,277
1.0
TaacKNet and LPS actually have a decreased sparsity after Tearning every task, we use 0,50 to roughly represent the average sparsity.
splitting CIFAR-I0 and Tiny-ImageNet into 5 and 10 tasks, each of which consists of 2 and 20 classes respectively. Dataset licensing information can be found in Appendix A.
Comparing methods. We select several CL methods including regularization based (EWC [32]. LwF [37], architecture-bant
(PackNet [42], LPS [57]), and rehearsal-based (A-GEM [13],
iCaRL [44], FDR [5], ER [14], DER++ [8]) methods. Note that PackNet and LPS are only compatible with task-incremental learning. We also adapt representative sparse training methods (SNIP [35],
RigL [20]) to the CL setting by combining them with DER++ (SNIP-DER++, RigL-DER++).
Variants of our method. To show the generality of SparCL, combine it with DER++ (one of the SOTA CL methods), and ER (simple and widely-used) as SparCL-DER++ and SparCL-ER, respectively. We also vary the weight sparsity ratio (0.75, 0.90, 0.95) of SparCL for comprehensive evaluation.
Evaluation metrics. We use the average accuracy on all tasks to evaluate the performance of the final model. Moreover, we measure the training FLOPS| [20], and memory footprint [67] (including feature map pixels and model parameters during training) to demonstrate the efficiency of each method. Please see Appendix B.1 for detailed definitions of these metrics.
Experiment details. For fair comparison, we strictly follow the settings in prior CL work [8, 29]. We set the per task training epochs 50 and 100 for Split CIFAR-I0 and Tiny-ImageNet, respectively, with a batch size of 32. For the model architecture, we follow [8, 50] and adopt the ResNet- 18 [26] without any pre-training. We also use the best hyperparameter setting reported [8. 57] for CL methods, and in [20, 35] for CL-adapted sparse training methods. For SparCL and its competing CL-adapted sparse training methods, we adopt uniform sparsity ratio for all convolutional layers. Please see Appendix B for further details.
5.2 Main Results
Comparison with CL methods. Table summarizes the results on Split CIFAR-10 and Tiny- ImageNet, under both class-incremental (Class-II.) and task-incremental (Task-Il.) settings. From Table 1, we can clearly tell that SparCL significantly improves upon ER and DER++, while also
7

---

Table 2: Comparison with Cl-aadated sparse training methods. All methods are combined with DER++ with 500 buffer size. SparCI. outperforms all methods in both accuracy and training efficiency, under all sparsity ratios, three methods here can save 20% 51% memory footpriat, please see Appendix B.2 for details.
Split CIEAR-10
Split Tiny-ImageNett Net
Method
Spasity
FLOPs Train x10 ())
FLOPs Train
Class-IL.(t)
Class-II.(t)
x10 (U)
DER++ [8]
0.00
72.70=136
13,9
19.38
22.2
SNIP-DER++ [35]
69.82=072
1.6
13+061
2.5
Rigl-DERR+ [20]
0.90
69.86
1.6
18.364049
2.5
SparCI-DER+to
73.424005
1.1
19.624067
1.8
SNIP-DER++ [35]
66.07=001
0.9
14. 44.762032
1.5
Rigl-DERR+ [20]
0.95
66.53+118
0.9
1.5. 2068
1.5
SparCL-DER++gg
72.14
0.6
19.01
1.0
Table 3: Ablation study on Split-CIFR100 with 0.75 sparsity ratio. All components contributes to the overall
performance, in terms of both accuracy and efficiency (training FLOPs and memory footprint).
FLOPs Train x10 (4)
Memory Footpriat (4)
TDM
DDR
DGM
Class-IL (f)
x
x
72.70
13,9
247MB
x
73.37
3.6
180MB
/
x
73.80
2.8
180MB
X
V
73.97
3.3
177MB
74.09
2.5
177MB
outperforming other CL baselines, in terms of training efficiency (measured in FLOPs). With higher sparsity ratio, SparCL leads to fewer training FLOPs. Notably. SparCL achieves 23x training efficiency improvement upon DER++ with a sparsity ratio of 0.95. On the other hand, our framework also improves the average accuracy of ER and DER++ consistently under all cases with a sparsity ratio of 0.75 and 0.90, and only slight performance drop when sparsity gets larger as 0.95. In particular, SparCL-DER++ with 0.75 sparsity ratio sets new SOTA accuracy, with all buffer sizes under both benchmarks. The outstanding performance of SparCL indicates that our proposed strategies successfully preserve accuracy by further mitigating catastrophic forgetting with a much sparser model. Moreover, the improvement that SparCL brings two different existing CL methods shows the generalizability of SparCI. as a unified framework, ie., has the potential to be combined with wide array of existing methods.
We also take a closer look at PackNet and LPS, which also leverage the idea of sparsity to split the model by different tasks, different motivation from training efficiency. Firstly, they are only compatible with the Task-IL setting, since they leverage task identity at both training and test time. Moreover, the model sparsity of these methods reduces with the increasing number of tasks, which still leads to much larger overall training FLOPs than that of SparCL. This further demonstrates the importance of keeping a sparse model without permanent expansion throughout the CL process.
Comparison with CL-adapted sparse training methods. Table shows results under the more difficult Class-IL setting. SparCL outperforms all CL-adapted sparse training methods in both accuracy and training FLOPs. The performance gap between SparCL-DER++ and other methods gets larger with higher sparsity. SNIP- and RigL-DER++ achieve training acceleration at the cost of compromised accuracy, which suggests that keeping accuracy is a non-trivial challenge for existing sparse training methods under the CL setting. SNIP generates the static initial mask after network initialization which does not consider the structure suitability among tasks. Though RigL adopts a dynamic mask, the lack of fask-awarenes prevents it from generalizing well to the CL setting.
5.3 Effectiveness of Key Components
blation study. We provide a comprehensive ablation study Table 3 using SparCL-DER++ with 0.75 sparsity on Split CIFAR10. Table 3 demonstrates that all components of our method contribute both efficiency and accuracy improvements. Comparing rows I and 2, we can see that the majority of FLOPs decrease results from TDM. Interestingly, TDM leads to increase in accuracy, indicating
8

---

RgL
SparCL
Average Accuracy
73
DDR
One-shol
Acceleration rate (Himes)
No removal
72
2.5
71
201P-000-16
Reforeace Meti'". Spertt out
2.0
Fsl.
70
5NP ceR 3y
1.5
1st 1g1
69
20
40
60
80
1.0
Remove Percentage
66
67
68
69
70 Accuracy
71
72
73
Comparison between DDR and One- shot [67] data removal strategy w.t.t different data removal proportion p. DDR outperforms One-shot and also achieves better accuracy when p S 30%.
Figure 3:
Figure 4: Comparison with CL-adapted sparse training methods in training acceleration rate and accuracy results. radius circles are measured by memory footprint.
TDM generates a sparse model that is even more suitable for learning all tasks than then full dense model. Comparing rows and 3, we can see that DDR indeed further accelerates training by removing less informative examples. As discussed in Section 4.2, when we remove certain number of samples (30% here), we achieve point where we keep much informative samples as we need, and also balance the current and buffered data. Comparing rows and 4, DGM reduce both training FLOPs and memory footprint while improve the performance of the network. Finally, the last row demonstrates the collaborative performance of all components. We also show the same ablation study with 0.90 sparsity in Appendix B.4 for reference. Details can be found in Appendix B.I.
Exploration on DDR. To understand the influence of the data removal proportion p. and the cutoff stage for each task, we show corresponding experiment results in Figure 3 and Appendix B.3, respectively. In Figure 3, we fix cutoff 4, ie.. gradually removing equal number of examples every 5 epochs until epoch 20, and vary from 10% to 90%. We also compare DDR with One-shot removal strategy [67], which removes examples at once at cutoff. DDR outperforms One-shot consistently with different in average accuracy. Also note that since DDR removes the examples gradually before the cutoff stage, DDR more efficient than One-shot. When p S 30,, we also observe increased accuracy of DDR compared with the baseline without removing any data. When p 2 40%, the accuracy gets increasingly lower for both strategies. The intuition is that when DDR removes proper amount of data, it removes redundant information while keeps the most informative examples. Moreover, as discussed in Section 4.2, it balances the current and buffered data, while also leave informative samples in the buffer. When DDR removes too much data, it will also lose informative examples, thus the model has not learned these examples well before removal.
Exploration on DGM. We test the efficacy of DGM at different sparsity levels. Detailed exploratory experiments are shown Appendix B.5 for reference. The results indicate that by setting the proportion within appropriate range, DGM can consistently improve the accuracy performance regardless the change of weight sparsity.
5.4 Mobile Device Results
The training accelcration results are measured on the CPU of an off-the-shelf Samsung Galaxy S20 smartphone, which has the Qualcomm Snapdragon 865 mobile platform with a Qualcomm Kryo 585 Octa-core CPU. We run each test on a batch of 32 images to denote the training speed. The detail of on-mobile compiler-level optimizations for training acceleration can be found in Appendix C.I.
The acceleration results are shown in Figure 4. SparCL can achieve approximately 3.1 and 2 training acceleration with 0.95 sparsity and 0.90 sparsity, respectively. Besides, our framework can also save 51 % and 48% memory footprint when the sparsity is 0.95 and 0.90. Furthermore, the obtained sparse models save the storage consumption by using compressed sparse row (CSR) storage and can be accelerated to speed up the inference on-the-edge. We provide on-mobile inference acceleration results in Appendix C.2.

---

Conclusion
This paper presents a unified framework named SparCL for efficient CL that achieves both learning accelcration and accuracy preservation. It comprises three complementary strategies: task-aware dynamic masking for weight sparsity, dynamic data removal for data efficiency, and dynamic gradient masking for gradient sparsity. Extensive experiments standard CL benchmarks and real-world edge device evaluations demonstrate that our method significantly improves upon existing CL methods and outperforms CL-adapted sparse training methods. We discuss the limitations and potential negative social impacts of our method in Sections and 8, respectively.
7 Limitations
One limitation of our method is that we assume a rehearsal buffer available throughout the CL process. Although the assumption is widely-accepted, there are still situations that a rehearsal buffer is not allowed. However, as a framework targeting for efficiency, our work has the potential to accelerate all types of CL methods. For example, simply removing the terms related rehearsal buffer in equation 1 and equation 3 could serve as naive variation of our method that is compatible with other non-rehearsal methods. It is interesting to further improve SparCL to be more generic for all kinds of CL methods. Moreover, the benchmarks we use are limited to vision domain. Although using vision-based benchmarks has been a common practice in the CL community, we believe evaluating our method, as well as other CL. methods, on datasets from other domains such as NLP will lead to more comprehensive and reliable conclusion. We will keep track of newer CL. benchmarks from different domains and further improve our work correspondingly.
8 Potential Negative Societal Impact
Although SparCL is general framework to enhance efficiency for various CL methods, we still need to be aware its potential negative societal impact. For example, we need to be very careful about the trade-off between accuracy and efficiency when using SparCL. If one would like to pursue efficiency by setting the sparsity ratio too high, then even SparCL will result in significant accuracy drop, since the over-sparsified model does not have enough representation power. Thus, should pay much attention when applying SparCI. on accuracy-sensitive applications such as healthcare [66]. Another example is that, SparCI. powerful tool to make CL. methods efficient, also strengthen models for malicious applications [7]. Therefore, we encourage the community to come up with more strategies and regulations prevent malicious of artificial intelligence.
Acknowledgement
The authors gratefully acknowledge support by the National Science Foundation under grants CCF- 1937500, CCT-1919117 and CN5-2112471.
References
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang. Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In CVPR, 2021.
[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. NeurIPS, 2019.
[4] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In ICLR, 2018.
[5] Ari Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint
2018.
10

---

[6] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. NeurIPS, 2020.
[7] Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe. Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. The malicious use of artificial intelligence:
Forecasting, prevention, and mitigation. arXiv preprint wrXiv:/802.07225. 2018.
[8] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS, 2020.
[9] Pietro Buzzega, Matteo Boschíni, Angelo Porrello, and Simone Calderara. Rethinking ex- perience replay: a bag of tricks for continual learning. In ICPR, pages 2180-2187. IEEE, 2021.
[10] Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. arXiv preprint arXiv: 1605.00777 2016.
[11] Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In ICCV, 2021.
[12] Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip Torr, and David Lopez-Paz Paz. Us- ing hindsight to anchor past knowledge in continual learning. arXiv preprint arXiv:2002.08165, 2(7), 2020.
[13] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a a-gem. arXiv preprint 17X.::08/2.00420, 2018.
[14] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 0ati::9900 /902 10486, 2019.
[15] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In ICLR, 2020.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, leee, 2009.
[17] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840. 2019.
[18] Peiyan Dong, Siyue Wang, Wei Niu, Chengming Zhang, Sheng Lin, Zhengang Li, Yifan Gong, Bin Ren, Xue Lin, and Dingwen Tao. Rtmobile: Beyond real-time mobile acceleration of mns for speech recognition. DAC, pages 1-6. IEEE, 2020.
[19] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In NeurIPS, pages 759-770, 2019.
[20] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In JCML, pages 2943-2952. PMLR, 2020.
[21] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019.
[22] Yifan Gong, Geng Yuan, Zheng Zhan, Wei Niu, Zhengang Li, Pu Zhao, Yuxuan Cai, Sijia Liu, Bin Ren, Xue Lin, al. Automatic mapping of the best-suited dnn pruning schemes for real-time mobile acceleration. ACM Transactions on Design Automation of Electronic Systems (TODAES), 27(5):1 26, 2022.
[23] Yifan Gong, Zheng Zhan, Zhengang Li. Wei Niu, Xiaolong Ma. Wenhao Wang, Bin Ren, Caiwen Ding, Xue Lin, Xiaolin Xu, et al. A privacy-reeervvingorreene dnn pruning and mobile acceleration framework. In GLSVLSI, pages 119-124, 2020.
[24] Song Han, Jeff Pool, et al. Learning both weights and connections for efficient neural network.
In NewrIPS, pages 1135-1143, 2015.
11

---

[25] Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In ICRA, 2019.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 06866, 2018.
[28] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In CVPR, pages 4340-4349, 2019.
[29] Yen- Yen-Chang Hsu, Yen Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating con- tinual learning scenarios: A categorization and case for strong baselines. arXiv preprint 1810. 12488, 2018.
[30] Tong Jian, Yifan Gong, Zheng Zhan, Runbin Shi, Nasim Soltani, Zifeng Wang, Jennifer G Dy, Kaushik Roy Chowdhury, Yanzhi Wang, and Stratis Ioannidis. Radio frequency fingerprinting on the edge. IEEE Transactions on Mobile Computing, 2021.
[31] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of a mixed sequence of similar and dissimilar tasks. NeurlPS, 2020.
[32] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-BBrrinska. al. Overcoming catastrophic forgetting in neural networks. PNAS, 1 11((13)335213322 2017.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
https///www.css.oronto..ee
2009.
[34] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistt/ 1998.
[35] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
[36] Tuanhui Li, Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convolutional neural networks via factorized convolutional filters. In CVPR, pages 3977-3986, 2019.
[37] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, (0(12) 2017.
[38] Xiaolong Ma, Fu-Ming Guo, Wei Niu, Xue Lin, Jian Tang, Kaisheng Ma, Bin Ren, and Yanzhi Wang. Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices. In AAAI, pages 5117-5124, 2020.
[39] Xiaolong Ma, Wei Niu, Tianyun Zhang, Sijia Liu, Sheng Lin, Hongjia Li, Wujie Wen, Xiang Chen, Jian Tang, Kaisheng Ma, et al. An image enhancing pattern-based sparsity for real-time inference on mobile devices. In ECCV, pages 629-645. Springer, 2020.
[40] Zheda Mai, Ruiwen Li. Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. arXiv preprint arXiv:2101.1042 10423, 2021.
[41] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In ECCV, pages 67-82, 2018.
[42] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In CVPR, 2018
[43] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
12

---

[44] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training lifelong learning. ICML Workshop, 2021.
[45] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.
[46] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML, pages 4646-4655, PMLR, 2019.
[47] Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang, Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Patdnn: Achieving real-time dnn execution mobile devices with pattern-based weight pruning. arXiv preprint arXiv:2001.00138, 2020.
[48] Lorenzo Pellegrini, Vincenzo Lomonaco, Gabriele Graffieti, and Davide Maltoni. Con- tinual learning at the edge: Real-time training on smartphone devices. arXiv preprint 0rXiv:2105.33177 2021.
[49] Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. NeurlPS, 2021.
[50] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 2001-2010, 2017.
[51] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606. 04671, 2016.
[52] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In /CML, 2018
[53] Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Spacenet: Make free space for continual learning. Neurocomputiing. 131: 2021.
[54] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. NeurIPS, 33:6377- 2020.
[55] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05155 2018
[56] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2019.
[57] Zifeng Wang, Tong Jian, Kaushik Chowdhury, Yanzhi Wang, Jennifer Dy, and Stratis Ioannidis. Learn-prune-share for lifelong learning. In ICDM, 2020.
[58] Zifeng Wang, Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. ECCV, 2022.
[59] Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. CVPR, 2022.
[60] Paul Wimmer, Jens Mehnert, and Alexandru Condurache. Freezenct: Full performance by reduced storage costs. In ACCV, 2020.
[61] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pages 374-382, 2019.
[62] Yushu WU, Yifan Gong, Pu Zhao, Yanyu Li, Zheng Zhan, Wei Niu, Hao Tang, Minghai Qin, Bin Ren, and Yanzhi Wang. Compiler-aware neural architecture search for on-mobile real-time super-resolution. ECCV, 2022.
13

---

[63] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In CVPR, pages 3014-3023, 2021.
[64] Li Yang, Sen Lin, Junshan Zhang, and Deliang Fan. Grown: Grow only when necessary for continual learning. arXiv preprint arXiv:21 10. 10.00908, 2021.
[65] Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.0108 2021.
[66] Kun-Hsing Yu, Andrew L Beam, and Isaac S Kohane. Artificial intelligence in healthcare. Nature biomedical engineering, 2(10): 2018.
[67] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, al. Mest: Accurate and fast memory-economic sparse training framework on the edge. NeurIPS, 34, 2021.
[68] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, 2017.
[69] Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kaeli, Bin Ren, al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In ICCV, pages 1-231, 2021.
[70] Tingting Zhao, Zifeng Wang, Aria Masoomi, and Jennifer Dy. Deep bayesian unsupervised lifelong learning. Neural Networks, 149:95-106, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The claims match the experimental results and it is expected to generalize according to the diverse experiments stated our paper. We include all of our code, data, and models in the supplementary materials, which can reproduce our experimental results.
(b) Did you describe the limitations of your work? [Yes] See Section 6 and Appendix 7. (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6 and Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A] Our paper is based on the experimental results and do not have any theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] Our paper is based on the experimental results and we do not have any theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material a URL.? [Yes] See Section 5.1, Section 5.4 and we provide code to reproduce the main experimental results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5. and Section 5.4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table 1, Table 2, fig fig 3.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.1, Section 5.4.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
14

---

(a) If your work uses existing assets, did you cite the creators? [Yes] We mentioned and and all comparing methods
cited the datasets (Split CIFAR- 10 and with their paper and github in it.
(b) Did you mention the license of the assets? [Yes] The licences of used datasets/models are provided in the cited references and we state them explicitly Appendix A.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We provide code for our proposed method in the supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "hI5fvrY_8cm", "reviewer": "Reviewer_igAw", "summary": "The authors discuss whether continual learning can be investigated under the perspective of training efficiency and propose combining network sparsification techniques in a CL setting. As a result, they notice that sparser network are not necessarily worse continual learners and that they can possibly surpass some recent non-sparse approaches. The effectiveness of the proposed SparCL method is investigated by means of ablative studies and a conclusive test on an edge device.", "questions": "Please refer to the weakness section above, I am mostly interested in seeing the following point addressed:\n\n- Why is DER++ not among competitors?\n- Can any hypothesis be made on whether representation learning approaches CO2L/Dualnet can also benefit from sparsification?\n- Where are weights pruned the most in the backbone, is there a pattern?\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work. It is recommended to do so in the main paper, not in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts\n- At the core of this work is a very well-thought intuition: the learning process is noisy/overparameterised and pruning can be beneficial in CL, where we need to cram knowledge in waves into a model and possibly retain generalisation capabilities to facilitate upcoming tasks.\n- the presentation of this work is clear, the exposition is easy to follow.\n\nWeaknesses (in decreasing order of importance)\n- While the authors claim that they are using a SOTA roster of competitors, I believe that this is not exactly the case. Some recent methods that are missing and could provide an interesting comparison are: LUCIR [a]/BiC [b] (somewhat stronger baselines w.r.t. iCaRL that additionally manage the bias problem), MIR [c]/GSS [d] (which similarly to DDR operate a selection on data which is memorised), CO2L [e]/DualNet [f] (recent works focussing on representation learning, which could or could not benefit from sparsification, as their inner working is rather different from standard replay methods).\n- In line with the previous point, the authors take DER as one of their reference baselines. In doing so, however, they are omitting the improved DER++ method proposed in the same paper that appears to be better in all respects w.r.t. DER++. Is there a reason for this exclusion? If not, I take that DER++ should be included instead.\n- It would be very interesting to have an additional study on where the sparsification takes place in the network. The authors describe very practical routines to establish which weights to prune. Linked to the recent trend discussing the effect of catastrophic forgetting on different layers of the networks [g], it would be very interesting to have an analysis of at which level the network is pruned the most and whether this changes at all within or between tasks.\n- an experimental comparison could be made between DDR and the approach for buffer reduction delineated in [d] (gradient based) or the one described in [h] (loss based)\n- there seem to be some problems with Alg.1: the second if-then (if $t=\\delta$ k) is missing its end and I cannot understand whether the following $t mod \\delta k = 0$ also applies for tasks > 1. If this is the case -- as I understood from the text -- then this is not an else-if.\n- I recommend that the authors include figures in this paper in vectorial format so as to facilitate reading.\n- as per neurips policies, limitations and societal impact should be in the main paper, not in the appendix\n- Typos: line 95 (thus not applicable - verb missing), line 161 (missing escape of space after w.r.t. - write w.r.t.\\ in latex), line 282 (contributes -> contribute), line 288 (as much as informative -> as much informative)\n\n[a] Hou, Saihui, et al. \"Learning a unified classifier incrementally via rebalancing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.   \n[b] Wu, Yue, et al. \"Large scale incremental learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.  \n[c] Aljundi, Rahaf and Belilovsky, Eugene and Tuytelaars, Tinne and Charlin, Laurent and Caccia, Massimo and Lin, Min and Page-Caccia, Lucas, \"Online Continual Learning with Maximal Interfered Retrieval\", NEURIPS 2019  \n[d] Aljundi, Rahaf, et al. \"Gradient based sample selection for online continual learning.\" Advances in neural information processing systems 32 (2019).  \n[e] Cha, Hyuntak, Jaeho Lee, and Jinwoo Shin. \"Co2l: Contrastive continual learning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.  \n[f] Pham, Quang, Chenghao Liu, and Steven Hoi. \"Dualnet: Continual learning, fast and slow.\" Advances in Neural Information Processing Systems 34 (2021): 16131-16144.  \n[g] Ramasesh, Vinay V., Ethan Dyer, and Maithra Raghu. \"Anatomy of catastrophic forgetting: Hidden representations and task semantics.\" arXiv preprint arXiv:2007.07400 (2020).  \n[h] Buzzega, Pietro, et al. \"Rethinking experience replay: a bag of tricks for continual learning.\" 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gkGZ8Lfgd9-", "reviewer": "Reviewer_8efq", "summary": "The paper addresses continual learning in edge devices. It claims that previous work on CL didn't report the actual FLOPs in training which is very important in real life applications on edge devices. The paper includes adaptations of sparse training to CL . It applies three techniques:  1)dynamic weight sparsification based on the magnitude of the weights and magnitude of the gradients on the novel task and on the buffer (it combines it with collapse and expand approach on class boundaries). 2)It reduces the data based on misclassification. 3)It applies less weight updates by not updating parameters with low gradients. Applying these techniques results in significant time and memory savings without loss of accuracy when integrated with rehearsal methods for CL. It even shows slight improvement in forgetting. \n\n", "questions": "There is no section 6 in the Appendix. ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: reduces the resources in CL, while previous work focused on not increasing them. Useful in real-life applications. Novel in terms of focus. \nWeakness: The proposed techniques  are straightforward and  do not offer technical novelty.\nHowever, suggesting a straightforward solution in a paper that raises a new problem/challenge (to the best of knowledge) could still be OK.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8T1I-3103Lz", "reviewer": "Reviewer_g1tN", "summary": "The paper  proposes Sparse Continual Learning (SparCL in which they wanted to do cost-effective continual learning on edge devices by leveraging the  sparsity. Author's also claim that they achieve both training acceleration and accuracy preservation through the synergy of three aspects: weight sparsity, data efficiency, and gradient sparsity. Including TDM process for dynamic data removal (DDR) to remove less informative training data, and dynamic gradient masking (DGM) to sparsify the gradient updates. Authors also claim that the proposed method is uses 23Xless training FLOPs, and, achieving the SOTA accuracy by at most 1.7%.  The authors also tested the method on two datasets split cifar-10, spilt tiny Image-net in Class Incremental and Task Incremental setting.", "questions": "\nPlease refer \"Strengths And Weaknesses\"", "limitations": "Please see \"Strengths And Weaknesses\"", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nStrengths:\n\n+ Inclusion of Dynamic data removal along with sparsity is good benefit to overall system.\n+ Sparsity in gradient and propose dynamic gradient masking \n+ Method also considers the  importance of weights w.r.t. data saved in rehearsal buffer.\n\nCon's:\n- While reading the paper, I really felt that some sections of the paper are not clearly written specifically Sec 4.1 Task-aware Dynamic Masking.\n\n- I am not really clear/convincing about the authors claim about the memory foot print and training FLOPS. Consider one of the longback papers basic paper on CL using Neural Pruning (https://arxiv.org/abs/1903.04476)(You can choose any other regular CL on pruning paper) and some other CL using sparsity papers which does CL on split CIFAR-10/CIFAR-100 using pruning and they achieve the CL on them without using memory buffer of 500 images. Similarly in the method proposed by authors in which it is also showed that that they need to do extra steps of dynamic Gradient Masking,Dynamic Data Removal which add the overhead too. Clarify this if I am wrong in my understanding on how these additional overheads won't effect the training FLOPS and memory footprint.\n\n- I am also not clear with authors point of view of choosing the \"Rehearsal based methods also\" for comparing for  \"memory foot print and training\" as it is obviously clear that rehearsal based methods does need  additional amount of training and memory.\n\n- I am also very curious to know what happens when buffer size of the proposed method is 0. What happens to accuracies as shown in the Table 1 when buffer size increased from 200--->500 the acc increased from 65-->72\n\n- The improvement in the acc w.r.t to SOTA minimal as PackNet shown in table 1 does show packnet, LPF achieves 93.73%,94.50% acc when compared to best SparCL-DER75(95.19±0.34)(500 buffer) and   SparCL-DER75(94.06±0.45)(200 buffer).\n\n- One more limitation I see regarding the proposed approach is towards the scalability as you can see current papers such as ANML(https://ecai2020.eu/papers/939_paper.pdf), of CL does show CL >100's of tasks. But how would given proposed scale comparing the overhead of buffer, memory masks while trying to learn huge no of tasks.\n\n\n\nOverall the paper is generally well written and easy to follow apart from few sections., and the experiments are thorough and well-executed but limited to fewer datasets, lesser tasks. and minimal improvement on SOTA There are extensive ablation experiments demonstrating some of the key components the proposed approach. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2BM3ldFnnb-", "reviewer": "Reviewer_BMhS", "summary": "This paper introduces sparse continual learning, a continual learning framework that overcomes catastrophic forgetting of deep networks in the learning of a task stream and accelerates the training and inference. \nThe efficiency and effectiveness are achieved through three key components that encourage sparse network weight connection, replay buffer selection, and sparse gradient truncation. \nThe effectiveness is validated on both public benchmarks on real-world edge deivce. ", "questions": "The proposed method relies on the usage of a reply buffer. As also discussed by the authors in Appendix Section A, the proposed method can potentially be applied to more advanced methods that do not rely on any previous data. Showing the proposed method to be more 'model agnostic might help improve the paper. \n\n", "limitations": "The authors discussed the limitations in Appendix Section A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The presentation of the paper is clear and easy to follow. \n\n- The results on mobile devices are a plus\n\n\nWeakness:\n\n- I see sparsity as a universal and generic way of improving deep model efficiency. Therefore it is not surprising to see introducing sparsity to CL can improve efficiency.  \n\n- Among the three techniques proposed by this paper, only the weight sparsity part seems to have a direct impact on overcoming catastrophic forgetting, which is the key to improving continual learning. \n\n- In Table 1, the latest method included in the comparison is from 2020. The quantitative comparison are clearly insufficient to support the claim that the proposed framework 'further improves the SOTA accuracy by at most 1.7%.'\n\n- Overall, this paper delivers a useful message that sparsity can improve the efficiency of CL, and even slightly improve the accuracy. However, the novelty and soundness of the paper do not meet the standard of NeurIPS. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept