Exclusive Supermask Subnetwork Training for Continual Learning
Authors: Prateek Yadav
Mohit Bansal
Venue: Submitted to ICLR 2023 (2023)

Abstract: Continual Learning (CL) methods mainly focus on avoiding catastrophic forgetting and learning representations that are transferable to new tasks. Recently, Wortsman et al. (2020) proposed a CL method, SupSup, which uses a randomly initialized, fixed base network (model) and finds a supermask for each new task that selectively keeps or removes each weight to produce a subnetwork. They prevent forgetting as the network weights are not being updated. Although there is no forgetting, the performance of supermask is sub-optimal because fixed weights restrict its representational power. Furthermore, there is no accumulation or transfer of knowledge inside the model when new tasks are learned. Hence, we propose ExSSNeT (Exclusive Supermask SubNEtwork Training), that performs exclusive and non-overlapping subnetwork weight training. This avoids conflicting updates to the shared weights by subsequent tasks to improve performance while still preventing forgetting. Furthermore, we propose a novel KNN-based Knowledge Transfer (KKT) module that dynamically initializes a new task's mask based on previous tasks for improving knowledge transfer. We demonstrate that ExSSNeT outperforms SupSup and other strong previous methods on both text classification and vision tasks while preventing forgetting. Moreover, ExSSNeT is particularly advantageous for sparse masks that activate 2-10% of the model parameters, resulting in an average improvement of 8.3% over SupSup. Additionally, ExSSNeT scales to a large number of tasks (100), and our KKT module helps to learn new tasks faster while improving the overall performance.

Under review as a conference paper at ICLR 2023
EXCLUSIVE SUPERMASK SUBNETWORK TRAINING FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual Learning (CL) methods mainly focus avoiding catastrophic forgetting and learning representations that are transferable to new tasks. Recently, Wortsman et al. (2020) proposed CL method, SupSup. which uses randomly initialized, fixed base network (model) and finds a supermask for each new task that selectively keeps or removes each weight to produce subnetwork. They prevent forgetting as the network weights are not being updated. Although there is no forgetting, the performance of supermask sub-optimal because fixed weights restrict its representational power. Furthermore, there is no accumulation or transfer of knowl- edge inside the model when new tasks are learned. Hence, we propose EXSSNET (Exclusive Supermask SubNetwork Training), that performs exclusive and non- overlapping subnetwork weight training. This avoids conflicting updates to the shared weights by subsequent tasks improve performance while still preventing forgetting. Furthermore, we propose a novel KNN-based Knowledge Transfer (KKT) module that dynamically initializes tasks for improving knowledge transfer. We demonstrate that EXSSNET outper- forms SupSup and other strong previous methods on both text classification and vision tasks while preventing forgetting. Moreover, EXSSNET is particularly ad- vantageous for sparse masks that activate 2-10% of the model parameters, resulting in an average improvement of 8.3% over SupSup. Additionally, EXSSNET scales to a large number of tasks (100) and our KKT module helps to learn new tasks faster while improving the overall performance.
task's 's mask based on previous
INTRODUCTION
In artificial intelligence, the overarching goal is develop autonomous agents that can learn to accomplish a set of tasks. Continual Learning (CL) (Ring, 1998; Thrun, 1998; Kirkpatrick et al., 2017) key ingredient for developing agents that can accumulate expertise on new tasks. However, when a model is sequentially trained on tasks ( and l2 with different data distributions, the model's ability to extract meaningful features for the previous task t1 degrades. This loss in performance on the previously learned tasks, is referred to as catastrophic forgettt (CF) (McCloskey & Cohen, 1989; Zhao Schmidhuber, 1996; Thrun, 1998; Goodfellow et al., 2013). Forgetting is a consequence of two phenomena happening in conjunction: (1) not having access to the data samples from the previous tasks, and (2) multiple tasks sequentially updating shared model parameters resulting in conflicting updates, which is called as parameter interference (McCloskey & Cohen, 1989).
Recently, some CL methods avoid parameter interference by taking inspiration from the Lottery Ticket Hypothesis (Frankle & Carbin, 2018) and Supermasks (Zhou et al., 2019) to exploit the expressive power of sparse subnetworks. Zhou et al. (2019) observed that the number of sparse subnetwork combinations is large enough (combinatorial) that within randomly weighted neural networks, there exist supermasks that achieve good performance. A supermask is a sparse binary mask that selectively keeps or removes each connection in fixed and randomly initialized network to produce a subnetwork with good performance on given task. We call this the subnetwork as supermask sub- network that is shown in Figure I, highlighted in red weights. Building upon this idea, Wortsman et al. (2020) proposed a CL method, SupSup, which initializes network with fixed and random weights
Our code is uploaded supplementary material.

---

Under review as a conference paper at ICLR 2023
Mask Mi over waights Wrll
Mask Mr over weights 10000)
Maak Mj over waights Writ)
Task1
Task2
Taskk
Randomly initializzd weights Wr(1)
Masde
Tran Inoe pworloping weights
Find Mast
Trih
Fird Mask
Task1 Traning
Taek2 Traning
Traning
Untrined weights
Trained weights
Over axprapping waightt
(Mi, wat)
M2, Wr(a
(Ms. W
Oeoflipping indigtt--an updettd
Figure 1: EXSSNET diagram. We start with random weights W For task I, we first learn a supermask Mi (the corresponding subnetwork is marked by red color, column row 1) and then train the weight corresponding to M1 resulting in weights W(() (bold red lines, column row 2). Similarly, for task 2, we Iearn the mask M2 over fixed weights W mask M2 weights overlap with M, (marked by bold dashed green lines in column 3 row 1), then only the non-overlapping weights (solid green lines) of the task subnetwork are updated (as shown by bold and solid green lines column 3 row 2). These already trained weights (bold lines) are not updated by any subsequent task. Finally, for task 3, we learn the mask Ms (bluc lines) and update the solid blue weights.
and then learns different supermask for each new task. This allows them to prevent catastrophic forgetting (CF) as there is no parameter interference (because the model weights are fixed).
Although SupSup (Wortsman et al., 2020) prevents CF, there are some problems to using supermasks for CL: (1) Fixed random model weights in SupSup limits the supermask subnetwork 's representa- tional power resulting in sub-optimal performance. As shown in Figure 2, for single task the test accuracy of SupSup is approximately 10% worse compared to a fully trained model where all model weights are updated. As a possible remedy, one could try to naively train the weights corresponding to supermask subnetworks of different tasks; however, it can lead to CF as shown in Figure 3. This happens because subnetworks for different tasks can overlap and training subnetworks weights might result in parameter interference. (2) When learning a task, there is no mechanism for transferring knowledge from previously learned tasks to better learn the current task. Moreover, the model not accumulating knowledge over time as they weights are not being updated.
We overcome the aforementioned issues, we propose our method, EXSSNET (Exclusive Supermask SubNFtwort T Training), pronounced as 'excess-net', which first learns mask for task and then selectively trains a subset of weights from the supermask subnetwork. We train the weights of this subnetwork via exclusion that avoids updating parameters from the current subnetwork that have already been updated by any of the previous tasks. This helps us to prevent forgetting. This procedure is demonstrated in Figure 1 for learning three tasks sequentially. Training the supermask subnetwork s weights increases its representational power and allows EXSSNET to encode task task-specific knowledge inside the subnetwork. This solves the first problem and allows EXSSNET to perform comparable to a fully trained network on individual tasks; and when learning multiple tasks, the exclusive subnetwork training improves the performance of each task while still preventing forgetting.
To address the second problem of knowledge transfer, we propose a k-nearest neighbors based knowledge transfer (KKT) module that transfers relevant information from the previously learned tasks to improve performance on new tasks while learning them faster. Our KKT module uses KNN classification select a subnetwork from the previously learned tasks that has better than random predictive power for the current task and use it as starting point to learn the new tasks.
Next, we show our method's 's advantage by experimenting with both natural language and vision tasks. For natural language, we evaluate on WebNLP classification tasks (de Masson d' Autume et al., 2019; Huang et al., 2021) and GLUE benchmark tasks (Wang et al., 2018), whereas, for vision, we evaluate on SplitMNiST (Zenke et al., 2017; De Lange & Tuytelaars, 2021). SplitCIFAR 100 (Chaudhry et al., 2018; De Lange & Tuytelaars, 2021), 1), and SplifTinylmageNer (Buzzega et al., 2020) datasets. We show that for both language and vision domains, EXSSNET outperforms multiple strong and recent continual learning methods based on replay, regularization, distillation, and parameter isolation. For the vision domain, EXSSNET outperforms the strongest baseline by 4.8% and 1.4% on SplitCIFAR and Ingigeenylmaeeeet datasets respectively, while surpassing multitask model and bridging the gap to training individual models for each task. In addition, for GLUE datasets, EXSSNET is 2%
2

---

Under review as a conference paper at ICLR 2023
Average Test Accuracy
Average Test Apcuracy
60-
50
Methods
Methods
SSNeT
EXBSINTT
30
Fully Tished
SupSup
30-
SupSup
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0
10
20
30
40
50
60
70
80
Mask Density
Average Sparse Overlap
Figure 2: Test accuracy versus the mask den- sity for 100-way CIFAR 100 classification. Re- sults averaged over three seeds.
Figure 3: Average Test accuracy on five 20-way tasks from SplitCIFARIO R00 as function of sparse overlap. Results are averaged over three seeds.
better than the strongest baseline methods and surpasses the performance of multitask learning (that has access to all the tasks at once and does not suffer from CF). Moreover, EXSSNET obtains an average improvement of 8.3% over SupSup for sparse masks with 2 10% of the model parameters and scales to large number of tasks (100). Furthermore, EXSSNET with the KKT module learns new tasks in as few as 30 epochs compared to 100 epochs without it, while achieving 3.2% higher accuracy on the SplitCIFAR 100 dataset. In summary, our contributions are listed below:
We propose a simple and novel method improve mask learning by combining it with exclusive subnetwork weight training to improve CL performance while preventing CF.
We propose a KNN-based Knowledge Transfer (KKT) module for supermask initialization that dynamically identifies previous tasks to transfer knowledge learn new tasks better and faster.
Extensive experiments on language and vision tasks show that EXSSNET outperforms strong baselines and comparable to multitask model for language tasks while surpassing for vision tasks. Moreover, EXSSNET works well for sparse masks and scales to large number of tasks.
2 MOTIVATION
Using sparsity for CL is an effective technique to learn multiple tasks, i.e. by encoding them in different subnetworks inside a single model. SupSup (Wortsman al., 2020) is an instantiation of this that initializes the network weights randomly and then learns a separate supermask for each task as shown in the SupSup diagram in Appendix Figure 7. They prevent CF because the weights of the network are fixed and never updated. However, this is crucial problem as discussed below.
Problem 1- Sub-Optimal Performance of Supermask: Although fixed network weights SupSup prevent CF, this also restricts the network's representational capacity, leading worse performance compared to a fully trained network. We demonstrate this Figure 2, where we report the test accuracy of different methods with respect to the fraction of network parameters selected by the mask, ie., the mask density for an underlying ResNet18 model (Wortsman et al., 2020) on a single 100-way classification CIFAR 100 dataset. The fully trained ResNet18 model (dashed green line) achieves an accuracy of 63.9%. Similar to Zhou et al. (2019), we observe that the performance of SupSup (yellow dashed line) is least 8.3% worse compared to fully trained model. As possible partial remedy, we propose simple solution, SSNET (Supermask SubNerwork Training), that first finds subnetwork for task and then trains the subnetwork's weights. This increases the representational capacity of the subnetwork because there are more trainable parameters. For a single task, the test accuracy of SSNET is better than SupSup for all mask densities and matches the performance the fully trained model beyond density threshold. But as shown below, when learning multiple tasks sequentially, SSNET gives rise to parameter interference that results CF
Problem 2 Parameter Interference Due to Subnetwork Weight Training for Multiple Tasks: Next, we demonstrate that when learning multiple tasks sequentially, SSNET can still lead CF In Figure 3, we report the average test accuracy versus the fraction of overlapping parameters between the masks of different tasks, i.e., the sparse overlap (see Equation 1) for five different 20-way classification tasks from SplitCIFAR 100 dataset with ResNet18 model. We observe that SSNET outperforms SupSup for lower sparse overlap but as the sparse overlap increases, the performance declines because the supermask subnetworks for different tasks have overlapping (common)
3

---

Under review as a conference paper at ICLR 2023
weights (bold dashed lines in Figure 1). This leads to higher parameter interference resulting in increased forgetting which suppresses the gain from subnetwork weight training.
To address this second problem, final proposed method, ExSSNET selectively trains subset of the weights in the supermask subnetwork to prevent parameter interference. When learning multiple tasks, this prevents CF resulting in strict improvement in performance over SupSup shown in Figure 3 while having the representational power to bridge the gap with fully trained models, see Figure 2.
3 METHOD
As shown in Figure I, when learning supermask M₁ for the task; (2) We use all the previous tasks' masks M1,..., M to create a free parameter mask Mfrr that finds the parameters selected by the mask M, that were not selected by any of the previous masks; (3) We update the weights corresponding to the mask Mefree this avoids parameter interference. Now, we formally describe all the step of our method EXSSNET (Exclusive Supermask SubNetwork Training) for Multi-layer perceptron (MLP).
task t,, ExSSNET follows three steps: (1) We learn
When finding supermasks and training subnetworks weight, we can treat each layer of an MLP network separately. An intermediate layer has ny nodes denoted by V() = {vI...... Uny For node v in layer let In denote its input and 2, a (I,) denote its output, where a(.) is the activation function. Given this notation, In can be written as Ip = Eueva 1) Wur Z.. where who is the network weight connecting node u to node The complete network weights for the MLP are denoted by W. When training the task 4, we have access to the supermasks from all previous tasks {M, 1jzl and the model weights ( obtained after learning task t;
Notation:
3.1 EXSSNET: EXCLUSIVE SUPERMASK SUBNETWORK TRAINING
Finding Supermasks: Following Wortsman et al. (2020), we use the algorithm of Ramanujan et al. (2019) to learn a supermask Mj for the current task t,. The supermask M, learned with respect to the underlying model weights W((-1) and the mask selects fraction of weights that lead to good performance on the task without training the weights. To achieve this, we learn score Suy for each weight wav, and once trained, these scores are thresholded to obtain the mask. Here, the input to node v is I. = Eueva function which outputs 1 for top-k% of the scores in the layer with & being the mask density. Next, we use straight-through gradient estimator (Bengio al., 2013) and iterate over the current task's data samples to update the scores for the corresponding supermask M, as follows,
Wae Mu where mae = A(swv) is the binary mask value and l((.) is
OL OI, OC VI, Ozuv aI, Wuo Zn.
Sun = Sup arsavi
Finding Exclusive Mask Parameters: Given learned mask M,, we use all the previous tasks' masks Mi...., MM-1 to create a free parameter mask M that finds the parameters selected by the mask M, that were not selected by any of the previous masks. We do this by (1) creating new mask M1 containing all the parameters already updated by any of the previous tasks by taking a union of all the previous masks {M by using the logical or operation, and (2) Then we obtain a
mask "rre by taking the intersection of all the network parameters not used by any previous task which is given by the negation of the mask M1--1 with the current task mask M, via logical and operation. Next, we use this mask M, free for the exclusive supermask subnetwork weight training.
Exclusive Supermask Subnetwork Weight Training: For training the subnetwork parameters for task t, given the free parameter mask free we we perform the forward pass on the model after applying the mask Mfree free as model(a, W @ Mfrree, where @ is the element-wise multiplication. Hence, dur- ing the weight training, only the parameters corresponding the mask M are updated because the gradient value is 0 for all the weights Wwe where freee use the mask Md. In contrast, SSNET uses the task mask M, both during the training and inference as model(x, (x. ((
While during the inference on task i; we
This updates all the parameters in the mask including the parameters that are already updated by previous tasks that result in CF. Therefore, in cases where the sparse overlap is high,
4

---

Under review as a conference paper at ICLR 2023
EXSSNET is preferred over SSNET. To summarize, ExSSNET circumvents the CF issue of SSNET while benefiting from the subnetwork training to improve overall performance as shown in Figure 3.
3.1.1 SPACE, TIME, AND MEMORY COMPLEXITY OF EXSSNET
For training, we store an additional set of scores on GPU with size as the model weight. The additional GPU memory required is a small fraction because the model activations account for a huge fraction of the total GPU memory. Our runtime is similar to training the weight of model with < 5% overhead due to the logical operations on masks and masking weight during the forward passes. For training time comparisons refer Appendix Table 10. On the disk, need to store k |W| updated weights of 32-bits and boolean mask which takes 1-bit for each parameter. Hence, we take max [W] * 1, [W]) 32 + [W| 1 bits in total as in the worst case need store all [W] model weights.
3.2 KKT:
KNOWLEDGE TRANSFER MODULE
When learning multiple tasks, is a desired property to transfer information Iearned by the previous tasks to achieve better performance on new tasks and to learn them faster (Biesialska et al., 2020). Hence, we propose a K-Nearest Neighbours (KNN) based knowledge transfer (KKT) module that uses KNN classification to dynamically find the most relevant previous task (Veniat et al., 2021) to initialize the supermask for the current task. To be more specific, before learning the mask M, for the current task 4, we randomly sample small fraction of data from task 4, and split it into a train and test set. Next, we use the trained subnetworks of each previous task f1.... f to obtain features on this sampled data. Then we learn i 1 independent KNN classifssiiin models using these features. Then we evaluate these - models on the sampled test set to ohtain accuracy scores which denote the predictive power of features from each previous task for the current task. Finally, we select the previous task with the highest accuracy on the current task. If this accuracy is better than random then we use its mask initialize the current task's supermask. This enables EXSSNET to transfer information from the previous task to learn new tasks better and faster.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP AND TRAINING DETAILS
Datasets: For natural language domain, follow the shared text classification setup of IDBR (Huang et al., 2021). LAMOL (Sun et al., 2019), 9), and MBPA++ (De Lange et al., 2019) to sequentially learn five text classification tasks; (1) Yelp Sentiment analysis (Zhang et al., 2015); (2) DBPedia for Wikipedia article classification (Mendes et al., 2012) (3) Yahoo! Answer for Q&A classification (Chang et al., 2008); (4) Amazon sentiment analysis (McAuley & Leskovec, 2013) (5) AG News for news classification (Zhang et al., 2015). We call them WebNLP classification tasks for easier reference. While comparing with the previous ofate-of-the-art text methods, we use the same training and test set as IDBR and LAMOL containing 1 10000005000760000 Train/Val/Tes examples. For our ablation studies, we follow IDBR and use a sampled dataset, please see Appendix Table for statistics. Additionally, we create a CL. benchmark using the popular GLUE classification tasks (Wang et al., 2018) consisting of more than 5k train samples. We use the official validation split as test data and use 0.1% of the train data to create a validation set. Our final benchmark includes five tasks; MNLI (353k/39k/9.8) (k)) QQP (327k//6k/40k), QNLI (94k/100 10/05.4554 5k)),, SST-2 600k/6.7k/872) CoLA (7.6k/856/1k) k)). For vision experiments, we follow SupSup and use three CL benchmarks, SplitMNIST (Zenke et al., 2017) SplitCIFAR 100 (Chaudhry et al., 2018). and SplicTinylmageNet (Buzzega et al., 2020) datasets with 10, 100 and 200 total classes respectively.
Metrics: We follow Chaudhry et al. (2018) and evaluate our model after learning task on all the tasks, denoted by TT This gives an accuracy matrix A Rann where 0z,j represents the classification accuracy on task j after learning task i. We want the model to perform well on all the tasks it has been learned. This is measured by the average accurucy, A(T) x E aN,K: where N is the number of tasks. Next, we want the model to retain performance on the previous tasks when learning multiple tasks. This is measured by the forgetting metric (Lopez-Pat -Pzz & Ranzato, 2017), F(T): N-1 E -1 (max (1 N-1.) 1) ag,t - ax.)). This is the average difference between the maximum accuracy obtained for task and its final accuracy. Higher accuracy and lower forgetting are desired.
5

---

Under review as a conference paper at ICLR 2023
Method (1)
GLUE
WebNLP
Order (-)
S1
S2
S3
S4
S5
Average
Random
33.3
(-14
(-)
7-14
(-14
7-)
Mullilask
(0.0)
77.2 (0.0)
(0.0)
77.2 (0.0)
77.2 (0.0)
77.2 (0.0)
Individual
87.7 (0.0)
79.5
79.5
79.5 (0.0)
79.5
FT
14.1 (86.0)
26.9 (62.1)
22.8 (67.6)
(55.9)
15.6 (76.8)
24.0 (65.6)
AdaptBERT + FT
24.7 (53.4)
20.8 (68.4)
19.1 (70.9)
(645)
14.6 (76.0)
19.6 (70.0)
AdaptBERT + Replay
78.5 (3.8)
(3.0)
(2.0)
(2.0)
(2.6
74.2 (2.3)
MuliAAaPtBERT
76.7 (0.0)
76.7 (0.0)
(0.0
76.7 (0.0)
76.7 (0.0)
Prompt Tuning
76.3
66.3 (0.0)
(0.0)
66.3 (0.0)
66.3 (0.0)
66.3 (0.0)
Regularization
(8.8)
74.9 (2.0)
72.2 (6.8
76.5 (2.0)
76.0
Replay
77.7 (4.8)
(3.1)
(3.5)
75.7 (3.1)
(3.0)
MBPA++1
(-
())
(-)9
7-
74.3 (-)
LAMOL;
7-)1
7-)1
77.2
76.7
76.5 (-)
IDBR
73.0 (6.8)
(2.7)
(3.5)
(1.6)
(1.9)
76.0 (2.4)
SupSup
78.3 (0.0)
(0.0)
76.1 (0.0)
76.0 (0.0)
75.9 (0.0)
(0.0)
SSNET
78.4 (3.6)
76.3 (0.8)
76.3 (0.8)
(0.3)
76.3 (0.3)
76.3 (0.6)
EXSSNET
80.5 (0.0)
77.0 (0.0)
77.1 (0.0)
76.7 (0.0)
76.9 (0.0)
76.9 (0.0)
Table 1: Comparing average test accuracy 1 (and forgetting metrie J) for multiple tasks and sequence orders with statc-of-thc-art (SotA) methods. Results with f are taken from (Huang et al., 2021).
Sparse Overlap to Quantify Parameter Interference: Next, we propose measure quantify pa- rameter interference for task i, i.e., the fraction of the parameters in mask M, that are already updated by some previous task. We define sparse overlap the difference between the number of parameters selected by mask M, and " divided by the total parameters selected by Mr. Formally, we define sparse overlap (SO) between current supermask Mj and supermasks for previous tasks { {M,} 1 as,
SO(M, (MG){=)) swm(M.) su
M = M A (V)
swm(M.)
where A, V, - are logical and, or, and not operations.
For both vision and language (VL) tasks, we compare with: (VL.I) Naive Training/Finetuning (Yogatama et al., 2019): where for the vision domain we train all model parameters from scratch whereas for the language domain we finetune BERT model weights for each task sequentially. (VL.2) Experience Replay (ER) (de Masson d'Autume et al., 2019): we replay previous tasks examples when we train new tasks; (VL.3) Multitask Learning (Crawshaw, 2020): where all the tasks are used jointly to train the model and have strong performance; (VL.44 Individual Models: where we train separate model for each task. This is considered an upper bound for CL; (VL.5) Supsup (Wortsman et al., 2020). For natural language (L), we further compare with the follow- ing methods: (L.66 Regularization (Huang et al., 2021): Along with the Replay method, we regularize the hidden states the BERT classifier with an L.2 loss term; We show three Adapter BERT (Houlsby et al.. 2019) variants, (L.7) AdaptBERT + FT where we have single adapter which is finetuned for all task; (L.8) AdaptBERT + ER where a single adapter is finctuned with replay; (L.9) Multi- AdaptBERT where separate adapter is finetuned for each task; (L.10) Prompt Tuning (Li & Liang, 2021) that learns 50 different continuous prompt tokens for each task. (I..1)) MBPA++ (de Mas- son d' d'Autume et al., 2019) perform replay with random examples during training and does local adap- tation during inference to select replay example; (L.12) LAMOL (Sun et al., 2019) uses a language model to generate pseudo-samples for previous tasks for replay; (L.13) IDBR (Huang et al., 2021) dis- entangles hidden representations into generic and task mask-specifii representations and regularizes them while also performing replay. For vision task (V), we additionally compare with two popular regular- ization based methods, (V.6) Online EWC (Schwarz et al., 2018), (V.7) Synaptic Intelligence (SI) (Zenke et al., 2017); one knowledge distillation method, (V:8) Learning without Forgetting (LwF) (Li & Hoiem, 2017), three additional experience replay method, (V.9) AGEM (Chaudbry et al., 2018). (V.10) Dark Experience Replay (DER) (Buzzega et al., 2020), (V.11) DER++ (Buzzega et al., 2020).
Previous Methods and Baselines:
We focus on the task incremental setting (Hsu et al., 2018) and unless otherwise specified, we obtain supermasks with a mask density of 0. For language tasks, unless specified otherwise initialize the token embedding for methods using frozen BERT-base- uncased (Devlin et al., 2018) model's 's representations using Huggingface (Wolf et al., 2020). We use static CNN model from Kim (2014) 4) as our text classifier. Following Sun al. (2019); Huang et al.
Implementation Details:

---

Under review as a conference paper at ICLR 2023
Method
S-MNIST
S-CIFARI00
S-TinylmugeNe
Val. Accuracy
70-
Muliiask
(0.5(
(3.0
(5.9
68
60
Individual
(0.0)
75.5 (0.0)
53.7 (0.0)
50
Naive Sequential
(25.0)
19.3 (73.7)
11.5 (43.9)
40
40
EWC
96.1
(4.5) (4.5)
32.4 (60.5)
20.5 (52.1)
20
40
60
00
0
20
40
60
80
(0.6)
46.1 (47.8)
(46.2)
LwF
(0.))
29.5 (70.2)
18.1 (56.5)
AGEM
(1.3
52.1 (42.0)
21.6 (54.9)
Val. Accuracy
70
DER
ER
(1.9().2)
(0.6
60.1 (27-5)
62,5 (28.4)
35.9 (37.7)
35.6 (36.0)
60
60-
70
Speedtp
DER++
(1.8)
62.5 (27.5)
36.2 (35.7)
50
50
bitshee sK1
SupSup
99.6 (0.0)
62.1 (0.0)
50.6 (0.0)
-0
40
SSNET
(0.0)
23.9 (54.4)
(1.9)
EXSSNET
99.7 (0.0)
67.3 (0,0)
52.0 (0.0)
20
40
60
20
Epochs
Epochs
Table 2: Average accuracy t (Forgetting metric L) on all tasks for vision. For our method we report the results are averaged over three random seeds.
Figure 4: We plot validation accuracy vs Epoch for ExSSNET and ExSSNET KKT. We ob- serve that KKT helps to learn the subsequent tasks faster and improves performance.
(2021), for WebNLP datasets we learn different task orders S1-S52 that are provided in Appendix Table Following (Wortsman et al., 2020), we use LeNet (Lecun et al., 1998) for SplitMNIST, ResNet50 model (He et al., 2016) for
Resnet-18 18 model (Wortsman et al., 2020) for SplitCIFARIOM TinylmageNet datasets. Unless specified, we randomly split all the vision datasets to obtain five tasks with disjoint classes. In all our experiments, all methods perform an equal number of epochs over the datasets. For additional details on implementation and hyperparameters refer to Appendix A.1.1.
4.2
MAIN RESULTS
Q1. Does Supermask Subnetwork Training Help?
In these experiments, we show that EXSSNET outperforms multiple strong baseline methods including SupSup. For our main language experiments in Table sequentially learn multiple task orders, $1 S52 corresponding to the GLUE and WebNLP benchmarks. These task orders are listed in Appendix Table 6. We report the average test accuracy (and forgetting in parentheses). For natural language, we perform better than previous state-of-the-art CL methods in four out of five cases, across multiple task orders, and in aggregate. Specifically, the GLUE benchmark, EXSSNET is least 2.0% better than other methods while avoiding CF. Furthermore, EXSSNET either outperforms or is close to the performance of the multitasking bascline which is a strong bascline for CL methods.
For vision tasks, we split the MNIST, CIFARI00, and TinyImageNet datasets into five different tasks with an equal number of disjoint classes and report results. We report these results in Table 2 and observe that EXSSNET leads to 4.8% and 1.4% improvement over the strongest baseline for Split-IIFAR 100 and Split-TinylmageNee datasets. Furthermore, both EXSSNET and SupSup outperform the multitask baseline. Moreover, EXSSNET bridges the gap to individually trained models significantly, for TinyImageNer we reach within 1.7% of individual models' performance. The average sparse overlap of EXSSNET is 19.4% across all three datasets implying that there is a lot more capacity in the model. See appendix Table for sparse overlap of other methods.
We note that past methods employ specific tricks like local adaptation in MBPA++, generative replay module in LAMOL, and experience replay in AGEM, DER, and ER. In contrast, EXSSNET does not require replay and simply trains different subnetworks for tasks.
Q2. Can KKT Knowledge Transfer Module Share Knowledge Effectively?
In Section 3.2, presented our KKT module for enabling knowledge sharing across tasks. In Table 3, we show that adding the KKT module to EXSSNET, SSNET, and SupSup improves performance on vision benchmarks. The experimental setting here is similar Table 2. We observe across all methods and datasets that the KKT module improves average test accuracy. Specifically, for the Split-CIFAR 100 dataset, the KKT module results in 5.0%, and 3.2% improvement for SupSup
For example, in S2 order the model learns the task in this order, ag - yelp amazon - yahoo dbpedia

---

Under review as a conference paper at ICLR 2023
Method
S-MNIST
S-CIFARI00
S-TinyImageNt
Avergee Tesss Accurccy
SupSup KKT
99.6
62.1
50.6
99.6
67.1 [45.0]
53.3 [+2.7]
SSNET
99.7
23,9
49,6
60-
+ KKT
99.3 [-0.4]
23.5 [-0.4]
51.8 [+2.2]
Methods
EXSSNoT
EXSSNET
99.7
67.3
52.0
SSNeT
KKTT
99.7 [+0.0]
70.5 [+3.2]
54.0 [+2.0]
SupSup
Table 3: We report average test accuracies f [and gains from KKTJ when using the KKT knowledge sharing module for Vision datasets. The overall best method is highlighted in gray.
800
Mask Density (Log Scale)
Figure 5: Average test accuracy versus mask density on SplitCIFAR 100 datasct.
Method
S-Tnnymageeeet
Avg. Sparse Overlap
Method
FastText
Glove
BERT
SupSup
90.34 (0.0)
SupSup
54.01
55.52
74.0
SSNET
89.02 (2.2)
90.0
SSNET
60.41 [+6.4]
59.78 [+4.3]
74.5 [+0.5]
EXSSNET
91.21 (0.0)
90.0
EXSSNET
62.52 [+8.5]
62.81 [+7.3]
74.8 [+0.8]
Table 4:
Average accuracy f (forgetting metric J) and average sparse overlap when learning 100 tasks.
Table 5:
Ablation result for token embeddings. We
report average accuracy 1 [and gains over SupSup)
and EXSSNET respectively; while for Split-TinylmageNe, EXSSNET + KKT outperforms the individual models. We observe performance decline for SSNET when using KKT because KKT promotes sharing of parameters across tasks which can lead to worse performance for SSNET. Furthermore, EXSSNET + KKT outperforms all other methods on both the Split-CIFAR 100 and Split- Iint--Tinlmggeeee datasets. For EXSSNET + KKT, the average sparse overlap is 49.6% across all three datasets (see appendix Table 8). These results indicate that the benefits of weight training and the KKT module can be coupled together to further improve performance.
Q3. Can KKT Knowledge Transfer Module Improve Learning Speed of Subsequent Tasks?
Next, we show that the KKT module enables us to learn new tasks faster. To demonstrate this, in Figure 4 we plot the running mean of the validation accuracy epochs for different tasks from the Split- CIFARIO0 experiment in Table 3. We show curves for EXSSNET with and without the KKT module and omit the first task as both these methods are identical for Task because there is no previous task to transfer knowledge. For all the subsequent tasks (Task 2,5, we observe that - (1) EXSSNET + KKT starts off with a much better initial performance compared EXSSNET (2) given fixed number of epochs for training. EXSSNET+ + KKT always learns the task better because it has a better accuracy at all epochs; and (3) EXSSNET KKT can achieve similar performance as EXS in much fewer epochs as shown by the green horizontal arrows. This clearly illustrates that using the KKT knowledge- transfer module not only helps to learn the tasks better (see Table 3) but also learns them faster.
4.3 ADDITIONAL RESULTS AND ANALYSIS
Q4. Effect of Mask Density on Performance: Next, show the advantage of using EXSSNET when the mask density is low. In Figure 5, we show the average accuracy for the Split-CIFARI00 dataset function of mask density. We observe that EXSSNET obtains 7.9%, 18.4%, 8.4%, and 4.7% improvement over SupSup for mask density values 0.02, 0.04, 0.06, 0.08 respectively. This is a highly desirable property as tasks select fewer parameters which inherently reduces sparse overlap allowing EXSSNET to learn a large number of tasks.
Q5. Can EXSSNET Learn Large Number of Tasks? SupSup showed that it can scale a large number of tasks. Next, we show that this property is preserved by EXSSNET. We perform experiments to learn 100 tasks created by splitting the TinylmageNet dataset. Note that, as the number of task increase, the sparse overlap between the masks also increases resulting in fewer free model weights. In the extreme case where there are no free weights, EXSSNET by design reduces to SupSup because there will be weight training. From Table 4, conclude that EXSSNET can learn 100 tasks while still improving performance over SupSup and preventing forgetting.
Q6. Effect of Token Embedding Initialization for NLP: For our language experiments, a pretrained BERT model (Devlin et al., 2019) to obtain the initial token representations. Using the powerful representations from multitask model like BERT overshadows the superiority of
8

---

Under review as a conference paper at ICLR 2023
EXSSNET over SupSup in terms of performance. Hence, perform ablations on the token embedding initialization to further highlight these differences. In Table 5, we present the average test accuracy on the S2 task-order sequence of the sampled version of WebNLP dataset (see Section 4.1, Datasets). We initialize the token representations using FastText (Bojanowski et al., 2016), Glove (Pennington et al., 2014), and BER/' embeddings. From Table 5, observe that - (1) the performance gap between EXSSNET and SupSup increases from 0.8% + 7.3% and 0.8% - 8.5% when moving from BERT to Glove and FastText initializations respectively. These gains imply that it is even more beneficial to use EXSSNET in absence of good initial representations, and (2) the performance trend, ExSSNET > SSNET SupSup is consistent across initialization.
RELATED WORK
Continual Learning methods fall into three main categories: Regularization, Replay, and Architec- ture based methods. We point the readers to Delange al. (2021); Biesialska et al. (2020) for a comprehensive survey of all continual learning methods. Next, we discuss these main categories.
Regularization-base -ased methods estimate the importance of model components and add importance regularization terms to the loss function. Zenke et al. (2017) regularize based on the distance of weights from their initialization, whereas Kirkpatrick et al. (2017); Schwarz et al. (2018) use an approximation of the Fisher information matrix (Pascanu & Bengio, 2013) to regularize the parameters. In NLP, regularization methods (Han et al., 2020; Wang et al., 2019) are used to constrain the relevant information from the huge amount of knowledge inside large language models (LLM). Huang et al. (2021) first identifying hidden spaces that need to be updated versus retained via information disentanglement (Fu et al., 2017: Li et al., 2020) and then regularize these hidden spaces separately. Our method inherently avoids the need for regularization as it avoids parameter interference.
Replay based methods maintain a small memory buffer of data samples (De Lange et al., 2019) or their relevant proxies (Rebuffi et al., 2017) from the previous tasks and retrain for them later to prevent CF. Lopez-Paz & Ranzato (2017); Chaudhry et al. (2018) use the buffer during optimization to constrain parameter gradients. Shin et al. (2017); Kemker & Kanan (2018) uses a generative model to sample and replay pseudo-data during training, whereas Rebuffi et al. (2017) replay distilled knowledge from the previous tasks. de Masson d' d'Autume et al. (2019) employ episodic memory along with local adaptation for CL in the NI.P domain, whereas Sun et al. (2019) trains a language model to generate a sample for replay. These methods usually have associated memory and runtime costs whereas our method works without a replay buffer.
Architecture based methods can be divided into two categories: (1) methods that add new modules over time (Sodhani et al., 2019; Veniat et al., 2021; Douillard et al., 2022); and (2) methods that isolate the network's 's parameters for different tasks (Fernando et al., 2017; Mallya & Lazebnik, 2018; Mallya et al., 2018). Rusu et al. (2016) introduce a new network for each task that is connected to all the previous tasks resulting in super-linear growth in network size. Schwarz et al. (2018) fix this issue by distilling the new network after each task into the original one. Recent prompt models for vision (Wang et al., 2022a;b) assume access a pre-trained model learn a set of prompts that can potentially be shared across tasks to perform CL this is orthogonal to our method that trains from scratch. Methods like Fernando et al. (2017) initialize a fixed-size model and reuse subset of modules for each task by finding path in the graph of neural network modules. Mallya & Lazebnik (2018) allocates parameters to specific tasks and then trains them in isolation which limits the number of tasks that can be learned. In contrast, Mallya et al. (2018) use frozen pretrained model and learns new mask for each task but a pretrained model is crucial for their method's good performance. Worts- man et al. (2020) removes the pretrained model dependence and learns mask for each task over fixed randomly initialized network. EXSSNET avoids the shortcomings of Mallya & Lazebnik (2018); Mallya et al. (2018 8) and performs supermask subnetwork training to increase the representational capacity compared to (Wortsman et al., 2020) while performing knowledge transfer and avoiding CF
seamiing-hssed CL
DISCUSSION AND CONCLUSION
In this paper, we define a simple yet effective method EXSSNET and the KKT knowledge transfer module that leverages sparsity to learning tasks while resolving the shortcomings of supermasks by performing Exclusive Subnetwork Training. EXSSNET improves performance, prevents forgetting, accumulates and transfers knowledge across tasks, and works for both vision and language domains.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL
http:// /ttt://arxiv.. orooorg/a//eooo httg/abs/1300 3432.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jusså, Continual lifelong learning in natural language processing: survey. In Proceedings of the 28th International Conference on Computational Lingwistics, pp. 6523-6544 Barcelona. Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10.18653/v 1/20000cl/2000 57. URL https: haa/aaaaannttoooooo org/ 200. mamainn........
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. arXiv preprint
2016.
Pietro Buzzega, Mattco Boschini, Angelo Porrello, Davide Abati, and Simone Calderara, Dark experience for general continual learning: strong, simple baseline. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. -5920-15930.
Curran Associates, Inc., 2020.
Ming- Wiig-Weii Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence. Volume 2, AAAr08, pp. 830-835, AAAI Press, 2008. ISBN 9781577353683.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint 1eXiv:/812.00420 2018.
Michael Crawshaw. Multi-task learning with deep neural networks: A survey. ArXiv, abs/2009.09796. 2020.
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9250-8259, October 2021.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv: 1903:v:/999.00000 08383, 2(6), 2019.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic mem- ory in lifelong language learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL https: http://///ooopoppppoeooooooeeeet neurips. ps. http:ccpcppp......./// pdf.
hilee/fddee80ce 2ww.2440000000200aaa2a22233333
M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 00.119//PAAM222213305555
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18853/1//119111 1423. URL https: // //www. ac web. wwwebboorge. org/ aat/tntooooooo hocooologyy///ll - 1423.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
10

---

Under review as a conference paper at ICLR 2023
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017. URL http::////// //apivvaaivvv//////////eaa org/ aaaaabbs/11// 08734.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, arXiv preprint arXiv:1803.0363 2018.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Explo- ration and evaluation. arXiv preprint arXiv:/711.000 2017.
lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting gradient-bant baadee neural networks. arXiv preprint arXiv: 1312.6211, 2013.
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Continual relation learning via episodic memory activation and reconsolidation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 6429-6440, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/w/2002...cm m773 URL https: //www www.ww.ww..................ww. wellleeb.b... org/ antatotthhloyyyoo / 2020 acl -maimiann 573.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 09-15 Jun 2019. URL https: http://ooooceeeeeeeeooeooo mlr.
press/ (ress/v970 houl sby1 9a htmll.
Yen-Chang Hsu, Yen-Cheng Liu, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. CoRR, abs/1810.12455 2018. URL http://
wexivvorrge abs /1100 12488.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 7736,2746, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653/v1/222 maiel-mmin.2113 URL https: acl anthology org/ 2021 www.clmaaaaa........ 218.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations, 2018. URL /ttpps///penreviee. net/forum? -iw.umndd==Sdimmmm..m
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014
Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-177 1751, Doha. Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115//1/114-11111/
URL https:ttpp........... / antho ooy. .rggy..rrg/nn
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412:61412.690 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Grabska-Barwinska.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2227) (11(1)):278823333 2324, 1998. doi: 10. 109//.72679111

---

Under review as a conference paper at ICLR 2023
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 wwt.l.n..c-oo.35.... 333.. URL https::: / ac anthology. /t0/20211112 2021 aacll 333..
Yuan Li, Chunyuan Li, Yizhe Zhang, Xiujun Li, Guoqing Zheng, Lawrence Carin, and Jianfeng Gao. Complementary auxiliary classifiers for label-conditional text generation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05) Apr. 2020. doi: (0.1609/aai.vv34006 sil.aaaiii3i00..6444444 6644. URL https: /ojs aaai. org/ index. php /tpp artaacleeeeeee
6346.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2016.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 67-82, 2018.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, pp. 165-172, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450324090. doi: 10.114575507157225071633 URL https: //tp:///oiii//.. org/ 2507166..
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Pablo Mendes, Max Jakob, and Christian Bizer. DBpedia: A multilingual cross-domain knowledge base. In Proceedings of the Eighth International Conference on Language Resources and Eval- uation (LREC" /2), pp. 1813-1817, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA), URL http:/ /www. lrec- conf. inn.on org pttpppppocccc////oeepppp lrt.nngggg/eee/1//// pdf/570__Ppeee pdf.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint 10X.::1301.3584, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational URL https::t//// / a anthol ogy www.oooog/---/... 62.
Linguisties. doi: 10.31 10.3115/v1/1441110
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's 's hidden in a randomly weighted neural network? arXiv preprint 19Xiv: 2019.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Mark Ring. Child: A first step towards continual learning. In Learning to learn, pp. 261-292.
Springer, 1998.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606606666.6 2016.
12

---

Under review as a conference paper at ICLR 2023
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki. Agnieszka Yee whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning. arXiv preprint
2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Shagun Sodhani, Sarath Chandar, and Yoshua Bengio. Toward training recurrent neural networks for lifelong learning. Neural Computation, 32(1):1-35, 2019.
Fan-Keng Sun, Cheng-Hao Ho, and Hung- Yi Lee. Lamol: Language modeling for lifelong language learning. In International Conference on Learning Representations, 2019.
Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pp. 181-209. Springer, 1998.
Tom Veniat, Ludovic Denoyer, and MarcAurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. In International Conference on Learning Representations, 2021.
URL https::t///... wpppreveeeeeeeeeeeeeppe net forum? id= EKV158t
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 1/W//////I18/ 1446. URL https::t//// /aaaaaaaaatttaaae ho ogy org /W18- 5446.
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. Sentence embedding alignment for lifelong relation extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 2019. Association for Computational Linguistics. doi: 10. 10.11863/v1/1191111 1086. URL https: // //wwwww.o a cl web. org/ antholo ogy/ / /1 9-199- 1086.
Minncapolis, Minnesota, June
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference on Computer Vision, 2022a.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.006/55 15, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick vOn Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 48-45, Online, October 2020. Association for Computational Linguistics. doi: 1tt55/3/1122222mmmmll.... aclweb. org/ antho wow.hhloggy2200000 1m demos.emsssso.ooo 6.
URL https:////// /www.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 15173-15184, Curran Asso- ciates, Inc., 2020. URL https:// httttps:///ppoceeee http///ooeeenns..ececeeopoo...se neurips hccppaper//020/////eccp 221////20////2////////// pdf.
Dani Yogatama, Cyprien de Masson d' Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating general linguistic intelligence. arXiv preprint arxiv: /901. 1373, 2019.
13

---

Under review as a conference paper at ICLR 2023
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657, 2015.
Jieyu Zhao and Jurgen Schmidhuber. Incremental self-improvement for life-time multi-agent re- inforcement learning. In From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, Cambridge, MA, pp. 515,525, 1996.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, pp. 3592-3602, 2019.
A APPENDIX FOR EXSSNET
A.1
ADDITIONAL EXPERIMENTS AND DETAILS
ID
Task Sequence
S1
mnli +QP + -qali ss12 cola (Dec. data Size)
Dataset
Cluss
Type
Train
Validation
Test
ag yelp - amazon yahoo - dbpedia
AGNews
News
8k.
Sk.
7.6k
S3
yclp -elpooyhoo--a amazon dbpedia ag
Yelp
Sentiment
10k
10k
7.6k
S4
dbpediu - yahoo
amazon yelp
Amazon
Sentiment
7.6k
S5
yclp ag - dbpedia--aaaa- - -aazziiaaammzonn - yahoo
DBPedia
Wikipedia
28k
28k
7.6k
ag yel- yahoo
Yahoo
Q&AA
20k
20k
7.6k
S7
yahoo-oho
S8
yahoo aa yelp
Table 7: Statistics for sampled data used from Huang et al. (2021) for hyperparameter tuning. The validation is the same size the train set. Class means the number of output classes for text classification task. Type the domain of text classification.
Table 6: Task sequences used in text experiments. order corresponding
For the GLUE dataset, to decreasing train data size, Sequence $2-S8 are from (Huang et al., 2021: de Masson d' l'Autume et al., 2019; Sun et al., 2019).
A.1.1 EXPERIMENTAL SETUP AND HYPERPARAMETERS
Implementation Details: Unless otherwise specified, we obtain supermasks with a mask density of 0.1. In our CNN models, we use non-affine batch normalization to avoid storing their means and variance parameters for all tasks (Wortsman et al., 2020). Similar to (Wortsman et al., 2020), bias terms in model are 0 and we randomly initialize the model parameters using signed kaiming constant (Ramanujan et al., 2019). We use Adam optimizer (Kingma Ba, 2014) along with cosine decay (Loshchilov & Hutter, 2016) and conduct experiments GPUs with 12GB of memory. For our main experiment, we rum three independent runs for each experiment and report the averages for all the metries and experiments. For natural language tasks, unless specified otherwise the token embedding for our methods using a frozen BERT-base--nn -assu-a-uunceeed (Devlin et al., 2018) model's representations using Huggingface (Wolf al., 2020). We use static CNN model from Kim (2014) 4) as our text classifier over BERT representations. The model employs ID convolutions along with Tanh activation. Following Sun al. (2019); Huang et al. (2021), we evaluate our model on various task sequences as provided in Appendix Table 6, while limiting the maximum number of tokens to 256. Following (Wortsman al., 2020), we use LeNet (Lecun et al., 1998) for SplitMNIST dataset, Resnet- 18 model with fewer channels (Wortsman et al., 2020) for SplitCIFAR 100 dataset, ResNer50 model (He et al., 2016) for TinyImageNe dataset. Unless specified, we randomly split all the vision datasets obtain five tasks with disjoint classes. We use the codebase of DER (Buzzega al., 2020) to obtain the vision baselines. In all our experiments, all methods perform an equal number epochs over the datasets.
initialize
For the ablation experiment on natural language data, following Huang et al. (2021), we use sampled version of the WebNLP datasets due limited resources. The reduced dataset contains 2000 training and validation examples from each output class. The test set is the same as the main experiments. The dataset statisties summarized in Table 7. For WebNLP datasets, tune the learning rate
14

---

Under review as a conference paper at ICLR 2023
80-
LaF
Table 8: report the average sparse overlap for
Averoge Acourncy
method and datasct combinations reported in Table
AGEM
3.
80-
ER
Method
S-MNIST
S-CIFARI00
S-TinyImageNee
DER+
SupSop
22.6
18.1
+ KKT
46.4
48.3
52.4
Ex5SN21
SSNET
22.5
17.6
18.6
20
40
80
100
KKT
52.7
49.9
52.4
Num. Seen Classes
EXSSNET
22.5
17.3
18.5
Figure 6: Average Accuracy of all seen tasks as function of number of learned classes for the Split- CIFAR 100 dataset.
on the validation set across the values {0.01, 0.001, 0.0001), for GLUE datasets we use the default learning rate of the BERT model. For our vision experiments, use the default learning rate for the dataset provided in their original implementation. For TinyImageNet, SplicCIFAR 100, SplitMNIST dataset, for 30, 100, and 30 epochs respectively. We store 0.1% of our vision datasets for replay while for our language experiments we use 0.01 % of the data because of the large number of datasets available for them.
A.1.2 SPARSE OVERLAP NUMBERS
In Table 8, report the sparse overlap numbers for SupSup, SSNET, and EXSSNET with and without the KKT knowledge transfer module. This table corresponds to the results main paper Table 3.
A.1.3 AVERAGE ACCURACY EVOLUTION
In Figure 6, plot Eiso Au that is the average accuracy function of observed classes. This plot corresponds to the SplitCIFARI 100 results provided in the main paper Table 2. We can observe from these results that Supsup and ExSSNeT performance does not degrade when we learn new tasks leading very stable curve whereas for other methods the performance degrades learn new tasks indicating some degree of forgetting.
d
Model (1)
Length-5 WebNLP
Length-3 WebNLP
Order (-+)
S2
S3
S4
Average
S6
S7
S8
Average
Random
7.14
7.14
7.14
7.14
10.0
10.0
10.0
10.0
MTL.
75.09
75.09
75.09
75.09
74.16
74.16
74.16
74.16
Finetune
32.37
32.22
26.44
30.34
25.79
36.56
41.01
34.45
Replay
68.25
70.52
70.24
69.67
69.32
70.25
71.31
70.29
Regularization/
72.28
73.03
72.92
72.74
71.50
70.88
72.93
71.77
AdaptBERT
30.49
20.16
23.01
24.55
24.48
31.08
26.67
27.41
AdaptBERT+ Replay
69.30
71.98
69.73
66.12
71.62
68.96
IDBRI
72.63
73.72
73.23
73.19
71.80
72.72
73.08
72.53
SupSup
74.01
74.04
74.18
74.08
72.01
72.35
72.53
72.29
SSNET
74.5
74.5
74.65
74.55
73.1
72.92
73.07
73.03
EXSSNET
74.78
74.72
74.71
74.73
72.67
72.99
73.24
72.97
Table Average test accuracy reported over task sequences three independent runs on sub-sampled data. Results with are taken from Huang et al. (2021).
15

---

Under review as a conference paper at ICLR 2023
Method (J)
GLUE
WehNLP
Method
Runtime (in minules)
Order (-)
S1
S2
53
S4
S5
Average
Mulidaak
200
Finetune
33.3
Replay
204
Mulillsk
80.6
77.4
77.5
76.9
76.8
77.1
AdapterBERT
AdapterBERT Replay
173
Replay
79.7
75.2
74.5
75.2
75.5
75.1
MuldAAdapBERR
170
AdapBBERT FT
25.1
20.8
19.1
23.6
14.6
19.5
Regularization
257
AdaptBERT Replay
IDBR
258
MultAdaptBEKI
83.6
76.7
76.7
76.7
76.7
76.7
SupSup
117
Regularization
75.5
75.9
75.0
76.5
76.3
75.9
IDBR
77.5
75.8
75.4
76.4
76.4
76.0
SSNET
117
SupSup
78.1
75.7
76.0
76.0
75.9
75.9
EXSSNET
117
27.2
76.3
77.0
76.1
764
EXSSNET
80.1
77.1
71,3
71.2
77.1
77.2
Table 10: Runtime comparison of different methods used in the text experiments.
Table 11: Average validation accuracy (1) for multiple tasks and
sequence orders with previous tate-of-the--a (SotA) methods.
A.1.4 EFFECT OF TASK ORDER AND NUMBER TASKS
Following Huang et al. (2021), we conduct experiments to study the effect of task length and order in the language domain. We use task sequences of lengths three and five, with multiple different task orders on the sampled data (Section 1, Table 6, and Appendix) these variables on the performance. In Table 9, we present the average test accuracy averaged over three different random seeds. We observe that across all six different settings our method performs better compared to all the baseline methods. Our methods bridge the gap toward multitask methods' performance, leaving a gap of 0.36% and 1.19% for lengths three and five sequences, respectively.
characterize the impact of
A.1.5 RUNTIME COMPARISON ACROSS METHODS
In this Section, we provide the result to compare the runtime of various methods used in the paper. We ran each method on the sampled version of the WebNLP dataset for the task order as defined in Table 6. We report the runtime of methods for four epochs over each dataset in Table 10. Note that the masking-bass based method, SupSup, SSNET, EXSSNET takes much lower time because they are not updating the BERT parameters and are just finding a mask over much smaller CNN-based classification model using pretrained representation from BERT. This gives our method an inherent advantage that we are able to improve performance but with significantly lower runtime while learning a mask over much fewer parameters for the natural language setting.
A.1.6 VALIDATION RESULTS
In Table 11, we provide the average validation accuracies for the main natural language results presented in Table I. We do not provide the validation results of LAMOL (Sun et al., 2019) and MBPA+ (de Masson d' d'Autume et al., 2019) as we used the results provided in their original papers. For the vision domain, we did not use a validation set because no hyperparameter tuning was performed as we used the experimental setting and default parameters from the original source code from (Wortsman et al., 2020; Wen et al., 2020).
A.2 ADDITIONAL. MODEL DETAILS
A.2.1 ALGORITHM FOR EXSSNET
In Algorithm 1, we provide a pseudo-code for our method ExSSNET for easier reference and under- standing. We also attach our working code supplementary material to encourage reproducibility.
16

---

Under review as a conference paper at ICLR 2023
Algorithm 1 EXSSNET training procedure.
Input: Tisks T.
model model M,, mask spursity &s, exclusive-Tuee
Outgut: Trained model
Tritfall modal ghts W-(0)
initialize_ moode__eighhss[MM
forall ronge([7[]
b. Set the mask M, corresponding task f, for optimization. Laink_oot_ppaamss M;
the supeznask M; using edge-popup
forall em mavk spochss do
M: leam sumesrsssssskmddell tark__opp_aammsss.p f.)
end
Kodel weight at this point it's > and exchniive then
as the last iteration W0-1)
> Find mask for all the weights used provicua tasks.
M14-1 == V)z)(M))
> Get mask for weights in M; which are not in (M:)
Mfrree M, -Mi--1
Pind nen-oveslapping weight for updating.
w
free -M @ W
not exclasive then W(9 free W(f-1)
end
weight, paramss W free
b learn the free weight in the supernask M, forall em weightt_pooche do
w( update weights(mudel,, weight pprppans,, t.)
end
end
A.2.2 MODEL DIAGRAM FOR SUPSUP
In Figure 7, we provide the canonical model diagram for SupSup. Please read the figure description for more details regarding the distinctions between SupSup and ExSSNeT.
Randomly initialized weights W
Untraiined
Task
Task
Findt Mask for Task
Find Mazk for Tauk 3
Find Mask for Task
Mask M, over weights (!!9)
Mask M2 over weights (1)))
Mask Mg over weights (9)
Figure 7: This is canonical model diagram for SupSup. In SupSup. the model weights are always fixed at the random initialization (())) each task SupSup learns a new mask (in this case M1, M2, Mz) over the weights ( A mask selectively activates subset of weights for particular task. This subset of selected weights forms subnetwork inside the full model which refer the supermask subnetwork. For example, when learning Task SupSup learns mask M2 (the weights activated by the mask are highlighted green) over the fixed weight (1)). These highlighted weights along with the participating nodes are the subnetwork formed by mask M2. Whenever prediction is made for Task 2 samples, this mask is selected and used obtain the predictions. Please note that the model weights W/()) are never updated after their random initialization. Hence, for SupSup there is leamned knowledge sharing across tasks, This is in contrast I, where for the first task the mask is learned over the weights ()))) but once the mask is selected the weights of the corresponding subnetwork over these new set of weights W1(1) and on. Also note that in Figure 1, we do not show the KKT knowledge transfer module here avoid confusion.
setup in Figure
also updates obtain weight (7(1), Then the next task's mask learned
17


Reviews:
[{"review_id": "xglMBeZioIy", "reviewer": "Reviewer_fPix", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an Exclusive Supermask Subnetwork Training framework for continual learning of both text classification and vision tasks. Compared to the previous method SupSup, the proposed ExSSNet makes fixed weights trainable thus facilitating the knowledge transfer from previously learned tasks to new tasks. The further proposed KKT module can be regarded as a better Initialization mechanism, which helps to learn new tasks faster and better. Extensive experiments and impressing performance validate the effectiveness of the proposed method.", "summary_of_the_review": "I think this paper can be regarded as an good improvements of the previous method SupSup thus I tend to accept. But my main concern is about the claim of learning a large number of tasks. If the author can give good feedback, I will improve the score.", "strength_and_weaknesses": "Strength:\n- The idea of the overall framework is novel compared to previous work SupSup. It is a natural idea to expend fixed weights to trainable weights but it brings a large boost and more advantages, e.g., under the condition of sparse masks.\n- The experiments are sufficient and can effectively validate the claimed points. The informative figures and tables are also helpful for illustration and understanding.\n- The paper is well organized and comprehensive in terms of experimental settings and details.\n\nWeakness:\n- Although the idea is novel, most of the specific implementation of the framework is based on existing work. For example, the supermask learning follows Ramanujan et al.(2019), the training mechanism using exclusive mask is similar as Learn to Grow[A].\n- The description of the paper can be polished. Some long sentences interfere with reading, and there are also a few typos, e.g., the first sentence of the fourth paragraph, “We overcome the aforementioned issues, we propose our method,...”.\n- The main concern is about the claim that ExSSNet can learn 100 tasks. First, as listed in Table 4, SupSup actually achieves a good results (90.34%) and the improvement of ExSSNet (91.21%) is not noticeable. Therefore, the ability to learn 100 tasks is what SupSup have achieved, but not unique for ExSSNet. Second, I wonder that this kind of ability is related to the size of the model since the extreme case is there are no free weights. How is the performance of using a small model like LeNet for SplitMNIST and ResNet18 for SplitCIFAR100. Furthermore, how is the performance of using a larger model like Resnet101? Can larger models learn more than 100 tasks since they have more parameter spaces? \n\nReference:\n- [A] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher. \"Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.\" International Conference on Machine Learning. PMLR, 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "qcGHRDOctQ8", "reviewer": "Reviewer_XDRc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the author focused on developing the solution for mitigating the catastrophic forgetting problem in the continuous learning problem. The proposed framework is mainly composed of two components, the first component is exclusive and non-overlapping subnetwork weight training, The second component is the KNN knowledge transfer module. The authors have evaluated the proposed pipeline on several benchmarks and demonstrate the improvement over listed benchmarks. ", "summary_of_the_review": "My rating is between the borderline and weakly reject since the evaluation data is relative small. Not sure if it can be generalized to more complicated task with big data. ", "strength_and_weaknesses": "Strength:\n1.\tThe framework has adopted and expanded previous works’ solution and achieved better performance than listed benchmarks.\n2.\tThe idea is straightforward and easy to reproduce.\n\nWeakness:\n1.\tThe listed image benchmark is relatively simple, hard to justify the effectiveness of the proposed solution for dealing with big dataset. \n2.\tOne of the most important steps in this pipeline is the supermask learning. It needs to be learned when adding new task. The author has listed the operations after 3 tasks in figure 1. What will be the parameters complexity after N tasks? Will that be a problem in the long run of continuous learning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jD_hFLfazZF", "reviewer": "Reviewer_jWtC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is proposing a model for continual learning based on subnetwork isolation and training. Specifically, the authors build on top of SupSup, a prior method relying on a randomly initialized backbone where task-specific subnetworks are discovered and isolated, showcasing good performances on a stream of tasks to be learned. This paper extends the method by i) allowing subnetwork parameters to be updated by gradient descent, and therefore achieving better performance on their task and ii) introducing a KNN-based mask initialization method for bootstrapping subnetwork selection on the next task. Experiments are carried out for natural language processing and computer vision tasks.", "summary_of_the_review": "I appreciate the quality of the research and of the experimental section. However, in my opinion, the technical contribution of the paper is not worth for publication in a venue such as ICLR. I really think this work could benefit more discussion and comparisons with the parameter isolation methods I mentioned above.", "strength_and_weaknesses": "+ +The paper is very clear and well written, one read suffices to the interested reader to grasp the main ideas and technical contributions.\n+ +Experiments showcase encouraging results on two different data modalities and tasks, for a total of 5 different datasets among NLP and vision.\n+ +Experimental results are very encouraging, as the model is able to outperform a number of recent works, including some relying on replay buffers.\n+ +An ablation study grounds the effectiveness of KKT, grounding its beneficial effect.\n- -The biggest weakness of the paper is its technical novelty, that I'll discuss in the next point.\n- -The model, needing to separate the relevant task-specific subnetwork during inference, can only be employed in task-incremental learning settings, which represent the easiest and more controlled protocol in CL literature. This raises questions about the suitability of the model for real problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KigaYTrwZAV", "reviewer": "Reviewer_7z7s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a mask-based approach to obtain subnetworks for specific tasks in continual learning (CL). It is an improvement of the previous method SupSup, which finds fixed supermask for each task to alleviate forgetting. This paper proposes to perform exclusive and nonoverlapping subnetwork training to avoid conflicting updating on the shared parameters. A specific KNN based method is proposed to enhance parameter sharing. The experiments show that the proposed method can produce better performance than other methods. ", "summary_of_the_review": "As discussed above, the work is well motivated and justified, and the paper is well-written. \nThe paper provides enough novel insights and observations and some empirical experimental results and analyses on the proposed method. The significance is influenced by its relationship with SupSup and that it is restricted to the task-incremental setting, where task identifiers are required in both training and testing. \n\n---\nI appreciate the authors' response, which addressed part of my concerns, especially on the KKT module. I am still concerned by the novelty (mainly about the relationship with SupSup) and technical contributions. After further discussions, I am further convinced that the experiment part also needs to be improved, especially in comparison with other parameter isolation-based methods, instead of only SupSup.  \nThe work may be further improved by fixing the issues and conducting some more work, such as some attempts to extend the method to other settings beyond the task-incremental setting.\n", "strength_and_weaknesses": "Strength\n- The work is well motivated. It starts from the limitation of the SupSup method and designs well-motivated and well-justified techniques to improve the supermask based method. \n- The proposed method performs better than the compared methods with a similar setting. \n- The paper conducted carefully designed analyses and discussions on the experiments. \n\nWeakness\n- The whole framework is based on SupSup with similar systematic limitations and benefits, which influences the significance at some level. Specifically, the task identifier is required in both training and testing, restricting the methods on the task-incremental setting. \n- As a result, all the experiments are restricted to the task-incremental setting. Please correct me if I overlooked other settings.\n- Some related works are not discussed, such as (but not limited to) the following ones. There are a series of approaches to generate subnetworks at a module or neuron level to alleviate the forgetting and parameter interference issue in CL. Although the settings may not be the same, they may be discussed. \n\n    - Hurtado, Julio, Alain Raymond, and Alvaro Soto. \"Optimizing reusable knowledge for continual learning via metalearning.\" Advances in Neural Information Processing Systems 34 (2021): 14150-14162.\n\n    - Veniat, Tom, Ludovic Denoyer, and Marc'Aurelio Ranzato. \"Efficient continual learning with modular networks and task-driven priors.\" arXiv preprint arXiv:2012.12631 (2020).\n\n    - Yan, Qingsen, Dong Gong, Yuhang Liu, Anton van den Hengel, and Javen Qinfeng Shi. \"Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 109-118. 2022.\n    - Golkar, Siavash, Michael Kagan, and Kyunghyun Cho. \"Continual learning via neural pruning.\" arXiv preprint arXiv:1903.04476 (2019).\n\n- How is the efficiency of the KKT module? It needs to run training and testing on multiple splits. Can the running time on the vision dataset be reported? How does the setting for the KKT module (like number of splits) influence the running time and performance? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject