Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks
Authors: Zixuan Ke
Bing Liu
Wenhan Xiong
Asli Celikyilmaz
Haoran Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Continual Learning
Heterogeneous Tasks
Sub-network Discovery

Abstract: Continual learning (CL) has two main objectives: preventing catastrophic forgetting (CF) and encouraging knowledge transfer (KT). The existing literature mainly focused on overcoming CF. Some work has also been done on KT when the tasks are similar. To our knowledge, only one method has been proposed to learn a sequence of mixed tasks. However, these techniques still suffer from CF and/or limited KT. This paper proposes a new CL method to achieve both. It overcomes CF by isolating the knowledge of each task via discovering a sub-network for it. A soft-masking mechanism is also proposed to preserve the previous knowledge and to enable the new task to leverage the past knowledge to achieve KT. Experiments using classification, generation, information extraction, and their mixture (i.e., heterogeneous tasks) show that the proposed method consistently outperforms strong baselines.

Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks
Zixuan Ke'; Bing Liu', Wenhan Xiong", Asli Celikyilmaz' and Haoran Li²
*Department of Computer Science, University of Illinois at Chicago
Meta
'[zke4, hitllluuuuuue liu,l@ic.eeecduuu 2 [xwhan, aatpwann,sllice aslic, aimeeli peateeeeel@eeee.eeeeaee Jemeta. com
Abstract
age knowledge transfer (KT) across tasks, i.e., the learning of task should be able to leverage the knowledge learned from previous tasks?; and (3) be able to learn a mixed sequence of similar and dissimilar tasks and achieve both (1) and (2).
Continual learning (CL) has two main objec- tives: preventing calastrophic forgetting and encouraging knowledge transfer (KT). The existing literature mainly focused on overcom- ing CF. Some work has also been done on KT when tasks are similar. To our knowledge, only one method has been proposed to learn a sequence of mixed tasks. However, these techniques still suffer from CF and/or limited KT. This paper proposes new CL method to achieve both. It overcomes CF by isolating the knowledge of each task via discovering a sub- network for it. A soft-masking mechanism is also proposed to preserve the previous knowl- edge and to enable the new task to leverage the past knowledge to achieve KT. Experiments using classification, generation, information ex- traction, and their mixture (i.e., heterogeneous tasks) show that the proposed method consis- tently outperforms strong baselines.!
forgetttnn (CF)
Most existing approaches for TIL address one of the three the detriment of the others. For exam- ple, HAT (Serrà et al., 2018) and SupSup (Worts- man et al., 2020) can overcome CF by isolating the parameters of each task, which makes KT difficult. CUBER (Lin et al., 2022a) and DAS (Ke et al., 2023) can achieve KT by allowing some updates to existing knowledge, which causes CF. Progres- siveNet (Rusu et al., 2016) assures no CF by fixing all the learned parameters and expanding the net- work, but limits KT. To our knowledge, CAT (Ke et al., 2020) is the only system that tries to have all three capabilities. However, it still suffers from CF due to its weak task similarity detection (for KT) and parameter sharing across dissimilar tasks.
1 Introduction
This paper takes major step to achieve all three objectives without suffering from the above drawbacks. The proposed system called TSS (TIL based on Sub-network discovery and Soft- masking). TSS performs two functions, (A) sub- network discovery with ooft-masking and (B) im- portance computation, to learn mixed sequence. where some tasks may be similar (which enable KT) and some may be dissimilar.
One of the overarching goals of AI is develop agents that can continually learn diverse tasks. To- ward this goal, continual learning (CL. has been proposed, which incrementally learns a sequence of tasks, ...... (Chen and Liu, 2018). Once a task t is learned, its training data D: (at least majority of it) is no longer accessible. This paper studies CL of NLP tasks (Ke and Liu, 2022), where task is end-task, e.g., text classification, summarization information extraction, in the popular setting of iassiinnremental learning (TIL), in which the task-id id is given in both training and testing. ? Ideally, CL should (I) overcome catastrophic forgetting (CF), i.e., it should not degrade the per- formance of previously learned tasks; (2) encour-
Given a fixed and randomly initialized back- bone network N, function (A) finds a separate sub-network as the model for each task. A sub- network is indicated by set of binary gates, one for each parameter of N, specifying whether the corresponding parameter in N should be in the sub-network for the task. In training, N is fixed and the binary gates are obtained by learning a set of positive and real-valued popup scores (one for
"The work done mainly during Meta AI internship. 'The code TSS can found at ht tps::// thwb.thcoo.o com/ Zixuanke/Py PeteePCyonnuuua
'Another popular CL setting is elass-incrrmental learn- ing (CII.), which provides no task-id in testing and solves a different type of problems (Ke and Liu, 2022).
"This is called forward transfer in the literature. There is also the backward transfer, where the performance of previous tasks may he improved after learning similar new task.

---

each parameter of N) and then applying a threshold on the popup scores." 4 In other words, we convert the network Iraining into the popup scores train- ing. Only the binary gates (1's) obtained from the trained popup scores are stored for each task as its model. Thus, there is no interference across tasks to cause CF as the backbone network is fixed.
which computes the importance of the popup scores after training a task. The importance is a number between 0 and 1. After training popup scores for a task, we input the training data again to compute the importance of each popup score for the task based on its gradient. We save the accumulated im- portance over all previous tasks to keep track of the important popup scores so far. The accumulated importance (within 0 and 1; thus "soft") is used to guide soft-masking in (A) in training a new task. This paper makes three key contributions.
This sub-network discovery effective for over- coming CF, but it prevents KT if the popup scores of each task are independently trained with no shar- ing across tasks. A naive way to share knowledge is to initialize the popup scores of the new task t with the trained popup scores of the previous task t 1. However, this is problematic for mixed sequence of similar and dissimilar tasks because the training of task can only leverage the knowledge from the (t- t)th task. This can cause negative transfer if task ( - is dissimilar to task L. To address this, we propose a soft- masking mechanism to make the ini- tialization of popup scores for the new task contain the learned knowledge from all previous tasks so that the new task training can leverage the knowl- edge of all previous similar tasks. This is key novelty of our approach. is achieved by reducing (called "soft--askingg) the gradient corresponding to the subset of popup scores that are important to previous tasks (1...t - 1) in training the current task so that the previously learned knowledge is well protected. In this way, the trained popup scores for task f, which will be used as the ini- tialization for the next task, contain not only the knowledge for task t but also the knowledge from all previous tasks. This is eritical for learning a mixed sequence of similar and dissimilar tasks be- cause the system does not know a priori which tasks' knowledge can be transferred to the current task. Note that soft-masking is only applied in backward propagation to preserve old knowledge. In the forward pass, the training can still use all popup scores corresponding to all parameters of the backbone network. Additionally, soft-masking reduces the gradient instead of fully blocking the gradient, which gives the model the flexibility to adjust any popup scores when training the current task I, which further encourages KT.
1. It studies an important but under-studied prob- lem of continually learning a mixed sequence, where some tasks may be similar and some may be dissimilar. Existing CL methods suffer from either CF or limited KT in this scenario. 2. It proposes novel method TSS consisting of sub-network discovery with soft-masking and importance computation to prevent CF and to encourage KT for a mixed sequence of tasks. 3. It evaluates TSS using datasets consisting of classification, generation and information ex- traction tasks and their mixture. The results demonstrate the effectiveness of TSS.
Related Work
Forgetting prevention in continual learning (CL). There are four main families of approaches: (1) Regularization-bass patrick al., 2016; Lee al., 2017: Seff et al., 2017; Zenke et al., 2017: Rusu et al., 2016) add a regularization in the loss to penalize changes to parameters that are important to previous tasks. Gradient projection (Zeng et al., 2019) ensures the gradient updates occur in the orthogonal direction to the input of old tasks and in trust region (Lin et al., 2022a.b) TSS uses no regularization. (2) Replay-based approaches (Rebuffi et al.,
approaches (Kirk-
2017: Lopez-Paz and Ranzato, 2017: Chaudhry et al., 2019; Wang et al., 2020) retain some training data of old tasks and use them in learning a new task. The methods in (Shin et al., 2017; Kamra et al., 2017; Rostami et al., 2019; He and Jaeger, 2018) learn data generators and generate old task data for learning a new task. These are clearly different from TSS as it uses no any replay data. (3) Parameter isolation (Serrà et al., 2018 8; Ke et al., 2020, 2021a; Mallya and Lazebnik, 2018; Fernando et al., 2017; Wortsman et al., 2020) learns and masks a dedicated sub- uub-network for each task, but has limited KT. TSS leverages this approach to
The question is how to decide the important popup scores for each task. This is done by (B),
'The popup score means that the score is used select a parameter or popup the edge from the backbone network (Ra- manujan et al., 2020).
'Therefore, training task f is the same as training the popup scores for task f. We thus use the two terms interchangeubly.

---

isolate knowledge for different tasks, but enables KT by using a soft-mask mechanism to preserve the learned knowledge and uses the old knowledge as the initialization for learning the new task.
2021: Lai al., 2021; Chen al., 2020; Lin al.,
2020; Gao al., 2021; Voita et al., 2019). For Transformer, one can prune the attention head (Michel et al., 2019; Voita et al., 2019; McCar- ley et al., 2019) and sub-layers (Fan et al., 2020; Sajjad et al., 2020). However, these methods are not directly applicable to us as we need to compute the importance of each popup score instead of each parameter in the network. We use the importance as soft-masks leverage all existing neb-networks for KT rather than to compress the LM.
(4) Parameter isolation plus out-of- -fittiibbiiiio (OOD) detection is new and theoretical grounded approach (Kim et al., 2022, 2023). It mainly used for class-incremental learning (CIL). The main idea is to use parameter isolation approach for task incremental learning (TIL) to overcome CF and the model for each task is an OOD detection model. During inference, the system performs both task- id prediction and within-task prediction for CIL classification. However, our work is about TIL.
3 Proposed TSS Technique
TSS designed for both CF prevention and for- ward KT. Figure gives an overview of TSS. The backhone network consists of the transformer and adapters, which are indicated by the transformer parameters wrransformrr of layer 1 and by the pa- rameters of adapter w walaapter of layer respectively (see the grey boxes). Note again that these are fixed in the entire learning process. On the left, we show the forward pass and backward propagation of (A) sub-network discovery with soft-masking (Sec. 3.1). In the forward pass, set of learnable popup scores 8 are initialized (Sec. 3.1.2) and fed into a step function. The output of the step function (t) that element-wise mul- tiply (8) with the parameters of the adapter of the backbone network (w0 ddppatt As a result, the binary gates indicate a sub-network for task t within the un adapter (Sec. 3.1.1). In the backward propagation (Sec. 3.1.2), we first accumulate the importance of popup scores for all previous tasks by normaliza- tion and element-wise maximum ("EMax"). The accumulated importance r( f(st-1) is converted into soft-mask by the operation mask is then element- wise multiplied with the orig- inal gradient of the popup scores s(" in layer 1, V, (computed by ttraight-through estimator in Eq. 3). The soft-masked gradient, Vi. is the final gradient that is used to optimize the popup scores 8 the important popup scores have lower gradients in V,, the popup scores that are important to previous tasks are preserved and are used to initialize the new task to encourage KT (not shown in Figure 1).
Knowledge transfer in CL. There are two main approaches for KT: (1) replay-based (Huang et al., 2021; Wang et al.. 2021, 2020; de Mas- son d' Autume et al., 2019; Zhu et al., 2022; Yin et al., 2022), which use replay data to help the transfer. TSS is replay-free. (2) mmmilarity-baserr which uses task similarities using features (Ke et al., 2021a,b, 2020; Wang et al., 2022: Zhang et al., 2022) or gradients (Lin al.. 2022a). However, these methods are not always reliable, which results in CF. CAT (Ke et al., 2020) considers mixed se- quence of tasks by first detecting previous tasks that are similar to the current task and then opening the masks of these tasks so that the new task learning can modify their parameters to achieve KT. How- ever, this causes CF for dissimilar tasks that share parameters with those similar tasks. DAS (Ke et al., 2023) encourages KT by allowing previous knowl- edge updates based on importance, but this also causes CF because previous parameters change- able and there is no easy way to separate the pa- rameters that are used for different tasks. In con- trast, TSS does not require any explicit similarity detection but finds sub-network for each task to guarantee no CF. Its soft-masks allow the initial- ization contains all previously learned knowledge and thus encourage KT to the new task. Konishi et al. (2023) recently proposed a parameter-level soft-masking method using AlexNet as the back- bone. The method has difficulties working with more complex architectures. Our soft-maske are set on the popup scores rather than network param- eters and our approach is based on sub-network discovery and knowledge sharing.
is a set of binary gates g
r((--1) This soft-
(t) Since
After (A), we show (B) importance computa- tion (Sec. 3.2). The forward pass is the same as (A). However, in backward propagation, the gra- dient of popup scores, Vt, is not used to update anything (red cross in the arrow) but computes
Network pruning as importance computation. It is known that many parameters in a neural net- work are redundant and can be pruned (Li et al.,

---

(A)
(B)
Sub-network Discovery with Soft-masking
Importance Computation
LcE
Lee
(s
adapter
adapter
wrannffrrmer
Transformer
Forward Propagation
Forward Propagation
L
()--))
Normalize
(tt-1
V
W Adapter
1/400
Adapter
(1)
Pansformer
transtormer
(2--))
Backward Propagation
Backward Propagation
Figure 1: Illustration of TSS in training task t. The detailed description is in Sec. 3. The grey boxes indicate that the parameters of the adapters and backbone language model (LM) are all frozen. The only trainable parameters are the popup scores (t) In (A) sub-network discovery with soft-masking (Sec. 1), we remove the step function in the backward propagation to reflect the straight-through estimator Eq. 3, where the gradient "skips" the step function as if is an identity function. The original gradient for popup scores, Vi, is not directly used to update the popup scores but soft- masked based on the accumulated importance (Sec. 3.1.2). In (B) Importance Computation (Sec. 3.2), the red cross in backward propagation indicates that the gradient is not used update the popup scores (+) 8) but only to compute the importance.
the importance using its absolute average value, E [Vtl The resulting importance of task t in layer I, I(0) is saved for (A) in learning the next task. The following sub-sections present each function and step in detail.
of for layer for task has the same size as the parameters of the adapters w adapter (which is fixed throughout learning). In the forward pass, si is constantly thresholded,
g - 1.
(1)
3.1 Sub-network Discovery and Soft-maskin
where II is a step function that outputs 1 for the scores that are larger than the threshold (a hyper- parameter). Since the gate g, is always binary, it can naturally indicate a sub-network within the pa- rameters of the fixed adapter, w, pddapter by element- wise multiplication,
Sub-network discovery finds a sub-network (repre- sented as binary gates) for the current task, which is saved to overcome CF. It relies on the training of popup scores and its training also involves soft- masking to encourage knowledge transfer. In this section, will present how discover the sub- network via learning popup scores and how we en- courage knowledge transfer (KT) via voft-masking.
adapter
adapter i
wi
ndupter
(2)
3.1.1 From Popup Scores to Sub-network
where is element-wise multiplication. w indicates the selected parameters in the adapter that form the discovered sub-network.
addapter
Since the pre-trained language model (LM) con- tains useful general knowledge, we fix and use the full LM. The sub-network discovery is only done on adapters (Houlsby et al., 2019), which are in- serted into each layer of the LM. 6 Note again that throughout the training process, both the adapters (randomly initialized) and backbone LM are fixed. We now define a of learnable popup scores
Training the popup scores. 5/6 cannot be directly optimized because the derivative of its threshold-based step functions is zero. Then noth- ing can be learned. To solve the problem straight-through estimators (Bengio al., 2013). Specifically, we ignore the derivative of the thresh- old function and pass on the incoming gradient as if the threshold function was an identity function.
"See Appendix A for additional details about adapters.

---

where 1 2) refers to the previously accumulated importance at task t 2. We do not need to save Consequently, for each parameter o/ in 8/ it updated as follows,
(1) 8) (1) aV;
all {I, ) for Eq. 4, but only the incrementally ]k=1 accumulated importance after training each task. Soft-masking the popup scores. Given the ac- cumulated importance 1 ((tt1)) of layer and the cross-entropy loss LCE, we reduce (or soft-mask) (t's "s gradient (V, in Eq. 3) flow as follows,
V Oc
Oc 0I,
ac
w1, Or.
JI, 0
(1)
0I,
where I and O, are the input and output of neurons and j. Vi is the gradients on the popup scores sf") and a the learning rate. After training, the binary gate, Di is saved (taking only 1-bit per element).
V = (1 1 11 VI,
(5)
Since gi is saved, and all the backbone parameters are fixed (including the adapter's and backbone LM's), TSS does not suffer from forgetting of the previously learned knowledge.
The importance I (((t-1) has the same size as V, and the associated popup scores. This is soft- masking as each element in I r(5t-1) is real num- ber in [o, 1] (not binary (0, 1)), which gives the model the flexibility to adjust any popup scores. We note that the above soft-masks are only applied in the backward pass, but not in the forward pass, which encourages knowledge transfer because each task training can leverage all popup scores that are corresponding to all parameters of the backhone network and the popup scores contain the knowl- edge from all previously learned tasks.
3.1.2 Preserving Previous Knowledge for New Task Initialization
While the sub-network discovery in Sec. 3.1.1 can help prevent forgetting (CF), it does not do knowl- edge transfer (KT). As discussed earlier, the naive way for KT to initialize the popup scores s) for the new task & with the learned scores 8i for task & 1. However, s/ 1) contains only the knowledge from task 1, but we have no idea whether any knowledge from task - can be transferred to task because tasks t and t - may be entirely different. To address this, we preserve the learned knowledge from all previous tasks from in s/ (t-1), In this way, task t can leverage all previously learned knowledge for KT.
(t-1)
Popup scores initialization. Thanks the soft- (1-1) contains masking mechanism, the trained 81 knowledge from all previous tasks. We use it to initialize the popup scores for the new task 8 to encourage knowledge transfer. Note that we apply Kaiming Initialization for 8)
3.2 Computing the Importance of Popup Scores the Current Task
To make the preservation possible, inspired by (Ke et al., 2023), we propose soft-masking mech- anism based on the importance of each popup score to all previous tasks. We compute the importance in Sec. 3.2. For now, assume that we already (k) J==1 for s)
To apply soft-masking (Sec. 3.1.2), we need to de- termine the importance of popup scores for the previous tasks. This is done after training task. We adapt the gradient-based importance detection method in (Michel et al., 2019) for our purpose. Given dataset D {(x;, )= of n samples (3/ is the class label of æx)) and the trained popup scores 8) we input the data again and the impor- tance of pupup scores in the layer I is estimated with a gradient- based proxy score
have the set of importance value (())t_1 for each previously learned task k. The preserva- tion is achieved by soft- soft-masking the learning based on the accumulated importance as follows:?
Accumulating importance. We accumulate the importance after task ( 1 is learned via element- wise max (FMax),
N dLer(Eas Um))
(1--)) EMax ((I(1-)) (1-1) (51-2)
(4)
1
1
(6)
2 (0)
"Before accumulation, we normalized the the importance values in each layer / for task & by making the importance values for all popup scores in the layer have the mean of O and standard deviation of To further facilitate soft- masking, the normalized importance values rounded by Tanh activation and the absolute operation that the values are in the interval of [0.1]. To simplify the notation, we still use (()) to represent the resulting importance.
Note the average gradient computed in Eq. over all the data is only used to compute the importance and will not be used to optimize the popup scores.
The resulting ((()) is of the same size as s) each entry corresponding to the importance of a popup

---

score. It is used in the next task by accumulating with the previously accumulated importance (Eq. 4) and soft-masking the learning (Bq. 5). Note that 1, is all 0 as we do not know which popup scores are important before the training of the first task.
(ASC and NER) are datasets consisting of similar tasks as the tasks share similar task labels (with some exceptions, see the statistic in Appendix B) but from different domains. Our goal is to achieve both CF prevention and KT. Three of them (SUM, CCD, and DRG) are dissimilar tasks as the distri- bution shift across tasks is relatively large. They have little shared knowledge to transfer and the main goal is to ensure there is little or no CF. Heterogeneous tasks sequence. This sequence is constructed by mixing all the above similar and dissimilar tasks of different types from all the 5 datasets in random order. It has a total of 40 tasks. This is a challenging and more realistic setting where the system needs to prevent CF and encour- age KT dynamically.
4 Experiments
We now evaluate the proposed system TSS. We first learn all tasks sequentially. After that, their task models are tested using their respective test sets. TSS does not use any replay data.
4.1 Datasets and Baselines
Datasets: We use five datasets covering a wide range of NLP problems, including classification, generation, and information extraction. For de- tailed datasets statistics, please see Appendix B. (1) ASC (Aspect Sentiment Classification) is from (Ke et al., 2021a) and has 19 tasks. Each task clas- sifies the opinion (positive, negative, or neutral) in review sentence at the aspect-level of a prod- uct. (2) CCD (Continual Classification Dataset) is popular continual text classification dataset (de Masson d' Autume al., 2019). (3) SUM (ConvoSum) is a conversational abstractive summa- rization dataset with 6 tasks/domains (Fabbri et al.. 2021). Given conversations from a domain, the system generates its summary. (4) DRG (Dialogue Response Generation) is popular task-oriented di- alogue response dataset Wo7ti) (Ramadan et al., 2018) with 5 tasks/domains. Given the intent and dialogue state (slot-value pairs containing mes- sages to express), the system generates a response. (5) NER (Named Entity Recognition)) classifies mentions into pre-defined entity types in each task.
Baselines. We use 14 baselines with both non- continual and continual learning (CL) methods.
MTL and MTL (Adapter) train tasks in a multi-task or data combined setting, where the former trains the whole LM and the latter trains only the adapter. These two are widely accepted as the upper bounds of CL. ONE builds a separate model for each task by fine-tuning the LM, which has no KT CF. ONE (Adapter) (Madotto et al., 2020) trains an adapter for each task separately (called AdapterCI. in its original paper). ONE (Prompt) (Zhu et al., 2022) trains a prompt for each task (called C-PT in its original paper).
Non-CL baselines:
CL baselines. The CL baselines include an naive continual learning (NCL) method where the sys- tem learns the tasks one by one with mechanism to deal with CF or to encourage KT, and 9 state-of- the-art TIL methods: 5 adapter-based methods: CTR (Ke et al., 2021a), HAT (Serrà et al., 2018), SupSup (Wortsman et al., 2020), CAT (Ke et al., 2020) and CUBER (Lin et al., 2022a). 1 prompt- based method: L.P (Wang et al., 2022). 3 base- lines that modify the Transformer: LAMOL (Sun et al., 2020), EWC (Kirkpatrick et al., 2016) and DAS (Ke et al., 2023). Readers can refer to Ap- pendix D for the details of these baselines.
Since TSS aims at (1) preventing forgetting, (2) encouraging knowledge transfer and (3) learning a mixed sequence of similar and dissimilar tasks, we consider two types of sequences.
Homogeneous tasks sequences. In each such task sequence, all tasks are from the same dataset. Among the aforementioned 5 datasets, two of them
"It contains 5 tasks: AGNews (news classification), Yelp (sentiment analysis), Amazon (sentiment analysis), DBpedia (Wikipedia article classification) and Yahoo (questions and answers categorization). Since each of these datascts quite large, we randomly sampled 500 samples from each class for each task due our resource limitations.
LM and hyperparameters. Since we need a backbone LM that can do classification, generation.
Fooor classification datasets (ASC, CCD and NER), we con- duct multi task learning (MTL) experiment. For generation datasets (SUM and DRG). MTL is not possible as the lan- guage modeling head on top of BART is a linear layer with weights tied to the input embeddings. We follow the standard practice (c.g.. (Qin and Joty. 2022; Madotto et al., 2020)) and pool all data together to train a single shared head (we still called this MTL for simplicity).
"This data consists of 5 tasks, including conl103 (Sang and Meulder, 2003), wikigold (Balasuriya a et al., 2009). bte (Derczynski et al., 2016), re3d (Laboratory, 2017), and gum (7eldes, 2017). Due to resource limitations, we randomly sampled 200 samples for each task.

---

Similar
Dissimilar
Similar ASC
Dissimilar CCD
Average
Average
Average
Averuge
Dataset Model
ASC
NER
SUM
CCD
DRG BLEU
NER
SUM
DRG
MFI
F1
R1
MF1
Main
FR
MFI
F1
R1
MF1
BLEU
Main
FR
Non-continual learning
MTL
92.28
63.33
39.39
90.57
25.29
62.17
92.28
63.33
39.39
90.57
25.29
62.17
MIL (Adapter)
92.17
60.61
38.84
91.09
24.50
61.44
92.28
63.33
39.39
90.57
25.29
62.17
ONE
85,55
59.33
39.07
91.07
24.14
59.83
85,55
59.33
39.07
91.07
24.14
59.83
ONE (Adupter)
83,95
56.69
38.90
90.78
23.42
58.75
83,95
56.69
38,90
90.78
23.42
58.75
ONE (Prompt)
76,46
45,90
30.67
86.23
12.67
50.39
76.46
45,90
30.67
86,23
12.67
50.39
Continual learning of beterugenenux tusks
Continual Tearning of homeggeeeuu tasks
NCL
88.05
34,56
22.86
81.86
8.68
47,20
13.94
89.25
49.19
32.68
85.08
22.31
55.70
6.42
EWC
88.73
28.75
18.65
83.24
7.99
45.47
13.43
88.36
51.76
32.64
87.27
18.30
55.66
5.53
HAT
87.45
50.16
35.21
89.85
20.98
56.73
0.97
89.33
52.31
37.11
90.21
21.47
58.08
0.49
DAS
90.62
43.38
25.45
82.66
15.21
51.47
10.49
90,94
42.12
31.31
87.97
22.25
54,92
9.26
SupSup
85.83
58.93
38.23
90.66
24.71
59.67
0.00
85,83
58.93
38.23
90.66
24.71
59.67
0.00
LAMOL
84.62
10.88
54.44
19.96
CAT
45.79
35,24
17.44
36.62
10:21
29,06
30.27
84.31
50.73
37.24
90.82
21.72
56.96
1.37
CTR
89.37
50.16
33.98
89.96
17.63
56.22
0.95
88.86
51.85
37.34
90.54
21.39
58.00
0.65
CUBER
90.58
46.67
30.94
81.54
15:68
53.08
7.27
91.25
47.19
30.44
89.19 19
21.66
55.95
5.82
L2P
78.10
39.03
28. 34
72.35
4.55
44.47
6.44
74.81
44.22
26. 65
85,35
8.52
47.91
2.58
TSS
90.61
62.13
38,29
90.70
24.56
61.26
0.00
91.28
63.96
38.39
90.89
24.75
61,85
0.00
Table 1: Performance for heterogeneous tasks (a sequence of 40 tasks from all 5 datasets) and homogeneous tasks (each sequence consisting of all tasks from one dataset), averaged over 5 random sequences (the standard deviations are given in Appendix H due to space limits). " means not applicable. We bold the best results within CL baselines. The smaller forgetting rate (FR) means the system deal with forgetting better. Note, the results for non-continual learning are the same in both settings. Execution time and memory need are given in Appendix J.
and information extraction. We adopt BART LARGE (Lewis et al., 2020) as our LM. Fine-tuning of BART follows the standard practice." 1I Detailed hyperparameters are given in Appendix C.
LAMOL is not included in heterogeneous tasks as is not obvious how to adapt LAMOL for NER.
Heterogeneous Tasks. The left section in Table shows that TSS outperforms all CL baselines for the mixed sequence of 40 heterogeneous tasks, with similar tasks (from ASC, NER) and dissimilar tasks (from SUM, CCD and DRG). Note, although we have only one task sequence, we report the results separately for each dataset. Other observations are: (1). TSS is more effective than CL base- lines that only deal with CF (EWC, HAT, Sup- Sup). In the two similar datasets (ASC and NER), TSS clearly wins because regularization-bast EWC sacrifices accuracy for overcoming CF and parameter-isolation based SupSup prevents any possible KT. In the 3 dissimilar datasets which have little shared knowledge to transfer, TSS is similar to the baseline that can prevent CF (like SupSup). This confirms TSS can achieve both CF prevention and KT the challenging heterogeneous setting. 14 (2). TSS is more effective than CL baselines that deal with both CF and KT (CAT, CTR, DAS, CUBER and L.2P). Among these systems, CAT performs the worst due to its inaccurate task sim- ilarity detection and its difficulty to deal with CF. L2P performs better, but still has a large gap to TSS due to the poor prompt selection (we can see it is
4.2 Evaluation Results and Analysis
Since the order of the tasks in a sequence may impact the final result, we ran 5 randomly sampled task sequences (results of individual sequences are given in Appendix I). For different types of tasks, we use their standard evaluation metries." 12 Ta- ble 1 gives the average result of each system over 5 random task sequences after continual learning of heterogeneous task sequences (left section) and homogeneous task sequences (right section). We report the main performance of each dataset, i.e., MF1 (Macro-F1) in ASC and CCD, Fl in NER, RI in SUM and BLEU in DRG and their average in the Average Main column (the second from the right in each section). We also report the average forget- ting rate (FR) on the same metrics in the Average FR column to evaluate the forgetting prevention." 13
"I For ASC, we adopt the ASC formulation in (Xu et al., 2019). where the aspect term and sentence concatenated via 
. Opinion predicted as the average over all tokens.
We use Macro-Fl and accuracy for the sequence- classification lasks (ASC and CCD), where Macro-F1 (MF1) is primary metric because highly imbalanced classes biases in accuracy. We use Rouge score (RI, R2 and RL) for SUM. BLEU score for DRG and for NER. "For the detailed metrics and results for each system, please see Appendix F. For the detailed forgetting rate for each dataset, pleuse see Appendix G.
ASC
"Other baselines perform poorly: HAT has little KT in clussification tasks, which makes ASC poorer. It has forgetting in generation tasks as it cannot isolate parameters in the shared LM head.

---

Similar
Dissimilar
Similar NER
Dissinilar
Dataset
ASC
NER
SUM
CCD MFI
DRG BLEU
Average
Average
ASC
SUM
CCD MFI
DRG
Average
Average
Model
MF1
Main
F1
Non-continual learning
ONE
85.55
39.07
00
24.14
83
85.55
59.35
39:07
91.07
Continual learning of heterogencous tasks
Continual learning of homogeneous tasks
ISS (wlo SDJ
86,54
37.03
28.04
78.
14.85
48.95
9.02
91.06
40.47
30.42
88. 34
22.77
54.61
9.61
1SS (wlo SM)
85.83
58.93
90.66
24.71
59.67
0.00
85.83
58.93
90.66
24.71
59.67
TSS (wlo SM: Naive)
88,89
34,79
36
89.99
24.19
54.80
0.00
90.38
62.42
58,08
90.63
24.59
61.22
0.00
TSS
90.61
62.13
38.29
90.70
24.56
61.26
0.00
91.28
63.96
38.39
90.89
24.75
61,85
0.00
Table 2: Ablation experiment results for heterogemeous and homogeneous tasks averages over 5 random sequences (the standard deviations are reported in Appendix H due to space limits).
even poorer than ONE (prompt), indicating that its selection causes CF). DAS performs well on ASC, but poorly on other datasets, indicating it cannot effectively prevent CF. CUBER has strong perfor- mance similar tasks (e.g., ASC) but performs poorly on dissimilar tasks. CTR is the best among the three, but it is still worse than TSS due to its inaccurate instance-level similarity detection.
mechanism are effective. For (I), we conduct the ablation experiment TSS (w/o SD), where we re- move the sub-network discovery and directly train the parameters of adapters with soft-masking. For (2), we conduct the experiment TSS (w/o SM) and TSS (w/o SM; Naive). They both do not use the soft-masking mechanism. The former initializes the popup scores with Kaiming Initialization for all tasks while the latter initializes the scores with only those of the last task.
Knowledge transfer (KT) and CF prevention. To validate TSS's effectiveness in dealing with CF with a sequence of dissimilar tasks, we can first compare TSS with ONE. We can see TSS achieves similar results to ONE in the three dissimilar tasks datasets, indicating effective CF prevention. Ad- ditionally, can evaluate the continual learning process to see whether forgetting occurs during the training of each system. To this end, we com- pute Forgetting R in Table I (the right-most column in the CL of heterogeneous tasks section). Clearly, TSS has 0 forgetting. SupSup also has 0 forgetting because it also tries to find a uub-network for each task. While this is certainly good for CF prevention but makes KT impossible. We can other baselines all suffer from forgetting (positive forgetting rate) on average.
The right section (heterogeneous) in Table 2 shows the ablation results and the corresponding forgetting rates. The full TSS gives the best average result, indicating that every component helps. We further observe: (1) TSS's s gain is partially from the sub-network discovery as TSS (w/o SD) is poorer on average, particularly for those datasets having little shared knowledge; (2) soft-masking helps as TSS (w/o SM) gives a worse performance; (3) soft- masking can help make the initialization for the new task better as TSS (w/o SM; Naive) is clearly worse than TSS. Note that both TSS (wlo SM) and TSS (w/o SM; Naive) have 0 forgetting rate, due to the effectiveness of sub-network discovery.
Homogeneous Tasks. For continual learning of homogeneous tasks, conducted 5 experiments, one for each for the 5 datasets, i.e., each task se- quence contains only the same type of tasks (see Sec. 4... The right section of Table shows the av- erage results for each system. The observations are similar to those for heterogeneous tasks, i.e., TSS outperforms all CL baselines. We further observe that TSS has larger improvements on similar tasks (ASC and NER), comparing to the improvements in the mixed sequence. This is expected.
Regarding KT, we can again use ONE as the con- trol and see whether there is effective KT (learned tasks help the new task). Clearly, we can see TSS outperforms ONE in two datasets (ASC and NER) with similar tasks, indicating effective KT. Thanks to the sub-network. discovery and soft-masking, also see TSS is very similar to MTL/Comb, which again shows the effectiveness of TSS.
A blation study. We want know whether (1) the sub-network discovery and (2) the soft-masking
The forgetting rate (Liu et al., 2020) is defined as FR E Σi=1 A,,, A,,, where A,, the test performance each task when it was first learned and A..; is the performance of task after training the last task t. We average over all tasks except the last one as the last task obviously has no forgetting. The detailed forgetting rate for each datasct is given in Ap- pendix G. (Mehta et al., 2021) defined a different forgetring rate. Appendix F will argue that ours is more effective.
The right section of Table 2 shows the ablation study. The observations are also similar those for heterogeneous tasks, i.e., every component helps. We also observe that the naive transfer (TSS (w/o SM; Naive)) works similarly to TSS. This indicates that the proposed initialization is more important

---

in the heterogeneous tasks setting because two ad- jacent tasks can be very dissimilar (c.g., the current task is NER but its previous/last task may be sum- marization) and causes negative transfer.
References
Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009. Named entity recognition in wikipedia. In Proceedings of the 1st 2009 Workshop on The People's 's Web Meets NLP: Collaboratively Constructed Semantic Resources @ IJCNLP 2009, Suntec, Singapore, Au- gust 7, 2009, pages 10-18. Association for Computa- tional Linguisties.
In summary, we can say that TSS works well for both the homogeneous tasks and the challenging heterogeneous tasks scenarios.
5 Conclusion
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computa- tion. CoRR, abs/1308.33 338..343..
This paper studied task incremental learning (TIL) on a range of NLP problems. We first presented the three desired capabilities: no forgetting, knowledge transfer and learning a mixed sequence of similar and dissimilar tasks. To our knowledge, only one system (CAT) aims to achieve all these objectives but it suffers from forgetting. We then propose a novel method, TSS, to achieve all three. Experi- mental results showed that TSS achieves KT and no forgetting even for the challenging mixed (or heterogeneous) task sequence setting.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019, Efficient lifelong learning with A-GEM. In ICLR.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang. and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural informa- tion processing systems, 33: 1834-15846.
Zhiyuan Chen and Bing Liu. 2018. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12
6 Limitation
While effective, TSS has some limitations. First, al- though TSS shows strong performance in forward transfer (old knowledge helps the new task), how to enable backward transfer (new knowledge im- proves the trained tasks) is left to future work. Sec- ond, while empirically soft-masking the gradient does not harm task learning, there no theoretical guarantee that the task with soft-mast soft-masking can learn as well as without soft- foft-masking. is interesting to investigate whether this is true for any scenario.
Cyprien de Masson d' Autume, Sehastian Ruder, Ling- peng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: An- nual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada.
Leon Derczynski, Kalina Bontcheva, and Tan Roberts. 2016. Broad twitter corpus: A diverse named entity recognition resource. In COLING 2016, 26th Inter- national Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pages 1169- 1179. ACL.
7 Ethics Statement
This paper proposes novel busk-incremental learn- ing model that can achieve both forgetting preven- tion and knowledge transfer for mixed sequence of heterogencous tasks. We do not foresee any neg- ative consequences for any individual as a result of this research. The consequence of the failure of the system is that the system makes some incor- rect predictions, which, we believe, do not have any ethic implications. All the data and pre-trained models are publicly available and our method does not leverage biases in the data.
Alexander R. Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mchdad, and Dragomir R. Radev. 2021. Convosumm: Conversation summa- rization benchmark and improved abstractive sum- marization with argument mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing, ACL/ICCNPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 6866-6880. Associa- tion for Computational Linguistics.
Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with struc- tured dropout. In International Conference on Learn- ing Representations.
A Acknowledgements
The work of Bing Liu was supported in part by four National Science Foundation (NSF) grants (1910424, 1838770, 2225427, and 2229876) and a research contract from KDDI.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. 2017, Pathnet: Evolution

---

channels gradient descent in super neural networks, CoRR.
Tatsuya Konishi. Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu, 2023. Parameter- level soft-masking for continual learning. Proceed- hawpeeefeeehleeeerraaaaooooorooo Learning (ICML-2023).
Yang Gao, Nicolo Colombo, and Wei Wang. 2021. Adapting by pruning: A case study on bert. arXiv preprint urXiv:2105.. 105.03343.
Defence Science Technology Laboratory. 2017. Rela- tionship and entity extraction evaluation dataset.
Xu He and Herbert Jaeger. 2018. Overcoming catas- trophic interference using conceptor-aided backprop- agation. In ICLR.
Cheng-I Jeff Lai, Yang Zhang. Alexander H Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang. Kaizhi Qian, Sameer Khurana, David Cox, and Jim Glass, 2021. Parp: Prune, adjust and re-prune for self- supervised speech recognition. NewrIPS, 34.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, 2019, Parameter-efficiee transfer learning for NLP. In ICML.
Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. 2017. Overcom- ing catastrophic forgetting by incremental moment matching. In NIPS.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text clas- sification with information disentanglement based regularization. arXiv preprint 1rXiiv22004.86 05489.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-ro-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguisties.
Nitin Kamra, Umang Gupta, and Yan Liu. 2017. Deep generative dual memory network for continual lear- ing. CoRR.
Zixuan Ke and Bing Liu. 2022. Continual learning of natural language processing tasks: A survey, arXiv prepriat 1rX1122211.1201.
Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2021. Differentiable subset pruning of transformer heads. Transactions of the Association for Computational Linguistics, 9:1442-1453 1459.
Zixuan Ke, Bing Liu, and Xingchang Huang. 2020. Continual learning of a mixed sequence of similar and dissimilar tasks. In NeurIPS.
Zixuan Ke, Bing Liu, Nianzu Ma. Hu Xu, and Lei Shu. 2021a. Achieving forgetting prevention and knowl- edge transfer in continual learning. Advances in Neu- ral Informatoon Processing Systems, 34.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. 2022a. Beyond not- forgettiig:: Continual learning with backward knowledge transfer. NeurIPS.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. 2022b. Trgp: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.000 022931.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre- training of language models. In The Eleventh Inter- national Conference on Learning Representations.
Zi Lin, Jeremiah Zhe Liu, Zi Yang, Nan Hua, and Dan Roth. 2020. Pruning redundant mappings in trans- former models via speetral-normalle identity prior. arXiv preprint arXiv:0010.01991.
Zixuan Ke, Hu Xu, and Bing Liu. 2021b. Adapting bert for continual learning of sequence of aspect sentiment classification tasks. In NAACI.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun, 2020. Mnemonics training: Multi-class incremental learning without forgetting. In CVPR.
Gyuhak Kim. Changman Xiao, Tatsuya Konishi, Zix- Ke, and Bing Liu. 2022. A theoretical study on solving continual learning. Advances in Neural Information Processing Systems.
David Lopez-Paz and Mare' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In NIPS.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, and Bing Liu. 2023. Learnability and algorithm for con- tinual learning, arXiv preprint 230.. 12646.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Se- ungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. 2020. Continual learning in task-oriented dialogue systems. arXiv preprini XXXx:2012211504
James Kirkpatrick, Razvan Pascamu, Neil C. Rabi- nowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan. Tiago Ramalho. Ag- nieszka Traaska-BBarwinske Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. CoRR.
Arun Mallya and Svetlana Lazebnik. 2018 Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR.

---

JS McCarley, Rishav Chakravarti, and Avirup Sil. 2019. Structured pruning of a bert-based question answer- ing model. arXiv preprint arXiv: 1910.06360.
Joan Serrà, Didac Suris, Marius Miron, and Alexan- dros Karatzoglou. 2018. Overcoming catastrophic forgetting with hard attention to the task. In ICMI.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021. An empirical investigation of the role of pre-training in lifelong learning. CoRR, abs/21 12.09133.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In NIPS.
Kang Sun, Cheng-Har Banng-Hoooue Ho, and Hung- Yi Lee. 2020. Lamol: Language modeling is all you need for life- long language learning. In ICLR.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? Advances in neural information processing systems, 32.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi- head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418
Chengwei Qin and Shafiq R. Joty. 2022. LFPT5: unified framework for lifelong few-shot language learning based on prompt tuning of T5. In The Tenth International Conference on Learning Representa- fions, ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net.
Chengyu Wang, Haojie Pan, Yuan Liu, Kehan Chen, Minghui Qiu, Wei Zhou, Jun Huang, Haiqing Chen, Wei Lin, and Deng Cai. 2021. Mell: Large-scale extensible user intent classification for dialogue sys- tems with meta lifelong learning. In KDD '21: The 27th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, Virtual Event, Singapore, August 14- 14-18, 2021.
Osman Ramadan, Pawel Budzianowski, and Milica Ga- sic 2018. Large-scale multi multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 2. pages 432 437.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kem- bhavi, Ali Farhadi, and Mohammad Rastegari. 2020. What's hidden in randomly weighted neural net- work? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seat- tle, WA, USA, June 13-19, 2020, pages 11890-11899, Computer Vision Foundation/ / IEEE.
Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022. Learning to prompt for continual learning. In CVPR.
Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. 2020. Efficient meta lifelong- learning with limited memory. In EMNLP.
Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: In- cremental classifier and representation learning. In CVPR.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. 2020. Supermasks in su- perposition. In NeurlPS.
Mohammad Rostami, Soheil Kolouri, and Praveen K. Pilly. 2019. Complementary learning for overcoming catastrophic forgetting using experience replay. In IJCAI.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehen- sion and aspect-based sentiment analysis. In NAACI- HLT.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Des- jardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. CoRR.
Wenpeng Yin, Jia Li, and Caiming Xiong. 2022. Con- tintin: Continual learning from task instructions. In Proceedings of the 60th Annual Meeting of the As- for sociation for Computational Linguistics (Volume 1: Long Papers), ACL2022 Dublin, Ireland, May 22-27, 2022.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020. On the effect of dropping layers of praited transformer models. arXiv preprint arXiv:2004.00 00344444.4..4.....
Amir Zeldes. 2017. The GUM corpus: creating mul- tilayer resources in the classroom. Lang. Resour. Evaluation, 51(3):581-612.
Brik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31- June 1, 2003, pages 142-147. ACL.
Guanxiong 7eng, Yang Chen, Bo Cui, and Shan Yu. 2019. Continuous learning of context-dependent pro- cessing in neural networks. Nature Machine Intelli- gence.
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu, 2017. Continual learning in generative adversarial nets. CoRR, abs/ 700.08995.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelli- gence. In ICMI.

---

sentence Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2022.
A Additional Details about Adapters
Continual sequence generation with adaptive compo- sitional modules. arXiv preprint arXiv:2203. 70652.
TSS leverages adapters to do masking. An adapter is simply 2-layer fully connected network with layer normalization and residual connections in- serted into each Transformer layer, which adapts the output distribution the pre-trained Trans- former language model (LM) without modifying LM's original weights (the original LM is fixed), Figure 2 illustrates the LM with adapters.
Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022. Continual prompt tuning for dialog state tracking. arXiv preprini arXiv:2203.0662
B Additional Details about the Datasets
Recall that TSS uses five datasets. Here we give their detailed statistics.
(1) ASC. ASC is more than the traditional senti- ment classification because of the additional input of the aspect and the fact that in the different aspects can have different opinions. For example, "The picture is good but the sound poor" about TV expresses a positive opinion about the aspect "picture" and negative opinion about the aspect "sound". We show more details about these datasets in Table 3.
(2) CCD, SUM, DRG and NER. We give their detailed statistics in Table 4.
C Hyperparameters
Unless otherwise stated, the same hyper- parameters used in all experiments. The maximum input length is set to 128 for all datasets except for SUM which to its longer sequences. AdamW optimizer used. The learning rate set to 5c-5 for Transformer (search within 3,e5 prompt (search within
1024 due
3e-2 for adapter and classifier. The prompt length is set to 20 (search within (10.20.50.8.100 1000,0)0)),)) and adapter bottleneck size is 64, following the original paper (Houlsby al., 2019). The batch size is 32 and the number of training epochs is set to 50 with early stopping. т in Eq. is set O. For classification tasks, a separate classification head is used for each task in the sequence. For generation tasks, shared LM head is used for all tasks in sequence. we further set the number of beams to 4 for beam search and constrain the target length in between 30 to 200. For image-based (EWC, Cuber, DAS, HAT. SupSup, CAT, L2P) and RoBERTa-based (CTR) systems, we adapt them for text classification and generation by replacing their feature extractors with BART. For

---

Output
Transformer Layer
Foed-forward
Layer Norm
Layer Norm
Feed-forward
layer
layer
Adapter
Multi-headed Attention
Adapter
Feed forward
Feed-forward
layers
layer
Transformer with adapter
Adapter
Figure 2: Architecture Transformer with adapters. An adapter (blue component) inserted in each layer. Only the blue and green boxes trainable while the LM is fixed during training.
Tasks/Domains
#Training
#Validating
#Testing
Speaker
233 S.352 A.2877 P/65 NJO Ne.
30 S./4 A./3 P.99 N.0 No.
38 S,44 A.40 P/4 NJO Ne.
Router
200 S./45 A/142 P/103 N. /0 Ne.
24 S.31 A./1 P/12 NJO Ne.
22 2./11 A,24 P/7 NJO Ne.
Computer
187 S.283 A./118 P/65 N.O Ne.
25 S.35 P/12
Ne.
29 S.33 A.229 P/7
Ne.
Nokia6610
A./19 P/73 Ne.
2./4 A.33
S./3 P.9
Nikon4300
131 S/162 A//15 P/27 Nc.
5./0 A//1 N.
1./2 A//19 P/2 Ne.
Creative
582 S.677 A,422 P/255 N./O Ne.
68 S./85 A./42 P/43 N.O Ne.
5./5 A./2 P/33 N.O Ne.
CanonG3
190 S./228 A/180 P/48 NJO Ne.
25 $./29 A/21 P/8 N.O Ne.
24 S./29 A.24 P/5 NO Ne.
ApexAD
281 S//43 A/146 P.117 N./O Ne.
35 S./43 A/16 P/27 NJO No.
28 S./43 A./31 P//2 N./O Ne.
C5nonD500
103 S/118 18 A./9 P/22 N. Ne.
11 S./15 A/14 P/1 N.O0 Ne.
13 S./5 A//1 P/4 N. Ne.
CanonI00 100
137 S./175 A./123 P/52 N.O Nc.
1./2 A/20 P/2 N./0 No. 22 S./24 A/18 P/6 N./0 No.
16 S,/2 A.21 P/1 N.O Ne.
Diaper
166 5 A./143 P/48 NJO No.
24 S./24 A./2 P/2 N.O Ne.
Hitachi
152 S.212 A./153 P/59 NJO No.
SJ26 A/19 P/7 N.O
S.27 A/14 P./3 N.O Ne. 19 S.20 A./15 P/S N.JO Ne.
Ipod
124 S./133 A./101 P/52 NJO Nc.
S./1 A//4 P/5 N.O
Linksys
152 SJ/176 A./128 P/48 NJJ
S,/2 AJ13 N.O
20 S./2 A./16 P.7 N.JO Ne.
MicroMP3
384 S./484 A./40 P//44 N./O No.
42 S./61 A./8 P/13 No.
S./6 A./3 P/22 N.OO Ne.
Nokia6600
298 S./62 A/244 P.1 18 N./O Ne.
26 S./45 A./2 P/13 N.O Ne.
S./4 A./3 P//6 N.OO Ne.
Norton
168 S./194 A./5 P/140 N.J0 Ne.
17 S./24 A/15 P/9 N.O Ne.
24 S./25 N./ P/20 N.JO Ne.
Restaurant
1893 S.34552 A,/2094 P/779 N./79 Ne.
84 S./15 A/70 P/26 N.S4 Ne.
600 S./1 120 A/728 P/196 NJ/96 Ne. 411 S.638 A./31 P/128 N./169 Ne.
Lpptop
1360 63/2663 A./90 P/800 N.433 Ne.
98 S./50 A.57 P/66 N.27 Ne.
Table 3: Statistics of the ASC tasks. S.: number of sentences; A: number of aspects; P., N., and Ne.: number aspects with positive, negative and neutral opinions, respectively. Note that the "Restaurant" and "Laptop" have 3 classes of opinion polarities (positive, negative and neutral) while the others have only classes (positive and negative).
Dataset
Dasks/Domains
#Training
#Validating
#Testing #Clastes
Yahoo
4500
500
4840
10
AGnows
1785
199
1756
898
100
998
Dhpedia
6237
693
6748
14
Yelp
900
100
984
icsi
43
10
6
ami
97
20
20
reddit
201
250
SUM
stack
205
50
250
nyt
200
50
250
emails
56
hotel
3366
143
177
DRG
attraction
298
27
28
train
1954
196
262
569
39
con112003
3453
wikigoold
200
170
170
9
NER
btc
200
934
934
9
re3d
200
77
200
gum
250
1000
Table 4: Statistics of the CCD. SUM. DRE and NER datasets. Number of classes is not applicable to SUM and DRG because they are generation datasets.

---

LAMOL, we directly run the author provided code. Except for the aforementioned hyper-parameters. follow those
is more reasonable. Let us use some examples illustrate,
all saselice-specific in their original papers.
(I). if task archives the accuracy 0.5 right af- ter its training, it achieves 0.4 after task 2, and it achieves 0.3 after task (final task), In this case, both measures give the same forgetting 0.2.
D Details of CL Baselines
5 adapter-based methods (CTR (Ke et al., 2021a), HAT (Serrà al., 2018), SupSup (Wortsman et al.. 2020). CAT (Ke et al., 2020) and CUBER (Lin al., 2022a)). They all train a shared adapter while SupSup trains popup scores to indicate sub-network on the fixed adapter. HAT one of the most ef- fective TIL methods with little forgetting. CTR encourages transfer via capsule networks and trans- fer routing. SupSup uses the similar sub-network discovery method TSS but cannot do KT at all. CAT and CTR are two systems that deal with both CF and KT. CUBER also deals with CF and KT, but a hard threshold is needed.
(2). if task archives the accuracy 0.5 right after its training, achieves 0.8 after task 2. and it achieves 0.82 after task 3 (final task). If we take max (F)), then the forgetting is 0.02, but our method will give -0.32. In this case, our method is more reasonable because it precisely shows how much backward trans- fer (negative value here means backward trans- fer) has been achieved for task after task 2 and task are learned since both measures evaluate from the same reference point, i.e., after all tasks are learned (the last task t both measures, i.e., task in this example).
prompt-based method (L2P (Wang et al., 2022)), which trains a prompt pool to transfer task knowledge and a key-value pair prompt selection strategy to select the task-specific prompt (it thus deals with both KT and CF).
(3). If task archives the accuracy 0.5 right after its training. 0.8 after task 2 and 0.4 after task (final task), our metric will give the forgetting 0.1. Fy will give 0.4. This case is more debatable because Fý catches the worst forgetting. But since we evaluate after all tasks are learned (the reference point is when the last task is learned), again we believe that method is more reasonable.
3 baselines that modify the Transformer (LAMOL (Sun et al., 2020), EWC (Kirkpatrick et al., 2016) and DAS (Ke et al., 2023)). LAMOL is a pspudo-replay method using GPT-2. EWC a regularization method. DAS is a soft-masking method aims to achieve both forgetting prevention and knowledge transfer for continual pre- pre-training. Unlike TSS, it does not discover any sub-network and thus still suffers from forgetting.
(4). task archives the accuracy 0.5 right after its training, 0.1 after task 2 and 0.4 after task (final task), both metrics will give the forgetting 0. In this case, F/ does not catch the worst forgetting of 0.4 (0.5-0.1) in the process. Again, if we agree that we evaluate from the reference point of when the last task is learned, then both measures are fine in this case.
E Detailed Results for Each System
In Tables and 2 in the main paper, we only report the results of the main metries (MF1 in ASC and CCD, Fl in NER, RI in SUM and BLEU in DRG). In this section, we give all the results of all the other metrics. We report the detailed results in Tables and 6, other metrics.
see TSS is effective in all
In summary, while we believe ours more rea- sonable, a better metric may be designed in the future to characterize forgetting and knowledge transfer in the continual learning process.
F Difference between our forgetting rate and the one in (Mehta et al., 2021)
G Detailed Forgetting Rate for Each Dataset
Unlike our forgetting rate in See. 4, the forget- ting in (Mehta al., 2021) is defined as Fl - I max. (Syr,r Sr
We report the average forgetting rate of each system in Tables and 2 (in the main paper). In this section, we report the detailed forgetting rate of each system corresponding to each of the above tables.
Since both metrics measure everything from the standing point of the end of continual learning, i.c., after all tasks are learned, we believe our measure

---

Similar
Dissimilar
Average
Average
Dataset Model
ASC
NER
SUM
CCD
DRG
MF1
F1
MHI
BLEU
Main
FR
Non-continual learning
MTL
92.28
94,22
63.33
39.39
10.33
35,05
90.57
90.55
25.29
62.17
MTL (Adapter)
92.17
94.65
60.61
38.84
11.38
34.81
91.09
91.14
24.50
61.44
ONE
85.55
91.30
59.33
39.07
10.71
35.25
91.07
91.09
24.14
59.83
ONE (Adapter)
83,95
56.69
38,90
35.23
90.78
90.81
23.42
58.75
ONE (Prompt)
76.46
85.54
45.90
30.67
7.23
27.53
86.23
86.28
12.67
50.39
Continual learning of heterogeneous
Tasses
88.05
92.60
34.56
22.86
20.61
81 16.86
82.70
8.68
47.20
13.94
EWC
88.73
92.86
28.75
18.65
21.22
16.36
83.24
83.92
7.99
45.47
13.43
HAT
87.45
92.77
50.16
35.21
9.96
31.83
89.85
90.03
20.98
56.73
0.97
DAS
90.62
93.73
43.38
25.45
5.88
23.08
82.66
84.39
15.21
51.47
10:49
SupSup
85.83
92.41
58.93
38.23
11.44
34.88
90.66
90.68
24.71
59.67
0.00
CAT
45.79
50.08
15.24
17.44
2.43
15.50
36.62
39.01
10.21
29.06
30.27
CTR
89.37
93.28
50.10 16
33.98
9.97
30.80
89.96
89.98
17.63
56,22
0.95
CUBER
46.67
81.54
15.6
53.08
7.52
L2P
78.10
87.97
39.03
28 34
5.93
25.44 44
72.35
76.99
4.55
44.47
6.44
TSS
90.61
94.47
62.13
38.29
11.77
35.26
90,70
90.70
24.56
61.26
0.00
Continual learning of homoggrncous
Tasks
CL
89.25
93.04
49.19
32.68
6.84
29.19
85.08
85.10
22.31
55.70
6.42
EWC
88.36
92.60
$1.76
32.64
7.12
29.00
87.27
87.37
18.30
58.06
5.53
HAT
89.33
93.28
52.31
37.11
10:40
33.53
90.21
90.23
21.47
58,08
0.49
90.94
31.31
87.97
88.1
54.92
9.26
SupSup
85.83
92.41
58.93
38.23
11.44
34.88
90.66
90.68
24.71
59.67
0.00
LAMOL
84.62
90.17
10.88
1.39
6.87
54.44
67.04
19.96
50.73
37.24
90.82
21.72
1.37
CTR
88.86
92.94
51.85
37.34
10.73
33.69
90.54
90.58
21.39
58.00
0.65
CUBER
91.25
94.33
47. 19
30.44
6.75
27.51
89.19
89:27
21 66
55,95
5.82
84.64
44
26.65
23.90
85.35
85
8.52
47.91
2.58
TSS
91.28
94.06
63.96
38.39
11.60
35.22
90.89
90.92
24.75
61,85
0.00
Table 5: Performance for heterogeneous (40 tasks in total) and homogeneous tasks, averaged over 5 random
sequences (the standard deviation is reported in Table 9). meaas not applicable. We bold the best performance within CL baselines.
Similar
Dissimilar
Average Main
Averuge
Dataser
ASC
NER
SUM
CCD
DRG BLEU
Model
RI.
MFI
Acc
FR
Non-continnal learning
ONE
85.55
91.50
59.33
39.07
10.71
35.25
91.07
91.09
24.14
59.83
Continual
learning of heterogeneous Tasks
37.03
28.04
6.39
14.85
9.02
TSS (wlo SM)
85.83
92.41
58.93
38.23
11.44
34.88
90.66
90.68
24.71
59.67
0.00
Naive)
93.36
34.79
36.11
10.27
33.15
89.99
89.99
24.19
54.80
0.00
TSS
90.61
94.47
62.13
38.29
11.77
35.26
90.70
90.70
24.56
61,26
0,00
Continual learning of homogenoous Tasks
94.24
40.47
7.30
22.77
9.61
TSS (wlo SM)
85.83
92.41
58.93
38.23
11.44
34.88
90.66
90.68
24.71
59.67
0.00
Naive)
94.25
62.42
38.06
11.52
90.63
90.74
24.59
61.22
0.00
TSS
91.28
94.116
63.96
38.39
11.60
35.22
911.9
90.12
24.75
61.85
0.00
Table 6: Ablation experiment results for heterogeneous and homogeneous tasks averages over 5 random sequences (the standard deviation is reported in Table 10).
Table 7 and Table 8 show the detailed forget- ting rate for both heterogeneous and homogeneous tasks.
SupSup have the lowest forgetting rate (TSS outper- forms SupSup in final performances due to knowl- edge transfer). We also notice the forgetting is in general less than heterogencous tasks because the tasks heterogeneous sequence can be more dissimilar (e.g., from summarization to NER).
In heterogeneous tasks, we can see on average TSS and SupSup have the lowest forgetting rate (TSS outperforms SupSup in final performances due to knowledge transfer). While some baselines (e.g. DAS, CUBER, EWC) have a negative for- getting rate (indicating the new knowledge helps similar old tasks), they suffer from large forgetting in dissimilar tasks because they cannot do well in forgetting.
H Standard Deviations
This section reports the standard deviations of the corresponding results in Tables and 2 (in the main paper) of TSS and the considered baselines over 5 runs with random sequences. We only report the CL baselines since they are related to the task order. We can see the results of TSS is stable. SupSup
In homogeneous tasks, similar to the hetero- geneous tasks, we can see on average TSS and

---

Similar
Dissimilar
Average
Dataset Model
ASC
NER
SUM
CCD
DRG BLEU
MF1
Acc
F1
RI
RL
MFI
Acc
FR
Continual
learning of beterogencous Tasks
NCL
1.89
3.41
26.89
15.79
6.16
8.77
8.65
7.35
16.47
13.94
EWC
-0.16
0.57
29.29
19.50
6.70
10.33
3.06
6.53
15.46
13.43
HAT
-0.98
0.11
1.54
2.21
1.20
1.49
0.62
1.12
1.44
0.97
DAS
-0.44
0.01
21.66
13.48
5.65
7.32
8.02
7.86
9.73
10.49
SupSup
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0..00
0.00
CAT
39.99
36.92
18.30
25.23
9.99
13.93
53.85
32.99
13.96
30.27
CTR
-0.15
-0.13
0.72
2.31
0.64
1.39
0.47
0.38
1.42
0.95
CUBER
-1.84
-1.72
12.98
11.54
5.09
6.65
1.56
1.55
12.09
7.27
L2P
2.53
0.09
4.04
5.84
2.87
3,99
12.66
11.31
7.16
6.44
TSS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.000
0.00
Continual
learning of homogencous Tasks
NCL.
-0.44
0.31
15.49
8.37
5.54
5,90
5.67
5.64
3.01
6.42
EWC
7.81
2.78
7.53
2.58
1.68
2.04
8.94
8.17
0.78
5.53
HAT
-0.88
-0.36
0.79
1.10
1.22
0.78
-0.07
-0.08
1.48
0.49
DAS
-0.81
-0.77
31.88
7.87
3.89
4.98
3.73
3,56
3.61
9.26
SupSup
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
LAMOL
-1.88
-1.52
3.90
1.42
2.32
32,26
27.30
2.29
CAT
2.05
0.81
0.81
1.85
1.41
1.37
-0.13
-0.21
2.26
1.37
CTR
-0,38
-0.08
0,58
1.43
0.90
0.86
-0.27
-0.27
1.91
0.65
CUBER
-0,38
-0.08
0.58
1.43
0.90
0.86
-0.27
-0.27
1.91
0.65
L2P
0.30
0.54
0.96
7.27
3.53
4.68
0.94
0.82
3.42
2.58
TSS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 7: Forgetting rate for heterogeneous (40 tasks in total) and homogeneous tasks, averaged over 5 random sequences.
Similar
Dissinilar
Average
Dataset Model
ASC
NER
SUM
CCD
DRG BLEU
MFI
Acc
F1
R1
R2
RL.
MF1
Ace
FR
Continual learning of Intterogeneous Tasks
TSS (wla SD)
.0.88
0.24
20.90
8.88
4.19
4.57
8,38
8.53
7.41
9.02
TSS (w/o SM)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
TSS SM; Naive)
0.00
0.00
TSS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Continual learning of homogeneous Tasks
ISS (wlo SD)
-1.11
-0.77
34.14
10.18
4.94
6.62
2,83
2.74
2.62
9.61
TSS (wlo SM)
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
TSS Naive)
0.00
0.00
TSS
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 8: Forgetting rate for ablation of heterogencous tasks (40 tasks in total) and homogeneous tasks, averaged over 5 random sequences.
and TSS (w/o SM) discover different sub-networks for different tasks, so they are not related to the task order and are not reported in the table. We report the standard deviations for all metrics for completeness.
Results for Individual Sequences
In Table 1 in the main text, we reported the results averaged over 5 random sequences (different task orders). In this section, we give the results of TSS of each sequence in Table 11. We can see that the order indeed affects the results but not by much. In summary, we believe the average over random sequences in Table (in the main text/paper) can show the effectiveness of TSS.
In heterogeneous tasks, Table and Table 10 re- port the standard deviations TSS and the consid- ered baselines over 5 runs with random sequences. We can see the results of TSS and its variants are stable.
Execution Time and Number of Parameters
Table 12 reports the number of parameters (regard- less of trainable or non-trainable), training execu- tion times for different CL models. Our experi- ments were run on GeForce GTX 2080 Ti with
In homogeneous tasks, similar to heteroge- neous tasks, we can see the results of TSS and its variants are stable.

---

Similar
Dissimilar
Dataset
ASC
NER
SUM
CCD
DRG
Model
MP1
Ace
F1
R
R2
RI.
MPI
Ace
BLBU
Continual
learning of beferogeneous tasks
NCL
20,033
20.1397
=0.0779
100051
200054
100150
20004
100527
EWC
=0,114
+0.0500
=0.1278
=0.1047
+0.2167
+0.0947
200171
+00155
20,0914
HAT
20,035
100012
200078
000041
10.0024
10.0046
200000
10.0023
100119
DAS
20,0226
100072
-00348
00.116
10.0300
10,0000
100041
100375
10.0638
CAT
20.1752
102217
20003
0.0549
10.0318
10.0744
10.1176
102263
10.0333
CTR
+0.447
+00023
-00712
-00278
+0.058
+0.029
+0,1492
+0.085
+00704
CUBER
+00175
+0.107
-00589
0.0682
+00008
+0.098
+0.994
+0.096
+00631
1.2P
+00173
+00060
-00225
-00328
+00153
+0,759
+00267
TSS
10.0058
10.0124
20.050
20.0025
10.0018
10.021
10.0047
100034
10.005
Contional
learning of homogeneuus tacks
NCL
10,0040
10.027
-0,777
0.0000
10.0055
10.0066
10.0233
100050
10,0800
EWC
10.0093
10.0063
000707
000132
10.0053
10.0113
10.0227
10,0216
100217
HAT
+0,0330
100042
00006
000130
10.0049
100115
10,028
10,0088
10.084
DAS
20.0755
10.0024
000647
000091
1000700
10.0074
200178
10.0755
10.0044
LAMOL
10.042
20.0317
+0.0288
200068
CAT
20.009
10.0211
20010
000104
20.0000
10,0000
20.0019
10.0011
20.0511
CTR
10,0080
10.0052
CUBER
20.0072
10.0031
20.0543
0.0155
10.0061
10.0119
10.0069
10.0062
10.0000
L2P
L0,0550
-0.0022
000225
10.0047
10.0203
10.081
100076
100016
TSS
-00631
+0.0055
-0.0152
-0.021
+0.020
+0.015
+00024
+0.026
+0.055
Table 9: Standard deviations of the corresponding metrics the proposed TSS model and the baselines on the heterogeneous and homogeneous tasks.
Similar
Dissimilar
Dotaset Model
MF1
Acc
F1
82
RL
MPL
Ave
BLEU
Tearning beterogeeeun
7SS twto SD)
20.866
20,3637
20.332
20.033
200016
20,075
2009M2
321322
-0,841
TSS (wl SM, Naivey
=0,009
#0,0033
+003307
=0.010
-0.0033
=0.0007
#0
=0.0049
20,025
100007
13003
Continual learning of homegeneous tasks
TSS (wio SD)
000000
10,0066
100089
0.0088
0,0056
10.0071
100274
1,0028
0.00399
SM: Namve
00,0102
1000
-0,024
100006
ESS
20,001
20.005
200022
-0,02]
20,0000
20,0115
2500024
200000
20,033
Table 10: Standard deviations of the corresponding metries of the proposed TSS model and the ablation experiments on the heterogeneous and homogeneous tasks.
Sümilar
Dissimilar
Average
SUM R2
MP1
Acc
F1
R1
R1.
MP1
Acc
BL.EU
Main
90.61
94.47
62.13
38.29
11:77
35.25
90.70
90.20
24.56
61.26
90.07
93.66
62.00
38.56
11.65
35.35
90.21
90.23
24.25
61.00
Heterogenoons
89.34
93:46
48.83
37,80
11.33
34.75
91.35
91.20
25.92
58.65
89.59
93.65
58.53
38.39
11:78
35.24
90:24
90.81
25.25
60.40
88.95
90.82
61.25
38.31
11,45
35.09
90:07
90.38
25.07
60.73
91.57
94.40
63.96
38.46
11.38
35.19
91.02
91.08
24.89
61.98
91.65
63.19
38.53
11.62
35.28
90.84
90.86
24.07
Homogeneoms
90.96
93.62
63.26
38.58
11.85
35.40
18
91. 34
24.81
61.78
91.48
94.39
63.01
38.14
11.72
35.05
90.65
90.65
25.02
61.66
90.94
93.78
63.37
38.06
11.33
34.97
90.65
90.67
24.97
61.60 61.60
Table 11: Results for TSS in different orders for heterogeneous and homogeneous tasks.
48G GPUs memory. We only report the training time for heterogenous tasks because the training time is related to datasets and one of the main goal of TSS to work well in the challenging hetero- geneous scenario. we can see TSS is very efficient, ranking among the least number of parameters and shortest training time among all the baselines con- sidered.
constant, which has the same size the adapter.
Saved binary gates for each task. For each task, the memory usage for the saved binary gates is on average 3.1M bytes, which is 7 times less than saving all the information of the adapter (25.2M), not to mention if one saves the whole LM (3.6G). The number of accumulated importance values is

---

Model
#Params (M)
Training time (min)
NCL
896.9
600
TWC
896.9
600
HAT
958.0
600
DAS
896.9
600
LAMOI
124.4
550
CAT
896.9
24,000
1,536
CUBER
896.9
1.2P
899.0
TSS 896.9
600
Table 12: Network size (#parameters in millions, regard- less of trainable or non-trainable) and average training time per task of each CL model measured minutes for heterogenous tasks. Here we report #parameters without including the memory buffer.


Reviews:
[{"review_id": "SlqDH2faCG", "reviewer": "Reviewer_dnuC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for appreciating our work and for your careful reading. We will fix the typos, grammar style and presentation issues in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons To Accept:\n* New continual learning algorithm which try to overcome the problems faced by all other algorithms in the field.\n* Nice algorithm scheme illustrations.\n* Extensive experiments with many models and datasets.\n* Ablation study.", "reasons_to_reject": "Except for minor linguistic/presentation flaws, I see no reason to reject the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper's area is Continual Learning (CL), where a model is supposed to solve a sequence of similar or not-quite-similar tasks. The authors propose a new way to train such a model. The approach is designed in order to overcome two common Continual Learning problems: catastrophic forgetting (when the model \"forgets\" how to solve the previously seen tasks) and limited knowledge transfer (when the model's \"experience\" of solving tasks does not help it to solve a new task).\n\nContributions:\n* New continual learning algorithm which try to overcome the problems faced by all other algorithms in the field.\n* The core idea of the method (as far as I understood) is based on \"attention\", which is quite a popular and wide-used concept.\n* Detailed description of the area and approaches previously used (not a survey study but sometimes feels like it).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 22: \"AI\". Probably also needs full naming when abbreviation is first used.\n* Line 34 (also line 178): \"should (1) overcome\". It took some time before I realized the meaning of number \"(1)\". If this is arranged as a paragraph (not a numbered list), then maybe better use words like \"first\", \"second\".\n* Line 123: Better say \"real number\", not just \"number\".\n* Figure 1: Right column's captions not centered (forward propagation, backward propagation).\n* Line 333: \"Importance values\" (plural).\n* Line 336: \"as follows.\" (point not colon).\n* Footnote 7: duplicate \"the\"\n* Line 367: \"Kaiming Initialization\". Provide a reference?\n* Line 378: \"pupup scores\"\n* Line 380 (eq. 6): Small vertical braces. Also, maybe something is wrong with the denominator.\n* Lines 446 - 474 (and many others). I highly recommend reconsidering the use of bold fony. In my opinion, currently it is used too much.\n* Line 460: excess \"dot\" symbol.\n* Table 2: not all best values in \"FR\" column are typed on bold.\n* Line 527: \"Knowledge transfer (KT)\" In my opinion, better to omit \"(KT)\" here. (Or, on the contrary, omit \"Knowledge transfer\")\n* Line 993: \"Let us use some examples to illustrate:\" (colon, not comma). Also, probably better use a numbered list below, not bulleted.\n* Lines 1003, 1004: Wrong formatting of negative numbers (this is not a minus sign).\n* Table 12: Training time 48000 min... Maybe I missed something but such an ominous number just stunned me for a while :)\n"}}, {"review_id": "g7F0PuXVrL", "reviewer": "Reviewer_Naqz", "summary": "", "questions": "1) Why were small training samples (less than 30,000) used from 40 different tasks in the experiments? Did the authors attempt to use all the training samples for tasks in a smaller homogeneous sequence for a fair comparison of TSS against other continual learning methods? Was the relationship between the proposed method's performance and the number of training samples analyzed?\n\n2) Was the method compared with a replay-based baseline given the small size of the training dataset (less than 30,000 samples)?\n\n3) Did the authors analyze the relationship between popup scores of different tasks? Are popup scores closer for similar tasks? How does the proposed initialization and soft gradient masking technique affect the learned popup scores?\n\n4) In Section 3.1.1, why was there no specific explanation for the chosen value of the threshold parameter (ε)? Is this parameter set uniformly for all tasks, or do different tasks have different settings?\n\n5) In Section 3.1.2, what is the calculation method for normalizing and rounding the gradient-based importance using Tanh()? Why do the importance scores after Tanh() activation fall within the [0, 1] range, despite the original range being [-1, 1]? What impact does normalizing the importance values to -μ for parameters that initially had an importance of 0 have on the subsequent importance calculations?\n\n6) In Section 4.2, the authors reported average performance on all datasets. However, since different tasks have different evaluation metrics, why did they choose to average F1, BLEU, and ROUGE-1 scores? Why not normalize task performances before averaging?\n\n7) In Section 4.2, why did the authors use ONE as a control for analyzing Knowledge Transfer (KT) when ONE and TSS have different model architectures? Wouldn't it be more reasonable to compare each task's performance independently with its performance in the task sequence?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Questionable Applicability. Continual learning are commonly applied in scenarios with large training datasets...\n\nThere may be a misunderstanding here. Continual learning research does not have any requirement or restriction on the dataset size. If we think about human learning, we learn continually all the time and we almost always learn in a few-shot manner rather than collect a large amount of data before learning. \n\nFor the datasets we used in the paper, they have been used extensively in the literature, e.g., ASC has been used in [1,2], CCD has been used in [3,4], NER and SUM have been used in [5], and DRG has been used in [6]. Except for NER and CCD, where we do random sampling due to our compute resource limitations, we directly use the original full datasets (ASC, SUM, DRG). We regard TSS’s ability to work well with “small” datasets as an advantage rather than disadvantage since labeling data is very expensive. Nevertheless, we agree that experiments on larger datasets can help strengthen our work.\nFollowing your suggestion, we ran experiments on the large-CCD [3] (76k samples) and large-NER (25k samples). The latest dataset statistic for large-CCD and large-NER are:\n| Large-CCD   | Tasks/Domains | #Train \t| #Test   | #Class |\n|-----------|---------------|------------|------------|--------|\n| AGNews\t| News      \t| 8,000  \t| 7,600  \t| 4  \t|\n| Yelp  \t| Sentiment \t| 10,000 \t| 7,600  \t| 5  \t|\n| Amazon\t| Sentiment \t| 10,000 \t| 7,600  \t| 5  \t|\n| DBPedia   | Wikipedia \t| 28,000 \t| 7,600  \t| 14 \t|\n| Yahoo \t| QA        \t| 20,000 \t| 7,600  \t| 10 \t|\n| **Total** | **---**   \t| **76,000** | **38,000** | **38** |\n\n| Large-NER   | #Train \t| #Test  | #Class |\n|-----------|------------|-----------|--------|\n| conll2023 | 14,041 \t| 3,453 \t| 9  \t|\n| wikigold  | 1,356  \t| 170   \t| 9  \t|\n| btc   \t| 7,470  \t| 934   \t| 9  \t|\n| re3d  \t| 688    \t| 200   \t| 21 \t|\n| gum   \t| 2,245  \t| 1,000 \t| 23 \t|\n| **Total** | **25,800** | **5,757** | **71** |\n\nDue to the time constraint, we compare with popular baselines NCL and ONE as well as the top competitive baselines SupSup and HAT in homogenous tasks (average Macro-F1 over 5 random sequences). \n|  Model  | Large-NER | FR       | Large-CCD | FR       |\n|:-------:|-----------|----------|-----------|----------|\n| ONE     | 76.68     | ---      | 78.14     | ---      |\n| NCL     | 42.57     | 43.65    | 72.93     | 6.62     |\n| SupSup  | 73.96     | 0.00     | 78.15     | 0.00     |\n| HAT     | 61.87     | 0.36     | 77.05     | 0.36     |\n| **TSS** | **76.84** | **0.00** | **78.38** | **0.00** |\n\nWe can observe that TSS is still outperforming other baselines, illustrating the effectiveness of TSS in larger datasets. Unlike the improvement observed in Table 1 on NER, we note that the TSS is similar to ONE in the large-NER. This is because knowledge transfer is less obvious when more data for each task is available. \n\n[1]: Achieving forgetting prevention and knowledge transfer in continual learning, Ke et al., NeurIPS 2021  \n[2]: Adapting BERT for continual learning of a sequence of aspect sentiment classification tasks, Ke et al., NAACL 2021  \n[3]: Continual Learning for Text Classification with Information Disentanglement Based Regularization, Huang et al., NAACL 2021  \n[4]: Episodic memory in lifelong language learning, de Masson d’auttume et al., NeurIPS 2019  \n[5]: LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5, Qin et al., ICLR 2022  \n[6]: Continual Learning in Task-Oriented Dialogue Systems, Madotto et al., EMNLP 2021  \n\n> Q2: Impact on CL Baselines. Existing methods may suffer from insufficient data...\n\nRelated to Q1, we believe that working well on less data is an advantage. We also want to note that we do not reduce the data size for ASC, SUM and DRG. In fact, random sampling is a common preprocessing in continual learning papers to reduce the amount of computation. For example, [3] samples 2000 samples per class for CCD. Following your suggestion, we have conducted additional experiments on larger datasets in the answer to Q1. \n\n> Q3: Missing Replay-based Baseline\n\nWe didn’t include replay-based methods as we thought it was unnecessary because we have included multitask learning (MTL) (which replays all the past data) as the upper bound, and TSS is comparable to MTL. Also, LAMOL is a popular pseudo-replay method and TSS outperforms LAMOL by a large margin. We agree that adding replay-based baselines is beneficial. We added a popular replay-based method DER++ [8] in homogenous sequences (50 replay samples are saved for each task, which is the largest memory we can apply). We could not run on heterogeneous task sequences on DER++ due to a huge memory requirement by DER++. DER++ has been used in [1,7] and has been shown to be the best replay-based method [7]. All hyperparameters follow those of [1,7]. Results are averaged over 5 random sequences. \n\n|  Model  | ASC   \t| NER   \t| SUM   \t| CCD   \t| DRG   \t|\tAvg.   | Avg. \t|\n|:-------:|-----------|-----------|-----------|-----------|-----------|:---------:|----------|\n|     \t|\tMF1\t| \tF1\t| \tR1\t|\tMF1\t|\tBLEU   |\tMain   | FR   \t|\n| DER++   | 90.88 \t| 61.74 \t| 36.45 \t| 86.51 \t| 20.65 \t| 59.25 \t| 5.11 \t|\n| **TSS** | **91.28** | **63.96** | **38.39** | **90.89** | **24.75** | **61.85** | **0.00** |\n\nWe can see that TSS outperforms DER++, which further illustrates the effectiveness of TSS. \n\n[7]: Continual Pre-training of Language Models, Ke et al., ICLR 2022  \n[8]: Dark Experience for General Continual Learning: a Strong, Simple Baseline, Buzzega et al., NeurIPS 2020  \n\n> Q4: Ignoring Resource Disparities. By limiting the training data size uniformly across all tasks...\n\nIn fact, most of our datasets are imbalanced except one. Thus, we have a good mix of balanced and imbalanced datasets, which is desirable for good evaluation. ASC is highly imbalanced (see Table 3 in Appendix for statistics) and TSS works very well on this data set. SUM and DRG are imbalanced too. The only balanced dataset is NER (see Table 4 in Appendix for statistics). TSS actually works well on the resource disparities scenario.\n\n> Q5: Why were small training samples (less than 30,000) used from 40 different tasks in the experiments? Did the authors attempt to use all the training samples for tasks in a smaller homogeneous sequence for a fair comparison of TSS against other continual learning methods? \n\nWe have added experiments with larger datasets in the answer of Q1. Based on the new results and those in Table 1 in the paper, we can see that TSS outperforms baselines under both “small” and “large” datasets. Along with the added datasets, experiments on SUM, NER, DRG and ASC all use their full datasets. For CCD, to our knowledge, none of the existing continual learning work uses the full dataset [3,4] as it is huge (> 500k training samples). We followed [3] and evaluated our method on the large-CCD datasets as well (see the dataset sizes in our answer to your Q1). For the ”smaller sequence”, we have evaluated our method with sequences of various lengths, ranging from 5 to 40 in the paper, which we believe is desirable for good evaluation. \n\n> Q6: Was the relationship between the proposed method's performance and the number of training samples analyzed? \n\nComparing the added large-NER with the NER (small) in Table 1, we observe that larger datasets improve the performance of both TSS and the baselines, but TSS continues to outperform baselines. Note that the results for large-CCD and CCD (small) in Table 1 are not comparable because CCD (small)  uses a binary classification version of Amazon and Yelp (2 datasets in CCD) while large-CCD from [3] uses the 5-class classification version of these two datasets. \n\n> Q7: Did the authors analyze the relationship between popup scores of different tasks? Are popup scores closer for similar tasks? How does the proposed initialization and soft gradient masking technique affect the learned popup scores?\n\nPopup scores are used to identify what network parameters are useful for each task. We do not believe there is any relationship between them for different tasks. We also note that TSS does not employ any task similarity detection method and thus popup scores are not related to task similarity. They simply indicate what parameters should be included in the sub-network for the current task. As a result, the popup scores are not necessarily closer for similar tasks because there could be many possible sub-networks even for the same task. Our method only finds one. The initialization and soft masking are mainly used for dealing with forgetting and knowledge transfer. They only indirectly affect the current popup scores as those parameters/knowledge that can be shared across tasks should have higher popup scores. They will not affect those already learned popup scores for previous tasks because we save the sub-networks identified by popup scores for previous tasks (L308).\n\n> Q8: why was there no specific explanation for the chosen value of the threshold parameter (ε)?  Is this parameter set uniformly for all tasks, or do different tasks have different settings?\n\nSince we proposed a novel method to enable the supermask method [9] to transfer knowledge across tasks, which originally learns each task completely independently, we follow [9] to set the value of the threshold. The same threshold is used across all tasks\n\n[9]: Supermasks in Superposition, Wortsman et al., NeurIPS 2020\n\n> Q9: what is the calculation method for normalizing and rounding the gradient-based importance using Tanh()? Why do the importance scores after Tanh() activation fall within the [0, 1] range, despite the original range being [-1, 1]? What impact does normalizing the importance values to -μ for parameters that initially had an importance of 0 have on the subsequent importance calculations?\n\nWe apply normalization to the importance of each layer, i.,e., $\\text{Norm}(I_l^{(k)})= \\frac{I_l^{(k)} - I_l^{(k)}.mean()}{I_l^{(k)}.std()}$. We noticed that we had a typo in Eq. 6. The absolute operation should not be taken in Eq. 6 but after the Tanh so that the importance scores fall within [0,1]. Let the “initial importance” be the importance score computed after (the corrected) Eq. 6. The normalized importance for the 0 initial importance is around 0 (may not be exactly 0) because the mean is always close to 0. As a result, $Norm(I_l^{(k)})\\approx 0$ and its corresponding parameter can be updated with almost no masking (Eq.5) by the subsequent task.\n\n> Q10: since different tasks have different evaluation metrics, why did they choose to average F1, BLEU, and ROUGE-1 scores? Why not normalize task performances before averaging?\n\nWe reported the average score mainly for illustration purposes as the range of all metrics fall in [0, 1]. We agree that the readers should compare the individual results. We see the TSS works the best in terms of individual results as well.\n\n> Q11:  why did the authors use ONE as a control for analyzing Knowledge Transfer (KT) when ONE and TSS have different model architectures? Wouldn't it be more reasonable to compare each task's performance independently with its performance in the task sequence?\n\nAll systems (including ONE) use the same backbone BART-large. We use ONE as control because ONE trains model independently, with no transfer or forgetting.   \n\nWe believe your second question is about comparing TSS with TSS training each task independently. TSS training each task independently is the same as removing soft-masking (which is TSS (w/o SM) in Table 2). Table 2 shows that TSS outperforms “TSS (w/o SM)”, which also indicates forward knowledge transfer. \n\n> Q12: Missing Reference: Konishi, T., Kurokawa, M., Ono, C., Ke, Z., Kim, G., & Liu, B. (2023). Parameter-Level Soft-Masking for Continual Learning. arXiv preprint arXiv:2306.14775.\n\nThis paper was published in ICML-2023, which took place (July 23-29, 2023) after the EMNLP submission deadline. Our method is also different. The main difference is that their method cannot guarantee no forgetting as their soft-masking allows changes to parameters of previous tasks. On the contrary, TSS guarantees no forgetting with the help of sub-network discovery. What’s more, their soft-masking is designed only for classification tasks, which is not applicable to generation tasks like SUM and DRG. Not to mention it is also designed for image datasets. We will include a discussion about the paper in our revision. \t\n\n> Q13: In Equation (3), the subscripts on the left and right sides are inconsistent, and the variable definition for w_{i,j} has not been declared.\n\nWe didn’t explicitly declare $w_{i,j}$ because we thought it can be inferred from $s_{i,j}$. $s_{i,j}$ means a popup score in layer $l$ and similarly, $w_{i,j}$ means a weight (in our case, it is the adapter weight, which is always fixed) in layer $l$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-structured and easy to understand.\n- Introducing a parameter importance-based gradient soft-mask on top of the SupSup method has enhanced the method's knowledge transfer capability.\n- Empirical study shows that the proposed method is effective and outperforms CL baselines.", "reasons_to_reject": "The experiment setting raises concerns regarding using small training set sizes for the 40 task datasets involved in the experiments. This approach gives rise to several issues that should be addressed:\n\n1) **Questionable Applicability**: Continual learning methods are commonly applied in scenarios with large training datasets, where the storage of data and training of new models become crucial considerations. However, the experimental setup in the paper employs less than 30k training data, significantly deviating from real-world application scenarios. This poses doubts about the applicability of the findings to practical situations.\n2) **Impact on CL Baselines**: Many CL baselines rely on using the training set of the current task to train adapters or perform fine-tuning on pre-trained models. With the training sets being drastically reduced in size, these methods may suffer from insufficient data, resulting in weakened performance.\n3) **Missing Replay-based Baseline**: The paper fails to include a replay-based method as a baseline, which is a challenging approach to beat and should be considered for a comprehensive evaluation. \n4) **Ignoring Resource Disparities**: Real-world applications often involve tasks with varying resource availability, directly impacting their relative difficulty. By limiting the training data size uniformly across all tasks, the paper fails to account for the resource differences among them, potentially obscuring important insights.\n\nIn conclusion, the experiment setting requires revision to address the aforementioned issues and enhance the validity and applicability of the study's conclusions. Consideration should be given to utilizing more substantial training datasets that align better with real-world scenarios, ensuring the evaluation of CL baselines is based on sufficient data, including the replay-based method as part of the baseline comparisons, and acknowledging and accounting for the resource disparities among different tasks.", "questions_to_authors": "1) Why were small training samples (less than 30,000) used from 40 different tasks in the experiments? Did the authors attempt to use all the training samples for tasks in a smaller homogeneous sequence for a fair comparison of TSS against other continual learning methods? Was the relationship between the proposed method's performance and the number of training samples analyzed?\n\n2) Was the method compared with a replay-based baseline given the small size of the training dataset (less than 30,000 samples)?\n\n3) Did the authors analyze the relationship between popup scores of different tasks? Are popup scores closer for similar tasks? How does the proposed initialization and soft gradient masking technique affect the learned popup scores?\n\n4) In Section 3.1.1, why was there no specific explanation for the chosen value of the threshold parameter (ε)? Is this parameter set uniformly for all tasks, or do different tasks have different settings?\n\n5) In Section 3.1.2, what is the calculation method for normalizing and rounding the gradient-based importance using Tanh()? Why do the importance scores after Tanh() activation fall within the [0, 1] range, despite the original range being [-1, 1]? What impact does normalizing the importance values to -μ for parameters that initially had an importance of 0 have on the subsequent importance calculations?\n\n6) In Section 4.2, the authors reported average performance on all datasets. However, since different tasks have different evaluation metrics, why did they choose to average F1, BLEU, and ROUGE-1 scores? Why not normalize task performances before averaging?\n\n7) In Section 4.2, why did the authors use ONE as a control for analyzing Knowledge Transfer (KT) when ONE and TSS have different model architectures? Wouldn't it be more reasonable to compare each task's performance independently with its performance in the task sequence?", "paper_topic_and_main_contributions": "This paper proposed a task incremental learning method (TSS) to achieve both forgetting prevention and knowledge transfer on a range of NLP problems. Specifically, TSS finds a sub-network for each task to avoid catastrophic forgetting and soft-mask the gradient to encourage knowledge transfer.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1) Konishi, T., Kurokawa, M., Ono, C., Ke, Z., Kim, G., & Liu, B. (2023). Parameter-Level Soft-Masking for Continual Learning. arXiv preprint arXiv:2306.14775.", "Typos_Grammar_Style_and_Presentation_Improvements": "1) In Equation (3), the subscripts on the left and right sides are inconsistent, and the variable definition for w_{i,j} has not been declared."}, "extra_text": {}}, {"review_id": "2acABjlOJD", "reviewer": "Reviewer_jmje", "summary": "", "questions": "Question A: The paper states that \"To our knowledge, only one method has been proposed to learn a sequence of mixed tasks.\". Why other task-incremental learning methods are not suitable for learning a sequence of mixed tasks? What are the special designs in your method enable learning a sequence of mixed tasks?\nQuestion B: Why the proposed method is built upon adapter? Is it an empirical choice?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The significance of studying continual learning of a mixed sequence of tasks is unclear\n\nThe concepts of “mixed sequence”, “similar” and “dissimilar tasks” have been used in the literature [1,2]. “mixed sequence” is a very practical and general scenario as we cannot guarantee that all tasks are similar or dissimilar in a continual learning application. It is very true that continual learning should not assume the types of tasks to be learned. Precisely because of this, we follow [1,2] to create similar and dissimilar tasks and mix them to create complex task sequences. We defined the similar tasks as tasks sharing the same or similar labels from different domains (see the details in L428-429). In fact, many existing continual learning works implicitly assume tasks are dissimilar and focus on forgetting prevention only. In this work, we do not have such an assumption. Instead our tasks in a sequence can be similar, dissimilar or a mixture of them, which is the most general case.  We also allow the tasks to be homogeneous (same type of tasks) or heterogeneous (different types of tasks). As a result, we believe that our settings are much more general and realistic than the existing work that assumes tasks are dissimilar and homogeneous. \n\n[1]: Continual learning of a mixed sequence of similar and dissimilar tasks, Ke et al., NeurIPS 2020  \n[2]: Continual Sequence Generation with Adaptive Compositional Modules, Zhang et al., ACL 2022\n\n> Q2: The core idea of the proposed method (e.g., subnetwork discovery and importance score computation) is not innovative. \n\nWe are eager to address your concerns. We hope you can give us more details and also prior work references that are similar to our work.  \n\nTSS addresses both forgetting prevention and knowledge transfer in a wide range of homogeneous and heterogeneous tasks. Only CAT [1] tries to do the same but its similarity detection can go very wrong and TSS does not need such task similarity detection.\n\n> Q3: Furthermore, there are no strong connections between the proposed method and the setting of continual learning of a mixed sequence of tasks. It seems that the proposed method can be apply to conventional task-incremental learning settings.\n\nThe conventional TIL setting uses dissimilar and homogeneous tasks. Yes, TSS can work in this setting. We can see from Table 2 that TSS works very well. What’s more, TSS also works well in the other much more challenging settings (i.e., heterogeneous, similar tasks, and mixed sequences). TSS can achieve these because our soft-masking mechanism can transfer knowledge and sub-network discovery can prevent forgetting, which are the two key aspects needed in the general setting beyond the conventional TIL. \n\n> Q4: The method part of the paper is hard to follow (e.g., Figure 1 is difficult to understand). Besides, the rationale of some model design is quite confusing.\n\nWe will improve our writing to make everything clearer. We noticed that reviewers dnuC and Naqz thought our paper was easy to understand. \n\nAbout Figure 1, we use one adapter layer as an example to show the two functions of TSS. For sub-network discovery (left-hand side), we train the popup scores ($s_l^{(t)}$) with soft-masking. The forward propagation on the top left shows how the popup scores are used in the computation graph and the backward propagation on the bottom left shows how we use importance to soft-mask the gradients. For importance computation (right-hand side), the forward propagation is exactly the same as the one in sub-network discovery but we use the gradients of the popup scores to indicate the importance (bottom right) instead of updating the popup scores as in sub-network discovery. \n\n> Q5: The paper claim that the proposed method encourage knowlege transfer, but the results of some key metrics (e.g., forward/backward transfer) are not shown in the experiments.\n\nWe reported forgetting rate (L535), which is essentially a backward transfer metric. If it is negative, there is backward transfer; if it is positive, there is forgetting. In Table 1, we can see that none of the systems has backward transfer and most baselines systems have forgetting except SupSup. Our TSS performs better than all baselines including SupSup.\n\nAs for the forward transfer, we use ONE as control (L544-L551) and observe that TSS can do better in similar tasks which indicates positive forward transfer. We can also report forward performance as in [1], i.e., comparing the performance of ONE and the performance of the proposed method when it learns each task. To strengthen our paper, below we report the forward results as in [1] for the popular baseline NCL as well as the top performing baselines CTR and HAT (average over 5 random sequences). Note that SupSup trains individual tasks independently, so it has no forward transfer.\n\nHeterogeneous tasks\n|  Model  | ASC       | NER       | SUM       | CCD       | DRG       |           |\n|:-------:|-----------|-----------|-----------|-----------|-----------|-----------|\n|         |    MF1    |     F1    |     R1    |    MF1    |    BLEU   |  Average  |\n| NCL     | 89.14     | 61.80     | 38.36     | 89.78     | 24.96     | 60.81     |\n| HAT     | 88.33     | 51.47     | 37.77     | 90.87     | 23.28     | 58.35     |\n| CTR     | 89.22     | 50.88     | 36.29     | 90.44     | 19.05     | 57.18     |\n| **TSS** | **90.61** | **62.13** | **38.39** | **90.90** | **24.98** | **61.40** |\n\nHomogenous tasks\n| Model | ASC       | NER       | SUM       | CCD       | DRG       |           |\n|:-----:|-----------|-----------|-----------|-----------|-----------|-----------|\n|       |    MF1    |     F1    |     R1    |    MF1    |    BLEU   |  Average  |\n| NCL   | 88.84     | 63.96     | 38.98     | 90.39     | 24.69     | 61.37     |\n| HAT   | 88.56     | 52.89     | 38.58     | 90.04     | 22.79     | 58.57     |\n| CTR   | 88.45     | 52.39     | 38.86     | 90.31     | 23.18     | 58.64     |\n| TSS   | **91.28** | **63.96** | **38.99** | **90.89** | **24.75** | **61.97** |\n\nWe can see the forward transfer performance on both heterogenous and homogenous tasks which again show that TSS has strong forward knowledge transfer.\n\n> Q6: The paper states that \"To our knowledge, only one method has been proposed to learn a sequence of mixed tasks.\". Why other task-incremental learning methods are not suitable for learning a sequence of mixed tasks? What are the special designs in your method enable learning a sequence of mixed tasks? \n\nThe reason is that most existing task-incremental learning (TIL) methods implicitly assume tasks to be dissimilar and homogeneous. As a result, they only focus on forgetting prevention. While some also do knowledge transfer, they do not consider the heterogeneous tasks and mixed sequences of similar and dissimilar tasks, which are significantly more challenging. Thus, they do not have mechanisms to automatically balance knowledge transfer and forgetting prevention when both are needed in a single sequence. In this work, we argue that the key to learning a mixed sequence of tasks is that the system should not have a priori (L109) information about what knowledge can be transferred and what knowledge should only be learned from the current task data. One way to address this is to detect task similarity as CAT [1], but the detection could be wrong, which causes serious forgetting. TSS proposes to maintain information from previous tasks via soft-masks and use all the information as initialization (L361-L367) to let the current task automatically choose the previous knowledge to use or to transfer. This naturally gets rid of task similarity detection and achieves both forgetting prevention and knowledge transfer in a mixed sequence. We can also see from the experimental results that our method outperforms strong baselines. \n\n> Q7: Why the proposed method is built upon adapter? Is it an empirical choice?\n\nAs a parameter-efficient tuning method, the adapter is widely used, including in continual learning [3,4,5]. But we believe our method is not limited to adapters. Other parameter-efficient tuning methods and even the conventional fine-tuning can be used too, as the sub-network discovery and soft-masking are model-agnostic by design. \n\n[3]: Achieving forgetting prevention and knowledge transfer in continual learning, Ke et al., NeurIPS 2021  \n[4]: Adapting BERT for continual learning of a sequence of aspect sentiment classification tasks, Ke et al., NAACL 2021  \n[5]: Continual Sequence Generation with Adaptive Compositional Modules, Zhang et al., ACL 2022  \n\n> Q8: Missing references   \n[R1] Ermis B, Zappella G, Wistuba M, et al. Memory efficient continual learning with transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 10629-10642.  \n[R2] Zheng J, Liang Z, Chen H, et al. Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 3602-3615.  \n[R3] Xia Y, Wang Q, Lyu Y, et al. Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 2291-2300.  \n\nThanks for pointing out these related works. [R1] (ADA) extends conventional adapters by proposing a method to distill knowledge between adapters in continual learning.  It keeps a fixed stack of K adapters, and when a new Adapter needs to be added to the stack, ADA takes the adapter that has the highest similarity with the new task and distills its knowledge to the new task. This is very different from TSS as TSS uses only a single adapter but discovers different sub-networks. In ADA, while distillation can help mitigate forgetting, it is insufficient. Its similarity detection could also be wrong like CAT and cause forgetting. TSS does not have these problems because it discovers different sub-networks and does not do similarity comparison. [R2] and [R3] are both about entity-type incremental learning which is a kind of class-incremental learning (CIL). We focus on task-incremental learning (TIL), which is a different setting than CIL as stated in footnote 2. We will include a discussion about these papers in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The extensive experiments show the effectiveness of the proposed method under the setting of learning mixed sequence continually.", "reasons_to_reject": "1. The significance of studying continual learning of a mixed sequence of tasks is unclear. This paper does not provide a formal definition of  \"continual learning of a mixed sequence\" and clearify what is the exact meaning of \"similar tasks\" and \"disimilar tasks\". In a practical scenario of task-incremental learning, the model should not have assumptions about the tasks to be learned. Therefore, I think the setting is strange and unrealistic. \n2. The core idea of the proposed method (e.g., subnetwork discovery and importance score computation) is not innovative. Furthermore, there are no strong connections between the proposed method and the setting of continual learning of a mixed sequence of tasks. It seems that the proposed method can be apply to conventional task-incremental learning settings.\n3. The method part of the paper is hard to follow (e.g., Figure 1 is difficult to understand). Besides, the rationale of some model design is quite confusing. \n4. The paper claim that the proposed method encourage knowlege transfer, but the results of some key metrics (e.g., forward/backward transfer) are not shown in the experiments.", "questions_to_authors": "Question A: The paper states that \"To our knowledge, only one method has been proposed to learn a sequence of mixed tasks.\". Why other task-incremental learning methods are not suitable for learning a sequence of mixed tasks? What are the special designs in your method enable learning a sequence of mixed tasks?\nQuestion B: Why the proposed method is built upon adapter? Is it an empirical choice?", "paper_topic_and_main_contributions": "This paper focus on the continually learning a mixed sequence. They propose two subnetwork discovery with soft-masking to allevaite catastrophic forgetting and importance computation to learn a mixed sequence. They also conduct extensive experiments on different NLP tasks and compare with many methods.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {"Missing_References": "[1] Ermis B, Zappella G, Wistuba M, et al. Memory efficient continual learning with transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 10629-10642.\n\n[2] Zheng J, Liang Z, Chen H, et al. Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 3602-3615.\n\n[3] Xia Y, Wang Q, Lyu Y, et al. Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 2291-2300."}}]

Author rebuttal:

Meta-review: The authors proposed a new methodology for continual learning, which introduces an importance-based gradient soft-masking on top of SupSup to enhance transfer capacity. The paper itself in general is well-structured and easy to understand. In terms of weaknesses, there was disagreement among the reviewers ranging from the presentation only having minor presentation flaws, to criticism about the applicability of the proposed method, concerns about the impact of reduced data on the baselines results that needs to be further discussed, and one reviewer arguing lack of novelty, that the methodology was hard to follow, and raising that results concerning improved transfer ability and replay-based methods are not shown in the experiments. Moreover, one reviewer suggested the authors may consider including results for settings with resource disparities. In response, the authors clarify their setting and its applicability, addressed the criticism about the impact of the size of the dataset on the baselines and also that about resource disparities, provided additional experiments including a replay-based method, and addressed concerns related to novelty, clarity and provided additional results to highlight the improved transfer capabilities of the proposed methodology. It is worth noting that the authors presented a comprehensive rebuttal including detailed responses to the reviewers concerns and questions and included substantial new experimental results. Moreover, the reviewers who raised concerns engaged in discussion with the authors. However, one of the reviewers still insisted that the proposed approach is incremental and of limited value to the NLP community.
Decision: Accept-Findings