Title: Enhancing Knowledge Transfer for Task Incremental Learning with Data-free Subnetwork
Venue: NeurIPS 2023
Accepted: yes
Keywords: data-free subnetwork

task-incremental learning

knowledge transfer

mask
Abstract: As there exist competitive subnetworks within a dense network in concert with Lottery Ticket Hypothesis, we introduce a novel neuron-wise task incremental learning method, namely Data-free Subnetworks (DSN), which attempts to enhance the elastic knowledge transfer across the tasks that sequentially arrive. Specifically, DSN primarily seeks to transfer knowledge to the new coming task from the learned tasks by selecting the affiliated weights of a small set of neurons to be activated, including the reused neurons from prior tasks via neuron-wise masks. And it also transfers possibly valuable knowledge to the earlier tasks via data-free replay. Especially, DSN inherently relieves the catastrophic forgetting and the unavailability of past data or possible privacy concerns. The comprehensive experiments conducted on four benchmark datasets demonstrate the effectiveness of the proposed DSN in the context of task-incremental learning by comparing it to several state-of-the-art baselines. In particular, DSN enables the knowledge transfer to the earlier tasks, which is often overlooked by prior efforts.

Enhancing Knowledge Transfer for Task Incremental Learning with Data-free Subnetwork

Abstract

As there exist competitive subnetworks within a dense network in concert with Lottery Ticket Hypothesis , we introduce a novel neuron-wise task incremental learning method, namely Data-free Subnetworks (DSN) , which attempts to enhance the elastic knowledge transfer across the tasks that sequentially arrive. Specifically, DSN primarily seeks to transfer knowledge to the new coming task from the learned tasks by selecting the affiliated weights of a small set of neurons to be activated, including the reused neurons from prior tasks via neuron-wise masks. And it also transfers possibly valuable knowledge to the earlier tasks via datafree replay. Especially, DSN inherently relieves the catastrophic forgetting and the unavailability of past data or possible privacy concerns. The comprehensive experiments conducted on four benchmark datasets demonstrate the effectiveness of the proposed DSN in the context of task-incremental learning by comparing it to several state-of-the-art baselines. In particular, DSN enables the knowledge transfer to the earlier tasks, which is often overlooked by prior efforts.

1 Introduction

Continual learning (CL), as one of the human-like lifelong learning or incremental learning paradigms, has received enormous attention in Artificial Intelligence (AI) community due to its capability to incrementally learn a sequence of tasks in a (deep) neural network and to keep accumulating knowledge throughout its lifetime [1]. According to the context of whether the task identity is provided during model training and inference, the majority of prior works categorize continual learning into three main practical problems by specific settings, including task-incremental learning, class-incremental learning, and task-free incremental learning. This study was prepared for Task Incremental Learning or TIL for short. Not limited to TIL, existing solutions for incremental learning are put forward to solve the catastrophic forgetting (CF) issue [2] that is a common phenomenon in CL scenarios. Usually, these approaches can be broadly categorized into three directions, i.e., regularization-based [3, 4, 5], rehearsal-based [6, 7, 8], and architecture-based [9, 10, 11, 12].

Motivations: Recently, several studies have demonstrated that deep neural networks are usually over-parameterized, whereby the redundant or unuseful weights can be pruned, allowing efficient computation and on-par or even better performance [13, 14, 15]. Likewise, researchers stated this phenomenon as Lottery Ticket Hypothesis (LTH) [16, 17] , i.e., a randomly-initialized neural network contains a subnetwork such that, when trained in isolation, can match the performance of the original network. With this in mind, more recent efforts turn to randomly initialize a large but sparse neural network and employ it to sequentially discover compact subnetworks for (task) incremental learning,

aiming at providing room to learn new tasks as well as enabling forward knowledge transfer from the prior tasks [18, 19, 20]. Specifically, they concentrate on weight-wise search using an adaptive binary mask that determines which weight should be activated ( winning tickets ). Nevertheless, we consider that it is not preferable to maintain such a binary mask for retrieving a subnetwork, especially in incremental learning. Since it will destroy the continuous structure of a layer-dependent subnetwork. Meanwhile, maintaining a binary mask whose size equals the number of parameters also results in significant resource consumption. In addition to this, knowledge transfer across different tasks is the core work of incremental learning. Existing solutions primarily seek to enable the knowledge transfer from earlier tasks to the new coming task (i.e., forward knowledge transfer ), along with considering the CF problem [21, 22, 23]. Like human cognitive processes [24], we debate whether the newly learned knowledge has the potential to help improve past tasks (i.e., backward knowledge transfer ). Intuitively, it is natural and practical that we can replay the old samples from earlier tasks, like rehearsal-based approaches, to trigger the ability to tackle old tasks much better. However, this is not applicable in the actual incremental learning scenario due to computational overhead and privacy concerns. With those in mind, we ask a much more ambitious question: Does LTH hold in the setting of TIL concerning elastic knowledge transfer when different tasks arrive sequentially?

Contributions: To answer this question, we propose a novel and practical approach named Data-free Subnetworks (DSN), enabling the elastic knowledge transfer across the tasks that sequentially arrive. Motivated by LTH, we assume that there exists a hypernetwork (Fig. 1 (a)) that contains a sequence of competitive "ticket" sub-networks, where each of them can perform well on the affiliated task with the knowledge transfer in mind. DSN mainly contains two components, i.e., neuron-wise mask and data-free memory reply. Specifically, we first randomly initialize a hypernetwork and incrementally learn the model parameters and task-specific masks, where each mask determines which neurons and their corresponding weights should be used for a new coming task (Fig. 1 (b)). To allow the forward knowledge transfer on a new task, we use the mask to adaptively select the neurons that have been used in the earlier tasks and make their corresponding weights unchangeable for the purpose of addressing the CF problem. To permit the backward knowledge transfer, we first measure the mask similarity scores and craft the impressions of the most similar task via data-free memory replay. In contrast to sample replay paradigms, we treat the subnetwork as a past experience and use it to recall past impressions, avoiding incrementally preserving the past actual samples. Then, we fine-tune the most similar task and make backward knowledge transfer possible(Fig. 1 (c)). We summarize our contributions as follows:

In concert with Lottery Ticket Hypothesis (LTH), we introduce a novel and practical taskincremental learning solution DSN, which aims to learn a compact subnetwork (winning ticket) for each task from the perspective of knowledge transfer, including forward and backward transfer.

We devise a neuron-wise mask mechanism to adaptive select neuron-affiliated weights to transfer the learned knowledge to the new task, where the used neuron-affiliated weights in the past tasks are frozen to eliminate the CF problem. Besides, our proposed data-free replay mechanism regards the trained subnetwork as a past experience and uses it to craft impressions regarding the past samples, which does not require holding any actual samples related to past tasks.

The comprehensive experiments conducted on four benchmark datasets demonstrate the effectiveness of our proposed DSN against the state-of-the-art baselines.

2 Related Work

Catastrophic Forgetting. Existing approaches to address the CF problem can be broadly categorized into three directions. (1) Regularization-based approaches that add a selective penalty in the loss function, which penalizes the variation of network parameters depending on its importance in performing previous tasks [3, 25, 4, 26, 27]. EWC operates the Fisher information matrix to estimate the importance weights [3]. Synaptic Intelligence (SI) provides an online approximation of parameter importance by assessing their contributions to the variations of total loss [28]. (2) Rehearsal-based approaches typically incrementally hold a few representative old training samples as a small memory buffer and replay it to retain the past knowledge when learning a new task [6, 7, 8, 29, 30, 31]. For example, GEM proposes to further guarantee an equal number of old training samples per class [32]. DGR presents a framework that replays generated data sampled from the old generative model while learning each generation task to inherit the previous knowledge [6]. Also, other

t

-1

t

1

…

…

t

-1

t

1

…

…

t

-1

t

1

…

…

(a) Dense Initialization

(b) Selection for Task

t

(c) Consolidation for Task 1

00000

00000

researchers [33, 34, 35] perform data-free generative replay to alleviate memory space or privacy concerns. For instance, [34] employs two teacher models to enhance the quality of generated samples. However, it requires an additional generative model, yielding additional trainable parameters. [35] can generate old samples merely through the main network. Besides, [33, 34, 35], which directly employ synthesized samples to fine-tune the model, have limitations in achieving forget-free performance and are susceptible to interference from other tasks. (3) Architecture-based approaches assign different model parameters to different tasks, freeze the parameters for old tasks by masking them out when training the new task, and expand the network when necessary [10, 11, 12, 8, 19]. DEN [36] can dynamically adjust its network capacity as it trains on a sequence of tasks to learn a compact overlapping knowledge-sharing structure among tasks. WSN aims to explore the subnetworks sequentially for a sequence of tasks from a sparse neural network [19]. It also maintains a binary mask whose size equals the number of parameters, resulting in significant resource consumption. We argue the architecture-based approaches provide us with the possibility to explore better subnetworks while they either confront the redundant neurons (weights) or shallow knowledge transfer.

Knowledge Transfer. Recently, researchers and practitioners have argued that the ability to learn from a sequence of tasks is in essence to complement and reinforce each other through knowledge transfer. Several studies have proposed to improve task performance by maximizing shareable knowledge [37, 38, 39, 40]. For instance, Bayes model [41, 42] and regression method [43] concentrate on transferring the learned knowledge to facilitate learning in the new domains or new tasks. BNS [12] presents a Task-CL algorithm that can adaptively determine the network structure for each new task and selectively transfer previously learned knowledge. However, it employs massive past samples to address the negative knowledge transfer concern. HAT [44] and WSN [19] selectively reuse the optimized parameters via a binary masking mechanism, enabling significant forward knowledge transfer. However, backward knowledge transfer is not considered. [45] focuses on a few-shot scene, which is accomplished by jointly learning model weights and adaptive non-binary soft masks, with the major subnetwork minimizing catastrophic forgetting and the minor subnetwork preventing overfitting. Also, it fails to take backward knowledge transfer into consideration. [46] proposes two distillation-based objectives for class incremental learning. This method can utilize the feature space structure of the previous model to preserve the representation of previous classes. Besides, a controlled transfer is introduced to approximate and condition the current model on the semantic similarities between incoming and prior classes, maximizing positive forward transfer and minimizing negative backward transfer. However, it can not achieve a positive backward transfer. [38] and [47] preserve previous data as a replay buffer for retraining old tasks, which raises concerns regarding data privacy and memory overhead. [38] utilizes a CL-plugin to explicitly identify and transfer useful features/knowledge from previous tasks to new ones, designed specifically for LLMs. [48] characterizes task correlations, identifies positively correlated old tasks, and selectively modifies the learned model of the old tasks when learning new tasks. In sum, the above solutions concentrate on either addressing severe negative transfer using pruning methods or using additional data to help knowledge consolidation. In contrast, our DSN attempts to select subnetworks from a sparse network for a sequence of tasks while using a data-free mechanism to enable the possibly positive knowledge transfer to the old tasks.

3 The Proposed Method

3.1 Problem Statement

Without loss of generality, a classical TIL scenario, typically in a supervised learning setting (e.g., classification), deals with a set D = { D 1 , D 2 , · · · , D T } of T tasks that come in turn to the learner or task network (e.g., a deep neural network). Let D t = { ( x i t , y i t ) } n t i =1 denote the dataset of task t , containing n t pairs of raw instances and independent label space. In our context, assume that we have a neural network, named hypernetwork H . H ( · , θ ( n ) ) is composed of layered neurons n where learnable weights θ bridge them together. TIL follows the common continual learning paradigm and attempts to sequentially learn a set of task T with the following objective for each task t :

where L ( · , · ) is a loss function such as cross-entropy loss. It is important to note that D t for task t is only accessible during task t learning, and any instances in D t cannot be accessed while learning future tasks. Furthermore, assume that task identity is given in both the training and inference processes in the TIL scenario. In addition, we follow most of the previous studies and employ multi-head settings for TIL, with each task having its own single but non-overlapping head.

The Lottery Ticket Hypothesis (LTH) hypothesizes the existence of competitive subnetworks in a randomly initialized dense neural network [16], which motivates us to ingeniously devise an incremental learner that employs an over-parameterized deep neural network to allow more room for future task learning. Therefore, our goal is to discover an optimal subnetwork along with a neuron-wise mask m t ∈ { 0 , 1 } | n | for each task t such that | m ∗ t | ≪| n | , summarized as follows:

3.2 Neuron-wise Mask

To obtain an optimal subnetwork from the hypernetwork H for any new coming task, we are motivated by HAT [44] and inhibitory synapse [49] and devise a neuron-wise differentiable mask mechanism to search a small set of neurons with the objective Eq.(2). Specifically, for each layer in the hypernetwork H , let each layer associated with a learnable embedding e l t ( l ∈ { 1 , 2 , · · · , L } ) before training a coming task t , where L is the number of layers. To determine if a neuron in n should be used for task t , we adopt the binary mask m t to produce the architecture of the subnetwork. However, neurons typically have only two states, i.e., activated or not activated . To this end, we use the following function to bridge the gap between the layer embedding e l t and layer mask m l t ∈ m t :

where σ is the sigmoid function. γ is the scaling factor to control the sharpness of the function that can make the binarized progressively [44, 50]. We note that the optimal mask m ∗ t can be received by jointly optimizing the model parameters θ of task t , as the task learning is conditioned on mask m t . For instance, the forward propagation of hypernetwork H during the training of task t can be expressed as follows:

where h l t denotes the output of neruons in the layer l . Correspondingly, the backward propagation can be summarized as follows,

where θ lij ∈ θ refers to a parameter (i.e., weight) between the neuron i of l -th layer and the neuron j of ( l -1) -th layer. m l,i t denotes the mask value of i -th neuron. max( m l,i t , m l -1 ,j t ) indicates that the parameter will not be updated if neither the neuron i of l -th layer nor the neuron j of l -1 -th

layer is activated for task t . Otherwise, the parameter will be updated. However, the above backward propagation inevitably results in the catastrophic forgetting issue since the weights trained in the previous tasks could be changed. Hence, we regard the used neurons in the previous tasks as the synapses that only take the role of sending messages between different layers. To this end, we devise a cumulative mask to maintain the signal that all previously learned parameters should be frozen. The cumulative mask m ≤ t for any task t can be recursively obtained as follows,

Note that we set m ≤ 0 = 0 , indicating all neurons can be used before training task 1. Thus, the Eq.(5) can be revised as follows,

It is worth noting that mask operation may also pose capacity problems. To this end, we add a regularization term L r to the objective L to hold more room for future tasks as follows,

where n l denotes the neuron number of l -th layer in hypernetwork H and η is a hyperparameter that controls the capacity preference for the current task t . Note that we provide more details of the hyperparameter setting in Appendix A.

3.3 Data-free Replay

To enhance the knowledge transfer from the current task to the earlier tasks, data-free replay recalls network experiences to address the unavailability of old samples or possible data privacy concerns. Recent zero-shot learning solutions [51, 35] provide us with a new perspective. That is, we craft past knowledge through the subnetwork rather than seeking past samples.

Output Space Modeling. Let o t denote the output space (i.e., Softmax space for classification problems) of task t over the subnetwork H ( · , θ ( n ⊙ m t )) . Recent studies [51, 35] reveal that any output representation can be sampled from a Dirichlet distribution as its ingredients fall in the range of [0,1] and their sum is 1. Specifically, the distribution to represent the (Softmax) outputs o c t of c -th class can be modeled as Dir ( C t , β × α c ) , where c ∈ { 1 , 2 , · · · , C t } is the class index regarding task t , α c ∈ R C t is the concentrate vector to model c -th class, β is a scaling parameter [52, 51], and any real value in α c is greater than 0. It should be noted that there could exist interactive correlations between different classes in a single task since the samples related to different classes may be similar. To mitigate the risk of ignoring the inherent distribution of different classes of samples, we are inspired by [51] and additionally preserve a class similarity matrix M t after training each task t to describe the correlation between different classes. As M t of task t is a class-wise table, it will not bring the significant capacity issue. We detail the motivation of the similarity matrix in Appendix B.

Replay with Impression Craft. Due to the unavailability of old task samples, like the human memory mechanism, we attempt to synthesize the input of H ( · , θ ( n ⊙ m t )) , which can be denoted as an impression regarding a raw sample in an earlier task. To be specific, we synthesize an impression set I t based on the Dirichlet sampling. For any synthesized sample ˆ x c,i t corresponding to c -th class in task t , we initialize ˆ x c,i t to random noise, e.g., by sampling from a uniform distribution. And we optimize it with the following objective over a sampled outputs ˆ o c t and the subnetwork H ( · , θ ( n ⊙ m t )) , i.e.,

where τ is a temperature value [53] used in the output (Softmax) head. In this manner, we can successively produce impressions for each class. Intuitively, we can produce the impressions of different classes equally. However, the subnetwork may show different performances in different classes. That is, we need to consider the emergence of hard classes. Thus, we generate a biased number of impressions of different classes based on the accuracy performance. For each task learning, we report the error rate of each class and normalize them as the distribution of the sampling rate.

Algorithm 1: Incremental learning with DSN.

Input: { D t } T t =1 , a hypernetwork H with neurons n and associated parameters θ , a cumulative mask m ≤ 0 . Randomly initialize θ and set m ≤ 0 = 0 ; for t = 1 , 2 , . . . , T do Initialize layer embeddings e t = { e l t } L l =1 ; Use e t = { e l t } L l =1 to obtain the mask m t according to Eq.(3); Train the hypernetwork H with θ ( n ⊙ m t ) on D t according to Eq.(2) and Eq.(8); Obtain the optimal layer embeddings e t and layer masks m ∗ t ; Update the cumulative mask m ≤ t via Eq.(6); if t >1 then Measure the task similarity scores S t via Eq.(10); Obtain the most similar task argmax ( S t ) with number of class C argmax ( S t ) ; Compute the replay number of each class { B i } C argmax ( St ) 1 according to the class-specific error rate; Initialize impression set I argmax ( S t ) ←{} ; for c = 1 : C argmax ( S t ) do Set the concentration parameter α c = M c argmax ( S t ) ; for b = B 1 , B 2 , · · · , B C argmax ( St ) do for i = 1 : b do Sample ˆ o c argmax ( S t ) ∼ Dir ( C argmax ( S t ) , β b × α c ) ; Initialize ˆ x c,i argmax ( S t ) to random noise and craft ˆ x c,i argmax ( S t ) via Eq.(9); I argmax ( S t ) ←I argmax ( S t ) ∪ ˆ x c,i argmax ( S t ) ; Merge mask for task argmax ( S t ) : ˆ m argmax ( S t ) = max( m ∗ argmax ( S t ) , m ∗ t ) ; Fine-tune the task argmax ( S t ) based on I argmax ( S t ) and ˆ m argmax ( S t ) ; if accuracy performance of task argmax ( S t ) is higher than before then update the mask m ∗ argmax ( S t ) ;

3.4 Knowledge Transfer-enhanced Incremental Learning with DSN

Incremental Learning ( t ≥ 1 ). For the first task, i.e., t = 1 , we randomly initialize the parameters θ in the hypernetwork H . For each task t including the first task, we start with randomly initializing a task-specific classifier (e.g., a fully-connected layer with a Softmax function) and the corresponding layer embeddings e t . Then, we attempt to jointly learn the model parameters in the hypernetwork H and task-specific masks of subnetworks regarding each task, where each task-specific mask determines which neurons will be used in the current task. Specifically, given a task t , we optimize the parameter θ and its mask m t with the objectives of Eq.(2) and Eq.(8) where the update of parameters will follow the rule of Eq.(7) for the purpose of addressing the CF problem. After the training, we likewise build the class similarity matrix M t and update the cumulative mask m ≤ t .

Knowledge Transfer ( t > 1 ). To consolidate the possibly useful knowledge behind the newly learned task to the earlier tasks, we first measure the task similarity and then employ our date-free replay to transfer the knowledge to the most similar task. Notably, we could choose multiple similar tasks to transfer, but this would take more transferring time than it is worth due to the cost of memory replay. Since the unavailability of any real samples in the earlier tasks, it is impossible to use the naive solution for task similarity measurement, i.e., estimating the similarity between the instance distributions of two tasks. Fortunately, we can use the masks of tasks to measure the task similarities. For a current task t , we can compute the similarity scores S t as follows,

where cosine is the cosine distance. As such, we can obtain the most similar task that owns the largest similarity score, i.e., argmax ( S t ) . After that, we can use the data-free memory replay to produce impression crafts as the input samples of the most similar task by optimizing the objective of Eq.(9). It is noted that we only use the impressions to determine whether we need to adjust the subnetwork architecture of the most similar task. Meanwhile, any parameter update in the hypernetwork will cause interference with the current task as well as all the previous tasks. Instead, we only allow the parameter update in the task-specific classifier (head) while freezing any parameters in the subnetwork.

Specifically, we first merge the optimal mask of current task t to the most similar task argmax ( S t ) , i.e., ˆ m argmax ( S t ) = max( m ∗ argmax ( S t ) , m ∗ t ) . Then, we fine-tune the task argmax ( S t ) based on ˆ m argmax ( S t ) and make knowledge transfer possible. We will update the architecture of the subnetwork regarding task argmax ( S t ) if it performs better. Notably, the cumulative mask m ≤ t does not need to change due to the fact of ˆ m argmax ( S t ) ⊆ m ≤ t . The complete workflow is summarized in Algorithm 1.

4 Experiments

Datasets. We employ four benchmark datasets for TIL problem as follows: Permuted MNIST ( PMNIST ) [3], Rotated MNIST ( RMNIST ) [21], Incremental CIFAR-100 [54, 55], and TinyImageNet [19]. PMNIST encompasses 10 variations of MNIST [56], wherein each task is transformed by a fixed permutation of pixels. RMNIST also comprises ten versions of MNIST, with each task rotated by a specific angle between 0 and 360 degrees. The original CIFAR-100 was divided into 20 tasks, each containing 5 different categories. TinyImageNet constitutes a variant of ImageNet [57], we construct twenty 10-way classification tasks for consistency in our experiments.

Baselines. We compare our DSN with the following representative baselines: SGD [58] utilizes a feature extractor without any parameter update and a learnable classifier to solve a series of tasks incrementally. EWC [3] purposefully uses Fisher information as a simple and effective standard baseline to alleviate CF. IMM [25] aims to penalize parameter modifications, yielding two variants, i.e., Mean-IMM and Mode-IMM. PGN [9] attempts to expand the task network by incorporating a fixed number of neurons. DEN [36] dynamically decides the number of new neurons via selective retraining and network split. RCL [21] is a reinforcement learning method that controls the scale of newly added neurons while rendering weights relative to the used neurons unchangeable. HAT [44] leverages a masking mechanism to prevent the optimization of old neurons. SupSup [22] solves the CF problem when sequentially learning tasks by discovering super masks (sub-networks). WSN [19] tries to learn the model parameters sequentially and select the optimal sub-network for each task.

Implementations and Evaluation Metrics. In accordance with the task-incremental continual learning framework, all methods used in the experiments employ a multi-head configuration. For two variants of MNIST, we adopt the experimental setup outlined in [44] in which we use a two-layer MLP, a multi-head classifier, and begin with 2000-2000-10 neurons for the first task. For the other datasets, we refer to previous works [19, 44] and use the modified model with three convolutional layers and two fully-connected layers. We follow previous studies [3, 12, 19] and evaluate all methods on three metrics: ACC , BWT , and Trans . ACC is a common CL metric that reports the average accuracy of all tasks validated on their respective test sets after training on all tasks. BWT measures how the model performance changes for the old tasks once the model is trained on the new task. Trans measures the proficiency to transfer knowledge from previously learned tasks to newer ones, indicating the usefulness of the acquired knowledge in facilitating new learning tasks. Notice that more details of the implementation setup are specified in Appendix C. The source codes are available at https://github.com/shanxiaojun/DSN .

Overall Performance. To examine the influence of task mixture, we shuffled the tasks with five different seeds, resulting in five lists with different task orders. The averaged results over five runs are presented in Table 1, where the deviation results are reported in Appendix D. We have the following observations: First, not all CL solutions outperform the naive SGD method, even mean-IMM performs slightly worse on PMNIST than SGD. We consider that this observation may be due to the sequential

1

2

3

4

5

6

7

8

9

10

Increasing No. of T asks

94

95

96

97

98

Average Acc(%)

EWC

RCL

HAT

WSN

DSN

(a) RMNIST

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Increasing No. of T asks

57

60

63

66

69

72

75

Average Acc(%)

EWC

RCL

HAT

WSN

DSN

(b) CIFAR100

1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Increasing No. of T asks

33

36

39

42

45

48

51

54

Average Acc(%)

EWC

RCL

HAT

WSN

DSN

(c) TinyImageNet

weight penalty during knowledge transfer, which could significantly hinder knowledge transfer between old and new tasks. Second, we find that CL methods with network expandability such as PGN and RCL do not perform as well as EWC. This indicates that merely iterative network expansion, without network pruning or neuron selection, has the potential of incurring an over-parameterized risk [13, 59]. On the other hand, HAT, SupSup, and WSN adhere to the merit of network pruning and mainly seek to produce a sub-network by partially selecting some neurons (weights), resulting in higher accuracy performance. However, they can only facilitate forward knowledge transfer, yielding only narrow knowledge consolidation. In contrast, our DSN outperforms all the baselines on all metrics such as achieving the 3.20% improvements on CIFAR-100 regarding ACC. In addition, to show the superiority of DSN in incremental learning from the fine-grained aspect, Fig. 2 presents the averaged results as the number of tasks increases, we can clearly observe that DSN outperforms other baselines during the entire incremental learning process.

Forward Knowledge Transfer. The results of Trans in Table 1 demonstrate that DSN is substantially better overall due to the flexible neuron selection. Specifically, DSN performed on PMNIST and RMNIST shows only a slight degradation over the single-task learning scenario. This finding indicates that we can use more compact sub-networks to take on multiple tasks incrementally, which is more practical in future computing contexts, such as edge computing and low resource constraints.

















































(a) RMNIST



















          



























(b) CIFAR100



















          































(c) TinyImageNet

Backward Knowledge Transfer. The experimental results on BWT indicate that SGD suffers from severe knowledge forgetting or negative knowledge transfer issues as it does not take any effort to avoid or alleviate the catastrophic forgetting problem in task incremental learning. Other traditional CL approaches, such as EWC and IMM, choose to confine the shift of model parameters to maintain the old task's performance while learning new tasks sequentially. However, the penalty of parameters still affects the ability of the model to recall old tasks, and these methods also confront severe catastrophic forgetting problems. In comparison with forget-free methods like RCL, HAT, and WSN (BWT=0), DSN can even promote the previous task performance on four datasets (BWT>0). There are two reasons for this observation. First, existing forgetting-free solutions concentrate on simply freezing the neurons or weights used in the previous tasks while making the architecture of the task network regarding each old task unchangeable throughout the entire incremental learning process. As a result, it offers us only barely forgetting or no-forgetting gains. Second, our DSN owning data-free

replay enables the possibility of actual backward transfer, i.e., transferring new knowledge to previous tasks. Fig. 3 shows the performance of the first task throughout the incremental learning. It is obvious that HAT and WSN can only maintain the model performance on the first task while they cannot promote the performance of the first task when incrementally receiving new tasks. This suggests that newly acquired knowledge can help address inadequate experience or knowledge of past tasks.

FC1                                      FC2

0

20

40

60

80

100

task1

task3

task4

task5

task6

task8

task9

task10

(a) PMNIST

FC1                                      FC2

0

20

40

60

80

100

task1

task3

task4

task5

task6

task8

task9

task10

(b) RMNIST

FC1                                      FC2

0

20

40

60

80

100

task1

task3

task4

task5

task6

task8

task9

task10

task11

task12

task13

task14

task15

task16

task17

task18

task19

task20

(c) TinyImageNet

Capacity issues. Fig. 4 shows the capacity monitoring as the sequence of tasks emerges, where we only show the first two fully-connected layers of TinyImageNet for better visualization. From a macro perspective, we can find that the total usage (dashed line) of neurons increases incrementally as more task comes. But we discover that the used neurons for each task are gradually increased, which suggests that DSN prefers to reuse more neurons from earlier tasks when a new task arrives. Especially, DSN reuses more neurons from the past on RMNIST as the total usage increases slowly. The rationale behind this is that although different tasks have different angles of images, they do not change the goal of image classification and those that arrive in sequence are implicitly similar. Furthermore, we investigate the scale impact of our defined hypernetwork. Specifically, we vary the neuron number of each layer in the hypernetwork with different initial learning rates. As shown in Fig. 5, it is evident to observe that a larger over-parameterized hypernetwork enables higher accuracy performance. Interestingly, we find that a smaller learning rate brings us higher gains. We conjecture this phenomenon indicates that over-fitting problem could degrade the model performance.

Learning Rate

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Number of Neurons

500

1000

1500

2000

2500

3000

3500

4000

Accuracy(%)

92

93

94

95

96

97

98

99

100

(a) RMNIST

Learning Rate

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Number of Neurons

500

1000

1500

2000

2500

3000

3500

4000

Accuracy(%)

40

45

50

55

60

65

70

75

80

85

(b) CIFAR-100

Learning Rate

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Number of Neurons

500

1000

1500

2000

2500

3000

3500

4000

Accuracy(%)

10

15

20

25

30

35

40

45

50

(c) TinyImageNet

Efficiency Issue. We investigate the efficiency concern regarding our DSN and the representative baselines. First, Table 2 reports the runtime of all approaches including DSN. We first observe that solutions with network expandability take significantly more time, especially for RCL with reinforcement learning. The methods with parameter penalty spend less time while subnetwork selection methods such as WSN cost slightly higher. As for DSN, it costs more time but less than network expandability-based methods. There are two reasons: 1) recent mask mechanisms in recent subnetwork selection methods such as WSN are weight-wise, where the constructed binary masks have the same shape as the weights. As such, we can couple with them in a parallel manner. 2) DSN needs to craft the impressions regarding the old tasks, yielding a higher time cost. Nevertheless, Table 3 shows that DSN and HAT need less trainable masks than WSN due to the neuron-wise masks.

Sensitivity Analysis. We evaluate the impact of η that aims to hold more room for future tasks. As illustrated in Fig. 6, we can find that a larger η will leave more room but bring a slight performance drop. η = 0 indicates that we do not penalize the number of activated neurons for each task. We can find that the capacity is full while the accuracy performance is the worst. We note that we evaluated the impacts of other key hyperparameters. However, due to space limitations, more details are described in Appendix D.







































(a) PMNIST





































(b) TinyImageNet

5 Conclusions

Motivated by Lottery Ticket Hypothesis, we introduced a Data-free Subnetworks (DSN) approach for task incremental learning, aiming at enhancing knowledge transfer across sequentially arrived tasks. DSN consists of two components, i.e., neuron-wise mask and data-free replay. The former aims to find an optimal subnetwork using task-specific masks for a new arriving task while the latter attempt to craft the impressions for transferring the knowledge to the past tasks. DSN allows for more flexible knowledge transfer between old and new tasks, whereas backward knowledge transfer was usually ignored before. The experimental results on four datasets show the superiority of our proposed DSN. As part of our future work, we plan to investigate how to devise a knowledge-inspired mask mechanism to enhance knowledge transfer as more tasks arrive sequentially.

References

M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(07):3366-3385, 2022.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems , volume 33, pages 3647-3658, 2020.

Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. In International Conference on Learning Representations , pages 1-18, 2021.

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2994-3003, 2017.

Lukasz Korycki and Bartosz Krawczyk. Class-incremental experience replay for continual learning under concept drift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3649-3658, 2021.

Haiyan Yin, Ping Li, et al. Mitigating forgetting in online continual learning with neuron calibration. In Advances in Neural Information Processing Systems , pages 10260-10272, 2021.

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016.

Shuaicheng Niu, Jiaxiang Wu, Guanghui Xu, Yifan Zhang, Yong Guo, Peilin Zhao, Peng Wang, and Mingkui Tan. Adaxpert: Adapting neural architecture for growing data. In International Conference on Machine Learning , pages 8184-8194. PMLR, 2021.

Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for class-incremental learning. In Advances in Neural Information Processing Systems , volume 34, pages 478-3490, 2021.

Qi Qin, Wenpeng Hu, Han Peng, Dongyan Zhao, and Bing Liu. Bns: Building network structures dynamically for continual learning. In Advances in Neural Information Processing Systems , volume 34, pages 20608-20620, 2021.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems , pages 1135-1143, 2015.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations , 2017.

Qiang Gao, Zhipeng Luo, Diego Klabjan, and Fengli Zhang. Efficient architecture search for continual learning. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning , pages 6682-6691. PMLR, 2020.

Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference on Learning Representations , 2021.

Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning , pages 10734-10750. PMLR, 2022.

Jian Jiang and Oya Celiktutan. Neural weight search for scalable task incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1390-1399, 2023.

Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages 907-916, 2018.

Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems , volume 33, pages 15173-15184, 2020.

Pratik Mazumder, Pravendra Singh, and Piyush Rai. Few-shot lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 2337-2345, 2021.

Almut Hupbach, Rebecca Gomez, Oliver Hardt, and Lynn Nadel. Reconsolidation of episodic memories: A subtle reminder triggers integration of new information. Learning & memory , 14(1-2):47-53, 2007.

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems , 30, 2017.

Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-based continual learning with adaptive regularization. In Advances in Neural Information Processing Systems , volume 32, pages 4392-4402, 2019.

Jie Zhang, Junting Zhang, Shalini Ghosh, Dawei Li, Jingwen Zhu, Heming Zhang, and Yalin Wang. Regularize, expand and compress: Nonexpansive continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 854-862, 2020.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning , pages 3987-3995. PMLR, 2017.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , pages 350-360, 2019.

Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing HONG, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations , pages 1-25, 2022.

Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems , volume 35, pages 11602-11614, 2022.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 6470-6479, 2017.

Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class-incremental learning via entropy-regularized data-free replay. In European Conference on Computer Vision , pages 146-162. Springer, 2022.

Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Dual-teacher class-incremental learning with data-free generative replay. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3543-3552, 2021.

Mozhgan PourKeshavarzi, Guoying Zhao, and Mohammad Sabokrou. Looking back on learned experiences for class/task incremental learning. In International Conference on Learning Representations , 2022.

Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations , pages 1-11, 2018.

Zixuan Ke, Bing Liu, Hao Wang, and Lei Shu. Continual learning with knowledge transfer for sentiment classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 683-698. Springer, 2020.

Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. In Advances in Neural Information Processing Systems , volume 34, pages 22443-22456, 2021.

Hao Wang, Bing Liu, Shuai Wang, Nianzu Ma, and Yan Yang. Forward and backward knowledge transfer for sentiment classification. In Asian Conference on Machine Learning , pages 457-472. PMLR, 2019.

Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Avoiding forgetting and allowing forward transfer in continual learning via sparse networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 85-101. Springer, 2022.

Zhiyuan Chen, Nianzu Ma, and Bing Liu. Lifelong learning for sentiment classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 750-756, 2015.

Hao Wang, Shuai Wang, Sahisnu Mazumder, Bing Liu, Yan Yang, and Tianrui Li. Bayesenhanced lifelong attention networks for sentiment classification. In Proceedings of the 28th International Conference on Computational Linguistics , pages 580-591, 2020.

Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In International conference on machine learning , pages 507-515. PMLR, 2013.

Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning , pages 4548-4557. PMLR, 2018.

Haeyong Kang, Jaehong Yoon, Sultan Rizky Hikmawan Madjid, Sung Ju Hwang, and Chang D. Yoo. On the soft-subnetwork for few-shot class incremental learning. In The Eleventh International Conference on Learning Representations , pages 1-23, 2023.

Arjun Ashok, KJ Joseph, and Vineeth N Balasubramanian. Class-incremental learning with cross-space clustering and controlled transfer. In European Conference on Computer Vision , pages 105-122, 2022.

Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of a mixed sequence of similar and dissimilar tasks. In Advances in Neural Information Processing Systems , volume 33, pages 18493-18504, 2020.

Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. In Advances in Neural Information Processing Systems , volume 35, pages 16165-16177, 2022.

Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics , 5:115-133, 1943.

Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3014-3023, 2021.

Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International Conference on Machine Learning , pages 4743-4751. PMLR, 2019.

Jiayu Lin. On the dirichlet distribution. Department of Mathematics and Statistics, Queens University , pages 10-11, 2016.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Tront , 2009.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

A Kirzhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , volume 25, pages 1097-1105, 2012.

Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 , 2013.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , pages 1-42, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Data-free Subnetworks (DSN) approach for task incremental learning. With a random initialized neural network, DSN learns a task-specific neuron-wise mask to find an optimal subnetwork for a new arriving task, and performs data-free replay for transferring the knowledge to the past tasks. DSN achieves superior performance on four benchmarks.\n\nSTRENGTHS:\n- The paper's approach of designing a task-incremental learning method motivated by the Lottery Ticket Hypothesis is reasonable and novel.\n- The emphasis on backward knowledge transfer, often overlooked in previous incremental learning methods, is commendable.\n- The proposed method outperforms many State-Of-The-Art methods across four benchmarks under multiple evaluation metrics, demonstrating its effectiveness.\n\nWEAKNESSES:\n- It is confusing how backward knowledge transfer is achieved through data-free replay. By definition, backward knowledge transfer refers to the process in which new knowledge is transferred to old tasks to potentially improve their performance. However, it seems that the replayed samples (impression crafts) only represent knowledge of old tasks and may not incorporate new knowledge for improving performance in those tasks. \n\n&nbsp;\n- The related work section can be further improved by providing detailed comparisons with 1) architecture-based methods (such as [39][41]) that also learn masks to designate different subnetworks for different tasks; 2) incremental learning methods [40][1*] that also consider backward knowledge transfer; and 3) [2*] that is also motivated by Lottery Ticket Hypothesis.\nRef. \n[1*] Class-incremental learning with cross-space clustering and controlled transfer, ECCV 2022;\n[2*] On the Soft-subnetwork for few-shot class incremental learning, ICLR 2023\n\n&nbsp;\n- The method details and formulas need clarification. For example, Eq. (9) lacks an explanation of how to calculate L_{IC}, the optimization objective does not include the variable x, and the discussion on the validity of the Dirichlet distributions representing the task output is less convincing.\n\n&nbsp;\n- Knowledge transfer is only performed between current task and the most similar previous task. Considering more similar tasks and evaluating the associated cost could potentially lead to improvements. \n\n&nbsp;\n- Some minors: 1) Case errors, such as “Without loss of generality, A classical TIL scenario, ...”; 2) Brackets missing on line 185; 3) Fig. 6 is not referred to in the manuscript.\n\n&nbsp;\n\nQUESTIONS:\n- Which direction(s) does the proposed method belong to, regularization-based, rehearsal-based, or architecture-based? \n- What does the f represent in Eq. (1) and (2)?\n- What is the difference between a neuron-wise mask and a weight-wise mask?\n- What does “bridge the gap between the layer embedding and layer mask” mean?\n- Could you provide more details about the structure of the over-parameterized deep neural network used in this method, such as the number of layers and neurons in each layer?\n- In Algorithm 1, what do B1, ..., B_{argmax(S_t)} represent?\n\nLIMITATIONS:\nYes. The authors have discussed the limitations regarding the efficiency and capacity issue of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors explore task-incremental learning through the lens of the Lottery Ticket Hypothesis (LTH). They contend, primarily from an LTH perspective, that a distinct task necessitates merely a sparse collection of neurons, hence using only a compact sub-network for its operation. Subsequently, every new task can integrate itself within these compact sub-networks, thereby enabling the overall network to learn new tasks sequentially.\n\nThe methodology employed by the authors is intriguing. Initially, a hypernetwork is set up at random, followed by the gradual learning of the model's parameters in tandem with masks that are specific to each task. These masks play a crucial role, determining which neurons and corresponding weights should be utilized or rendered static for the impending task.\n\nNotably, the authors strive to enhance previously acquired knowledge via a process they term 'backward knowledge transfer'. This involves determining the most similar task previously encountered, creating a data impression of that task, and subsequently refining the mask/weights of the preceding tasks. This concept stands as a key contribution of this study.\n\nSTRENGTHS:\n1. The organization of the paper is commendable, effectively guiding readers through the progression of ideas, hypotheses, methodology, results, and conclusions. This well-structured approach allows for a clear understanding of the study and its outcomes.\n2. The concept of data-free backward transfer presented in this work is notably intriguing. This aspect, overlooked by existing methods, has been adeptly integrated into the authors' framework, contributing to its uniqueness and potential impact in the field.\n\n\n3. The authors have executed a compelling array of experiments in their work. Their holistic analysis of their method in relation to Forward Knowledge Transfer, Backward Knowledge Transfer, capacity issues, efficiency, and sensitivity analysis, is thoroughly detailed and insightful. This comprehensive evaluation underscores the robustness of their methodology and enhances the overall credibility of their findings.\n\nWEAKNESSES:\n1. The present work bears a significant similarity to the research conducted by Kang et al. [1], which also advocates for a similar framework leveraging the Lottery Ticket Hypothesis (LTH) and employs binary masks to learn task-oriented sub-networks. It would be advantageous for the authors to highlight their unique additions, modifications, or contributions vis-à-vis this recent analogous work, as this would underscore the true value and novelty of the current work.\n2. For each new subtask, the presented method learns a mask embedding corresponding to the task. The current manuscript would have been more complete if the authors had discussed the space complexity of the masked embeddings learned with respect to the tasks, comparing it to [1].\n\nMinor:\n1. A more concrete review/literature survey of data-free replay should have been performed, For example, somewhat related works like [2] and [3] have not been discussed.  Nonetheless, I encourage authors to explore more related works have utilized data-free replay in various other contexts.\n\n[1] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, Chang D. Yoo. Forget-free Continual Learning with Winning Subnetworks, In ICML, 2022.\n\n[2] Liu, Huan, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. \"Few-shot class-incremental learning via entropy-regularized data-free replay.\" In ECCV, 2022.\n\n[3] Choi, Yoojin, Mostafa El-Khamy, and Jungwon Lee. \"Dual-teacher class-incremental learning with data-free generative replay.\" In CVPR Workshops, 2021.\n\nQUESTIONS:\n1. In connection with Objective 8, the authors introduce a capacity constraint to the current task, which may impact the task's performance due to the limitation it imposes on the neural network's capacity. I would be keen to hear an explanation from the authors on the extent to which this constraint influences the efficacy of the current task.  \n\n2. Considering the methodology used to measure the accuracy of task $argmax(S_t)$, I notice that the authors operate under the assumption of not having access to the previous task's data. Given this constraint, how do the authors determine the accuracy of the task post fine-tuning $argmax(S_t)$ and subsequently compare it with its preceding accuracy? Could they elaborate on their approach to evaluating accuracy under these specific conditions and how they ensure the validity of their comparisons?\n\nLIMITATIONS:\nIn relation to the proposed method, I understand that tasks are stored as parts of a subnetwork. This is an intriguing concept, yet I find myself questioning its scalability when considering a large number of diverse tasks. Given that the total upper capacity of the neural network is fixed, I am uncertain if the proposed method would accommodate scaling up effectively. It seems to me that a broad and varied set of tasks may exceed the inherent capacity limits of the neural network, potentially leading to capacity exhaustion or compromise in task performance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for task incremental learning that uses a subnetwork for each task. The method is based on neuron masking obtained by learnable embedding. The method also finds the most similar task from the past tasks and allows for backward transfer. The method is tested on four benchmarks: PMNIST, RMNIST, CIFAR-100, and TinyImageNet. The results show an improvement in performance over other methods.\n\nSTRENGTHS:\n-   The proposed method achieves a better performance than other methods on the three studied benchmarks.\n-   Up to my knowledge, few papers addressed positive backward transfer in CL. It is nice to see more work addressing this aspect.\n-   Analysis of the reusability of the neurons over tasks is provided.\n-   Efficiency analysis is considered.\n-   Different metrics are evaluated Accuracy, BWT, and Trans, which assess different requirements in CL.\n\nWEAKNESSES:\n-  Backward transfer to one only past task is not that convincing. Multiple previous tasks could benefit from the new knowledge if they are similar enough. This is my main concern. Since backward transfer is one of the main contributions, I expect more investigation into that point.\n-  Despite the fact that most of the paper is easy to follow, Section 3.3 lacks an overview at the beginning that can link the following information to how it contributes to the method. It becomes more clear after reading Section 3.4.\n\n [suggestion] Maybe the authors can consider reordering these two sections or clarify a bit more in Sec 3.3\n-  The paragraph explaining the backward transfer is very brief (Line 125-221), and some sentences are not clear. More elaboration could be useful. Algorithm 1 is helpful, though.\n-  An analysis that assesses the validity of the proposed similarity measure (even on a toy experiment) could be helpful.\n-  Minor: text in figures 3 and 4 is not readable.\n-  Minor: typo in the last line in Algo.1 “hyep”\n-  Some closely related works are missing. Not necessarily to empirically compare with them as you provide a comparison with one recent related method (WSN), but at least to have a discussion in the related work section. Examples of the missing works are below.\n\no\tGurbuz, Mustafa B., and Constantine Dovrolis. \"NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks.\" International Conference on Machine Learning. PMLR, 2022.\n\no\tSokar, Ghada, Decebal Constantin Mocanu, and Mykola Pechenizkiy. \"Spacenet: Make free space for continual learning.\" Neurocomputing 439 (2021): 1-11.\n\no\tSokar, Ghada, Decebal Constantin Mocanu, and Mykola Pechenizkiy. \"Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer Nature Switzerland, 2022.\n\no\tWang, Zifeng, et al. \"SparCL: Sparse continual learning on the edge.\" Advances in Neural Information Processing Systems 35 (2022): 20366-20380.\n\no\tYin, Haiyan, and Ping Li. \"Mitigating forgetting in online continual learning with neuron calibration.\" Advances in Neural Information Processing Systems 34 (2021): 10260-10272.\n\nQUESTIONS:\n-   What is the contribution of each proposed components (neuron wise mask and data free replay) to the performance gain in DSN? Do you have an ablation for that?\n-   L 215 “Instead, we only allow the parameter update in the task-specific classifier (head) while freezing any parameters in the subnetwork.” This sentence is not clear. Does it mean that in backward transfer, you only update the head?\n-   Looking at Table 1 and Table 2 together and comparing WSN to DSN, do you think that the improvement gain in performance worths the additional costs (i.e., double the training cost in some cases). In other words, Are there specific cases in that one can definitely choose DSN over WSN?\n-   Is the proposed method applicable for more realistic scenarios i.e. class-incremental learning?\n-   Do you have some thoughts on how this approach can be further extended/improved to allow for backward transfer to multiple tasks?\n\nLIMITATIONS:\n-\tLimitations and social impact are not discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper focuses on task continual learning and attempts to enhance the elastic knowledge transfer across the tasks that sequentially arrive. With the help of masks, achieve forward and backward knowledge transfer.

STRENGTHS:
This paper is well written and easy to follow. The proposed method can achieve backword knowledge transfer with the help of mask similarity.

WEAKNESSES:
1. Why can masks identify the similarity of tasks? The mask of the new task before learning should not be determined, so how to complete knowledge transfer at beginning.
2. During the forward transfer, the parameters of the old task were fixed to prevent catastrophic forgetting, while during the reverse transfer, the parameters of the old task were optimized. The two seem somewhat contradictory. Can you unify the two to improve learning efficiency?
3. The EWC and IMM algorithms used in the experiment are both class incremental learning. How can they be compared to Task incremental learning?
4. The operation of convolutional layers is somewhat different from fully connected, and the paper lacks effective descriptions of other structures. Does the convolutional part regards each convolutional kernel as fully connected int the experiments?

QUESTIONS:
Please refer to the Weakness part.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuilding upon the principles of the Lottery Ticket Hypothesis, this paper introduces a hyper network model embedded with a series of competitive \"ticket\" sub-networks. Each of these sub-networks is designed to excel at their corresponding tasks, with a particular emphasis on knowledge transfer. Furthermore, this model promotes not just forward knowledge transfer, but also supports backward transfer of knowledge.\n\nSTRENGTHS:\nPros:\n1. The DSN method facilitates not only forward knowledge transfer but also backward transfer, offering a more dynamic and comprehensive learning paradigm that closely mimics human cognitive processes.\n\n2. The DSN method uses a neuron-wise mask and data-free memory replay, which can significantly save computational resources compared to maintaining a binary mask equal to the number of parameters.\n\n3. By using the mask to select neurons that have been used in earlier tasks and keeping their corresponding weights unchanged, the DSN method effectively addresses the issue of catastrophic forgetting that is common in continual learning scenarios.\n\n4. The data-free replay mechanism treats the trained subnetwork as a past experience and uses it to craft impressions of past samples, which bypasses the need to retain actual past samples, thus addressing privacy concerns and computational overhead.\n\nWEAKNESSES:\nCons:\n\n1. The DSN method may involve complex processes and computations, such as measuring mask similarity scores and fine-tuning the most similar tasks, which might make the method computationally intensive and challenging to implement.\n\n2. While task-specific masks help in achieving better model performance, they add another layer of complexity to the training process and may lead to overfitting if not handled correctly.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}