BNS: Building Network Structures Dynamically for Continual Learning
Authors: Qi Qin
Wenpeng Hu
Han Peng
Dongyan Zhao
Bing Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: continual learning

Abstract: Continual learning (CL) of a sequence of tasks is often accompanied with the catastrophic forgetting(CF) problem. Existing research has achieved remarkable results in overcoming CF, especially for task continual learning. However, limited work has been done to achieve another important goal of CL,knowledge transfer.In this paper, we propose a technique (called BNS) to do both.  The novelty of BNS is that it dynamically builds a network to learn each new task to overcome CF and to transfer knowledge across tasks at the same time. Experimental results show that when the tasks are different (with little shared knowledge), BNS can already outperform the state-of-the-art baselines. When the tasks are similar and have shared knowledge, BNS outperforms the baselines substantially by a large margin due to its knowledge transfer capability.

BNS: Building Network Structures Dynamically for Continual Learning
Qi Qin Han Peng', Wenpeng Hu', Dongyan Zhao> and Bing Liu
Center for Data Science, AAIS, Peking University
Department of Information Science, School of Mathematical Sciences, Peking University
Wangxuan Institute of Computer Technology, Peking University
4 Department of Computer Science. University of Illinois at Chicago (qinqi, phan, wenpeng.hu, zhaody) @pku.edu.en, liub @ uic. edu
Abstract
Continual learning (CL) of a sequence of tasks is often accompanied with the catastrophic forgetting (CF) problem. Existing research has achieved remarkable results in overcoming CF, especially for task continual learning. However, limited work has been done to achieve another important goal of CL, knowledge transfer. In this paper, we propose a technique (called BNS) to do both. The novelty BNS is that it dynamically builds network to learn each new task to overcome CF and to transfer knowledge across tasks at the same time. Experimental results show that when the tasks are different (with little shared knowledge), BNS already outperform the state-of-the-art baselines. When the tasks are similar and have shared knowledge, BNS outperforms the baselines substantially by large margin due to its knowledge transfer capability.
1 Introduction
Continual learning (CL) incrementally learns a sequence of tasks in a neural network. Each task consists of set of classes to be learned together. CL is often accompanied by the catastrophic forgetting (CF) problem (38]. Two main settings for CL have been extensively studied, namely, task continual learning (Task-CL) and class continual learning (Class-CL.). Task-CL. learns classifier for each task. In testing, the task id is provided for the test data so that the specific task model can be applied to assign a class to the test sample. Although Class-CL. also learns sequence of tasks, in testing the task id is not provided for each sample. Note that there is also a less known or studied CL setting called domain continual learning (Domain-CL), in which all tasks (or domains) share the same set of classes 22 and no task id is provided in testing. This paper focuses on Task-CL.
A large amount of research has been done overcoming CF [8]. The main existing techniques include the regularization-bases methods 63 1 25 18] replay-based methods 43 18 56) and
methods primarily designed to evaluate the importance of the parameters learned from old tasks, and then use some mechanisms to protect those important parameters so that the performance of old tasks will not be affected much in the process of lcarning new tasks. Replay-based methods use part of the data cached in the past or train generator generate some similar past data to help maintain the performance of the old tasks in learning new tasks. Structure-based methods allow the neural network to adaptively adjust the network structure during the learning process or to add learning module that does not share in
truettre-based methods (50) 61 39 Regularization-bant
"Corresponding authors. The main part of the work was done when Bing Liu was at Peking University on leave of absence from University of Illinois at Chicago.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the continual learning process. In the Task- CL setting, several techniques have overcome CF, e.g. (see Section
HAT [50]
4.3)
However, another major objective of CL., knowledge transfer across tasks g 21 47 has received relatively little attention. An ideal Task-CL. algorithm must do well on both (preventing CF and transferring knowledge). Being able to prevent CF alone is far from satisfactory. We will see in Section 4.3 that the knowledge transfer ability of the current Task-CL algorithms is very weak. This paper proposes a technique based on reinforcement learning (RL) called BNS (Building Network Structures dynamically for CL) to explicitly perform both CF prevention and knowledge transfer at the same time and automatically. When the tasks are different with little shared knowledge, BNS should prevent CF. When the tasks are similar with shared knowledge, BNS should transfer knowledge across tasks to achieve higher accuracy than without knowledge transfer.
BNS has three components: neural structure search agent, a set of actions, and the enviroument. The environment includes the current task data, a continual learner, a knowledge repository and a replay buffer. We propose to use the similarity of the new task and the old tasks as the state representation in the environment. This enables knowledge transfer as more similar tasks tend to have more shared knowledge to be transferred across tasks. For each new task, the agent uses the state to sample a sequence of actions to construct new continual learner for the task, which includes building a new network structure and selecting the past knowledge (i.e., parameters) from the knowledge repository to initialize the parameters. The continual learner then learns the current task. After learning, specially designed reward computed based on the current task validation data and the saved data of old tasks in the replay buffer (to avoid CF and to transfer knowledge to the current task). The reward is used to guide the training of the agent through RL. Thus, BNS achieves both CF avoidance and knowledge transfer at the same time. The final model learned by the continual learner not only performs the current new task well but also maintains the performance of the old tasks.
Experimental results on five datasets MNIST, CIFAR IO, CIFAR-100, F-EMNIST and F-Celeba show that BNS markedly outperforms the existing state-of-the-art Task-CL baselines. For datasets (F-EMNIST and F-Celeba) with high task similarities, BNS can leverage knowledge transfer to improve the accuracy substantially compared to the baselines. Even for datasets (MNIST. CIFARIO. and CIFAR-100) 100) with very different/dissinee
tasks, apart from overcoming CF, it can also improve the accuracy in most cases, which existing Task-CI. baselines have difficulty to do.
2 Related work
Existing CL approaches mainly focus on overcoming CF. Limited work has been done to explicitly leverage the knowledge gained in the continual learning process. In general, these methods can be divided into three categories: cuuularization-basee memory-based, and structure-based methods.
Regularization-bat methods deal with CF by keeping the important parameters for old tasks minimally modified in the new task learning. For example, EWC [25] uses fisher information to quantify the importance of parameters to old tasks, and selectively alters the learning rates of parameters to protect the parameters. Many papers dealing with CF employ similar ideas [63] 48) In the extreme, [62] finds orthogonal projections of weight updates that do not disturb the weights of old tasks. Many techniques also use knowledge distillation 33 59 6 4
3. [44 60, 23 40
11
35
29 [33. which is another family of methods that relies on regularization, i.e.,
knowledge distillation loss.
Memory-bused methods save a small number of training examples of each old task or generate some pscudo-cxample of the old tasks to be used to jointly train with the current task data (56)) This approach is also called experience replay. In the data memorization paradigm, the representative work includes GEM (36), A-GEM 77). and many others 46
TO, [7 In the data generation 39 15), which typically learns generator to generate pseudo old data. Our work also saves some training examples, but they are used in reward calculation in our setting rather than in training as in the existing methods.
43
59
45
paradigm, there is also extensive research [5] 49
20
58
31
Structure-based methods dynamically expand the network in learning each new task 33
60
32
39 or use masks over parameters to activate a subset of the network [50 21 37 which BNS also does. particular, MNTDP (55) learns each task by expanding the existing network resulted from the previous tasks by adding modules and selectively connect them to the existing network. DEN [61

---

is similar to MNTDP, although the specific details are different. Both these approaches are not based on reinforcement learning (RL). Our BNS very different as BNS is based on RL and it constructs a new network (continual learner) for the new task and initializes it based on 5 actions, not by adding modules to the existing network. RCL. [60] is an RL--based method. However, its state a fixed empty embedding. RCL has only one action, which determines how many filters should be added in each layer for the new task. Its reward is only the new task validation data accuracy and the model complexity, Our BNS uses task similarity as the state of the environment and has 5 actions. BNS's s reward considers both forgetting and transfer and thus can simultaneously tackles both catastrophic forgetting and knowledge transfer. Recently, pre- pre-trained feature extractors have also been employed to improve the CL accuracy [19 22 BNS does not use any pre-trained feature extractor.
Earlier works in lifelong learning focused mainly on knowledge transfer [4 47 g However, since
they use traditional learning methods such as regression 47 and naive Bayes 9 independent model for each task, there is no CF. In deep learning, the recent system CAT [2] learns a sequence of mixed similar and dissimilar tasks and does knowledge transfer among detected similar tasks. However, it does not do knowledge transfer among dissimilar tasks. BNS can transfer knowledge regardless of task similarity. Progressive Network [46) tries to perform forward transfer. It builds a model for each task and then connects all the models. However, it cannot do backward transfer and its network size grows quadratically in the number of tasks. It is not scalable. Several continual sentiment classification methods have focused on knowledge transfer as sentiment tasks are similar 42 22 However, they do not deal with dissimilar tasks for which CF is a major issue.
to build an
Our method is also related to neural architecture search (NAS) as we search for an optimal network structure. The goal of traditional NAS is to replace the manually designed models to find an advanced network structure for a specific task. A NAS method usually employs a reinforcement learning (RL) method [64 [52 to select actions to construct the final neural architecture. However, this method usually requires considerable computing power. To solve the problem of high computational cost, weight sharing strategy has been proposed +1 34 14. ENAS 41 is a weight sharing algorithm using RL. After training, the parameters of the previously sampled network structure are retained. When the newly sampled network contains some substructure of the previous network, the new network will reuse some parameters of the previous network substructure. Our method borrows the idea of ENAS, which has not been used in CL before. The major difference between our work and ENAS is that our policy exploits task similarities to enable our method to transfer knowledge across tasks as similar tasks tend to have shared knowledge that can be transferred to enable new tasks to learn better. Further, our actions and reward computation are specifically designed to enable BNS to achieve both knowledge transfer and CF avoidance at the same time, which ENAS does not do.
3 Method
Our Task-CL. setting is as follows. We incrementally learn sequence of N tasks, Each task has where z) is an input instance and v/ is its class label, and n, is the number of training examples of the t-th task. Similarly, the test dataset and validation dataset of task f are denoted by Diest and D (the use of the validation data will become clear shortly). We denote the datasets of task t by D' = [Dirain Diest: D D- Our goal to design a Task-CL. algorithm that can adaptively determine the network structure for each new task t to selectively transfer the previously learned knowledge to overcome CF and to learn the new task t better. We use reinforcement learning to help achieve our goal.
a training dataset Dirain = {(]), } 1=1
Overview of BNS: As mentioned in the introduction, the proposed reinforcement learning based Task-CL algorithm BNS consists of three components: a neural structure search agent, a set of actions, and an environment. When learning the new task t, the environment consists of the current task dataset Dt, continual learner f(:;O),, a knowledge repository K. and replay buffer T. At the high level, BNS works as follows. First, the current task dataset 1) and the cached data of each old task in the replay buffer T are used to compute their similarity. The set of similarities of the new task with all old tasks forms the current environment state representation St. Second, the agent takes S; as input to sample an action sequence a to build new continual learner f(:: Bz). The actions determine the network structure and parameter initialization of the continual learner. Third, the new continual learner is trained using the training set D Dirain in D', and then calculates the reward r for the model using the validation set Dr Diaalid of D) and the saved old task data in the replay buffer T. Fourth, it uses the reward to update the agent via reinforcement learning.
3

---

The above four steps are repeated to train the agent I iterations or times on task until it converges. After the agent is well trained, we select the continual learner with the highest reward as the final model for task I. This model not only performs well on the current task but also maintains the performance of the old tasks. After learning task I, the validation dataset D' saaid and the parameters of [:; 0₂) are saved in the replay buffer T and the knowledge repository K. respectively. Note that at any time, we only keep the trained model of the current continual learner for the last task.
3.1 Environment, Agent, Action, State and Reward
Environment: It contains the current task dataset D the current continual learner f(;;0e), the replay buffer 7 and the knowledge repository K. BNS uses a simple network topology, [(: (2) i.e., a network with L shared layers (shared among all the learned tasks) and a task specific layer (i.e.. the last classification layer) for each task. In learning the current task t. we calculate the similarity $1,4 between task t and each old task to obtain S. = [st,0- ..... $t,t-1]. We regard S, as the current state representation of the environment. Specifically, we use pre-trained feature extracto (-) to encode each instance of the current task's validation set D Dinaid and the saved data of previous tasks in the replay buffer T. We then compute the current data representation X, using the validation set Dt palid of task t and the data representation Xx of the saved validation set D valid" E T of the old task k in the replay buffer T. After that, we compute the similarity $t,k of X, and Xk.
X,
me Σ F(as), X2
F(TEs),
8t, = Similarity Xx,, Xx)-
IEE
where mg. my are the number of validation examples and Similarity can be any similarity function such as cosine, KL divergence or L, norm. When using KL divergence, we use the softmax function to convert X2 and Xg into probability distributions. This paper uses the three functions together through concatenation as the measure of similarity.
Agent: The agent aims to produce sequence of actions build an optimal network for the continual learner {(; fr)) for the current task t. In each training iteration of the agent for task t, it samples an action sequence to construct each layer of the network of f(;; 0e) using the current environment state S, Since there is correlation between the adjacent layers in the neural network, the actions for building the neural network should have front-to-back dependency. Thus, BNS uses a LSTM [16) based network as the agent to decide the action for each layer of the current continual learner f(: 0.). The agent trained with policy gradient by maximizing the expected reward computed using the current task performance and the CF avoidance on the old task data in the replay buffer T.
BNS has two types of actions: layer-wise actions (i.e., "reuse", "adaptation", "new", "fuse") and element-wise action (i.e., "mask"). They decide how each layer of the continual learner should be constructed by leveraging the shared layers' weights of each old task learned by its continual learner stored in K. For the layer-wise actions: "reuse" makes the new task use the same parameters as a previous task (sampled over the corresponding layers of all the old tasks according to the "reuse" probability). Since it can reuse any previous task's ' corresponding layer, the number of reuse increases as the task increases. "adaptation" expands the network by adding a small number of parameters. Specifically, we add some new additional neural units to the original layer to increase the capacity of the current continual learner. "new" spawns new parameters of exactly the same size as that of the current layer's parameters, and all the new parameters are randomly initialized. "fuse" makes the new task use the average parameter values of all previous tasks' corresponding layers. For each layer, the agent also needs to decide whether to use "mask" to protect some units so that its previous parameters (knowledge) will not be updated and forgotten (see Eq. (2) for details). Figure (AA) shows the schematic diagram of continual learner composed of these actions.
Action:
We use the convolution neural network as an example introduce how each action works. For the I-th layer of the convolutional neural network, the default kernel size is 5 x 5. The action "reuse" (or "fuse") is use a previous task's 4e layer's 's existing weights (or the average of all previous tasks I-th layers' existing weights) for the len layer of the current continual learner. For "adaptation", we use 1 convolution layer added to the original x 5 convolution layer in parallel. For the "new" action, we introduce replicated 5 x 5 layer that is initialized randomly. Given the output of of the
In this paper, we ResNet18 pre- trained on ImageNet in Pytorch calculate the task similarity.

---

b
LSTM
LSTM
Eiee
Mx
Layer
Layer+11 (++
(A)
(B)
Figure 1: (A): The schematic diagram of the continual learner after the agent learned task t. (B): Agent of BNS, where S; is the environment state of the learning task f; is the layer-wise action of layer 1 of the continual learner; by the element-wiss action of layer I of the continual learner.
units of the 1-th layer, our clement-wise "mask" mf and the layer final output of are
mi (>e)), yf mi @ vl.
(2)
where a, y and of are the sigmoid function, hyper-parameter and the 4n layer embedding of task I, respectively. It can also be seen from Eq. ( that "mask" can generate task specific features. From another perspective, the five actions defined for each layer of the continual learner can be divided into two categories: changing the network structure (i.e., "adaptation", "mask") and initializing the parameters (i.e., "reuse", "new" and "fuse").
Reward: Once continual learner is built in an iteration, it is trained with the current task training data D Divain and then reward is computed for the model of the continual learner for the iteration. We design the reward by considering both the current task t's performance improvement over the previous iteration computed using a Up-ratio function on its validation data and the forgetting rate of all previous tasks using Drop-ratio function. The Up-ratio of the continual learner constructed by the agent in the current iteration i of action sampling as follows,
acc
(3)
acc.
where i, acct,, (ac))-1,) are the sample step or iteration the agent and the accuracy the current (previous) iteration continual learner ((.00) on the current task t's validation set D' The Drop-ratio, which measures f('00,'s remembering (CF avoidance) ability, is as follows
acci,e
df
(4)
acci
where acei (or accç, is the accuracy of the final continual learner f(,, 02) (or the current f(; 0,)) on the previous task i's validation set D valid T. Our reward for the current iteration is
r = de + Bu
(5)
where 3 is syper-parammter. For simplicity, we omit the subscript from now on.
3.2 Training BNS
Figure [B) gives our LSTM based agent network. The LSTM samples actions via softmax classifiers an auto-regressive fashion: the action from the previous step and environment state are fed into the next step. In the first step, the agent network receives an empty action embedding with state S; as input. BNS has two sets of learnable parameters: the parameters of the agent LSTM л (S,, az). denoted by and the parameters of the current continual learner f(, 0t), denoted by 0.. The training procedure of BNS consists of two interleaving phases. The first phase trains O., the parameters of the current learner, using whole pass over the training dataset Dirain- The second phase trains the parameters the agent I. LTM. These two phases are alternated during the training of BNS and trained for fixed number of steps, 200 for each task in experiments. More details follows.
5

---

Table 1: Datasets details. For each of the first three datasets, the number of classes is the total number of classes the dataset. For each of the last two datasets, the number of classes represents the number of classes per task and each dataset has 10 tasks, not shown in the table.
Dataset MNIST CIFAR10 10
Classes
Training
Testing
10
60,000
10,000
10
50,000
10,000
CIFARI00
100
50,000
10,000
F-EMNIST
(per task)
7,673
858
F-Celeba
(per task)
960
110
Training the continual learner [( 0²): In this step, the agent first samples action sequence
which is used reconstruct a continual learner [(, 0c) for learning the current task. Then, we fix the agent's s policy Tw (St, ae) and minimize the expected loss
1
n2 Σ j=1
[(0.)
(6)
14
where, é the standard cross-entropy loss, computed on a minibatch of the training data, with a model f(., 0e) sampled from Tw (S,, de). Our policy is to output action for each layer of the current continual learner. When the action is "reuse" or "fuse", in order to ensure that these parameters will not change much and thus alleviate forgetting when training the current task, we introduce L. transfer regularization [12] 24] to constrain the training of the corresponding layer's parameters. In particular, we mark μ the parameters obtained from the knowledge repository AC when using the action "reuse" or "fuse". Finally, we perform stochastic gradient descent (SGD) on 8, to optimize the following loss function train each task
((f(zf.00):)) - -n -n. - --a112 ((0,) = The j=1
where 77 is a hyper-parameter.
Training the agent: When the continual learner f(, 02) converges, we use Eq. 5 to calculate the reward r (rf with iteration omitted), and then use the standard policy gradient algorithm to update the agent's 's parameters. In this step, we fix 6g and only update the policy parameters aiming to maximize the expected reward. Formally, we update the parameters w using the following function,
w + VulogT uz) .
where a is the learning rate of the agent.
Deriving the continual learner for the current task: We first sample several models from the trained policy T.. (S,, dz). For each sampled model, first train it on the current task and then compute its reward using the current task's ' validation set and replay buffer T. We then take only the model with the highest reward the final learned model.
4 Experiments
We evaluate the proposed B3 on five image classification datasets, three standard benchmarks MNIST 288, CIFARIO and CIFAR-100 [26] and two additional datasets F-EMNIST and F-
Celeba 21 originally used for federated learning, which have similar tasks and thus allow us to evaluate the knowledge transfer ability of BNS. F-EMNIST has 10 tasks and each task contains the written digits and characters (62 classes) of one individual writer. F-Celeba also has 10 tasks and each of them contains images of a celebrity labeled by whether he/she is smiling or not. The datasets statistics are given Table [ Our goal two-fold: (1) to verify whether searching specific network for each task through reinforcement learning can deal with CF, and (2) to verify whether similarities between tasks can help build a good network for knowledge transfer achieve improved accuracy.
'The code of BNS be found at: https: //gpt/h///tttb///// www.eo/mo/aaa. haommlalllaan comp/alalaupp6

---

4.1 Baselines
We consider the following 9 baselines for comparison with BNS, which is task continual learning (Task-CL) system. SGD uses a feature extractor and a classifier as the continual learner to incremen- tally learn all tasks. There is no mechanism for dealing with CF or knowledge transfer. LWF (33 uses knowledge distillation to deal with CF. EWC [63] is a commonly used baseline in most continual learning papers. IMM 30 combines the sequentially trained independent models for different tasks to perform all the tasks in the sequence. HAT [50] uses masks to protect task parameters to deal with CF. GEM [36] a replay method for Task-CL. UCL [2 is Task-CL method that tries improve HAT. HNET+R [56] is a latest replay Task-CL method (HNET+R performs better than HNET), is a recent Task-CL method that learns new tasks by taking gradient steps in orthogonal direction to the gradient subspaces deemed important for the past tasks. Note, LWF, EWC and IMM were originally designed for class continual learning (Class- CL). They were adapted for Task-CL. in We use these versions and run their code in the HAT repository.
GPM
48
HAT [50]
4.2 Implementation Details and Evaluation Metries
For experiments using MNIST and F-EMNIST, our BNS and all baselines except HNET+R use the same three fully connected layers as the shared feature extractor, and classifier on top with one fully connected layer for each task. In these two datasets, BNS does not use the action "adaptation" in agent to sample actions for the continual learner. For experiments with F-Celeba, CIFAR 10 and CIFARTO0 100, BNS and all baselines except HNET+R use AlexNet [7 as the basic network, which is borrowed from HAT [50]. Its shared feature extractor consists of three convolutional layers followed by two fully connected layers. The task specific classifier is one fully connected layer. In these three datasets, BNS uses all actions to search for a suitable architecture for the continual learner. For all experiments, we use a single Nvidia RTX 2080Ti GPU.
Parameter settings: For BNS, we use SGD as the optimizer with learning rate 0.1 to train the continual learner f(, 6.) except E-ENNIST and F-Celeba (learning rate 0.01). The parameters of the agent (LSTM) is updated by the Adam optimizer using the learning rate 0.0001. The hyper- parameters 17 and 3 are set 0.001 and 0.003 respectively. To be consistent with the baseline settings, our continual learner trains 100 epochs for all datasets except MNIST (10 epochs). We use ten percent of the training data of each task as the validation set for reward computation. After a task is learned, its validation set is saved in the replay buffer.
We adopt the final classification accuracy (ACC) as the basic evaluation metric. For each dataset, we report the average accuracy of all tasks tested using their respective test data after all tasks have been learned, thus the final accuracy. In order to compare the ability in overcoming CF and knowledge transfer, we introduce two other indicators. The first one is backward transfer (BWT) 33) to measure forgetting, which indicates how much the learning of new tasks has influenced the accuracy of the models for the previous tasks. BWT < O means that the learning of new tasks leads to some forgetting, BWT > indicates that the learning of new tasks helps the previous tasks. The second one is a new metric (Trans), which measures the forward transfer ability, showing how helpful the knowledge learned from previous tasks is to each new task in learning. In particular, Trans compares the accuracy of the final continual learner on each task with the accuracy of training an independent model for each task separately (no continual learning).
Evaluation metries:
N ACC where N is the total number of tasks learned; acc,,, is the accuracy on the test set of task right after task l is learned; acc, is the accuracy on the test set of task / of the independent model trained on task Accordingly, a.C,, is the accuracy of task after all N tasks have been learned. Note that since our BNS is built on HAT's architecture and may also sample the action "mask", the key mechanism in HAT for protecting the sub-network of each task to prevent forgetting. So comparing with HAT, we will see the knowledge transfer effect of BNS compared to a Task-CL system.
N
N Σ acON,
BWT
Σ 1
aceN,1
acC,,
Trans
acCN,
acc,-
N
N
'Experiments with H use its network as it is highly complex and is very difficult to change.
7

---

Table 2: Average results of ACC, BWT and Trans for BNS and baselines on MNIST, CIFARIO and CIFARTO0 with dissimilar tasks over 5 runs, where As. A2 and Aso represent the number of tasks that are continually learned on the corresponding dataset. Standard deviation, number of model parameters, and execution time are given in Appendix A.3.
Madel
ACC
MNIST BRYT
Trass
ACC
MNIST As
NWT
Trats
ACE
CIEARIO
BWIT
Trans
ACE:
CIPARIO As
RWT
Torn
ATT:
CTEAR
NWT
Tian
SUD
65.27
17:45
14.05
90.47
11 2020
-5.53
73.54
25.45
115.53
70.03
25.25
10..55
01.14
50.24
-20.22
533
-3.36
-0.28
-0-16
-18.60
-14.05
63.23
-6.62
57.81
-37 -05
-32.
15.61
12.06
9.37
6.63
22.00
10.58
DCL
95.74
1.22
1.07
97:40
0.07
1:30
26.20
0.98
6.52
88.00
3.04
2.95
64.25
20.84
26.21
GIM
196.00 56
-3.65
-1:25
98:22
-0.37
-1.03
08.02
-2.06
-3.39
55.770
-4.72
-2.90
7h-03
-3.70
-11.3 11.10
120M
-3.30
-1.33
46
-U.77
-2.63
77:84
-4.10
-14.37
02.87
-2.09
-25.77
73.05
-0.78
-12.30
HNET-R
0.31
0.31
3.89
0.00
-7 7.00
Table 3: Average results of ACC, BWT and Trans for BNS and the baselines on F-EMNIST and F-Celeba with similar tasks over 5 runs.
F-EMNIST
F-Celeba BWT
Model
ACC
BWT
Trans
ACC
SGD
13.49
-26.66
-25.87
54.93
-1.62
-6.8
LWF
8.11
-2.20
-31.25
57.18
-1.61
-4.63
FWC
43.47
-8 -8.67
4.10
55.27
-1.22
- 6.53
IMM
17.52
25.74
21.84
57.81
-0.00
4.00
UCL
49.75
-1.63
10.38
55.26
-3.65
-6.55
GPM
57.84
6.40
19.07
58.08
-3.69
GEM
53.82
21.75
15.02
54.51
-3.62
-7.77
HIET+ R
24.95
43.69
13.85
55.84
3.23
5.02
HAT
56.02
0.00
16.65
59.82
0.00
- -1.99
BNS
60.38
0.53
21.01
65.44
-2.59
3.62
4.3 Experimental Results
To evaluate the effectiveness of forgetting avoidance and knowledge transfer of the continual learner of BNS, we use the 3 metrics in Eq. Each row in the table gives the average results over 5 runs of all compared systems. The mean of results are summarized in Table 2 and Table 3 The standard deviations are summarized in Tables we evaluate the effectiveness under three settings, tasks (As), 2 tasks (A2) and 50 tasks (A5o). For example, CIFARIO 10 As means that the original dataset is divided into 5 tasks and each task contains two classes. For F-MNIST and F-Celeba, the 10 tasks are fixed in the original datasets. We analyze the results in the two tables from the following perspectives.
Appendix A.3. For MNIST, CIFARIO and CIFARI00,
Final accuracy (ACC): BNS performs very well. Only HNET+R performs slightly better than BNS on CIFAR10 A5. However, HNET+R performs poorly in the other experiments in Table 2 especially on CIFAR 100 A50- On the two similar task datasets F-EMNIST and F-Celeba (Table 3 the current state-of-the- art models, HNET+R and GPM, are very weak compared to BNS. They than HAT. BNS outperforms all baselines by a large margin, which shows that its ability to handle multiple tasks using the final continual learner is powerful.
poorer
With and without continual learning (CL): Here we compare the results of learning each task separately (without CL) and CL and study the impact of CL. The Trans metric is for this purpose. Recall Trans the difference between the final accuracy ACC of the continual learner after all tasks are learned and the accuracy of learning model for each task separately and independently (which is ACC- Trans). The Trans column Table shows that BNS produces similar or better results than learning separately when the tasks are different. The baselines are clearly poorer. For CIFAR 100 Aso. there are some loss of accuracy, but the baselines lose even more. The Trans column in Table B shows a very different story when we have similar tasks. We see that five CL basclines can improve the accuracy (positive Trans values) of learning separately without CL (ACC Trans) for F-EMNIST, which show some knowledge transfer. But the knowledge transfer ability of BNS is significantly better than all baselines. For example, the Trans values of GPM on F-EMNIST is 19.07% and on F-Celeba is 3.66% (negative transfer), but these of BNS are 21.01% and 3.62% respectively. Note that the final accuracy ACC of all other baselines are weaker than HAT, which is markedly poorer than BNS. This again demonstrates the knowledge transfer ability of BNS.
Overcoming catastrophic forgetting (CF): BWT measures the ability of preventing CF. SGD performs extremely poorly with very negative values (negative backward transfer or CF) for BWT for
8

---

1.0
POBC
0.8
Parecutage of actions
0.6
0.4
0.2
0.0
MINIST A,5
MNIST A_2
CIFARIO
CIFAR1O A,2
A FEMNIST
F-Coleba
CIF 1RD0
Figure 2: The percentage of each action used in the continual learner. Note that "mask" co-exists with other actions. At the same time, "adaptation" is accompanied by "reuse".
all datasets. This is because SGD has no mechanism deal with CF. Comparing the BWT values of BNS and the baselines, we can see that BNS performs very well except few cases with GPM and HAT (no CF), but their final accuracy ACC values are substantially lower.
Backward knowledge transfer: As indicated in Section 4.2 BWT not only measures CF avoidance (i.e., BWT O), but also measures backward knowledge transfer from new tasks old tasks (iee, BWT > 0). For all datasets, HAT has no backward transfer (BWT = 0.00) and no forgetting. Our BNS achieves none negative backward transfer in of the 7 experiments. The baseline GPM achieves only in 3, out of which 2 (F-EMNIST and F-CEleba) still result in much lower final accuracy (ACC) than BNS, indicating that the forward transfer capability of GPM poor (see below).
Forward knowledge transfer: This is defined as the difference between the accuracy a task when is first learned (ACC - BWT) and the accuracy of learning the task separately (ACC Trans), N-1 BWT, which shows whether the knowledge learned from previous tasks can help i... Trans- N
learn the new task. We summarize the experimental results of this metric in Appendix A.2. We can again see that BNS is substantially better overall. As discussed above, for the two similar tasks datasets in Table 3 GPM's backward transfer is more effective, but GPM's forward transfer N B BW) results are only 13.31% and -4.61 % but those of BNS are 20.53% and 5.95%
(Trans N respectively, which explains why GPM's final accuracy ACC results are much lower than BNS. For the best baseline HAT, they are respectively 16.65% and 1.99%, which are also substantially lower than those of BNS.
In summary, we can conclude that for dissimilar tasks with little shared knowledge, BNS can deal with CF as well as state-of-the of-the-art baselines and in many cases, it can even achieve some positive knowledge transfer. For similar tasks with shared knowledge, the knowledge transfer capability of BNS is considerably better than the baselines. This indicates that BNS is much better model for achieving the two objectives of Task-CL, CF avoidance and knowledge transfer.
4.4 Actions Used by the Agent
In Figure 2 we plot the percentage of each action used in the action sequences generated by the agent for all tasks in our experiments. It can be seen that in the datasets with similar tasks (F-EMNIST and F-Celeba), the percentage of action "masks" used is much lower. This is because for similar tasks sharing and knowledge transfer are needed but masks are for protecting previous knowledge. In Appendix A. we will present case study to show the action selection of the agent.
4.5 A Ablation Experiments and Analysis
As discussed earlier. BNS produces a task-related network structure and gives specific initialization method for each task according to similarity of the task to previous tasks. order to verify the effectiveness of including the task similarity, we compare the results of three settings five datasets MNIST As. CIFARTO As. CIFAR100 Aso. F-EMNIST and F-Celeba. The first setting is without task similarity the agent input, which degenerates into traditional neural architecture search algorithm,

---

Table 4: Ablation study, BNS (zero-smi) means that no task similarity is used as an input to the agent. BNS (one-smi) (or BNS) means that only the cosine similarity (or cos, KL and norm together) is (are) used to calculate task correlation and is used as an input to the agent.
Components
MNIST
CIFARIO
CIFARIO0
F-EMNIST
F-Celeba
SGD
85.77
73.57
61.14
13.49
54.93
RNS (sen-sim)
99.24
88.99
78.50
57.75
62.28
BNS (one-sim)
99.85
90.73
80.21
58.41
66.62
BNS
99.87
91.40
82.39
60.38
65.44
referred as BNS (zero-sim). The second setting uses similarity, cosine similarity, the agent input, referred to as BNS (one-sim). The third setting uses all similarity measures as an input to the agent, which is our final BNS model. Table gives the results of the ablation experiments.
By comparing the results of BNS (zero-sim) and SGD, it can be seen that after storing the parameters learned in the past, using NAS (neural architecture search) to search for each task has significant positive effect on overcoming catastrophic forgetting (CF). That is, designing a new initialization method for each task using the parameters of previous tasks and appropriately changing the network capacity can already help overcome CF significantly. Additionally, introducing one or more similari- ties as the input the agent further improves the model's s ability to overcome CF (see the results of BNS (one-sim) and BNS). This demonstrates that it is highly effective to generate the network structure and initialization method based on task similarity.
5
Conclusion
This paper proposed a novel reinforcement learning hased method, BNS, for task continual learn- ing (Task-CL... The proposed model can dynamically adjust the network structure and parameter initialization method of each task by exploiting the similarities and differences of the tasks to achieve both objectives of Task-CL, overcoming catastrophic forgetting and transferring knowledge across tasks to improve accuracy. Experimental results showed that the performance of BNS is superior to the state-of-the-ar baselines in both forgetting prevention and knowledge transfer. The knowledge transfer ability of the baselines are especially weak compared to BNS. One limitation of work is that it requires a large amount of memory and compute. In our future work, we plan to try generative approach to generate the parameters of the past models.
Acknowledgments
This work was supported in part by an internal grant of Peking University and National Key R&D Program of China 2020AAA0106600.
References
[1] Tameem Adel, Han Zhao, and Richard E Turner. Continual learning with adaptive weights (claw). ICLR, 2020.
[2] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty- continual learning with adaptive regularization. In Advances in Neural Information Processing Systems, pages 4394 4404, 2019.
[3] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory Aware Synapses: Learning what (not) forget. arXiv preprint 171.017111.06111. 2017.
[4] Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In ICCV, October 2019.
[5] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Under- standing and simplifying one-shot architecture search. volume 2, pages 883-893, 2018.
[6] Francisco Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning, In ECCV, pages 233-248 2018.
10

---

[7] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a a-gem. In ICLR, 2019.
[8] Zhiyuan Chen and Bing Liu. Lifelong Machine Learning. Morgan & Claypool Publishers, 2018.
[9] Zhiyuan Chen, Nianzu Ma, and Bing Liu. Lifelong learning for sentiment classification. In ACL, pages 750-756, 2015.
[10] Cyprien de Masson d' A'Auumme, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In NeurlPS, 2019.
[11] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, 2019.
[12] Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In KDD, pages 109-117, 2004.
[13] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08755 2017.
[14] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling, 2020.
[15] Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. arXiv preprint arXiv:1910.02509, 2019.
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 133::173- 1997.
[17] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, pages 831-839, 2019.
[18] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR,, 2019.
[19] Wenpeng Hu, Qi Qin, Mengyu Wang, Jinwen Ma, and Bing Liu. Continual learning by using information of each class holistically. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7797-7805, 2021.
[20] Nitin Kamra, Umang Gupta, and Yan Liu. Deep Generative Dual Memory Network for Continual Learning. arXiv preprint arXiv:1710.10360 2017.
[21] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of a mixed sequence of similar and dissimilar tasks. In NewrIPS, 2020.
[22] Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu. Classic: Continual and contrastive learning of aspect sentiment classification tasks. In Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP-202)) 2021.
[23] Ronald Kemker and Christopher Kanan. FearNet: Brain-Inspired Model for Incremental Learning. In ICLR, 2018
[24] Wolf Kienzle and Kumar Chellapilla. Personalized handwriting recognition via biased regular- ization. In ICML, pages 457-464, 2006.
[25] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, and Others. Overcoming catastrophic forgetting in neural networks. volume 114, pages 3521-3526. National Acad Sciences, 2017.
[26) Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, Toronto., 2009.
11

---

[27] Alex Krizhevsky, I. Sutskever. and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[28] Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits. http:/lyannleccu.comm www.exxl///mmiis//. 1998.
[29] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, 2019.
[30] Sang-Woo Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tai Tak Zhang. Overcoming catastrophic forgetring by incremental moment matching. In NIPS, pages 4655-4666 2017.
[31] Timothée Lesort, Hugo Caselles-Dupprs Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning, https:/aarii.orggab/////10.0111 2018.
[32] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In ICML, 2019.
[33] Zhizhong Li and Derek Hoiem. Learning without forgetting. PAMI, 40(12):9935-2447) 2017.
[34] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search.
2018.
[35] Yaoyao Liu, An-An Liu, Yuting Su, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. arXiv preprint arXiv:2002. /02/1, 2020.
[36]
David Lopez-Paz and Marc' MurciAurelio Ranzato. Gradient episodic memory for continual learning. In NIPS, 2017.
[37] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[38] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning & motiv., volume 24. 1989.
[39] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In CVPR, pages 1 13221-11329, 2019.
[40] German I Parisi, Jun Tani, Comelius Weber, and Stefan Wermter. Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization. arXiv preprint, 2018.
[41] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018.
[42] Qi Qin, Wenpeng Hu, and Bing Liu. Using the past knowledge to improve sentiment classifica- tion. In EMNLP-findingg 2020.
[43] Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, pages 5533-5542, 2017.
[44] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In NIPS, pages 3738-3748, 2018.
[45] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Greg Wayne. Experi- ence replay for continual learning. In NewrIPS, 2019.
[46] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins. Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606 2016,
[47] Paul Ruvolo and Eric Eaton. ELLA: An efficient lifelong learning algorithm. In ICML, 2013.
12

---

[48] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In ICLR, 2021.
[49] Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual learning in generative adversarial nets. arXiv preprint arXiv: 077700000395 2017.
[50] Joan Serrå, Didac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, 2018.
[51] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. NIPS, pages 2994-3003, 2017.
[52] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V. Le. Mnasnet: Platform-awe aware neural architecture search for mobile, 2019.
[53] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, and Yihong Gong. Bi-objective continual learning: Learning' new" while consolidating'knowwe In AAAI, pages 5989-5996, 2020.
[54] Sebastian Thrun. Lifelong learning algorithms. In Learning to Learn, pages 181-209. Springer, 1998.
[55] Tom Veniat, Ludovic Denoyer, and Mare' Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. In ICLR, 2021.
[56] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. ICLR, 2020.
[57] Hao Wang, Bing Liu, Shuai Wang, Nianzu Ma, and Yan Yang. Forward and backward leogledge transfer for sentiment classification. In ACML, 2019.
[58] Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, al. Memory replay gans: Learning to generate new categories without forgetting. In NIPS, pages 5962-5972, 2018.
[59] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.
[60] Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NIPS, pages 899-908, 2018.
[61] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong Learning with Dynamically Expandable Networks. ICLR, 2018
[62] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continuous learning of context-dependent processing in neural networks. Nature Machine Intelligence, 2019.
[63] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987-3995, 2017.
[64] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition, 2018.
13


Reviews:
[{"review_id": "bq2kdh8gKYC", "reviewer": "Reviewer_FW7Z", "summary": "The paper proposes an approach for continual learning that builds dynamic network structures such that it minimizes catastrophic forgetting but maximizes forward transfer. The approach involves neural architecture search where an LSTM policy optimizes for selection of network layers (essentially their size) and their initialization with a reward that calculates forward improvement (from the architecture) over iterations as well as backward transfer (across tasks). The results on some standard (albeit toy-ish) CL datasets show that the approach mitigates catastrophic forgetting while also inducing forward transfer in tasks that can use it.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros \n- The paper is well written. Even though the approach is quite involved, the details are well explained and easy to follow along.\n- Experimental results are strong and confirm that the approach can learn architectures for forward transfer while also minimizing forgetting. However, some recent related approaches and baselines are not discussed (see below).\n- Ablations are insightful. I also liked the qualitative analysis in the supplementary of a sequence of architectures found by the agent.\n\nCons\n- The datasets used are really small toy datasets, with only a maximum of 10 tasks in a sequence. While these are common datasets used in CL literature, it will be better to move beyond these for more impactful/useful results.\n- A recent work [1] seems very closely related. It will be good to compare with this or at the least discuss differences with it. I also see additional baselines in that paper, such as RCL (Xu and Jhu, 2018) and DEN (Yoon et al., 2018) that are also relevant here.\n- The approach seems to keep architectures for all previous tasks in memory which would give a prohibitively high memory requirement (order of number of tasks)\n\nQuestions for Authors\n- The “reuse” action seems to require all parameters of previous tasks in memory. That will have an order of magnitude larger memory footprint than the baselines. Can you comment on this?\n- The \"new\" action seems to double the number of parameters in a layer. This has the potential to blow up the number of parameters in the architecture. How is this managed? Is there a cap on max number of parameters?\n- Please clearly characterize the memory requirement, in terms of both the number of parameters as well as the size of replay buffer, that the approach requires.\n- Task representations seem to be computed by averaging all the representations of all instances in the task, regardless of their labels (see eq 1). This seems weak and uninformative. Can you comment on what similarities you hope to capture through such representations? Since this is the most important part of the input to the policy, it could be helpful to visualize this similarity for tasks to understand what it is capturing.\n- Can you comment on the relevance of [1] and the baselines of RCL and DEN in the context of the proposed approach? \n\n[1] EFFICIENT CONTINUAL LEARNING WITH MODULAR NETWORKS AND TASK-DRIVEN PRIORS, ICLR 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Since the architecture required to solve each task could be expensive (in terms of memory and compute), the authors should potentially discuss this as a limitation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "SpldLAq0dLY", "reviewer": "Reviewer_mjUa", "summary": "The paper focuses on transfer of knowledge across tasks in a continual learning setting without catastrophic forgetting. The proposed method dynamically builds neural network structures. For each new task an agent samples a sequence of actions that integrates knowledge transfer from the old tasks to construct the continual learner. Old data are stored in a replay buffer, which are used for reward calculation rather than in training. The model thus incrementally learns a sequence of $N$ tasks. The continual learner is trained in a reinforcement learning setting where the reward is a combination of performance increase compared to a previous iteration and the forgetting rate of all previous tasks promoting the accuracy on all previous tasks.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides a clear definition of task-continual learning and class-continual learning, although the paper focuses on the easier problem of task-continual learning where at test time the task id is provided. \n\nThe model exploits task similarities to enable the model to transfer knowledge across tasks. \nThe task similarity is based on the current data of the new task and the data stored in the memory buffer of the old tasks and is computed as the combination of three classical similarity metrics applied on the representation of these data. The paper would benefit from adding a clear motivation of the way this similarity is computed and the intuition behind it. \n\nNow, the overview of the model and the training of the continual learner are described in text. For clarity, this text could be replaced by an algorithm in pseudocode. \n\nThe work has value as transfer learning model where you learn to dynamically build your model for a new task exploiting properties of older learned models classified by a number of layer-wise discrete action labels (e.g., reuse, fuse, mask). These action labels are predicted by means of an LSTM that decides which action for which layer. The transfer learning capabilities of the proposed model are clearly shown in the experiments. \n\nThe model is evaluated with several datasets commonly used in continual learning and the results of several metrics (accuracy, forgetting metric and transfer ability metric) are convincing that the claims of the paper are accomplished when compared with many baselines.  \n\nRemarks to further improve the paper:\n-\tLine 122: Clarify better why the agent for a task $t$ is trained for $N$ iterations until it converges, where $N$ is the number of tasks.\n-\tTraining time is giving in the supplementary material. An analysis of the computational complexity of the training procedure and ideas for reducing this complexity would strengthen the paper. \n-\tFigure 1 (A): the font size of the legend could be improved; Figure 1 (B): More explanation in the caption would aid the reader to understand this figure early on when reading the paper.\n-\tAre the validation sets for reward computation released after publication of the paper? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitations of their work in the supplementary material. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "FmnMb_uPA2", "reviewer": "Reviewer_t7Yn", "summary": "This paper studies continual learning when the model is required to fit a sequence of tasks (with task identifiers) and maximize the average performance on all tasks. The goal is to prevent both catastrophic forgetting (CF), and improve forward knowledge transfer. It presents a neural architecture search (NAS) approach, namely BNS, to train an agent to design the new architecture for each new task. The idea of applying NAS in lifelong learning setting is interesting and novel, and the experimental results do show performance gains.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the paper is novel with good results. But there are certain technical details that are not immediately clear to me (and the link to code is empty), that I would appreciate if the author can clarify a bit in the rebuttal, and I am happy to adjust my score based on that.\n\n1. How do you evaluate the score for, say the first task, after training on all tasks? Do you save the best model for it in the memory buffer? Or do you utilize the agent to create a new model? If it's the former, wouldn't that requires linear memory in terms of number of tasks? If it's the later, what is the environment, or the base learner for the input to the agent? Does that mean the agent can utilize knowledge learned in the later task to construct model for earlier tasks?\n2. It seems to me that the agent is the most prone to CF. Essentially, it is trained to optimize the current task performance in a lifelong learning setup. Although preventing CF is included as part of the reward, I am not super convinced that this is a signal that is strong enough. Have you tried use other lifelong learning methods to regularize the agent?\n3. A big concern of NAS is its efficiency, and this is even more serious the case here since you are learning to search for not just one but many tasks. Given the lifelong learning setting is often used in a online setting in reality, it seems to me that the computational complexity may be prohibitively large for your method. Can you provide the runtime compared with other methods for your experiments?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "9uHOAoFZ8Qy", "reviewer": "Reviewer_W4Fy", "summary": "The paper proposes BNS  for task continual learning, which can dynamically adjust the network structure and parameter initialization method of each task by exploiting the similarities and differences of the tasks to achieve both objectives of Task-CL, overcoming catastrophic forgetting and transferring knowledge across tasks to improve the performance.\nThe contribution of the paper is to propose a knowledge transfer method across different tasks for continual learning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper address the problem of knowledge transfer across tasks in continual learning, which has received relatively little attention.  This problem is worth to be studied.\n2. The paper formulates continual learning as a reinforcement learning problem and defines a novel RL setting.\n3. The experiment results show the proposed method is effective.\n4. The proposed method combines the structure search and element-wise action to better find the effective subnetwork for each task.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The main concern is the RL usually needs large trial and error, and thereby low-efficient. The paper should give the learning cost.\n2. The number of datasets is relatively small and hard to verify the effectiveness of continual learning. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an NAS approach (namely BNS) for continual learning that builds dynamic network structures such that it minimizes catastrophic forgetting but maximizes forward transfer. The experiment results on several CL tasks show the effectiveness of the proposed method.

Overall, all reviewers found the paper to be well written and easy to follow, and focuses on an important research problem with relatively little attention from the area. Based on the reviewer comments, the authors further provided missing details pointed out by the reviewers and some reviewers increased the review score. After the rebuttal, all the reviewers remained positive. I recommend accepting this paper.

Decision: Accept (Poster)