BNS: Building Network Structures Dynamically for Continual Learning

1 Introduction . Continual learning ( CL ) incrementally learns a sequence of tasks in a neural network . Each task consists of a set of classes to be learned together . CL is often accompanied by the catastrophic forgetting ( CF ) problem [ 38 ] . Two main settings for CL have been extensively studied , namely , task continual learning ( Task-CL ) and class continual learning ( Class-CL ) . Task-CL learns a classifier for each task . In testing , the task id is provided for the test data so that the specific task model can be applied to assign a class to the test sample . Although Class-CL also learns a sequence of tasks , in testing the task id is not provided for each sample . Note that there is also a less known or studied CL setting called domain continual learning ( Domain-CL ) , in which all tasks ( or domains ) share the same set of classes [ 22 ] and no task id is provided in testing . This paper focuses on Task-CL . A large amount of research has been done on overcoming CF [ 8 ] . The main existing techniques include the regularization-based methods [ 63 , 11 , 25 , 18 ] , replay-based methods [ 43 , 18 , 56 ] and structure-based methods [ 50 , 61 , 39 ] . Regularization-based methods are primarily designed to evaluate the importance of the parameters learned from old tasks , and then use some mechanisms to protect those important parameters so that the performance of old tasks will not be affected much in the process of learning new tasks . Replay-based methods use part of the data cached in the past or train a generator to generate some similar past data to help maintain the performance of the old tasks in learning new tasks . Structure-based methods allow the neural network to adaptively adjust the network structure during the learning process or to add a learning module that does not share in ∗Corresponding authors . The main part of the work was done when Bing Liu was at Peking University on leave of absence from University of Illinois at Chicago . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the continual learning process . In the Task-CL setting , several techniques have overcome CF , e.g , HAT [ 50 ] ( see Section 4.3 ) . However , another major objective of CL , knowledge transfer across tasks [ 8 , 21 , 47 ] , has received relatively little attention . An ideal Task-CL algorithm must do well on both ( preventing CF and transferring knowledge ) . Being able to prevent CF alone is far from satisfactory . We will see in Section 4.3 that the knowledge transfer ability of the current Task-CL algorithms is very weak . This paper proposes a technique based on reinforcement learning ( RL ) called BNS ( Building Network Structures dynamically for CL ) to explicitly perform both CF prevention and knowledge transfer at the same time and automatically . When the tasks are different with little shared knowledge , BNS should prevent CF . When the tasks are similar with shared knowledge , BNS should transfer knowledge across tasks to achieve higher accuracy than without knowledge transfer . BNS has three components : a neural structure search agent , a set of actions , and the environment . The environment includes the current task data , a continual learner , a knowledge repository and a replay buffer . We propose to use the similarity of the new task and the old tasks as the state representation in the environment . This enables knowledge transfer as more similar tasks tend to have more shared knowledge to be transferred across tasks . For each new task , the agent uses the state to sample a sequence of actions to construct a new continual learner for the task , which includes building a new network structure and selecting the past knowledge ( i.e. , parameters ) from the knowledge repository to initialize the parameters . The continual learner then learns the current task . After learning , a specially designed reward is computed based on the current task validation data and the saved data of old tasks in the replay buffer ( to avoid CF and to transfer knowledge to the current task ) . The reward is used to guide the training of the agent through RL . Thus , BNS achieves both CF avoidance and knowledge transfer at the same time . The final model learned by the continual learner not only performs the current new task well but also maintains the performance of the old tasks . Experimental results on five datasets MNIST , CIFAR10 , CIFAR-100 , F-EMNIST and F-Celeba show that BNS markedly outperforms the existing state-of-the-art Task-CL baselines . For datasets ( F-EMNIST and F-Celeba ) with high task similarities , BNS can leverage knowledge transfer to improve the accuracy substantially compared to the baselines . Even for datasets ( MNIST , CIFAR10 , and CIFAR-100 ) with very different/dissimilar tasks , apart from overcoming CF , it can also improve the accuracy in most cases , which existing Task-CL baselines have difficulty to do . 2 Related work . Existing CL approaches mainly focus on overcoming CF . Limited work has been done to explicitly leverage the knowledge gained in the continual learning process . In general , these methods can be divided into three categories : regularization-based , memory-based , and structure-based methods . Regularization-based methods deal with CF by keeping the important parameters for old tasks minimally modified in the new task learning . For example , EWC [ 25 ] uses fisher information to quantify the importance of parameters to old tasks , and selectively alters the learning rates of parameters to protect the parameters . Many papers dealing with CF employ similar ideas [ 63 , 13 , 3 , 44 , 60 , 23 , 40 , 11 , 1 , 48 ] . In the extreme , [ 62 ] finds orthogonal projections of weight updates that do not disturb the weights of old tasks . Many techniques also use knowledge distillation [ 33 , 59 , 6 , 4 , 35 , 29 , 53 ] , which is another family of methods that relies on regularization , i.e. , knowledge distillation loss . Memory-based methods save a small number of training examples of each old task or generate some pseudo-examples of the old tasks to be used to jointly train with the current task data [ 56 ] ) This approach is also called experience replay . In the data memorization paradigm , the representative work includes GEM [ 36 ] , A-GEM [ 7 ] , and many others [ 46 , 43 , 59 , 45 , 10 , 17 ] . In the data generation paradigm , there is also an extensive research [ 51 , 49 , 20 , 58 , 18 , 31 , 39 , 15 ] , which typically learns a generator to generate pseudo old data . Our work also saves some training examples , but they are used in reward calculation in our setting rather than in training as in the existing methods . Structure-based methods dynamically expand the network in learning each new task [ 33 , 61 , 60 , 32 , 39 ] or use masks over parameters to activate a subset of the network [ 50 , 21 , 37 ] , which BNS also does . In particular , MNTDP [ 55 ] learns each task by expanding the existing network resulted from the previous tasks by adding modules and selectively connect them to the existing network . DEN [ 61 ] is similar to MNTDP , although the specific details are different . Both these approaches are not based on reinforcement learning ( RL ) . Our BNS is very different as BNS is based on RL and it constructs a new network ( continual learner ) for the new task and initializes it based on 5 actions , not by adding modules to the existing network . RCL [ 60 ] is an RL-based method . However , its state is a fixed empty embedding . RCL has only one action , which determines how many filters should be added in each layer for the new task . Its reward is only the new task validation data accuracy and the model complexity . Our BNS uses task similarity as the state of the environment and has 5 actions . BNS ’ s reward considers both forgetting and transfer and thus can simultaneously tackles both catastrophic forgetting and knowledge transfer . Recently , pre-trained feature extractors have also been employed to improve the CL accuracy [ 19 , 22 ] . BNS does not use any pre-trained feature extractor . Earlier works in lifelong learning focused mainly on knowledge transfer [ 54 , 47 , 8 ] . However , since they use traditional learning methods such as regression [ 47 ] and naive Bayes [ 9 , 57 ] to build an independent model for each task , there is no CF . In deep learning , the recent system CAT [ 21 ] learns a sequence of mixed similar and dissimilar tasks and does knowledge transfer among detected similar tasks . However , it does not do knowledge transfer among dissimilar tasks . BNS can transfer knowledge regardless of task similarity . Progressive Network [ 46 ] tries to perform forward transfer . It builds a model for each task and then connects all the models . However , it can not do backward transfer and its network size grows quadratically in the number of tasks . It is not scalable . Several continual sentiment classification methods have focused on knowledge transfer as sentiment tasks are similar [ 42 , 22 ] . However , they do not deal with dissimilar tasks for which CF is a major issue . Our method is also related to neural architecture search ( NAS ) as we search for an optimal network structure . The goal of traditional NAS is to replace the manually designed models to find an advanced network structure for a specific task . A NAS method usually employs a reinforcement learning ( RL ) method [ 64 , 52 ] to select actions to construct the final neural architecture . However , this method usually requires considerable computing power . To solve the problem of high computational cost , a weight sharing strategy has been proposed [ 41 , 34 , 5 , 14 ] . ENAS [ 41 ] is a weight sharing algorithm using RL . After training , the parameters of the previously sampled network structure are retained . When the newly sampled network contains some substructure of the previous network , the new network will reuse some parameters of the previous network substructure . Our method borrows the idea of ENAS , which has not been used in CL before . The major difference between our work and ENAS is that our policy exploits task similarities to enable our method to transfer knowledge across tasks as similar tasks tend to have shared knowledge that can be transferred to enable new tasks to learn better . Further , our actions and reward computation are specifically designed to enable BNS to achieve both knowledge transfer and CF avoidance at the same time , which ENAS does not do . 3 Method . Our Task-CL setting is as follows . We incrementally learn a sequence of N tasks . Each task t has a training dataset Dttrain = { ( xtj , ytj ) } nt j=1 , where x t j is an input instance and y t j is its class label , and nt is the number of training examples of the t-th task . Similarly , the test dataset and validation dataset of task t are denoted by Dttest and D t valid ( the use of the validation data will become clear shortly ) . We denote the datasets of task t by Dt = { Dttrain , Dttest , Dtvalid } . Our goal is to design a Task-CL algorithm that can adaptively determine the network structure for each new task t to selectively transfer the previously learned knowledge to overcome CF and to learn the new task t better . We use reinforcement learning to help achieve our goal . Overview of BNS : As mentioned in the introduction , the proposed reinforcement learning based Task-CL algorithm BNS consists of three components : a neural structure search agent , a set of actions , and an environment . When learning the new task t , the environment consists of the current task dataset Dt , a continual learner f ( · ; θt ) , a knowledge repository K and a replay buffer T . At the high level , BNS works as follows . First , the current task dataset Dt and the cached data of each old task in the replay buffer T are used to compute their similarity . The set of similarities of the new task with all old tasks forms the current environment state representation St. Second , the agent takes St as input to sample an action sequence at to build a new continual learner f ( · ; θt ) . The actions determine the network structure and parameter initialization of the continual learner . Third , the new continual learner is trained using the training set Dttrain in D t , and then calculates the reward rt for the model using the validation set Dtvalid of D t and the saved old task data in the replay buffer T . Fourth , it uses the reward to update the agent via reinforcement learning . The above four steps are repeated to train the agent I iterations or times on task t until it converges . After the agent is well trained , we select the continual learner with the highest reward as the final model for task t. This model not only performs well on the current task but also maintains the performance of the old tasks . After learning task t , the validation dataset Dtvalid and the parameters of f ( · ; θt ) are saved in the replay buffer T and the knowledge repository K respectively . Note that at any time , we only keep the trained model of the current continual learner for the last task .

Summary: The paper focuses on transfer of knowledge across tasks in a continual learning setting without catastrophic forgetting. The proposed method dynamically builds neural network structures. For each new task an agent samples a sequence of actions that integrates knowledge transfer from the old tasks to construct the continual learner. Old data are stored in a replay buffer, which are used for reward calculation rather than in training. The model thus incrementally learns a sequence of $N$ tasks. The continual learner is trained in a reinforcement learning setting where the reward is a combination of performance increase compared to a previous iteration and the forgetting rate of all previous tasks promoting the accuracy on all previous tasks.