Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning

1 Introduction . This paper studies continual learning ( CL ) of a sequence of natural language processing ( NLP ) tasks in the task continual learning ( Task-CL ) setting . It aims to ( i ) prevent catastrophic forgetting ( CF ) , and ( ii ) transfer knowledge across tasks . ( ii ) is particularly important because many tasks in NLP share similar knowledge that can be leveraged to achieve better accuracy . CF means that in learning a new task , the existing network parameters learned for the previous tasks may be modified , which degrades the performance of previous tasks [ 40 ] . In the Task-CL setting , the task id is provided for each test case in testing so that the specific model for the task in the network can be applied to classify the test case . Another popular CL setting is class continual learning , which does not provide the task id during testing but it is for solving a different type of problems . Most existing CL papers focus on dealing with CF [ 21 , 5 ] . There are also some papers that perform knowledge transfer . To achieve both objectives is highly challenging . To overcome CF in the Task-CL setting , we don ’ t want the training of the new task to update the model parameters learned for previous tasks to achieve model separation . But to transfer knowledge across tasks , we want the new task to leverage the knowledge learned from previous tasks for learning a better model ( forward transfer ) and also want the new task to enhance the performance of similar previous tasks ( backward transfer ) . ∗Work was done prior to joining Amazon . 2The code of CTR can be found at https : //github.com/ZixuanKe/PyContinual 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . This means it is necessary to update previous model parameters . This is a dilemma . Although several papers have tried to deal with both [ 22 , 37 ] , they were only tested using sentiment analysis tasks with strong shared knowledge . When tested with tasks that don ’ t have much shared knowledge , they suffer from severe CF ( see Sec . 5.4 ) . Those existing papers that focus on dealing with CF do not do well with knowledge transfer as they have no explicit mechanism to facilitate the transfer . Another observation about the current CL research is that most techniques do not use pre-trained models . But such pre-trained models or feature extractors can significantly improve the CL performance [ 18 , 24 ] . An important question is how to make the best use of pre-trained models in CL . This paper studies the problem as well using NLP tasks , but we believe that the developed ideas are also applicable to computer vision tasks because most pre-trained models are based on the transformer architecture [ 60 ] . We will see that the naive or the conventional way of directly adding the CL module on top of a pre-trained model is not the best choice ( see Sec . 5.4 ) . In NLP , fine-tuning a BERT [ 8 ] like pre-trained language model has been regarded as one of the most effective techniques in applications [ 65 , 57 ] . However , fine-tuning works poorly for continual learning . This is because the fine-tuned BERT for a task captures highly task-specific information [ 41 ] , which is difficult to be used by other tasks . When fine-tuning for a new task , it has to update the already fine-tuned parameters for previous tasks , which causes serious CF ( see Sec . 5.4 ) . This paper proposes a novel neural architecture to achieve both CF prevention and knowledge transfer , which also deals with the CF problem with BERT fine-tuning . The proposed system is called CTR ( Capsules and Transfer Routing for continual learning ) . CTR inserts a continual learning plug-in ( CL-plugin ) module in two locations in BERT . With the pair of CL-plugin modules added to BERT , we no longer need to fine-tune BERT for each task , which causes CF in BERT , and yet we can achieve the power of BERT fine-tuning . CTR has some similarity to Adapter-BERT [ 16 ] , which adds adapters in BERT for parameter efficient transfer learning such that different end tasks can have their separate adapters ( which are very small in size ) to adapt BERT for individual end tasks and to transfer the knowledge from BERT to the end tasks . Then , there is no need to employ a separate BERT and fine-tuning it for each task , which is extremely parameter inefficient if many tasks need to be learned . An adapter is a simple 2-layer fully-connected network for adapting BERT to a specific end task . A CL-plugin is very different from an adapter . We do not use a pair of CL-plugin modules to adapt BERT for each task . Instead , CTR learns all tasks using only one pair of CL-plugin modules inserted into BERT . A CL-plugin is a full CL network that can leverage a pre-trained model and deal with both CF and knowledge transfer . Specifically , it uses a capsule [ 15 ] to represent each task and a proposed transfer routing algorithm to identify and transfer knowledge across tasks to achieve improved accuracy . It further learns and uses task masks to protect task-specific knowledge to avoid forgetting . Empirical evaluations show that CTR outperforms strong baselines . Ablation experiments have also been conducted to study where to insert the CL-plugin module in BERT in order to achieve the best performance ( see Sec . 5.4 ) . 2 Related Work . Catastrophic Forgetting : Existing work in CL mainly focused on overcoming CF using the following approaches . ( 1 ) Regularization-based approaches , such as those in [ 27 , 30 , 51 , 69 ] , add a regularization in the loss to consolidate weights for previous tasks when learning a new task . ( 2 ) Replay-based approaches , such as those in [ 45 , 36 , 4 , 63 ] , retain some training data of old tasks and use them in learning a new task . The methods in [ 54 , 20 , 47 , 14 ] learn data generators and generate old task data for learning a new task . ( 3 ) Parameter isolation-based approaches , such as those in [ 52 , 21 , 39 , 10 ] , allocate model parameters dedicated to different tasks and mask them out when learning a new task . ( 4 ) Gradient projection-based approaches [ 68 ] ensure the gradient updates occur only in the orthogonal direction to the input of old tasks and thus will not affect old tasks . Some recent papers used pre-trained models [ 18 , 23 , 24 ] and learn one class per task [ 18 ] . Tackling CF only deals with model deterioration . These methods perform worse than learning each task separately . An empirical study of the cause of CF and the impact of task similarity on CF was done in [ 44 ] . Some NLP applications have also dealt with CF . For example , CL models have been proposed for sentiment analysis [ 23 , 24 , 37 , 43 ] , dialogue slot filling [ 53 ] , language modeling [ 58 , 7 ] , language learning [ 31 ] , sentence embedding [ 33 ] , machine translation [ 25 ] , cross-lingual modeling [ 35 ] , and question answering [ 12 ] . A dialogue CL dataset is also reported in [ 38 ] . Knowledge Transfer : Ideally , learning from a sequence of tasks should also allow multiple tasks to support each other via knowledge transfer . CAT [ 21 ] ( a Task-CL system ) works on a mixed sequence of similar and dissimilar tasks and can transfer knowledge among similar tasks detected automatically . Progressive Network [ 48 ] does forward transfer but it is for class continual learning ( Class-CL ) . Knowledge transfer in this paper is closely related to lifelong learning ( LL ) , which aims to improve the new/last task learning without handling CF [ 56 , 49 , 5 ] . In the NLP area , NELL [ 3 ] performs LL information extraction , and several other papers worked on lifelong document sentiment classification ( DSC ) and aspect sentiment classification ( ASC ) . [ 6 ] and [ 61 ] proposed two Naive Bayesian methods to help improve the new task learning . [ 64 ] proposed a LL approach based on voting . [ 55 ] used LL for aspect extraction . [ 43 ] and [ 62 ] used neural networks for DSC and ASC , respectively . Several papers also studied lifelong topic modeling [ 5 , 13 ] . However , all these works do not deal with CF . SRK [ 37 ] and KAN [ 22 ] try to deal with both CF and knowledge transfer in continual sentiment classification . However , they have two critical weaknesses : ( i ) Their RNN architectures can not use plug-in or adapter modules to tune BERT , which significantly limits their power . ( ii ) Since they were mainly designed for knowledge transfer , they suffer from serious CF ( see Sec . 5.4 ) . B-CL [ 24 ] uses the adapter idea [ 16 ] to adapt BERT for sentiment analysis tasks , which are similar to each other . However , since its mechanism of dynamic routing for knowledge transfer is very week , its knowledge transfer ability is markedly poorer than CTR ( see Sec . 5.4 ) . CLASSIC [ 23 ] is another recent work on continual learning for knowledge transfer , but its CL setting is domain continual learning . Its knowledge transfer method is based on contrastive learning . AdapterFusion [ 42 ] used adapters proposed in [ 16 ] . It proposes a two-stage method to learn a set of tasks . In the first stage , it learns one adapter for each task independently using the task ’ s training data . In the second stage , it uses the training data again to learn a good composition of the learned adapters in the first stage to produce the final model for all tasks . AdapterFusion basically tries to improve multi-task learning . It is not for continual learning and thus has no CF . As explained in Sec . 1 , the CL-plugin concept in CTR is different from that of adapters for adapting BERT for each task . CL-plugins are continual learning systems that make use of a pre-trained model . 3 CTR Architecture . This section describes the general architecture of CTR . The details about its key component CLplugin is presented in the next section . Due to its good performance , BERT [ 8 ] and its transformer [ 60 ] architecture are used as the base in our model CTR . Since BERT fine-tuning is prone to CF ( Sec . 1 ) , we propose the CL plug-in idea , which is inspired by Adapter-BERT [ 16 ] . CL-plugin is a full continual learning module designed to interact with a pre-trained model , in our case , BERT . Inserting CL-plugins in BERT . A commonly used method of leveraging a pre-trained model is to add the end task module on top of the pre-trained model . However , as explained in Sec . 1 , fine-tuning the pre-trained model can cause serious CF for CL . The CL system PCL [ 18 ] , which uses this approach , has the pre-trained model frozen to avoid forgetting . But as we will see in Sec . 5.4 , this is not the best choice . CTR inserts the proposed CL-plugin in two locations in BERT , i.e. , in each transformer layer of BERT . We will also see in Sec . 5.4 that inserting only one CL-plugin in one location is sub-optimal . Figure 1 gives the CTR architecture and we can see the two CL-plugins are added into BERT . In learning , only the two CL-plugins and the classification heads are trained . The components of the original pre-trained BERT are fixed . Continual learning plug-in ( CL-plugin ) . CL-plugin employs a capsule network ( CapsNet ) [ 15 , 50 ] like architecture . In the classic neural network , a neuron outputs a scalar , real-valued activation as a feature detector . CapsNets replaces that with a vector-output capsule to preserve additional information . A simple CapsNet consists of two capsule layers . The first layer stores low-level feature maps , and the second layer generates the classification probability with each capsule corresponding to one class . CapsNet uses a dynamic routing algorithm to make each lower-level capsule to send its output to a similar ( or “ agreed ” , computed by dot product ) higher-level capsule . This property can already be used to group similar tasks and their shareable features to produce a CL system ( see the ablation study in Sec . 5.4 ) . One of the key ideas of CL-plugin ( see Figure 2 ( A ) ) is a transfer capsule layer with a new transfer routing algorithm to explicitly identify transferable features/knowledge from previous tasks to transfer to the new task . Additionally , transfer routing avoids the need for hyper-parameter tuning on the number of iterations of dynamic routing [ 50 ] to update the agreements .

Summary: This work studies the continual learning of a sequence of NLP tasks. A novel model called AFK is proposed, which contains a series of hand-crafted modules to prevent forgetting and help with knowledge transfer. A number of baselines are compared with, and strong CL performance are shown in the experiments.