Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning

1 Introduction . This paper studies continual learning ( CL ) of a sequence of natural language processing ( NLP ) tasks in the task continual learning ( Task-CL ) setting . It aims to ( i ) prevent catastrophic forgetting ( CF ) , and ( ii ) transfer knowledge across tasks . ( ii ) is particularly important because many tasks in NLP share similar knowledge that can be leveraged to achieve better accuracy . CF means that in learning a new task , the existing network parameters learned for the previous tasks may be modified , which degrades the performance of previous tasks [ 40 ] . In the Task-CL setting , the task id is provided for each test case in testing so that the specific model for the task in the network can be applied to classify the test case . Another popular CL setting is class continual learning , which does not provide the task id during testing but it is for solving a different type of problems . Most existing CL papers focus on dealing with CF [ 21 , 5 ] . There are also some papers that perform knowledge transfer . To achieve both objectives is highly challenging . To overcome CF in the Task-CL setting , we don ’ t want the training of the new task to update the model parameters learned for previous tasks to achieve model separation . But to transfer knowledge across tasks , we want the new task to leverage the knowledge learned from previous tasks for learning a better model ( forward transfer ) and also want the new task to enhance the performance of similar previous tasks ( backward transfer ) . ∗Work was done prior to joining Amazon . 2The code of CTR can be found at https : //github.com/ZixuanKe/PyContinual 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . This means it is necessary to update previous model parameters . This is a dilemma . Although several papers have tried to deal with both [ 22 , 37 ] , they were only tested using sentiment analysis tasks with strong shared knowledge . When tested with tasks that don ’ t have much shared knowledge , they suffer from severe CF ( see Sec . 5.4 ) . Those existing papers that focus on dealing with CF do not do well with knowledge transfer as they have no explicit mechanism to facilitate the transfer . Another observation about the current CL research is that most techniques do not use pre-trained models . But such pre-trained models or feature extractors can significantly improve the CL performance [ 18 , 24 ] . An important question is how to make the best use of pre-trained models in CL . This paper studies the problem as well using NLP tasks , but we believe that the developed ideas are also applicable to computer vision tasks because most pre-trained models are based on the transformer architecture [ 60 ] . We will see that the naive or the conventional way of directly adding the CL module on top of a pre-trained model is not the best choice ( see Sec . 5.4 ) . In NLP , fine-tuning a BERT [ 8 ] like pre-trained language model has been regarded as one of the most effective techniques in applications [ 65 , 57 ] . However , fine-tuning works poorly for continual learning . This is because the fine-tuned BERT for a task captures highly task-specific information [ 41 ] , which is difficult to be used by other tasks . When fine-tuning for a new task , it has to update the already fine-tuned parameters for previous tasks , which causes serious CF ( see Sec . 5.4 ) . This paper proposes a novel neural architecture to achieve both CF prevention and knowledge transfer , which also deals with the CF problem with BERT fine-tuning . The proposed system is called CTR ( Capsules and Transfer Routing for continual learning ) . CTR inserts a continual learning plug-in ( CL-plugin ) module in two locations in BERT . With the pair of CL-plugin modules added to BERT , we no longer need to fine-tune BERT for each task , which causes CF in BERT , and yet we can achieve the power of BERT fine-tuning . CTR has some similarity to Adapter-BERT [ 16 ] , which adds adapters in BERT for parameter efficient transfer learning such that different end tasks can have their separate adapters ( which are very small in size ) to adapt BERT for individual end tasks and to transfer the knowledge from BERT to the end tasks . Then , there is no need to employ a separate BERT and fine-tuning it for each task , which is extremely parameter inefficient if many tasks need to be learned . An adapter is a simple 2-layer fully-connected network for adapting BERT to a specific end task . A CL-plugin is very different from an adapter . We do not use a pair of CL-plugin modules to adapt BERT for each task . Instead , CTR learns all tasks using only one pair of CL-plugin modules inserted into BERT . A CL-plugin is a full CL network that can leverage a pre-trained model and deal with both CF and knowledge transfer . Specifically , it uses a capsule [ 15 ] to represent each task and a proposed transfer routing algorithm to identify and transfer knowledge across tasks to achieve improved accuracy . It further learns and uses task masks to protect task-specific knowledge to avoid forgetting . Empirical evaluations show that CTR outperforms strong baselines . Ablation experiments have also been conducted to study where to insert the CL-plugin module in BERT in order to achieve the best performance ( see Sec . 5.4 ) . 2 Related Work . Catastrophic Forgetting : Existing work in CL mainly focused on overcoming CF using the following approaches . ( 1 ) Regularization-based approaches , such as those in [ 27 , 30 , 51 , 69 ] , add a regularization in the loss to consolidate weights for previous tasks when learning a new task . ( 2 ) Replay-based approaches , such as those in [ 45 , 36 , 4 , 63 ] , retain some training data of old tasks and use them in learning a new task . The methods in [ 54 , 20 , 47 , 14 ] learn data generators and generate old task data for learning a new task . ( 3 ) Parameter isolation-based approaches , such as those in [ 52 , 21 , 39 , 10 ] , allocate model parameters dedicated to different tasks and mask them out when learning a new task . ( 4 ) Gradient projection-based approaches [ 68 ] ensure the gradient updates occur only in the orthogonal direction to the input of old tasks and thus will not affect old tasks . Some recent papers used pre-trained models [ 18 , 23 , 24 ] and learn one class per task [ 18 ] . Tackling CF only deals with model deterioration . These methods perform worse than learning each task separately . An empirical study of the cause of CF and the impact of task similarity on CF was done in [ 44 ] . Some NLP applications have also dealt with CF . For example , CL models have been proposed for sentiment analysis [ 23 , 24 , 37 , 43 ] , dialogue slot filling [ 53 ] , language modeling [ 58 , 7 ] , language learning [ 31 ] , sentence embedding [ 33 ] , machine translation [ 25 ] , cross-lingual modeling [ 35 ] , and question answering [ 12 ] . A dialogue CL dataset is also reported in [ 38 ] . Knowledge Transfer : Ideally , learning from a sequence of tasks should also allow multiple tasks to support each other via knowledge transfer . CAT [ 21 ] ( a Task-CL system ) works on a mixed sequence of similar and dissimilar tasks and can transfer knowledge among similar tasks detected automatically . Progressive Network [ 48 ] does forward transfer but it is for class continual learning ( Class-CL ) . Knowledge transfer in this paper is closely related to lifelong learning ( LL ) , which aims to improve the new/last task learning without handling CF [ 56 , 49 , 5 ] . In the NLP area , NELL [ 3 ] performs LL information extraction , and several other papers worked on lifelong document sentiment classification ( DSC ) and aspect sentiment classification ( ASC ) . [ 6 ] and [ 61 ] proposed two Naive Bayesian methods to help improve the new task learning . [ 64 ] proposed a LL approach based on voting . [ 55 ] used LL for aspect extraction . [ 43 ] and [ 62 ] used neural networks for DSC and ASC , respectively . Several papers also studied lifelong topic modeling [ 5 , 13 ] . However , all these works do not deal with CF . SRK [ 37 ] and KAN [ 22 ] try to deal with both CF and knowledge transfer in continual sentiment classification . However , they have two critical weaknesses : ( i ) Their RNN architectures can not use plug-in or adapter modules to tune BERT , which significantly limits their power . ( ii ) Since they were mainly designed for knowledge transfer , they suffer from serious CF ( see Sec . 5.4 ) . B-CL [ 24 ] uses the adapter idea [ 16 ] to adapt BERT for sentiment analysis tasks , which are similar to each other . However , since its mechanism of dynamic routing for knowledge transfer is very week , its knowledge transfer ability is markedly poorer than CTR ( see Sec . 5.4 ) . CLASSIC [ 23 ] is another recent work on continual learning for knowledge transfer , but its CL setting is domain continual learning . Its knowledge transfer method is based on contrastive learning . AdapterFusion [ 42 ] used adapters proposed in [ 16 ] . It proposes a two-stage method to learn a set of tasks . In the first stage , it learns one adapter for each task independently using the task ’ s training data . In the second stage , it uses the training data again to learn a good composition of the learned adapters in the first stage to produce the final model for all tasks . AdapterFusion basically tries to improve multi-task learning . It is not for continual learning and thus has no CF . As explained in Sec . 1 , the CL-plugin concept in CTR is different from that of adapters for adapting BERT for each task . CL-plugins are continual learning systems that make use of a pre-trained model . 3 CTR Architecture . This section describes the general architecture of CTR . The details about its key component CLplugin is presented in the next section . Due to its good performance , BERT [ 8 ] and its transformer [ 60 ] architecture are used as the base in our model CTR . Since BERT fine-tuning is prone to CF ( Sec . 1 ) , we propose the CL plug-in idea , which is inspired by Adapter-BERT [ 16 ] . CL-plugin is a full continual learning module designed to interact with a pre-trained model , in our case , BERT . Inserting CL-plugins in BERT . A commonly used method of leveraging a pre-trained model is to add the end task module on top of the pre-trained model . However , as explained in Sec . 1 , fine-tuning the pre-trained model can cause serious CF for CL . The CL system PCL [ 18 ] , which uses this approach , has the pre-trained model frozen to avoid forgetting . But as we will see in Sec . 5.4 , this is not the best choice . CTR inserts the proposed CL-plugin in two locations in BERT , i.e. , in each transformer layer of BERT . We will also see in Sec . 5.4 that inserting only one CL-plugin in one location is sub-optimal . Figure 1 gives the CTR architecture and we can see the two CL-plugins are added into BERT . In learning , only the two CL-plugins and the classification heads are trained . The components of the original pre-trained BERT are fixed . Continual learning plug-in ( CL-plugin ) . CL-plugin employs a capsule network ( CapsNet ) [ 15 , 50 ] like architecture . In the classic neural network , a neuron outputs a scalar , real-valued activation as a feature detector . CapsNets replaces that with a vector-output capsule to preserve additional information . A simple CapsNet consists of two capsule layers . The first layer stores low-level feature maps , and the second layer generates the classification probability with each capsule corresponding to one class . CapsNet uses a dynamic routing algorithm to make each lower-level capsule to send its output to a similar ( or “ agreed ” , computed by dot product ) higher-level capsule . This property can already be used to group similar tasks and their shareable features to produce a CL system ( see the ablation study in Sec . 5.4 ) . One of the key ideas of CL-plugin ( see Figure 2 ( A ) ) is a transfer capsule layer with a new transfer routing algorithm to explicitly identify transferable features/knowledge from previous tasks to transfer to the new task . Additionally , transfer routing avoids the need for hyper-parameter tuning on the number of iterations of dynamic routing [ 50 ] to update the agreements .

Summary: This paper studies the problem of continual learning in a popular NLP task, text classification.  It proposes a framework called AFK, that utilizes capsule networks to dynamically route task-specific knowledge among task-shared knowledge. The author claims two major benefits of doing this: (1) this essentially establishes an attention effect that allows the model to more effectively conduct forward transfer, and (2) the assignment of new parameters for new tasks and the masking mechanism can mitigate catastrophic forgetting. Overall, the application of capsule network in continual learning is novel and the framework obtains good performance. However, I feel that more comprehensive analysis could further supported the presentation, and some could be critical for understanding these models (see below).