Venue: NAACL 2022

# Parameter-efficient Continual Learning Framework in Industrial Real-time Text Classification System

Tao Zhu, Zhe Zhao , Weijie Liu, Jiachi Liu, Yiren Chen, Weiquan Mao, Haoyan Liu, Kunbo Ding, Yudong Li, and Xuefeng Yang

Tencent Research, Beijing, China

{mardozhu, nlpzhezahao, jaegerliu, jiachiliu, yjrenchen, weiquanmao, haoyanliu, karlding, yudongli, ryanxfyang}@tencent.com

Corresponding author.

###### Abstract

Catastrophic forgetting is a challenge for model deployment in industrial real-time systems, which requires the model to quickly master a new task without forgetting the old one. Continual learning aims to solve this problem; however, it usually updates all the model parameters, resulting in extensive training times and the inability to deploy quickly. To address this challenge, we propose a parameter-efficient continual learning framework, in which efficient parameters are selected through an offline parameter selection strategy and then trained using an online regularization method. In our framework, only a few parameters need to be updated, which not only alleviates catastrophic forgetting, but also allows the model to be saved with the changed parameters instead of all parameters. Extensive experiments are conducted to examine the effectiveness of our proposal. We believe this paper will provide useful insights and experiences on developing deep learning-based online real-time systems.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

In industry, many text-related applications have enjoyed a superior performance boost from the emerging of pre-trained language models, such as word2vec Mikolov et al. (2013); Ma et al. (2017) , ELMo Peters et al. (2018), GPT Radford et al. (2018, 2019), and BERT Devlin et al. (2019); Liu et al. (2019); Sun et al. (2019). However, when a fine-tuned model needs to be updated to master a new task swiftly, it usually loses the ability to handle previous tasks. This phenomenon is known as catastrophic forgetting French (1999), and it poses a significant issue in industrial settings.

Continual learning aims to incrementally expand acquired knowledge for future learning Chen and Liu (2018), and mitigate the impact of catastrophic forgetting in the meantime. Existing continual learning methods usually use data replay Rebuffi et al. (2017), parameter isolation Rusu et al. (2016); Fernando et al. (2017), and regularization Kirkpatrick et al. (2017); Li and Hoiem (2017) to make models adapt to new tasks without catastrophic forgetting. However, these approaches lack research on implementing continual learning in industrial scenarios, where endowing models with continual learning capabilities meets numerous practical constraints. For time constraints, when new data or tasks arrive, the model should be launched in minutes or even seconds, which is common in time-sensitive scenarios, e.g., blocking certain rumors content. For space constraints, the strong demand for tracing tasks makes it necessary to save every model once it is changed. So for the current large-scale pre-trained models, storage becomes an industrial challenge with the increase of new tasks.

To solve these industrial challenges, we propose a parameter-efficient continual learning framework based on an offline parameter selection strategy. The framework consists of two parts, i.e., offline calculation and online training. In the offline calculation part, all the parameters that are important to the old task are selected to be fixed, while the remaining parameters are employed to learn the new task. Since in a real industrial scenario, the arrival of a new task will have an interval of hours or even days, we can make full use of this interval to advance the selection of parameters. During the online training phase, the model is parameter- efficiently trained on a new task within a small set of parameters and further combines multiple regularization-based methods Kirkpatrick et al. (2017); Li and Hoiem (2017) to overcome catastrophic forgetting. To alleviate storage costs, we only save the modified parameters for each snapshot. Extensive experiments demonstrate that our framework can maintain the old task performance while learning a new task quickly. Our implementa

tion is based on UER-py pre-training toolkit1(Zhao et al., 2019).

Footnote 1: [https://github.com/dibiir/UER-py/](https://github.com/dibiir/UER-py/)

The main contributions of this paper can be summarized as follows:

* We are the first to explore continual learning with only a few model parameters, and show that updating \(0.1\%\) parameters of BERT can achieve competitive performance.
* We propose a parameter-efficient continual learning framework that solves issues in real-world industrial settings by utilizing parameter-efficient-based offline parameter selection strategies and regularization-based online training methods.
* Extensive experiments on a real-world domain incremental text classification task verify the effectiveness of our proposed framework.

## 2 Related Work

### Continual Learning

The major challenge of continual learning is catastrophic forgetting McCloskey and Cohen (1989); French (1999), which occurs when optimizing for a new task causes performance degradation on a task learned previously. Methods designed to mitigate catastrophic forgetting mainly fall into three categories: replay methods, parameter isolation methods, and regularization-based methods Delange et al. (2021).

**Replay methods** explicitly retrain on a subset of stored old task samples while training on new tasks. Instead of selecting samples at random, Rebuffi et al. (2017b) incorporated the Herding technique Welling (2009) to choose samples that best approximate the mean feature vector of a class, and it is widely used in Castro et al. (2018), Wu et al. (2019), Hou et al. (2019), Zhao et al. (2020), Mi et al. (2020a, b). Ramalho and Garnelo (2019) proposed to store samples that the model is least confident. However, replay methods exploit samples from old tasks, which will slow down the online training. To meet the time constraint, they are not used in our framework.

**Parameter isolation methods** dedicate different model parameters to each task, which are divided in two directions. One is growing a new branch network for a new task, while freezing previous task parameters Rusu et al. (2016); Xu and Zhu (2018). The other one is masking out parameters of previous task during new task training, which is imposed either at parameters level Fernando et al. (2017); Mallya and Lazebnik (2018), or unit level Serra et al. (2018). Parameter isolation is unsuitable for usage in industrial scenario. It is difficult to keep track of the model's scale if the number of used parameters is continually accumulated as the number of tasks increases.

**Regularization-based methods** add an additional regularization term in the loss function, which will consolidate previous knowledge when learning on new data Delange et al. (2021). Elastic weight consolidation EWC Kirkpatrick et al. (2017) is a well-known regularization-based method, which introduces network parameter uncertainty in the Bayesian framework. LwF Li and Hoiem (2017) is another regularization method, using the previous model to infer current data and taking the outputs as soft labels to mitigate forgetting and transfer knowledge.

### Parameter-efficient Training

Training a model with a few parameters is useful in many applications. Not only does the model have the potential to achieve better performance, but also disk space can be saved by only saving the updated parameters for each task. Recent work has shown that it is possible to update only a small subset of the model's parameters during training. This kind of work could heavily alleviate storage and deployment communication requirements . For example, Adapters Houlsby et al. (2019); Rebuffi et al. (2017a); Bapna et al. (2019) introduce additional trainable parameters into a pre-trained model in the form of small task-specific modules while the rest of the model's parameters are kept fixed. Many works like Diff Pruning Guo et al. (2020) and BitFit Ben Zaken et al. (2021) have shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements. Xu et al. (2021) and Sung et al. (2021) even show a acceptable performance on random selection of parameters. Therefore, we choose to perform parameter-efficient training on continual learning to quickly master a new task while avoid catastrophic forgetting.

## 3 Methodology

We introduce a parameter-efficient continual learning framework, as shown in figure 1. Our frame 

work is divided into two components. The first is offline computation, which makes use of the interval between tasks to evaluate the data and select the parameters that are crucial to old tasks. These parameters are kept fixed in the new task. The other part is online training. In this stage, we utilize parameter-efficient training to perform new task training on the parameters that are not fixed. In addition, we introduce some well-known regularization-based methods in our framework for further improvement.

### Offline calculation

The goal of offline calculation is to select the subset of parameters that are (in some sense) the most important to all of the old tasks, and fix them. Therefore, we make full use of the interval between tasks to review the previous training data, and calculate the parameters that are important to the previous tasks in the latest model (snapshot). These parameters will be fixed in the new task, and the remaining parameters will participate in the training.

As for the method of measuring the importance of parameters, we consider the indicator of how much changing the parameter will impact the model's output. The Fisher information is particularly well suited to identifying the highly relevant subset of parameters for previous tasks. It serves as an useful tool for estimating how much information a random variable contains about a parameter of the distribution (Tu et al., 2016). The Fisher information assumes that the more important the parameter towards the target task, the higher value it conveys. Formally, the Fisher information for the parameter \(\theta_{i}\) is as follows:

\[F\left(\theta_{i}\right)=\frac{1}{|D|}\sum_{j=1}^{|D|}\left(\frac{a\partial \log p\left(y_{j}|x_{j};\theta\right)}{\partial\theta_{i}}\right)^{2} \tag{1}\]

where \(D\) denotes the task-specific training data, \(x\) and \(y\) denote the input and the output respectively.

### Parameter-efficient Online training

Given the important parameters selected in offline stage, we use the remaining parameters for online training. Updating subset of the parameters can avoid catastrophic forgetting to some extent and largely decrease the storage space required by the snapshot. In addition, we combine the parameter-efficient training with two typical regularization-based continual learning methods. The combination of multiple orthogonal techniques can further improve the performance of our system. The overview optimization objective is as follows:

\[L(\theta^{*})=L_{CE}(\theta^{*})+L_{regul-based}(\theta^{*}),\theta^{*}\in S_{\theta} \tag{2}\]

where \(S_{\theta}\) are parameters selected from the off-line calculation stage, which have small Fisher values. \(L_{CE}\) denotes the cross-entropy loss, and \(L_{regul-based}\) denotes regularization-based method loss.

#### 3.2.1 Parameter-efficient learning

As shown in figure 2, in traditional training setting (left), all of the model's parameters are updated. But in our online training (right), we only train a

Figure 1: The overall architecture of the parameter-efficient continual learning framework.

Figure 2: Illustration of parameter-efficient training.

 

few parameters, and most of the parameters are fixed according to the result of offline calculation to avoid forgetting old tasks. In general, the model will be easy to forget old tasks while learning new tasks, if more parameters are updated. Therefore, in our framework, we choose to perform parameter-efficient training on parameters that are not important to previous tasks, which are not fixed. Refer to previous experience (Ben Zaken et al., 2021; Xu et al., 2021), we chose a layer-wise strategy to select important parameters. We fixed the parameters from large to small (Fisher information) in a certain proportion at each layer.

On the other hand, updating a small number of parameters is beneficial for storage purpose and rapid deployment. Sometimes we need to deploy our model on thousands of servers. So the model size needs to be as small as possible (around 1.2 GB for BERT-Large 400 MB for BERT-Base). Our framework only needs to store the values and indices denoting the position of the updated parameters. Our experimental results demonstrate that only 0.1% trainable parameters of the original model can achieve competitive performance.

#### 3.2.2 Regularization-based method

EWC and LwF are two representative approaches for preventing catastrophic forgetting in neural networks. They respectively add restrictions on model parameters and output activation. The two methods are orthogonal and we combine them as follows:

\[L_{regul-based}(\theta)=\lambda_{1}L_{EWC}(\theta)+\lambda_{2}L_{LwF}(\theta) \tag{3}\]

Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) introduces network parameter uncertainty in the Bayesian framework. Intuitively, this approach consists of a \(L2\) penalty on the difference between the parameters for the old \(\theta^{*}_{i}\) (\(i\) denotes the indexes of the parameters) and the new \(\theta_{i}\). It uses the diagonal of the Fisher information matrix \(F_{i}\) (2) to weight different parameters. The EWC loss (4) slows down the learning process of task-relevant parameters, which contains knowledge learned previously.

\[L_{EWC}(\theta)=\sum_{i}F_{i}(\theta_{i}-\theta^{*}_{i})^{2} \tag{4}\]

From formula (4), we can see that if most of the parameters are fixed, it is equivalent to reducing the EWC loss, which is beneficial to preventing catastrophic forgetting.

Learning without forgetting (LwF) (Li and Hoiem, 2017) is another method for continual learning. Before training the new task, network outputs for the new task data are recorded, which is denoted by \(y^{\prime}_{o}\). It will be subsequently used during training to distill prior task knowledge. LwF employs a variant of knowledge distillation. In our framework, we use \(L2\) (5) loss to regulate the outputs:

\[L_{LwF}(\theta)=\sum_{k}(y_{o}-y^{\prime}_{o})^{2} \tag{5}\]

## 4 Experiment

In this section, we empirically verify the effectiveness and efficiency of our framework under the setting of incremental text classification tasks in the industrial scenarios.

### Dataset & Implementation Details

In the experiments, we utilize the Amazon Reviews dataset (He and McAuley, 2016) to examine our method, which is widely used in text classification tasks. The original dataset contains 142.8 million product reviews collecting from 29 different domains. To be consistent with our industrial scenario, we firstly create a reduced dataset by randomly selecting 10000 pieces of data from 12 domains as the base task, 6000 of which are used as training set, 2000 as validation set, 2000 as test set. In the incremental learning session, we construct 17 subsequent tasks with 300 examples for training, 100 for validation and 100 for test from the rest domains.

We adopt the BERT-Base model (Devlin et al., 2019) and the BERT default uncased vocabulary. All runs use the AdamW optimizer2Kingma and Ba, 2014; Devlin et al., 2019) with 5 epochs, 32 batch size and 0.1 dropout rate. For the base task, we set learning rate as 2e-5. For the incremental tasks, we set learning rate as 2e-3, \(\lambda_{1}\) as 0.35 and \(\lambda_{2}\) as 1. Based on offline calculation results, we only pick up 0.1% parameters to participate in each training process. Our experiments are conducted in Intel(R) Xeon(R) CPU E5-2699 v4 at 2.20GHz, 2 Nvidia Tesla P40 GPU with 24 GB of RAM.

 



### Models

We compare our proposed models with the a series of baseline methods in our experiments:

* lower-bound: a standard classification model is fine-tuned on the individual task without any continual learning strategy, which can be considered as the lower-bound method.
* upper-bound: a model is trained on all tasks simultaneously, which can be considered as the upper-bound method since it has access to the whole dataset.
* EWC & LwF: Two classical regularization-based methods for continual learning.
* PE-rand: Our proposal model is trained by randomly choosing some parameters and keeping them unchanged during online training stage, instead of using the offline calculation strategy.
* PE-*: Our continual learning framework, including offline calculation and parameter-efficient online training.

### Results

The models are trained on the current training set and evaluated on the union of all the test sets. To ensure the robustness of the task ordering, we evaluate our methods on the four different orderings (chosen randomly), which are shown in Appendix A.

Table 1 provides a summary of our main results. We report the micro-averaged accuracy for the classification task. The lower bound is trained in the current task without using any continual learning strategy to overcome catastrophic forgetting, while the upper bound is trained on all data after the new task comes, which can be considered multi-task method. There is a significant gap between the lower bound and the upper bound, which illustrates the need for continual learning. As the classical CL methods, EWC and LwF outperform the standard model without any specific continual learning, but still suffer from catastrophic forgetting in the order IV. It can be seen that our proposed PE-* achieves a better performance than EWC, LwF and their combination. This is because most of the parameters in BERT are fixed, which is equivalent to posing a strict regularization to the parameters to prevent catastrophic forgetting while using the remaining parameters to learn new tasks. Compared to PE-rand, PE-* has a better average accuracy, which verifies the importance of parameter selections. Although the random selection method outperforms PE-* in order I, it is difficult to obtain a suitable set of parameters in most cases for models to learn new tasks while maintaining previous knowledge.

Moreover, according to the principles of EWC and LwF, the former records the initial model parameters, and the latter records the data features of new tasks. As the training progresses, their regular loss terms especially \(L_{EWC}(\theta)\) will get bigger and bigger in model like BERT-Base with 110M parameters. What's more, in real industrial scenarios, each new task may have different suitable hyper parameters. We do not have time to do grid search of the best hyper parameters, so \(\lambda_{EWC}\) and \(\lambda_{LwF}\) may not be optimal solutions. This results in an unbalanced ratio of \(L(\theta)\) to \(L_{EWC}(\theta)\) and \(L_{LwF}(\theta)\), where \(L(\theta)\) may much smaller than \(L_{EWC}(\theta)\) and \(L_{LwF}(\theta)\). Therefore, as the number of tasks increases, the training of new tasks will become more and more difficult with the same set of hyperparameters. However, the previous tasks have not been fully learned. This problem accumulates gradually in regularization-based method and leads to results that are not as good as our method (PE-*) which just handles a very small amount parameters.

Figure 3 shows the accuracy of model on the first task test set as the model are trained on more tasks. The figure illustrates how well each model retains its previously acquired knowledge as it learns new knowledge. We can see that our framework is con

\begin{table}
\begin{tabular}{c|l l l l l|r r} \hline Order \(\downarrow\sim\text{Method}\rightarrow\) & EWC & LwF & EWC\&LwF & PE-rand & PE-* & Lower Bound & Upper Bound \\ \hline I & 47.39 & 47.6 & 45.76 & **50.6** & 50.46 & 45.14 & 51.59 \\ II & 48.54 & 47.95 & 48.3 & 51.35 & **51.63** & 45.81 & 53.18 \\ III & 40.44 & 45.23 & 42.11 & 45.42 & **49.51** & 38.29 & 51.1 \\ IV & 43.27 & 45.16 & 41.04 & 42.07 & **46.8** & 41.27 & 54.78 \\ \hline Average Acc & 44.91 & 46.49 & 43.30 & 47.36 & **49.6** & 42.63 & 52.66 \\ \hline \end{tabular}
\end{table}
Table 1: Main result of text classification (above) averaged accuracy score respectively (see Appendix A for the dataset orderings).



sistently better and more stable compared to other methods.

### Parameter-efficient Strategy

Figure 4 shows the weight map of Fisher information. It can be seen that in the BERT model, the parameters of the embedding-layer have little effect on our classification task. Most of the important parameters are concentrated in the transformer block layer, and the importance of the attention layer is higher than that of the feed forward layer. In addition, according to our statistics, we found that 13%(about 14M) of the parameters' Fisher information is 0, and most of them are in embedding-layer.

In our experiments, we found that the parameters to be fixed cannot be determined simply in order of magnitude. According to Jawahar et al. (2019), BERT encodes rich linguistic information in different transformer blocks. Therefore, refer to previous experience Ben Zaken et al. (2021); Xu et al. (2021), we chose a layer-wise strategy to select important parameters. We fixed the parameters from large to small in a certain proportion at each layer of the model. To this end, each layer has parameters for new tasks to learn.

### Model Size

We set layer-wise parameter reserved for new task to different values, 0.5%, 1%, 5% and 10%, and the percent of parameters fixed for old task are 99.5%, 99%, 95% and 90%. The advantage of our parameter-efficient continual learning becomes more pronounced at extreme sparsity rates. In Table 2, we report the accuracy across different task orders and reserved rates. We can observe that the more parameters fixed, the better the effect on alleviating catastrophic forgetting.

In the above experiments, we only trained 0.1% of the parameters in the BERT model. We use the sparse-matrix method to store the model, and only store the index and value each time, occupying about 1.2 Mb of space, which is 0.3% of the entire model, as shown in Table 3. Parameter-efficient strategy greatly saves network bandwidth and storage requirements.

## 5 Conclusion

This paper introduces a parameter-efficient continual learning framework, which is designed for real-time incremental learning system. In offline stage, the framework identifies the parameters that are less important to the old tasks. By updating these

\begin{table}
\begin{tabular}{c|c c c c} \hline Reserved Percent & Order1 & Order2 & Order3 & Order4 \\ \hline
0.50\% & **50.46** & **51.63** & **49.51** & 46.8 \\
1.00\% & 50.04 & 51.63 & 49.05 & **47.34** \\
5.00\% & 45.11 & 48.61 & 46.44 & 44.43 \\
10.00\% & 39.08 & 38.36 & 34.38 & 37.61 \\ \hline \end{tabular}
\end{table}
Table 2: The effect of layer-wise parameter reserved percentage on accuracy.

Figure 4: Calculating the model Fisher information by review old tasks, where dark colors are weighted more than light colors. The layers of BERT are ordered from bottom to top (i.e. the embedding layer is shown at the top).

\begin{table}
\begin{tabular}{c|c|c|c} \hline Method & Acc & Saved Parameters & Saved model size \\ \hline EWC & 47.49 & 110B & 421MB \\ LwF & 47.95 & 110B & 421MB \\ EWC\&LwF & 48.65 & 110B & 421MB \\ PE-* & **51.63** & 0.1B & 1.2MB \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of storage costs.

Figure 3: Performance on the first task test examples of order I during training as training progresses.

 

parameters in online training stage, the model is able to learn new tasks in short time without forgetting the old ones. Furthermore, we surprisingly find that decent results can be achieved by only training a small subset of parameters (e.g. 0.1%). This observation enables us to largely decrease the storage of the snapshot, which is important for the system requiring frequent update.

## References

* A. Bapna, N. Arivazhagan, and O. Firat (2019)Simple, scalable adaptation for neural machine translation. arXiv preprint arXiv:1909.08478. Cited by: SS3.5.
* E. B. Zaken, S. Ravfogel, and Y. Goldberg (2021)Bitfit: simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv e-prints, pp. arXiv-2106. Cited by: SS1.
* F. M. Castro, M. J. Marin-Jimenez, N. Guil, C. Schmid, and K. Alahari (2018)End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248. Cited by: SS1.
* Z. Chen and B. Liu (2018)Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning12 (3), pp. 1-207. External Links: ISSN 0021-909X, Link, Document Cited by: SS1.
* M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars (2021)A continual learning survey: defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Vol. 4171-4186, pp. 4171-4186. Cited by: SS1.
* C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra (2017)PathNet: evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734. Cited by: SS1.
* R. M. French (1999)Catastrophic forgetting in connectionist networks. Trends in cognitive sciences3 (4), pp. 128-135. Cited by: SS1.
* D. Guo, A. M. Rush, and Y. Kim (2020)Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463. Cited by: SS1.
* R. He and J. McAuley (2016)Ups and downs: modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517. Cited by: SS1.
* S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin (2019)Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839. Cited by: SS1.
* N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019)Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. Cited by: SS1.
* G. Jawahar, B. Sagot, and D. Seddah (2019)What does BERT learn about the structure of language?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3651-3657. Cited by: SS1.
* D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: SS1.
* J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017)Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences114 (13), pp. 3521-3526. Cited by: SS1.
* Z. Li and D. Hoitem (2017)Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence40 (12), pp. 2935-2947. Cited by: SS1.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1.
* A. Mallya and S. Lazebnik (2018)Packnet: adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773. Cited by: SS1.
* M. McCloskey and N. J. Cohen (1989)Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, Vol. 24, pp. 109-165. Cited by: SS1.
* F. Mi, L. Kong, T. Lin, Kaicheng Yu, and Boi Faltings (2020)Generalized class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 240-241. Cited by: SS1.
* F. Mi, X. Lin, and B. Faltings (2020)AdER: adaptively distilled exemplar replay towards continual learning for session-based recommendation. In 

Fourteenth ACM Conference on Recommender Systems, pp. 408-413. Cited by: SS2.
* T. Mikolov, K. Chen, G. Corrado, and J. Dean (2013)Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. Cited by: SS5.
* T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean (2013)Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems26. Cited by: SS2.
* M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer (2018)Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227-2237. Cited by: SS2.
* A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018)Improving language understanding by generative pre-training. External Links: Link Cited by: SS2.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)Language models are unsupervised multitask learners. OpenAI blog1 (8), pp. 9. Cited by: SS2.
* T. Ramalho and M. Garnelo (2019)Adaptive posterior learning: few-shot learning with a surprise-based memory module. arXiv preprint arXiv:1902.02527. Cited by: SS2.
* S. Rebuffi, H. Bilen, and A. Vedaldi (2017)Learning multiple visual domains with residual adapters. arXiv preprint arXiv:1705.08045. Cited by: SS2.
* S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert (2017)ICARI: incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010. Cited by: SS2.
* A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell (2016)Progressive neural networks. arXiv preprint arXiv:1606.04671. Cited by: SS2.
* J. Serra, D. Suris, M. Miron, and A. Karatzoglou (2018)Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548-4557. Cited by: SS2.
* Y. Sun, S. Wang, Y. Li, S. Feng, X. Chen, H. Zhang, X. Tian, D. Zhu, H. Tian, and H. Wu (2019)Ernie: enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. Cited by: SS2.
* Yi-Lin Sung, Varun Nair, and Colin A. Raffel (2021)Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems34. Cited by: SS2.
* M. Tu, V. Berisha, M. Woolf, J. Seo, and Y. Cao (2016)Ranking the parameters of deep neural networks using the fisher information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2647-2651. Cited by: SS2.
* M. Welling (2009)Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1121-1128. Cited by: SS2.
* Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu (2019)Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374-382. Cited by: SS2.
* J. Xu and Z. Zhu (2018)Reinforced continual learning. arXiv preprint arXiv:1805.12369. Cited by: SS2.
* R. Xu, F. Luo, Z. Zhang, C. Tan, B. Chang, S. Huang, and F. Huang (2021)Raise a child in large language model: towards effective and generalizable fine-tuning. arXiv preprint arXiv:2109.05687. Cited by: SS2.
* B. Zhao, X. Xiao, G. Gan, B. Zhang, and S. Xia (2020)Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13208-13217. Cited by: SS2.
* Zhe Zhao, H. Chen, J. Zhang, W. Xin Zhao, T. Liu, W. Lu, X. Chen, H. Deng, Q. Ju, and X. Du (2019)UCF: an open-source toolkit for pre-training models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pp. 241-246. Cited by: SS2.
* Z. Zhao, T. Liu, S. Li, B. Li, and X. Du (2017)Ngram2vec: learning improved word representations from ngram co-occurrence statistics. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 244-253. Cited by: SS2.

## Appendix A Task order

We use the following task orders (chosen randomly) for text classification:

* Kindle_Store\(\rightarrow\)Arts_Crafts_and_Sewing\(\rightarrow\)Electronics\(\rightarrow\)Magazine_Subscriptions\(\rightarrow\)Pet_Supplies\(\rightarrow\)Sports_and_Outdoors\(\rightarrow\)Prime_Pantry\(\rightarrow\)Office_Products 

* \(\rightarrow\) Movies_and_TV\(\rightarrow\) Automotive\(\rightarrow\) CDs_and_Vinyl\(\rightarrow\) Gift_Cards\(\rightarrow\) Digital_Music\(\rightarrow\) Clothing_Shoes_and_Jewelry\(\rightarrow\) Home_and_Kitchen\(\rightarrow\) Software\(\rightarrow\) Grocery_and_Gourmet_Food
* Pet_Supplies\(\rightarrow\) Gift_Cards\(\rightarrow\) Electronics\(\rightarrow\) Prime_Pantry\(\rightarrow\) Office_Products\(\rightarrow\)Digital_Music\(\rightarrow\) Magazine_Subscriptions\(\rightarrow\)Home_and_Kitchen\(\rightarrow\) CDs_and_Vinyl\(\rightarrow\) Grocery_and_Gourmet_Food
* Digital_Music\(\rightarrow\) Arts_Crafts_and_Sewing\(\rightarrow\) Office_Products\(\rightarrow\)Magazine_Subscriptions\(\rightarrow\)Kindle_Store\(\rightarrow\)Software\(\rightarrow\)Automotive\(\rightarrow\)Prime_Pantry\(\rightarrow\) Grocery_and_Gourmet_Food\(\rightarrow\)Movies_and_TV\(\rightarrow\)Electronics\(\rightarrow\)Home_and_Kitchen\(\rightarrow\)Pet_Supplies\(\rightarrow\)CDs_and_Vinyl\(\rightarrow\)Clothing_Shoes_and_Jewelry\(\rightarrow\)Gift_Cards
* Magazine_Subscriptions\(\rightarrow\) Sports_and_Outdoors\(\rightarrow\)Digital_Music\(\rightarrow\)Electronics\(\rightarrow\)Prime_Pantry\(\rightarrow\)CDs_and_Vinyl\(\rightarrow\)Grocery_and_Gourmet_Food\(\rightarrow\)Home_and_Kitchen\(\rightarrow\)Software\(\rightarrow\)Arts_Crafts_and_Sewing\(\rightarrow\)Clothing_Shoes_and_Jewelry\(\rightarrow\)Pet_Supplies\(\rightarrow\)Office_Products\(\rightarrow\)Kindle_Store\(\rightarrow\)Gift_Cards