Venue: NAACL 2021

# Continual Learning for Text Classification with Information Disentanglement Based Regularization

 Yufan Huang, Yanzhe Zhang\({}^{*1}\), Jiaao Chen, Xuezhi Wang\({}^{2}\), Diyi Yang

Georgia Institute of Technology, \({}^{1}\)Zhejiang University, \({}^{2}\)Google

{yhuang704, jiaaochen, dyang888}@gatech.edu

\({}^{1}\)z_yanzhe@zipu.edu.cn, \({}^{2}\)xuezhiw@google.com

Equal contribution.

###### Abstract

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an _information disentanglement_ based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at [https://github.com/GT-SALT/IDBR](https://github.com/GT-SALT/IDBR).

## 1 Introduction

Computational systems in real world scenarios face changing environment frequently, and thus are often required to learn continually from dynamic streams of data building on what was learnt before Biesialska et al. (2020). For example, a tweeter classifier needs to deal with trending topics which are constantly emerging. While being an intrinsic nature of human to continually acquire and transfer knowledge throughout lifespans, most machine learning models often suffer from _catastrophic forgetting_: when learning on new tasks, models dramatically and rapidly forget knowledge from previous tasks McCloskey and Cohen (1989). As a result, Continual Learning (CL) Ring (1998); Thrun (1998) has received more attention recently as it can enable models to perform positive transfer Perkins et al. (1992) as well as remember previously seen tasks.

A growing body of research has been conducted to equip neural networks with the ability of continual learning abilities Kirkpatrick et al. (2017); Lopez-Paz and Ranzato (2017); Aljundi et al. (2018). Existing continual learning methods on NLP tasks can be broadly categorized into two classes: purely _replay_ based methods de Masson d'Autume et al. (2019); Sun et al. (2019) where examples from previous tasks are stored and re-trained during the learning of the new task to retain old information, and _regularization_ based methods Wang et al. (2019); Han et al. (2020) where constraints are added on model parameters to prevent them from changing too much while learning new tasks. The former usually stores an extensive amount of data from old tasks de Masson d'Autume et al. (2019) or trains language models based on task identifiers to generate sufficient examples Sun et al. (2019), which significantly increases memory costs and training time. While the latter utilizes previous examples efficiently via the constraints added on text hidden space or model parameters, it generally views them as equally important and regularize them to the same extent Wang et al. (2019); Han et al. (2020), making it hard for models to differentiate informative representation that needs to be retained from ones that need a large degree of updates. However, we argue that when learning new tasks, task generic information and task specific information should be treated differently, as these generic representation might function consistently while task specific representations might need to be changed significantly.

To this end, we propose an information disentanglement based regularization method for continual learning on text classification. Specifically, we first disentangle the text hidden representation

space (e.g., the output representation of BERT Devlin et al. (2019)) into a **task generic** space and a **task specific** space using two auxiliary tasks: next sentence prediction for learning task generic information and task identifier prediction for learning task specific representations. When training on new tasks, we constrain the task generic representation to be relatively stable and representations of task specific aspects to be more flexible. To further alleviate catastrophic forgetting without much increases of memory and training time, we propose to augment our regularization-based methods by storing and replaying only a small amount of representative examples (e.g., 1% samples selected by memory selection rules like K-Means MacQueen et al. (1967)). To sum up, our contributions are threefold:

* We propose an information disentanglement based regularization method for continual text classification, to better learn and constrain task generic and task specific knowledge.
* We augment the regularization approach with a memory selection rule that requires only a small amount of replaying examples.
* Extensive experiments conducted on five benchmark datasets demonstrate the effectiveness of our proposed methods compared to state-of-the-art baselines.

## 2 Related work

Continual LearningExisting continual learning research can be broadly divided into four categories: (i) replay-based method, which remind models of information from seen tasks via experience replay de Masson d'Autume et al. (2019), distillation Rebuffi et al. (2017), representation alignment Wang et al. (2019) or optimization constraints Lopez-Paz and Ranzato (2017); Chaudhry et al. (2019) using examples sampled from previous tasks Rebuffi et al. (2017); de Masson d'Autume et al. (2019) or synthesized with generative models Shin et al. (2017); Sun et al. (2019); (ii) regularization-based method, which constrains model's output Li and Hoiem (2018), hidden space Rannen et al. (2017), or parameters Lopez-Paz and Ranzato (2017); Zenke et al. (2017); Aljundi et al. (2018) from changing too much to retain learned knowledge; (iii) architecture-based method, where different tasks are associated with different components of the overall model to directly minimize the interference between new tasks and old tasks Rusu et al. (2016); Mallya and Lazebnik (2018); (iv) meta-learning-based method, which directly optimizes the knowledge transfer among tasks Riemer et al. (2019); Obamuyide and Vlachos (2019), or learns robust data representations Javed and White (2019); Holla et al. (2020); Wang et al. (2020) to alleviate forgetting.

Among these different approaches, replay-based methods and regularization-based methods have been widely applied to NLP tasks to enable large pre-trained models Devlin et al. (2019); Radford et al. (2019) to continually acquire novel world knowledge from streams of textual data without forgetting the already learned knowledge. For instance, replaying examples have shown promising performance for text classification de Masson d'Autume et al. (2019); Sun et al. (2019); Holla et al. (2020), relation extraction Wang et al. (2019) and question answering de Masson d'Autume et al. (2019); Sun et al. (2019); Wang et al. (2020). However, they often suffer from large memory costs or considerable training time, due to the requirements of storing an extensive amount of texts de Masson d'Autume et al. (2019) or training language models to generate a sufficient number of examples Sun et al. (2019). Recently, regularization-based methods Wang et al. (2019); Han et al. (2020) have also been applied to directly constrain knowledge deposited in model parameters without abundant rehearsal examples. Despite better efficiency compared to replay-based methods, current regularization-based approaches often fail to generalize well to new tasks as they treat and constrain all the information _equally_ and thus limit the needed updates for parameters that are specific to different tasks. To overcome these limitations, we propose to first distinguish hidden spaces that need to be retained from those that need to be updated substantially through information disentanglement, and then regularize different spaces _separately_, to better remember previous knowledge as well as transfer to new tasks. In addition, we enhance our regularization method by replaying only a limited amount of examples selected by K-means as the memory selection rule.

Textual Information DisentanglementOur work is related to information disentanglement for text data, which has been extensively explored in generation tasks like style transfer Fu et al. (2017); Zhao et al. (2018); Romanov et al. (2019); Li 

et al., 2020), where text hidden representations are often disentangled into sentiment Fu et al. (2017); John et al. (2019), content Romanov et al. (2019); Bao et al. (2019) and syntax Bao et al. (2019) information through supervised learning from pre-defined labels John et al. (2019) or unsupervised learning with adversarial training Fu et al. (2017); Li et al. (2020). Building on these prior works, we differentiate task generic space from task specific space via supervision from two simple yet effective auxiliary tasks: next sentence prediction and task identifier prediction.

Related Learning ParadigmsThere exists some other learning paradigms also dealing with multiple tasks, such as multi-task learning Yu et al. (2020) and transfer learning Houlsby et al. (2019); Pfeiffer et al. (2021). However, neither can fit in the scenario of learning multiple tasks sequentially. The former could be adapted to dynamic environments by storing all seen training data and retraining the model after the arrival of new tasks, which highly decreases efficiency and is impractical in deployment. The latter only focuses on the target tasks and ignores catastrophic forgetting on the source tasks. A more thorough discussion can be found in Biesialska et al. (2020).

## 3 Problem Formulation

In this work, we focus on continual learning for a sequence of text classification tasks \(\{T_{1},...T_{n}\}\), where we learn a model \(f_{\theta}(.)\), \(\theta\) is a set of parameters shared by all tasks and each task \(T_{i}\) contains a different set of sentence-label training pairs, \((x^{i}_{1:m},y^{i}_{1:m})\). After learning all tasks in the sequence, we seek to minimize the generalization error on all tasks Biesialska et al. (2020) :

\[R(f_{\theta})=\sum_{i=1}^{n}\mathbb{E}_{(x^{i},y^{i})\sim T_{i}}\mathcal{L}( f_{\theta}(x^{i}),y^{i})\]

We use two commonly-used techniques for this problem setting in our proposed model:

* **Regularization**: in order to preserve knowledge stored in the model, regularization is a constraint added to model output Li and Hoiem (2018), hidden space Zenke et al. (2017) and parameters Lopez-Paz and Ranzato (2017); Zenke et al. (2017); Aljundi et al. (2018) to prevent them from changing too much while learning new tasks.
* **Replay**: when learning new tasks, Experience Rebuffin et al. (2017) is commonly used to recover knowledge from previous tasks, where a memory buffer is first adopted to store seen examples from previous tasks and then the stored data is replayed with the training set for the current task. Formally, after training on task \(t-1~{}(t\geq 2)\), \(\gamma|S_{t-1}|\) examples are _randomly sampled_ from the \(t\)-th training set \(S_{t-1}\) into the memory buffer \(\mathcal{M}\), where \(0\leq\gamma\leq 1\) is the store ratio. Data from \(\mathcal{M}\) is then merged with the \(t\)-th training set \(S_{t}\) when learning from task \(t\).

## 4 Method

In continual learning, the model needs to adapt to new tasks quickly while maintaining the ability to recover information from previous tasks, hence not all information stored in the hidden representation space should be treated equally. In previous work like style transfer John et al. (2019) and controlled text generation Hu et al. (2017), certain information (such as content and syntax) is extracted and shared among different categories and other information (such as style and polarity) is manipulated for each specific category. Similarly, in our continual learning scenario, there is shared knowledge among different tasks as well while the model needs to learn and maintain specific knowledge for each individual task in the learning process. This key observation motivates us to propose an _information-disentanglement_ based regularization for continual text classification to retain shared knowledge while adapting specific knowledge to streams of tasks (Section 4.1). We also incorporate a small set of representative replay samples to alleviate catastrophic forgetting (Section 4.3). Our model architecture is shown in Figure 1.

### Information Disentanglement (ID)

This section describes how to disentangle sentence representations into task generic space and task specific space, and how separate regularizations are imposed on them for continual text classification. Formally, for a given sentence \(x\), we first use a multi-layer encoder \(B(.)\), e.g., BERT Devlin et al. (2019), to get the hidden representations \(r\) which contain both task generic and task specific information. Then we introduce two disentanglement networks \(G(.)\) and \(S(.)\) to extract the generic representation \(g\) and specific representation \(s\) from \(r\) 

For new tasks, we learn the classifiers by utilizing information from both spaces, and we allow different spaces to change to different extents to best retain knowledge from previous tasks.

Task Generic SpaceTask generic space is the hidden space containing information generic to different tasks in a task sequence. During switching from one task to another, the generic information should roughly remain the same, e.g., syntactic knowledge should not change too much across the learning process of a sequence of tasks. To extract task generic information \(g\) from hidden representations \(r\), we leverage the _next sentence prediction_ task Devlin et al. (2019) 1 to learn the generic information extractor \(G(.)\). More specifically, we insert a [SEP] token into each training example during tokenization to form a sequence pair labeled _IsNext_, and switch the first sequence and the second sequence to form a sentence pair labeled _NotNext_. In order to distinguish _IsNext_ pairs and _NotNext_ pairs, extractor \(G(.)\) needs to learn the context dependencies between two segments, which is beneficial to understand every example and generic to any individual task.

Footnote 1: Note that the word “sentence” here refers to an arbitrary span of continuous text Devlin et al. (2019), it could be several linguistic sentences or part of a linguistic sentence.

Denote \(\tilde{x}\) as the _NotNext_ example corresponding to \(x\) (_IsNext_), and \(l\in\{0,1\}\) as the label for next sentence prediction. We build a sentence relation predictor \(f_{nsp}\) on the generic feature extractor \(G(.)\):

\[\mathcal{L}_{nsp}=\mathbb{E}_{x\in S_{i}\cup\mathcal{M}}(\mathcal{ L}(f_{nsp}(G(B(x)),0)\\ +\mathcal{L}(f_{nsp}(G(B(\tilde{x})),1))\]

where \(\mathcal{L}\) is the cross entropy loss, \(\mathcal{M}\) is the memory buffer and \(S_{t}\) is the \(t\)-th training set.

Task Specific SpaceModels also need task specific information to perform well over each task. For example, on sentiment classification words like "good" or "bad" could be very informative, but they might not generalize well for tasks like topic classification. Thus we employ a simple task-identifier prediction task on the task specific representation \(s\), which means for any given example we want to distinguish which task this example belongs to. This simple auxiliary setup will encourage \(s\) to embed different information from different tasks. The loss for task-identifier predictor \(f_{task}\) is:

\[\mathcal{L}_{task}=\mathbb{E}_{(x,z)\in S_{t}\cup\mathcal{M}}\mathcal{L}(f_{ task}(S(B(x)),z)\]

where \(z\) is the corresponding task id for \(x\).

Text ClassificationTo adapt to the \(t\)-th task, we combine the task generic representation \(g=G(B(x)\) and task specific representation \(s=S(B(x))\) to perform text classification, where we minimize the cross entropy loss:

\[\mathcal{L}_{cls}=\mathbb{E}_{(x,y)\in S_{i}\cup\mathcal{M}}\mathcal{L}(f_{ cls}(g\circ s),y))\]

Here \(y\) is the corresponding class label for \(x\), \(f_{cls}(.)\) is the class predictor. \(\circ\) denotes the concatenation of the two representations.

Figure 1: Our proposed model architecture. We disentangle the hidden representation into a task generic space and a task specific space via different induction biases. When training on new tasks, different spaces are regularized separately. Also, a small portion of previous data is stored and replayed.

 



### ID Based Regularization

To further prevent severe distortion when training on new tasks, we employ regularization on both generic representations \(g\) and specific representations \(s\). Different from previous approaches Li et al. (2018); Wang et al. (2019) which treat all the spaces equally, we allow regularization to different extents on \(g\) and \(s\) as knowledge in different spaces should be preserved separately to encourage both more positive transfer and less forgetting. Specifically, before training all the modules on task \(t\), we first compute the generic representations and specific representations of all sentences \(x\) from the training set \(S_{t}\) of current task \(t\) and memory buffer \(\mathcal{M}_{t}\). Using the trained \(B^{t-1}(.)\), \(G^{t-1}(.)\) and \(S^{t-1}(.)\) from previous task \(t-1\), for each example \(x\) we calculate the generic representation as \(G^{t-1}(B^{t-1}(x))\), and the specific representation as \(S^{t-1}(B^{t-1}(x))\) to hoard the knowledge from previous models. The computed generic and specific representations are saved. During the learning from training pairs from task \(t\), we impose two regularization losses separately:

\[\begin{split}&\mathcal{L}_{reg}^{g}=\mathbb{E}_{x\in S_{i}\cup \mathcal{M}_{t}}\|G^{t-1}(B^{t-1}(x))-G(B(x))\|_{2}\\ &\mathcal{L}_{reg}^{s}=\mathbb{E}_{x\in S_{i}\cup\mathcal{M}_{ t}}\|S^{t-1}(B^{t-1}(x))-S(B(x))\|_{2}\end{split}\]

### Memory Selection Rule

Since we only store a small number of examples as a way to balance the replay as well as the extra memory cost and training time, we need to carefully select them in order to utilize the memory buffer \(\mathcal{M}\) efficiently. Considering that if two stored examples are very similar, then only storing one of them could possibly achieve similar results in the future. Thus, those stored examples should be as diverse and representative as possible. To this end, after training on \(t\)-th task, we employ K-means MacQueen et al. (1967) to cluster all the examples from current training set \(S_{t}\): For each \(x\in S_{t}\), we utilize its embedding \(B(x)\) as its input feature to conduct K-means. We set the numbers of clusters to \(\gamma|S_{t}|\) and only select the example closest to each cluster's centroid, following Wang et al. (2019); Han et al. (2020).

### Overall Objective

We can write the final objective for continual learning on text classification as the following:

\[\begin{split}\mathcal{L}=\mathcal{L}_{cls}+\mathcal{L}_{nsp}+ \mathcal{L}_{task}\\ +\lambda^{g}\mathcal{L}_{reg}^{g}+\lambda^{s}\mathcal{L}_{reg}^{ s}\end{split} \tag{1}\]

```
Input Training sets \(\{S_{1},...,S_{n}\}\), Replay Frequency \(\beta\), Store ratio \(\gamma\), Coefficients \(\lambda_{g},\lambda_{s}\) Output Optimal models \(B\), \(G\), \(S\), \(f_{nsp}\), \(f_{task}\), \(f_{cls}\) \(\mathcal{M}=\{\}\) \(\triangleright\) Initialize memory buffer  Initialize \(B\) using pretrained BERT  Initialize \(G,S,f_{nsp},f_{task},f_{cls}\) for\(t=1,\ldots,n\)doif\(t\geq 2\)then  Store \(G(B(x))\), \(S(B(x))\), \(\forall x\in S_{t}\cup\mathcal{M}\) for batches \(\in S_{t}\)do  Optimize \(\mathcal{L}\) in Equation 1 ifstep\(~{}~{}~{}\mod~{}\beta=0\)then\(\triangleright\) Replay  Sample \(t-1\) batches from \(\mathcal{M}\)  Optimize \(\mathcal{L}\) in Equation 1 endif endfor else\(\triangleright\) No regularization on 1st task forbatches \(\in S_{t}\)do  Optimize \(\mathcal{L}=\mathcal{L}_{cls}+\mathcal{L}_{nsp}+\mathcal{L}_{task}\) endfor endif \(\mathcal{C}\) = K-Means(\(S_{t}\), \(n_{clusters}\)=\(\gamma|S_{t}|\)) \(\triangleright\)\(\mathcal{C}\) :  centroid \(\mathcal{C}^{\prime}\) = { Examples closest to centers \(\in C\) } \(\mathcal{M}\leftarrow\mathcal{M}\cup\mathcal{C}^{\prime}\)\(\triangleright\) Add to memory endfor return\(B\), \(G\), \(S\), \(f_{nsp}\), \(f_{task}\), \(f_{cls}\)
```

**Algorithm 1**IDBR

## 5 Experiment

### Datasets

Following MBPA++ de Masson d'Autume et al. (2019), we use five text classification datasets Zhang et al. (2015); Chen et al. (2020) to evaluate our methods, including AG News (news classification), Yelp (sentiment analysis), DBPedia (Wikipedia article classification), Amazon (sentiment analysis), and Yahoo! Answer (Q&A classification). A summary of the datasets is shown in Table 1. We merge the label space of Amazon and Yelp considering their domain similarity, with 33

classes in total.

### Experiment Setup

Due to the limitation of resources, for most of our experiments, we create a reduced dataset by randomly sampling 2000 training examples and 2000 validation examples per class for every task. See Table 1 for the train/test size of each dataset. We name this setting Setting (Sampled). We tune all the hyperparameters on the basis of Setting (Sampled). Beyond that, to have a comparison with previous State-of-the-art, we also conduct experiments on the same training set and test set as MbPA++ de Masson d'Autume et al. (2019) and LAMOL Sun et al. (2019), which contains 115,000 training examples and 7,600 test examples for each task. For every task, we randomly hold out 500 examples per class from training examples for validation purpose. We name the latter Setting (Full). During training, we evaluate our model on validation sets from all seen tasks, following Kirkpatrick et al. (2017).

Our experiments are mainly conducted on the task sequences shown in Table 2. To minimize the effect of task order and task sequence length on the results, we examine both length-3 task sequences and length-5 task sequences in various orders. The first 3 task sequences are a cyclic shift of ag \(\rightarrow\) yahoo, which are three classification tasks in different domains (news classification, sentiment analysis, Q&A classification). The last four length-5 task sequences follows de Masson d'Autume et al. (2019).

### Baselines

We compare our proposed model with the following baselines in our experiments:

* **Finetune**Yogatama et al. (2019): fine-tune BERT model sequentially without the episodic memory module and any other loss.
* **Replay**Wang et al. (2019); de Masson d'Autume et al. (2019): **Finetune** model augmented with an episodic memory. Replay examples from old tasks while learning new tasks.
* **Regularization**: On top of **Replay**, with an L2 regularization term added on the hidden state of the classifier following BERT.
* **MBPA++**de Masson d'Autume et al. (2019): augment BERT model with an episodic memory module and store **all** seen examples. MBPA++ performs experience replay at training time, and uses K-nearest neighbors to select examples for local adaptation at test time.
* **LAMOL**Sun et al. (2019): train a language model that simultaneously learns to solve the tasks and generate training samples, the latter is for generating pseudo samples used in experience replay. Here the text classification is performed in Q&A formats.
* **Multi-task Learning** (MTL): The model is trained on all tasks simultaneously, which can be considered as an upper-bound for continual learning methods since it has access to data from all tasks at the same time.

### Implementation Details

We use pretrained BERT-based-uncased from HuggingFace Transformers Wolf et al. (2020) as our base feature extractor. The task generic encoder and task specific encoder are both one linear layer followed by activation function \(Tanh\), their output size are both 128 dimensions. The predictors built on encoders are all one linear layer followed by activation function \(softmax\).

All experiments are conducted on NVIDIA RTX 2080 Ti with 11GB memory with the batch size of

\begin{table}
\begin{tabular}{l l l c r} \hline
**Dataset** & **Class** & **Type** & **Train** & **Test** \\ \hline AGNews & 4 & News & 8000 & 7600 \\ Yelp & 5 & Sentiment & 10000 & 7600 \\ Amazon & 5 & Sentiment & 10000 & 7600 \\ DBPedia & 14 & Wikipedia & 28000 & 7600 \\ Yahoo & 10 & Q\&A & 20000 & 7600 \\ \hline \end{tabular}
\end{table}
Table 1: Dataset statistics we used for Setting (Sampled). Type means the domain of task classification. Note that the size of the validation set is the same as the size of the training set.

\begin{table}
\begin{tabular}{l p{142.3pt}} \multicolumn{1}{c}{**Order**} & **Task Sequence** \\ \hline
1 & ag \textgreater{} yellow \textasciode{}y Sahoo \textasciode{}ag \\
2 & yelp \textasciode{}yahoo \textasciode{}ag \\
3 & graoo \textasciode{}ag \textasciode{}yelp \\
4 & ag \textgreater{} pelty \textasciodes{}amazon \textasciode{}yahoo \textasciode{}dbpedia \\
5 & yelo \textasciode{}yahoo \textasciode{}ag \textasciode{}amazon \textasciode{}dbpodia \textasciode{}ag \\
6 & dbpedia \textasciode{}yahoo \textasciode{}ag \textasciode{}amazon \textasciode{}yelp \\
7 & yelp \textasciode{}ag \textasciode{}dbpedia \textasciode{}amakazon \textasciode{}yahoo \\ \hline \end{tabular}
\end{table}
Table 2: Seven random different task sequences used for experiments. The first 6 are used in Setting (Sampled). The last 4 are used in Setting (Full).

 

8 and the maximum sequence length of 256 (use the first 256 tokens if one's length is beyond that). We use AdamW Loshchilov and Hutter, 2019) as optimizer. For all modules except the task id predictor, we set the learning rate \(lr=3e^{-5}\); for task id predictor, we set its learning rate \(l_{Task}=5e^{-4}\). The weight decay for all parameters are 0.01.

For experience replay, we set the store ratio \(\gamma=0.01\), i.e. we store 1% of seen examples into the episodic memory module. Besides, we set the replay frequency \(\beta=10\), which means we do experience replay once every ten steps.

For information disentanglement, we mainly tune the coefficients of the regularization loss. For batches from memory buffer \(\mathcal{M}\), we set \(\lambda^{g}\) to 2.5, select best \(\lambda^{s}\) from \(\{1.5,2.0,2.5\}\). For batches from current training set \(S\), we set \(\lambda^{g}\) to 0.25, select best \(\lambda^{s}\) from \(\{0.15,0.20,0.25\}\).

## 6 Results and Discussion

We evaluate models after training on all tasks and report their average accuracies on all test sets as our metric. Table 3 summarizes our results in Setting (Sampled). While continual finetuning suffered from severe forgetting, experience replay with 1% stored examples achieves promising results, which demonstrates the importance of experience replay for continual learning in NLP. Beyond that, simple regularization turns out to be a robust method on the basis of experience replay, which shows consistent improvements on all 6 orders. Our proposed Information Disentanglement Based Regularization (IDBR) further improves regularization consistently under all circumstances.

Table 4 compares IDBR with previous SOTA: MBPA++ and LAMOL in Setting (Full). Note that although we use the same training/testing data, there is some inherent differences between our settings and previous SOTA methods. Despite the fact that MBPA++ applies local adaptation when testing, IDBR still outperforms it by an obvious margin. We achieve comparative results with LAMOL, despite that LAMOL requires task identifiers during inference which makes its prediction task easier.

### Impact of the Lengths of Task Sequences

Comparing results of length-3 sequences and length-5 sequences in Table 3, we found that the gap between IDBR and multi-task learning became bigger when the length of task sequence changed from 3 to 5. To better understand how IDBR gradually forgot, we followed Chaudhry et al. (2018) to measure _forgetting_\(\mathcal{F}_{k}\) after trained on task \(k\) as follows:

\[\begin{split}&\mathcal{F}_{k}=\mathbb{E}_{j=1...t-1}f_{j}^{k},\\ & f_{j}^{k}=\max_{l\in\{1...k-1\}}a_{l,j}-a_{k,j}\end{split}\]

where \(a_{l,j}\) is the is the model's accuracy on task \(j\) after trained on task \(l\). On order 4, 5 and 6, we calculate the forgetting every time after IDBR was trained on a new task and summarize them in Table 5. For continual learning, we hypothesize that the model is prone to suffer from more severe forgetting as the task sequence becomes longer. We found that although there was some big drop after training on the 3rd task, IDBR maintained stable performance as the length of task sequence increased, especially after training on 4-th and 5-th task, the forgetting increment was relatively small, which demonstrated the robustness of IDBR.

### Visualizing Disentangled Spaces

To study whether our task generic encoder \(G\) tends to learn more generic information and task specific encoder \(S\) captures more task specific information, we used t-SNE van der Maaten and Hinton, 2008) to visualize the two hidden spaces of IDBR, using the final model trained on order 2, and the results are shown in Figure 2, where Figure 1(a) visualizes task generic space and Figure 1(b) visualizes task specific space. We observe that compared with task specific space, generic features from different tasks were more mixed, which demonstrates that the next sentence prediction helped task generic space to be more task-agnostic than task specific space, which was induced to learn separated representations for different tasks. Considering we only employed two simple auxiliary tasks, the effect of information disentanglement was noticeable.

Figure 2: t-SNE visualization of task generic hidden space and task specific hidden space of IDBR.

 



### Ablation Studies

Effect of DisentanglementIn order to demonstrate that each module of our information disentanglement helps the learning process, we performed ablation study on the two auxiliary tasks using order 5 as a case study. The results are summarized in Table 6. We found that both task-id prediction and next sentence prediction contribute to the final performance. Furthermore, the performance gain was much larger by combing these two auxiliary tasks together. Intuitively, the model needs both tasks to disentangle the representation well, since it is easy for the model to ignore one of the spaces if the constraint is not imposed appropriately. The results show that the two tasks are likely complimentary to each other in helping the model learn better disentangled representations.

Impact of RegularizationTo study the effect of regularization on task generic hidden space \(g\) and task specific hidden space \(s\), we performed an ablation study which only applied regularization on \(g\) or \(s\), and compared the results with regularization on both in Table 7. We found that regularization on both spaces results in a much better performance than regularization on one of them only, which demonstrates the necessity of both regulariz

\begin{table}
\begin{tabular}{l l} \hline
**Model** & **Accuracy** \\ \hline Regularization & 73.03 \\ IDBR w/o \(\mathcal{L}_{nsp}\) & 73.17 \\ IDBR w/o \(\mathcal{L}_{tsk}\) & 73.29 \\ IDBR & **73.72** \\ \hline \end{tabular}
\end{table}
Table 6: Comparison among using task-id prediction only, next sentence prediction only and both of them. All results are averaged over 3 runs.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline
**Model** & **TT** & **TI** & **LA** & **PM** & \multicolumn{4}{c}{**Length-5 Task Sequences**} \\ \hline
**Order** & & & & **4** & **5** & **6** & **7** & **Average** \\ MBPA++ † & & ✓ & BERT & 70.7 & 70.2 & 70.9 & 70.8 & 70.7 \\ MBPA++ † † & & ✓ & BERT & 74.9 & 73.1 & 74.9 & 74.1 & 74.3 \\ LAMOL ‡† & ✓ & ✓ & & GPT-2 & **76.1** & 76.1 & **77.2** & **76.7** & **76.5** \\ IDBR & ✓ & & & BERT & 75.9 & **76.2** & 76.4 & **76.7** & 76.3 \\ \hline  & & & & & & & & \\ \end{tabular}
\end{table}
Table 4: Summary of results on Setting (Full) using averaged accuracy after training on the last task. Our results are averaged over 2 runs. † means we fetch numbers from de Masson d’Autume et al. (2019). † means we fetch numbers from Sun et al. (2019). **TT**: whether task-id is available during training. **TI**: whether task-id is available during inference. **LA**: whether need local adaptation during inference. **PM**: pretrained models used for continual learning.

\begin{table}
\begin{tabular}{l l l l l} \hline
**Model** & **4** & **5** & **6** & **Avg** \\ \hline Reg only on \(s\) & 72.05 & 72.54 & 72.61 & 72.40 \\ Reg only on \(g\) & 72.01 & 72.98 & 72.73 & 72.57 \\ Reg on both & **72.63** & **73.72** & **73.23** & **73.19** \\ \hline \end{tabular}
\end{table}
Table 7: Comparison among using regularization on task specific space only, task generic space only and both of them. All results are averaged over 3 runs.

\begin{table}
\begin{tabular}{l l l l l l l l l} \hline
**Model** & **TT** & **TI** & **LA** & **PM** & \multicolumn{4}{c}{**Length-5 Task Sequences**} \\ \hline
**Order** & & & & & **4** & **5** & **6** & **7** & **Average** \\ MBPA++ † † & & & ✓ & BERT & 70.7 & 70.2 & 70.9 & 70.8 & 70.7 \\ MBPA++ † † † † & & & ✓ & BERT & 74.9 & 73.1 & 74.9 & 74.1 & 74.3 \\ LAMOL ‡�� & ✓ & ✓ & & GPT-2 & **76.1** & 76.1 & **77.2** & **76.7** & **76.5** \\ IDBR & ✓ & & & BERT & 75.9 & **76.2** & 76.4 & **76.7** & 76.3 \\ \hline \end{tabular}
\end{table}
Table 3: Summary of results on Setting (Sampled) using averaged accuracy after training on the last task. All results are averaged over 3 runs. The \(p\)-values of paired \(t\)-test between nine numbers of Regularization and IDBR are 0.018 on Length-3 and 0.009 on Length-5, demonstrating the significant differences.



ers. While we may expect to give more tolerance to specific space for changing, we found that no regularization on it would lead to severe forgetting of previously learnt good task specific embeddings, hence it is necessary to add a regularizer over this space as well. Beyond that, we also observed that under most circumstances, adding regularization on the task generic space \(g\) results in a more significant gain than adding regularization on the task specific space \(s\), consistent with our intuition that task generic space changes less across tasks and thus preserving it better helps more in alleviating catastrophic forgetting.

Impact of K-MeansTo demonstrate our hypothesis that when the memory budget is limited, selecting the most representative subset of examples is vital to the success of continual learning, we performed an ablation study on order 1,2,3 using IDBR with and without K-Means. The result is shown in Table 8. From the table, we found that using K-Means helps boost the overall performance. Specifically, the improvement brought by K-Means was larger on those challenging orders, i.e. orders on which IDBR had worse performance. This is because for these challenging orders, the forgetting is more severe and the model needs more examples from previous tasks to help it retain previous knowledge. Thus with the same memory budget constraint, diversity across saved examples will help the model better recover knowledge learned from previous tasks.

## 7 Conclusion

In this work, we introduce an information disentanglement based regularization (IDBR) method for continual text classification, where we disentangle the hidden space into task generic space and task specific space and further regularize them differently. We also leverage K-Means as the memory selection rule to help the model benefit from the augmented episodic memory module. Experiments conducted on five benchmark datasets demonstrate that IDBR achieves better performances compared to previous state-of-the-art baselines on sequences of text classification tasks with various orders and lengths. We believe the proposed approach can be extended to continual learning for other NLP tasks such as sequence generation and sequence labeling as well, and plan to explore them in the future.

## Acknowledgment

We would like to thank the anonymous reviewers for their helpful comments, and the members of Georgia Tech SALT group for their feedback.

## References

* Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoesiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 139-154.
* Bao et al. (2019) Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai, and Jiajun Chen. 2019. Generating sentences from disentangled syntactic and semantic spaces. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6008-6019, Florence, Italy. Association for Computational Linguistics.
* Biesialska et al. (2020) Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussa. 2020. Continual lifelong learning in natural language processing: A survey. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 6523-6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Chaudhry et al. (2018) Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2018. Riemannian walk for incremental learning: Understanding forgetting and intransignence. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
* Chaudhry et al. (2019) Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with a-GEM. In _International Conference on Learning Representations_.
* Chen et al. (2020) Jiaoo Chen, Zichao Yang, and Diyi Yang. 2020. MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2147-2157, Online. Association for Computational Linguistics.
* de Masson d'Autume et al. (2019) Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Goyal et al. (2017)

\begin{table}
\begin{tabular}{l l l l l} \hline
**Rules** & **1** & **2** & **3** & **Average** \\ \hline Random & 71.52 & 72.60 & 73.03 & 72.38 \\ K-Means & **71.80** & **72.72** & **73.08** & **72.53** \\ \hline \end{tabular}
\end{table}
Table 8: Comparison between different selection rules: select stored examples randomly or select by K-Means. All results are averaged over 3 runs.

 

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fu et al. (2017) Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2017. Style transfer in text: Exploration and evaluation. _arXiv preprint arXiv:1711.06861_.
* Han et al. (2020) Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Continual relation learning via episodic memory activation and reconsolidation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6429-6440, Online. Association for Computational Linguistics.
* Holla et al. (2020)Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020. Meta-learning with sparse experience replay for lifelong language learning. _CoRR_, abs/2009.04891.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 2790-2799. PMLR.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70 of _Proceedings of Machine Learning Research_, pages 1587-1596. PMLR.
* Javed and White (2019) Khurram Javed and Martha White. 2019. Meta-learning representations for continual learning. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* John et al. (2019) Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 424-434, Florence, Italy. Association for Computational Linguistics.
* Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. _Proceedings of the National Academy of Sciences_, 114(13):3521-3526.
* Li et al. (2020) Yuan Li, Chunyuan Li, Yizhe Zhang, Xiujun Li, Guoqing Zheng, Lawrence Carin, and Jianfeng Gao. 2020. Complementary auxiliary classifiers for label-conditional text generation. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8303-8310.
* Li and Hoitem (2018) Zhizhong Li and Derek Hoitem. 2018. Learning without forgetting. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 40(12):2935-2947.
* Lopez-Paz and Runzato (2017) David Lopez-Paz and Marc'Aurelio Ronzato. 2017. Gradient episodic memory for continual learning. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* MacQueen et al. (1967) James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_, volume 1, pages 281-297. Oakland, CA, USA.
* Mallya and Lazebnik (2018) Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to a single network by iterative pruning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 7765-7773.
* McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In _Psychology of leaning and motivation_, volume 24, pages 109-165. Elsevier.
* Obamuyide and Vlachos (2019) Abioda Obamuyide and Andreas Vlachos. 2019. Meta-learning improves lifelong relation extraction. In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_, pages 224-229.
* Perkins et al. (1992) David N Perkins, Gavriel Salomon, et al. 1992. Transfer of learning. _International encyclopedia of education_, 2:6452-6457.
* Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwaya Kamath, Andreas Ruckle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 487-503, Online. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Rannen et al. (2017) Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. 2017. Encoder based lifelong learning. In _Proceedings of the IEEE International Conference on Computer Vision_, pages 1320-1328 

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_, pages 2001-2010.
* Riemer et al. (2019) Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu,, and Gerald Tesauro. 2019. Learning to learn without forgetting by maximizing transfer and minimizing interference. In _International Conference on Learning Representations_.
* Ring (1998) Mark B Ring. 1998. Child: A first step towards continual learning. In _Learning to learn_, pages 261-292. Springer.
* Romanov et al. (2019) Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019. Adversarial decomposition of text representation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 815-825, Minneapolis, Minnesota. Association for Computational Linguistics.
* Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. _arXiv preprint arXiv:1606.04671_.
* Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In _Advances in Neural Information Processing Systems_, pages 2990-2999.
* Sun et al. (2019) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. In _International Conference on Learning Representations_.
* Thrun (1998) Sebastian Thrun. 1998. Lifelong learning algorithms. In _Learning to learn_, pages 181-209. Springer.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(86):2579-2605.
* Wang et al. (2019) Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sentence embedding alignment for lifelong relation extraction. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 796-806, Minneapolis, Minnesota. Association for Computational Linguistics.
* Wang et al. (2020) Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, and Jaime Carbonell. 2020. Efficient meta lifelong-learning with limited memory. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 535-548, Online. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remit Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Iernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Yogatama et al. (2019) Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. _arXiv preprint arXiv:1901.11373_.
* Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. In _Advances in Neural Information Processing Systems_, volume 33, pages 5824-5836. Curran Associates, Inc.
* Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. _Proceedings of machine learning research_, 70:3987.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in neural information processing systems_, pages 649-657.
* Zhao et al. (2018) Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. In _International Conference on Machine Learning_, pages 5902-5911. PMLR.