SlowGuess/NAACL_2021_OCR huggingface dataset:
Yufan Huang*, Yanzhe Zhang\*1, Jiao Chen, Xuezhi Wang2, Diyi Yang
Georgia Institute of Technology,  $^{1}$ Zhejiang University,  $^{2}$ Google
{yhuang704, jiaaochen, dyang888}@gatech.edu
$^{1}$ z_yanzhe@zju.edu.cn,  $^{2}$ xuezhiw@google.com
Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.
Computational systems in real world scenarios face changing environment frequently, and thus are often required to learn continually from dynamic streams of data building on what was learnt before (Biesialska et al., 2020). For example, a tweeter classifier needs to deal with trending topics which are constantly emerging. While being an intrinsic nature of human to continually acquire and transfer knowledge throughout lifespans, most machine learning models often suffer from catastrophic forgetting: when learning on new tasks, models dramatically and rapidly forget knowledge from previous tasks (McCloskey and Cohen, 1989). As a result, Continual Learning (CL) (Ring, 1998; Thrun,
1998) has received more attention recently as it can enable models to perform positive transfer (Perkins et al., 1992) as well as remember previously seen tasks.
A growing body of research has been conducted to equip neural networks with the ability of continual learning abilities (Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017; Aljundi et al., 2018). Existing continual learning methods on NLP tasks can be broadly categorized into two classes: purely replay based methods (de Masson d'Autume et al., 2019; Sun et al., 2019) where examples from previous tasks are stored and re-trained during the learning of the new task to retain old information, and regularization based methods (Wang et al., 2019; Han et al., 2020) where constraints are added on model parameters to prevent them from changing too much while learning new tasks. The former usually stores an extensive amount of data from old tasks (de Masson d'Autume et al., 2019) or trains language models based on task identifiers to generate sufficient examples (Sun et al., 2019), which significantly increases memory costs and training time. While the latter utilizes previous examples efficiently via the constraints added on text hidden space or model parameters, it generally views them as equally important and regularize them to the same extent (Wang et al., 2019; Han et al., 2020), making it hard for models to differentiate informative representation that needs to be retained from ones that need a large degree of updates. However, we argue that when learning new tasks, task generic information and task specific information should be treated differently, as these generic representation might function consistently while task specific representations might need to be changed significantly.
To this end, we propose an information disentanglement based regularization method for continual learning on text classification. Specifically, we first disentangle the text hidden representation
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2736-2746 June 6-11, 2021. ©2021 Association for Computational Linguistics
space (e.g., the output representation of BERT (Devlin et al., 2019)) into a task generic space and a task specific space using two auxiliary tasks: next sentence prediction for learning task generic information and task identifier prediction for learning task specific representations. When training on new tasks, we constrain the task generic representation to be relatively stable and representations of task specific aspects to be more flexible. To further alleviate catastrophic forgetting without much increases of memory and training time, we propose to augment our regularization-based methods by storing and replaying only a small amount of representative examples (e.g.,  $1\%$  samples selected by memory selection rules like K-Means (MacQueen et al., 1967)). To sum up, our contributions are threefold:
- We propose an information disentanglement based regularization method for continual text classification, to better learn and constrain task generic and task specific knowledge.
- We augment the regularization approach with a memory selection rule that requires only a small amount of replaying examples.
- Extensive experiments conducted on five benchmark datasets demonstrate the effectiveness of our proposed methods compared to state-of-the-art baselines.
Continual Learning Existing continual learning research can be broadly divided into four categories: (i) replay-based method, which remind models of information from seen tasks via experience replay (de Masson d'Autume et al., 2019), distillation (Rebuffi et al., 2017), representation alignment (Wang et al., 2019) or optimization constraints (Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019) using examples sampled from previous tasks (Rebuffi et al., 2017; de Masson d'Autume et al., 2019) or synthesized with generative models (Shin et al., 2017; Sun et al., 2019); (ii) regularization-based method, which constrains model's output (Li and Hoiem, 2018), hidden space (Rannen et al., 2017), or parameters (Lopez-Paz and Ranzato, 2017; Zenke et al., 2017; Aljundi et al., 2018) from changing too much to retain learned knowledge; (iii) architecture-based method, where different tasks are associated with different components of the overall model to directly
minimize the interference between new tasks and old tasks (Rusu et al., 2016; Mallya and Lazebnik, 2018); (iv) meta-learning-based method, which directly optimizes the knowledge transfer among tasks (Riemer et al., 2019; Obamuyide and Vlachos, 2019), or learns robust data representations (Javed and White, 2019; Holla et al., 2020; Wang et al., 2020) to alleviate forgetting.
Among these different approaches, replay-based methods and regularization-based methods have been widely applied to NLP tasks to enable large pre-trained models (Devlin et al., 2019; Radford et al., 2019) to continually acquire novel world knowledge from streams of textual data without forgetting the already learned knowledge. For instance, replaying examples have shown promising performance for text classification (de Masson d'Autume et al., 2019; Sun et al., 2019; Holla et al., 2020), relation extraction (Wang et al., 2019) and question answering (de Masson d'Autume et al., 2019; Sun et al., 2019; Wang et al., 2020). However, they often suffer from large memory costs or considerable training time, due to the requirements of storing an extensive amount of texts (de Masson d'Autume et al., 2019) or training language models to generate a sufficient number of examples (Sun et al., 2019). Recently, regularization-based methods (Wang et al., 2019; Han et al., 2020) have also been applied to directly constrain knowledge deposited in model parameters without abundant rehearsal examples. Despite better efficiency compared to replay-based methods, current regularization-based approaches often fail to generalize well to new tasks as they treat and constrain all the information equally and thus limit the needed updates for parameters that are specific to different tasks. To overcome these limitations, we propose to first distinguish hidden spaces that need to be retained from those that need to be updated substantially through information disentanglement, and then regularize different spaces separately, to better remember previous knowledge as well as transfer to new tasks. In addition, we enhance our regularization method by replaying only a limited amount of examples selected by K-means as the memory selection rule.
Textual Information Disentanglement Our work is related to information disentanglement for text data, which has been extensively explored in generation tasks like style transfer (Fu et al., 2017; Zhao et al., 2018; Romanov et al., 2019; Li
et al., 2020), where text hidden representations are often disentangled into sentiment (Fu et al., 2017; John et al., 2019), content (Romanov et al., 2019; Bao et al., 2019) and syntax (Bao et al., 2019) information through supervised learning from pre-defined labels (John et al., 2019) or unsupervised learning with adversarial training (Fu et al., 2017; Li et al., 2020). Building on these prior works, we differentiate task generic space from task specific space via supervision from two simple yet effective auxiliary tasks: next sentence prediction and task identifier prediction.
Related Learning Paradigms There exists some other learning paradigms also dealing with multiple tasks, such as multi-task learning (Yu et al., 2020) and transfer learning (Houlsby et al., 2019; Pfeiffer et al., 2021). However, neither can fit in the scenario of learning multiple tasks sequentially. The former could be adapted to dynamic environments by storing all seen training data and retraining the model after the arrival of new tasks, which highly decreases efficiency and is impractical in deployment. The latter only focuses on the target tasks and ignores catastrophic forgetting on the source tasks. A more thorough discussion can be found in Biesialska et al. (2020).
In this work, we focus on continual learning for a sequence of text classification tasks  $\{T_1,\dots T_n\}$  where we learn a model  $f_{\theta}(.)$ $\theta$  is a set of parameters shared by all tasks and each task  $T_{i}$  contains a different set of sentence-label training pairs,  $(x_{1:m}^{i},y_{1:m}^{i})$  . After learning all tasks in the sequence, we seek to minimize the generalization error on all tasks (Biesialska et al., 2020):
$$
R (f _ {\theta}) = \sum_ {i = 1} ^ {n} \mathbb {E} _ {(x ^ {i}, y ^ {i}) \sim T _ {i}} \mathcal {L} (f _ {\theta} (x ^ {i}), y ^ {i})
$$
We use two commonly-used techniques for this problem setting in our proposed model:
- Regularization: in order to preserve knowledge stored in the model, regularization is a constraint added to model output (Li and Hoiem, 2018), hidden space (Zenke et al., 2017) and parameters (Lopez-Paz and Ranzato, 2017; Zenke et al., 2017; Aljundi et al., 2018) to prevent them from changing too much while learning new tasks.
- Replay: when learning new tasks, Experience Replay (Rebuffi et al., 2017) is commonly used to recover knowledge from previous tasks, where a memory buffer is first adopted to store seen examples from previous tasks and then the stored data is replayed with the training set for the current task. Formally, after training on task  $t - 1$  ( $t \geq 2$ ),  $\gamma | S_{t - 1} |$  examples are randomly sampled from the  $t$ -th training set  $S_{t - 1}$  into the memory buffer  $\mathcal{M}$ , where  $0 \leq \gamma \leq 1$  is the store ratio. Data from  $\mathcal{M}$  is then merged with the  $t$ -th training set  $S_{t}$  when learning from task  $t$ .
In continual learning, the model needs to adapt to new tasks quickly while maintaining the ability to recover information from previous tasks, hence not all information stored in the hidden representation space should be treated equally. In previous work like style transfer (John et al., 2019) and controlled text generation (Hu et al., 2017), certain information (such as content and syntax) is extracted and shared among different categories and other information (such as style and polarity) is manipulated for each specific category. Similarly, in our continual learning scenario, there is shared knowledge among different tasks as well while the model needs to learn and maintain specific knowledge for each individual task in the learning process. This key observation motivates us to propose an information-disentanglement based regularization for continual text classification to retain shared knowledge while adapting specific knowledge to streams of tasks (Section 4.1). We also incorporate a small set of representative replay samples to alleviate catastrophic forgetting (Section 4.3). Our model architecture is shown in Figure 1.
4.1 Information Disentanglement (ID)
This section describes how to disentangle sentence representations into task generic space and task specific space, and how separate regularizations are imposed on them for continual text classification. Formally, for a given sentence  $x$ , we first use a multi-layer encoder  $B(\cdot)$ , e.g., BERT (Devlin et al., 2019), to get the hidden representations  $r$  which contain both task generic and task specific information. Then we introduce two disentanglement networks  $G(\cdot)$  and  $S(\cdot)$  to extract the generic representation  $g$  and specific representation  $s$  from  $r$ .
Figure 1: Our proposed model architecture. We disentangle the hidden representation into a task generic space and a task specific space via different induction biases. When training on new tasks, different spaces are regularized separately. Also, a small portion of previous data is stored and replayed.
For new tasks, we learn the classifiers by utilizing information from both spaces, and we allow different spaces to change to different extents to best retain knowledge from previous tasks.
Task Generic Space Task generic space is the hidden space containing information generic to different tasks in a task sequence. During switching from one task to another, the generic information should roughly remain the same, e.g., syntactic knowledge should not change too much across the learning process of a sequence of tasks. To extract task generic information  $g$  from hidden representations  $r$ , we leverage the next sentence prediction task (Devlin et al., 2019)<sup>1</sup> to learn the generic information extractor  $G(\cdot)$ . More specifically, we insert a [SEP] token into each training example during tokenization to form a sequence pair labeled IsNext, and switch the first sequence and the second sequence to form a sentence pair labeled NotNext. In order to distinguish IsNext pairs and NotNext pairs, extractor  $G(\cdot)$  needs to learn the context dependencies between two segments, which is beneficial to understand every example and generic to any individual task.
Denote  $\tilde{x}$  as the NotNext example corresponding to  $x$  (IsNext), and  $l\in \{0,1\}$  as the label for next sentence prediction. We build a sentence relation
predictor  $f_{nsp}$  on the generic feature extractor  $G(\cdot)$ :
$$
\begin{array}{l} \mathcal {L} _ {n s p} = \mathbb {E} _ {x \in S _ {t} \cup \mathcal {M}} (\mathcal {L} (f _ {n s p} (G (B (x)), 0) \\ + \mathcal {L} (f _ {n s p} (G (B (\tilde {x})), 1)) \\ \end{array}
$$
where  $\mathcal{L}$  is the cross entropy loss,  $\mathcal{M}$  is the memory buffer and  $S_{t}$  is the  $t$ -th training set.
Task Specific Space Models also need task specific information to perform well over each task. For example, on sentiment classification words like "good" or "bad" could be very informative, but they might not generalize well for tasks like topic classification. Thus we employ a simple task-identifier prediction task on the task specific representation  $s$ , which means for any given example we want to distinguish which task this example belongs to. This simple auxiliary setup will encourage  $s$  to embed different information from different tasks. The loss for task-identifier predictor  $f_{task}$  is:
$$
\mathcal {L} _ {t a s k} = \mathbb {E} _ {(x, z) \in S _ {t} \cup \mathcal {M}} \mathcal {L} (f _ {t a s k} (S (B (x)), z)
$$
where  $z$  is the corresponding task id for  $x$ .
Text Classification To adapt to the  $t$ -th task, we combine the task generic representation  $g = G(B(x))$  and task specific representation  $s = S(B(x))$  to perform text classification, where we minimize the cross entropy loss:
$$
\mathcal {L} _ {c l s} = \mathbb {E} _ {(x, y) \in S _ {t} \cup \mathcal {M}} \mathcal {L} (f _ {c l s} (g \circ s), y))
$$
Here  $y$  is the corresponding class label for  $x$ ,  $f_{cls}(.)$  is the class predictor.  $\circ$  denotes the concatenation of the two representations.
<sup>1</sup>Note that the word "sentence" here refers to an arbitrary span of continuous text (Devlin et al., 2019), it could be several linguistic sentences or part of a linguistic sentence.
4.2 ID Based Regularization
To further prevent severe distortion when training on new tasks, we employ regularization on both generic representations  $g$  and specific representations  $s$ . Different from previous approaches (Li and Hoiem, 2018; Wang et al., 2019) which treat all the spaces equally, we allow regularization to different extents on  $g$  and  $s$  as knowledge in different spaces should be preserved separately to encourage both more positive transfer and less forgetting. Specifically, before training all the modules on task  $t$ , we first compute the generic representations and specific representations of all sentences  $x$  from the training set  $S_{t}$  of current task  $t$  and memory buffer  $\mathcal{M}_t$ . Using the trained  $B^{t - 1}(.)$ ,  $G^{t - 1}(.)$  and  $S^{t - 1}(.)$  from previous task  $t - 1$ , for each example  $x$  we calculate the generic representation as  $G^{t - 1}(B^{t - 1}(x))$ , and the specific representation as  $S^{t - 1}(B^{t - 1}(x))$  to hoard the knowledge from previous models. The computed generic and specific representations are saved. During the learning from training pairs from task  $t$ , we impose two regularization losses separately:
$$
\mathcal {L} _ {r e g} ^ {g} = \mathbb {E} _ {x \in S _ {t} \cup \mathcal {M} _ {t}} \| G ^ {t - 1} (B ^ {t - 1} (x)) - G (B (x)) \| _ {2}
$$
$$
\mathcal {L} _ {r e g} ^ {s} = \mathbb {E} _ {x \in S _ {t} \cup \mathcal {M} _ {t}} \| S ^ {t - 1} (B ^ {t - 1} (x)) - S (B (x)) \| _ {2}
$$
4.3 Memory Selection Rule
Since we only store a small number of examples as a way to balance the replay as well as the extra memory cost and training time, we need to carefully select them in order to utilize the memory buffer  $\mathcal{M}$  efficiently. Considering that if two stored examples are very similar, then only storing one of them could possibly achieve similar results in the future. Thus, those stored examples should be as diverse and representative as possible. To this end, after training on  $t$ -th task, we employ K-means (MacQueen et al., 1967) to cluster all the examples from current training set  $S_{t}$ : For each  $x \in S_{t}$ , we utilize its embedding  $B(x)$  as its input feature to conduct K-means. We set the numbers of clusters to  $\gamma | S_{t}|$  and only select the example closest to each cluster's centroid, following Wang et al. (2019); Han et al. (2020).
4.4 Overall Objective
We can write the final objective for continual learning on text classification as the following:
$$
\begin{array}{l} \mathcal {L} = \mathcal {L} _ {c l s} + \mathcal {L} _ {n s p} + \mathcal {L} _ {t a s k} \tag {1} \\ + \lambda^ {g} \mathcal {L} _ {r e g} ^ {g} + \lambda^ {s} \mathcal {L} _ {r e g} ^ {s} \\ \end{array}
$$
We set the coefficient of the first three loss terms to 1 for simplicity and only introduce two coefficients to tune:  $\lambda^g$  and  $\lambda^s$ . In practice,  $\mathcal{L}_{task}$  and  $\mathcal{L}_{cls}$  are also conducted on each generated NotNext example  $\hat{x}$ ,  $\mathcal{L}_{reg}^g$  and  $\mathcal{L}_{reg}^s$  are only optimized starting from the second task. The full information disentanglement based regularization (IDBR) algorithm is shown in Algorithm 1.
Algorithm 1 IDBR
Input Training sets  $\{S_1,\dots,S_n\}$ , Replay Frequency  $\beta$ , Store ratio  $\gamma$ , Coefficients  $\lambda_g, \lambda_s$
Output Optimal models  $B,G,S,f_{nsp},f_{task},f_{cls}$ $\mathcal{M} = \{\}$  ▷ Initialize memory buffer
Initialize  $B$  using pretrained BERT
Initialize  $G, S, f_{nsp}, f_{task}, f_{cls}$
for  $t = 1,\dots ,n$  do
if  $t\geq 2$  then
Store  $G(B(x)), S(B(x)), \forall x \in S_t \cup \mathcal{M}$
for batches  $\in S_t$  do
Optimize  $\mathcal{L}$  in Equation 1
if step mod  $\beta = 0$  then  $\triangleright$  Replay
Sample  $t - 1$  batches from  $\mathcal{M}$
else  $\triangleright$  No regularization on 1st task for batches  $\in S_{t}$  do
Optimize  $\mathcal{L} = \mathcal{L}_{cls} + \mathcal{L}_{nsp} + \mathcal{L}_{task}$
$$
\mathcal {C} = \mathrm {K} - \text {M e a n s} \left(S _ {t}, n _ {\text {c l u s t e r s}} = \gamma \mid S _ {t}\right) \quad \triangleright \mathcal {C}:
$$
$$
\mathcal {C} ^ {\prime} = \{\text {E x a m p l e s c l o s e s t t o c e n t e r s} \in C \}
$$
$\mathcal{M}\gets \mathcal{M}\cup \mathcal{C}^{\prime}$  Add to memory
return  $B,G,S,f_{nsp},f_{task},f_{cls}$
Following MBPA++ (de Masson d'Autume et al., 2019), we use five text classification datasets (Zhang et al., 2015; Chen et al., 2020) to evaluate our methods, including AG News (news classification), Yelp (sentiment analysis), DBPedia (Wikipedia article classification), Amazon (sentiment analysis), and Yahoo! Answer (Q&A classification). A summary of the datasets is shown in Table 1. We merge the label space of Amazon and Yelp considering their domain similarity, with 33
Dataset	Class	Type	Train	Test
AGNews	4	News	8000	7600
Yelp	5	Sentiment	10000	7600
Amazon	5	Sentiment	10000	7600
DBPedia	14	Wikipedia	28000	7600
Yahoo	10	Q&A	20000	7600
Table 1: Dataset statistics we used for Setting (Sampled). Type means the domain of task classification. Note that the size of the validation set is the same as the size of the training set.
classes in total.
Due to the limitation of resources, for most of our experiments, we create a reduced dataset by randomly sampling 2000 training examples and 2000 validation examples per class for every task. See Table 1 for the train/test size of each dataset. We name this setting Setting (Sampled). We tune all the hyperparameters on the basis of Setting (Sampled). Beyond that, to have a comparison with previous State-of-the-art, we also conduct experiments on the same training set and test set as  $\mathrm{MbPA}++$  (de Masson d'Autume et al., 2019) and LAMOL (Sun et al., 2019), which contains 115,000 training examples and 7,600 test examples for each task. For every task, we randomly hold out 500 examples per class from training examples for validation purpose. We name the latter Setting (Full). During training, we evaluate our model on validation sets from all seen tasks, following Kirkpatrick et al. (2017).
Our experiments are mainly conducted on the task sequences shown in Table 2. To minimize the effect of task order and task sequence length on the results, we examine both length-3 task sequences and length-5 task sequences in various orders. The first 3 task sequences are a cyclic shift of  $\mathrm{ag} \rightarrow$  yelp  $\rightarrow$  yahoo, which are three classification tasks in different domains (news classification, sentiment analysis, Q&A classification). The last four length-5 task sequences follows de Masson d'Autume et al. (2019).
We compare our proposed model with the following baselines in our experiments:
- Finetune (Yogatama et al., 2019): finetune BERT model sequentially without the episodic memory module and any other loss.