SlowGuess/NAACL_2021_OCR huggingface dataset:
Zixuan Ke $^{1}$ , Hu Xu $^{2}$  and Bing Liu $^{1}$
$^{1}$ Department of Computer Science, University of Illinois at Chicago
1{zke4, liub}@uic.edu
$^{2}$ huxu@fb.com
This paper studies continual learning (CL) of a sequence of aspect sentiment classification (ASC) tasks. Although some CL techniques have been proposed for document sentiment classification, we are not aware of any CL work on ASC. A CL system that incrementally learns a sequence of ASC tasks should address the following two issues: (1) transfer knowledge learned from previous tasks to the new task to help it learn a better model, and (2) maintain the performance of the models for previous tasks so that they are not forgotten. This paper proposes a novel capsule network based model called B-CL to address these issues. B-CL markedly improves the ASC performance on both the new task and the old tasks via forward and backward knowledge transfer. The effectiveness of B-CL is demonstrated through extensive experiments.
Continual learning (CL) aims to incrementally learn a sequence of tasks. Once a task is learned, its training data is often discarded (Chen and Liu, 2018). This is in contrast to multi-task learning, which assumes the training data of all tasks are available simultaneously. The CL setting is important in many practical scenarios. For example, a sentiment analysis company typically has many clients and each client often wants to have their private data deleted after use. In the personal assistant or chatbot context, the user does not want his/her chat data, which often contains sentiments or emotions, uploaded to a central server. In such applications, if we want to improve sentiment analysis accuracy for each user/client without breaching confidentiality, CL is a suitable solution.
There are two main types of continual learning: (1) Task Incremental Learning (TIL) and (2) Class Incremental Learning (CIL). This work focuses
Task ID	Domain/Task	One Training Example(in that domain/task)
1	Vacuum Cleaner [CF]	This vacuum cleaner sucks !!!
2	Desktop [KT]	The keyboard is clicky .
3	Tablet [KT]	The soft keyboard is hard to use.
4 (new task)	Laptop	The new keyboard sucks and is hard to click!
Table 1: Tasks 2 and 3 have shareable knowledge to transfer (KT) to the new task, whereas Task 1 has specific knowledge that is expected to be isolated from the new task to avoid catastrophic forgetting (CF) (although they use the same word). Note that here we use only one sentence to represent a task, but each task actually represents a domain with all its sentences.
on TIL, where each task is a separate aspect sentiment classification (ASC) task. An ASC task is defined as follows (Liu, 2015): given an aspect (e.g., picture quality in a camera review) and a sentence containing the aspect in a particular domain (e.g., camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) about the aspect. TIL builds a model for each task and all models are in one neural network. In testing, the system knows which task each test instance belongs to and uses only the model for the task to classify the instance. In CIL, each task contains one or more classes to be learned. Only one model is built for all classes. In testing, a test case from any class may be presented to the model to classify without giving it any task information. This setting is not applicable to ASC.
Our goal of this paper is to achieve the following two objectives: (1) transfer the knowledge learned from previous tasks to the new task to help learn a better model for the new task without accessing the training data from previous tasks (in contrast to multi-task learning), and (2) maintain (or even improve) the performance of the old models for previous tasks so that they are not forgotten. The focus of the existing CL (TIL or CIL) research has been on solving (2), catastrophic forgetting (CF) (Chen and Liu, 2018; Ke et al., 2020a). CF means that when a network learns a sequence of tasks, the learning of each new task is likely to change the net
$^{1}$ https://github.com/ZixuanKe/PyContinual
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4746-4755 June 6-11, 2021. ©2021 Association for Computational Linguistics
work parameters learned for previous tasks, which degrades the model performance for the previous tasks (McCloskey and Cohen, 1989). In our case, (1) is also important as ASC tasks are similar, i.e., words and phrases used to express sentiments for different products/tasks are similar. To achieve the objectives, the system needs to identify the shared knowledge that can be transferred to the new task to help it learn better and the task specific knowledge that needs to be protected to avoid forgetting of previous models. Table 1 gives an example.
Fine-tuned BERT (Devlin et al., 2019) is one of the most effective methods for ASC (Xu et al., 2019; Sun et al., 2019). However, our experiments show that it works very poorly for TIL. The main reason is that the fine-tuned BERT on a task/domain captures highly task specific information which is difficult to transfer to a new task.
In this paper, we propose a novel model called B-CL (BERT-based Continual Learning) for ASC continual learning. The key novelty is a building block, called Continual Learning Adapter (CLA) inspired by the Adapter-BERT in (Houlsby et al., 2019). CLA leverages capsules and dynamic routing (Sabour et al., 2017) to identify previous tasks that are similar to the new task and exploit their shared knowledge to help the new task learning and uses task masks to protect task-specific knowledge to avoid forgetting (CF). We conduct extensive experiments over a wide range of baselines to demonstrate the effectiveness of B-CL.
In summary, this paper makes two key contributions. (1) It proposes the problem of task incremental learning for ASC. (2) It proposes a new model B-CL with a novel adapter CLA incorporated in a pre-trained BERT to enable ASC continual learning. CLA employs capsules and dynamic routing to explore and transfer relevant knowledge from old tasks to the new task and uses task masks to isolate task-specific knowledge to avoid CF. To our knowledge, none of these has been done before.
Continual learning (CL) has been studied extensively (Chen and Liu, 2018; Parisi et al., 2019). To our knowledge, no existing work has been done on CL for a sequence of ASC tasks, although CL of a sequence of document sentiment classification tasks has been done.
Continual Learning. Existing work has mainly focused on dealing with catastrophic forgetting (CF).
Regularization-based methods, such as those in (Kirkpatrick et al., 2016; Lee et al.; Seff et al., 2017), add a regularization in the loss to consolidate previous knowledge when learning a new task.
Parameter isolation-based methods, such as those in (Serrà et al., 2018; Mallya and Lazebnik, 2018; Fernando et al., 2017), make different subsets of the model parameters dedicated to different tasks and identify and mask them out during the training of the new task.
Gradient projection-based method, such as that in (Zeng et al., 2019), ensures the gradient updates occur only in the orthogonal direction to the input of the old tasks and thus will not affect old tasks.
Replay-based methods, such as those in (Rebuffi et al., 2017; Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019), retain an exemplar set of old task training data to help train the new task. The methods in (Shin et al., 2017; Kamra et al., 2017; Rostami et al., 2019; He and Jaeger, 2018) build data generators for previous tasks so that in learning the new task, they can use some generated data for previous tasks to help avoid forgetting.
As these methods are mainly for avoiding CF, after learning a sequence of tasks, their final models are typically worse than learning each task separately. The proposed B-CL not only deals with CF, but also performs knowledge transfer to improve the performance of both the new and the old tasks.
Lifelong Learning (LL). LL is now regarded the same as CL, but early LL mainly aimed at improving the new task learning through forward transfer without tackling CF (Silver et al., 2013; Ruvolo and Eaton, 2013; Chen and Liu, 2018).
Several researchers have used LL for document-level sentiment classification. Chen et al. (2015) and Wang et al. (2019) proposed two Naive Bayes (NB) approaches to help improve the new task learning. A heuristic NB method was also used in (Wang et al., 2019). Xia et al. (2017) presented a LL approach based on voting of individual task classifiers. All these works do not use neural networks, and are not concerned with the CF problem.
Shu et al. (2017) used LL for aspect extraction, which is a different problem. Wang et al. (2018) used LL for ASC, but improved only the new task and did not deal with CF. Existing CL systems SRK (Lv et al., 2019), KAN (Ke et al., 2020b) and L2PG (Qin et al., 2020) are for document sentiment classification, but not ASC. Ke et al. (2020a) also performed transfer in the image domain.
Recently, capsule networks (Hinton et al., 2011) have been used in sentiment classification and text classification (Chen and Qian, 2019; Zhao et al., 2019). But they have not been used in CL.
This section introduces BERT, Adapter-BERT and Capsule Network as they are used in our model.
BERT for ASC. Due to its superior performance, this work uses BERT (Devlin et al., 2019) and its transformer (Vaswani et al., 2017) architecture as the base. We also adopt the ASC formulation in (Xu et al., 2019), where the aspect term and review sentence are concatenated via [SEP]. The sentiment polarity is predicted on top of the [CLS] token. Although BERT can achieve impressive performance on a single ASC task, its architecture and fine-tuning paradigm are not suitable for CL (see Sec. 1). Experiments show that it performs very poorly for CL (Sec. 5.4). We found that Adapter-BERT (Houlsby et al., 2019) is a better fit for CL.
Adapter-BERT. Adapter-BERT basically inserts a 2-layer fully-connected network (adapter) in each transformer layer of BERT (see Figure 1(A)). During training for the end-task, only the adapters and normalization layers are trained, no change to any other BERT parameters, which is good for CL because fine-tuning BERT itself causes serious forgetting. Adapter-BERT achieves similar performances to fine-tuned BERT (Houlsby et al., 2019). We propose to exploit the adapter idea and the capsule network to achieve effective CL for ASC tasks.
Capsule Network. Capsule network (CapsNet) is a relatively new classification architecture (Hinton et al., 2011; Sabour et al., 2017). Unlike CNN, CapsNet replaces the scalar feature detectors with vector capsules that can preserve additional information such as position and thickness in images. A typical CapsNet has two capsule layers. The primary layer stores low-level feature maps and the class layer produces the probability for classification with each capsule corresponding to one class. It uses a dynamic routing algorithm to enable each lower level capsule to send its output to the similar (or "agreed", computed by dot product) higher level capsule. This is the key property that we exploit to identify and group similar tasks and their shared features or knowledge.
Note that the proposed B-CL does not adopt the whole capsule network as we are only interested in the capsule layers and dynamic routing instead of
Figure 1: (A). Adapter-BERT (Houlsby et al., 2019) and its adapters in a transformer (Vaswani et al., 2017) layer. An adapter is a 2-layer fully connected network with a skip-connection. It is added twice to each Transformer layer. Only the adapters (yellow boxes) and layer norm (green boxes) layers are trainable. The other modules (grey boxes) are frozen. (B). Proposed B-CL, which replaces the adapter with CLA. CLA has two sub-modules: knowledge sharing module (KSM) and task specific module (TSM). Each of these modules has a skip-connection.
the max-margin loss and the classifier.
4 Continual Learning Adapter (CLA)
Recall the proposed B-CL aims to achieve (1) knowledge transfer between related old tasks and the new task through knowledge sharing and (2) forgetting avoidance through preventing task specific knowledge of previous tasks from being overwritten by the new task learning. Inspired by Adapter-BERT, we propose the continual learning adapters (CLA) to replace the adapters in Adapter-BERT to enable CL as in Figure 1(B) to achieve BERT based continual learning for ASC.
The architecture of CLA is shown in Figure 2(A). It contains two modules: (1) knowledge sharing module (KSM) for identifying and exploiting shareable knowledge from the similar previous tasks and the new task, and (2) task specific module (TSM) for learning task specific neurons and protecting them from being updated by the new task.
CLA takes two inputs: (1) hidden states  $h^{(t)}$  from the feed-forward layer inside a transformer layer and (2) task ID  $t$ . The outputs are hidden states with features good for the  $t$ -th task. KSM leverages capsule layers (see below) and dynamic routing to group similar tasks and the shareable knowledge, whereas TSM takes advantage of task mask (TM) to protect neurons for a particular task and leave other neurons free. Those free neurons are later used by TSM for a new task. Since TMs are differentiable, the whole system B-CL can be
trained end-to-end. We detail each module below.
4.1 Knowledge Sharing Module (KSM)
KSM groups similar tasks and shared knowledge (features) among them to enable knowledge transfer among similar tasks. This is achieved through two capsule layers (task capsule layer and knowledge sharing capsule layer) and the dynamic routing algorithm of the capsule network.
4.1.1 Task Capsule Layer (TCL)
Each capsule in TCL represents a task and TCL prepares low-level features derived from each task (Figure 2(A)). As such, a capsule is added to TCL for every new task. This incremental growing is efficient and easy because these capsules are discrete and do not share parameters. Also each capsule is simply a 2-layer fully connected network with a small number of parameters. Let  $h^{(t)} \in \mathbb{R}^{d_t \times d_e}$  be the input of CLA, where  $d_t$  is the number of tokens and  $d_e$  the number of dimensions. Let the set of tasks learned so far be  $T_{\text{prev}}$  (before learning the new task  $t$ ) and  $|T_{\text{prev}}| = n$ . In TCL, we have  $n + 1$  different capsules representing all past  $n$  learned tasks as well as the new task  $t$ . The capsule for the  $i$ -th ( $i \leq n + 1$ ) task is
$$
p _ {i} ^ {(t)} = f _ {i} \left(h ^ {(t)}\right), \tag {1}
$$
where  $f_{i}(\cdot) = \mathrm{MLP}_{i}(\cdot)$  denotes a 2-layer fully-connected network.
4.1.2 Knowledge Sharing Capsule Layer (KCL)
Each knowledge sharing capsule in KCL captures those tasks (i.e., their task capsules  $\{p_i^{(t)}\}_{1}^{n + 1}$ ) with similar features or shared knowledge. This is automatically achieved by the dynamic routing algorithm. Recall dynamic routing encourages each lower level capsule (task capsule in our case) to send its output to the similar (or "agreed") higher level capsule (knowledge sharing capsule in our case).
Essentially, the similar task capsules (with many shared features) are "clustered" together by higher coefficients (which determine how much a task capsule can go to the next layer) while dissimilar tasks (with few shared features) are blocked via low coefficients. Such clustering identifies the shared features or knowledge from multiple task capsules as well as helps backward transfer across the similar tasks.
KCL first turns each task capsule  $p_i^{(t)}$  into a temporary feature  $u_{j|i}^{(t)}$  as:
$$
u _ {j \mid i} ^ {(t)} = W _ {i j} p _ {i} ^ {(t)}, \tag {2}
$$
where  $W_{ij} \in \mathbb{R}^{d_s \times d_k}$  is the weight matrix,  $d_s$  and  $d_k$  are the dimensions of task capsule  $i$  and knowledge sharing capsule  $j$ . The number of knowledge sharing capsules is a hyperparameter detailed in the experiment section. The temporary features are summed up with weights  $c_{ij}^{(t)}$  to obtain the initial knowledge sharing capsule  $s_j^{(t)}$ :
$$
s _ {j} ^ {(t)} = \sum_ {i} c _ {i j} ^ {(t)} u _ {j | i} ^ {(t)}, \tag {3}
$$
where  $c_{ij}^{(t)}$  is a coupling coefficient summed up to 1 and we detail how to compute it later. Note that the task capsule for each task in Eq. 1 is mapped to the knowledge sharing capsule in Eq. 3 and  $c_{ij}^{(t)}$  indicates how much or how informative the representation of the  $i$ -th task is to the  $j$ -th knowledge sharing capsule. As a result, a knowledge sharing capsule can represent diverse sharable knowledge. For those tasks with a very low  $c_{ij}^{(t)}$ , their representations are less considered in the  $j$ -th knowledge sharing capsule. This makes sure only task capsules for tasks that are salient or similar to the new task are used and the others task capsules are ignored (and thus protected) to learn more general shareable knowledge. Recall that the ASC tasks are similar and thus such learning of task sharing features can be very important.
Note that in backpropagation, the dissimilar tasks with low  $c_{ij}^{(t)}$  are updated with a low gradient while the similar tasks with high  $c_{ij}^{(t)}$  are updated with a larger gradient. This encourages backward transfer across similar tasks.
Dynamic Routing. The coupling coefficient in Eq. 3 is essential for the quality of shareable knowledge. This is computed by a "routing softmax":
$$
c _ {i j} ^ {(t)} = \frac {\exp \left(b _ {i j} ^ {(t)}\right)}{\sum_ {o} \exp \left(b _ {i o} ^ {(t)}\right)}, \tag {4}
$$
where each  $b_{ij}$  is the log prior probability showing how salient or similar a task capsule  $i$  is to a knowledge sharing capsule  $j$ . It is initialized to 0 indicating no salient connection between them at the beginning. We apply the dynamic routing algorithm in (Sabour et al., 2017) to update  $b_{ij}$ :
$$
b _ {i j} ^ {(t)} \leftarrow b _ {i j} ^ {(t)} + a _ {i j} ^ {(t)}, \tag {5}
$$
Figure 2: (A) Architecture of CLA: the skip-connection is not shown for clarity. (B) illustration of task masking: a (learnable) task mask is applied after the activation function to selectively activate a neuron (or feature). Some notes about (B) are: the two rows of each task corresponds to  $k_0^{(t)}$  and  $k_1^{(t)}$  in TSM. In the cells before training, those with 0's are the neurons to be protected (masked) and those cells without a number are free neurons (not used). In the cells after training, those cells with 1's show neurons that are important for the current task, which are used as a mask for the future. Those cells with more than one color indicate that they are shared by more than one task. Those 0 cells without a color are not used by any task.
where  $a_{ij}$  is the agreement coefficient (see below). Intuitively, this step tends to aggregate the similar (or "agreed") tasks on a knowledge sharing capsule with a higher agreement coefficient  $a_{ij}$  and thus a higher logit  $b_{ij}^{(t)}$  (Eq. 5) or coupling coefficient  $c_{ij}^{(t)}$  (Eq. 4). The agreement coefficient is computed as
$$
a _ {i j} ^ {(t)} = u _ {j | i} ^ {(t)} \cdot v _ {j} ^ {(t)}, \tag {6}
$$
where  $v_{j}^{(t)}$  is a normalized representation by applying the non-linear "squash" function (Sabour et al., 2017) to  $s_{j}^{(t)}$  (for the first task,  $s_{j}^{(t)} = u_{j|i}^{(t)}$ ):
$$
v _ {j} ^ {(t)} = \frac {\left| \left| s _ {j} ^ {(t)} \right| \right| ^ {2}}{1 + \left| \left| s _ {j} ^ {(t)} \right| \right|} \frac {s _ {j} ^ {(t)}}{\left| \left| s _ {j} ^ {(t)} \right| \right|}, \tag {7}
$$
where the length of  $v_{j}^{(t)}$  is normalized to [0,1] to represent the active probability of a knowledge sharing capsule  $j$ .
Finally, note that the dynamic routing procedure (Eq.  $(3)\rightarrow (7))$  is repeated for  $r$  iterations.
4.2 Task Specific Module (TSM)
Although knowledge sharing is important for ASC, it is equally important to preserve task specific
knowledge for previous tasks to prevent forgetting (CF). To achieve this, we use task masks (Figure 2(B)). Specifically, we first detect the neurons used by each old task, and then block off or mask out all the used neurons when learning a new task.
The task specific module consists of differentiable layers (CLA uses a 2-layer fully-connected network). Each layer's output is further applied with a task mask to indicate which neurons should be protected for that task to overcome CF and forbids gradient updates for those neurons during backpropagation for a new task. Those tasks with overlapping masks indicate knowledge sharing. Due to KSM, the features flowing in those overlapping neurons enable the related old tasks to also improve in learning the new task.
4.3 Task Masks
Given the knowledge sharing capsule  $s_{j}^{(t)}$ , TSM maps them into input  $k_{l}^{(t)}$  via a fully-connected network, where  $l$  is the  $l$ -th layer in TSM. A task mask (a "soft" binary mask)  $\mathfrak{m}_{l}^{(t)}$  is trained for each task  $t$  at each layer  $l$  in TSM during training task  $t$ 's
classifier, indicating the neurons that are important for the task in the layer. Here we borrow the hard attention idea in (Serrà et al., 2018) and leverage the task ID embedding to the train the task mask.
For a task ID  $t$ , its embedding  $e_l^{(t)}$  consists of differentiable deterministic parameters that can be learned together with other parts of the network. It is trained for each layer in TSM. To generate the task mask  $\mathfrak{m}_l^{(t)}$  from  $e_l^{(t)}$ , Sigmoid is used as a pseudo-gate function and a positive scaling hyperparameter  $s$  is applied to help training. The  $m_l^{(t)}$  is computed as follows:
$$
m _ {l} ^ {(t)} = \sigma \left(s e _ {l} ^ {(t)}\right). \tag {8}
$$
Note that the neurons in  $m_{l}^{(t)}$  may overlap with those in other  $m_{l}^{(i_{\mathrm{prev}})}$ s from previous tasks showing some shared knowledge. Given the output of each layer in TSM,  $k_{l}^{(t)}$ , we element-wise multiply  $k_{l}^{(t)} \otimes m_{l}^{(t)}$ . The masked output of the last layer  $k^{(t)}$  is fed to the next layer of the BERT with a skipconnection (see Figure 1). After learning task  $t$ , the final  $m_{l}^{(t)}$  is saved and added to the set  $\{m_{l}^{(t)}\}$ .
4.4 Training
For each past task  $i_{\mathrm{prev}} \in \mathcal{T}_{\mathrm{prev}}$ , its mask  $m_l^{(i_{\mathrm{prev}})}$  indicates which neurons are used by that task and need to be protected. In learning task  $t$ ,  $m_l^{(i_{\mathrm{prev}})}$  is used to set the gradient  $g_l^{(t)}$  on all used neurons of the layer  $l$  in TSM to 0. Before modifying the gradient, we first accumulate all used neurons by all previous tasks' masks. Since  $m_l^{(i_{\mathrm{prev}})}$  is binary, we use max-pooling to achieve the accumulation:
$$
m _ {l} ^ {\left(t _ {\mathrm {a c}}\right)} = \operatorname {M a x P o o l} \left(\left\{m _ {l} ^ {\left(i _ {\text {p r e v}}\right)} \right\}\right). \tag {9}
$$
The term  $m_l^{(t_{\mathrm{ac}})}$  is applied to the gradient:
$$
g _ {l} ^ {\prime (t)} = g _ {l} ^ {(t)} \otimes \left(1 - m _ {l} ^ {\left(t _ {\mathrm {a c}}\right)}\right). \tag {10}
$$
Those gradients corresponding to the 1 entries in  $m_{l}^{(t_{\mathrm{ac}})}$  are set to 0 while the others remain unchanged. In this way, neurons in an old task are protected. Note that we expand (copy) the vector  $m_{l}^{(t_{\mathrm{ac}})}$  to match the dimensions of  $g_{l}^{(t)}$ .
Though the idea is intuitive,  $e_l^{(t)}$  is not easy to train. To make the learning of  $e_l^{(t)}$  easier and more stable, an annealing strategy is applied (Serrà et al., 2018). That is,  $s$  is annealed during training, inducing a gradient flow and set  $s = s_{\mathrm{max}}$  during testing. Eq. 8 approximates a unit step function as the mask, with  $m_l^{(t)} \to \{0,1\}$  when  $s \to \infty$ . A
training epoch starts with all neurons being equally active, which are progressively polarized within the epoch. Specifically,  $s$  is annealed as follows:
$$
s = \frac {1}{s _ {\operatorname* {m a x}}} + \left(s _ {\operatorname * {m a x}} - \frac {1}{s _ {\operatorname* {m a x}}}\right) \frac {b - 1}{B - 1}, \tag {11}
$$