SlowGuess/NAACL_2021_OCR huggingface dataset:
Ting Hua
Yilin Shen
Changsheng Zhao
Yen-Chang Hsu
Hongxia Jin
{ting.hua,yilin.shen,changsheng.z,yenchang.hsu,hongxia.jin}@samsung.com
Domain classification is the fundamental task in natural language understanding (NLU), which often requires fast accommodation to new emerging domains. This constraint makes it impossible to retrain all previous domains, even if they are accessible to the new model. Most existing continual learning approaches suffer from low accuracy and performance fluctuation, especially when the distributions of old and new data are significantly different. In fact, the key real-world problem is not the absence of old data, but the inefficiency to retrain the model with the whole old dataset. Is it potential to utilize some old data to yield high accuracy and maintain stable performance, while at the same time, without introducing extra hyperparameters? In this paper, we proposed a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. Specifically, we utilize Fisher information to select exemplars that can "record" key information of the original model. Also, a novel scheme called dynamical weight consolidation is proposed to enable hyperparameter-free learning during the retrain process. Extensive experiments demonstrate that baselines suffer from fluctuated performance and therefore useless in practice. On the contrary, our proposed model CCFI significantly and consistently outperforms the best state-of-the-art method by up to  $20\%$  in average accuracy, and each component of CCFI contributes effectively to overall performance.
Catastrophic forgetting is the well-known Achilles' heel of deep neural networks, that the knowledge learned from previous tasks will be forgotten when the networks are retrained to adapt to new tasks. Although this phenomenon has been noticed as early as the birth of neural networks (French, 1999; McCloskey and Cohen, 1989), it didn't attract much
attention until deep neural networks have achieved impressive performance gains in various applications (LeCun et al., 2015; Krizhevsky et al., 2012).
Domain classification is the task that mapping the spoken utterances to natural language understanding domains. It is widely used in intelligent personal digital assistant (IPDA) devices, such as Amazon Alexa, Google Assistant, and Microsoft Cortana. As many IPDAs now allow third-party developers to build and integrate new domains (Kumar et al., 2017), these devices are eager to continual learning technologies that can achieve high performance stably (Kim et al., 2018a,b). However, most traditional IPADs only work with well-separated domains built by specialists (Tur and De Mori, 2011) or customized designed for specific datasets (Li et al., 2019).
There is still a lack of continual learning methods that capable of general domain classification. Most previous approaches capable of continual learning focus on the scenario that the new model should be retrained without any access to old data (Li and Hoiem, 2016; Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017). However, these methods often involve more parameters that require extensive efforts in expert tuning. And when data distributions of new tasks are obviously different from the original task (e.g. class-incremental learning), these approaches can hardly maintain good accuracy for both tasks and may suffer from fluctuations in performance. On the other hand, old data are not unavailable in many practical cases. The main concerns arise from the tremendous cost in memory and computation resources, if the model is updated with huge previous datasets. Also, most continual learning approaches are applied to image data that little attention has been paid to text data. Is it possible to develop a desirable model capable of continual learning that satisfies the following qualities? 1)
High accuracy with limited old data. Compared to the extreme cases that no access or full access to
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2669-2678 June 6-11, 2021. ©2021 Association for Computational Linguistics
old data, it is more practical to put models under the setting with a limited amount of old data available (e.g., no more than  $10\%$  of original data). In this case, models can achieve good performance without too much additional cost in physical resources and can be conveniently renewed with periodical system updates. 2) High stability with zero extra parameters. Many continual learning models can perform well only under restricted experiment settings, such as specific datasets or carefully chosen parameters. However, practical datasets are usually noisy and imbalanced distributed, and inexperienced users can't set suitable parameters in real-world applications. Therefore, it is desirable to develop a hyperparameter-free model that can work stably under various experimental environments.
To achieve these goals, we proposed a Continuous learning model based on weight Consolidation and Fisher Information sampling (CCFI), with application to domain classification. The main challenge is how to "remember" information from original tasks, not only the representative features from data, but also the learned parameters of the model itself. This is a non-trivial contribution since "uncontrollable changes" will happen to neural network parameters whenever the feature changes. To avoid such "uncontrollable changes", previous work iCarL even discards deep neural networks as its final classifier and turns to k-nearest neighbors algorithm for actual prediction (Rebuffi et al., 2017). Our work demonstrates that these changes are "controllable" with exemplar selected by Fisher information and parameters learned by Dynamical Weight Consolidation. Our contributions can be summarized as follows.
- Fisher information sampling. Good exemplars are required to "remember" key information of old tasks. Unlike previous work using simple mean vectors to remember the information of old data, exemplars selected by Fisher information record both the features of data and the information of the original neural network.
- Dynamical weight consolidation. The need for hyperparameter is an inherited problem of regularization-based continual learning. Previous works search for this hyperparameter by evaluating the whole task sequence, which is supposed not to be known. This work provides a simple auto-adjusted weighting mechanism, making the regularization strategy possible for a practical application. Also, traditional weight consolida
tion methods such as EWC (Kirkpatrick et al., 2017) are designed for sequential tasks with similar distributions. We extend it to the incremental learning scenario and add more regularity to achieve better stability.
- Extensive experimental validation. Most existing continual learning methods are designed for image data, while few previous attempts working on text data are often limited to specific usage scenarios and rely on fine-tuned parameters. Our proposed CCFI model is a general framework that can be efficiently applied to various environments with the least efforts in parameter tuning. Our extensive experimental results demonstrate the proposed CCFI can outperform all state-of-the-art methods, and provide insights into the working mechanism of methods.
Although most of the existing approaches are not directly applicable to our problem, several main branches of research related to this work can be found: exemplar selection, regularization of weights, and feature extraction or fine-tune method based on pre-trained models.
Our problem is closest to the setting of exemplar selection methods (Rebuffi et al., 2017; Li et al., 2019). These approaches store examples from original tasks, and then combine them together with the new domain data for retraining. iCarL (Rebuffi et al., 2017) discards the classifier based on neural network to prevent the catastrophic forgetting, and turns to traditional K-nearest neighbor classifier.
To avoid large changes of important parameters, regularization models (Kirkpatrick et al., 2017; Li and Hoiem, 2016; Zenke et al., 2017) add constraints to the loss function. They usually introduce extra parameters requiring careful initialization. And it has been shown that their performance will drop significantly if the new tasks are drawn from different distributions (Rebuffi et al., 2017). On the contrary, our proposed CCFI is a parameter-free model that can produce stable performance under various experimental environments.
Feature extraction methods utilize pre-trained neural networks to calculate features of input data (Donahue et al., 2014; Sharif Razavian et al., 2014). They make little modifications to the original network but often result in a limited capacity for learning new tasks. Fine-tuning models (Girshick et al., 2014) can modify all the parameters in order to
achieve better performance in new tasks. Although starting with a small learning rate can indirectly preserve the knowledge learned from the original task, fine-tuning method will eventually tend to new tasks. Adapter tuning (Houlsby et al., 2019) can be viewed as the hybrid of fine-tune and feature extraction. Unlike our model that makes no changes to the backbone model, the Adapter tuning method increases the original model size and introduces more parameters by inserting adapter modules to each layer.
3 Our CCFI Model
Given data stream  $D = \{x_{i},y_{i}\}_{i = 1}^{N}$ , the classification tasks in deep learning neural networks are equal to find the best parameter set  $\Theta$  that can maximize the probability of the data  $p(\mathcal{D}|\Theta)$ . Namely, the classifier can make predictions  $\hat{Y}$  that best reproduce the ground truth labels  $Y$ . Under the continual learning setting, new data  $\mathcal{D}^n$  of additional classes will be added to the original data stream  $\mathcal{D}^o$  in an incremental form. Our goal is to update the old parameters  $\Theta^o$  (trained on original data stream  $\mathcal{D}^o$ ) to the new parameters  $\Theta^n$ , which can work well on both new data  $\mathcal{D}^n$  and old data  $\mathcal{D}^o$ .
In this paper, the initial model is trained with the original data set  $\mathcal{D}^o$ , and will output the trained model  $\Theta^o$ . In this training process, Fisher Information Sampling is conducted to select the most representative examples that can help to "remember" the parameters of the initial trained model. In the retraining process, the renewed model is learned based on Dynamical Weight Consolidation, and evaluated on the training set consisted of new classes and the old exemplars.
3.1 Fisher Information Sampling
The critical problem of exemplar set selection is: what are good examples that can "maintain" the performance of old tasks? The state-of-the-art method iCaRL (Rebuffi et al., 2017) selects examples close to mean feature representation, and CoNDA (Li et al., 2019) follows the same scheme to domain adaptation on text data. To utilize the advantage of the mean feature and avoid catastrophic forgetting, iCaRL chooses k-nearest neighbors algorithm as the classifier rather than deep neural networks, although the latter is proved to be a much better performer. Is there any exemplar selection method that can utilize the powerful deep learning models as the classifier, and at the same time, "remember"
the key information of old tasks?
Fisher information measures how much information that an observable random variable carries about the parameter. For a parameter  $\theta$  in the network  $\Theta$  trained by data  $\mathcal{D}$ , its Fisher information is defined as the variance of the gradient of the log-likelihood:
$$
\begin{array}{l} I (\theta) = \operatorname {V a r} (s (\theta)) \tag {1} \\ { = } { E \Bigg [ \left( \frac { \partial } { \partial \theta } \log p ( \mathcal { D } | \theta ) \right) \left( \frac { \partial } { \partial \theta } \log p ( \mathcal { D } | \theta ) \right) ^ { T } \Bigg ] . } \\ \end{array}
$$
Fisher information can be calculated directly, if the exact form of  $\log p(\mathcal{D}|\theta)$  is known. However, the likelihood is often intractable. Instead, empirical Fisher information  $\hat{I} (\theta)$  is computed through data  $d_{i}\in \mathcal{D}$  drawn from  $p(\mathcal{D}|\theta)$ :
$$
\hat {I} (\theta) = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\frac {\partial}{\partial \theta} \log p (d _ {i} | \theta)\right) \left(\frac {\partial}{\partial \theta} \log p (d _ {i} | \theta)\right) ^ {T}. \tag {2}
$$
From another point of view, when  $\log p(\mathcal{D}|\theta)$  is twice differentiable with respect to  $\theta$ , Fisher information can be written as the second derivative of likelihood:
$$
I (\theta) = - E \left[ \frac {\partial^ {2}}{\partial \theta^ {2}} \log p (\mathcal {D} | \theta) \right]. \tag {3}
$$
According to Equation 3, three equivalent indications can be made to a high value in Fisher information  $I(\theta)$ :
- a sharp peak of likelihood with respect to  $\theta$ ,
-  $\theta$  can be easily inferred from data  $\mathcal{D}$
data  $\mathcal{D}$  can provide sufficient information about the correct value for parameter  $\theta$
Jointly thinking about the calculation form introduced by empirical Fisher information (Equation 2) and the physical meaning of Fisher information revealed by the second derivative form (Equation 3), we can find a way to measure how much information each data  $d_{i}$  carries to the estimation of parameter  $\theta$ , which we call as empirical Fisher information difference:
$$
\Delta \hat {I} _ {i} (\theta) = \left(\frac {\partial}{\partial \theta} \log p \left(d _ {i} \mid \theta\right)\right) \left(\frac {\partial}{\partial \theta} \log p \left(d _ {i} \mid \theta\right)\right) ^ {T}. \tag {4}
$$
Instead of simple mean feature vectors used in previous work (Rebuffi et al., 2017; Li et al., 2019), we use the empirical Fisher information difference to select exemplar set. Specifically, CCFI model makes use of BERT (Devlin et al., 2019) for text
classification. The base BERT model is treated as feature extractor  $\Phi : X \to H$ , which takes input token sequences  $X$ , and outputs the hidden representations  $H$ . To predict the true label  $Y$ , a softmax classifier is added to the top of BERT:
$$
p (\hat {Y} | X, \Theta) = \sigma (W ^ {T} \cdot H) = \sigma (W ^ {T} \cdot \Phi (X)), \tag {5}
$$
where  $W$  is the task-specific parameter matrix for classification. The trained parameters  $\Theta$  can therefore be split into the fixed feature extraction part  $\Phi$  and variable weight parameter  $W$ . In continual learning setting, we denote  $W^{k} \in R^{h \times k}$  as the most up-to-date weight matrix, where  $k$  is the number of classes that have been so far observed, and  $h$  is the size of the final hidden state  $H$ .
Remember that, for the classification task, the best parameters that can maximize the probability of the data  $p(\mathcal{D}|\Theta)$  are also the parameters that make predictions  $\hat{Y}$  closest to the ground truth label  $Y$ . Therefore, we can take Equation 5 into Equation 4, in order to get the practical computation of empirical Fisher information difference for data  $d_{i}$  on parameter  $\theta$ . Since the parameters of feature extractor  $\Phi$  are fixed, only empirical Fisher information difference of parameters in weight matrix  $w_{j}\in W$  are calculated:
$$
\Delta \hat {I} _ {i} (w _ {j}) = \left(\frac {\partial}{\partial w _ {j}} \log [ \sigma (W ^ {T} \cdot \Phi (x _ {i})) ] ^ {(y _ {i})}\right) ^ {2}, \tag {6}
$$
where the likelihood  $p(d_i|\theta)$  is calculated via the log-probability value of the correct label  $y_{i}$  of input  $x_{i}$ . And the total empirical Fisher information difference data  $d_{i}$  carrying is the sum over all  $w_{j}\in W$ :
$$
\Delta \hat {I} _ {i} = \sum_ {j = 1} ^ {h \times k} \Delta \hat {I} _ {i} \left(w _ {j}\right). \tag {7}
$$
Algorithm 1 describes the exemplar selecting process. Within each class  $k$ , the samples top ranked by empirical Fisher information difference are selected as exemplars, till the targeted sample rate (e.g.,  $1\%$ ) is met.
3.2 Dynamical Weight Consolidation
The main goal of retraining process is: how to achieve good performance for both new and old tasks? EWC (Kirkpatrick et al., 2017) is proved to be a good performer that can balance the performance of old and new task. However, rather
Algorithm 1: Construction of exemplar set
Input: original data stream  $\mathcal{D}^o$
Input: trained neural network  $\Theta^o = \{\Phi, W^o\}$
Input: sample rate  $r$
1 for each data  $d_{i}$  do
for each parameter  $w_{j}\in W^{o}$  do 3 calculate  $\Delta \hat{I}_i(w_j)$  using Equation 6
4 calculate  $\Delta \hat{I}_i$  using Equation 7
5 for each class  $k$  do
rank the samples  $d_{i}$  by  $\Delta \hat{I}_i$
7 select the top  $|\mathcal{D}_k| \times r$  examples as  $E_k$
Output: exemplar set  $E \gets \{E_1, \dots, E_k\}$
than incremental learning problem studied in this paper, EWC is designed for the tasks with same class number but different in data distributions. Furthermore, EWC requires careful hyperparameter setting, which is unrealistic to be conducted by inexperienced users. In this section, we introduce a scheme named Dynamical Weight Consolidation, which can avoid the requirement of such hyperparameter setting. Also, this scheme is demonstrated to perform more stably than traditional EWC in the experimental part.
Specifically, our loss function during the retraining process can be viewed the sum of two parts: loss  $\ell_{n}$  calculated by the correct class indicators of new data and loss  $\ell_{e}$  to reproduce the performance of old model:
$$
\ell_ {n} = - \sum_ {y \in Y ^ {n}} y \log \hat {y}, \tag {8a}
$$
$$
\ell_ {e} = - \sum_ {y \in E} y \log \hat {y} + \frac {\lambda}{2} \sum_ {j = 1} ^ {h \times \left(k _ {o} + k _ {n}\right)} \hat {I} \left(w _ {j}\right) \left(\hat {w} _ {j} ^ {n} - w _ {j} ^ {o}\right) ^ {2}. \tag {8b}
$$
The loss function  $\ell_{e}$  can be further divided into two parts: the cross entropy of exemplar set, and the consolidation loss caused by modifying parameters with high Fisher information. In traditional EWC model, the weight  $\lambda$  that balances cross entropy and consolidation loss is a fixed value. In our CCFI model,  $\lambda$  is updated dynamically based on current values of cross entropy and consolidation loss:
$$
\lambda = \left\lfloor \lg \frac {- \sum_ {y \in Y ^ {n}} y \log \hat {y}}{\sum_ {j = 1} \hat {I} \left(w _ {j}\right) \left(\hat {w} _ {j} ^ {n} - w _ {j} ^ {o}\right) ^ {2}} \right\rfloor . \tag {9}
$$
Note that, the  $\hat{I} (w_j)$  is the element in the updated
(a) Overall accuracy.
Figure 1: Performance of class-incremental learning on the 73-domain dataset. The observable ratio of old data is kept as  $1\%$ , 43 out of 73 domains are used for initial training, and then 30 domains are added one-by-one for retraining.
(b) New task accuracy.
(c) Old task accuracy.
parameter information table  $T^n$ . The details can be found in the section 3.3.2.
3.3 Overall Process
This part describes the overall process of the CCFI model. First we list the outputs of the old tasks, then we introduce the detailed implements of retraining.
3.3.1 Initial training
After training the model with old data ( $k_{o}$  classes), the outputs of the old task include: 1) trained model  $\Theta^{o}$ ; 2) exemplars  $E$  of old data, and 3) parameter information table  $T^{o}$ . Each element in the parameter information table  $T^{o}$  is the empirical Fisher information  $\hat{I}(w_{j}^{o})$  of the old task, which can be computed through Equation 2 during the initial training process.
3.3.2 Retraining
The retraining process can be described as follows:
1. Load freeze feature extractor: The feature extractor  $\Phi$  is kept unchanged, which means the BERT encoder with transformer blocks and self-attention heads are freezed.
2. Update variable weight matrix: To adopt the new classes data  $X^n$ , the original variable weight matrix  $W^{k_o}$  is extended to  $W^{k_o + k_n} \in R^{h \times (k_o + k_n)}$ , where the first  $k_o$  columns are kept the same with the original model and the new  $k_n$  columns are initialized with random numbers.
3. Update parameter information table: Similar to variable weight matrix, parameter information table  $T^o$  is a matrix with dimension  $h \times k_o$ . To adopt the new data, it is extended to the new matrix  $T^n$  with dimension  $h \times (k_o + k_n)$ , where the first  $k_o$  columns are same with  $T^o$  and the new  $k_n$  columns are initialized with zero. In this way, the new model can freely update the the new  $k_n$  columns to lower classification loss,
but will receive penalty when modifying important parameters in the original  $k_{o}$  columns.
In this section, the CCFI model is first compared with the state-of-the-art methods under a continual setting. And further evaluations are conducted to examine the effectiveness of the individual components within CCFI model.
Datasets. We evaluated our proposed CCFI and comparison methods on public available 150-class dataset (Larson et al., 2019) and real-world (even product) 73-domain dataset The details of datasets can be found in Appendix A.1.
Baselines. iCaRL (Rebuffi et al., 2017) and CoNDA (Li et al., 2019), are the closest continual learning approaches to CCFI, which are designed for the scenario with access to old data. We also add fine-tune and the fixed feature method as baselines. To make fair comparisons, CCFI and all the baselines use the same BERT backbone (Devlin et al., 2019), and observe the same amount of old data in all learning tasks. The implementation details can be found in Appendix A.2.
In the main body of experiments, we report the results with the framework consisted of BERT backbone and one-layer linear classifier. We also conducted experiments with a multiple-layer classifier, which can be found in Appendix A.3.
Two key factors play in the performance of continual learning: 1) the number of new classes for retraining, and 2) the amount of old observable data. In this section, we first validate our model through a class-incremental learning task, by keep
(a) Overall accuracy.
Figure 2: Performance of class-incremental learning on the 150-class dataset. The observable ratio of old data is kept as  $10\%$ , 90 out of 150 classes are used for initial training, and then 30 domains are added as additional classes.
ing the amount of old observable data fixed and changing the number of new classes. We also study the effects of different exemplars by keeping the number of new classes unchanged but varying old observable data.
Class-incremental Learning. In this part, we conduct the class-incremental learning evaluation on both 150-class and 73-domain dataset. Class-incremental learning can be viewed as the benchmark protocol for continual learning with access to old data (Rebuffi et al., 2017; Li et al., 2019). Specifically, after the initial training, new domains will be added in random order. After adding each batch of new data, the results are evaluated on the current data set, considering all classes have been trained so far.
Figure 1 and Figure 2 show the performance of class-incremental learning on 73-domain dataset and 150-class dataset. CCFI outperforms other methods in all tasks on both datasets. Specifically, several observations can be made as follows.
- Overall performance. Under the same new class number, CCFI always achieves the best overall accuracy among all methods. And this performance gap is enlarged with more new classes added for retraining.
- Performance fluctuations. Fine-tune method is unstable in performance. It is the second performer on the 73-domain dataset. However, it quickly drops to almost zero and displays fluctuations on the 150-class dataset, even if the experiment conducted on the 150-class dataset is set with a higher observable ratio of old data.
- Accuracy stage. Both the fixed feature method and CoNDA display the pattern of "performance stage" with more new classes added, and CoNDA enjoys a "larger" stage than the fixed feature method. For example, as shown in Figure 1a, CoNDA maintains stable performance with