SlowGuess/NAACL_2021_OCR huggingface dataset:
Yue Cao, $^{1,2,3}$  Hao-Ran Wei, $^{4}$  Boxing Chen, $^{4}$  Xiaojun Wan $^{1,2,3}$
$^{2}$ Center for Data Science, Peking University
<sup>3</sup>The MOE Key Laboratory of Computational Linguistics, Peking University
4 Alibaba Group
{yuecao,wanxiaojun}  $@$  pku.edu.cn,{funan.whr,boxing.cbx} $@$  alibaba-inc.com
Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus. However, this bears the risk of catastrophic forgetting that the performance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT models. We consider a scenario where the training is comprised of multiple stages and propose a dynamic knowledge distillation technique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and propose a bias-correction module to eliminate the bias. We conduct experiments on three representative settings of NMT application. Experimental results show that the proposed method achieves superior performance compared to baseline models in all settings.
Continual learning, which is also referred to as incremental learning or lifelong learning, is a learning paradigm that allows the agent to continuously learn from new knowledge without forgetting previously learned knowledge. Humans naturally have the ability to continually acquire knowledge while preserving old knowledge throughout their lifespan.
In real-world applications, data is usually given in a continuous stream form, and only part of the data is available at the beginning of training. Therefore, the ability to learn from continuous streams of information is crucial for artificial intelligence systems. However, continual learning remains a
big challenge for artificial intelligence systems and models since they suffer from the problem of catastrophic forgetting (French, 1993), i.e., the learning of new tasks may cause the model to forget the knowledge learned from previous tasks. This phenomenon typically leads to a significant performance decrease in previously learned tasks. One trivial solution to avoid catastrophic forgetting is to retrain from scratch by combining old and new tasks. However, this methodology is computationally inefficient and needs to store old data all the time.
Recently, continual learning has received increasing attention in the artificial intelligence field. Most of existing works focus on computer vision tasks (Zenke et al., 2017; Aljundi et al., 2017; Triki et al., 2017; Hou et al., 2018; Aljundi et al., 2018; Hou et al., 2019; Wu et al., 2019).
In the natural language processing area, several methods have been proposed to alleviate the problem of catastrophic forgetting for Neural Machine Translation (NMT) models. For example, Freitag and Al-Onaizan (2016) propose to ensemble models trained on different domains. However, this brings a storage issue: as the number of domains increases, the number of stored models also increases. Saunders et al. (2019) and Thompson et al. (2019) add an L2 or EWC regularization to each parameter to prevent the model's parameters from changing too much. However, for those transformer models with more than 100 million parameters, the time and space cost for computing L2 or EWC regularization is expensive. Khayrallah et al. (2018) propose a regularized training objective that minimizes the cross-entropy between in domain model's output distribution and that of the out-of-domain model. This method can essentially be regarded as a kind of knowledge distillation.
The above works assume that the training is divided into two stages, i.e., out-of-domain training and in-domain fine-tuning. In this work, we extend
This work was done when Yue Cao was an intern at Alibaba. Codes are available at https://github.com/caoy1996/CL-NMT.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3964-3974 June 6-11, 2021. ©2021 Association for Computational Linguistics
these works and propose a new continual learning framework for NMT models. We consider a more general scenario where the training is comprised of multiple stages. We propose a dynamic knowledge distillation-based method to alleviate the problem of catastrophic forgetting in a systematic and principled way.
We also find that when fine-tuning on new data, there exists a strong bias towards the new words in the output embedding layer (i.e. the linear projection before the last softmax layer) of the decoder, which results in the bias in the generation that favors words from new data. To address this issue, we incorporate the model with a bias-correction module that normalizes the weights in the projection layer. The bias-correction module can effectively eliminate the bias of significant differences in magnitudes.
We consider three continual learning scenarios: (1) in-domain multi-stage training, where  $m$  streams of data from the same domain are fed to the model sequentially, (2) domain-incremental training, where  $m$  streams of data from different domains are fed to the model sequentially, and (3) time-incremental training, where  $m$  streams of data from different time are fed to the model sequentially. Experimental results show that the proposed method can effectively address the catastrophic forgetting issue and balance the weights in the projection layer, thus achieving superior results compared to the competitive models.
In summary, the prime contributions of this paper are as follows:
- We propose a novel continual learning framework for neural machine translation. Compared with existing works, we consider a more general scenario where the training is comprised of multiple stages.
- We propose a novel method to alleviate the problem of catastrophic forgetting in a systematic way. We also find the existence of bias in the output embedding layer and propose a bias-correction module to address this issue.
- Experimental results in three different settings all show that the proposed method obtains superior performance compared to competitive models.2
The task of machine translation is to automatically translate a written text from one natural language into another. Early machine translation systems are mostly built upon statistical learning techniques, which mainly rely on various count-based features (Brown et al., 1990; Och, 2003; Koehn et al., 2007). Recently, statistical machine translation (SMT) has largely been superseded by neural machine translation (NMT), which tackles machine translation with deep neural networks (Luong et al., 2015; Vaswani et al., 2017). Most NMT models either use LSTM (Luong et al., 2015) or Transformer (Vaswani et al., 2017) architectures.
NMT systems are sensitive to the data distributions (Stahlberg, 2019). To improve the performance of NMT models in low-resource domains, a widely-used technique is to train the model on a general domain corpus, and then fine-tune it on the in-domain corpus via continual training (Sennrich et al., 2016; Luong and Manning, 2015). However, this suffers from the problem of catastrophic forgetting (French, 1993) that the performance of the model on the general domain has decreased drastically. In this work, we aim to mitigate the catastrophic forgetting for NMT models.
As for the bias in NMT systems, Michel and Neubig (2018) 2018 adapt the bias of the output softmax to build a personalized NMT model. Different from their work, we propose to eliminate the bias in the output layer.
Most of continual learning models are proposed for computer vision tasks. These models mainly fall into parameter-based methods (Aljundi et al., 2018; Kirkpatrick et al., 2016; Zenke et al., 2017) and distillation-based methods (Aljundi et al., 2017; Triki et al., 2017; Hou et al., 2018, 2019; Wu et al., 2019). The parameter-based methods estimate the importance of each parameter and penalize the model once it updates the important parameters. The distillation-based methods transfer important knowledge from an old model to a new model through a teacher-student framework. Usually, a modified cross-entropy loss is adopted to preserve the knowledge of the old model.
In the field of natural language processing, there are some researches on solving catastrophic forgetting problem in lifelong learning (Freitag and
2Codes and data will be released once this paper gets accepted.
Al-Onaizan, 2016; Khayrallah et al., 2018; Saunders et al., 2019; Thompson et al., 2019). However, these works only consider the scenario of one-stage incremental training. To the best of our knowledge, there is no previous work that takes into account the scenario in which the training consists of multiple stages.
Domain adaptation learning (or transfer learning) is a task similar to continual learning. The difference is that domain adaptation learning only cares about the performance of in-domain data, while continual learning cares about not only the performance on in-domain data, but also the performance on out-of-domain data.
3.1 Overall
Given a bilingual translation pair  $(x,y)$ , the NMT model  $g$  learns the parameter  $\xi$  and  $\theta$  to maximize the conditional log-likelihood  $\log P(y|x,\xi,\theta)$ . Generally, the probability of generating  $i$ -th word is computed as
$$
p \left(y _ {i} \mid y _ {1: i - 1}, x\right) = \frac {\exp \left\{\theta^ {\top} \phi \left(x _ {i} , y _ {1 : i - 1} , \xi\right) \right\}}{\sum_ {j} \exp \left\{\theta^ {\top} \phi \left(x _ {j} , y _ {1 : i - 1} , \xi\right) \right\}} \tag {1}
$$
where  $x_{i}, y_{i}$  is the  $i$ -th word in  $x$  and  $y$ ,  $\phi(\cdot, \xi)$  is a nonlinear function that maps an input  $x$  into a dense representation. The linear projection parameterized by  $\theta$  maps the dense representation to the word distributions, followed by a softmax activation to output the probability of generating each word. For NMT models, the nonlinear function  $\phi(\cdot, \xi)$  is usually chosen as the encoder-decoder framework. In the following text, for the convenience of narration, we use  $w$  to refer to  $\xi$  and  $\theta$ , i.e.,  $w = \xi \cup \theta$ .
Under the continual training setting, the encoder and decoder  $\phi(\cdot, \xi)$  is trained on data of different domains successively. When fine-tuning on new data, the learned parameters  $\xi$  may overfit new data and degrade the performance on old data, which is known as the problem of catastrophic forgetting. On the other hand, when fine-tuning on a new domain corpus, we need to add new words from the new domain to the vocabulary, so we need to expand the projection matrix in the linear projection. At this stage, the model always samples new words to generate, and the ground truths for those old words are always 0. After several epochs, the model may mistakenly believe that the old words are no longer used and thus reduce the probability
of old words to be 0 for all samples. This causes the biased weights issue.
Our goal is twofold: (1) for the parameters  $\xi$  in the encoder and decoder  $\phi (\cdot ,\xi)$ , we aim to alleviate the catastrophic forgetting problem, and (2) for the linear projection  $\theta$ , we aim to eliminate the bias generated during continuous training. For the former, we propose a dynamic knowledge distillation-based technique to alleviate the catastrophic forgetting problem during multi-stage continual training (Section 3.2). For the latter, we incorporate the model with a bias-correction module that eliminates the bias of projection weights (Section 3.3).
3.2 Alleviate the Catastrophic Forgetting Issue
As discussed above, we propose to alleviate the catastrophic forgetting in the encoder and decoder under the continual training setting.
3.2.1 Definition
We consider the scenario where the training is comprised of  $m$  stages, denoted by  $k = 1,\dots ,m$ . At  $k$ -th stage, a subset of data  $\{x_{k}^{(i)},y_{k}^{(i)}\}_{i = 1}^{T_{k}}$  are fed to the model, where  $T_{k}$  refers to the number of samples at  $k$ -th stage,  $x_{k}^{(i)}$  refers to  $i$ -th sample at  $k$ -th stage.
Assuming that  $u_{k}(\cdot)$  is a gold function sampled from an unknown distribution  $\mathbb{P}_y$  that maps each  $x_{k}^{(i)}$  to  $y_{k}^{(i)}$  at stage  $k$ , i.e.,  $y_{k}^{(i)} = u_{k}(x_{k}^{(i)})$ . Under the continual learning setting, our goal is to learn a deep neural model  $g(\cdot; w)$  parameterized by  $w$ , such that  $g(\cdot; w)$  not only fits well to  $u_{k}(\cdot)$ , but also  $u_{k-1}(\cdot), u_{k-2}(\cdot), \dots, u_{1}(\cdot)$  received in early stages to alleviate the catastrophic forgetting.
3.2.2 Formulation
Considering that in some cases, recent data is more important than early data, we set a discount (Sutton and Barto, 1998)  $\alpha^s$  to  $u_{k - s}(\cdot)$ , and minimize the cross-entropy loss between model output  $g(\cdot; w)$  and weighted sum of  $u_k(\cdot)$ :
$$
\min  \mathcal {L} _ {k} \left(w _ {k}\right) \triangleq - \sum_ {i = 1} ^ {T _ {k}} z _ {k} \left(x _ {k} ^ {(i)}\right) \times \log g \left(x _ {k} ^ {(i)}; w _ {k}\right) \tag {2}
$$
where  $z_{k}(x)$  is the normalized sum of  $u_{k}(\cdot)$ :
$$
z _ {k} (x) = \frac {1 - \alpha}{1 - \alpha^ {k}} \sum_ {s = 0} ^ {k - 1} \alpha^ {s} u _ {k - s} (x) \tag {3}
$$
Notice that with  $\alpha$  close to 1, minimizing  $\mathcal{L}_k(w_k)$  is closely related to minimizing
$\sum_{i=1}^{T_k} \left( \mathrm{E}_{u \sim \mathbb{P}_y} u(x) \right) \times \log g(x^{(i)}; w_k)$ . In our experiments, we set  $\alpha = 0.999$  for the case which the data from different stages have no priority.
For an input  $x$  in stage  $k$ , the computation of  $z_{k}(x)$  needs us to get the value of  $\{u_{s}(x)\}_{s = 1}^{k}$  first. A simple but inefficient way is to store the outputs or a learned approximation of  $u_{s}(x)$  of every stages, which means that we need to store  $m$  models if the training is comprised of  $m$  stages. To reduce the space overhead, we rewrite Eq. 3 as
$$
\begin{array}{l} z _ {k} (x) = \frac {1 - \alpha}{1 - \alpha^ {k}} [ u _ {k} (x) + \alpha u _ {k - 1} (x) + \dots + \alpha^ {k - 1} u _ {1} (x) ] \\ = \frac {1 - \alpha}{1 - \alpha^ {k}} \left[ u _ {k} (x) + \alpha \left(u _ {k - 1} (x) + \dots + \alpha^ {k - 2} u _ {1} (x)\right) \right] \\ = \frac {1 - \alpha}{1 - \alpha^ {k}} \left[ u _ {k} (x) + \alpha \frac {1 - \alpha^ {k - 1}}{1 - \alpha} z _ {k - 1} (x) \right] \\ = \frac {1 - \alpha}{1 - \alpha^ {k}} u _ {k} (x) + \alpha \frac {1 - \alpha^ {k - 1}}{1 - \alpha^ {k}} z _ {k - 1} (x) \tag {4} \\ \end{array}
$$
Let  $\lambda_{k} = \alpha \frac{1 - \alpha^{k - 1}}{1 - \alpha^{k}}$ , notice that  $\frac{1 - \alpha}{1 - \alpha^k} + \alpha \frac{1 - \alpha^{k - 1}}{1 - \alpha^k} = 1$ , we have:
$$
z _ {k} (x) = \left(1 - \lambda_ {k}\right) u _ {k} (x) + \lambda_ {k} z _ {k - 1} (x) \tag {5}
$$
Eq. 5 reveals that  $z_{k}(x)$  can be derived from  $z_{k-1}(x)$  and  $u_{k}(x)$ , so we can instead seek to calculate  $z_{k-1}(x)$  to avoid storing too many sub-models.
Since in the last stage, we make the distribution of  $g(x; w_{k-1})$  be as similar to  $z_{k-1}(x)$  as possible by minimizing their cross-entropy. Therefore, in  $k$ -th stage, we use  $g(x; w_{k-1})$  to approximate  $z_{k-1}(x)$ .
The training objective of our model at  $k$ -th stage can be written as:
$$
\begin{array}{l} \min  \hat {\mathcal {L}} _ {k} \left(w _ {k}\right) \triangleq - \sum_ {i = 1} ^ {T _ {k}} \left[ \left(1 - \lambda_ {k}\right) u _ {k} \left(x _ {k} ^ {(i)}\right) + \lambda_ {k} g \left(x _ {k} ^ {(i)}; w _ {k - 1}\right) \right] \\ \times \log g \left(x _ {k} ^ {(i)}; w _ {k}\right) \tag {6} \\ \end{array}
$$
3.2.3 Relevance to Knowledge Distillation
The proposed method can also be regarded as a special kind of knowledge distillation. To explain this, we rewrite Eq. 6 as
$$
\begin{array}{l} \hat {\mathcal {L}} _ {k} (w _ {k}) = - \sum_ {x _ {k}} z _ {k} (x _ {k}) \times \log g (x _ {k}; w _ {k}) \\ = - \sum_ {x _ {k}} [ (1 - \lambda_ {k}) u _ {k} (x _ {k}) + \lambda_ {k} z _ {k - 1} (x) ] \times \log g (x _ {k}; w _ {k}) \\ = - \left(1 - \lambda_ {k}\right) \sum_ {x _ {k}} u _ {k} \left(x _ {k}\right) \times \log g \left(x _ {k}; w _ {k}\right) \\ - \lambda_ {k} \sum_ {x _ {k}} z _ {k - 1} \left(x _ {k}\right) \times \log g \left(x _ {k}; w _ {k}\right) \tag {7} \\ \end{array}
$$
The first term in Eq. 7 minimizes a cross-entropy loss between gold label  $y_{k} = u_{k}(x_{k})$  and model output  $g(x_{k};w_{k})$ , which is a standard translation loss. The second term in Eq. 7 minimizes the cross-entropy between the model's output of last stage  $g(x;w_{k - 1})$  and current stage  $g(x;w_k)$ . If we consider the trained model of last stage as the "teacher", and the model of current stage as the "student", then this is a standard knowledge distillation loss.
Therefore, the proposed method can also be seen as optimizing a weighted sum of translation and distillation loss, which is similar to Khayrallah et al. (2018). The difference is that Khayrallah et al. (2018) only consider the case where the training is comprised of two stages, and thus they use a fixed  $\lambda = 0.1$  in Eq. 7, i.e.,
$$
\begin{array}{l} \hat {\mathcal {L}} ^ {\prime} \left(w _ {k}\right) = - (1 - \lambda) \sum_ {x _ {k}} u _ {k} \left(x _ {k}\right) \times \log g \left(x _ {k}; w _ {k}\right) \tag {8} \\ - \lambda \sum_ {x _ {k}} z _ {k - 1} (x _ {k}) \times \log g (x _ {k}; w _ {k}) \\ \end{array}
$$
When applying Eq. 8 to multi-stage incremental training, it is easy to deduce that they actually fit a  $z_{k}^{\prime}(x) = \sum_{s=0}^{k-1} \lambda^{s} (1 - \lambda) u_{k-s}(x)$  at the  $k$ -th stage, which means that the weights of old knowledge are always lower. When  $\lambda < 1$ , the model will always pay more attention to new data and decay the weights of old knowledge at an exponential rate. Under this case, the model will quickly forget the general knowledge learned from earliest stage and overfit the new data. On the other hand, if choose  $\lambda$  close to 1, the model hardly learns new knowledge as the weight of translation loss close to 0. During experiments, we find that  $\lambda = 0.7$  works well for this method, so we set  $\lambda = 0.7$  in the following experiments.
Our method adjusts the weight  $\lambda_{k}$  dynamically and gradually increases the weight of distilled loss  $(\lambda_{k} = \alpha \frac{1 - \alpha^{k - 1}}{1 - \alpha^{k}})$ . Therefore, our model can balance the learning of new knowledge and memorization of old knowledge. We name the proposed method as "dynamic knowledge distillation".
3.3 Eliminate the Bias in Linear Projection
3.3.1 Biased Weights In the Linear Projection
To reveal the bias weights phenomenon in the linear projection in continual training, we conduct a test that first trains an English-German NMT model on an IT-related corpus, and then fine-tunes it on law-related corpus. We find that after fine-tuning on law-related data, the model will no longer generate IT-specific words even we feed an IT-related
3The number of training samples for IT and law corpus are K and 205K respectively.
Figure 1: The changes of  $\eta$  with the training of Model-1 and Model-2. This figure shows that directly finetuning the model on new data will cause the biased weights problem.
source sentence to the model. As a consequence, the model performs extremely poorly on the IT test set.
We hypothesize that the model reduces the old words' probability by shrinking their corresponding weights in the last linear projection  $\Theta$ . To verify this, we train two models simultaneously: one is trained on combined IT-related and law-related corpus (referred to as Model-1), and the other is trained on IT-related corpus first, and then fine-tuned on the law-related corpus (referred as Model-2).
Denote  $\eta$  as the ratio of new words weights and old words weights in the last linear projection:
$$
\eta = \left(\frac {1}{n _ {\text {n e w}}} \sum_ {\theta \in \Theta_ {\text {n e w}}} \| \theta \|\right) / \left(\frac {1}{n _ {\text {o l d}}} \sum_ {\theta \in \Theta_ {\text {o l d}}} \| \theta \|\right) \tag {9}
$$
We calculate the changes of  $\eta$  with the training of Model-1 and Model-2 respectively and plot the results in Fig. 1. Since Model-1 can achieve good performance on both IT and law test sets, we consider its weights' ratio as the "ground truth".
Fig. 1 shows that compared to Model-1, Model-2's norm of the weights for new words is much higher than those for old words as the training goes by. In Eq. 1, if  $i$ -th word should be picked out, then  $\theta_i^\top \phi(x, w)$  should be a positive number. In this case, decreasing  $\|\theta_i\|$  will reduce the probability of generating  $i$ -th word. This results in the bias in the generation that favors new words.
3.3.2 Weight Normalization for Bias Correction
Based on the above observation, we propose to add a weight normalization module similar to Nguyen and Chiang (2018) in the linear projection.
Concretly, we normalize the weights for all words by:
$$
\hat {\theta} _ {i} = \theta_ {i} / \| \theta_ {i} \| \tag {10}
$$
and compute the probability of generating each word as:
$$
\hat {p} _ {i} (x) = \frac {\exp \left\{\gamma \cdot \hat {\theta} _ {i} ^ {\top} \phi (x , w) \right\}}{\sum_ {j} \exp \left\{\gamma \cdot \hat {\theta} _ {j} ^ {\top} \phi (x , w) \right\}} \tag {11}
$$
where  $\gamma$  is a (learnable) scaling scalar. The introduction of  $\gamma$  is to control the peakiness of the softmax distribution.
Notice that since the encoder and decoder are shared and always used for data from different domains, they do not suffer the biased weights problem.
We consider the following three representative training scenarios in NMT systems:
- In-domain incremental training: We split the training data in the same domain into  $m$  sets, and fed one set of data to the model at each stage. We share the same validation and test sets among different stages in this setting. Notice that since the data in different stages are from the same domain, we do not incorporate the bias-correction module under this setting.
- Domain-incremental training: We first train the model on a large-scale general domain corpus<sup>5</sup>, and then fine-tune it on  $m$  new domains successively. We calculate the model's performance on the test sets of general and the new domains at each stage.
- Time-incremental training: Time-incremental training is a special case of in-domain incremental training, where the training data come from different time and are fed to the model in chronological order. We set this scenario to simulate the training of NMT model on real-world time streaming data.
WMT14 News Commentary.
4In transformer model,  $\theta_i^\top \phi (x,w)$  will  $\approx 0$  for most words, but for those words that are likely generated,  $\theta_i^\top \phi (x,w)$  will  $>0$
IWSLT2013	20%	40%	60%	80%	100%
Combined	24.98	32.18	35.19	37.00	37.72
Fine-tuning	24.98	29.25*	32.57*	34.09*	34.51*
+ knowledge distill.	24.98	30.69 (+1.44)	33.46 (+0.89)	34.61 (+0.52)*	34.85 (+0.34)*
+ EWC reg.	24.98	29.65 (+0.40)*	32.75 (+0.18)*	34.13 (+0.04)*	34.43 (-0.08)*
Ours	24.98	30.94 (+1.69)	33.49 (+0.92)	34.96 (+0.87)	35.20 (+0.69)
WMT14	20%	40%	60%	80%	100%
Combined	16.75	19.91	23.82	25.36	27.14
Fine-tuning	16.75	18.67*	22.07*	23.36*	25.25*
+ knowledge distill.	16.75	19.19 (+0.52)*	22.82 (+0.75)	23.88 (+0.52)*	25.90 (+0.65)*
+ EWC reg.	16.75	18.70 (+0.03)*	22.41 (+0.34)*	23.84 (+0.48)*	25.54 (+0.29)*
Ours	16.75	19.44 (+0.77)	23.02 (+0.95)	24.17 (+0.81)	26.22 (+0.97)