SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1: Experiment results of different models under in-domain incremental training setting on IWSLT2013 and WMT14 datasets. Best results are highlighted in bold. Statistically significant improvements  $(p < 0.1)$  over our method are marked with  $*$ .
Method	+ It	+ Koran	+ Law	+ Medical	+ Subtitles
Fine-tuning	44.38	23.41	57.71	54.65	30.02
+ knowledge distill.	44.36 (-0.02)	23.50 (+0.09)	57.54 (-0.17)	54.49 (-0.16)	29.91 (-0.11)
+ EWC reg.	44.12 (-0.26)	22.94 (-0.47)	57.10 (-0.61)	54.03 (-0.62)	29.51 (-0.51)
Ours	44.41 (+0.03)	23.49 (+0.08)	57.52 (-0.19)	54.58 (-0.07)	29.87 (-0.15)
w/o Dynamic KD.	44.09 (-0.29)	23.09 (-0.32)	57.24 (-0.47)	54.03 (-0.62)	29.43 (-0.59)
w/o BiC.	44.34 (-0.04)	23.36 (-0.05)	57.46 (-0.25)	54.43 (-0.22)	29.73 (-0.29)
Table 2: Experiment results of different models under domain-incremental training setting. The best results are highlighted in bold.
In our experiments, we set  $m = 5$ . Following previous works on lifelong learning (Aljundi et al., 2017; Triki et al., 2017; Aljundi et al., 2018; Hou et al., 2019; Wu et al., 2019), we use a memory with fixed capacity to reserve the training examples sampled from old data. The data stored in the memory and the new data are together fed to the model at each stage. The memory size is set to 50, 000 in our experiments.
4.1.1 Data Preparation
We use the IWSLT2013 de-en translation data<sup>6</sup> and WMT14 de-en translation data<sup>7</sup> for in-domain incremental training. The number of training samples of IWLST2013 dataset is 206,122 in total, and we use 41,224 samples to train the model at each stage. The validation and test sets are shared among all stages, and the numbers of validation and test samples are 3,000. The number of training samples of WMT14 dataset is 4,500,000 in total.
We use the new data split of OPUS multi-domain dataset released by Aharoni and Goldberg for domain-incremental training. This dataset con
tains de-en data from IT, koran, law, medical, and subtitles fields. The numbers of training samples for these domains are 222,927, 17,982, 467,309, 248,099 and 500,000, respectively. The numbers of validation and test samples are 2,000 for each domain.
We use WMT news-commentary 2015-2019 deen translation data for time-incremental training. The WMT news-commentary data was first built in 2015 and some new data was added in each subsequent year. News-commentary 2015 contains 216,897 training samples, and 26,576, 27,999, 12,774 and 54,038 new samples are added in 2016-2019, respectively. The test sets contain 3,000 samples for each year. Notice that each year's test set may contain test samples from previous years. For example, the 2017 test set contains both new test samples from 2017 and some old test samples from 2015 and 2016.
We use the following competitive models for comparison in experiments:
- Fine-tuning This model is directly fine-tuned on new data.
http://www.statmt.org/
$^{6}$ http://workshop2013.iwslt.org/59.php
7http://www.statmt.org/wmt14/
<sup>8</sup>https://github.com/roeeaharoni/
unsupervised-domain-clusters
- Combined This model is trained on combined new data and old data from scratch, which is considered the upper bound in the field of continual learning.
- Knowledge Distillation (KD) (Khayrallah et al., 2018) When fine-tuning on current set of data, this model optimizes a weighted sum of NLL loss and regularization term:  $\mathcal{L}(w) = (1 - \alpha)\mathcal{L}_{nll}(w) + \alpha \mathcal{L}_{reg}(\theta)$ . The regularization term is formulated in the spirit of knowledge distillation that minimizes the cross-entropy between in-domain (teacher) model's output distribution and that of the out-of-domain (student) model. The value of  $\alpha$  is fixed at every stage.
- Elastic Weight Consolidation (EWC) (Saunders et al., 2019; Thompson et al., 2019) This model optimizes a weighted sum of NLL loss and EWC term. We recommend readers refer to their papers for more details.
For the convenience of narration, we refer to the knowledge distillation, elastic weight consolidation, and our proposed method as "learningwithout-forgetting (LWF)"-based methods. To study the effectiveness of different components of our proposed method, we also test the following variants of our model:
- w/o dynamic knowledge distillation It removes the dynamic knowledge distillation module from the proposed model.
- w/o bias correction It removes the bias correction module from the proposed model.
We use the Fairseq toolkit (Ott et al., 2019) to implement the proposed model. We process the text into subword units by using the subword-nmt toolkit<sup>10</sup>.
We adopt the transformer (Vaswani et al., 2017) as the model architecture. We set the model's hidden size, feed-forward hidden size to 512, 2048, and set the number of layers and the number of heads to 6 and 8, respectively. We use the same configuration for all encoders and decoders.
For training and inference, we use Adam optimizer (Kingma and Ba, 2014) and use the same parameters and learning rate schedule as previous
work (Vaswani et al., 2017). We use warm-up learning rate (Goyal et al., 2017) for the first 3,000 steps, and the initial warm-up learning rate is set to  $1e - 7$ . We use the dropout technique and set the dropout rate to 0.4. We use beam search for inference, and the beam size is set to 5.
The max update steps of each model are different, depending on when they converge.
5.1 In-Domain Incremental Training
The experimental results of in-domain incremental training are shown in Table 1. Notice that the combined model is trained on all data observed so far, and it serves as the upper bound in this setting and will not participate in the comparison.
It first can be seen that there is a gap between the fine-tuning model and combined model, which suggests that there is some amount of general knowledge that has been forgotten by the model during fine-tuning. The performance improved when incorporating knowledge distillation, EWC regularization, or the proposed dynamic knowledge distillation techniques into the fine-tuning process, which shows that learning-without-forgetting strategies can help the model remember the general knowledge and benefit the fine-tuning. The improvement is less significant for the EWC-based model.
By comparing results of our model with the knowledge distillation-based and EWC regularization-based methods, we can see that our model outperforms them in all cases. The proposed model achieves an average improvement of 0.3 and 0.8 BLEU scores compared to the knowledge distillation-based and EWC regularization-based methods, respectively.
The above results confirm the finding of prior works that the learning-without-forgetting strategies can benefit the continual training, and demonstrate that the proposed method adds more gains.
We also study the effect of  $\alpha$  in Eq. 3. A small value of  $\alpha$  indicates that the model will pay more attention to new data, and penalize less for forgetting old knowledge. The detailed experiment results are shown in Table 3. We can observe that when  $\alpha$  is larger than 0.5, the proposed method can achieve good performance, and the model achieves the best BLEU scores when  $\alpha = 0.5$  or  $\alpha = 0.7$ .
$^{10}$ https://github.com/rsennrich/subword-nmt
Method	20%	40%	60%	80%	100%
Upper bound	24.98	32.18	35.19	37.00	37.72
Baseline	24.98	29.25	32.57	34.09	34.51
Ours (α = 0.999)	24.98	30.59	33.22	34.45	35.04
Ours (α = 0.9)	24.98	30.71	33.19	34.51	35.12
Ours (α = 0.7)	24.98	31.04	33.52	34.91	35.12
Ours (α = 0.5)	24.98	30.96	33.49	34.96	35.20
Ours (α = 0.3)	24.98	30.23	33.12	34.82	34.88
Ours (α = 0.1)	24.98	30.18	32.99	34.29	34.34
Table 3: The effect of  $\alpha$  in the dynamic knowledge distillation module. The proposed method can achieve good results when  $\alpha >0.5$
Figure 2: The performance of our model and competitive models on general test set after each stage. Baseline is the model that directly fine-tuned on new data. The proposed method significantly outperform competitive methods in all cases.
5.2 Domain-Incremental Training
In this setting, we first train a general NMT model on the large-scale WMT16 de-en dataset, and then fine-tune the model on IT, koran, law, medical, and subtitles domain sequentially. Considering that these domains have no priority to each other, so we set  $\alpha = 0.999$  (approximate 1) in Eq. 3.
To explore the degree to which the model forgets old knowledge during incremental training, after each incremental training phase, we report the results of the models on the general domain (WMT16 de-en) test set. We present the experimental results of this part in Fig. 2, and we also present the results of the ablation study in Fig. 3. Due to the forgetting of old knowledge, the result is a descending curve of the BLEU score after each phase.
We can see from Fig. 2 that our model outperforms all competitive models at any stage. Incorporating the proposed method to the fine-tuning can bring an improvement of 3-4 BLEU scores in the general domain, indicating that our proposed method can effectively alleviate the catastrophic forgetting issue, and maintain the performance of
the model on old data.
It seems that the largest drop in performance happens at the first training step. This is because the "private knowledge" of the general domain will be covered by the new knowledge mostly at the first training step, while the few remaining knowledge will be gradually covered in the later steps. The results also show that when fine-tuning on the new domain that contains more training samples, the occurrence of catastrophic forgetting would be more obvious, and our method can gain more improvements.
The knowledge distillation-based method can also improve the results on the general domain, but the improvement is lower than ours. This is because the underlying thought of Eq. 8 is to attenuate old knowledge at an exponential rate (when  $k = 5$ , the coefficient of  $u_{1}(x)$  is 0.072). Thus after several stages, the model will focus more on new data and neglect old data.
We also analyze the representations of sentences in different stages and investigate how they evolve over time. For this purpose, we compute the average sentence representation  $s$  in general domain, and compute the ratio of changes  $\left\| s_{t + 1} - s_t\right\| /\left\| s_t\right\|$  at each stage. We find that our method leads to fewer changes compared to baseline model (0.16 vs. 0.21), indicating that our method is better at preserving previously learned knowledge.
We also study whether the introduction of these "learning-without-forgetting" strategies will harm the domain transfer, i.e., decreasing the results of the model on the current/new domain. Therefore, we also report the results of the model on the current domain. These results are shown in Table 2. Due to the imbalanced training data in different domains, the combined model performs poorly in some domains, especially those with small training samples, so we do not report the results of the combined model under this setting.
The results in Table 2 show that our model performs slightly better or at least comparable to the model that is directly fine-tuned on new data. We hypothesize that this is because the proposed method reserves general knowledge learned from the general domain corpus, such as the basic grammar and word semantics, to the continual training model when fine-tuned on new data. Therefore encouraging the model to remember this knowledge can better help the model leverage general knowledge to improve performance on new do
Method	2015	+ 2016	+ 2017	+ 2018	+ 2019
Combined (Upper Bound)	29.03	32.41	37.69	46.22	35.38
Fine-tuning	29.03	31.97	37.07	45.34	34.51
+ knowledge distill.	29.03	32.29 (+0.32)	37.43 (+0.36)	45.83 (+0.49)	34.81 (+0.30)
+ EWC reg.	29.03	32.07 (+0.10)	37.38 (+0.31)	45.69 (+0.35)	34.67 (+0.16)
Ours	29.03	32.27 (+0.30)	37.55 (+0.48)	46.08 (+0.74)	35.06 (+0.55)
w/o Dynamic KD.	29.03	32.04 (+0.07)	37.19 (+0.12)	45.51 (+0.17)	34.63 (+0.12)
w/o BiC.	29.03	32.19 (+0.22)	37.45 (+0.38)	46.06 (+0.72)	34.91 (+0.40)
Table 4: Experiment results of different models in time-incremental training setting. Best results are highlighted in bold. The combined model serves as the upper bound in this setting and will not participate in the comparison.
mains. This observation is consistent with some previous work (Khayrallah et al., 2018).
The results of the ablation study in Fig. 3 show that both the dynamic knowledge distillation and bias correction module contribute to the improvement of the results. Although the bias correction module is simple, it plays a very important role in the proposed model. After removing the bias correction module, the result of the model drops by 0.9-2.1 BLEU scores.
5.3 Time-Incremental Training
Table 4 shows the results of different models in time-incremental training setting. Since the test set of each year is a combination of old and new test samples, we directly report the results of different models on current year's test set. The combined model serves as the upper bound and will not participate in the comparison.
As expected, the proposed model outperforms competitive models in most cases. There is an improvement of 0.3-0.8 BLEU scores over the finetuned model, 0-0.3 BLEU scores over the knowledge distillation-based model, and 0.2-0.5 BLEU scores over the EWC regularization-based model. These results show that the proposed method for continual training is effective.
The results of ablation study show that the bias correction module is less beneficial for the model under this setting as the removal of bias correction module only results in a decrease of 0.1-0.2 BLEU score to the performance. We hypothesize that this is because the domain variation among test sets from 2015 to 2019 is smaller than that in domain-incremental experiments. Therefore, the biased weights phenomenon is slighter in this case.
In this paper, we propose a new continual learning framework for neural machine translation. We
first propose a dynamic knowledge distillation-based method to alleviate the problem of catastrophic forgetting in a multi-stage view, and then propose a bias-correction module to address the biased weights issue. To verify the effectiveness of the proposed method, we conduct experiments in three different settings: in-domain incremental training, time-incremental training, and domain-incremental training. Experimental results show that the proposed method can obtain superior performance compared to competitive models.
In the future, we will apply the proposed method to other NLP tasks to test its robustness.
This work was partially supported by National Natural Science Foundation of China (61772036), Beijing Academy of Artificial Intelligence (BAAI) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Roeaharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7747-7763. Association for Computational Linguistics.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part III, volume 11207 of Lecture Notes in Computer Science, pages 144-161. Springer.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuyte-laars. 2017. Expert gate: Lifelong learning with
a network of experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 7120-7129. IEEE Computer Society.
Peter F. Brown, John Cocke, Stephen Della Pietra, Vincent J. Della Pietra, Frederick Jelinek, John D. Lafferty, Robert L. Mercer, and Paul S. Roossin. 1990. A statistical approach to machine translation. Comput. Linguistics, 16(2):79-85.
Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for neural machine translation. CoRR, abs/1612.06897.
Robert M. French. 1993. Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented? In Advances in Neural Information Processing Systems 6, [7th NIPS Conference, Denver, Colorado, USA, 1993], pages 1176-1177. Morgan Kaufmann.
Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2018. Lifelong learning via progressive distillation and retrospection. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part III, volume 11207 of Lecture Notes in Computer Science, pages 452-467. Springer.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2019. Learning a unified classifier incrementally via rebalancing. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 831-839. Computer Vision Foundation / IEEE.
Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. 2018. Regularized training objective for continued training for domain adaptation in neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018, pages 36-44. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic. The Association for Computational Linguistics.
Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In International Workshop on Spoken Language Translation.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421. The Association for Computational Linguistics.
Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 312-318. Association for Computational Linguistics.
Toan Q. Nguyen and David Chiang. 2018. Improving lexical choice in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 334-343. Association for Computational Linguistics.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 7-12 July 2003, Sapporo Convention Center, Sapporo, Japan, pages 160-167. ACL.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the NAACL-HLT, pages 48-53.
Danielle Saunders, Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2019. Domain adaptive inference for neural machine translation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 222-228. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the
54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Felix Stahlberg. 2019. Neural machine translation: A review. CoRR, abs/1912.02047.
Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 9(5):1054-1054.
Brian Thompson, Jeremy G Winnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2062-2068. Association for Computational Linguistics.
Amal Rannen Triki, Rahaf Aljundi, Matthew B. Blaschko, and Tinne Tuytelaars. 2017. Encoder based lifelong learning. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 1329-1337. IEEE Computer Society.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. 2019. Large scale incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 374-382. Computer Vision Foundation / IEEE.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995. PMLR.