SlowGuess/NAACL_2022_OCR huggingface dataset:
For the task of continual MRC, the overall results under CDA-C and CDA-Q setting and are shown in Table 2 and Table 3. Taking  $F1_{avg}$  and  $F1_{all}$  as two overall performance evaluations, we have the following findings: (1) The proposed MA-MRC methods outperform other baselines by a significant margin on both two continual settings. (2) The performance trend of different methods are not consistent across two continual settings. For exam
Table 5: The ablation study results under CDA-Q setting.
Data	D1	D2	D3	D4	D5
MA-MRC:
step 1:	80.13	-	-	-	-
step 2:	74.92	69.16	-	-	-
step 3:	64.94	59.53	69.25	-	-
step 4:	74.13	62.87	65.12	60.36	-
step 5:	70.11	59.35	61.92	52.18	59.51
UpperBound:
step 1:	80.13	-	-	-	-
step 2:	79.51	68.66	-	-	-
step 3:	78.42	67.64	70.03	-	-
step 4:	76.91	66.60	67.51	59.36	-
step 5:	78.24	67.41	68.00	58.63	57.59
MA-MRC w/o Adv:
step 1:	80.13	-	-	-	-
step 2:	76.58	68.82	-	-	-
step 3:	68.83	61.14	69.74	-	-
step 4:	73.67	60.37	66.5	60.13	-
step 5:	65.68	56.99	59.81	52.47	59.93
Table 6: Catastrophic forgetting phenomenon of proposed MA-MRC(w/o Adv) and UpperBound mehtods under CDA-C setting.
ple, OnlineEWC performs well under CDA-C but performs poorly CDA-Q, and all memory-based methods outperform consolidation-based methods under CDA-Q but not CDA-C. The reason is that the characteristics of the domains and the continual learning difficulty are different. Concretely, it is obvious that the domain data in CDA-C is more different and more difficult than that in CDA-Q. (3) There is a big gap between all the models and the upper bound. We further demonstrate the evaluation results of the proposed MA-MRC and Upper-Bound methods on each domain at each continual learning step  $t$  in Table 6. The results indicate that MA-MRC indeed forgets knowledge learned on previously seen domain data, and there remain lots of things to be explored for continual MRC.
Besides, we plot the average F1 performance of models during the whole continual learning process in Figure 3 to investigate how performance
Sampling	SQuAD	NaturalQA	HotpotQA	NewsQA	TriviaQA	F1avg	F1all
uentropy, norm1	68.13	60.33	61.25	52.2	60.03	60.39	61.57
uentropy, norm2	70.11	59.35	61.92	52.18	59.51	60.61	61.77
uprob, norm1	68.71	59.30	62.00	52.79	59.94	60.55	61.55
uprob, norm2	63.78	60.42	59.88	52.72	59.54	59.27	60.25
Random	65.68	56.99	59.81	52.47	59.93	58.97	59.71
Table 7: The results of different sampling straties under CDA-C setting.
M	methods	F1avg	F1all
200	MA-MRC(norm1)	53.76	53.91
MA-MRC(norm2)	56.13	56.66
300	MA-MRC(norm1)	58.39	59.08
MA-MRC(norm2)	58.14	58.88
400	MA-MRC(norm1)	60.39	61.57
MA-MRC(norm2)	60.62	61.77
Table 8: Results with different  $\mathcal{M}$  under CDA-C.
F1avg	F1all
order 1	DER++	70.21	64.35
AGEM	57.57	30.18
MA-MRC(norm1)	71.83	71.41
MA-MRC(norm2)	72.77	72.48
order 2	DER++	69.9	76.2
AGEM	74.97	77.02
MA-MRC(norm1)	75.41	77.5
MA-MRC(norm2)	75.27	78.01
order 3	AGEM	66.82	68.82
MA-MRC(norm1)	72.64	74.27
MA-MRC(norm2)	72.21	73.73
changes. We observe that the performance of all models decreases in some degree with increasing numbers of domains under both CDA-C and CDA-Q settings. However, the proposed MA-MRC methods outperform other baselines and achieve better performance on the whole domain data.
4.5.2 Ablation Study
To better understand our proposed model, we conduct ablation studies to see the effectiveness of each model component. The results in Table 4 and Table 5 demonstrate that both knowledge distillation and adversarial training contribute to avoiding largely forgetting. For incremental domain learning, knowledge distillation is a naive way to enforce the model to remember its original responses for previous domains. What is more, the adversarial domain adaptation, can make MRC model learn domain-invariant and transfer representations better. Table 6 shows the evaluation results of w/o Adv at each continual step. Though w/o Adv obtains a slightly higher performance on the last two domains than MA-MRC model, it has a more harm-
Table 9: The results of different domain order under CDA-Q setting.
Para	Training Time / 1 epoch
CDA-C	CDA-Q
AGEM	θT,AV	77 m	49 m
DER	θT,AV	57 m	38 m
DER++	θT,AV	56 m	38 m
MA-MRC(norm1)	θT,AV,D	61 m	40 m
MA-MRC(norm2)	θT,AV,D	60 m	39 m
Table 10: Parameters and speed comparison.
ful forgetfulness on the very previous seen domain. The results prove that the adversarial learning indeed helps for remembering previous knowledge.
4.5.3 Effect of Memory
Memory Size  $\mathcal{M}$ . Table 8 shows the performance with three different memory size  $\mathcal{M}$ : 200, 300, and 400. In low memory scenario, i.e.,  $\mathcal{M} = 200$ , the proposed method performs poorly. The reason is that the adversarial domain adaption has difficulty transferring well with a too small memory. Therefore, as the number of memory samples increases, it will be more conducive to transfer to a new domain, and the overall performance will be better. We believe that an appropriate memory size could lead to better performance.
Uncertainty-aware Sampling. We replace the uncertainty-aware memory updating strategy with another two strategies. First, we use another uncertainty measurement that takes heuristic max softmax probability of spans as the uncertainty:  $u^{prob} = \max_{i,j}(p_i^{start} + p_j^{end})$ . The second strategy is random sampling. The experimental results in Table 7 indicate that the uncertainty-aware sampling (both Entropy-based and Probability-based uncertainty) is better than random sampling.
4.5.4 Effect of Domain Order
Table 9 shows the results of different domain orders. Order 1 is a descending order based on the number of training samples in each domain and order 2 is an ascending order, and order 3 is a random order. The performance of AGEM and DER++ degrades severely in order 1. However, the proposed MA-MRC methods is superior to baselines and are stable and robust under different orders.
4.5.5 Efficiency Analysis
We compare the parameters and training speed of methods with the same size of the memory in Table 10. The MA-MRC have more parameters for domain discriminator. Nevertheless, considering the larger number of parameters of the Transformer, we conclude all methods have almost the same number of parameters. As for training time per epoch, MA-MRC is slower than DER++ (4%/5% under CDA-C/CDA-Q). It can be accepted because of the significant improvement of MA-MRC.
In this paper, an incremental learning MRC model with uncertainty-aware fixed memory and adversarial domain adaptation, MA-MRC, is proposed for continual MRC and alleviating catastrophically forgetting. Inspired by the human learning process, There are two main ideas of MA-MRC: a memory that stores a small number of samples in previous seen domain data and always focuses on what the model most needs to replay; and adversarial learning the domain adaptation in a two-player game to learn better transfer representations between previous and current domain data. Experimental results show that the proposed MA-MRC can achieve a good continuous learning performance without catastrophically forgetting under CDA-C and CDA-Q settings.
In the future, we would like to explore a more effective sampling strategy, domain adaptation strategy, and balance training strategy for multiple objectives to enhance the continual MRC model.
This paper is funded by National Natural Science Foundation of China (Grant No. 62173195). This work is partly supported by seed fund of Tsinghua University (Department of Computer Science and Technology) -Siemens Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things.
Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and Gerhard Weikum. 2018. Never-ending learning for open-domain question answering over knowledge bases. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1053-1062.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Mach. Learn., 79(1-2):151-175.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yu Cao, Meng Fang, Baosheng Yu, and Joey Tianyi Zhou. 2020. Unsupervised domain adaptation on reading comprehension. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7480-7487.
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13122-13131.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019, Hong Kong, China, November 4, 2019, pages 1-13.
Robert M. French. 1999. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-135.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. 2016. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17:59:1-59:35.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601-1611.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526.
Seanie Lee, Donggyu Kim, and Jangwon Park. 2019. Domain-agnostic question-answering with adversarial training. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, MRQA@EMNLP 2019, Hong Kong, China, November 4, 2019, pages 196-202.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6467-6476.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul A. Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. 2021. Continual learning in task-oriented dialogue systems. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7452-7467.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,  $000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. 2019. Learning to learn without forgetting by maximizing transfer and minimizing interference. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & compress: A scalable framework for continual learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, pages 4535-4544.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017.
Yeon Seonwoo, Ji-Hoon Kim, Jung-Woo Ha, and Alice Oh. 2020. Context-aware answer extraction in question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2418-2428.
Lixin Su, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2020. Continual domain adaptation for machine reading comprehension. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 1395-1404.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017, Vancouver, Canada, August 3, 2017, pages 191-200.
Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, and Hongning Wang. 2019. Adversarial domain adaptation for machine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2510-2520.
Zhijing Wu and Hua Xu. 2020. Improving the robustness of machine reading comprehension model with hierarchical knowledge and auxiliary unanswerability prediction. Knowl. Based Syst., 203:106075.
Ju Xu and Zhanxing Zhu. 2018. Reinforced continual learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 907-916.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotq: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369-2380.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. 2018. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2021. Retrospective reader for machine reading comprehension. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 14506-14514.