SlowGuess/NAACL_2021_OCR huggingface dataset:
We speculate that the second phase of pre-training will result in the catastrophic forgetting for the pre-trained model, which could hurt the adaptation performance. Figure 2 illustrates that the performance of TAPT without RecAdam keeps dropping as the pre-training continues, and it starts to perform worse than the fine-tuning method after three epochs' pre-training, while the performance of TAPT with RecAdam remains stable at around a 25.5 ROUGE-1 score. We conjecture that excessive pre-training makes the pre-trained model overfit to the pre-training data and partially lose its language
To ensure the comparison between DAPT and TAPT is fair, we sample partial data from the DAPT corpus to make its size comparable to the TAPT corpus and create vocabularies for each based on the top 5K most frequent words (excluding stopwords). The vocabulary for the validation set of the summarization task is also created in the same way.
Source Domains	Target Domains
XSum	CNN/DM	Dialog	Email	Movie R.	Debate	Social M.	Science
Document Summary	354.16	676.03	91.64	124.47	2112.97	196.69	229.31	633.03
21.13	57.91	20.28	4.10	21.28	11.07	6.31	150.01
Table 5: Averaged length of the input documents and output summaries for the source and target domains.
Domains	BART	SDPT	DAPT	SDPT+DAPT
Dialog	39.95	42.84	41.22	42.27
Email	24.71	25.16	26.50	23.71
Movie R.	25.13	25.45	24.25	22.20
Debate	24.48	25.61	26.71	25.16
Social M.	21.76	22.43	22.95	22.03
Science	72.76	73.09	71.88	71.56
Average	34.80	35.76	35.59	34.49
Table 6: ROUGE-1 results for SDPT+DAPT compared to the SDPT, DAPT and BART fine-tuning.
understanding and generation ability. However, the model is required to possess both language ability and domain knowledge for better performance in the domain adaptation task. RecAdam helps the pre-trained model preserve its original language ability while continuing pre-training on a new corpus, which boosts the effectiveness of pre-training. However, as we can see from Table 2, RecAdam fails to improve the performance on DAPT using large corpora. We speculate that the catastrophic forgetting issue does not do much harm to the performance of DAPT because pre-training on the large corpus enables the pre-trained model to possess a good language understanding ability in the target domain even though it could lead to partial forgetting in previous domains, and RecAdam makes DAPT stay somewhere in the middle (not forgetting much the previous learned knowledge, but not learning well in the target domain, either). It indicates that more advanced learning methods are needed for coping with the second pre-training phase on a large corpus.
6.4 Incorporating SDPT and DAPT
Intuitively, incorporating both the summarization task and target domain knowledge into the pretrained model could further boost the domain adaptation performance in the summarization task. Therefore, we propose to combine SDPT and DAPT in the second pre-training step. Since SDPT and DAPT use different objective functions, jointly learning these two tasks will make BART confused about what to generate (summarization or sentence
Domains	BART	SDPT (XSum)	SDPT (CNN)
Dialog	39.95	42.84	43.13
Email	24.71	25.16	23.81
Movie R.	25.13	25.45	24.51
Debate	24.48	25.61	23.98
Social M.	21.76	22.43	22.56
Science	72.76	73.09	72.41
Average	34.80	35.76	35.07
Table 7: ROUGE-1 results for SDPT based on the XSum and CNN/DM (denoted as CNN in the table) datasets.
reconstruction) given the input sequences. To cope with this issue, we use two BART models (one for SDPT and one for DAPT) and share their encoders in this joint pre-training process to learn the knowledge from both the task and domain. Then, we use the BART model for SDPT to fine-tune to the summarization task in the target domain.
As shown in Table 6, the experimental results are contradictory to the intuition. We find that SDPT+DAPT can not further improve upon the performance of SDPT and DAPT. For the dialog and social media domains, the performances of SDPT+DAPT stay between those of SDPT and DAPT, while for the science, movie review and email domains, the performances of SDPT+DAPT are even lower than that of the BART fine-tuning. We conjecture that SDPT and DAPT are two completely different tasks, and jointly pre-training based on them could confuse the model about the knowledge that it learns. However, integrating the task and domain knowledge is still a promising direction for domain adaptation. We leave how to incorporate SDPT and DAPT for future work.
6.5 Different Source Domain Data for SDPT
To explore how different source domain data can affect the performance of SDPT, we use another News domain dataset, CNN/Daily Mail (DM) dataset (Hermann et al., 2015; Nallapati et al., 2016), as the labeled summarization data for SDPT. As we can see from Table 7, SDPT based on CNN/DM only achieves marginal improvements upon the BART fine-tuning baseline in terms of
Figure 3: ROUGE-1 results of BART fine-tuning, DAPT and SDPT over different numbers of training data for email (left) and dialog (right) domains. We consider both low-resource settings (50, 100, 200 and 300 ( $\sim$ 2%) samples), medium-resource settings (25% and 50% samples), and high-resource settings (75% and 100% samples).
the averaged score, and for all the domains, it generally performs worse or similar compared to SDPT based on XSum. Since both of them are from the News domain but the number of training samples in CNN/DM (287k) is higher than that in XSum (204k), pre-training on CNN/DM should have achieved better performance than pre-training on XSum. To further analyze the reason, we calculate the averaged length of input documents and output summaries for the source and target domains. From Table 5, we find that the averaged length of XSum is much shorter than that of CNN/DM in terms of both document and summary, and surprisingly, SDPT based on XSum can outperform SDPT based on CNN/DM in domains with short length document and summary (e.g., debate and email) as well as the domains with long length document or summary (e.g., movie review and science). Hence, we conjecture that pre-training with relatively short document and summary is more effective for SDPT. Another reason can be attributed to the fact that the summaries of the CNN/DM tend to copy the content in the input documents, while XSum has larger amounts of novel tokens in the summaries. Therefore, we conjecture that XSum enables model learn a more powerful summarization ability, which helps it to better adapt to low-resource target domains. We leave investigating the effectiveness of different source domain datasets in SDPT for future work.
6.6 Performance vs. Training Sample Size
We investigate how well models perform in an extremely low-resource scenario (e.g., 50 training samples) and the performance discrepancies among different levels of resources. The performance over different numbers of training samples is illustrated in Figure 3. We find that BART fine-tuning with
the  $25\%$  data samples significantly outperforms that with  $\sim 2\%$  data samples in the dialog domain, but such improvements are not remarkable in the email domain. We conjecture that the input and output lengths for the email domain are relatively short compared to the dialog domain (according to Table 5), making the domain adaptation easier.
Interestingly, DAPT outperforms other models in the medium-resource and high-resource settings in the email domain but not in the dialog domain. We speculate the reasons are twofold. First, based on the vocabulary overlaps from Table 4, the email corpus is more effective for DAPT than the dialog domain. Second, email corpus is much larger than the dialog corpus from Table 3. However, the performance of DAPT using a high-quality corpus will be still limited by the low-resource scenario, and it needs large enough training samples to achieve remarkable improvements. Moreover, the performance of TAPT is better than BART fine-tuning in the low-resource setting, while it becomes worse in the medium-resource and high-resource settings. We conjecture that training with more data will aggravate the catastrophic forgetting caused by TAPT, which leads to the worse performance.
Surprisingly, the performance of DAPT with medium-resource is close to that with high-resource, which can be attributed to the combination of the powerful adaptation ability of the large pre-trained generative model and the effectiveness of the second phase of pre-training. However, there is still a large performance gap for all the models between the low-resource and high-resource settings and all the models perform badly when there is only 50 training samples, which highlights the needs for more advanced domain adaptation models for the summarization task.
In this paper, we present AdaptSum, the first benchmark to simulate the low-resource setting for the abstractive summarization task with a combination of existing datasets across six diverse domains. We systematically study three different methods for a second phase of pre-training (i.e., SDPT, DAPT and TAPT), and propose to leverage RecAdam to alleviate the catastrophic forgetting issue caused by the continuing pre-training. Experiments show that SDPT and TAPT can generally improve on the performance of the fine-tuning method, while the effectiveness of DAPT depends on the similarity between the pre-training data and the target domain task data, which is different from the insights into DAPT for classification tasks. Further analysis illustrates that RecAdam successfully alleviates the catastrophic forgetting issue for TAPT and further boost its performance.
Finally, our work highlights several research challenges in low-resource domain adaptation for the abstractive summarization task: (1) How to construct an effective corpus for DAPT; (2) How to better cope with the catastrophic forgetting issue for the second pre-training phase on a large corpus; (3) How to effectively integrate the task and domain knowledge (i.e., incorporate SDPT and DAPT); (4) How to choose better source domain datasets for conducting SDPT; (5) How to build a more powerful domain adaptation models for the extremely low-resource summarization task. We hope that the proposed dataset and the highlighted research directions will accelerate the studies in this area.
We want to thank the anonymous reviewers for their constructive feedback. This work is partially funded by ITF/319/16FP, ITS/353/19FP and MRP/055/18 of the Innovation Technology Commission, the Hong Kong SAR Government.
Yamen Ajjour, Henning Wachsmuth, Johannes Kiesel, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Data acquisition for argument search: The args. me corpus. In Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), pages 48-59. Springer.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint arXiv:1409.0473.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3606-3611.
Steven Bird, Robert Dale, Bonnie J. Dorr, Bryan Gibson, Mark T. Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R. Radev, and Yee Fan Tan. 2008. The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics. In Proc. of the 6th International Conference on Language Resources and Evaluation Conference (LREC'08), pages 1755-1759.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440-447.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870-7881, Online. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint arXiv:1805.11080.
Hal Daumé III. 2009. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907.1815.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13063-13075.
George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 451-459.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Xinyu Hua and Lu Wang. 2017. A pilot study of domain adaptation effect for neural abstractive summarization. arXiv preprint arXiv:1707.07062.
Shruti Jadon. 2020. An overview of deep learning architectures in few-shot learning domain. arXiv preprint arXiv:2008.06365.
Chen Jia, Xiaobo Liang, and Yue Zhang. 2019. Cross-domain ner using cross-domain language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2464-2474.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519-2531.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Socratic. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150-157.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019b. Zero-shot cross-lingual dialogue systems with transferable latent variables. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1297-1303.
Zihan Liu, Genta Indra Winata, and Pascale Fung. 2020a. Zero-resource cross-domain named entity recognition. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 1-6.
Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020b. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8433-8440.
Zihan Liu, Genta Indra Winata, Peng Xu, and Pascale Fung. 2020c. Coach: A coarse-to-fine approach for cross-domain slot filling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 19-25, Online. Association for Computational Linguistics.
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020d. Crossner: Evaluating cross-domain named entity recognition. arXiv preprint arXiv:2012.04373.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pages 6467-6476.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150.
Ahmed Magooda and Diane Litman. 2020. Abstractive summarization for low resource data using domain transfer and data synthesis. arXiv preprint arXiv:2002.03407.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. 2008. Domain adaptation with multiple sources. Advances in neural information processing systems, 21:1041-1048.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Preksha Nema, Mitesh M Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063-1072.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381.
Oana Sandu, Giuseppe Carenini, Gabriel Murray, and Raymond Ng. 2010. Domain adaptation to summarize human conversations. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 16-22.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J Barezi, and Pascale Fung. 2020. Cairecovid: A question answering and multi-document summarization system for Covid-19 research. arXiv preprint arXiv:2005.03975.
Baochen Sun, Jiashi Feng, and Kate Saenko. 2016. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. arXiv preprint arXiv:1601.04811.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019. Exploring domain shift in extractive text summarization. arXiv preprint arXiv:1908.11664.
Lu Wang and Claire Cardie. 2013. Domain-independent abstract generation for focused meeting summarization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1395-1405.