SlowGuess/NAACL_2021_OCR huggingface dataset:
Tiezheng Yu*, Zihan Liu*, Pascale Fung
Center for Artificial Intelligence Research (CAIRE)
{tyuah,zliucr}@connect.ust.hk, Pascale@ece.ust.hk
State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.
Abstractive summarization models aim to extract essential information from long documents and to generate short, concise and readable text. Recently, neural abstractive summarization models have achieved remarkable performance (Gehrmann et al., 2018; Paulus et al., 2018), and large-scale generative pre-training (Lewis et al., 2019; Raffel et al., 2019) has shown itself to be surprisingly effective at generation tasks, including abstractive summarization. However, these models generally require large numbers of human-annotated summaries to achieve state-of-the-art performance,
which makes them not scalable to low-resource domains where only a few labeled data are available.
Domain adaptation methods have naturally arisen to tackle the low-resource issue and enable models to quickly adapt to target domain tasks. Yet, despite their practicality, very few studies have used domain adaptation methods on the low-resource scenario for the abstractive summarization task. To address this research gap, we present AdaptSum, the first benchmark to simulate the low-resource domain Adaptation setting for abstractive Summarization systems with a combination of existing datasets across six diverse domains (dialog (Gliwa et al., 2019), email (Zhang and Tetreault, 2019), movie review (Wang and Ling, 2016), debate (Wang and Ling, 2016), social media (Kim et al., 2019), and science (Yasunaga et al., 2019)), and for each domain, we reduce the number of training samples to a small quantity so as to create a low-resource scenario.
Recently, conducting a second pre-training step on large-scale language models (e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019a)) has proven to be effective for domain adaptation tasks (Lee et al., 2020; Gururangan et al., 2020). However, the current methods incorporating such a step are mainly focused on classification or classification-based (e.g., named entity recognition) tasks, leaving a research gap in exploring their use for generation tasks. In this paper, we systematically investigate adding a second phase of pre-training on large-scale generative models under three settings: 1) source domain pre-training (SDPT) based on a labeled source domain summarization dataset; 2) domain-adaptive pre-training (DAPT) based on an unlabeled substantial domain-related corpus; and 3) task-adaptive pre-training (TAPT) based on an unlabeled small-scale task-related corpus. The second phase of pre-training could cause the catastrophic forgetting in the pre-trained model. Thus, we propose to apply
* Equal contributions. Listing order is random.
1The code and data are released at: https://github. com/TysonYu/AdaptSum
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5892-5904 June 6-11, 2021. ©2021 Association for Computational Linguistics
RecAdam (Chen et al., 2020) into the pre-training process to alleviate this issue and further improve the adaptation performance.
Experimental results show that SDPT and TAPT can generally improve on the performance of the fine-tuning method, while the effectiveness of DAPT is correlated to the similarity between the pre-training data and the target domain task data. Different from previous insights into adaptive pretraining on classification tasks (Gururangan et al., 2020), we find that in the summarization task, DAPT could make the adaptation performance worse, even though the pre-training corpus is collected from domain-related sources. Furthermore, we show that RecAdam can further boost the performance of the second pre-training step by effectively maintaining the pre-trained model's knowledge gained in the first phase of pre-training.
- We introduce a low-resource domain adaptation scenario for the abstractive summarization task to move towards the fast adaptation of summarization systems.
- To the best of our knowledge, we are the first to systematically study the domain- and task-adaptative pre-training for a low-resource generation task.
- Our work highlights the research questions and challenges in the low-resource abstractive summarization task, which we hope will catalyze research in this area.
Abstractive summarization aims to generate short, concise and readable text that captures the core meaning of the input documents. Neural networks have achieved remarkable results for the abstractive summarization due to the emergence of Seq2Seq models (Sutskever et al., 2014) and attention mechanisms (Bahdanau et al., 2014). See et al. (2017), Paulus et al. (2017) and Gehrmann et al. (2018) applied a pointer network to solve the out-of-vocabulary issue. Further, See et al. (2017) used a coverage mechanism (Tu et al., 2016) to keep track of the already summarized content, which discourages repetition, while Paulus et al. (2017) and Chen and Bansal (2018) combined reinforcement learning into an end2end setting. Recently,
Domain	Unlabeled Corpus	Labeled data
# Tokens	Size	Train	Valid	Test
Dialog	44.96M	212MB	300	818	819
Email	117.54M	705MB	300	1960	1906
Movie R.	11.36M	62MB	300	500	2931
Debate	122.99M	693MB	300	956	1003
Social M.	153.30M	786MB	300	1000	1000
Science	41.73M	291MB	100	350	497
Table 1: Data statistics of AdaptSum for the unlabeled corpus and labeled summarization data across the six domains ("R." and "M." are the abbreviations for Review and Media, respectively).
pre-trained language models (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Dong et al., 2019; Lewis et al., 2019) have achieved impressive gains in a wide variety of natural language tasks. Many studies on the use of pre-trained language models in the abstractive summarization task (Liu and Lapata, 2019; Yan et al., 2020; Su et al., 2020; Yu et al., 2020) have been undertaken and have achieved the state-of-the-art performance.
Domain adaptation for natural language processing and computer vision tasks is widely studied (Blitzer et al., 2007; Mansour et al., 2008; Daume III, 2009; Sandu et al., 2010; Foster et al., 2010; Wang and Cardie, 2013; Sun et al., 2016; Liu et al., 2019b, 2020b; Gururangan et al., 2020; Winata et al., 2020; Jadon, 2020; Yin, 2020; Liu et al., 2020a,d). However, little has been done to investigate domain adaption for the abstractive summarization task. Hua and Wang (2017) first studied the adaptation of neural summarization models and showed that the models were able to select salient information from the source domain data. Wang et al. (2019) investigated the domain shift problem for the extractive summarization task. Recently, Magooda and Litman (2020) studied cross-domain transfer between two entirely different domains and introduced data synthesis methods. To the best of our knowledge, we are the first to systematically study the domain- and task-adaptative pre-training based on the pre-trained generative model in the low-resource abstractive summarization task across multiple diverse domains.
3 AdaptSum
The goal of AdaptSum is to provide an accessible benchmark for the evaluation of low-resource domain adaptation for abstractive summarization on a
diverse set of domains. The vocabulary overlaps between domains are shown in Figure 1. AdaptSum consists of six diverse target domains and the corresponding unlabeled domain-related corpora for DAPT. We provide the data statistics of all domains in Table 1, and the details are as follows.
Dialog Gliwa et al. (2019) introduced a human-annotated abstractive chat dialog summarization dataset. The unlabeled dialog corpus from different sources, namely, Reddit conversations, $^{2}$  personalized dialogs (Zhang et al., 2018), empathetic dialogs (Rashkin et al., 2019), and Wizard of Wikipedia dialogs (Dinan et al., 2019).
Email Zhang and Tetreault (2019) introduced an abstractive business and personal email summarization dataset which consists of email and subject pairs. We collect the unlabeled email corpus from the Enron Email Dataset.<sup>3</sup>
Movie Review Wang and Ling (2016) introduced a human-annotated abstractive movie review summarization dataset. We collect the unlabeled corpus for this domain from IDMB Movie Review (Maas et al., 2011).
Debate Wang and Ling (2016) introduced an abstractive debate summarization dataset which consists of arguments and the debate topic pairs. The unlabeled corpus is from Ajjour et al. (2019).
Social Media Kim et al. (2019) introduced an abstractive summarization dataset of Reddit TIFU posts, where the summary for each post come from its title. We collect the unlabeled corpus directly from Reddit TIFU.<sup>4</sup>
Science Yasunaga et al. (2019) introduced a human-annotated abstractive summarization dataset on computational linguistics. We collect the unlabeled domain corpus from the ACL anthology (Bird et al., 2008).
In this section, we will first introduce the three different settings that we investigate for a second pre-training step. Then, we will discuss how we
Figure 1: Vocabulary overlaps of the summarization validation set between domains. The News domain is the source domain and the other six domains are low-resource target domains. Vocabularies for each domain are created by considering the top 10K most frequent words (excluding stopwords). We observe that the vocabulary overlaps between domains are generally small, which illustrates that the overlaps between domains are comparably small and the chosen domains are diverse.
cope with the catastrophic forgetting issue in the second phase of pre-training.
4.1 A Second Phase of Pre-Training
We conduct a second pre-training phase based on a pre-trained generative model, BART (Lewis et al., 2019), on three different settings. Then, we fine-tune it to the summarization task in the target domains. The three settings are described as follows.
Source Domain Pre-Training (SDPT) Inspired by the cross-domain setting (Jia et al., 2019; Liu et al., 2020c,d), we leverage substantial training samples from a source (News) domain (XSum (Narayan et al., 2018)), to aid in the fast adaptation in target domains. We choose the News domain as the source domain because it is a rich resource domain in the summarization task, and from Figure 1, the similarity between this domain and target domains is generally low which increases the challenge of the domain adaptation.
Our method to conduct SDPT is straightforward. We continue pre-training BART using the source domain summarization data. The objective function for this pre-training is not the sentence reconstruction, as in the original pre-training of BART. Instead, we utilize the supervisions from the source domain summarization data to train BART on the summarization task. The purpose of this
$^{2}$ https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reudit
 $^{3}$ https://www.cs.cmu.edu/~enron/
 $^{4}$ https://convokit.cornell.edu/documentation/subreddit.html
pre-training is to inject the task knowledge into the pre-trained language model so that the model can quickly adapt to the same task in target domains.
Domain-Adaptive Pre-Training (DAPT) We leverage an unlabeled domain-related corpus to continue pre-training BART using its original pretraining objective function (corrupting documents and then optimizing a reconstruction loss—the cross-entropy between the decoder's output and the original document). The intuition behind this method is to introduce the domain knowledge into the pre-trained language model so as to enable its fast adaptation to the target domains.
Task-Adaptive Pre-Training (TAPT) The size of the domain-related corpus for DAPT is usually enormous, which results in two potential drawbacks. First, such a large corpus might not be always available, especially for the low-resource domains. Second, pre-training on such a large corpus is time-consuming and requires excessive computational resources. Therefore, investigating pretraining on a smaller unlabeled corpus is a practical and beneficial research direction. TAPT refers to pre-training on a set of the unlabeled documents in the target domain's summarization task. Compared to DAPT, TAPT uses a much smaller but far more task-relevant pre-training corpus since it directly uses the input documents from summarization task. This setting makes TAPT much less expensive to run and independent of the collection of the large domain-related corpus.
4.2 Recall and Learn
Although the second pre-training step allows the pre-trained model to learn the task or domain knowledge, it might lead to the catastrophic forgetting issue and cause the pre-trained model to partly lose the language understanding ability that it gains in the first pre-training step. To alleviate this issue, we expect the pre-trained model to recall the previously learned knowledge during the process of learning new knowledge. A straightforward way to achieve this goal is to borrow the idea of continual learning methods (Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017; Chen et al., 2020). In this paper, we adopt RecAdam from Chen et al. (2020) for the second phase of pretraining to weaken the catastrophic forgetting issue. The reason for choosing RecAdam is twofold: 1) it does not require the first step pre-training data from
the pre-trained model, which is usually not available; 2) it is the most recent approach that is being successfully applied to natural language processing tasks. The RecAdam is introduced as follows.
Based on the Adam optimizer (Kingma and Ba, 2015), RecAdam reconstructs the objective function to allow it to gradually shift to the target task:
$$
L o s s = \lambda (t) \cdot L o s s _ {T} + (1 - \lambda (t)) \cdot L o s s _ {S}, \tag {1}
$$
$$
\lambda (t) = \frac {1}{1 + \exp (- k \cdot (t - t _ {0}))}, \tag {2}
$$
where  $k$  and  $t_0$  are the hyper-parameters controlling the annealing rate and time steps,  $Loss_{T}$  represents the target task objective function, and  $Loss_{S}$  is used to simulate the first pre-training step of the pre-trained model.  $Loss_{S}$  can be simplified as:
$$
\operatorname {L o s s} _ {S} = \frac {1}{2} \gamma \sum_ {i} \left(\theta_ {i} - \theta_ {i} ^ {*}\right) ^ {2}, \tag {3}
$$
where  $\frac{1}{2}\gamma$  is the coefficient of the quadratic penalty,  $\theta$  is the parameters of the model, and  $\theta^{*}$  (fixed) is the original parameters of the pre-trained model.
Although RecAdam has shown its effectiveness in fine-tuning BERT-like models (e.g., BERT (Devlin et al., 2019) and ALBERT (Lan et al., 2020)) to the GLUE benchmark (Wang et al., 2018), exploring the effectiveness of RecAdam in the second phase of pre-training for generative pre-trained models is not trivial. First, the second pre-training step of a language model is a completely different task compared to fine-tuning to downstream tasks. Second, a generative model (e.g., BART) is structurally different from BERT-like models. Third, the corpus sizes for SDPT and DAPT are generally much larger than the sizes of GLUE tasks, which could affect the learning process.
Training Details We evaluate all of our models on AdaptSum. For the dialog and email domains, we use the standard splits of (Gliwa et al., 2019; Zhang and Tetreault, 2019), while for movie review, debate, social media and science domains, we split the whole dataset into training, validation and test sets by ourselves since the original works do not specify how to split these datasets or the published datasets do not contain the split training, validation and test sets. Since the dataset sizes are limited for science, movie review and dialog domains, the maximum training samples for these domains are
Models	Dialog	Email	Movie R.	Debate	Social M.	Science	Average
BART Fine-tuning	39.95	24.71	25.13	24.48	21.76	72.76	34.80
SDPT	42.84	25.16	25.45	25.61	22.43	73.09	35.76
w/ RecAdam	45.23	26.97	26.06	25.17	23.25	72.60	36.55
DAPT	41.22	26.50	24.25	26.71	22.95	71.88	35.59
w/ RecAdam	40.05	25.66	25.78	25.01	21.51	72.23	35.04
TAPT	40.15	25.30	25.27	24.59	22.81	73.08	35.20
w/ RecAdam	41.34	25.73	25.65	24.70	23.01	72.80	35.54
Table 2: ROUGE-1 scores on different pre-training methods compared to the baseline BART over all domains.
Corpus	Dialog	Email	Movie R.	Debate	Social M.	Science	Average
DAPT	212MB	705MB	62MB	693MB	786MB	291MB	458.2MB
TAPT	7.9MB	14MB	3.3MB	2.4MB	74MB	384KB	17.0MB
Table 3: Corpus size comparisons between DAPT and TAPT.
100, 300, and 300, respectively, while for dialog, email, and social media domains, the maximum training samples for them are 14732, 14436, and 60354, respectively, and we select 300 samples for each domain to construct a low-resource setting. We truncate the input documents into 1024 tokens due to the limitation of the maximum input length for BART. For all the experiments, we use the BART-base version to implement our models. We use a mini-batch size of 4 with a gradient accumulation for 10 iterations. We use Adam optimizer with momentum  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.998$  and noam decay with warm up steps of 1000. In the decoding stage, we use beam search with a beam size of 4. The decoding process will not stop until an end-of-sequence (EOS) token is emitted or the length of the generated summary reaches to 256 tokens. As for the hyperparameters of RecAdam, we select the best  $t_0$  and  $k$  in  $\{500, 600, 700, 800, 900, 1,000\}$  and  $\{1e - 2, 1e - 3, 1e - 4, 1e - 5, 1e - 6\}$ , respectively, for the annealing coefficient  $\lambda(t)$  (Eq. 2).
Baseline As our baseline, we use an off-the-shelf BART model (Lewis et al., 2019) and perform supervised fine-tuning of its parameters for the summarization task in each domain. BART serves as a good baseline since it provides the state-of-the-art performance in the summarization task. And, as a single generative language model, it can be easily adapted to different target domains.
Evaluation Metrics We use ROUGE (Lin and Hovy, 2003) to measure the quality of the summary produced in our experiments. Following the previous work (Nema et al., 2017), we report ROUGE
F1 (ROUGE-1) on the AdaptSum dataset.5
From Table 2, we can see that SDPT is able to generally improve the summarization performance of the fine-tuning method for all domains. This is because SDPT teaches the model how to do the task using large numbers of annotated examples, which enables the model to adapt to target domains faster than the fine-tuning method, and SDPT is able to outperform both DAPT and TAPT in terms of the averaged ROUGE-1 score. The enormous unlabeled corpus makes DAPT quite effective in certain domains, such as email, debate and social media, with close to or more than 2 ROUGE-1 scores improvements over the fine-tuning baseline. As we can see from Table 3, although TAPT uses a far smaller pre-training corpus than DAPT, the performance of TAPT is on par with that of DAPT, which accords with the results in Gururangan et al. (2020), where the experiments are conducted for domain adaptation in classification tasks. Additionally, adding RecAdam into the second phase of pre-training can generally further boost the adaptation performance for SDPT and TAPT, while it only boost the performance on the movie review and science domains for DAPT. We conjecture that a relatively large corpus can potentially weaken the effectiveness of RecAdam, and we observe that
5We use pyrouge to compute all ROUGE scores, with parameters "c 95 -24 -U -r 1000 -n 4 -w 1.2 -a". The full results of all the models with ROUGE-2 and ROUGE-L are reported in the Appendix.
Domains	DAPT Corpus	TAPT Corpus
Dialog	37.56 (-7.04)	44.60
Email	51.87 (-5.93)	57.80
Movie R.	46.63 (-14.59)	61.22
Debate	53.49 (-8.99)	62.48
Social M.	48.10 (-3.82)	51.92
Science	36.94 (-20.90)	57.84
Average	45.44 (-10.54)	55.98
Table 4: Vocabulary overlaps (\%) between the pretraining corpus (for DAPT or TAPT) and the validation set of the summarization task for each domain. The numbers in the brackets denote the vocabulary overlap differences between the two pre-training corpora, and the bold numbers denote the large discrepancies.
the corpus used for DAPT is comparably small for movie review and science domains, and the number of data samples for XSum (204k) is also much smaller than those of DAPT corpora in many domains (e.g., email), which have more than 1M sentences. According to Eq. 1, extensive training data could result in a comparatively large  $Loss_{s}$  (the model's parameters tend to be greatly modified) which lead to an unstable loss and a negative effect to the pre-training process. In addition, we find that RecAdam is originally shown to be effective at fine-tuning to the downstream GLUE tasks (Chen et al., 2020), the sizes of which are much smaller than the datasets used for SDPT and DAPT.
6.2 How Pre-training Data Affects DAPT
According to prior experiments on domain adaptation for classification or classification-based tasks (Beltagy et al., 2019; Lee et al., 2020; Gururangan et al., 2020), DAPT improves the performance for all domains on the fine-tuning baseline. However, as we can see from Table 2, DAPT causes the performance to drop for the movie review and science domains in the summarization task, while TAPT boosts the performance for all the domains. To further investigate the reasons, we aim to analyze the similarity (e.g., vocabulary overlap) between the pre-training corpus for DAPT and the summarization task in the target domain, which we represent with the target domain validation set of the summarization task to represent. We notice that it is difficult to justify how much overlap is large enough for DAPT to be considered as effective. Hence, we add the TAPT corpus, which is directly related to the target domain's summarization task, as an upper bound for the comparison.
Figure 2: TAPT performance over different pretraining epoch numbers in the email domain in terms of using and not using RecAdam.
Table 4 illustrates the vocabulary overlaps for DAPT and TAPT for each domain. We find large discrepancies between the DAPT corpus and TAPT corpus on the movie review and science domains, which indicates that the domain-related corpora in these two domains are not quite related to the task domains, and pre-training on a domain-unrelated or less related corpus can lead to a performance drop compared to the fine-tuning method. Given that the corpus construction is done by looking for the domain-related sources (as mentioned in Section 3), the experimental results point out that collecting a domain-related corpus for DAPT in the summarization task is not straightforward. Thus, we leave exploring how to construct an effective corpus for DAPT for future work.
6.3 Catastrophic Forgetting Issue