SlowGuess/NAACL_2021_OCR huggingface dataset:
Alexander R. Fabbri $^{\dagger}$  Simeng Han $^{\dagger\dagger}$  Haoyuan Li $^{\ddagger\ddagger}$  Haoran Li $^{\ddagger}$  Marjan Ghazvininejad $^{\ddagger}$  Shafiq Joty $^{\dagger\dagger}$  Dragomir Radev $^{\dagger}$  Yashar Mehdad $^{\ddagger}$
†† Nanyang Technological University ‡‡ Renmin University of China {alexander.fabbri,dragomir.radev}@yale.edu {aimeeli,ghazvini,mehdad}@fb.com
Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. However, these models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. In this work, we introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction of the desired summaries. WikiTransfer models achieve state-of-the-art, zero-shot abstractive summarization performance on the CNN-DailyMail dataset and demonstrate the effectiveness of our approach on three additional diverse datasets. These models are more robust to noisy data and also achieve better or comparable few-shot performance using 10 and 100 training examples when compared to few-shot transfer from other summarization datasets. To further boost performance, we employ data augmentation via round-trip translation as well as introduce a regularization term for improved few-shot transfer. To understand the role of dataset aspects in transfer performance and the quality of the resulting output summaries, we further study the effect of the components of our unsupervised fine-tuning data and analyze few-shot performance using both automatic and human evaluation.
Automatic text summarization aims to distill the most salient content of a given text in a compact form. Recent advances in summarization have been driven by the availability of large-scale datasets such as the CNN-DailyMail (CNNDM) corpus (Nallapati et al., 2016) and the New York Times
corpus (Sandhaus, 2008) as well as by the introduction of large pretrained models such as BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2019), in some cases resulting in summaries which are even favored over the human-written reference summaries. Creating data for every new domain, however, is infeasible and highly costly. Thus, the ability to transfer large pretrained models to new domains with little or no in-domain data is necessary, especially as such models make their way into real-world applications.
Unsupervised summarization approaches include autoencoders to mirror the information compression inherent in summarization (Baziotis et al., 2019; Chu and Liu, 2019; Bražinskas et al., 2020b) as well as large-scale pretraining for domain-specific adaptation (Yang et al., 2020). However, little work has focused on domain adaptation in summarization. Wang et al. (2019) examine domain adaptation for extractive summarization. Hua and Wang (2017) showed that summarization models have difficulty generating text in the style of the target domain, while more recently, Zhang et al. (2019) report strong performance of pretrained models when trained in few-shot settings and (Bražinskas et al., 2020a) fine-tune dataset-specific components of a model for few-shot learning. We aim to build on recent work in pretrained models and improve zero-shot and few-shot summarization by encoding characteristics of the target summarization dataset in unsupervised, intermediate fine-tuning data.
Summarization can be seen as a function of subfunctions of the input, called subaspects, which determine the output form. Jung et al. (2019) define three subaspects for summarization: position, importance, and diversity, and study how these subaspects manifest themselves in summarization corpora and model outputs. For example, a common subaspect for the CNNDM dataset is position; earlier sentences tend to constitute a good sum
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 704-717 June 6-11, 2021. ©2021 Association for Computational Linguistics
mary. Inspired by this view of summarization as subaspects, we aim to encode subaspects of a target dataset into unlabeled data to allow a model finetuned on this data to learn characteristics of the target dataset to improve zero-shot and few-shot transfer of the model. In our work, we focus on the subaspects of extractive diversity, as determined by how well an extractive model performs on the data, compression ratio between the source document and summary, and, in the case of CNNDM, the lead bias. We assume knowledge of the target dataset such as the size of input documents, the size of the desired summaries, and the extent to which the summary is abstractive, all of that can be treated as prior knowledge if the task is to be well-defined (Kryscinski et al., 2020). We encode this knowledge into Wikipedia article data by extracting summaries of the desired output length and filtering examples based on the desired level of abstraction.
Our contributions are the following: 1) We introduce a novel method, called WikiTransfer, which creates pseudo-summaries with subaspects of the target dataset which can be used as unlabeled data for intermediate fine-tuning. We show that this method improves zero-shot domain transfer over transfer from other domains, achieving state-of-the-art unsupervised abstractive summarization performance on the CNNDM dataset while generalizing to other domains, and we perform extensive hyperparameter studies on the factors influencing zero-shot performance 2) We demonstrate the benefits of WikiTransfer in few-shot settings, and show additional improvements when applying WikiTransfer with data augmentation and a regularization term for training with potentially noisy augmented data. We show robustness in these settings and analyze differences in performance in both automatic and human assessments.
While advances have been made in neural techniques for summarization due in part to large datasets, less work has focused on domain adaptation of such methods in the zero and few-shot settings. Wang et al. (2019) examine domain adaptation, but in extractive summarization. Hua and Wang (2017) examine domain adaptation between opinion and news summarization, observing that models trained on one domain and applied to another domain can capture relevant content but differ
in style in generating the summary.
Bražinskas et al. (2020a) introduce plug-in networks, small finetune-able layers that aim to reproduce characteristics of the target dataset as seen in a small set of labeled examples. In contrast, we aim to encode the characteristics of our target dataset, such as level of extraction and compression, a priori in the intermediate training phase. In other work, Lebanoff et al. (2018) adapt a single-document summarization model to multi-document settings, while Zhu et al. (2019) use Wikipedia reference data for downstream query-based summarization
Several approaches for unsupervised summarization have made use of variational autoencoders (Baziotis et al., 2019; Chu and Liu, 2019; Bražinskas et al., 2020b). Zhou and Rush (2019) makes use of pretrained language models for unsupervised text summarization by aligning the coverage of the generated summary to the source document. Laban et al. (2020) train an unsupervised summarization model with reinforcement learning rewards. In another line of work, extractive models such as TextRank, (Mihalcea and Tarau, 2004), LexRank (Erkan and Radev, 2004), and more recently PacSum (Zheng and Lapata, 2019), make use of graph centrality for modeling salience.
The power of pretrained models for few-shot transfer was shown for abstractive summarization in Zhang et al. (2019) and extractive summarization in Desai et al. (2020). Our work focuses on the zero-shot abstractive summarization setting and the transferability of models fine-tuned on task-specific data from a generic corpus, rather than just the transferability of a single pretrained model. The closest work to ours for zero-shot transfer is Yang et al. (2020), which uses the lead-bias in news to pretrain an unsupervised model on a large dataset of news articles. Our approach, however, focuses on fine-tuning an already-pretrained model specifically for summarization on a downstream dataset by leveraging a generic text corpus (Wikipedia) to create auxiliary fine-tuning data that transfers across domains, allowing for more fine-grained control over the transfer process. We show the generalizability of such fine-tuning across domains. BART (Lewis et al., 2020) is a pretrained denoising autoencoder and achieved state-of-the-art performance when fine-tuned on summarization tasks at the time. In this work, we use BART as our base pretrained model but in future work will experiment with other pretrained models.
WikiTransfer Intermediate Fine-tuning: We propose a method for fine-tuning pretrained models using unsupervised Wikipedia data. We create dataset-specific unsupervised data for this intermediate fine-tuning, by making use of characteristics of the target dataset such as the average length of input documents, the average summary length, and the general bin of whether the summaries desired are very abstractive or very extractive, as discussed above. Assume that we want a summary of M sentences from source documents of N sentences on average, and that we know approximately how extractive the summaries are in the target dataset, as defined as the upper bound ROUGE (Lin, 2004) performance of an extractive model, the extractive oracle, on that dataset. We bin the level of extraction of the target summaries into extremely abstractive (ROUGE oracle 10-30), more abstractive (ROUGE oracle 20-30), more extractive (ROUGE oracle 30-50), and extremely extractive (ROUGE oracle 40-60). We then iterate the following procedure on all Wikipedia articles available in a Wikipedia dump: We remove the first M sentences from the Wikipedia article for use as a summary and the following N sentences for use as a source document. Then, we want to check whether this pseudo data point matches the level of extraction of the target dataset. We select the M sentences in the pseudo source document with the highest individual ROUGE scores against the pseudo summary and calculate the ROUGE score between those M sentences concatenated and the pseudo summary, which amounts to a greedy upper bound of the performance of an extractive model on this example. The example will be kept if this ROUGE score falls into the general range of the extractive oracle of the target dataset defined previously and otherwise discarded. We use knowledge of how abstractive a dataset is as a type of summary style which an end-user would know ahead of time. We filter the Wikipedia data points so that only those which fall into the bin for a given dataset are used for fine-tuning. For datasets that are extremely abstractive, such examples may be hard to find, so we remove high-ROUGE sentences from the input until the desired ROUGE oracle score is reached. From here on we refer to data created through this process as WikiTransfer. We then fine-tune a pretrained model on this dataset-specific WikiTransfer data to transfer to a target domain.
Data Augmentation via Round-Trip Translation: In addition to fine-tuning on WikiTransfer data for zero-shot domain transfer, we test the ability of our model to transfer when we have few examples and whether data augmentation further improves these results. In few-shot fine-tuning, we conduct data augmentation to reduce brute-force memorization and introduce a regularization effect. Specifically, we perform round-trip translation (Yu et al., 2018) to generate paraphrases of both the source documents and summaries, as previous work has found this approach creates diverse paraphrase for augmentation while preserving semantic meaning (Yu et al., 2018; Xie et al., 2019). Our examination found that round-trip translation increased the number of novel n-grams while preserving semantic meaning. Given a dataset of  $N$  data points, we translate the source and target sentence-wise into a non-English language and keep the top  $k$  beam hypotheses from beam search as output. We then do likewise for the backtranslation to English. This results in  $N * k^2$  augmented data points in addition to the  $N$  original supervised data points. We align a single beam from the translation to non-English text to a single beam in the backtranslation to English; using all combinations of beams for augmented data did not result in an improvement in initial experiments. We refer to the training setting of  $N$  supervised data points with this additional augmented data as N-a.
Data Augmentation Consistency: While data augmentation may introduce a regularization effect, naively training with augmented data does not necessarily account for noise introduced in the augmented examples. To balance learning from the examples while not overfitting to the small number of supervised samples, the model must learn to be robust to small changes in input examples. We thus investigate the effect of using a consistency loss (Xie et al., 2019; Athiwaratkun et al., 2019) for few-shot training which enforces consistency between the original and round-trip translated documents with respect to the original summary. Let  $x = \{x_{1}, x_{2}, \dots, x_{i}, \dots, x_{n}\}$  be a source document with  $n$  words and  $N$  sentences, where  $x_{i}$  represents the  $i$ -th word in  $x$ . It could also be represented as  $\{s_{1}, s_{2}, \dots, s_{j}, \dots, s_{N}\}$ , where  $s_{t}$  represents the  $j$ -th sentence in  $x$ . The corresponding target summary  $y$  contains  $m$  words and  $M$  sentences, and  $y_{i}$  denotes the  $i$ -th token of  $y$ . Standard training, used in the above sections, minimizes the negative log-
likelihood loss using supervised teacher forcing (Williams and Zipser, 1989), which we label  $L_{sup}$ :
$$
L _ {s u p} (x, y) = - \sum_ {t = 1} ^ {m} \log \left(f \left(y _ {t} \mid y _ {0: t - 1}, x, \theta\right)\right) \tag {1}
$$
where  $f(\cdot |\cdot ,\theta)$  represents the distribution among the vocabulary predicted by our model with parameter  $\theta$ . In our formulation, the output (summary) distribution given an augmented (round-trip translated) example should not diverge much from the distribution given the original document, with teacher forcing, so that the model learns to be resilient to small perturbations. Let  $\hat{x}$  be a paraphrase of input document  $x$  generated via round-trip translation as described in the previous section. In addition to the supervised loss  $L_{sup}(x,y)$ , we introduce another loss  $L_{cons}(x,\hat{x},y)$ :
$$
\sum_ {t = 1} ^ {m} K L \left(f \left(\cdot \mid y _ {0: t - 1}, x\right) \mid \mid f \left(\cdot \mid y _ {0: t - 1}, \hat {x}\right), \theta\right) \tag {2}
$$
where  $KL$  is the KL divergence, which penalizes the model if the probability distribution of the output using the original input is far from the distribution using the round-trip translated input document. Following Xie et al. (2019), the gradient does not backpropagate through the model for the distribution of the original input while it does propagate through to the round-trip translated input. The total loss  $L^{\prime}$  for training with consistency then is:
$$
L ^ {\prime} (x, \hat {x}, y) = L _ {s u p} (x, y) + \lambda L _ {c o n s} (x, \hat {x}, y) \quad (3)
$$
We note that the original formulation of Unsupervised Data Augmentation (UDA) (Xie et al., 2019) enforces consistency in a semi-supervised framework. We also experiment with this setup using unlabeled examples from the target dataset with pseudo labels (for teacher forcing) generated by a model trained on the associated few-shot subset, although this approach is very sensitive to the quality of the pseudo labels (see Appendix). We refer to the training setting of  $N$  supervised data points with consistency training as  $\mathbf{N}\cdot \mathbf{c}$ .
Datasets: We experiment with four datasets, CNNDM, XSum (Narayan et al., 2018), Reddit_tifu (Reddit) (Kim et al., 2019), and BigPatent (Sharma et al., 2019). The datasets were chosen as they all
differ in their abstractiveness, output length (from one sentence in XSum to on average four in BigPatent), and cover multiple domains from news (CNNDM and XSum) to social media (Reddit) to patent documents (BigPatent), to show the generalizability of our results. Each of the datasets falls into a different extractive bin, from the most extractive CNNDM to the more abstractive XSum; we discuss these settings further in the Appendix.
Model Selection and Metric: For the experiments which follow, we first choose the model with the best zero-shot performance on a given domain. We test the zero-shot performance from all four domains onto every other domain. For models from our WikiTransfer subset, we choose the best model based on performance on an unsupervised WikiTransfer validation subset. We find that fine-tuning the model longer does not result in performance gains in few-shot transfer, and the checkpoints chosen were typically fine-tuned from 2 to 5 epochs. Results from hyperparameter studies for zero-shot transfer from WikiTransfer data are shown on the validation set of that given target dataset. Unless otherwise stated, all results reported are ROUGE-1/2/L. We run all few-shot transfer experiments on five subsets of supervised data, and the reported numbers, unless zero-shot, are the average of the top three results of the five runs following previous work (Gunel et al., 2020). The 10 data point sets are subsets of the 100 data point sets.
Data Augmentation Parameters: For data augmentation via round-trip translation, we use a beam size of 10 and  $k$  of 10 on German and Russian translation models; faireq provides bidirectional pretrained translation models (Edunov et al., 2018) from WMT19 (Ng et al., 2019) for these language pairs. For both 10 and 100 data points, this resulted in 2010 and 20100 total data points. For consistency loss, we use the same augmented data.
Model Hyperparameters: We use the faireq codebase (Ott et al., 2019) for our experiments. Our base abstractive text summarization model is BART-large (Lewis et al., 2020), a pretrained denoising autoencoder with 336M parameters that builds off of the sequence-to-sequence transformer of Vaswani et al. (2017). We fine-tune BART using a polynomial decay learning rate scheduler using the Adam optimizer (Kingma and Ba, 2015). We mainly vary the learning-rate scheduler, warm-up updates, and total updates. As in the previ
ous few-shot summarization work (Zhang et al., 2019) and work in unsupervised machine translation (Conneau and Lample, 2019), we use a subset of the target-domain validation set for early stopping based on the validation loss. We used the following (warmup updates, total updates, learning rate) parameter tuples based on an examination of the validation curves in initial experiments: 10: (25, 100, 3e-5); 10-a: (20, 200, 3e-5); 100 (20, 200, 3e-5); 100-a: (200, 1000, 1e-5). For consistency loss experiments, we use the  $\lambda$  values of 0.1 and 0.5 for experiments with 10 and 100 data points, respectively, chosen manually based on Xie et al. (2019). See the Appendix for more details.
5 Zero-shot Transfer Results
We compare the zero-shot performance of BART fine-tuned on WikiTransfer data to that of one transferred from other summarization datasets. We also show the effect of different choices for WikiTransfer fine-tuning data on CNNDM and XSum.
5.1 Zero-shot Transfer Comparison
We aim to show that a model fine-tuned on WikiTransfer data has better zero-shot performance than models transferred from other summarization datasets. We fine-tune BART on WikiTransfer data for each of the four target datasets described above and also fine-tune a model on each of the fully-supervised datasets. We compare the zero-shot performance of transferring from WikiTransfer against the best zero-shot transfer performance from another dataset in Table 1. Zero-shot transfer from WikiTransfer notably outperforms transferring from other datasets on CNNDM, XSum, and BigPatent. On Reddit, we perform better on ROUGE-1 and comparably on ROUGE-2/L, which may be due to distinct writing style on Reddit data, as noted in Zhang et al. (2019). We also experimented with training a model on data combined from multiple datasets for zero-shot transfer, but this does not report improved results, so for the experiments which follow we use the best performing single-domain transfer model. Details of the fully-supervised BART models are in the Appendix.
We compare our model to the state-of-the-art unsupervised abstractive model on CNNDM in Table 2. We outperform the recently-introduced TED model (Yang et al., 2020) which was specifically motivated for the news domain. We believe the creation of task-specific data from a generic corpus
Target Dataset	WikiTransfer	Other Transfer
CNNDM	39.11	17.25	35.73	36.81	14.18	32.62	(Reddit)
XSum	31.85	10.44	23.75	24.04	6.43	18.99	(Reddit)
Reddit	21.47	4.10	17.62	21.37	4.14	17.76	(CNNDM)
BigPatent	35.58	10.91	31.53	33.57	9.34	25.76	(CNNDM)
Table 1: Comparison of ROUGE-1/2/L zero-shot transfer performance from dataset-specific WikiTransfer vs. transfer from another dataset. The dataset from which zero-shot transfer performed the best is in parentheses.
Model	ROUGE-1/2/L
WikiTransfer	39.11	17.25	35.73
TED (Yang et al., 2020)	38.73	16.84	35.40
Table 2: A comparison of our approach to the unsupervised pretraining of TED (Yang et al., 2020), showing the superior performance and generalizability of our approach versus the TED model, which focused specifically on the news domain.
such as Wikipedia allows for more control over the transfer process than relying on the autoencoder objective of TED, and more generalizable cross-domain results.
5.2 Effect of WikiTransfer Hyperparameters
We study the effect the characteristics of our intermediate fine-tuning data have on downstream zero-shot performance on CNNDM and XSum to compare highly extractive and abstractive datasets. Effect of learning rate in intermediate finetuning: We examine the extent to which overfitting to the unsupervised WikiTransfer data occurs by examining the effect of the learning rate in intermediate fine-tuning on zero-shot transfer performance. We finetune the models on the CNNDM and XSum WikiTransfer data respectively each with a maximum learning rate of 3e-6 and 3e-5. Results are shown in Table 3. Using a smaller learning rate in intermediate fine-tuning improves results on CNNDM, but not on XSum, likely due to the simple extractive and lead bias objectives which can easily overfit during fine-tuning. We see a similar trend with the effect of dataset size. For datasets other than CNNDM, we use a learning rate of 3e-5 in intermediate fine-tuning.
Effect of extractive oracle bin use and the choice of M: We tested whether using the extractive bin to filter examples in the unsupervised data affected zero-shot transfer. For this experiment, we used the first M sentences from the Wikipedia article as the summary and the remaining N as the source, but do not filter examples according to how extractive they are. From Table 3, we see that the extractive bin has a very noticeable effect on
Ablation	CNNDM	XSum
LR=3e-6	40.14	17.71	36.66	27.60	8.62	20.93
LR=3e-5	39.73	16.94	36.24	31.80	10.46	23.66
LR=3e-6, No-bin	39.11	16.98	35.66	22.78	5.66	17.16
LR=3e-6, bin, M=1	37.45	14.72	32.52	27.60	8.62	20.93
LR=3e-6, bin, M=3	40.14	17.71	36.66	27.98	9.59	23.11
transfer results for XSum and a moderate effect on CNNDM. This is to be expected, as the model otherwise is missing information about XSum's distinctive output style.