SlowGuess/NAACL_2021_OCR huggingface dataset:
We examine how the choice of M affected performance. We set  $M = 1$  for CNNDM and  $M = 3$  for XSum and filtered examples in a similar way based on the extractive bin of the target dataset. We see that the choice of M has a large impact on CNNDM performance but no decrease on XSum. This result, combined with the effect of filtering examples based on the extractive bin, gives insight into the importance of the subaspect of abstractiveness over compression for XSum performance.
Effect of intermediate pretraining dataset size: We examined the effect of the size of the WikiTransfer data on downstream performance. Results are shown in Table 4. We see a general increase with the addition of more data, although smaller increases after 100k data points and even a decrease in 250k on XSum, likely due to noise variation. The performance with 10k data points on CNNDM is already much closer to the best performance than the XSum case. We believe that this is due to the highly extractive nature of CNNDM, which is especially easy for a model such as BART to learn, as it is pretrained as a denoising autoencoder. For XSum, we see a noticeable improvement from 10k to 100k examples. We suspect that the abstractive objective is harder for the model to learn with small datasets. As we add more examples, we do not see a noticeable improvement. Such observations agree with our observation of the effect of learning rate and overfitting to the easier CNNDM objective. For the remaining experiments, we use 400k data points based on initial experiments.
Effect of summary sentence choice: The first M sentences of a given Wikipedia article were chosen as this introduction intuitively form a coherent summary of the article. We examine the effect of choosing the first sentences compared to choosing based
Table 3: Ablation studies on the effect of learning rate, the use of extractive bin for data filtering and the choice of M in intermediate fine-tuning on ROUGE-1/2/L performance on CNNDM and XSum validation sets.
Intermediate Dataset Size	CNNDM	XSum
10k	39.48	17.79	36.3	21.59	4.85	16.28
100k	39.92	17.65	36.5	31.52	10.86	23.94
250k	40.10	17.70	36.62	31.39	10.27	23.43
400k	40.14	17.71	36.66	31.80	10.46	23.66
Table 4: A comparison of the effect of dataset size of the unsupervised intermediate fine-tuning data on the zero-shot transfer ROUGE-1/2/L performance.
Target Dataset	First M Sents	IND-ORIG	IND-ORIG-P
CNNDM	40.14	17.71	36.66	37.62	15.15	34.21	37.85	15.32	34.39
XSum	31.80	10.46	23.66	29.95	9.37	21.78	30.22	9.79	23.23
Table 5: A comparison of the effect of summary sentence choice for WikiTransfer on zero-shot transfer ROUGE-1/2/L performance.
on other criteria. As an alternative, we pick the sentences with the highest self-ROUGE (ROUGE score of a sentence when using all other sentences as the reference summary) in a greedy fashion (the equivalent of the IND-ORIG settings in Zhang et al. (2019)). As in Zhang et al. (2019), we use ROUGE-1 F1. The sentences chosen under this heuristic consistently corresponded to those which were longest, and the resulting summaries were hence longer. Thus, we also experimented with choosing important sentences by using ROUGE-1 Precision, IND-ORIG-P. The comparison of these methods is shown in Table 5. The choice of the summary sentence has a noticeable impact on performance. We hypothesize that the coherence lost in the summaries is especially important for the longer CNNDM summaries. Using important sentences other than the first sentence likely adds more diversity in the data, and finding a balance between coherence and output style is an interesting direction for additional work (Christensen et al., 2013).
Effect of lead bias on CNNDM fine-tuning: We examined the effect of selecting the M sentences greedily chosen for calculating the extractive oracle and inserting them at the beginning of the unsupervised source document versus leaving them in place for CNNDM intermediate fine-tuning. This is meant to mirror the lead bias present in the dataset. This had a slight impact on performance (40.14 vs 39.74 without this bias), and thus we keep the lead bias for CNNDM experiments.
Wikipedia vs target domain unlabeled data: While Wikipedia is a natural source of unlabeled data, we tested whether creating unsupervised data from unlabeled in-domain data improved results. We performed the same dataset creation treating
the source data of the target domain as we did the Wikipedia data. This resulted in about 60k examples for CNNDM and 200k examples for XSum. Fine-tuning on this data, however, resulted in a performance of 38.08/25.83 ROUGE-1 for CNNDM and XSum (vs 39.11/31.85 on WikiTransfer data). The removal of the first sentences may remove too much information in the case of CNNDM, while for XSum, which already has an initial sentence headline removed as the summary, the first sentence may not constitute a very good summary of the remaining document. Wikipedia data often contains multi-paragraph introductions; thus the removal of the first few sentences may still leave a pyramid-structured document with coherent informative content placed at the front. This result supports the emphasis on learning the sub-aspects of the target domain over simply in-domain training. An analysis of the output of intermediate fine-tuning on CNNDM reveals that the output was more abstractive, due to information present in the summary not being directly stated in the source, than fine-tuning on Wikipedia. We also experiment with further in-domain pretraining of BART before zero-shot transfer, but this does not result in consistent improvements across datasets.
6 Few-Shot Transfer Results
We examine whether zero-shot transfer improvements also carry over to the few-shot setting. Also, we explore the effect of data augmentation and consistency regularization techniques. The results of our experiments with varying training data sizes and augmentation methods for all 4 datasets are shown in Figure 1 and the Appendix.
10 and 100-shot performance with round-trip translation augmentation: We see that in few-shot settings, without data augmentation or consistency training, our model outperforms transferring from another domain or vanilla BART. In the case of transfer to Reddit, we observe that despite similar zero-shot performance with transfer from CNNDM, there is a more sizeable gap with 10-shot transfer. This suggests that our intermediate finetuning does more closely align the BART model with the target domain. Furthermore, when training on augmented data from round-trip translation, we see the best performance in transfer from WikiTransfer in all cases except BART transfer to CNNDM on 10-aug, which is likely due to the autoencoder pretraining objective of BART which biases
it towards copying and lead bias, allowing it to perform well in applications to CNNDM. We see improvements when training with augmented data in 10-example cases and most 100-example cases for WikiTransfer. Less improvement is seen in the 100-aug setting when transferring from BART or another domain. We hypothesize that the noise present in the larger augmented dataset causes this occasional performance drop, while the WikiTransfer models appear more robust to potential noise. We also found model robustness as the standard deviation of top-performing WikiTransfer models was least among all models in the majority of cases. Interestingly, for transfer from BART and another domain 100-aug only improves on CNNDM, the most extractive dataset, while the largest drop in performance from augmented data occurs on XSum. This XSum performance drop may be caused by the high compression in the XSum summaries which leaves less room for noisy output when compared to the longer CNNDM and BigPatent summaries which may still preserve the main meaning of the original summary better despite backtranslation noise. In most cases, 100-aug with WikiTransfer results in the best performance, only several points from the state-of-the-art supervised performance.
Transfer with Consistency Training: We find contrasting trends with the added consistency loss compared to data augmentation via round-trip translation. We note the most sizeable improvements in the more abstractive cases of XSum and Reddit. We hypothesize that the consistency loss promotes better abstraction as the model learns to be invariant to noise which does not change the meaning of the text, and is thus equipped with a better notion of paraphrasing. The consistency loss allows for better training of vanilla BART as well as in general better transfer from other domains than without consistency loss. The loss likely provides a regularization factor which prevents the models from overfitting to the supervised examples. As the WikiTransfer model is already more closely tuned to the target domain, this regularization may not make as large of a difference. This aligns with our observation of WikiTransfer models being more robust to noisy backtranslated data on XSum and Reddit. Transfer to Reddit shows similar results across models for consistency loss with 100 examples (better ROUGE-L for WikiTransfer, better ROUGE-1/2 for Reddit); vanilla BART's strong performance at 100 examples suggests that the in
Figure 1: ROUGE-1 scores across datasets, training dataset size, data augmentation  $(^{*} - \mathbf{a})$ , and consistency loss  $(^{*}$  c) showing the generalizable and robust performance of models transferred from WikiTransfer. Standard deviation bars are also plotted.
formation provided in this subset is sufficient for good performance, thus diminishing the gains from the head-start the WikiTransfer model provides in zero and 10-shot transfer. We leave aspects of the consistency training such as the role of the quality of the round-trip translation data and its relation to the transfer domain to future work.
6.1 Human Quality Assessment
We examine how the improved performance from WikiTransfer manifests itself in qualitative annotations when varying the amount of training data. We collect human judgment annotations for two of the four quality dimensions studied in Kryscinski et al. (2019); Fabbri et al. (2020), namely consistency and relevance. Consistency is defined as the factual alignment between the summary and the summarized source text, while relevance is defined as the selection of important content; only relevant
Target Dataset	CNNDM	XSum
Relevance	Consistency	Relevance	Consistency
0	4.37	4.71	3.75*	3.75
10-a	4.31	4.76	3.77*	4.10
100-a	4.25	4.86	4.00	4.04
Full supervision	4.31	4.86	4.11	3.98
Table 6: Summary relevance and factual consistency across CNNDM and XSum datasets with varying amounts of training data. All results except those with an asterisks do not differ in a statistically significant way (p-value of 0.05) from the full supervision score. Bold results emphasize the least amount of data to achieve statistically indistinguishable results from the fully-supervised results.
information should be included in the summary. We did not include fluency as a dimension as an initial inspection of the data found fluency to be of very high quality, and we did not include coherence due to our inclusion of single-sentence XSum summaries where coherence is not a factor. We randomly select 50 examples per dataset and collect the model output from the best-performing zero-shot, 10-aug, 100-aug, and fully supervised models on CNNDM and XSum. The annotator sees the source article and randomly-ordered output from the four models rates the summaries for relevance and consistency on a Likert from 1-5, with 5 being the best score. We averaged the score of two native English-speaking annotators on each example and then across examples, and found moderate and strong annotator correlations for relevance and consistency, respectively. Results are shown in Table 6. For CNNDM, we see an increase in consistency as more training data is added but not a statistically significant difference (using a Student's t-test with a p-value of 0.05) between 100 and full supervision for any of the relevance or consistency results. The relevance of the full model does not outperform the others, likely because the model output was more concise and was judged as not including source information, while the zero-shot output more closely resembles the lead-three bias, so was judged as more informative. For XSum, we see that relevance improves noticeably as more training data is used. We see varied results for consistency, although without statistically significant differences. This fluctuation in scores may be due to the transition of the model from using knowledge from pretraining in its output versus knowledge from the target dataset obtained during fine-tuning, which we discuss in the Appendix.
We introduced WikiTransfer, a novel and generalizable method for fine-tuning pretrained models on dataset-specific unsupervised data obtained from generic Wikipedia data. WikiTransfer models achieve state-of-the-art zero-shot abstractive summarization performance on the CNN-DailyMail dataset and generalize across three additional datasets. In few-shot settings, WikiTransfer models are robust to noise introduced through data augmentation and benefit from consistency loss on more abstractive datasets. Furthermore, human assessments of the resulting summaries do not show significant differences between the WikiTransfer few-shot summaries and fully-supervised summaries, demonstrating the efficiency of our approach.
We make use of existing datasets available through libraries such as huggingface's datasets library. Biases may exist in the datasets, such as political bias in the news datasets as well as gender bias in potentially all of the datasets. Thus, models trained on these datasets may propagate these biases. When used as intended, applying the summarization models described in this paper can save people much time. However, the current models are still prone to producing hallucinated summaries, and in such a case may contribute to misinformation on the internet. Further research is needed for ensuring the faithfulness of abstractive summaries to address this issue, as this issue is present among all current abstractive summarization models.
The experiments make use of V100 GPUs. We used up to 8 GPUs per experiment (depending on the experiment; sometimes a single GPU was used to run the maximum number of experiments in parallel). The experiments may take from several minutes in the case of few-shot experiments without augmentation to a couple of hours for the larger augmented datasets, and up to one day for full-dataset training. Over 400 experiments were run due to our requirement of averaging across multiple experiments. Future work should experiment with distilled models for more light-weight training. We note that while our work required extensive experiments to draw sound conclusions, future work will be able to draw on these insights and need not run as many large-scale comparisons, and models in production may be trained once for use using the most promising settings.
We thank Griffin Adams, Shrey Desai, and the NAACL reviewers for their constructive feedback.
Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. 2019. There are many consistent explanations of unlabeled data: Why you should average. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019. SEQ^3: Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 673-681, Minneapolis, Minnesota. Association for Computational Linguistics.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020a. Few-shot learning for abstractive multi-document opinion summarization. In EMNLP.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020b. Unsupervised opinion summarization as copycat-review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151-5169, Online. Association for Computational Linguistics.
Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163-1173, Atlanta, Georgia. Association for Computational Linguistics.
Eric Chu and Peter J. Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1223-1232. PMLR.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067.
Shrey Desai, Jiacheng Xu, and Greg Durrett. 2020. Compressive summarization with plausibility and salience modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 6259-6274, Online. Association for Computational Linguistics.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020. Gsum: A general framework for guided neural abstractive summarization. arXiv preprint arXiv:2010.08014.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457-479.
Alexander R Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626.
Belize Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020. Supervised contrastive learning for pretrained language model fine-tuning. arXiv preprint arXiv:2011.01403.
Xinyu Hua and Lu Wang. 2017. A pilot study of domain adaptation effect for neural abstractive summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 100-106, Copenhagen, Denmark. Association for Computational Linguistics.
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019. Earlier isn't always better: Subaspect analysis on corpus and system biases in summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3324-3335, Hong Kong, China. Association for Computational Linguistics.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519-2531, Minneapolis, Minnesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. 2020. The summary loop: Learning to write abstractive summaries without examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5135-5150, Online. Association for Computational Linguistics.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131-4141, Brussels, Belgium. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary!
topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314-319, Florence, Italy. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Florence, Italy. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818-2826. IEEE Computer Society.
Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019. Exploring domain shift in extractive text summarization.
Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280.