SlowGuess/NAACL_2021_OCR huggingface dataset:
Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. 2020. TED: A pretrained unsupervised summarization model with theme modeling and denoising. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1865-1874, Online. Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Hao Zheng and Mirella Lapata. 2019. Sentence centrality revisited for unsupervised summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6236-6247, Florence, Italy. Association for Computational Linguistics.
Jiawei Zhou and Alexander Rush. 2019. Simple unsupervised summarization by contextual matching. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5101-5106, Florence, Italy. Association for Computational Linguistics.
Haichao Zhu, Li Dong, Furu Wei, Bing Qin, and Ting Liu. 2019. Transforming wikipedia into augmented data for query-focused summarization. arXiv preprint arXiv:1911.03324.
A.1 Comparison to Previous Work
We show a comparison of our best-performing WikiTransfer few-shot results with those from Zhang et al. (2019) in Table 7. The Pegasus numbers were obtained by a single run as opposed to our average of the best three over 5 subsets. We show large improvements with our few-shot approach compared to previous numbers, except for the 100-shot experiment on XSum. The XSum dataset has the highest overlap with the Pegasus pretraining dataset of all datasets explored in Zhang et al. (2019), although that work states that the effect of removing this overlap does not affect the full-dataset performance. We hope that this comparison promotes future benchmarking of few-shot results.
A.2 Sample Summary Outputs
We include an example of model output summaries on the XSum dataset in Table 8. The example serves to demonstrate how output style varies as
Target Dataset	WikiTransfer	Pegasus (Zhang et al., 2019)
# training samples	0	10	100	0	10	100
CNNDM	39.11	17.25	35.73	39.39	16.92	36.00	42.08	18.93	38.83	32.90	13.28	29.38	37.25	15.84	33.49	40.28	18.21	37.03
XSum	31.85	10.44	23.75	35.17	12.76	26.80	37.26	14.20	28.85	19.27	3.00	12.72	19.39	3.45	14.02	39.07	16.44	31.27
Reddit	21.47	4.10	17.62	28.42	7.88	22.32	30.56	9.22	24.38	14.66	3.06	10.17	15.36	2.91	10.76	16.64	4.09	12.92
BigPatent	35.58	10.91	31.53	37.73	12.40	32.89	40.95	14.05	35.03	25.61	6.56	17.42	28.87	8.30	19.71	33.52	10.82	22.87
Table 7: A comparison of zero and few-shot performance between our best-performing WikiTransfer model (-a in the case of CNNDM and BigPatent and -c for XSum and Reddit) and the zero and few-shot results reported in Zhang et al. (2019).
Source Document: Ms Jones told BBC Radio Wales she did not want to give up being an AM to go to Brussels to replace Nathan Gill, UKIP Wales leader. Mr Gill has been told by the UKIP assembly group and the UKIP party chairman Steve Crowther to stop "double-jobbing" as an AM and MEP. Mr Gill said those making such calls were doing it out of "malice". "We've got Brexit now and I think that, possibly, it may be best to leave that role unfilled," Ms Jones told the Good Morning Wales programme. "I'm surprised I've not been formally asked what I'd like to do." Ms Jones, the South Wales West AM, is one of two people who could take up the role of UKIP Wales MEP if Mr Gill made it vacant - the other being South Wales East AM David Rowlands...
0: Lorraine Jones is a Welsh Labour Party Member of the Welsh Assembly for South Wales West.
10-a: Lorraine Jones is a Welsh Labour member of the Welsh Assembly for South Wales West.
100-a: Wales Assembly Member for South Wales West Rachel Jones says she has not been formally asked to become a UKIP MEP.
Full supervision: First Minister Carwyn Jones has said she is "surprised" she has not been asked to become a UKIP MEP.
Gold Summary: UKIP's Welsh MEP post may be better left unfilled as a result of Brexit , party AM Caroline Jones has said .
the amount of training data is increased and how the source of pretraining or fine-tuning data affects this style and model hallucinations. The source document does not state the first name of Ms. Jones, yet every model output, and the gold target, give her one. For zero and 10-aug, the model outputs Lorraine Jones, likely still under the influence of BART Wikipedia pretraining, as there is a Wikipedia article on the Welsh politician Ruth Lorraine Jones (although it does not appear in our intermediate fine-tuning subset). The zero and 10-aug also most resemble Wikipedia introduction sentences; although the output is compact and abstractive like an XSum target sentence, the "X is Y" format of Wikipedia appears. We see at 100-aug examples
Table 8: An example of WikiTransfer model output across dataset size used in fine-tuning, illustrating how model output style and hallucinated entities differ as the model moves from Wikipedia pretraining as a source of knowledge to the target dataset. Text not stated in the source document is highlighted in red.
Target Dataset	CNNDM
Transfer from	WikiTransfer	Reddit	BART
0	39.11	17.25	35.73	36.81	14.18	32.62	35.98	15.10	32.97
10	39.10	16.98	35.84	38.26	16.34	34.76	38.55	16.56	34.97
10-a	39.39	16.92	36.00	39.12	16.90	35.44	39.78	17.11	36.38
10-c	39.16	16.96	35.92	38.99	16.83	35.43	38.98	16.68	35.41
100	40.55	18.01	37.03	40.13	17.88	36.67	40.14	17.88	36.62
100-a	42.08	18.93	38.83	40.94	18.52	37.00	40.47	18.18	37.07
100-c	41.12	18.34	37.51	40.84	18.09	37.28	41.36	18.59	37.77
Target Dataset	XSum
Transfer from	WikiTransfer	Reddit	BART
0	31.85	10.44	23.75	24.04	6.43	18.99	19.87	2.75	15.66
10	34.95	12.61	26.58	30.69	10.22	23.29	22.45	5.94	17.23
10-a	34.98	12.73	26.79	31.03	10.23	23.29	26.10	8.19	20.18
10-c	35.17	12.76	26.80	31.25	10.54	23.73	28.28	9.13	21.61
100	36.92	14.09	28.44	34.17	12.64	26.37	35.17	13.29	27.20
100-a	36.87	14.18	28.62	31.75	11.12	24.49	28.85	9.46	22.28
100-c	37.26	14.20	28.85	36.14	13.65	27.97	36.65	14.05	28.57
Target Dataset	Reddit
Transfer from	WikiTransfer	CNDM	BART
0	21.47	4.10	17.62	21.37	4.14	17.76	18.66	2.90	15.33
10	27.88	7.62	22.09	26.55	6.83	21.29	19.37	3.51	15.72
10-a	28.07	7.70	22.47	26.88	6.95	21.46	21.39	4.57	17.22
10-c	28.42	7.88	22.32	27.20	7.14	21.67	20.42	3.97	16.45
100	29.87	8.93	23.31	28.90	8.42	22.56	29.66	8.88	23.12
100-a	30.54	9.24	24.31	29.28	8.51	23.28	28.96	8.39	22.80
100-c	30.56	9.22	24.38	30.78	9.45	24.14	30.78	9.22	23.32
Target Dataset	BigPatm
Transfer from	WikiTransfer	CNNDM	BART
0	35.58	10.91	31.53	33.57	9.34	25.76	32.56	9.64	29.27
10	37.06	11.58	32.37	35.76	10.62	30.63	34.48	10.76	30.56
10-a	37.73	12.40	32.89	36.83	11.33	30.95	36.11	11.40	32.04
10-c	37.64	12.24	33.05	36.11	10.84	30.64	33.99	10.48	30.45
100	39.61	13.53	33.86	39.35	13.03	33.88	39.06	13.04	33.61
100-a	40.95	14.05	35.03	38.88	12.69	32.88	38.77	12.88	33.55
100-c	39.87	13.76	34.32	39.74	13.45	34.49	39.46	13.37	34.28
Table 9: A comparison of transfer results across datasets, training dataset size, data augmentation techniques, showing the generalizable and robust performance of our models transferred from WikiTransfer.
that the model output is stylistically already much like that of the fully-supervised output and gold summary. This stylistic change is also reflected in the change in hallucination; the use of Rachel Jones is likely caused by the appearance of the name of a minister Rachel Haves in an article on Welsh politics found in the 100-aug subset. The model at this point is already fitting strongly to the target domain. For the fully supervised output, we see the use of Carwyn Jones, which does not match the gender of Ms. Jones but which is found 1090 times in the training source documents. Caroline Jones, the actual person in question, only appears 21 times in the training set. This phenomenon points to two interesting research directions for future work, how to properly preserve world knowledge from pretraining and improvement faithfulness to the source text in knowing when to insert world knowledge.
A.3 Additional Training Setting Details
We provide additional details regarding the training and validation of models. We also provide the exact numbers for few-shot transfer in Table 9.
WikiTransfer Data: We use the statistics from the original papers to determine the extractive bin of the dataset except for the case of Reddit; upon seeing the strong zero-shot performance of the CNNDM, we investigated the extractive oracle of the Reddit dataset and found it to be much higher (about 31 ROUGE-1) than that stated in the original paper. We select the first M sentences for the pseudo-summaries from Wikipedia except in the case of Reddit, where we choose the IND-ORIG setting; this did not result in a difference in zero-shot performance but upon a qualitative inspection of the output, we found the IND-ORIG to be less biased towards Wikipedia style with the coherence of the summaries not being an issue.
We believe that the approximate level of extraction of desired summaries should be treated as prior knowledge. We also examine, however, how many data points are needed to accurately find the extractive oracle bin from target datasets. We found that using 10 data points sufficed to accurately estimate the bin of the extractive oracle.
Using the first M sentences does not produce ideal summaries of the remaining Wikipedia article, but experiments comparing the WikiTransfer approach on Wikipedia data as opposed to using in-domain data, as well as manual inspection of the data showed the validity of using Wikipedia data for proxy summaries. While the extractive oracle provides some measure of overlap, this heuristic does not ensure deeper semantic overlap or faithfulness between the pseudo summary and the rest of the article. We believe a valuable direction for future work is improving the target-specific data as well as encoding additional semantics and style-based subaspects into the pseudo summaries.
Training and Validation Hyperparameters: We found that full-precision floating-point gave slightly better, and more stable, results, so we report full-precision floating-point numbers. We set a maximum tokens-per-batch of 1024 and use gradient accumulation with an update frequency of 8 for all experiments with 10 data points, and 32 for 10-aug as well as all experiments with 100 (+ augmented) data points. For CNNDM 10 examples, we found it necessary to use a smaller learning rate (3e-6) to avoid immediate overfitting. We per
form validation after each model update, as the models typically converge in under 50 iterations. For the 100-aug setting, we begin validation checking after 50 iterations as the models typically converged around 100 iterations. We train with label-smoothed cross-entropy (Szegedy et al., 2016) loss for few-shot transfer. We found that models can be sensitive to the choice of hyperparameters in the few-shot settings, hence the averaging over 5 subsets to reduce variation.
We use the standard training and testing splits of each dataset (for Reddit, we use the same 80-10-10% split as in Zhang et al. (2019)), and thus refer the reader to the original papers for detailed statistics. For validation, we used a subset of the target-dataset validation set consisting of 4k examples. While this matches previous unsupervised and transfer settings, we understand that the use of a large validation set is not ideal. We experimented with smaller validation sets on Reddit transfer and found that the results did not change using a validation set of only 10 data points, although we leave a further examination of the effect of validation set size for future work.
We provide the range of the label-smoothed cross-entropy validation loss by taking the average validation loss (over five subsets) from the best-performing and worst-performing transfer models on a given dataset. The range of validation losses for CNNDM is (4.49, 5.05), for XSum (4.63, 5.45), for Reddit (5.98, 6.65), and for BigPatent (4.88, 6.40).
Full Supervision and Additional Experiments: For zero and few-shot transfer, we compare transfer from BART trained on WikiTransfer data to the best-transferring BART model trained on the datasets. The following numbers are ROUGE-1. Our application of BART on fully-supervised data achieves state-of-the-art performance on Reddit (32.74). We perform slightly worse on CNNDM (44.16 vs 45.94 from Dou et al. (2020)). Lower performance when compared to Pegasus-large (Zhang et al., 2019) on XSum (45.14 vs 47.21) and BigPatent (43.34 vs 53.63) is likely due to differences in capacity and training batch size, as our performance is comparable to Pegasus-base. Our approach is not model-specific to BART, so we leave the application of other models such as Pegasus to future work and do not focus on achieving state-of-the-art on the fully-supervised individual datasets.
We limit our primary few-shot experiments to
Target Dataset	CNNDM
Transfer from	WikiTransfer	Reddit	BART
10-UDA	39.51	17.2	35.93	38.86	16.83	35.26	37.81	16.82	34.64
100-UDA	39.89	17.27	36.26	40.19	17.91	36.98	38.43	17.22	35.49

Target Dataset	XSum
Transfer from	WikiTransfer	Reddit	BART
10-UDA	35.09	12.86	26.94	29.34	9.56	22.6	19.75	3.19	15.09
100-UDA	36.57	13.89	28.42	33.91	12.23	26.05	26.44	7.97	20.35

Target Dataset	Reddit
Transfer from	WikiTransfer	CNNDM	BART
10-UDA	27.6	7.19	22.37	25.21	5.96	20.63	20.90	4.16	16.82
100-UDA	29.91	8.35	23.93	28.28	7.68	22.83	27.67	7.46	21.81

Target Dataset	BigPatent
Transfer from	WikiTransfer	CNNDM	BART
10-UDA	36.58	11.45	32.29	33.77	9.45	29.19	32.32	10.0	28.98
100-UDA	40.25	13.77	35.09	39.04	12.99	34.41	38.2	12.7	34.16 5
Table 10: Results from experiments using the original formulation of UDA Xie et al. (2019) on 10 examples.
10 and 100 data points, as we are primarily interested in real-world few-shot applications where we likely do not have 1k data points. Initial experiments using 1k and 10k data points on CNNDM showed that WikiTransfer still outperforms transfer from other domains, although both remain below state-of-the-art performance. We leave a further examination of fine-tuning on larger training sets for future work.
A.4 Semi-supervised UDA experiments
We experimented with the original formulation of UDA in a semi-supervised setting. In this framework, the label (summary) outputted by the model for an augmented example should be the same as the label of the original document on unlabeled examples. Let  $x_{U}$  be an unsupervised source document from the target dataset other than our supervised few-shot examples. Let  $\hat{x}_U$  be a paraphrase of input  $x_{U}$  generated via round-trip translation as in our above data augmentation experiments. To apply teacher forcing, we require a label  $y_{U}$ , which we obtain for each model by applying the model fine-tuned on the analogous few-shot subset. In addition to the supervised loss  $L_{sup}(x,y)$ , we thus introduce another loss  $L_{uda}(x_U,\hat{x}_U,y_U) =$ :
$$
\sum_ {t = 1} ^ {m} K L \left(f \left(\cdot \left| y _ {U 0: t - 1}, x _ {U}\right) \right| \mid f \left(\cdot \left| y _ {U 0: t - 1}, \hat {x} _ {U}\right)\right)\right) \tag {4}
$$
In practice, for an epoch, we iterate through the supervised examples with loss  $L_{sup}$  followed by iterating over the unsupervised examples  $L_{uda}$ . We sampled 1k unlabeled data points for 10-UDA experiments and 3k unlabeled data points for 100-UDA. Results of initial experiments are shown in Table 10. We find that the performance of the UDA models is very dependent on the quality of the pseudo-labels generated. We chose the model
trained on the first data subset of the 5 runs to generate the pseudo-labels and if this model had higher performance then this model likely performed better in UDA (this occurred in our Reddit transfer to CNNDM with 100 data points. As a result, as the quality of the pseudo-labels improves with 100-shot training the UDA performance improves and is more comparable to the unaugmented performance in Table 9.