SlowGuess/NAACL_2021_OCR huggingface dataset:
In all knowledge distillation experiments, student models have the same neural network architecture with their teachers and are trained with the same hyperparameters as the teacher models. The best teacher and student model are selected by evaluating perplexity on the development set. For noisy distillation models, word drop probability  $p_d$  was set to 0.1. The candidate length  $k$  for word replacement was 10 and word replacement probability  $p_r$  was 0.1. Sentence drop probability  $p_s$  was 0.05.
During decoding we used beam search (size 5), and tuned  $\alpha$  for the length penalty (Wu et al., 2016) between 0.6 and 1 on the validation set; we decode until an end-of-sequence token is emitted. Repeated trigrams are blocked (Paulus et al., 2018).
We evaluated summarization quality automatically using ROUGE (Lin, 2004). We report unigram and
bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency. Examples of system output are shown in Table 5.
Table 1 summarizes our results on the CNN/DailyMail and XSum (single document) datasets. The first block includes the results of non-pretrained models. We present the LEAD baseline (which simply selects the first three sentences in a document for CNN/DailyMail and the first sentence for XSum). We also report the results of See et al.'s (2017) pointer generator network (PTRNET), and an abstractive system from Liu and Lapata (2019) based on Transformers (TransformerAbs; see Section 4.2 for details). The latter forms the backbone of our self-knowledge distillation models (SKD). We present a variant without noise (+SKD), a variant with noise in the teacher training signal (+Noisy T), and a third variant where the student is additionally trained on noisy data (+Noisy S).
The second and third blocks in Table 1 include the results of pretrained models. To make comparisons fairer, we separate LARGE- (second block) from BASE-size (third block) pretrained models based on parameter size (shown within parentheses). With regard to LARGE-size models, we report the results of three very strong summarization systems finetuned with UNILMLarge (Bao et al., 2020), BARTLarge (Lewis et al., 2020), and T511B (Raffel et al., 2019). Our BASE-size models include BERTSUMBASE (Liu and Lapata, 2019), a
Models	CNN/DailyMail	XSum
TRANSFORMERABS	20.8	32.7
+Noisy SKD	21.4	33.6
UNILMv2BASE	23.7	38.7
+Noisy SKD	24.8	39.9
summarizer based on a BASE-size BERT encoder and a randomly initialized decoder,  $\mathrm{MASS}_{\mathrm{BASE}}$  (Song et al., 2019) and  $\mathrm{UNILM}_{\mathrm{BASE}}$  which are both finetuned with BASE-size pretrained models.
As can be seen in Table 1, SKD improves over teacher models in both pretrained (BASE-size) and non-pretrained settings. We also observe that injection of noise brings further improvements with noise in the training signal (+Noisy T) seeming more effective compared to noisy data augmentation (+Noisy S). Overall, we obtain competitive results with SKD and BASE-size pretrained models and even manage to outperform UNILMLARGE and  $\mathrm{T5}_{11\mathrm{B}}$  on the CNN/DailyMail dataset.
Table 2 presents experimental results on the WikiCatSum dataset. The first block in the table includes results for non-pretrained models. CV-S2S and CV-S2D (Perez-Beltrachini et al., 2019) are convolutional encoder-decoder models. The former is a standard convolutional decoder, while the latter adopts a hierarchical convolutional decoder which first generates target sentence vectors, and then generates target words based on sentence vectors. TF-S2S is a standard Transformer encoder-decoder model trained on WikiCatSum (Perez-Beltrachini et al., 2019). TF-S2S is the model used in our SKD system and its noisy version (+Noisy T, +Noisy S). The second block includes the results of a system using the BASE-size pretrained model UNILM<sub>BASE</sub> on its own and with SKD. Results are reported per domain (Company, Film, and Animal) and across domains (All).
Under pretrained and non-pretrained settings, we observe that SKD boosts the performance of the teacher model (UNILM<sub>BASE</sub> and TF-S2S, respectively) and that the injection of noise is beneficial. Improvements in performance vary across domains, with Film showing the least gains. Column All in Table 2 shows average ROUGE across domains. Although SKD and noise injection improve results, we observe that non-pretrained models benefit more.
Table 3: Factual correctness on CNN/DailyMail and XSum test set. +Noisy SKD are students trained on noisy signals and noisy data.
CNN/DailyMail	Succinct	Inform	Fluent
UNILMv2BASE	0.47	0.40	0.54
+Noisy SKD	0.53	0.60	0.46
XSum	Succinct	Inform	Fluent
UNILMv2BASE	0.46	0.36	0.53
+Noisy SKD	0.54	0.64	0.47
WikiCatSum	Company	Film	Animal
UNILMv2BASE	0.62	0.47	0.45
+Noisy SKD	0.38	0.53	0.55
Table 4: Human evaluation on CNN/DailyMail, XSum, and WikicatSum test sets. +Noisy SKD is UNILMv2<sub>BASE</sub> trained with self-knowledge distillation (on noisy signals and noisy data). All pairwise differences between systems are significant ( $p < 0.05$ ) using a paired  $t$ -test.
5.2 Factual Consistency Evaluation
Besides ROUGE, we also use FactCC (Krysciński et al., 2019) to evaluate the factual correctness of the generated summaries. FactCC is a BERT-based classifier trained to identify conflicts between a source document and a generated summary. Given a document-sentence pair as input, it assigns a positive label if factual information mentioned in a summary sentence is consistent with the document, otherwise it assigns a negative label. We view the percentage of positive labels assigned by FactCC to all generated summaries as a factual correctness score for a summarization system.
We performed experiments with the publicly released version of FactCC. Our results on the CNN/DailyMail and XSum datasets are presented in Table 3. Here, we only focus on single-document summarization, as there is no version of FactCC trained on multi-document datasets. As can be seen, the application of SKD (trained with noisy signals and on noisy data) improves factual consistency for non-pretrained and pretrained models on both datasets. All +Noisy SKD students are significantly  $(p < 0.05)$  more factually correct compared to their teachers (TransformerAbs and UNILMv2_BASE), using a paired student  $t$ -test.
In addition to automatic evaluation, we also assessed system output by eliciting human judgments. We compared the quality of the summaries produced by a teacher model (UNILMv2BASE)
$^{2}$ https://github.com/salesforce/factCC
CNN/Daily Mail
GOLD	LZ Granderson: millennials say they'll marry if and when they want. He says that's not the case; they're happily single and happy. Granderson says marriage is about family, not money.
UNILMv2	LZ Granderson: millennials say they don't care what their generation thinks about marriage. He says they'll get married if and when they want. LZ: marriage is linked to economic well-being, but it's not clear if that's true.
+Noisy SKD	Carol Costello: talk to any millennial and you can envision an America virtually marriage-free. In countries like Sweden or Denmark, people don't feel pressured to marry even if they have kids together.
XSum
GOLD	More than half of pupils in Wales have passed their GCSE exam for the third year running.
UNILMv2	More than 66.6% of pupils in Wales have achieved the top grades in their GCSE exams.
+Noisy SKD	Two thirds of Welsh pupils who took GCSEs got A* to C grades, according to this year's results.
WikiCatSum (Animal)
GOLD	The Conception Bank silver boa (Chilabothrus Argentum) is a species of boa described in May 2016. It is only known from the conception island bank in the Bahamas. It is the first known discovery of a West Indian boa species in 73 years. It is named for its unique silver color.
UNILMv2	The Conception Bank silver boa (Chilabothrus Argentum) is a species of snake in the family Boidae. It is endemic to the Bahamas. The species was discovered on Conception Island Bank, which comprises uninhabited islets.
+Noisy SKD	The Conception Bank silver boa (Chilabothrus Argentum) is a species of nonvenomous boa endemic to the Bahamas. It was discovered in 2016 on Conception Island Bank, an uninhabited islet in the Bahamas.
Table 5: GOLD reference summaries and automatic summaries produced by UNILMv2<sub>BASE</sub> and its distilled student on the CNN/DailyMail, XSum, and WikiCatSum datasets.
against its distilled student (+Noisy SKD). For CNN/DailyMail and XSum, human participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the following criteria: Succinctness (Does the summary avoid repetition?), Informativeness (Does the summary capture the document's most important information?), and Fluency (Is the summary fluent and grammatical?). Evaluation was conducted on the Amazon Mechanical Turk crowdsourcing platform. We used the same test documents (20 in total) from Liu and Lapata (2019) for both CNN/DailyMail and XSum. We elicited five responses per HIT. Systems were rated along each dimension, and assigned a score corresponding to the proportion of times a system was selected as better against another.
Human evaluation results are shown in Table 4 (upper part). On both CNN/DailyMail and
XSum datasets participants perceive the student (+Noisy SKD) as significantly  $(p < 0.05)$  more succinct and informative compared to the teacher (UNILMv2BASE). However, on Fluency, the student tends to be worse. Upon inspection we found student summaries to be rather telegraphic, and hypothesize that crowdworkers tend to penalize them in terms of fluency, even though they are grammatical.
Human evaluation was performed slightly different for WikiCatSum. Recall that this is a multi-document dataset, where input documents are discontinuous webpage fragments. To allow participants to perform the experiment in a timely fashion, we used the gold summary as a proxy for the content of the input. Crowdworkers were presented with the output of two systems (again UNI L M v2 BASE and +Noisy SKD) and asked to decide which one was better according to the in
formation contained in the gold summary. Evaluation was conducted on AMT, we randomly selected 20 samples from the test set and elicited three responses per HIT. For each domain, we report the proportion of times a system was chosen as better.
Human evaluation results are shown in Table 4 (lower part). AMT Crowdworkers prefer the summaries produced by the student for the Animal and Film domains, but not for Company; we found that the distilled model tends to generate too many entities in one sentence which render the summaries too dense for this domain.
In this paper we advocated the use of self-knowledge distillation for abstractive summarization, as a means to alleviate problems associated with maximum-likelihood training for this task. We also introduced several noise functions (in the training signal and training data) which help regularize training and further boost performance. Experiments on three benchmark datasets demonstrate that our framework can improve both non-pretrained and pretrained summarizers. In the future we would like to investigate more thoroughly which aspects of pretrained models improve and how self-knowledge distillation can be enhanced with more sophisticated noise functions.
Acknowledgments We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, "Translating Multiple Modalities into Text").
Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. 2019. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9155-9163, Long Beach, California.
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27, pages 2654-2662. Curran Associates, Inc.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2020. UniLMv2: Pseudo-masked language models for unified language model pre-training. CoRR, abs/2002.12804.
Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, page 535-541, New York, NY, USA. Association for Computing Machinery.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. 2016. Dropout distillation. In Proceedings of the International Conference on Machine Learning, pages 99-107, New York, New York.
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1662-1675, New Orleans, Louisiana.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. 2019. BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5931-5937, Florence, Italy. Association for Computational Linguistics.
Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu. 2017. Sobolev training for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4278-4287. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. 2019. Distillation  $\approx$  early stopping? harvesting dark knowledge utilizing anisotropic information retrieval. In NeurIPS 2019 Workshop on Machine Learning with Guarantees, Vancouver, Canada.
Maha Elbayad, Laurent Besacier, and Jakob Verbeek. 2018. Token-level and sequence-level loss smoothing for RNN language models. CoRR, abs/1805.05062.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, pages 1602-1611, Stockholm, Sweden.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, Louisiana. OpenReview.net.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Sangchul Hahn and Heeyoul Choi. 2019. Self-knowledge distillation in natural language processing. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 423-430, Varna, Bulgaria. INCOMA Ltd.
Donna Harman and Paul Over. 2004. The effects of human variation in DUC summarization evaluation. In Text Summarization Branches Out, pages 10-17, Barcelona, Spain. Association for Computational Linguistics.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. CoRR, abs/1909.13788.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Wojciech Krysciński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. CoRR, abs/1908.08960.
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into
one MST parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1744-1753, Austin, Texas. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by summarizing long sequences. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6035-6044, Online. Association for Computational Linguistics.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487-4496, Florence, Italy. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. 2019. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. CoRR, abs/1902.03393.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett Bartlett. 2020. Self-distillation amplifies regularization in Hilbert space. CoRR, abs/2002.05715.
Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, and Zhi Jin. 2016. Distilling word embeddings: An encoding approach. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 1977-1980, Indianapolis, Indiana.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Ani Nenkova. 2006. Summarization evaluation for text and speech: issues and approaches. In Proceedings of the 9th International Conference on Spoken Language Processing, Pittsburgh, Pennsylvania.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. 2015. Actor-mimic: Deep multitask and transfer reinforcement learning. CoRR, abs/1511.06342.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada.
Laura Perez-Beltrachini, Yang Liu, and Mirella Lapata. 2019. Generating summaries with topic templates and structured convolutional decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5107-5116, Florence, Italy. Association for Computational Linguistics.
Mary Phuong and Christoph Lampert. 2019. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning, pages 5142-5151, Long Beach, California.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram