SlowGuess/NAACL_2021_OCR huggingface dataset:
Yang Liu $^{\text{心}}$ , Sheng Shen $^{\text{◇}}$ , and Mirella Lapata $^{\text{△}}$
Microsoft Cognitive Services Research
$\diamond$  University of California, Berkeley
$\triangle$  School of Informatics, University of Edinburgh
yaliu10@microsoft.com sheng.s@berkeley.edu mlap@inf.ed.ac.uk
In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summarization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.
Automatic summarization has enjoyed renewed interest in recent years, thanks to the popularity of neural network models and their ability to learn continuous representations without recourse to preprocessing tools or linguistic annotations. The availability of large-scale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018; Narayan et al., 2018) containing hundreds of thousands of document-summary pairs has driven the development of neural architectures for summarization. Several approaches have been proposed, in the vast majority sequence-to-sequence models which are trained in an end-to-end fashion with a maximum likelihood estimation loss (See et al., 2017; Celikylmaz et al., 2018; Paulus et al., 2018; Gehrmann et al., 2018).
Despite promising results, there are specific characteristics of the summarization task which render it ill-suited to standard sequence-to-sequence training. For instance, maximum-likelihood training on single reference datasets might not be optimal for summarization which is subject to a great
deal of human variation (Harman and Over, 2004; Nenkova, 2006). In the context of extractive summarization, different people select different sentences to include in a summary (Rath et al., 1961), and when writing abstracts, disagreement exists both in terms of writing style and the specific content deemed important for the summary (Harman and Over, 2004). Although summarization models would naturally benefit from multiple target references, it is unrealistic to expect that multi-reference datasets can be created at scale for neural network training. In fact, most popular benchmarks are collated opportunistically, based on summaries which only loosely correspond to the source input.
For example, Narayan et al. (2018) create a dataset by pairing the first sentence of a news article with the rest of the document under the assumption that the introductory sentence expresses the gist of the article. Grusky et al. (2018) pair articles with metadata available in HTML pages under the assumption that HTML tags (e.g., description) denote summary-like content. In other work (Liu et al., 2018; Perez-Beltrachini et al., 2019), multi-document summarization datasets are created by viewing lead sections in Wikipedia articles as summaries of documents cited therein. The inherent noise in the data collection process further hampers training with models often being prone to hallucination (Song et al., 2018; Maynez et al., 2020), and struggling to identify which content units are salient (Tan et al., 2017).
In this paper, we propose to alleviate these problems by turning to knowledge distillation (Bucilu et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015; Kim and Rush, 2016). Knowledge distillation transfers knowledge from a larger "teacher" network to a smaller "student" model by training the student to imitate the teacher's outputs (in addition to learning from the training data set). In "born-again networks", (Furlanello et al., 2018) the teacher and student have the same neural archi
1Our code is available at https://github.com/nlpyang/NoisySumm.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 692-703 June 6-11, 2021. ©2021 Association for Computational Linguistics
tecture and model size, and yet surprisingly the student is able to surpass the teacher's accuracy. Intuitively, such self-knowledge distillation is effective because the teacher's output distribution provides a richer training signal capturing additional information about training examples. In the context of summarization, the teacher can benefit student training in two ways. It provides a softened distribution over reference summaries thereby enriching the single reference setting. Moreover, the teacher's distribution is (to a certain extent) denoised enabling the student to circumvent inaccuracies in the training data. We further capitalize on the idea that both the teacher and the student should be robust to noise and introduce several noise injection techniques which together with knowledge distillation improve model generalization and performance.
We present experiments on several summarization benchmarks (Narayan et al., 2018; Perez-Beltrachini et al., 2019; Hermann et al., 2015) covering single- and multi-document summarization settings as well as different types of summaries (e.g., verbose or more telegraphic). Across datasets, the proposed framework boosts the performance of pretrained and non-pretrained abstractive summarizers, achieving new state-of-the-art results.
2.1 Neural Abstractive Summarization
Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where the encoder maps the sequence of tokens in the source document  $\mathbf{x} = [x_1,\dots,x_n]$  to a sequence of continuous representations  $\mathbf{z} = [z_{1},\ldots ,z_{n}]$ , and the decoder autoregressively generates the target summary  $\mathbf{y} = (y_{1},\dots,y_{m})$  token-by-token, hence modeling the conditional probability  $p(y_{1},\dots,y_{m}|x_{1},\dots,x_{n})$ .
Rush et al. (2015) and Nallapati et al. (2016) were among the first to apply the neural encoder-decoder architecture to text summarization. See et al. (2017) enhance this model with a pointer-generator network which allows to copy words from the source text, and a coverage mechanism which keeps track of words that have been summarized. Other work develops abstractive models trained end-to-end with reinforcement learning based on multiple encoders and hierarchical attention (Celikyilmaz et al., 2018) or a coverage mechanism where the decoder attends over previously
generated words (Paulus et al., 2018). Gehrmann et al. (2018) follow a bottom-up approach where a content selector first determines which phrases in a source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding. Although the majority of summarization systems are composed of LSTM units, Narayan et al. (2018) and (Perez-Beltrachini et al., 2019) propose abstractive models based on convolutional neural networks.
Pretrained language models have recently emerged as a key technology for achieving impressive gains in abstractive summarization (Liu and Lapata, 2019; Lewis et al., 2020; Song et al., 2019). These models first pretrain a language model with self-supervised objectives on large corpora and then fine-tune it on summarization datasets. Liu and Lapata (2019) combine a pretrained encoder based on BERT (Devlin et al., 2019) with a randomly initialized decoder, demonstrating substantial gains on summarization performance. Song et al. (2019) pretrain an encoder-decoder framework to reconstruct (masked) fragments within a sentence and then fine-tune it on summarization datasets. In the same vein, Lewis et al. (2020) present BART, an encoder-decoder Transformer (Vaswani et al., 2017), pretrained by reconstructing a text corrupted with several arbitrary noisy functions. Bao et al. (2020) design UNILMv2, a Transformer-based neural network pretrained as a pseudo-masked language model. Qi et al. (2020) introduce their own novel self-supervised task based on future  $n$ -gram prediction.
Knowledge Distillation refers to a class of methods for training a new smaller student network by learning from a teacher network (in addition to learning from the training data). It is generally assumed that the teacher has been previously trained, and the parameters for the student are estimated by matching the student's predictions to the teacher.
Let  $T$  and  $S$  denote teacher and student models, respectively. Let  $f_{T}$  and  $f_{S}$  be functions of the teacher and student. The models are typically neural networks and function  $f$  can be in principle defined using the output of any network layer (e.g., a hidden or softmax layer). Knowledge distillation methods are commonly expressed as minimizing
an objective function over training set  $\mathcal{X}$ :
$$
\mathcal {L} _ {K D} = \sum_ {x _ {i} \in \mathcal {X}} l \left(f _ {T} \left(x _ {i}\right), f _ {S} \left(x _ {i}\right)\right) \tag {1}
$$
where  $l()$  is a loss function that penalizes the difference between the teacher and the student.
Specific instantiations of this general framework include minimizing the teacher/student difference based on output logits, intermediate hidden representations, attention maps, and derivatives of the loss to the input (Ba and Caruana, 2014; Romero et al., 2014; Zagoruyko and Komodakis, 2017; Czarnecki et al., 2017). Other work integrates an ensemble of teachers in order to improve the student (Urban et al., 2016), trains a succession of students (Furlanello et al., 2018), introduces a "teacher assistant" for better knowledge transfer (Mirzadeh et al., 2019), and regularizes multi-task agents (Parisotto et al., 2015; Teh et al., 2017) in reinforcement learning. Compared to direct training, knowledge distillation provides a more stable training process which leads to better performing student models (Hinton et al., 2015; Phuong and Lampert, 2019). Recent work (Furlanello et al., 2018; Hahn and Choi, 2019) also sheds light on leveraging knowledge distillation for training a high-performing student model with the same size as the teacher (see the discussion in the next section).
Knowledge distillation has been also shown to improve results for various NLP tasks. Tan et al. (2019) use it to transfer knowledge from BERT to smaller models, helping them approach or exceed the quality of much larger pretrained neural networks. Aside from distilling large models into smaller ones (Kim and Rush, 2016; Mou et al., 2016) or ensembles of models into single models (Kuncoro et al., 2016; Liu et al., 2019), knowledge distillation has been further used in multi-task learning, e.g., to teach a multi-task student from single-task teachers (Clark et al., 2019).
3 Self-Knowledge Distillation for Text Summarization
Self-knowledge distillation refers to the special case where the teacher and student have identical neural network architectures. Surprisingly, perhaps, it has been consistently observed (Furlanello et al., 2018; Yang et al., 2019; Ahn et al., 2019; Liu et al., 2020) that students trained with self-knowledge distillation outperform their teachers by significant margins in several computer vision and language
modeling tasks. Recent efforts have also focused on understanding why this happens, e.g., by observing that knowledge transferred by the teacher is localized mainly in higher layers and does not affect early (feature extraction) layers much (Gotmare et al., 2019), by interpreting the teacher's knowledge as importance weighting (Furlanello et al., 2018), by showing that early-stopping is crucial (Dong et al., 2019), and by studying how self-distillation modifies regularization (Mobahi et al., 2020).
For text summarization, we argue that self-knowledge distillation can potentially alleviate problems in conventional maximum likelihood training. Summarization models are typically trained on single reference document-summary pairs, however considering a single summary as the only correct reference during maximum likelihood training can harm model generalization (Elbayad et al., 2018) and is counter-intuitive. There can be multiple valid summaries for a source input (Harman and Over, 2004; Nenkova, 2006) and even the single reference summaries available are not entirely goldstandard due to the inherent noise in the automatic construction of large-scale summarization datasets (Krysciński et al., 2019). With self-knowledge distillation, teacher outputs provide softened distributions of the reference summaries, which can be viewed as an enrichment of the single reference setting and a reweighting of gold summaries to prevent the student from becoming overconfident in its predictions.
The standard objective for an abstractive summarization model is negative log likelihood:
$$
\mathcal {L} _ {\mathrm {N L L}} = - \sum_ {t = 1} ^ {T} \log \left(p \left(y _ {t} \mid y _ {1} ^ {t - 1}, x\right)\right) \tag {2}
$$
where  $x$  indicates the source document,  $y_1^t$  indicates the  $t$ -th token in the target summary and  $y_1^{t-1}$  are the first  $t-1$  tokens in the target summary. We further assume that the teacher is a fully trained neural model, the student has the same architecture with the teacher, and access to the learned teacher's output distribution  $p_T(y_t | y_{1:t-1}, x)$ :
$$
\mathcal {L} _ {\mathrm {K D}} = \sum_ {t = 1} ^ {T} \mathrm {K L} \left(p _ {T} \left(y _ {t} \mid y _ {1} ^ {t - 1}, x\right), p _ {S} \left(y _ {t} \mid y _ {1} ^ {t - 1}, x\right)\right) \tag {3}
$$
where  $p_T(y_t | y_1^{t-1}, x)$  and  $p_S(y_t | y_1^{t-1}, x)$  are model outputs from the teacher and student, respectively.
It is common practice to compensate for no direct access to the training data (see Equation (3)) by interpolating between the two losses in Equations (3) and (2). So, the final objective for training the student becomes:
$$
\mathcal {L} _ {\text {F I N A L}} = (1 - \lambda) \mathcal {L} _ {\text {N L L}} + \lambda \mathcal {L} _ {\text {K D}} \tag {4}
$$
where  $\lambda$  is a mixture parameter combining the one-hot distribution and the teacher distribution.
We further want our summarization systems to be robust to natural noise found in existing datasets. Injecting noise onto training samples has been proven useful for improving model generalization (Xie et al., 2019). We extend this idea for knowledge distillation, and propose a novel framework for introducing noise to both distillation signals and training data. We design different noise mechanisms for the teacher and student, and select the best noise configuration experimentally.
Noisy Teacher To inject noise into the distillation signals, we incorporate a teacher dropout mechanism (Bulò et al., 2016), where dropout is kept active while generating teacher predictions for training the student. In this manner, the teacher generates variable supervision labels for the student with some degree of uncertainty, alleviating the problem of overfitting to the teacher predictions. Meanwhile, it can also be considered as approximating an average ensemble from many neural networks (Bulò et al., 2016).
The knowledge distillation loss now becomes:
$$
\mathcal {L} _ {\mathrm {K D}} = \sum_ {t = 1} ^ {T} \operatorname {K L} \left(\tilde {p} _ {T} ^ {\alpha} \left(y _ {t} \mid y _ {1} ^ {t - 1}, x\right), p _ {S} \left(y _ {t} \mid y _ {1} ^ {t - 1}, x\right)\right) \tag {5}
$$
where  $\tilde{p}_T^\alpha$  indicates the predictions from the teacher model with active dropout  $\alpha$ .
Noisy Student To inject noise into the training data, we propose various mechanisms to perturb the source input. Random perturbation is effective in enforcing local smoothness for training text generation models under the assumption that semantically similar inputs can be mapped to the same or similar targets. A related approach has been shown to improve the performance of machine translation models in self-training settings (He et al., 2019). For text summarization, where the input is usually a long document, we design the following perturbation policies:
1. Word Drop: a word in the source document is removed with probability  $p_d$ .
2. Word Replacement: for each word  $x_{i}$  in the source document, we calculate a candidate replacement list by selecting  $k$  words most similar to  $x_{i}$  from the vocabulary. The similarity is calculated as the cosine distance between the embedding of  $x_{i}$  and embeddings of all other words in the vocabulary. Then, a source word is replaced with a word randomly selected from its candidate replacement list with probability  $p_{r}$ .
3. Sentence Drop: a sentence in the source document is removed with probability  $p_s$ .
4. Gaussian Noise: a Gaussian noise vector  $\mathbf{e}$  is multiplied with the embeddings  $\mathbf{x}$  of input words:  $\mathbf{x} \gets \mathbf{x} \otimes \mathbf{e}, \mathbf{e} \sim N(I, \sigma^2 I)$ .
These perturbation policies can be applied simultaneously or successively as a pipeline. We experimentally found the best combination for our task to be the sequential application of word drop, followed by word replacement, and sentence drop. Although Gaussian noise has been effective in natural language understanding tasks (Zhang and Yang, 2018), we found it not to be helpful in our summarization experiments. The knowledge distillation loss with a student trained on noisy data becomes:
$$
\mathcal {L} _ {\mathrm {K D}} = \sum_ {t = 1} ^ {T} \mathrm {K L} \left(\tilde {p} _ {T} ^ {\alpha} \left(y _ {t} \mid y _ {1} ^ {t - 1}, x\right), p _ {S} \left(y _ {t} \mid y _ {1} ^ {t - 1}, \tilde {x}\right)\right) \tag {6}
$$
where  $\tilde{x}$  indicates perturbed source input.
In this section, we describe the summarization datasets used in our experiments and discuss various implementation details.
4.1 Summarization Datasets
We evaluated our model on two single-document summarization datasets, namely the CNN/DailyMail news highlights (Hermann et al., 2015) and XSum (Narayan et al., 2018), and one multi-document summarization dataset, i.e., WikiCatSum (Perez-Beltrachini et al., 2019). These datasets represent different summary styles ranging from highlights to very brief-one sentence summaries. The summaries also vary with respect to the type of rewriting operations they exemplify (e.g., CNN/DailyMail showcases more cut and paste operations while XSum is genuinely abstractive). Finally, two of these
Without Pretraining	CNN/DailyMail	XSum
R1	R2	RL	R1	R2	RL
LEAD	40.42	17.62	36.67	16.30	1.60	11.95
PTRNET	39.53	17.28	36.38	28.10	8.02	21.72
TransformerAbs	40.21	17.76	37.09	31.04	10.48	24.54
+SKD	40.64	18.10	37.43	32.22	11.45	25.56
+SKD +Noisy T	40.79	18.24	37.57	32.32	11.56	25.72
+SKD +Noisy T +Noisy S	40.86	18.27	37.66	32.76	11.88	26.07
BASE-size Pretrained Models	R1	R2	RL	R1	R2	RL
MASSBASE	(123M)	42.12	19.50	39.01	39.75	17.24
BERTSUMABS	(156M)	41.72	19.39	38.76	38.76	16.33
UNILMv2BASE	(110M)	43.45	20.71	40.49	43.69	20.71
+SKD	(110M)	43.44	20.68	40.51	43.76	21.04
+SKD +Noisy T	(110M)	43.59	21.01	40.66	44.11	21.30
+SKD +Noisy T +Noisy S	(110M)	43.77	20.98	40.82	44.14	21.34
LARGE-size Pretrained Models	R1	R2	RL	R1	R2	RL
UNILMLARGE	(340M)	43.08	20.43	40.34	—	—
BARTLARGE	(400M)	44.16	21.28	40.90	45.14	22.27
T511B	(11B)	42.05	20.34	39.40	—	—
Table 1: ROUGE F1 results on CNN/DailyMail and XSUM test sets (R1 and R2 are shorthands for unigram and bigram overlap; RL is the longest common subsequence).  $\mathit{SKD}$  refers to a system trained with self-knowledge distillation, Noisy  $T$  are SKD models trained with noisy signals while Noisy  $S$  are student models trained on noisy data. Results for comparison systems are taken from the authors' respective papers or obtained on our data by running publicly released software.
datasets (XSum and WikiCatSum) were created automatically following various assumptions about the correspondence of purported summaries to the source input.
CNN/DailyMail contains news articles and associated highlights, i.e., a few bullet points written by journalists which give a brief overview of the article. We used the standard splits of Hermann et al. (2015) for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents). We did not anonymize entities. Sentences were split with the Stanford CoreNLP toolkit (Manning et al., 2014) and the dataset was pre-processed following See et al. (2017). Input documents were truncated to 512 tokens.
XSum contains 226,711 news articles accompanied with a one-sentence summary, answering the question "What is this article about?". We used the splits of Narayan et al. (2018) for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work. Input documents were also truncated to 512 tokens.
WikiCatSum is a multi-document summarization dataset derived from WikiSum (Liu et al.,
2018). The target summary is the lead section of a Wikipedia article, and the source input are webpages related to this article. WikiCatSum (Perez-Beltrachini et al., 2019) represents three domains from the original Wikisum dataset under the assumption that these vary in terms of the topics the summaries discuss and their linguistic characteristics. Aside from the summaries, the dataset contains the input webpages whose length is truncated to the first 800 tokens. WikiCatSum contains 62,545 samples for the Company domain, 59,973 samples for the Film domain, and 60,816 samples for the Animal domain.
For all datasets, we evaluated our self-knowledge distillation framework in two settings. In the first setting, our models are non-pretrained while in the second setting we take advantage of pretrained language models which have demonstrated impressive improvements in summarization (Lewis et al., 2020; Liu and Lapata, 2019; Bao et al., 2020).
Specifically, we adopt UNILMv2 (Bao et al., 2020) as the pretrained model. UNILMv2 is a Transformer-based neural network (Vaswani et al., 2017) with 12 Transformer layers and 12 attention
Without Pretraining	Company	Film	Animal	All
R1	R2	RL	R1	R2	RL	R1	R2	RL	R1	R2	RL
CV-S2S	24.5	9.4	19.9	34.6	19.8	30.7	42.2	28.4	38.5	33.8	19.2	29.7
CV-S2D	27.6	10.5	21.3	37.7	20.8	32.0	42.3	27.3	37.1	35.9	19.5	30.1
TF-S2S	26.0	9.5	20.4	36.5	18.8	31.0	44.0	28.8	40.0	35.5	19.0	30.5
+SKD	26.8	9.9	20.9	37.2	19.3	31.8	44.3	29.0	40.3	36.1	19.4	31.0
+SKD +Noisy T	27.2	10.3	21.0	37.7	20.6	32.0	44.6	29.1	40.4	36.5	20.0	31.1
+SKD +Noisy T +Noisy S	27.4	10.4	21.3	37.9	21.0	32.2	44.6	29.0	40.4	36.6	20.1	31.3
With Pretraining	R1	R2	RL	R1	R2	RL	R1	R2	RL	R1	R2	RL
UNILMv2BASE	33.32	14.36	25.39	42.51	25.92	36.54	45.45	31.69	40.91	40.4	24.0	34.3
+SKD	33.20	14.66	25.53	42.39	25.90	36.53	45.59	31.87	41.12	40.4	24.1	34.4
+SKD +Noisy T	33.42	14.87	25.80	42.60	26.02	36.65	45.75	32.19	41.30	40.6	24.4	34.6
+SKD +Noisy T +Noisy S	33.50	14.95	25.85	42.71	26.09	36.77	45.86	32.23	41.40	40.7	24.4	34.7
Table 2: ROUGE F1 results on WikiCatSum test sets (R1 and R2 are shorthands for unigram and bigram overlap; RL is the longest common subsequence). Results are reported separately on three domains and in combination (All).  ${SKD}$  refers to systems trained with self-knowledge distillation,  ${NoisyT}$  are SKD systems trained with noisy signals,and Noisy  $S$  are SKD students trained on noisy data. Results for comparison systems are taken from the authors' respective papers or obtained on our data by running publicly released software.
heads. It is pretrained as a pseudo-masked language model on a large corpus (label smoothing is applied with smoothing factor 0.1). We fine-tuned our teacher models following the procedure outlined in Bao et al. (2020). In the non-pretrained setting, we adopt a Transformer encoder-decoder model with 6 layers, 768 hidden size and 2,048 feed-forward filter size. Label smoothing was also used with smoothing factor 0.1. All teacher models in this setting were trained from randomly initialized parameters following Liu and Lapata (2019).