Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
Authors: Chenguang Zhu
Ziyi Yang
Robert Gmyr
Michael Zeng
Xuedong Huang
Venue:  (2020)
Keywords: Summarization
Pretraining

Abstract: Lead bias is a common phenomenon in news summarization, where early parts of an article often contain the most salient information. While many algorithms exploit this fact in summary generation, it has a detrimental effect on teaching the model to discriminate and extract important information. We propose that the lead bias can be leveraged in a simple and effective way in our favor to pretrain abstractive news summarization models on large-scale unlabelled corpus: predicting the leading sentences using the rest of an article. Via careful data cleaning and filtering, our transformer-based pretrained model without any finetuning achieves remarkable results over various news summarization tasks. With further finetuning, our model outperforms many competitive baseline models. For example, the pretrained model without finetuning outperforms pointer-generator network on CNN/DailyMail dataset. The finetuned model obtains 3.2% higher ROUGE-1, 1.6% higher ROUGE-2 and 2.1% higher ROUGE-L scores than the best baseline model on XSum dataset.

Under review as a conference paper at ICLR 2020
MAKE LEAD BIAS IN YOUR FAVOR: A SIMPLE AND EFFECTIVE METHOD FOR NEWS SUMMARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Lead bias is common phenomenon in summarization, where early parts of an article often contain the most salient information. While many algorithms exploit this fact in summary generation, it has detrimental effect on teaching the model to discriminate and extract important information. We propose that the lead bias can be leveraged in simple and effective way our favor to pretrain abstrac- tive news summarization models on large-scale unlabeled corpus: predicting the leading sentences using the rest of an article. Via careful data cleaning and filter- ing, our transformer-baser pretrained model without any finetuning achieves re- markable results over various news summarization tasks. With further finetuning, our model outperforms many competitive baseline models. For example, the pre- trained model without finetuning achieves state-of-the-an results on DUC-2003 and DUC-2004 datasets. The finetuned model obtains 3.2% higher ROUGE-I, 1.6% higher ROUGE-2 and 2. % higher ROUGE-L scores than the best baseline model XSum dataset. Human evaluations further show the effectiveness of our method.
INTRODUCTION
The goal of text summarization to condense piece of text into a shorter version that contains the salient information. Due to the prevalence of news articles and the need to provide succinct sum- maries for readers, majority of existing datasets for summarization come from the news domain (Hermann et al. 2015 Sandhaus 2008 Narayan et al. 2018). However, according to journalistic conventions, the most important information in a news report usually appears near the beginning of the article (Kedzie et al. [0188] Jung al. 2019). While it facilitates faster and easier understanding of the news for readers, this lead bias causes undesirable consequences for summarization models. The output of these models is inevitably affected by the positional information of sentences. Fur- thermore, the simple baseline of using the top few sentences as summary can achieve a stronger performance than many sophisticated models (See et al. 2017). It can take lot of effort for models to overcome the lead bias Kedzie et al. (2018).
Additionally, existing summarization models fully supervised and require time and labor- intensive annotations to feed their insatiable appetite for labeled data. For example, the New York Times Annotated Corpus (Sandhaus (2088) contains 1.8 million news articles, with 650,000 sum- maries written by library scientists. Therefore, some recent work (Gusev) 2019) explores the effect of domain transfer utilize datasets other than the target one. But this method may be affected by the domain drift problem and still suffers from the lack of labelled data.
The recent promising trend of pretraining models (Devlin et al. 2018, Radford al.) 2018 proves that a large quantity of data can be used to boost NLP models' performance. Therefore, we put forward a novel method to leverage the lead bias of news articles in our favor conduct large-seale pretraining of summarization models. The idea is to leverage the top few sentences of news article as the target summary and use the rest as the content. The goal of our pretrained model to generate an abstractive summary given the content. Coupled with careful data filtering and cleaning, the lead bias can provide a delegate summary of sufficiently good quality, and it immediately renders the large quantity of unlabeled news articles corpus available for training news summarization models.
We employ this pretraining idea on a three-year collection of online news articles. We conduct thorough data cleaning and filtering. For example, to maintain quality assurance bar for using

---

Under review as a conference paper at ICLR 2020
leading sentences as the summary, compute the ratio of overlapping non-stopping words between the top sentences and the rest of the article. As higher ratio implics a closer semantic connection, only keep articles for which this ratio is higher than a threshold.
We end up with 21.4M articles based on which we pretrain a transformer-basee encodcr-decoder summarization model. We conduct thorough evaluation of our models five benchmark news summarization datasets. Our pretrained model achieves a remarkable performance on various target datasets without any finetuning. This shows the effectiveness of leveraging the lead bias to pretrain on large-scale news data. We further finetune the model on target datasets and achieve better results than a number of strong baseline models. For example, the pretrained model without finetuning obtains state-of- of-the-art results on DUC-2003 and DUC-2004. The finetuned model obtains 3.2% higher ROUGE-1, 1.6% higher ROUGE-2 and % higher ROUGE-I. scores than the best baseline model on XSum dataset (Narayan et al. 2018). Human evaluation results also show that our models outperform existing baselines like pointer-generato network.
The rest of paper is organized as follows. We introduce related work in news summarization and pretraining in Section 2 We describe the details of pretraining using lead bias in Section 3 We introduce the transformer-based summarization model in Section 4 We show the experimental results in Section and conclude the paper in Section 6
2
RELATED WORK
2.1 DOCUMENT SUMMARIZATION
End-to-end abstractive text summarization has been intensively studied in recent literature. To gener- ate summary tokens, most architectures take the encoder-decoder approach (Sutskever et al.] 2014). Rush al. 20155) first introduces attention-based seq2seq model to the abstractive sentence summarization task. However, its output summary degenerates as document length increases, and out-of-vocabulary (OOV) words cannot be efficiently handled. To tackle these challenges, Seee al. (017)) proposes a pointer-generator network that can both produce words from the vocabulary via generator and copy words from the source article via a pointer. |auluus al. (2017): ): Li et al. (2018) utilize reinforcement learning to improve the result. [Gehrmann et al. (2018) uses a con- tent selector to over-determine phrases in source documents that helps constrain the model to likely phrases. You et al. (2019) adds Gaussian focal bias and salience-selection network to the trans- former encoder-decoder structure (Vaswani et al. 2017) for abstractive summarization. Grenander al. (2019) randomly reshuffles the sentences in news articles to reduce the effect of lead bias in extractive summarization.
2.2 PRETRAINING
In recent years, pretraining language models have proved to be quite helpful in NLP tasks. The state- of-the-art pretrained models include CoVe (McCann et al. 2017). ELMo (Peters et al. 2018). GPT (Radford et al. 2018). BERT (Devlin et al. 2018) and M (Dong et al. 2019). Built upon large- scale corpora, these pretrained models learn effective representations for various semantic structures and linguistic relationships. As a result, pretrained models have been widely used with considerable success in applications such as question answering (Zhu et al. 2018), sentiment analysis (Peters) al. 2018) and passage reranking (Nogueira & Chol 2019). Furthermore, UniLM (Dong et al., 2019 leverages its sgguence-to-squeeeee capability for abstractive summarization: the BERT model has been employed as an encoder in BERTSUM (Liu & Lapata 2019) for extractive/absstrative summarization.
Compared to our work, UniLM (ong et al., 2019) is a general language model framework and does not take advantage of the special semantic structure of news articles. Similarly, BERTSUM (Liu & Lapata 2019) directly copies the pretrained BERT structure into its encoder and finetunes on labelled data instead of pretraining with the large quantity of unlabeled news corpus available.
2

---

Under review as a conference paper at ICLR 2020
Original Artide (truncated):
though she is na langer an offical member of the royal family. [2] The Duchess of York's assistant, Antonia Marshall, confirmed that Ferguson has launched her own lifesaye brand [3] also uploaded sorre photos of the products under Ferguson's label via Instagram. that next year. [5] Ats of late, it is still unclea what products will be aveilable from Sereh Senses, Bui based Marshall's photo, it has been confirmed
[]] Sarah
continues
[4] Marshall
Senses
starting
there are scont diffusars and tea. [6] likely for part of Uhe proceeds from Sarah Semes lee donated to Street Child UK charity that Fergusen supports.
article truncatad);
[4] Marshell also revealed that Sarah Senses will avallable
Leed-3 Summary:
latn, still urcesr what products will be ovailable from Sarah Senses. But besed Warshall's photo, confirmed
year.
[1] Sarah Ferguson continues to make a name herself even though she is no longer an official member of the royal family.
Modal
York's assistant, Antonia Morshall, confirmed that Ferguson has laurched her own photos of products un ded Ferpuson's label vis instagram.
diffusers and tea. [6] It is also likely for part of the proceeds from Sarah
lnfestyle brand. [3] She also uplozded
to donated Street Child UK, charity that Ferguson supports.
Figure 1: Using Lead-3 summary as target in pretraining.
3 PRETRAINING WITH LEADING SENTENCES
News articles usually follow the convention of placing the most important information early in the content, forming an inverted pyramid structure. This lead bias has been discovered number of studies (Kedzie et al., 2018 Jung et al., 2019 Grenander et al. 2019). One of the consequences is that the lead baseline, which simply takes the top few sentences as the summary, can achieve summarization. For instance, in the CNN/Daily Mail dataset (Hermann et al. (2013), using the top three sentences as summaries can get a higher ROUGE score than many deep learning based models. This positional bias brings lots of difficulty for models to extract salient information from the article and generate high-quality summaries. For instance, Grenander et al. (201)) discovers that most models' performances drop significantly when random sentence is inserted in the leading position, or when the sentences in a news article are shuffled.
rather strong performance in
On the other hand, news summarization, just like many other supervised learning tasks, suffers from the scarcity of labelled training data. Abstractive summarization is especially data-hungry since the efficacy of models depends on quah-qualtty handerafted summaries.
We propose that the lead bias in news articles can be leveraged in our favor to train an abstractive summarization model without human labels. Given news article, we treat the top three sentences, denoted by Lead-3, as the target summary, and use the rest of the article as news content. The goal of the summarization model produce Lead-3 using the following content, as illustrated in Figure]]
The benefit of this approach is that the model can leverage the large number of unlabeled news articles for pretraining. In the experiment, we find that the pretrained model alone can have a strong performance on various news summarization datasets, without any further training. We also finetune the pretrained model on downstream datasets with labelled summaries. The model can quickly adapt to the target domain and further increase its performance.
is worth noting that this idea of utilizing structural bias for large-scale summarization pretraining is not limited to specific types of models, and it can be applied other types of text well: academic papers with abstracts, novels with editor's notes, books with tables of contents.
However, one should carefully examine and clean the source data to take advantage of lead bias, as the top three sentences may not always form good summary. We provide more details in the experiments about the data filtering and cleaning mechanism apply.
3

---

Under review as a conference paper at ICLR 2020
4 MODEL
In this section, we introduce our abstractive summarization model, which has transformer-based encoder-decoder structure. We first formulate the supervised summarization problem and then present the network architecture.
4.1 PROBLEM FORMULATION
We formalize the problem of supervised abstractive summarization follows. The input consists of a pairs of articles and summaries: {[X,, Y1), (X2, Y2).... Y.)) Each article and summary are tokenized: X; (x1 (#1;--.- .... -., FL,) and Y; (y1,..., (N,) In abstractive summarization, the summary tokens need not be from the article. For simplicity, we will drop the data index subscript. The goal of the system generate summary Y (, Vm)) given the transcript X (nn
4.2 NETWORK STRUCTURE
We utilize a transformer-baseee encoder-decodes structure that maximizes the conditional probability of the summary: [XY|X,0 0), where 0 represents the parameters.
4.2.1 ENCODER
The encoder maps each token into a fixed-length vector using a trainable dictionary D randomly initialized using normal distribution with zero mean and a standard deviation of 0.02. Each trans- former block conducts multi-head self-attention. And we use sinusoidal positional embedding in order to process arbitrarily long input. the end, the output of the encoder is set of contextualized vectors:
Encoder- Transformer( zn}) = {uf u
4.2.2 DECODER
The decoder is transformer that generates the summary tokens time, based the input and previously generated summary tokens. Each token is projected onto a vector using the same dictionary as the encoder.
The decoder transformer block includes an additional cross-attention layer to fuse in information from the encoder. The output of the decoder transformer is denoted as:
Decoder-Transformee
( {up u
To predict the next token Wk, we reuse the weights of dictionary D as the final linear layer to decode uL into probability distribution over the vocabulary: P(wk| of softmax (Du
Training. During training, we seek to minimize the cross-entropy loss:
L(0)
log P(yklyck X)
We use teacher-forcing in decoder training, i.c. the decoder takes ground-truth summary tokens as input. The model has 10 layers of 8-headed transformer blocks both its encoder and decoder, with 154.4M parameters.
Inference. During inference, we employ beam search select the best candidate. The search starts with the special token (BEGIN). We ignore any candidate word which results in duplicate trigrams. We select the summary with the highest average ligeliieldhood per token.
EXPERIMENTS
5.1 DATASETS
We evaluate our model on five benchmark summarization datasets: the New York Times Annotated Corpus (NYT) (Sandhaus 2008), XSum (Narayan al., 2018), the CNN//ailyMail dataset (Her-
4

---

Under review as a conference paper at ICLR 2020
mann et al. 2015 DUC-2003 and DUC-2004 (Over al. 2007). These datasets contain 104K, 227K, 312K, 624 and 500 news articles and human- summaries respectively, covering differ- the same train/val/test split
ent topics and various summarization styles. For NYT dataset, and filtering methods following [Drrrett et al. (2016). As DUC-2003/2004 datasets are very small, follow West al. (20T9) to employ them as test set only.
5.2 IMPLEMENTATION DETAILS
We use SentencePiece (Kudo & Richardson, 2018) for tokenization, which segments any sentence into subwords. We train the SentencePiece model on pretrained data generate vocabulary of size 32K and of dimension 720. The vocabulary stays fixed during pretraining and finetuning.
Pretraining. We collect three years of online news articles from June 2016 to June 2019. We filter out articles overlapping with the evaluation data on media domain and time range. We then conduct several data cleaning strategies.
First, many news articles begin with reporter names, media agencies, dates or other contents irrele- vant to the content, e.g. "New York (CNN) -", "Jones Smith, May 10th, 2018:". We therefore apply simple regular expressions to remove these prefixes.
Second, to ensure that the summary is concise and the article contains enough salient information, we only keep articles with 10-150 words in the top three sentences and 150- 200 words in the rest, and that contain least 6 sentences in total. In this way, filter out i) articles with excessively long content to reduce memory consumption; ii) very short leading sentences with little information which are unlikely to good summary. To encourage the model to generate abstrative summaries, we also remove articles where any of the top three sentences is exactly repeated in the rest of the article.
Third, we try to remove articles whose top three sentences may not form a relevant summary. For this purpose, we utilize a simple metrie: overlapping words. We compute the portion of non-stopping words in the top three sentences that are also in the rest of an article. A higher portion implies that the summary representative and has higher chance of being inferred by the model using the rest of the article. To verify, compute the overlapping ratio of non-stopping words between human-edited summary and the article in CNN/Daily dataset, which has median value of 0.87. Therefore, in pretraining, we keep articles with overlapping word ratio higher than 0.65.
These filters rule out around 95% of the raw data and we end up with 21.4M news articles, 12,000 of which are randomly sampled for validation.
We pretrain the model for 10 epochs and evaluate its performance on the validation set at the end of each epoch. The model with the highest ROUGE-L score is selected.
During pretraining, we use a dropout rate of 0.3 for all inputs to transformer layers. The batch size is 1,920. We use RAdam (Liu et al. 2019) as the optimizer, with learning rate of 10-4. Also, due to the different numerical scales of the positional embedding and initialized sentence piece embeddings, divide the positional embedding by 100 before feeding into the transformer. The beam width is set to 5 during inference.
Finetuning. During finetuning, we keep the optimizer, learning rate and dropout rate unchanged as in pretraining. The batch size is 32 for all datasets. We pick the model with the highest ROUGE-L score on the validation set and report its performance on the test set. More details are given in the Appendix.
Our strategy Pretraining with unlabeled Lead-3 summaries called PL. We denote the pretrained model with finetuning on target datasets as PL-FT. The model with only pretraining and no finetun- ing denoted PL-NoFT, which is the same model for all datasets.
5.3 BASELINE
To compare with our model, we select a number of strong summarization models as baseline sys- tems. LFADDXX uses the top X sentences summary (Liu & Lapata 2019). The value of X is
5

---

Under review as a conference paper at ICLR 2020
Model
R1
R2
RL
Model
R1
R2
RL
LEAD-3
39.58
20.11
35.78
LEAD-1
16.30
1.60
11,95
PTGEN
42.47
25.61
PTGEN
29.70
9.21
23.24
PTGEN Cov
43.71
26,40
PTGEN+COV
28.10
8.02
21.72
DRM
42.94
26.02
TCONVS2S
31.89
11.54
25.75
PL-NoFT
35.32
17.80
31.88
PL-NoFT
24.12
5.59
19.20
PL-FT
44.18
27.49*
40.65
PL-FT
35.06 TM
13.12
27.86
Table 1: ROUGE recall scores on NYT test set. Table 2:
ROUGE FI results on XSum test set.
Model
RI
R2
RL
LEAD-3
40.5
17.7
36.7
Unsupervised
SEQ
17.85
3.94
19.53
GPT-2
29.34
8.27
26.58
PL-NoFT
38.95
16.27
35.11** **
Supervised
PTGEN
36.44
15.66
33.42
PTGEN+COV
39.53
17.28
36.38
DRM
39.87
15.82
36.90
BOTTOMUP
41.22
18.68
38.34
PLAFT
40.41
17.81
37.19
Table 3:
ROUGE Fl results on CNN/DailyMaiil test set.
DUC-2003
DUC-2004
Model
R1
R2
RL
RI
R2
RL
Supervised
ABS
28.48
8.91
23.97
28.18
8.49
23.81
DRGD
31.79
10.75
27.48
Unsupervised
SEQ
20.90
6.08
18.55
22.13
6.18
19.3
BottleSum Ex
21.80
5.63
19.19
22.85
5.71
19.87
Bottleum Self
21.54
5.93
18.96
22.30
5.84
19.60
GPT-2
4.98
0.37
4.63
5.29
0.28
4.90
PL-NoFT
23.23
6.64
20.42
22.71
6.37*
19.97
Table 4: ROUGE recall scores DUC-2003 and DUC-2004 test set.
3 for NYT and CNN/DailyMaii and for XSum to accommodate the nature of summary length. PTGEN (See et al. 2017) the pointer-generator network. DRM (Paulus et al. 2017) leverages deep reinforcement learning for summarization. TCoNvS2S (Narayan et al. 2018) based convolutional neural networks. BOTTOMUP (Gehrmann et al.) 2018) uses a bottom-up approach to generate summarization. ABS (Rush et al. 2015a) uses neural attention for summary generation. DRGD [Li et al. 2017) is based on a deep recurrent generative decoder.
To compare with pretrain-only model, include several unsupervised abstractive baselines: SEQ (Baziotis et al. 2019) employs the reconstruction loss and topic loss for summarization. Bot- tleSum (West et al.) 2019) leverages unsupervised extractive and self-supervised abstractive meth- ods. GPT-2 (Radford et al. 2018) large-scaled pretrained language model which can be directly
used generate summaries
'The ROUGE scores here CNN/Daily Mail higher than those reported the original paper, because we extract sentences in Daily Mail rather than
We follow GPT-2's approach to add TI; after the article for summary generation. And we use the GPT-2 small model available.

---

Under review as a conference paper at ICLR 2020
Ratio of Novel n-grams Summaries
Reference PL-NoFT
PL-FT
Percentage
40
30
20
1-gram
2-gram
3-gram
4-gram
Figure 2: Ratio of novel n-grams in summaries from reference, PL-NoFT and PL-FT models in NYT test set.
5.4 METRICS
We employ the standard ROUGE-I, ROUGE-2 and ROUGE-L metrics (Lin) 2004) to evaluate all summarization models. These three metrics respectively evaluate the accuracy unigrams, bigrams and longest common subsequence. ROUGE metrics have been shown to highly correlate with the human judgment (Lin 2004). Following Durrett al. (206)); West et al. (2019), use F-measure ROUGE on XSUM and CNN/DailyMaii, and use limited-length recall-measure ROUGE on NYT and DUC. In NYT, the prediction is truncated to the length of the ground-truth summaries; in DUC, the prediction is truncated to 75 characters.
5.5 RESULTS
The results are displayed in Table 1 Table Table 3 and Table As shown, both NYT and XSum dataset, PL-FT outperforms all baseline models by large margin. For instance, PL-FT obtains 3.2% higher ROUGE-1, 1 1.6% higher ROUGE-2 and 2. 1% higher ROUGE-L scores than the best baseline model on XSum dataset. We conduct statistical test and found that the results are all significant with p-value smaller than 0.05 (marked by #) 0.01 (marked by **), compared with previous best scores. On CNN/DailyMail dataset, PL-FT outperforms all baseline models except BottomUp (Gehrmann al. 2018).
PL-NoFT, the pretrained model without any finetuning, also gets remarkable results. On XSum dataset, PL-NoFT is almost 8% higher than Lead-1 in ROUGE-1 and ROUGE-L. On CNN/DailyMail dataset, PL-NoFT significantly outperforms unsupervised models SEQ and GPT- and even surpasses the supervised pointer-generator network. PL-NoFT also achieves state- of-the-art results on DUC-2003 and DUC-2004 among unsupervised models (except ROUGE-1 DUC-2004), outperforming other carefully designed unsupervised summarization models. It's worth noting that PL-NoFT is the same model for all experiments, which proves that our pretrain strategy effective across different news corpus.
5.6 ANALYSIS
Example Summaries. We showcase two examples of LEAD-33, reference summaries and the sum- maries generated by our models in Figure As shown, without finetuning the target dataset, PL-NoFT model can generate summaries with salient information, often shortening and reorganiz- ing the original article's contents. After further finetuning. PL-FT captures relevant content from the article that is similar to that in the reference summary.
Abstractiveness. We measure the abstractiveness of our model via the ratio of novel n-grams in summaries, i.e. the percentage of n-grams in the summary that are not present in the article. Figure shows this ratio in summaries from reference and generated by PL-NoFT and PL-FT in NYT dataset.

---

Under review as a conference paper at ICLR 2020
Lead-3:
Lead-3:
china's recent test of antisatellite weapon sent confusing message to the world about its military intentions, but the united states and china are slowly building stronger military-to-miline ties, the top-ranking united states military officer said friday.
the first major investigation of medicare mar- keting, the oklahoma insurance commissioner has documented widespread misconduct by agents working for humana and has ordered the com- pany take corrective action to protect consumers against nigh-pressure sales tactics.
gen. peter pace, chairman of the joint chiefs of staff, said he urged his chinese counterparts in meetings thursday and friday to provide more information about the aims of their military buildup.
the commissioner, kim holland, said some agents had enrolled medicare recipients in humana prod- uets that "they did not understand and did want."
at least 68 agents did not have the licenses needed to sell insurance in oklahoma, ms. holland said monday in an interview.
he called the meetings "encouraging" but said he did not get concrete answers.
Reference:
Reference:
oklahoma insurance commissioner kim holland, in first major probe of medicare marketing, docu- ments widespread misconduct by agents working for humana.
gen peter pace says urged his chinese counter- parts to provide more information about aims of their military buildup.
calls meetings in beijing encouraging, but says he did not get concrete answers.
orders company take corrective action to protect consumers against high-pressure sales tactics. contends some agents enrolled medicare recipi- ents in humana products they did not understand want.
says china's recent test of antisatellite weapon sent confusing message to world about its military intentions, but us and china are slowly building stronger iilitaay-toomiiitaer tics.
PL.NooT Summary:
the state's top insurance regulator said monday that it had found widespread misconduct by agents working for humana and has ordered the com- pany to take corrective action to protect consumers against high-pressure sales tactics.
PL-NoFT Summary:
china's 's recent test of an antisatellite weapon sent a confusing message the world about its military intentions, but the united states and china are slowly building stronger military toaary-ttomiinne to-mititary ties. the test, which china said nothing about for more than a week, raised alarms in washington that the chinese military might seek the capacity cripple the pentagon.
the oklahoma state insurance commissioner said that some agents had enrolled medicare recipients humana products that they did not understand or want.
PL-FT Summary:
oklahoma insurance comr kim holland, in first major investigation of medicare marketing, docu- mented widespread misconduct by agents working for humana and has ordered company to take cor- rective action to protect consumers against high- pressure sales tacties.
PL-FT Summary:
chairman gen peter pace urges chinese counter- parts in mcetings about aims of their military buildup.
provide more information
china's 's recent test of antisatellite weapon sends confusing message to world about its military intentions, but us and china are slowly building stronger military- to military ties.
says some agents had enrolled medicare recipients humana products that they did not understand or want.
Two summary examples in NYT test set. The summaries are from the leading three sen- tences of the article, the reference, the pretrained-only model PL-NoFT and the pretrained+finetuned model PL-FT.
Figure 3:
Both PL-NoFT and PL-FT yield more novel I-grams in summary than the reference. And PL-NoFT has similar novelty ratio with the reference in other n-gram categories. Also, we observe that the novelty ratio drops after finetuning. We attribute this to the strong lead bias in the NYT dataset which affects models trained on
5.7 HUMAN EVALUATION
We conduct human evaluation of the generated summaries from our models and the pointer generator network with coverage. We randomly sample 100 articles from the CNN//ailyMaii test set and ask 3 human labelers from Amazon Mechanical Turk assess the quality of summaries with score from to 5 (5 means perfect quality. The labelers need to judge whether the summary can express the salient information from the article in a concise form of fluent language. We put the details of
8

---

Under review as a conference paper at ICLR 2020
Model
Average Score
Standard deviation
PTGEN+COv
3.24
1.17
PL-NoFT
3.47
1.12
PL-FT
4.09
0.88
Table 5: Average and standard deviations of human evaluation scores for summaries on CNNiDailyMaill test set. Scores range from to with 5 being perfect. Each summary is judged by human evaluators. PL-FT's result statistically significant compared with pointer-generate network with coverage with a p-value less than 10-7.
evaluation guidelines in Appendix. To reduce bias, we randomly shuffle summaries from different sources for each article.
As shown in T both of our models PL-NoFT and PL-FT outperform the pointer generator net- work (PTGen+Cov). and PL-FT's advantage over PTGen+Cov is statistically significant. This shows the effectiveness of both our pretraining and finetuning strategy. To evaluate the inter-annotator agreement, we compute the kappa statistics among the labels and the score 0.34.
CONCLUSIONS
In this paper, we propose simple and effective pretraining method for news summarization. By employing the leading sentences from news article as its target summary, turn the problematic lead bias for news summarization in our favor. Based on this strategy, we conduct pretraining for abstractive summarization in a large-scale news corpus. We conduct thorough empirical tests on five benchmark news summarization datasets, including both automatic and human evaluations. Results show that the same pretrained model without any finetuning can achieve state-of-the afrt results among unsupervised methods over various news summarization datasets. And finetuning on target domains can further improve the model's performance. We argue that this pretraining method can be applied in more scenarios where structural bias exists.
REFERENCES
Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. Seq" 3: Dif- ferentiable sequence-tor sequence-to-sqqeeneer compression. arXiv preprint arXiv: 03350. 2019.
autoencoder for unsupervised abstractive sentence
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800, 2018.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Sisao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03199 2019.
Greg Durrett, Taylor Bing-Kirkpatrick, and Dan Klein. Learning-based single-document summa- rization with compression and anaphoricity constraints. arXiv preprint 1.XXv:160330888,, 2016.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. arXiv preprint 1rXiv:1808.10777 2018.
Matt Grenander, Yue Dong, Jackie C.K. Cheung, and Annie Louis. Countering the effects of lead bias in news summarization via multi-stage training and auxiliary losses. EMNLP, 2019.
Ilya Gusev. Importance of copying mechanism for news headline generation. arXiv preprint arXiv: 1475, 2019.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, pp. 1693-1701, 2015. 2015.

---

Under review as a conference paper at ICLR 2020
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. Earlier isnt always better: Sub- aspect analysis on corpus and system biases in summarization. EMNLP, 2019.
Chris Kedzie, Kathleen McKeown, and Hal Daume III. Content selection in deep learning models of summarization. arXiv preprint arXiv: 0.210.12344 12343, 2018.
Taku Kudo and John Richardson. Sentencepicce: simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 06226, 2018.
Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. Deep recurrent generative decoder for abstractive text summarization. arXiv preprint 17Xiv:/70..008 2017.
Piji Li, Lidong Bing, and Wai Lam. Actor-critic based training framework for abstractive summa- rization. arXiv preprini arXiv: 2018.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization
Branches Out,, 2004.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:/908 1908.0365, 2019.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. EMNLP, 2019.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances Neural Information Processing Systems, pp. 6294- 2017.
Shashi Narayan, Shay В Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint 1881::1808.08745. 2018.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint 190.:0000 2019.
Paul Over, Hoa Dang, and Donna Harman. Due in context. Information Processing & Management, 4366):150- 2007. 2007.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.043000 2017.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.0068. 2015a.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00623 2015b.
Evan Sandhaus. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752, 2008.
Abigail See, Peter J Liu, and Christopher Manning. Get the point: Summarization with pointer- generator networks. arXiv preprint
2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems,, pp. 3104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. pp. 5998-599999 2017.
10

---

Under review as a conference paper at ICLR 2020
Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. Bottlesum: Unsupervised and self- supervised sentence summarization using the information bottleneck principle. arXiv preprint arXiv:1909.07405. 2019.
Yongjian You, Weijia Jia, Tianyi Liu, and Wenmian Yang. Improving abstractive document sum- marization with salient information modeling. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2132-2141, 2019.
Chenguang Zhu, Michael Zeng, and Xuedong Huang. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv: 2018.
APPENDIX
MODEL SPECIFICATIONS IN FINETUNING
For NYT dataset, apply minimum and maximum summary generation length of 50 and 150 sentence pieces, respectively. The article is truncated to the first 400 sentence pieces. The beam width 8.
For XSum dataset, we apply a minimum and maximum summary generation length of 30 and 150 sentence pieces, respectively. The article is truncated to the first 400 sentence pieces. The beam width is 1.
For Mail dataset, we apply minimum and maximum summary generation length of 60 and 150 sentence pieces, respectively. The article is truncated to the first 350 sentence pieces. The beam width
For DUC-2003/2004 datasets, apply minimum and maximum summary generation length of 12 and 20 sentence pieces, respectively. The article is truncated the first 250 sentence pieces. The beam width is 3.
HUMAN EVALUATION SETUP
We ask human labelers from Amazon Mechanical Turk to assess the quality of summaries with a score from I to 5. Here's the evaluation guideline shown the labelers:
Score
Criteria
The summary contains all key points of the news.
The summary misses one key point of the news.
The summary misses two key points the news.
The summary misses all key points of the news. The summary is hardly related to the news the language is not natural and fluent.
Table 6: Scoring criteria for human evaluation of summaries.


Reviews:
[{"review_id": "HkgUIDhoFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests generating a large news summarization dataset by taking advantage of the fact that in news articles it is often the case that first few sentences contain the most important information. I have the following criticisms of this paper:\n- the idea is not novel. The XSUM dataset cited had used this to create a large dataset based on BBC articles as the editorial guidelines are such that the first sentence is a summary of the article. The lead1 baseline doesn't make sense, as it is the actual reference of the dataset. As implemented, it actually picks the second sentence of the original article, and unsurprisingly works worse than the lead-X for the other two datasets.\n- the filtering based on word overlap between the initial sentences and the rest of the document means that the training dataset will encourage models copying words; good summaries don't have high word overlap necessarily.\n- no human evaluation is not conducted; ROUGE indicates small differences, but it can't be trusted without confirmation by human evaluation\n- I don't agree that using positional information is bad for the models. The point is that we need to do better than that, but we should still take it into account", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJxhRlhaKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "For news article it has been know since long that the LEAD baseline is a tough-to-beat competitor. This paper proposes to use this knowledge as self-supervision for training summarization models.\nFor this the author download and clean 3 years of news articles and use this to (pre-)train a Tranformer model. This alone already provides a competitive baseline, which is greatly improved by fine-tuning it on 3 different data-sets. While the data-set can probably not be released, it would be very helpful to have the model available for reproductivity and benchmarking.\n\nThe paper is clear and well-written. Section 4 I believe is very redundant for an ICLR audience and could be moved to the appendix, making space for a more detailed analysis. One criticism is that the paper is light: the author show that a simple idea works (this is a compliment), but I would have expected to have used the remaining space for ablation studies or a discussion on where this leads.\nOne important point which I would like to see before recommending acceptance is a comparison to know if what is helping is just more data, or the summarization objective. Using lots of more data beats all those numbers (see BERTSUM paper, Liu & Lapata 2019). The comparison I am missing is training BERT on your crawled data-set, and use that for BERTSUM (the code is available). If that helps as much as the summarization pre-training then it would be disappointing but a nice result in favor of language modeling. If not, then it is a strong support for your idea. \n\nTwo other points which should at least be discussed, as it gives the impression of cherry-picking results instead: \n1/ Table 1 is recall; Table 2&3 F1. Why?\n2/ The parameters of fine-tuning of the appendix vary wildly depending on the data-set (in particular, the difference in the width of the beam search is striking). Was this optimized on test-data? What is the sensitivity of the summaries to this?\n\nI do not understand the last two sentences of Sect 4 (\"A candidate word leading...). Could you explain?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgO4kMOFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an interesting idea on how we can leverage the lead bias in summarization datasets to pretrain abstractive news summarization models on large-scale unlabelled corpus in simple and effective way. \n\nFor pre-training, they collected three years of online news articles data. Then, they take the top 3 sentences of the article as summary and the rest of the article as input document. For better choosing such article-summary pairs, they employ effective data cleaning and filtering process. Overall, they collected 21.4M articles for the pretraining. \n \nOverall, the pretrained model does decent on three summarization datasets without any fine-tuning. After fine-tuning the respective datasets, the gains seem significant. Especially on the XSum dataset, the improvements are remarkable. \n\nI believe that the idea is interesting but the experiments are incomplete and more investigation is required to make this paper stronger. Therefore I suggest to reject this paper. \n\nArguments:\n1) The important experiments that are missing in this paper are evaluating the proposed method on better human written summarization datasets -- DUC. The real world summarizations resemble more like the ones in the DUC dataset and it would be interesting to see if the transfer results of the pretrained model on the DUC datasets. The important question is to understand whether the pretrained model which took advantage of lead-bias could achieve good summaries on real summarization samples. This would also answer whether the pretrained just took advantage of the lead-bias issue of many large summarization datasets or does it really learn good summarization model. \n\n2) This paper has good idea but mainly missing ablation studies. For example, how does the proposed model do compared with GPT-2 in the fine-tuning setting, and how do these two models perform on the DUC datasets. \n\n3) During the dataset filtering/collection, a check on the quality of the filtering process by doing a small human study would have been a great addition. Also, instead of showing the output examples (which can go in the supplementary), human study comparing the quality of the pretrained model with fine-tuning and a baseline (can be from previous work) would have been better. \n\nOther minor questions\n1) “we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest” -- is there any reason on fixing to these numbers? How did you make this decision ?\n\n2) Even though the performance gains look visibly significant, I would suggest to report the statistical significance scores.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to leverage the Lead (i.e., first sentence of an article) in training a model for abstractive news summarization. 

Reviewers' initial recommendations were weak reject to weak accept, pointing out the limitations of the paper including 1) little novelty in modeling, 2) weak evaluation, and 3) lack of deep analysis. After the author rebuttal and revised paper, one of the reviewers increased the score and were leaning toward weak accept. 

However, reviewers noted that there was significant overlap with another submission, and we discussed that it would be best to accept one of the two, incorporating the contributions of both papers. Hence, I recommend that this paper not be accepted, and perhaps some of the non-overlapping contents of this paper can be included in the other, accepted paper.

Thank you for submitting this paper. I enjoyed reading it.
Decision: Reject