TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Authors: Ziyi Yang
Chenguang Zhu
Michael Zeng
Xuedong Huang
Eric Darve
Venue:  (2020)
Keywords: text summarization
unsupervised learning
natural language processing

Abstract: Text summarization aims to extract essential information from a piece of text and transform it into a concise version. Existing unsupervised abstractive summarization models use recurrent neural networks framework and ignore abundant unlabeled corpora resources. In order to address these issues, we propose TED, a transformer-based unsupervised summarization system with dataset-agnostic pretraining. We first leverage the lead bias in news articles to pretrain the model on large-scale corpora. Then, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are abstractive and containing even higher proportions of novel tokens than those from supervised models.

Under review as a conference paper at ICLR 2020
TED: A PRETRAINED UNSUPERVISED SUMMARIZA- TION MODEL WITH THEME MODELING AND DENOIS- ING
Anonymous authors Paper under double-biee blind review
ABSTRACT
Text summarization aims to extract essential information from a piece of text and transform it into a concise version. Existing unsupervised abstractive summa- rization models use recurrent neural networks framework and ignore abundant unlabeled corpora resources. order to address these issues, we propose TED, a transformer-based unsupervised summarization system with pretraining on large- scale data. We first leverage the lead bias in news articles to pretrain the model large-scale corpora. Then, finetune TED target domains through theme modeling and a denoising autoencoder to enhance the quality of summaries. No- tably, TED outperforms all unsupervised abstractive baselines NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are abstractive and containing even higher proportions of novel tokens than those from supervised models.
INTRODUCTION
Summarization refers to the task of condensing document into shorter version. Summarization models can be categorized into two classes: abstractive and extractive. Extractive models select sen- tences from the input article as the summary. Such process ensures a basic level of grammaticality and accuracy, but also limit the model to copying. contrast, abstractive models summarize doc- uments using tokens and phrases that may not be found in the input article, process requiring an advanced ability to refine, paraphrase and re-organize information (See al, 2017 Narayan al,) 2018).
Like most machine learning algorithms, summarization models can also be divided into supervised and unsupervised categories. Supervised approaches require in-domain parallel data, i.e. both input articles and corresponding reference summaries must be present for training (Hermann et al.) 2015] Liu & Lapata 2019). Unfortunately, high-quality paired data not always available across differ- ent text domains and styles. Moreover, considering the fact that summarization not an easy task for people, reliable human-labeled data also difficult obtain. Therefore, several unsuper- vised summarization approaches have been proposed, which do not require reference summaries for the target domain. We introduce these methods as follows.
Unsupervised extractive models. TextRank (Mihalcea & Tarau, 2004) encodes sentences in the article as nodes in an undirected graph. The weights of edges measured by sentences similar- ity. The centrality of a node (sentence) is computed by PageRank (Brin & Page 1998) to decide whether sentence should be included in the final summary. Zheng Lapata (2019) 9 advances upon
TextRank by using BERT (Devlin et al. 2018) to compute sentence similarity and build graphs with directed edges decided by the relative positions sentences.
Unsupervised abstractive models. Baziotis et al. (2019) leverages differentiable sampling and op- timizes by re-constructing the input article from the generated summary. Chu Liu (2018) proposes similar idea in the multi-document summarization setting. Wang & Lee (2018) uses adversarial training and reinforcement learning to make the summary human-readable. Févry & Phang (2018) adopts denoising autoencoders originally used in sentence compression. However, most of these models are only tested on datasets with considerably small article/summary length. Also, previ- models usually utilize the recurrent neural networks (RNNs). However, transformers (Vaswani

---

Under review as a conference paper at ICLR 2020
et al. 2017 Devlin al. 2018) have shown superior performances over RNNs on various NLP tasks, including machine translation, reading comprehension, sentiment analysis, etc.
In this paper, we present TED, unsupervised abstractive summarization model with theme mod- cling and denoising that uses a transformer based encoder-decoder structure and the pretraining leverages large scale unlabeled corpora. Our main contributions two-fold follows.
First, we leverage the lead bias in news articles for model pretraining. The lead bias is introduced by the journalistic convention of writing using an inverted pyramid structure, placing the most important information in the beginning of an article. We propose to use the leading sentences as the target summary and train the model predict it during pretraining. In this way, unlabeled corpora. Without any finetuing, the model pretrained in this way on 21.4M news articles can yield better performance than most existing unsupervised methods.
utilize large-scale
Second, to finetune on specific datasets, TED is further trained with theme modeling loss and denoising autoencoder. The role of the theme modeling module is to make the generated summary semantically close to the article. The module uses semantic classifier trained using a discrimina- tive objective function. Furthermore, to optimize on the generated summary tokens, we adopt the Gumbel-Softmax (Jang et al. 2016) estimator to replace the non-differentiable arg max. The de- noising autoencoder has been previously used unsupervised machine translation (Lample et al.] 2017) and sentence compression (Févry & Phang 2018), and we employ it to help the model extract salient information from corrupted text.
Also, instead of classical word tokenization, we adopt the SentencePiece tokenization (Kudo & Richardson, 2018) to alleviates the -ta-ditan voc-ol-vocabulary (OOV) problem in language generation tasks (Luong al. 2014 [Sennrich et al. 2013).
We test TED on several benchmark datasets. The experimental results show that TED outperform all unsupervised abstractive baselines all datasets. For example, the CNN/DM dataset, it outperforms the state-of-the-art unsupervised abstractive model by more than 9 ROUGE- points and compares favorably with most unsupervised extractive models. We further show that TED is capable of generating novel words and phrases in the summaries, and is a highly abstractive system even compared with supervised systems.
2 METHODOLOGY
In this section, we will go through the model structure TED, i.e. the transformer encoder and de- coder, theme modelling and the denoising autoencoder. The overall architecture of TED is illustrated in Fig.
2.1 TRANSFORMER ENCODER AND DECODER
Previous unsupervised summarization methods are based on the sequence to sequence (seq2seq) model (Sutskever et al. 2014) that primarily uses the RNN model. As the transformer structure (Vaswani et al.) 2017) has been successfully used a large number of NLP tasks, our model employs the multi-layer transformer encoder-decoder architecture. We follow the standard transformer design in our network and refer readers to Vaaswani et al. (2017) for more technical details. Denote the number of layers (i.e., Transformer blocks) T., the number of self-attention heads H and the hidden size as N. We explore two different configurations in experiments, 4 layers 4 heads (4L4H) with N 512 and 10 layers heads (10L8H) with N 720.
and each token is transferred to a vector by trainable embeddings matrix V. The output from transformer encoder is a sequence of encoded vectors E(X) {uf u ..., u The decoder can be viewed as a conditional language model to generate the summary. Given input summary tokens W = {wi, W2, ..., w the cross attention layer in the decoder D attends with encoder outputs {u," pm The decoder outputs are D([w) W2, ... ) = {up, ub ..... u The probability distribution over the vocabulary for Wk+1 | given by:
Denote the input article token sequence as X = {x T2,, ..).
P(wK+1 1li: Fim)) softmax( (
2

---

Under review as a conference paper at ICLR 2020
In our model, the text are not tokenized by spaces but by the SentencePiece (Kudo & Richardson 2018) model, in order to address the challenging vocaaullocaa (OOV) words issue. Efforts have been made to address this issue at the cost of losing semantic information, such as mapping OOV words to a special "UNK" token. To mitigate the open vocabulary problem, we adopt SentencePiece, data-driven method that trains tokenization models from sentences in large-scale corpora. The advantage of the SentencePiece model is that its subwords can cover all possible word forms and the subword vocabulary size is controllable. our experiments, we train SentencePiece subword vocabulary of size 32,000.
Note for supervised summarization models, the inputs to the decoder are the groundrruhs/reference summary tokens; for unsupervised learning, input tokens are generated in the previous pass. More details are available in section 23.11
2.2 PRETRAINING WITH UNLABELED CORPORA
Leveraging large scale unlabeled text corpora to pretrain models has been proven as an effective method in multiple NLP tasks (Devlin et al. 2018). However, such approach has not yet been utilized in text summarization.
News articles follow inverted pyramid structure, i.c. front loading the most salient information. This so-called "lead bias" for news summarization is so strong that See et al. (2017) have shown that using the first 3 sentences in deep learning models. Although this poses great challenge to previous research, we leverage this property in favor in the pretraining phase.
article summary can score higher than many sophisticated
For news article, we set the target summary to be the first three sentences. This allows the model exploit the structural bias of the news domain and infer the most important information using the background materials in the remainder of the article. For pretraining, we obtain three years of online news articles from 2016 to 2019 via an industrial search engine. The search engine indexes major online news domain, for instance, New York Times and Bloomberg. Then we collect the parsed articles within the 2016-2019 time range as the raw data. Note that this time span does not overlap any of three test datasets we use in this paper, therefore the pretraining should not lead to data leakage in test.
Next we conduct data cleaning to remove irrelevant distracting content and filter out articles whose top three sentences do not form a good summary. First, many news articles begin with media names, reporter names, dates or other irrelevant information for summarization, e.g. "New York (CNN) "Adam Smith, June 3rd 2018:". We automatically clean these using regular expressions. Second, we only include articles whose top three sentences contain between 10 and 150 words, and remaining sentences contain between 150 and 1,200 words. Third, we try to remove articles for which the first three sentences may contain the major information in the article. We compute metric: overlapping words. We compute the portion of non-stopping words in the top three sentences that also appear in the rest of an article. A higher ratio indicates that the rest of the article is likely to elaborate on the beginning part. We keep those articles with this ratio of overlapping words higher than 0.65.
simple and easy-to-
Finally, we end up with 21.4M articles. We randomly sample 12,000 from the data for validation. We conduct pretraining for 10 epochs and pick the model with the best ROUGE-L score on the validation set. The pretraining idea is also used in (Anonymous) 2020). After pretraining, we finetune TED on target datasets in an unsupervised manner. This includes two modules: theme modeling and denoising autoencoder.
2.3 THEME MODELING
Theme modeling aims to make the generated summary semantically close to the input article. We employ differential sampling to enable optimization on generated summaries and train a classifier to improve the semantic relatedness between the summary and article.
3

---

Under review as a conference paper at ICLR 2020
theme loss
article
Pretrained Transformer Encoders/Decoders
generated summary
Transformer
Encoders/Decoders
pretrain
eoossyytropy
add nolse
loss
noisy
denoised sent.
sent.
Figure I:
Overall structure of our model. TED first pretrains on news articles and then finetunes with theme modeling and denoising. (from left to right).
2.3.1 DIFFERENTIABLE SAMPLING
In order to optimize the network on output summaries, we need to make the generation of summary tokens differentiable. Recall the probability distribution of token Wk+1 is P(wx+1 (1.) softmax(V ( Let π denote P(wx+1|uu W1::- X1sn). One can use arg max on π to obtain the to- ken Wk+11 in the forward pass, however,
not differentiable in the gradient pack-propagation.
Although one can get around by obtaining the embedding of Wk+11 as a weighted sum of the vocabu- lary embeddings V. this results in an undesirable gap between the forward pass in training (weighted sum) and inference (discrete sampling). ) To solve this issue, we employ the straight- thhruuh Gumbel- Softmax estimator (Jang et al. 2016) as in Yang et al. (2018); Baziotis et al. (2019). Specifically, the forward pass in training still uses arg max sampling, but for gradient computation, the following Gumbel-Softman distribution is used as a differentiable approximation for the arg max operation:
exp(log(m:) m: + 9:)T))
(2)
exp(log(TF)) (T)) + 9;)/T)
where 91,'. 9k are i.i.d samples drawn from the Gumbel distribution G(O, 1) and T denotes the -> 0, the Caumbel-Softnat distribution
softmax temperature. As shown in Jang al. (2016), converges to the categorical (one-hot) distribution; as r + inf, the Gumbel-Softmax distribution converges the uniform distribution. Although this gradient estimator is biased, we find that this method works well in practice. We choose = 0.1 based on the CNN/DM validation set and use this value in all the experiments. Denote the input article as d, the generated summary as s= fwi, W2, .... }. The generation of follows the recursive process that input W1:k to the transformer decoder obtain Wk++1, then imput w1s++1 compute Wk+42 and on. The first input token wi is always the special beginning token [START].
2.3.2 ENCODER TRANSFORMER AS A SEMANTIC CLASSIFIER
We frame the semantic similarity problem in a discriminative setting. As the generated summary may be off the article theme the beginning of training, add sentence pairs from the article to facilitate similarity computation.
Concretely, during training, pick two consecutive sequences of tokens a₁ and a2 from an article to form a positive sequence pair (a), )2). Second, sequence by is chosen from a random article from the dataset form the negative sequence pair {a,, b, Following Devlin et al. (2018), each sequence pair is packed into one single sequence by inserting a special token [SEP] between them and adding trainable segment embeddings. A special classification token [CLS] is also added to the beginning. As shown in Fig. 2 the packed sequence is fed as input into TED's 's encoder. The output vector associated with the token [CLS] is then elassified into similar/distinet categories by two-layer fully connected network. We use the following cross-entropy loss to optimize the encoder. Note that the theme modeling loss does not involve the transformer decoder.
Ltheme log(p(y 1/a1, az)) log(p(y 1/s, d)) log(p(y 00a,, bi))
4

---

Under review as a conference paper at ICLR 2020
similar/distinee /distinct
classify
[CLS]
Add & Normalize
Feed Forward
Add Normalize
Self Celf-Attention
[CL5]
my
dog
is
cuite
[SEP]
hee
likes
play
ing
(CLS)
dog
work
Theme modeling is essentially a semantic classifier. The input sentence pair is first pro- cessed by adding "class" token in the beginning and "separation" token between. Then the sentence pair is fed into the transformer encoder, and then linear classifier.
Figure 2:
2.4 DENOISING AUTOENCODER
The idea of denoising autoencoder (Vincent et al. 2008) has been used in unsupervised machine translation (Artetxe al.) 2017 Lample al. 2017) to prevent the model learning to merely copy every input word one by one. This denoising process imitates text simplification and helps to refine essential semantic information.
In detail, sequence of consecutive tokens æ from the input article is injected with two types of noise. First, we insert noisy tokens sampled from other articles in the same dataset into the original sequence at random positions, obtaining a new sequence with length n', where n' is 40%-50% larger than n. Next, similar to Lample et (2017), the sequence is slightly shuffled by applying permutation such that Vi E 1,2, ,n],, o (i)
S k, where the permutation distance k is set to be 20% of the length of æ. The final corrupted sequence is denoted as x'.
The TED model is trained to recover the original token sequence given the corrupted sequence:
Ldenoise CE(a TED(x)))
(4)
where CE denotes the mean of token-level cross-entropy loss. TED(x') denotes the sequence of probability distribution outputs (T) from the decoder with inputing x' the encoder and æ to the decoder for teacher forcing.
The final objective function is the mean of Eq. 3 and Eq.
Ltheme + Ldencise
TED
(5)
2
EXPERIMENTS
3.1 IMPLEMENTATION DETAILS
For pretraining, we use dropout rate 0.3 for all inputs to transformer layers. We use RAdam (Liu al. 2019) the optimizer, with a learning rate of 10-4, Also, due to the different numerical scales of the positional embedding and initialized sentence piece embeddings, we divide the positional embedding by 100 before feeding into the transformer.
For unsupervised finetuning on specific datasets, the learning rate is set to 2 x 10-4 and dropout ratio stays the same as in pretraining. The batch size is 16, and the vocabulary embeddings also updated in the training process. During test, we generate the summarization from trained encoder and decoder by beam search.

---

Under review as a conference paper at ICLR 2020
avg. document words
avg. summary
Dataset
# does
sen.
words
sen.
CNN/DM NYT English Gigaword
11,490 4,375
641.9
28.0
54.6
3.9
1,290.5
50.7
79.8
3.5
1,937
29
8
Table 1: Average document and summary length in number of words and sentences on NYT, CNN/DM, and English Gigaword datasets (test set).
3.2 RESULTS
We evaluate our model on three summarization datasets: NYT, CNN/DM and English Gigaword. The text statistics these datasets are summarized in Table 1 Numbers of NYT and CNN/DM are collected from Zheng & Lapata (2019).
We compare TED with the following baselines: Brief WWag & Lee, 2018). , SEQ (Baziotis et al. 2019). GPT-2 [Radford et al. 2019). TextRank (Mihalcea & Tarau 2004). PACSUM (Zheng & Lapata 2019), PGNet (See et al. 2017) and REFRESH (Narayan et al. (2018). These models cover both unsupervised and supervised categories, and include abstractive and extractive methods.
We measure the quality of generated summaries ROUGE F1 score (Lin 2004), including unigram (ROUGE-1), bigram (ROUGE-2) and longest common subsequence (ROUGE-L).
The NYT dataset (Durrett et al. 2016) contains 110,540 news articles with 100,834/9,700 train/test split. Following Liu & Lapata 2019), we choose 4,000 examples as the validation set and filter out examples with summaries of fewer than 50 words. As demonstrated by the results in table 2 the unsupervised fine-tuning of TED improves upon the pretrained model by 2.75%/..06%//... .06%/2237%% on respectively. Note that ROUGE metric prefers extractive systems that preserve original phrasing (See et al. (017).. Considering this factor, TED achieves results that are competitive with unsupervised extractive baselines and surpasses all unsupervised abstractive models.
The CNN/DM dataset (Hermann et al. 2015) composed of articles from CNN and Daily Mail, and uses associated highlights reference summaries. We use the same training, validation and test split (287,227/13,5)
1,490) as other baselines. Similar See et (217)) and[Liu Lapata (219)), input articles are truncated to 500 tokens. Results are shown in Table 2 TED with larger model size (10L8H) outperforms all unsupervised abstractive methods and compares favorably with unsupervised extractive baselines. Note that TED outperforms GTP-2, powerful transformer-bases language generation model pretrained on large scale webpage textual data, by siginificant margins. Again, TED further improves upon pretrained models on both 10L8H and 4L4H configurations.
For the English Gigaword sentence compression dataset, the input text is the first sentence of article, and the reference summary is the article's headline. The size of train/val/test is 3.8M/189K/1, 1,997 respectively. after filtering out data examples with articles containing only "UNK" tokens. As shown in Table 3 TED outperforms all the unsupervised baselines.
DISCUSSION
4.1
ABLATION STUDY
The ablation studies shown in Table f verify the effectiveness of each module in TED. Training the transformer encoder-decoder from scratch yields reasonable performance. Pretraining large- scale data results in more than 10% improvement on all three metrics on training TED from scratch. Pretraining plus either theme modeling denoising improves upon the pretrained model by more than 2%. The full TED model, pretraining with theme modeling and denoising, produces the best result overall.

---

Under review as a conference paper at ICLR 2020
CNN/DM
NYT
Model
R1
R2
RL
R1
R2
RL
Unsupervised Abstractive
TED 10L8H (ours)
38.73
16.84
35,40
37.78
17.63
34.33
Pretrained 10L8H (ours)
38.38
16.49
35.08
35.03
16.57
31.96
TED 4L4H (ours)
34.38
9.56
30.10
Pretrained 4L4H (ours)
31.20
10.05
27.80
SEQ
23.24
7.10
22.15
17.85
3.94
19.53
Brief
28.11
9.97
25.41
GPT-2
29.34
8.27
26.58
Unsupervised Extructive
LEAD-3
40.50
17.70
36.70
35.50
17.20
32.00
TextRank + tf-idf
33.20
11.80
29.60
33.20
13.10
29.00
TextRank + skip-thought
31.40
10.20
28.20
30.10
9.60
26.10
TextRank + BERT
30.80
9.60
27.40
29.70
9.00
25.30
PACSUM tf-idf
39.20
16.30
35.30
40.40
20.60
36.40
PACSUM + skip-thought
38.60
16.10
34.90
38.30
18.80
34.50
PACSUM BERT
40.70
17.80
36.90
41.40
21.70
37.50
Supervised Abstractive
Extractive
SUMO
41.00
18.40
37.20
42.30
22.70
38.60
PGNet
39.50
17.30
36.40
42.70
22.10
38.00
REFRESH
41.30
18.40
37.50
41.30
22.00
37.80
Table 2: ROUGE Fj scores on NYT and CNN/DM datasets.
R1/R2/RL stands for ROUGE- I/ROUGE PROUGE-2//OUGE/ respectively. Best results in each unsupervised category is in bold. Results of other models are obtained from original papers running open-sourced software.
Model
R1
R2
RL
TED 10L8H (ours)
25.58
8.94
22.83
Pretrained 10L8H (ours)
25.23
8.84
22.56
TED 4L4H (ours)
24.59
8.10
21.91
Pretrained 4L4H (ours)
22.52
7.46
20.09
LEAD-8
21.86
7.66
20.45
SEQ
25.39
8.21
22.68
Brief
21.26
5.60
18.89
Table 3: Results on the English Gigaword dataset. Numbers are collected from original papers. The best performance is in bold.
Model
R1
R2
RL
train from scratch pretrained only
24.49
4.41
20.14
35.03
16.57
31.96
pretrained w/ theme modeling
37.16
18.18
34.15
pretrained w/ denoise loss full model
37.48
17.83
34.05
37.78
17.63
34.33
Table 4: Ablation study of different components in TED on the NYT dataset. We test with the 10LSH model configuration.

---

Under review as a conference paper at ICLR 2020
PG Net
LDD
Reference
Progettiols roud begrame
50
60
1-grams
2-grams
3-grams
4-grams
Figure 3: Proportion novel grams in summaries on the CNN/DM test set. We compare three systems, TED, PGNet and reference summaries. Numbers of PGNet are computed from its publicly released output.
4.2 MODEL ANALYSIS
Novel N-grams. examine how abstractive TED is, we compute the proportion of novel N-grams in the summary output (Fig. 3) The reference summary and the output from PGNet included for comparison. Although TED is unsupervised, includes more novel grams than the supervised model PGNet. The reference summaries have the highest proportion of n-grams.
Example. We showcase a sample summary from CNN/DM dataset along with the input article and the reference summary (Fig- 4). As shown, TED is able to capture and organize the essential information into fluent language. We attribute the grammatical correctness to the pretraining process and the denoising autoencoder. However, we also note that although TED manages to recognize the temporal information related to reported event (a few hours after Fox news reports), it makes mistake by summarizing as "a few hours after a report about roberts' research was released... shows that fact cross-checking is potential future research direction.
Article
after exposing potential security risks with airlines' in-flight entertainment systems, one of the top experts on counter-threat intelligence in the world was pulled off flight by fbi agents. chris roberts, who featured string of fox reports, was yanked off his plane after it landed in syracuse, new york, wednesday night by two fbi agents and two uniformed officers. roberts, works security intelligence company one world labs, was questioned for the next four hours
TED Summary
chris roberts, who works for security intelligence company one world labs, was pulled off plane in syracuse, new york, on wednesday night by two fhi agents and two uniformed officers. the incident occurred only few hours after a report about roberts' research was released by the government accountability office earlier this week.
Reference
chris roberts of one world labs grabbed after plane landed in syracuse. two fbi agents spent four hours questioning him about cyberhacking, agents confiscated electronic devices and computer files from roberts. he fiew in give talk acrospace conference about plane vulnerabilities. roberts featured fox news' the record with greta susteren. regarded of the world's top experts on counter-threat intelligence.""
Figure 4: An example of a generated summary by TED. The reference summary and parts of the input article are also included.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this paper, we propose TED, an unsupervised abstractive text summarization model. First, we introduce an effective and powerful pretraining approach leveraging the lead bias news articles. We then develop finetuning scheme to induce the semantic similarity between summaries and input articles, together with a denoising autoencoder. Experiments across three datasets show that TED outperforms unsupervised abstractive baselines. For future work, we would like to encode the criteria of relevance, informativeness and importance proposed Peyrard (2019) into TED. Fact cross checking is another interesting direction mentioned in the section 4.2.
REFERENCES
Anonymous. Make lead bias in your favor: simple and effective method for news summarization. In Submitted to International Conference on Learning Representations, 2020. URL https: under review.
net. www.://oommdddyyyaaataeea
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. arXiv preprint 17X.v 2017.
Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. Seq 3: Dif- ferentiable sequence-tor grguence-to-sqqueneee sequence-to--sequeeeee autoencoder for unsupervised abstractive sentence compression. arXiv preprint arXiv: 2019.
Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. In COMPUTER NETWORKS AND ISDN SYSTEMS, pp. 107-117. Elsevier Science Publishers B. V., 1998.
Eric Chu and Peter Liu. Meansum: A neural model for unsupervised multi-document abstractive summarization. arXiv preprint arXiv: 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. Learning-base single-document summa- rization with compression and anaphoricity constraints. arXiv preprint arXiv: 2016.
Thibault Févry and Jason Phang. Unsupervised Sentence Compression using Denoising Auto- Encoders. arXiv e-prints, art. arXiv:1809.00 02069,,, Sep 2018.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv
preprint arXiv:161 1.0| 144, 2016.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprini arXiv: 100.1808...2202 2018.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.0004 2017.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:190s 03265,3685,0 2019.
Yang Liu and Mirella Lapata. Text Summarization with Pretrained Encoders. arXiv e-prints, art. arXiv: 1908.0000 08345, Aug 2019.

---

Under review as a conference paper at ICLR 2020
Kinh-TTang Luong, llya Sutskever, Quoe V Le, Oriol Vinyals, and Wojeiech Zaremba. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206, 2014.
Rada Mihalcea and Paul Tarau. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pp. 404-411, 2004.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Ranking sentences for extractive summariza- tion with reinforcement learning. arXiv preprint arXiv: /0X.v08000006630 2018.
Maxime Peyrard. A simple theoretical model of importance for summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1059-1073, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 18)), 2019.
Abigail See, Peter J Liu, and Christopher D Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- lional Linguistics (Volume 1: Long Pupers), pp. 1073-1083, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.000 07909, 2015.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL
attention is all you-need.e...
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103. ACM, 2008
Yau-Shian Wang and Hung- Yi Lee. Learning to encode text as human-readable summaries using generative adversarial networks. arXiv preprint arXiv:1810.02851 2018.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg- Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, pp. 7287-7298, 2018.
Hao Zheng and Mirella Lapata. Sentence centrality revisited for unsupervised summarization. arXiv preprint arXiv:/9066 03508, 2019.
10


Reviews:
[{"review_id": "rJxaA5iecS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to improve abstractive summarization models by using pretrained embeddings, theme modeling and denoising.\n\nThey propose a very interesting idea: to leverage the lead bias in news article to build supervized summarization task from 21.4 M of articles. Details are given how to produce this supervized data using simple heuristics.\n\nThe  model is  train with a denoising loss, by introducing 2 types of noise (tokens from other article and sequence shuffle). Theme modeling is also introduced as a classification problem  (same as BERT) :  the system must learn to classify pairs of sentences from the same article and pairs from different articles. \n\nExperiments are conducted on 3 datasets. The proposed model outperforms the other unsupervized abstractive models and provides results closed to unsupervized extractive models, with a metrics which favors extractive models. Ablation study shows that pretraining yields most of the impact, whereas improvements due to theme modeling and denoising loss are marginal. \n\nIn the Article example : \n\"in the wold\"  ?\n\nConclusion : \n- dataset-agnostic : I don't see why since the approach take advantage of the lead bias.\n- \"outperforms previous systems by significant margins\" : excessive. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hke_3SSAYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper's Claims\n\nThe paper introduces a new unsupervised abstractive summarization approach called TED, using a Transformer encoder and decoder. Their main contributions are as follows:\n1) Pretraining the encoder and decoder on news articles using the first beginning as the target summary.\n2) Fine-tune on other datasets using so-called theme modeling, and separately a denoising loss.\n3) TED's performance is claimed to significantly improve over GPT-2 while not being too far from the best unsupervised extractive summarization results.\n\nDecision\n\nEdit: After revisions and discussions, I recommend we accept this paper.\n\nI am leaning towards accepting this paper mostly because of the contribution #1 above. Unsupervised learning using large quantities of text that have the property of being typically written in a style that synthesizes information in the first 1-3 sentences is a powerful idea. That the performance is improved compared to other unsupervised abstractive summarization confirms the importance of this approach.\n\nHowever the importance of and justification for the fine-tuning steps are comparatively much more limited in my opinion. Also, some important details about the preprocessing for pre-training appear to be missing and they could be quite important. \n\nDetailed arguments for decision\n\nI view this effort as aiming to reproduce the BERT approach in the context of abstractive summarization, which is a good idea. The most clever contribution is in leveraging un-labeled text using the first few sentences as the target summary for pretraining. The results of just this part are already beating previous approaches, while not requiring any in-domain data, which is quite powerful. \n\nHowever, some relatively important details regarding the methodology are omitted or only glossed over and it would greatly contribute to making this work more reproducible if the details were included (see my detailed notes below, notably regarding section 2.2). \n\nOn the fine-tuning steps, I have several worries. First, why not fine-tune using supervised learning, as would be the analog to the BERT approach? Instead the authors go out of their way to do in-domain unsupervised learning, which provides a boost, yes, but still doesn't compare positively to extractive and/or supervised methods. Second, why not perform the theme modeling and denoising also -- or rather only -- on the unlabelled pretraining data? Why should it be done on the in-domain fine-tuning data instead (while not using the most valuable piece of in-domain information, namely the example summaries)? After all, it's a fully unsupervised approach and it can actually be performed on any text at all, whether a summary for it exists or not.\n\nAgain regarding the unsupervised approach, and to push the BERT analogy further, I'm wondering why not initialize the pretraining model with a BERT-style trained model? After all we could imagine building a system that adds more and more in-domain characteristics sequentially: first pretrain a BERT model, then fine-tune to summarization using what this paper calls pretraining, and then finally fine-tune again to a specific summarization domain. \n\nSo, to conclude, I find that this paper goes in the right direction and introduces important ideas for pretraining and fine tuning unsupervised abstractive summarization models, but that some decisions about how to use the various ideas (theme and denoising but no supervised learning, in-domain vs during pretraining) have not been explored enough.\n\nExtra notes\n\npage 2, second line: pretrainleverages (typo)\nsection 2.1: fix first sentence to make it an actual sentence.\nsection 2.2: \"we obtain three years of online new articles ... via a search engine\" please be more specific about your methodology.\nsection 2.2: You should double check more throughly that there is no data leakage in test. There could be articles about the same exact events, years apart, for example. I doubt that this would be a big effect, but there are easily ways to find highly similar articles between the pretraining data and test data to make sure.\nsection 2.2: \"Next we conduct following data cleaning\" fix (typo?). Also that sentence probably belongs to the next paragraph.\nsection 2.2: Why did you pick the values that you did for the preprocessing heuristics (such as between 10-150 words, 150-1200 words, 3 sentences and not 2 or 1 or 4, the ratio 0.65, etc.)? Were other values tried?\nsection 2.2: You mention you end up with 21.4M articles. How many were there to start with? What's the filtering ratio?\nsection 2.2: You mention that you pick the model with the best ROUGE-L score on the validation set. How many models were there? What was different between them?\nsection 2.2, OOV Problem: the information in this whole subsection would fit better in 2.1 where 'tokens' are left generic without specifying which type of token you're considering.\nFigure 1: I find the upper part of this figure very confusing. Why are there arrows going from the encoder/decoder to a summary, to theme loss, to article and back to encoder/decoder? It's important that the summary is never seen by the theme loss otherwise it's not unsupervised anymore, and I also don't see why the arrow would go through article *after* theme loss. I assume there must have been a mistake, please fix.\nsection 2.4: \"the sequence is slightly shuffled by applying a permutation /sigma such that ...\" The formula given here tells me that all token indices are shuffled with another token within a window k. That seems like a lot of moving around, and also depending on the implementation a token from the beginning could possibly end up at the very tail of the sentence by being picked iteratively again and again, thus falling outside the permutation distance k. Please provide more details on how this is done and a justification for why it was decided to do it this way.\nSection 3.1: I'd like to know how long (preferably number of words, or at least number of wordpiece tokens) the summaries generated are. What determines how long they are, is it a fixed size, or the model decides to stop on his own (or when hitting some limit), or something else?\nsection 4.2: Do you have any idea why your unsupervised approach yields more novel n-grams than a the supervised model you compare against? This can be good as much as it can be bad, in that it could be going off-track. Yes humans have high novelty, but high novelty in itself isn't necessarily good. I don't find the argument that have more novel ngrams is intrinsically, necessarily good, compelling. If I'm wrong, then it would be nice to have better explanation in the paper.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xJ3ZoaFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POS-DISCUSSION\nI thank the authors for their answer. I updated my score assuming ryxAY34YwB does not exist, and would encourage authors to discuss in more details the relationship with MeanSum if this gets accepted\n\nPRE-DISCUSSION\n\nThis is an important contribution for the field of unsupervised summarization. \"Unsupervised *\" is trendy in NLP so this is a timely contribution. Furthermore, doing this for summarization is important because of the cost of getting gold summaries and the model used in translation is harder (impossible?) to adapt to this setting where there is information loss in one direction.\n\nHowever, I find major drawbacks in the current state of this paper. They are best related to the three contributions the author claim:\n - Contribution3: the use of BPE. \"BPE for X\", with X being an NLP task can hardly count as a contribution today. If we are counting who did it first, then this is taken at least by Liu & Lapata 2019 through their use of BERT\n - Contribution1: leveraging the lead bias for pre-training. This is a great idea! However, this seems to be covered by an accompanying paper (ICLR submission ryxAY34YwB) which is not referenced. Because of common paragraphs and experimental setting I am assuming there is an overlap of the author sets in two papers. PLEASE CORRECT IF THIS IS NOT THE CASE. As you don't get to claim the same contribution twice, this contribution should go all to the benefit of the other paper.\n - Contribution2: the use of combining reconstruction loss and theme loss for summarization is another great idea. However, the paper that introduced this for summarization (as far as I know) is not cited nor compared too (MeanSum: https://arxiv.org/abs/1810.05739). This seems like a major issue considering the similarity in the approach (including the use of the straight-through Gumbel softmax estimator).\n\nOther comments:\n\n - Being a growing topic of study, I appreciated in particular the care taken to report a number of other approaches. Could you please clarify which version of ROUGE was used in each case? There are significant differences in the different implementations being used.\n - Please also specify the version of ROUGE you used. \n - Your numbers in Table 2 do not coincide with Table 3 of ryxAY34YwB (eg: LEAD-3 for CNN/DM). Can you explain?\n - Your ablation study (Sect 4.1) focuses on CNN/DM (NOTE: the caption of Table 4 says NYT, but the number correspond to CNN/DM. I guess this is an error), where the topic & reconstruction loss indeed helps. However this is not the case for NYT, where LEAD-3 actually beats any of your approach. This is not mention nor discussed.\n - The example of Fig 4 reveals a major problem. The summary states an incorrect fact: the gov accountability had indeed released a report earlier that week; but this was NOT a few hours before the reported incident. What happened a few hours before was a report on Fox News.\n\n\nIn a summary: a good idea combining ideas of ryxAY34YwB and adapting MeanSum. However, this is in my opinion not enough material for a full paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an abstractive text summarization model that takes advantage of lead bias for pretraining on unlabeled corpora and a combination of reconstruction and theme modeling loss for finetuning. Experiments on NYT, CNN/DM, and Gigaword datasets demonstrate the benefit of the proposed approach. 

I think this is an interesting paper and the results are reasonably convincing. My only concern is regarding a parallel submission that contains a significant overlap in terms contributions, as originally pointed out by R2 (https://openreview.net/forum?id=ryxAY34YwB). All of us had an internal discussion regarding this submission and agree that if the lead bias is considered a contribution of another paper this paper is not strong enough. 

Due to space constraint and the above concern, along with the issue that the two submissions contain a significant overlap in terms of authors as well, I recommend to reject this paper.
Decision: Reject