Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose to improve abstractive summarization models by using pretrained embeddings, theme modeling and denoising.

They propose a very interesting idea: to leverage the lead bias in news article to build supervized summarization task from 21.4 M of articles. Details are given how to produce this supervized data using simple heuristics.

The  model is  train with a denoising loss, by introducing 2 types of noise (tokens from other article and sequence shuffle). Theme modeling is also introduced as a classification problem  (same as BERT) :  the system must learn to classify pairs of sentences from the same article and pairs from different articles. 

Experiments are conducted on 3 datasets. The proposed model outperforms the other unsupervized abstractive models and provides results closed to unsupervized extractive models, with a metrics which favors extractive models. Ablation study shows that pretraining yields most of the impact, whereas improvements due to theme modeling and denoising loss are marginal. 

In the Article example : 
"in the wold"  ?

Conclusion : 
- dataset-agnostic : I don't see why since the approach take advantage of the lead bias.
- "outperforms previous systems by significant margins" : excessive. 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Paper's Claims

The paper introduces a new unsupervised abstractive summarization approach called TED, using a Transformer encoder and decoder. Their main contributions are as follows:
1) Pretraining the encoder and decoder on news articles using the first beginning as the target summary.
2) Fine-tune on other datasets using so-called theme modeling, and separately a denoising loss.
3) TED's performance is claimed to significantly improve over GPT-2 while not being too far from the best unsupervised extractive summarization results.

Decision

Edit: After revisions and discussions, I recommend we accept this paper.

I am leaning towards accepting this paper mostly because of the contribution #1 above. Unsupervised learning using large quantities of text that have the property of being typically written in a style that synthesizes information in the first 1-3 sentences is a powerful idea. That the performance is improved compared to other unsupervised abstractive summarization confirms the importance of this approach.

However the importance of and justification for the fine-tuning steps are comparatively much more limited in my opinion. Also, some important details about the preprocessing for pre-training appear to be missing and they could be quite important. 

Detailed arguments for decision

I view this effort as aiming to reproduce the BERT approach in the context of abstractive summarization, which is a good idea. The most clever contribution is in leveraging un-labeled text using the first few sentences as the target summary for pretraining. The results of just this part are already beating previous approaches, while not requiring any in-domain data, which is quite powerful. 

However, some relatively important details regarding the methodology are omitted or only glossed over and it would greatly contribute to making this work more reproducible if the details were included (see my detailed notes below, notably regarding section 2.2). 

On the fine-tuning steps, I have several worries. First, why not fine-tune using supervised learning, as would be the analog to the BERT approach? Instead the authors go out of their way to do in-domain unsupervised learning, which provides a boost, yes, but still doesn't compare positively to extractive and/or supervised methods. Second, why not perform the theme modeling and denoising also -- or rather only -- on the unlabelled pretraining data? Why should it be done on the in-domain fine-tuning data instead (while not using the most valuable piece of in-domain information, namely the example summaries)? After all, it's a fully unsupervised approach and it can actually be performed on any text at all, whether a summary for it exists or not.

Again regarding the unsupervised approach, and to push the BERT analogy further, I'm wondering why not initialize the pretraining model with a BERT-style trained model? After all we could imagine building a system that adds more and more in-domain characteristics sequentially: first pretrain a BERT model, then fine-tune to summarization using what this paper calls pretraining, and then finally fine-tune again to a specific summarization domain. 

So, to conclude, I find that this paper goes in the right direction and introduces important ideas for pretraining and fine tuning unsupervised abstractive summarization models, but that some decisions about how to use the various ideas (theme and denoising but no supervised learning, in-domain vs during pretraining) have not been explored enough.

Extra notes

page 2, second line: pretrainleverages (typo)
section 2.1: fix first sentence to make it an actual sentence.
section 2.2: "we obtain three years of online new articles ... via a search engine" please be more specific about your methodology.
section 2.2: You should double check more throughly that there is no data leakage in test. There could be articles about the same exact events, years apart, for example. I doubt that this would be a big effect, but there are easily ways to find highly similar articles between the pretraining data and test data to make sure.
section 2.2: "Next we conduct following data cleaning" fix (typo?). Also that sentence probably belongs to the next paragraph.
section 2.2: Why did you pick the values that you did for the preprocessing heuristics (such as between 10-150 words, 150-1200 words, 3 sentences and not 2 or 1 or 4, the ratio 0.65, etc.)? Were other values tried?
section 2.2: You mention you end up with 21.4M articles. How many were there to start with? What's the filtering ratio?
section 2.2: You mention that you pick the model with the best ROUGE-L score on the validation set. How many models were there? What was different between them?
section 2.2, OOV Problem: the information in this whole subsection would fit better in 2.1 where 'tokens' are left generic without specifying which type of token you're considering.
Figure 1: I find the upper part of this figure very confusing. Why are there arrows going from the encoder/decoder to a summary, to theme loss, to article and back to encoder/decoder? It's important that the summary is never seen by the theme loss otherwise it's not unsupervised anymore, and I also don't see why the arrow would go through article *after* theme loss. I assume there must have been a mistake, please fix.
section 2.4: "the sequence is slightly shuffled by applying a permutation /sigma such that ..." The formula given here tells me that all token indices are shuffled with another token within a window k. That seems like a lot of moving around, and also depending on the implementation a token from the beginning could possibly end up at the very tail of the sentence by being picked iteratively again and again, thus falling outside the permutation distance k. Please provide more details on how this is done and a justification for why it was decided to do it this way.
Section 3.1: I'd like to know how long (preferably number of words, or at least number of wordpiece tokens) the summaries generated are. What determines how long they are, is it a fixed size, or the model decides to stop on his own (or when hitting some limit), or something else?
section 4.2: Do you have any idea why your unsupervised approach yields more novel n-grams than a the supervised model you compare against? This can be good as much as it can be bad, in that it could be going off-track. Yes humans have high novelty, but high novelty in itself isn't necessarily good. I don't find the argument that have more novel ngrams is intrinsically, necessarily good, compelling. If I'm wrong, then it would be nice to have better explanation in the paper.



review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: POS-DISCUSSION
I thank the authors for their answer. I updated my score assuming ryxAY34YwB does not exist, and would encourage authors to discuss in more details the relationship with MeanSum if this gets accepted

PRE-DISCUSSION

This is an important contribution for the field of unsupervised summarization. "Unsupervised *" is trendy in NLP so this is a timely contribution. Furthermore, doing this for summarization is important because of the cost of getting gold summaries and the model used in translation is harder (impossible?) to adapt to this setting where there is information loss in one direction.

However, I find major drawbacks in the current state of this paper. They are best related to the three contributions the author claim:
 - Contribution3: the use of BPE. "BPE for X", with X being an NLP task can hardly count as a contribution today. If we are counting who did it first, then this is taken at least by Liu & Lapata 2019 through their use of BERT
 - Contribution1: leveraging the lead bias for pre-training. This is a great idea! However, this seems to be covered by an accompanying paper (ICLR submission ryxAY34YwB) which is not referenced. Because of common paragraphs and experimental setting I am assuming there is an overlap of the author sets in two papers. PLEASE CORRECT IF THIS IS NOT THE CASE. As you don't get to claim the same contribution twice, this contribution should go all to the benefit of the other paper.
 - Contribution2: the use of combining reconstruction loss and theme loss for summarization is another great idea. However, the paper that introduced this for summarization (as far as I know) is not cited nor compared too (MeanSum: https://arxiv.org/abs/1810.05739). This seems like a major issue considering the similarity in the approach (including the use of the straight-through Gumbel softmax estimator).

Other comments:

 - Being a growing topic of study, I appreciated in particular the care taken to report a number of other approaches. Could you please clarify which version of ROUGE was used in each case? There are significant differences in the different implementations being used.
 - Please also specify the version of ROUGE you used. 
 - Your numbers in Table 2 do not coincide with Table 3 of ryxAY34YwB (eg: LEAD-3 for CNN/DM). Can you explain?
 - Your ablation study (Sect 4.1) focuses on CNN/DM (NOTE: the caption of Table 4 says NYT, but the number correspond to CNN/DM. I guess this is an error), where the topic & reconstruction loss indeed helps. However this is not the case for NYT, where LEAD-3 actually beats any of your approach. This is not mention nor discussed.
 - The example of Fig 4 reveals a major problem. The summary states an incorrect fact: the gov accountability had indeed released a report earlier that week; but this was NOT a few hours before the reported incident. What happened a few hours before was a report on Fox News.


In a summary: a good idea combining ideas of ryxAY34YwB and adapting MeanSum. However, this is in my opinion not enough material for a full paper.
review_title: Official Blind Review #2

Meta-review: This paper proposes an abstractive text summarization model that takes advantage of lead bias for pretraining on unlabeled corpora and a combination of reconstruction and theme modeling loss for finetuning. Experiments on NYT, CNN/DM, and Gigaword datasets demonstrate the benefit of the proposed approach. 

I think this is an interesting paper and the results are reasonably convincing. My only concern is regarding a parallel submission that contains a significant overlap in terms contributions, as originally pointed out by R2 (https://openreview.net/forum?id=ryxAY34YwB). All of us had an internal discussion regarding this submission and agree that if the lead bias is considered a contribution of another paper this paper is not strong enough. 

Due to space constraint and the above concern, along with the issue that the two submissions contain a significant overlap in terms of authors as well, I recommend to reject this paper.
Decision: Reject