Title: Classify or Select: Neural Architectures for Extractive Document Summarization
Authors: Ramesh Nallapati
Bowen Zhou and Mingbo Ma
Venue: ICLR 2017
Keywords: Natural language processing
Supervised Learning
Applications
Deep learning

Abstract: We present two novel and contrasting Recurrent Neural Network (RNN) based architectures for extractive summarization of documents. The Classifier based architecture sequentially accepts or rejects each sentence in the original document order for its membership in the summary. The Selector architecture, on the other hand, is free to pick one sentence at a time in any arbitrary order to generate the extractive summary. 

Our models under both architectures jointly capture the notions of salience and redundancy of sentences. In addition, these models have the advantage of being very interpretable, since they allow visualization of their predictions broken up by abstract features such as information content, salience and redundancy. 

We show that our models reach or outperform state-of-the-art supervised models on two different corpora. We also recommend the conditions under which one architecture is superior to the other based on experimental evidence.

TL;DR: This paper presents two different neural architectures for extractive document summarization whose predictions are very interpretable, and show that they reach or outperform state-of-the-art supervised models.

Review discussion:
Based on the experiments, looks like the classifier architecture is better than the selector architecture in every case.
So, is there any advantage of using the selector architecture?
Dear AnonReviewer3,
It is true that the Classifier architecture is better than the Selector architecture on real-world single-document summarization datasets. Our hypothesis is that the Classifier exploits the document structure (such as discourse, and story development patterns used by journalists in news stories) in its sequential decision making, but may not offer advantages in other settings where such structure is less apparent. To validate this hypothesis, we randomly shuffled all the sentences in each document at training time, and trained both architectures on the shuffled data. As reported in Table 3 and discussed in section 5 under the heading "Impact of Document Structure", we do see a role reversal as predicted by our hypothesis, where the Selector indeed outperforms the Classifier on shuffled data.
We argued that this simulation experiment offers preliminary evidence that Selector can be superior to Classifier in real-world scenarios such as multi-document summarization (where there is no logical ordering of sentences across document boundaries) and summarization of Tweets on a given topic (where each Tweet is usually independent of other Tweets).
Although beyond the scope of this paper, we do intend to run the Selector architecture on the problems mentioned above as part of our future work.
Thanks
-Ramesh
(1). In your ablation experiments, position features seem to be quite important for the selector architecture. This makes sense since important sentences often appear earlier in the documents. Did you observe whether the classifier architecture tends to pick sentences that appear earlier in the documents more due to its sequential nature compared to the selector architecture, since removing these features does not seem to hurt the classifier architecture as much?
(2). Do you have any intuitions on how well these models would perform for other related tasks such as sentence compression?
Thanks!
Dear AnonReviewer2,
(1) This is a very perceptive observation, and you are absolutely spot-on in terms of your conclusion. I just ran some analysis on the output of the Classifier and Selector, and what I found is that the Classifier picks the first sentence of the document as its top prediction for summary 38% of the time, while the Selector picks it only 16% of the time.
Also, the first two sentences of the document are found in the Classifier's top 5 predictions 80% of the time, whereas the Selector includes them in the top 5 predictions only 65% of the time.
Therefore, we do have some evidence that the Classifier is learning the pattern that the beginning sentences are important for summary simply by virtue of its architecture, and independent of the positional features. This hypothesis is also vindicated by the simulation experiment we ran in Section 5 where random shuffling of sentences in the training data makes the Classifier degrade much more than the Selector, since the former's architecture is no longer applicable to the input data.
(2) Sentence compression is definitely a related task, but it is not immediately obvious how our models could apply to this task since they only do extraction at sentence level. Perhaps you are referring to extracting words in a sentence in an analogous manner to extracting sentences in a document? If yes, this is a very interesting idea, but one limitation could be that our compression will only consist of deletion and re-ordering of words in the original sentence. State-of-the-art abstractive sentence compression models such as NAMAS [Rush et al, ACL 2015] and [Nallapati et al, CoNLL `16] achieve compression by also paraphrasing, which lends them more modeling strength.
However, this idea is certainly worth trying, and we may consider this as part of our future work. Thanks for the suggestion!
A difference with the recent work of Cheng & Lapata (2016) is that this paper generates oracle extractive summaries by a "unsupervised greedy" approach (similarly to other previous works). In the empirical analysis the authors say that "one potential reason why our models do not consistently outperform the extractive model of Cheng & Lapata (2016) is the additional supervised training they used to create sentence-level extractive labels to train their model". Did you try to do use the same supervised training procedure as Cheng & Lapata to verify this claim?
Dear AnonReviewer1,
We could not verify this claim since the data released by Cheng and Lapata (2016) did not include the supervised labels for training the classifier, to the best of our knowledge. It is possible we could handcraft our own training data, but at present, it seems unlikely before the review deadline given our limited labeling resources. This is the reason why we qualified our claim using the phrase "one *potential* reason". If you think it is still too strong a claim, we are happy to delete it from the main body. Please let us know.
Thanks
-Ramesh
This paper provides two RNN-based architectures for extractive document summarization. The first, "Classify", reads in the whole document and traverses the sentences a second time to decide whether to include them or not (0/1 decisions). The second, "Select",  reads in the whole document and picks the most relevant sentence one at the time. The models assume that oracle extractive summaries exist, and a pseudo ground-truth generation procedure is used, which mimics Svore et al. (2007) among others.
Overall, this paper seems a small increment over Cheng & Lapata (2016) and performance is similar or worse to that paper. The problem of single document extractive summarization is not particularly exciting since in DUC 2002 (14 years ago) existing models could not beat the lead baseline (which selects the first sentences of the document). It's a pity that this paper doesn't address the most interesting problems of abstractive summarization or apply the proposed approach to multi-document summarization. It's also a little disappointing that the maximum sentence length had to be capped to 50, which suggests the model has some trouble to scale.
Dear AnonReviewer1,
Thanks for your comments.
* Regarding your comment on single document extractive summarization not being exciting:
We believe single document extractive summarization is still interesting because the summaries produced by an extractive system are always grammatically and semantically correct and coherent, and therefore may deliver better user experience than abstractive systems which tend to be brittle in terms of both grammar and semantics. Modern datasets such as CNN/DailyMail open up new opportunities to train deep learning models for extractive summarization compared to traditional DUC datasets that have very little training data, and it is now more possible than ever to substantially outperform the LEAD baseline (which our models as well as that of Cheng and Lapata do).
Hence we think research on using deep learning models for extractive single document summarization is both desirable and necessary.
* Regarding your comment on multi-document summarization:
We are still not aware of any large scale datasets for this problem on which we could train our deep learning models. The newly released MS-MARCO dataset is the only one that comes close but it was released in parallel to our paper.
* Regarding your comment on sentence length capped to 50:
This is more of an approximation to increase training speed than any real limitation. More than 90% of the sentences have fewer than 50 words, so this approximation has very little effect in terms of accuracy, but increases training speed by restricting the tensor size of each batch.
This paper presents two RNN architectures for extractive document summarization. The first one, Classifier, takes into account the order in which sentences appear in the original document, whereas the second one, Selector, chooses sentences in an arbitrary order. For each architecture, the concatenated RNN hidden state from a sentence forward and backward pass  is used as features to compute a score that captures content richness, salience, positional importance, and redundancy. Both models are trained in a supervised manner, so the authors used "pseudo-ground truth generation" to create training data from abstractive summaries. Experiments show that the Classifier model performs better, and it achieves near state-of-the-art performance for some evaluation metrics.
The proposed model is in general an extension of Cheng and Lapata, 2016. Unfortunately, the performance is only slightly better or sometimes even worse. The authors mentioned that one key difference how they transform abstractive summaries to become gold labels for the supervised method.
However, in the experiment results, the authors described that one potential reason their models do not consistently outperform the extractive model of Cheng & Lapata, 2016 is that the unsupervised greedy approximation may generate noisier ground truth labels than Cheng & Lapata. Is there a reason to construct the training data similar to Cheng & Lapata, if that turns out to be a better method?
In order for the proposed models to be convincing, they need to outperform this baseline that's very similar to the proposed methods more consistently, since the main contribution is improved neural architectures for extractive document summarization.
Dear AnonReviewer2,
Thanks for your comments.
We agree our models do not consistently outperform the model of Cheng and Lapata in all settings. Where they fail to outpeform, they are still more or less statistically indistinguishable from their system. We would like to gently reiterate that the main contribution of our work is comparison and analysis of the Classifier and Selector architectures, and establishing the conditions under which one may outperform the other. Cheng and Lapata's model is subsumed under the Classifier architecture, and therefore outperforming this system was not the main goal of this work.
This paper presents two models for extractive document summarization: the classifier architecture and the selector architecture. These two models basically use either classification or ranking in a sequential order to pick the candidate sentences for summarization. Experiments in this paper show the results are either better or close to the SOTA.
Technical comments:
- In equation (1), there is a position-relevant component call "positional importance". I am wondering how important this component is? Is it possible to show the performance without this component? Especially, for the discussion on impact of document structure, when the model is trained on the shuffled order but tested on the original order.
- A similar question about equation (1), is the content-richness component really necessary? Since the score function already has salience part, which could measure how important of $h_j$ with respect to the whole document.
- For the dynamic summary representation in equation (3), why not use the same updating equation for both training and test procedures? During test time, the model actually knows the decisions that have been made so far by the decoder. In this way, the model will be more consistent during training and test.
- I think section 5 is the most interesting part of this paper, and it is also convincing on the difference between the two architectures.
- It is a little disappointing that the decoding algorithm used in this paper is too simple. In a minimal case, both of them could use beam search and the results could be better.
Dear AnonReviewer3,
Thanks for your feedback.
*Regarding your questions/comments on the positional features:
We have displayed the importance weights of the various abstract features of the deep Selector model on both original and shuffled data in Table 4 in the Appendix section. The results show that the positional features have high absolute weight (31.09) in the original data but close to zero weight (0.20) in the shuffled data, confirming our intuition that the document structure is quite important in modeling summaries. We also performed ablation experiments removing one feature at a time for both Classifier and Selector architectures as reported in Table 5. These results show that removing positional features hurts the Selector the most. Removing these features also hurts the Classifier, but not as much, since the architecture of the sequence classifier already captures the structural properties of the document.
*Regarding your questions/comments on the content-richness feature:
The importance weights of the features in Table 4 show that content richness is not as important as salience, position or redundancy, but the weights are not insignificant showing that they indeed add some value. The ablation tests in Table 5 also demonstrate the removing the content features hurts performance of the Classifier as well as the Selector.
*Regarding your question on inconsistency between training and testing for Eq (3):
The update for summary representation is consistent between training and testing for the Selector architecture. For the Classifier architecture, we use hard-updates at training time and soft-updates at testing time for the following reason: since the classifier model makes binary decision for each sentence, it may end up selecting too few or too many sentences for summary (this is not the case for Selector because we stop selecting sentences once the desired summary length is reached).
Therefore we run the classifier on all sentences, and sort them in the decreasing order of their probability of being in summary, and chose the top sentences in that order until the desired summary length is reached. Since we use soft probabilities in making decisions, we also use soft updates of the summary representation at test time so that it is consistent with our sentence selection mechanism.
* Regarding your comment on beam search:
We agree beam search will improve the performance of both models.
However, so far as the comparison between the two architectures is concerned, greedy algorithm is used on both of them, so it is fair. We will definitely update the numbers with beam search upon acceptance.

Meta-review:
role: PC
data: comment: Reviewers found this paper clear to read, but leaned negative on in terms of impact and originality of the work. Main complaint is that the paper is neither significantly novel in terms of modeling (pointing to Cheng & Lapata), nor significantly more performative on this task ("only slightly better"). One reviewer also has a side complaint that the task itself is also somewhat simplistic and simplified, and suggests other tasks. This comment is perhaps harsh, but reflects a mandate for revisiting "old" problems to provide significant improvements in accuracy or novel modeling.

Decision: Reject