Writing a summary is a different task compared to producing a longer article. As a consequence, it is likely that the topic and discourse moves made in summaries differ from those in regular articles. In this work, we present a powerful extractive summarization system which exploits rich summary-internal structure to perform content selection, redundancy reduction, and even predict the target summary length, all in one joint model.Text summarization has been addressed by numerous techniques in the community BIBREF0 . For extractive summarization, which is the focus of this paper, a popular task setup is to generate summaries that respect a fixed length limit. In the summarization shared tasks of the past Document Understanding Conferences (DUC), these limits are defined in terms of words or bytes. As a result, much work has framed summarization as a constrained optimization problem, in order to select a subset of sentences with desirable summary qualities such as informativeness, coherence, and non-redundancy within the length budget BIBREF1 , BIBREF2 , BIBREF3 .One problem with this setup is that it does not match many real-world summarization settings. For example, writers can tailor the length of their summaries to the amount of noteworthy content in the source article. Summaries created by news editors for archives, such as the New York Times Annotated Corpus BIBREF4 , exhibit a variety of lengths. There is also evidence that in the context of web search, people prefer summaries of different lengths for the documents in search results depending on the type of the search query BIBREF5 . More generally, current systems focus heavily on properties of the source document to learn to identify important sentences, and score the coherence of sentence transitions. They reason about the content of summaries primarily for purposes of avoiding redundancy, and respecting the length budget. But they ignore the idea that it might actually be useful to learn content structure and discourse planning for summaries from large collections of multi-sentence summaries.This work proposes an extractive summarization system that focuses on capturing rich summary-internal structure. Our key idea is that since summaries in a domain often follow some predictable structure, a partial summary or set of summary sentences should help predict other summary sentences. We formalize this intuition in a model called NextSum, which selects the next summary sentence based not only on properties of the source text, but also on the previously selected sentences in the summary. An example choice is shown in Table 1 . This setup allows our model to capture summary-specific discourse and topic transitions. For example, it can learn to expand on a topic that is already mentioned in the summary, or to introduce a new topic. It can learn to follow a script or discourse relations that are expected for that domain's summaries. It can even learn to predict the end of the summary, avoiding the need to explicitly define a length cutoff.The core of our system is a next-sentence prediction component, which is a feed-forward neural network driven by features capturing the prevalence of domain subtopics in the source and the summary, sentence importance in the source, and coverage of the source document by the summary so far. A full summary can then be generated by repeatedly predicting the next sentence until the model predicts that the summary should end.Since summary-specific moves may depend on the domain, we first explore domain-specific summarization on event-oriented news topics (War Crimes, Assassinations, Bombs) from the New York Times Annotated Corpus BIBREF4 . We also train a domain-general model across multiple types of events. NextSum predicts the next summary sentence with remarkably high accuracies, reaching 67% compared to a chance accuracy of 9%. The generated summaries outperform the lead baseline as well as domain-specific summarization baselines without requiring explicit redundancy check or a length constraint. Moreover, the system produces summaries of variable lengths which correlate with how long human summaries are for the same texts.Many approaches to extractive summarization are unsupervised, and focus on the role of word frequency and source document representation for selecting informative and non-redundant content BIBREF6 , BIBREF7 , BIBREF8 . More recently, supervised approaches are popular, which view content selection as a sentence-level binary classification problem, typically using a neural network BIBREF9 , BIBREF10 . Using source structure. Source structure is a common cue for summarization. Relative word frequency and position of sentences are standardly used in many systems. Discourse- and graph-based summarization techniques explicitly focus on computing document structure BIBREF11 , BIBREF12 , BIBREF13 . Other techniques include learning probabilistic topic models over source articles within a domain to capture subtopics and transitions between them BIBREF14 , BIBREF15 , BIBREF16 . But, the use of structure from summaries is less explored. Using summary structure. Actually, almost all systems maintain some representation of the partial summary at a timestep. At the very least, it is needed for respecting a length limit and for preventing redundancy. Even in recent neural network based extractive summarization, a representation of the summary so far has been proposed to allow redundancy checks BIBREF10 . However, current methods do not focus on capturing rich summary discourse and content structure.Recent abstractive neural summarization models based on encoder-decoder frameworks actually have greater scope for capturing summary structure and content. The use of techniques such as attention and pointer mechanisms can be viewed as a form of summary structure modelling BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 . However, because such systems currently operate at the word level, these mechanisms are mostly used for handling issues such as grammaticality, out-of-vocabulary items, predicate-argument structure, and local coherence. By contrast, we aim to capture higher-level transitions in the contents of a summary. Next-sentence prediction. The way we learn summary structure is by training a module for next summary sentence prediction. A parallel idea can be found in the form of next-utterance prediction in retrieval-based dialogue systems BIBREF21 , BIBREF22 , BIBREF23 . There have also been recent attempts at predicting the next sentence in text. The skip-thought model BIBREF24 is trained to predict a sentence from its neighbouring sentences to produce sentence representations. CLSTM2016 and pichotta:acl16 evaluate neural language models on next-sentence and event prediction. In contrast, we aim to predict the next output sentence within the tangible application of summarization.We first present the key ideas, and the next section explains how we implement the model.NextSum comprises two components, a next-sentence prediction system, and a summary generation module. The first is a supervised system trained to select the next summary sentence, given a set of candidate sentences from the source, and the summary generated so far. NextSum's generation component builds a summary by making repeated calls to the next-sentence predictor.The next-sentence predictor is trained on a corpus of source articles and their gold-standard summaries written by humans. In this work, we focus on single-document summarization.Consider a source article $X = \lbrace s_1,..,s_M\rbrace $ containing $M$ sentences, and a gold-standard extractive summary $G=g_1..g_N$ , a sequence of $N$ sentences. Since $G$ is extractive, $G \subseteq X$ .In NextSum, summaries are created by adding one sentence at a time. Let $Y_T=y_1..y_T$ be the partial summary at timestep $T$ ; $Y_T$ has $T$ sentences. At time $T+1$ , the goal of NextSum is to score a set of candidate sentences from the source, $C_{T+1} =\lbrace s_1,..,s_K\rbrace $ , $C_{T+1} \subseteq X$ and find the best next sentence to follow $Y_T$ . Let the gold-standard next sentence be $g_{T+1}$ . The set $C_{T+1}$ may either be all of the source sentences which have not yet been included in the summary, or be limited to a smaller size $T$0 . For now, assume that all the unselected source sentences are in the candidate set, and thus $T$1 .The model selects the next summary sentence from $C_{T+1}$ such that: $ {\hat{y}}_{T+1} = {\arg \max }_{s_i \in C_{T+1}} Pr(s_i|X,Y_T;\theta )$ When there is a tie, the earlier sentence in the article is selected. In this work, $Pr(s_i|X,Y_T;\theta )$ is estimated by a neural network parameterized by $\theta $ . Recall that the oracle next sentence $g_{T+1}$ is in $C_{T+1}$ . Hence one approach to learn the parameters of $Pr(s_i|X,Y_T;\theta )$ is to frame it as a binary classification problem where the label for sentence $g_{T+1} \in C_{T+1}$ is 1, and 0 for all $s_w \in C_{T+1}$ where $s_w \ne g_{T+1}$ . We implement this classifier using a feed-forward neural network which takes the encoded representations of ( $X$ , $Y_T$ and $\theta $0 ), and outputs the probability of label 1, $\theta $1 , which we use as $\theta $2 . The loss for the classification at timestep $\theta $3 is the binary cross-entropy loss: $ L = -\log p_{g_{T+1}} -\sum _{s_i \in C_{T+1}; s_i \ne g_{T+1}} \log (1-p_{s_i})$ .One of the special features of NextSum is that we model the end of the summary within the same setup. To do so, we introduce a special sentence $\langle \textrm {EOS}\rangle $ (End of Summary) to mark the end of every gold-standard summary, i.e. $G=y_1..y_N\langle \textrm {EOS}\rangle $ . In the model, $\langle \textrm {EOS}\rangle $ is included in candidate sets at every timestep. This inclusion allows the model to learn to discriminate between selecting a sentence from the source versus ending the summary by picking the $\langle \textrm {EOS}\rangle $ marker. Thus our candidate set is in fact $C^{\prime }_{T+1} = C_{T+1} \cup \lbrace \langle \textrm {EOS}\rangle \rbrace $ .After the next sentence prediction model is trained, it can be used to generate a complete summary for a source article. The model performs this task by iteratively predicting the next sentence until $\langle \textrm {EOS}\rangle $ is selected. Note that, unlike previous work, the generation component is not given the target length of the summary.To generate the full summary, the model employs a greedy method that simply calls the next-sentence prediction module repeatedly until $\langle \textrm {EOS}\rangle $ is selected. We also tried beam search decoding for a more globally optimal sequence of sentences, but we found in preliminary experiments that this search did not improve our results.We developed two versions of our system. Previous methods of summary content selection assume a fixed length limit. To compare against these systems, in one version of our model, NextSum $_{L}$ , the length limit is provided as a constraint. If, after the model generates a summary sentence, the word count exceeds the given length, we stop generation and truncate the last sentence so the summary is within the length limit. The second version, NextSum, is the full model which predicts the summary length. Both systems have no access to the oracle partial summary, and use their own previous decisions to construct the partial summary.We evaluate all the summaries by comparing them with gold-standard abstracts using ROUGE BIBREF30 . We use ROUGE-2 F-score, as NextSum generates summaries of varied length.In all these systems, the target length of the summary is given as a constraint. We set the length $k$ to the average length (in words) of summaries in the training dataset for each domain (Table 3 ).Lead takes the first $k$ words from the source article. For single-document extractive summarization, the lead is a very strong baseline which many systems fail to beat BIBREF31 .CHMM is the approach used by barzilay04 for extractive summarization using content models. CHMM computes an importance score for each topic $v$ . This score is a probability computed by: 1) counting the articles in the training set where $v$ appears in both the article and its summary, 2) and normalizing by the number of articles containing $v$ . To generate a summary, the model ranks the topics in order of decreasing importance, and adds one sentence from the source for each topic (breaks ties randomly if multiple sentences decoded into the same topic). The generation stops upon reaching the length limit. This method scores the summary-worthy nature of sentences based solely on their topic.Transition is an iterative greedy approach based on the transition probability of topics from the content model. It selects $\hat{y}_{T+1} = \arg \max _{s_j \in C_{T+1}}P(\textrm {topic}(s_j)|\textrm {topic}(y_{T}))$ at each timestep until the length limit is reached. This baseline simulates a degenerate version of next-sentence prediction, where the choice is based on a single feature at topic level; i.e., the probability of transitioning from the topic of the last summary sentence to the topic of the candidate. Like our model, this baseline has no access to the oracle partial summary, and uses its previous decisions for next sentence selection.CHMM-T is also an iterative greedy approach where the evaluation function is the product between topic transition probability (Transition) and topic importance (CHMM).Apart from the above domain baselines, we also compare with two other types of summaries.General is based on a recent competitive neural network based extractive system BIBREF9 . This model is designed to be domain-general. We trained it on the DailyMail dataset BIBREF28 , containing around 200K articles and their highlights, without using pretrained embeddings. Our systems are not directly comparable, because NextSum is trained on much less data, but we show this result to give an idea of the performance of recent methods.Oracle is the gold-standard extractive summary created from abstracts using the mapping method from Section "Obtaining extractive summaries" . It represents an upper bound on the performance of any extractive summary.Table 5 shows the ROUGE-2 F-score results for all the systems. The baselines, NextSum $_L$ , oracle and general are fixed length summaries.Among the baselines, we see that the simple lead summary comprising the first $k$ words of the source article is the strongest, outperforming domain-trained content model systems in all the domains. The oracle results, however, show that there is still considerable scope for the improvement of automatic systems performing sentence extraction. The oracle extractive summary (which was chosen to maximize similarity with the abstract) gets close to double the ROUGE score of lead baseline in the crime domain.Both NextSum $_L$ and NextSum outperform the lead (with statistical significance) in all cases except the bombs domain. Importantly, NextSum, which does automatic length prediction, outperforms NextSum $_L$ , indicating that automatically tailoring summaries to different lengths is clearly of value. In the next section, we examine this length prediction ability in detail.Comparing performance across domains, the source articles in bombs domain are on average longer than the other domains (refer Table 3 ), which could be a reason that content selection performance is lower here. This domain also has longer gold standard summaries and the correlation between the length of human abstracts and source articles is also the lowest in this domain.The domain-general system of cheng-lapata:2016 is trained on a much larger general corpus of summary-article pairs. While our results are not directly comparable, we see that NextSum's performance is competitive with current methods, and since it is based on a new outlook and no explicit constraints, it provides much scope for future improvements.In this section, we explain how we select the candidate set, what features we use in the neural network for next sentence prediction, and the design of the generation component.Some source articles are very long, which means that $C^{\prime }_{T+1}$ can contain many candidate sentences if we take all of the unselected sentences as candidates. In practice, we limit the size of $C^{\prime }_{T+1}$ in order to reduce the search space of the model, which improves running time.In the single-document scenario, the source text sentences are in a natural discourse, and thus in a logical and temporal order. Hence, it is not unreasonable to assume that a good summary is a subsequence of the source. Given this assumption, suppose the last sentence chosen for the summary is $s_j$ at timestep $T$ , then we consider the $K$ sentences in the source immediately following $s_j$ as the candidate set at time $T+1$ .During development, we found that when $K=10$ , the gold-standard next summary sentence is in the candidate set 90% of the time, and is present 80% of the time when using $K$ =5. Based on this empirical support for the subsequence hypothesis, we use $K=10$ plus the end of summary marker for all the experiments in this paper, for a total candidate set size of 11. For comparison, a source article in our corpus has on average 33 sentences, and the maximum is as high as 500 sentences. During training, when fewer than 10 sentences remain, we randomly sample other sentences from the entire article to ensure having enough negative samples. The model is trained on balanced dataset by downsampling, and tested on the distribution where each candidate set has size 11.We have a source document $X=\lbrace s_1..s_M\rbrace $ with $M$ sentences, $Y_T=y_1..y_T$ is a partial summary at time $T$ , and let $s$ be a sentence (or $\langle $ EOS $\rangle $ ) in the candidate set ${C^{\prime }}_{T+1}$ . NextSum's next sentence prediction relies on computing $Pr(s|X,Y_T;\theta )$ using a feedforward neural network with parameters $\theta $ . This network learns from rich feature-based representations of $M$0 , $M$1 , $M$2 , and their interactions. Domain subtopics. These features are based on topics induced from a large collection of documents in the same domain as the source article.These topics are obtained using the content-model approach of barzilay04. The content model is a Hidden Markov Model (HMM), where the states correspond to topics, and transitions between them indicate how likely it is for one topic to follow another. The emission distribution from a state is a bigram language model indicating what lexical content is likely under that topic. Each sentence in the article is emitted by one state (i.e., one topic). The probability of an article $T=s_1...s_N$ under a HMM with $M$ states $\lbrace \textrm {topic}_1,..,\textrm {topic}_M\rbrace $ is given by: $\sum _{\textrm {topic}_1..\textrm {topic}_n} \prod _{i=1}^{N} P(\textrm {topic}_i|\textrm {topic}_{i-1})P(s_i|\textrm {topic}_i)$ Content models can be trained in an unsupervised fashion to maximize the log likelihood of the articles from the domain. We choose the number of topics on a development set.Once trained, the model can compute the most likely state sequence for sentences in the source document, and in the partial summary, using Viterbi decoding. Based on the predicted topics, we compute a variety of features:[noitemsep]the proportion of source sentences assigned to each topicthe proportion of sentences in the partial summary assigned to each topicthe most likely topic of the candidate $s$ given by ${\arg \max }_{i \in \textrm {Topics}} P(\textrm {topic}_i|s)$ the emission probability of $s$ from each topicthe transition probability between the topic of the previous summary sentence $y_T$ , and the topic of $s$ , $P(\textrm {topic}(s)|\textrm {topic}(y_T))$ a global estimation of observing the candidate $s$ , $P(s)= \sum _{i \in \textrm {Topics}}P(s|\textrm {topic}_i)$  Content. We compute an encoding of source, summary so far, and the candidate sentence by averaging the pretrained word2vec embeddings BIBREF25 (trained on Google News Corpus) of each word in the span (900 features in total, 300 each for the source, summary so far, and the candidate). We also add features for the 1,000 most frequent words in the training articles, in order to encode their presence in $s$ , and in the sentence previous to $s$ in the source article, i.e. ( $s-1$ ). Similarly, for $s$ and $s-1$ , we record the presence of each part-of-speech tag and named entity. We expect these features for $s$ and $s-1$ are useful for predicting $\langle \textrm {EOS}\rangle $ , since the last sentence in a summary might contain some lexical cues. Redundancy. These features calculate the degree to which the candidate sentence overlaps with the summary so far. They include $\textit {sim}(s, y_t)$ for $t = T$ , $T-1$ , $T-2$ (3 features), where $\textit {sim}(p,q)$ is computed using cosine similarity between count vector representations of the words in $s$ and $y_t$ . We also include the number of overlapping nouns and verbs between $s$ and $Y_T$ (2 features). Position. The position of a sentence in the source document is an important indicator for content selection and is widely used in systems. We indicate the position in the source of the last generated summary sentence $y_T$ (as one of 5 bins, the size of each bin depends on the length of the source article). We also indicate the position of the candidate sentence, and its distance to $y_T$ in the source (normalized by the length of the source). Length. We include features for the length of the source, both as number of sentences, and number of words (binned into 5 bins). We also include the number of sentences and words in the summary so far. The length measures for the partial summary are not binned. Coverage. These features compute how much of the source will be covered by the summary when a candidate sentence is added to it. We use the KL divergence between the source and candidate summary when $s$ is included in it: $D_{KL} (X\Vert Y_T \cup \lbrace s\rbrace )$ where the distribution of $X$ and $Y_T$ are unigram language models. Sentence importance. We also indicate the individual importance of a candidate sentence. The frequency of a word in the source is known to be a strong feature for importance BIBREF26 . With this intuition, we include the $ \frac{1}{|s|}\sum _{w \in s} \textrm {uni}_X(w)$ where $w$ is a token in the candidate sentence, and $\textrm {uni}_X(w)$ is the unigram probability of $w$ in the source $X$ .We also use a separate pre-trained model of word importance. This model feeds the context of a target word (the two words before and two words after) into a LSTM model which outputs the probability of the target word appearing in a summary. The importance score of a sentence is then the average and maximum of the predicted scores of each word in the sentence. This model is trained on the same training and development data sets.We hypothesize that next-sentence prediction is more likely to be successful in event-oriented domains (describing events as opposed to explanations and opinions). Moreover, summary-specific moves may be more prominent and learnable from summary-article pairs within specific domains compared to a general corpus.So we create three domain-specific and one domain-general dataset, all focusing on events. We use the New York Times Annotated Corpus (NYtimes) BIBREF4 since it provides topic metadata, has thousands of article-summary pairs on different topics, and summaries are not written to set lengths. We selected three topics: “War Crimes and Criminals” (crime), “Assassinations and Attempted Assassinations” (assassin.), and “Bombs and Explosives” (bombs). We also create a more general dataset (mixed) by randomly sampling from all the three domains.We sample a similar number of articles across each domain, and randomly split each domain into 80% training, 10% development and 10% test data. Table 2 shows the sizes of these datasets.We use the Stanford CoreNLP toolkit BIBREF27 to tokenize, segment sentences, and assign part of speech tags to all the texts.As previously mentioned, summaries are often written to express the summary-worthy content of an article, and not restricted to an arbitrary length. This property can be seen in our data (Table 3 ).The NYTimes summaries are abstractive in nature and range from a minimum of 2 words to as many as 278 words. The last column of the table gives the Kendall Tau correlation (corrected for ties) between the length of the source and the summary. There is a significant positive correlation, implying that the length of the article is indicative of its information content. This finding motivates us to include the length of the source article as a feature for next sentence prediction, though we note that the source length by itself is not enough to determine the summary length without doing further analysis of the source content.The summaries from NYTimes are abstractive in nature. Our system is extractive, and for training the next sentence selection from the source, we need a mapping between the abstractive summary and the sentences in the source article. Note that we create these extractive summaries only for training our model. We will evaluate NextSum's output by comparing with the abstractive human summaries as is standard practice.We map each sentence in the abstract to the most similar sentence in the source article. Let $A=a_1..a_n$ be the sequence of sentences in the abstract. For each $a_i$ , we find $y_i = \arg \max _{s_j \in X}\cos (a_i,s_j)$ where $X$ is the set of source sentences, and $\cos (p,q)$ is the cosine similarity between the word unigrams of $p$ and $q$ .The sequence $Y=y_1..y_n$ corresponding to $A=a_1..a_n$ forms the gold standard extractive summary. Since the extractive summary mirrors the sequence of content in the abstract, the structure of the summary is preserved, allowing our next sentence prediction system to be trained on the extractive sequence of sentences. It is also for this reason that we do not use summarization datasets such as the CNN/Daily Mail corpus BIBREF28 where summaries are three-sentence highlights, and do not have any discernible discourse structure as a whole.We first evaluate our model intrinsically on the next-sentence prediction task, then test its performance on the full summary generation problem.Here, the goal is to select the best sentence to follow the partial summary from a candidate set of 11 options (see Section "Data" ). For evaluating this part of our system, we assume that we have oracle partial summaries; i.e., the partial summary at timestep $T$ , $Y_T=y_1..y_T$ is the same as the gold summary sequence up to time $T$ . The question is how well we can predict the next sentence in this sequence from the candidate set $C^{\prime }_{T+1}$ . The correct answer is the sentence in the gold standard at position $T+1$ . The prediction at each timestep is a separate classification example.Recall that we framed the machine learning problem as one of binary classification. We thus present two sets of results: a) on the binary task, and b) on the final choice of one sentence from the candidate set (among the 11 candidates). In task (a.), the binary evaluation, the model discriminates among the 2 classes by thresholding at $p_s>0.5$ . The best setting has 4 hidden layers, each layer comprising between 500 to 1,500 neurons. We trained the model by backpropagation using the Adam optimizer BIBREF29 for up to 75 epochs. Hyperparameters were tuned on the development set. The choice of a final sentence, Task (b.), is made by picking the candidate sentence with highest $p_s$ .Table 4 shows the accuracy on binary classification task and 1-of-11 task, on the different domains. In the 1-of-11 task, the expected chance-level accuracy is roughly 9.1%, since we force every candidate set to have size 11. Our next-utterance prediction system's accuracy is between 60 to 67% on the different domains, showing that there are distinctive clues on summary internal structure and content, which can be learned by a model. Note also that the accuracy numbers are consistent across all domains and the mixed case indicating that the patterns are fairly domain-general within event-oriented documents.These evaluations are somewhat idealistic in that the model has access to oracle partial summaries during prediction. We next evaluate NextSum on the full summarization task.NextSum requires neither redundancy removal nor length constraints. In this section, we show that our system produces summaries of varied lengths which correlate with the lengths of human-written summaries of the same source article.Figure 1 shows the distribution of the length (in words) of NextSum summaries (all domains put together). The generated lengths vary greatly, and span the average range covered by the summaries in the training data. The majority of lengths are in the 30 to 50 words limit. Hence NextSum is specializing summary lengths to cover a wide range.Next, we measure how well these summary lengths correlate with the lengths of the human-written abstracts. Table 6 shows the Kendall Tau correlation (corrected for ties) between length in words of the NextSum summary and the length of the abstract for the same source.NextSum's summary lengths correlate fairly well with those of the abstracts, leading to significant numbers in all the domains and the mixed case. Again, the length prediction is worse on the Bombs domain compared to the rest. Overall, this result shows promise that we can develop summarization systems which automatically tailor their content based on properties of the source.In this work, we have presented the first summarization system which integrates content selection, summary length prediction, and redundancy removal. Central to this system is the use of a next-sentence prediction system which learns summary-internal discourse transitions. We show that NextSum outperforms a number of baselines on ROUGE-2 F-scores even when the summary length is not provided to the system. Furthermore, the lengths of the predicted summaries correlate positively with the lengths of human-written abstracts, indicating that our method implicitly captures some aspect of how much summary-worthy content is present in the source article.In future work, we plan to investigate whether this approach also leads to more coherent summaries. This issue will be especially important in the multi-document setting, which we would also like to investigate using an extension of our model.

Q: How does nextsum work?