# Diverse Decoding for Abstractive Document Summarization
## 1 Introduction  
Document summarization is the task of generating a condensed and coherent summary of a document while retaining the salient information. There are two broad types of summarization: extractive and abstractive. Extractive summarization systems generate summaries by extracting sentences from the original documents, while abstractive summarization systems use words or phrases that may not appear in the original document like the human-written summaries. Abstractive summarization is considered to be much more difficult, for it involves sophisticated techniques like natural language understanding, knowledge representation, natural language generation, etc. Over the past few years, deep neural network approaches have shown encouraging results in various natural language generation (NLG) tasks like machine translation [1; 2; 3], image captioning [4], and abstractive summarization [5]. In particular, the attention-based sequence-to-sequence (seq2 seq) framework with recurrent neural networks (RNNs) [6; 7; 5] prevails at the task of abstractivesummarization. While promising, these methods still fail to provide satisfactory performance when faced with a long input sequence of multiple sentences, with a considerable gap with respect to extractive approaches. In this work, we study the key factors that keep abstractive models from achieving better performance. The sequence-to-sequence framework consists of two parts: an encoder that processes the input and a decoder that generates the output. There have been extensive works on the design and alignment of the encoder, like the choice of the neural network structure [5; 6; 7], the specific design of the attention mechanism [8; 9; 10; 11; 12; 13; 14], etc. In contrast, the decoding process, which directly affects the results of the summarization task, has received relatively much less attention. In the decoding stage, nearly all these seq2 seq models use a heuristic algorithm called beam search, which generates sequences word-by-word in a greedy left-to-right fashion while keeping a fixed number (top \(N\)) of candidates at each time step. At the end of the beam search, the top sequence with the highest conditional probability would be selected as the summary of the input document. However, in practice, with the increasing length of generated sequences, candidate sequences expanded from a single beam would gradually take up the positions of top \(N\). As a result, the beam search decoder always produces nearly identical sequences that only differ by several words in the endings. Figure 1 shows an example of the outputs of the neural decoder applying the standard beam search method. As the decoding process is in fact a process of iterative word prediction, the lack of diversity of beam search means that this widely-applied method only covers a small portion of the search space. Therefore, the standard beam search is not only a waste of computational resources, but more importantly, eliminates the chance of generating more distinct and maybe better sequences for the summarization systems. Another common weakness of previous works is the lack of an explicit method for estimating the salient information preservation of the input document, as saliency is one of the core requests for the summarization system [15; 16]. During the whole process of generating the sequences, the only feature that neural decoders rely on is the conditional probability of the trained language model, which may result in the output summary being fluent in language, yet missing most of the salient points of the original document. In this paper, we study how to improve abstractive document summarization by focusing on addressing the issues of decoding and propose a method called Neural Abstractive Summarization  
Figure 1: Example from the CNN/Daily Mail test dataset showing the outputs of the standard beam search decoder. with Diverse Decoding (NASDD).