# Diverse Decoding for Abstractive Document Summarization
## 2 Related Work  
Early summarization works mostly focused on extractive methods with human-engineered features, for example parts of speech [17] and term frequency [18]. Classical approaches include graph-based methods [19], integer linear programming [20] and classifier-based methods [21; 22]. On the other hand, there has been much less research on abstractive summarization. Early works on abstractive summarization were mostly restricted to the domain of sentence compression, using methods like syntactic tree pruning [23; 24] and machine translation [25]. A more systematic review of these classical approaches can be found in [15]. In recent years, neural networks have been widely investigated both on extractive and abstractive summarization tasks. In terms of neural extractive models, impressive performance gains have been made by applying deep learning techniques in the traditional framework [26; 27; 28; 29] or using a more data-driven way, i.e., the encoder-decoder approach [30; 31]. In the meantime, neural sequence-to-sequence models have provided a viable new approach for abstractive summarization. As this paper is about abstractive summarization at the document level, the following sections will mainly focus on the related works of neural abstractive summarization. [PERSON] et al. [5] were the first to apply modern neural networks in the task of abstractive text summarization and achieved state-of-the-art performance on the sentence-level summarization datasets DUC-2004 and Gigaword. Based on the neural encoder-decoder architecture with the attention mechanism, their model used a Convolutional Neural Network (CNN) encoder and a neural language model decoder. [PERSON] et al. [6] extended this work by replacing the decoder with a Recurrent Neural Network (RNN). [PERSON] et al. [7] further improved the performance by using a full RNN encoder-decoder framework. These neural models often use a fixed vocabulary, which leads to the Out-Of-Vocabulary (OOV) problem when there are new words in the input. One way to fix this is to enable the decoder network to copy these OOV words into the output sequence. Based on this copying idea, [PERSON] et al. [32] proposed CopyNet and [PERSON] et al. [33] proposed the pointer network. Another challenge for abstractive summarization is the lack of large-scale datasets for longer document-level summarization. [PERSON] et al. solved this problem by introducing the CNN/Daily Mail dataset, which was originally the DeepMind question-answering dataset [34]. In their work, they provided the first abstractive baseline for this dataset and illustrated a key problem in this document-level summarization: these attentional neural models often generate unnatural summaries with repeated phrases. [PERSON] et al. [16] proposed a graph-based attention mechanism to address saliency and a hierarchical beam search algorithm to alleviate the repeated phrases problem. [PERSON] et al. [35] proposed the intra-attention mechanism and a new training method with reinforcement learning to generate more readable long sequences. [PERSON] et al. [36] augmented the standard seq2 seq model with a pointer network [37] for the OOV problem and the coverage mechanism [38; 39] to discourage repetition. These models were able to achieve state-of-the-art performance on the CNN/Daily Mail dataset, generating summaries with basic readability. Unfortunately, these state-of-the-art models have overlooked the potential value of improving beam search, which keeps these models from getting better performance. There have been some works on producing diverse decoding sequences from recurrent models in other Natural Language Processing (NLP) tasks like machine translation and conversation modeling. [PERSON] et al. [40] were the first to address the issue of output diversity in the neural sequence generation framework; they proposed using maximum mutual information as the objective function in neural models to produce more diverse and interesting responses in conversation modeling.