# Abstract Text Summarization with a Convolutional Seq2 seq Model
## 1 Introduction  
Automatic abstract text summarization plays an essential part in natural language processing (NLP) and information retrieval (IR) tasks. Abstract text summarization is to generate a short summary which covers the core information of a passage or an article, consisting of a few keywords and sentences. It can be harnessed for other analyses or applications, e.g., text categorization [1], opinion mining [2] and so on. In this task, we do not simply extract the existing sentences or words from the original text, instead we generate a highly compressed paraphrasing of the core information of the text, even leveraging vocabulary absent from the source if needed. Due to the fast development of deep learning, many sequence to sequence (in the following, we use seq2 seq for simplicity) models are proposed to project an input sequence into another output sequence. They have been successful in many applications or tasks like speech recognition [3], video captioning [4] and machine translation [5]. Unlike the above tasks, in text summarization, the output sequence is much shorter than the input source sequence (document). To realize the context summarization, [6] proposes an attentional seq2 seq model based on RNNs which was originally developed for machine translation. It projects the the original document into low-dimensional embeddings. However, RNNs tend to have low-efficiency problems as they rely on the previous steps when training. Therefore, though not commonly used in sequence model, we propose a seq2 seq model based on CNN to create the representations for the source texts. However, traditional CNNs can only encode fixed size contexts. To address this issue, we propose to enlarge the effective text by stacking CNN layers over each other. By doing so, the length of the sequence being dealt with can be easily controlled, and each element in the sequence could be computed parallelly. In the contrast, RNN has to preserve the hidden state of the whole past, thus not able to perform parallelization operations. Multi-layer CNNs create hierarchical representations on the input, i.e., close elements in a sequence interact at lower layers whereas distant elements interact at higher layers [7]. Unlike the chain structure of RNN, our multi-layer CNN offers a shortcut to parallelly express long-range sequences. For instance, to represent a sequence with \(n\) words, CNNs only need \(O(\frac{n}{k})\) operations, and \(k\) is the kernel width, while RNNs need \(O(n)\) operations. In addition, kernels and non-linearities numbers are fixed in CNN, whereas in RNN, the first word will need \(n\) operations and non-linearities, and the last word will also need a single set of operations [8]. We also add gated linear units (GLU) [9] and residual connections [10] in this convolutional model. In the source document, the sentences could be very long, hence a summary could be drawn from not only the keywords but also the key sentences. So we firstly apply two CNNs on the source text, at word level and sentence level, respectively. Then to capture such hierarchical document structure, we propose a hierarchical attention mechanism applied on both levels simultaneously. We first calculate the word-level attention, then it will be re-weighted by its corresponding sentence-level attention. In summarization task, it is common that the named-entities and keywords in the test text are essential to the summary, but in training data they may actually be rare or even unseen, and we call these words as out-of-vocabulary (OOV) words. To solve the issue, we introduce a copying mechanism [11] which enables the seq2 seq model to extract the OOV words. It can distinguish the keywords based on their position or syntactic information from original text, even not knowing their exact meanings. In this work we have three major contributions:  
* We apply a convolutional seq2 seq model to realize the text summarization which could help improve the model efficiency.