Venue: NAACL 2019

# Abstractive Summarization of _Reddit_ Posts

with Multi-level Memory Networks

 Byeongchang Kim Hyunwoo Kim Gunhee Kim

Department of Computer Science and Engineering & Center for Superintelligence

Seoul National University, Seoul, Korea

{byeongchang.kim,hyunwoo.kim}@vision.snu.ac.kr gunhee@snu.ac.kr

###### Abstract

We address the problem of abstractive summarization in two directions: proposing a novel dataset and a new model. First, we collect _Reddit TIFU_ dataset, consisting of 120K posts from the online discussion forum Reddit. We use such informal crowd-generated posts as text source, in contrast with existing datasets that mostly use formal documents as source such as news articles. Thus, our dataset could less suffer from some biases that key sentences usually locate at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Second, we propose a novel abstractive summarization model named _multi-level memory networks_ (MMN), equipped with multi-level memory to store the information of text from different levels of abstraction. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the _Reddit TIFU_ dataset is highly abstractive and the MMN outperforms the state-of-the-art summarization models. The code and dataset are available at [http://vision.snu.ac.kr/projects/reddit-tifu](http://vision.snu.ac.kr/projects/reddit-tifu).

## 1 Introduction

Abstractive summarization methods have been under intensive study, yet they often suffer from inferior performance compared to extractive methods Allahyari et al. (2017); Nallapati et al. (2017); See et al. (2017). Admittedly, by task definition, abstractive summarization is more challenging than extractive summarization. However, we argue that such inferior performance is partly due to some biases of existing summarization datasets. The source text of most datasets Over et al. (2007); Hermann et al. (2015); Cohan et al. (2018); Grusky et al. (2018); Narayan et al. (2018) originates from formal documents such as news articles, which have some structural patterns of which extractive methods better take advantage.

In formal documents, there could be a strong tendency that key sentences locate at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Hence, summarization methods could generate good summaries by simply memorizing keywords or phrases from particular locations of the text. Moreover, if abstractive methods are trained on these datasets, they may not show much abstraction See et al. (2017), because they are implicitly forced to learn structural patterns Kedzie et al. (2018). Grusky et al. (2018) and Narayan et al. (2018) recently report similar extractive bias in existing datasets. They alleviate this bias by collecting articles from diverse news publications or regarding intro sentences as gold summary.

Different from previous approaches, we propose to alleviate such bias issue by changing the source of summarization dataset. We exploit user-generated posts from the online discussion forum Reddit, especially TIFU subreddit, which are more casual and conversational than news articles. We observe that the source text in Reddit does not follow strict formatting and disallows models to simply rely on locational biases for summarization. Moreover, the passages rarely contain sentences that are nearly identical to the gold summary. Our new large-scale dataset for abstractive summarization named as _Reddit TIFU_ contains 122,933 pairs of an online post as source text and its corresponding long or short summary sentence. These posts are written by many different users, but each pair of post and summary is created by the same user.

Another key contribution of this work is to propose a novel memory network model named _multi-level memory networks_ (MMN). Our model is equipped with multi-level memory networks, storing the information of source text from different

levels of abstraction (_i.e._ word-level, sentence-level, paragraph-level and document-level). This design is motivated by that abstractive summarization is highly challenging and requires not only to understand the whole document, but also to find salient words, phrases and sentences. Our model can sequentially read such multiple levels of information to generate a good summary sentence.

Most abstractive summarization methods [15, 20, 38, 46] employ sequence-to-sequence (seq2seq) models [7] where an RNN encoder embeds an input document and another RNN decodes a summary sentence. Our MMN has two major advantages over seq2seq-based models. First, RNNs accumulate information in a few fixed-length memories at every step regardless of the length of an input sequence, and thus may fail to utilize far-distant information due to vanishing gradient. It is more critical in summarization tasks, since input text is usually very long (\(>\)300 words). On the other hand, our convolutional memory explicitly captures long-term information. Second, RNNs cannot build representations of different ranges, since hidden states are sequentially connected over the whole sequence. This still holds even with hierarchical RNNs that can learn multiple levels of representation. In contrast, our model exploits a set of convolution operations with different receptive fields; hence, it can build representations of not only multiple levels but also multiple ranges (_e.g._ sentences, paragraphs, and the whole document).

Our experimental results show that the proposed MMN model improves abstractive summarization performance on both our new Reddit TIFU and existing Newsroom-Abs [7] and XSum [29] datasets. It outperforms several state-of-the-art abstractive models with seq2seq architecture such as [15, 38, 46]. We evaluate with quantitative language metrics (_e.g._ perplexity and ROUGE [38]) and user studies via Amazon Mechanical Turk (AMT).

The contributions of this work are as follows.

1. We newly collect a large-scale abstractive summarization dataset named _Reddit TIFU_. As far as we know, our work is the first to use non-formal text for abstractive summarization.
2. We propose a novel model named _multi-level memory networks_ (MMN). To the best of our knowledge, our model is the first attempt to leverage memory networks for the abstractive summarization. We discuss the unique updates of the MMN over existing memory networks in Section 2.
3. With quantitative evaluation and user studies via AMT, we show that our model outperforms state-of-the-art abstractive summarization methods on both Reddit TIFU, Newsroom abstractive subset and XSum dataset.

## 2 Related Work

Our work can be uniquely positioned in the context of the following three topics.

**Neural Abstractive Summarization**. Many deep neural network models have been proposed for abstractive summarization. One of the most dominant architectures is to employ RNN-based seq2seq models with attention mechanism such as [14, 46, 47, 48, 49, 50].In addition, recent advances in deep network research have been promptly adopted for improving abstractive summarization. Some notable examples include the use of variational autoencoders (VAEs) [45, 46], graph-based attention [20], pointer-generator models [10], self-attention networks [12], reinforcement learning [21, 22], contextual agent attention [31] and integration with extractive models [14, 15].

Compared to existing neural methods of abstractive summarization, our approach is novel to replace an RNN-based encoder with explicit multi-level convolutional memory. While RNN-based encoders always consider the whole sequence to represent each hidden state, our multi-level memory network exploits convolutions to control the extent of representation in multiple levels of sentences, paragraphs, and the whole text.

**Summarization Datasets**. Most existing summarization datasets use formal documents as source text. News articles are exploited the most, including in DUC [34], Gigaword [26], CNN/DailyMail [20, 21], Newsw 

room (Grusky et al., 2018) and XSum (Narayan et al., 2018a) datasets. Cohan et al. (2018) introduce datasets of academic papers from arXiv and PubMed. Hu et al. (2015) propose the LC-STS dataset as a collection of Chinese microblog's short text each paired with a summary. However, it selects only formal text posted by verified organizations such as news agencies or government institutions. Compared to previous summarization datasets, our dataset is novel in that it consists of posts from the online forum Reddit.

Rotten Tomatoes and Idebate dataset Wang and Ling (2016) use online text as source, but they are relatively small in scale: 3.7K posts of RottenTomatoes compared to 80K posts of TIFU-short as shown in Table 1. Moreover, Rotten Tomatoes use multiple movie reviews written by different users as single source text, and one-sentence consensus made by another professional editor as summary. Thus, each pair of this dataset could be less coherent than that of our TIFU, which is written by the same user. The Idebate dataset is collected from short arguments of debates on controversial topics, and thus the text is rather formal. On the other hand, our dataset contains the posts of interesting stories happened in daily life, and thus the text is more unstructured and informal.

**Neural Memory Networks**. Many memory network models have been proposed to improve memorization capability of neural networks Kaiser et al. (2017); Na et al. (2017); Yoo et al. (2019). Weston et al. (2014) propose one of early memory networks for language question answering (QA); since then, many memory networks have been proposed for QA tasks Sukhbaatar et al. (2015); Kumar et al. (2016); Miller et al. (2016). Park et al. (2017) propose a convolutional read memory network for personalized image captioning. One of the closest works to ours may be Singh et al. (2017), which use a memory network for text summarization. However, they only deal with extractive summarization by storing embeddings of individual sentences into memory.

Compared to previous memory networks, our MMN has four novel features: (i) building a multi-level memory network that better abstracts multi-level representation of a long document, (ii) employing a dilated convolutional memory write mechanism to correlate adjacent memory cells, (iii) proposing normalized gated tanh units to avoid covariate shift within the network, and (iv) generating an output sequence without RNNs.

## 3 Reddit TIFU Dataset

We introduce the Reddit TIFU dataset whose key statistics are outlined in Table 1. We collect data from Reddit, which is a discussion forum platform with a large number of subreddits on diverse topics and interests. Specifically, we crawl all the posts from 2013-Jan to 2018-Mar in the TIFU subreddit, where every post should strictly follow the posting rules, otherwise they are removed. Thanks to the following rules1, the posts in this subreddit can be an excellent corpus for abstractive summarization: _Rule 3: Posts and titles without context will be removed. Your title must make an attempt to encapsulate the nature of your f***up. Rule 11: All posts must end with a TL;DR summary that is descriptive of your f***up and its consequences_. Thus, we regard the body text as source, the title as short summary, and the TL;DR summary as long summary. As a result, we make two sets of datasets: _TIFU-short_ and _TIFU-long_. Figure 1 shows an example post of the TIFU subreddit.

Figure 1: An example post of the TIFU subreddit.

 



### Preprocessing

We build a vocabulary dictionary \({\cal V}\) by choosing the most frequent \(V\)(=15K) words in the dataset. We exclude any urls, unicodes and special characters. We lowercase words, and normalize digits to 0. Subreddit names and user ids are replaced with @subreddit and @userid token, respectively. We use markdown2 package to strip markdown format, and spacy3 to tokenize words. Common prefixes of summary sentences (_e.g._ ifu by, tit-tl;dr, etc) are trimmed. We do not take OOV words into consideration, since our vocabulary with size 15K covers about 98% of word frequencies in our dataset. We set the maximum length of a document as 500. We exclude the gold summaries whose lengths are more than 20 and 50 for _TIFUshort_ and _TIFUlong_, respectively. They amount to about 0.6K posts in both datasets (_i.e._ less than 1% and 3%). We use these maximum lengths, based on previous datasets (_e.g._ 8, 31, 56 words on average per summary in Gigaword, DUC, and CNN/DailyMail datasets, respectively). We randomly split the dataset into 95% for training, 5% for test.

Footnote 2: [https://python-markown.github.io/](https://python-markown.github.io/).

Footnote 3: [https://spacy.io](https://spacy.io).

### Abstractive Properties of Reddit TIFU

We discuss some abstractive characteristics found in Reddit TIFU dataset, compared to existing summarization datasets based on news articles.

**Weak Lead Bias**. Formal documents including news articles tend to be structured to emphasize key information at the beginning of the text. On the other hand, key information in informal online text data are more spread across the text. Figure 2 plots the density histogram of the relative locations of bigrams of gold summary in the source text. In the CNN/DailyMail and Newsroom, the bigrams are highly concentrated on the front parts of documents. Contrarily, our Reddit TIFU dataset shows rather uniform distribution across the text.

This characteristic can be also seen from the ROUGE score comparison in Table 2. The Lead baseline simply creates a summary by selecting the first few sentences or words in the document. Thus, a high score of the Lead baseline implicates a strong lead bias. The Lead scores are the lowest in our TIFU dataset, in which it is more difficult for models to simply take advantage of locational bias for the summary.

**Strong Abstractness**. Besides the locational bias, news articles tend to contain wrap-up sentences that cover the whole article, and they often have resemblance to its gold summary. Its existence can be measured by the score of the Ext-Oracle baseline, which creates a summary by selecting the sentences with the highest average score of F1 ROUGE-1/2/L. Thus, it can be viewed as an upper bound for extractive models Narayan et al. (2018a, b); Nallapati et al. (2017).

\begin{table}
\begin{tabular}{c|c c c c|c c c|c c c|c} \hline  & \multicolumn{4}{c|}{PG} & \multicolumn{4}{c|}{Lead} & \multicolumn{4}{c|}{Ext-Oracle} & PG/Lead & PG/Oracle \\ \hline Dataset & R-1 & R-2 & R-L & R-1 & R-2 & R-L & R-1 & R-2 & R-L & Ratio (R-L) & Ratio (R-L) \\ \hline CNN/DMNallapati et al. (2016) & 36.4 & 15.7 & 33.4 & 39.6 & 17.7 & 36.2 & 54.7 & 30.4 & 50.8 & 0.92x & 0.66 x \\ NY Times Sandhaus (2008) & 44.3 & 27.4 & 40.4 & 31.9 & 15.9 & 23.8 & 52.1 & 31.6 & 46.7 & 1.70x & 0.87x \\ Newsroom Grusky et al. (2018) & 26.0 & 13.3 & 22.4 & 30.5 & 21.3 & 28.4 & 41.4 & 24.2 & 39.4 & 0.79x & 0.57x \\ \hline Newsroom-Abs Grusky et al. (2018) & **14.7** & **2.2** & **10.3** & 13.7 & 2.4 & 11.2 & 29.7 & 10.5 & 27.2 & 0.92x & 0.38x \\ XSum Narayanan et al. (2018a) & 29.7 & 9.2 & 23.2 & 16.3 & 1.6 & 12.0 & 29.8 & 8.8 & 22.7 & 1.93x & 1.02x \\ \hline TIFU-short & 18.3 & 6.5 & 17.9 & **3.4** & **0.0** & **3.3** & **8.0** & **0.0** & **7.7** & **5.42x** & **2.32x** \\ TIFUlong & 19.0 & 3.7 & 15.1 & **2.8** & **0.0** & **2.7** & **6.8** & **0.0** & **6.6** & **5.59x** & **2.29x** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of F1 ROUGE scores between different datasets (row) and methods (column). PG is a state-of-the-art abstractive summarization method, and Lead and Ext–Oracle are extractive ones. PG/Lead and PG/Oracle are the ROUGE-L ratios of PG with Lead and Ext–Oracle, respectively. We report the numbers for each dataset (row) from the corresponding cited papers.

Figure 2: Relative locations of bigrams of gold summary in the source text across different datasets.



In Table 2, the ROUGE scores of the Ext-Oracle are the lowest in our TIFU dataset. It means that the sentences that are similar to gold summary scarcely exist inside the source text in our dataset. This property forces the model to be trained to focus on comprehending the entire text instead of simply finding wrap-up sentences.

Finally, PG/Lead and PG/Oracle in Table 2 are the ROUGE-L ratios of PG with Lead and Ext-Oracle, respectively. These metrics can quantify the dataset according to the degree of difficulty for extractive methods and the suitability for abstractive methods, respectively. High scores of the TIFU dataset in both metrics show that it is potentially an excellent benchmark for evaluation of abstractive summarization systems.

## 4 Multi-level Memory Networks (MMN)

Figure 3 shows the proposed _multi-level memory network_ (MMN) model. The MMN memorizes the source text with a proper representation in the memory and generates a summary sentence one word at a time by extracting relevant information from memory cells in response to previously generated words. The input of the model is a source text \(\{x_{i}\}=x_{1},...,x_{N}\), and the output is a sequence of summary words \(\{y_{t}\}=y_{1},...,y_{T}\), each of which is a symbol from the dictionary \(\mathcal{V}\).

### Text Embedding

Online posts include lots of morphologically similar words, which should be closely embedded. Thus, we use the fastText Bojanowski et al. (2016) trained on the Common Crawl corpus, to initialize the word embedding matrix \(\mathbf{W}_{emb}\). We use the same embedding matrix \(\mathbf{W}_{emb}\) for both source text and output sentences. That is, we represent a source text \(\{x_{i}\}^{N}_{i=1}\) in a distributional space as \(\{\mathbf{d}_{i}^{0}\}^{N}_{i=1}\) by \(\mathbf{d}_{i}^{0}=\mathbf{W}_{emb}\mathbf{x}_{i}\) where \(\mathbf{x}_{i}\) is a one-hot vector for \(i\)-th word in the source text. Likewise, output words \(\{y_{t}\}^{T}_{t=1}\) is embedded as \(\{\mathbf{o}_{t}^{0}\}^{T}_{t=1}\), and \(\mathbf{d}_{i}^{0}\) and \(\mathbf{o}_{t}^{0}\in\mathbb{R}^{300}\).

### Construction of Multi-level Memory

As shown in Figure 3(a), the multi-level memory network takes the source text embedding \(\{\mathbf{q}_{i}^{0}\}^{N}_{i=1}\) as an input, and generates \(S\) number of memory tensors \(\{\mathbf{M}_{s}^{a/c}\}^{S}_{s=1}\) as output, where superscript \(a\) and \(c\) denote input and output memory repre

Figure 4: Comparison between (a) the gated linear unit Gehring et al. (2017) and (b) the proposed normalized gated tanh unit.

Figure 3: Illustration of the proposed _multi-level memory network_ (MMN) model.

 

sentation, respectively. The multi-level memory network is motivated by that when human understand a document, she does not remember it as a single whole document but ties together several levels of abstraction (_e.g_. word-level, sentence-level, paragraph-level and document-level). That is, we generate sets of memory tensors, each of which associates each cell with different number of neighboring word embeddings based on the level of abstraction. To build memory slots of such multi-level memory, we exploit a multi-layer CNN as the write network, where each layer is chosen based on the size of its receptive field.

However, one issue of convolution is that large receptive fields require many layers or large filter sizes. For example, stacking layers with a filter size of 3 results in a receptive field size of 13, _i.e_. each output depends on 13 input words. In order to grow the receptive field without increasing the computational cost, we exploit the _dilated_ convolution Yu and Koltun [2016] for the write network.

**Memory Writing with Dilated Convolution**. In dilated convolution, the filter is applied over an area larger than its length by skipping input values with a certain gap. Formally, for a 1-D -length and a filter . , the dilated convolution operation on elements of a sequence is defined as

(1)

where is the dilation rate, is the filter size, accounts for the direction of and and are the parameters of the filter. With , the dilated convolution reduces to a regular convolution. Using a larger dilation enables a single output at the top level to represent a wider range of input, thus effectively expanding the receptive field.

To the embedding of a source text , we recursively apply a series of dilated convolutions . We denote the output of the -th convolution layer as .

**Normalized Gated Tanh Units**. Each convolution is followed by our new activation of _normalized gated tanh unit_ (NGTU), which is illustrated in Figure 4(b):

(2) (3)

where is a sigmoid, is the element-wise multiplication and denote the filter and gate for -th layer dilated convolution, respectively.

The NGTU is an extension of the existing gated tanh units (GTU) Oord et al. [2016a, b] by applying weight normalization Salimans and Kingma [2016] and layer normalization Ba et al. [2016]. This mixed normalization improves earlier work of Gehring et al. [2017], where only weight normalization is applied to the GLU. As in Figure 4(a), it tries to preserve the variance of activations throughout the whole network by scaling the output of residual blocks by . However, we observe that this heuristic does not always preserve the variance and does not empirically work well in our dataset. Contrarily, the proposed NGTU not only guarantees preservation of activation variances but also significantly improves the performance.

**Multi-level Memory**. Instead of using only the last layer output of CNNs, we exploit the outputs of multiple layers of CNNs to construct sets of memories. For example, memory constructed from the 4-th layer, whose receptive field is 31, may have sentence-level embeddings, while memory from the 8-th layer, whose receptive field is 511, may have document-level embeddings. We obtain each -th level memory by resembling key-value memory networks Miller et al. [2016]:

(4)

Recall that and are input and output memory matrix, respectively. indicates an index of convolutional layer used for the -th level memory. For example, if we set and , we make three-level memories, each of which uses the output of the 3-rd, 6-th, and 9-th convolution layer, respectively. To output memory representation , we add the document embedding as a skip connection.

### State-Based Sequence Generation

We discuss how to predict the next word at time step based on the memory state and previously generated words . Figure 3(b) visualizes the overall procedure of decoding.

We first apply max-pooling to the output of the last layer of the encoder network to build a whole document embedding :

(5) 

The decoder is designed based on WaveNet (Oord et al., 2016) that uses a series of causal dilated convolutions, denoted by \(\hat{\mathcal{F}}(\mathbf{o}^{l}_{1:t})\in\mathbb{R}^{t\times 300}\). We globally condition \(\mathbf{d}^{whole}\) to obtain embeddings of previously generated words \(\mathbf{o}^{l}_{1:t}\) as:

\[\mathbf{h}^{l}_{f/g}=\hat{\mathcal{F}}^{l}_{f/g}(\mathbf{o}^{l}_{1: t}+\mathbf{W}^{l}_{ff/g}\mathbf{d}^{whole}), \tag{6}\] \[\mathbf{h}^{l}_{a}=\text{tanh}(\mathbf{h}^{l}_{f})\circ\sigma( \mathbf{h}^{l}_{g}),\] (7) \[\mathbf{o}^{l+1}_{1:t}=\text{LayerNorm}(\mathbf{o}^{l}_{1:t}+ \mathbf{h}^{l}_{a}), \tag{8}\]

where \(\mathbf{h}^{l}_{f/g}\) are the filter and gate hidden state respectively, and learnable parameters are \(\mathbf{W}^{l}_{f}\) and \(\mathbf{W}^{l}_{g}\in\mathbb{R}^{300\times 300}\). We initialize \(\mathbf{o}^{0}_{t}=\mathbf{W}_{emb}y_{t}\). We set the level of the decoder network to \(L=3\) for TIFU-short and \(L=5\) for TIFU-long.

Next, we generate \(S\) number of query vectors \(\{\mathbf{q}^{s}_{t}\}_{s=1}^{S}\) at time \(t\) to our memory network as

\[\mathbf{q}^{s}_{t}=\text{tanh}(\mathbf{W}^{s}_{q}\mathbf{o}^{L}_{t}+\mathbf{ b}^{s}_{q}), \tag{9}\]

where \(\mathbf{W}^{s}_{q}\in\mathbb{R}^{300\times 300}\) and \(\mathbf{b}^{s}_{q}\in\mathbb{R}^{300}\).

Each of these query vectors \(\{\mathbf{q}^{s}_{t}\}_{s=1}^{S}\) is fed into the attention function of each level of memory. As in (Vaswani et al., 2017), the attention function is

\[\mathbf{M}^{s}_{o_{t}}=\text{softmax}(\frac{\mathbf{q}^{s}_{t}(\mathbf{M}^{a} _{s})^{T}}{\sqrt{d^{emb}}})\mathbf{M}^{c}_{s}, \tag{10}\]

where we set \(d^{emb}=300\) for the embedding dimension and \(\mathbf{M}^{s}_{o_{t}}\in\mathbb{R}^{300}\).

Next, we obtain the output word probability:

\[\mathbf{s}_{t}=\text{softmax}(\mathbf{W}_{o}[\mathbf{M}^{1}_{o_{t}};...; \mathbf{M}^{S}_{o_{t}};\mathbf{o}^{L}_{t}]), \tag{11}\]

where \(\mathbf{W}_{o}\in\mathbb{R}^{(300\times(S+1))\times V}\). Finally, we select the word with the highest probability \(y_{t+1}=\text{argmax}_{\mathbf{s}\in\mathcal{V}}(\mathbf{s}_{t})\). Unless \(y_{t+1}\) is an EOS token, we repeat generating the next word by feeding \(y_{t+1}\) into the output convolution layer of Eq.(8).

### Training

We use the softmax cross-entropy loss from estimated \(y_{t}\) to its target \(y_{GT,t}\). However, it forces the model to predict extremes (zero or one) to distinguish among the ground truth and alternatives. The label smoothing alleviates this issue by acting as a regularizer that makes the model less confident in its prediction. We smooth the target distribution with a uniform prior distribution \(u\)(Pereyra et al., 2017; Edunov et al., 2017; Vaswani et al., 2017). Thus, the loss over the training set \(\mathcal{D}\) is

\[\mathcal{L}=-\sum\log p_{\theta}(\mathbf{y}|\mathbf{x})-D_{KL}(u||p_{\theta}( \mathbf{y}|\mathbf{x})).\]

We implement label smoothing by modifying the ground truth distribution for word \(y_{GT,t}\) to be \(p(y_{GT,t})=1-\epsilon\) and \(p(y^{\prime})=\epsilon/\mathcal{V}\) for \(y^{\prime}\neq y_{GT,t}\) where \(\epsilon\) is a smoothing parameter set to 0.1. Further details can be found in the Appendix.

## 5 Experiments

### Experimental Setting

**Evaluation Metrics**. We evaluate the summarization performance with two language metrics: perplexity and standard F1 ROUGE scores Lin (2004). We remind that lower perplexity and higher ROUGE scores indicate better performance.

**Datasets**. In addition to Reddit TIFU, we also evaluate on two existing datasets: abstractive subset of NewsroomGrusky et al. (2018) and XSumNarayan et al. (2018). These are suitable benchmarks for evaluation of our model in two aspects. First, they are specialized for abstractive summarization, which meets well the goal of this work. Second, they have larger vocabulary size (40K, 50K) than Reddit TIFU (15K), and thus we can evaluate the learning capability of our model.

**Baselines**. We compare with three abstractive summarization methods, one basic seq2seq model, two heuristic extractive methods and variants of our model. We choose PGSee et al. (2017), SEASSZhou et al. (2017), DRGDLi et al. (2017) as the state-of-the-art methods of abstractive summarization. We test the attention based seq2seq model denoted as s2s-attChopra et al. (2016). As heuristic extractive methods, the Lead-1 uses the first sentence in the text as summary, and the Ext-Oracle takes the sentence with the highest average score of F1 ROUGE-1/2/L with the gold summary in the text. Thus, Ext-Oracle can be viewed as an upper-bound for extractive methods.

We also test variants of our method MMN-*. To validate the contribution of each component, we exclude one of key components from our model as follows: (i) -NoDilated with conventional convolutions instead, (ii) -NoMulti with no multi-level memory (iii) -NoNGTU with existing gated linear units Gehring et al. (2017). That is, -NoDilated quantifies the improvement by the dilated convolution, -NoMulti assesses the effect of multi-level memory, and -NoNGTU validates the normalized gated tanh unit.

Please refer to the Appendix for implementation details of our method 



### Quantitative Results

Table 3 compares the summarization performance of different methods on the TIFU-short/long dataset. Our model outperforms the state-of-the-art abstractive methods in both ROUGE and perplexity scores. PG utilizes a pointer network to copy words from the source text, but it may not be a good strategy in our dataset, which is more abstractive as discussed in Table 2. SEASS shows strong performance in DUC and Gigaword dataset, in which the source text is a single long sentence and the gold summary is its shorter version. Yet, it may not be sufficient to summarize much longer articles of our dataset, even with its second-level representation. DRGD is based on the variational autoencoder with latent variables to capture the structural patterns of gold summaries. This idea can be useful for the similarly structured formal documents but may not go well with diverse online text in the TIFU dataset.

These state-of-the-art abstractive methods are not as good as our model, but still perform better than extractive methods. Although the Ext-Oracle heuristic is an upper-bound for extractive methods, it is not successful in our highly abstractive dataset; it is not effective to simply retrieve existing sentences from the source text. Moreover, the performance gaps between abstractive and extractive methods are much larger in our dataset than in other datasets See et al. (2017); Paulus et al. (2018); Cohan et al. (2018), which means too that our dataset is highly abstractive.

Table 4 compares the performance of our MMN on Newsroom-Abs and XSum dataset. We report the numbers from the original papers. Our model outperforms not only the RNN-based abstractive methods but also the convolutional-based methods in all ROUGE scores. Especially, even trained on single end-to-end training procedure, our model outperforms T-ConvS2S, which necessitates two training stages of LDA and ConvS2S. These results assure that even on formal documents with large vocabulary sizes, our multi-level memory is effective for abstractive datasets.

### Qualitative Results

We perform two types of qualitative evaluation to complement the limitation of automatic language metrics as summarization evaluation.

**User Preferences**. We perform Amazon Mechanical Turk (AMT) tests to observe general users' preferences between the summarization of different algorithms. We randomly sample 100 test examples. At test, we show a source text and two summaries generated by our method and one baseline in a random order. We ask turkers to choose the more relevant one for the source text. We obtain answers from three different turkers for each test example. We compare with four abstractive baselines (s2s-att, PG, SEASS and DRGD)

\begin{table}
\begin{tabular}{|c|c c c|c c c|} \hline  & \multicolumn{3}{c|}{**Newsroom-Abs**} & \multicolumn{3}{c|}{**XSum**} \\ \hline Methods & R-1 & R-2 & R-L & R-1 & R-2 & R-L \\ \hline s2s-att & 6.2 & 1.1 & 5.7 & 28.4 & 8.8 & 22.5 \\ PG & 14.7 & 2.2 & 11.4 & 29.7 & 9.2 & 23.2 \\ ConvS2S & - & - & - & 31.3 & 11.1 & 25.2 \\ TMNN (Ours) & **17.5** & **4.7** & **14.2** & **32.0** & **12.1** & **26.0** \\ \hline \end{tabular}
\end{table}
Table 4: Summarization results in terms of ROUGE-1/2/L on Newsroom-Abs Grusky et al. (2018) and XSum Narayan et al. (2018a). Except MMN, all scores are referred to the original papers. T–ConvS2S is the topic-aware convolutional seq2seq model.

\begin{table}
\begin{tabular}{|c|c c|c c c|} \hline  & \multicolumn{3}{c|}{**TIFU-short**} & \multicolumn{3}{c|}{**TIFU-long**} \\ \hline Methods & PPL & R-1 & R-2 & R-L \\ \hline Lead-1 & n/a & 3.4 & 0.0 & 3.3 \\ Ext-Oracle & n/n & 8.0 & 0.0 & 7.7 \\ \hline s2s-att Chopra et al. (2016) & 46.2 & 18.3 & 6.4 & 17.8 \\ PG See et al. (2017) & 40.9 & 18.3 & 6.5 & 17.9 \\ SEASS Zhou et al. (2017) & 62.6 & 18.5 & 6.4 & 18.0 \\ DRGD Li et al. (2017) & 69.2 & 14.6 & 3.3 & 14.2 \\ \hline MMN & 32.1 & **20.2** & **7.4** & **19.8** \\ MMN-NoDilated & **31.8** & 19.5 & 6.8 & 19.1 \\ MMN-NoMulti & 34.4 & 19.0 & 6.1 & 18.5 \\ MMN-NoMGTU & 40.8 & 18.6 & 5.6 & 18.1 \\ \hline \multirow{2}{*}{**TIFU-long**} & \multirow{2}{*}{Lead-1} & n/a & 2.8 & 0.0 & 2.7 \\ Ext-Oracle & n/a & 6.8 & 0.0 & 6.6 \\ \hline s2s-att Chepora et al. (2016) & 180.6 & 17.3 & 3.1 & 14.0 \\ PG See et al. (2017) & 175.3 & 16.4 & 3.0 & 13.5 \\ SEASS Zhou et al. (2017) & 387.0 & 17.5 & 2.9 & 13.9 \\ DRGD Li et al. (2017) & 176.6 & 16.8 & 2.0 & 13.6 \\ \hline MMN & **114.1** & **19.0** & **3.7** & **15.1** \\ MMN-NOMaltted & 124.2 & 17.6 & 3.4 & 14.1 \\ MMN-NOMulti & 124.5 & 14.0 & 1.5 & 11.8 \\ MMN-NonSGIU & 235.4 & 14.0 & 2.6 & 12.1 \\ \hline \end{tabular}
\end{table}
Table 3: Summarization results measured by perplexity and ROUGE-1/2/L on the TIFU-short/long dataset.



and the gold summary (Gold).

Table 5 summarizes the results of AMT tests, which validate that human annotators significantly prefer our results to those of baselines. As expected, the gold summary is voted the most.

**Summary Examples**. Figure 5 shows selected examples of abstractive summarization. Baselines often generate the summary by mostly focusing on some keywords in the text, while our model produces the summary considering both keywords and the whole context thanks to multi-level memory. We present more examples in the Appendix.

## 6 Conclusions

We introduced a new dataset _Reddit TIFU_ for abstractive summarization on informal online text. We also proposed a novel summarization model named _multi-level memory networks_ (MMN). Experiments showed that the Reddit TIFU dataset is uniquely abstractive and the MMN model is highly effective. There are several promising future directions. First, ROUGE metrics are limited to correctly capture paraphrased summaries, for which a new automatic metric of abstractive summarization may be required. Second, we can explore the data in other online forums such as Quora, Stackoverflow and other subreddits.

## Acknowledgments

We thank Chris Dongjoo Kim, Yunseok Jang and the anonymous reviewers for their helpful comments. This work was supported by Kakao and Kakao Brain corporations and IITP grant funded by the Korea government (MSIT) (No. 2017-0-01772, Development of QA systems for Video Story Understanding to pass the Video Turing Test). Gunhee Kim is the corresponding author.

## References

* M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. D. Trippe, J. B. Gutierrez, and K. Kochut (2017)Text Summarization Techniques: A Brief Survey. In arXiv:1707.02268, Cited by: SS3.4.
* J. L. Ba, J. R. Kiros, and G. E. Hinton (2016)Layer Normalization. In Stat, Cited by: SS3.4.
* P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov (2016)Enriching Word Vectors with Subword Information.. In TACL, Cited by: SS3.4.
* A. Celikyilmaz, A. Bosselut, X. He, and Y. Choi (2018)Deep Communicating Agents for Abstractive Summarization. In NAACL-HLT, Cited by: SS3.4.
* S. Chopra, M. Auli, and A. M. Rush (2016)Abstractive Sentence Summarization With Attentive Recurrent Neural Networks. In NAACL-HLT, Cited by: SS3.4.
* F. Coutin, D. Soon Kim, T. Bui, S. Kim, W. Chang, and N. Goharian (2018)A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In NAACL-HLT, Cited by: SS3.4.
* S. Edunov, M. Ott, M. Auli, D. Grangier, and M. Runzato (2017)Classical Structured Prediction Losses for Sequence to Sequence Learning. In NAACL-HLT, Cited by: SS3.4.
* J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin (2017)Convolutional sequence to Sequence Learning. In ICML, Cited by: SS3.4.
* S. Gehrmann, Y. Deng, and A. Rush (2018)Bottom-up abstractive summarization. In EMNLP, Cited by: SS3.4.

Figure 5: Examples of abstractive summary generated by our model and baselines. In each set, we too show the source text and gold summary.

 

Xavier Glorot and Yoshua Bengio. 2010. Understanding the Difficulty of Training Deep Feedforward Neural Networks. In _AISTATS_.
* Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies. In _NAACL-HLT_.
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In _NIPS_.
* Hsu et al. (2018) Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss. In _ACL_.
* Hu et al. (2015) Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. Lcsts: A large scale chinese short text summarization dataset. In _EMNLP_.
* Kaiser et al. (2017) Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. 2017. Learning to Remember Rare Events. In _ICLR_.
* Kedzie et al. (2018) Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content Selection in Deep Learning Models of Summarization. In _EMNLP_.
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In _ICLR_.
* Kumar et al. (2016) Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2016. Ask me Anything: Dynamic Memory Networks for Natural Language Processing. In _ICML_.
* Li et al. (2017) Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. 2017. Deep Recurrent Generative Decoder for Abstractive Text Summarization. In _EMNLP_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In _TSBO_.
* Liu et al. (2018) Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences. In _ICLR_.
* Miao and Blunsom (2016) Yishu Miao and Phil Blunsom. 2016. Language as a Latent Variable: Discrete Generative Models for Sentence Compression. In _EMNLP_.
* Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amiri Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-Value Memory Networks for Directly Reading Documents. In _EMNLP_.
* Na et al. (2017) Seil Na, Sangho Lee, Jisung Kim, and Gunhee Kim. 2017. A Read-Write Memory Network for Movie Story Understanding. In _ICCV_.
* Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. SummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents. In _AAAI_.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive Text Summarization Using Sequence-to-sequence RNNs and Beyond. In _CoNLL_.
* Napoles et al. (2012) Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In _NAACL-HLT AKBC-WEKEEX_.
* Narayan et al. (2018a) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In _EMNLP_.
* Narayan et al. (2018b) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018b. Ranking Sentences for Extractive Summarization with Reinforcement Learning. In _NAACL-HLT_.
* van den Oord et al. (2016a) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016a. WaveNet: A Generative Model for Raw Audio. In _SSW_.
* von den Oord et al. (2016b) Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. 2016b. Conditional Image Generation With Pixelcnn Decoders. In _NIPS_.
* Over et al. (2007) Paul Over, Hoa Dang, and Donna Harman. 2007. DUC in Context. In _IPM_.
* Park et al. (2017) Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. 2017. Attend to You: Personalized Image Captioning with Context Sequence Memory Networks. In _CVPR_.
* Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-Reward Reinforced Summarization with Saliency and Entailment. In _NAACL-HLT_.
* Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A Deep Reinforced Model for Abstractive Summarization. In _ICLR_.
* Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing Neural Networks by Penalizing Confident Output Distributions. In _ICLR_.
* Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In _EMNLP_.
* Salimans and Kingma (2016) Tim Salimans and Diederik P Kingma. 2016. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In _NIPS_.
* Sandhaus (2008) Evan Sandhaus. 2008. New York Times Annotated Corpus. In _LDC_.
* See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the Point: Summarization with PointerGenerator Networks. In _ACL_.

 

Abhishek Kumar Singh, Manish Gupta, and Vasudeva Varma. 2017. Hybrid MemNet for Extractive Summarization. In _CIKM_.
* Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end Memory Networks. In _NIPS_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In _NIPS_.
* Tan et al. (2017) Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive Document Summarization with a Graph-based Attentional Neural Model. In _ACL_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In _NIPS_.
* Wang and Ling (2016) Lu Wang and Wing Ling. 2016. Neural Network-Based Abstract Generation for Opinions and Arguments. In _NAACL-HLT_.
* Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory Networks. In _ICLR_.
* Yoo et al. (2019) Seungjoo Yoo, Hyojin Bahng, Sunghyo Chung, Junsoo Lee, Jaehyuk Chang, and Jaegul Choo. 2019. Coloring with Limited Data: Few-shot Colorization via Memory-Augmented Networks. In _CVPR_.
* Yu and Koltun (2016) Fisher Yu and Vladlen Koltun. 2016. Multi-scale Context Aggregation by Dilated Convolutions. In _ICLR_.
* Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective Encoding for Abstractive Sentence Summarization. In _ACL_.

## Appendix A Implementation Details

All the parameters are initialized with the Xavier method Glorot and Bengio (2010). We apply the Adam optimizer Kingma and Ba (2015) with \(\beta_{1}=0.9,\beta_{2}=0.999\) and \(\epsilon=1e-8\). We apply weight normalization Salimans and Kingma (2016) to all layers. We set learning rate to 0.001 and clip gradient at 0.3. At every 4 epochs, we divide learning rate by 10 until it reaches 0.0001. We train our models up to 12 epochs for TIFUshort and 60 epochs for TIFUlong.

Table 6 summarizes the setting of hyperparameters for our model in all experiments on TIFUshort/long dataset, Newsroom abstractive subset and XSum.

## Appendix B Novel N-gram Ratios

Table 7 compares the ratios of novel N-grams in the reference summary between datasets. Following See et al. (2017); Narayan et al. (2018a), we compute this ratio as follows; we first count the number of N-grams in the reference summary that do not appear in the source text and divide it with the total number of N-grams. The higher the ratio is, the less the identical N-grams are in the source text. The CNN/DailyMail, New York Times, Newsroom datasets all, for example, exhibit low novel 1-gram ratios as 10.3%, 11.0%, 15.6 % respectively. This means that about 90% of the words in reference summary already exist inside the source text. It is due to that the summaries from formal documents (news and academic papers) tend to have same expressions with the source documents. Therefore, these datasets may be more suitable for extractive summarization than abstractive one; on the other hand, our dataset is more abstractive.

We also compare the novel N-gram ratio for XSum and three subsets of Newsroom;

(i) Newsroom-Ext, a subset favorable for extractive methods, (ii) Newsroom-Mix, a subset favorable for mixed methods, and (iii) Newsroom-Abs, a subset favorable for abstrac

\begin{table}
\begin{tabular}{|c|c|c|} \hline
**Description** & \multicolumn{2}{l|}{**Common Configurations**} \\ \hline Initial learning rate & 0.001 \\ Embedding dimension (\(d^{emb}\)) & 300 \\ Kernel size (\(k\)) & 3 \\ Dilation rate (\(d\)) & \(2^{l}\) \\ \hline \hline
**Description** & **TIFUshort** & **TIFUlong** \\ \hline Grad clip & 0.3 & 0.3 \\ \# of encoder layers & 9 & 8 \\ \# of decoder layers & 3 & 5 \\ Layers used for memory (**m**) & \{3, 6, 9\} & \{4, 8\} \\ Smoothing parameter (\(\epsilon\)) & 0.1 & 0.05 \\ \hline
**Description** & **Newsroom-Abs** & **XSum** \\ \hline Grad clip & 0.3 & 0.8 \\ \# of encoder layers & 10 & 9 \\ \# of decoder layers & 6 & 6 \\ Layers used for memory (**m**) & \{3, 6, 10\} & \{4, 7, 9\} \\ Smoothing parameter (\(\epsilon\)) & 0.05 & 0.05 \\ \hline \end{tabular}
\end{table}
Table 6: Model hyperparameters in experiments on TIFUshort/long, Newsroom abstractive subset and XSum.

\begin{table}
\begin{tabular}{|c|c c c c|} \hline  & \multicolumn{4}{c|}{**Novel N-gram Ratio**} \\ \hline Dataset & 1-gram & 2-gram & 3-gram & 4-gram \\ \hline CNN/DailyMail & 10.3 & 49.9 & 70.5 & 80.3 \\ NY Times & 11.0 & 45.5 & 67.2 & 77.9 \\ Newsroom & 15.6 & 45.4 & 57.2 & 62.2 \\ Newsroom-Ext & 1.5 & 5.9 & 8.9 & 11.1 \\ Newsroom-Mix & 11.6 & 47.0 & 66.5 & 76.8 \\ \hline Newsroom-Abs & 33.9 & 83.9 & 97.1 & 99.5 \\ XSum & 35.8 & 83.5 & 95.5 & 98.5 \\ \hline TIFUshort & 29.7 & 71.5 & 88.1 & 93.8 \\ TIFUlong & 27.4 & 76.7 & 92.5 & 97.0 \\ \hline \end{tabular}
\end{table}
Table 7: Comparison of novel N-gram ratios between Reddit TIFU and other summarization datasets.

 



**[Source Text] (...)** We use an internal messaging application software at work which has been great for communicating with other teammates. A lot of us have started using it to complain about things we are not happy about at work. (...) This leads me to today where just as i am about to go home my manager calls me in to a private meeting looking really upset. Then they mentioned the program name and that they had received an email, and suddenly I realized I had fucked up one of the quirks of this program is that when someone is offline it entails them the message. A recently ex co-worker is still active on the chat for quick questions for the next 2 weeks. They came online so we started having a conversation, then another co-worker walked up to me for a chat who has been having a rough week and complained about our boss. When they finished their rant, I then messaged my cx co-worker that my boss wasn't popular with the staff at the moment as that was the second minor complaint I had heard that week. They had gone offline, so an email was sent to their old work email. Past employees can get sent to the boss, in case important emails are sent to them. So after the meeting i still have my job. I had an awkward conversation with my boss (...)

**[Long Summary] (Gold)** message program at work emailed a private message between a past co worker and myself to my boss saying how people where not happy with them

**(Ours)** sent a message to my boss and now i 'm in a meeting with my boss

**(s12*atl)** i label about my boss to get my job and now i 'm in a job with a new job

**(PG)** i got a program that sacked from work and i got sent to a metting by my boss

**Figure 6: Examples of abstractive summary generated by our model and baselines. In each set, we too show the source text and reference summary.



tive methods. We summarize two interesting observations as follows. First, as expected, the more favorable for abstractive methods is, the higher novel n-gram ratio is. Second, novel n-gram ratios of Newsroom-Abs and XSum are higher than those of our dataset, even though their data sources are news publications. Thus, we argue that novel n-gram ratios are pretty good but not a sufficient measure to find extractive bias in the summarization dataset.

## Appendix C More Examples

Figure 6 illustrates selected examples of summary generation. In each set, we show a source text, a reference summary and generated summaries by our method and baselines. In the examples, while baselines generate summary by mostly focusing on some keywords, our model produces summary considering both keywords and the whole context thanks to the multi-level memory.