Title: MEM2MEM: LEARNING TO SUMMARIZE LONG TEXTS WITH MEMORY-TO-MEMORY TRANSFER

Abstract: We introduce the Mem2Mem mechanism, a conditional memory-to-memory mechanism that can be appended to general sequence-to-sequence frameworks, and demonstrate its effectiveness in improving long text neural abstractive summarization. Mem2Mem seamlessly transfers "memories" via readable/writable external memory modules that augment both the encoder and decoder. By enabling a memory transfer, Mem2Mem uses representations of highly salient input sentences and performs an implicit sentence extraction step. By allowing the decoder to read and write over encoded input memories, the models learn to store information about the input sequence while keeping track of what has been generated by the decoder. We evaluate Mem2Mem on abstractive text summarization and surpass the current state-of-the-art with less model capacity than competing models and with a full end-to-end training setup. To our knowledge, Mem2Mem is the first mechanism that can effectively use and update memory cells filled with different contextual information.

1. INTRODUCTION:

Automatic summarization is the automated process of reducing the size of an input text while preserving its most relevant information content and its core semantics. Abstractive text summarization requires a source document to be understood, the most important parts to be prioritized and key concepts to be paraphrased into coherent sentences. Most recent approaches to solving abstractive summarization tasks have relied on the sequence-to-sequence (seq2seq) paradigm (Sutskever et al., 2014) , with an encoder that generates vector representations of the input tokens and a decoder that generates an output text conditioned on the encoded representations. Advances in the conditional Natural Language Generation (NLG) of seq2seq architectures have typically focused on improving the decoder and have been particularly successful when applied to problems such as machine translation (Cho et al., 2014; Bahdanau et al., 2014) , abstractive summarization (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017) and semantic parsing (Dong & Lapata, 2016) .

Several attention-based encoder-decoders were introduced to tackle varying text generation issues of standalone seq2seq models. Gu et al. (2016) proposed a copy mechanism to address difficulties in generating out-of-vocabulary words and named entities. See et al. (2017) used both a pointer mechanism (Vinyals et al., 2015) and a coverage mechanism (Tu et al., 2016) to address repetitive generation of word sequences in abstractive summarization. For very large input documents, however, such attention based techniques can still suffer from uninformative encoding. It is well studied that Recurrent Neural Networks (RNN) trained with stochastic gradient descent have difficulty learning long-term dependencies encoded in the input sequences due to vanishing gradients (Hochreiter, 1998; Pascanu et al., 2013) . While attention mechanisms (Bahdanau et al., 2014; Luong et al., 2015) in the decoder provide more precise conditioning and facilitate gradient flow into the encoder, it is still unclear how trained encoders improve over word embeddings and how linguistic information of the input text is effectively represented (Wieting & Kiela, 2019; Park et al., 2019) . Memory-Augmented Neural Networks (MANN) and Memory-Augmented Encoder-Decoder (MAED) have also been seen as potential solutions to long term dependency issues in RNNs and seq2seqs respectively (Wang et al., 2016; Yogatama et al., 2018; Ma & Príncipe, 2018; Le et al., 2018) . Using differentiable read and write operations to an external module, MAED can represent non-local context of RNNs using its larger memory capacity. Such models are able to store tempo-rally distant information of large input sequences. However, the role in memory for input encoding and sequence generation had not yet been fully understood or exploited.

Geared towards generating concise summaries and focusing summary generation on the most relevant information, alternative ideas introduced an intermediate sentence extraction step. Most recently, hybrid abstractive and extractive architectures (Chen & Bansal, 2018; Subramanian et al., 2019) were proposed and have proven to outperform competing methods on a number of summarization datasets. In such set-ups, a downstream attention based model first selects salient sentences and then rewrites them abstractively without relying on the encoded input representation. However, the two-step approach not only makes strong assumptions that all of the information necessary to generate a summary is contained within a subset of sentences, but it also relies on suboptimal labels (i.e. individual sentence level ROUGE scores) to identify high saliency sentences during the extraction step. Such targets for the extractive step do not necessarily ensure minimal content overlap and maximal ROUGE scores on the whole set of extracted sentences (Narayan et al., 2018) . Furthermore, it is assumed that the performances of the extractive model is independent of the behavior of the abstractive model. In experiments by Chen & Bansal (2018) ; Subramanian et al. (2019) for instance, most if not all of the training cycles were performed separately for the abstractive and extractive models.

In this paper, we propose a memory based end-to-end approach trained from scratch that implicitly performs an extraction step without needing extraction labels. The architecture, that is called Mem2Mem, was inspired by the reconsolidation hypothesis, which states that the incorporation of new information can happen when memory is recalled in the brain (Sara, 2000; Coccoz et al., 2011) . To our knowledge, Mem2Mem is the first MAED that creates a long term memory of the encoded input and that allows the decoder to modify its "memories" during the summary generation process. We empirically demonstrate the merits of this approach by setting a new state of the art on long text abstractive summarization results on the Pubmed dataset (Cohan et al., 2018) . Our results show large improvements on all ROUGE scores, especially on ROUGE-L, the longest common subsequence and "a means of assessing fluency" (Narayan et al., 2018) over competing models. We further provide evidence that our new memory-based technique performs an implicit extraction step by choosing sentence representations to add to memory and to use during summary generation. Our contributions are three folds:

1. We introduce a novel mechanism, called Mem2Mem, to transfer memories created during the encoding process to the decoder, that is able to read and modify memory contents. 2. We show that Mem2Mem performs an implicit information retrieval step akin to extractive summarization, without needing extraction labels. 3. We achieve state-of-the art results with a fully differentiable model trained end-to-end.

2. BACKGROUND:

The methodology proposed in this paper can be attached to general seq2seq models that have either RNN or Convolutional encoder/decoders. Since we are interested in solving very long text summarization, we adopt a baseline seq2seq model based on the hierarchical recurrent encoder-decoder structure (HRED) from Nallapati et al. (2016) and Cohan et al. (2018) .

The hierarchical recurrent encoder has two encoder GRUs: a sentence encoder and a document encoder. Given an input sentence of length N , the sentence encoder takes a sequence of token embeddings x and transforms it into a sequence of hidden states.

h (w) 1 , h (w) 1 , . . . h (w) N = GRU sen (x 1 , x 2 , . . . , x N ) (1)

The last hidden state of the sentence encoder is used as a corresponding sentence embedding s. Note that the sentence encoder is shared across every sentence in the input document. The sequence of sentence embeddings s are then processed by the document encoder.

h (s) 1 , h (s) 1 , . . . h (s) L = GRU doc (s 1 , s 2 , . . . , s L ) (2)

where s j is the j th sentence embedding, h

j is the associated document encoder hidden state and L is the total number of sentences in the document.

The decoder GRU generates a target summary one token at a time. At each decoding step t, the decoder creates a decoder hidden state h (d) t . The decoder then obtains the context vector c t :

c t = N d i=1 γ ti h (w) i (3)

where γ ti is the attention weight that represents the alignment between h (w) i and h (d) t . γ ti is computed by combining token level attention α and sentence level attention β.

EQUATION

EQUATION

EQUATION

m(l) denotes the index of the sentence corresponding to the l th word, and N d is the total number of tokens in the input document. Attn in equation ( 4) and ( 5) is defined as in Bahdanau et al. (2014) :

EQUATION

Finally, the pointer generator and decoder coverage method in See et al. (2017) are used for the baseline HRED. More details of the baseline architecture can be found in Appendix.

3. CONDITIONAL MEMORY-TO-MEMORY MECHANISM (MEM2MEM):

In this section, we describe the external memory modules that extend the baseline HRED to create Mem2Mem. Mem2Mem has three main features. (1) An encoder memory bank that compresses a large set of encoder hidden representations into a smaller subset of mutually exclusive representations.

(2) Read/write operations that allow the decoder to update the encoder memory bank with new information during summary generation.

(3) Word generation that is conditioned on the encoded sentence representations accessed from the dynamic memory. In essence, the whole process can be seen as extraction followed by generation. The architecture is depicted in Figure 1 .

3.1. NEURAL ENCODER MEMORY BANK:

The aim of having an external memory on the encoder is to create a fixed size representation that reduces the set of L hidden representations from GRU doc to a subset of r representations. From a sequence of sentence-level document encoder hidden representations h

EQUATION

L , we construct an intermediate 2-D matrix H ∈ R L×d , where d is the document encoder hidden size.

H =     h (s) 1 h (s) 2 . . . h (s) L     (8)

A smaller sized memory bank is generated by taking a linear combination of each row in H. The weight vector for the linear combination a is computed with self-attention mechanism.

EQUATION

where w a1 ∈ R da , W a2 ∈ R da×d , and d a is the size of the hidden layer which is a hyperparameter.

To capture various aspects of the input document, a is extended to a multi-headed attention matrix A with r heads.

EQUATION

where W a1 ∈ R r×da and r is a hyperparameter. This results in r different weighted sums of H, which gives us the final multi-headed encoder memory matrix M (e) ∈ R r×d .

M (e) = AH (11) Lin et al. ( 2017) used a similar approach to create multiple types of sentence embeddings for one particular sentence with a "structured self-attentive sentence embedding" matrix. Different from their work, we use the external memory block with r number of slots to store a smaller set (much less than L) sentence representations. We further integrate the memory module onto the baseline HRED architecture. Additionally, in our approach, the memory representation M gets updated dynamically by the decoder while the "structured self-attentive sentence embedding" matrix remains static.

To ensure that attention weights of different heads are not redundant, a regularization term P is added to the cross entropy training objective to encourage diversity of memorized encoded states. Similar to Lin et al. (2017) , we use the following regularization loss to constrain the attention weight matrix A:

P = ||(AA T -I)|| F 2 (12)

where || • || F is the Frobenius norm. The regularization loss P achieves two goals simultaneously:

(1) It promotes diversity over the r representations stored in memory, thereby reducing the risk of storing redundant information.

(2) It "hardens" the attention probabilities by reducing variance across the categorical attention distribution, thereby assuring that each memory slot is associated to approximately one sentence representation. As a result, the encoder memory M (e) essentially performs implicit extraction over the encoder hidden states. Note that no supervision exists on this extractive step. In Section 5, we will demonstrate that the encoder memory bank learns to choose amongst a large set of input sentences representations, using only the back-propagated error signals from the target summary generation objective as a guide.

3.2. READABLE/WRITABLE DECODER MEMORY:

Most MAED architectures are not equipped with a Decoder memory but only query the Encoder Memory. Our MAED stores decoder hidden states with a write operation that is different than the one described in the previous section 3.1. As a first step, the encoder memory matrix M (e) is transferred to the decoder and used as an initial state of the decoder memory M (d) . At every time step t, the decoder reads from the memory and generates a memory read vector m t . Specifically, the decoder takes the weighted sum over r memory slots via a RAM like attention mechanism:

EQUATION

m t = r k=1 ψ tk M (d) k (14)

where M

(d)

k is the vector representation of the k th slot of M (d) . To condition the decoder attention mechanism on the memory read, h

(d) t

is combined with m t and generate a memory augmented decoder hidden state h

EQUATION

h (d) t

in the equation ( 4) and ( 5) of the baseline system is then replaced with h (m) t

. Similarly, the probability distribution of the next target word y t is estimated using the memory augmented decoder hidden state h (m) t and the context vector c t , such that:

EQUATION

As a consequence, the attention over the source text and the prediction of the target token are conditioned on the memory read m t , a weighted sum of the memorized sentence representations. There is a direct link between the contents of the memory and text generation. The training loss L is the average negative log likelihood of the target word y t over the whole ground truth summary of length T . The semantics of the source sequence that is kept in the decoder memory can also be modified during the conditional text generation process. The decoder memory write operation outlined below removes and adds information using a gated mechanism to "forget" past memories and "update" each memory slot. Essentially, the memory write operation enables the memory to log the history of what has been attended and generated. The gating mechanism is conditioned on the the memory content M (d) , the sentence level context vector c (s)

t , and the decoder hidden state h

EQUATION

)

EQUATION

EQUATION

A similar memory write mechanism, called scratchpad, has been proposed in Benmalek et al. (2019) . The write operation updates all encoder hidden states given the decoder hidden state and the context vector. In our work, we condition updates on memory-augmented context and content. Moreover, the scratchpad mechanism scales linearly with the input document length and output summary length. To be more precise, the complexity of scratchpad is O(N T ) where N and T are the length of the input text and the target summary respectively. Our Mem2Mem, however, is O(rT ) where r is the number of memory slots. Given that r is a fixed hyperparameter and typically r N , our method is an ideal candidate for large output text generation conditioned on even larger input documents.

4. RELATED WORK:

Memory based models have been used in several NLG tasks. Sukhbaatar et al. (2015) , using a continuous memory representation similar to Graves et al. (2014) 's Neural Turing Machines, show the importance of allowing multiple reads and writes to memory between inputs in language modeling experiments. Yogatama et al. (2018) further improved on such experiments by using an LSTM language model equipped with a multihop adaptive continuous stack memory. When experimenting with the multihop adaptive continuous stack memory for abstractive summarization in a HRED context1 , we have found two issues (1) only the top 2 slots of the encoder's stack memory were filled (2) the decoder memory stack rarely used or mixed memories transferred from the encoder.

Other MAED have been proposed for conditional NLG, such as machine translation (Kaiser et al., 2017) , image captioning (Park et al., 2017) , QA (Sukhbaatar et al., 2015; Kumar et al., 2015; Miller et al., 2016; Na et al., 2017; Han et al., 2019) . In summarization, MAED have been used on extractive summarization (Singh et al., 2017) and on short text abstractive summarization (Kim et al., 2018) . These approaches excel at tasks where it is necessary to store some parts of a sequential input in a representation that can later be precisely queried. We found however that most MAED innovations focused on improving long term memories of the encoded input or read/write operations. In this work, we suggest solutions in these areas: (1) a memory creation mechanism that condenses the encoded input akin to an extraction step; (2) a conditional read/write mechanism that allows the decoder to use/update encoder memories. Further, we propose another feature -Mem2Mem transfers memories from different context. The encoder memory stores input sentence-level information while the decoder updates are conditioned on output word-level information.

5. RESULTS AND DISCUSSION:

We evaluate Mem2Mem on PubMed (Cohan et al., 2018) which is a large scale summarization dataset comprised of biomedical literature. The average lengths of source documents and target summaries are 3016 and 203 respectively. In comparison, the widely used CNN/DailyMail summarization dataset (Hermann et al., 2015; See et al., 2017) is almost 4 times smaller with average lengths of source documents and target summaries at 781 and 56 respectively. Our pre-processing and training details are identical to Cohan et al. (2018) . The readers are encouraged to refer to the appendix for training and evaluation details. For the quantitative evaluation, we use the ROUGE metric (Lin, 2004) and report full-length F-1 ROUGE scores.

Table 1 recaps the ROUGE scores of Mem2Mem along with a baseline model and previous works. We first note that the baseline HRED outperforms similar hierarchical recurrent encoder summarization model suggested in Cohan et al. (2018) 2 . The results also show that Mem2Mem surpasses the state of the art for abstractive summarization on the PubMed dataset. Mem2Mem even outperforms the hybrid extractive-abstractive TLM-I+E (Subramanian et al., 2019) on all scores except for ROUGE-1. We may say that Mem2Mem is much more "fluent" than TLM-I+E as evidence by the 13.2 percentage point gain in ROUGE-L scores. Interestingly, Mem2Mem achieves such performances over TLM-I+E while needing 6 times less parameters (14.7M vs. 93.3M). TLM-I+E also benefits from a language modeling step on its input text that can be considered a form of preprocessing. We train our model from scratch. Finally, we reiterate that Mem2Mem is trained completely end-to-end while TLM requires separate training for the extractive model and the conditional transformer language model.

5.1. ABLATION STUDY:

To assess the importance of different components in Mem2Mem, we conduct an ablation study on PubMed dataset results. Table 2 shows the affects of adding various Mem2Mem features on ROUGE scores . Encoder Mem is the baseline HRED augmented with the neural encoder memory bank described in Section 3.1. The result demonstrates that the memory context indeed enhances the quality of generated summaries. Furthermore, it can be observed that discouraging redundancies in the encoder memory cells via the regularization loss P leads to additional improvements on all ROUGE scores. Figure 2 shows the effect of regularization on the encoder memory selection of encoded sentence representations. We observe that each memory head attends to a single sentence of the Table 2 : Model ablation study on the PubMed dataset. With Encoder Mem, the baseline HRED is augmented with a neural encoder memory bank as described in Section 3.1. We then regularize the encoder memory, Regularize Mem, using the equation ( 12). We also experiment with the Decoder Mem that performs read/write mechanism described in Section 3.2 without memory transfer. Finally, with Mem Transfer, we obtain the whole Mem2Mem setup.

input. This is due to the regularization term P that encourages sparse, non-redundant attention distributions over different memory heads or slots. Analysing multiple test samples, a trend emerges.

The encoder memory typically attends to first and last few encoded sentences representations. This aligns with known statistic that existing summarization datasets tend to have key information at the front (i.e. the introduction) and last parts (i.e. the conclusion) of the document (Kim et al., 2018) . Decoder Mem is a writable decoder memory but without memory transfer. In this case, the encoder memory is no longer used to initialize the decoder memory state at the start of the summary generation process. The decoder memory is simply initialized with zeros. With over one percentage point increases in both ROUGE-1 and ROUGE-L scores, the result shows that read/write mechanism on the decoder brings substantial improvements even without leveraging encoded sentence representations. This shows that the summary generation process largely benefits from accessing long term contextual information of the output text. The sparsity of the chosen content as seen Figure 2 (a) encourages us to think that presenting the decoder with a small subset of input sentence information helps summary generation in similar way to models that use a hybrid extractive-abstractive approach. Adding the memory transfer further improves ROUGE scores, which could imply that the soft extractive step of the encoder memory remains effective even with write operations on the decoder.

5.2. IMPLICIT EXTRACTION IN THE ENCODER MEMORY:

Our initial hypothesis was that the neural encoder memory bank would automatically pick a subset of the most salient sentences for a given task. To confirm this hypothesis, we analyze the quality of the sentences chosen by the encoder memory. As seen in equation ( 16), the Markovian probability of output tokens is conditioned on both the previous token and the decoder memory read m t . In Mem2Mem, the memory content is initialised with a sparse set of encoded sentence representations (see Figure 2 ). As explained previously, the encoder memory attention A is the mechanism used to choose a subset of sentence representations. To better understand the link between encoder memory creation and the summary generation process, we compute the ROUGE scores of input sentence tokens that had above 80% probability encoder memory attention A. That is, we calculated in Table 3 the ROUGE scores of input sentences whose representation was the most likely stored in the encoder memory bank.

Table 3 shows the ROUGE scores of different unsupervised extractive summarization methods on the PubMed dataset. The summaries generated with extracted sentences outperform existing unsupervised extractive summarization baselines on Pubmed. The result demonstrates that the encoder memory bank is able to prioritize amongst a large set of input sentences.

6. CONCLUSION:

In this paper, we proposed a novel framework called Mem2Mem, which augments existing hierarchical seq2seq architectures and validated its efficacy on abstractive summarization task. From a long source document, the encoder memory learns to locate salient information without explicit labels for the extraction. The decoder memory, which is transferred from the encoder memory, enhances the quality of generated summaries by means of memory-augmented attention and hidden representations. The proposed Mem2Mem established new state of the art results on the challenging long-text summarization PubMed dataset. We also provided in-depth analyses of different features of Mem2Mem and show that the decoder memory along with memory transfer has the largest impact on performance. The Mem2Mem's notion of implicit extraction followed by generation can be generalized to a wide range of seq2seq tasks such as question generation and answering. In future work, we would like to extend this approach and validate the strength of memory transfer on a variety of sequential learning tasks.

A APPENDIX:

A.1 BASELINE ARCHITECTURE

In addition to the hierarchical recurrent encoder-decoder (HRED) architecture described in Section 2, following features are used for both baseline and Mem2Mem architecture.

A.1.1 POINTER GENERATOR NETWORK

In order to handle out-of-vocabulary (OOV) token predictions, pointer generator in See et al. (2017) is used to copy words directly from the input document. At each step t, the decoder decides whether to predict the next target word from the generation the pointer generator computes z t which denotes the probability of choosing P g for sampling the next target word.

EQUATION

The probability z t is used as a variable for soft switch between generating a word from the vocabulary (P g ) or directly copying from the source document (P c ). The probability of copying a word w from the source text is calculated based on the attention weights γ.

P c (y t = w|y 1:t-1 ) = i:xi=w

EQUATION

Note that P c (y t = w|y 1:t-1 ) = 0 if w does not exist in the source document. Likewise, P g (y t = w|y 1:t-1 ) = 0 if w is an out of vocabulary word. Combining two probability distributions, the final probability of the next word y t being w is as follows.

P (y t = w|y 1:t-1 ) = z t P g (y t = w|y 1:t-1 ) + (1 -z t )P c (y t = w|y 1:t-1 )

A.1.2 DECODER COVERAGE It is well known that sequence-to-sequence models tend to suffer from repeated phrases when generating long target sequences. See et al. (2017) tackled this issue by keeping track of the attention coverage. More concretely, the coverage vector cov t at decoding step t is computed by taking the summation of the token-level attention weights α until the last step t -1.

EQUATION

To inform the decoder of the history of attention weights, the coverage vector is fed into the tokenlevel attention mechanism, which modifies the equation (4) to the following equation.

EQUATION

A.2 TRAINING DETAILS For pre-processing, we constrain the maximum number of sections to 4 and the maximum number of tokens for each section to 500. The length of the target summary is limited to 200 tokens.

Single-layer bidirectional GRUs (Cho et al., 2014) are used for sentence and document encoders.

The decoder is also a single layer GRU. All GRUs have hidden size of 256. The dimensionality of token embeddings is 128 and embeddings are trained from scratch. The vocabulary size is limited to 50,000. Batch size is 16 and Adam (Kingma & Ba, 2014) with learning rate 2e -4 is used for training. Maximum gradient norm is set to 2. We train all models for 15 epochs. At the test time, beam search with the beam size 4 is used for decoding.

For Mem2Mem hyperparameters, the number of heads for the encoder memory module is 10 and the self-attention hidden size is 64. The weight for the regularization P is 0.01.

:

We used a GRU with a stack memory on the sentence encoder and decoder. More details can be found in the appendix.

The difference may arise from employing a different hierarchical structure: Ours used the word-sentence hierarchy whileCohan et al. (2018) used the word-section discourse. We also trained more epochs (15) than their work (10) to build a strong baseline model.

Figures and Tables:

Figure 1: The proposed Mem2Mem architecture for abstractive summarization. Sentence-level representations from the document encoder hidden states are concatenated and reduced to a fix sized memory M0. Mi is read using a RAM like mechanism. The memory readout vector is used to condition word-level attention. The decoder hidden states modulated by word-level attention is then used to update the memory state Mi+1 via a gated write operation (z i ).

Figure 2: An example of encoder memory attention A. Rows denote different memory slots and columns indicate input sentence representations with indices. Note that the regularization loss removes the redundancy on different memory slots and helps each slot to focus on a single sentence.

Results on the PubMed dataset. TLM uses a GPT-like transformer language models (Radford et al., 2019) conditioned on introduction (I) and with extracted sentences (E). The highest ROUGE scores for abstractive methods are bold-faced. Hybrid refers to models that use two-step extractive-abstractive summarization.

Table content: <table><tr><td>Model</td><td>Type</td><td># of params.</td><td>1</td><td>ROUGE 2 3</td><td>L</td></tr><tr><td>SumBasic</td><td>Extractive</td><td>-</td><td colspan="3">37.15 11.36 5.42 33.43</td></tr><tr><td>LexRank</td><td>Extractive</td><td>-</td><td colspan="3">39.19 13.89 7.27 34.59</td></tr><tr><td colspan="2">Ptr-Gen-Seq2Seq (Cohan et al., 2018) Abstractive</td><td>-</td><td colspan="3">35.86 10.22 7.60 29.69</td></tr><tr><td colspan="6">Discourse-aware (Cohan et al., 2018) Abstractive 14.3M 38.93 15.37 9.97 35.21</td></tr><tr><td>TLM-I (Subramanian et al., 2019)</td><td colspan="5">Abstractive 71.2M 36.63 11.67 5.40 21.00</td></tr><tr><td>TLM-I+E (Subramanian et al., 2019)</td><td>Hybrid</td><td colspan="4">93.3M 41.43 15.89 8.62 24.32</td></tr><tr><td>Baseline HRED</td><td colspan="5">Abstractive 14.0M 40.02 15.82 9.30 36.28</td></tr><tr><td>Mem2Mem (our model)</td><td colspan="5">Abstractive 14.7M 41.35 16.09 9.00 37.53</td></tr><tr><td>Model</td><td>1</td><td>ROUGE 2 3</td><td>L</td><td/><td/></tr><tr><td>Baseline HRED</td><td colspan="3">40.02 15.82 9.30 36.28</td><td/><td/></tr><tr><td>+ Encoder Mem</td><td colspan="3">40.39 16.00 9.39 36.58</td><td/><td/></tr><tr><td colspan="4">+ Regularize Mem 40.72 16.06 9.53 36.90</td><td/><td/></tr><tr><td>+ Decoder Mem</td><td colspan="3">41.09 16.00 9.02 37.34</td><td/><td/></tr><tr><td>+ Mem Transfer</td><td colspan="3">41.35 16.09 9.00 37.53</td><td/><td/></tr></table>

Rouge scores of unsupervised extracted input sentences with respect to the ground truth summaries. For Mem2Mem Encoder Memories, we used the sentences that had encoder memory attention A above 80%. Gold Ext is the gold extracted ROUGE scores of all sentences selected by a greedy selection from the input article that have the highest per-sentence ROUGE scores, as inSubramanian et al. (2019).

Table content: <table><tr><td>Model</td><td>1</td><td>ROUGE 2 3</td><td>L</td></tr><tr><td>Lead-10</td><td colspan="3">37.45 14.19 8.26 34.07</td></tr><tr><td>LexRank</td><td colspan="3">39.19 13.89 7.27 34.59</td></tr><tr><td colspan="4">Mem2Mem Encoder Memories 40.11 15.56 8.99 31.90</td></tr><tr><td>Gold Ext</td><td colspan="3">47.76 20.36 11.52 39.19</td></tr></table>

Peer review (iclr2020):
This paper introduces a novel way of augmenting a seq2seq model by an external memory module. The obtained system achieves promising results on the PubMed datasets. Although the paper is clear-written and the idea is well-received, I found the idea proposed is relatively incremental, given the advances brought by the end-to-end memory network. The experimental improvement also seems a bit marginal, compared to the chosen baseline model. The ablation study is necessary for architecture like this. The result of the regularization term is interesting. In theory, the architecture can be plugged in any seq2seq system. Would this technique help to enhance other tasks, such as machine translation or other? With that being said, I*d recommend trying adding the memory module to other architectures and experiment it on tasks with more variety.