Title: Efficient Summarization with Read-Again and Copy Mechanism
Authors: Wenyuan Zeng
Wenjie Luo
Sanja Fidler
Raquel Urtasun
Venue: ICLR 2017

Abstract: Encoder-decoder models have been widely used to solve sequence to sequence prediction tasks. However current approaches suffer from two shortcomings. First, the encoders compute a representation of each word taking into account only the history of the words it has read so far, yielding suboptimal representations. Second, current models utilize large vocabularies in order to minimize the problem of unknown words, resulting in slow decoding times and large storage costs. In this paper we address both shortcomings. Towards this goal, we first introduce a simple mechanism that first reads the input sequence before committing to a representation of each word. Furthermore, we propose a simple copy mechanism that is able to exploit very small vocabularies and handle out-of-vocabulary words. We demonstrate the effectiveness of our approach on the Gigaword dataset and DUC competition outperforming the state-of-the-art.


Review discussion:
What is the difference between the copy mechanism in this paper and the other baselines‘? Have you compared the baseline models (with copy mechanism) with your model on the smaller vocabularies?
Thank you for the valuable comments and questions.
->What is the difference between the copy mechanism in this paper and the other baselines‘?
Vinyals et al. (2015) use a pointer network to copy entities from the input, but they cannot predict an entity out of the input sequence. Gulcehre et al. (2016); Nallapati et al. (2016) use a gating mechanism to leverage two modes, either copy a word from the input or generate a word from a predefined dictionary. Towards this goal, they train different classifiers for each mode and another MLP gating network to combine them. Different from their gating mechanism, we directly apply a softmax layer over all possible candidate words to select the appropriate word. This includes words in the predefined dictionary as well as words in the input sequence. The difference compared to Gu et. al. (2016) is that we equipped our decoder with a mechanism to extract the meaning of copied words from their context in the input text, represented by vectors p_1, p_2, ..., p_n defined in Eq (15). We argue that this will further help the decoder to generate fluent summaries.
->Have you compared the baseline models (with copy mechanism) with your model on the smaller vocabularies?
We will conduct the experiments and post them in a revised version.
-". Gulcehre et al. (2016); Nallapati et al. (2016) add a hard gate to allow the model to decide whether generate a target
word from the fixed-size dictionary or from the input sequence." What do you mean by hard-gating here? If they use discrete actions, how did they train it?
- Can you add more explanations and the description to the caption of Figure 1?
- Have you done exhaustive and contrastive comparisons against deep stacked LSTMs or bidirectional encoders?
- Do you have visualizations of the attention of the model, how do they compare against to the model without copying mechanism?
- Have you checked/tested the embeddings learned by the decoder? E.g. TSNE visualizations, analogy-making ...etc.
Thank you for the valuable comments and questions.
(1) It’s not a discrete action. We use ‘hard-gating’ here simply to point out that they use explicit gating mechanism to leverage two distinct modes, as compared to softmax in Gu et al. (2016) with no gating involved.
However, the gating they use is still a continuous function. We’ll clarify this in the next version.
(2) Fig1 (a) shows an overview of our model. Both encoder1 and encoder2 take the same sentence as input.
Further, encoder2 takes the hidden vectors from encoder1 as addional input.  The decoder generates a summary with both attention and our copy mechanism as shown in Fig1 (b). In this figure, c_t is a weighted average of hidden vectors from encoder2, i.e. h_1^2, h_2^2, …, h_n^2 corresponding to all input words. As shown in Fig 1 (b), the word embedding we use for generating summaries could be either ‘Copied Word Embedding’ or ‘Decoder Word Embedding’. The 'Copied Word Embedding' are vectors p_1, p_2, ..., p_n defined in Eq (15), and ‘Decoder Word Embedding’ are learnable parameters for each word in a pre-defined dictionary. y_{t-1} in Fig 1 (b) is the word embedding of the previously generated word during decoding. Notice y_{t-1} = p_i if the previous word is copied from i^{th} input word, or equal to ‘Decoder Word Embedding’ as in normal sequence to sequence tasks if it’s not copied.
(3) We want to point out that our contribution is orthogonal to deep stacked LSTMs and bidirectional encoders, as we can use our approach with these two types of architectures. What we propose is a self-attention mechanism, where the first read is used to bias the second read. The key point here is to be able to filter out less important words and generate a more relevant encoding vector. We will add this to the next version.
(4)  We use two forms of attention in our model.
First, our novel self-attention mechanism is used in the encoder to enable the encoder to attentively select important words during encoding. This is visualized in Fig 4. The second attention, which is used in the decoder is the same as Bahdanau et al. (2014).
(5) No, we haven't, but this is a very interesting experiment which we will add to a revised version.
This work explores the neural models for sentence summarisation by using a read-again attention model and a copy mechanism which grants the ability of direct copying word representations from the source sentences. The experiments demonstrate the model achieved better results on DUC dataset.
Overall, this paper is not well-written. There are confusing points, some of the claims are lack of evidence and the experimental results are incomplete.
Detailed comments:
-Read-again attention. How does it work better than a vanilla attention? What would happen if you read the same sentences multiple times? Have you compared it with staked LSTM (with same number of parameters)? There is no model ablation in the experiment section.
-Why do you need reading two sentences? The Gigaword dataset is a source-to-compression dataset which does not need multiple input sentences. How do you compare your model with single sent input and two sent input?
-Copy mechanism. What if there are multiple same words appeared in the source sentences to be copied? According to equation (5), you only copy one vector to the decoder.
However, there is no this kind of issue for a hard copy mechanism.
Besides, there is no comparison between the hard copy mechanism and this vector copy mechanism in the experiment section
-Vocabulary size. This part is a bit off the main track of this paper. If there is no evidence showing this is the special property of vector copy mechanism, it would be trivial in this paper.
-Experiments. On the DUC dataset, it compares the model with other up-to-date models, while on the Gigaword dataset paper only compares the model with the ABS Rush et al. (2015) and the GRU (?), which are quite weak baseline models. It is irresponsible to claim this model achieved the state-of-the-art performance in the context of summarization.
Typos: (1) Tab. 1. -> Table 1. (2) Fig. 3.1.2.?
Thank you for the valuable comments and questions. Here are some rebuttals.
- Read-Again attention:
There are two different and orthogonal attention mechanism in our model. The first one models attention between the decoder and the encoder, as commonly used in sequence to sequence models, and might be what you referred to as vanilla attention. The second attention mechanism we use is our Read-Again attention, which is orthogonal and thus cannot be compared.  One could use multiple passes of reading, but that would be expensive.
Instead, we expect the first pass to capture enough global information about the full sentence.
- Two sentences:
Gigaword is actually a document to sentence summarization dataset.
However, previous work was mostly concerned with compressing the first sentence in a document to a shorter summary, and thus they cut off the remaining part of document in their experiments. Following this convention, we first evaluate our models on the task of summarizing the first sentence. We also demonstrated that our model could utilize information from more than 1 sentence, achieving better results when feeding 2 sentences rather than one.
- Copy mechanism:
The copy anchor (output-input copied word alignment) is constructed if a word in the output is Out of Vocabulary (OOV) for the decoder but it appears in the input.
When OOV words appear multiple times, we use its first appearance in the input
Experiments:
Previous state-of-the-art work is evaluated on an internal testing set.
However, this set and the source code are not openly available at the time we submitting this paper, and thus it’s hard to directly compare with previous work on Gigaword.
Therefore, we first compare our Read-Again attention with GRU baseline, which is a fair comparison with our models, to testify the effectiveness of our attention mechanism. We then conduct the comparison of our model with state-of-the-art models on the standard DUC summarization task.
Typos:
Thank you for pointing them out. We’ll fix them.
This paper proposed two incremental ideas to extend the current state-of-the-art summarization work based on seq2seq models with attention and copy/pointer mechanisms.
1. This paper introduces 2-pass reading where the representations from the 1st-pass is used to  re-wight the contribution of each word to the sequential representation of the 2nd-pass. The authors described how such a so-called read-again process applies to both GRU and LSTM.
2. On the decoder side, the authors also use the softmax to choose between generating from decoder vocabulary and copying a source position, with a new twist of representing the previous decoded word Y_{t-1} differently. This allows the author to explore a smaller decoder vocabulary hence led to faster inference time without losing summarization performance.
This paper claims the new state-of-the-art on DUC2004 but the comparison on Gigaword seems to be incomplete (missing more recent results after Rush 2015 etc).
While the overall work is solid, there are also other things missing scientifically.
For example,
- how much computational costs does the 2nd pass reading add to the end-to-end system?
- How does the decoder small vocabulary trick work without 2nd-pass reading on the encoder side for both summarization performance and runtime speed?
- There are other ways to improve the embedding of a sentence. How does the 2nd-pass reading compare to recent work from multiple authors on self-attention and/or LSTMN?
For example, Cheng et al. 2016, Long Short-Term Memory-Networks for Machine Reading; Parikh et al. 2016, A Decomposable Attention Model for Natural Language Inference?
Thank you for the valuable comments and questions. Here are some rebuttals.
- Experiments on Gigaword:
Previous state-of-the-art work is evaluated on an internal testing set.
However, this set and the source code are not openly available at the time we submitting this paper, and thus it’s hard to directly compare with previous work on Gigaword. We have to compare our model with our self-implemented baselines on a randomly sampled testing set, and then finish the comparison with other state-of-the-art models on DUC2004.
- Computation Cost:
The training time of vanilla GRU is around 2 days while our Read-Again GRU takes less than 3 days. More specifically, it takes 0.20s per minibatch for a GRU  and 0.27s for a Read-Again GRU. Both models achieve the best validation perplexity after running for 10 epochs over the whole training set.
- Small vocabulary:
We argue that the copy mechanism is orthogonal to the read-again mechanism. We empirically observe in our experiments that our copy mechanism consistently improves the model performance with or without read-again mechanism.
- Other work:
Thanks for pointing out to additional papers. The main idea of the self-attention is to expand the memory capacity in vanilla LSTM, while the output of the model during each step is still only dependent on the past history. Our attention mechanism tries to address this issue, and utilize the information from both past and future words (in a self-attention maner differed from bi-direction GRU).
Summary: This paper proposes a read-again attention-based representation of the document with the copy mechanism for the summarization task. The model reads each sentence in the input document twice and creates a hierarchical representation of it instead of a bidirectional RNN. During the decoding, it uses the representation of the document obtained via the read-again mechanism and points the words that are OOV in the source document. The model does abstractive summarization. The authors show improvements on DUC 2004 dataset and provide an analysis of their model with different configurations.
Contributions:
The main contribution of this paper is the read-again attention mechanism where the model reads the same sentence twice and obtains a better representation of the document.
Writing:
The text of this paper needs more work. There are several typos and the explanations of the model/architecture are not really clear, some parts of the paper feel somewhat bloated.
Pros:
- The proposed model is a simple extension to the model to the model proposed in [2] for summarization.
- The results are better than the baselines.
Cons:
- The improvements are not that large.
- Justifications are not strong enough.
- The paper needs a better writeup. Several parts of the text are not using a clear/precise language and the paper needs a better reorganization. Some parts of the text is somewhat informal.
- The paper is very application oriented.
Question:
- How does the training speed when compared to the regular LSTM?
Some Criticisms:
A similar approach to the read again mechanism which is proposed in this paper has already been explored in [1] in the context of algorithmic learning and I wouldn’t consider the application of that on the summarization task a significant contribution.  The justification behind the read-again mechanism proposed in this paper is very weak. It is not really clear why additional gating alpha_i is needed for the read again stage.
As authors also suggest, pointer mechanism for the unknown/rare words [2] and it is adopted for the read-again attention mechanism.
However, in the paper, it is not clear where the real is the gain coming from, whether from “read-again” mechanism or the use of “pointing”.
The paper is very application focused, the contributions of the paper in terms of ML point of view is very weak.
It is possible to try this read-again mechanism on more tasks other than summarization, such as NMT, in order to see whether if those improvements are
The writing of this paper needs more work. In general, it is not very well-written.
Minor comments:
Some of the corrections that I would recommend fixing,
On page 4: “… better than a single value … ” —> “… scalar gating …”
On page 4: “… single value lacks the ability to model the variances among these dimensions.” —> “… scalar gating couldn’t capture the ….”
On page 6: “ … where h_0^2 and h_0^'2 are initial zero vectors … “ —> “… h_0^2 and h_0^'2 are initialized to a zero vector in the beginning of each sequence …"
There are some inconsistencies for example parts of the paper refer to Tab. 1 and some parts of the paper refer to Table 2.
Better naming of the models in Table 1 is needed.
The location of Table 1 is a bit off.
[1] Zaremba, Wojciech, and Ilya Sutskever. "Reinforcement learning neural Turing machines." arXiv preprint arXiv:1505.00521 362 (2015).
[2] Gulcehre, Caglar, et al. "Pointing the Unknown Words." arXiv preprint arXiv:1603.08148 (2016).
Thank you for the valuable comments and questions. Here are some rebuttals.
- How does the training speed when compared to the regular LSTM?
We compare the training time of our read-again model with our GRU baseline. A vanilla GRU encoder-decoder architecture takes 0.2s per minibatch, while our Read-Again GRU encoder-decoder takes 0.27s for the same minibatch.
- Justification of Read-Again
Vanilla RNN calculates the representation of each word conditioned only on the past history (words). This might be sub-optimal especially when the input sequence is long and future words also bring important information. This is indeed the case of summarization where the input text is lengthy.  Our Read-Again mechanism tries to address this issue by building the representation of each world once the full input text has been read.
For example, the gating alpha_i depends not only in the current world but also in future words. We plan to generalize our model to perform document-level summarization. We expect and even larger improvement as the inputs are much longer.
- Copy Mechanism
Although our copy mechanism is inspired by Pointer Network, the main difference is that our model selects  automatically whether to copy a word from the input sequence or to generate a new word from the pre-defined dictionary, while Pointer Network can only select words from the input sequences.
Therefore, our copy mechanism is more natural for abstractive summarization task.
Furthermore, our model can extract embeddings of rare-words in the input sequence. This help us decrease the dictionary size of the decoder as well as encoder. (See extra experiment response).
As for the reason of gain in the experiments, we refer the reader to Table.1. Our Read-Again models (Ours-GRU and Ours-LSTM) defeat vanilla RNNs.
Besides, incorporating the Copy mechanism further improves performance.
- Other comments
Thanks for your advice. We’re improving our writing.
In the paper, we've shown that our copy mechanism can reduce the decoder size. Here, we show that it can also help to reduce the encoder vocabulary size. We fixed the decoder size as 15K, and we can see that with copy mechanism we are able to reduce encoder vocabulary size from 110k to 15k without loss of performance, while model without copy has severe decreasing. The reason is that our copy mechanism is able to extract an rare word’s embedding from its context, and thus it is enough to learn and store the high-frequency words’ embeddings in our model.
Rouge1		Rouge2		    RougeL
size	    nocopy    copy	nocopy	copy    nocopy	    copy
5k	    21.8	    26.2	9.8		12.0	    21.6	    25.0
15k	    23.8	    27.8	10.7		12.5	    22.5	    26.0
30k	    23.8	    27.5	10.7		12.6	    22.3	    26.0
110k    25.3         27.4     11.8	        12.6	    23.7	    25.7

Meta-review:
role: AC
data: comment: Reviewers are currently in discussion. Please post a rebuttal to any comments or questions in their reviews.

Thanks!
role: PC
data: comment: This work presents a method for reducing the target vocabulary for abstractive summarization by employing a read-again copy mechanism. Reviewers felt that the paper was lacking in several regards particularly focusing on issues clarity and originality. 
 
 Issues raised:
 - Several reviewers focused on clarity/writing issues of the work, highlighting inconsistencies of notation, justifications, and extraneous material. They recommend a rewrite of the work. 
 - The question of originality and novelty is important. The copy mechanism has now been invented many times. Reviewers argue that simply applying this to summary is not enough, or at least not the focus of ICLR. The same question is posed about self-attention.
 - In terms of quality, there are concerns about comparisons to more recent baselines and getting the experiments to run correctly on Gigaword.

Decision: Reject