Title: READ, HIGHLIGHT AND SUMMARIZE: A HIERARCHI-CAL NEURAL SEMANTIC ENCODER-BASED APPROACH

Abstract: Traditional sequence-to-sequence (seq2seq) models and other variations of the attention-mechanism such as hierarchical attention have been applied to the text summarization problem. Though there is a hierarchy in the way humans use language by forming paragraphs from sentences and sentences from words, hierarchical models have usually not worked that much better than their traditional seq2seq counterparts. This effect is mainly because either the hierarchical attention mechanisms are too sparse using hard attention or noisy using soft attention. In this paper, we propose a method based on extracting the highlights of a document; a key concept that is conveyed in a few sentences. In a typical text summarization dataset consisting of documents that are 800 tokens in length (average), capturing long-term dependencies is very important, e.g., the last sentence can be grouped with the first sentence of a document to form a summary. LSTMs (Long Short-Term Memory) proved useful for machine translation. However, they often fail to capture long-term dependencies while modeling long sequences. To address these issues, we have adapted Neural Semantic Encoders (NSE) to text summarization, a class of memory-augmented neural networks by improving its functionalities and proposed a novel hierarchical NSE that outperforms similar previous models significantly. The quality of summarization was improved by augmenting linguistic factors, namely lemma, and Part-of-Speech (PoS) tags, to each word in the dataset for improved vocabulary coverage and generalization. The hierarchical NSE model on factored dataset outperformed the state-of-the-art by nearly 4 ROUGE points. We further designed and used the first GPU-based self-critical Reinforcement Learning model.

1. INTRODUCTION:

When there are a very large number of documents that need to be read in limited time, we often resort to reading summaries instead of the whole document. Automatically generating (abstractive) summaries is a problem with various applications, e.g., automatic authoring (Banerjee & Mitra, 2015) . We have developed automatic text summarization systems that condense large documents into short and readable summaries. It can be used for both single (e.g., Rush et al. (2015) , See et al. (2017) and Nallapati et al. (2017) ) and multi-document summarization (e.g., Celikyilmaz et al. (2018) , Nallapati et al. (2017) , Henß et al. (2015) ).

Text summarization is broadly classified into two categories: extractive (e.g., Nallapati et al. (2017) and (Narayan et al., 2018) ) and abstractive summarization (e.g., Nallapati et al. (2016) , Chopra et al. (2016) and Chen & Bansal (2018) ). Extractive approaches select sentences from a given document and groups them to form concise summaries. By contrast, abstractive approaches generate human-readable summaries that primarily capture the semantics of input documents and contain rephrased key content. The former task falls under the classification paradigm, and the latter belongs to the generative modeling paradigm, and therefore, it is a much harder problem to solve. The backbone of state-of-the-art summarization models is a typical encoder-decoder (Sutskever et al., 2014) architecture that has proved to be effective for various sequential modeling tasks such as machine translation, sentiment analysis, and natural language generation. It contains an encoder that maps the raw input word vector representations to a latent vector. Then, the decoder usually equipped with a variant of the attention mechanism (Bahdanau et al., 2014) uses the latent vectors to generate the output sequence, which is the summary in our case. These models are trained in a supervised learning setting where we minimize the cross-entropy loss between the predicted and the target summary. Encoder-decoder models have proved effective for short sequence tasks such as machine translation where the length of a sequence is less than 120 tokens. However, in text summarization, the length of the sequences vary from 400 to 800 tokens, and modeling long-term dependencies becomes increasingly difficult.

Despite the metric's known drawbacks, text summarization models are evaluated using ROUGE (Lin, 2004) , a discrete similarity score between predicted and target summaries based on 1-gram, 2-gram, and n-gram overlap. Cross-entropy loss would be a convenient objective on which to train the model since ROUGE is not differentiable, but doing so would create a mismatch between metrics used for training and evaluation. Though a particular summary scores well on ROUGE evaluation comparable to the target summary, it will be assigned lower probability by a supervised model. To tackle this problem, we have used a self-critic policy gradient method (Rennie et al., 2016) to train the models directly using the ROUGE score as a reward. In this paper, we propose an architecture that addresses the issues discussed above.

1.1. PROBLEM FORMULATION:

Let D = {d 1 , d 2 , ..., d N } be the set of document sentences where each sentence d i , 1 ≤ i ≤ N is a set of words and S = {s 1 , s 2 , ..., s M } be the set of summary sentences. In general, most of the sentences in D are a continuation of another sentence or related to each other, for example: in terms of factual details or pronouns used. So, dividing the document into multiple paragraphs as done by Celikyilmaz et al. (2018) leaves out the possibility of a sentence-level dependency between the start and end of a document. Similarly, abstracting a single document sentence as done by Chen & Bansal (2018) cannot include related information from multiple document sentences. In a good human-written summary, each summary sentence is a compressed version of a few document sentences. Mathematically,

∀s ∈ S, ∃d 1 , d 2 , ..., d K ∈ D, | C(d 1 , d 2 , ..., d K ) = s (1)

Where C is a compressor we intend to learn. Figure 1 represents the fundamental idea when using a sequence-to-sequence architecture. For a sentence s in summary, the representations of all the related document sentences d 1 , d 2 , ..., d K are expected to form a cluster that represents a part of the highlight of the document.

Figure 1 : Document sentences are first projected into a semantic space typically by an encoder in a sequence-to-sequence model. g 1 , g 2 , g 3 are highlights of a document representing closely related sentence-semantics {h

EQUATION

3 }, {h

EQUATION

3 }, {h

EQUATION

3 } respectively. These highlights are then used by the decoder to form concise summaries.

First, we adapt the Neural Semantic Encoder (NSE) for text summarization by improving its attention mechanism and compose function. In a standard sequence-to-sequence model, the decoder has access to input sequence through hidden states of an LSTM (Hochreiter & Schmidhuber, 1997) , which suffers from the difficulties that we discussed above. The NSE is equipped with an additional memory, which maintains a rich representation of words by evolving over time. We then propose a novel hierarchical NSE by using separate word memories for each sentence to enrich the word representations and a document memory to enrich the sentence representations, which performed better than its previous counterparts (Nallapati et al. (2016) , Nallapati et al. (2017) , Ling & Rush (2017) ). Finally, we use a maximum-entropy self-critic model to achieve better performance using ROUGE evaluation.

2. RELATED WORK:

The first encoder-decoder for text summarziation is used by Rush et al. (2015) coupled with an attention mechanism. Though encoder-decoder models gave a state-of-the-art performance for Neural Machine Translation (NMT), the maximum sequence length used in NMT is just 100 tokens. Typical document lengths in text summarization vary from 400 to 800 tokens, and LSTM is not effective due to the loss in memory over time for very long sequences. Nallapati et al. (2016) used hierarchical attention (Yang et al., 2016) to mitigate this effect where, a word LSTM is used to encode (decode) words, and a sentence LSTM is used to encode (decode) sentences. The use of two LSTMs separately for words and sentences improves the ability of the model to retain its memory for longer sequences. Additionally, Nallapati et al. (2016) explored using a hierarchical model consisting of a feature-rich encoder incorporating position, Named Entity Recognition (NER) tag, Term Frequency (TF) and Inverse Document Frequency (IDF) scores. Since an RNN is a sequential model, computing at one time-step needs all of the previous time-steps to have computed before and is slow because the computation at all the time steps cannot be performed in parallel. Chopra et al. (2016) used convolutional layers coupled with an attention mechanism (Bahdanau et al., 2014) to increase the speed of the encoder. Since the input to an RNN is fed sequentially, it is expected to capture the positional information. But both works Nallapati et al. (2016) and Chopra et al. (2016) found positional embeddings to be quite useful for reasons unknown. Nallapati et al. (2017) proposed an extractive summarization model that classifies sentences based on content, saliency, novelty, and position. To deal with out-of-vocabulary (OOV) words and to facilitate copying salient information from input sequence to the output, See et al. (2017) proposed a pointer-generator network that combines pointing (Vinyals et al., 2015) with generation from vocabulary using a soft-switch. Attention models for longer sequences tend to be repetitive due to the decoder repeatedly attending to the same position from the encoder. To mitigate this issue, See et al. (2017) used a coverage mechanism to penalize a decoder from attending to same locations of an encoder. However, the pointer generator and the coverage model (See et al., 2017) are still highly extractive; copying the whole article sentences 35% of the time. Paulus et al. (2018) introduced an intra-attention model in which attention also depends on the predictions from previous time steps.

One of the main issues with sequence-to-sequence models is that optimization using the crossentropy objective does not always provide excellent results because the models suffer from a mismatch between the training objective and the evaluation metrics such as ROUGE (Lin, 2004) and METEOR (Banerjee & Lavie, 2005) . A popular algorithm to train a decoder is the teacher-forcing algorithm that minimizes the negative log-likelihood (cross-entropy loss) at each decoding time step given the previous ground-truth outputs. But during the testing stage, the prediction from the previous time-step is fed as input to the decoder instead of the ground truth. This exposure bias results in error accumulation over each time step because the model has never been exposed to its predictions during training. Instead, recent works show that summarization models can be trained using reinforcement learning (RL) where the ROUGE (Lin, 2004) score is used as the reward (Paulus et al. (2018) , Chen & Bansal (2018) and Celikyilmaz et al. (2018) ). Henß et al. (2015) made such an earlier attempt by using Q-learning for single-and multi-document summarization. Later, Ling & Rush (2017) proposed a coarse-to-fine hierarchical attention model to select a salient sentence using sentence attention using REINFORCE (Williams, 1992) and feed it to the decoder. Narayan et al. (2018) used REINFORCE to rank sentences for extractive summarization. Celikyilmaz et al. (2018) proposed deep communicating agents that operate over small chunks of a document, which is learned using a self-critical (Rennie et al., 2016) training approach consisting of intermediate rewards. Chen & Bansal (2018) used a advantage actor-critic (A2C) method to extract sentences followed by a decoder to form abstractive summaries. Our model does not suffer from their limiting assumption that a summary sentence is an abstracted version of a single source sentence. Paulus et al. (2018) trained their intra-attention model using a self-critical policy gradient algorithm (Rennie et al., 2016) . Though an RL objective gives a high ROUGE score, the output summaries are not readable by humans. To mitigate this problem, Paulus et al. (2018) used a weighted sum of supervised learning loss and RL loss.

Humans first form an abstractive representation of what they want to say and then try to put it into words while communicating. Though it seems intuitive that there is a hierarchy from sentence representation to words, as observed by both Nallapati et al. (2016) and Ling & Rush (2017) , these hierarchical attention models failed to outperform a simple attention model (Rush et al., 2015) . Unlike feedforward networks, RNNs are expected to capture the input sequence order. But strangely, positional embeddings are found to be effective (Nallapati et al. (2016) , Chopra et al. (2016) , Ling & Rush (2017) and Nallapati et al. (2017) ). We explored a few approaches to solve these issues and improve the performance of neural models for abstractive summarization.

3. PROPOSED MODELS:

In this section, we first describe the baseline Neural Semantic Encoder (NSE) class, discuss improvements to the compose function and attention mechanism, and then propose the Hierarchical NSE. Finally, we discuss the self-critic model that is used to boost the performance further using ROUGE evaluation.

3.1. NEURAL SEMANTIC ENCODER::

A Neural Semantic Encoder (Munkhdalai & Yu, 2017 ) is a memory augmented neural network augmented with an encoding memory that supports read, compose, and write operations. Unlike the traditional sequence-to-sequence models, using an additional memory relieves the LSTM of the burden to remember the whole input sequence. Even compared to the attention-model (Bahdanau et al., 2014) which uses an additional context vector, the NSE has anytime access to the full input sequence through a much larger memory. The encoding memory is evolved using basic operations described as follows:

EQUATION

EQUATION

EQUATION

EQUATION

Where, x t ∈ R D is the raw embedding vector at the current time-step.

f LST M r , f M LP c (Multi-Layer Perceptron), f LST M w

be the read, compose and write operations respectively. e l ∈ R l , e k ∈ R k are vectors of ones, 1 is a matrix of ones and ⊗ is the outer product.

Instead of using the raw input, the read function f LST M r in equation 2 uses an LSTM to project the word embeddings to the internal space of memory M t-1 to obtain the hidden states o t . Now, the alignment scores z t of the past memory M t-1 are calculated using o t as the key with a simple dot-product attention mechanism shown in equation 3. A weighted sum gives the retrieved input memory that is used in equation 5 by a Multi-Layer Perceptron in composing new information. Equation 6 uses an LSTM and projects the composed states into the internal space of memory M t-1 to obtain the write states h t . Finally, in equation 7, the memory is updated by erasing the retrieved memory as per z t and writing as per the write vector h t . This process is performed at each time-step throughout the input sequence. The encoded memories {M } T t=1 are similarly used by the decoder to obtain the write vectors {h} T t=1 that are eventually fed to projection and softmax layers to get the vocabulary distribution.

3.2. IMPROVED NSE:

Although the vanilla NSE described above performed well for machine translation, just a dot-product attention mechanism is too simplistic for text summarization. In machine translation, it is sufficient to compute the correlation between word-vectors from the semantic spaces of different languages. In contrast, text summarization also needs a word-sentence and sentence-sentence correlation along with the word-word correlation. So, in search of an attention mechanism with a better capacity to model the complex semantic relationships inherent in text summarization, we found that the additive attention mechanism (Bahdanau et al., 2014) given by the equation below performs well.

EQUATION

Where, v, W, U, b attn are learnable parameters. One other important difference is the compose function: a Multi-layer Perceptron (MLP) is enough for machine translation as the sequences are short in length. However, text summarization consists of longer sequences that have sentence-tosentence dependencies, and a history of previously composed words is necessary for overcoming repetition (Rush et al., 2015) and thereby maintaining novelty. A powerful function already at our disposal is the LSTM; we replaced the MLP with an LSTM, as shown below:

EQUATION

In a standard text summarization task, due to the limited size of word vocabulary, out-of-vocabulary (OOV) words are replaced with [UNK] tokens. pointer-networks (Vinyals et al., 2015) facilitate the ability to copy words from the input sequence to the output via pointing. Later, See et al. (2017) proposed a hybrid pointer-generator mechanism to improve upon pointing by retaining the ability to generate new words. It points to the words from the input sequence and generates new words from the vocabulary. A generation probability p gen ∈ (0, 1) is calculated using the retrieved memories, attention distribution, current input hidden state o t and write state h t as follows:

EQUATION

Where, W m , W h , W o , b ptr are learnable parameters, and σ is the sigmoid activation function. Next, p gen is used as a soft switch to choose between generating a word from the vocabulary by sampling from p vocab , or copying a word from the input sequence by sampling from the attention distribution z t . For each document, we maintain an auxiliary vocabulary of OOV words in the input sequence. We obtain the following final probability distribution over the total extended vocabulary:

p(w) = p gen p vocab + (1 -p gen ) i:w=wi z t i (11)

Note that if w is an OOV word, then p vocab (w) is zero; similarly, if w does not appear in the source document, then i:w=wi z t i is zero. The ability to produce OOV words is one of the primary advantages of the pointer-generator mechanism. We can also use a smaller vocabulary size and thereby speed up the computation of output projection and softmax layers.

3.3. HIERARCHICAL NSE:

When humans read a document, we organize it in terms of word semantics followed by sentence semantics and then document semantics. In a text summarization task, after reading a document, sentences that have similar meanings or continual information are grouped together and then expressed in words. Such a hierarchical model was first introduced by Yang et al. (2016) for document classification and later explored unsuccessfully for text summarization (Nallapati et al., 2017) . In this work, we propose to use a hierarchical model with improved NSE to take advantage of both augmented memory and also the hierarchical document representation. We use a separate memory for each sentence to represent all the words of a sentence and a document memory to represent all sentences. Word memory composes novel words, and document memory composes novel sentences in the encoding process that can be later used to extract highlights and decode to summaries as shown in Figure 2 .

Let D = {(w ij ) Tin j=1 } Sin i=1 be the input document sequence, where S in is the number of sentences in a document and T in is the number of words per sentence. Let {M i } Sin i=1 , M i ∈ R Tin×D be the sentence memories that encode all the words in a sentence and M d , M d ∈ R Sin×D be the document memory that encodes all the sentences present in the document. At each time-step, an input token x t is read and is used to retrieve aligned content from both corresponding sentence memory M i,s t and document memory M d t . Please note that the retrieved document memory, which is a weighted combination of all the sentence representations forms a highlight. After composition, both the sentence and document memories are written simultaneously. This way, the words are encoded with contextual meaning, and also new simpler sentences are formed. The functionality of the model is as follows:

EQUATION

EQUATION

EQUATION

)

EQUATION

EQUATION

EQUATION

Where, f attn is the attention mechanism given by equation( 8). U pdate remains the same as the vanilla NSE given by equation( 7)and Concat is the vector concatenation. Please note that NSE (Munkhdalai & Yu, 2017) has a concept of shared memory but we use multiple memories for representing words and a document memory for representing sentences, this is fundamentally different to a shared memory which does not have a concept of hierarchy.

3.4. SELF-CRITICAL SEQUENCE TRAINING:

As discussed earlier, training in a supervised learning setting creates a mismatch between training and testing objectives. Also, feeding the ground-truth labels in training time-step creates an exposure bias while testing in which we feed the predictions from the previous time-step. Policy gradient methods overcome this by directly optimizing the non-differentiable metrics such as ROUGE (Lin, 2004) and METEOR (Banerjee & Lavie, 2005) . It can be posed as a Markov Decision Process in which the set of actions A is the vocabulary and reward R is the ROUGE score itself. So, we should find a policy π(θ) such that the set of sampled words ỹ = {ỹ 1 , ỹ2 , ..., ỹT } achieves highest ROUGE score among all possible summaries.

We used the self-critical model of Rennie et al. (2016) proposed for image captioning. In self-critical sequence training, the REINFORCE algorithm (Williams, 1992 ) is used by modifying its baseline as the greedy output of the current model. At each time-step t, the model predicts two words: ŷt sampled from p(ŷ t |ŷ 1 , ŷ2 , ..., ŷt-1 , x), the baseline output that is greedily generated by considering the most probable word from the vocabulary and ỹt sampled from the p(ỹ t |ỹ 1 , ỹ2 , ..., ỹt-1 , x). This model is trained using the following loss function:

EQUATION

Using the above training objective, the model learns to generate samples with high probability and thereby increasing r(ỹ) above r(ŷ).

Additionally, we have used enthttps://stackoverflow.com/questions/19053077/looping-over-data-and-creating-individualfiguresropy regularization.

EQUATION

EQUATION

Where, p(ỹ t ) = p(ỹ t |ỹ 1 , ỹ2 , ..., ỹt-1 , x) is the sampling probability and V is the size of the vocabulary. It is similar to the exploration-exploitation trade-off. α is the regularization coefficient that explicitly controls this trade-off: a higher α corresponds to more exploration, and a lower α corresponds to more exploitation. We have found that all TensorFlow based open-source implementations of self-critic models use a function (tf.py func) that runs only on CPU and it is very slow. To the best of our knowledge, ours is the first GPU based implementation.

4. EXPERIMENTS AND RESULTS:



4.1. DATASET:

We used the CNN/Daily Mail dataset (Nallapati et al., 2016) , which has been used as the standard benchmark to compare text summarization models. This corpus has 286,817 training pairs, 13,368 validation pairs, and 11,487 test pairs, as defined by their scripts. The source document in the training set has 766 words spanning 29.74 sentences on an average while the summaries consist of 53 words and 3.72 sentences (Nallapati et al., 2016) . The unique characteristics of this dataset such as long documents, and ordered multi-sentence summaries present exciting challenges, mainly because the proven sequence-to-sequence LSTM based models find it hard to learn long-term dependencies in long documents. We have used the same train/validation/test split and examples for a fair comparison with the existing models.

The factoring of lemma and Part-of-Speech (PoS) tag of surface words, are observed (Bandyopadhyay, 2019) to increase the performance of NMT models in terms of BLEU score drastically. This is due to the improvement of the vocabulary coverage and better generalization. We have added a pre-processing step by incorporating the lemma and PoS tag to every word of the dataset and training the supervised model on the factored data. The process of extracting the lemma and the PoS tags has been described in Bandyopadhyay (2019) . Please refer to the appendix for an example of factoring.

4.2. TRAINING SETTINGS:

For all the plain NSE models, we have truncated the article to a maximum of 400 tokens and the summary to 100 tokens. For the hierarchical NSE models, articles are truncated to have a maximum of 20 sentences and 20 words per sentence each. Shorter sequences are padded with 'PAD' tokens.

Since the factored models have lemma, PoS tag and the separator '|' for each word, sequence lengths should be close to 3 times the non-factored counterparts. For practical reasons of memory and time, we have used 800 tokens per article and 300 tokens for the summary.

For all the models, including the pointer-generator model, we use a vocabulary size of 50,000 words for both source and target. Though some previous works (Nallapati et al., 2016) have used large vocabulary sizes of 150,000, since our models have a copy mechanism, smaller vocabulary is enough to obtain good performance. Large vocabularies increase the computation time. Since memory plays a prominent role in retrieval and update, it is vital to start with a good initialization. We have used 300-dimensional pre-trained GloVe (Pennington et al., 2014) word-vectors to represent the input sequence to a model. Sentence memories are initialized with GloVe word-vectors of all the words in that sentence. Document memories are initialized with vector representations of all the sentences where a sentence is represented with the average of the GloVe word-vectors of all its words. All the models are trained using the Adam optimizer with the default learning rate of 0.001. We have not applied any regularization as the usage of dropout, and L 2 penalty resulted in similar performance, however with a drastically increased training time.

The Hierarchical models process one sentence at a time, and hence attention distributions need less memory, and therefore, a larger batch size can be used, which in turn speeds up the training process.

The non-factored model is trained on 7-NVIDIA Tesla-P100 (Nallapati et al., 2016) 32.75 12.21 29.01 abstractive model (Nallapati et al., 2016) 35.46 13.30 32.65 Pointer Generator (See et al., 2017) 36 

4.3. EVALUATION:

All the models are evaluated using the standard metric ROUGE; we report the F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L, which quantitively represent word-overlap, bigram-overlap, and longest common subsequence between reference summary and the summary that is to be evaluated. The results are obtained using pyrouge package 1 . The performance of various models and our improvements are summarized in Table 2 . A direct implementation of NSE performed very poorly due to the simple dot-product attention mechanism. In NMT, a transformation from word-vectors in one language to another one (say English to French) using a mere matrix multiplication is enough because of the one-to-one correspondence between words and the underlying linear structure imposed in learning the word vectors (Pennington et al., 2014) . However, in text summarization a word (sentence) could be a condensation of a group of words (sentences). Therefore, using a complex neural network-based attention mechanism proposed improved the performance. Both dot-product and additive (Bahdanau et al., 2014) mechanisms perform similarly for the NMT task, but the difference is more pronounced for the text summarization task simply because of the nature of the problem as described earlier. Replacing Multi-Layered Perceptron (MLP) in the NSE with an LSTM further improved the performance because it remembers what was previously composed and facilitates the composition of novel words. This also eliminates the need for additional mechanisms to penalize repetitions such as coverage (See et al., 2017) and intra-attention (Paulus et al., 2018) . Finally, using memories for each sentence enriches the corresponding word representation, and the document memory enriches the sentence representation that help the decoder. Please refer to the appendix for a few example outputs. 

5. CONCLUSION:

In this work, we presented a memory augmented neural network for the text summarization task that addresses the shortcomings of LSTM-based models. We applied a critical pre-processing step by factoring the dataset with inherent linguistic information that outperforms the state-of-the-art by a large margin. In the future, we will explore new sparse functions (Martins & Astudillo, 2016) to enforce strict sparsity in selecting highlights out of sentences. The general framework of preprocessing, and extracting highlights can also be used with powerful pre-trained models like BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019) . 

Original Article:

The build-up for the blockbuster fight between Floyd Mayweather and Manny Pacquiao in Las Vegas on May 2 steps up a gear on Tuesday night when the American holds an open workout for the media. The session will be streamed live across the world and you can watch it here from 12am UK.

Factored Article:

The Hier-NSE output the build-up for the blockbuster fight between floyd mayweather and manny pacquiao in las vegas on may 2 steps up a gear on tuesday night . the session will be streamed live across the world and you can watch it here from uk . the session will be the media 's open workout for the media .

Hier-NSE-SC:

the floyd mayweather and manny pacquiao in las vegas . the american holds an open workout for the media . will be streamed live across the world and . 

-LRB-CNN -RRB-Justin Timberlake and Jessica Biel , welcome to parenthood . The celebrity couple announced the arrival of their son , Silas Randall Timberlake , in statements to People . " Silas was the middle name of Timberlake 's maternal grandfather Bill Bomar , who died in 2012 , while Randall is the musician 's own middle name , as well as his father 's first , " People reports . The couple announced the pregnancy in January , with an Instagram post . It is the first baby for both .

Factored Article

-LRB-| -lrb-| JJ CNN | cnn | NNP -RRB-| -rrb-| NNP Justin | justin | NNP Timberlake | timberlak | NNP and | and | CC Jessica | jessica | NNP Biel | biel | NNP , | , | , welcome | welcom | NN to | to | TO parenthood | parenthood | NN . | . | . The | the | DT celebrity | celebr | NN couple | coupl | NN announced | announc | VBD the | the | DT arrival | arriv | NN of | of | IN their | their | PRP son | son | NN , | , | , Silas | sila | NNP Randall | randal | NNP Timberlake | timberlak | NNP , | , | , in | in | IN statements | statement | NNS to | to | TO People | peopl | NNS . | . | . " | " | " Silas | sila | NNP was | wa | VBD the | the | DT middle | middl | JJ name | name | NN of | of | IN Timberlake | timberlak | NNP 's | 's | POS maternal | matern | JJ grandfather | grandfath | NN Bill | bill | NNP Bomar | bomar | NNP , | , | , who | who | WP died | die | VBD in | in | IN 2012 | 2012 | CD , | , | , while | while | IN Randall | randal | NNP is | is | VBZ the | the | DT musician | musician | NN 's | 's | POS own | own | JJ middle | middl | NN name | name | NN , | , | , as | as | RB well | well | RB as | as | IN his | hi | PRP father | father | NN 's | 's | POS first | first | JJ , | , | , " | " | " People | peopl | NNP reports | report | NNS . | . | . The | the | DT couple | coupl | NN announced | announc | VBD the | the | DT pregnancy | pregnanc | NN in | in | IN January | januari | NNP , | , | , with | with | IN an | an | DT Instagram | instagram | NNP post | post | NN . | . | . It | It | PRP is | is | VBZ the | the | DT first | first | JJ baby | babi | NN for | for | IN both | both | DT . -. | .

GT Summary:

timberlake and biel welcome son silas randall timberlake . the couple announced the pregnancy in january .

Hier-NSE Output:

" silas was the middle name of timberlake 's maternal grandfather bill bomar ' the couple announced the pregnancy in january , with an instagram post . it is the first baby for both .

Hier-NSE-SC justin timberlake and jessica biel the couple of their son , . silas randall timberlake , in . the first baby for both . 

-LRB-CNN -RRB-Once Hillary Clinton 's official announcement went online , social media responded in a big way , with terms like " Hillary Clinton , " " Hillary2016 , " and yes , even " WhyImNotVotingforHillary " trending . Certainly , you could n't go far on Twitter -LRB-even before Clinton tweeted her announcement -RRB-, without an opinion or thought on her new campaign -LRB-there were over 3 million views of her announcment tweets in one hour , and 750,000 Facebook video views so far by Sunday evening -RRB-. Some tweeted their immediate support , with one word :

GroundTruth Summary response across social media led to multiple trending topics for hillary clinton 's presidential announcement . some responded to her video and her new campaign logo .

hillary clinton tweeted her announcement without an opinion or thought on her new campaign . some tweeted their immediate support , with one word : " hillary clinton , " yes .

Hier-NSE-SC hillary clinton 's official announcement . clinton " hillary clinton , " . " ' ' in the .

Under review as a conference paper at ICLR 2020 

thomas has been training with the championship club this week . the former austria u21 international had a spell with inverness caledonian thistle . blackpool are in talks to sign austria defender thomas .

Hier-NSE-SC blackpool are in talks to sign austria defender thomas . has been training with the championship club this week . is a free agent and .

:

https://pypi.org/project/pyrouge/

Figures and Tables:

Figure 2: Hierarchical NSE: From a given article, all the M sentences consisting of N words each are processed by the NSE using read (R), compose (C) and write (W) operations. Each sentence memory is updated N times by each word in the sentence ({M (k) si } N k=1). After the last encoder step, all the updated sentence memories M N s1 , M N s2 , ..., M N s M are concatenated to form the cumulative sentence memory M s . The decoder then uses the cumulative sentence memory M s and document memory M d in a similar fashion to produce the write vectors h t that are passed through a softmax layer to obtain the vocabulary distribution.

Figure 3 below shows the self-critical model. All the examples shown in Tables 3-8 are chosen as per the shortest article lengths available due to space constraints.

Figure 3: Self-Critic training reduces exposure bias and by learning a policy whose samples score better than the greedy samples that are used during test time in a supervised learning setting.

| the | DT build-up | build-up | NN for | for | IN the | the | DT blockbuster | blockbust | NN fight | fight | NN between | between | IN Floyd | floyd | NNP Mayweather | mayweath | NNP and | and | CC Manny | manni | NNP Pacquiao | pacquiao | NNP in | in | IN Las | la | NNP Vegas | vega | NNP on | on | IN May | may | NNP 2 | 2 | CD steps | step | NNS up | up | RB a | a | DT gear | gear | NN on | on | IN Tuesday | tuesday | NNP night | night | NN when | when | WRB the | the | DT American | american | NNP holds | hold | VBZ an | an | DT open | open | JJ workout | workout | NN for | for | IN the | the | DT media | media | NNS . | . | . The | the | DT session | session | NN will | will | MD be | be | VB streamed | stream | VBN live | live | JJ across | across | IN the | the | DT world | world | NN and | and | CC you | you | PRP can | can | MD watch | watch | VB it | it | PRP here | here | RB from | from | IN 12am | 12am | . | . GT Summary floyd mayweather holds an open media workout from 12am uk -lrb-7pm edt -rrb-. the american takes on manny pacquiao in las vegas on may 2 . mayweather 's training is being streamed live across the world .

GT summary (factored)floyd | floyd | nnp mayweather | mayweath | nnp holds | hold | vbz an | an | dt open | open | jj media | media | nns workout | workout | nn from | from | in 12am | 12am | cd uk | uk | nnp -lrb-| -lrb-| vbd 7pm | 7pm | cd edt | edt | nnp -rrb-| -rrb-| nn . the | the | dt american | american | jj takes | take | vbz on | on | in manny | manni | nnp pacquiao | pacquiao | nnp in | in | in las | la | nnp vegas | vega | nnp on | on | in may | may | nnp 2 | 2 | cd . mayweather | mayweath | nnp 's | 's | pos training | train | nn is | is | vbz being | be | vbg streamed | stream | vbn live | live | jj across | across | in the | the | dt world | world | nn .Hier-NSE output (factored) the | the | dt session | session | nn will | will | md be | be | vb streamed | stream | vbn live | live | jj across | across | in the | the | dt world | world | nn and | and | cc you | you | prp can | can | md watch | watch | vb it | it | prp here | here | rb from | from | in 12am | 12am | nnp nnp | nnp | nnp . the | the | dt american | american | nnp holds | hold | vbz an | an | dt open | open | jj workout | workout | nn for | for | in the | the | dt media | media | nns .

GT summary (factored) timberlake | timberlak | nnp and | and | cc biel | biel | nnp welcome | welcom | vbp son | son | nn silas | sila | nnp randall | randal | nnp timberlake | timberlak | nnp . the | the | dt couple | coupl | nn announced | announc | vbd the | the | dt pregnancy | pregnanc | nn in | in | in january | januari | nnp . Hier-NSE Output (factored) justin | justin | nnp timberlake | nnp | nnp and | and | cc jessica | jessica | nnp nnp | nnp | nnp are | are | [UNK] in | in | in statements | statement | nns to | to | to people | peopl | nns . he | he | nnp is | is | vbz the | the | dt first | first | jj baby | vbz | nn for | for | in both | both | dt . timberlake | [UNK] | jj bill | bill | nn , | , | , the | the | dt couple | | nn 's | 's | pos son | son | nn .

Table 1 shows the results in comparison to the previous methods. Our hierarchical model outperforms Nallapati et al. (2016) (HIER) by 5 ROUGE points. Our factored model achieves the new state-of-the-art (SoTA) result, outperforming Celikyilmaz et al. (2018) by almost 4 ROUGE points. Performance of various NSE models on CNN/Daily Mail corpus. Please note that the data is not factored here.

Table content: <table><tr><td>Model</td><td colspan="3">ROUGE (% F-score)</td></tr><tr><td/><td>1</td><td>2</td><td>L</td></tr><tr><td>Plain NSE</td><td>7.99</td><td>0.86</td><td>7.52</td></tr><tr><td colspan="4">NSE -improved attention 25.47 8.96 24.01</td></tr><tr><td colspan="4">NSE -improved compose 30.86 11.42 29.04</td></tr><tr><td>Hierarchical NSE</td><td colspan="3">38.31 16.34 35.26</td></tr></table>

Sample outputs for both non-factored and factored input articles. While factoring, each surface word is augmented with lemma and PoS tag separated by |.

Table content: <table/>

Sample outputs for both non-factored and factored input articles. While factoring, each surface word is augmented with lemma and PoS tag separated by |.

Table content: <table/>

Sample outputs from the hierarchical NSE and self-critical model.

Table content: <table/>

Sample outputs from the hierarchical NSE and self-critical model.Blackpool are in talks to sign Austria defender Thomas Piermayr . The 25-year-old has been training with the Championship club this week and they are keen to get him on board for what is expected to be confirmed as a campaign in League One next season . Piermayr is a free agent and had been playing for Colorado Rapids . The former Austria U21 international had a spell with Inverness Caledonian Thistle in 2011 . Thomas Piermayr -LRB-left , in action for the Colorado Rapids -RRBtries to tackle Obafemi Martins last year GroundTruth Summary thomas piermayr has been training with blackpool this week . austrian defender is a free agent after leaving mls side colorado rapids . blackpool are bottom of the championship and look set to be relegated .

Table content: <table><tr><td>Original Article</td></tr></table>

Peer review (iclr2020):
This submission claims to adapt and improve on neural semantic encoders for the task of abstractive summarization. The submission: - briefly describes the neural semantic encoder (NSE) model (Munkhdalai and Yu, 2017) - claims that the dot-product attention mechanism in NSE is too simplistic for the task of summarization because dot-product attention does not capture word-sentence and sentence-sentence correlations, the latter two being important to summarize a document, and suggests utilizing the additive attention mechanism along with the pointer-generator mechanism instead - Introduces the “hierarchical NSE” model and a self-critical sequence training scheme to optimize for ROUGE - Provides empirical results for these models and for other relevant baseline models on the CNN/Daily Mail dataset At a high-level, this submission is unpolished, imprecise, lacks technical (both experimental and theoretical) rigor in its description and explanation of its methods, and generally does not have a significant contribution, which is why I believe it should be rejected. The primary motivation underlying this submission is strong: modeling long-range dependencies in a document is very critical for abstractive summarization. However, the submission does not make a convincing case for its proposed methods/models, in the context of this motivation and otherwise. The whole premise behind the NSE (according to Munkhdalai and Yu’s paper) is that its variable-sized encoding memory allows it to access the entire input sequence during reading, thus providing for long-range dependencies and helping with tasks like summarization. So the exceptionally low ROUGE scores for plain NSE in Table 2 seemed at odds with the motivation and premise. The authors attribute this very briefly to the dot-product attention mechanism and show empirically that changing it to an additive attention mechanism leads to a significant increase in ROUGE. At an intuitive level, the notion of “highlighting” seems to be congruous to “attending”. The submission (with the title Read, Highlight and Summarize) seems to be suggesting that they are incongruous and that there is novelty in this notion itself, however there is no clear mathematical description or rigorous explanation of this notion. The suggested improvements beyond the standard NSE architecture (augmented with the well-known additive attention and the pointer-generator mechanism) are limited to extending NSE in a “hierarchical” fashion and optimizing for ROUGE, both of which have not been described in sufficient detail or clarity for the task at hand. Munkhdalai and Yu evaluated the NSE approach for document sentiment analysis (among other tasks) and for that task, they used two NSEs: one for obtaining sentence representations and another for document representations using sentence representations. This submission’s “hierarchical NSE” seems to overlap with Munkhdalai and Yu’s approach for document sentiment analysis, except that the document memory seems to be constructed in a different fashion that is not clearly described. In Table 1, Hier-NSE seems to be performing worse than the pointer-generator with coverage loss and only slightly better without coverage loss. Since Hier-NSE contains the pointer-generator mechanism, it seems the Hier-NSE isn’t really adding any non-trivial value. What would be interesting to try out as an exercise is to see if the factoring of lemma and PoS tags described helps significantly improve the pointer-generator with coverage loss. Some typos: - Equation 5 should have m_{r,t}, not m_{t,t} - Page 5 right above equation 22 has a Stack Overflow link that has nothing to do with the submission Suggestions on improving the submission that are here to help, not necessarily part of my decision assessment: - Provide dimensions for every non-scalar parameter in the Proposed Models section: while I see dimensions for some parameters, I do not see dimensions for others - Use accessible notation that makes it easy to distinguish between a non-scalar parameter and a scalar parameter (boldface non-scalars, use capitalized letters for only matrices/tensors, not scalars or vectors, etc.) - Use mathbb{R} for depicting the set of real numbers in LaTeX