Entity Commonsense Representation for Neural Abstractive Summarization

Abstract: A major proportion of a text summary includes important entities found in the original text. These entities build up the topic of the summary. Moreover, they hold commonsense information once they are linked to a knowledge base. Based on these observations, this paper investigates the usage of linked entities to guide the decoder of a neural text summarizer to generate concise and better summaries. To this end, we leverage on an off-the-shelf entity linking system (ELS) to extract linked entities and propose Entity2Topic (E2T), a module easily attachable to a sequence-to-sequence model that transforms a list of entities into a vector representation of the topic of the summary. Current available ELS's are still not sufficiently effective, possibly introducing unresolved ambiguities and irrelevant entities. We resolve the imperfections of the ELS by (a) encoding entities with selective disambiguation, and (b) pooling entity vectors using firm attention. By applying E2T to a simple sequence-to-sequence model with attention mechanism as base model, we see significant improvements of the performance in the Gigaword (sentence to title) and CNN (long document to multi-sentence highlights) summarization datasets by at least 2 ROUGE points.

Text summarization is a task to generate a shorter and concise version of a text while preserving the meaning of the original text. The task can be divided into two subtask based on the approach: extractive and abstractive summarization. Extractive summarization is a task to create summaries by pulling out snippets of text form the original text and combining them to form a summary. Abstractive summarization asks to generate summaries from scratch without the restriction to use the available words from the original text. Due to the limitations of extractive summarization on incoherent texts and unnatural methodology BIBREF0 , the research trend has shifted towards abstractive summarization.
Sequence-to-sequence models BIBREF1 with attention mechanism BIBREF2 have found great success in generating abstractive summaries, both from a single sentence BIBREF3 and from a long document with multiple sentences BIBREF4 . However, when generating summaries, it is necessary to determine the main topic and to sift out unnecessary information that can be omitted. Sequence-to-sequence models have the tendency to include all the information, relevant or not, that are found in the original text. This may result to unconcise summaries that concentrates wrongly on irrelevant topics. The problem is especially severe when summarizing longer texts.
In this paper, we propose to use entities found in the original text to infer the summary topic, mitigating the aforementioned problem. Specifically, we leverage on linked entities extracted by employing a readily available entity linking system. The importance of using linked entities in summarization is intuitive and can be explained by looking at Figure 1 as an example. First (O1 in the Figure), aside from auxiliary words to construct a sentence, a summary is mainly composed of linked entities extracted from the original text. Second (O2), we can depict the main topic of the summary as a probability distribution of relevant entities from the list of entities. Finally (O3), we can leverage on entity commonsense learned from a separate large knowledge base such as Wikipedia.
To this end, we present a method to effectively apply linked entities in sequence-to-sequence models, called Entity2Topic (E2T). E2T is a module that can be easily attached to any sequence-to-sequence based summarization model. The module encodes the entities extracted from the original text by an entity linking system (ELS), constructs a vector representing the topic of the summary to be generated, and informs the decoder about the constructed topic vector. Due to the imperfections of current ELS's, the extracted linked entities may be too ambiguous and coarse to be considered relevant to the summary. We solve this issue by using entity encoders with selective disambiguation and by constructing topic vectors using firm attention.
We experiment on two datasets, Gigaword and CNN, with varying lengths. We show that applying our module to a sequence-to-sequence model with attention mechanism significantly increases its performance on both datasets. Moreover, when compared with the state-of-the-art models for each dataset, the model obtains a comparable performance on the Gigaword dataset where the texts are short, and outperforms all competing models on the CNN dataset where the texts are longer. Furthermore, we provide analysis on how our model effectively uses the extracted linked entities to produce concise and better summaries.
In the next subsections, we present detailed arguments with empirical and previously examined evidences on the observations and possible issues when using linked entities extracted by an entity linking system (ELS) for generating abstractive summaries. For this purpose, we use the development sets of the Gigaword dataset provided in BIBREF5 and of the CNN dataset provided in BIBREF6 as the experimental data for quantitative evidence and refer the readers to Figure 1 as the running example.
As discussed in Section "Introduction" , we find three observations that show the usefulness of linked entities for abstractive summarization.
First, summaries are mainly composed of linked entities extracted from the original text. In the example, it can be seen that the summary contains four words that refer to different entities. In fact, all noun phrases in the summary mention at least one linked entity. In our experimental data, we extract linked entities from the original text and compare them to the noun phrases found in the summary. We report that $77.1\%$ and $75.1\%$ of the noun phrases on the Gigaword and CNN datasets, respectively, contain at least one linked entity, which confirms our observation.
Second, linked entities can be used to represent the topic of the summary, defined as a multinomial distribution over entities, as graphically shown in the example, where the probabilities refer to the relevance of the entities. Entities have been previously used to represent topics BIBREF7 , as they can be utilized as a controlled vocabulary of the main topics in a document BIBREF8 . In the example, we see that the entity “Jae Seo” is the most relevant because it is the subject of the summary, while the entity “South Korean” is less relevant because it is less important when constructing the summary.
Third, we can make use of the entity commonsense that can be learned as a continuous vector representation from a separate larger corpus BIBREF9 , BIBREF10 . In the example, if we know that the entities “Los Angeles Dodgers” and “New York Mets” are American baseball teams and “Jae Seo” is a baseball player associated with the teams, then we can use this information to generate more coherent summaries. We find that $76.0\%$ of the extracted linked entities are covered by the pre-trained vectors in our experimental data, proving our third observation.
Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness.
First, the extracted entities may be ambiguous. In the example, the entity “South Korean” is ambiguous because it can refer to both the South Korean person and the South Korean language, among others. In our experimental data, we extract (1) the top 100 entities based on frequency, and (2) the entities extracted from 100 randomly selected texts, and check whether they have disambiguation pages in Wikipedia or not. We discover that $71.0\%$ of the top 100 entities and $53.6\%$ of the entities picked at random have disambiguation pages, which shows that most entities are prone to ambiguity problems.
Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, “Wednesday” is erroneous because it is wrongly linked to the entity “Wednesday Night Baseball”. Also, “swap” is irrelevant because although it is linked correctly to the entity “Trade (Sports)”, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\%$ , $13.7\%$ , $30.0\%$ , and $26.9\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities.
To solve the issues described above, we present Entity2Topic (E2T), a module that can be easily attached to any sequence-to-sequence based abstractive summarization model. E2T encodes the linked entities extracted from the text and transforms them into a single topic vector. This vector is ultimately concatenated to the decoder hidden state vectors. The module contains two submodules specifically for the issues presented by the entity linking systems: the entity encoding submodule with selective disambiguation and the pooling submodule with firm attention.
Overall, our full architecture can be illustrated as in Figure 2 , which consists of an entity linking system (ELS), a sequence-to-sequence with attention mechanism model, and the E2T module. We note that our proposed module can be easily attached to more sophisticated abstractive summarization models BIBREF13 , BIBREF14 that are based on the traditional encoder-decoder framework and consequently can produce better results. The code of the base model and the E2T are available online.
As our base model, we employ a basic encoder-decoder RNN used in most neural machine translation BIBREF2 and text summarization BIBREF15 tasks. We employ a two-layer bidirectional GRU (BiGRU) as the recurrent unit of the encoder. The BiGRU consists of a forward and backward GRU, which results to sequences of forward and backward hidden states $(\overrightarrow{h}_1, \overrightarrow{h}_2, ..., \overrightarrow{h}_n)$ and $(\overleftarrow{h}_1, \overleftarrow{h}_2, ..., \overleftarrow{h}_n)$ , respectively: $
\overrightarrow{h}_i &= GRU(x_i, \overrightarrow{h}_{i-1}) \\
\overleftarrow{h}_i &= GRU(x_i, \overleftarrow{h}_{i+1}) \nonumber $ 
The forward and backward hidden states are concatenated to get the hidden state vectors of the tokens (i.e. $h_i = [\overrightarrow{h}_i; \overleftarrow{h}_i]$ ). The final states of the forward and backward GRU are also concatenated to create the final text representation vector of the encoder $s = [\overrightarrow{h}_n; \overleftarrow{h}_1]$ . These values are calculated per layer, where $x_t$ of the second layer is $h_t$ of the first layer. The final text representation vectors are projected by a fully connected layer and are passed to the decoder as the initial hidden states $s_0 = s$ .
For the decoder, we use a two-layer uni-directional GRU with attention. At each time step $t$ , the previous token $y_{t-1}$ , the previous hidden state $s_{t-1}$ , and the previous context vector $c_{t-1}$ are passed to a GRU to calculate the new hidden state $s_t$ , as shown in the equation below. 
$$s_t = GRU(w_{t-1}, s_{t-1}, c_{t-1}) \nonumber $$   (Eq. 9) 
The context vector $c_t$ is computed using the additive attention mechanism BIBREF2 , which matches the current decoder state $s_t$ and each encoder state $h_i$ to get an importance score. The scores are then passed to a softmax and are used to pool the encoder states using weighted sum. The final pooled vector is the context vector, as shown in the equations below. $
g_{t,i} &= v_a^\top tanh(W_a s_{t-1} + U_a h_i) \\
a_{t,i} &= \frac{exp(g_{t,i})}{\sum _i exp(g_{t,i})} \\
c_t &= \sum _i a_{t,i} h_i \nonumber $ 
Finally, the previous token $y_{t-1}$ , the current context vector $c_t$ , and the current decoder state $s_t$ are used to generate the current word $y_t$ with a softmax layer over the decoder vocabulary, as shown below. $
o_t &= W_w w_{t-1} + W_c c_t + W_s s_t \\
p(y_t | y_{<t}) &= softmax(W_o o_t) \nonumber $ 
After performing entity linking to the input text using the ELS, we receive a sequential list of linked entities, arranged based on their location in the text. We embed these entities to $d$ -dimensional vectors $E = \lbrace e_1, e_2, ..., e_m\rbrace $ where $e_i \in \mathbb {R}^d$ . Since these entities may still contain ambiguity, it is necessary to resolve them before applying them to the base model. Based on the idea that an ambiguous entity can be disambiguated using its neighboring entities, we introduce two kinds of disambiguating encoders below.
One way to disambiguate an entity is by using all the other entities, putting more importance to entities that are nearer. For this purpose, we employ an RNN-based model to globally disambiguate the entities. Specifically, we use BiGRU and concatenate the forward and backward hidden state vectors as the new entity vector: $
\overrightarrow{h}_i &= GRU(e_i, \overrightarrow{h}_{i-1}) \\
\overleftarrow{h}_i &= GRU(e_i, \overleftarrow{h}_{i+1}) \\
e^{\prime }_i &= [\overrightarrow{h}_i; \overleftarrow{h}_i] \nonumber $ 
Another way to disambiguate an entity is by using only the direct neighbors of the entity, putting no importance value to entities that are far. To do this, we employ a CNN-based model to locally disambiguate the entities. Specifically, we do the convolution operation using filter matrices $W_f \in \mathbb {R}^{h \times d}$ with filter size $h$ to a window of $h$ words. We do this for different sizes of $h$ . This produces new feature vectors $c_{i,h}$ as shown below, where $f(.)$ is a non-linear function: $
c_{i,h} = f([e_{i-(h-1)/2}; ...; e_{i+h(+1)/2}]^\top W_f + b_f) \nonumber $ 
The convolution operation reduces the number of entities differently depending on the filter size $h$ . To prevent loss of information and to produce the same amount of feature vectors $c_{i,h}$ , we pad the entity list dynamically such that when the filter size is $h$ , the number of paddings on each side is $(h-1)/2$ . The filter size $h$ therefore refers to the number of entities used to disambiguate a middle entity. Finally, we concatenate all feature vectors of different $h$ 's for each $i$ as the new entity vector: $
e^{\prime }_i = [c_{i,h_1}; c_{i, h_2}; ...] \nonumber $ 
The question on which disambiguating encoder is better has been a debate; some argued that using only the local context is appropriate BIBREF16 while some claimed that additionally using global context also helps BIBREF17 . The RNN-based encoder is good as it smartly makes use of all entities, however it may perform bad when there are many entities as it introduces noise when using a far entity during disambiguation. The CNN-based encoder is good as it minimizes the noise by totally ignoring far entities when disambiguating, however determining the appropriate filter sizes $h$ needs engineering. Overall, we argue that when the input text is short (e.g. a sentence), both encoders perform comparably, otherwise when the input text is long (e.g. a document), the CNN-based encoder performs better.
It is obvious that not all entities need to be disambiguated. When a correctly linked and already adequately disambiguated entity is disambiguated again, it would make the entity very context-specific and might not be suitable for the summarization task. Our entity encoding submodule therefore uses a selective mechanism that decides whether to use the disambiguating encoder or not. This is done by introducing a selective disambiguation gate $d$ . The final entity vector $\tilde{e}_i$ is calculated as the linear transformation of $e_i$ and $e^{\prime }_i$ : $
e^{\prime }_i &= encoder(e_i) \\
d &= \sigma (W_d e^{\prime }_i + b_d) \\
\tilde{e}_i &= d \times f(W_x e_i + b_x) + \\ & \quad (1-d) \times f(W_y e^{\prime }_i + b_y) \nonumber $ 
The full entity encoding submodule is illustrated in Figure 3 . Ultimately, the submodule outputs the disambiguated entity vectors $\tilde{E} = \lbrace \tilde{e}_1, \tilde{e}_2, ..., \tilde{e}_m\rbrace $ .
The entity vectors $\tilde{E}$ are pooled to create a single topic vector $t$ that represents the topic of the summary. One possible pooling technique is to use soft attention BIBREF18 on the vectors to determine the importance value of each vector, which can be done by matching each entity vector with the text vector $s$ from the text encoder as the context vector. The entity vectors are then pooled using weighted sum. One problem with soft attention is that it considers all entity vectors when constructing the topic vector. However, not all entities are important and necessary when generating summaries. Moreover, a number of these entities may be erroneous and irrelevant, as reported in Section "Related work" . Soft attention gives non-negligible important scores to these entities, thus adds unnecessary noise to the construction of the topic vector.
Our pooling submodule instead uses firm attention mechanism to consider only top $k$ entities when constructing the topic vector. This is done in a differentiable way as follows: $
G &= v_a^\top tanh(W_a \tilde{E} + U_a s) \\
K &= top\_k(G) \\
P &= sparse\_vector(K, 0, -\infty ) \\
g^{\prime }_i &= g_i + p_i \\
a_i &= \frac{exp(g^{\prime }_i)}{\sum _i exp(g^{\prime }_i)} \\
t &= \sum _i a_i \tilde{e}_i \nonumber $ 
 where the functions $K = top\_k(G)$ gets the indices of the top $k$ vectors in $G$ and $P = sparse\_vector(K,0,-\infty )$ creates a sparse vector where the values of $K$ is 0 and $-\infty $ otherwise. The sparse vector $P$ is added to the original importance score vector $G$ to create a new importance score vector. In this new vector, important scores of non-top $k$0 entities are $k$1 . When softmax is applied, this gives very small, negligible, and close-to-zero values to non-top $k$2 entities. The value $k$3 depends on the lengths of the input text and summary. Moreover, when $k$4 increases towards infinity, firm attention becomes soft attention. We decide $k$5 empirically (see Section "Experimental settings" ).
Entity2Topic module extends the base model as follows. The final text representation vector $s$ is used as a context vector when constructing the topic vector $t$ in the pooling submodule. The topic vector $t$ is then concatenated to the decoder hidden state vectors $s_i$ , i.e. $s^{\prime }_i = [s_i; t]$ . The concatenated vector is finally used to create the output vector: 
$$o_i = W_w w_{i-1} + W_c c_i + W_s s^{\prime }_i \nonumber $$   (Eq. 18) 
Due to its recent success, neural network models have been used with competitive results on abstractive summarization. A neural attention model was first applied to the task, easily achieving state-of-the-art performance on multiple datasets BIBREF5 . The model has been extended to instead use recurrent neural network as decoder BIBREF3 . The model was further extended to use a full RNN encoder-decoder framework and further enhancements through lexical and statistical features BIBREF15 . The current state-of-the-art performance is achieved by selectively encoding words as a process of distilling salient information BIBREF13 .
Neural abstractive summarization models have also been explored to summarize longer documents. Word extraction models have been previously explored, performing worse than sentence extraction models BIBREF19 . Hierarchical attention-based recurrent neural networks have also been applied to the task, owing to the idea that there are multiple sentences in a document BIBREF15 . Finally, distraction-based models were proposed to enable models to traverse the text content and grasp the overall meaning BIBREF4 . The current state-of-the-art performance is achieved by a graph-based attentional neural model, considering the key factors of document summarization such as saliency, fluency and novelty BIBREF14 .
Previous studies on the summarization tasks have only used entities in the preprocessing stage to anonymize the dataset BIBREF15 and to mitigate out-of-vocabulary problems BIBREF14 . Linked entities for summarization are still not properly explored and we are the first to use linked entities to improve the performance of the summarizer.
We report the ROUGE F1 scores for both datasets of all the competing models using ROUGE F1 scores BIBREF27 . We report the results on the Gigaword and the CNN dataset in Table 2 and Table 3 , respectively. In Gigaword dataset where the texts are short, our best model achieves a comparable performance with the current state-of-the-art. In CNN dataset where the texts are longer, our best model outperforms all the previous models. We emphasize that E2T module is easily attachable to better models, and we expect E2T to improve their performance as well. Overall, E2T achieves a significant improvement over the baseline model base, with at least 2 ROUGE-1 points increase in the Gigaword dataset and 6 ROUGE-1 points increase in the CNN dataset. In fact, all variants of E2T gain improvements over the baseline, implying that leveraging on linked entities improves the performance of the summarizer. Among the model variants, the CNN-based encoder with selective disambiguation and firm attention performs the best.
Automatic evaluation on the Gigaword dataset shows that the CNN and RNN variants of base+E2T have similar performance. To break the tie between both models, we also conduct human evaluation on the Gigaword dataset. We instruct two annotators to read the input sentence and rank the competing summaries from first to last according to their relevance and fluency: (a) the original summary gold, and from models (b) base, (c) base+E2Tcnn, and (d) base+E2Trnn. We then compute (i) the proportion of every ranking of each model and (ii) the mean rank of each model. The results are reported in Table 4 . The model with the best mean rank is base+E2Tcnn, followed by gold, then by base+E2Trnn and base, respectively. We also perform ANOVA and post-hoc Tukey tests to show that the CNN variant is significantly ( $p<0.01$ ) better than the RNN variant and the base model. The RNN variant does not perform as well as the CNN variant, contrary to the automatic ROUGE evaluation above. Interestingly, the CNN variant produces better (but with no significant difference) summaries than the gold summaries. We posit that this is due to the fact that the article title does not correspond to the summary of the first sentence.
We proposed to leverage on linked entities to improve the performance of sequence-to-sequence models on neural abstractive summarization task. Linked entities are used to guide the decoding process based on the summary topic and commonsense learned from a knowledge base. We introduced Entity2Topic (E2T), a module that is easily attachable to any model using an encoder-decoder framework. E2T applies linked entities into the summarizer by encoding the entities with selective disambiguation and pooling them into one summary topic vector with firm attention mechanism. We showed that by applying E2T to a basic sequence-to-sequence model, we achieve significant improvements over the base model and consequently achieve a comparable performance with more complex summarization models.
We would like to thank the three anonymous reviewers for their valuable feedback. This work was supported by Microsoft Research, and Institute for Information communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2017-0-01778 , Development of Explainable Humanlevel Deep Machine Learning Inference Framework). S. Hwang is a corresponding author.

Figure and table captions:
Figure 1: Observations on linked entities in summaries. O1: Summaries are mainly composed of entities. O2: Entities can be used to represent the topic of the summary. O3: Entity commonsense learned from a large corpus can be used.
Figure 2: Full architecture of our proposed sequence-to-sequence model with Entity2Topic (E2T) module.
Figure 3: Entity encoding submodule with selective disambiguation applied to the entity 3©. The left figure represents the full submodule while the right figure represents the two choices of disambiguating encoders.
Table 1: Dataset statistics.
Table 2: Results on the Gigaword dataset using the fulllength F1 variants of ROUGE.
Table 3: Results on the CNN dataset using the fulllength F1 ROUGE metric.
Table 4: Human evaluations on the Gigaword dataset. Bold-faced values are the best while red-colored values are the worst among the values in the evaluation metric.
Table 5: Examples from Gigaword and CNN datasets and corresponding summaries generated by competing models. The tagged part of text is marked bold and preceded with at sign (@). The red color fill represents the attention scores given to each entity. We only report the attention scores of entities in the Gigaword example for conciseness since there are 80 linked entities in the CNN example.
Table 6: Examples with highest/lowest disambiguation gate d values of two example entities (United States and gold). The tagged part of text is marked bold and preceded with at sign (@).

Questions about this paper:
Why are current ELS's not sufficiently effective?

Answers:
Linked entities may be ambiguous or too common