Introduction
Traditional approaches to abstractive summarization have relied on interpretable structured representations such as graph based sentence centrality BIBREF0, AMR parses BIBREF1, discourse based compression and anaphora constraints BIBREF2. On the other hand, state of the art neural approaches to single document summarization encode the document as a sequence of tokens and compose them into a document representation BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7. Albeit being effective, these systems learn to rely significantly on layout bias associated with the source document BIBREF8 and do not lend themselves easily to interpretation via intermediate structures.
Recent work provides evidence that structured representation of text leads to better document representations BIBREF9, BIBREF10. However, structured representations are under-explored in the neural summarization literature. Motivated by this, we propose a structure-aware end-to-end model (§SECREF2) for summarization. Our proposed model, StructSum, augments the existing pointer-generator network BIBREF3 with two novel components: (1) a latent-structure attention module that adapts structured representations BIBREF11, BIBREF12 for the summarization task, and (2) an explicit-structure attention module, that incorporates a coreference graph. The components together model sentence level dependencies in a document generating rich structured representations. The motivation of this work is to provide a framework to induce rich interpretable latent structures and inject external document structures that can be introduced into any document encoder model.
Encoders with induced latent structures have been shown to benefit several tasks including document classification, natural language inference BIBREF12, BIBREF13, and machine translation BIBREF11. Building on this motivation, our latent structure attention module builds upon BIBREF12 to model the dependencies between sentences in a document. It uses a variant of Kirchhoff’s matrix-tree theorem BIBREF14 to model such dependencies as non-projective tree structures(§SECREF3). The explicit attention module is linguistically-motivated and aims to incorporate sentence-level structures from externally annotated document structures. We incorporate a coreference based sentence dependency graph, which is then combined with the output of the latent structure attention module to produce a hybrid structure-aware sentence representation (§SECREF5).
We evaluate our model on the CNN/DM dataset BIBREF15 and show in §SECREF4 that it outperforms strong baselines by up to 1.1 ROUGE-L. We find that the latent and explicit structures are complementary, both contributing to the final performance improvement. Our modules are also independent of the underlying encoder-decoder architectures, rendering them flexible to be incorporated into any advanced models. Our analysis quantitatively compares our generated summaries with the baselines and reference documents (§SECREF5). It reveals that structure-aware summarization reduces the bias of copying large sequences from the source inherently making the summaries more abstractive by generating $\sim $15% more novel n-grams compared to a competitive baseline. We also show qualitative examples of the learned interpretable sentence dependency structures, motivating further research for structure-aware modeling.
StructSum Model
Consider a source document $\mathbf {x}$ consisting of $n$ sentences $\lbrace \mathbf {s}\rbrace $ where each sentence $\mathbf {s}_i$ is composed of a sequence of words. Document summarization aims to map the source document to a target summary of $m$ words $\lbrace y\rbrace $. A typical neural abstractive summarization system is an attentional sequence-to-sequence model that encodes the input sequence $\mathbf {x}$ as a continuous sequence of tokens $\lbrace w\rbrace $ using a BiLSTM. The encoder produces a set of hidden representations $\lbrace \mathbf {h}\rbrace $. An LSTM decoder maps the previously generated token $y_{t-1}$ to a hidden state and computes a soft attention probability distribution $p(\mathbf {a}_t \mid \mathbf {x}, \mathbf {y}_{1:t-1})$ over encoder hidden states. A distribution $p$ over the vocabulary is computed at every timestep $t$ and the network is trained using negative log likelihood loss : $\text{loss}_t = - \mathrm {log}\:p(y_t) $. The pointer-generator network BIBREF3 augments the standard encoder-decoder architecture by linearly interpolating a pointer based copy mechanism. StructSum uses the pointer-generator network as the base model. Our encoder is a structured hierarchical encoder BIBREF16, which computes hidden representations of the sequence both at the token and sentence level. The model then uses the explicit-structure and implicit-structure attention modules to augment the sentence representations with rich sentence dependency information, leveraging both learned latent structure and additional external structure from other NLP modules. The attended vectors are then passed to the decoder, which produces the output sequence for abstractive summarization. In the rest of this section, we describe our model architecture, shown in Figure FIGREF2, in detail.
StructSum Model ::: Encoder
Our hierarchical encoder consists of a BiLSTM encoder over words, followed by sentence level BiLSTM encoder. The word encoder takes a sequence of words in a sentence $\mathbf {s}_i = \lbrace w\rbrace $ as input and produces contextual hidden representation for each word $\mathbf {h}_{w_{ik}}$, where $w_{ik}$ is the $i^{th}$ word of the $k^{th}$ sentence, $k=1:q$ and $q$ is the number of words in the sentence $\mathbf {s}_i$. The word hidden representations are max-pooled at the sentence level and the result is passed to a BiLSTM sentence-encoder which produces new hidden sentence representations for each sentence $\mathbf {h}_{\mathbf {s}_i}$. The sentence hidden representations are then passed as inputs to latent and explicit structure attention modules.
StructSum Model ::: Latent Structure (LS) Attention
We model the latent structure of a source document as a non-projective dependency tree and force a pair-wise attention module to automatically induce this tree. We denote the marginal probability of a dependency edge as $a_{ij} = p(z_{ij}=1)$ where $z_{ij}$ is the latent variable representing the edge from sentence $i$ to sentence $j$. We parameterize with a neural network the unnormalized pair-wise scores between sentences and use the Kirchoff's matrix tree theorem BIBREF14 to compute the marginal probability of a dependency edge between any two sentences.
We decompose the representation of sentence $\mathbf {s}_i$ into a semantic vector $\mathbf {g}_{\mathbf {s}_i}$ and structure vector $\mathbf {d}_{\mathbf {s}_i}$ as $\mathbf {h}_{\mathbf {s}_i} = [\mathbf {g}_{\mathbf {s}_i}; \mathbf {d}_{\mathbf {s}_i}]$. Using the structure vectors $\mathbf {d}_{\mathbf {s}_i}, \mathbf {d}_{\mathbf {s}_j}$, we compute a score $f_{ij}$ between sentence pairs $(i,j)$ (where sentence $i$ is the parent node of sentence $j$) and a score for sentence $\mathbf {s}_i$ being the root node $r_i$:
where $F_p, F_c$ and $F_r$ are linear-projection functions to build representations for the parent, child and root node respectively and $W_a$ is the weight for bilinear transformation. Here, $f_{ij}$ is the edge weight between nodes $(i,j)$ in a weighted adjacency graph $\mathbf {F}$ and is computed for all pairs of sentences. Using $f_{ij}$ and $r_i$, we compute normalized attention scores $a_{ij}$ and $a_{i}^r $ using a variant of Kirchhoff’s matrix-tree theorem BIBREF12, BIBREF14 where $a_{ij}$ is the marginal probability of a dependency edge between sentences $(i,j)$ and $a_{i}^r $ is the probability of sentence $i$ being the root.
Using these probabilistic attention weights and the semantic vectors $\lbrace \mathbf {g}_{\mathbf {s}}\rbrace $, we compute the attended sentence representations as:
where $\mathbf {p}_{\mathbf {s}_i}$ is the context vector gathered from possible parents of sentence $i$, $\mathbf {c}_{\mathbf {s}_i}$ is the context vector gathered from possible children, and $\mathbf {g}_{root}$ is a special embedding for the root node. Here, the updated sentence representation $\textit {l}_{\mathbf {s}_i}$ incorporates the implicit structural information.
StructSum Model ::: Explicit Structure (ES) Attention
BIBREF2 showed that modeling coreference knowledge through anaphora constraints led to improved clarity or grammaticality in summaries. Taking inspiration from this, we choose coreference links across sentences as our explicit structure. First, we use an off-the-shelf coreference parser to identify coreferring mentions. We then build a coreference based sentence graph by adding a link between sentences $(\mathbf {s}_i, \mathbf {s}_j)$, if they have any coreferring mentions between them. This representation is then converted into a weighted graph by incorporating a weight on the edge between two sentences that is proportional to the number of unique coreferring mentions between them. We normalize these edge weights for every sentence, effectively building a weighted adjacency matrix $\mathbf {K}$ where $k_{ij}$ is given by:
where $m_i$ denotes the set of unique mentions in sentence $\mathbf {s}_i$, ($m_i$ $\bigcap $ $m_j$) denotes the set of co-referring mentions between the two sentences and $z$ is a latent variable representing a link in the coreference sentence graph. $\epsilon = 5e-4$ is a smoothing hyperparameter.
StructSum Model ::: Explicit Structure (ES) Attention ::: Incorporating explicit structure
Given contextual sentence representations $\lbrace \mathbf {h}_{\mathbf {s}}\rbrace $ and our explicit coreference based weighted adjacency matrix $\mathbf {K}$, we learn an explicit-structure aware representation as follows:
where $F_u$ and $F_e$ are linear projections and $\mathbf {e}_{\mathbf {s}_i}$ is an updated sentence representation which incorporates explicit structural information.
Finally, to combine the two structural representations, we concatenate the latent and explicit sentence vectors as: $\mathbf {h}_{\mathbf {s}_i} = [\mathbf {l}_{\mathbf {s}_i};\mathbf {e}_{\mathbf {s}_i}]$ to form encoder sentence representations of the source document. To provide every token representation with context of the entire document, we keep the same formulation as pointer-generator networks, where each token $w_{ij}$ is mapped to its hidden representation $\mathbf {h}_{w_{ij}}$ using a BiLSTM. The token representation is concatenated with their corresponding structure-aware sentence representation: $\mathbf {h}_{w_{ij}} = [\mathbf {h}_{w_{ij}};\mathbf {h}_{\mathbf {s}_i}]$ where $\mathbf {s}_i$ is the sentence to which the word $w_{ij}$ belongs. The resulting structure-aware token representations can be used to directly replace previous token representations as input to the decoder.
Experiments ::: Dataset:
We evaluate our approach on the CNN/Daily Mail corpus BIBREF15, BIBREF17 and use the same preprocessing steps as shown in BIBREF3. The CNN/DM summaries have an average of 66 tokens ($\sigma = 26$) and 4.9 sentences. Differing from BIBREF3, we truncate source documents to 700 tokens instead of 400 in training and validation sets to model longer documents with more sentences.
Experiments ::: Baselines:
We choose the following baselines based on their relatedness to the task and wide applicability:
BIBREF3 : We re-implement the base pointer-generator model and the additional coverage mechanism. This forms the base model of our implementation and hence our addition of modeling document structure can be directly compared to it.
BIBREF6 : This is a graph-based attention model that is closest in spirit to the method we present in this work. They use a graph attention module to learn attention between sentences, but cannot be easily used to induce interpretable document structures, since their attention scores are not constrained to learn structure. In addition to learning latent and interpretable structured attention between sentences, StructSum also introduces an explicit structure component to inject external document structure.
BIBREF7 : We compare with the DiffMask experiment with this work. This work introduces a separate content selector which tags words and phrases to be copied. The DiffMask variant is an end-to-end variant like ours and hence is included in our baselines.
Our baselines exclude Reinforcement Learning (RL) based systems as they aren't directly comparable, but our approach can be easily introduced in any encoder-decoder based RL system. Since we do not incorporate any pretraining, we do not compare with recent contextual representation based models BIBREF18.
Experiments ::: Hyperparameters:
Our encoder uses 256 hidden states for both directions in the one-layer LSTM, and 512 for the single-layer decoder. We use the adagrad optimizer BIBREF19 with a learning rate of 0.15 and an initial accumulator value of 0.1. We do not use dropout and use gradient-clipping with a maximum norm of 2. We selected the best model using early stopping based on the ROUGE score on the validation dataset as our criteria. We also used the coverage penalty during inference as shown in BIBREF7. For decoding, we use beam-search with a beam width of 3. We did not observe significant improvements with higher beam widths.
Results
Table TABREF8 shows the results of our work on the CNN/DM dataset. We use the standard ROUGE-1,2 and L BIBREF20 F1 metric to evaluate all our summarization output. We first observe that introducing the capability to learn latent structures already improves our performance on ROUGE-L. It suggests that modeling dependencies between sentences helps the model compose better long sequences w.r.t reference compared to baselines. We do not see a significant improvement in ROUGE-1 and ROUGE-2, hinting that we retrieve similar content words as the baseline but compose them into better contiguous sequences.
We observe similar results when using explicit structures only with the ES attention module. This shows that adding inductive bias in the form of coreference based sentence graphs helps compose long sequences. Our results here are close to the model that uses just LS attention. This demonstrates that LS attention induces good latent dependencies that make up for pure external coreference knowledge.
Finally, our combined model which uses both Latent and Explicit structure performs the best with a strong improvement of 1.08 points in ROUGE-L over our base pointer-generator model and 0.6 points in ROUGE-1. It shows that the latent and explicit information are complementary and a model can jointly leverage them to produce better summaries.
Modeling structure and adding inductive biases also helps a model to converge faster where the combined LS+ES Attention model took 126K iterations for training in comparison to 230K iterations required to train the plain pointer-generator network and an additional 3K iterations for the coverage loss BIBREF3.
Analysis
We present below analysis on the quality of summarization as compared to our base model, the pointer-generator network with coverage BIBREF3 and the reference.
Analysis ::: Analysis of Copying
Despite being an abstractive model, the pointer-generator model tends to copy very long sequences of words including whole sentences from the source document (also observed by BIBREF7). Table TABREF15 shows a comparison of the Average Length (Copy Len) of contiguous copied sequences greater than length 3. We observe that the pointer-generator baseline on average copies 16.61 continuous tokens from the source which shows the extractive nature of the model. This indicates that pointer networks, aimed at combining advantages from abstractive and extractive methods by allowing to copy content from the input document, tend to skew towards copying, particularly in this dataset. A consequence of this is that the model fails to interrupt copying at desirable sequence length.
In contrast, modeling document structure through StructSum reduces the length of copied sequences to 9.13 words on average reducing the bias of copying sentences in entirety. This average is closer to the reference (5.07 words) in comparison, without sacrificing task performance. StructSum learns to stop when needed, only copying enough content to generate a coherent summary.
Analysis ::: Content Selection and Abstraction
A direct outcome of copying shorter sequences is being able to cover more content from the source document within given length constraints. We observe that this leads to better summarization performance. In our analysis, we compute coverage by computing the number of source sentences from which sequences greater than length 3 are copied in the summary. Table TABREF15 shows a comparison of the coverage of source sentences in the summary content. We see that while the baseline pointer-generator model only copies from 12.1% of the source sentences, we copy content from 24.0% of the source sentences. Additionally, the average length of the summaries produced by StructSum remains mostly unchanged at 66 words on average compared to 61 of the baseline model. This indicates that StructSum produces summaries that draw from a wider selection of sentences from the original article compared to the baseline models.
BIBREF21 show that copying more diverse content in isolation does not necessarily lead to better summaries for extractive summarization. Our analysis suggests that this observation might not extend to abstractive summarization methods. The proportion of novel n-grams generated has been used in the literature to measure the degree of abstraction of summarization models BIBREF3. Figure FIGREF17 compares the percentage of novel n-grams in StructSum as compared to the baseline model. Our model produces novel trigrams 21.0% of the time and copies whole sentences only 21.7% of the time. In comparison, the pointer-generator network has only 6.1% novel trigrams and copies entire sentences 51.7% of the time. This shows that StructSum on average generates 14.7% more novel n-grams in comparison to the pointer-generator baseline.
Analysis ::: Layout Bias
Neural abstractive summarization methods applied to news articles are typically biased towards selecting and generating summaries based on the first few sentences of the articles. This stems from the structure of news articles, which present the salient information of the article in the first few sentences and expand in the subsequent ones. As a result, the LEAD 3 baseline, which selects the top three sentences of an article, is widely used in the literature as a strong baseline to evaluate summarization models applied to the news domain BIBREF22. BIBREF8 observed that the current summarization models learn to exploit the layout biases of current datasets and offer limited diversity in their outputs.
To analyze whether StructSum also holds the same layout biases, we compute a distribution of source sentence indices that are used for copying content (copied sequences of length 3 or more are considered). Figure FIGREF19 shows the comparison of coverage of sentences. The coverage of sentences in the reference summaries shows a high proportion of the top 5 sentences of any article being copied to the summary. Additionally, the reference summaries have a smoother tail end distribution with relevant sentences in all positions being copied. It shows that a smooth distribution over all sentences is a desirable feature. We notice that the sequence-to-sequence and pointer-generator framework (with and without coverage enabled) have a stronger bias towards the beginning of the article with a high concentration of copied sentences within the top 5 sentences of the article. In contrast, StructSum improves coverage slightly having a lower concentration of top 5 sentences and copies more tail end sentences than the baselines. However, although the modeling of structure does help, our model has a reasonable gap compared to the reference distribution. We see this as an area of improvement and a direction for future work.
Analysis ::: Document Structures
Similar to BIBREF12, we also look at the quality of the intermediate structures learned by the model. We use the Chu-Liu-Edmonds algorithm BIBREF23, BIBREF24 to extract the maximum spanning tree from the attention score matrix as our sentence structure. Table TABREF20 shows the frequency of various tree depths. We find that the average tree depth is 2.9 and the average proportion of leaf nodes is 88%, consistent with results from tree induction in document classification BIBREF25. Further, we compare latent trees extracted from StructSum with undirected graphs based on coreference and NER. These are constructed similarly to our explicit coreference based sentence graphs in §SECREF5 by linking sentences with overlapping coreference mentions or named entities. We measure the similarity between the learned latent trees and the explicit graphs through precision and recall over edges. The results are shown in Table TABREF22. We observe that our latent graphs have low recall with the linguistic graphs showing that our latent graphs do not capture the coreference or named entity overlaps explicitly, suggesting that the latent and explicit structures capture complementary information.
Figure FIGREF24 shows qualitative examples of our induced structures along with generated summaries from the StructSum model. The first example shows a tree with sentence 3 chosen as root, which was the key sentence mentioned in the reference. We notice that in both examples, the sentences in the lower level of the dependency tree contribute less to the generated summary. Along the same lines, in the examples source sentences used to generate summaries tend to be closer to the root node. In the first summary, all sentences from which content was drawn are either the root node or within depth 1 of the root node. Similarly, in the second example, 4 out of 5 source sentences were at depth=1 in the tree. In the two examples, generated summaries diverged from the reference by omitting certain sentences used in the reference. These sentences appear in the lower section of the tree giving us some insights on which sentences were preferred for the summary generation. Further, in example 1, we notice that the latent structures cluster sentences based on the main topic of the document. Sentences 1,2,3 differ from sentences 5,6,7 on the topic being discussed and our model has clustered the two sets separately.
Related Work
Prior to neural models for summarization, document structure played a critical role in generating relevant, diverse and coherent summaries. BIBREF26 formulated document summarization using linguistic features to construct a semantic graph of the document and building a subgraph for the summary. BIBREF27 leverage language-independent syntactic graphs of the source document to do unsupervised document summarization. BIBREF1 parse the source text into a set of AMR graphs, transform the graphs to summary graphs and then generate text from the summary graph. While such systems generate grammatical summaries and preserve linguistic quality BIBREF2, they are often computationally demanding and do not generalize well BIBREF21.
Data-driven neural models for summarization fall into extractive BIBREF13, BIBREF28 or abstractive BIBREF29, BIBREF3, BIBREF7, BIBREF30. BIBREF3 proposed a pointer-generator framework that learns to either generate novel in-vocabulary words or copy words from the source. This model has been the foundation for a lot of follow up work on abstractive summarization BIBREF7, BIBREF31, BIBREF32. Our model extends the pointer-generator model by incorporating latent structure and explicit structure knowledge, making our extension applicable to any of the followup work. BIBREF6 present a graph-based attention system to improve the saliency of summaries. While this model learns attention between sentences, it does not induce interpretable intermediate structures. A lot of recent work looks into incorporating structure into neural models. BIBREF32 infuse source side syntactic structure into the copy mechanism of the pointer-generator model. They identify explicit word-level syntactic features based on dependency parses and parts of speech tags and augment the decoder copy mechanism to attend to them. In contrast, we model sentence level dependency structures in the form of latent or induced structures and explicit coreference based structures. We do not identify any heuristic or salient features other than linking dependent sentences. BIBREF33 propose structural compression and coverage regularizers to provide an objective to neural models to generate concise and informative content. Here, they incorporate structural bias about the target summaries but we choose to model the structure of the source sentence to produce rich document representations. BIBREF34 induce latent document structure for aspect based summarization. BIBREF35 use present long document summarization model applicable for scientific papers, which attends to discourse sections in a document, while BIBREF36 propose an unsupervised model for review summarization which learns a latent discourse structure and uses it to summarize a review. BIBREF37 use discourse structures to improve coherence in blog summarization. These are all complementary directions to our work. To our knowledge, we are the first to simultaneously incorporate latent and explicit document structure in a single framework for document summarization.
Conclusion and Future Work
To summarize, our contributions are three-fold. We propose a framework for incorporating latent and explicit document structure in neural abstractive summarization. We introduce a novel explicit-attention module which can incorporate external linguistic structures, and we show one such application where we use coreference to enhance summarization. We show quantitative improvements on the ROUGE metric over strong summarization baselines and demonstrate improvements in abstraction and coverage through extensive qualitative analysis.
StructSum has demonstrated performance gain and higher quality output summaries; with a potential direction to study the role of latent structures in the interpretability of models in the future. Another possible direction is to investigate whether structured representations allow better generalization for transfer learning and summarization in other domains with limited data.

Question: By how much they improve over the previous state-of-the-art?
Answer: 1.08 points in ROUGE-L over our base pointer-generator model , 0.6 points in ROUGE-1

Question: Is there any evidence that encoders with latent structures work well on other tasks?
Answer: Yes