SlowGuess/ACL_2020_OCR huggingface dataset:
Wei Li $^{1}$ , Xinyan Xiao $^{1*}$ , Jiachen Liu $^{1}$ , Hua Wu $^{1}$ , Haifeng Wang $^{1}$ , Junping Du $^{2}$
<sup>1</sup>Baidu Inc., Beijing, China
{liwei85,xiaoxinyan,liujiachen,wu_hua,wanghaifeng}@baidu.com
junpingdu@126.com
Graphs that capture relations between textual units have great benefits for detecting salient information from multiple documents and generating overall coherent summaries. In this paper, we develop a neural abstractive multi-document summarization (MDS) model which can leverage well-known graph representations of documents such as similarity graph and discourse graph, to more effectively process multiple input documents and produce abstractive summaries. Our model utilizes graphs to encode documents in order to capture cross-document relations, which is crucial to summarizing long documents. Our model can also take advantage of graphs to guide the summary generation process, which is beneficial for generating coherent and concise summaries. Furthermore, pre-trained language models can be easily combined with our model, which further improve the summarization performance significantly. Empirical results on the WikiSum and MultiNews dataset show that the proposed architecture brings substantial improvements over several strong baselines.
Multi-document summarization (MDS) brings great challenges to the widely used sequence-to-sequence (Seq2Seq) neural architecture as it requires effective representation of multiple input documents and content organization of long summaries. For MDS, different documents may contain the same content, include additional information, and present complementary or contradictory information (Radev, 2000). So different from single document summarization (SDS), cross-document links are very important in extracting salient information, detecting redundancy and generating overall coherent summaries for MDS. Graphs that capture
relations between textual units have great benefits to MDS, which can help generate more informative, concise and coherent summaries from multiple documents. Moreover, graphs can be easily constructed by representing text spans (e.g. sentences, paragraphs etc.) as graph nodes and the semantic links between them as edges. Graph representations of documents such as similarity graph based on lexical similarities (Erkan and Radev, 2004) and discourse graph based on discourse relations (Christensen et al., 2013), have been widely used in traditional graph-based extractive MDS models. However, they are not well studied by most abstractive approaches, especially the end-to-end neural approaches. Few work has studied the effectiveness of explicit graph representations on neural abstractive MDS.
In this paper, we develop a neural abstractive MDS model which can leverage explicit graph representations of documents to more effectively process multiple input documents and distill abstractive summaries. Our model augments the end-to-end neural architecture with the ability to incorporate well-established graphs into both the document representation and summary generation processes. Specifically, a graph-informed attention mechanism is developed to incorporate graphs into the document encoding process, which enables our model to capture richer cross-document relations. Furthermore, graphs are utilized to guide the summary generation process via a hierarchical graph attention mechanism, which takes advantage of the explicit graph structure to help organize the summary content. Benefiting from the graph modeling, our model can extract salient information from long documents and generate coherent summaries more effectively. We experiment with three types of graph representations, including similarity graph, topic graph and discourse graph, which all significantly improve the MDS performance.
6232
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6232-6243 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Additionally, our model is complementary to most pre-trained language models (LMs), like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019b). They can be easily combined with our model to process much longer inputs. The combined model adopts the advantages of both our graph model and pre-trained LMs. Our experimental results show that our graph model significantly improves the performance of pre-trained LMs on MDS.
- Our work demonstrates the effectiveness of graph modeling in neural abstractive MDS. We show that explicit graph representations are beneficial for both document representation and summary generation.
- We propose an effective method to incorporate explicit graph representations into the neural architecture, and an effective method to combine pre-trained LMs with our graph model to process long inputs more effectively.
- Our model brings substantial improvements over several strong baselines on both WikiSum and MultiNews dataset. We also report extensive analysis results, demonstrating that graph modeling enables our model process longer inputs with better performance, and graphs with richer relations are more beneficial for MDS.<sup>1</sup>
2.1 Graph-based MDS
Most previous MDS approaches are extractive, which extract salient textual units from documents based on graph-based representations of sentences. Various ranking methods have been developed to rank textual units based on graphs to select most salient ones for inclusion in the final summary.
Erkan and Radev (2004) propose LexRank to compute sentence importance based on a lexical similarity graph of sentences. Mihalcea and Tauru (2004) propose a graph-based ranking model to extract salient sentences from documents. Wan (2008) further proposes to incorporate document-level information and sentence-to-document relations into the graph-based ranking process. A series of variants of the PageRank algorithm has been
further developed to compute the salience of textual units recursively based on various graph representations of documents (Wan and Xiao, 2009; Cai and Li, 2012). More recently, Yasunaga et al. (2017) propose a neural graph-based model for extractive MDS. An approximate discourse graph is constructed based on discourse markers and entity links. The salience of sentences is estimated using features from graph convolutional networks (Kipf and Welling, 2016). Yin et al. (2019) also propose a graph-based neural sentence ordering model, which utilizes entity linking graph to capture the global dependencies between sentences.
2.2 Abstractive MDS
Abstractive MDS approaches have met with limited success. Traditional approaches mainly include: sentence fusion-based (Banerjee et al., 2015; Filippova and Strube, 2008; Barzilay and McKeown, 2005; Barzilay, 2003), information extraction-based (Li, 2015; Pighin et al., 2014; Wang and Cardie, 2013; Genest and Lapalme, 2011; Li and Zhuge, 2019) and paraphrasing-based (Bing et al., 2015; Berg-Kirkpatrick et al., 2011; Cohn and Lapata, 2009). More recently, some researches parse the source text into AMR representation and then generate summary based on it (Liao et al., 2018).
Although neural abstractive models have achieved promising results on SDS (See et al., 2017; Paulus et al., 2018; Gehrmann et al., 2018; Celikyilmaz et al., 2018; Li et al., 2018a,b; Narayan et al., 2018; Yang et al., 2019a; Sharma et al., 2019; Perez-Beltrachini et al., 2019), it's not straightforward to extend them to MDS. Due to the lack of sufficient training data, earlier approaches try to simply transfer SDS model to MDS task (Lebanoff et al., 2018; Zhang et al., 2018; Baumel et al., 2018) or utilize unsupervised models relying on reconstruction objectives (Ma et al., 2016; Chu and Liu, 2019). Later, Liu et al. (2018) propose to construct a large scale MDS dataset (namely WikiSum) based on Wikipedia, and develop a Seq2Seq model by considering the multiple input documents as a concatenated flat sequence. Fan et al. (2019) further propose to construct a local knowledge graph from documents and then linearize the graph into a sequence to better sale Seq2Seq models to multi-document inputs. Fabbri et al. (2019) also introduce a middle-scale (about 50K) MDS news dataset (namely MultiNews), and propose an end-to-end model by incorporating traditional MMR-based
$^{1}$ Codes and results are in: https://github.com/PaddlePaddle/Research/tree/master/NLP/ACL2020-GraphSum
6233
Figure 1: Illustration of our model, which follows the encoder-decoder architecture. The encoder is a stack of transformer layers and graph encoding layers, while the decoder is a stack of graph decoding layers. We incorporate explicit graph representations into both the graph encoding layers and graph decoding layers.
extractive model with a standard Seq2Seq model. The above Seq2Seq models haven't study the importance of cross-document relations and graph representations in MDS.
Most recently, Liu and Lapata (2019a) propose a hierarchical transformer model to utilize the hierarchical structure of documents. They propose to learn cross-document relations based on self-attention mechanism. They also propose to incorporate explicit graph representations into the model by simply replacing the attention weights with a graph matrix, however, it doesn't achieve obvious improvement according to their experiments. Our work is partly inspired by this work, but our approach is quite different from theirs. In contrast to their approach, we incorporate explicit graph representations into the encoding process via a graph-informed attention mechanism. Under the guidance of explicit relations in graphs, our model can learn better and richer cross-document relations, thus achieves significantly better performance. We also leverage the graph structure to guide the summary decoding process, which is beneficial for long summary generation. Additionally, we combine the advantages of pretrained LMs into our model.
2.3 Summarization with Pretrained LMs
Pretrained LMs (Peters et al., 2018; Radford et al.; Devlin et al., 2019; Dong et al., 2019; Sun et al., 2019) have recently emerged as a key technology for achieving impressive improvements in a wide variety of natural language tasks, including both language understanding and language generation (Edunov et al., 2019; Rothe et al., 2019). Liu and
Lapata (2019b) attempt to incorporate pre-trained BERT encoder into SDS model and achieves significant improvements. Dong et al. (2019) further propose a unified LM for both language understanding and language generation tasks, which achieves state-of-the-art results on several generation tasks including SDS. In this work, we propose an effective method to combine pretrained LMs with our graph model and make them be able to process much longer inputs effectively.
In order to process long source documents more effectively, we follow Liu and Lapata (2019a) in splitting source documents into multiple paragraphs by line-breaks. Then the graph representation of documents is constructed over paragraphs. For example, a similarity graph can be built based on cosine similarities between tfidf representations of paragraphs. Let  $\mathbb{G}$  denotes a graph representation matrix of the input documents, where  $\mathbb{G}[i][j]$  indicates the relation weights between paragraph  $P_{i}$  and  $P_{j}$ . Formally, the task is to generate the summary  $S$  of the document collection given  $L$  input paragraphs  $P_{1},\ldots ,P_{L}$  and their graph representation  $\mathbb{G}$ .
Our model is illustrated in Figure 1, which follows the encoder-decoder architecture (Bahdanau et al., 2015). The encoder is composed of several token-level transformer encoding layers and paragraph-level graph encoding layers which can be stacked freely. The transformer encoding layer follows the Transformer architecture introduced in Vaswani et al. (2017), encoding contextual information for tokens within each paragraph. The
6234
graph encoding layer extends the Transformer architecture with a graph attention mechanism to incorporate explicit graph representations into the encoding process. Similarly, the decoder is composed of a stack of graph decoding layers. They extend the Transformer with a hierarchical graph attention mechanism to utilize explicit graph structure to guide the summary decoding process. In the following, we will focus on the graph encoding layer and graph decoding layer of our model.
3.1 Graph Encoding Layer
As shown in Figure 1, based on the output of the token-level transformer encoding layers, the graph encoding layer is used to encode all documents globally. Most existing neural work only utilizes attention mechanism to learn latent graph representations of documents where the graph edges are attention weights (Liu and Lapata, 2019a; Niculae et al., 2018; Fernandes et al., 2018). However, much work in traditional MDS has shown that explicit graph representations are very beneficial to MDS. Different types of graphs capture different kinds of semantic relations (e.g. lexical relations or discourse relations), which can help the model focus on different facets of the summarization task. In this work, we propose to incorporate explicit graph representations into the neural encoding process via a graph-informed attention mechanism. It takes advantage of the explicit relations in graphs to learn better inter-paragraph relations. Each paragraph can collect information from other related paragraphs to capture global information from the whole input.
Graph-informed Self-attention The graph-informed self-attention extends the self-attention mechanism to consider the pairwise relations in explicit graph representations. Let  $x_{i}^{l - 1}$  denotes the output of the  $(l - 1)$ -th graph encoding layer for paragraph  $P_{i}$ , where  $x_{i}^{0}$  is just the input paragraph vector. For each paragraph  $P_{i}$ , the context representation  $u_{i}$  can be computed as a weighted sum of linearly transformed paragraph vectors:
$$
\begin{array}{l} \alpha_ {i j} = \operatorname {s o f t m a x} \left(e _ {i j} + \mathfrak {R} _ {i j}\right) \\ e _ {i j} = \frac {\left(x _ {i} ^ {l - 1} W _ {Q}\right) \left(x _ {j} ^ {l - 1} W _ {K}\right) ^ {T}}{\sqrt {d _ {\text {h e a d}}}} \tag {1} \\ u _ {i} = \sum_ {j = 1} ^ {L} \alpha_ {i j} (x _ {j} ^ {l - 1} W _ {V}) \\ \end{array}
$$
where  $W_{K}, W_{Q}$  and  $W_{V} \in \mathbb{R}^{d*d}$  are parameter weights.  $e_{tj}$  denotes the latent relation weight between paragraph  $P_{i}$  and  $P_{j}$ . The main difference of our graph-informed self-attention is the additional pairwise relation bias  $\Re_{ij}$ , which is computed as a Gaussian bias of the weights of graph representation matrix  $\mathbb{G}$ :
$$
\mathfrak {R} _ {i j} = - \frac {\left(1 - \mathbb {G} [ i ] [ j ]\right) ^ {2}}{2 \sigma^ {2}} \tag {2}
$$
where  $\sigma$  denotes the standard deviation that represents the influence intensity of the graph structure. We set it empirically by tuning on the development dataset. The gaussian bias  $R_{ij} \in (-inf, 0]$  measures the tightness between the paragraphs  $P_i$  and  $P_j$ . Due to the exponential operation in softmax function, the gaussian bias approximates to multiply the latent attention distribution by a weight  $\in (0, 1]$ .
In our graph-attention mechanism, the term  $e_{ij}$  in Equation 1 keeps the ability to model latent dependencies between any two paragraphs, and the term  $\Re_{ij}$  incorporates explicit graph representations as prior constraints into the encoding process. This way, our model can learn better and richer inter-paragraph relations to obtain more informative paragraph representations.
Then, a two-layer feed-forward network with ReLU activation function and a high-way layer normalization are applied to obtain the vector of each paragraph  $x_{i}^{l}$ :
$$
p _ {i} ^ {l} = W _ {o 2} \operatorname {R e L U} \left(W _ {o 1} \left(u _ {i} + x _ {i} ^ {l - 1}\right)\right) \tag {3}
$$
$$
x _ {i} ^ {l} = \operatorname {L a y e r N o r m} \left(p _ {i} ^ {l} + x _ {i} ^ {l - 1}\right)
$$
where  $W_{o1} \in \mathbb{R}^{d_{ff}*d}$  and  $W_{o2} \in \mathbb{R}^{d*d_{ff}}$  are learnable parameters,  $d_{ff}$  is the hidden size of the feedforward layer.
3.2 Graph Decoding Layer
Graphs can also contribute to the summary generation process. The relations between textual units can help to generate more coherent or concise summaries. For example, Christensen et al. (2013) propose to leverage an approximate discourse graph to help generate coherent extractive summaries. The discourse relations between sentences are used to help order summary sentences. In this work, we propose to incorporate explicit graph structure into the end-to-end summary decoding process. Graph edges are used to guide the summary generation process via a hierarchical graph attention, which
6235
is composed by a global graph attention and a local normalized attention. As other components in the graph decoding layer are similar to the Transformer architecture, we focus on the extension of hierarchical graph attention.
Global Graph Attention The global graph attention is developed to capture the paragraph-level context information in the encoder part. Different from the context attention in Transformer, we utilize the explicit graph structure to regularize the attention distributions so that graph representations of documents can be used to guide the summary generation process.
Let  $y_{t}^{l - 1}$  denotes the output of the  $(l - 1)$ -th graph decoding layer for the  $t$ -th token in the summary. We assume that each token will align with several related paragraphs and one of them is at the central position. Since the prediction of the central position depends on the corresponding query token, we apply a feed-forward network to transform  $y_{t}^{l - 1}$  into a positional hidden state, which is then mapped into a scalar  $s_{t}$  by a linear projection:
$$
s _ {t} = L * s i g m o i d (U _ {p} ^ {T} t a n h (W _ {p} y _ {t} ^ {l - 1})) \qquad (4)
$$
where  $W_{p} \in \mathbb{R}^{d* d}$  and  $U_{p} \in \mathbb{R}^{d}$  denote weight matrix.  $s_{t}$  indicates the central position of paragraphs that are mapped by the  $t$ -th summary token. With the central position, other paragraphs are determined by the graph structure. Then an attention distribution over all paragraphs under the regularization of the graph structure can be obtained:
$$
\beta_ {t j} = s o f t m a x (e _ {t j} - \frac {(1 - \mathbb {G} [ s _ {t} ] [ j ]) ^ {2}}{2 \sigma^ {2}}) \qquad (5)
$$
where  $e_{tj}$  denotes the attention weight between token vector  $y_{t}^{l - 1}$  and paragraph vector  $x_{j}$ , which is computed similarly to Equation 1. The global context vector can be obtained as a weighted sum of paragraph vectors:  $g_{t} = \sum_{j = 1}^{L}\beta_{tj}x_{j}$
In our decoder, graphs are also modeled as a Gaussian bias. Different from the encoder, a central mapping position is firstly decided and then graph relations corresponding to that position are used to regularize the attention distributions  $\beta_{tj}$ . This way, the relations in graphs are used to help align the information between source input and summary output globally, thus guiding the summary decoding process.
Local Normalized Attention Then, a local normalized attention is developed to capture the token-level context information within each paragraph.
The local attention is applied to each paragraph independently and normalized by the global graph attention. This way, our model can process longer inputs effectively.
Let  $\gamma_{t,ji}$  denotes the local attention distributions of the  $t$ -th summary token over the  $i$ -th token in the  $j$ -th input paragraph, the normalized attention is computed by:
$$
\hat {\gamma} _ {t, j i} = \gamma_ {t, j i} \beta_ {t j} \tag {6}
$$
and the local context vector can be computed as a weighted sum of token vectors in all paragraphs:  $l_{t} = \sum_{j = 1}^{L}\sum_{k = 1}^{n}\hat{\gamma}_{t,ji}x_{ji}$
Finally, the output of the hierarchical graph attention component is computed by concatenating and linearly transforming the global and local context vector:
$$
d _ {t} = U _ {d} ^ {T} \left[ g _ {t}, l _ {t} \right] \tag {7}
$$
where  $U_{d}\in \mathbb{R}^{2d*d}$  is a weight matrix. Through combining the local and global context, the decoder can utilize the source information more effectively.
3.3 Combined with Pre-trained LMs
Our model can be easily combined with pre-trained LMs. Pre-trained LMs are mostly based on sequential architectures which are more effective on short text. For example, both BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are pre-trained with maximum 512 tokens. Liu and Lapata (2019b) propose to utilize BERT on single document summarization tasks. They truncate the input documents to 512 tokens on most tasks. However, thanks to the graph modeling, our model can process much longer inputs. A natural idea is to combine our graph model with pretrained LMs so as to combine the advantages of them. Specifically, the token-level transformer encoding layer of our model can be replaced by a pre-trained LM like BERT.
In order to take full advantage of both our graph model and pre-trained LMs, the input documents are formatted in the following way:
[CLS] first paragraph [SEP] [CLS] second paragraph [SEP] ... [CLS] last paragraph [SEP]
Then they are encoded by a pre-trained LM, and the output vector of the “[CLS)” token is used as the vector of the corresponding paragraph. Finally, all paragraph vectors are fed into our graph encoder to learn global representations. Our graph decoder is further used to generate the summaries.
6236
Graph Representations We experiment with three well-established graph representations: similarity graph, topic graph and discourse graph. The similarity graph is built based on tfidf cosine similarities between paragraphs to capture lexical relations. The topic graph is built based on LDA topic model (Blei et al., 2003) to capture topic relations between paragraphs. The edge weights are cosine similarities between the topic distributions of the paragraphs. The discourse graph is built to capture discourse relations based on discourse markers (e.g. however, moreover), co-reference and entity links as in Christensen et al. (2013). Other types of graphs can also be used in our model. In our experiments, if not explicitly stated, we use the similarity graph by default as it has been most widely used in previous work.
WikiSum Dataset We follow Liu et al. (2018) and Liu and Lapata (2019a) in treating the generation of lead Wikipedia sections as a MDS task. The source documents are reference webpages of the Wikipedia article and top 10 search results returned by Google, while the summary is the Wikipedia article's first section. As the source documents are very long and messy, they are split into multiple paragraphs by line-breaks. Further, the paragraphs are ranked by the title and top ranked paragraphs are selected as input for MDS systems. We directly utilize the ranking results from Liu and Lapata (2019a) and top-40 paragraphs are used as source input. The average length of each paragraph and the target summary are 70.1 tokens and 139.4 tokens, respectively. For the seq2seq baselines, paragraphs are concatenated as a sequence in the ranking order, and lead tokens are used as input. The dataset is split into 1,579,360 instances for training, 38,144 for validation and 38,205 for testing, similar to Liu and Lapata (2019a). We build similarity graph representations over paragraphs on this dataset.
MultiNews Dataset Proposed by Fabbri et al. (2019), MultiNews dataset consists of news articles and human-written summaries. The dataset comes from a diverse set of news sources (over 1500 sites). Different from the WikiSum dataset, MultiNews is more similar to the traditional MDS dataset such as DUC, but is much larger in scale. As in Fabbri et al. (2019), the dataset is split into 44,972 instances for