SlowGuess/ACL_2020_OCR huggingface dataset:
Hanqi Jin, Tianming Wang, Xiaojun Wan
{jinhanqi,wangtm,wanxiao jun}@pku.edu.cn
In this paper, we propose a multi-granularity interaction network for extractive and abstractive multi-document summarization, which jointly learn semantic representations for words, sentences, and documents. The word representations are used to generate an abstractive summary while the sentence representations are used to produce an extractive summary. We employ attention mechanisms to interact between different granularity of semantic representations, which helps to capture multi-granularity key information and improves the performance of both abstractive and extractive summarization. Experiment results show that our proposed model substantially outperforms all strong baseline methods and achieves the best results on the Multi-News dataset.
Document summarization aims at producing a fluent, condensed summary for given documents. Single document summarization has shown promising results with sequence-to-sequence models that encode a source document and then decode it into a summary (See et al., 2017; Paulus et al., 2018; Gehrmann et al., 2018; Celikyilmaz et al., 2018). Multi-document summarization requires producing a summary from a cluster of thematically related documents, where the given documents complement and overlap each other. Multi-document summarization involves identifying important information and filtering out redundant information from multiple input sources.
There are two primary methodologies for multi-document summarization: extractive and abstractive. Extractive methods directly select important sentences from the original, which are relatively simple. Cao et al. (2015) rank sentences with a recursive neural network. Yasunaga et al. (2017)
employ a Graph Convolutional Network (GCN) to incorporate sentence relation graphs to improve the performance for the extractive summarization. Abstractive methods can generate new words and new sentences, but it is technically more difficult than extractive methods. Some works on multi-document summarization simply concatenate multiple source documents into a long flat sequence and model multi-document summarization as a long sequence-to-sequence task (Liu et al., 2018; Fabbri et al., 2019). However, these approaches don't take the hierarchical structure of document clusters into account, while the too-long input often leads to the degradation in document summarization (Cohan et al., 2018; Liu and Lapata, 2019). Recently, hierarchical frameworks have shown their effectiveness on multi-document summarization (Zhang et al., 2018; Liu and Lapata, 2019). These approaches usually use multiple encoders to model hierarchical relationships in the discourse structure, but other methods to incorporate the structural semantic knowledge have not been explored. The combination of extractive and abstractive has been explored in single document summarization. Chen and Bansal (2018) use the extracted sentences as the input of the abstractive summarization. Subramanian et al. (2019) concatenate the extracted summary to the original document as the input of the abstractive summarization.
In this work, we treat documents, sentences, and words as the different granularity of semantic units, and connect these semantic units within a three-granularity hierarchical relation graph. With the multi-granularity hierarchical structure, we can unify extractive and abstractive summarization into one architecture simultaneously. Extractive summarization operates on sentence-granularity and directly supervises the sentence representations while abstractive summarization operates on word-granularity and directly supervises the word repre
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6244-6254 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
sentations. We propose a novel multi-granularity interaction network to enable the supervisions to promote the learning of all granularity representations. We employ the attention mechanism to encode the relationships between the same semantic granularity and hierarchical relationships between the different semantic granularity, respectively. And we use a fusion gate to integrate the various relationships for updating the semantic representations. The decoding part consists of a sentence extractor and a summary generator. The sentence extractor utilizes the sentence representations to select sentences, while the summary generator utilizes the word representations to generate a summary. The two tasks are trained in a unified architecture to promote the recognition of important information simultaneously.
We evaluate our model on the recently released Multi-News dataset and our proposed architecture brings substantial improvements over several strong baselines. We explore the influence of semantic units with different granularity, and the ablation study shows that joint learning of extractive and abstractive summarization in a unified architecture improves the performance.
In summary, we make the following contributions in this paper:
- We establish multi-granularity semantic representations for documents, sentences, and words, and propose a novel multi-granularity interaction network to encode multiple input documents.
- Our approach can unify the extractive and abstractive summarization into one architecture with interactive semantic units and promote the recognition of important information in different granularities.
- Experimental results on the Multi-News dataset show that our approach substantially outperforms several strong baselines and achieves state-of-the-art performance. Our code is publicly available at https://github.com/zhongxia96/MGSum.
The methods for multi-document summarization can generally be categorized to extractive and abstractive. The extractive methods produce a summary by extracting and merging sentences from
the input documents, while the abstractive methods generate a summary using arbitrary words and expressions based on the understanding of the documents. Due to the lack of available training data, most previous multi-document summarization methods were extractive (Erkan and Radev, 2004; Christensen et al., 2013; Yasunaga et al., 2017).
Since the neural abstractive models have achieved promising results on single-document summarization (See et al., 2017; Paulus et al., 2018; Gehrmann et al., 2018; Çelikyilmaz et al., 2018), some works trained abstractive summarization models on a single document dataset and adjusted the model to adapt the multi-document summarization task. Zhang et al. (2018) added a document set encoder into the single document summarization framework and tuned the pre-trained model on the multi-document summarization dataset. Lebanoff et al. (2018) combined an extractive summarization algorithm (MMR) for sentence extraction to reweight the original sentence importance distribution learned in the single document abstractive summarization model. Recently, two large scale multi-document summarization datasets have been proposed, one for very long input, aimed at generating Wikipedia (Liu et al., 2018) and another dedicated to generating a comprehensive summarization of multiple real-time news (Fabbri et al., 2019). Liu et al. (2018) concatenated multiple source documents into a long flat text and introduced a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder-decoder architectures. Liu and Lapata (2019) introduced intermediate document representations and simply add the document representations to word representations for modeling the cross-document relationships. Compared with our proposed multi-granularity method, Liu and Lapata (2019) inclined to the traditional bottom-up hierarchical method and don't effectively utilize the hierarchical representations while ignoring the hierarchical relationships of sentences. Fabbri et al. (2019) incorporated MMR into a hierarchical pointer-generator network to address the information redundancy in multi-document summarization.
Our model consists of a multi-granularity encoder, a sentence extractor, and a summary generator. Firstly, the multi-granularity encoder reads multiple input documents and learns the multi
6245
granularity representations for words, sentences, and documents. Self-attention mechanisms are employed for capturing semantic relationships of the representations with same granularity, while cross-attention mechanisms are employed for the information interaction between representations with different granularity. Fusion gates are used for integrating the information from different attention mechanisms. Then the sentence extractor scores sentences according to the learned sentence representations. Meanwhile, the summary generator produces the abstractive summary by attending to the word representations. In the following sections, we will describe the multi-granularity encoder, the sentence extractor, and the summary generator, respectively.
3.1 Multi-Granularity Encoder
Given a cluster of documents, we establish explicit representations for documents, sentences, and words, and connect them within a hierarchical semantic relation graph. The multi-granularity encoder is a stack of  $L_{1}$  identical layers. Each layer has two sub-layers: the first is the multi-granularity attention layer, and the second is multiple fully connected feed-forward networks. The multi-granularity attention sub-layer transfers semantic information between the different granularity and the same granularity, while the feed-forward network further aggregates the multi-granularity information. We employ multi-head attention to encode multi-granularity information and use a fusion gate to propagate semantic information to each other. Figure 1 shows the overview of the multi-granularity encoder layer, and Figure 2 illustrates how the semantic representations are updated, which takes the sentence representation as an example.
Let  $w_{i,j,k}$  be the  $k$ -th word of the sentence  $s_{i,j}$  in the document  $d_i$ . At the bottom of the encoder stack, each input word  $w_{i,j,k}$  is converted into the vector representation  $e_{i,j,k}$  by learned embeddings. We assign positional encoding to indicate the position of the word  $w_{i,j,k}$  and three positions need to be considered, namely  $i$  (the rank of the document),  $j$  (the position of the sentence within the document),  $k$  (the position of the word within the sentence). We concatenate the three position embedding  $PE_i, PE_j,$  and  $PE_k$  to get the final position embedding  $p_{i,j,k}$ . The input word representation can be obtained by simply adding the word
Figure 1: The overview of the multi-granularity encoder layer.
embedding  $e_{i,j,k}$  and the position embedding  $p_{i,j,k}$ :
$$
p _ {i, j, k} = \left[ P E _ {i}; P E _ {j}; P E _ {k} \right] \tag {1}
$$
$$
h _ {w _ {i, j, k}} ^ {0} = e _ {i, j, k} + p _ {i, j, k}
$$
where the definition of positional encoding  $PE$  is consistent with the Transformer (Vaswani et al., 2017). For convenience, we denote the output of  $l$ -th multi-granularity encoder layer as  $h^l$  and the input for the first layer as  $h^0$ . Symbols with subscripts  $w_{i,j,k}, s_{i,j}$  and  $d_i$  are used to denote word, sentence, and document granularities, respectively. Both sentence representations  $h_{s_i,j}^0$  and document representations  $h_{d_i}^0$  are initialized to zeros.
In each multi-granularity attention sub-layers, the word representation is updated by the information of word granularity and sentence granularity. We perform multi-head self-attention across the word representations in the same sentence  $h_{w_i,j,*}^{l-1} = \{h_{w_i,j,k}^{l-1} | w_{i,j,k} \in s_{i,j}\}$  to get the context representation  $\tilde{h}_{w_i,j,k}^l$ . In order to propagate semantic information from sentence granularity to the word granularity, we duplicate sentence-aware representation  $\overleftarrow{h}_{w_i,j,k}^l$  from corresponding sentence  $s_{i,j}$  and employ a fusion gate to integrate  $\tilde{h}_{w_i,j,k}^l$  and  $\overleftarrow{h}_{w_i,j,k}^l$  to get the updated word representation  $f_{w_i,j,k}^l$ .
$$
f _ {w _ {i, j, k}} ^ {l} = \operatorname {F u s i o n} \left(\tilde {h} _ {w _ {i, j, k}} ^ {l}, \overleftarrow {h} _ {w _ {i, j, k}} ^ {l}\right)
$$
$$
\tilde {h} _ {w _ {i, j, k}} ^ {l} = \text {M H A t t} \left(h _ {w _ {i, j, k}} ^ {l - 1}, h _ {w _ {i, j, *}} ^ {l - 1}\right) \tag {2}
$$
$$
\stackrel {\leftarrow} {h} _ {w _ {i, j, k}} ^ {l} = h _ {s _ {i, j}} ^ {l - 1}
$$
where MHAtt denotes the multi-head attention proposed in Vaswani et al. (2017) and Fusion denotes the fusion gate.  $h_{w_{i,j,k}}^{l-1}$  is the query and  $h_{w_{i,j,*}}^{l-1}$  are
6246
Figure 2: The multi-granularity encoder layer for updating sentence representation. The sentence representation is updated by using two fusion gates to integrate the information from different granularities.
the keys and values for attention. The fusion gate works as
$$
\mathrm {z} = \sigma \left([ x; y ] W _ {f} + b _ {f}\right)
$$
$\operatorname {Fusion}(x,y) = \operatorname {z}x + (1 - \operatorname {z})y$
$$
(3)
$$
where  $\sigma$  is the sigmoid function, parameters  $W_{f}\in \mathbb{R}^{2*d_{model}\times 1}$  and  $b_{f}\in \mathbb{R}$ .
The sentence representation is updated from three sources: (1) We take the sentence representation  $h_{s_i,j}^{l-1}$  as the query, the word representations  $h_{w_{i,j,*}}^{l-1} = \{h_{w_{i,j,k}}^{l-1}|w_{i,j,k}\in s_{i,j}\}$  as the keys and values, to perform multi-head cross-attention to get the intermediate word-aware representation  $\overrightarrow{h}_{s_i,j}^{l-1}$ ; (2) Multi-head self-attention across sentence representations  $h_{s_i,*}^{l-1} = \{h_{s_i,j}^{l-1}|s_{i,j}\in d_i\}$  is performed to get the context representation  $\tilde{h}_{s_i,j}^l$ ; (3) In order to propagate document granularity semantic information to the sentence, we duplicate the document-aware representation  $\overline{h}_{s_i,j}^l$  from corresponding document  $d_j$ .
$$
\begin{array}{l} \vec {h} _ {s _ {i, j}} ^ {l} = \mathrm {M H A t t} \left(h _ {s _ {i, j}} ^ {l - 1}, h _ {w _ {i, j, *}} ^ {l - 1}\right) \\ \tilde {h} _ {s _ {i, j}} ^ {l} = \mathrm {M H A t t} \left(h _ {s _ {i, j}} ^ {l - 1}, h _ {s _ {i, *}} ^ {l - 1}\right) \tag {4} \\ \stackrel {\leftarrow} {h} _ {s _ {i, j}} ^ {l} = h _ {d _ {i}} ^ {l - 1} \\ \end{array}
$$
Semantic representations from the three sources are fused by two fusion gate to get the updated sentence representation  $f_{s_{i,j}}^{l}$ .
$$
f _ {s _ {i, j}} ^ {l} = \text {F u s i o n} \left(\text {F u s i o n} \left(\overrightarrow {h} _ {s _ {i, j}} ^ {l}, \overleftarrow {h} _ {s _ {i, j}} ^ {l}\right), \tilde {h} _ {s _ {i, j}} ^ {l}\right) \tag {5}
$$
To update the document representation, multihead self-attention across all document representa
tions  $h_{d_*}^{l-1} = \{h_{d_i}^{l-1}\}$  is performed to get the context representation  $\tilde{h}_{d_i}^l$ . Meanwhile, we take the document representation  $h_{d_i}^{l-1}$  as the query, sentence representations  $\{h_{s_{i,j}}^{l-1}|s_{i,j} \in d_i\}$  as the keys and values to perform multi-head cross-attention to get the intermediate sentence-aware representation  $\overrightarrow{h}_{d_i}^l$ . A fusion gate is used to aggregate the above outputs  $\tilde{h}_{d_i}^l$  and  $\overrightarrow{h}_{d_i}^l$ .
$$
f _ {d _ {i}} ^ {l} = \operatorname {F u s i o n} \left(\tilde {h} _ {d _ {i}} ^ {l}, \overrightarrow {h} _ {d _ {i}} ^ {l}\right)
$$
$$
\tilde {h} _ {d _ {i}} ^ {l} = \mathrm {M H A t t} \left(h _ {d _ {i}} ^ {l - 1}, h _ {d _ {*}} ^ {l - 1}\right) \tag {6}
$$
$$
\overrightarrow {h} _ {d _ {i}} ^ {l} = \mathrm {M H A t t} \left(h _ {d _ {i}} ^ {l - 1}, h _ {s _ {i, *}} ^ {l - 1}\right)
$$
The feed-forward network FFN is used to transform multiple-granularity semantic information further. To construct deep network, we use the residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) to connect adjacent layers.
$$
\tilde {h} = \text {L a y e r N o r m} \left(h ^ {l - 1} + f ^ {l}\right) \tag {7}
$$
$$
h ^ {l} = \operatorname {L a y e r N o r m} (\tilde {h} + \operatorname {F F N} (\tilde {h}))
$$
where  $l \in [1, L_1]$ , FFN consists of two linear transformations with a ReLU activation in between. Note that we used different FFN and LayerNorm for the different granularity. The final representation  $h_s^{L_1}$  is fed to the sentence extractor while  $h_w^{L_1}$  is fed to the summary generator. For convenience, we denote  $h_s^{L_1}$  as  $o_s$ , and  $h_w^{L_1}$  as  $o_w$ .
3.2 Sentence Extractor
we build a classifier to select sentences based on the sentence representations  $o_{s}$  from the multi-granularity encoder. The classifier uses a linear transformation layer with the sigmoid activation function to get the prediction score for each sentence
$$
\tilde {y} _ {s} = \sigma \left(o _ {s} W _ {o} + b _ {o}\right) \tag {8}
$$
where  $\sigma$  is the sigmoid function, parameters  $W_{o}\in \mathbb{R}^{d_{model}\times 1}$  and  $b_{o}\in \mathbb{R}$ .
These scores are used to sort the sentences of multiple documents and produce the extracted summary.
3.3 Summary Generator
The summary generator in our model is also a stack of  $L_{2}$  identical layers. The layer consists of three parts: a masked multi-head self-attention mechanism, a multi-head cross-attention mechanism, and
6247
a fully connected feed-forward network. As the input and output of multi-document summarization are generally long, the multi-head attention degenerates as the length increases (Liu and Lapata, 2019). Following Zhao et al. (2019) 's idea, we adopt a sparse attention mechanism where each query only attends to the top-  $k$  values according to their weights calculated by the keys rather than all values in the original attention (Vaswani et al., 2017). And  $k$  is a hyper-parameter. This ensures that the generator focuses on critical information in the input and ignores much irrelevant information. We denote the multi-head sparse attention as MSAttn.
Similar to the multi-granularity encoder, we add the positional encoding of words in the summary to the input embedding at the bottom of the decoder stack. We denote the output of the  $l$ -th layer as  $g^l$  and the input for the first layer as  $g^0$ . The self-attention sub-layer with masking mechanism is used to encode the decoded information. The masking mechanism ensures that the prediction of the position  $t$  depends only on the known output of the position before  $t$ .
$$
\tilde {g} = \operatorname {L a y e r N o r m} \left(g ^ {l - 1} + \operatorname {M S A t t n} \left(g ^ {l - 1}, g ^ {l - 1}\right)\right) \tag {9}
$$
The cross-attention sub-layer take the self-attention output  $\tilde{g}$  as the queries and the multi-granularity encoder output  $o_w$  as keys and values to performs multi-head sparse attention. The feedforward network is used to further transform the outputs.
$$
\begin{array}{l} c = \text {L a y e r N o r m} (\tilde {g} + \text {M S A t t} (\tilde {g}, o _ {w})) \\ g ^ {l} = \text {L a y e r N o r m} (c + \operatorname {F F N} (c)) \tag {10} \\ \end{array}
$$
The generation distribution  $p_t^g$  over the target vocabulary is calculated by feeding the output  $g_t^{L_2}$  to a softmax layer.
$$
p _ {t} ^ {g} = \operatorname {s o f t m a x} \left(g _ {t} ^ {L _ {2}} W _ {g} + b _ {g}\right) \tag {11}
$$
where  $W_{g}\in \mathbb{R}^{d_{model}\times d_{vocab}},b_{g}\in \mathbb{R}^{d_{vocab}}$  and  $d_{vocab}$  is the size of target vocabulary.
The copy mechanism (Gu et al., 2016) is employed to tackle the problem of out-of-vocabulary (OOV) words. We compute the copy attention  $\varepsilon_{t}$  with the decoder output  $g^{L_2}$  and the input representations  $o_w$ , and further obtain copy distribution  $p_t^c$ .
$$
\begin{array}{l} \varepsilon_ {t} = \operatorname {s o f t m a x} \left(g _ {t} ^ {L _ {2}} o _ {w} ^ {\top} + b _ {\varepsilon}\right) \\ p _ {t} ^ {c} = \sum_ {i, j, k} \varepsilon_ {t} z _ {i, j, k} ^ {\top} \tag {12} \\ \end{array}
$$
where  $z_{i,j,k}$  is the one-hot indicator vector for  $w_{i,j,k}$  and  $b_{\varepsilon} \in \mathbb{R}^{d_{\text{vocab}}}$ .
A gate is used over the the decoder output  $g^{L_2}$  to control generating words from the vocabulary or copying words directly from the source text. The final distribution  $p_t$  is the "mixture" of the two distributions  $p_t^g$  and  $p_t^c$ .
$$
\eta_ {t} = \sigma \left(g _ {t} ^ {L _ {2}} W _ {\eta} + b _ {\eta}\right) \tag {13}
$$
$$
p _ {t} = \eta_ {t} * p _ {t} ^ {g} + (1 - \eta_ {t}) * p _ {t} ^ {c}
$$
where  $\sigma$  is the sigmoid function,  $W_{\eta} \in \mathbb{R}^{d_{model} \times 1}, b_{\eta} \in \mathbb{R}$ .
3.4 Objective Function
We train the sentence extractor and the summary generator in a unified architecture in an end-to-end manner. We use the cross entropy as both the extractor loss and the generator loss.
$$
\begin{array}{l} L _ {e x t} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \left(y _ {s} ^ {(n)} \log \tilde {y} _ {s} ^ {(n)} + \right. \\ \left(1 - y _ {s} ^ {(n)}\right) \log \left(1 - \tilde {y} _ {s} ^ {(n)}\right)) \tag {14} \\ \end{array}
$$
$$
L _ {a b s} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \log p (y _ {w} ^ {(n)})
$$
where  $y_{s}$  is the ground-truth extracted label,  $y_{w}$  is the ground-truth summary and  $N$  is the number of samples in the corpus.
The final loss is as below
$$
L _ {m i x} = L _ {a b s} + \lambda L _ {e x t} \tag {15}
$$
We experiment with the latest released Multi-News dataset (Fabbri et al., 2019), which is the first large scale multi-document news summarization dataset. It contains about 44972 pairs for training, 5622 pairs for development, and 5622 for the test. Each summary of the average of 264 words is paired with a documents cluster of average 2103 words discussing a topic. The number of source documents per summary presents as shown in Table 1. While the dataset contains abstractive gold summaries, it is not readily suited to training extractive models. So we follow the work of Zhou et al. (2018) on extractive summary labeling, constructing gold-label sequences by greedily optimizing ROUGE-2 F1 on the gold-standard summary.
6248
# of source	Frequency	# of source	Frequency
2	23,894	7	382
3	12,707	8	209
4	5,022	9	89
5	1,873	10	33
6	763
We set our model parameters based on preliminary experiments on the development set. We prune the vocabulary to 50k and use the word in the source documents with maximum weight in copy attention to replace the unknown word of the generated summary. We set the dimension of word embeddings and hidden units  $d_{model}$  to 512, feed-forward units to 2048. We set 8 heads for multi-head self-attention, masked multi-head sparse self-attention, and multi-head sparse cross-attention. We set the number of multi-granularity encoder layer  $L_{1}$  to 5 and summary decoder layer  $L_{2}$  to 6. We set dropout (Srivastava et al., 2014) rate to 0.1 and use Adam optimizer with an initial learning rate  $\alpha = 0.0001$ , momentum  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and weight decay  $\epsilon = 10^{-5}$ . When the valid loss on the development set increases for two consecutive epochs, the learning rate is halved. We use a mini-batch size of 10, and set the hyper-parameter  $k = 5$  and  $\lambda = 2$ . Given the salience score predicted by the sentence extractor, we apply a simple greedy procedure to select sentences. We select one sentence based on the descending order of the salience scores and append to the extracted summary until the summary reaches 300 words. We disallow repeating the same trigram (Paulus et al., 2018; Edunov et al., 2019) and use beam search with a beam size of 5 for summary generator.
We use ROUGE (Lin, 2004) to evaluate the produced summary in our experiments. Following previous work, we report ROUGE  $\mathrm{F1}^1$  on MultiNews dataset. We compare our model with several typical baselines and several baselines proposed in the latest years.
Lead-3 is an extractive baseline which concatenates the first-3 sentences of each source document as a summary. LexRank (Erkan and Radev, 2004)