SlowGuess/ACL_2020_OCR huggingface dataset:
Ming Zhong*, Pengfei Liu*, Yiran Chen, Danqing Wang, Xipeng Qiu†, Xuanjing Huang
Shanghai Key Laboratory of Intelligent Information Processing, Fudan University
825 Zhangheng Road, Shanghai, China
{mzhong18,pfliu14,yrchen19,dqwang18,xpqiu,xjhuang}@fudan.edu.cn
This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. Instead of following the commonly used framework of extracting sentences individually and modeling the relationship between sentences, we formulate the extractive summarization task as a semantic text matching problem, in which a source document and candidate summaries will be (extracted from the original text) matched in a semantic space. Notably, this paradigm shift to semantic matching framework is well-grounded in our comprehensive analysis of the inherent gap between sentence-level and summary-level extractors based on the property of the dataset.
Besides, even instantiating the framework with a simple form of a matching model, we have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1). Experiments on the other five datasets also show the effectiveness of the matching framework. We believe the power of this matching-based summarization framework has not been fully exploited. To encourage more instantiations in the future, we have released our codes, processed dataset, as well as generated summaries in https://github.com/maszhongming/MatchSum.
The task of automatic text summarization aims to compress a textual document to a shorter highlight while keeping salient information on the original text. In this paper, we focus on extractive summarization since it usually generates semantically and grammatically correct sentences (Dong et al., 2018; Nallapati et al., 2017) and computes faster.
Currently, most of the neural extractive summarization systems score and extract sentences (or smaller semantic unit (Xu et al., 2019)) one by
Figure 1: MATCHSUM framework. We match the contextual representations of the document with gold summary and candidate summaries (extracted from the document). Intuitively, better candidate summaries should be semantically closer to the document, while the gold summary should be the closest.
one from the original text, model the relationship between the sentences, and then select several sentences to form a summary. Cheng and Lapata (2016); Nallapati et al. (2017) formulate the extractive summarization task as a sequence labeling problem and solve it with an encoder-decoder framework. These models make independent binary decisions for each sentence, resulting in high redundancy. A natural way to address the above problem is to introduce an auto-regressive decoder (Chen and Bansal, 2018; Jadhav and Rajan, 2018; Zhou et al., 2018), allowing the scoring operations of different sentences to influence on each other. Trigram Blocking (Paulus et al., 2017; Liu and Lapata, 2019), as a more popular method recently, has the same motivation. At the stage of selecting sentences to form a summary, it will skip the sentence that has trigram overlapping with the previously selected sentences. Surprisingly, this simple method of removing duplication brings a remarkable performance improvement on CNN/DailyMail.
The above systems of modeling the relationship between sentences are essentially sentence-level extractors, rather than considering the semantics
*These two authors contributed equally.
6197
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208
of the entire summary. This makes them more inclined to select highly generalized sentences while ignoring the coupling of multiple sentences. Narayan et al. (2018b); Bae et al. (2019) utilize reinforcement learning (RL) to achieve summary-level scoring, but still limited to the architecture of sentence-level summarizers.
To better understand the advantages and limitations of sentence-level and summary-level approaches, we conduct an analysis on six benchmark datasets (in Section 3) to explore the characteristics of these two methods. We find that there is indeed an inherent gap between the two approaches across these datasets, which motivates us to propose the following summary-level method.
In this paper, we propose a novel summary-level framework (MATCHSUM, Figure 1) and conceptualize extractive summarization as a semantic text matching problem. The principle idea is that a good summary should be more semantically similar as a whole to the source document than the unqualified summaries. Semantic text matching is an important research problem to estimate semantic similarity between a source and a target text fragment, which has been applied in many fields, such as information retrieval (Mitra et al., 2017), question answering (Yih et al., 2013; Severyn and Moschitti, 2015), natural language inference (Wang and Jiang, 2016; Wang et al., 2017) and so on. One of the most conventional approaches to semantic text matching is to learn a vector representation for each text fragment, and then apply typical similarity metrics to compute the matching scores.
Specific to extractive summarization, we propose a Siamese-BERT architecture to compute the similarity between the source document and the candidate summary. Siamese BERT leverages the pre-trained BERT (Devlin et al., 2019) in a Siamese network structure (Bromley et al., 1994; Hoffer and Ailon, 2015; Reimers and Gurevych, 2019) to derive semantically meaningful text embeddings that can be compared using cosine-similarity. A good summary has the highest similarity among a set of candidate summaries.
We evaluate the proposed matching framework and perform significance testing on a range of benchmark datasets. Our model outperforms strong baselines significantly in all cases and improve the state-of-the-art extractive result on CNN/DailyMail. Besides, we design experiments to observe the gains brought by our framework.
1) Instead of scoring and extracting sentences one by one to form a summary, we formulate extractive summarization as a semantic text matching problem and propose a novel summary-level framework. Our approach bypasses the difficulty of summary-level optimization by contrastive learning, that is, a good summary should be more semantically similar to the source document than the unqualified summaries.
2) We conduct an analysis to investigate whether extractive models must do summary-level extraction based on the property of dataset, and attempt to quantify the inherent gap between sentence-level and summary-level methods.
3) Our proposed framework has achieved superior performance compared with strong baselines on six benchmark datasets. Notably, we obtain a state-of-the-art extractive result on CNN/DailyMail (44.41 in ROUGE-1) by only using the base version of BERT. Moreover, we seek to observe where the performance gain of our model comes from.
2.1 Extractive Summarization
Recent research work on extractive summarization spans a large range of approaches. These work usually instantiate their encoder-decoder framework by choosing RNN (Zhou et al., 2018), Transformer (Zhong et al., 2019b; Wang et al., 2019) or GNN (Wang et al., 2020) as encoder, non-auto-regressive (Narayan et al., 2018b; Arumae and Liu, 2018) or auto-regressive decoders (Jadhav and Rajan, 2018; Liu and Lapata, 2019). Despite the effectiveness, these models are essentially sentence-level extractors with individual scoring process favor the highest scoring sentence, which probably is not the optimal one to form summary<sup>1</sup>.
The application of RL provides a means of summary-level scoring and brings improvement (Narayan et al., 2018b; Bae et al., 2019). However, these efforts are still limited to auto-regressive or non-auto-regressive architectures. Besides, in the non-neural approaches, the Integer Linear Programming (ILP) method can also be used for summary-level scoring (Wan et al., 2015).
In addition, there is some work to solve extractive summarization from a semantic perspective before this paper, such as concept coverage (Gillick
<sup>1</sup>We will quantify this phenomenon in Section 3.
and Favre, 2009), reconstruction (Miao and Blunsom, 2016) and maximize semantic volume (Yogatama et al., 2015).
2.2 Two-stage Summarization
Recent studies (Alyguliyev, 2009; Galanis and Androutsopoulos, 2010; Zhang et al., 2019a) have attempted to build two-stage document summarization systems. Specific to extractive summarization, the first stage is usually to extract some fragments of the original text, and the second stage is to select or modify on the basis of these fragments.
Chen and Bansal (2018) and Bae et al. (2019) follow a hybrid extract-then-rewrite architecture, with policy-based RL to bridge the two networks together. Lebanoff et al. (2019); Xu and Durrett (2019); Mendes et al. (2019) focus on the extract-then-compress learning paradigm, which will first train an extractor for content selection. Our model can be viewed as an extract-then-match framework, which also employs a sentence extractor to prune unnecessary information.
3 Sentence-Level or Summary-Level? A Dataset-dependent Analysis
Although previous work has pointed out the weakness of sentence-level extractors, there is no systematic analysis towards the following questions: 1) For extractive summarization, is the summary-level extractor better than the sentence-level extractor? 2) Given a dataset, which extractor should we choose based on the characteristics of the data, and what is the inherent gap between these two extractors?
In this section, we investigate the gap between sentence-level and summary-level methods on six benchmark datasets, which can instruct us to search for an effective learning framework. It is worth noting that the sentence-level extractor we use here doesn't include a redundancy removal process so that we can estimate the effect of the summary-level extractor on redundancy elimination. Notably, the analysis method to estimate the theoretical effectiveness presented in this section is generalized and can be applicable to any summary-level approach.
We refer to  $D = \{s_1, \dots, s_n\}$  as a single document consisting of  $n$  sentences, and  $C = \{s_1, \dots, s_k, |s_i \in D\}$  as a candidate summary in
cluding  $k$ $(k\leq n)$  sentences extracted from a document. Given a document  $D$  with its gold summary  $C^*$ , we measure a candidate summary  $C$  by calculating the ROUGE (Lin and Hovy, 2003) value between  $C$  and  $C^*$  in two levels:
1) Sentence-Level Score:
$$
\mathrm {g} ^ {\text {s e n}} (C) = \frac {1}{| C |} \sum_ {s \in C} \mathrm {R} (\mathrm {s}, \mathrm {C} ^ {*}), \tag {1}
$$
where  $s$  is the sentence in  $C$  and  $|C|$  represents the number of sentences.  $\mathrm{R}(\cdot)$  denotes the average ROUGE score<sup>2</sup>. Thus,  $\mathrm{g}^{sen}(C)$  indicates the average overlaps between each sentence in  $C$  and the gold summary  $C^*$ .
2) Summary-Level Score:
$$
\mathrm {g} ^ {\text {s u m}} (C) = \mathrm {R} (C, C ^ {*}), \tag {2}
$$
where  $\mathrm{g}^{sum}(C)$  considers sentences in  $C$  as a whole and then calculates the ROUGE score with the gold summary  $C^*$ .
Pearl-Summary We define the pearl-summary to be the summary that has a lower sentence-level score but a higher summary-level score.
Definition 1 A candidate summary  $C$  is defined as a pearl-summary if there exists another candidate summary  $C'$  that satisfies the inequality:  $\mathrm{g}^{sen}(C') > \mathrm{g}^{sen}(C)$  while  $\mathrm{g}^{sum}(C') < \mathrm{g}^{sum}(C)$ .
Clearly, if a candidate summary is a pearl-summary, it is challenging for sentence-level summarizers to extract it.
Best-Summary The best-summary refers to a summary has highest summary-level score among all the candidate summaries.
Definition 2 A summary  $\hat{C}$  is defined as the best-summary when it satisfies:  $\hat{C} = \operatorname*{argmax}_{C\in \mathcal{C}}\mathrm{g}^{sum}(C)$
where  $\mathcal{C}$  denotes all the candidate summaries of the document.
3.2 Ranking of Best-Summary
For each document, we sort all candidate summaries in descending order based on the sentence-level score, and then define  $z$  as the rank index of the best-summary  $\hat{C}$ .
Here we use mean  $\mathrm{F}_1$  of ROUGE-1, ROUGE-2 and ROUGE-L.
3We use an approximate method here: take #Ext (see Table 1) of ten highest-scoring sentences to form candidate summaries.
6199
Datasets	Source	Type	Train	# Pairs Valid	Test	# Tokens	# Ext
Doc.	Sum.
Reddit	Social Media	SDS	41,675	645	645	482.2	28.0	2
XSum	News	SDS	203,028	11,273	11,332	430.2	23.3	2
CNN/DM	News	SDS	287,084	13,367	11,489	766.1	58.2	3
WikiHow	Knowledge Base	SDS	168,126	6,000	6,000	580.8	62.6	4
PubMed	Scientific Paper	SDS	83,233	4,946	5,025	444.0	209.5	6
Multi-News	News	MDS	44,972	5,622	5,622	487.3	262.0	9
Table 1: Datasets overview. SDS represents single-document summarization and MDS represents multi-document summarization. The data in Doc. and Sum. indicates the average length of document and summary in the test set respectively. # Ext denotes the number of sentences should extract in different datasets.
(a) Reddit
(b) XSum
(c) CNN/DM
(d) WikiHow
(e) PubMed
(f) Multi-News
Figure 2: Distribution of  $z(\%)$  on six datasets. Because the number of candidate summaries for each document is different (short text may have relatively few candidates), we use  $z /$  number of candidate summaries as the X-axis. The Y-axis represents the proportion of the best-summaries with this rank in the test set.
Intuitively, 1) if  $z = 1$  ( $\hat{C}$  comes first), it means that the best-summary is composed of sentences with the highest score; 2) If  $z > 1$ , then the best-summary is a pearl-summary. And as  $z$  increases ( $\hat{C}$  gets lower rankings), we could find more candidate summaries whose sentence-level score is higher than best-summary, which leads to the learning difficulty for sentence-level extractors.
Since the appearance of the pearl-summary will bring challenges to sentence-level extractors, we attempt to investigate the proportion of pearl-summary in different datasets on six benchmark datasets. A detailed description of these datasets is displayed in Table 1.
As demonstrated in Figure 2, we can observe that for all datasets, most of the best-summaries are not made up of the highest-scoring sentences. Specifically, for CNN/DM, only  $18.9\%$  of best-summaries are not pearl-summary, indicating sentence-level extractors will easily fall into a local optimization, missing better candidate summaries.
Different from CNN/DM, PubMed is most suitable for sentence-level summarizers, because most of best-summary sets are not pearl-summary. Additionally, it is challenging to achieve good performance on WikiHow and Multi-News without a summary-level learning process, as these two datasets are most evenly distributed, that is, the appearance of pearl-summary makes the selection of the best-summary more complicated.
In conclusion, the proportion of the pearlsummaries in all the best-summaries is a property to characterize a dataset, which will affect our choices of summarization extractors.
3.3 Inherent Gap between Sentence-Level and Summary-Level Extractors
Above analysis has explicated that the summary-level method is better than the sentence-level method because it can pick out pearl-summaries, but how much improvement can it bring given a specific dataset?
Based on the definition of Eq. (1) and (2), we can characterize the upper bound of the sentence-level and summary-level summarization systems for a document  $D$  as:
Figure 3:  $\Delta (\mathcal{D})$  for different datasets.
$$
\alpha^ {s e n} (D) = \max  _ {C \in \mathcal {C} _ {D}} \mathrm {g} ^ {s e n} (C), \tag {3}
$$
$$
\alpha^ {s u m} (D) = \max  _ {C \in \mathcal {C} _ {D}} \mathrm {g} ^ {s u m} (C), \tag {4}
$$
where  $\mathcal{C}_D$  is the set of candidate summaries extracted from  $D$ .
Then, we quantify the potential gain for a document  $D$  by calculating the difference between  $\alpha^{sen}(D)$  and  $\alpha^{sum}(D)$ :
$$
\Delta (D) = \alpha^ {s u m} (D) - \alpha^ {s e n} (D). \tag {5}
$$
Finally, a dataset-level potential gain can be obtained as:
$$
\Delta (\mathcal {D}) = \frac {1}{| \mathcal {D} |} \sum_ {D \in \mathcal {D}} \Delta (D), \tag {6}
$$
where  $\mathcal{D}$  represents a specific dataset and  $|\mathcal{D}|$  is the number of documents in this dataset.
We can see from Figure 3, the performance gain of the summary-level method varies with the dataset and has an improvement at a maximum 4.7 on CNN/DM. From Figure 3 and Table 1, we can find the performance gain is related to the length of reference summary for different datasets. In the case of short summaries (Reddit and XSum), the perfect identification of pearl-summaries does not lead to much improvement. Similarly, multiple sentences in a long summary (PubMed and Multi-News) already have a large degree of semantic overlap, making the improvement of the summary-level method relatively small. But for a medium-length summary (CNN/DM and WikiHow, about 60 words), the summary-level learning process is rewarding. We will discuss this performance gain with specific models in Section 5.4.
4 Summarization as Matching
The above quantitative analysis suggests that for most of the datasets, sentence-level extractors are
inherently unaware of pearl-summary, so obtaining the best-summary is difficult. To better utilize the above characteristics of the data, we propose a summary-level framework which could score and extract a summary directly.
Specifically, we formulate the extractive summarization task as a semantic text matching problem, in which a source document and candidate summaries will be (extracted from the original text) matched in a semantic space. The following section will detail how we instantiate our proposed matching summarization framework by using a simple siamese-based architecture.
4.1 Siamese-BERT
Inspired by siamese network structure (Bromley et al., 1994), we construct a Siamese-BERT architecture to match the document  $D$  and the candidate summary  $C$ . Our Siamese-BERT consists of two BERTs with tied-weights and a cosine-similarity layer during the inference phase.
Unlike the modified BERT used in (Liu, 2019; Bae et al., 2019), we directly use the original BERT to derive the semantically meaningful embeddings from document  $D$  and candidate summary  $C$  since we need not obtain the sentence-level representation. Thus, we use the vector of the '[CLS]' token from the top BERT layer as the representation of a document or summary. Let  $\mathbf{r}_D$  and  $\mathbf{r}_C$  denote the embeddings of the document  $D$  and candidate summary  $C$ . Their similarity score is measured by  $f(D,C) = \mathrm{cosine}(\mathbf{r}_D,\mathbf{r}_C)$ .
In order to fine-tune Siamese-BERT, we use a margin-based triplet loss to update the weights. Intuitively, the gold summary  $C^*$  should be semantically closest to the source document, which is the first principle our loss should follow:
$$
\mathcal {L} _ {1} = \max  (0, f (D, C) - f (D, C ^ {*}) + \gamma_ {1}), \tag {7}
$$
where  $C$  is the candidate summary in  $D$  and  $\gamma_{1}$  is a margin value. Besides, we also design a pairwise margin loss for all the candidate summaries. We sort all candidate summaries in descending order of ROUGE scores with the gold summary. Naturally, the candidate pair with a larger ranking gap should have a larger margin, which is the second principle to design our loss function:
$$
\begin{array}{l} \mathcal {L} _ {2} = \max  (0, f (D, C _ {j}) - f (D, C _ {i}) \\ \quad + (j - i) * \gamma_ {2}) \quad (i <   j), \end{array} \tag {8}
$$
6201
where  $C_i$  represents the candidate summary ranked  $i$  and  $\gamma_{2}$  is a hyperparameter used to distinguish between good and bad candidate summaries. Finally, our margin-based triplet loss can be written as:
$$
\mathcal {L} = \mathcal {L} _ {1} + \mathcal {L} _ {2}. \tag {9}
$$
The basic idea is to let the gold summary have the highest matching score, and at the same time, a better candidate summary should obtain a higher score compared with the unqualified candidate summary. Figure 1 illustrate this idea.
In the inference phase, we formulate extractive summarization as a task to search for the best summary among all the candidates  $\mathcal{C}$  extracted from the document  $D$ .
$$
\hat {C} = \underset {C \in \mathcal {C}} {\arg \max } f (D, C). \tag {10}
$$
4.2 Candidates Pruning
**Curse of Combination** The matching idea is more intuitive while it suffers from combinatorial explosion problems. For example, how could we determine the size of the candidate summary set or should we score all possible candidates? To alleviate these difficulties, we propose a simple candidate pruning strategy.
Concretely, we introduce a content selection module to pre-select salient sentences. The module learns to assign each sentence a salience score and prunes sentences irrelevant with the current document, resulting in a pruned document  $D' = \{s_1', \dots, s_{ext}' | s_i' \in D\}$ .
Similar to much previous work on two-stage summarization, our content selection module is a parameterized neural network. In this paper, we use BERTSUM (Liu and Lapata, 2019) without tri-gram blocking (we call it BERTEXT) to score each sentence. Then, we use a simple rule to obtain the candidates: generating all combinations of sel sentences subject to the pruned document, and re-organize the order of sentences according to the original position in the document to form candidate summaries. Therefore, we have a total of  $\binom{ext}{sel}$  candidate sets.
In order to verify the effectiveness of our framework and obtain more convincing explanations, we perform experiments on six divergent mainstream datasets as follows.
Reddit	XSum	CNN/DM	Wiki	PubMed	M-News
Ext	5	5	5	5	7	10
Sel	1,2	1,2	2,3	3,4,5	6	9
Size	15	15	20	16	7	9
Table 2: Details about the candidate summary for different datasets.  $Ext$  denotes the number of sentences after we prune the original document,  $Sel$  denotes the number of sentences to form a candidate summary and  $Size$  is the number of final candidate summaries.
CNN/DailyMail (Hermann et al., 2015) is a commonly used news summarization dataset modified by Nallapati et al. (2016). PubMed (Cohen et al., 2018) is collected from scientific papers. We modify this dataset by using the introduction section as the document and the abstract section as the corresponding summary. WikiHow (Koupaee and Wang, 2018) is a diverse dataset extracted from an online knowledge base. XSum (Narayan et al., 2018a) is a one-sentence summary dataset to answer the question "What is the article about?". Multi-News (Fabbri et al., 2019) is a multi-document news summarization dataset, we concatenate the source documents as a single input. Reddit (Kim et al., 2019) is a highly abstractive dataset collected from social media platform. We use the TIFU-long version of Reddit.
We use the base version of BERT to implement our models in all experiments. Adam optimizer (Kingma and Ba, 2014) with warming-up is used and our learning rate schedule follows Vaswani et al. (2017) as:
$$
\mathrm {l r} = 2 \mathrm {e} ^ {- 3} \cdot \min  (\text {s t e p} ^ {- 0. 5}, \text {s t e p} \cdot \mathrm {w m} ^ {- 1. 5}), \tag {11}
$$
where each step is a batch size of 32 and  $wm$  denotes warmup steps of 10,000. We choose  $\gamma_{1} = 0$  and  $\gamma_{2} = 0.01$ . When  $\gamma_{1} < 0.05$  and  $0.005 < \gamma_{2} < 0.05$  they have little effect on performance, otherwise they will cause performance degradation. We use the validation set to save three best checkpoints during training, and record the performance of the best checkpoints on the test set. Importantly, all the experimental results listed in this paper are the average of three runs. To obtain a Siamese-BERT model on CNN/DM, we use 8 TeslaV100-16G GPUs for about 30 hours of training.
For datasets, we remove samples with empty document or summary and truncate the document
6202