SlowGuess/ACL_2020_OCR huggingface dataset:
Yuning Mao $^{1}$ , Liyuan Liu $^{1}$ , Qi Zhu $^{1}$ , Xiang Ren $^{2}$ , Jiawei Han $^{1}$
$^{1}$ Department of Computer Science, University of Illinois at Urbana-Champaign, IL, USA
$^{2}$ Department of Computer Science, University of Southern California, CA, USA
$^{1}\{\mathrm{yuningm2}, \mathrm{II2}, \mathrm{qiz3}, \mathrm{hanj}\} @$ illinois.edu  $^{2}$ xiangren@usc.edu
Commonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage in extracted summaries. Specifically, we treat each sentence in the reference summary as a facet, identify the sentences in the document that express the semantics of each facet as support sentences of the facet, and automatically evaluate extractive summarization methods by comparing the indices of extracted sentences and support sentences of all the facets in the reference summary. To facilitate this new evaluation setup, we construct an extractive version of the CNN/Daily Mail dataset and perform a thorough quantitative investigation, through which we demonstrate that facet-aware evaluation manifests better correlation with human judgment than ROUGE, enables fine-grained evaluation as well as comparative analysis, and reveals valuable insights of state-of-the-art summarization methods.
Text summarization has enjoyed increasing popularity due to its wide applications, whereas the evaluation of text summarization remains challenging and controversial. The most commonly used evaluation metric of summarization is lexical overlap, i.e., ROUGE (Lin, 2004), which regards the system and reference summaries as sequences of tokens and measures their n-gram overlap.
However, recent studies (Paulus et al., 2017; Schluter, 2017; Kryscinski et al., 2019) reveal the limitations of ROUGE and find that in many cases, it fails to reach consensus with human judgment. Since lexical overlap only captures information
Reference: Three people in Kansas have died from a listeria outbreak.
Lexical Overlap: But they did not appear identical to listeria sam
ples taken from patients infected in the Kansas outbreak . ( ROUGE J )
pies taken from patients infected in the Kansas outbreak. (ROI F1=37.0, multiple token matches but totally different semantics)
Manual Extract: Five people were infected and three died in the past
year in Kansas from listeria that might be linked to blue bell creameries
year in Kansas from history that might be linked to blue shellcreameriesproducts , according to the CDC ( POUCF . FL . 360 . commodity covered
products, according to the CDC (ROUGE-2013: 38.7, see but lower ROUGE due to the presence of other details)
Reference: Chelsea boss Jose Mourinho and United manager Louis van Gaal are pals.
Lexical Overlap: Gary Neville believes Louis van Gaal's greatest achievement as a football manager is the making of Jose Mourinho.
Manual Extract: The duo have been friends since they first worked
together at Barcelona in 1997 where they enjoyed a successful relationship
together at Barcelona in 1997 where they enjoyed a successful relation with the Cerran News (POWERS PUBL/TEL 03) (in German, English and Italian).
Table 1: Lexical overlap — finding the document sentence with the highest ROUGE against one reference sentence — could be misleading. Examples are from the CNN/Daily Mail dataset (Nallapati et al., 2016).
coverage at the surface (token) level, ROUGE favors system summaries that share more tokens with the reference summaries. Nevertheless, such summaries may not always convey the desired semantics. For example, in Table 1, the document sentence with the highest ROUGE score has more lexical overlap but expresses rather different semantic meaning. In contrast, the sentence manually extracted from the document by our annotators, which conveys similar semantics, is over-penalized as it involves other details or uses alternative words.
In this paper, we argue that the information coverage in summarization can be better evaluated by facet overlap, i.e., whether the system summary covers the facets in the reference summary. Specifically, we treat each reference sentence as a facet, identify document sentences that express the semantics of each facet as support sentences of the facet, and measure information coverage by Facet-Aware Recall (FAR), i.e., how many facets are covered. We focus on extractive summarization for the following two reasons. Theoretically, since extractive methods cannot paraphrase or compress the document sentences as abstractive methods, it is somewhat unfair to penalize them for extracting long sentences that cover the facets. Pragmatically,
<sup>1</sup>Data can be found at https://github.com/morningmoni/FAR.
4941
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4941-4957
we can evaluate extractive methods automatically by comparing the indices of extracted sentences and support sentences. We denote the mappings from each facet (sentence) in the reference summary to its support sentences in the document as Facet-Aware Mappings (FAMs). FAMs can be used as labels indicating which sentences should be extracted but they are grouped with respect to each facet, while conventional extractive labels correspond to the entire reference summary rather than individual facets (detailed explanations in Sec. 2.1). Compared to treating one summary as a sequence of n-grams, facet-aware evaluation considers information coverage at a semantically richer granularity, and thus can contribute to a more accurate assessment on the summary quality.
To verify the effectiveness of facet-aware evaluation, we construct an extractive version of the CNN/Daily Mail dataset (Nallapati et al., 2016) by annotating its FAMs (Sec. 2). We revisit state-of-the-art extractive methods using this new extractive dataset (Sec. 3.2), the results of which show that FAR correlates better with human evaluation than ROUGE. We also demonstrate that FAMs are beneficial for fine-grained evaluation of both abstractive and extractive methods (Sec. 3.3). We then illustrate how facet-aware evaluation can be useful for comparing different extractive methods in terms of their capability of extracting salient and non-redundant sentences (Sec. 3.4). Finally, we explore the feasibility of automatic FAM creation by evaluating sentence regression approaches against the ground-truth annotations (i.e., FAMs), and generalize facet-aware evaluation to the entire CNN/Daily Mail dataset without any human annotation (Sec. 4). We believe that the summarization community will benefit from the proposed setup for better assessment of information coverage and gain deeper understandings of the current benchmark dataset and state-of-the-art methods through our analysis.
Contributions. (1) We propose a facet-aware evaluation setup that better assesses information coverage for extractive summarization. (2) We build the first dataset designed specifically for extractive summarization by creating facet-aware mappings from reference summaries to documents. (3) We revisit state-of-the-art summarization methods in the proposed setup and discover valuable insights. (4) To our knowledge, our work is also the first thorough quantitative analysis regarding the char
Figure 1: An illustration of facet-aware evaluation. Two of three support groups of facet  $1(r_{1})$  are covered. Facet  $2(r_{2})$  cannot be covered as document sentence  $4(d_{4})$  is missing in the extracted summary. The illustration corresponds to the example in Sec. 3.1.
acteristics of the CNN/Daily Mail dataset.
2 Dataset Creation
In this section, we describe the process of creating an extractive summarization dataset to facilitate facet-aware evaluation, which involves annotating FAMs between the documents and abstractive reference summaries. We first formalize the FAMs and then describe the FAM annotation on the CNN/Daily Mail dataset (Nallapati et al., 2016).
2.1 FAMs: Facet-Aware Mappings
We denote one document-summary pair as  $\{\mathbf{D},\mathbf{R}\}$ , where  $\mathbf{D} = [d_1,d_2,\dots,d_D]$ ,  $\mathbf{R} = [r_1,r_2,\dots,r_R]$  and  $D,R$  denote the numbers of document sentences and reference sentences, respectively. We conceptualize facet as one unique semantic aspect presented in the summary. In practice, we hypothesize that each reference sentence  $r_i$  corresponds to one facet. We define support sentences as the sentences in the document that express the semantics of one facet  $r_i$ . We define support group  $\mathcal{S}$  of facet  $r_i$  as a set of support sentences that can fully cover the information of  $r_i$ . For each facet  $r_i$  in the reference summary, we try to find all its support sentences in the document and put them into support groups. Since we focus on single-document
It is possible to define facet at sub-sentence or multisentence level as in Pyramid (Nenkova and Passonneau, 2004). However, such definitions inevitably incur more annotation effort and lower inter-annotator agreement, while the current definition balances cost and effectiveness.
Category	#Samples	#Facets	Example (full documents, reference summaries, and the FAMs can be found in Appendix C)
Noise (N)	41 (27.3%)	137 (27.1%)	• Reference: “Furious 7” opens Friday. (unimportant detail)
• Reference: Click here for all the latest Floyd Mayweather vs Manny Pacquiao news. (not found in the document)
• Reference: Vin Diesel: “This movie is more than a movie”. (random quotation)
• Reference: “I had a small moment of awe,” she said. (random quotation)
Low Abstraction (L)	89 (59.3%)	310 (61.2%)M=1:275 (88.7%)M=2:35 (11.3%)	• Reference: Willis never trademarked her most-famous work, calling it “my gift to the city”.
• Support: Willis never trademarked her most-famous work, calling it “my gift to the city.” (identical)
• Reference: Thomas K. Jenkins, 49, was arrested last month by deputies with the Prince George’s County sheriff’s office, authorities said.
• Support: Authorities said in a news release Thursday that 49-year-old Thomas K. Jenkins of capitol heights, Maryland, was arrested last month by deputies with the Prince George’s County sheriff’s office. (compression)
High Abstraction (H)	20 (13.3%)	59 (11.7%)	• Reference: College-bound basketball star asks girl with down syndrome to high school prom. Pictures of the two during the “prom-posal” have gone viral. (highly abstractive)
• Reference: While Republican Gov. Asa Hutchinson was weighing an Arkansas religious freedom bill, Walmart voiced its opposition. Walmart and other high-profile businesses are showing their support for gay and lesbian rights. (unable to find support sentences)
Table 2: Category breakdown of Facet-Aware Mappings (FAMs). Nearly  $60\%$  samples are of low abstraction while more than a quarter of samples contain noisy facets.  $\overline{M}$  denotes the average number of support sentences.
summmarization in this work, most facets only have one support group. But some may contain multiple and extracting any of them would suffice (see example in Appendix C Table 10). Allowing multiple support groups also makes FAMs easily extendable to multi-document summarization where redundant sentences prevail.
Formally, for each  $r_i$ , we annotate a Facet-Aware Mapping (FAM)  $r_i \rightarrow \{S_1^i, S_2^i, \dots, S_N^i\}$ , where  $N$  is the number of support groups. Each  $S_j^i = \{d_{I_1}, d_{I_2}, \dots, d_{I_{M_j}}\}$  is a support group, where  $I_1, I_2, \dots, I_{M_j}$  are the indices of support sentences and  $M_j$  is the number of support sentences in  $S_j^i$ . One illustrative example is presented in Fig. 1. The support sentences are likely to be verbose, but we consider whether the support sentences express the semantics of the facet regardless of their length. The reason is that we believe extractive summarization should focus on information coverage since it cannot alter the original sentences and once salient sentences are extracted, one can then compress them in an abstractive manner (Chen and Bansal, 2018; Hsu et al., 2018).
Relation w. Extractive Labels. Extractive methods (Nallapati et al., 2017; Chen and Bansal, 2018; Narayan et al., 2018c) typically require binary labels of every document sentence indicating whether it should be extracted during model training. Such labels are called extractive labels and usually created heuristically based on reference summaries
since existing datasets do not provide extractive labels but only abstractive references. Our assumption that each reference sentence corresponds to one facet is similar to that during the creation of extractive labels. The major differences are that (1) We allow an arbitrary number of support sentences while extractive labels usually limit to one support sentence for each reference sentence, i.e., we do not specify  $M_{j}$ . For example, we would put two support sentences to one support group if they are complementary and only combining them can cover the facet. (2) We try to find multiple support groups ( $N > 1$ ), as there could be more than one set of support sentences that cover the same facet. In contrast, there is no notion of support group in extractive labels as they inherently form one such group ( $N = 1$ ). Also, we allow  $N = 0$  if such a mapping cannot be found even by humans. (3) The FAMs are more accurate as they are created by human annotators while extractive methods use sentence regression approaches (which we evaluate in Sec. 4.1) to obtain extractive labels approximately.
Comparison w. SCUs. Some may mistake FAMs for Summarization Content Units (SCUs) in Pyramid (Nenkova and Passonneau, 2004), but they are different in that (1) FAMs utilize both the documents and reference summaries while SCUs ignore the documents; (2) FAMs are at the sentence level and can thus be used to automatically evaluate extractive methods once created — simply by matching sentence indices we can know how many facets are covered, while SCUs have to be manually annotated for each system (refer to Appendix B Fig. 4).
<sup>3</sup>We ignore coreference (e.g., "he" vs. "the writer") and short fragments when considering the semantics of one facet, as we found that the wording of the reference summaries regarding such choices is also capricious.
2.2 Creation of Extractive CNN/Daily Mail
To verify the effectiveness of facet-aware evaluation, we annotate the FAMs of 150 document-summary pairs from the test set of CNN/Daily Mail. Specifically, we take the first 50 samples in the test set, the 20 samples used in the human evaluation of Narayan et al. (2018c), and randomly draw another 80 samples. The annotators are graduate students who are required to read through the document and mark support groups for each facet. The most similar document sentences to each facet found by ROUGE and cosine similarity of average word embeddings are provided as the baselines for annotation. 310 non-empty FAMs are created by three annotators with high agreement (pairwise Jaccard index 0.714) and further verified to reach consensus. On average, 5.44 (6.04 non-unique) document sentences are included as the support sentences in each document-summary pair.
To summarize, we found that the facets can be divided into three categories based on their quality and degree of abstraction as follows.
Noise: The facet is noisy and irrelevant to the main content, either because the document itself is too hard to summarize (e.g., a report full of quotations) or the human editor was too subjective when writing the summary (See et al., 2017). Another possible reason is that the so-called "summaries" in CNN/Daily Mail are in fact "story highlights", which seems reasonable to include certain details. We found that 41/150  $(27.3\%)$  samples have noisy facet(s), indicating that the reference summaries of CNN/Daily Mail are rather noisy. We show in Sec. 3.2 that existing summarization methods perform poorly on this category, which justifies our judgment of "noisy facets" from another aspect. Also note that there would not be a "noise" category in a "clean" dataset. However, given the creation process of popular summarization datasets (Nallapati et al., 2016; Narayan et al., 2018b), it is unlikely that all of their samples are of high quality.
Low Abstraction: The facet can be mapped to its support sentences. We denote the (rounded) average number of support sentences for each facet as  $\overline{M} (= \frac{1}{N}\sum_{j=1}^{N} M_j, N$  represents the number of support groups). As shown in Table 2, all the facets with non-empty FAMs in CNN/Daily Mail are paraphrases or compression of one to two sentences in
the document without much abstraction.
High Abstraction: The facet cannot be mapped to its support sentences ( $N = 0$ ) by humans, which indicates that the writing of the facet requires deep understanding of the document rather than simply reorganizing several sentences. The proportion of this category ( $13.3\%$ ) also indicates how often extractive methods would not work (well) on CNN/Daily Mail.
We found it easier than previously believed to create the FAMs on CNN/Daily Mail, as it is uncommon (average number of support groups  $\overline{N} = 1.6$ ) to detect multiple sentences with similar semantics. In addition, most support groups only have one or two support sentences with large lexical overlap, which coincides with the fact that extractive methods work quite well on CNN/Daily Mail and abstractive methods are often hybrid and learn to copy words directly from the documents. That said, we try to automate the FAM creation and scale facet-aware evaluation to the whole test set of CNN/Daily Mail using machine-created FAMs (Sec. 4).
3 Facet-Aware Evaluation
In this section, we introduce the facet-aware evaluation setup (Sec. 3.1) and demonstrate its effectiveness by revisiting state-of-the-art summarization methods under this new setup (Sec. 3.2). We then illustrate the additional benefits of facet-aware evaluation, including fine-grained evaluation (Sec. 3.3) and comparative analysis (Sec. 3.4).
3.1 Proposed Metrics
As current extractive methods are facet-agnostic, i.e., their output is not nested (organized by facets) but a flat set of extracted sentences, we consider one facet as being "covered" if any of its support groups can be found in the whole extracted summary. Formally, we define the Facet-Aware Recall (FAR) as follows.
$$
\mathrm {F A R} = \frac {\sum_ {i = 1} ^ {R} \mathbf {A n y} (\mathbf {I} (\mathcal {S} _ {1} ^ {i} , \mathcal {E}) , . . . , \mathbf {I} (\mathcal {S} _ {N} ^ {i} , \mathcal {E}))}{R},
$$
where  $\mathbf{Any}(\mathcal{X})$  returns 1 if any  $x\in \mathcal{X}$  is 1 and 0 otherwise,  $\mathbf{I}(\mathcal{X},\mathcal{Y})$  returns 1 if set  $\mathcal{X}\subset \mathcal{Y}$  and 0 otherwise,  $\mathcal{E}$  denotes the set of extracted sentences, and  $R$  is the number of facets. Intuitively, FAR does not over-penalize extractive methods for extracting long sentences as long as the extracted sentences cover the semantics of the facets. FAR
One alternative way is to store multiple FAMs for each sample (like multiple reference summaries) and average their results as in ROUGE.
also treats each facet equally, whereas ROUGE weighs higher the facets with more tokens since they are more likely to incur lexical overlap.
To further measure model capability of retrieving salient (support) sentences without considering redundancy as FAR does, we merge all the support sentences of one document-summary pair to one single support set and define the Support-Aware Recall (SAR) as follows. SAR is used in Sec. 3.4 for the comparative analysis of extractive methods.
$$
\mathsf {S A R} = \frac {\mid \bigcup_ {i = 1} ^ {R} \bigcup_ {j = 1} ^ {N} \mathcal {S} _ {j} ^ {i} \cap \mathcal {E} \mid}{\mid \bigcup_ {i = 1} ^ {R} \bigcup_ {j = 1} ^ {N} \mathcal {S} _ {j} ^ {i} \mid}.
$$
Example (Fig. 1). Assume that  $R = 2$ ,  $r_1 \to \{\{d_1\}, \{d_3\}, \{d_4\}\}$ ,  $r_2 \to \{\{d_2, d_4\}\}$ , and  $\mathcal{E} = \{d_1, d_2, d_3\}$ . Then  $\mathrm{FAR} = \frac{1}{2}$  as  $\mathcal{E}$  covers  $\{d_1\}$  (or  $\{d_3\}$ ) for  $r_1$  but cannot cover  $\{d_2, d_4\}$  for  $r_2$ .  $\mathrm{SAR} = \frac{|\{d_1, d_2, d_3, d_4\} \cap \{d_1, d_2, d_3\}|}{|\{d_1, d_2, d_3, d_4\}|} = \frac{3}{4}$ . Note that  $d_1$  and  $d_3$  are salient (support sentences) and both considered positive in SAR, while they only contribute to the coverage of one facet in FAR.
3.2 Automatic Evaluation with FAR
By utilizing the low abstraction category on the extractive CNN/Daily Mail dataset, we revisit extractive methods to evaluate how they perform on information coverage. Specifically, we compare Lead-3 (that extracts the first three document sentences), FastRL(E) (E for extractive only) (Chen and Bansal, 2018), BanditSum (Dong et al., 2018), NeuSum (Zhou et al., 2018), Refresh (Narayan et al., 2018c), and UnifiedSum(E) (Hsu et al., 2018) using both ROUGE and FAR. For a fair comparison, each method extracts three sentences  $\left( |\mathcal{E}| = 3 \right)$ .
Results on Neural Extractive Methods. As shown in Table 3, there is almost no discrimination among the last four methods under ROUGE-1 F1, and the rankings under ROUGE-1/2/L often contradict with each other. The observations on ROUGE Precision/Recall are similar. We provide them as well as more comparative analysis under facet-aware evaluation in Sec. 3.4. For facet coverage, the upper bound of FAR by extracting 3 sentences (Oracle, given the ground-truth FAMs) is 84.8, much higher than all the compared methods. The best performing extractive method under FAR
is UnifiedSum(E), which indicates that it covers the most facets semantically.
Method	ROUGE-1	ROUGE-2	ROUGE-L	FAR
Lead-3	41.9	19.6	34.8	50.6
FastRL(E)	41.6	20.3	35.5	50.8
BanditSum	42.7	20.2	35.8	44.7
NeuSum	42.7	22.1	36.4	51.2
Refresh	42.8	20.3	39.3	51.3
UnifiedSum(E)	42.6	20.7	35.5	54.8
Oracle	53.8	32.1	48.1	84.8
FAR's Correlation w. Human Evaluation. Although FAR is supposed to be favored as the FAMs are manually labeled and indicate accurately whether one sentence should be extracted (assuming the annotations are in high quality), to further verify that FAR correlates with human preference, we ask the annotators to rank the outputs of UnifiedSum(E), NeuSum, and Lead-3 and measure ranking correlation. As listed in Table 4, we observe that the method with the most 1st ranks in the human evaluation coincides with FAR. We also find that FAR has higher Spearman's coefficient  $\rho$  than ROUGE (0.457 vs. 0.44).<sup>6</sup>
Table 3: Performance comparison of extractive methods under ROUGE F1 and Facet-Aware Recall (FAR).
Method	1st	2nd	3rd
Lead-3	26.8%	46.3%	26.8%
NeuSum	29.3%	39.0%	31.7%
UnifiedSum(E)	37.8%	52.4%	9.8%
Table 4: Proportions of system ranking in human evaluation. FAR shows better human correlation than ROUGE and prefers UnifiedSum(E).
3.3 Fine-grained Evaluation
One benefit of facet-aware evaluation is that we can employ the category breakdown of FAMs for fine-grained evaluation, namely, how one method performs on noisy / low abstraction / high abstraction samples, respectively. Any metric of interest can be used for this fine-grained analysis. Here we consider ROUGE and additionally evaluate several abstractive methods: PG (PointerGenerator) (See et al., 2017), FastRL(E+A) (extractive+abstractive) (Chen and Bansal, 2018), and UnifiedSum(E+A) (Hsu et al., 2018).
As shown in Table 5, extractive methods perform poorly on high abstraction samples, which
Extracting all the sentences results in a perfect FAR, which is expected as FAR measures recall. One can also normalize FAR by the number of extracted sentences.
<sup>6</sup>We expect that one can observe larger gains on datasets with less lexical overlap than CNN/Daily Mail.