Building Extractive CNN/Daily Mail
In this section, we describe the procedure of annotating CNN/Daily Mail. For each facet (sentence) in the reference summary, we find all its support sentences in the document that can cover its meaning. Note that the support sentences are likely to be more verbose, but we only consider if the sentences cover the semantics of the facet regardless of their length. The reason is that we believe extractive summarization should focus on information coverage and once salient sentences are extracted, one can then compress them in an abstractive way BIBREF0, BIBREF1. Formally, we denote one document-summary pair as $\lbrace d, r\rbrace $, where $d = \lbrace d^j\rbrace _{j=1}^D$, $r = \lbrace r^j\rbrace _{j=1}^R$, and $D$, $R$ denote the number of sentences. We define one support group of facet $\mathcal {F}$ as a minimum set of sentences in the document that express the meaning of $\mathcal {F}$. For each $r^j$, we annotate a FAM $r^j \rightarrow \lbrace \lbrace d^{s_{j, 1}^k}\rbrace _{k=1}^{\textrm {K}_1}, \lbrace d^{s_{j, 2}^k}\rbrace _{k=1}^{\textrm {K}_2}, ..., \lbrace d^{s_{j, N}^k}\rbrace _{k=1}^{\textrm {K}_N}\rbrace $ in which each $\lbrace d^{s_{j, n}^k}\rbrace _{k=1}^{\textrm {K}_n}$ is a support group and $s_{j, n}^k$ is the index of the $k$-th support sentence in group $n$.
One may regard the procedure as creating extractive labels, which is widely used in extractive summarization since only abstractive references are available in existing datasets. The major differences are that 1) We label all the support sentences instead of just one or fixed number of sentences, i.e., we do not specify $\textrm {K}_n$. For example, we would put two sentences to one support group if they are complementary and only combining them can cover the facet. 2) We find multiple support groups ($N > 1$), as there could be more than one set of sentences that cover the same facet and extracting any one of them is acceptable. In contrast, there is no concept of support group in extractive labels as they inherently form one such group. We sampled 150 document-summary pairs from the test set of CNN/Daily Mail. 344 FAMs were created by three annotators with high agreement (pairwise Jaccard index 0.71) and further verified to reach consensus. We found that the facets can be divided into three categories based on their quality and degree of abstraction as follows.
Random: The facet is quite random, either because the document itself is too hard to summarize (e.g., a report full of quotations) or the human editor was too subjective when writing the summary BIBREF2. Another possible reason is that the so-called “summaries” are in fact “story highlights”, which seems reasonable to contain details. We found that 41/150 (26%) samples have random facet(s), implying there are severe issues in the reference summaries of CNN/Daily Mail.
Low Abstraction: The facet can be mapped to its support sentences. We further divide this category by the (rounded) average number of support sentences K of $N$ support groups ($\textrm {K}=\frac{\sum _{n=1}^N |\lbrace d^{s_{j, n}^k}\rbrace _{k=1}^{\textrm {K}_n} \rbrace |}{N})$. As in Table TABREF1, most facets (93%) in the reference summaries are paraphrases or compression of one to two sentences in the document without much abstraction.
High Abstraction: The facet cannot be mapped to its support sentences, which indicates that its writing requires deep understandings of the document rather than reorganizing several sentences. The proportion of this category (7%) also indicates how often extractive methods would not work (well) on CNN/Daily Mail.
Surprisingly, we found it easier than previously believed to create the FAMs on CNN/Daily Mail, as it is uncommon ($\overline{N} = 1.56$) to detect multiple sentences with similar semantics (compared to multi-document summarization). In addition, most support groups only have one or two support sentences with large lexical overlap.
Revisit of State-of-the-art Methods
By utilizing the FAMs, we revisit extractive methods to see how well they perform on facet coverage. Specifically, we compare Lead-3, Refresh BIBREF3, FastRL(E) (E for extractive only) BIBREF0, UnifiedSum(E) BIBREF1, NeuSum BIBREF4, and BanditSum BIBREF5 using both ROUGE and FAMs. As these methods are facet-agnostic (i.e., their outputs are not organized by facets but flat extract sets), we consider one facet is covered as long as one of its support groups is extracted and measure the Facet-Aware Recall ($\textbf {FAR} = \frac{\textrm {\#covered}}{R}$). For a fair comparison, each method extracts three sentences since extracting all would result in a perfect FAR.
As shown in Table TABREF13, there is almost no discrimination among the last four methods under ROUGE-1 F1, and their rankings under ROUGE-1/2/L are quite different. In contrast, FAR shows that UnifiedSum(E) covers the most facets. Although FAR is supposed to be favored as FAMs are already manually labeled and tell exactly if one sentence should be extracted (assuming our annotations are in agreement), to further verify that FAR correlates with human preference, we rank UnifiedSum(E), NeuSum, and Lead-3 in Table TABREF15. The order of the 1st rank in the human evaluation coincides with FAR. FAR also has higher Spearman's coefficient $\rho $ than ROUGE (0.457 vs. 0.44, n=30, threshold=0.362 at 95% significance).
Another benefit of the FAMs is that one can employ the category breakdown for fine-grained analysis under any metrics of interest. Here we consider ROUGE and additionally evaluate several abstractive methods: Pointer-Generator (PG) BIBREF2, FastRL(E+A)(extractive+abstractive) BIBREF0, and UnifiedSum(E+A) BIBREF1. As depicted in Table TABREF16, not only extractive methods fail on high abstraction samples, but there is also a huge performance gap between low and high abstraction samples for abstractive methods, which suggests that existing methods achieve decent performance mainly by extraction rather than abstraction. We also found that all the compared methods perform much worse on the documents with “random” summaries, implying that the randomness in the reference summaries might introduce noise to both model training and evaluation. Despite the fact that the sample size is relatively small, we observed consistent results when analyzing different subsets of the data.
Analysis of Approximate Approaches to Mapping Generation
Although the FAMs only need to be annotated once, we investigate whether such human efforts can be further reduced by evaluating approximate approaches that generate extractive labels. Approximate approaches typically transform one abstractive summary to extractive labels heuristically using ROUGE. Previously one could only estimate the quality of these labels by evaluating the extractive models trained using such labels, i.e., comparing the extracted and reference summaries (also approximately via ROUGE). Now that the FAMs serve as ground-truth extractive labels, we can evaluate how well each approach performs accurately. Since the approximate approaches do not have the notion of support group, we flatten all the support sentences in one FAM to a label set.
Due to limited space, we leave the details of the approximate approaches (most of them are self-evident) to Appendix . The comparison results are shown in Table TABREF17. On the bright side, approximate approaches perform relatively well (e.g., 90.6% selected sentences of BIBREF3 indeed contain salient information). This is explainable as ROUGE is good at capturing lexical overlap and as we have shown, there are many copy-and-paste reference summaries in CNN/Daily Mail. On the other hand, these approaches are not perfect and the low recall suggests that simply mapping each facet with one support sentence would miss plenty of salient sentences, which could worsen the performance of extractive models trained on such labels. That said, how to find more than one support group for each facet or multiple support sentences in one support group automatically and accurately remains an open question.
Conclusions and Future Work
We presented the promising results towards the facet-aware evaluation for extractive summarization. In the future, we will conduct large-scale human annotations in a crowd-sourcing way on the whole test set of CNN/Daily Mail. We will also investigate benchmark multi-document summarization datasets such as DUC BIBREF8 and TAC BIBREF9 to see if the findings coincide and how we can leverage the multiple references provided for each document set in those datasets.

Question: What is the problem with existing metrics that they are trying to address?
Answer: Answer with content missing: (whole introduction) However, recent
studies observe the limits of ROUGE and find in
some cases, it fails to reach consensus with human.
judgment (Paulus et al., 2017; Schluter, 2017).

Question: How do they evaluate their proposed metric?
Answer: manually labeled and tell exactly if one sentence should be extracted (assuming our annotations are in agreement), to further verify that FAR correlates with human preference,

Question: What is a facet?
Answer: Unanswerable