Abstract: Commonly adopted metrics for extractive text summarization like ROUGE focus on the lexical similarity and are facet-agnostic. In this paper, we present a facet-aware evaluation procedure for better assessment of the information coverage in extracted summaries while still supporting automatic evaluation once annotated. Specifically, we treat \textit{facet} instead of \textit{token} as the basic unit for evaluation, manually annotate the \textit{support sentences} for each facet, and directly evaluate extractive methods by comparing the indices of extracted sentences with support sentences. We demonstrate the benefits of the proposed setup by performing a thorough \textit{quantitative} investigation on the CNN/Daily Mail dataset, which in the meantime reveals useful insights of state-of-the-art summarization methods.\footnote{Data can be found at \url{this https URL}.

Introduction: In this section, we describe the procedure of annotating CNN/Daily Mail. For each facet (sentence) in the reference summary, we find all its support sentences in the document that can cover its meaning. Note that the support sentences are likely to be more verbose, but we only consider if the sentences cover the semantics of the facet regardless of their length. The reason is that we believe extractive summarization should focus on information coverage and once salient sentences are extracted, one can then compress them in an abstractive way BIBREF0, BIBREF1. Formally, we denote one document-summary pair as $\lbrace d, r\rbrace $, where $d = \lbrace d^j\rbrace _{j=1}^D$, $r = \lbrace r^j\rbrace _{j=1}^R$, and $D$, $R$ denote the number of sentences. We define one support group of facet $\mathcal {F}$ as a minimum set of sentences in the document that express the meaning of $\mathcal {F}$. For each $r^j$, we annotate a FAM $r^j \rightarrow \lbrace \lbrace d^{s_{j, 1}^k}\rbrace _{k=1}^{\textrm {K}_1}, \lbrace d^{s_{j, 2}^k}\rbrace _{k=1}^{\textrm {K}_2}, ..., \lbrace d^{s_{j, N}^k}\rbrace _{k=1}^{\textrm {K}_N}\rbrace $ in which each $\lbrace d^{s_{j, n}^k}\rbrace _{k=1}^{\textrm {K}_n}$ is a support group and $s_{j, n}^k$ is the index of the $k$-th support sentence in group $n$. One may regard the procedure as creating extractive labels, which is widely used in extractive summarization since only abstractive references are available in existing datasets. The major differences are that 1) We label all the support sentences instead of just one or fixed number of sentences, i.e., we do not specify $\textrm {K}_n$. For example, we would put two sentences to one support group if they are complementary and only combining them can cover the facet. 2) We find multiple support groups ($N > 1$), as there could be more than one set of sentences that cover the same facet and extracting any one of them is acceptable. In contrast, there is no concept of support group in extractive labels as they inherently form one such group. We sampled 150 document-summary pairs from the test set of CNN/Daily Mail. 344 FAMs were created by three annotators with high agreement (pairwise Jaccard index 0.71) and further verified to reach consensus. We found that the facets can be divided into three categories based on their quality and degree of abstraction as follows. Random: The facet is quite random, either because the document itself is too hard to summarize (e.g., a report full of quotations) or the human editor was too subjective when writing the summary BIBREF2. Another possible reason is that the so-called “summaries” are in fact “story highlights”, which seems reasonable to contain details. We found that 41/150 (26%) samples have random facet(s), implying there are severe issues in the reference summaries of CNN/Daily Mail. Low Abstraction: The facet can be mapped to its support sentences. We further divide this category by the (rounded) average number of support sentences K of $N$ support groups ($\textrm {K}=\frac{\sum _{n=1}^N |\lbrace d^{s_{j, n}^k}\rbrace _{k=1}^{\textrm {K}_n} \rbrace |}{N})$. As in Table TABREF1, most facets (93%) in the reference summaries are paraphrases or compression of one to two sentences in the document without much abstraction. High Abstraction: The facet cannot be mapped to its support sentences, which indicates that its writing requires deep understandings of the document rather than reorganizing several sentences. The proportion of this category (7%) also indicates how often extractive methods would not work (well) on CNN/Daily Mail. Surprisingly, we found it easier than previously believed to create the FAMs on CNN/Daily Mail, as it is uncommon ($\overline{N} = 1.56$) to detect multiple sentences with similar semantics (compared to multi-document summarization). In addition, most support groups only have one or two support sentences with large lexical overlap.

Question: What is the problem with existing metrics that they are trying to address?
Answer: Answer with content missing: (whole introduction) However, recent
studies observe the limits of ROUGE and find in
some cases, it fails to reach consensus with human.
judgment (Paulus et al., 2017; Schluter, 2017).