SlowGuess/ACL_2020_OCR huggingface dataset:
is somewhat expected since they cannot perform abstraction. Abstractive methods, however, also exhibit a huge performance gap between low and high abstraction samples, which suggests that existing abstractive methods achieve decent overall performance mainly by extraction rather than abstraction, i.e., performing well on low abstraction samples of CNN/Daily Mail. We also found that all the compared methods perform much worse on the documents with "noisy" reference summaries, implying that the randomness in the reference summaries might introduce noise to both model training and evaluation. Note that although the sample size is relatively small, we observe consistent results when analyzing different subsets of the data.
Method	N	L	H	L + H
Extractive	Lead-3	34.1	41.9	24.9	38.9
FastRL(E)	33.5	41.6	31.2	39.8
BanditSum	35.3	42.7	34.1	41.2
NeuSum	34.9	42.7	30.7	40.6
Refresh	35.7	42.8	32.2	40.9
UnifiedSum(E)	34.2	42.6	31.3	40.6
Abstractive	PG	32.6	40.6	27.5	38.2
FastRL(E+A)	35.1	40.8	29.9	38.8
UnifiedSum(E+A)	34.2	42.4	29.2	40.1
Table 5: ROUGE-1 F1 of extractive and abstractive methods on noisy (N), low abstraction (L), high abstraction (H), and high quality  $(\mathrm{L} + \mathrm{H})$  samples.
3.4 Comparative Analysis
Facet-aware evaluation is also beneficial for comparing extractive methods regarding their capability of extracting salient and non-redundant sentences. We show the FAR, SAR, and ROUGE scores of various extractive methods in Fig. 2. We next illustrate how one can leverage these scores under different metrics for comparative analysis. For brevity, we denote ROUGE Precision and ROUGE Recall as RP and RR, respectively.
FAR vs. ROUGE. By comparing the scores of extractive methods under FAR and ROUGE, one can discover useful insights. For example, we observe that the performance of Refresh, FastRL(E), NeuSum are quite close to Lead-3 under FAR, but they generally have higher RR. Such results imply that these methods might have learned to extract sentences that are not the support sentences, i.e., sentences that do not directly contribute to the facet coverage, but still have lexical overlap with reference summaries. It is also likely that they extract
redundant support sentences that happen to have token matches with other facets. Overall, Unified-Sum(E) covers the most facets (high FAR) and also has decent lexical matches (high RR).
SAR vs. ROUGE. By comparing SAR with RP, one can find that UnifiedSum(E) extracts salient but possibly redundant support sentences, as it has higher SAR but similar RP to Lead-3. On the contrary, Refresh has similar SAR with Lead-3 but higher RP, which again implies that it might extract non-support sentences that contain token matches but irrelevant semantics. Similarly, BanditSum is capable of lexical overlap (high RP), but the matched tokens may not contribute much to the major semantics (low SAR).
FAR vs. SAR. By comparing FAR with SAR (Fig. 3), we observe that FastRL(E) and NeuSum have FAR scores similar to Lead-3 and Refresh, but higher SAR scores. One possible explanation is that FastRL(E) and NeuSum are better at extracting support sentences, but they do not handle redundancy very well, i.e., the extracted sentences might contain multiple support groups of the same facet (recall the example in Sec. 3.1). For instance, there are  $30.3\%$  extracted summaries of FastRL(E) that can cover more than one support group of the same facet while there are  $19.1\%$  for Lead-3.
4 Evaluation without Human Annotation
In the previous sections, we have demonstrated the effectiveness and benefits of facet-aware evaluation. One remaining issue that might prevent facet-aware evaluation from scaling is the need of human-annotated FAMs. We thus study the feasibility of automatic FAM creation with sentence regression and present a pilot study of conducting facet-aware evaluation without any human annotation in this section.
4.1 Sentence Regression for FAM Creation
Similar to most benchmark constructions, facet-aware evaluation requires one-time annotation once the FAMs are annotated, we can reuse them for automatic evaluation. That said, we explore various approaches to automate this one-time process. Specifically, we investigate whether facet-aware evaluation can be conducted without any human effort by utilizing sentence regression (Zopf et al., 2018) to automatically create the FAMs.
Sentence regression is widely used to create extractive labels. Sentence regression approaches typ
Figure 2: Performance of extractive methods under ROUGE, FAR, and SAR. The results under ROUGE-1/2/L often disagree with each other. UnifiedSum(E) generally performs the best in the facet-aware evaluation.
Figure 3: Comparison of extractive methods under FAR and SAR reflects their capability of extracting salient and non-redundant sentences.
ically transform abstractive reference summaries to extractive labels heuristically using ROUGE. Previously, one could only estimate the quality of these labels by evaluating the extractive models trained using such labels, i.e., comparing their extracted summaries with the reference summaries (also approximately via ROUGE). Now that the human-annotated FAMs serve as ground-truth extractive labels, we can evaluate how well each approach performs accurately.
Sentence Regression Approaches. We briefly review recent sentence regression approaches as follows. Nallapati et al. (2017) greedily select sentences that maximize ROUGE-1 F1 until adding
another sentence decreases it. Chen and Bansal (2018) find for each reference sentence the most similar sentence in the document by ROUGE-L recall. Zopf et al. (2018) argue that precision is a better measure than recall because it aims not at covering as much information but at wasting as little space as possible. Narayan et al. (2018c) measure sentence similarity by the average of ROUGE-1/2/L F1. We also test other variants of ROUGE and TF-IDF, which represents sentences by TF-IDF features and measures their cosine similarity.
4.2 Evaluation with Machine-Created FAMs
Results on Support Sentence Discovery. We first evaluate sentence regression with its original function, i.e., creating extractive labels (finding support sentences). We merge the support groups of each sample and calculate precision and recall (i.e., SAR). The performance of sentence regression approaches is shown in Table 6. The relatively low recall suggests that simply finding one support sentence for each facet as most existing approaches do would miss plenty of salient sentences, which could possibly worsen the models trained on such labels since the models would treat missed support sentences as unimportant ones. On the bright side, many sentence regression approaches achieve high
precision. For instance,  $90.0\%$  document sentences labeled positive by Narayan et al. (2018c) indeed contain salient information. This is to some extent explainable as ROUGE captures lexical overlap and as we have shown, there are many copy-and-paste reference summaries in CNN/Daily Mail.
Method	Precision	Recall	F1
Lead-3	61.0	33.7	43.4
Greedy ROUGE-1 F1	58.2	30.8	40.3
TF-IDF	83.7	51.9	64.0
ROUGE-1 F1	88.9	53.1	66.5
ROUGE-2 F1	86.6	52.3	65.2
ROUGE-L Recall	89.3	53.7	67.1
ROUGE-L Precision	77.2	45.5	57.2
ROUGE-L F1	87.8	53.5	66.5
ROUGE-AVG F1	90.0	53.9	67.4
Correlation w. Human-Annotated FAMs. We then explore the correlation between human-annotated and machine-created FAMs by evaluating extractive methods against both of them. This time we extend to find for each facet multiple support sentences and put each support sentence into a separate support group. We measure the correlation between estimated and ground-truth FAR by Pearson's  $r$ . We measure the correlation between system rankings induced from estimated and ground-truth FAR by Spearman's  $\rho$  and Kendall's  $\tau$ . The detailed correlation results of representative approaches are listed in Table 7. We observe that creating three support groups consistently shows the highest correlation for the same sentence regression approach. Also, the FAMs created by ROUGE-1 F1 and ROUGE-AVG F1 have very high correlation with human annotation, indicating the usability and reliability of machine-created FAMs for system ranking.
Table 6: Performance of sentence regression approaches regarding support sentence discovery. High precision and low recall are often observed.
Method	N = 1	N = 2	N = 3
r	ρ	τ	r	ρ	τ	r	ρ	τ
ROUGE-1 F1	70.5	37.1	33.3	72.0	71.4	60.0	88.4	94.3	86.7
ROUGE-2 F1	11.0	25.7	20.0	43.4	65.7	46.7	88.4	65.7	60.0
ROUGE-L F1	34.0	54.3	46.7	37.5	42.9	20.0	62.3	42.9	46.7
ROUGE-AVG F1	49.6	54.3	46.7	46.1	65.7	46.7	83.2	82.9	73.3
FAR Prediction. Despite the high correlation, we also find that the estimated FAR scores may vary in
range compared to the ground-truth FAR. $^{7}$  Therefore, we further use the estimations of different sentence regression approaches to train a linear regression model to fit the ground-truth FAR (denoted as AutoFAR). We then calculate the estimated FAR scores on the whole test set of CNN/Daily Mail and use the trained linear regressor to predict a (supposedly) more accurate FAR score (denoted as AutoFAR-L). As shown in Table 8, the fitting of AutoFAR is very close to the ground-truth FAR, and the system ranking on the large-scale evaluation under AutoFAR-L follows a similar trend to that under FAR with Spearman's  $\rho = 54.3$ . On the other hand, although our preliminary analysis on AutoFAR-L shows promising results, we also note that since the human annotation on the whole test set is lacking, the reliability of such extrapolation is not guaranteed and we leave more rigorous study with a larger number of systems and samples as future work.
Table 7: Correlation between ground-truth and estimated FAR scores by Pearson's  $r$ , Spearman's  $\rho$ , and Kendall's  $\tau$ .  $N$  denotes the number of support groups.
Method	FAR	AutoFAR	AutoFAR-L	FAR vs. AutoFAR(-L)
BanditSum	44.7	44.8	44.7	Pearson's r
Lead-3	50.6	51.3	45.6	97.6 (42.9)
FastRL(E)	50.8	51.0	43.1	Spearman's ρ
NeuSum	51.2	49.9	44.3	77.1 (54.3)
Refresh	51.3	51.7	46.2	Kendall's τ
UnifiedSum(E)	54.8	54.5	46.9	60.0 (46.7)
Table 8: FAR prediction via linear regression. AutoFAR(-L) denotes the results on the human-annotated subset (entire CNN/Daily Mail dataset).
Evaluation Metrics for Text Summarization. ROUGE (Lin, 2004) is the most widely used evaluation metric for text summarization. Extensions of ROUGE include ROUGE-WE (Ng and Abrecht, 2015) that incorporated word embedding into ROUGE, ROUGE 2.0 (Ganesan, 2018) that considered synonyms, and ROUGE-G (ShafieiBavani et al., 2018) that applied graph analysis to WordNet for lexical and semantic matching. Nevertheless, these extensions did not draw enough attention as the original ROUGE and recent advances (Gu et al., 2020; Zhang et al., 2019a) are still primarily evaluated by the vanilla ROUGE.
Another popular branch is Pyramid-based metrics (Nenkova and Passonneau, 2004; Yang et al., 2016), which annotate and compare the Summarization Content Units (SCUs) in the summaries.
The raw estimated FAR scores are provided in Appendix B Fig. 5 in the interest of space.
FAR is related to Pyramid and HighRES (Hardy et al., 2019) in that Pyramid employs the summaries to annotate SCUs and HighRES highlights salient text fragments in the documents, while FAR considers both the summaries and documents.
Beyond lexical overlap, embedding-based evaluation metrics (Zhang et al., 2019b; Zhao et al., 2019; Sun and Nenkova, 2019; Xenouleas et al., 2019) are gaining more traction along with the dominance of pre-trained language models. One straightforward way to incorporate embedding-based metrics into FAR is to use them as similarity measures instead of the ROUGE-based approaches tested in Sec. 4.1 for automatic FAM creation (i.e., finding support sentences for each facet by the scores of embedding-based metrics). Such similarity measures are especially beneficial when the facet and its support sentences are not similar at the lexical level.
Reflections on Text Summarization. There has been increasing attention and critique to the issues of existing summarization metrics (Schluter, 2017), methods (Kedzie et al., 2018; Shapira et al., 2018), and datasets (Jung et al., 2019). Notably, Kryscinski et al. (2019) conducted a comprehensive critical evaluation for summarization from various aspects. Zopf et al. (2018) investigated sentence regression approaches in a manner similar to ours but they could only evaluate them approximately against ROUGE as no ground-truth labels (FAMs) existed.
Annotation and Analysis. Many recent studies conduct human annotation or evaluation on text summarization and other NLP tasks to gain useful insights. Hardy et al. (2019) annotated 50 documents to demonstrate the benefits of highlight-based summarization evaluation. Recent summarization methods (Paulus et al., 2017; Narayan et al., 2018c; Chen and Bansal, 2018) generally sampled 50 to 100 documents for human evaluation in addition to ROUGE in light of its limitations. Chen et al. (2016); Yavuz et al. (2018) inspected 100 samples and analyzed their category breakdown for reading comprehension and semantic parsing, respectively. We observed similar trends when analyzing different subsets of the FAMs, indicating that our findings are relatively stable. We thus conjecture that our sample size is sufficient to verify our hypotheses and benefit future research.
We propose a facet-aware evaluation setup for better assessment of information coverage in extractive summarization. We construct an extractive summarization dataset and demonstrate the effectiveness of facet-aware evaluation on this newly constructed dataset, including better human correlation on the assessment of information coverage, and the support for fine-grained evaluation as well as comparative analysis. We also evaluate sentence regression approaches and explore the feasibility of fully-automatic evaluation without any human annotation. In the future, we will investigate multi-document summarization datasets such as DUC (Paul and James, 2004) and TAC (Dang and Owczarzak, 2008) to see whether our findings coincide when multiple references are provided. We will also explore better sentence regression approaches for the use of both extractive summarization methods and automatic FAM creation.
We thank Woojeong Jin and Jiaming Shen for the valuable feedback on the paper draft. We thank anonymous reviewers for the constructive comments. Research was sponsored in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and SocialSim Program No. W911NF-17-C0099, National Science Foundation IIS 16-18481, IIS 17-04532, and IIS-17-41317, and DTRA HDTRA11810026. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of DARPA or the U.S. Government.
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. In TAC.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3739-3748, Brussels, Belgium. Association for Computational Linguistics.
Kavita Ganesan. 2018. Rouge 2.0: Updated and improved measures for evaluation of summarization tasks. arXiv preprint arXiv:1803.01937.
Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, Hongkun Yu, You Wu, Cong Yu, Daniel Finnie, Ji-aqi Zhai, and Nicholas Zukoski. 2020. Generating representative headlines for news stories. WWW.
Hardy Hardy, Shashi Narayan, and Andreas Vlachos. 2019. HighRES: Highlight-based reference-less evaluation of summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3381-3392, Florence, Italy. Association for Computational Linguistics.
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 132-141, Melbourne, Australia. Association for Computational Linguistics.
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019. Earlier isn't always better: Sub-aspect analysis on corpus and system biases in summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3315-3326, Hong Kong, China. Association for Computational Linguistics.
Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818-1828, Brussels, Belgium. Association for Computational Linguistics.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI, pages 3075-3081.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Ronald Cardenas, Nikos Papasarantopoulos, Shay B. Cohen, Mirella Lapata, Jiangsheng Yu, and Yi Chang. 2018a. Document modeling with external attention for sentence extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2020-2030, Melbourne, Australia. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018c. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925-1930, Lisbon, Portugal. Association for Computational Linguistics.
Over Paul and Yen James. 2004. An introduction to duc-2004. In Proceedings of the 4th Document Understanding Conference (DUC 2004).
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41-45, Valencia, Spain. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, and Fang Chen. 2018. A graph-theoretic summary evaluation for ROUGE. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 762-767, Brussels, Belgium. Association for Computational Linguistics.
Ori Shapira, David Gabay, Hadar Ronen, Judit Bar-Ilan, Yael Amsterdamer, Ani Nenkova, and Ido Dagan. 2018. Evaluating multiple system summary lengths: A case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 774-778, Brussels, Belgium. Association for Computational Linguistics.
Simeng Sun and Ani Nenkova. 2019. The feasibility of embedding based automatic evaluation for single document summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1216-1221, Hong Kong, China. Association for Computational Linguistics.
Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. SUMQE: a BERT-based summary quality estimation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6004-6010, Hong Kong, China. Association for Computational Linguistics.
Qian Yang, Rebecca J Passonneau, and Gerard De Melo. 2016. Peak: Pyramid evaluation via automated knowledge extraction. In Thirtieth AAAI Conference on Artificial Intelligence.
Semin Yavuz, Izzeddin Gur, Yu Su, and Xifeng Yan. 2018. What it takes to achieve  $100\%$  condition accuracy on WikiSQL. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1702-1711, Brussels, Belgium. Association for Computational Linguistics.