EntSUMv2: Dataset, Models and Evaluation for More Abstractive Entity-Centric Summarization
Authors: Dhruv Mehra
Lingjue Xie
Ella Hofmann-Coyle
Mayank Kulkarni
Daniel Preotiuc-Pietro
Venue: EMNLP 2023 Main (2023)
Keywords: summarization
entity
dataset
evaluation
TL;DR: This paper presents EntSUMv2, a more abstractive version of the original EntSUM dataset, where the summaries are intentionally made shorter to benefit more specific and useful entity-centric summaries for downstream users.

Abstract: Entity-centric summarization is a form of controllable summarization that aims to generate a summary for a specific entity given a document. Concise summaries are valuable in various real-life applications, as they enable users to quickly grasp the main points of the document focusing on an entity of interest. This paper presents ENTSUMV2, a more abstractive version of the original entity-centric ENTSUM summarization dataset. In ENTSUMV2 the annotated summaries are intentionally made shorter to benefit more specific and useful entity-centric summaries for downstream users. We conduct extensive experiments on this dataset using multiple abstractive summarization approaches that employ supervised fine-tuning or large-scale instruction tuning. Additionally, we perform comprehensive human evaluation that incorporates metrics for measuring crucial facets. These metrics provide a more fine-grained interpretation of the current state-of-the-art systems and highlight areas for future improvement.

ENTSUMv2:
Data, Models and Evaluation for More Abstractive Entity-Centric Summarization
Dhruv Mehra Lingjue Xie Ella Hofmann-Coyle"*
Mayank Kulkarni*) Daniel Preofiuc-Pietro" Bloomberg Amazon Alexa Al
dmehra wwmbbbbbrbbrrooo.ooooooooooorrr. net, cxieerebloomberg.. net, ehofmanncoyl www.ll@ebbeggmeeeeee
maykul@amazon. com, dpreotiucpie@hl .nnber...ett net
Abstract
and Gillick, 2014), aspect-based sentiment classifi- cation (Pontiki et al., 2016) or question answering. Abstractive summarization methods aim pro- duce new summaries (Nenkova et al., 2011), which can be obtained through selection, compression and reformulation of the given source document. Compared to extractive summarization, abstractive summarization can produce concise summaries that capture the essence of the source text using fewer words, making them more efficient for users consume. However, abstractive summarization is prone to suffer from issues in consistency with the source document (or factual errors), coherence or fluency (Cao et al., 2018; Kryscinski et al., 2019; Lebanoff et al., 2019). To evaluate abstractive sum- maries, automatic metrics have been proposed, al- though their correlation with human evaluation on the desirable facts for summary not always high or consistent (Fabbri al.. 2021).
Entity-centric summarization is a form of con- trollable summarization that aims to generate summary for specific entity given a document. Concise summaries are valuable in various real- life applications, as they enable users quickly grasp the main points of the document focusing on an entity of interest. This paper presents ENTSUMV2,
abstractive version of the original entity- centtic ENTSUM summa- rization dataset. ENTSUMM2 the annotated summaries intentionally made shorter to benefit more specific and useful entity-centric summaries for downstream users. We conduct extensive experiments this dataset using mul- tiple abstractive summarization approaches that employ supervised fine-tuning large-scale in- struction tuning. Additionally, we perform com- prehensive human evaluation that incorporates metrics for measuring crucial facets. These met- rics provide a more fine-grained interpretation the current state-of-the -frthe-art systems and high- light areas for future improvement.
In this paper, focus on the task of abstrac- tive entity-centric summarization. Past research on this topic was limited by the ability to comprehen- sively evaluate models, relying cither on single- faceted human quality judgments (Fan et al., 2018; He et al., 2020; Goyal et al., 2023) or reference entity-centric summaries which were very extrac- tive (Maddela et al., 2022). To this end, we release updated version of the ENTSUM dataset (Mad- dela et al., 2022), named ENTSUMV2, where sum- maries are deliberately made shorter and more ab- stractive. Moreover, we enhance the evaluation pro- cess of entity-centric summarization methods by in- corporating a comprehensive multi-faceted human evaluation, specifically designed for this task. This human evaluation complements the standard auto- matic metrics, including ROUGE and BERTScore. By incorporating both automatic metries and hu- man evaluation, we aim to provide a thorough and robust evaluation of summarization model perfor- mance and show the path forward to improving models for this task.
1 Introduction
Controllable summarization is rapidly expanding field of research that deals with creating summaries tailored to different elements (Fan et al., 2018; He et al., 2020; Hofmann Coffann--oolle et al., 2022). The con- trollable elements include entities (Maddela et al., 2022), aspects (Amplayo et al., 2021; Ahuja et al.. 2022), users' preferred style (Fan et al., 2018) and length (Kikuchi et al., 2016; Dou et al., 2021). Con- trollable summarization has the promise to increase the utility and usability of summarization systems by enabling users to obtain summaries that align with their specific needs and preferences (Maddela et al., 2022). Further, controllable summaries can be used in downstream applications like search (Varadarajan and Hristidis, 2006; Turpin et al., 2007), entity salience (Gamon et al., 2013: Dunietz
"The authors contributed equally
Work done while at Bloomberg

---

Separately, we explore training several model ar- chitectures on this task and propose several im- proveeents to the training process, which sub- stantially outperform the existing state-of-the-ae art (+2.5 BERTScore, +4.4 Rouge-L), instruction- tuned models and the strong entity-centric Lead3 heuristic.
3 Methods
We experiment with several methods for abstractive summarization as follows:
3.1 Heuristics
Lead3oor is generic summarization approach that disregards the target entity and simply selects the first three sentences from the document.
2 Data
Lead3 selects the first three sentences in the document specifically mentioning the given entity following entity detection and coreference resolu- tion, as described ENTSUM (Maddela et al., 2022).
In this paper, we introduce the ENTSUMv2 dataset which contains more compressed abstractive sum- maries when compared to the original ENTSUM dataset. We build the ENTSUM dataset on top of The New York Times' summarization corpus (hereafter referred to as NYT) which avail- able to use via the LDC.! The dataset shares the same set of documents as ENTSUM, but with a stricter length constraint of up to 60 words, half of ENTSUM's. The annotations are performed by annotators trained over multiple rounds pro- prietary annotation platform. The annotators are presented with the original document, the target entity and the salient sentences for the target entity as annotated in the original ENTSUM dataset. A diagram with the annotation process is presented in Appendix A. For quality control, we addition- ally calculated the inter-annotator agreement for the EnISUMv2 dataset in the final training round using ROUGE-[1,22 and BERTScore between the abstractive summary and the proxy summary (entity salient sentences) provided annotators at annotation time. The EntSUMv2 Krippendorff's alpha for ROUGE-[1,2,L] and BERTscore are 0.75, 0.84, 0.85 and 0.81 respectively, indicating a high overlap. We collect single summary for each doc- ument and entity pair.
3.2 GSum
We start with GSument-eent: an entity-centric sum- marization version of GSum (Dou et al., 2021) which obtained the best performance on abstrac- tive summarization on ENTSUM (Maddela et al., 2022). GSum is summarization framework that incorporates two encoders: one for the source doc- ument and another for the guidance signal. Our GSum setup closely follows the setup outlined in ENTSUM (Maddela et al., 2022), where the model weights are initialized with BART (Lewis et al., 2019) with a few modifications which we find lead to improved performance. First, we incorporate a dropout layer (p=0.5) into the guidance signal encoder stack of the model architecture. Second, we experiment with two step training process (wo-step) motivated by an analysis on the GSum results which showed the entity-centric model's ability select key information from the source document could be improved. In Step 1, we train the GSum model to generate generic document summaries by providing only the source document and an empty guidance signal input. In Step 2, we load the best generic GSum summarization check- point and fine-tune with the entity guidance signal and proxy entity-centric summaries, described in (Maddela al., 2022), to produce entity-centric summaries.
Table displays summary statistics for the newly introduced ENTSUMV2 dataset in comparison to ENTSUM and other public datasets for summa- rization. There is a notable increase in the occur- rence of novel n-grams compared to ENTSUM, albeit still less than other datasets like NYT or CNN/Daily Mail (Nallapati al., 2016). Moreover, the average summary length in ENTSUMV2 is sig- nificantly shorter, with an average of 46 words com- pared to the 81 words in ENTSUM. This stricter length constraint presents challenge for the model to effectively select the most essential information within the summarized output.
3.3 T5
T5 (Raffel et al., 2020) is a transformer-based encoder-decoder model that is pretrained using text with dropped token sequences as input and the dropped tokens delimited by their sentinel tokens as output. We fine-tune two base versions of the T5 model for entity-centric summarization. The first is b5-base, trained with a combination
'https://catalog. wpw..pp....dccccoomm http:cc.ppeeeeeeee/0/00000011

---

Avg. summary len.
Avg. article len.
novel ngram
Dataset
Size
sents.
words
char.
sents.
words
char.
unigram bigram
NYT
41265
4.9
117
677
36.9
1021
5471
11.5
39.5
CNN/DAILY MAIL
312085
3.7
56
297
33.1
782
3998
13.3
49.95
ENTSUM
2788
2.5
81
444
34.4
1002
5319
0.82
5.93
ENTSUMV2
2788
1.8
46
251
34.4
1002
5319
1.69
10.72
Table 1: Comparison of the existing document summarization datasets with ENTSUM. We report the corpus size, average article and summary length (in terms of words, sentences, and characters), and percentage of novel n-grams in the summary when compared to the article.
supervised tasks including summarization. The second is T5-v1. 1--vse, pretrained solely on the un- supervised objective, allowing us to assess its per- formance independently of mixed task fine-tuning other summarization data. In addition, inves- tigate training setups: experiment with fine- tuning the model with proxy entity-centric marization only (proxy), or train in two steps (two-step), wherein we initially train the model to generate generic summaries and subsequently fine-tune for proxy entity-centric summarization. The second approach aims to provide the models with additional contextual understanding through the first step of training.
Experimental Setup
Training Data We employ proxy summaries from the NYT corpus during the fine-tuning process of GSum and T5 models for entity-centriy sum- marization. The original corpus comprised 44,382 training and 5,523 validation pairs (document, sum- mary) for generic summary. To generate entity- centric summaries, we select the first three sen- tences that mention the target entity. This selection is based on the entity recognition and coreference resolution methods described in (Maddela al., 2022). Given that each document the corpus con- tained multiple entities, the training set expanded to 464,339 pairs, while the validation set grows to 58,991 pairs.
3.4 Flan-T5
Test Data We use the ENTSUMv2 dataset for eval- uation only, following (Maddela et al., 2022). We conduct experiments by splitting this dataset into training and test sets. However, training on this dataset, even when combined with the additional proxy summaries, does not result any perfor- mance improvements. Therefore, to ensure more ac- curate and dependable evaluations of model perfor- mance, we utilize the entire ENTSUMV2 dataset exclusively for testing purposes.
Large-scale instruction tuning using diverse NLP tasks has emerged as an alternative single-task fine-tuning. We examine the efficacy of Flan- T5 (Chung et al., 2022), an enhanced version of the T5 model, which has undergone instruction-tuning us- ing a wide range of tasks and instructions, including several summarization datasets such as CNN/Daily Mail (Nallapati et al., 2016), Gigaword (Rush et al., 2015), MultiNews (Fabbri et al., 2019), Sam- Sum (Gliwa et al., 2019) and XSum (Narayan et al., 2018). To facilitate zero-shot entity-centric summa- rization, we employ prompt engineering techniques to guide the model in generating entity-centric sum- maries. We develop entity-centric summarization prompt templates, inspired by the Flan Collection templates² and explore two input strategies, as the model not originally trained for the entity- centric summarization task. In the first, provide the complete source document input, and in the second only sentences containing the entity and its coreference are provided. Appendix C, we present the performance evaluation of the Flan-T5 T5 model across different prompts.
Implementation Details The T5-base 3, T5-v1.1 and Flan-T5-bur models are obtained from the HuggingFace model repository. We use the GSum implementation provided by the authors." 6 In our GSum experiments, we adhere to the hyperparam- eters and implementation details outlined in the GSum framework. We conduct fine-tuning of the T5 and T5 v1.1 models for 2 epochs with a learn- ing rate of 2e-5. The batch size is set to 32, and the experiments are performed on Nvidia Tesla V 1000 GPUs, During inference, we impose a constraint
hnttps://huggingface co/t5- base "https::/huggiingface. ct/
http:///wwgggggaacc.oo////////ee adde_ddoc/t5v1.1
hhttps://huggingface http://uuggingaaccccoooggg////// cowgooolle/llanntt555.. (5-baae
'https://gi thub. cww/neul httpg/uuaaaagggd......
hhttps://gtthbb.com//ggg//// httpr/hhw..mom/oogleeeeeeeerrrre
summarization

---

the T5 models to limit the generated output to maximum of 60 tokens.
mary) (Hofmann-Coyle et al., 2022). Despite evaluating on abstractive summaries, there re- mains a gap compared these oracle extractive methods, highlighting that abstractive methods still need to be further enhanced to identify key entity information.
5 Results
We evaluate all models using both automatic and human evaluation for a more comprehensive view on model performance.
5.2 Faceted Human Evaluation
5.1 Automatic Evaluation
We conduct human evaluation of three top-performing methods of each type (GSum stop dropout. T5-v1.1-baseeuo-str Flan-TS-basep2 paae----aaSe--entttt (ent2tttett)) for more comprehensive assessment. T5-v1.1-base is selected for a fair comparison with GSum as T5-base is trained with additional summarization datasets. The T5v1.1 output is restricted to the first 60 tokens for a fair evaluation, as tends to produce longer summaries. Three independent raters evaluated all 480 summaries each. Summaries are ranked on Likert scale of to 5, with a focus on crucial aspects: entity-specificity (or relevance), factuality (or consistency), and fluency aligning with previous work on human evaluation for sum- marization (Kryseinski et al., 2019; Fabbri et al., 2021). We also include completeness, specifically for measuring the controllability aspect and an overall quality score. The evaluation guidelines are provided in Appendix E. The trained annotators achieve a Krippendorf Alpha (Krippendorfff, 2011) of 0.48 with the authors subset of 100 annotations. The inter-annotator agreement between annotators on the five aspects is 0.73. The agreement numbers are in line to past research (Fabbri et al., 2021). The facet based scores indicate that:
The results of automatic evaluation are reported in Table 2. We employ the same set of automated metrics used in ENTSUM, namely ROUGE-I, ROUGE-2, ROUGE-L (Lin and Hovy, 2003) and BERTScore (Zhang et al., 2020). The results show: GSum and T5 based methods perform similarly in their best configurations, with GSum slightly better on BERTScore.
The best performing summarization model out- performs the strong Lead3 entity-centric base- line on R-1 (+2.2), R-2 (+3), R-L (+2.2) and BERTScore (+0.9).
Two step training on generic, then entity-centric summaries is beneficial, improving results on GSum. Since GSum takes in 2 inputs (source document and guidance signal) as opposed T5 which only takes a single input, we suspect that the two step training process acts like a curricu- lum based learning approach which helps the model learn more effectively. The GSum model first learns to summarize the overall key informa- tion from the provided source document. Then, uses the additional provided signal to summarize the information relevant to the provided entity.
random
Instruction tuned models obtain decent results but only as part of pipeline that selects the entity related sentences priori. Otherwise, their performance is similar or lower to the Lead3 generic summary heuristic, showing they can not perform the entity control aspect.
GSum and T5 demonstrate divergent facet-level performance, notably overall quality, despite similar overall ROUGE and BERTScore results.
The different architectures of these models lead to distinct summary patterns, with GSum ex- celling in factuality and completeness.
Both T5 and T5-v1.1 achieve comparable per- formance after being fine-tuned on proxy entity- centric summarization, despite T5-base being ini- tially fine-tuned with multiple supervised tasks, including summarization. This shows that fur- ther training on out-of-domain summaries pro- vides diminishing gains.
Despite the 10-point R-1 score difference be- tween T5 and Flan-T5, the performance gap nar- rows in human evaluation. Flan-T5 is trained on larger corpus and diverse tasks, which aid in sentence fluency but inhibit its performance in other areas due to the generic nature of its pre- trained tasks. Additionally, both T5 and Flan-T5 struggle more with factuality, generating inaceu- rate or fictional information.
We also compare the summarization mod- els oracle extractive summarization models that rely on identifying the Lead3 salient sen- tences (Lead3ent Salient) and Lead3 sentences used to write the summary (Lead3 Sum-
All models are able to obtain controllability, al-

---

ROUGE-1
ROUGE-2
ROUGE-L.
BERTScore
Avg. Len Sent. Word
Heuristics
Lead3mur
28.8
15.1
25.6
55.3
3.0/ 3.0/99.38
99.38
LeadBent
57.5
48.5
54.5
73.3
2.76 92.31
Abstractive Summarization Methods with Fine-tuning
GSument-sent
55.1g.17
17
12.3n.1a
71.7o.
3.05p.21
/99.660.2
GSum
59.7o.04
51.56.03
56.7o.03
74.20. 04
2.88o.01
190.0o.13
TS- 1S-baaeprory
60.80.34
50.20.39
56.0u.34
72.8u.16
1.50u oz 43. 33..o..7
TS-
1S-base/wo-xtep
21.60.78
51.0p.65
56.7o.re
73.20.30
1.51o.ca 02 /44.55..7 71
1.I1-basprrazy
61.5p.24
51.1o.28
56.7o.28
13.1
1.550.01
43.70o.
15-v1l1-baseruo- b-baScruo-step
61.3p.12
50.9p.17
56.60.12
73.00.12
1.55p.01 /43 43.48u.ue
Abstractive Summarization Methods with Instruction-tuned Models and Zero-Shot Inference
Saan-T--base2---o
25.3
11.9
21.6
54.6
1.24 35.0
Flan-TS Taa--S-basey2-entt basep2-entity
$2.1
40,9
47.8
69.8
1.09 / 32.2
TS- IS-baseentity
48.2
34.1
43.4
65.6
1.78 / 36.9
Methods using Oracle Entity Sentence Information
Lead3ene Salient
62.8
55.4
59.6
76.3
2.73 /9131
Lead3ent Summary
69.6
63.5
66.6
86.0
Table 2: Automatic evaluation results of different summarization models the ENTSUMv2 dataset. Bold typeface denotes the best performance overall and underlined numbers represent best performance within class of methods. The fine-tuning results are averaged over 3 with different seeds and standard deviation is provided in the subscript.
Model
Speett-Speciffcity
Factuality
Completeness
Fluency
Quality
p.85
4.690.62
3.7l0.82
4.17a.us
3.170.67
T5-v1. T5-vl.1-bastsoo- ---basttwo-step
4.720.96
4.171.43
3.1 1.11
4.36p.6
2.771.u
4.581.12
4.061.9
3.061.06
4.640.88
2.760.97
Table 3: Human evaluation results (average score,tdes) of three types of summarization models on a subset of the ENTSUMv2 dataset. Bold typeface denotes the best performance.
though Flan-T5 lags behind the other models, even if fed with sentences that contain the entity,
as base for writing our summaries. Thus, the paper does not test the generalizability of our models and findings to other languages.
6 Conclusions
We train the model for predetermined num- ber of epochs without task specific validation validation dataset for entity-centric summariza- tion is not available and we only use the entire ENTSUMv2 dataset for evaluation.
This paper presents comprehensive analysis of abstractive entity-centric summarization. We intro- duce a new dataset - ENTSUMV: with summaries that are more abstractive and almost half the length of the summaries in ENTSUM, posing additional challenges to summarization models. We explore different model types, improving upon previous -ep-performing models through data insights and training techniques, well surpassing the strong Lead3 cntity-centric bascline. Finally, we conduct the first multi-faceted human evaluation on entity- centric summarization, revealing detailed insights into model behavior and trade-offs, suggesting po- tential avenues for further enhancement.
We limit our experimentation to the T5-base model due to its comparable number of parameters with the GSum model and due to limited compute resources. However, exploring the training of larger T5 models can provide valuable insights into the impact of model size on task performance.
We only use arguably the most popular metrics for automatic summarization (ROUGE, BERTScore). Using more metrics could provide more complete picture of model performance.
Limitations
Acknowledgements
We only study the task of entity-centric summariza- tion in English, as this relatively new task and there are no other datasets to build on with relevant and salient entity sentences selected, which we use
We would like thank Rajarshi Bhowmik and the many members of Bloomberg AT group who pro- vided invaluable feedback on this paper. We are grateful to our annotators for their diligence in per-

---

forming this annotation task and human evaluation of the summaries.
Angela Fan, David Grangier, and Michael Auli 2018, Controllable abstractive summarization. In Proceed- ings of the 2nd Workshop on Neural Machine Transla- tion and Generation, pages 45-54, Melbourne, Aus- tralia. Association for Computational Linguistics.
References
Michael Gamon, Tae Yano, Xinying Song, Johnson Apacible, and Patrick Pantel. 2013. Identifying salient entities in web pages. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2375-2380.
Ojas Ahuja, Jiacheng Xu, Akshay Gupta, Kevin Horecka, and Greg Durrett. 2022. ASPECTNEWS: Aspect-oriented summarization of news documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 6494 6506, Dublin, Ireland. Association for Computational Linguistics.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summariza- tion. In Proceedings of the 2nd Workshop on New Frontiers Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-cont opinion summa- rization. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing. pages 6578-6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News summarization and evaluation the era of gpt- 3.
Junxian He, Wojeiech Kryscinski, Bryan McCann,
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. Thirty-second AAAI Conference on Artificial Inteliignce, AAAL.
Nazneen Fatema Rajani, and Caiming Xiong. 2020. Ctrlsum: Towards generic controllable text summa- rization. CoRR, abs/2012.04281.
Ella Hofmann- Mayank Kulkarni. Lingjue Xie, Mounica Maddela, and Daniel Perootuu--Pittro 2022. Extractive entity-centric summarization as sentence selection using bi-encoders. In Proceedings of the 2nd Conference of the sociation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 326-333, Online only. Association for Computational Linguis- tics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling cnarruction-fintunee language models. arXiv preprint arXiv:2210.11416
Chapter of the As-
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhenghao Jiang. and Graham Neubig. 2021. GSum: A gen- eral framework for guided neural abstractive summa- rization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4830-4842, Online. Association for Computational Linguistics.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1328-13333 Austin, Texas. Association for Computational Linguis- tics.
Jesse Dunietz and Daniel Gillick. 2014. A new en- tity salience task with millions of training examples. In Proceedings of the 14th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, volume 2: Short Papers. pages 205-209. Gothenburg, Sweden. Association for Computational Linguistics.
Klaus Krippendorff. 2011. Computing Krippendorff's Alpha-Reliability
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP) pages 540-551, Hong Kong, China. Association for Computational Linguis- tics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Associ- ation for Computational I. Linguistics.
Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Analyzing sentence fusion in abstrac- tive summarization. In Proceedings of the 2nd Work- shop on New Frontiers in Summarization, pages 104- 110, Hong Kong, China. Association for Computa- tional Linguistics.
Alexander R Fabbri, Wojeiech Kryšcińíski, Bryan Mc- Cann, Caiming Xiong, Richard Sacher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke 7ettlemoyer. 2019, Bart: Denoising sequence-to-sequenee pre-training for nat- ural language generation, translation, and comprehen- sion. arXiv preprini arXiv: 1910. 13461.
Conference on Empirical Methods Natural Lan- guage Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Andrew Turpin, Yohannes Tsegay, David Hawking, and Hugh E Williams. 2007. Fast generation of result snip- pets in web search. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127-134.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North American Chapter of the Association for Computational Linguis- tics, pages 150-157.
Ramakrishna Varadarajan and Vagelis Hristidis. 2006. A system for query- uver-sspecific document summarization. In Proceedings of the 15th ACM international con- ference on Information and knowledge management, pages 622-631.
Mounica Maddela, Mayank Kulkarni, and Daniel Preotiuc-Piete Pietro. 2022. EntSUM: A data for entity- centric extractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics VVolume Long Papers), pages 1365,,366, Dublin, Ireland. Association for Compu- tational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulgehre, and Bing Xiang. 2016. Abstrac- tive text summarization using -eeuueeee RNNS and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
A Dataset Annotation Process
In Figure 1, illustrate example showcasing the multiple stages of annotation implemented in the ENTSUM dataset. The example encompasses four distinct stages annotation. In this paper, our experiments focus on two particular stages: salient sentences and entity-centric summary. We use the entity-centric summary to evaluate the model per- formance. During the evaluation process, we com- pare the models' performance when provided with either the entire article or only the salient sentences as input.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguisties.
Ani Nenkova, Kathleen McKeown, et al. 2011, Auto- matic summarization. Foundations and Trends in Information Retrieval, 52--3)110--2) 233.
B Qualitative Comparison of EntSUM and EntSUMv2
Maria Pontiki, Dimitris Galanis. Haris Papageor- giou, Ion Androutsopoulos, Suresh Manandhar, Mo- hammad AL-Smadi. Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clereq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Na- talia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud María Jiménez-Zate Zafra, and Gülşen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment anal- ysis. In Proceedings of the 10th International Work- shop on Semantic Evaluation 'SemEEal-22100 pages 19-30, San Diego, California. Association for Com- putational Linguistics.
In Table 4, we illustrate the qualitative difference between the ENTSUM and ENTSUMv2 datasets. In ENTSUMv2 the abstractive entity-centrie sum- maries are constrained to 60 tokens, resulting more abstractive and specific summaries. Entity- centric summaries in ENTSUMv2 are on average 33% shorter than the cooresponding summary in ENTSUM.
C Prompt comparison for Flan-T5
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 55511
Table 5 compares the performance of the Flan- T5 model when selecting diffcrent prompts for entity-centric summarization. The evaluation of the prompts is conducted on the NYT valida- tion dataset using proxy entity-centric summaries. Prompt and Prompt adopt the summarization prompts employed in Flan-T5 instruction tenniigg,
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015

---

Entity: Federal Emergeney Medical Response.
Annotate entity salience with respect to the documeent
The Homeland Security Depariment and its Federal Emerggnncy Management Agency will continue to be the lead federal player responee efforts, according
proposed Townsend.
Select sentences salient to the entity the document
The Depariment of Hesith, meanokile, would take back from FEMA the disaster medical teams it weed to ripervioe before Departmout Homeland Security was vstabitshed. And Department of Housing and Urban Development expected to find sonporary housing for vietins, duty also handled FEMA, which Mr. Tounsond said placed mach empharis on buying travel trailers and mobile homes rather than on finding spartments or other options.
Sclect sentences the summary among salient sentences
Homeland Security Depuriment Fuderal Emergaacy Managenent Agency continue player in disaster responu efforts, acconding the bluqprint progosed Traasood.
fonkrut
The Department of Health, meanwkile, would take back from PEMA disaster medical toams if used to rupervise before Departmnet Homeland Security estabitrhed. And Department Housing Urban Development expected to find temporary housing for victims, duyy also now hanalled FEMA, which Mr. Townsend said placed too much empharis on buying travel truilers and mobile homes rother than on finding apartments or other options.
Write entity-centric summary from the selected summary sentences.
The Department of Heals would take back the disazter medical soans from FEMA The Departmuu of Houzing and Urhen Development find tomporary housing for victima, duty handled FEMA.
Figure 1: Annotation Pipeline as described in Maddela et al. (2022)
Average Score
Crowd-Source Annotations
400
350
1
350
300
175
250
300
250
150
125
200
200
150
150
150
75
100
100
100
50
50
50
25
Entity-Specificity
Factuality
Quality
Completeness
Fluency
Average Score Author Annotations
60
25
25
30
40
20
20
30
20
30
15
20
10
10
10
10
5-
10
2
Specii-Seeciiicity
Factuality
Quality
Completeness
Fluency
Figure 2: Histogram of average scores for trained annotator and author annotations, respectively

---

Entity Bush
EntSUM
EntSUMv2
Focusing his prioiiies. President Bush invited ordinary people like teacher, physicist, an Afghan politician, the family of a fallen soldier to the State of the Union address. But Democratic congresswoman turned the tables on Mr. Bush by inviting guest of her own Cindy Sheehan, the antiwar protester who has determined Mr. Bush from his Texas ranch the White House. When he entered the House chamber, his latest political trophy Samuel A. Alito Jr., newly confirmed and sworn in justice of the Supreme Court was full display, a powerful reminder Capitol Hill.
President Bush invited ordinary people to the State of the Union address. But a Democratic congresswoman tumned the tables on Mr. Bush by inviting guest of her own Cindy Shechan, the anti- protester who has determined Mr. Bush from his Texas ranch to the White House.
that Mr. Bush can still flex his muscles
Jennifer Baker
The audience for the first day of the retrial, according to several people in the courtroom that day, included Mr. Giuca's 's mother, Doreen Giuliano, and Jennifer Baker, a young woman who had handed out pamphlets at Mr. Giuca's sentencing, She identified herself as a reporter for a weekly campus newspaper at Brooklyn College and said she was writing ahout the case. Annamaria Scaccia, editor-in-chicf of The Kingsman, and Lauren Darson, managing editor of The Excelsior, said they did not emplay anyone named Jennifer Baker and did not have reporters assigned to the courts or the district attorney's 's office.
Jennifer Baker, a young woman who had handed out pamphlets at Mr. Giuca's sentencing, said she was a student re- porter from Brooklyn College. Anna- maria Scaccia, editor-in-chief of The Kingsman, and Lauren Darson, manag- ing editor of The Excelsior, said they did not employ anyone named Jennifer Baker.
Glen A. Rosen- baum
partner the powerful law firm of Vincent & Elkins and spokesman for 18 top Texas law firms that have complained inequities in the new taxing formula. While maintaining that they were willing to taxed for the first time, Mr. Rosenbaum said, one way making the plan fairer would be to raise the deduction per lawyer to at least $500.000 from the proposed $300,000.
Glen A. Rosenbaum
Glen A. Rosenbaum is partner the powerful law firm Vincent & Elkins and spokesman for 18 top Texas law firms that have complained inequities in the new taxing formula.
Byun Ia
Byun Ha Jung, a senior manager at the Hyundai Asan Corporation, the South Korean company and unit of the Hyundai Corporation that is developing the park, said that for South Korean companies. the reality is that one doesn't 't have to go to China. He asked reporters how much they have invested in China and whether it is one billion dollars.
Byun Ha Jung, senior manager at the Hyundai Asan Corporation said that for South Korean companies, the reality is that one doesn't 't have to go to China. He asked reporters how much they have in- vested in China and whether it is one billion dollars.
Jung
Table 4: Comparison of EmSUM and EntSUMv2
accompanied by additional entity-related informa- tion. Prompt 3 introduces explicit word con-
still pertains to the entity.
If all of the content pertains to the entity, but is not factually correct according the source text, please score this metric 4 (All content is about the entity but the sentences may not be salient) Scale anchors:
straint. Prompt 4 adopts guestion-answering style prompt, utilizing the 5WIH framework. The re- sults show that the design of the prompts has a significant impact on the performance of the model. Prompts that closely resemble the task-specific prompts used during model training yield more accurate and relevant summaries. Prompt 2 is the selected prompt for the following evaluations.
1. None of the content is about the entity
2. Most of the content is not about the entity
3. Some but not all of the content is about the entity
4. All content is about the entity but the sentences
may not be salient
D Extended Human Evaluation Results
5. All content is about the entity and is salient
The results of the authors human evaluation results can be found in Table 6. The histograms of the trained annotators and author Likert scores for each facet are included in Figure 2.
Fluency: this metric measures whether the sum- mary is grammatically correct and easy under- stand. Please do not penalize the score if the sum- mary about the entity is incomplete (i.e., should include more details from the source text). The completeness metric measures this instead. Scale anchors:
E Human Evaluation Guidelines
for this metric we are determin- ing what extent the content pertains to the entity
1. The summary is incomprehensible
and is salient (relevant) in a summary about the
2. Disfluent
entity. Please note the following:
3. Understandable
Please do not penalize the score for the entity name not being mentioned so long as the content
4. Good
5. Flawless

---

Prompt
ROUGE-1
ROUGE-2
ROUGE-L
BERTScore
Summarize the Tollowing article locusing
(entity]: [text]
34.2
23.3
30.6
61.7
Write a short summary about (entity) ] based on
the following article: (text)
41.6
32.5
38.6
66.3
Generate summary under 60 words that describes [entity], based on the following article: (text]
38.3
27.7
34,6
63.8
[text) 'n Based the article,
the
following question: (eatity) did what to whom, when, where and how?
13.7
10.2
13.3
47.7
Table 5: Comparison Flan- T5 model performance on entity-centric summarization with different prompts.
Model
Factuality
Completeness
Fluency
Quality
GSum steepppoootteee
4.75g.62
4.660.66
3.710.83
4.13g.56
3.13/0.73
T5-v1.1 F...l,1-basezuo-ee -basecuo-stes
4. 4.09
4.14
3.071.19
4.34p.c4
2.691.05
Flan-T5-base,2 entilty
4.481.17
4.021.aa
3.041.08
4.62a.7
2.71.01
Table Authors human evaluation results (average scorextdev) of three types of summarization models on a subset of the ENTSUMv2 dataset. typeface denotes the best performance.
Factuality: this metric measures whether the sum- mary is true to the source text. Please penalize the score if the summary introduces new facts that were not present in the source text.
Overall Quality: this measures, from a reader's point of view, whether a reader would be able to gain overview of the essential information from the original source text that pertains to the entity if they did not have access to the original source and the entity name Scale anchors:
Scale anchors:
1. Very untrue of the source text
Mostly untrue of the source text Somewhat true of the source text
1. Poor
Mostly true of the source text
2. Fair
Very true of the source text
3. Good
Completeness: this metric measures whether the summary includes a comprehensive overview of the source text that pertains to the entity.
4. Very good
5. Excellent
Scale anchors:
1. Does not capture entity-specific overall im-
portant information
Captures overall important information, but does not capture entity-specific information Captures some entity-specific information
4. Mostly captures the -ppecy-ppecific information
5. Completely captures the entity-speciff informa- tion


Reviews:
[{"review_id": "RqAn5rmdlP", "reviewer": "Reviewer_d8Nz", "summary": "", "questions": "Are extractive summaries of the document provided to annotators while writing the abstractive summaries?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and feedback. Please see our responses to your questions below:\n1. Our annotation process followed the same steps and guidelines described in the original EntSUM paper and is additionally detailed in Appendix A of our paper. The summaries in the EntSUM v2 dataset were annotated by a single annotator on a proprietary platform (details on the platform would break anonymity but we will provide more information in the final version). To ensure quality of the annotations, we conducted three rounds of annotator training with data not included in the evaluation dataset and provided the most salient sentences about the entity along with the source document to the annotators. We have additionally calculated the inter-annotator agreement for the EntSUMv2 dataset in the final training round using ROUGE-[1,2,L] and BERTScore between the abstractive summary and the proxy summary (entity salient sentences) provided to annotators at annotation time. The EntSUMv2 Krippendorff's alpha for ROUGE-[1,2,L] and BERTscore are 0.75, 0.84, 0.85 and 0.81 respectively, which we will include in the final paper. \n\n2. We will attempt to release the code. Meanwhile we have provided specifics on the implementation details in Appendix E, and we will provide further model-specific details in addition to references to the open source libraries used in our experiments.\n\n3. Thank you for the suggestion. The paper currently includes some dataset level statistics, such as average summary length and novel n-grams between for the two versions of the dataset in Table 1. We will do as suggested and augment the quantitative analysis to include additional comparison statistics and also add qualitative analysis with examples to the final paper. \n\nRegarding your question on whether extractive summaries are provided to annotators, we provided annotators with a reference of the sentences that were salient to the entity at the time of annotation, which serves as a proxy extractive summary. Please see Figure 1 in Appendix A for more details on the same. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper creates a new dataset for entity-centric summarization which will be useful to the NLP community. Currently, there are no good abstractive entity-centric summarization datasets.\n\n2) The paper proposed a new multi-faceted human evaluation setup for entity-centric summarization.\n\n3) The paper is well-written and easy to understand. The experiment setup is sound and the experiments are described thoroughly.\n\n4) The results provide good insights into the existing models.", "reasons_to_reject": "1) Few annotations details are not clear. Is the dataset single-annotated or is more than one summary collected for each entity-document pair? Which platform was used to collect the data? Was any other quality control done other than training the annotators? \n\n2) Will the authors make their code available? Otherwise, it will be difficult to replicate the results. \n\n3) It is unclear from the novel ngram how much the abstractive summaries deviate from the extractive summaries. It would help to do more qualitative analysis and add examples from ENTSUM and ENTSUMv2 to understand the difference between the two datasets.", "questions_to_authors": "Are extractive summaries of the document provided to annotators while writing the abstractive summaries?", "paper_topic_and_main_contributions": "The paper proposes EntSUM-v2, a dataset for entity-centric text summarization. It revises the existing ENTSUM dataset (with extractive summaries) to include abstractive summaries. The paper finetunes different summarization systems for the task and benchmarks them on the dataset. Finally, the paper conducts the first multi-faceted human evaluation on entity-centric summarization, which reveals detailed insights into model behavior.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yS4A6KNvEB", "reviewer": "Reviewer_gpN3", "summary": "", "questions": "Please answer the questions above.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and feedback. Please see our responses to your questions below:\n1.  Our annotation process followed the same steps and guidelines described in the original EntSUM paper and is additionally detailed in Appendix A of our paper. The summaries in the EntSUM v2 dataset were annotated by a single annotator on a proprietary platform (details on the platform would break anonymity but we will provide more information in the final version). To ensure quality of the annotations, we conducted three rounds of annotator training with data not included in the evaluation dataset and provided the most salient sentences about the entity along with the source document to the annotators. \n\n2. The paper currently includes dataset-level statistics like average summary length between the two versions of the dataset and number of novel n-grams in the summary in Table 1, which provide insights as to how much EntSUMv2 differs from EntSUM. We can incorporate more comparison statistics across the two versions of the datasets, such as n-gram overlap, and also add qualitative analysis with examples in the final paper. We have additionally calculated the inter-annotator agreement for the EntSUMv2 dataset in the final training round using ROUGE-[1,2,L] and BERTScore between the abstractive summary and the proxy summary (entity salient sentences) provided to annotators at annotation time. The EntSUMv2 Krippendorff's alpha for ROUGE-[1,2,L] and BERTscore are 0.75, 0.84, 0.85 and 0.81 respectively, which we will include in the final paper.\n\n3. Please find the explanation of our choice of models to evaluate in section 5.2. T5-v1.1-base was selected for a fair comparison with GSum, as T5-base is trained with additional summarization datasets whereas T5-v1.1-base was not.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiment setups looks fine. The idea of explicitly shortening the gold-standard summaries to make summaries more abstractive sounds good. Overall the performance improvement is also quite significant.", "reasons_to_reject": "1. It is unclear how the annotators are trained and what was the overall agreement during the data creation stage (cf. line 98). Could you please elaborate on annotation guidelines and what was the overall\n\n2. I am curious as to how much abstraction is still introduced. There is no analysis to show comparision on word-level or entity level b/w the summaries of two dataset. Please discuss more on this part as there needs to be some insight into how much the new data differs from old data.\n\n3. The choice of models for faceted-human evaluation is not clear. Although it is discussed in text it is likely incorrect. As per the table T5-base should have been considere but it was not the case. Can you elaborate on this choice?", "questions_to_authors": "Please answer the questions above.", "paper_topic_and_main_contributions": "In this paper, authors focus on the task of abstract entity-centric summarization. They update a version of a well-known existing ENTSUM data set where summaries are deliberately made shorter and more abstractive.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d00MCzXmwm", "reviewer": "Reviewer_fMbs", "summary": "", "questions": "Please address Point 1 under \"Reasons to reject\".", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and feedback. We appreciate your suggestion and will incorporate your feedback to include standard deviation values to Table 2 in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy-to-understand. The key details are well-explained.\n2. The EntSumv2 dataset is an important contribution, and they evaluate a wide range of models using it. \n3. The discussion of automatic evaluation results and faceted human evaluation is quite comprehensive and provides interesting insights.", "reasons_to_reject": "1. The standard deviation values in Table 2 are not provided.", "questions_to_authors": "Please address Point 1 under \"Reasons to reject\".", "paper_topic_and_main_contributions": "The authors contribute a new dataset for the task of entity-centric abstractive summarization. They evaluate the performance of different abstractive summarization models and perform detailed empirical evaluations and a crucial human evaluation study.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper conducts the first multi-faceted human evaluation on entity-centric summarization, which reveals detailed insights into model behavior. And, all reviewers have expressed positive opinions about this paper. Therefore, I suggest this paper accepted as a short paper in the conference.
Decision: Accept-Main