USB: A Unified Summarization Benchmark Across Tasks and Domains
Authors: Kundan Krishna
Prakhar Gupta
Sanjana Ramprasad
Byron C Wallace
Jeffrey P. Bigham
Zachary Chase Lipton
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
large language models
pretraining
benchmarks
factual correctness
TL;DR: We introduce a Wikipedia-derived benchmark, complemented by a rich set of crowd-sourced annotations, that supports 8 summarization-related tasks focusing on factuality and controllability of summaries.

Abstract: While the NLP community has produced numerous summarization benchmarks,
none provide the rich annotations required 
to simultaneously address many important problems related 
to control and reliability.
We introduce a Wikipedia-derived benchmark,
complemented by a rich set of crowd-sourced annotations,
that supports $8$ interrelated tasks:
(i) extractive summarization;
(ii) abstractive summarization;
(iii) topic-based summarization;
(iv) compressing selected sentences into a one-line summary;
(v) surfacing evidence for a summary sentence;
(vi) predicting the factual accuracy of a summary sentence;
(vii) identifying unsubstantiated spans in a summary sentence;
(viii) correcting factual errors in summaries.
We compare various methods on this benchmark and discover
that on multiple tasks, moderately-sized fine-tuned models 
consistently outperform much larger few-shot prompted language models.
For factuality-related tasks, we also evaluate existing heuristics 
to create training data and find that training on them 
results in worse performance than training on $20\times$ less human-labeled data.
Our articles draw from 6 domains,
facilitating cross-domain analysis.
On some tasks, the amount of training data 
matters more than the domain where it comes from,
while for other tasks training specifically on data from the target domain, 
even if limited, is more beneficial.

USB:
A Unified Summarization Benchmark
Across Tasks and Domains
Kundan Krishna Byron C. Wallace 9
Prakhar Gupta
Sanjana Ramprasad
Jeffrey P. Bigham Zachary C. Lipton
Cameeiee Mellon University
o Northeastern University
(kundank, prakharg, Jbigham, zlipton] Sandrew. cnt.reewwcccccom { tramprasad. sa, b. wal lace w @northeastern.edi
Abstract
previous work has primarily focused on evaluating the models' ability to generate summaries similar to reference summaries, neglecting key auxiliary properties of text summarization systems.
While the NLP community has produced nu- merous summarization benchmarks, none pro- vide the rich annotations required simulta- neously address many important problems re- lated to control and reliability, We introduce Wikipedia-deriver benchmark, complemented by rich set crowd-sourced annotations, that supports 8 interrelated tasks: extractive sum- marization; (ii) abstractive summarization: (iii) topic-based summarization; (iv) compressing selected sentences into one-line summary; (v) surfacing evidence for a summary sentence; (vi) predicting the factual accuracy of a sum- mary sentence; (vii) identifying unsubstanti- ated spans in a summary sentence; (viii) cor- recting factual errors in summaries. We pare various methods on this benchmark and discover that on multiple tasks, moderately- sized fine-tuned models consistently outper- form much larger few-shot prompted language models. For factuality- evaluate existing heuristics to create training data and find that training on them results in worse performance than training on 20x less human-labeled data. Our articles draw from 6 domains, facilitating cross-domain analysis. On some tasks, the amount of training data mat- ters more than the domain where it comes from, while for other tasks training specifically on data from the target domain, even if limited, is more beneficial.
Recent research has highlighted the importance of addressing additional aspects in text summa- rization. These aspects include the ability to steer summaries by controlling its focus on topic specific parts of the source text (Gchrmann et al., 2019). Furthermore, there is increasing empha- sis dasuringffctuce correctness and implement- ing mechanisms to eliminate factual errors from model outputs (Scialom et al., 2021; Balachandran et al., 2022). Similarly, to foster trust in the outputs, it is desirable for summarization systems to present evidence from sources that corroborate the gener- ated summaries. As models have improved in gen- erating coherent and readable summaries (Goyal et al., 2022), these auxiliary considerations have gained importance. Aligning summaries with user requirements and ensuring sufficient factual sup- port are critical frontiers in summarization research. The current summarization benchmarks fail to pro- vide comprehensive evaluation of model capabil- ities across various summarization tasks, encom- passing properties such factuality and controlla- bility.
tasks, we also
In this work, we introduce USB, comprehen- sive benchmark for text summarization that sup- ports eight auxiliary tasks. The benchmark includes labeled datasets with high-quality human annota- tions collected from diverse documents across six domains. To create the benchmark, we sampled Wikipedia articles from various categories, such as people, organizations, and events. We utilized the introductory section of the articles summary and the remaining content as the source text, resulting in imperfect source- "arrcee"uumary""" pairs. Human annotators then searched for evidence support each summary sentence. If evidence was lacking, corresponding spans or entire sentences
1 Introduction
Automatic text summarization has been impor- tant, active research sub-area in NLP for over two decades (Radev et al., 2002; Nenkova et al., 2011; El-Kassas al., 2021). Numerous summarization benchmarks have been proposed to facilitate the development of summarization methods (Nallapati et al., 2016. Narayan et al., 2018; Wang and Ling. 2016; Gliwa et al., 2019). However, the majority of
reference
'The dataset can be downloaded from https:// gi github. www.ab..co/////o kukkikkii iirrishaaa shna/ush

---

life.
Basic tasks
Fatrisse Bealiedge was som hibruary Sirketwwal.
A. Document
retter naterdsater and gentieren's exttitter. Ste siccated atresstedd ligh School, and the intoenstty of Liverponl
Abstractive summerization
A16
Filn and television.
Extractive summarization
A+C
Specints
caunly serties "Gopping Apporancon
the portreger rornerzy vorking cinno vaner sostas pretenatoos (Irstating pronouread "touquet") and reduntors of graadear aft-nentioned "candlalight sppers)). feesleage tesightee in portrayōng sysstrts, as clataed couldn's stand people like 1ife-
Summary
Factuality
In 1953, dec use eratsse conddy Rxard For hor partrayal, счттA Auures in 1332 and 3323.
was lateo nostroted
Evidence extraction
A1RxC
Evidence
The seraes srese et trstaedge recasat, in 2035 deapise Lte engekng popularity. Ste uisted to pursue other rolme ctaracter actress.
annotation
Factuality clessification
A-B+C=D
Unsupported span prediction
B1C4D
B
Fixing factuelity
A+BIC-D
Done Kotheriin Potriiclo Rottledge fime born If fedewarry 1929 an tngltsh actress
D. summary
singerr and becoddons for bisttl
For her role at Mrahhnhh Butker in the comendy aidine Keeoting Up Accesrattee
Controllability based tasks
yoon the was romtrated for the Wen tV waveeeeeer
hentormence tong and
Section
Topic based summarization
A+B-C4E
Her +ile
Include Str. with 1967 and
labels
tridgo Lower Mover 1963
Mulli-sentence compression
AIBIC
Figure I: A schematic of our dataset, annotations, and the supported tasks. The example shown (abridged) displays the edits made by a human annotator on the initial candidate summary (deletions in red with strike-throuu through; additions green). Every summary sentence is supported by one or more evidence sentences highlighted in blue.
were removed. Whenever conflicting evidence was encountered, the summary was revised with mini- mal edits align with the available evidence. The resulting annotations can be repurposed to create labeled datasets for useful tasks (Figure 1).
those trained on heuristically generated labeled datasets, even when the latter are 20x larger. A common challenge real-world adoption of models their use in resource-poor domains where one does not have access to abundant la- beled training data. We compare how the size of available training data matters vis-a-vis its do- main for different summarization tasks. We found that for tasks related to factual correctness of sum- maries, the amount of training data matters more than its domain; but for other tasks having domain- specific training data matters more. Our benchmark explicitly segmented into domains based on Wikipedia categories, and hence provides a natural test-bed for such domain transfer studies.
We offer the first human-labeled training datasets for various summarization tasks, including evi- dence extraction and identifying spans in maries without supporting evidence. These datasets enable the training and evaluation of models specif- ically for these crucial aspects. We benchmark the performance of several models such as instruction- tuned encoder-decoder models and LLMs on our tasks, including both fine-tuning and well as few- shot prompting based approaches. Notably, found that fine-tuning even small models (fewer than a billion parameters) substantially outperforms few-shot prompting of much larger open- and private large language models.
Summary of contributions:
Multi-domain benchmark for training and evaluating models on different tasks dealing with some critical but understudied aspects of text summarization.
Prior efforts have relied heuristics generate synthetic training data for certain tasks included in our benchmark. For instance, a common heuristic employed is lexical overlap to identify and extract supporting evidence (Chen and Bansal, 2018). Sim- ilarly, artificial factual errors have been introduced into summaries to train models for factuality classi- fication or correction (Kryšciński et al., 2020; Bal- achandran et al., 2022). Although such automatic approaches enable easy creation of large training datasets, heuristically derived annotations are typi- cally noisy compared to human annotations. Our findings demonstrate that models trained on min- imal amount of human-labeled data outperform
Comprehensive evaluation of models and training strategics, including fine-tuning, few- shot prompting, and multi-task training.
Comparison of relative value of training data labels generated by humans versus hcuristics, showing that for multiple tasks, human anno- tations yield better models even with 20 less training data.
Practical insights about out-of-domain gener- alization for different tasks, identifying the

---

tasks for which the size of the training data matters more than it being from a specific tar- get domain.
overlapping parts. To derive a good document- summary pair from an article, there should ideally be a large amount of overlap between the overview part and remaining article. Otherwise, after human annotation (to remove parts of the summary unsup- ported by the corresponding document) one would be left with little text in the summary.
2 Dataset Curation
To create the USB benchmark, we first collected set of manual annotations Wikipedia articles. We then used the collected annotations to create la- beled data for the benchmark tasks. In this section we describe the process of collecting these manual annotations. We consider the text in a Wikipedia article overview (leading) section as the target sum- mary S, and the rest of the article as D. In well- written articles, the overview section (5) provides broad summary of the article, and the rest of the article (D) provides specifics. Hence, the content in S which highlights parts of D can be effectively considered its summary. However, for S to be a valid summary of D, we need to remove contents within that mention new information that is not present in D and cannot be inferred from it.
Given article, with the overview section represented by S and the remaining part repre- sented by D, we broke the summary into sentences 81 8283...8p using Spacy². We calculated how many of the summary sentences have at least one entity which is also present in D. For this step, we auto- matically marked entities in S and D by consider- ing all the words with internal hyperlinks to other Wikipedia pages entities. If two hyperlinked words pointed to the same page, they were con- sidered the same entity. For annotation, we only retained articles that have more than 75% of sen- tences with at least one entity overlapping with D. We also controlled for summary length by discarding any article where S has fewer than 4 or more than 12 sentences.
We recruited annotators and asked them to ecute the following tasks: (1) Find and annotate evidence in D for each summary sentence of S, and; (2) Delete parts of S that are not supported by D. This yields a document-summary pair where the summary is fully supported by the document, and the supporting evidence is explicitly marked. We provide a detailed description of our data cre- ation process below.
Flagging entity overlaps help annotators find evidence To help annotators find evidence sup- porting any given summary sentence, our interface highlights entities present in that sentence and also in the source document, with a different color for each entity. To maximize the number of entities de- tected, we took union of entities extracted using Wikipedia's hyperlinks, Spacy and DBpedia.
Retrieval of Wikipedia articles We down- loaded the Wikipedia English articles dump from July 2022. We extracted the articles from this corpus using the Wikiextractor tool. We dropped tables and lists during extraction, but retained sec- tion headers. We used a set of category filters to retrieve pages about specific types of entities which helps us in creating a dataset with diverse domains. We manually filtered domains to select those in which articles generally had substantial part of S supported by evidence present in D. We retrieved articles for the following 6 domains: biographies, companies, schools, newspapers, landmarks, and disasters.
Selection and monitoring of Mechanical Turk Workers We ran a qualification task on Me- chanical Turk, tasking workers with annotating one document-summary pair according to the provided instructions. To take this qualifier, workers have a HIT approval rate 95%, and have more than 1000 approved HITS. Each worker was allowed to take the qualification task only once. All workers were given the same document-summary pair for annotation. total of 174 workers took the qualification task. Out of these, 28 workers were approved by checking their annotation quality man- ually. The approved workers were then permitted to work on the main task where they were asked to annotate different document-suummrry pairs. Each pair was annotated by exactly one worker. After 300 annotations for the main task, we analyzed
required
Selecting documents for annotation Our heuristic is to assume that the overview section of a Wikipedia article will feature a significant amount of overlap with the remaining part which would be retained after the annotators remove non-
attps::/ sattps://sgacy..1
Pnt / httthub con wttpttttttttttttttttttttetteette ractor
A Anttps wwwp://wwo....... spot Lorgt.org

---

the annotation quality of the responses again. For many approved workers, the annotation quality on the main task was significantly worse than the qual- ification task, and hence we restricted the worker set to only 3 workers whose annotation quality was much better than the rest (hereafter referred to primary workers). The remaining annotations were done by these workers, and a total of 1988 document-summary pairs were annotated.
that it contains. We define the ideal "reference" ex- tractive summary the set of all source sentences marked as evidence for the summary.
A bstractivv Summarization (ABS): Generate a mulli-sentenee summary of the source document by not just simply selecting important content, but also rephrasing it for succinctness and coherence. The ground truth is the full-length summary created after the annotators' edits.
Verify ing annotations Due to the complexity of the annotation task, evidence has not been anno- tated in some parts in the summaries after the first round. To address this, trained model to pre- dict unsupported spans in summaries. Specifically, we trained models that accept an initial summary sentence s and the evidence annotated by the work- ers as the input, and then predict which spans in s were deleted by the annotator to in their submit- ted version s'. We applied this model to the sum- mary sentences submitted by annotators to predict unsupported spans in them. We fine-tuned Flan- T5 XL. (Chung et al., 2022) for this task. We di- pairs annotated
Factuality Classification (FAC): Predict if a sum- mary sentence is factually correct and sufficiently supported by the information present in the source. We create labeled data by assigning non-faciual and factual labels the before and after versions of each edited summary sentence, with the marked evidence as source context fed in the input.
Fixing Factuality (FIX): Given factually incor- rect summary sentence, edit make it factually correct, with reference to the source text. We cre- ate annotations using pre-edited summary sentence and the marked evidence as the input, and the post- edited sentence as the target.
vided the set of dooument-summann by our primary workers into two halves, trained a model on each half, and used it to predict the unsup- ported spans in the other half. We used one of these models for prediction on the remaining document- summary pairs submitted by other workers. Us- ing these model predictions, we selected around 20% of the total summary sentences most likely contain unsupported spans, and flagged them for verification. This included about 15% of the sentences annotated by primary workers and 15% of sentences annotated by other workers, which aligns with our manual inspection of quality of the workers' annotations. We then designed a slightly modified interface for the verification task, where summary sentences have highlights showing po- tentially unsupported content, and the workers can select additional evidence or edit the summary as before. After incorporating the changes made in this verification round, we arrived at the final ver- sion of the annotated corpus.
Topic-based Summarization (TOPIC): Given the source article and a topic, the task is to generate a summary for given topic from souree article. We use Wikipedia section headers as topics and select summary sentences from our labeled dataset that have evidence from single section only. These sentences act as target summaries, while the full document and section header serve as input.
Multi-sentence Compression (COMP): Given cluster of sentences from the source document, gen- erate a single sentence summary that incorporates information from all of them. We create labeled data for this by using each summary sentence as a target and its marked evidence as the input.
Evidence Extraction (EVEXT):
Given a source document and a summary sentence, identify a min- imal set of source sentences which collectively pro- vide supporting evidence for all facts present in that summary sentence. The labeled data consists of each summary sentence and the full source doc- ument as input, and the evidence links marked by annotators as the ground truth.
3 Task Definitions
We derived labeled datasets for tasks using the col- lected annotations. The resulting benchmark con- sists of the following tasks:
Unsupported Span Prediction (UNSUP): Given summary sentence and a set of sentences from the source providing evidence, predict spans in the summary which are not supported by the evidence. To create labeled data, we select those summary
Extractive Summarization (EXT): Given the full document as input, extract all important sentences

---

Domain
Count
Domain
Count
Buchholz, 2000). For the remaining tasks we use standard binary classification metries.
Biographies 1514
Companies
97
Schools
150
Landmarks
50
For the classificatiion/sann prediction tasks in our benchmark, we fine-tune Roberta-Large (Liu et al. 2019; Table 2). We recast these as seq2seq tasks and fine-tune variants of T5 models on each of the 8 tasks. We include the original (Raffel et al., 2020) and the instruction-tunet Flan version (Chung et al., 2022). T5 Large outperforms Roberta-Large on all the classification/span prediction tasks. Flan- T5 Large performs similarly to T5 Large, though achieves notable gains on Unsupported Span Pre- diction. Flan-T5 XI. consistently improves perfor- mance over larger models on almost all tasks, sug- gesting model size helps (Table 2). We also train multi-task variant of Flan-T5-XL (on all tasks jointly). This mostly retains similar performance as a dedicated XL model trained only on that task, except for Evidence Extraction and Unsupported Span Prediction (Table 2).
Disasters
145
Newspapers
32
Table I: Number of annotated documents in various domain splits of our benchmark. Total 1988 documents across all domains.
sentences where annotators only made deletions (no additions or replacements). The imput is the pre- edit summary sentence and the marked evidence. and the gold target the set of spans that were deleted from the summary by annotators.
4 Dataset Overview and Statistics
The USB is a benchmark comprising 6 different domains with a varying number of instances in each (Table 1). We use 40:20:40 split for train, validation and test set size for each domain, cept the landmarks and newspapers domains due to small size. Articles from these two domains are kept as challenging test sets to measure the out- of-domain generalization. Length distributions of source documents and their summaries are shown in Figure 3 in the Appendix. Both exhibit long-tail distributions with lengthy sequences - about 32% of source documents have more than 2000 words and 10% of summaries have more than 200 words. We also find that 27% of summary sentences cor- respond to 1 or more marked evidence sentences (Figure 3 in the Appendix). This suggests a high de- gree of abstractiveness, because information needs to be combined from many source sentences and expressed in a single sentence. Annotators deleted about 22% of the words average from the initial summary presented to them, while adding about 2% new words.
We run large language models including publicly released models (for research purposes) such Llama (Touvron al., 2023) and Vicuna (Chiang et al., 2023), and closed models such as OpenAl's gpt-3.5-turbe i.c., CharGPT. For tasks where the full document is fed as input. we use 4 examples for few-shot prompting owing to limitations in the maximum feasible sequence length for these mod- els, while for the rest we use 16 examples (for details, see the Appendix). ChatGPT consistently outperformed Vicuna-13B and Llama-13B on all tasks except Fixing Factuality. This is because for the Fixing Factuality task, ChatGPT almost always adds new unnecessary information to the summary, even after prompting it to not do that. Compared to ChatGPT, finetuned models perform better on ev- ery task. The performance difference is largest for factuality based tasks such Unsupported Span Prediction, Evidence Extraction, and Fixing Factu- ality. ChatGPT does comparatively well on tasks that involve generating summaries.
5 Benchmarking Different Models
We run a suite of models on all tasks in our bench- mark and present the results Table 2. For this set of experiments, we use the consolidated train, vali- dation and test splits, which are a union of the cor- responding splits from all domains. For tasks that involve generation of summaries, we use Rouge score (Lin, 2004) the metric. We show geomet- ric mean of the 1,2, and L variants for succinetness (Table 2). One exception is the Fixing Factuality task for which we use exact match as the metric. For Unsupported Span Prediction, we measure the Fl score based on BIO tagging format (Sang and
Since automatic metrics for measuring summary quality like ROUGE (Lin, 2004) do not necessar- ily mirror human preference (Cohan and Goharian, 2016), we conducted human evaluation of the gen- crated summaries in the COMP, ABS and TOPIC tasks. We collect ratings for summaries generated by Flan- T5 and ChatGPT for 50 randomly selected documents from the test set, using questionnaire (see the Appendix for more details), We found that
'ee used the frozen version codenamed gpt-3.5-mm 3.5.tarbo-0201

---

Model
COMP
EVEXT
EXT
FAC
FIX
ABS
TOPIC
UNSUP
Metric->
Rouge
FI
AUC
AUC
ExactMatch
Rouge
Rouge
F1
Fine-tuned models
RoBERTa-Large
71.01
84.06
92.69
49.21
T5- Ta-Large
41.97
77.22
87.00
94.89
31.26
33.44
23.81
51.71
Flan-TS-Large
43.23
77.71
87.99
95.15
32.94
32.05
23.62
58.57
Flan-TS-XL
44.87
79.23
87.81
95.30
35.10
32.69
24.26
64.94
Flan-T5-XL (multitask)
44.32
76.64
86.44
95.38
36.71
31.83
23.46
58.51
Few-shot prompted L.LMS
Llama- 13B
28.12
5.56
52.90
49.34
8.20
5.51
2.47
0.63
Vicuna-13B
31.35
6.65
52.76
55.28
4.28
5.56
2.84
1.47
GPT-33.5-uut
33.21
26.78
61.63
60.81
3.29
29.77
14.59
7.80
Table Performance of models on different tasks evaluated on the full test dataset. Tasks: COMP: Multi-sentence Compression EVEXT: Evidence Extraction FAC: Factuality Classification FIX: Fixing Factuality ABS: Abstractive Summarization (of full document) EXT: Extractive Summarization TOPIC: Topic-based Summarization UNSUP: Unsupported Span Prediction
on average, ChatGPT's summaries are mostly pre- ferred over Flan-T5-XL model's summaries for all 3 summary generation tasks in terms of relevance and factuality (Table 3). This suggests that while fine-tuned models produce summaries closer to the ground truth in the dataset (thus achieving high ROUGE), humans may find the summaries of few- shot prompted LLMs better. For example, in the Topic-based summarization task, while Flan-T5- XL produces summaries with an average length of 46.3 words, ChatGPT generates summaries with an average length of 110.2 words. The ground truth summaries for that task are 36.9 words long average, which is more closely matched by Flan- T5-XL, but the much longer summaries of Chat- GPT are considered better by human annotators as reflected the human ratings (Table 3).
what they were trained on. Our benchmark has training data from domains (i.e. excluding land- marks and newspapers), with different amounts of labeled data for each. To control for training set size, randomly subsample annotated documents for each domain isolate 40, 19, and 38 docu- ments for train, validation and test splits. These sizes of the splits were chosen to match the smallest of the 4 domains i.e. companies (Table I).
We train and evaluate Flan-T5 Large models on different domains and plot average scores for all tasks training and test domain pair in Figure 2. Models trained on the same domain the test do- main perform best or negligibly close to it. But across test domains, the best out-of-domain trained model has < 15% performance drop compared this, showing respectable average out-of-domain performance. Going by the in-domain performance of models trained equal amounts of data, the biographies domain is the easiest and the disasters domain is the most difficult. One distinction be- tween the disasters domain and others which might explain difficulty is that deals with summariz- ing an event rather than an entity.
For the Fixing Factuality (FIX) task, we compare fixed summaries generated by Flan-T5-XL. and ChatGPT, asking which of them (i) remove more factual errors; (ii) mistakenly remove more cor- rect information; (ii)) add more new information; the initially provided incorrect summary. We found that while ChatGPT removes more factual errors from summaries than Flan-T5, it often does so by removing lots of (even factual) information altogether, and replacing it with new content to ef- fectively make new different summary (Table 3).
For each task in our benchmark, we investigate whether having access to large training dataset (irrespective of domain) is more important than having training data from the test domain. We use the test splits of domains (companies, disasters. and schools), and on each of them we evaluate 3 different models trained on: (I) The training split of the same domain; (2) The training split of the
6 Out-Of-Domain Performance on Tasks
We next evaluate the performance of fine-tuned models when tested on a domain different from

---

Abstractive Summarization (ABS)
Question
Flan-T5
GPT-3,5-turbo
Which of the following summarics is better in terms of effectively summarizing the given
36.4%
39.7%
full content?
Which of the following summaries is more factual, accurately representing the information
33.8%
33.1%
presented the given full content?
Multi-sentence CompresionnCOMMP)
Question
Flan- T5
GPT-3.5-turbo
Which of two summaries covers more information touching upon highlighted
27.6%
50.0%
sentences?
Which the following summaries more factual, accurately representing the information presented in the document?
21.1%
38.8%
Topic-based Summarization(TOPICC)
Question
Flan-T5
GPT-3,5-turbo
Which of the two summaries is better in terms of effectively summarizing the given topic?
10.0%
85.3%
Which of the two summaries is more related to and exclusive to the given topic?
11.3%
77.3%
Fixing Factuality(FIX)
Question
Flan-Tr T5
GPT-3.5-turbo
Which the two summarics removes more contradiccorr/uussppooodd information from the incorrect summary, in reference to the context?
18.0%
38.0%
Which of the two summaries removes more correct information (which is actually well- supported by the context) from the incorrect summary?
3.0%
24.0%
Which of the summaries adds
facts compared the incorrect summary?
2.0%
67.0%
Table Win rate model outputs along different aspects as indicated in human evaluation for different tasks
Training Domain
CОMP
EVEXT
EXT
FAC
FIX
ABS
TOPIC
UNSUP
Metric+
Rouge
AUC
AUC
ExactMatch
Rouge
Rouge
F1
Companies
Companies
30.02
61.61
66.36
90.10
11.76
19.30
18.51
7.41
Biographies
-1.83
+2.07
+2.22
-2.80
-5.88
-3.19
-3.62
-7.41
Biographies (full)
+0.67
+6.42
+16.57
+3.84
+20.59
+0.40
-2.92
446.85
Disasters
Disasters
31.69
52.89
77.89
77.67
3.03
21.68
16.95
5.80
Biographies
-2.75
+7.15
-9.84
+6.91
-1.01
-5.31
-1.54
-5.80
Biographies (full)
-2.09
+12,52
+6,36
+12,55
+15.15
+0.45
+0.78
+40.22
Schools
Schools
38.63
62.72
73.92
88.89
3.19
28.89
25.04
2.70
Biographies
-2.09
-0.24
-0.72
-1.84
+2.13
-8.45
-5,67
+0.12
Biographies (full)
+0.69
+5.20
+10.60
+3.44
+26.60
-4.88
-4.51
+37.98
Table 4: Of-o-mai-mm evaluation of fine-tuned Flan-T5-Large models. In each section of the table, evaluate 3 variants A) Model trained on the test domain (Companies, Disasters & Schools), B) Model trained on the Biographies domain (training sets validation-19 19, test-38), and C) Model variant trained the full biographies dataset with 607 datapoints for training. Factuality related tasks benefit greatly from an abundance of training data, even if it's not from the target domain.
and are subsampled have equal number of datapoints: train-40,
biographies domain, and; (3) The full training split of the biographies domain (before subsampling) which contains 607 annotated documents. Training equivalent amounts of data from the test domain
and biographics domain leads to comparable or worse performance of the latter (Table 4).
However, training on the full train set of the biographies domain achieves much higher perfor-

---

biogr biographies
companies
Evidence Extraction
disasters
schools
F1
SuperPAL (Emst al., 2021) ROUGE (Chen and Bansal, 2018)
53.8
44
40.9
blographies
Entity overlap
47.0
Train Domaln
Human annotations 100% (N=765)
77.7
companies
39.22
38.13
35.36
40.71
Human annotations 5%
70.9
40
Factuality Classification
disasters
34.36
35.95
AUC
schools
38.29
36.36
35.22
40.50
36
FactEdit (Balachandran et al., 2022)
74.6
FactCC (Kryšcinsski al., 2020)
68.9
Test Domain
Human annotations 100%
95.1
Human annotation 5%
90.4
Figure 2: Average cross-domain model performance (us- ing Flan-T5- on benchmark tasks. All domains are subsampled to equal number of annotated docu- ments (train-40, validation- test-38).
Fix factuality
Exact Match
FactFddt (Balachandran et al., 2022)
1.0
FactCC (Kryšcińsski al., 2020)
0.8
mance on many tasks, despite the domain shift (Ta- ble 4). Gains most visible the Unsupported Span Prediction and Fixing Factuality tasks. By contrast, for tasks requiring summary generation, using the large biographies training set often does worse than using 15x smaller in-domain train set. This might happen because domain-specine knowledge is required to learn the style of sum- maries to generate for a given domain. On the other hand, factuality related tasks tend to be more objective (e.g., judging factual correctness), and so model skills are transferrable across domains.
Human annotations 100%
32.9
Human annotations 5%
11.2
Table 5: Comparing use of human annotations vs heuris- tie annotations for finetuning Flan-T5 Large models. We also report performance when finetuning on 5% of training set with human annotations.
ROUGE-L to derive references. Finally, we use SuperPAL. (Ernst et al., 2021) as an out-of-the-box solution to predict evidence labels for our dataset's summaries, and then use them for model training.
To train models to detect or fix factual errors, we artificially introduce errors into summaries be used exemplars. We do this via transformations such as swapping entities, numbers, pronouns, in- troducing negation, and so on, inspired by prior work (Kryšciński et al., 2020). To generate diverse errors and hallucinations, we follow Balachandran et al. (2022); we mask parts of the summary out and then use a language model to infill these with (mostly unsupported) information.
7 Comparison with Heuristic Annotations
For some lasks in our benchmark, past works have used heuristics to create large labeled training data sets as an alternative to collecting manual anno- tations (Chen and Bansal, 2018; Kryšcinski et al., 2020; Balachandran et al., 2022). We use such proposed heuristics to train models and compare them with models trained on high-quality, human annotated data. We conduct experiments the Evidence Extraction, Factuality Classification and Fixing Factuality tasks. Because the primary ad- vantage heuristic-generateed training sets their size, we also assess how smaller training sets fare in comparison.
We train models for 3 tasks on both heuristically- generated and manually annotated training datasets. and evaluate them on clean human-labeled test sets. Training on human-annotated data performs better than all souristic-based alternatives across all tasks (Table 5). Next, we train models on subsets of the manually annotated datasets with varying sizes and compare their performance the test sets; this shows how even little human-annotated data can outperform large amounts of heuristic-gent -eneerttee data for different tasks. For each the three tasks, the performance achieved using only 5% of the hu-
For the Evidence Extraction task, we use lexical overlap proxy to derive "reference" evidence alignments. For example, we select the source sentence with the highest ROUGE-L score with summary sentence as evidence, as outlined in Chen and Bansal (2018). We also create a training set variant where entity overlap is used instead of

---

man annotated training set, still outperforms the heuristically labeled full training set (Table 5). This highlights the value in collecting manual annota- tions, even if in small quantities, over using heuris- tics to generate training data labels.
sion we introduce a much larger manually labeled dataset than prior works (Slobodkin al., 2022). Prior research has mostly approached topic based summarization by adopting a predefined set of top- ics (Krishna and Srinivasan, 2018; Akhtar et al., 2017; Hayashi et al., 2021). However, we did not restrict the set of topics in our dataset, resulting in long tail of (potentially challenging) rare topics.
8 Related Work
The tasks in our benchmark have been studied in prior work to varying degrees. The greatest amount of attention has gone the tasks of extractive sum- marization (Wong et al., 2008; Kágebäck al., 2014; Zhang et al., 2016; Nallapati et al., 2017). and abstractive summarization (Liu and Lapata, 2019; Zhang et al., 2020; Raffel et al., 2020; Lewis et al., 2020; Goyal et al., 2022). There exist plenty of datasets for abstractive summarization (Narayan et al., 2018; See et al., 2017; Kim et al., 2019; Wang and Ling, 2016). However, many of them were created heuristically, with "targets" being automat- ically extracted via rules from documents pulled from the web. This can lead to poor quality ref- erence summaries (Bommasani and Cardie, 2020; Krishna et al.. 2022), and training on them can yield models prone to generating hallucinations (Nan et al., 2021; Ji et al., 2022). By contrast, we manual annotation ensure that summaries are fully supported by sources, resulting in a high quality abstractive summarization dataset.
Conclusion
We introduced the USB benchmark comprising tasks to measure model performance across dif- ferent text summarization sub-tasks. We showed that fine-tuned smaller models outperform few-shot prompting of much larger LLMs by large margin tasks related to appraising the factuality of sum- maries. We studied how fine-tuned summarization models perform on out-of-domain data, and identi- fied several tasks where the training dataset size is more important than its domain.
Finally, we showed that rather than training mod- els on large volumes of heuristically labeled data, one can get better performance by creating a much smaller (+= 20x smaller) manually labeled training instead. The resultant USB benchmark permits the training of models for useful tasks such as ex- tracting evidence for a summary, correcting factual errors in it, and generating summaries focused on specific topics. Our hope is that this benchmark spurs further research on these tasks and will serve as barometer for progress in them.
Past works for predicting factual correctness of summaries incorporate question- answering models and natural language inference methods (Scialom et al., 2021; Fabbri et al., 2022; Goyal and Durrett, 2021). or use synthetically introduced factual er- rors (Kryšciński et al., 2020) to train models. In contrast, the USB benchmark introduces a high- quality manually annotated dataset for predicting factual correctness. For the task of editing sum- maries to fix factual errors, datasets with both syn- thetic and model-generated errors have been cre- ated (Balachandran al., 2022; Liu et al., 2022). The task of unsupported span prediction is akin detecting hallucinated content in generated sum- maries, and to the best of our knowledge, no labeled dataset exists for this task.
guestion-answering
10 Acknowledgements
We gratefully acknowledge the National Science Foundation (RI 2211954, RI 2211955, and FAI 2040929), UPMC, Highmark Health, Abridge, Ford Research, Mozilla, the PwC Center, Ama- zon AI, JP Morgan Chase, the Block Center, the Center for Machine Learning and Health, and the CMU Software Engineering Institute (SEI) via De- partment of Defense contract FA8702-15-D-0002
11 Limitations
Despite efforts to collect a diverse dataset, the benchmark used in this study may still exhibit cer- tain biases. The sampling process and the selection of Wikipedia articles as the primary data source could introduce inherent biases, potentially affect- ing the generalizability of the results. These biases may stem from the specific domains or topics cov- ered in the dataset, as well as the way in which
For extracting evidence for a summary, past works have used lexical overlap based heuris- ties (Chen and Bansal, 2018: Lebanoff et al.. 2019). A manually annotated dataset for the task was in- troduced by Ernst et al. (2021), albeit our work provides a substantially larger manually annotated dataset. Similarly, for multe-sentence compres-

---

Wikipedia articles are written and formatted. The dataset's reliance on Wikipedia articles as the pri- mary source of data might not adequately represent the nuances and challenges encountered in differ- ent domains or sources. One prominent example is conversations which are frequently used in sum- marization research but are not represented in the benchmark. Similarly, model's ability to detect er- rors/hallucinations in summaries in the benchmark may not necessarily reflect its ability to detect er- rors more broadly in summaries generated by a variety of models.
the summary which were flagged for verification. which can be as low as I sentence. The creation of the entire dataset costed about 6000 USD includ- ing platform fees paid to AMT and server hosting costs.
Use of proprietary LLMs: We included the GPT- 3.5-turbo arge-languageemmoee from OpenAl our experiments since has demonstrated excellent performance on diverse NLP tasks in zero-shot and few-shot settings. Unfortunately, OpenAl could discontinue hosting the model in future which point it may not be possible to reproduce its sults on the tasks proposed in this work. For this reason we have also included results with public open-source LLMs like Llama and Vicuna, as these models are publicly available and hence their re- sults can always be reproduced.
While the benchmark dataset was annotated by human annotators, it is important to acknowledge the possibility of annotation errors or inconsisten- cies. Despite efforts ensure high-quality annota- tions, the presence of errors should be taken into account when interpreting the results. Human anno- tation subjective by nature, and different annota- tors may have varying interpretations in some situa- tions, e.g.. deciding whether a fact in the summary requires explicit evidence or should be presumed as common knowledge.
References
Nadcem Akhtar, Nashez Zubair, Abhishek Kumar, and Tameem Ahmad. 2017. Aspect based sentiment ori- ented summarization of hotel reviews. Procedia Computer Science, 115(C):563 -7516
12 Ethics Statement
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. 2022. Correcting diverse factual errors in abstractive summarization via post- editing and language model infilling. arXiv preprint arXiv:2210.1 12.778..
Potential biases: When selecting the pool of an- notators Amazon Mechanical Turk (AMT) for creating the dataset, we required their location be the United States. This was done since the US has a very large population of native English speakers, which can help in getting high quality annotations. However, this geographical restriction can also lead to biases in the annotation process. For example, it would affect what's considered common knowl- edge when assessing evidence for summaries. An annotator from the United States would likely con- sider person's 's birth in Los Angeles as evidence of them being from California, because they know Los Angeles is in that state. However, if it were some other city and state a country unfamiliar to them, they may not make a similar inference.
Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075-8096.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstrac- tive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686.
Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vieuna: An open- source chatbot impressing gpt-4 with 90%* chattpt quality.
Compensation for annotators: For the initial qualification task, workers were paid? 2 USD. After selecting the qualified workers, for the main an- notation task workers were paid 2 to 3 USD per document-summnn pair, depending on the num- ber of sentences in the summary and the domain where came from (we observed that some do- mains were more difficult). For the second round for verification, we paid annotators between 0.3 to 1.0 USD depending on the number of sentences in
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction-fineture language models. arXiv preprint arXiv: 20.0. 1416.
Arman Cohan and Nazli Goharian. 2016. Revisiting summarization evaluation for scientific articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation 16). pages 806-813.

---

Wafaa El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021. Automatic text sum- marization: A comprehensive survey. Expert systems with applications, 165:113679.
Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, and Zachary C Lipton. 2021. Generating soap notes from doctor-patient conversations using modular summa- rization techniques. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4958 4972.
Ori Ersst Ori Shapira, Ramakanth Pasunuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, and Ido Dagan. 2021. Summary-soun
proposition- alignment: Task, datasets and supervised baseline. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 310-322.
Kundan Krishna and Balaji Vasan Srinivasan. 2018. Generating topic-oriented summaries using neural attention. In NAACL-HLT.
Alexander Richard Fabbri, Chien- Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Im- proved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2587-22 2507-2601.
Kundan Krishna, Yao Zhao, Jie Ren, Balaji Lakshmi- narayanan, Jiaming Luo, Mohammad Saleh, and Pe- ter J Liu. 2022. Improving the robustness of sum- marization models by detecting and removing input noise. arXiv preprint arXiv:2212.0992
Wojciech Krysciński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332 9346.
Sebastian Gehrmann, Hendrik Strobelt, Robert Krüger, Hanspeter Pfister, and Alexander M Rush. 2019. Vi- sual interaction with deep learning models through collaborative semantic inference. IEEE transactions on visualization and computer graphics, 26(1):884- 894.
Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 2175-2189.
Bogdan Gliwa, Iona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. EMNLP-IJCNI 2019, page 70.
Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1449-1462.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequenc pre-training for nat- ural language generation. translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209.12333
Chin- Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. Wikiasp: A dataset for multi-domain aspect- based summarization. Transactions of the Associa- fion for Computational Linguistics, 9:211 1-225.
Yang Liu and Mirella I. Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of EMNLP- IJCNLP 2019, pages 3721-3731.
Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu. Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung, 2022. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:
Mikael Kägebäck, Olof Mogren, Nina Tahmasebi. and Devdatt Dubhashi. 2014. Extractive summarization using continuous vector space models. In Proceed- ings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), pages 31-39.
Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Hal- faker. Dragomir Radev, and Ahmed H Awadallah. 2022. On improving sunmarization factual consis- tency from natural language feedback. arXiv preprint 122.9922
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of doc- uments. In Thirty-first AAAI conference on artificial intelligence.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of reddit posts with multi-level memory networks. In Proceedings of NAACL-HLL, pages 2519-2531.

---

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequeee rans and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1 13971.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen Mckeown, and Bing Xiang. 2021. Entity-lvvel fac- tual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 2727-20 2733.
Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies. pages 47-57.
Kam-Fai Wong, Mingli Wu, and Wenjie Li. 2008. Ex- tractive summarization using supervised and semi- supervised learning. In Proceedings of the 22nd in- ternational conference on computational linguistics (Coling 2008), pages 985-992.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't 't give the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. ArXiv, flss/1808.0877
Ani Nenkova. 2006. Summarization evaluation for text and speech: issues and approaches. In Ninth Interna- tional Conference on Spoken Language Processing.
Jingqing Zhang. Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-traine -r-iariiing with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339, PMLR.
Ani Nenkova, Kathleen McKeown, et al. 2011, Auto- matic summarization. Foundations and Trends@ in Information Retrieval, 5(2-3): (12-3):033-233.
Yong Zhang, Joo Er Meng, and Mahardhika Pratama. 2016. Extractive document summarization based on convolutional neural networks. In IECON 2016-42nd Annual Conference of the IEEE Industrial Electronics Society, pages 918 918-922. IEBE
Dragomir Radev, Eduard Hovy, and Kathleen McKe- own. 2002. Introduction to the special issue on sum- marization. Computational linguistics, 28(4):399- 408.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Erik Tjong Kim Sang and Sabine Buchholz. 2000, In- troduction to the conll-2000 shared task chunking. In Fourth Conference on Computational Natural Lan- guage Learning and the Second Learning Language in Logic Workshop.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Questeval: Summariza- tion asks for fact- evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083.
Aviv Slobodkin, Paul Roit, Eran Hirsch, Ori Ernst, and Ido Dagan. 2022. Controlled text reduction. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5699- 5715, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

cation, the target is single Yes/No based on the label. During inference, we measure the probabili- ties of generated Yes/No tokens which allows us to measure AUC scores too. For Unsupported Span Prediction, we generated the ground truth target by surrounding the unsupported spans in the summary with begin-span and end-span tags.
A Appendix
A.1
Sample datapoints for different tasks
We show sample labeled datapoint for cach task from the validation set of USB in Figure and Figure 6.
Llama/Vieuna For Llama and Vicuna we use the exact same input formatting. Compared to the Flan-T3 data formatting, we use different set of instructions for these models, after trying out plau- sible variants for each task on the validation set. We provide 4 different instances as few-shot ex- amples following the instruction each datapoint for cach task. The few-shot examples are chosen by sampling from the training set without replace- ment. Due to limitations in sequence length, we only use a maximum of 2048 tokens for the few- shot examples. For the tasks which require the full document in the input (i.e. ABS, EXT, EVEXT, TOPIC), we use 4 examples with each having a maximum of 512 tokens. For the remaining tasks, we use 16 examples each with a maximum length of 128 tokens. The few-shot examples are sampled (without replacement) from the training set while creating the prompt for each datapoint in the test set. Since these are decoder-only models which essentially generate plausible completions of the input string, we preface each output with word (e.g. "SUMMARY:", "LABELS:") in the few-shot examples and at the end of the prompt to trigger the generation of the required summary/labels.
A.2 Instructions used in model inputs
We list the instructions used in the inputs to Flan-T5 models in Table 6, Llama- 13B in Table 7, Vicuna- 13B in Table 8, and GPT-3.5-turbo in Table
A.3 Implementation details for models
In this section outline the architectures, and input/output formatting used for different models report
used in our experiments. Additionally, the hyperparameters used during training and infer- ence for each model in Table 10.
Roberta For the Factuality Classification task, we feed in the evidence and summary separated by the SEP token into a standard classifier setup, which applies a linear layer with sigmoid activa- tion on top of the CLS embedding. For Evidence Extraction, we use the same architecture and input individual pairs of summary sentence with each source sentence to make prediction for each of them. For the Extractive Summarization task, we use a hierarchical architecture identical the one described as BERT-LSTM in Krishna et al. (2021), except that we use a Roberta encoder instead of BERT. For Unsupported Span Prediction, we frame it as a sequence tagging problem where the given summary sentence and evidence are passed through Roberta and linear layer with sigmoid predicts whether each token is supported or not. The con- secutive positive predictions are concatenated to turn them into spans.
GPT-3.5-turbo
The formatting of input and out- put is exactly the same as for Llama/Vicuns for all tasks except Evidence Extraction and Extractive Summarization, For these two tasks, we found that this model performed much better if we prompted it to generate the source sentence ids which should be assigned the positive label, instead of generating Yes/No prediction for each source sentence. So we changed the output formatting in our few-shot ex- amples accordingly. For this model too, we choose a different set instructions for the tasks by exper- imenting with different options on the validation set.
T5/Flan-T5 We preface each input with an in- struction for the task to be done, followed by the text from the source/summary to be input. We frame the Evidence Extraction and Extractive Sum- marization tasks as sequence of Yes/No predic- tions for each sentence in the source. Each source sentence in the input is prefixed by an enumer- ated sentence id (e.g. SENT34), and the ground truth target is the sequence of all sentence ids, with a Yes/No following each according to it's posi- tive/negative label (c.g. "SENTO Yes SENTI No SENT2 No..."). Similarly, for Factuality Classifi-
A.4 Human evaluation of model outputs
It well-acknowledged that ROUGE (Lin, 2004) an imperfect automatic metric to assess sum- mary quality, and may not accurately reflect human preferences (Nenkova, 2006; Cohan and Goharian,

---

2016; Goyal al., 2022). Hence, we also con- ducted human evaluation for some tasks, where we show summaries generated by the best fine- tuned model (Flan-T5-XL) and the best fewshot- prompted LLM (GPT-3.5-turby) and ask annotators to choose the better one along different dimensions (Table 3).
For the tasks of Abstractive Summarization (ABS), Multi-sentence Compression (COMP), and Topic-based Summarization (TOPIC), we collected annotations for 50 pairs of summaries, with 3 notators rating each pair. For these tasks, we did not screen workers based on qualification tasks since evaluating overall summary quality is sub- jective task and it is better to have a diverse opinion from a large population, rather than small set of manually selected people.
Evaluating model outputs for the Fixing Factual- ity (FIX) task is a more difficult but objective job. The increased difficulty comes from the need to carefully note the edits made by the models on the original incorrect summary and then decide on the factual validity and necessity of each edit. So we screened annotators via qualification task on Me- chanical Turk and selected annotators to conduct the human evaluation for this specific task. Each pair of model outputs was rated by both annotators.

---

Source length
Summary length
Evidence sentences per summary sentence
3030
100
75-
2000
50-
50-
1000
2000
4000
6000
8000
10000
100
200
500
Figure 3: Distribution of number of words in the source and the summary, and the numher of source sentences marked as evidence per summary sentence.
Task
Instruction
Multi-sentence Compression (COMP)
Summarize the following content in a single line.
Abstractive Summarization (ABS)
Summarize the following content.
Fixing Factuality (FIX)
Rewrite the given summary of the content to make it factually correct.
Unsupported Span Prediction (UNSUP)
Annotate parts of the summary which are not sup- ported by evidence from the content.
Topic-based Summarization (TOPIC)
Summarize the given content for the following topic.
Factuality Classification (FAC)
Is there sufficient evidence for the summary in the content?
Extractive Summarization (EXT)
For each sentence, predict it is important.
Evidence Extraction (EVEXT)
For each sentence in the content, predict it provides any evidence for the claim.
Table 6: Instructions used in inputs Flan-TS T5 models
Assignment ID: STONTTTEENE222SSSOOOMMMES Time rerraiing 003403
company which deonloped high-praccctton digital-to- scamers, well
Indmuation
Sihmt
in 1958, Tredtan raturned to assoents, starting at the Nossochsettss Iratitute faclmologe (M21) as a fill professor.
Tdward Freckin (horn Octaber 1931) drectaped profecese
13 froe 1971 to 1999, treskin vas the oirector of Preject NN at NiI.
sigitsl
10 (Profect PAC was year
MET Latorstary for computee Science 1976.)
atyuica.
brisences narked: 2 Park a3. Bores
tobel Prize-wirnin physicist Richars Fayrmar, and was Protesse of Plyntes
Freditn's peinary contributions include vork on reversible corpoting and rettilaee automata.
toston Universsty for aix yours.
fredia has hal formal and informal conniatiinn Eecasgie Bellon University
sexeral
While karrad Zane's a bask, "Calculating Space" (1920), sentiones the importance of reversible corpotation, the Freckin gate represented the
eigital nechantce,
study of diserate rodale of fundamental process ir Bhystee.
rssential treakthreugh.
Fredzin has bean Seatingesshes Career Frotessor ot Zompeter Sclerce at Drl.
24 als Wisdting Sclentia: NET Medie Laboratury.
te recent work, uses
"digital philososty" (wr).
Profossce Robotics BM oistlspddsed foueder
Evicences marked: a Mark as Done:
Fredete
cospartes, including Ieferatier Isternattoral, These Rivers Compater Cocsoration, Ergland
divense
5 buring his carser, Fradkin has teen profecsoe of conputer science at the
Talevistor Corgoration (owter of Doston's a then aa atriliate NAV on chernel 7), The ielliatle Natur consary (sanofectiree asvanced
Scratchoad:
desilination
Frediln
Fereegla
warly stonere digtral plyctes.
Frediin broally Interested cosptaction
and saftaree
ke the storetor the trie data structure, racio transponders for vehicle
Dridence selected:
contificattion the concept of compater ravtgattor for autowattes, the Freskin and the silliard-tall computer Nodel resersible compiting.
hredt has had tormal and intormal exocuation with Carrage Malion Univarstty (CMU) over several decade..
corputar artificial Intelligoncs reseanch.
on fredisn also worked at the intersection theonetical issues in the Physics of cosputattir with cosputsttoral nodsle of physses.
Figure 4: Screenshot of the interface used for collecting annotations. The summary is shown on the left and the source on the right. Entities in the active summary line are highlighted to help find evidence quickly. A scratchpad is provided where users can keep track of the parts of the summary for which evidence has been marked.

---

Task
Instruction
Multi-sentent Compression (COMP)
Write one-line summary of the content shown be- low.
Evidence Extraction (EVEXT)
Go over each sentence in the content, and decide if it supports the claim not. Answer in Yes for sentence if it supports the claim, and answer No otherwise.
Factuality Classification (FAC)
Is there sufficient evidence for the summary in the content?
Fixing Factuality (FIX)
Rewrite the given summary of the content to make it factually correct.
Abstractive Summarization (ABS)
Write concise summary of the following paragraph
Topic-based Summarization (TOPIC)
Summarize the given content for the following topic.
Extractive Summarization (EXT)
For each sentence in the given content, label it as Yes if it is noteworthy enough to be included in a summary, or No otherwise.
Unsupported Span Prediction (UNSUP)
Regenerate the given summary, while surrounding those parts which do not have any supporting evi- dence in the content using and [/] tags
Table 7: Instructions used in inputs to the Llama- 13B model
Task
Instruction
Multi-sentence Compression (COMP)
Write a single sentence summarizing the important points in the given content.
Evidence Extraction (EVEXT)
Predict which sentences in the given content can be used to infer facts in the claim.
Factuality Classification (FAC)
Decide the following summary is consistent with the corresponding content. Note that consistency means all information in the summary is supported by the content. Explain your reasoning step by step then answer (yes no) the question
Fixing Factuality (FIX)
Rewrite the following summary to make factually accurate
Abstractive Summarization (ABS)
Draft summary for the given document.
Topic-based Summarization (TOPIC)
Generate a summary of the given content covering the given topic.
Extractive Summarization (EXT)
For each sentence, predict if it is important.
Unsupported Span Prediction (UNSUP)
Annotate parts the summary which are not sup- ported by evidence from the content
Table Instructions inputs Vicuna- 13B model

---

Task
Instruction
Multi-sentence Compression (COMP)
Summarize the following content in a single line.
Evidence Extraction (EVEXT)
Below is claim along with its corresponding content. Identify and list the sentences within the content that partially or entirely support the claim.
Factuality Classification (FAC)
Decide if the following summary consistent with the corresponding content. Note that consistency means all information in the summary is supported by the content. Answer yes or no.
Fixing Factuality (FIX)
The summary might be incorrect. How would you rewrite to make it factually accurate? Make as little changes as possible. Do not add any new information to the summary.
Abstractive Summarization (ABS)
Draft summary for the given document.
Topic-based Summarization (TOPIC)
Create a short summary of the given content that touches upon information which fall under the speci- fied topic.
Extractive Summarization (EXT)
For the task of extractive summarization, list all the SENTs of the content which would be included in its summary.
Unsupported Span Prediction (UNSUP)
Go over the given summary carefully, and regenerate it while surrounding any parts which are not sup- ported by the content using [] and [/] tags
Table 9: Instructions used in inputs to the GPT-3.5-turbo model
Model
Task
Learning rate
Batch Size
Max input length
Max output length
Roberta-Large
FAC
le-5
32
512
Roberta-Large
EXT
le-5
32
128x 128
Raagrta-Large
EVEXT
2e-5
2048
I28
Roberta-Large
UNSUP
2e-5
32
512
Large
(AIl)
5c-5
32
8192
768
Llarge-LLage
(All) 5e-5
32
8192
768
FlanT5-XL
(All)
5e-5
64
1536
512
Llama-13B
(All)
6144
512
Vicuna-133
(All)
6144
512
GPT-3.5-turbo
(AIl)
Variable
Variable
Table 10: Hyperparameters used for training and inference with different models. ú: 128 sentences each with maximum of 128 tokens fed into hierarchical model. o: GPT-3.5- Sttro-ttooooooo has relatively small limit of 4096 tokens including both the input (with fhw-shot examples) the output, and so we truncate input on per-task basis leave token budget equal to the maximum output length in the train split for that task.

---

Evidence Extraction
Accuracy
AUC
F1
Precision
Recall
SuperPAL (Ernst et al., 2021)
98.1
95.8
53.8
82.1
40.0
ROUGE (Chen and Bansal, 2018)
95.9
88.5
40.9
33.7
52.1
Entity overlap
95.7
92.5
47.0
35.6
69.4
Human annotations 100% (N=765)
98.8
99.0
77.7
77.0
78.4
Human annotations 20%
98.7
98.4
74.7
78.9
70.8
Human annotations 10%
98.5
98.1
72.4
73.0
71.8
Human annotations 5%
98.4
97.7
70.9
70.8
70.9
Factuality Classification
Accuracy
AUC
F1
Precision
Recall
FactEdit (Balachandran al., 2022)
55.7
74.6
29.4
72.6
18.4
FactCC (Kryšciniski et al., 2020)
52.9
68.9
20.1
66.3
11.8
Human annotations 100%
88.1
95.1
87.5
92.3
83.2
Human annotations 20%
86.7
93.9
86.1
90.6
82.0
Human annotations 10%
83.4
91.8
81.6
91.7
73.5
Human annotations
82.6
82.5
81.7
Fix factuality
Exact Match
Rouge-1
Rouge-2
Rouge-I.
FactEdit (Balachandran al., 2022)
1.0
81.6
73.0
FacCCC (Kryšcińskii et al., 2020)
0.8
81.9
73.6
81.4
Human annotations 100%
32.9
91.9
86.5
91.4
Human annotations 20%
28.8
90.3
84.3
89.8
Human annotations 10%
15.3
85.7
78.5
85.1
Human annotations 5%
11.2
83.9
76.1
83.3
Table 11: Comparision between using human annotations vs heuristic annotations for training models Flan-TT-
Large. We also report performance when finetuning on smaller fractions of the training set with human annotations.

---

Abstractive
INPUT
DOCUMENT:
Summarization
D'Vauntes Smith-Rivera High school career
(ABS)
Smith-Rlvera started high school at North Central High School in Indianapolis, and led team to state championship in his sophomore year. transferred to basketball specialty Oak Academy in Virginia senior year, and helped lead team recruited by Xavier, UCLA, Louisville, Memphis, State, and Georgetown.
national championship
TARGET
D'Vauntes Smith Rivera professional basketball player last played played high school basketball for North Central in Indianapolis and Hill Academy Virginia.
League.
Multi-sentence Compression (COMP)
SOURCE SENTENCES:
Odenkirk hired writer "Saturday Night Live" 1987 and worked there through 1991.
Odenkirk's friendship with Stiller, with whom he briefly shared office at "SNL", would lead to his being hired for the cast of "The Ben Stiller Show" in 1992.
Working both writer and actor the show, he created and starred in the memorable sketch "Manson Lassie", and helped the show win Emmy Award for writing.
TARGET
From the late 1980s 1990s, Odenkirk wrote for television shows 'Saturday Night Live" and "The Ben Stiller Show". winning Emmy Award for writing.
Extractive
INPUT
DOCUMENT:
Summarization
SENTO: D'Vauntes Smith-Rivera
(EXT)
SENT1: High school career
SENT2: Smith-miiee Rivera started high school at North Central High School in Indianapolis, and led team to state championship in his sophomore year. SENT3: He transferred to the basketball specialty Oak Hill Academy Virginia for his senior year, and he helped lead the team to the 2012 national championship.
He was recruited by Xavier, UCLA, Louisville, Memphis, NC State, and Georgetown.
SENT4:
TARGET
SENTO SENT2 SENT4...
Topic-based
INPUT
DOCUMENT:
Summarization
Arkema S.A.
(TOPIC)
Arkera created when French oil major Total restructured chemicals business.
The restructuring gradual that began many earlier:
TOPIC NAME: Organization
TARGET
Arkema organized into three business segments: Coating Solutions, Industrial Chemicals, and Performance Products.
Figure 5: Sample onput-output pairs for different tasks from the validation set of USB

---

Factuality
INPUT
EVIDENCE:
Classification
2014 YG also expanded into the beauty industry with the creation its cosmetics brand Moonshot.
(FAC)
YG Plus Inc., previously named Phoenix Holdings Inc., is a publicly traded media
advertising company acquired by YG Entertainment November 2014.
SUMMARY:
addition, the company operates a number of subsidiary ventures under separate public traded company, YG Plus, which includes clothing line, golf management agency, and a cosmetics brand.
TARGET
Incorrect
Unsupported Span
INPUT
EVIDENCE:
Prediction
David Martin Mcintosh
(UNSUP)
Mcintosh born in Oakland, California, the of Jean Marie (Slough), judge, and Norman Mcintosh.
He graduated with B.A. (cum laude) 1980, and later received J.D. from University of Chicago Law School 1983.
Incumbent Democrat U.S. Congressman Philip Sharp of Indiana's 2nd congressional district decided to retire. Mcintosh decided to and the Republican primary with plurality 43% in four candidate field.
the general election, defeated Democratic Secretary of State of Indiana Hogsett
SUMMARY: David
attorney and Republican Party politician who served as representative Indiana's
(born 1958)
congressional district from
TARGET
David Martin Mcintosh ( born June 8, 1958 is an American attorney and Republican Party politician who served as the U.S. representative for Indiana 's 2nd congressional district from 1995-to 2001.
Fixing Factuality (FIX)
INPUT
EVIDENCE:
In 2009, Jordan returned to the F1 scene as pundit for BBC Sport F1 coverage alongside Jake Humphrey (who was later replaced by Suzi Perry) and David Coulthard.
In March 2016 he was announced as Channel 4's lead analyst for C4F1.
SUMMARY:
He was the chief analyst for Formula One coverage the BBC from 2009 to 2015 before joining Channel after BBC pulled out in 2016.
TARGET
He was the pundit for Formula One coverage on the BBC from 2009 before joining Channel
2016.
Evidence Extraction
INPUT
DOCUMENT:
(EVEXT)
SENTO:
2012 Istanbul rally commemorate the Khojaly massacre
SENT1: "Justice Khojaly" campaign.
SENT2:
"Justice for Khojaly", "JFK" for short, International Awareness Campaign, 'Justice Khojaly, Freedom Karabakh".
initiated May 2008 under the
SENT6:
Around 200,000 participants for 20th anniversary remembrance of the Khojaly victims, dozens youth student organizations, public unions, Turkish organizations movements participated in the rally.
Various slogans included, "We are all from Khojaly", "Stop Armenian aggression", not forget Turkic people genocide by Armenian gangs southern Azerbaijan', "One nation, two countries, Justice for Khojaly!", and "Stop Armenian lies".
SENT17:
SUMMARY:
The demonstration with slogan "We are all from Khojaly" had around 200,000 participants.
TARGET
SENT6 SENT17
Figure 6: Sample input-output pairs for different tasks from the validation set of USB


Reviews:
[{"review_id": "qL89Gu8xqd", "reviewer": "Reviewer_WFLP", "summary": "", "questions": "could you please give more details about bout the annotators and how human build the dataset? More evidences for the quality of the datasets could be provided.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on the uniqueness of the tasks, the evaluation of various models, and the insights provided by our paper.\n\n### Quality of the Dataset\nTo address the point that the summaries are not written but edited by human, we note that the introduction part of the Wikipedia articles even before the edits is actually a summary written collaboratively by humans (the many contributors who authored the Wikipedia page). The official Wikipedia manual of (writing) style (https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style) dictates that “An article's content should begin with an introductory lead section – a concise summary of the article”; hence we believe this is a good summary of the topic of the article covering the most important information, whereas the long remaining part of the article goes into more details. The only issue is that the introduction section may also contain things not present in the remainder of the article, in which case it might be considered “unfaithful” to the full article. To address this, we employed annotators to edit the introduction section and remove information not supported by the remainder of the article, thus creating properly aligned document-summary pairs.\n\n\nWe employed rigorous quality assurance measures outlined as follows:\nWe conducted rigorous screening of annotators via an initial qualification task which was attempted by 174 AMT workers and only 16% of them passed after manual grading by us. \nDuring the main task, we further monitored the annotations submitted by the workers and removed the ones whose annotations weren’t up to the mark, retaining only the best annotators.\nFinally, after all documents were annotated, we again ran a new task to verify the annotations. We used a trained model to identify summary sentences with potentially missing evidence, and got them re-checked by annotators who made the appropriate edits if there was a legitimate error in the prior annotation.\n\n\n### Lack of Depth in Experimental Analysis\nWe feel we have performed a robust set of experiments (though would be happy to add additional, specific experiments if suggested). To reiterate, we performed 3 sets of experiments. \n\n(1) We benchmarked several latest models for the tasks introduced and compare their performance especially comparing fine-tuning and few-shot prompting based approaches.\n\n(2) We evaluated the performance of fine-tuned models when tested on a domain different from what they were trained on.\n\n(3) We demonstrated the usefulness of our annotations by comparing against heuristic annotations done using previously proposed approaches. \n\n### More details about the annotators\nIn the Mechanical Turk interface, we filtered workers by the following criteria: (1) they should be located in the United States, (2) they need to have a HIT approval rate of >95%, and (2) they must have at least 1000 approved HITs. To ensure the expertise of the workers, we conducted an extensive qualification round where 174 annotators participated and 28 of them were selected by manually grading their responses by the authors. They were allowed to do the main task but their annotation quality was spot-checked, after about 15% of the dataset was collected, and 3 annotators were hand-picked because of their excellent annotation quality. The remaining 85% of the dataset was annotated by those 3 annotators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The benchmark offers a diverse platform for training and assessment on 8 unique tasks focusing on essential yet overlooked facets of text summarization.\n2. Various models and training approaches, such as fine-tuning, few-shot prompting, and multi-task training, were evaluated.\n3. They provided insights on how different tasks generalize outside their original domain, pinpointing which tasks prioritize the volume of training data over its specific domain origin.\n4. The paper is well-written.", "reasons_to_reject": "1.The dataset is not written but edited by humans, and the quality of the dataset is unknown. Therefore, the evaluations done on this dataset might not be so trustworthy.\n2. The experimental analysis lacks depth, so the findings based on using this benchmark is not exciting enough.\n\n", "questions_to_authors": "could you please give more details about bout the annotators and how human build the dataset? More evidences for the quality of the datasets could be provided.", "paper_topic_and_main_contributions": "They present a benchmark based on Wikipedia, enhanced with a comprehensive set of crowd-sourced annotations, supporting 8 interconnected tasks: (i) extractive summarization, (ii) abstractive summarization, (iii) topic-centered summarization, (iv) compressing selected sentences to a single line summary, (v) identifying evidence backing a summary, (vi) determining a summary's factual accuracy, (vii) spotting unsupported segments in a summary, and (viii) rectifying factual errors in summaries. When comparing different approaches on this benchmark, they find that medium-sized fine-tuned models consistently surpass larger few-shot prompted models across several tasks. Regarding tasks tied to factuality, heuristics-based training data perform worse than training on much fewer human-labeled datasets. Their sourced articles span 6 fields, enabling cross-domain studies. Depending on the task, the quantity of training data can be more influential than its domain origin, but for some tasks, domain-specific training, even if scarce, proves more effective.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uKganOG6f8", "reviewer": "Reviewer_mh1S", "summary": "", "questions": "The Landmarks and Newspapers are not included in any experiments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the value of our work in contributing a high-quality, multifaceted benchmark for summarization tasks. We appreciate your positive comments on the paper's writing, the importance of quality datasets, and the range of tasks our benchmark supports.\n\n### Absence of Human Assessment for the Ultimate Version of the Proposed Dataset\nWe understand your concerns about the absence of human evaluations for the final version of the dataset. We can break down the quality of the dataset into 2 aspects: (1) the summary must have good coverage of the important aspects of the source, and (2) everything in the summary should have corresponding evidence in the source. While it is impossible to achieve perfection in both these criteria for any dataset, we have designed our dataset collection process to ensure them as follows :\n\n(1) We took the introduction section of the  Wikipedia article as the summary which covers most of the important content in the remaining article according to the official Wikipedia manual of (writing) style (https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style) : “An article's content should begin with an introductory lead section – a concise summary of the article”;  and hence we believe this is a good summary of the topic of the article covering the most important information, whereas the long remaining part of the article goes into more details. \n\n(2) We explicitly tasked annotators with editing summaries to remove unsupported facts from the source, which is rarely done in most widely used summarization datasets such as CNN-DailyMail, XSum, Reddit-TIFU etc. To ensure good quality of annotations we eventually selected the best 3 annotators out of a total 174 initial candidates based on manual assessment of their work. Even after all documents were annotated, we again ran a new task on Mechanical Turk to verify the annotations. \n\nSince the notion of a good summary varies from one person to another, a human assessment of the appropriateness of summary content is unlikely to reveal any meaningful results. This is further complicated by the fact that there is no reference point to compare the scores given by humans in such an evaluation.\n\nSimilarly there is subjectiveness associated with deciding if any given part in the summary is unsupported or not. This is because often \nsome “background knowledge” is required to infer the summary from the source (we talk about this in lines  642-652).  For example, if the source mentions that a person was born in Los Angeles, and the summary says that they are American, then annotators who know Los Angeles is in that state would consider that as sufficient evidence. However, another annotator who does not have that background knowledge might consider it an unsupported fact and rate it as a bad summary.\n\n### Good human ratings of ChatGPT despite low ROUGE score\nPrior work has made very similar observation that ChatGPT summaries are preferred to that of fine-tuned models’ summaries by humans on popular benchmarks such as XSum, CNN-Dailymail (Goyal et al 2022, \"News summarization and evaluation in the era of GPT-3”). This does not necessarily suggest that the ground truth summaries are low quality, but rather suggests that their form (e.g. length, choice of important content) is different than that of ChatGPT summaries (leading to low ROUGE scores). Humans prefer ChatGPT summaries, which is in line with observations made in prior literature.\n\n\n### Poor performance of LLMs on factuality-related tasks\nWe experimented with 3 different LLMs and tried extensive prompt-tuning to try and improve performance of fewshot prompting but the performance stayed far below that of finetuned models. This simply shows that fewshot prompting is not enough to teach models to do factuality related tasks. There is no reason why a model should be capable of learning any arbitrary task from a small number of demonstrations, and perhaps these tasks are ones where it is necessary to train the model with thousands of examples to teach them.\n\n\n### Landmarks and Newspapers in Experiments\nThe Landmarks and Newspapers datasets were not included in the training phase due to their small size. Instead, they were used only as challenging Out-Of-Domain (OOD) examples in the testing phase to evaluate the model's generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written. High quality summarization datasets are always important for the research community. This paper integrated eight very interesting summarization-related tasks and annotated a relatively large amount of examples for training and evaluation. This will be helpful for everyone in the research community.", "reasons_to_reject": "My main worry is the absence of human assessment for the ultimate version of the proposed dataset. Maynez et al. 2020 noted that ground truth summaries are prone to hallucinations. I believe that incorporating thorough human evaluation of the dataset would provide researchers with confidence to explore this dataset extensively. \nDue to the lack of human confirmation, I currently have some doubts about the quality of the annotated summaries. This is because, in Table 2, the difference in performance between ChatGPT and the fine-tuned models on generation tasks, as assessed w.r.t. the reference summaries, is quite substantial. \nThis indicates that summaries generated by fine-tuned models are more aligned with the annotated ground truth summaries, whereas summaries produced by ChatGPT do not closely resemble the ground truth. Interestingly, human evaluation highlighted a preference for summaries generated by ChatGPT. This suggests that the proximity to human-annotated summaries may not necessarily correlate with higher human satisfaction. Perhaps it's necessary to conduct a more thorough analysis of the distinctions between human-annotated summaries and the summaries generated by ChatGPT.\n\nAdditionally, the paper lacks an in-depth examination of the experimental outcomes. Of particular note is the notably poor performance of LLMs on tasks other than generation (such as EVEXT, EXT, FAC, UNSUP).", "questions_to_authors": "The Landmarks and Newspapers are not included in any experiments?", "paper_topic_and_main_contributions": "In this paper, a benchmark derived from human-annotated Wikipedia content is presented, encompassing eight closely connected tasks: extracted summarization, abstractive summarization, topic-based summarization, sentence compressing, evidence selection, factual accuracy predicting, unsupported span prediction, factual errors correction. The authors also conduct a comprehensive comparison between traditional fine-tuned models and the most recent LLMs across these tasks, using both automatic and human evaluations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MnOesQCWw9", "reviewer": "Reviewer_T3pT", "summary": "", "questions": "- QA line 80, if evidence was lacking, do you directly remove all sentences? Or do you remove only some related spans?\n- QB line 205, do you only consider hyperlinks as entities? Have you ever done an experiment to show its coverage of all entities?\n- QC line 248, how do you deal with the sentences that are verified to be wrong?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the importance of our work and its contributions to the field of summarization, particularly in detecting the factual issues of LLMs.\n\n### Considering the introduction section of article as summary\nThe official Wikipedia manual of (writing) style (https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style) dictates that “An article's content should begin with an introductory lead section – a concise summary of the article”;  and hence we believe this is a good summary of the topic of the article covering the most important information, whereas the long remaining part of the article goes into more details. The only issue is that the introduction section may also contain things not present in the remainder of the article, in which case it might be considered “unfaithful” to the full article. due to which it could not be considered a valid summary of the remaining article. To address exactly this, we employed annotators to edit the introduction section and remove information that’s not supported by the remainder of the article, thus creating properly aligned document-summary pairs.\n\n\n### Ground truth for extractive summarization task (EXT) is not natural \nGiven the abstractive summary of an article, we believe it is logical that the parts of the article that provide evidence for the summary should collectively be a good extractive summary of the article. This is because all the information which is present in the abstractive summary is also present in the corresponding evidence sentences from the source.\n\n### Domain Bias\nThe skew in domain samples is due to the difference in the amount of available data in Wikipedia (there are many more articles in the biographies category than the others). We specifically included rare domains to perform low-resource domain transfer experiments, as highlighted in Table 3 of our paper.\n\n### Answers to more specific questions\n\n(Line 80) Only the unsupported span is removed. In some cases, the entire sentence may be removed if it is entirely unsupported.\n\n(Line 205) We considered only hyperlinks as entities to ensure a high level of certainty, as this information would be sent to human annotators. This approach was chosen to be time- and cost-efficient.\n\n(Line 248) Human annotators are instructed to remove parts of the summary sentence for which no evidence is present, and otherwise annotate unmarked evidence where it exists.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is important to create a manually labeled summarization benchmark that includes various tasks. Especially for detecting the factual issues of the LLMs.\n2. Overall, the paper is well-written and easy to follow. I can understand the motivation for each step of dataset construction and experiment. \n3. The proposed benchmark is relatively comprehensive. It covers 8 tasks and 6 domains with an evaluation of two types of SOTA models. A number of insightful findings are proposed by analyzing experimental results.", "reasons_to_reject": "1. Quality of the summaries is unknown. Although there is verification for annotation of the evidence, it is not clear about the quality of the summaries themselves. How could we know that the overview section of Wikipedia can serve as a good summary of the rest of the source text? Probably, some key information in the content sections is missing as a summary. The overall section is not designed to serve as a summary initially. That's why we can find factual errors in it. So I think the authors also need to prove they can serve as summaries by some human annotation.\n\n2. Some proposed tasks are not natural. Since the dataset is not written but edited by humans, some tasks seem pseudo data as well. For instance, for EXT task, evidence is just for the overall section which may not be a good extractive summary. And for the TOPIC task, the topic-based summary is directly extracted from the overall section. I believe the readability will decrease. It is also a pseudo-summary rather than a human-written one. Similarly, in the COMP task, some coreference issues may happen. And it may not be a good summary of evidence without rewriting.\n\n3. Domain bias. Biographies contain 1514 samples which is much larger than the sum of the rest domains. The second largest domain only contains 150 samples. This may weaken your conclusions on the domain experiments.\n\nOverall, although some manual labels are involved, the construction of the dataset still contains some bias and noises which weaken the conclusions. Among the 8 tasks, the most useful or clean ones are fact-check-related tasks. \n\n\n", "questions_to_authors": "- QA line 80, if evidence was lacking, do you directly remove all sentences? Or do you remove only some related spans?\n- QB line 205, do you only consider hyperlinks as entities? Have you ever done an experiment to show its coverage of all entities?\n- QC line 248, how do you deal with the sentences that are verified to be wrong?", "paper_topic_and_main_contributions": "The authors propose a unified summarization benchmark (USB). They use the overall section of Wikipedia pages as the candidate summary of the rest of the source. Then, they ask annotators to find the evidence for each sentence in the candidate summary and correct the sentences with no evidence. They propose 8 tasks over the annotated dataset and evaluate it on fine-tuned models and LLMs. Results show that fine-tuned models outperform few-shot LLMs. They also split the dataset into 6 domains and evaluated the cross-domain performances.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "For figure 2, I suggest to use 30 as pure white and 50 as pure blue to make the figure more clear."}, "extra_text": {}}, {"review_id": "Io5usUG5cO", "reviewer": "Reviewer_Ur5L", "summary": "", "questions": "Q1 Instead of stating they are Mechanical Turk workers, could the authors provide more detailed ethical consideration that discusses the detailed human annotation process. For example, what are the background of human annotators? Are they linguists or just native speakers? What are their education level? Are they compensated and how?\n\nQ2 How reliable are the human evaluation results? Have the authors measure their agreements (e.g. kappa)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments about the motivation and value of our paper. \nRegarding your concern about the absence of an ethics section, we recognize the importance of ethical considerations, especially when human annotators are involved. We will add an ethics section in the revised paper to discuss this in detail. For now we have mentioned the details below.\n\n### Detailed Ethical Consideration for Human Annotation\nWe exercised considerable care in the recruitment of workers for annotation. The workers were required to be from the United States to do our task. Workers needed to have a HIT approval rate of >95% and must have at least 1000 approved HITs. To ensure the expertise of the workers, we conducted an extensive qualification round where 174 annotators participated and 28 of them were selected by manually grading their responses by the authors. They were allowed to do the main task but even that their annotation quality was spot-checked, after about 15% of the dataset was collected, and 3 annotators were hand-picked because of their excellent annotation quality. The remaining 85% of the dataset was annotated by those 3 annotators. We also conducted a second round of verification where we had got part of the annotations re-checked by annotators again. \n\nRegarding worker compensation, for the initial qualification task, workers were paid 2 USD. After selecting the qualified workers, for the main annotation task workers were paid 2 to 3 USD per document-summary pair, depending on the number of sentences in the summary and the domain where it came from (we observed that some domains were more difficult to work with). For the second round for verification, we paid annotators between 0.3 to 1.0 USD depending on the number of sentences in the summary which were flagged for verification, which can be as low as 1 sentence. \n\n\n### Agreement between raters for the human evaluation results\nYour question seems to pertain to the human evaluations comparing FlanT5 and ChatGPT (Table 5 in appendix). Below we show the agreement metrics among human evaluators on different tasks. \n\nFor the multi-sentence compression (COMP), abstractive summarization (ABS) and topic-based summarization (TOPIC) tasks, each example was rated by 3 different raters, while for the Fix Factuality (FIX) task, two annotators rated each example.\nBelow we show how often the raters agreed on the answer to the questions asked for each task:\n\n-----------------------------------------------------------------------------------------------------------------------\n\nMulti-sentence compression (COMP)\n\nWhich of the two summaries covers more information touching upon all the highlighted sentences?\nAll 3 raters agree: 46%\nTwo raters agree: 52%\nNobody agrees: 2%\n\nWhich of the following summaries is more factual, accurately representing the information presented in the document?\nAll 3 raters agree: 12%\nTwo raters agree: 78%\nNobody agrees: 10%\n\n-----------------------------------------------------------------------------------------------------------------------\n\nAbstractive summarization (ABS)\n\nWhich of the following summaries is better in terms of effectively summarizing the given full content?\nAll 3 raters agree: 38%\nTwo raters agree: 52%\nNobody agrees: 10%\n\nConsider the factuality of the summaries. Which of the following summaries is more factual, accurately representing the information presented in the given full content?\nAll 3 raters agree: 20%\nTwo raters agree: 70%\nNobody agrees: 10%\n\n-----------------------------------------------------------------------------------------------------------------------\n\nTopic-based summarization (TOPIC)\n\nWhich of the two summaries is better in terms of effectively summarizing the given topic?\nAll 3 raters agree: 60%\nTwo raters agree: 40%\nNobody agrees: 0%\n\nWhich of the two summaries is more related to and exclusive to the given topic?\nAll 3 raters agree: 42%\nTwo raters agree: 58%\nNobody agrees: 0%\n\n---------------------------------------------------------------------------------------------------------------------------\n\nFixing Factuality (FIX)\n\nWhich of the two summaries removes more contradictory/unsupported information from the incorrect summary, in reference to the highlighted text?\nBoth raters agree: 76%\nNobody agrees: 24%\n\nWhich of the two summaries removes more correct information (which is actually well-supported by the highlighted context) from the incorrect summary?\nBoth raters agree: 88% \nNobody agrees: 12%\n\nWhich of the two summaries adds more new facts compared to the incorrect summary?\nBoth raters agree: 98% \nNobody agrees: 2%", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and easy to follow.\n2. The USB benchmark can serve as a valuable data source for future research on evidence mining and factuality.\n3. Extensive experiments and detailed analysis.", "reasons_to_reject": "Generally, this paper is well motivated and can be valuable to the filed.\nThe only concern is the lack of an ethics consideration section to discuss the human annotation and evaluation detailedly, which correspondingly leads to the data quality concern.", "questions_to_authors": "Q1 Instead of stating they are Mechanical Turk workers, could the authors provide more detailed ethical consideration that discusses the detailed human annotation process. For example, what are the background of human annotators? Are they linguists or just native speakers? What are their education level? Are they compensated and how?\n\nQ2 How reliable are the human evaluation results? Have the authors measure their agreements (e.g. kappa)?", "paper_topic_and_main_contributions": "This paper introduces a unified summarization benchmark (USB) that supports 9 summarization-related tasks.\nTo construct USB, the authors first conduct a sophisticated preprocessing pipeline to contain source text of their interest. Then, given a wikipedia article, the authors perform human annotation by asking human annotators 1) identify relevant evidence for each sentence in the summary (wikipedia leading paragraph); 2) edit (delete) contents in the summary, which cannot be supported by the wikipedia article (input text). In this way, they obtain parallel data, including article, leading paragraph, modified leading paragraph, supporting evidence, etc, and design 8 tasks using those data.\nAfter building the USB, the authors conduct extensive experiments on it, and specifically analyze the OOD performance and influence of human annotated data compared with silver data.\nGenerally, this paper is well written and can provide some insights to the filed.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some statements are not alway true.\nL548-L554: \"There exist plenty of datasets... many of them were created heuristically\". However, *they are also many datasets that are human annotated*, for example, SAMSUM, QMSum, DialogSum. The authors could discuss more the difference compared with those abstractive datasets, instead of extractive ones.\n\nAlso, more recently, there is SummZoo benchmark, and more relevantly, there is MACSum, which contains multiple controllable attributes, and *is a unified benchmark as well*. However, they are not cited.\n\n\nZhang, Yusen, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. \"Macsum: Controllable summarization with mixed attributes.\" Transactions of the Association for Computational Linguistics 11 (2023): 787-803.\n\nZhong, Ming, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan et al. \"QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization.\" In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5905-5921. 2021.\n\nGliwa, Bogdan, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. \"SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization.\" In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79. 2019.\n\nChen, Yulong, Yang Liu, Liang Chen, and Yue Zhang. \"DialogSum: A Real-Life Scenario Dialogue Summarization Dataset.\" In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 5062-5074. 2021.\n\nChen, Yulong, Yang Liu, Ruochen Xu, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Yue Zhang. \"Unisumm: Unified few-shot summarization with multi-task pre-training and prefix-tuning.\" arXiv preprint arXiv:2211.09783 (2022).", "Justification_for_Ethical_Concerns": "The only concern is the lack of an ethics consideration section to discuss the human annotation and evaluation detailedly, which correspondingly leads to the data quality concern.\n\nIn particular, the authors did not provide a detailed ethical consideration section that discusses the detailed human annotation and evaluation process. For example, what are the background of human annotators? Are they linguists or just native speakers? What are their education level? Are they compensated and how?"}}]

Author rebuttal:

Meta-review: The paper introduces a Unified Summarization Benchmark (USB) encompassing 8 interconnected summarization tasks based on Wikipedia content, with a focus on factuality and correctness. The authors compare fine-tuned models and large language models (LLMs) on these tasks, revealing that fine-tuned models outperform LLMs, particularly in tasks tied to factuality. However, concerns regarding the quality of the human-annotated dataset and the need for more extensive human evaluation are raised, emphasizing the importance of assessing the dataset's reliability. Additionally, the paper lacks an in-depth analysis of experimental outcomes, especially the notable performance gaps observed in tasks other than generation. Some issues were resolved during the response period, specifically on human annotation.
Decision: Accept-Findings