SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization
Authors: Philippe Laban
Wojciech Maciej Kryscinski
Divyansh Agarwal
Alexander Fabbri
Caiming Xiong
Shafiq Joty
Chien-Sheng Wu
Venue: EMNLP 2023 Main (2023)
Keywords: factual consistency
faithfulness
summarization
LLMs
benchmark
TL;DR: We create a new benchmark -- SummEdits -- to measure whether LLMs can reason about the consistency of facts in summarization

Abstract: With the recent appearance of LLMs in practical settings, having methods that can effectively detect factual inconsistencies is crucial to reduce the propagation of misinformation and improve trust in model outputs.
When testing on existing factual consistency benchmarks, we find that a few large language models (LLMs) perform competitively on classification benchmarks for factual inconsistency detection compared to traditional non-LLM methods. However, a closer analysis reveals issues with existing evaluation benchmarks, affecting evaluation precision.
To address this, we propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. This new benchmark is 20 times more cost-effective per sample than previous benchmarks and highly reproducible, as we estimate inter-annotator agreement at about 0.9.
Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance, highlighting the gaps in LLMs' ability to reason about facts and detect inconsistencies when they occur.

SUMMEDITS: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization
Philippe Laban
Wojciech Kryšciński
Divyansh Agarwal Alexander R. Fabbri
Caiming Xiong Shafiq Joty Chien-Sheng Wu
Salesforce Al {plaban, divyansh.agarwal afabbri, cxiong, sjoty, wu.jason] ww.jason]@sslesliiiin @salesforce.com
Abstract
100%
SummEdits Benchmark Performance
With the recent appearance LLMs in prac- tical settings, having methods that effec- tively detect factual inconsistencies is crucial to reduce the propagation of misinformation and improve trust in model outputs. When testing on existing factual consistency bench- marks, we find that a few large language mod- els (LLMs) perform competitively classifi- cation henchmarks for factual inconsistency de- tection compared traditional non-LLM meth- ods. However, closer analysis reveals issues with existing evaluation benchmarks, affecting evaluation precision. To address this, we pro- pose new protocol for inconsistency detec- tion benchmark creation and implement it in a 10-domain benchmark called SUMMEDITS. This new benchmark is 20 times more cost- effective per sample than previous benchmarks estimate inter-
70.7
Cleud
68.0
Tuto
Blaon
$5.7
85.7
60
98.8
Bard
d8 12e
50% Frandon chenoe)
51.9
20.4
OpenAl Anthropic
Google
Meta
Mistral Vicuna Cohere
Plrfform.
O
M
Figure I: SUMMEDITS is benchmark evaluate the factual reasoning abilities of LLMs, measuring if mod- detect factual inconsistencies when they occur summaries. Capable detection models can help build more reliable NL. systems.
sistency, finding that most abstractive summariza- tion systems produce a non-negligible portion of inconsistent summaries. In turn, such corpora used to instantiate tasks such inconsistency detection (ID) (Laban al., 2022a; Tang et al., 2022), in which models are given (docurent, s summary) pairs, and must identify whether the summary is consistent with the document.
and highly reproducible, annotator agreement at about 0.9. Most LLMs struggle on SUMMEDITS, with performance close to random chance. The best-performing model, GPT-4, still 8% below estimated human performance, highlighting the gaps in LLMs' ability to reason about facts and detect inconsistencies when they occur.
Recent investigations of using LLMs for evalua- tion have shown promising results across different NL.P tasks (Liu et al., 2023; Fu et al., 2023), includ- ing factual consistency (Luo et al., 2023). In this work, continue this line of research and explore applying LLMs factuality evaluators in the con- text of text summarization. We first establish base- line performance for a suite of LLMs on two exist- ing benchmarks - AggreFact (Tang et al., 2022) and DialSummEva (Gao and Wan, 2022) with results confirming that some LLMs perform competitively with state-of-the-art specialized methods such as Q FvattEvvll (Fabbri et al., 2022). However, man- ual analysis of conflict cases in AggreFact reveals significant number of mislabeled samples (7+%) of factual inconsistencies undetected by annotators during dataset creation that the GPT4 explanations reveal. This lack of quality of benchmarks lim-
1 Introduction
With recent progress in generation capabilities of LLMs, automatic summarization is making its appearance in practical information consump- tion situations such as summarizing work meet- ings (Arabzadeh et al., 2022), health records (Jain et al,, 2022), or scientific documents (Cachola et al., 2020). To ensure the safe and effective implementa- tion of these applications, it is essential to limit the reach of factually inconsistent summaries, known issue with generated summaries (Kryšcifiski et al., 2019; Maynez et al., 2020).
Prior work (Kryšciński al., 2020; Fabbri et al., 2021; Gao and Wan, 2022) has annotated corpora of model summaries with labels of factual con-

---

its the precise evaluation of model performance at factual inconsistency detection.
produce non-negligible portion of inconsistent summaries. Although most annotation effort has focused on the summarization of news, some prior work also looked at dialogue summarization (Gao and Wan, 2022), or the medical domain (Tang et al., 2023). In most work, scalable high-quality anno- tation is challenging, due to low inter-annotator agreement when relying on crowd-workers, with some work showing that 10+ annotators quired achieve some level of consensus (Falke et al., 2019), and some work recommending solely relying on experts (Fabbri et al., 2021). At the heart of the issue, annotating the factual consistency of a summary is challenging; it requires careful reading of long documents and the detection and interpre- tation of nuanced facts. In this work, we propose protocol to annotate factual consistency sources and show that lowers the cost and in- creases reproducibility by minimizing the amount of reasoning required for each annotation.
To address this issuc, we introduce a protocol designed to create challenging benchmarks while ensuring the reproducibility of the labels. The pro- tocol involves manually verifying the consistency of small set of seed summaries and subsequently generating numerous edited versions of these sum- maries. We discover that assessing the consistency of edited summaries is relatively straightforward and easy to scale for human annotators, thus guar- anteeing low cost and high agreement among anno- tators, yet keeping the task challenging for models.
We create the SUMMEDITS benchmark by imple- menting the protocol in ten diverse textual domains, including the legal, dialogue, academic, financial, and sales domains. Figure summarizes experi- mental results on the benchmark, which indicate that SUMMEDITS presents challenge for all mod- els, with only four LLMs outperforming the special- ized model QAFactEval. Our estimate of human performance of 90%+ largely outperforms models, suggesting current LLMs are not yet proficient at complex factual reasoning, and cannot assess the factual validity of summaries with precision.
Some work has also annotated inconsistency of pairs on text on non-summarization tasks, such as paraphrasing (Zhang et al., 2019), document- grounded dialogue (Honovich et al., 2021; Dziri et al., 2022), and Wikipedia-editingg (Schuster al., 2021). Follow-up work has then aggregated and standardized annotations into benchmarks such as SummaC (Laban et al., 2022a), AggreFact (Tang et al., 2022) and TRUE (Honovich et al., 2022).
We believe SUMMEDITS can serve as a tool to evaluate LLMs' abilities to detect factual inconsis- tencies when they (inevitably) occur and encourage LLM developers to report their performance on the benchmark. For practitioners requiring specific domain expertise, the protocol can be adapted to generate low-cost, in-domain benchmarks that can check model capabilities prior to production use. We release the code and benchmark publicly!.
Detecting Factual Errors. Some work has taken an automated approach to the detection of inconsis- tencies, with approaches falling into two main cate- gories: question and entailment-based. In question- based approaches, questions are generated with the expectation that paired documents and summaries should provide consistent answers. QvFactEval (Fabbri et al., 2022) unified prior work (Wang al., 2020; Scialom et al., 2021; Honovich al., 2021) by systematically evaluating each element of the pipeline and proposing a best-performing combi- nation. Entailment- based methods either rely on entailment on dependency parses - DAE (Goyal and Durrett, 2020) or directly leverage natural- language entailment models, such as SummaC (La- ban et al., 2022a). We include these three represen- tative models in our experiments, finding that even with several orders of magnitudes fewer parameters than LLMs, they can reach similar performances on benchmarks.
2 Related Work
Annotating Factuality of Summaries. With ad- vances in language models and the increase in flu- ency and abstractiveness of summarizers, prior work showed that one of the key challenges in summarization was enforcing factual consistency (Kryšciński et al., 2019), particularly with mod- els trained on datasets with unfaithful references (Maynez et al., 2020). Several efforts such as FactCC (Kryšciński et al., 2020), SummEval (Fab- bri et al., 2021), Polytope (Huang et al., 2020), FRANK (Pagnoni et al., 2021), and CLIFF (Cao and Wang, 2021) annotated the generated sum- maries of tens of model, finding that most models
'https://g'' thub. com/ sal I est estorre/ FactualNiG

---

3 Limits of Crowd-Based Benchmarks
AggreFact
DialSummEEal
Model Name
% BAcc.
% BAcc.
Corr
We first analyze model performance on two popular benchmarks for factual consistency detection in summarization: AggreFact (Tang et al., 2022) and DialSummEvvl (Gao and Wan, 2022) and uncover limitations that guide the design principles of the SUMMEDITS benchmark.
DAE
76.0
56.2
0.44
SummaC
71.6
62.7
QAFactEval
73.9
64.4
0.59
Cohere-emd-X
63.1
56.6
0.36
Claude V1.3
50.6
56.8
0.30
Bard
62.7
0.26
PaL.M-Biison
57.0
55.6
0.57
Dav001
53.3
52.9
0.11
3.1 Experimental Setup
Dav002
54.3
59.2
0.49
Vicuna-13b
60.3
58.6
0.36
We include in our experiments three special- ized non-LLM approaches: DAE, SummaC, and QAFactEval and ten LLM models from recent LLM familics. We include Cohere's Command- XL, Anthropic's Claude V1.3 (Bai et al., 2022),
Dav0U3
GPT3.5-turbo
70.2
62.0
0.56
GPT4
73.6
68.4
0.58
Table I: Performance of models the AggreFact, Di- alSummEval consistency benchmarks reported in bal-
Google's Bard and PaLM2-Bison (Thoppilan et al., 2022), Vicuna- 13b (Chiang et al., 2023), and OpenAl's Da Vinci001 (Brown al., 2020), DaVinci002 (Ouyang et al., 2022), DaVinci003, GPT3.5-turbo and GPT-4. Appendix A provide each model's method of access and model card.
anced accuracy (% Bacc.) and correlation (corr.).
To inspect performance on the AggreFact bench- mark further, we hired an annotator to manually inspect the cases where GPT4 disagrees with the label of AggreFact. More precisely, we manually inspected the explanations provided by GPT4 for the 101 summaries judged were inconsistent but labeled as consistent in the dataset. Appendix B provides more detail on the annotation protocol.
To minimize the computational cost of experi- ments, we select single Zero-Shot prompt that is used for all LLM models. We make this choice in- stead of optimizing the prompt for each model for two reasons: (1) there's guarantee that prompt quality will transfer across benchmarks, and us- ing single common prompt removes variance from prompt optimization that does not measure underlying model ability, and (2) more complex prompts would require adaptation to each domain (c.g. domain-specifii few-shot examples), and re- strict the evaluation of models with shorter maxi- mum sequence lengths due to longer prompts.
Of the 101 samples, 80 were labeled by the an- notator correct or partially correct explanations that identify and explain factual inconsistency in the summary. other words, this manual anal- ysis of subset of AggreFact reveals that a min- imum of 6% of the samples in AggreFact are mislabeled. The low reliability of labels in crowd- sourced benchmarks like AggreFact is known issue (Pagnoni et al., 2021) stemming from task complexity that requires the annotator to carefully read and understand entire document and companying summary, leading to low repeatability and inter-annotator agreement.
3.2 AggreFact
AggreFact-SOTA (Tang et al., 2022) is factual consistency benchmark focused on the news do- main, modified from SummaC (Laban et al., 2022a) focus summaries generated by SOTA models (i.e., models based on pre-trained Transformers), as analysis showed older models' summaries were less relevant the field of consistency detection. Table 1 reports the balanced accuracy of spe- cialized models and LLMs on AggreFact. At first glance, the specialized models still outperform LLMs, even though incrcasing LLM size leads to performance improvements and helps close the gap. with GPT-4 performing within 2.4% points of the specialized DAE. However, all models perform rel- atively poorly, with no model reaching a balanced accuracy of 80% on a binary classification task.
This analysis reveals the potential for LLMs part of dataset creation. In some cases, LLM ex- planation that is verifiable such explanation for an identified factual inconsistency can accel- erate and improve the quality of annotation. L.M explanations might not be valuable in all cases, such as when a model asserts a summary is consis- tent, manual verification is still required to assure quality. In Section 5, we introduce a protocol for benchmark creation that can involve an LLM. Based on the low reliability of labels Ag- greFact, we note that a key requirement for future

---

Averuge Annotator
Source Document
Human Anno
Model
1.5
3.0
3.5
5.0
Dav001
90.2
Any flaws with the summary? (flvency, format, etc.)
Cohere-cmad
83.6
No
Pal.MM2bbson
25.3
78.7
77.7
90.0
Is the seed factually consistent document?
Seed Summary
Dav003
81.6
90.0
Yes
2.2
24.6
45.9
QAFactEvval
24.6
Edited Summaries
LLM
SummaC:
20.9
21.3
27.7
58.9
Claude
45.8
9.9
9.8
18.2
Table 2: Percent of summaries classified consistent in DialSummEval, bucketed by average Likert consistency score. Models interpret the Likert range differently.
Huran Anno
Consistent
Inconsistent
benchmarks to improve label reliability, which can be demonstrated with high annotator agreement when multiple annotators are involved.
SummEdits Benchmark
Doss the edit introduce factual inconsistency?
Figure SUMMEDITS protocol diagram, three-step protocol to create summarization ID benchmarks. See Table for example samples produced by the protocol.
3.3 DialSummEval
The DialSummEval (Gao and Wan, 2022) bench- mark is a summarization evaluation benchmark cre- ated following the format of SummEval (Fabbri et al., 2021) for the domain of dialogue summa- rization. In DialSummEval, each (dia a alouee, summary) tuple is evaluated by three annotators, each assigning a Likert score (1-5) assessing the consistency of the summary. The authors of the benchmark report an agreement level of 0.67 Krip- pendorff's s alpha on the labels, indicating a moder- ate amount of agreement among annotators.
cls assign large proportions of samples from each bucket into consistent and inconsistent classes.
We argue that annotating the consistency of summaries using a Likert scale limits the quality and interpretability of the benchmark, as it is not evident to interpret the differences between scores, limiting reproducibility, which is reflected in the moderate Kripendorff' 's alpha. Instead, we favor framing factual consistency benchmarks detection task. In the detection task, identifying any factual inconsistency between the document and summary leads to an overall assessment of the summary being inconsistent. If no inconsistency is detected, the summary is consistent. The detection framing also allows for models to provide natural language explanations when identifying summary as inconsistent, which can be manually verified to confirm model reasoning ability.
We evaluate model performance in two ways: (1) correlation between model predictions and the average annotator score, and (2) we follow Laban et al. (2022a) to transform the annotation into a binary classification task, amenable to the balanced accuracy metric. Results summarized in Table 1. Echoing results on AggreFact, increasing model size leads to minor performance gains, with most LLMs underperforming specialized methods. In ab- solute terms, all methods struggle to achieve strong performance, with accuracies below 70%.
In the next section, we propose novel protocol to create factual consistency benchmarks, incorpo- rating lessons learned from cxisting benchmarks.
In Figure 2, we aggregate model predictions into 0.5-width buckets on the Likert scale. We find that most models achieve strong performance borderline buckets (l1.0, 1.5), [1.5, 2.0], 4.5). [4.5, 5.0J), assigning a vast majority of samples to the correct class (inconsistent for low buckets, con- sistent for high buckets). The borderline buckets ([2.0, 4.0)) however are less clear-cut: most mod-
4 SUMMEDITS Protocol
4.1
Design Principles
We set several design principles that help create higher-quality factual consistency benchmark:
PI. Binary Classification Task; In the bench-

---

Consistent Edited Summary
Incomsistent Edited Summary
4.2 Creation Procedure
The characters discuss ponder the consequences of bunishing Mar- cius, with Cominiis warning that his alliance collahoration with the Volscians will hring great danger to Rome.
The characters discuss the con- sequences of banishing Marcius, with Cominius warning that his alliance with the Volkcians Ro: mans bring great danger to Rome the Volscions, intity
We now describe the creation procedure for SUMMEDITS illustrated in Figure which sat- isfies the design principles stated above.
The procedure consists of three steps: (1) seed summary verification,
Kanipul stiii
(2) generation of summary edits, and (3) annotation of edited summaries.
introduced novel new, sim- ple, efficient data augmenta lion method that boosis improves the performances of existing GANs when training data limited and
introduced novel, simple, and elficient data sugmentation method that boosts the performances of cx- isting GAN' when training data limited aburddant and diverse. Ant onyn Swap
Seed Summary Verification. Benchmark cre- ators select a small collection of documents in a domain of choice, and a seed summary for each document, which can be human-written or model generated. An annotator answers two questions about each (document, seed summary) tu- ple: (a) "Are there any flaws with the summary? (fluency, format, etc.)", (b) "Is the summary factu- ally consistent with the document?". If the annota- tor identifies a flaw or an inconsistency, the tuple is filtered out (P2), otherwise, it proceeds to Step 2.
Employycs of the European Coun- mission are now forced lnstrneted to delere remove TikTok from their work devices, and delete get rid of froen their personal dlevicus if they have work-related apps appli: cations installed.
Employees of the European Cou- mission now foroed re- quired to delete TikTok from their work devices, and delete but should still remove from their per- sonal devices
if they have work related apps installed. [a Halluciin 1uc Mallucinated inniiiaeed ed Pact
A conversation between sales agent a potential client possible customer, The sales agent provides information on different home in- surance plans options and pricing, well as available discounts for clients with good credit scores and other factors.
A conversation between sales agent and a potential client. The sales agent provides information on different home insurance plans and, but not on pricing, as well as or available discounts for clients with good credit scores and other factors. Magat Lon Tesert
Editing Summaries. The second step consists in generating multiple minor edits of the summary, which might or might not affect the summary's 's sistency. This step can be carried out manually, or automatically with an LLM. Proposed edits should be atomic and localized, not entirely rewriting a novel summary. Table 3 gives examples of edits.
Table 3: Example edited summaries deletions, inser- tions- for from SUMMEDITS (domains top-to-botoom: Shakespeare Plays, SciTIDDR, News, Sales Call). Incon- sistent summaries are labeled with an Edit Type.
mark, a summary should either be labeled as inconsistent if any factual inconsistency is identified with the document or consistent otherwise, improve label interpretability.
Annotation of Edited Summaries. The anno- tator who completed Step I reviews each edited summary, assigning one of three labels: (a) consis- lent if an edit does not lead to an inconsistency, (b) inconsistent if the edit modifies the seed summary in a way that introduces a factual inconsistency, (c) borderline for any other case such as the edit mak- ing the summary unclear, or requiring subjectivity. Crucially, a single annotator should complete both Steps and 3, as once they have invested the time in reading the (document, summary seed) tuple, judging the consistency of edits is simpler task. We recommend including a large number of edits (e.g.. 30 edits) to maximize edit diversity (P4) and encouraging annotators to assign the borderline label they are unsure about any aspect of an edit to maximize reproducibility (P3).
P2. Focus on Factual Consistency: Summaries in the benchmark should be flawless on as- pects unrelated to consistency, such as fluency, coherence, and formatting, to avoid confound- ing effects on the quality of the benchmark.
P3. Reproducibility: Benchmark labels should not depend on annotator identity, and high an- notator agreement should confirm the validity of the benchmark, as well as estimate human performance on the benchmark.
A benchmark can be formed by retaining edited summaries that are labeled as consistent and incon- sistent and filtering out borderline cases.
P4. Benchmark Diversity: Inconsistency errors in the benchmark should represent wide range of errors in realistic textual domains, to increase understanding of model strengths and weaknesses, and better establish gaps in performance between models and human an- notators at factual reasoning, if there are any.
The procedure requires a small number of docu- ments and seed summaries which are derived into many edited summaries. This flexibility facilitates the creation of factual consistency benchmarks in application domains that lack such resources.

---

Domain
N
% Balance
IAA
cent news articles. We obtained news articles from the Google News top events in February 2023, se- lecting at most one per news source to increase coverage diversity (Laban et al., 2023). Seed sum- maries are extracted from article metadata.
News
819
39.2%
0.91
Podcast
500
32.6%
0.91
Billsum
853
42.3%
0.90
Samsum
664
36.4%
0.90
Shakespeare
814
46.4%
0.96
Podcast (Clifton al., 2020) We collected 40 podcast transcripts from Spotify's podcast sum- marization dataset's test set. We generated seed summaries due to low reference summary quality.
SciTLDR
466
31.1%
0.93
QMSum
431
42.5%
0.92
ECTSum
668
38.0%
0.96
Sales Email
29.2%
0.87
Sales Call
520
33.3%
0.93
BillSum (Kornilova and Eidelman, 2019) We collected 40 US bills and their summaries as seeds from the training portion of BillSum, a challenging dataset for summarization in the legal domain.
Overall 6,348
37.10%
0.92
Table 4:
Statistics of the ten domains included in the SUMMEDITS benchmark, including the number of samples (N), the percentage consistent summaries (% Balance), and the inter-annotator agreement (IAA).
SamSum (Gliwa al., 2019) We collected 40 dialogues and summaries from the training portion of SamSum, a common dialogue summarization datasct for messenger-like conversations.
5 SUMMEDITS Benchmark
5.1 Benchmark Creation
Shakespeare (Karpathy, 2015) We collected 40 scenes from Shakespeare plays from the Tiny Shakespeare corpus, each roughly 700 words long. We generated seed summaries automatically.
We implemented the SUMMEDITS protocol on ten realistic summarization domains explore the re- liability of the protocol. For five domains, seed summaries are automatically generated due to the lack or low quality of existing reference summaries. In such cases, we used GPT3.5-turbo and domain- specific prompts to generate seed summaries. We note that the quality of seed summaries is ultimately manually confirmed in step 1 of the protocol. GPT3.5-turbo for Step 2. We experimented with integrating multiple LLMs in the edit generation process, but prelim- inary results indicated that many LLMs were not successful at generating minorly edited summaries and often attempted to write entirely novel sum- maries, which led us to solely use GPT3.5-turbo. More this choice in Section 7.
SciTLDR (Cachola et al., 2020) We collected 40 research paper abstracts and corresponding TLDRs from the training portion of SeiTLDR, a dataset for scientific paper summarization.
QMSum (Zhong et al., 2021) We collected 40 document and seed summaries from QMSum, dataset for query-based meeting summarization.
For all domains,
ECTSum (Mukherjee et al., 2022) We collected 40 documents from the ECTSum dataset, a sum- marization dataset for the financial earnings call transcripts. Due low reference summary quality, we generated seed summaries automatically.
We hired two professional annotators compen- sated at a rate of $20/hour to perform Steps 1 and Three authors of the paper also participated in the annotation for quality control purposes. Ap- pendix has further detail on annotation proto- col and an overview of the annotation interface. We next introduce the ten domains included in the SU SUMMEDITS benchmark.
Sales Call & Email We generated 40 fictional sales call transcripts, 40 sales emails, and corre- sponding seed summaries using ChatGPT. These domains evaluate the protocol's validity with en- tirely synthetic textual data in targeted domains that lack pre-existing summarization datasets.
5.2 SUMMEDITS Statistics
Table 4 provides statistics of the SUMMEDITS benchmark. Each domain yielded between 400- 900 edited summaries, depending on the fraction of seed summaries that pass Step (58% overall pass rate) and the percentage of edited summarics that are filtered out as borderline Step 3 (around
News To avoid selecting documents that are the training corpora of evaluated models, we follow prior work (Goyal et al., 2022) and se- lect (document, summary) tuples from re-
"The prompts will he listed in our open-source relcase.

---

Model
Podcast
BillSum
SAMSum
News
Sales C
Sales
Shkspr
SciTL
QMSum
ECTSum
Avg.
54.9
59.5
61.7
55.0
54.5
55.7
SummaC
58.5
55.7
54.7
62.1
59.0
57.7
59.3
59.7
56.6
64.4
58.8
QAFactEval
61.0
54.4
66.3
74.6
68.5
64.2
61.9
67.5
62.4
72.9
63.7
Dav001
53.3
50.2
51.0
54.4
55.3
52.5
50.0
51.0
50.3
50.9
51,9
Cohere-cmd-XL
51.1
52.7
52.0
52.6
60.3
59.5
50.0
60.5
53.9
60.5
55.1
Vicuna-13b
52.6
50.8
63.0
$1.8
55.5
Claude v1.3
59.9
52.1
64.1
63.3
61.7
56.6
58.0
57.6
56.9
67.8
59.8
Dav002
56.4
53.9
57.1
61.9
65.1
59.1
56.6
64.6
60.6
66,2
60.1
Bard
50.0
58.3
61.3
72.8
73.8
69.0
58.4
66.1
53.9
73.1
63.7
PaLM2-bison
66.0
62.0
69.0
68.4
74.5
68.1
61.6
78.1
70.2
72.3
69.0
Duv003
65.7
89.9
67.5
71.2
78.8
69.4
69.6
74.4
72.2
77.9
70.7
GPT3.5-turbo
68.4
63.6
68.1
78.9
GPT4
83.3
71.1
82.9
83.3
87.6
80.1
84.6
82.4
88.0
82.4
GPT4 Oracle
90.2
85.5
86.3
88.3
91.1
83.55
96.6
86.3
89.9
91.7
88.9
Human Perf.
90.8
87,5
89.4
90.0
91.8
87.4
96.9
89.3
90.7
95.4
90.9
Table 5: Balanced accuracy of models on the SUMMEDITS benchmark. Top three are non-LLM specialized models, middle section are LLMs, bottom section reports a GPT4 oracle performance and an estimate human performance.
6%). In the five domains where seed summaries were generated by GPT3.5-turke 17.8% of the seed summaries were labeled as inconsistent, indicating that modern LLMs like GPT3.5-turbo struggle to remain consistent when summarizing documents.
of annotator work. The average cost of adding a domain to SUMMEDITS is around USD 300, within reach for NLP practitioners looking evaluate the model ability in their domain of choice. Authors of the FRANK benchmark (Pagnoni et al., 2021) - samples which are in Aggrefaact estimate that each FRANK sample required 30 minutes of annotator time. At similar annotator pay, the anno- tation of a new domain of similar size to ones in SummEdits would cost estimated USD 6,000: twenty times more. This cost analysis reveals the dual advantage of our protocol: by focusing the notation task on atomic edits, the cost drastically reduced and high reproducibility is maintained.
For each domain, the seed summaries of at least ten seed summaries were annotated by multiple an- notators, corresponding for each domain to at least 20% of the samples in the benchmark. In total, 1,419 of the 6,348 samples in SummEdits received multiple annotations, allowing us measure agrec- ment levels. When considering all three labels (con- sistent, inconsistent, borderline), Cohen's is Kappa in each domain varies between 0.72-0.90, averag- ing 0.82. When removing samples annotated as borderline by any annotator, the average Cohen's s Kappa rises to 0.92, empirically validating the importance of filtering out borderline samples create a reproducible benchmark.
5.3 SUMMEDITS Results
Table 5 reports the performance of specialized mod- els, LLMs with a zero-shot prompt. an oracle ver- sion of GPT4, and an estimate of human perfor- mance on the samples with multiple annotations.
The edited summaries have and average of 3.6 words inserted, and 3.5 words deleted. These edit statistics do not vary widely based on the consis- tency label, as consistent edited summaries have an average of 3.6 words inserted, 3.7 words deleted, and inconsistent edited summaries have 3.6 words inserted, 3.4 words deleted. These statistics that models could not rely on structural signals pre- dict the consistency of a summary, and required factual reasoning to accomplish the task.
Overall, model performance on the benchmark is low, with only GPT4 getting within 10% of human performance. Larger more recent LLMs perform better on the benchmark, is illustrated by the gradual improvements observed with each model generation the OpenAl model family.
PaLM2-Bisso Dav003, ChatGPT, and GPT4 are the only four LLMs that outperform the best non- LLM approach QAFactEval, providing evidence that most LLMSs are not yet capable to reason nut-of-the-boos about the consistency of facts.
In the final benchmark, 37% of summaries are consistent, approaching our objective of balanced benchmark to facilitate robust evaluation and mini- mize metric fluctuations (Luque al., 2019).
All three specialized models achieve their high- est performance in the news domain, unlike L.LM models. The specialized models are likely cali- brated to the news domain, which they are most fre-
The total annotation cost of SUMMEDITS is around USD 3,000, representing around 150 hours

---

Inconsistent Edit Type
Antonym Swap when a word or phrase is replaced by a word of opposite meaning, (3) hallucinated fact insertion, when novel fact is introduced the summary which is not supported by the docu- ment, and (4) negation insertion when any negator word (c.g., not, neither) which modifies summary meaning is inserted. Figure 3 provides example of each edit type in SUMMEDITS.
Model
EntMod
Anto
Hallu
Neg
DAE
52.0
53.0
52.9
53.9
SummaC
56.8
56.8
55.3
57.3
QAFaciEval
61.4
65.0
64.3
70.4
Dav001
50.0
50.9
50.8
53.7
Cohere-cmd-XX
53.7
55.8
55.5
63.8
Vicuna-13b
55.2
57.1
56.2
61.0
Claude v1.3
58.8
60.3
61.5
66.7
To annotate the entire benchmark, one author of the paper first manually annotated 200 samples of the dataset, which was used to evaluate several GPT4-based Zero-Shot and Few-Shot approaches. The best perfforrngn prompt provides the defini- tion of each edit type and canonical example of each, and it achieved performance of 0.85 F-1 and 0.92 recall, which was deemed sufficient for analysis purposes. GPT4 was used to annotate all inconsistent summaries in SUMMEDITS.
Dav002
58.3
61,4
62.4
72.0
Bard
63.2
65.3
65.6
71.3
PaLM2-Bison
67.0
70.0
71.7
80.3
Dav003
69.2
71.1
76.3
83.3
GPt3.5-turbo
70.7
70.6
74.2
79.7
GPT4
82.2
81.3
87.0
92.7
Average
61.4
62.9
64.1
69.7
Table Balanced accuracy of models on the SUMMED- ITS benchmark, broken down by type of factual error: Entity Modification (Ent Antonyms (Ant Hal- lucination (Hallu) and Negation (Neg) insertion.
Overall, 78% of inconsistent summaries contain an entity modification, 48% an antonym swap, 22% hallucinated fact insertion, and 18% a negator in- sertion. The distribution of edit types is highly influenced by the LLM and prompt used to pro- duce the edits in Step 2 of the protocol. Table 6 summarizes model performance by the edit type.
quently tested on (Goyal and Durrett, 2020; Laban et al., 2022a; Fabbri et al., 2022). This confirms the importance of creating multi-domain benchmarks measure model ability in realistic scenarios.
All models detect inconsistencies due to nega- tor insertions the best, sign that such errors are more discernable to models. Fact hallucinations are relatively harder to detect for non-LLM models but gradually become more evident to more perfor- mant LLMs. Finally, the entity modification and antonym error types generally see the lowest rate of detection by models across the board, perhaps due such edits modifying an existing consistent fact more nuanced way.
Some domains such as Shakespeare's 's plays or the legal BillSum are more challenging to the ma- jority of models, with the latter seeing no model score higher than 71.1% Yet, factual reasoning in the legal domain is an important application area of NLP (Chalkidis et al., 2020; Shen et al., 2022).
We experiment with an oracle setting in which we append the seed summary to the end of the in- put document and input the concatenation to the model. The seed summary serves as an information scaffold, enabling the model to view modifications between the seed and edited summaries. GPT4 achieves significant boost under the oracle setting, with the model performing within 2% of human performance. This confirms that high model per- formance SUMMEDITS is attainable and that the challenge lies in aligning the facts of the edited summary with the document, without knowing that it has been edited.
5.5 Number of Edits Effect
In SUMMEDITS, it is common for the LLM to in- troduce multiple edits in each of its candidate sum- maries, can be seen in the examples in Table 3, in which each edited summary contains multiple inserted and deleted words. In Appendix D, we analyze the effect of the number of edit types on model performance. In short, as the number of edit types in a summary increases, most models see sizable performance improvements, with average performance increasing from 59.2 to 74.1 when the number of edit types goes from 1 to 4.
5.4 Edit Type Analysis
We annotated each inconsistent sample in SUMMEDITS with tags of edit types.
The four edit types are: (1) Entity Modifica- tion in which an entity or phrase in the summary has been changed a Seaning-alteringg way, (2)
This analysis confirms the perspective the task in SUMMEDITS corresponds to a detection task: as the number of introduced enrors increases, model

---

performance increases as there is generally more evidence of inconsistencies for the models de- tect. In turn, future work looking to create more challenging benchmarks using a similar protocol can focus on editing summaries with a single edit type.
has an effect on the benchmark which could favor subset of LLMs most similar the one used for
benchmark creation. Initial attempts to use pool of LLMs to produce edits were unsuccessful as we found that only ChatGPT and GPT4 were currently capable of following editing instructions that do not fully rewrite summaries. Future iterations on simi- lar benchmarks should consider including diverse pools of LLMs in benchmark creation processes to avoid model-specific bias. Beside the edit sum- maries, we leveraged ChatGPT to generate the seed summaries in five of the ten domains in SUMMED- ITS, due to the low-quality or non-existence of Noman-writted summaries. All seed summaries are manually inspected by our annotators, and we did not find a gap in model performance dependent on the origin of the seed summaries.
6 Conclusion
this work, we explore the capabilities of LLMs to act as factual reasoners through the lens of fac- tual evaluation in text summarization. As part of this analysis. we uncover and discuss shortcomings of existing benchmarks. Using those insights we develop new protocol for creating inconsistency detection benchmarks, which we implement in 10-domain benchmark called SUMMEDITS. The SUMMEDITS benchmark is highly reproducible and more cost-effective per sample than previous benchmarks. Our experiments show that the bench- mark is challenging for most current LLMs, with the best-performing model, GPT-4, still 8% below estimated human performance. We believe that SUMMÈDITS can serve valuable tool for eval- uating LLMs' abilities reason about facts, detect factual errors and promote more reliable NLG sys- tems. We encourage LLM developers to report their performance on the benchmark.
Beyond Binary Classification. SUMMEDITS fo- cuses binary classification formulation of fac- tual reasoning (i.e., determining whether a sum- mary consistent/inconiiseen))) Binary classifica- tion has multiple advantages, including the ability to benchmark both generative and non-generative models, requiring limited adaptation of previous systems, and supporting well-established evalua- tion metries such as balanced accuracy. However, the edit-based protocol of SUMMEDITS could be beneficial in instantiating more advanced factual inconsistency tasks. For example, SUMMEDITS could be modified into an "error localization" task which would require models to identify edit spans that render the summary inconsistent, or an "error correction" task, which would require a generative model to undo problematic edits, removing edit spans that lead to factual errors. These more ad- vanced task formulations would require crafting reliable metries, which was out of the scope of the current project.
7 Limitations
Why not fix existing benchmarks? In Section 3, analysis reveals limitations with existing bench- marks that in theory can be fixed to yield improved versions of known benchmarks. The analysis we performed however only helps us invalidate a sub- set of samples in opportunistic way. by looking samples where benchmark labels and GPT4 dis- agree. However, this methodology cannot help us efficiently correct or confirm all samples, and improving existing benchmarks would require re- annotating a large portion of the benchmarks, and we do not have a guarantee that new annotations would improve on previous ones. By designing a new protocol for sample annotation that relies on clear, atomic edits, simplify the annotation process, improving reproducibility.
Evaluating Summarizers. Previous annotation efforts in factual consistency of summarization were in part collected to evaluate which summa- rization models are least likely to generate fac- tual inconsistencies (Falke et al., 2019). Since the summaries in SUMMEDITS are synthetic modifi- cations of summaries, the benchmark cannot di- rectly provide insights on summarizers and their ability to remain consistent, and the main purpose of SUMMEDITS is measure L.M ability to rea- son about facts, and detect factual inconsistencies in text pairs. Future work could explore using meth-
Effect of LLM in benchmark creation. Step of the protocol described in Section 4 relies on an LLM to generate many edits of the seed summary, which are subsequently manually annotated and in- cluded in the benchmark. The choice of LLM likely

---

ods such as Near-Negative Distinction (NND) (La- ban et al., 2022b) adapt SUMMEDITS into a set of tests to evaluate summarizer performance, and model ability to avoid generating inconsistent sam- ples in the first place.
ing exploration through an interactive questionnaire. arXiv preprint arXiv:2205.02.77
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077
Build Your Own Benchmark. The initial release of SUMMEDITS consists of ten diverse domains we hope span common summarization domains. The current benchmark is however limited, as it only includes documents and summaries in English, and mostly limits document length to below 2,000 words. We have however shown that the protocol can be adapted to widely different textual domains- from US legal bills to Shakespeare plays and pro- duce domain-specific benchmarks at low cost. We hope that others will adopt and adapt the protocol to new domains, languages, and NLP tasks.
Tom Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-1991.
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S Weld. 2020. Tidr: Extreme summarization of sci- entific documents. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 4776.
Ethical Considerations
Shuyang Cao and Lu Wang. 2021. Cliff: Contrastive
learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649.
The models and datasets utilized in the project pri- marily reflect the culture of the English-speaking populace. Gender, age, race, and other socio- economic biases may exist in the dataset, and mod- els trained on these datasets may propagate these biases. Text generation tasks such as summariza- tion have previously been shown to contain these biases.
Iliss Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ton Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. ArXiv, abs/2010.02559.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
In Section 3 and Section 5, we recruited pro- fessional annotators to perform labeling with re- spect summaries' factual consistency label or LLM reasoning explaining factual inconsistencies. We ensured to remunerate the participants fairly ($20/hour). Participants could communicate with us to voice concerns, could work at their own pace, and choose to stop working on the project at any time. Finally, we ensured to anonymize the an- notations by not including personally identifiable information in any version of the dataset (annota- tor identity is instead marked annotatorl, annotator2, etc.).
Ann Clifton, Aasish Pappu, Sravana Reddy, Yongze Yu, Jussi Karlgren, Ben Carterette, and Rosie Jones. 2020. The spotify podcast dataset. arXiv preprint arXiv:2004.0427
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue sys- tems: The begin benchmark. Transactions of the Association for Computational Linguistics, 10:1066- 1083.
In our work, we relied on several datasets as well as pre-trained language models. We explicitly verified that all datasets and models are publicly released for research purposes and that we have proper permission to reuse and modify the datasets.
A lexanderr R Fabbri, Wojciech Kryšciñsski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguisties, 9:391-409
Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Im- proved qa- qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2587-2601.
References
Negar Arabzadeh, Ali Ahmadvand, Julia Kiseleva, Yang Liu, Ahmed Hassan Awadallah, Ming Zhong, and Mi- lad Shokouhi. 2022. Preme: Preference- meet-

---

Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Pruceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 22 12--2220..
Wojciech Kryšciński. Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP), pages 540-551.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302 04166.
Wojeiech Kryšciński, Bryan McCann, Caiming Xiong. and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346.
Mingqi Gao and Xiaojun Wan. 2022. Dialsummeval: Revisiting summarization evaluation for dialogues. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5693-5709.
Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022a. Summac: Re-visiting nli- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics, 10:16377
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. EMNLP-IJCNLP 2019, page 70.
Philippe Laban, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022b. Near-negative distinetion: Giving second life to human evaluation datasets. In Conference Empirical Methods in Natural Lan- guage Processing.
Tanya Goyal and Greg Durrett. 2020. Evaluating factu- ality in generation with dependency-level entailment.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209.11 12356.
Philippe Laban, Chien-Sheng Wu, Lidiya Mu- rakhovs' Ka, Xiang 'Anthony' Chen, and Caiming Xiong. 2023. Designing and evaluating interfaces that highlight news coverage diversity using discord questions. In Proceedings of the 2023 CHI Confer- ence on Human Factors in Computing Systems, pages 1-21.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. True: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3905-3920.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang 7hu. 2023. Gpteval: Nig evaluation using gpt-4 with better human align- ment. arXiv preprint 2xXiv:2303.1660
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization.
Or Honovich, Leshem Choshen, Roce Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q2:: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 7856-7870.
Amalia Luque, Alejandro Carrasco, Alejandro Martin, and Ana de Las Heras. 2019. The impact of class im- balance in classification performance metrics based on the binary confusion matrix. Pattern Recognition, 11:216-231.
Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang. Jun Xie, and Yue Zhang. 2020. What have we achieved on text summarization? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 449-
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 1906-1919.
Raghav Jain, Anubhay Jangra, Sriparna Saha, and Adam Jatowt. 2022. A survey medical document sum- marization. ArXiv, 1b./221..01669.
Rajdeep Mukherjee, Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde. Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Gan- guly, Saptarshi Ghosh, et al. 2022. Ectsum: A new benchmark dataset for bullet point summarization of long carnings call transcripts. arXiv preprint urXiv:2210.12466
Andrej Karpathy. 2015.
char-rnn. https:// gi github.c.. www ka rwwthaaayaaao char-rnn.
Anastassia Kornilova and Vladimir Fidelman. 2019. Billsum: A corpus for automatic summarization of us legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48-56.
Sharan Narang and Aakanksha Chowdhery. 2022. Path- ways language model (palm): Scaling to 540 billion parameters for breakthrough performance.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13977
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 4812-4829.
Alex Wang. Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin c! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624 643.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyil- maz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meet- ing summarization. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 5905-5921.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Questeval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594 6604.
Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022. Multi- lexsum: Real-world summaries of civil rights law- suits at multiple granularities. arXiv preprint arXiv:2206. 10883.
A Model Access Detail
We experiment with wide range of models. For each model, we specify its model card, and how it was accessed.
Liyan Tang. Tanya Goyal, Alexander R Fabbri, Philippe Laban, Jiacheng Xu, Semih Yahvuz, Wojeiech Krys- cińskii Justin F Rousseau, and Greg Durrett. 2022. Understanding factual emrors in summarization: Er- rors, summarizers, datasets, error detectors. arXiv preprint 2rXiv:2205.12854.
Non-LLM models. The three specialized models SummaC3 DAE and QAFactlival were im- plemented through their online public repositories, and run locally on a multi-GUU machine (with 2 V-100 GPUs).
Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu. Ying Ding, Greg Durrett, Justin Rousseau, et al. 2023. Eval- uating large language models on medical evidence summarization. medRxiv, pages 2023-04.
Models. We experimented with five open-source LLM models:
Open-source
LLama-13b (Touvron et al., 2023), Alpaca-
Rohan Taori. Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpaca: strong, replicable cnatruction-olllwwiin model, Stan- ford Center for Research Foundation Models. https://orfm. stanford. 3/alapaaa html.
13b (Taori et al., 2023), Dolly-V2-12b cdatabricks/dolly (ioolly-2212bb) 13b (Chiang et al., 2023), and MosaicML's MPT-7b-chat (Team, 2023). All models were accessed through the public, online demonstration of LMSys.org". Model responses were collected between April 15th, 2023, and May 15th, 2023.
Vicuna-
MosaicML NLP Team. 2023. Introducing mbt-7b: A new standard for open-source, ly usable llss. Ac- cessed: 2023-05-16.
hhttps://github.com com/ hwwp://///o//////oooo/ooo/ooo wrrourro/ooooooooooooooooooooo summac hhttps://github.com ccw.tagoyal// factualityyddtassett
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 2rXiv:2201.082.
'https://github wwm/ salesforee/ @Aracteval
/attps://chat..
aecchat.mmyss....m maays.org/

---

Google Models. We experiment with two Google models, the Bard (Thoppilan et al., 2022) which we accessed through web-based interface' which does not specify an exact model card, but model responses were collected between April 15th, 2023 and May 15th, 2023. Second, the PaLM-v2-bison model (Narang and Chowdhery, 2022) (model card text bisonn001), which was accessed through the Google Cloud VertexAl API.
if the explanation correctly identifies and explains one or more factual
Entirely Correct:
inconsistencies the summary.
Partially Correct: if the explanation pro- vided contains several elements and least one of them correctly identifies and explains a factual inconsistency in the summary,
the explanation given does not directly relate a factual inconsistency be- tween the summary and the document,
Unrelated:
Anthropic Model. We collected outputs of the Claude V1.3 model (model card: claude-vil vonodeevv.3)) the latest and largest Anthropic model the time of publication, using the official API hosted by Anthropic
if the explanation given does not correctly identify a factual inconsistency the summary, for example, making a logical error.
Incorrect:
We collected outputs of Cohere's command-xlarge model, the latest and largest Cohere model at the time of publication, using the official API hosted by Cohere".
Cohere Model.
An example for each type of explanation was provided during onboarding. Annotation was per- formed in batches, and the first two batches of annotation by the annotator were reviewed by the authors of the paper. Incorrect annotations were dis- cussed, allowing the annotator to better understand edge cases of the task, and modify their annotation in the first batches. Each annotator could commu- nicate with one of the authors to discuss edge cases and maintain a common understanding of the task. Annotators could not communicate with each other.
OpenAl Models. We collected outputs for eight OpenAl models. Six models from the GPT-3 family: Ada001 (text-ada--011,
Bab001
(text-babbage-001)
Cur001 (text-curie-001)
Dav001
(text-davinci-001), and Dav003
Dav002
We also include
C SUMMEDITS Annotation Guidelines
GT3.5-turbo 1ept-3.5-turbe. and GPT-4
(gpt-4). All models were accessed through OpenAl's official AP
We hired two professional annotators complete the annotation of Steps 1 and of the SUMMEDITS protocol (see Section 4). The annotators were com- pensated at $20/hour. They received onboarding documentation that introduced them to the task and used the interface shown in Figure 3.
B Explanation Annotation Guidelines
We hired professional annotator to complete the annotation of model-generated explanations for AggreFact. The annotators were compensated $20/hour. They received onboarding documenta- tion that introduced them to the task, and provided the following definition for each type of explana- tion:
Annotators were first assigned 10 warm-up seed summaries, each with roughly 30 edited summaries, which had been pre-annotated by the authors of the paper. The authors reviewed the completed warm- up exercises, and strong agreement level on the warm-up task with both annotators was observed. Annotators could communicate with one of the authors of the paper to discuss any edge case or domain-speciffe question. For example, the annota- tion for the QMSumm domain required additional instructions due query- -fccusd formulation of the task, and instructions were communicated on how deal with the "query" element when eval- uating summaries. Namely, during Step of the protocol, participants were asked to additionally judge whether the summary accurately responded
No Explanation: If the model did not provide any explanation. (For example just saying: "The summary is inconsistent").
'nttps://bard..goon le. com/ 'https://githabb com/ anthropics/
anthropic-sdk-pyrhho.
httt:sss///ccc..cheeee com/ does/ tte-cohezre-latfoor
"https:///' cwm/ooppoooooc
cow.://ea///oaaaooooooooooo opena1
opeai-python

---

Document:
Original Summary:
Simulation useful tool situations where training data for machine learning models is costly anmotate or even hard to acquire. In this work, propose
propose algoritim automatically adjusts parameters of a simulation engine to generate training data for neural network such that validation accuracy maximized.
Task
reinforeement learning- based method for automatically adjusting the purameters any (non- differentiable) simulator, thereby controlling distribution synthesized data in order to muximize accuracy model trained on that data, In contrust to prior that hand-crafts these simulation parameters adjusts only of the available parameters, our approach fully controls the simulator with the underlying goul maximizing accuracy, rather than distribution or randomly generating a large volume of data. We find that approach (i) quickly converges the optimal simulation parameters controlled experiments indeed discover good sets of parameters image rendering simulator in actual computer vision applications.
Is any of the information the summary not present in the document?
Are there any other issues with summury? (incomplete sentence, formatting, etc.)
O Yes X No
Submil
Task 2:
Mudified Summaries:
We propose algorithm that automatically adjusts parameters simulation engine to generate training neural network such that validation accuracy is maximized only slightly improved.
mimicking
Inconsistent O Consistent Borderline
can
We propose algorithm that automatically adjusts parameters of a simulation engine to generate training testing data for neural network such that validation accuracy maximized. Inconsistent O Consistent
We propose an algorithm that automatically adjusts changes parameters of simulation engine generate training data for neural network in such way that validation accuracy maximized.
Inconsisteat
Consistent
Borderline
Figure 3: Two-column annotation interface used to annotate samples in the SUMMEDITS benchmark. Participants could read the document on the left-hand column. Once they completed Task the right-hane hand column, the second annotation task became visible.

---

#Distinct Edit Types
Model
2
DAE
50.2
53.5
55.4
64.9
SummaC
58.2
56.3
57.6
67.3
QAFactEval
59.4
63.7
72.3
76.5
Dav001
50.0
50.5
53.9
63.1
Vicuna-133
52.8
57.0
60.2
58.5
Cohere-emd-Xi.
50.0
55.9
63.7
70.0
Claude v1.
57.5
60.6
65.4
64.3
Dav002
56.3
61.2
69.4
81.7
Bard
61.0
64.9
72.4
73.4
PaLM2-Bison
66.1
69.5
79.6
69.4
ChatGPT
68.5
71.4
82.0
86.6
Dav003
65.3
72.0
85.8
88.8
GPT4
81.0
83.0
92.0
94.3
Average 59.2
62.5
69.2
74.1
Table 7: Relationship between the number of edits types in the summary and balanced accuracy of models on SUMMEDITS. Models generally perform better the number of introduced edits in summary increases.
the query, and otherwise mark summaries as inadequate.
D Number of Edits Effect
Using the labels of edit types generated in Sec- tion 5.4, each edited summary labeled as inconsis- tent receives between one and four edit types. We group the summaries based the number of dis- tinct edit types that they contain, and report results on this axis in Table
In general, we find that as the number of edit types present a summary increases, the majority of detection models (both LLM and specialized) see sizable performance improvements, with an average performance of 59.2 for summaries con- taining single error type, compared to 74.1 for summaries containing all four error types.


Reviews:
[{"review_id": "3Z9rL9n0Mq", "reviewer": "Reviewer_eGPn", "summary": "", "questions": "Provide some further discussion on how this work is more resource effective than fixing existing benchmarks that have been validated for annotator labels. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work.\nRegarding the use of a single LLM to generate seed summaries. We did not use LLM-generated seed summaries for all of the 10 domains in the SummEdits benchmark.\n- We prioritized the use of human-written reference summaries, which we used for five of the domains in the benchmark. For the other 5 domains, there were no high-quality reference summaries available, and we used GPT3.5-turbo to generate the seed summaries. We note that all seed summaries were manually inspected by annotators in Step 1 of the protocol, and low-quality summaries were filtered out.\n\n- Our experiments did not reveal that this choice of auto-generated seed summaries in some domains led to qualitative differences in the benchmark: for instance, one could have expected that LLMs would perform better on domains where seed summaries were auto-generated, but this in not observed in Table 5. For this reason, we decided to maximize domain coverage of the benchmark and include domains of interest to the summarization community (Sales Call, Creative Domain) by leveraging an LLM to generate seed summaries.\n\n- We will add a paragraph on this point in our Discussion, and direct participants to use the five domains with human-written references if they want to remove auto-generated seed summaries from their analysis.\n\n- Another possible observation could have been that the model used to generate seed summaries (GPT3.5-Turbo in our case) would have an unfair advantage in the benchmark. Although we do not have a way to measure whether a model is advantaged under the hood, we do observe that some models are able to outperform GPT3.5-Turbo performance on the benchmark, simply due to better factual reasoning abilities (for example GPT4). Since the original submission, we have conducted experiments with LLMs that have been published since then, and have found that other models (such as Claude 2.0) are able to outperform GPT3.5-Turbo and achieve performance only a few points below GPT4. This new finding confirms that models can achieve strong performance on the benchmark based on their factual reasoning abilities, even if they were not used in the benchmark creation process. We will add this new finding to the paper, and provide these discussion points.\n\nRegarding the resource effectiveness compared to fixing existing benchmarks. This point is discussed in the “Why not fix existing benchmarks?” paragraph of the Discussion. In short, the LLM-enabled analysis we performed in Section 3 to correct AggreFact samples can only be applied to a subset of the samples (samples where an LLM disagrees with the reference label), and cannot be used to fix existing benchmarks holistically. Therefore, fixing existing benchmarks would require re-annotation (which could be ~20x costlier per sample than our protocol). Another limitation with fixing existing benchmarks is that they often rely on summaries from old models (such as Pointer Generator) which make easier-to-detect mistakes. In essence, fixing existing benchmarks would be costly, and would not necessarily yield a challenging benchmark for the community, and we opted to create a new benchmark that is effective in terms of cost, and challenging to current detection models.\n\nWe will include the missing reference provided by the reviewer, as it is related and relevant to the benchmark protocol we proposed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "SUMMEDITS presents a challenge for all models evaluated, with only four LLMs outperforming the specialized model, and human performance outperforming all models. This analysis reveals the potential for LLMs as part of dataset creation and to identify factual errors. ", "reasons_to_reject": "The LLM used to generate seed summaries could influence the objectivity of the benchmark as an evaluation tool. ", "questions_to_authors": "Provide some further discussion on how this work is more resource effective than fixing existing benchmarks that have been validated for annotator labels. ", "paper_topic_and_main_contributions": "This paper proposes a new protocol for inconsistency detection benchmark creation with a 10-domain benchmark called SUMMEDITS. The protocol involves manually verifying the consistency of a set of seed summaries and subsequently generating numerous edited versions of these summaries. This new benchmark also has high annotator agreement and is reproducible.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://arxiv.org/pdf/2302.04434"}, "extra_text": {}}, {"review_id": "YOksabBaFH", "reviewer": "Reviewer_hL41", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our work.\nRegarding the use of mainly a binary classification task to measure LLM performance, we agree that this is indeed an important choice in our work, which has implications.\nThere are indeed other related tasks in summarization factual consistency which might be more complex and offer insights into LLM abilities, these include (a) error localization (specifying which portion of a summary contains a factual inconsistency), (b) error explanation/justification (in which a model would generate a free-text explanation for a factual inconsistency, verifying not only that the model’s label is correct, but that the model is able to explain what the error is), (c) error correction (in which a model might rewrite inconsistent summaries into factually consistent summary while minimally modifying the content).\n\nThe three tasks described above are all more advanced than the binary classification task we implemented in our work, but the choice to focus on classification offers three main advantages:\n1. Using a classification task allows us to compare generative models (LLMs) and non-generative models (SummaC, DAE and QAFactEval), which offers a wider range of comparison, and helps us understand how LLMs compare to specialized models (and in particular find that most LLMs lag these specialized models). The specialized models are not configured to perform the more advanced task formulations.\n2. Classification comes with well-established metrics (Balanced Accuracy in our case), which make the future evaluation of models automated and reliable. More advanced tasks are more complex to evaluate, and could require manual annotation in the evaluation (for instance in the error explanation/justification task), which would limit the usability of the benchmark. Because we want to build a reusable benchmark, proposing an automatic and high-quality evaluation to accompany the benchmark is important.\n3. Because binary classification has been the most common task formulation in inconsistency detection in summarization, it will allow more researchers in the community to test their methods on our benchmark without requiring adaptation, making the benchmark more accessible.\n\nDespite the fact that we focus on a relatively simple task formulation, we find that many models (including modern LLMs) achieve close to random performance and that there remains a large gap between human performance and top-performing LLMs, providing evidence that the SummEdits benchmark is challenging.\n\nWe believe however that the edit-based protocol of SummEdits could be beneficial to instantiate more advanced factual inconsistency tasks. For example, SummEdits could be reused in an “error localization” task in which the model would have to identify edit spans that led to the inconsistency (since these are already labeled), or for an “error correction” task, in which one could observe whether the models remove the edit spans that led to an error. This would likely require the crafting of reliable metrics, which was out of the scope of the current project. We will add a paragraph to our Discussion to relay this future direction, and discuss more advanced task formulations that could be implemented based on the SummEdits benchmark we release.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clear explanation of the benchmark creation process. The paper is easy to follow.\n- Solid experiments which comprehensively compare the performances of most of the current popular LLMs.\n- High reproducibility and diversity, which is crucial for creating a fair and generalized benchmark.", "reasons_to_reject": "- The benchmark mainly uses binary classification task to measure the LLMs. I am curious that whether there are other more challenging tasks can also be integrated into the benchmark to further improve the diversity.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new benchmark to test the ability of LLMs in detecting factual inconsistencies, which is more challenging and is 20 times more cost-effective per sample than previous benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c3eoxoAjTk", "reviewer": "Reviewer_jN9Y", "summary": "", "questions": "1. How different from the original summary are the edited summaries on average? Was there a threshold above which you would reject an edited article for being to different?\n2. Can you describe the process of estimating human accuracy in more detail? Specifically, the proportion of data points that were used for each domain\n3. What was the prompt given to the oracle version of GPT-4? Was it the same prompt as the other models but with the original summary added, or was there additional instruction?\n4. Why did you try to use data that would not have been seen in training for news data but not for any other domains?\n5. Do you assess other qualities of the edited summaries? Is it possible that there are differences in coherence/fluency/relevance between consistent and inconsistent versions and this is what models pick up on?\n6. Relatedly, do you measure the average number of edits made to inconsistent models vs consistent summaries? Could it be that edits are identifiable and models have learned that consistent versions have more/less edits?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their in-depth review of our work.\nWe agree with the reviewer that the inclusion of TRUE as a comparison to SummEdits would be beneficial to the readers, and will add a Section comparing SummEdits data statistics to previous benchmarks (SummEval, TRUE), offering a comparison in terms of size, domains and task covered.\n\nRegarding the protocol and annotation procedure, we thank the reviewer for their careful readthrough, and have the following comments:\n- We used ambiguous wording when describing the communication protocol between annotators and authors of the paper: “The annotators were added to a Slack channel with one of the authors”. More precisely, each annotator was added to an *individual conversation* with the first author of the paper, and not to a common Slack channel (the term Slack channel here was used incorrectly). The two annotators we hired did not know each other and did not communicate, and were hired on the professional recruiting platform UserInterviews. We agree that the two annotators communicating during annotation would undermine the reliability of the results, and confirm that this was not the case.\n- Besides the two annotators and the paper’s first author, two additional authors participated in the annotation which was used in the agreement level analysis. These two authors did not communicate with the professional annotators.\n- We believe these details were not clear in our current draft and will be corrected and expanded to include the details listed above.\n\nQuestion 1: We computed that on average, edited summaries have: 3.64 words inserted, and 3.48 words deleted. When breaking down by label: (1) for consistent edited summaries (3.69 words inserted, 3.69 words deleted), (2) for inconsistent edited summaries (3.60 words inserted, 3.35 words deleted). These statistics show that the summaries are minorly edited and that the amount of editing is roughly equivalent in consistent and inconsistent summaries. We will add these statistics to the dataset description section. We did not have a strict maximum number of words to be added or deleted but filtered out samples generated by the LLM that had a char-level Levenshtein string alignment of less than 0.7 to the seed summary, as we found that these were typically entirely rewritten summaries, rather than edited summaries based on the seed summary. We will add this detail in the procedure description. \n\nQuestion 2: To estimate the agreement level within each domain, samples of ten to thirty seed summaries, and all their accompanying edited summaries were doubly annotated, corresponding to at least 20% of the samples in each domain. In total, 1,419 of the 6,348 samples in SummEdits (~22%) received multiple annotations and were used to compute the agreement levels presented in Table 4.\n\nQuestion 3: Regarding the prompt of the oracle GPT4 model, we used the same prompt, and concatenated the seed summary to the document. This ensured that the model had access to the information in the seed summary, and could more easily view the edits and potential inconsistencies with the edited summary. We agree with the reviewer that this detail is lacking from the draft, and will add it. We will also note that the prompt could be modified to explicitly differentiate between the document and the seed summary, which we did not attempt, but could yield additional gains in performance.\n\nQuestion 4: In the news domain, the timestamp of documents allows for a simple way to control whether LLMs are likely to have seen the data during training, and since we could control this aspect, we ensured to use recent articles. For other domains (such as Shakespeare) it is likely that the models are trained on portions of the documents in SummEdits, and that the models know the plots of Shakespeare's plays, which likely affects their performance positively, but since we do not have access to exact training corpora of many LLMs, it is challenging to assess exactly. We will add this justification to the domain selection section.\n\nQuestion 5: In Step 1 of the process, annotators verify that the summaries do not have any flaws related to fluency, formatting, coherence, etc. However, we do not perform such a check on each edited summary. We instead rely on the fact that edits to the seed summary are minor and modify on average 3-4 words in the summary. Since the seed summary does not exhibit issues in fluency, formatting, or coherence, the edited summaries typically do not either. From the annotations performed by the authors of the paper, none of the edited summaries (either consistent or inconsistent) exhibited a lack of fluency, coherence, or issues in formatting, and LLM is instructed to target its edits to affect the consistency of the summary. We will add this explanation to the benchmark creation procedure.\n\nQuestion 6: See reply in Question 1. On average, consistent and inconsistent edited summaries see almost identical numbers of words inserted or deleted from the original seed summary. Therefore, it is unlikely that the model could leverage a difference in editing amount to improve performance. Furthermore, none of the models were trained on SummEdits, and therefore could not have learned patterns within the benchmark. However this is indeed a confounding factor that could affect the validity of the benchmark, and we will add the statistics to confirm that such bias is absent from the benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Detecting factual inconsistency is becoming a more and more important task, and a benchmark to measure model ability at this would be a valuable contribution. While this is proposed specifically with LLMs in mind, smaller, more specialized approaches could also benefit from this dataset.\n\nThe initial quality filter that removes summaries that annotators judge to have fluency/coherence issues is a good step, and the annotation procedure appears sound for the most part.\n\nThere are lots of model comparisons from models of different families and sizes indicating that this dataset is sufficiently challenging to be informative.\n\nWhile I have some questions and concerns about various aspects of the paper, if the authors are able to adequately address these points, I would be inclined to raise my scores.", "reasons_to_reject": "This work should include a comparison to the TRUE dataset (Honovich et al., 2022), which aggregates factuality assessment datasets across summarization and other text generation tasks. \n\nI do find some issues in the annotation of the dataset. The entire dataset was annotated by only two people who were allowed to communicate both with each other and the authors of the paper during annotation. The reported inter-annotator agreement is quite high, but because of this, these scores are likely to be inflated due to communication between annotators and between annotators and authors throughout the process. It would have been a better practice to employ more than two annotators and not allow them to communicate with each other in order to get more accurate inter-annotator agreement scores. The authors argue that the high inter-annotator agreement scores indicate good potential for reproducibility and extending this task, but they may actually only indicate this if future annotators are also allowed to talk to each other and the authors. While the annotation questionnaire details appear clear to me, there is also no information on what criteria the authors applied to select qualified annotators or where annotators were hired from.", "questions_to_authors": "1. How different from the original summary are the edited summaries on average? Was there a threshold above which you would reject an edited article for being to different?\n2. Can you describe the process of estimating human accuracy in more detail? Specifically, the proportion of data points that were used for each domain\n3. What was the prompt given to the oracle version of GPT-4? Was it the same prompt as the other models but with the original summary added, or was there additional instruction?\n4. Why did you try to use data that would not have been seen in training for news data but not for any other domains?\n5. Do you assess other qualities of the edited summaries? Is it possible that there are differences in coherence/fluency/relevance between consistent and inconsistent versions and this is what models pick up on?\n6. Relatedly, do you measure the average number of edits made to inconsistent models vs consistent summaries? Could it be that edits are identifiable and models have learned that consistent versions have more/less edits?", "paper_topic_and_main_contributions": "This paper introduces a benchmark for measuring LM ability to detect factual (in)consistency in summaries. The authors construct this benchmark through a dual LLM, human pipeline wherein they select a set of summaries that annotators judge to be high quality (fluent, coherent, etc), ask an LLM to create several edited versions of the summary, and ask the same annotators to mark these edited summaries as factually (in)consistent. They perform this process for summaries in 10 domains. They assess a variety of LLMs and smaller models on the benchmark and find that smaller models trained specifically for factual consistency evaluation tend to outperform all but the largest models, while no model performs better than estimated human performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "* [Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://aclanthology.org/2021.emnlp-main.619) (Honovich et al., EMNLP 2021)\n\n* [TRUE: Re-evaluating Factual Consistency Evaluation](https://aclanthology.org/2022.naacl-main.287) (Honovich et al., NAACL 2022)"}}]

Author rebuttal:

Meta-review: This paper creates a new benchmark dataset for assessing factual accuracy in summarization. They show that the task is challenging for LLMs. 

Both reviewers hL41 and jN9Y agree on the soundness of the paper and the authors have addresses or promised to address the comments. In the rebuttal the authors seem to have addressed the concerns of eGPn.

This resource seems valuable to the community and could be quite impactful in judging new LLMs.
Decision: Accept-Main