Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
Year: 2023

Abstract:
The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems’ outputs. However, the everevolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (preTransformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.1

heading: 1 Introduction
text: Although abstractive summarization systems (Liu and Lapata, 2019; Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020) have improved dramatically in recent years, these models still often include factual errors in generated summaries (Kryscinski et al., 2020; Maynez et al., 2020). A number of metrics have emerged to detect factuality errors, including methods based on sentence entailment (Kryscinski et al., 2020), finer-grained entail-
1Code and data are available at https://github.com/ Liyan06/AggreFact.
ment (Goyal and Durrett, 2020; Zhao et al., 2020), question generation and answering (Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021), and discrimination of synthetically-constructed error instances (Cao and Wang, 2021). Despite recent analyses (Pagnoni et al., 2021; Laban et al., 2022), reliably comparing these metrics remains difficult.
In this paper, we provide a new benchmark that allows for finer-grained comparison between different factuality systems. We aggregate 9 existing annotated factuality datasets to create our benchmark AGGREFACT. We stratify it according to the underlying summarization model, categorized into FTSOTA, EXFORMER and OLD based on their development timeline (see Section 2). First, we ask: do factuality metrics perform equally well at identifying errors from state-of-the-art summarization models and from earlier models? For nine recent factuality metrics, including recent ChatGPT-based metrics, we show that metric performance varies substantially between different categories of summarization models. Most importantly, we found that the standard way of reporting improvements on category-agnostic benchmarks can be misleading, as most of these gains are on the OLD or EXFORMER subset of the data which are less important to detect. On summaries generated by FTSOTA models, we found that there is no single metric that is superior in evaluating summaries from both the CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets.
To better understand their behavior, we next analyze what error types are different factuality metrics capable of identifying (Section 4). To do this, we leverage datasets from our benchmark that have fine-grained error annotations and unify these into a single taxonomy. We find that the error type distribution changes over time and even differs between annotations of the same summarization models across factuality datasets. Analysis of the factuality metrics shows that metrics claim-
11626
ing SOTA performance can identify each error type better in general, but all metrics differ significantly in how they perform on the same error types across CNN/DM and XSum.
We conclude with the following recommendations for best practices in this area:

heading: 1. Evaluate factuality metrics on summaries
text: generated by the state-of-the-art summarization models. We found generally worse performance when evaluating factuality systems on summaries generated by FTSOTA models instead of less recent models (Section 3). We release AGGREFACT to support this, which combines existing benchmarks and stratifies them according to the base summarization model, summarization dataset and error types. We suggest future work to augment our benchmark with LLM-generated summaries, e.g. from ChatGPT, which is beyond the scope of this paper.

heading: 2. Choose an appropriate factuality metric for
text: your downstream task at hand. No one metric is superior across all settings (Section 4). Fine-grained insights offered by our benchmark can be useful to compare strengths of different factuality metrics and make this choice.

heading: 3. Annotate error types consistently with prior
text: work for better comparability. We found that
error type boundaries in existing works are not clear and are not easy to leverage for crossdataset metric comparisons (Section 4).
We hope that our analysis can shed light on what comparisons practitioners should focus on, how to understand the pros and cons of different metrics, and where metrics should go next. Further, we hope that future work would extend this to incorporate diverse summarization domains such as dialogue summarization (Tang et al., 2022; Fabbri et al., 2021a; Zhang et al., 2021) and medical evidence summarization (Tang et al., 2023). These would have different error distributions, and annotated datasets are needed to perform a more comprehensive comparison and design domain-invariant factuality metrics.

heading: 2 Benchmark

heading: 2.1 Benchmark Standardization
text: Current factuality metrics are evaluated without considering the types of summarization models used to generate the annotated summaries. In these annotated datasets, a large proportion of summaries are generated by older models, such as a pointergenerator network (See et al., 2017), that often make obvious errors that recent models do not make. We hypothesize that current factuality systems primarily make progress in identifying
factuality inconsistencies in summaries generated by out-of-date summarization models. If this hypothesis is correct, comparing factuality systems on such datasets provide us less useful information on how these metrics perform on modern summarization systems.
Summarization datasets splits We introduce a new benchmark AGGREFACT built on top of SummaC from Laban et al. (2022). The benchmark Aggregates nine publicly available datasets (see Table 1) that consist of human evaluations of Factual consistency on model generated summaries. We focus particularly on incorporating recent datasets annotated on top of state-of-the-art pre-trained Transformer models.
All datasets contain summaries generated from articles in CNN/DM and XSum. Given the unique characteristics of CNN/DM and XSum, our proposed benchmark includes two subsets, AGGREFACT-CNN and AGGREFACT-XSUM, that evaluate the performance of factuality metrics on these two datasets separately (Table 2; see also Table 6 and 7 in the Appendix). This facilitates a more fine-grained and rigorous analysis of the metric performance.
Our benchmark formulates factual consistency evaluation as a binary classification task, following Laban et al. (2022). The binary factuality labels for the summaries are determined by human evaluations on the annotated datasets (Section 2.2).
Summarization model splits To validate our hypothesis and make a careful comparison of factuality metrics, we further divide models that were used to generated summaries in the benchmark into three distinct categories: C = { FTSOTA, EXFORMER, OLD }, as seen in Table 2. FTSOTA represents state-of-the-art fine-tuned summarization models, including BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020) and T5 (Raffel et al., 2020). EXFORMER is a collection of early Transformerbased summarization models. Typical models that fit into this category include BERTSum (Liu and
Lapata, 2019), and GPT-2 (Radford et al., 2019). The remaining models, such as Pointer-Generator (See et al., 2017) and BottomUp (Gehrmann et al., 2018), are instances of OLD. A full description of the models in each category is found in Appendix A.

heading: 2.2 Benchmark Datasets
text: The SUMMAC benchmark (Laban et al., 2022) includes six annotated datasets for factual consistency evaluation. We directly include XSumFaith (Maynez et al., 2020), FactCC (Kryscinski et al., 2020), SummEval (Fabbri et al., 2021b), and FRANK (Pagnoni et al., 2021) from SUMMAC in our benchmark. We do not include the CoGenSumm (Falke et al., 2019) dataset as the original task is ranking pairs of generated summaries instead of detecting factually consistent summaries, and pairs of summaries can be both factually consistent or inconsistent. We modify the Polytope (Huang et al., 2020) dataset in SUMMAC where we view summaries annotated with addition, omission or duplication errors as factually consistent since these three error types are not related to factual consistency. We use the validation and test splits from SUMMAC for the above mentioned datasets.
In addition to modifying SUMMAC, we further include four annotated datasets. For Wang’20 (Wang et al., 2020), CLIFF (Cao and Wang, 2021) and Goyal’21 (Goyal and Durrett, 2021), we create data splits based on the parity of indices, following SUMMAC. For Cao’22 (Cao et al., 2022), we use the existing splits from the original work.
Deduplication and label disagreement correction Some examples may be labeled for errors in multiple datasets. We removed all duplicates so that each instance appears only once in our benchmark. During this deduplication process, we detected 100 instances of the same summaries that are annotated in different datasets with different factual consistency labels. 98 of them are between FRANK and XSumFaith, and 2 of them are between FRANK and SummEval. The authors of this work manually corrected the labels for these examples based on our judgment.

heading: 2.3 Benchmark Evaluation Metrics
text: We use balanced accuracy to evaluate the performance of factuality metrics due to the imbalance of factually consistent and inconsistent summaries. We refer readers to Laban et al. (2022) for further
justification of balanced accuracy as the evaluation metric. In each dataset, a factuality metric selects a threshold for FTSOTA, EXFORMER and OLD, respectively, based on the performance on the corresponding validation set. The chosen thresholds convert raw scores from metrics into binary labels for balanced accuracy evaluation. We provide a weighted average of performance across all datasets in the benchmark (see Table 3).

heading: 3 Comparison of Factuality Metrics
text: First, we evaluate several SOTA factual consistency metrics on our benchmark, namely DAE (Goyal and Durrett, 2020, 2021), QuestEval (Scialom et al., 2021), SummaC-ZS, SummaC-Conv (Laban et al., 2022) and QAFactEval (Fabbri et al., 2021c).2 We also benchmark recent ChatGPTbased evaluation metics from Luo et al. (2023) and Wang et al. (2023). ChatGPT-ZS and ChatGPTCoT (Luo et al., 2023) prompt LLMs to directly output a binary factuality decision. On the other hand, ChatGPT-DA and ChatGPT-Star (Wang et al., 2023) ask LLMs to score the factuality of generated summaries on a scale of 0-100 and 1-5 respectively. More details about these metrics, including exact prompts are included in Appendix B.
Unifying these metrics We consider each metric as a function f(d, s) → y, mapping each (document, summary) pair to a score y ∈ R. We convert each method into a binary classifier f ′(d, s) → {0, 1} by picking a threshold t such that we predict 1 if f(d, s) > t and 0 otherwise.3
All thresholds are set separately for each metric. We consider two ways of setting the threshold for a metric: threshold-per-dataset and singlethreshold. The first setting has thresholds {tmd,c} within each metric for every dataset we consider, where d, c and m are any dataset in D, any model category from C, and any factuality metric, respectively. This allows one to choose the right metric for the task at hand. The single-threshold setting defines one threshold {tm} per metric.
Threshold Analysis We analyze scores from factuality metrics using chosen thresholds {tmd,c} from the validation sets. Specifically, for each factuality
2We do not consider other common metrics like ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) or BERTScore (Zhang* et al., 2020) as prior work (Fabbri et al., 2021c) has shown that they have low correlation with factuality.
3CHATGPT-ZS and CHATGPT-COT do not require thresholds as they directly predict factual consistency labels.
metric, we average the values of thresholds for each of SOTA, EXFORMER and OLD across all datasets (Figure 1). For all facuality metrics, the average threshold values for AGGREFACT-CNN are greater than those for AGGREFACT-XSUM across all categories. This discrepancy of threshold values shows that evaluating on both of these datasets with a single threshold is a difficult balancing act and may lead to poor results on at least one dataset.
The higher threshold values on CNN/DM are connected to both the nature of the errors involved and overall extractiveness of the summaries XSum summaries are more abstractive and tend to contain a larger number of errors, making it harder for the metrics to verify the consistency of summaries with respect to the source text and resulting in lower scores in general, even for factual cases. For CNN/DM, smaller deviations from the source may indicate non-factuality.
Binary Classification Results A weighted average of performance in terms of balanced accuracy for AGGREFACT-CNN and AGGREFACT-XSUM is shown in Table 3.4 It shows results using both trained metrics (upper half) and ChatGPT-based metrics (bottom half).
Our results show that for AGGREFACT-CNN, both trained and ChatGPT-based factuality metrics achieve the best performance in evaluating the summaries in OLD. This result is intuitive: the sum-
4Dataset-wise comparison between factuality metrics is shown in Appendix Table 8.
maries in OLD contain obvious errors, such as repetition, that can be more easily detected compared to more nuanced errors made by more recent models. From Table 2, the majority of annotated summaries are generated by models from OLD, so category agnostic performance evaluation will weight these more heavily. There is a significant performance drop when evaluating the CNN/DM summaries generated by models from EXFORMER or FTSOTA instead. Approximately a 10% balanced accuracy decrease on average occurs from OLD to FTSOTA. Evaluating on entire datasets, as is standard in prior work, gives us limited information of how these metrics perform on the FTSOTA summaries that are of more interest.
We observe more mixed results for AGGREFACTXSUM. Here, the trained and ChatGPT-based metrics perform best on FTSOTA and EXFORMER respectively. In fact, the ChatGPT-ZS and ChatGPTStar metrics report new state-of-the-art results for the EXFORMER category.5 In the case of AGGREFACT-XSUM also, we advocate for comparing metrics according to such a category-wise view as it provides more information on the most suit-
5We found that using different prompts can substantially vary the performance of ChatGPT metrics on both datasets. In our work, we use the exact same prompts as the original papers. Check the prompts in Appendix B.
able metric to use while evaluating a given category of models.
Binary Classification: FTSOTA To encourage comparison of factuality metrics on FTSOTA summaries, we provide a separate benchmark which consists of two subsets AGGREFACT-CNNFTSOTA and AGGREFACT-XSUM-FTSOTA that only consider summaries generated by FTSOTA models. This benchmark consists of validation and test splits from the FTSOTA subsets of the two datasets. This setting allows for comparisons of metrics to be made using only a single threshold.
We show metric comparisons on the FTSOTA subset in Table 4. Note that the ranking of factuality metric here (single-threshold setting) is slightly different from the ranking in Table 3 (threshold-perdataset setting). For AGGREFACT-CNN-FTSOTA, QuestEval achieves the best performance amongst all metrics. We did not observe a statistically significant improvement over other trained evaluation metrics; however, its improvement over ChatGPT-based metrics is statistically significant. For AGGREFACT-XSUM-FTSOTA, the DAE metric is significantly better than all other metrics.
Interestingly, metrics such as SummaC-Conv, QAFactEval and the recent ChatGPT metrics were all proposed as improved factuality evaluation on the category-agnostic SummaC benchmark (different from the SummaC metric). However, our stratified analysis provides a much clearer picture and shows that metrics which claim improved performance on SUMMAC do not show similar gains when evaluated on FTSOTA summaries. We recommend that future work similarly focuses on the SOTA category of generated summaries when comparing factuality metrics.
use unique colors to represent entity - and predicate - related errors, as well as the mix of two . See Appendix C for citations of papers that use each error type.

heading: 4 Finer-grained Error Analysis
text: Having established differences among factuality metrics across underlying summarization models, we now explore differences in metrics according to factuality error types. To do this, we need a way to unify error types across datasets in our benchmark and map them into a shared taxonomy.

heading: 4.1 A Taxonomy of Error Types
text: We surveyed existing error type taxonomies in prior work and unified the types of factual errors among them into a hierarchical taxonomy in Figure 2. Arrows relate more specific error types to more general “parent” errors. The prior works that make use of each error type can be found in Appendix C. As shown in the figure, most error types related to factual consistency fall under the subset {intrinsic, extrinsic} × {noun phrase, predicate} if we consider the coarsest level of the hierarchy. We discard discourse errors as these are uncommon and not present in most of our datasets. Therefore, we consolidate all unique error type taxonomies from all four datasets we consider here into this error type subset (shown in the gray box in Figure 2). Descriptions and examples for these error types are in Table 9. Further, we introduce two additional error categories {intrinsic-entire sent., extrinsicentire sent.} if an entire sentence is annotated as erroneous.
We are able to map four of the datasets (see Section 4.2) in AGGREFACT that contain fine-grained annotations to our unified taxonomy. For all four datasets, if there are multiple annotators, we assign an error type to a summary if the error is annotated by more than one annotator. We allow one summary to have multiple error types. We call the
annotated subset related to CNN/DM and XSum as AGGREFACT-CNN-UNIFIED and AGGREFACTXSUM-UNIFIED, respectively.

heading: 4.2 Error Mapping
text: XSumFaith XSumFaith consists of 500 summaries each from human reference, two models in OLD, and two models in EXFORMER. All summaries are annotated with intrinsic and extrinsic errors, but no finer categories are distinguished. For error type mapping, we automatically detect predicates in a summary and assign each error span intrinsic- or extrinsic-predicate error if it contains a predicate. We map the remaining error spans to intrinsic- or extrinsic-noun phrase error.
FRANK The CNN/DM subset of FRANK consists of three models in OLD, and one model each in both EXFORMER and FTSOTA. The XSum portion of FRANK has two models each in OLD and EXFORMER. Each model contains 250 summaries in the dataset. We mapped Entity error and Out of Article error to extrinsic-noun phrase error; Predicate error and Grammatical error to extrinsic-predicate error; Circumstance error and Coreference error to intrinsic-noun phrase error; and other errors to intrinsic-predicate error.
Goyal’21 Authors of the original dataset manually identified all hallucinated text spans for each summary and classified hallucination types into {intrinsic, extrinsic} × {entity, event, noun phrase, others}. The dataset consists of summaries for both CNN/DM and XSum. For the CNN/DM susbset, the authors directly annotated 50 summaries from FactCC, where summaries were generated by OLD models. The XSum subset consists of summaries from FTSOTA models. We map entity-related and noun phrase-related errors to noun phrase errors, event errors to predicate errors and others to entire sentence errors.
CLIFF This dataset consists of 150 summaries each for both CNN/DM and XSum from two models in FTSOTA. We use the same approach for error mapping as we do for XSumFaith by only considering words labeled as extrinsic or intrinsic errors.
We evaluate the accuracy of our error type mapping via manual inspection. Specifically, the authors of this work inspect 30 factually inconsistent examples each for XSumFaith, FRANK and CLIFF. Those examples cover summaries generated by all
models used in the datasets. Results of the manual inspection show that the accuracy of our error type mapping is over 90%.
A common discrepancy noticed by annotators was that in several cases the examples were originally annotated as intrinsic/extrinsic but we believe those errors are extrinsic/intrinsic. These cases are not a result of error in our mapping, but instead disagreement or error in the original annotation itself. For error mapping, we found out mapping of FRANK to be least accurate among all 4 datasets. For example, we found that the entity error (EntE) can be either intrinsic or extrinsic even though FRANK explicitly defines an extrinsic error type, i.e. “out of article” error. For Goyal’21, we manually correct any mapping errors that occur in the 150 examples. Corrections mostly happen for the event-related error defined in Goyal’21 which can be either noun phrase- or predicate-related.

heading: 4.3 Distribution Shift of Error Types
text: Next, we explore how the number of errors in specific groups of models from FTSOTA, EXFORMER, and OLD has changed with the progress in the field. Specifically, for each of the FRANK, XSumFaith, Goyal’21, and CLIFF datasets, we calculate the ratio of error types from factually inconsistent summaries generated by each model. We then study any distribution shift of error types in AGGREFACTCNN-UNIFIED and AGGREFACT-XSUM-UNIFIED under FTSOTA, EXFORMER, and OLD.
Summaries generated by the same models consist of different error distributions over different annotated datasets. As shown in AGGREFACTXSUM-UNIFIED (Figure 3), BART summaries are annotated by both Goyal’21 and CLIFF. However, it is interesting that BART summaries were anno-
tated as making more intrinsic-noun phrase and intrinsic-predicate errors in Goyal’21 but more extrinsic-noun phrase errors in CLIFF. Similar observations can be found in AGGREFACT-CNNUNIFIED, where BART summaries have a higher proportion of extrinsic-predicate error in FRANK and more intrinsic-noun phrase error in CLIFF.
In addition, although XSumFaith and FRANK annotate the same set of model generated summaries in AGGREFACT-XSUM-UNIFIED, the distribution of error types looks dramatically different. The main discrepancy lies in the proportion of extrinsic-noun phrase and intrinsic-predicate errors. There are two possible reasons for such discrepancy. First, FRANK does not have “entire sent.” errors based on our conversation of its annotation schema to the unified taxonomy (Section 4.2). Second, and more important, it is not easy to map error types from FRANK directly to our unified error types in spite of our validation. For example, the “out of article error” in FRANK is defined as an error where some statements in the summary do not show up in the source text. We found this error can be mapped to either an extrinsic-noun phrase error or extrinsic-predicate error. These observations indicate that previous work disagrees about where the individual error class boundaries are, even when aligned with our taxonomy.
A combined meta-analysis shows shifts in error distributions. Figure 3 shows that error type distribution can vary among models from the same category. For example, summaries from BART contain a higher ratio of intrinsic-noun phrase errors than PEGASUS in AGGREFACTCNN-UNIFIED. We now combine all datasets together from AGGREFACT-CNN-UNIFIED and AGGREFACT-XSUM-UNIFIED and show the uni-
fied error distributions over three model categories.6 As shown in Figure 4, models make approximately 50% extrinsic errors in CNN/DM, with a slightly decrease from OLD to more recent models. For XSum, the proportion of extrinsic errors remains unchanged and is at 70%.

heading: 4.4 Error Detection by Type
text: In this section, we analyze how factuality metrics perform on summaries that contain certain error types. Specifically, we collect subsets of examples from four annotated datasets and group them into AGGREFACT-CNN-ERROR and AGGREFACTXSUM-ERROR.7 Every subset contains summaries that include only one error type defined in Sec-
6For AGGREFACT-XSUM-UNIFIED, since XSumFaith and FRANK annotated the same set of summaries, we only use the annotation results from XSumFaith since our error mapping is more accurate on the span-level annotations.
7We exclude FRANK for this analysis for the same reason as in Section 4.3.
tion 4.1. Each factuality metric assigns a binary label to an instance obtained directly from AGGREFACT-CNN and AGGREFACT-XSUM. Note that each subset only consists of test set examples from our benchmark since examples from the validation set were used to choose the optimal thresholds (Section 3). Since there are limited annotations for each model category after only considering examples from the test set of the benchmark, we decide not to split data by model categories in this part of the analysis. We calculate the recall of identifying error types from those subsets and show the results in Table 5. Summaries in AGGREFACT-CNN-ERROR and AGGREFACTXSUM-ERROR primarily come from non-FTSOTA models (89.6% and 92.1%, respectively). On AGGREFACT-CNN-ERROR, where 79.0% of summaries were generated from OLD, there are more extrinsic errors (349) than intrinsic errors (243). This agrees with our above analysis that also shows that errors in generated summaries from less recent models are more likely to be extrinsic (Figure 4).
Across both AGGREFACT-CNN-ERROR and AGGREFACT-XSUM-ERROR, we found that recent metrics like SummaC-Conv, QAFactEval and ChatGPT-based achieve higher recall for most error types. This indicates that more recent factuality metrics are better at capturing obvious errors generated by less recent models. This mirrors our earlier finding in Table 3 (column EXFORMER and OLD). Interestingly, we find that summarization datasets (CNN/DM and XSum) have a nonnegligible effect on the metrics’ capabilities of
detecting certain error types, even in the cases of out-of-date errors. For example, the recall of identifying extrinsic-noun phrase error drops 10-30% across all trained factuality metrics when evaluated on XSUM, compared to CNN/DM. Similarly, ChatGPT metrics report 20-30% higher recall on CNN/DM, compared to its XSUM counterparts.
Another observation is that although DAE is trained using annotations from XSumFaith, which provides supervision for multiple error types, it does not identify errors as well in AGGREFACTCNN-ERROR. These findings indicate that summarization models make fundamentally different errors for each error type, and current factuality metrics cannot be uniformly good at identifying certain error types across datasets. We believe this conclusion still holds when evaluating metrics on summaries generated from FTSOTA models since they generate less obvious errors.

heading: 5 Recommendations
text: Evaluate factuality models on modern summarization systems We have seen that FTSOTA yields significantly different results than EXFORMER or OLD. Because of the prevalence of these systems, we believe that any new work should prefer evaluating on these SOTA summaries.
Particularly for factuality metrics that are either based on latest LLMs or on pre-trained models, evaluating on modern summarization systems is needed to see if these metrics are actually improving from the current state-of-the-art or merely patching errors in outdated systems that have already been fixed by other advances.
Annotate factual consistency errors from summaries generated by LLMs Recent work (Goyal et al., 2022) shows that LLMs like GPT-3 are capable of generating summaries that are preferred over FTSOTA summaries by human annotators. Furthermore, they show that existing factuality metrics cannot reliably detect errors in summaries from GPT-3 models as these latter summaries differ substantially from existing benchmarks and training sets. We encourage future work to annotate errors from LLM-generated summaries and evaluate new factual consistency metrics on this set as well in addition to the FTSOTA set. As such, we believe that future work should construct “living” benchmarks for factuality evaluation that are consistently updated as more powerful summarization systems are introduced.
Choose the right metric for the job We note that there is no one clear winner among the metrics evaluated here (Section 3). Depending on the downstream application, different methods may be more or less appropriate, as our analysis shows. Moreover, none of current factuality metrics can identify certain error types across datasets equally well. As QG/QA and NLI models get better, we expect all of these methods to improve further. Alternatively, although recent ChatGPT-based metrics (Luo et al., 2023; Wang et al., 2023) do not perform well on modern summarization systems, they can be a starting point for leveraging LLMs to perform factual consistency evaluation.
Use more consistent error types With our taxonomy, we have mapped error types annotated in previous work. It is relatively easier and more accurate to map errors from XSumFaith, Goyal’21, and CLIFF to our unified error types as they have annotation granularity finer than sentence-level. We encourage future work to follow this taxonomy where possible and leverage definitions in prior work to make cross-dataset comparisons possible. Here also, we encourage future work to prioritize annotation and evaluation of SOTA summaries.
Annotate and evaluate on non-news datasets Most of current annotated datasets are within the news domain and factuality metrics are evaluated on news summaries accordingly. As there is a rising interest in other domains such as dialogue summarization (Tang et al., 2022; Fabbri et al., 2021a; Zhang et al., 2021), and medical evidence summarization (Tang et al., 2023), future work could annotate and analyze errors made by SOTA models there. We encourage future work to develop factuality metrics that have superior performance over cross-domain evaluation.

heading: 6 Conclusion
text: In this work, we analyzed several factuality metrics across a large meta-benchmark assembled from existing datasets. We find state-of-the-art fine-tuned summarization models still present challenges for detecting factual errors, and the performance of error detectors is often overestimated due to the reliance on older datasets. Furthermore, we unify existing datasets into a common taxonomy and use this to highlight differences between datasets and summarization models, as well as the complexity of unifying concepts in this problem space.
Limitations
There are a few limitations of our work. First, we focus on evaluating state-of-the-art factuality metrics on English newswire datasets. This setting restricts us to English-language data, a formal style of text, and topics consisting of what is discussed in US and UK-centric news sources. Moreover, other summarization domains such as dialogue summarization have different common error types such as wrong reference error (Tang et al., 2022), which are not fully evaluated under current metrics. As settings like this are studied in future work, we believe that the kinds of analysis we do here can be extended to these settings as well.
Second, since our work is built on top of previous work, some analysis such as the error type mapping is limited by the quality and annotation agreement from previous work. We chose not to undertake large-scale reannotation to avoid causing confusion in the literature with multiple versions of datasets reflecting divergent annotator opinions. In spite of these limitations, we believe that our reevaluation of these metrics and the analysis of error types under newswire data can bring insights for future works in choosing, designing and evaluating factuality metrics.

heading: Acknowledgments
text: The UT Austin team on this work was supported by a gift from Salesforce Inc., NSF Grant IIS-1814522, and a gift from Amazon.

heading: A Model Categories
text: In this section, we briefly describe the summarization models we use in this paper.
For FTSOTA, we include Transformer-based pretrained models like BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and PEGASUS (Zhang et al., 2020). They are pre-trained on massive text corpus and further fine-tuned on summarization datasets.
For EXFORMER, we use BERTSumExt and BERTSumAbs from Liu and Lapata (2019), GPT-2 (Radford et al., 2019), TransS2S (Vaswani et al., 2017), and BERTS2S (Devlin et al., 2019).
For OLD, we include models FastAbsRl (Chen and Bansal, 2018), TConvS2S (Narayan et al., 2018), BottomUp (Gehrmann et al., 2018), PGNet (See et al., 2017), NeuSUM (Zhou et al., 2018), BanditSum (Dong et al., 2018), SummaRuNNer (Nallapati et al., 2017), TextRank (Mihalcea and Tarau, 2004), CBDec (Jiang and Bansal, 2018), RNES (Wu and Hu, 2018), ROUGESal (Pasunuru and Bansal, 2018), ImproveAbs (Kryściński et al., 2018), MultiTask (Guo et al., 2018), and UnifiedExtAbs (Hsu et al., 2018).

heading: B Factuality Metrics
text: We show the descriptions of consistency metrics we considered in our benchmark.
DAE (Goyal and Durrett, 2020) propose an arc entailment approach that evaluates the factuality Fa(a, x) = P (entailment | a, x) of each dependency arc a ∈ Arc(s) of the generated summary s independently with respect to the input article x. It then uses their aggregation
1 |Arc(s)| ∑ a∈Arc(s) Fa(a, x) as the overall score. We use the default model and hyperparameters provided by the authors,8 described in Goyal and Durrett (2021), which is trained on data from XSumFaith, which we account for later in our comparisons.
QuestEval (Scialom et al., 2021) propose a QAbased metric that aggregates answer overlap scores from selected spans r and questions qi ∈ QG(x) that derived from the input article x and answered QA(s, qi) using the summary s (recall-based); and those derived from the summary qi ∈ QG(s) and answered QA(x, qi) using the input article x (precision-based). QG and QA denote question generation and question answering components, respectively. We use the implementation provided by the authors9 and apply the unweighted version of
8https://github.com/tagoyal/ factuality-datasets
9https://github.com/ThomasScialom/QuestEval
the metric as in Laban et al. (2022).
SummaC-ZS (Laban et al., 2022) is a zero-shot entailment metric that computes a sentence-level entailment score F (si, xj) between each summary sentence si and input sentence xj using an NLI model F . It first find the maximum entailment score score(si) = maxj F (si, xj) for each summary sentence si, and averaging over all summary sentences for the final score 1|s| ∑ i score(si). We use the default model and hyperparameters provided by the authors, which may return a negative score.
SummaC-Conv (Laban et al., 2022) extends SummaC-ZS by replacing the max operation with a binning of the entailment scores between each summary sentence si and all input sentences xj to create a histogram hist(si, x). The histogram is then passed through a learned 1-D convolution layer Conv to produce the summary sentence score score(si) = Conv(hist(si, x)). Parameters for the convolution layer are learned on synthetic data from FactCC (Kryscinski et al., 2020).
QAFactEval (Fabbri et al., 2021c) is a QA-based metric analogous to the precision-based component of QuestEval and includes optimized question answering, generation, and answer-overlap components. We do not make use of the variation of QAFactEval which combines QA and entailmentbased scores into a single metric.
ChatGPT-ZS (Luo et al., 2023) uses a zero-shot template and directly asks for a binary label of summary factuality.
Decide if the following summary is consistent with the corresponding article. Note that consistency means all information in the summary is supported by the article.
Article: [Article] Summary: [Summary] Answer (yes or no):
ChatGPT-CoT (Luo et al., 2023) also uses a zero-shot template but invokes chain-of-thought (CoT) style reasoning in its prompt. Similar to ChatGPT-ZS, it directly asks for a binary factuality label for a given summary.
Decide if the following summary is consistent with the corresponding article. Note that consistency means all information in the summary is supported by the article.
Article: [Article]
Summary: [Summary] Explain your reasoning step by step then
answer (yes or no) the question:
ChatGPT-DA (Wang et al., 2023) uses a direct assessment (DA) prompt template that asks to assign a factual consistency score to a summary on a continuous scale from 0 to 100.
Score the following news summarization given the corresponding news with respect to consistency on a continuous scale from 0 to 100, where a score of zero means “inconsistency” and score of one hundred means “perfect consistency”. Note that consistency measures whether the facts in the summary are consistent with the facts in the original article. Consider whether the summary does reproduce all facts accurately and does not make up untrue information.
Article: [Article] Summary: [Summary] Scores:
ChatGPT-Star (Wang et al., 2023) is an alternative version of ChatGPT-DA that asks LLMs to score summaries on a scale of one-to-five.
Score the following news summarization given the corresponding news with respect to consistency with one to five stars, where one star means “inconsistency” and five stars means “perfect consistency”. Note that consistency measures whether the facts in the summary are consistent with the facts in the original article. Consider whether the summary does reproduce all facts accurately and does not make up untrue information.
Article: [Article] Summary: [Summary] Stars:

heading: C Surveyed Error Types
text: Here are our surveyed error types that are related to factual inconsistency.
Negation Error (Zhang et al., 2020; Kryscinski et al., 2020; Huang et al., 2020; Zeng et al., 2021)
Adjective Error (Zhang et al., 2020)
Coreference Error (Zhang et al., 2020; Kryscinski et al., 2020; Pagnoni et al., 2021; Nan et al., 2021b)
Number error (Kryscinski et al., 2020; Nan et al., 2021b; Chen et al., 2021; Cao et al., 2020)
Entity error (Kryscinski et al., 2020; Pagnoni et al., 2021; Zeng et al., 2021; Wang et al., 2020; Nan et al., 2021b,a; Chen et al., 2021; Cao et al., 2020)
Attribute error (Pagnoni et al., 2021; Huang et al., 2020)
Pronoun error (Kryscinski et al., 2020; Zeng et al., 2021; Cao et al., 2020)
Commonsense error (Kryscinski et al., 2020)
Temporal error (Kryscinski et al., 2020; Cao et al., 2020)
Predicate error (Pagnoni et al., 2021)
Discourse link Error (Pagnoni et al., 2021)
Relation error (Nan et al., 2021a,b)
Quantity error (Zhao et al., 2020)
Event error (Goyal and Durrett, 2021),
Noun phrase error (Wang et al., 2020; Goyal and Durrett, 2021),
Circumstance error (Pagnoni et al., 2021)
Factuality Metric
SummaC ChatGPT
DAE QuestEval ZS Conv QAFactEval ZS COT DA Star Dataset Category Count
CNN /DM
FactCC OLD 503 0.704 0.655 0.835 0.891 0.843 0.793 0.697 0.686 0.743
Wang’20 OLD 117 0.586 0.552 0.655 0.672 0.754 0.758 0.599 0.695 0.652
SummEval OLD 548 0.661 0.649 0.773 0.801 0.814 0.735 0.680 0.735 0.713
EXFORMER 50 0.760 0.680 0.620 0.580 0.740 0.720 0.740 0.820 0.760 FTSOTA 200 0.452 0.649 0.622 0.827 0.652 0.783 0.401 0.453 0.568
Polytope OLD 450 0.779 0.687 0.802 0.791 0.824 0.768 0.695 0.741 0.752 EXFORMER 150 0.774 0.733 0.970 0.811 0.726 0.693 0.632 0.713 0.740
FTSOTA 34 0.294 0.176 0.971 0.735 0.324 0.941 0.735 0.206 0.412
FRANK OLD 523 0.704 0.669 0.692 0.728 0.773 0.694 0.628 0.695 0.672 EXFORMER 175 0.574 0.556 0.631 0.634 0.646 0.583 0.540 0.517 0.558
FTSOTA 175 0.699 0.626 0.570 0.601 0.547 0.519 0.514 0.523 0.531
Goyal’21 OLD 25 0.188 0.146 0.375 0.354 0.271 0.375 0.417 0.500 0.479
CLIFF FTSOTA 150 0.730 0.740 0.646 0.649 0.716 0.603 0.550 0.528 0.612
XSum
Wang’20 FTSOTA 119 0.756 0.560 0.698 0.721 0.756 0.608 0.514 0.533 0.620
Cao’22 FTSOTA 239 0.723 0.601 0.490 0.668 0.613 0.643 0.576 0.502 0.530
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: As settings like this are studied in future work, we believe that the kinds of analysis we do here can be extended to these settings as well. Second, since our work is built on top of previous work, some analysis such as the error type mapping is limited by the quality and annotation agreement from previous work. We chose not to undertake large-scale reannotation to avoid causing confusion in the literature with multiple versions of datasets reflecting divergent annotator opinions. In spite of these limitations, we believe that our reevaluation of these metrics and the analysis of error types under newswire data can bring insights for future works in choosing, designing and evaluating factuality metrics