Related Work
Automated analysis of (abstractive) summaries became more relevant in recent years, with the influx of generic summarization systems becoming available Nallapati et al. ( 2016 ); See et al. ( 2017 ); Lewis et al. ( 2020 ) .
In particular, Goodrich et al. ( 2019 ) were the first to propose a reference-based estimator for factuality of generated summaries. As mentioned, their approach is based on a tuple representation of "facts" in the generated and gold summary. Fact tuples are extracted based on a weakly supervised end-to-end tagger and subsequently compared on the basis of matching arguments. Notably, no readily available implementation of their method currently exists. Later work has proposed alternative metrics based on textual entailment Falke et al. ( 2019 ); Mishra et al. ( 2021 ) and Question Answering (QA) Wang et al. ( 2020 ); Durmus et al. ( 2020 ) , where agreement of answers to questions on the reference and summary are used for estimating factuality.
However, QA-based metrics require additional task-specific fine-tuning on generic datasets, which makes the adoption to new domains fairly expensive.

The only other work that to our knowledge utilizes some form of SRL-based factuality estimation is presented by Fischer et al. ( 2022 ) . In comparison to SRLScore , their method aggregates "role buckets" at the document level, instead of creating sentence-specific fact tuples.
Empirically, their implementation has lower correlation with human ratings than compared approaches, which is contrary to our own findings. Li et al. ( 2022 ) frame factuality estimation as an in-filling task, where fact statements are withheld as masked tokens in a generated summary, and a separate model is trained to predict missing facts. Notably, this relies on the assumption that the majority of factual mistakes stems from noun phrases and entity mentions Pagnoni et al. ( 2021 ) .

An alternative body of literature has explored the possibility to exploit Language Models (LMs) directly for estimating factual consistency: Some works, such as BertScore Zhang et al. ( 2020 ) , use LM-generated representations to generate alignments for scoring.
In comparison, PRISM Thompson and Post ( 2020 ) or BARTScore Yuan et al. ( 2021 ) directly use model perplexity as a factuality estimate. Xie et al. ( 2021 ) explore masking approaches, which fall somewhere between the works of Li et al. ( 2022 ) and BARTScore; their framing of counterfactual estimation still relies on model-based likelihood scores for computation.

The majority of prior work expresses metric performance in terms of correlation with human factuality ratings. Notably, annotations exist for subsets of the popular CNN/DailyMail Hermann et al. ( 2015 ); Nallapati et al. ( 2017 ) and XSUM summarization corpora Narayan et al. ( 2018 ) .
Where Wang et al. ( 2020 ) collect user annotations from crowd workers, Fabbri et al. ( 2021 ) additionally sample expert judgments, and find that expert ratings tend to be more representative. Maynez et al. ( 2020 ) study several aspects of summarization evaluation beyond just factuality, but do not disclose the background of annotators for evaluation.

Generally, reliably evaluating correlation of summarization metrics with human preferences is no easy task, either: Deutsch et al. ( 2022 ) show that system-level evaluation metrics for text summarization rarely outperform simplistic metrics, such as ROUGE Lin ( 2004 ) , to a statistically significant degree.
Partially, this can be attributed to the small number of human-annotated samples available, generally less than 1000 different instances.