Introduction
One of the remaining issues that prevents productive deployments of neural text summarization systems is the low correlation of system outputs with human preferences.
Among those, factuality , i.e., the agreement of facts in the generated summaries with those present in the input text, is not part of the general training objectives of models, which frequently leads to hallucinated facts that are detrimental to perceived system performance ter Hoeve et al. ( 2020 ); Fabbri et al. ( 2021 ) .
Prior work has therefore introduced metrics for automated testing of factuality in generated text Goodrich et al. ( 2019 ); Kryscinski et al. ( 2020 ); Yuan et al. ( 2021 ) , which allows for a more nuanced verification of model capabilities.
In particular, one of the first relevant works by Goodrich et al. ( 2019 ) introduces the idea of representing text as a series of "fact tuples", in their case as (subject, predicate, object) triplets.
Their method exhibits some assumptions about the underlying data, which hampers correlation with human ratings.
For example, subject or object may vary for the same sentence meaning expressed using different syntactic structures, e.g., active and passive forms.
Semantic Role Labeling (SRL), however, allows for a syntactically independent meaning representation.
Our metric, SRLScore , improves factuality evaluation, building on fact tuples similar to Goodrich et al. It distinguishes itself in several ways from existing approaches, though:

To account for a more nuanced fact representation, we employ SRL to produce abstract representations of sentences that are independent of their syntactic formulations .

Fact tuples in SRLScore are generated on the input text instead of gold summaries; as a consequence, our method is reference-free, and may be applied for evaluation irrespective of the availability of labeled datasets.

We introduce a novel weighting scheme for fact tuple comparison, where adjustable weights allow for user optimization.

Finally, we experiment with extensions along different parts of the pipeline, including an optional co-reference resolution step and alternative similarity scoring functions.

Notably, SRLScore entirely relies on publicly available software components and may be used without any further domain adaption required. While our experiments are performed on English, we argue that the transfer of our approach to other languages is possible given only the existence of a language-specific tokenizer and a sufficiently good SRL tagger.
Furthermore, SRLScore offers the additional benefit of being an interpretable metric, due to its composition on top of fact tuples. In comparison, metrics used for factuality evaluation that are based on the intermediate presentations of language models, e.g., generation perplexity Zhang et al. ( 2020 ); Thompson and Post ( 2020 ); Yuan et al. ( 2021 ) , cannot present insightful reasons why a particular score was achieved.
Furthermore, it has been empirically demonstrated that generation-based evaluators exhibit a self-preference of outputs generated by models similar to the factuality evaluator (Fabbri et al., 2021 ; Liu et al., 2023 ) . This makes them a questionable choice over interpretable metrics.
We empirically show that the correlation of SRLScore with human ratings is on par with existing methods, and perform several ablations to study the impact of algorithmic choices within our pipeline.