Evaluating Factual Consistency of Texts with Semantic Role Labeling
Year: 2023

Abstract:
Automated evaluation of text generation systems has recently seen increasing attention, particularly checking whether generated text stays truthful to input sources. Existing methods frequently rely on an evaluation using taskspecific language models, which in turn allows for little interpretability of generated scores. We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. Our approach generates fact tuples constructed from Semantic Role Labels, applied to both input and summary texts. A final factuality score is computed by an adjustable scoring mechanism, which allows for easy adaption of the method across domains. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods and exhibits stable generalization across datasets without requiring further training or hyperparameter tuning. We experiment with an optional co-reference resolution step, but find that the performance boost is mostly outweighed by the additional compute required. Our metric is available online at: https:// github.com/heyjing/SRLScore

heading: 1 Introduction
text: One of the remaining issues that prevents productive deployments of neural text summarization systems is the low correlation of system outputs with human preferences. Among those, factuality, i.e., the agreement of facts in the generated summaries with those present in the input text, is not part of the general training objectives of models, which frequently leads to hallucinated facts that are detrimental to perceived system performance (ter Hoeve et al., 2020; Fabbri et al., 2021). Prior work has therefore introduced metrics for automated testing of factuality in generated text (Goodrich et al., 2019; Kryscinski et al., 2020; Yuan et al., 2021),
∗Both authors contributed equally to this work.
which allows for a more nuanced verification of model capabilities. In particular, one of the first relevant works by Goodrich et al. (2019) introduces the idea of representing text as a series of "fact tuples", in their case as (subject, predicate, object) triplets. Their method exhibits some assumptions about the underlying data, which hampers correlation with human ratings. For example, subject or object may vary for the same sentence meaning expressed using different syntactic structures, e.g., active and passive forms. Semantic Role Labeling (SRL), however, allows for a syntactically independent meaning representation. Our metric, SRLScore, improves factuality evaluation, building on fact tuples similar to Goodrich et al. It distinguishes itself in several ways from existing approaches, though:
1. To account for a more nuanced fact representation, we employ SRL to produce abstract representations of sentences that are independent of their syntactic formulations. 2. Fact tuples in SRLScore are generated on the input text instead of gold summaries; as a consequence, our method is reference-free, and may be applied for evaluation irrespective of the availability of labeled datasets. 3. We introduce a novel weighting scheme for fact tuple comparison, where adjustable weights allow for user optimization. 4. Finally, we experiment with extensions along different parts of the pipeline, including an optional co-reference resolution step and alternative similarity scoring functions.
Notably, SRLScore entirely relies on publicly available software components and may be used without any further domain adaption required. While our experiments are performed on English, we argue that the transfer of our approach to other languages is possible given only the existence of a language-specific tokenizer and a sufficiently good SRL tagger. Furthermore, SRLScore offers the
89
additional benefit of being an interpretable metric, due to its composition on top of fact tuples. In comparison, metrics used for factuality evaluation that are based on the intermediate presentations of language models, e.g., generation perplexity (Zhang et al., 2020; Thompson and Post, 2020; Yuan et al., 2021), cannot present insightful reasons why a particular score was achieved. Furthermore, it has been empirically demonstrated that generationbased evaluators exhibit a self-preference of outputs generated by models similar to the factuality evaluator (Fabbri et al., 2021; Liu et al., 2023). This makes them a questionable choice over interpretable metrics. We empirically show that the correlation of SRLScore with human ratings is on par with existing methods, and perform several ablations to study the impact of algorithmic choices within our pipeline.

heading: 2 Related Work
text: Automated analysis of (abstractive) summaries became more relevant in recent years, with the influx of generic summarization systems becoming available (Nallapati et al., 2016; See et al., 2017; Lewis et al., 2020). In particular, Goodrich et al. (2019) were the first to propose a reference-based estimator for factuality of generated summaries. As mentioned, their approach is based on a tuple representation of "facts" in the generated and gold summary. Fact tuples are extracted based on a weakly supervised end-to-end tagger and subsequently compared on the basis of matching arguments. Notably, no readily available implementation of their method currently exists. Later work has proposed alternative metrics based on textual entailment (Falke et al., 2019; Mishra et al., 2021) and Question Answering (QA) (Wang et al., 2020; Durmus et al., 2020), where agreement of answers to questions on the reference and summary are used for estimating factuality. However, QA-based metrics require additional task-specific fine-tuning on generic datasets, which makes the adoption to new domains fairly expensive. The only other work that to our knowledge utilizes some form of SRL-based factuality estimation is presented by Fischer et al. (2022). In comparison to SRLScore, their method aggregates "role buckets" at the document level, instead of creating sentence-specific fact tuples. Empirically, their implementation has lower correlation with human ratings than compared approaches, which is contrary to our own findings. Li et al. (2022) frame factuality estimation as an in-filling task, where fact statements are withheld as masked tokens in a generated summary, and a separate model is trained to predict missing facts. Notably, this relies on the assumption that the majority of factual mistakes stems from noun phrases and entity mentions (Pagnoni et al., 2021). An alternative body of literature has explored the possibility to exploit Language Models (LMs) directly for estimating factual consistency: Some works, such as BertScore (Zhang et al., 2020), use LM-generated representations to generate alignments for scoring. In comparison, PRISM (Thompson and Post, 2020) or BARTScore (Yuan et al., 2021) directly use model perplexity as a factuality estimate. Xie et al. (2021) explore masking approaches, which fall somewhere between the works of Li et al. (2022) and BARTScore; their framing of counterfactual estimation still relies on model-based likelihood scores for computation. The majority of prior work expresses metric perfor-
mance in terms of correlation with human factuality ratings. Notably, annotations exist for subsets of the popular CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2017) and XSUM summarization corpora (Narayan et al., 2018). Where Wang et al. (2020) collect user annotations from crowd workers, Fabbri et al. (2021) additionally sample expert judgments, and find that expert ratings tend to be more representative. Maynez et al. (2020) study several aspects of summarization evaluation beyond just factuality, but do not disclose the background of annotators for evaluation.
Generally, reliably evaluating correlation of summarization metrics with human preferences is no easy task, either: Deutsch et al. (2022) show that system-level evaluation metrics for text summarization rarely outperform simplistic metrics, such as ROUGE (Lin, 2004), to a statistically significant degree. Partially, this can be attributed to the small number of human-annotated samples available, generally less than 1000 different instances.

heading: 3 SRLScore
text: Our factual consistency metric, called SRLScore, is implemented as a two-stage process: first, extracting fact tuples using Semantic Role Labeling (SRL) on both the source texts and the summary texts, and then determining a factuality score based on tuple comparison. The measure outputs humaninterpretable scores between 0 and 1, where a higher score indicates greater factual consistency of a summary text. In this section, we detail the algorithmic choices and present an adaptive weighting scheme for computing the final factuality scores.

heading: 3.1 Generating Fact Tuples with Semantic Role Labeling
text: As Figure 1 shows, we operate on the sentence level, primarily because existing SRL tools work well on this level of granularity (Shi and Lin, 2019; Xu et al., 2021). The goal of our fact extractor is to produce a fact database comprised of semantic role tuples for each input text. The primary task of SRL is to find all role-bearing constituents in a sentence and label them with their respective roles (Màrquez et al., 2008). Typical semantic roles include agent, patient/theme, recipient, goal, instrument, manner, time, location and so on. From the many semantic labels available, we include seven roles based on availability in tagging schemes to construct a fact tuple: agent, negation, relation, patient, recipient, time, and location. We further note that not every sentence needs to contain all of these roles; absent labels are represented by None in this work. Importantly, roles reveal the semantic relations between a predicate (verb) and its arguments, which implies that one can generate several fact tuples from a single sentence, depending on the number of verbs in it. To illustrate an exemplary fact tuple, the extracted semantic tuple from sentence 1 in Figure 2 is (Mueller, None, gave, a book, Mary, yesterday, in Berlin).

heading: 3.2 Scoring Texts by Comparing Fact Tuples
text: Once fact tuples for both the input and summary texts are generated, the second step in our pipeline is to compute a factual accuracy score. We implement a dynamic weighting system, which crucially improves over a naive comparison, as we empirically show in Section 4.6. Furthermore, we describe the drop-in replacements for exact matching during similarity computation.
Scoring Algorithm. Given an input text R and summary text S, let FR and FS be fact databases, representing the semantic information contained in R and S, respectively. Individual fact tuples are represented as an ordered list of fact arguments, e.g., f = (agent, negation, relation, patient, recipient, time, location) ∈ F . Particular arguments in a fact tuple are referred to by their index position, meaning agent = f0, negation = f1, and so on. We further assume that there exists a scoring function that expresses the factual support of summary tuple fs, given an input tuple fr, denoted as S(fs|fr). To obtain a factuality score, we attempt to extract the best match f̂r ∈ FR for each sum-
mary fact fs ∈ Fs where f̂r maximizes the support score S(fs|f̂r). Importantly, we differ from, e.g., Goodrich et al. (2019), by considering the entirety of FR, instead of subsets that match both the agent and relation of the fact tuple. The factual accuracy is then the average across all maximized tuple scores in FS . With that, SRLScore is defined as:
SRLScore(R,S) := 1 |FS | ∑
fs∈Fs max fr∈FR
S(fs|fr)
(1) The final part of this scoring system is the computation of factual support S(fs|fr). Tuples are scored by comparing the corresponding attributes of each tuple, formally:
S(fs|fr) := ∑
i
1f is 6=None · sim(f i s, f i r) ·wi, (2)
where the summation over i addresses all attributes of the fact tuples, 1f is 6=None represents an indicator function considering only non-empty arguments f is (zero otherwise), and wi assigns static weights to arguments in position i. Generally, it should be assumed that the weights allow for a maximum factuality score of 1, i.e., ∑ iwi = 1. Finally, sim(f is, f i r) is the pairwise argument similarity of f is and f i r. We consider different similarity metrics, as described in the following paragraphs.
Dynamic Weighting System. The generic weighting in Equation (2) does not necessarily apply to the particular case of evaluating factual consistency in summarization, since a summary is still factually correct even if it leaves out particular aspects (e.g., dropping the date of an event), which were present in the input text. With static weights, however, absent arguments are still contributing to the scoring of the tuple fs, which means that leaving arguments out might potentially be considered as a penalization of factuality. To address this issue, we introduce a weight re-normalization factor, Wnorm, that distributes the static weights wi across only those attributes that are present in the current summary fact. In particular, this also increases penalties for actual mistakes over simple fact omission. The weight normalization is defined as follows:
Wnorm := 1∑
i 1f is 6=None · wi
(3)
With re-normalization enabled, we replace the existing computation of S(fs|fr) by the product Wnorm · S(fs|fr).
String Similarity Methods. We experiment with different methods to calculate the pairwise similarity sim(f is, f i r): exact matching (in line with prior work), but also approximate matching functions, such as word vector similarity1 and ROUGE1 precision (Lin, 2004). Computation of similarity with vectors and ROUGE each have their own respective strengths. Word vectors offer the highest flexibility in terms of recognizing argument similarity, enabling semantic comparison instead of purely syntactic equivalence. ROUGE-1 similarity does not offer the same level of flexibility in terms of matching, but shines with its comparatively faster computation, while still recognizing partial matches.

heading: 3.3 Improved Surface Form Invariance with Co-reference Resolution
text: In light of the fact that sentence-level SRL extraction misses co-references of the same entity across the texts, we integrate an optional component that takes co-reference resolution into account during the tuple generation. Concretely, we employ an offthe-shelf co-reference resolution tool (Lee et al., 2017) to identify and store all reference clusters in an external entity dictionary. There, all linguistic expressions that refer to the same entity will be grouped together, which allows for later disambiguation. As shown in Figure 3, if an extracted semantic role tuple contains co-references, a single fact tuple will be expanded into multiple tuples, representing the Cartesian product over all synonymous entity surface forms. The key idea here is to enable a better matching of potential facts across input texts and summaries, effectively increasing the recall of matches. The disadvantage is that this directly affects the runtime of our method by a strong factor, since the additional tuples in FS and FR will undoubtedly increase the number of comparisons.

heading: 4 Experiments
text: We empirically demonstrate the performance of our method through a number of experiments on two popular datasets for factual consistency evaluation, which are covered in this section. We further share implementation details and the choices for extracting SRL tuples and extracting co-reference clusters.
1We use spaCy’s vector similarity, see https://SpaCy. io/usage/linguistic-features#vectors-similarity, last accessed: 2023-03-06.
In addition to the experimental analysis, we also study the behavior of SRLScore through a number of ablation experiments and a brief error analysis.

heading: 4.1 Evaluation Datasets
text: QAGS (Wang et al., 2020). The dataset comprises of two separate splits: the first contains 235 instances collected from the test split of CNN/DailyMail (Nallapati et al., 2016), where each instance contains a source article and a modelgenerated summary using the bottom-up approach by Gehrmann et al. (2018). A secondary set contains 239 further instances from the test split of XSUM (Narayan et al., 2018), with generated summaries sampled from BART (Lewis et al., 2020).
SummEval (Fabbri et al., 2021). It includes synthetic summaries from 16 different abstractive and extractive models of 100 randomly selected articles from the test split of CNN/DailyMail. Unlike QAGS, which collected annotations from MTurk2, each SummEval sample was evaluated by five crowd-sourced annotators and three experts. For each summary, judges were asked to evaluate the coherence, consistency, fluency and relevance. For our evaluation, we use the expert ratings with regard to factual consistency as the gold score, based on the recommendation by Fabbri et al. (2021).

heading: 4.2 Evaluation Metrics and Significance
text: In line with prior work, we evaluate metrics by computing Pearson correlation (denoted as ρ) and Spearman correlation (denoted as s) between model predictions and human reference ratings. Given the limited size of all considered evaluation datasets, we further test results for significance using permutation tests (Riezler and Maxwell, 2005; Deutsch et al., 2021), following the recommendation of Dror et al. (2018). In all tables, † denotes
2https://www.mturk.com/, last accessed: 2023-03-06.
a significance level of 0.05 (p < 0.05) and ‡ a level of 0.01 (p < 0.01). When testing significance against several systems, we further apply Bonferroni correction of significance levels (Dunn, 1961).

heading: 4.3 Implementation
text: We use AllenNLP (Gardner et al., 2018), specifically version 2.1.0, to extract semantic role labels. AllenNLP implements a BERT-based SRL tagger (Shi and Lin, 2019), with some modifications. The output of AllenNLP uses PropBank convention (Palmer et al., 2005; Bonial et al., 2012; Pradhan et al., 2022), which lists for each verb its permitted role labels using numbered arguments (ARG0, ARG1, ...) instead of names, due to the difficulty of providing a small, predefined list of semantic roles that is sufficient for all verbs. Since numbered arguments are meant to have a verb-specific meaning (Yi et al., 2007), this implies that our mapping between numbered arguments and semantic roles may not always be consistent. The exact mapping used in our experiments is detailed in Appendix A. For co-reference, we similarly use the model provided by AllenNLP (Lee et al., 2017), which matches the output format of the SRL tagger. All experiments were carried out on a system with an Intel Xeon Silver 4210 CPU, two TITAN RTX GPUs (24 GB GPU VRAM each) and 64 GB of main memory. We run inference for the SRL model and co-reference component on separate GPUs. We report scores of all system and baseline variants across a single random seed only. Since we are comparing provided "plug-and-play" metrics, it is reasonable to assume that these are the primary choice for others evaluating their own datasets. Particularly for SRLScore, we further note that due to the system design, no fine-tuning or training is necessary. The only parameters varied during the experiments are thus the argument weights, which we describe in the following section.

heading: 4.4 System Variants
text: We compare with a number of generic automatic evaluation metrics, including BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005). Besides, we also consider several metrics specifically developed for factuality estimation, which have reported prior state-of-the-art correlation. Wherever possible, we reproduce scores with the official scripts provided by authors. Comparison is done with three variants of BARTScore (Yuan et al., 2021), two variants of CoCo (Xie et al., 2021), and two variants of ClozE (Li et al., 2022). For more details on reproducibility, see Appendix B. We chose each variant such that the highest self-reported scores of each paper on all evaluated datasets are considered.
For our own method, SRLScorebase represents a default setting, assigning equal weights wi = 17 to all attributes (agent, negation, relation, patient, recipient, time, location); the respective similarity function (exact match, spaCy vector, or ROUGE similarity) is chosen to maximize dataset-specific performance (see results of Table 2). SRLScorecoref uses the same weights, with co-reference enabled. We further provide model ablations to test various specifications of our models. As we could not find a implementation based on the original tuple extraction approach by Goodrich et al. (2019), we introduce SRLScoreopenie and SRLScoregoodrich as approximations of their method. Here, fact tuples are reduced to (agent, relation, patient) triplets
(with equal weights wi = 13 ). We note that this is not a true equivalence to the original method, although "[i]n most English sentences the subject is the agent" (Bates and Macwhinney, 1982); in reality, a broader variety of roles in the subject position may be encountered. The same applies for our mapping between object and the patient role. However, by using the same upstream labeling tool (i.e., the SRL model provided by AllenAI), we may more accurately compare the algorithmic scoring methods, independent of the annotation accuracy. We argue that our SRL-based modeling of relationship triplets allows for a better generalization beyond Wikipedia, which Goodrich et al. were using in their own experiments. The difference of SRLScoreopenie and SRLScoregoodrich lies in the implemented scoring function, where the OpenIE variant employs our own scoring algorithm, SRLScoregoodrich uses the preliminary filtering step defined in Goodrich et al. (2019). We do not apply a co-reference system in either one of the two ablation settings. Finally, SRLScorecoref-optimized illustrates the possibility of adapting our method to a particular dataset. For this variant, we optimize available hyperparameters (weights, scoring function, co-reference) in order to obtain the highest possible scores.

heading: 4.5 Main Results
text: The central evaluation results with recommended default settings are shown in Table 1. In almost all cases, specialized factuality metrics show higher
correlation than generic summarization evaluation metrics (ROUGE-1, BLEU and METEOR). Notably, despite the high increase in absolute scores, we do not always detect a significant level of improvement between factuality-specific metrics and generic metrics, particularly on QAGS-XSUM; we will discuss further implications of this in more detail later. When testing our own method, SRLScorebase, against generic metrics, we find strongly significant improvements only for Pearson correlation of QAGS-CNN/DM and SummEval, as well as Spearman correlation on SummEval (p < 0.01, with Bonferroni correction). It should be further noted that BARTScorecnn and CoCo results use BART models (Lewis et al., 2020) that were fine-tuned on the CNN/DailyMail corpus (respectively a variant fine-tuned on XSUM for CoCo on QAGS-XSUM); this may shift the results in favor of these methods for the particular dataset. In comparison, SRLScore does not make such assumptions, which may indicate a potentially stronger generalization to unseen datasets. The results in Table 1 also show that there are no significant differences between any of the factuality-specific metrics (SRLScore, BARTScore, and CoCo), particularly after applying Bonferroni correction for the comparison against several methods. These insights open up discussions about the current claims of "state-of-theart" performance, which may not be easily distinguished on the current evaluation datasets. We admit that there is likely no trivial solution to this (besides further annotations), as the main problem seems to stem from the high variance on small sample sizes.

heading: 4.6 Ablation Study
text: Given the limited expressiveness of the generic result evaluation, we perform a series of ablation studies on SRLScore, to support the individual algorithmic choices made in our method.
Extending Tuple Attributes. We investigate the assumption that semantic representations of sentences are usually far more complicated than the simplistic view of (agent, relation, patient) triplets, and the fact that errors may involve further roles. To this end, we compared SRLScoreopenie, using a triplet representation, against SRLScorebase with seven roles. The results in Table 2 confirm that extending tuples to cover more semantic roles is effective across datasets and metrics; SRLScorebase
scores consistently better than SRLScoreopenie, with significant improvements primarily on SummEval (the largest considered dataset).
Performance of Similarity Functions. Also seen in Table 2 is the difference in scores across various similarity functions. SRLScore achieves generally higher correlation when using vector (spaCy) or ROUGE similarity over exact matching, although not to a significant degree. These observations can be attributed to the hypothesis that abstractive entity references will not be detected by exact matching. Also note that results on QAGSXSUM are particularly affected by this, which shows higher levels of abstraction than CNN/DMderived resources (Wang et al., 2020; Pagnoni et al., 2021). This is also visible for the SRLScorecoref variant, as seen in Table 1, which can further improve the matching of re-formulations.
Dynamic Weight Re-Normalization. We next analyze the contribution of our dynamic weighting scheme through removing the weight renormalization Wnorm and instead defaulting to a static weighting on SRLScorebase. Results in Table 3 demonstrate that re-distributing static weights dynamically to present roles is very effective, however, results show no statistical significance.
Scoring Method QCNNDM QXSUM SummE
Ablation of Goodrich Scoring Method. We finally examine the performance of our scoring system against the partial matching approach of Goodrich et al. For fairness, we compare results on the reduced triplet sets. SRLScoreopenie uses the presented weighting function, SRLScoregoodrich implements an equivalent scoring to Goodrich et al. Results in Table 4 show that the presented scoring algorithm performs better than the scores determined by Goodrich’s approach on different datasets, in most instances to a significant degree.
Performance of Co-reference Resolution System. Results in Table 1 reveal that the coreference system is not always improving scores, particularly on the CNN/DailyMail-derived datasets. However, the use of co-reference resolution will significantly increase the processing time, as shown in Table 5. This is expected, given that there are now more fact tuples due to the tuple expansion; since the presented scoring method requires the comparison of each fact tuple in the summary against all input text tuples. We further compare the runtime against BARTScore, which only requires a single forward-pass through a neural net and can be batched easily, resulting in a 10x speed-up. In contrast, SRLScore requires construction and comparison the fact tuples, which are the main contributors for slower inference times.

heading: 4.7 Error Analysis
text: To better understand the limitations of our presented methods, we examine a number of instances manually, particularly those where there are large
differences between model-generated scores and human annotations on QAGS-XSUM. Table 6 shows two instances, where SRLScore respectively predicts a much higher and lower factuality score than human annotators. Notably, human raters tend to drastically reduce factuality scores in the presence of even a single mistake (what we refer to as "strike-out scoring"). In comparison, SRLScore and other factuality metrics tend to be more heavily influenced by the correctness of the majority of attributes, which can be seen as a "bottom-up scoring" (scores are built up from a initial factuality of zero instead of deducing from an initial score of one). On the other hand, highly abstractive samples, which retain factuality according to human raters, may pose a challenge for tuple-based SRLScore. In the second example of Table 6, synonymous expressions like step down instead of resign cause low predicted similarity; potential solutions could be found in verb sense disambiguation (Brown et al., 2011, 2022).

heading: 5 Conclusion and Future Directions
text: In this work, we presented a semantically consistent metric for estimating the factual truthfulness of two pieces of text: we applied our presented metric to the problem of text summarization evaluation, and demonstrated that it performs on par with existing approaches. In fact, we find that due to the small sample sizes of evaluation datasets, there are no significant differences between any of the considered state-of-the-art factuality estimation metrics. Our approach strikes with its relative simplicity and interpretability due to the intermediate representation of "fact tuples", which makes it possible for human annotators to review how or why system decisions were made. Furthermore, we have demonstrated the suitability of our approach over more naive tuple-based scoring methods through a series of ablation experiments, which also show the adaptability of our method to particular unseen settings by simply adjusting a series of parameters.
In our opinion, there are two key challenges concerning the effective deployment of SRLScore. The current implementation still suffers from impractically long runtimes for longer input texts. Notably, however, both the tuple generation and comparison stages can be parallelized and we are currently working on improving the compute effi-
ciency of our method. Secondly, we have seen a general trend that factuality estimation metrics are scoring differently from human annotators, who are putting heavy emphasis on a completely factual summary instead. We suspect that adopting a similar strike-out scoring for estimation may better correlate with human ratings, although it will require sufficiently accurate taggers to ensure correct recognition of all entities.
Limitations
While the presented method exhibits stable correlation with human judgments on some of the evaluated datasets, it still exhibits instances under which it will predict opposing factuality scores. It should therefore be considered an addition to human evaluation, but at this point not fully replace it. We also want to point out that the underlying summarization datasets that were used to compare human ratings on are known for their own set of limitations, particularly being fairly extractive in nature. This plays well with SRLScore’s estimation of matching between individual tuples extracted from single sentences; on the other hand, if summary texts contain facts derived from multiple source sentences (or undergo otherwise complex structural changes), fact tuples may be insufficient in their current form. Another limitation is the expressiveness of results on the fairly small human-annotated datasets. Here, statistically significant differences can rarely be obtained. However, we are to our knowledge the first to demonstrate this insight about (significant) differences between existing methods, which we consider a particularly useful insight for future work. We further want to point out that our method was only evaluated on English datasets; we argue that it
can be applied to other languages, given a similarly performing SRL labeling model. In practice, however, the existence of available models is currently limited for non-English languages.
Ethics Statement
The paper considers the automated analysis of factuality in generated text. While we see no imminent risk in the development of our presented method, we want to point to the explicitly spelled out limitations of the current method (see the previous section). The blind application of factuality metrics could be considered harmful in instances where the predicted scores are differing strongly from human ratings. We therefore recommend that factuality metrics should be employed purely as a complementary evaluation, and never directly replace analysis with humans in the loop.

heading: Acknowledgments
text: We thank the anonymous reviewers for their helpful comments and suggestions. The work of Jing Fan is supported by a scholarship of the China Scholarship Council (CSC).

heading: A Mapping of PropBank Arguments to Semantic Role Tuple Attributes
text: In our implementation, we extract sentence spans with label ARG0 as agent and spans with label ARG1 as patient. The extraction of time and location also does not pose any difficulties, because ARGM-TMP and ARGM-LOC are both given as modifiers that remain relatively stable across predicates (Jurafsky and Martin, 2009). However, as shown in Table 7, there is no one-to-one relationship between numbered arguments and the recipient role. For the sake of simplicity, we extracted elements with label ARG2 as recipient, because the probability that ARG2 correlates to recipient is the highest among all other possible roles (Yi et al., 2007).

heading: B Reproducing Scores of Related Work
text: We use the official scripts provided by the authors of BARTScore3 and CoCo4. Unfortunately, no public implementation exists at the time of writing for the work of Li et al. (2022), which prevents significance testing against ClozE models. For the work by (Goodrich et al., 2019), we similarly found no publicly available implementation; however, we note their wikipedia-based training data for generating fact extractors is available online5. When attempting to reproduce the scores of Xie et al. (2021), based on their own implementation, we encountered wildly differing scores compared to the values reported by the authors. Some results show drastic improvements from a reported Pearson correlation 0.58 to a reproduced score of 0.68, while other values dropped (e.g., on QAGSXSUM, we see a reduction of scores from 0.24 to 0.16 in terms of Pearson correlation). For the sake of reproducibility, we have included the exact commands that were used to run the CoCo models in our repository. On the other hand, all of our reproduced scores for BARTScore (Yuan et al., 2021) match the available self-reported results by the authors. For significance testing, we use our own implementation of a permutation-based significance test, again included in the code repository. We fix the initial NumPy random seed to 256, and compute results over 10,000 iterations for each test.
3https://github.com/neulab/BARTScore, last accessed: 2023-02-01.
4https://github.com/xieyxclack/factual_coco, last accessed: 2023-03-16.
5https://github.com/google-research-datasets/ wikifact, last accessed: 2023-05-17

heading: Future Work
text: However, we are to our knowledge the first to demonstrate this insight about (significant) differences between existing methods, which we consider a particularly useful insight for future work. We further want to point out that our method was only evaluated on English datasets; we argue that it
can be applied to other languages, given a similarly performing SRL labeling model. In practice, however, the existence of available models is currently limited for non-English languages