RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
SRLScore
Our factual consistency metric, called SRLScore , is implemented as a two-stage process: first, extracting fact tuples using Semantic Role Labeling (SRL) on both the source texts and the summary texts, and then determining a factuality score based on tuple comparison.
The measure outputs human-interpretable scores between 0 and 1, where a higher score indicates greater factual consistency of a summary text.
In this section, we detail the algorithmic choices and present an adaptive weighting scheme for computing the final factuality scores.
Generating Fact Tuples with Semantic Role Labeling
As Figure 1 shows, we operate on the sentence level, primarily because existing SRL tools work well on this level of granularity (Shi and Lin, 2019 ; Xu et al., 2021 ) . The goal of our fact extractor is to produce a fact database comprised of semantic role tuples for each input text.

The primary task of SRL is to find all role-bearing constituents in a sentence and label them with their respective roles (Màrquez et al., 2008 ) . Typical semantic roles include agent , patient/theme , recipient , goal , instrument , manner , time , location and so on. From the many semantic labels available, we include seven roles based on availability in tagging schemes to construct a fact tuple: agent , negation , relation , patient , recipient , time , and location .
We further note that not every sentence needs to contain all of these roles; absent labels are represented by None in this work. Importantly, roles reveal the semantic relations between a predicate (verb) and its arguments, which implies that one can generate several fact tuples from a single sentence, depending on the number of verbs in it. To illustrate an exemplary fact tuple, the extracted semantic tuple from sentence 1 in Figure 2 is (Mueller, None, gave, a book, Mary, yesterday, in Berlin) .
Scoring Texts by Comparing Fact Tuples
Once fact tuples for both the input and summary texts are generated, the second step in our pipeline is to compute a factual accuracy score. We implement a dynamic weighting system, which crucially improves over a naive comparison, as we empirically show in Section 4.6 . Furthermore, we describe the drop-in replacements for exact matching during similarity computation.
Scoring Algorithm.
Given an input text R and summary text S , let {F_{R}} and {F_{S}} be fact databases , representing the semantic information contained in R and S , respectively. Individual fact tuples are represented as an ordered list of fact arguments, e.g., f = ( agent , negation , relation , patient , recipient , time , location ) \in F .
Particular arguments in a fact tuple are referred to by their index position, meaning agent =f^{0} , negation =f^{1} , and so on.
We further assume that there exists a scoring function that expresses the factual support of summary tuple f_{s} , given an input tuple f_{r} , denoted as S(f_{s}|f_{r}) .
To obtain a factuality score, we attempt to extract the best match \hat{f_{r}}\in F_{R} for each summary fact f_{s}\in F_{s} where \hat{f_{r}} maximizes the support score S(f_{s}|\hat{f_{r}} ).
Importantly, we differ from, e.g., Goodrich et al. ( 2019 ) , by considering the entirety of F_{R} , instead of subsets that match both the agent and relation of the fact tuple.
The factual accuracy is then the average across all maximized tuple scores in F_{S} .
With that, SRLScore is defined as:

\textbf{SRLScore}(R,S):=\frac{1}{|F_{S}|}\sum_{f_{s}\in F_{s}}\max_{f_{r}\in F_{R}}S(f_{s}|f_{r}) (1)

The final part of this scoring system is the computation of factual support S(f_{s}|f_{r}) . Tuples are scored by comparing the corresponding attributes of each tuple, formally:

S(f_{s}|f_{r}):=\sum_{i}\mathbbm{1}_{f_{s}^{i} eq None}\cdot sim({f_{s}^{i}},{f_{r}^{i}})\cdot{w_{i}}, (2)

where the summation over i addresses all attributes of the fact tuples, \mathbbm{1}_{f_{s}^{i}\neq None} represents an indicator function considering only non-empty arguments f^{i}_{s} (zero otherwise), and w_{i} assigns static weights to arguments in position i . Generally, it should be assumed that the weights allow for a maximum factuality score of 1, i.e., \sum_{i}w_{i}=1 . Finally, sim(f^{i}_{s},f^{i}_{r}) is the pairwise argument similarity of f^{i}_{s} and f^{i}_{r} . We consider different similarity metrics, as described in the following paragraphs.
Dynamic Weighting System.
The generic weighting in Equation 2 does not necessarily apply to the particular case of evaluating factual consistency in summarization, since a summary is still factually correct even if it leaves out particular aspects (e.g., dropping the date of an event), which were present in the input text.
With static weights, however, absent arguments are still contributing to the scoring of the tuple f_{s} , which means that leaving arguments out might potentially be considered as a penalization of factuality.
To address this issue, we introduce a weight re-normalization factor, {W_{norm}} ,
that distributes the static weights w_{i} across only those attributes that are present in the current summary fact.
In particular, this also increases penalties for actual mistakes over simple fact omission.
The weight normalization is defined as follows:

{W_{norm}}:=\frac{1}{\sum\limits_{i}\mathbbm{1}_{f_{s}^{i} eq None}\cdot{w_{i}}} (3)

With re-normalization enabled, we replace the existing computation of S(f_{s}|f_{r}) by the product W_{norm}\cdot S(f_{s}|f_{r}) .
String Similarity Methods.
We experiment with different methods to calculate the pairwise similarity sim(f^{i}_{s},f^{i}_{r}) : exact matching (in line with prior work), but also approximate matching functions, such as word vector similarity 1 1 1 We use spaCy’s vector similarity, see https://SpaCy.io/usage/linguistic-features#vectors-similarity , last accessed: 2023-03-06. and ROUGE-1 precision (Lin, 2004 ) . Computation of similarity with vectors and ROUGE each have their own respective strengths. Word vectors offer the highest flexibility in terms of recognizing argument similarity, enabling semantic comparison instead of purely syntactic equivalence. ROUGE-1 similarity does not offer the same level of flexibility in terms of matching, but shines with its comparatively faster computation, while still recognizing partial matches.
Improved Surface Form Invariance with Co-reference Resolution
In light of the fact that sentence-level SRL extraction misses co-references of the same entity across the texts, we integrate an optional component that takes co-reference resolution into account during the tuple generation.
Concretely, we employ an off-the-shelf co-reference resolution tool Lee et al. ( 2017 ) to identify and store all reference clusters in an external entity dictionary . There, all linguistic expressions that refer to the same entity will be grouped together, which allows for later disambiguation.
As shown in Figure 3 , if an extracted semantic role tuple contains co-references, a single fact tuple will be expanded into multiple tuples, representing the Cartesian product over all synonymous entity surface forms. The key idea here is to enable a better matching of potential facts across input texts and summaries, effectively increasing the recall of matches. The disadvantage is that this directly affects the runtime of our method by a strong factor, since the additional tuples in F_{S} and F_{R} will undoubtedly increase the number of comparisons.
Experiments
We empirically demonstrate the performance of our method through a number of experiments on two popular datasets for factual consistency evaluation, which are covered in this section. We further share implementation details and the choices for extracting SRL tuples and extracting co-reference clusters. In addition to the experimental analysis, we also study the behavior of SRLScore through a number of ablation experiments and a brief error analysis.
QAGS (Wang et al., 2020 ) .
The dataset comprises of two separate splits: the first contains 235 instances collected from the test split of CNN/DailyMail (Nallapati et al., 2016 ) , where each instance contains a source article and a model-generated summary using the bottom-up approach by Gehrmann et al. ( 2018 ) . A secondary set contains 239 further instances from the test split of XSUM (Narayan et al., 2018 ) , with generated summaries sampled from BART (Lewis et al., 2020 ) .
SummEval (Fabbri et al., 2021 ) .
It includes synthetic summaries from 16 different abstractive and extractive models of 100 randomly selected articles from the test split of CNN/DailyMail. Unlike QAGS, which collected annotations from MTurk 2 2 2 https://www.mturk.com/ , last accessed: 2023-03-06. , each SummEval sample was evaluated by five crowd-sourced annotators and three experts. For each summary, judges were asked to evaluate the coherence, consistency, fluency and relevance. For our evaluation, we use the expert ratings with regard to factual consistency as the gold score, based on the recommendation by Fabbri et al. ( 2021 ) .
Evaluation Metrics and Significance
In line with prior work, we evaluate metrics by computing Pearson correlation (denoted as \rho ) and Spearman correlation (denoted as s ) between model predictions and human reference ratings.
Given the limited size of all considered evaluation datasets, we further test results for significance using permutation tests Riezler and Maxwell ( 2005 ); Deutsch et al. ( 2021 ) , following the recommendation of Dror et al. ( 2018 ) .
In all tables, † denotes a significance level of 0.05 ( p<0.05 ) and ‡ a level of 0.01 ( p<0.01 ). When testing significance against several systems, we further apply Bonferroni correction of significance levels Dunn ( 1961 ) .
Implementation
We use AllenNLP Gardner et al. ( 2018 ) , specifically version 2.1.0, to extract semantic role labels. AllenNLP implements a BERT-based SRL tagger (Shi and Lin, 2019 ) , with some modifications. The output of AllenNLP uses PropBank convention (Palmer et al., 2005 ; Bonial et al., 2012 ; Pradhan et al., 2022 ) , which lists for each verb its permitted role labels using numbered arguments ( ARG0, ARG1, … ) instead of names, due to the difficulty of providing a small, predefined list of semantic roles that is sufficient for all verbs.
Since numbered arguments are meant to have a verb-specific meaning (Yi et al., 2007 ) , this implies that our mapping between numbered arguments and semantic roles may not always be consistent.
The exact mapping used in our experiments is detailed in Appendix A .
For co-reference, we similarly use the model provided by AllenNLP Lee et al. ( 2017 ) , which matches the output format of the SRL tagger. All experiments were carried out on a system with an Intel Xeon Silver 4210 CPU, two TITAN RTX GPUs (24 GB GPU VRAM each) and 64 GB of main memory. We run inference for the SRL model and co-reference component on separate GPUs. We report scores of all system and baseline variants across a single random seed only. Since we are comparing provided "plug-and-play" metrics, it is reasonable to assume that these are the primary choice for others evaluating their own datasets.
Particularly for SRLScore , we further note that due to the system design, no fine-tuning or training is necessary.
The only parameters varied during the experiments are thus the argument weights, which we describe in the following section.
System Variants
We compare with a number of generic automatic evaluation metrics, including BLEU (Papineni et al., 2002 ) , ROUGE (Lin, 2004 ) , and METEOR (Banerjee and Lavie, 2005 ) . Besides, we also consider several metrics specifically developed for factuality estimation, which have reported prior state-of-the-art correlation. Wherever possible, we reproduce scores with the official scripts provided by authors. Comparison is done with three variants of BARTScore (Yuan et al., 2021 ) , two variants of CoCo (Xie et al., 2021 ) , and two variants of ClozE (Li et al., 2022 ) . For more details on reproducibility, see Appendix B .
We chose each variant such that the highest self-reported scores of each paper on all evaluated datasets are considered.

For our own method, SRLScore base represents a default setting, assigning equal weights w_{i}=\frac{1}{7} to all attributes ( agent, negation, relation, patient, recipient, time, location ); the respective similarity function (exact match, spaCy vector, or ROUGE similarity) is chosen to maximize dataset-specific performance (see results of Table 2 ).
SRLScore coref uses the same weights, with co-reference enabled.
We further provide model ablations to test various specifications of our models.
As we could not find a implementation based on the original tuple extraction approach by Goodrich et al. ( 2019 ) , we introduce SRLScore openie and SRLScore goodrich as approximations of their method. Here, fact tuples are reduced to (agent, relation, patient) triplets (with equal weights w_{i}=\frac{1}{3} ).
We note that this is not a true equivalence to the original method, although "[i]n most English sentences the subject is the agent" (Bates and Macwhinney, 1982 ) ; in reality, a broader variety of roles in the subject position may be encountered. The same applies for our mapping between object and the patient role.
However, by using the same upstream labeling tool (i.e., the SRL model provided by AllenAI), we may more accurately compare the algorithmic scoring methods, independent of the annotation accuracy.
We argue that our SRL-based modeling of relationship triplets allows for a better generalization beyond Wikipedia, which Goodrich et al. were using in their own experiments.

The difference of SRLScore openie and SRLScore goodrich lies in the implemented scoring function, where the OpenIE variant employs our own scoring algorithm, SRLScore goodrich uses the preliminary filtering step defined in Goodrich et al. ( 2019 ) . We do not apply a co-reference system in either one of the two ablation settings.
Finally, SRLScore coref-optimized illustrates the possibility of adapting our method to a particular dataset. For this variant, we optimize available hyperparameters (weights, scoring function, co-reference) in order to obtain the highest possible scores.
Main Results
The central evaluation results with recommended default settings are shown in Table 1 .
In almost all cases, specialized factuality metrics show higher correlation than generic summarization evaluation metrics (ROUGE-1, BLEU and METEOR).
Notably, despite the high increase in absolute scores, we do not always detect a significant level of improvement between factuality-specific metrics and generic metrics, particularly on QAGS-XSUM;
we will discuss further implications of this in more detail later.
When testing our own method, SRLScore base , against generic metrics, we find strongly significant improvements only for Pearson correlation of QAGS-CNN/DM and SummEval, as well as Spearman correlation on SummEval ( p<0.01 , with Bonferroni correction). It should be further noted that BARTScore cnn and CoCo results use BART models (Lewis et al., 2020 ) that were fine-tuned on the CNN/DailyMail corpus (respectively a variant fine-tuned on XSUM for CoCo on QAGS-XSUM); this may shift the results in favor of these methods for the particular dataset. In comparison, SRLScore does not make such assumptions, which may indicate a potentially stronger generalization to unseen datasets.

The results in Table 1 also show that there are no significant differences between any of the factuality-specific metrics ( SRLScore , BARTScore, and CoCo), particularly after applying Bonferroni correction for the comparison against several methods.
These insights open up discussions about the current claims of "state-of-the-art" performance, which may not be easily distinguished on the current evaluation datasets.
We admit that there is likely no trivial solution to this (besides further annotations), as the main problem seems to stem from the high variance on small sample sizes.
Ablation Study
Given the limited expressiveness of the generic result evaluation, we perform a series of ablation studies on SRLScore , to support the individual algorithmic choices made in our method.
Extending Tuple Attributes.
We investigate the assumption that semantic representations of sentences are usually far more complicated than the simplistic view of ( agent , relation , patient ) triplets, and the fact that errors may involve further roles. To this end, we compared SRLScore openie , using a triplet representation, against SRLScore base with seven roles. The results in Table 2 confirm that extending tuples to cover more semantic roles is effective across datasets and metrics; SRLScore base scores consistently better than SRLScore openie , with significant improvements primarily on SummEval (the largest considered dataset).
Performance of Similarity Functions.
Also seen in Table 2 is the difference in scores across various similarity functions. SRLScore achieves generally higher correlation when using vector (spaCy) or ROUGE similarity over exact matching, although not to a significant degree. These observations can be attributed to the hypothesis that abstractive entity references will not be detected by exact matching. Also note that results on QAGS-XSUM are particularly affected by this, which shows higher levels of abstraction than CNN/DM-derived resources (Wang et al., 2020 ; Pagnoni et al., 2021 ) . This is also visible for the SRLScore coref variant, as seen in Table 1 , which can further improve the matching of re-formulations.
Dynamic Weight Re-Normalization.
We next analyze the contribution of our dynamic weighting scheme through removing the weight re-normalization W_{norm} and instead defaulting to a static weighting on SRLScore base . Results in Table 3 demonstrate that re-distributing static weights dynamically to present roles is very effective, however, results show no statistical significance.
Ablation of Goodrich Scoring Method.
We finally examine the performance of our scoring system against the partial matching approach of Goodrich et al. For fairness, we compare results on the reduced triplet sets. SRLScore openie uses the presented weighting function, SRLScore goodrich implements an equivalent scoring to Goodrich et al. Results in Table 4 show that the presented scoring algorithm performs better than the scores determined by Goodrich’s approach on different datasets, in most instances to a significant degree.
Performance of Co-reference Resolution System.
Results in Table 1 reveal that the co-reference system is not always improving scores, particularly on the CNN/DailyMail-derived datasets. However, the use of co-reference resolution will significantly increase the processing time, as shown in Table 5 . This is expected, given that there are now more fact tuples due to the tuple expansion ; since the presented scoring method requires the comparison of each fact tuple in the summary against all input text tuples.
We further compare the runtime against BARTScore, which only requires a single forward-pass through a neural net and can be batched easily, resulting in a 10x speed-up. In contrast, SRLScore requires construction and comparison the fact tuples, which are the main contributors for slower inference times.
Error Analysis
To better understand the limitations of our presented methods, we examine a number of instances manually, particularly those where there are large differences between model-generated scores and human annotations on QAGS-XSUM. Table 6 shows two instances, where SRLScore respectively predicts a much higher and lower factuality score than human annotators. Notably, human raters tend to drastically reduce factuality scores in the presence of even a single mistake (what we refer to as "strike-out scoring" ). In comparison, SRLScore and other factuality metrics tend to be more heavily influenced by the correctness of the majority of attributes, which can be seen as a "bottom-up scoring" (scores are built up from a initial factuality of zero instead of deducing from an initial score of one).
On the other hand, highly abstractive samples, which retain factuality according to human raters, may pose a challenge for tuple-based SRLScore .
In the second example of Table 6 , synonymous expressions like step down instead of resign cause low predicted similarity; potential solutions could be found in verb sense disambiguation (Brown et al., 2011 , 2022 ) .
Conclusion and Future Directions
In this work, we presented a semantically consistent metric for estimating the factual truthfulness of two pieces of text: we applied our presented metric to the problem of text summarization evaluation, and demonstrated that it performs on par with existing approaches.
In fact, we find that due to the small sample sizes of evaluation datasets, there are no significant differences between any of the considered state-of-the-art factuality estimation metrics.
Our approach strikes with its relative simplicity and interpretability due to the intermediate representation of "fact tuples", which makes it possible for human annotators to review how or why system decisions were made.
Furthermore, we have demonstrated the suitability of our approach over more naive tuple-based scoring methods through a series of ablation experiments, which also show the adaptability of our method to particular unseen settings by simply adjusting a series of parameters.

In our opinion, there are two key challenges concerning the effective deployment of SRLScore .
The current implementation still suffers from impractically long runtimes for longer input texts. Notably, however, both the tuple generation and comparison stages can be parallelized and we are currently working on improving the compute efficiency of our method.
Secondly, we have seen a general trend that factuality estimation metrics are scoring differently from human annotators, who are putting heavy emphasis on a completely factual summary instead.
We suspect that adopting a similar strike-out scoring for estimation may better correlate with human ratings, although it will require sufficiently accurate taggers to ensure correct recognition of all entities.
Limitations
While the presented method exhibits stable correlation with human judgments on some of the evaluated datasets, it still exhibits instances under which it will predict opposing factuality scores. It should therefore be considered an addition to human evaluation, but at this point not fully replace it. We also want to point out that the underlying summarization datasets that were used to compare human ratings on are known for their own set of limitations, particularly being fairly extractive in nature.
This plays well with SRLScore ’s estimation of matching between individual tuples extracted from single sentences; on the other hand, if summary texts contain facts derived from multiple source sentences (or undergo otherwise complex structural changes), fact tuples may be insufficient in their current form. Another limitation is the expressiveness of results on the fairly small human-annotated datasets. Here, statistically significant differences can rarely be obtained. However, we are to our knowledge the first to demonstrate this insight about (significant) differences between existing methods, which we consider a particularly useful insight for future work.

We further want to point out that our method was only evaluated on English datasets; we argue that it can be applied to other languages, given a similarly performing SRL labeling model. In practice, however, the existence of available models is currently limited for non-English languages.
Ethics Statement
The paper considers the automated analysis of factuality in generated text.
While we see no imminent risk in the development of our presented method, we want to point to the explicitly spelled out limitations of the current method (see the previous section). The blind application of factuality metrics could be considered harmful in instances where the predicted scores are differing strongly from human ratings. We therefore recommend that factuality metrics should be employed purely as a complementary evaluation, and never directly replace analysis with humans in the loop.