RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Acknowledgments
We thank the anonymous reviewers for their helpful comments and suggestions.
The work of Jing Fan is supported by a scholarship of the China Scholarship Council (CSC).
Figure 1: Visual explanation of SRLScore . An input text and its associated summary are transformed into a series of fact tuples ( SR Tuple ) through extraction from SRL (and optional co-reference) annotations. The final factuality score is computed based on the similarity of the summary facts with fact tuples generated from the input text.; Visual explanation of SRLScore . An input text and its associated summary are transformed into a series of fact tuples ( SR Tuple ) through extraction from SRL (and optional co-reference) annotations. The final factuality score is computed based on the similarity of the summary facts with fact tuples generated from the input text.
Figure 2: Examples of semantic role label annotations. Labels may remain consistent across different syntactic forms (Sentence 1 & 2). A single sentence can also include several relations at the same time (Sentence 3).; Examples of semantic role label annotations. Labels may remain consistent across different syntactic forms (Sentence 1 & 2). A single sentence can also include several relations at the same time (Sentence 3).
Figure 3: Example of the tuple expansion step through co-reference resolution. In addition to the original SR tuple, we add tuples with all possible permutations of the surface forms of mentioned entities.; Example of the tuple expansion step through co-reference resolution. In addition to the original SR tuple, we add tuples with all possible permutations of the surface forms of mentioned entities.
Pearson ( \rho ) and Spearman ( s ) correlation of metrics with human ratings on the evaluated datasets. Bold scores indicate highest absolute values. For SRLScore variants, we report highest scores across all similarity functions. No significant differences were found between the correlation scores of factuality-specific metrics. ∗ : results were taken from the respective paper, as there is no existing code to reproduce their results as of now.
Comparison of SRLScore with a simplified triplet representation (SRLScore openie ). Extending the fact tuples strictly improves correlation with human ratings across all similarity functions. Significance markers indicate improvements over the same similarity function of the openie variant.
Correlation scores of SRLScore base with and without weight re-normalization enabled.
Results of the ablation experiment comparing the scoring method by Goodrich et al. ( 2019 ) with our proposed scheme, based on triplet representations.
Average processing time (in seconds) per instance in QAGS-CNN/DM. SRLScore uses ROUGE similarity. BARTScore is run with a batch size of 4.
Examples from the QAGS-XSUM dataset where the majority vote of human ratings differs strongly from SRLScore ’s predicted factuality. Colored text segments highlight the position of relevant facts, where red text indicates a factual discrepancy between input and summary segments.
Mapping between numbered arguments in PropBank and semantic roles (Bonial et al., 2012 ) . Particularly the mapping of argument 2 makes simplifying assumptions about different verb forms.