FENICE: Factuality Evaluation of Summarization Based on Natural Language Inference and Claim Extraction (2024)

Abstract: Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automaticallygenerated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLMbased metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization. In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our factuality annotations of long-form summarization at https://github.com/Babelscape/FENICE.

heading: 1 Introduction
text: In recent years, Natural Language Generation (NLG) approaches have achieved impressive results in many areas of Natural Language Processing (NLP), including Machine Translation (MT), and Question Answering (QA). When it comes to Text Summarization, the landscape differs somewhat. Despite the high performance boasted by conventional metrics such as ROUGE (Lin, 2004), several empirical investigations (Maynez et al., 2020;
Durmus et al., 2020; Kryscinski et al., 2020) revealed that a significant portion of automaticallygenerated summaries contains factual inconsistencies, e.g., hallucinations. Addressing this issue is crucial, as it stands as a barrier preventing the widespread application of such systems in realworld scenarios.
With the emergence of Large Language Models (LLMs) the challenge still persists, as their outputs continue to exhibit inconsistencies (Tam et al., 2022). What is concerning is that the heightened fluency boasted by these models (Wang et al., 2023b) may inadvertently lead inexpert users to place excessive trust in their outputs, thereby fostering the spread of misinformation. For these reasons, the introduction of reliable automatic metrics that detect factual inaccuracies in summaries is becoming increasingly urgent.
As a response to this need, various factuality metrics have emerged, leveraging methodologies rooted in Question Answering (Scialom et al., 2021; Fabbri et al., 2022; Wang et al., 2020), Natural Language Inference (Zha et al., 2023; Chen and Eger, 2023; Laban et al., 2022, NLI), and semantic representations (Ribeiro et al., 2022; Goyal and Durrett, 2020; Fan et al., 2023). The latest trend in this domain involves works (Fu et al., 2023; Liu et al., 2023a; Gao et al., 2023) that introduce prompt engineering approaches using LLMs.
However, current factuality metrics suffer from one or more of the following drawbacks: i) they often assign a broad score to the entire summary without specifying which parts are factual or hallucinated, thereby lacking interpretability; ii) they are mainly designed for evaluating summaries of short documents, such as news articles, and iii) they are impractical from a computational standpoint, specifically in the case of LLM-based metrics.
To overcome these limitations, we introduce Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction
14148
(FENICE), a state-of-the-art factuality metric that is more interpretable and efficient. As depicted in Figure 1, FENICE evaluates the factuality of a summary by leveraging an NLI-based alignment between information in the source document and a collection of atomic facts, referred to as claims, extracted from the summary. Our metric aligns claims from the summary with text from the input at three different levels of granularity: sentence, paragraph, and document. This novel alignment strategy facilitates interpretability: when presented with a claim, the metric user can either verify which specific part of the input document entails it, or, in the presence of hallucinations, they can observe that no alignment is provided by the metric.
Not only does our approach enhance interpretability but it also yields a positive impact on overall performance: FENICE achieves the highest average result in the AggreFact benchmark (Tang et al., 2023), the de-facto benchmark for summarization factuality evaluation. Furthermore, we evaluate the performance of our metric in a more challenging context, specifically targeting long-form summarization. To accomplish this, we conduct a human annotation process aimed at assessing long-form summarization factuality. When confronting the metrics outputs with the human labels, FENICE exhibits higher accuracy compared to its competitors, paving the way for future approaches for long document summarization evaluation.
In summary, our contributions are the following:
• We introduce FENICE, a state-of-the-art factuality metric for summarization evaluation;
• The alignments provided by our metric facilitate interpretability, highlighting the input document sections that are useful for summary claims verification;
• FENICE is designed to be computationally efficient, utilizing approximately 700 million parameters, thereby ensuring its applicability in real-world applications.
• We show the effectiveness of FENICE in the evaluation of long-form summarization factuality through a human annotation process.
• In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our human annotations at https://github.com/Babelscape/ FENICE.

heading: 2 Related Work
text: NLI-based metrics. Drawing upon recent developments in the NLI field, several studies have introduced novel factuality metrics that exhibit stronger correlations with human judgments. Chen and Eger (2023) proposed a series of NLI metrics tailored for both MT and summarization evaluation. They conducted experiments in both reference-based and reference-less settings. In the latter setting, which serves as the standard for consistency evaluation,
they utilized the input document as the premise for the NLI system and the entire summary as the hypothesis for verification. While this approach correlates well with human consistency judgments, it lacks interpretability: on the one hand, when a summary is inconsistent, the end user is not provided with the parts of it that are deemed non-factual; on the other hand, when a summary is factually accurate, the metric provides no indication of the specific sections in the document that entail it.
In the same context, AlignScore (Zha et al., 2023) provides a novel alignment system, which is trained on several verification tasks, including NLI. While it achieves high performance in factuality evaluation, AlignScore also falls short in terms of interpretability, providing an overall score for the entire summary, which prevents users from identifying specific factual or non-factual spans within it. Moreover, it aligns the summary with large blocks of text from the input document, making it hard for users to gain insights from the output.
The Dependency Arc Entailment (Goyal and Durrett, 2020, DAE) metrics leverage the entailment between the dependency parse trees of document sentences and the summary. However, the highly abstractive nature of summaries, often encapsulating multiple facts in a single sentence, introduces a drawback, making their sentence-level comparison suboptimal.
Despite the significant progress achieved so far, we posit that there is still considerable room for enhancing NLI-based metrics, encompassing improvements in both performance and interpretability.
LLM-based metrics. Several studies (Shen et al., 2023; Chen et al., 2023; Wang et al., 2023a) have recently analyzed the efficacy of LLMs as summarization evaluators in multiple evaluation settings. Their experiments demonstrate that, while LLMs outperform existing automatic metrics, they are not yet ready to be considered reliable alternatives for human evaluation. Starting from these findings, TrueTeacher (Gekhman et al., 2023) analyzed the LLMs’ capability of generating large-scale factuality datasets and then trained smaller student models on such data, outperforming existing NLI metrics.
Another trend of approaches revolves around prompt engineering strategies (Luo et al., 2023; Fu et al., 2023; Liu et al., 2023a; Gao et al., 2023), where LLMs are typically instructed with the evaluation task description, the input document, and
the system-generated summary. Specifically, Fu et al. (2023) demonstrated the capabilities of such models in achieving multi-aspect, customized, and training-free evaluation. Relatedly, the recentlyintroduced G-Eval (Liu et al., 2023a) experimented with chain-of-thoughts (CoT) and form-filling instruction paradigms. They showed that G-Eval, configured with GPT-4, outperforms all previous methods in terms of correlation with the human consistency annotations in SummEval (Fabbri et al., 2021).
While achieving high correlations with human judgments, LLM-based metrics raise concerns about their real-world applicability. Firstly, many of these studies rely on closed-source LLMs, and the lack of information about the composition of their training dataset raises uncertainty: there is no guarantee that the test examples used in these studies are actually unseen at training time for these models. Additionally, their exceptionallyhigh computational cost poses a significant constraint on their practical deployment and hampers their reproducibility for research purposes.
Our approach differs from previous ones in that it utilizes an LLM solely for the task of claim extraction, rather than also utilizing it for generating the factuality annotations, which it may have already seen during training. Furthermore, we provide a distilled version of this LLM—a more compact generative Transformer—trained to emulate the output of the LLM, which ensures the practicality and reproducibility of our approach.
Claim extraction-based evaluation. A growing number of works explores the employment of claim extraction for the purpose of evaluating generated text. Among these, Min et al. (2023) proposed a method for evaluating long-form text generation, through the validation of their extracted claims given a reliable knowledge source, e.g., Wikipedia.
In the context of summarization evaluation, Nenkova et al. (2007) introduced Pyramid, the first manual annotation protocol in this context. However, the scope of Pyramid and its automatic approximations (Gao et al., 2019; Harnly et al., 2005) is to evaluate summarization relevance, which differs significantly from the factuality-oriented focus of our study.
To the best of our knowledge, we are the first to consider claims as the foundation for a factualityoriented summarization metric.

heading: 3 FENICE
text: We now provide an overview of the FENICE algorithm. Initially, we extract a list of atomic facts, referred to as claims, from a given summary (Section 3.1). Subsequently, we detail the process of obtaining a factuality score for each extracted claim. This involves leveraging NLI, with the claim serving as the hypothesis, and the document sentences (Section 3.2) or longer blocks of text at different levels of granularity (Section 3.4) serving as premises. In Section 3.3, we elaborate on a refinement of the obtained scores through coreference resolution.

heading: 3.1 Claim extraction
text: Following the Atomic Content Unit (ACU) definition outlined in Liu et al. (2023b), we use the term claim to denote an atomic fact, i.e., an elementary information unit found in a summary, that does not require further subdivision. Figure 1(a) depicts an example list of claims extracted from a summary.
We frame the claim extraction problem as an end-to-end autoregressive generation task. Given a prompt consisting of the task description and the summary to be processed, the model generates a sequence of claims expressed within the provided summary. To guide the model in meeting the desired output format and enhancing its task comprehension, we include an example of an input summary and the expected claims in the prompt (see Appendix A).
We opt to center our evaluation strategy around claims rather than sentences, primarily because of their enhanced interpretability and their atomic nature, which facilitates alignment with information from the input.

heading: 3.2 NLI-based claim scoring
text: The NLI formulation lies at the core of our novel claim scoring algorithm. NLI is a task that involves determining the logical relationship between two text segments: a premise and a hypothesis. Formally, given a premise pre and a hypothesis hyp:
NLI(pre, hyp) = (pent, pneu, pcon) (1)
where pent, pneu, and pcon denote the respective probabilities of pre entailing, being neutral about, or contradicting hyp.
Starting from Equation 1, we define NLIScore as the difference between the entailment and contradiction probabilities, obtained using the document
sentence as premise and the claim extracted from the summary as hypothesis:
NLIScore(s, c) = pent − pcon
where the premise s is a sentence of the input document, and the hypothesis c is a claim extracted from the summary. NLIScore measures the degree to which a specific premise can be utilized to verify the claim c. The inclusion of the contradiction probability serves the purpose of penalizing claims that are in conflict with any content in the source document.
Finally, we introduce FENICEsent as the function that assigns a score to a given claim c by considering the sentences in the source document D as premises:
FENICEsent(D, c) = max s∈D NLIScore(s, c) (2)
We also define alignsent as the corresponding alignment function:
alignsent(D, c) = argmax s∈D NLIScore(s, c) (3)
For each claim c from the summary, this establishes a one-to-one alignment between c and the highestscoring sentence from the input document.

heading: 3.3 Coreference Resolution
text: Importantly, given a claim c and its aligned sentence s = alignsent(D, c), NLIScore(s, c) may not reflect the true degree of entailment for the (s, c) pair due to an entity or concept being mentioned in the two texts with distinct surface forms. For instance, consider the claim in Figure 1(a): "England number eight Billy Vunipola is set to return from injury" and its aligned source document sentence: "There were initial fears the 24-years-old player would miss [...]". In this case, the NLI model assigns only a moderate score of 0.6 to this pair due to the limited context, making it challenging to establish a clear correspondence between "Billy Vunipola" and "the 24-years-old player".
To tackle this challenge, we apply a Coreference Resolution model on the source document, which identifies clusters of mentions that refer to the same entity or concept. We then introduce a coreferenceresolution-aware version of Equation 2 as follows:
s∗ = alignsent(D, c)
FENICEcoref(D, c) = max s′∈Co(s∗)
NLIScore(s′, c)
(4)
where Co(s∗) is the set of sentences obtained by replacing each entity occurring in sentence s∗ with all its coreferential mentions.1
For instance, given "the 24-years-old player" we identify its coreferential spans in the document, such as "Billy Vunipola". Subsequently, we generate an alternative version of the original sentence by replacing "the 24-years-old player" with "Billy Vunipola", resulting in the sentence: "There were initial fears Billy Vunipola would miss [...]". As depicted in Figure 1(b), this approach facilitates the alignment process of the claim "England number eight Billy Vunipola is set to return from injury" to the revisited version of the sentence, which contains the same span for the given named entity.

heading: 3.4 Aligning claims across multiple input text granularities
text: Given the abstract nature of summaries, the verification of a claim may require a context that extends beyond a single sentence from the input. For example, to verify the claim: "Regular exercise contributes to overall well-being", the following input passage would be necessary: "Studies have consistently shown that individuals who engage in regular physical activity experience improved mental health. Furthermore, regular exercise is linked to a reduced risk of chronic diseases and enhanced cardiovascular health.".
Therefore, we define the following variant of Equation 2 for longer passages of k sentences:
FENICEp(D, c, k) = max i=1,...,|D|−k+1
NLIScore(si+k−1i , c)
(5)
where si+k−1i is the concatenation of si, . . . , si+k−1, i.e., the k consecutive sentences starting from sentence i in document D.
Our approach involves choosing k such that k < |D|, alongside considering the entire document as premise, i.e., k = |D|. Thus, our function for scoring claims given passages of multiple lengths from the source document is given by the following:
FENICEmul(D, c, j) = max{FENICEp(D, c, j), FENICEp(D, c, |D|)} (6)
where j is the desired length for the contiguous blocks of the source sentences. Then, we define
1We limit this augmentation to the sentence s∗, which is the result of Equation 3.
FENICEcl as the factuality score assigned to a claim c:
FENICEcl(D, c, j, T ) =    FENICEmul(D, c, j) if FENICEcoref (D, c) < T ;
FENICEcoref (D, c) otherwise. (7)
where T is a threshold. We introduce T to limit the application of FENICEmul to only a subset of claims that requires further verification, i.e., the low-scoring ones. We set j = 5 and T = 0.8 in all our experiments (Section 4.2) as a result of tuning them on the validation splits.
Finally, given the set of extracted claims C from a summary S, and a source document D, we introduce FENICE as:
FENICE(D,C, j, T ) =
∑ c∈C FENICEcl(D, c, j, T )
|C| (8)
which is the factual consistency score attributed by our metric to summary S.

heading: 4 Experiments and Results

heading: 4.1 Claim extraction
text: Experimental setup. As detailed in Section 3.1, we formulate the claim extraction problem as a sequence generation task. To accomplish this, we designed a prompt that includes the task description, an input-output example, and the summary to be processed. The prompt utilized is available in Appendix A. We experiment with GPT-3.5-turbo as claim extractor.2 We additionally provide a distilled version of the claim extractor, which consists of a T5base (Raffel et al., 2019) model trained on the LLM outputs. To create the training and evaluation datasets for the distilled model, we apply the LLM-based claim extractor on the reference summaries from two widely-used summarization datasets, namely XSum (Narayan et al., 2018) and CNN/DailyMail (Nallapati et al., 2016). We split the dataset obtained into training, validation, and test sets following a 80%-10%-10% proportion, comprising 430,946 summary-claims pairs for training, and 53,868 for both the validation and test partitions. We train our system on a single GPU NVIDIA GeForce RTX 3090 for a total of 1M steps, using AdaFactor (Shazeer and Stern, 2018) as optimizer with learning rate of 1 · 10−5.
2We use the gpt-3.5-turbo API, version dated 2nd December 2023.
Datasets. We evaluate our distilled version of the claim extractor on the test split of the LLMgenerated claim extraction dataset. We additionally assess the performance of both the LLMbased claim extractor and our distilled version on ROSE (Liu et al., 2023b), a dataset that includes summaries and human-written claims among its annotations. While this dataset was designed to serve as the basis for a human annotation protocol for summarization evaluation, we utilize the released manually-extracted claims for assessing our systems’ performance.
Metrics. Given the extractive nature of this task, we consider ROUGE (Lin, 2004) as the basis for a suitable evaluation metric for claim extraction. Following (Zhang and Bansal, 2021) we define easinessP as the score obtained by computing the ROUGE-1 metric for each pair of system-generated and gold standard claims, subsequently calculating the maximum, and averaging over the total number of generated claims:
easinessP (S,H) = ∑ c∈S maxc∗∈H R1(c, c ∗)
|S| (9)
where R1 is the ROUGE-1-F1 metric, and S, H are the sets of system- and human-written claims, respectively. We then introduce easinessR as the recall-oriented version of Equation 9, obtained by averaging over the gold standard claims:
easinessR(S,H) = ∑ c∗∈H maxc∈S R1(c, c ∗)
|H| (10)
Results and discussion. In Table 1 we report the results of the easinessP , and easinessR metrics, along with the F1 version of Equations 9 and 10, i.e., easinessF1, obtained on ROSE. In Appendix B we additionally report the results obtained when using BERTScore (Zhang et al., 2020) as backbone metric for our evaluation instead of ROUGE.
As we can see from Table 1, our distilled version performs comparably with the original one based on GPT-3.5. These results indicate the effectiveness of the distillation process, ensuring that our model can be used reliably for extracting claims. To further evaluate the quality of the claims generated by our distilled model, we compare them to those generated by the original GPT-3.5 model, using the latter as references. The results of this evaluation are presented in Table 2, demonstrating the strong ability of our T5 model to replicate the claims generated by GPT-3.5.

heading: 4.2 Factuality Evaluation
text: Experimental setting. We leverage an off-theshelf model trained on NLI to align the summaryextracted claims with text from the input document. Specifically, we employ a version of DeBERTa-v3large (He et al., 2020) fine-tuned on multiple NLI datasets,3 comprising MultiNLI (Williams et al., 2017), LingNLI (Parrish et al., 2021), WANLI (Liu et al., 2022), and Fever-NLI, as well as AdversarialNLI, both introduced in Nie et al. (2019). We adopt spaCy’s en-core-web-sm to segment the input documents into sentences, and F-Coref (Otmazgin et al., 2022) for the extraction of the coreferential spans (Section 3.3).
Evaluation datasets. We assess the effectiveness of FENICE on AGGREFACT (Tang et al., 2023), the de-facto benchmark for evaluating summarization factuality metrics. AGGREFACT comprises multiple datasets of news articles and manually annotated summaries with factuality labels. The benchmark is categorized into three splits based on the summarization systems employed: i) FTSOTA, which includes the outputs of state-of-the-art pretrained summarizers, ii) EXFORMER, consisting of summaries of early Transformer-based summarization models, and iii) OLD, which comprises the
3We loaded the pre-trained model weights from the Hugging Face Hub, available at: https://huggingface.co/MoritzLaurer/ DeBERTa-v3-large-mnli-fever-anli-ling-wanli
outputs of older approaches.
Baselines. We evaluate the performance of FENICE against several baselines that align with recent trends in the field, specifically focusing on NLI, QA, and LLM-based metrics.
NLI-based metrics. We evaluate several NLIbased metrics, such as AlignScore (Zha et al., 2023), MENLI (Chen and Eger, 2023), SummaCZS and SummaC-Conv, both introduced in (Laban et al., 2022), and DAE (Goyal and Durrett, 2020).
QA-based metrics. Question Answeringbased factuality metrics typically depend on a Question Generation (QG) model designed to formulate questions based on the input document as context, and check whether information in the summary can be used to answer such questions, and vice versa. For our study, we choose the top-performing QA metric on AGGREFACT as baseline, namely QAFactEval (Fabbri et al., 2022).
LLM-based metrics. We benchmark recent ChatGPT-based evaluation metrics: ChatGPT-ZS and ChatGPT-CoT (Luo et al., 2023) which leverage GPT-3.5 to generate a binary factuality label, in zero-shot and Chain-of-Thought paradigms, respectively. We also include ChatGPT-DA and ChatGPT-
Star (Wang et al., 2023a), which prompt LLMs to generate factuality scores on a scale of 0-100 and 1-5, respectively. We additionally include the results of the 11B-parameter T5 version of the TrueTeacher metric.
Results and discussion. We assess the performance of two distinct variants of our metric: FENICEGPT_claims, which employs LLMbased claims, and FENICET5_claims, featuring the knowledge-distilled claim extractor. To convert FENICE scores into binary labels, we tune a different threshold on each validation split of the AGGREFACT datasets (i.e., FTSOTA, EXFORMER, and OLD) and then compute the balanced accuracy on the corresponding test set, as indicated in Tang et al. (2023).
In Table 3, we present the balanced accuracy (Brodersen et al., 2010) scores on the AGGREFACT test sets. FENICEGPT_claims sets a new state-of-the-art performance on AGGREFACT, resulting in the highest average balanced accuracy score.
Moreover, we specifically analyze the metric’s ability to evaluate the outputs of state-of-the-art models. Following the recommendation of Tang et al. (2023), we report the results of our metrics on AGGREFACT-FTSOTA split in the single-threshold setting. As outlined by Tang et al. (2023), we fine-tune a threshold on the validation sets of the CNN-FTSOTA and XSUM-FTSOTA splits and subsequently apply it to convert our metric scores into
binary labels on the test set. From the outcomes presented in Table 4, we note that FENICEGPT_claims obtains the best average results in this split, also due to its high performance on XSUM-FTSOTA. Notably, we consider this latter setting to be the most challenging for a metric, given that it comprises summaries of state-of-the-art systems trained on XSum references, which exhibit the highest level of abstraction. In this context, FENICET5_claims slightly outperforms TrueTeacher, despite utilizing significantly fewer parameters (700M vs 11B).

heading: 4.3 Ablation study
text: We conduct an analysis of the impact of our primary design choices, namely, claim extraction, coreference resolution, the alignment with multiple text granularities, and our backbone NLI model.
First, we introduce NLIsent, a simple baseline leveraging NLI-based sentence-level alignment between the summary and the input document. Specifically, the factuality score for a summary sentence s′ is obtained through the following variant of Equation 2, considering summary sentences as hypotheses instead of claims:
NLIsent(D, s′) = max s∈D NLIScore(s, s′) (11)
As outlined in Section 3.4, we obtain the factuality score for a summary by averaging the result of Equation 11 applied to its sentences.
We then assess the impact of focusing the factuality evaluation on claims extracted from the summary rather than its sentences. To achieve this, we introduce NLIclaim, a modification of the previous baseline that assigns factuality scores to the claims using Equation 2. Subsequently, we evaluate the performance of NLIcoref , derived by integrating the coreference resolution strategy (Section 3.3) into the previous baseline. Additionally, we assess the impact of including the alignment strategy with input text at multiple granularities (Section 3.4) by presenting the performance of FENICE. Table 5 displays the balanced accuracy scores of FENICE on the AGGREFACT- FTSOTA split, comparing them to the aforementioned baselines. The values in the ∆AV G column represent the differences in terms of average balanced accuracy resulting from the incremental inclusion of each component. The results indicate that each component contributes significantly to the final performance. The alignment strategy with input text at multiple granularity has the most substantial impact, resulting in a signifi-
cant gain of 10.1 points when included. Additionally, we evaluate the performance of FENICE utilizing RoBERTa as the NLI model for our alignments (FENICERoBERTa), instead of DeBERTa (Appendix C).

heading: 4.4 Long-form summarization evaluation
text: In recent years, there has been a noticeable shift in research focus away from summarizing short texts, predominantly news articles, to handling longer documents like scientific papers (Gupta et al., 2021) and books (Kryściński et al., 2022; Scirè et al., 2023). Despite this rapid evolution in the development of approaches and resources, the evaluation of summarization has been slower to adapt, with only a limited number of works, which are primarily aimed at defining the basic guidelines for this task (Krishna et al., 2023).
Human evaluation. As an initial step towards addressing this gap, we conducted an evaluation of our metric in the context of long-form text summarization. To do so, we curated a set of annotations for assessing summaries of lengthy texts.
Drawing upon Echoes from Alexandria (Scirè et al., 2023), a multilingual resource for book summarization, we selected 26 English narratives, each paired with a reference abstractive summary. The reference summaries were refined by removing irrelevant details such as the story title, the genre, and the author’s name. As an example, the initial summary "Isaac Asimov’s ’Let’s Get Together’ is a science fiction novel which depicts the continuation of Cold War hostilities..." underwent manual editing and was refined to "The story depicts the continuation of Cold War hostilities...". To augment our selection, we incorporated summaries generated by 12 book summarization systems, as presented by Scirè et al. (2023), applied to our se-
lected stories. The systems chosen were versions of BART (Lewis et al., 2020), LongT5 (Guo et al., 2022), and LED (Beltagy et al., 2020), fine-tuned on the Echo-XSum split of the Echoes from Alexandria dataset. This process resulted in a comprehensive set of 338 text-summary pairs.
Next, we tasked an expert annotator with reading all the stories and assigning a binary factuality label to each of the associated summaries.4 As the foundation for the annotation process, we considered a summary factual if its content was explicitly stated or implied by the source document; otherwise, it was labeled as non factual. Through this process, we acquired factuality labels for all the 338 text-summary pairs.
Moreover, in order to measure the interannotator agreement we asked a second expert to manually annotate a subset of 52 summaries. We then computed the Cohen’s kappa coefficient, obtaining a moderate agreement of 0.74. We deemed this outcome reasonable, particularly when considering the high complexity of assessing long-form summarization compared to the standard setting. Table 6 compares the length of source texts in our proposed long-form summarization evaluation dataset with those in existing datasets, specifically SummaC (Laban et al., 2022) and SummEval (Fabbri et al., 2021). In our dataset, annotators were tasked with reading very long source texts, averaging 664 sentences, in order to assess the factuality of the provided summaries. The distribution of factual and unfactual labels in our dataset is detailed in Appendix D. To align with the evaluation setup of AGGREFACT, we partitioned our annotations into test and validation sets using an 80/20 split.
Results and discussion. We follow the experimental setup outlined in Section 4.2. We employ
4We supplied the annotator with the PDF files containing the text of the stories. We paid the annotator according to the standard salary for their geographical location. The annotator took an average of 2 hours to read a story, and an average of 5 minutes to annotate the corresponding summaries.
AlignScore, MENLI, and DAE as baselines, considering their relatedness to our work. In Table 7, we present the balanced accuracy results on the test split of our dataset. The results obtained highlight that FENICE metrics outperform the baselines, with FENICEGPT_claims obtaining the highest performance and FENICET5_claims coming second. Upon manual inspection of our alignments, we observe that a significant proportion of claims are aligned with source text having the length of a paragraph. We posit that this behaviour is due to the high abstractiveness of the stories’ summaries, which typically require contexts that are longer than a sentence in order to be verified. We hypothesize that our metric is well-suited for both standard and long-form summarization thanks to its adaptive capacity to choose the appropriate input text granularity for claim verification.

heading: 5 Conclusion and Future Work
text: We introduced FENICE, a novel metric designed for the factuality evaluation of summarization. By leveraging NLI-based alignments between claims extracted from the summary and the source text at multiple levels of granularities, FENICE facilitates interpretability. Our experiments demonstrate that our metric attains state-of-the-art performance in standard summarization evaluation datasets. Additionally, we showcase the effectiveness of our metric in a more-challenging setting, i.e., the evaluation of long-form summarization. To do so, we conducted a human annotation process for this task. Our work paves the way for future research focused on assessing long-form summarization factuality. As future work, we plan to improve our zero-shot alignment system’s performance with a tailored fine-tuning on summarization datasets.
Limitations
While our approach incorporates multiple components, including NLI, coreference resolution and claim extraction, in order to enhance the robustness and comprehensiveness of the factuality evaluation metric, it is essential to acknowledge that the involvement of these diverse systems introduces a layer of complexity and potential sources of error. The inherent variability and imperfections in each of these components may impact the overall performance and necessitate careful consideration in the interpretation of the results.
Our experiments are also constrained by the fact that the models employed are designed for working in English only. In order to extend this approach to other languages, the multilingual variants of the NLI, claim extractor, and coreference resolution models will need to be employed. However, the performance of such multilingual models needs a proper evaluation that we defer to future work.
The claim-level granularity of the outputs of FENICE, and the alignment of claims with portions of the input document it provides, ensures a higher level of interpretability compared to previous metrics. Yet, defining a quantitative evaluation for this aspect remains an ongoing challenge.

heading: Acknowledgements
text: We gratefully acknowledge the support of the PNRR MUR project PE0000013-FAIR.
We gratefully acknowledge the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content), which is funded by the MUR Progetti di Ricerca di Rilevante Interesse Nazionale programme (PRIN 2020). Alessandro Scirè and Karim Ghonim conducted this work during their enrollment in the Italian National Doctorate on Artificial Intelligence at Sapienza University of Rome.

heading: A Prompt for claim extraction.
text: Figure 2 illustrates the formulated prompt designed for extracting claims from a given summary. This prompt was employed to generate the claims utilized in our evaluation metric and to train our distilled claim generation model.

heading: B Claim extraction evaluation with BERTScore.
text: In Table 8, we report the claim extraction evaluation of the LLM-based claim extractor compared to its T5-based distilled version, using BERTScore instead of ROUGE.

heading: C NLI model ablation.
text: We evaluate the performance of FENICE utilizing RoBERTa5 (Liu et al., 2019) as the NLI model for our alignments (FENICERoBERTa), instead of DeBERTa (Table 9). The results reported are directly comparable to the ones presented in Table 4.

heading: D Long-form summarization factuality annotations statistics.
text: In Table 10, we report the number of factual and unfactual summaries in our collection.
5We load the pre-trained model weights from the Hugging Face Hub, available at: ynie/roberta-large-snli_mnli_ fever_anli_R1_R2_R3-nli.

heading: Future Work
text: Our work paves the way for future research focused on assessing long-form summarization factuality. As future work, we plan to improve our zero-shot alignment system’s performance with a tailored fine-tuning on summarization datasets. Limitations
While our approach incorporates multiple components, including NLI, coreference resolution and claim extraction, in order to enhance the robustness and comprehensiveness of the factuality evaluation metric, it is essential to acknowledge that the involvement of these diverse systems introduces a layer of complexity and potential sources of error. The inherent variability and imperfections in each of these components may impact the overall performance and necessitate careful consideration in the interpretation of the results. Our experiments are also constrained by the fact that the models employed are designed for working in English only. In order to extend this approach to other languages, the multilingual variants of the NLI, claim extractor, and coreference resolution models will need to be employed. However, the performance of such multilingual models needs a proper evaluation that we defer to future work. The claim-level granularity of the outputs of FENICE, and the alignment of claims with portions of the input document it provides, ensures a higher level of interpretability compared to previous metrics. Yet, defining a quantitative evaluation for this aspect remains an ongoing challenge