ACUEVAL: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization (2024)

Abstract: The impressive generation capabilities of large language models (LLMs) have made it harder to detect the subtle hallucinations they make in abstractive summarization, where generated summaries consist of a blend of correct and incorrect information w.r.t. a given document. Recently-proposed LLM-based evaluation metrics attempt to capture this, but still face challenges: (1) they are biased towards summaries generated from the same underlying LLM, and (2) they lack interpretability, offering only a single score. In this work, we present ACUEVAL, a metric that leverages the power of LLMs to perform two sub-tasks: decomposing summaries into atomic content units (ACUs), and validating them against the source document. Compared to current strong LLM-based metrics, our two-step evaluation strategy improves correlation with human judgments of faithfulness on three summarization evaluation benchmarks by 3% in balanced accuracy compared to the next-best metric, and also shows reduced preference bias towards LLM-generated summary. Further, we show that errors detected by ACUEVAL can be used to generate actionable feedback for refining the summary, improving the faithfulness scores by more than 10%.1

heading: 1 Introduction
text: Hallucination in abstractive summarization, where the generation contains information that is inconsistent with the source document, remains a crucial problem despite the significant progress of large language models (LLM) (Goyal et al., 2022; Zhang et al., 2024). The problem has become more subtle, as the generations often contain a mixture of correct and hallucinated facts (Pagnoni et al., 2021; Min et al., 2023), making the detection of such errors
∗Work done during an internship at Meta AI. 1Our code is publicly available at https://github.com/
meetdavidwan/acueval.
harder. Recently-proposed evaluation metrics have achieved high correlations with human preferences with the aid of LLMs (Fu et al., 2023; Liu et al., 2023a). Nevertheless, similar to the observation by Tang et al. (2023) and Liu et al. (2023a), we find that such metrics generally have a preference-bias, where the metric favors generations from the same underlying LLM used for scoring. Furthermore, such metrics often output only a single numeric score, making them less interpretable to practitioners in understanding the precise location of the errors and the justification behind the score.
To address these problems, we present a new metric: ACUEVAL, which leverages the strong capability of LLMs to perform two fine-grained and structured sub-tasks instead of asking the model to directly provide a single score. We operate on the level of atomic content units (Liu et al., 2023b, ACUs), facts that can be verified and cannot be broken down further. ACUEVAL first generates these atomic facts from the system summary, and then validates each extracted fact against the source document. In Figure 1, we show that ACUEVAL successfully identifies that the second atomic fact is not consistent with the source document.
Operating on such fine-grained units as an intermediate representation instead of directly on the system summary reduces the preference bias of the metric in assigning high scores for summaries generated by the same underlying model. ACUEVAL involves two separate steps, each drawing on different input sources. The first step, ACU generation, relies solely on the system summary, while the second step, ACU verification, evaluates the consistency of the ACU with respect to the original document without the use of the summary. This separation ensures that the model does not implicitly assign the best score for the outputs generated by the same model. Moreover, the systematic matching between all extracted facts and the source document narrows down the issue of hallu-
10036

heading: Candidate Summary

heading: Generate Atomic Units

heading: Wrong Fact(s)

heading: Generate Feedback 
text: cination precisely to the specific fact, allowing for better hallucination detection ability given the subtle mistakes LLMs make. The strong zero-shot and few-shot ability of LLMs also allow us to design a robust metric that can detect hallucinations across different datasets and system summaries without modifying the prompts for each setting.
First, we demonstrate that ACUEVAL aligns closely with human judgments across three summarization evaluation benchmarks (Fabbri et al., 2021; Tang et al., 2023; Zhang et al., 2024) and two datasets (Hermann et al., 2015; Narayan et al., 2018) which include summaries ranging from traditional approaches as well as those by recent powerful LLMs. ACUEVAL achieves higher correlations than previous metrics, including the recently proposed powerful LLM-based metrics. We show especially large improvements in detecting hallucinations for summaries generated by LLM-based models as opposed to summaries generated by traditional, fine-tuned models. Our detailed analysis in Section 5.2 also reveals that ACUEVAL significantly reduces the preference bias towards the summaries generated by the underlying LLM, due to operating on fine-grained units, unlike metrics that directly evaluate on the generated summary.
A novel downstream application of ACUEVAL’s fine-grained error localization is to create detailed, structured feedback to improve faithfulness in the iterative summarization process (Zhang et al., 2023), where a refinement model addresses the problems listed in the comment to produce an enhanced summary. As shown at the bottom of Fig-
ure 1, all facts judged to be incorrect by ACUEVAL are incorporated into the feedback. By covering the detailed hallucinations detected by ACUEVAL, as demonstrated in Section 5.4, the targeted feedback informed by ACUEVAL enhances the model’s ability to generate more faithful summaries after revision, leading to a 10% and 23% improvement on G-Eval (Liu et al., 2023a) and ACUEVAL scores compared to using GPT-4’s feedback.
Finally, we provide an analysis using ACUEVAL to assess the capacity of various LLM’s to produce faithful summaries. We first confirm ACUEVAL’s effectiveness in identifying patterns consistent with those noted according to human annotations, specifically on the news summarization meta-evaluation benchmark (Zhang et al., 2024). Our findings, specifically that instructions-based models perform better and the reference summary achieves low faithfulness scores, align closely with human judgments. Next, we apply ACUEVAL to assess LLMs in the hallucination benchmark,2 and find that GPT4 exhibits the least hallucination among the tested models, in line with previous findings (Min et al., 2023; Laban et al., 2023).
In summary, our contributions are the following:
1. We introduce ACUEVAL, an interpretable, LLM-based faithfulness evaluation metric for summarization, with a structured, two-step evaluation strategy that first breaks the output into fine-grained ACUs and then verifies their
2https://github.com/vectara/ hallucination-leaderboard
presence in the source document.
2. We show that ACUEVAL achieves a higher correlation to human judgments than current LLM-based metrics, especially for LLMgenerated summaries. With ACUEVAL, we observe trends such as GPT-4 containing the least hallucinations when assessing LLMs’ capability to generate faithful summaries.
3. We show that the hallucinating ACUs detected by ACUEVAL can be in turn transformed into detailed actionable feedback for refining the summary for improved faithfulness.

heading: 2 Related Work

heading: Faithfulness evaluation for summarization.
text: Numerous metrics have been designed to assess the faithfulness of abstractive summarization. These range from entailment-based metrics (Kryscinski et al., 2020; Goyal and Durrett, 2020), to question-generation, question-answering metrics (Durmus et al., 2020; Scialom et al., 2021; Fabbri et al., 2022). More recently, the focus has shifted towards LLM-based metrics (Liu et al., 2023a; Fu et al., 2023; Gao et al., 2023; Luo et al., 2023) that leverage LLMs to assess the faithfulness of a summary. Our method uses an open-source LLM and splits the evaluation into two distinct sub-tasks, enhancing interpretability and mitigating the bias inherent to using the same LLM for both generation and evaluation.
Fine-grained metrics. The adoption of finegrained units in summarization evaluation has improved inter-annotator agreement for human evaluation (Liu et al., 2023b; Krishna et al., 2023) as well as metric performance for automatic evaluation. For instance, DAE (Goyal and Durrett, 2020) outperforms traditional entailment-based metrics by focusing on the entailments of dependency arcs. Similarly, fine-grained units are also effective for relevance, dating back to works including Nenkova and Passonneau (2004); Shapira et al. (2019) that generate ACUs from the reference summary and validates against the system summaries. Other sentence decomposition methods includes semantic role labeling (Xu et al., 2020b; Chan et al., 2023; Glover et al., 2022), OpenIE (Ernst et al., 2022, 2021), and Rhetorical Structure Theory (Xu et al., 2020a; Liu and Chen, 2019; Li et al., 2016). FactScore (Min et al., 2023) also uses a two-step approach for evaluating the factuality of people bi-
ographies. ACUEVAL, while sharing similarities with FactScore, leverages a single, open-sourced LLM for both generating and evaluating atomic facts, allowing for cost-effective and easily replicable future developments. For abstractive summarization, we do not need a separate retriever to source relevant passages, given that the source document is already provided. We also demonstrate that the fine-grained metrics can be useful beyond evaluation, improving the downstream summarization performance when used as feedback for refining the summary.

heading: 3 ACUEVAL
text: Figure 1 illustrates our metric ACUEVAL. Here, we assume that we have a source document X and a generated summary ŷ. ACUEVAL consists of two structured steps: (1) deconstructing the summary into fine-grained ACUs, and (2) predicting the presence of each ACU against the information presented in the source document. The result of these steps is a faithfulness score.
ACU generation. We first generate atomic content units (ACUs), or atomic facts, from the summary. We follow the definition of ACUs by Liu et al. (2023b): Elementary information units, which no longer need to be further split for the purpose of reducing ambiguity in human evaluation. We note that, unlike previous approaches where atomic facts were generated from reference summary y, we apply this method to the generated summary ŷ. This approach yields more fine-grained information of the summary, which has been shown to improve faithfulness evaluation (Goyal and Durrett, 2020; Durmus et al., 2020). Additionally, we opt for a textual representation over complex representations like dependency parses (Goyal and Durrett, 2020) or AMR graphs (Ribeiro et al., 2022), which simplifies the integration of error localization in LLMs. Formally, we break down a summary ŷ into a list of atomic facts Aŷ = {a1, a2, ..., aN}. We generate these facts by asking an LLM to break an utterance up using the prompt shown in Figure 5.
ACU verification. After generating the ACUs, we then verify whether they are consistent with the source document X . This is done by prompting an LLM to predict whether each fact is consistent with the information in the source document with either "Yes" or "No" as the answer (See Figure 6). To refine our accuracy, we normalize the probability
of the two labels and take the probability for "Yes" as the final score of the ACU. We use same LLM for both ACU generation and ACU verification. Formally, the score for each ACU is defined as:
si = p(LLM(X, ai, pt) = Yes)
where LLM(X, a, pt) is LLM’s prediction given the document X , the ACU a, and the prompt pt.
Final Score. The final score is the average across all ACU presence predictions:
ACUEVAL = 1
|A|
|A|∑
i=1
si
Fine-grained Feedback from ACUEVAL. Next, we also demonstrate a novel application of finegrained error localization with ACUEVAL: Generating detailed feedback based on the hallucinations identified by ACUEVAL for improving the summary. Inspired by Saunders et al. (2022), who demonstrated that model-generated critiques can guide humans to detect overlooked flaws, our approach similarly uses detailed critiques to assist the refinement model in identifying and correcting hallucination. The refinement process with ACUEVAL can be seen in the lower section of Figure 1. Unlike the original method where the critique model generates free-form feedback, our strategy involves listing all ACUs deemed inconsistent with the document as inconsistent facts that the refinement model needs to address (see Figure 10 for the prompt template). Since the original critique model itself is quite similar to the LLM-based metrics proposed in prior works,3 it might overlook certain hallucinations because of the model’s coarse-grained scope and inherent preference biases. In contrast, the advantage that ACUEVAL has over critique models when used for feedback mirrors its benefits for evaluation purposes, where ACUEVAL offers a more exhaustive detection of hallucinations with little preference bias.

heading: 4 Experiments

heading: 4.1 Implementation Details
text: We use StableBeluga 2 (Mahan et al., 2023) for both ACU generation and ACU verification, as
3Both the critique model and LLM-based metrics, such as G-Eval, take the document and summary as input and output a text. However, while LLM-based metrics generate a score reflecting the quality of the summary, the critique model produces a textual commentary of the summary’s content.
we find that this model follows the instruction reliably.4 The model uses Llama2 70B (Touvron et al., 2023) as the backbone, and fine-tuned on the ORCA (Mukherjee et al., 2023) dataset. We use greedy decoding to ensure determinism and set the maximum generation length to 256 for ACU generation and 5 for ACU verification. More details can be found in Appendix B.

heading: 4.2 Benchmarks
text: We focus on abstractive summarization benchmarks that measure summary faithfulness by collecting human judgments. All of the benchmarks consist of examples from two news summarization datasets CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018), containing news articles from CNN/Dailymail and BBC, respectively. We include benchmarks consisting of annotations on summaries generated by previous state-of-theart models, as well as those by recent LLM models. SUMMEVAL (Fabbri et al., 2021) consists of annotations from extractive and abstractive systems. AGGREFACT (Tang et al., 2023) consists of 9 faithfulness benchmark datasets. We use the FTSOTA split consisting of state-of-the-art fine-tuned summarization models, as the authors find that previous metrics, including LLM-based metrics, fall short when evaluating summaries from more recent models. LLMSUMMEVAL (Zhang et al., 2024) is our primary evaluation benchmark, consisting of similar human annotations on summaries generated by LLMs under both zero-shot and few-shot settings. More details can be found in Appendix C.

heading: 4.3 Evaluation
text: Given the issue of significant class imbalance in the data, computing correlations directly to human labels may not accurately reflect performance. This problem is particularly crucial in contexts like the LLMSUMMEVAL benchmark, where only 20% of annotations are marked as incorrect. To mitigate the impact of this imbalance, we follow Laban et al. (2022); Tang et al. (2023) and focus on computing balanced accuracy. To ensure a fair evaluation across the diverse scales of metric scores, we additionally split the annotations into validation and test sets based on whether their indices are odd or even, following Tang et al. (2023). This allows us to tune for the threshold for the optimal balanced ac-
4We have also tried Llama2-chat, Zhepyr, and Vicuna 33B, but we find that they do not follow the prompt consistently (i.e. only predicting numerical scores or only answering true/false).
curacy within the validation dataset. Experiments on standard correlations are in Appendix E.

heading: 4.4 Baseline Metrics
text: We include baseline metrics in the respective benchmarks as well as strong faithfulness metrics developed for summarization evaluation. Our primary focus is a comparison with LLM-based metrics, which have been shown to be better at detecting hallucination than traditional metrics. We include strong GPT-based metrics, including GEval (Liu et al., 2023a), ChatGPT-ZS (Luo et al., 2023). However, due to the high cost of running the metric across all benchmarks, we also explore an alternative, BelugaEval, our variant of G-Eval and ChatGPT-ZS based on StableBeluga 2. This opensource model offers a similar approach and performance to G-Eval. Finally, we also include standard faithfulness metrics, including DAE (Goyal and Durrett, 2020), QuestEval (Scialom et al., 2021), and QAFactEval (Fabbri et al., 2022). For more information, we refer the readers to Appendix A.

heading: 5 Results

heading: 5.1 Meta-Evaluation
text: We present the balanced accuracy results on the three benchmarks in Table 1. We first note that BelugaEval is a reliable alternative to G-Eval and ChatGPT-ZS, as it achieves similar balanced accuracy that differs at most by 1 point. For XSum split of AGGREFACT-FTSOTA, BelugaEval improves 3.4 points over ChatGPT-ZS. ACUEVAL consistently achieves the highest balanced accuracy on all three evaluation benchmarks. Notably, in LLMSUMMEVAL, our main benchmark, ACUEVAL surpasses the next-best metric by 3 points in both CNN/DM and XSum datasets, highlighting the high accuracy and robustness of ACUEVAL.
Interestingly, despite showing high correlations with human judgments, the LLM-based evaluation
metrics, including G-Eval, ChatGPT-ZS, and BelugaEval, do not outperform some of the more established baseline metrics in terms of balanced accuracy. Particularly, the LLM-based metrics’ performance is the lowest for the CNN/DM split of AGGREFACT-FTSOTA. This aligns with findings from Tang et al. (2023), which suggest that while these metrics excel in assessing outputs from older systems, they may not be as effective with content generated by more recent models.
Furthermore, the results reveal that different metrics show varied trends when assessing summaries produced by earlier systems compared to those generated by LLMs. For instance, while QuestEval has the lowest balanced accuracy for AGGREFACTFTSOTA XSum benchmark, it achieves the highest accuracy among traditional metrics in the LLMSUMMEVAL XSum benchmark. QAFactEval,
which performs best on the traditional benchmark, achieves the lowest balanced accuracy for LLMbased summaries on the LLMSummEval benchmark. This underscores the importance of reevaluating various metrics, especially in the context of LLM-generated content which differs from traditional benchmarks.

heading: 5.2 Preference over LLM-based Outputs
text: A key concern with metrics based on LLMs is their potential bias towards outputs generated by similar LLMs. This issue arises because these metrics often use the same or a related generation model for evaluation, leading to higher scores for outputs from similar models (Deutsch et al., 2022).
Liu et al. (2023a) observed a tendency for GEval to favor outputs from GPT-3.5 models over human-written summaries. To investigate this, we conducted similar experiments comparing the metric scores for human-written summaries with those generated by GPT-3.5. We split the GPT-3.5 summaries from LLMSUMMEVAL into three categories based on how they were rated against human summaries: higher, equal, or lower, and compare the average metric scores for human summaries and the GPT-3.5 summaries under the three cases.
We perform the analysis using BelugaEval and ACUEVAL, both of which use the same underlying LLM and present the result in Figure 2. We see a clear bias in BelugaEval: It often rates GPT-3.5 summaries higher than human-written ones, even when human annotators preferred the latter. In the figure, we see that the average BelugaEval scores of GPT-3.5 summary are always higher than that for the human summaries. However, our metric, ACUEVAL, demonstrated more balanced behavior, assigning higher scores to superior human summaries and vice versa. Nevertheless, it still shows a slight preference for GPT-3.5 summaries where the summaries were deemed equally good. We further analyze the preference bias and examine the impact
of atomic units on preference bias in Appendix F.5.

heading: 5.3 Ablations
text: ACU generation performance. We wish to evaluate the effectiveness of the ACU generation capability. Since there are no gold ACUs available for generated summaries for comparison, we use the ROSE dataset (Liu et al., 2023b) which provides expert-written ACUs for examples in the CNN/DM, XSum, and Samsum dataset (Gliwa et al., 2019). We compare the ACUs generated by ACUEVAL with expert-written ones. Following the authors, we calculate the recall using Rouge1-F1 (Lin, 2004) score for each generated ACU by greedily finding its best match among the reference ACUs and then taking the average across all generated ACUs. We additionally report precision and F1 scores in Appendix F.2.
We experiment with various prompts, including those from FactScore (Min et al., 2023), and create few-shot prompts using the gold ACUs from ROSE’s CNN/DM validation set. Details on our prompt design can be found in the Appendix H.1. We include AutoACU2-gen (Liu et al., 2023b), a T0-3B (Sanh et al., 2022) model fine-tuned on all the reference ACUs as a potential upper bound for this task. Table 2 show the impact of different prompt strategies on ACU quality.
We observe that providing more context-specific examples (from 3-shot to 5-shot) leads to marginal improvements for the CNN/DM and Samsum datasets. The FactScore prompt, which focuses on sentence-level generation, shows better results for XSum, which contains single-sentence summaries. However, since expert-written ACUs are typically based on multi-sentence summaries and include cross-sentence references, the FactScore approach falls short for CNN/DM and Samsum, often resorting to generic subject assignments.
In conclusion, though there still exists a large gap between the few-shot approach and the full fine-tuning method, our analysis indicates that the
5-shot variant is preferable. It not only achieves the highest Rouge1 score for the CNN/DM dataset among the different prompts but also demonstrates robust performance across different types of summaries, including those in the Samsum dataset. More importantly, we demonstrate the utility of our LLM-generated ACUs in Appendix F.1. Compared to ACUs generated by fine-tuned models, our ACUs result in stronger metric correlations when used with LLM-based verification methods.
ACU verification performance. Next, we evaluate the ACU verification capability. We again utilize the ROSE dataset containing expert labels for the presence of reference ACUs in candidate summaries and evaluated using the accuracy of assigning the correct label. It is important to note that this task is slightly different from the standard usage of the ACUEVAL: here, we generate ACUs from a reference summary (y) and compare them to a candidate summary (ŷ). In contrast, for evaluating faithfulness, ACUs are derived from the candidate summary (ŷ) and then matched against the original document (X).
Similar to the ablations on ACU generation capability, we explore different prompts, including FactScore-style prompts, as well as zero-shot and few-shot approaches. We also investigate the impact of incorporating the document as additional input in a few-shot prompt setup. To compare to other metrics for this subtask, we include AutoACU2match, a model using DeBERTa-XLarge, trained on all the ACU-summary pairs in the ROSE dataset, alongside the original pre-trained model.
We present our results in Table 3. Notably, the zero-shot prompt technique emerges as the most accurate, surpassing the results of the pre-trained DeBERTa-XLarge model by 12 points across the
three datasets, and is slightly better than the fewshot variant. The FactScore prompt here does not show adaptability to our model and task, as it does not achieve high accuracy. Adding the document to the prompt also results in a noticeable decrease in performance. This observation is consistent with the findings of Liu et al. (2023c), who noted that inputs with redundant information could negatively impact predictive performance.
ACU verification with different models. Next, we investigate how benchmark performance changes when using different LLMs for the verification process. We examine Mixtral 7Bx8 (Jiang et al., 2024), Qwen 70B (Bai et al., 2023), and StableBeluga 13B. As shown in Table 4, performance generally improves with larger model sizes. Specifically, while StableBeluga 13B achieves random performance, Qwen 70B demonstrates an improvement of over 20 points on LLMSUMMEVAL. Additionally, it is not necessary to use the same model for both ACU generation and verification, as using Qwen 70B for verification achieves comparable balanced accuracy to using StableBeluga 70B.
Evaluation pipeline. Next, we evaluate whether the pipeline can be made efficient by combining the verification of all the facts or even combining the ACU generation and verification into one single step. We refer to the variant where the model provides faithfulness judgments for all ACUs simultaneously during the verification stage as VERIFY ALL AT ONCE, and the variant where the model generates and verifies ACUs in a single run as ONE PASS. The corresponding prompts are detailed in Figure 8 and Figure 9, respectively. As shown in Table 4, there is a clear trade-off between performance and efficiency, particularly on LLMSUM-
MEVAL. Combining the steps of ACUEVAL results in poorer performance than BelugaEval on LLMSUMMEVAL, indicating the importance of having current models perform one task at a time.

heading: 5.4 Improving Generation via Feedback
text: In this section, we assess the impact of the feedback informed by ACUEVAL on improving summary faithfulness in the summary refinement pipeline discussed in Section 3. ACUEVAL feedback consists of a comprehensive list of the atomic facts that are judged as incorrect according to ACUEVAL. We compare our feedback generation method against the self-critique method, where GPT-4 is tasked to provide a critique of the summary directly. This is achieved by asking the model to continue producing content after it has assigned the faithfulness score with the G-Eval prompt. Additionally, we include feedback from QAFactEval by retrieving the incorrect question-answer pairs.
For a fair comparison, we use GPT-4 as the refinement model with the same refinement prompt but change the feedback depending on the method. Examples of the refinement prompt with ACUEVAL feedback and G-Eval feedback can be seen in Figure 10 and Figure 11, respectively. The refinement model takes as input a prompt consisting of the document, summary, and feedback. Note that the setup is similar to the iterative summarization process (Zhang et al., 2023), but we also include the original document as additional input. This is important for comparing the two methods fairly because ACUEVAL only highlights where the summary deviates from the source document but does not provide the correct content directly.
We randomly selected 50 summaries each from CNN/DM and XSum within the LLMSUMMEVAL dataset, all containing errors identified by ACUEVAL. We measure the faithfulness of the refined summary using ACUEVAL and G-Eval, which allows us to verify that the gain does not stem from optimizing on our proposed ACUEVAL. The result in Table 5 shows that all feedback types improve
summary faithfulness, but ACUEVAL feedback leads to the most substantial improvement. First, QAFactEval feedback is better than G-Eval feedback on XSum and overall, indicating that finegrained feedback is important for reducing the hallucinations in the summaries. Nevertheless, summaries refined with ACUEVAL feedback nearly reached perfect faithfulness score, achieving 4.88 out of 5 for G-Eval and 97.1 for ACUEVAL. This highlights the strength of ACUEVAL at providing nuanced feedback, significantly improving the faithfulness of LLM generations.

heading: 6 Benchmarking LLM for Faithfulness
text: Lastly, ACUEVAL can also serve as a powerful analytical tool for assessing the capacity of current Large Language Models (LLMs) to generate faithful summaries. Demonstrating strong correlations with human evaluations, particularly in recent models, ACUEVAL provides a practical and reliable alternative to human assessments of hallucinations.

heading: 6.1 LLMSUMMEVAL
text: We first examine the ACUEVAL scores of various models using LLMSUMMEVAL in Table 7, which allows us to compare against the provided human judgments. The high congruence of these scores with human ratings indicates our metric’s alignment with human judgment. Our findings echo the insights Zhang et al. (2024) in several ways: we find that Instruction-tuned models perform better, and reference summaries are less faithful. More detailed discussions can be found in Section D. In summary, ACUEVAL’s scoring closely aligns with human judgments, demonstrating its efficacy as a benchmarking tool for discovering informative trends among the models.

heading: 6.2 Hallucination Benchmark
text: To compare the faithfulness power of more recent popular LLMs, we also calculate the ACUEVAL scores on the hallucination benchmark. It contains summaries of 831 documents using 11 strong
LLMs. The result is presented in Table 6. Models are ranked based on their performance according to ACUEVAL. The benchmark originally uses the Hallucination Evaluation Model5 (HEM) as the benchmarking metric, which is trained on fact verification with DeBERTaV3 (He et al., 2023). Our ACUEVAL ranking reveals similar trends as observed with HEM: Models that maintain an optimal answer rate and adhere to average summary lengths tend to score higher in faithfulness.
In line with previous works (Min et al., 2023; Laban et al., 2023), GPT-4 and GPT-3.5 achieve the highest faithfulness scores among the models. Apart from these two models, there is no single model family that consistently shows improvement with scaling model size on HEM-based ranking. Nevertheless, our ACUEVAL-based ranking reveals a notable phenomenon - faithfulness scales with model size within the same model family. Our ranking underscores a clear correlation between model size and faithfulness. For instance, Llama 2 shows a definitive hierarchy in faithfulness: 70B outperforms 13B, which in turn surpasses 7B. Similarly, Mistral 7B aligns closely with Llama 2 7B in terms of ranking. This contrasts with the HEM ranking, where a distinct hierarchy is evident (GPT > Llama 2 > Cohere > Claude 2 > Mistral > Palm).

heading: 7 Conclusion
text: In this paper, we introduce ACUEVAL an interpretable, fine-grained metric for evaluating faithfulness for abstractive summarization. Our findings demonstrate that ACUEVAL achieves the highest balanced accuracies across diverse benchmarks and datasets, outperforming other recent, strong LLMbased metrics. Notably, ACUEVAL shows very low bias towards LLM-generated outputs, making
5https://huggingface.co/vectara/hallucination_ evaluation_model
it a fair tool for evaluation of summaries in the era of LLMs. Next, we also explore how ACUs that are considered not faithful to the input document can be incorporated as detailed feedback, which in turn enhances the correction model at refining the summary with little hallucination. Finally, we compare the average ACUEVAL scores of various LLMs, assessing their faithfulness in abstractive summarization. These comparisons align closely with human judgments and reveal that larger models tend to be more faithful. We hope that ACUEVAL can serve as a foundational guide for evaluating generated summaries. Looking forward, we propose expanding this framework to encompass additional facets of summarization evaluation and adapting it for more complex tasks like multi-document and long-form summarization.

heading: Limitations
text: One key challenge of our current approach is the slow computation stemming from the need to break down the evaluation into two sub-tasks. This issue becomes particularly evident as the length of the summary grows, resulting in an increased number of small elements that must be individually checked. A potential solution is to have the model verify a variable number of ACUs in a single step, rather than one at a time. Another possibility is to merge two separate steps: having the model both create and then verify these elements in one go. However, this may be not reliable, as current LLMs cannot accurately follow multiple steps at once. Another limitation is the need for a model that can accurately follow instructions. We tested various LLMs, but many struggled with either generating or verifying the ACUs accurately. Mistakes in the generation phase can lead to further errors down the line, magnifying the problem, which is true for all model-based evaluation methods.

heading: A Details on Baseline Metrics
text: G-Eval (Liu et al., 2023a) is a GPT4-based metric, which significantly outperforms traditional metrics. However, due to the high cost of running the metric across all benchmarks, we explore an alternative, BelugaEval. This open-source model offers a similar approach and performance. See Appendix G for more details on the comparison.
ChatGPT-ZS (Luo et al., 2023) is another metric that uses ChatGPT to evaluate summaries. We include the results included by AGGREFACT.
BelugaEval is our variant of G-Eval and ChatGPT-ZS based on StableBeluga 2. We use a similar prompt as G-Eval, which can be found in Figure 7. Following Liu et al. (2023a), we integrate a chain-of-thought prompt and utilize the score normalization technique, where the final score is calculated as the weighted summation of the 1-5 scale, each weighted by its respective normalized probability. We refer the readers to the original paper for more details.
DAE (Goyal and Durrett, 2020) is an entailment metric that evaluates the faithfulness of the summary’s dependency arcs.
QuestEval (Scialom et al., 2021) is a questiongeneration question-answering (QGQA) metric. It computes answer overlap scores by generating questions from a source document and then assessing how well these questions are answered by the summary, and vice-versa.
QAFactEval (Fabbri et al., 2022) is a highly optimized QGQA-based metric after extensive analysis of the individual components.
B Implementation Details
We use 8 A100s GPUs to run all experiments. Running the ACU generation for each benchmark takes around 8 hours and running the ACU verification takes around 10 hours. Running GPT-4 (gpt-40613) for refinement takes around 10 minutes for the 50 examples. For all of our experiments, we use the transformers package (Wolf et al., 2020). All baseline metrics are used with the corresponding official implementations. For calculating balanced accuracy and correlations, we use the official scripts from AggreFact (Tang et al., 2023) and ROSE dataset (Liu et al., 2023b), respectively. Since we use greedy decoding for all experiments for deterministic behavior, we only perform single runs for all experiments.

heading: C Benchmark Details
text: SUMMEVAL (Fabbri et al., 2021) consists of expert annotations of 100 samples from 17 different extractive and abstractive systems, all using the CNN/DM dataset (Hermann et al., 2015). To have a fair comparison to previous metrics, we use the first 16 systems that were part of the initial release. We use the consistency labels for assessing faithfulness. The labels are on a 1-5 Likert scale, and we convert the scores into binary labels following Laban et al. (2022): If the majority of the expert annotators award a summary a score of 5, the summary is categorized with 1.
AGGREFACT (Tang et al., 2023) consists of 9 faithfulness benchmark datasets on both CNN/DM and XSum (Narayan et al., 2018). This benchmark splits the summaries systems into three categories: FTSOTA, EXF, and OLD, representing state-of-the-art fine-tuned summarization models, early transformer models, and older models, respectively. All annotations are transformed to a binary label. We refer the readers to the original paper for more details. The authors find that previous metrics, including LLM-based metrics, tend to show high accuracy with older summaries but fall short when evaluating summaries from more recent models. We thus focus on the FTSOTA split, containing outputs generated by BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and PEGASUS (Zhang et al., 2020). The annotations are split according to the two datasets.
LLMSUMMEVAL (Zhang et al., 2024) is
our primary evaluation benchmark. It collects similar human annotations on summaries generated by LLMs under both zero-shot and few-shot settings. It includes 100 examples of 19 settings for both CNN/DM and XSum.6 Similar to AGGREFACT, we split the benchmark based on the two datasets.
All benchmarks use CNN/DM and XSum, which are under the MIT license. For the benchmarks, SummEval is under the MIT. We follow the authors’ instructions for download and usage.

heading: D Benchmarking on LLMSUMMEVAL
text: We first examine the ACUEVAL scores of various models using LLMSUMMEVAL in Table 7, which allows us to compare against the provided human judgments. The high congruence of these scores with human ratings indicates our metric’s align-
6For XSum, the authors note that the 350M GPT3 model provides only empty outputs, and thus the XSum annotations contain 1800 examples in total.
ment with human judgment. Our findings echo the insights Zhang et al. (2024) in several ways: Instruction-tuned models perform better. Instruction-tuned GPT-3 models, especially in zero-shot scenarios, surpass their non-instructiontuned counterparts and generally achieve the highest faithfulness scores across datasets. Similar observations can be made under the few-shot setting for XSum. This trend also manifests under ACUEVAL scores, which show higher scores for instruction-tuned models. The only exception is the few-shot 350M model on CNN/DM, where human scores also consider the non-instruction-tuned models to be better. Reference summaries are less faithful. Zhang et al. (2024) note that the reference summaries are poor for the two datasets. This can be directly verified, as the human scores for the reference summary are generally among the lowest ones, especially for XSum. ACUEVAL scores mirror this trend, placing reference summaries among the lowest.
In summary, ACUEVAL’s scoring closely aligns with human judgments, demonstrating its efficacy as a benchmarking tool for discovering informative trends among the models.

heading: E Results on Meta-Evaluation
text: Table 8 shows results with traditional metaevaluation metrics, i.e. Kendall correlations on SUMMEVAL and LLMSUMMEVAL. The correlation results mirror the results we previously ob-
SUMMEVAL LLMSUMMEVAL CNN/DM XSum
ACUEVAL w. AutoACU’s ACU 49.3 ± 1.1 49.2 ± 1.0 50.7 ± 0.8 ACUEVAL (our) 86.2 ± 2.1 89.5 ± 1.6 78.4 ± 1.5 SummaC 80.0 ± 1.4 84.5 ± 1.6 70.7 ± 1.5 w. AutoACU’s ACU 76.5 ± 1.5 84.6 ± 2.1 75.9 ± 1.7 w. ACUEval’s ACU 79.3 ± 1.8 87.1 ± 1.8 74.5 ± 1.5 BelugaEval 81.1 ± 1.6 77.0 ± 2.0 62.8 ± 1.7 w. AutoACU’s ACU 80.5 ± 2.1 82.3 ± 1.7 67.4 ± 1.7 w. ACUEval’s ACU 86.6 ± 1.9 88.9 ± 1.3 74.4 ± 1.5
Table 9: Results of AutoACU’s and our ACU decomposition with different ACU validation methods.
served in balanced accuracy shown in Table 1. Notably, BelugaEval, representing LLM-based approaches that generate direct scores, shows a weaker correlation for more recent outputs from LLMSUMMEVAL. These correlations generally fall below those of baseline metrics. However, ACUEVAL achieves the highest system-level and summary-level correlations on both LLMSUMMEVAL benchmarks, especially on the XSum dataset, corresponding to the larger presence of hallucinations in the XSum dataset. Interestingly, ACUEVAL does not show any improvement over BelugaEval on the SUMMEVAL dataset. We emphasize the importance of referring back to the balanced accuracy results in Section 4.3, especially considering the substantial class imbalance present in these datasets.

heading: F Additional Experiments

heading: F.1 Benefit of LLM-based Decomposition
text: Within the ACU-style framework, we also experiment with the use of ACUs generated by AutoACU, a model fine-tuned on the ROSE dataset, representing a more traditional approach to decomposition without the use of LLMs. We compare the performance of our ACUs and AutoACU’s ACUs with our validation model, SummaC (Laban et al., 2022), and BelugaEval. For SummaC and BelugaEval, we replace the summary with the list of ACUs. The results are presented in Table 9.
The ACUs generated by AutoACU decrease the balanced accuracy for ACUEval and also on SummEval when combined with BelugaEval. With SummaC, both methods decrease performance on SummEval, with AutoACU dropping by 3.5 points while our ACUs only drop by 0.7 points. This validates the usefulness of ACUs generated by our approach for LLM-based validation methods.
We observe that adding ACUEval’s ACUs improves performance on SummaC and BelugaE-
val on the LLMSUMMEVAL. Particularly for LLM-based validation techniques, such as ACUEval and BelugaEval, our ACUs result in significantly larger improvements compared to using AutoACU’s ACUs. Specifically, with BelugaEval, our ACUs improve (1) over using the summary on LLMSUMMEVAL by 11.9 and 11.6 points and (2) over using AutoACU’s ACUs by 6.6 and 7.7 points on CNNDm and XSum, respectively. This indicates the strength of combining the two LLMbased methods for further gain. based methods for further gain.
In summary, we demonstrate the superiority of ACUs generated by LLMs over traditional decomposition methods, particularly their additive benefit when combined with an LLM-based validator.

heading: F.2 Additional ACU Generation Results
text: We extend the ACU generation ablations in Table 2 beyond recall by adding precision and F1. The full table is in Table 10. We generally observe the same trend for precision and F1 for the different methods, and for precision and F1, the Rouge scores are similar to the recall values and quite high (> 65).

heading: F.3 Additional ACU Verification Results
text: Alternatively, we extract ACUs from the 100 documents by randomly sampling 50 documents each for the CNN and XSum datasets and then validate the generated ACUs with the documents. In this case, all the verification modules should always output “True,” as the fact is generated from the same document. We find that 98.73% of the ACUs are judged as correct, indicating the high accuracy of the validation module.

heading: F.4 ACUs Statistics
text: We present the number of words, sentences, ACUs, and ACUs per sentence in Table 11. There is a difference between the two datasets. For example, CNNDM usually consists of 3 sentences, while XSum contains a single sentence. By having more sentences/words, the number of ACUs also increases. When we average the number of ACUs by the number of sentences, we observe that XSum contains on average more ACUs, which echoes the more extreme compression nature of XSum.

heading: F.5 Additional Preference Bias Results
text: Bias by datasets. We conduct additional analysis by splitting the scores by the two datasets in Figure 3. Overall, ACUEVAL is good at reducing the
Human Summary GPT-3.5 Summary
Human Summary GPT-3.5 Summary
Human Summary GPT-3.5 Summary
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00 Average ACUEval's scores
Human Summary GPT-3.5 Summary
Human Summary GPT-3.5 Summary
Human Summary GPT-3.5 Summary
0.5
0.6
0.7
0.8
0.9
1.0 Average ACUEval's scores
Human Summary GPT-3.5 Summary
Human Summary GPT-3.5 Summary
Human Summary GPT-3.5 Summary
Human Summary is Better Equally Good LLM Summary is Better
0.70
0.75
0.80
0.85
0.90
0.95
1.00 Average BelugaEval s scores with ACU
Figure 4: Preference bias for BelugaEval with ACU.
bias, i.e. higher scores for human summaries when annotators think so too instead of always assigning higher scores for LLM summaries. Considering together with Table 3, we can observe that with
more ACUs (i.e. CNNDM), we have a larger score difference when the human summary is better and a smaller score gap when the two summaries are equally good. This suggests that ACUEval is indeed helping with reducing the bias and aligning closer to human judgments with more ACUs.
Bias with ACUs only. We investigate whether the preference bias is affected by just breaking down into atomic units with BelugaEval. To do so, instead of providing the summary, we provide the list of ACUs and ask the model to provide the same Likert score. Note that this does not use the ACU validation component. As shown in Figure 4, we can observe a similar reduction of bias, where the human summaries are rated higher when humans think so too. Interestingly, the same slight preference for GPT-3.5 summaries when the summaries are equally good is also present here. Taken
together, breaking the text into atomic units is helpful in reducing the preference bias.

heading: G Comparison between LLM-based Evaluation Metrics
text: The prompt used for BelugaEval can be found in Figure 7. This is very similar to the prompt used for G-Eval except that we change the chain-of-thought prompt to the instruction Fabbri et al. (2021) uses for human annotation. We notice that this more targeted prompt improves the performance. Since we use StableBeluga 2 as the LLM, we use greedy decoding for reliable predictions. We also use the original score normalization technique outlined in Liu et al. (2023a).

heading: H Prompts

heading: H.1 ACUEVAL Prompts
text: ACU generation. We show our 5-shot prompt for generating the ACUs in Figure 5. Examples are taken from the ROSE dataset. For 1-shot and 3- shot, we select the first one and first three examples, respectively. For FactScore-style prompt, we use the prompt in Min et al. (2023), which contains multiple human-written in-context examples.
ACU verification. The prompt is shown in Figure 6. For FactScore-style prompt, we use the provided prompt: "{{ACU}} True or False?"
ACUEVAL VERIFY ALL AT ONCE variant. The prompt is shown in Figure 8. The prompt is similar to the verification prompt except that we include all atomic facts.
ACUEVAL ONE PASS variant. The prompt is shown in Figure 9, combining atomic fact generation and verification.

heading: H.2 Refinement Prompts
text: For refinement, we use Figure 10 for ACUEVALstyle and Figure 11 for G-Eval-style prompt. We note that the two prompts have the same refinement prompt and differs only in the comment section: ACUEVAL-style comment contains a list of incorrect atomic facts, while the comment with G-Evalstyle is a free-form text generated by the scoring model.

heading: I Refinement Examples
text: We show examples of refinement in Figure 12.

heading: Future Work
text: ACUEVAL, while sharing similarities with FactScore, leverages a single, open-sourced LLM for both generating and evaluating atomic facts, allowing for cost-effective and easily replicable future developments. For abstractive summarization, we do not need a separate retriever to source relevant passages, given that the source document is already provided. We also demonstrate that the fine-grained metrics can be useful beyond evaluation, improving the downstream summarization performance when used as feedback for refining the summary