FineSurE: Fine-grained Summarization Evaluation using LLMs (2024)

Abstract: Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and timeconsuming nature of human evaluation. Traditional methods like ROUGE do not correlate well with human judgment, while recently proposed LLM-based metrics provide only summary-level assessment using Likertscale scores. This limits deeper model analysis, e.g., we can only assign one hallucination score at the summary level, while at the sentence level, we can count sentences containing hallucinations. To remedy those limitations, we propose FineSurE, a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs). It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment. We compare various open-source and proprietary LLMs as backbones for FineSurE. In addition, we conduct extensive benchmarking of FineSurE against SOTA methods including NLI-, QA-, and LLM-based methods, showing improved performance especially on the completeness and conciseness dimensions. The code is available at https://github.com/ DISL-Lab/FineSurE-ACL24.

heading: 1 Introduction
text: Text summarization stands out as an important task in natural language processing, aiming to generate a condensed summary of a provided text while retaining its essential information (Gupta and Gupta, 2019; Song et al., 2023). Despite the enhanced quality of summaries produced by LLMs, the development of automated methods for evaluation remains a challenge (Kryściński et al., 2020; Maynez et al., 2020). Conventional reference-based metrics, such as ROUGE (Lin, 2004), have exhibited a
∗Corresponding Author. †This work conducted independently and is not related to
the author(s) position at Amazon.
weak correlation with actual human judgments (Liu et al., 2023). Consequently, human evaluation remains an essential step for accurately assessing the quality of generated summaries, even considering its inherent costs and time-consuming nature.
Recently, the need for better automatic evaluators has become an important research topic, aiming to streamline evaluation processes and ease manual efforts in model development (Gao et al., 2023). This effort provides valuable insights into whether generated summaries align with predefined quality standards, including aspects like faithfulness. Various approaches have been explored, including approaches based on neural language inference (NLI) (Laban et al., 2022) and questionanswering (QA) (Fabbri et al., 2022; Zhong et al., 2022). In addition, LLMs have recently proven their potential to be an automated tool for humanlike evaluation (Liu et al., 2023; Wang et al., 2023). The latest LLM-based method, G-Eval (Liu et al., 2023), demonstrated a Spearman correlation coefficient of over 0.5 with Likert-scale human judgments on the news domain using GPT-4.
Despite these advancements, we contend that the current LLM-based automated methods still fall short in achieving precise evaluation, primarily attributed to the coarse-grained evaluation pipeline and the ambiguity in evaluation dimensions. Specifically for coarse-grained evaluation, the evaluation dimensions–namely faithfulness, coherence, and relevance 1–are frequently assessed at the summarylevel, resulting in Likert-scale scores for each summary (Gao and Wan, 2022; Shen and Wan, 2023; Liu et al., 2023; Wang et al., 2023). This Likertscale scoring method lacks fine-grained information on errors in generated summaries. For instance, it does not provide a breakdown of the number of summary sentences with quality issues or specify
1We omit fluency assessment as modern AI models typically generate highly fluent outputs (Liu et al., 2023).
906
the types of mistakes present in each sentence. Furthermore, regarding ambiguity, the evaluation of coherence and relevance is hindered by the lack of clarity in their definition of "the collective quality of all sentences" and "the selection of important content from the source" (Fabbri et al., 2021; Shen et al., 2023). Given that human can encounter challenges in evaluating summaries, it is inappropriate to expect a neural model to provide accurate and objective assessments. Hence, there is a need to develop a more precise evaluation framework that results in a more detailed assessment with clearly defined evaluation dimensions.
In this paper, we present FineSurE (Finegrained Summarization Evaluation) using LLMs, a novel automated approach designed to evaluate the summarization quality at a fine-grained level based on summary sentences or keyfacts2, as depicted in Figure 1. We aim to evaluate summaries using this framework along three vital criteria: the faithfulness of minimizing factuality errors, the completeness of encompassing the majority of keyfacts, and the conciseness of avoiding unnecessary details. Thus, our framework entails executing two finely grained procedures utilizing LLMs: (1) fact checking involves identifying specific factuality errors present in each summary sentence and (2) keyfact alignment focuses on aligning each keyfact with all summary sentences from which they are inferred. We leverage the outcomes from both procedures to calculate precise percentage scores, offering a more detailed assessment than Likertscale scoring. This fine-grained approach enables us to analyze the quality issues of generated texts at both the sentence and keyfact-level.
2A keyfact refers to a concise sentence conveying a single key information, comprising at most 2-3 entities, also referred to as a semantic content unit (Bhandari et al., 2020). The keyfact list can be generated automatically or by humans.
On top of keyfact alignment, two dimensions (completeness and conciseness) can serve as better replacements for coherence and relevance (Fabbri et al., 2021), as they evaluate two key aspects of a good summary, assessing the comprehensive inclusion and density of key information while excluding irrelevant content.
To summarize, our main contributions are as follows: (1) we argue that LLM-based summarization suffers from hallucination, information emission and verbosity hence requiring revisiting the evaluation dimensions, (2) we suggest three metrics targeting LLM output characteristics and tackling the aforementioned problems including faithfulness, completeness and conciseness, (3) we propose a novel automated evaluation framework - FineSurE, based on keyfact lists and using LLMs to generate the keyfacts, align them to the summary sentences and categorize the errors automatically, (4) we compare various open-source and proprietary LLMs to power FineSurE and analyze their correlation to human judgment at the summary and system levels, (5) we provide comprehensive results comparing our method with similarity-based, NLI-based, QAbased, and LLM-based automated methods and show improved human correlation for FineSurE over state-of-the-art methods.

heading: 2 Related Work
text: Efforts to assess the quality of texts generated by language models have led to numerous initiatives in designing effective automated evaluators across various research directions (Lin, 2004; Fabbri et al., 2022; Zhong et al., 2022; Wang et al., 2023).
Similarity-based Evaluator. The evaluation of generated text can be measured by the n-gram overlap with the reference text, employing met-
rics such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Banerjee and Lavie, 2005). In contrast to relying on exact matches, several evaluators have leveraged token similarity through contextual embeddings like BERTScore (Zhang et al., 2019), MoverScore (Zhao et al., 2019), and BARTScore (Yuan et al., 2021). However, these evaluators lack human correlation and a multi-dimensional assessment of text quality akin to real human evaluation, as they typically produce a single-dimensional score based on text similarity.
To assess text quality, primarily focusing on checking factual consistency, task-specific evaluators utilizing NLI and QA have been explored.
NLI-based Evaluator. This task involves factchecking and verification by retrieving relevant evidence from the input text to support the claim made in the generated text (Glover et al., 2022; Honovich et al., 2022; Utama et al., 2022). DAE (Goyal and Durrett, 2020) introduced the dependency arc entailment formulation, offering a more fine-grained approach to faithfulness evaluation. SummaC (Laban et al., 2022) presented a lightweight model that facilitates NLI by segmenting input text into sentence units and aggregating scores between pairs of sentences. Despite their enhanced performance, they only focus on assessing faithfulness.
QA-based Evaluator. This involves generating plausible questions from the reference text and then answering these questions considering the generated text (Scialom et al., 2021; Chen et al., 2021). QAGS (Wang et al., 2020) and QAFactEval (Fabbri et al., 2022) enhanced the accuracy of faithfulness evaluation, surpassing other similarity- and NLIbased evaluators in text summarization. UniEval (Zhong et al., 2022) proposed a unified evaluator capable of assessing multi-dimensional evaluation of text generation through the QA task. In text summarization, it evaluates four aspects: faithfulness, coherence, relevance, and fluency. Generally, these methods require training a neural model to generate questions and their corresponding answers.
LLM-based Evaluator. With the emergence of LLMs, there is a move toward utilizing them as reference-free automated evaluators in diverse scenarios (Shi et al., 2023; Lin and Chen, 2023; Chen et al., 2023; Fu et al., 2023). Recently, a few efforts have been made to evaluate faithfulness using edited text (Laban et al., 2023), atomic facts (Min et al., 2023), and external knowledge base (Feng et al., 2023), as well as to assess multi-dimensional
aspects (Liu et al., 2023; Shen et al., 2023; Wang et al., 2023). Although LLMs have shown promise as evaluators, they currently lack a fine-grained assessment and primarily focus on addressing faithfulness, without considering other important dimensions for high-quality summaries.
Unlike prior studies, we define crucial aspects for a detailed evaluation using LLMs and introduce a new fine-grained evaluation framework called FineSurE. This framework addresses numerous open questions regarding the capabilities of LLMs, including sentence-level fact-checking, classification of error types, and keyfact-level alignment.

heading: 3 FineSurE Framework

heading: 3.1 Evaluation Dimensions
text: LLMs enhance the quality of summarization, but they rather suffer from hallucination, information emission and verbosity (Ji et al., 2023; Saito et al., 2023), requiring revisiting evaluation dimensions. Therefore, we advocate for a thorough assessment of the two evaluation criteria, "completeness" and "conciseness," in addition to "faithfulness." These two dimensions can effectively assess both information emission and verbosity while also complementing each other in evaluating information inclusion and summary succinctness. • Faithfulness: The summarizer does not manip-
ulate the information in the input text (i.e., intrinsic) and add any information not directly inferable from the input text (i.e., extrinsic).
• Completeness: The summarizer ensures the inclusion of all keyfacts from the input text in the output summary.
• Conciseness: The summarizer refrains from incorporating information outside the keyfacts in the output, maintaining a succinct and focused summary.
Note that, adhering to the precise definition of faithfulness in the recent work (Pagnoni et al., 2021), we categorize error types into a total of seven categories, with "out of context" as an extrinsic error, and "predicate," "entity," "circumstance," "coreference," "discourse link," and "grammatical" as intrinsic errors. See examples in Appendix A.

heading: 3.2 Evaluation Pipeline
text: We discuss the evaluation pipeline implementing the dimensions discussed previously. We employ LLMs as a tool to conduct fact checking and keyfact alignment tasks. Specifically, we design two
prompts tailored for the two tasks, as shown in
Figures 3-4 of Appendix B. All prompts are customized to generate outputs in JSON format, enhancing the success ratio of following our instructions and facilitating parsing. The detailed analysis of the success ratio is provided in Section 4.2.
Task 1. Fact Checking. Figure 3 illustrates our prompt and its expected JSON output for fact checking. We convert the problem of fact checking into a categorization problem involving nine categories. These include the seven factuality errors, along with an additional category "other error" for errors outside the seven errors, and an additional category "no error" for cases where no error was detected. Therefore, given a pair of input text and model summary, the LLM is expected to output the error type classified into one of the nine categories for each sentence along with a concise reason.
Task 2. Keyfact Alignment. Figure 4 shows our prompt and its expected JSON output for keyfact alignment. We address the alignment problem through keyfact matching, a process that involves two sequential tasks: verifying if each keyfact is inferred from the summary and, if affirmative, specifying the line numbers for all the corresponding summary sentences. Thus, given a pair of keyfact list3 and model summary, the output should be the binary label and the list of line numbers of all summary sentences matched for each keyfact.
Parsing and Scoring. The evaluation scores are computed using the results from the two tasks. Given a document D, let S = {s1, . . . sN} be the generated summary with N sentences. By the fact checking task, we identify a subset of Sfact ⊆ S, which consists solely of summary sentences marked "no error". Then, the percentage score of faithfulness on S is determined by:
Faithfulness(D,S) = |Sfact|/|S|. (1)
Let K = {k1, . . . , kM} be the list of keyfacts with a size of M . Through the keyfact alignment, we construct a bipartite graph M = (K,S,E), where the edge set E = {(k, s) : k → s | k ∈ K ∧ s ∈ S} and k → s indicates that the keyfact k aligns with the summary sentence s. Then, the percentage scores of completeness and conciseness on S are computed at the summary level by: Completeness(K,S)= ∣∣{k|(k, s)∈ E}
∣∣/|K| Conciseness(K,S)= ∣∣{s|(k, s)∈ E} ∣∣/|S|, (2)
where the operator | · | returns the number of unique items within the provided set. Intuitively, the two scores represent completeness, indicating the degree to which keyfacts are included in the summary, and conciseness, reflecting the density of relevant sentences aligning with given keyfacts. Moreover, unlike existing LLM-based methods (Liu et al., 2023; Wang et al., 2023; Shen et al., 2023), we provide more detailed information about the error type associated with each sentence and the alignment of each keyfact with summary sentences.

heading: 3.3 Prompt Engineering
text: We explore various prompt engineering strategies (Wei et al., 2022; Yu et al., 2023) to identify the most suitable one for our evaluation pipeline:
• Basic Prompt: A default question prompt in plain text, e.g., is the summary sentence supported by the transcript?
• Instruction: The prompt is provided using a step-by-step instruction using "Instruction:".
• Categorization: The prompt solves a categorization task by providing target categories.
• Reasoning: The prompt uses a chain-of-thought approach, incorporating a reasoning step.
• Evidence Mapping: The prompt requests an exact quote from the input to confirm the decision made by LLMs.
Combining all the above techniques was not always superior. Evaluation prompts are recommended to use instruction format with categorization and reasoning for faithfulness evaluation, as in Figure 3, and only instruction format for completeness and conciseness evaluation, as in Figure 4. See the detailed ablation in Appendix G.

heading: 3.4 Keyfact Extraction
text: The list of keyfacts is essential for evaluating the completeness and conciseness using FineSurE. Humans are best suited to generate these keyfacts as they understand the priorities in different domains, such as medicine or sales. However, in some cases, obtaining human keyfacts can be challenging. FineSurE works with human keyfacts by default, but for cases where no human keyfacts are provided, it can employ the LLM to extract keyfacts automatically. This process is entirely automated, utilizing prompts tailored for keyfact
3The list of keyfacts is provided by humans; if unavailable, it can be automatically derived from the reference summary. See Appendix C for details.
extraction (see Figure 5). For further details, refer to Appendix C. The impact of employing automatic keyfact extraction on keyfact alignment is discussed in Section 4.1.2.

heading: 4 Evaluation
text: Datasets To evaluate the automated evaluator’s performance, we need datasets with human annotations for sentence-level faithfulness and keyfactlevel alignment. Since no single dataset includes both types of annotations, we opt for two separate datasets. FRANK (Pagnoni et al., 2021) is a benchmark dataset of 2, 246 summaries for factuality evaluation metrics. It encompasses summaries of nine summarization systems on CNNDM (Hermann et al., 2015) and XSUM (Narayan et al., 2018), providing fine-grained annotations of sentence-level factuality error types. On the other hand, REALSumm (Bhandari et al., 2020) is another dataset of 2, 500 summaries from 25 summarization systems for automated metrics based on CNNDM. It includes a list of human keyfacts, along with corresponding annotations indicating their presence in the summary. FRANK and REALSumm obtain the inter-annotator agreement (IAA) scores of 0.58 (cohen’s kappa) and 0.66 (Krippendorff’s alpha) for three annotators, respectively.
LLMs as Evaluators We use the GPT-4-turbo (gpt-4-1106-preview) (Achiam et al., 2023) by default in main evaluation, but test with various open-source and proprietary LLMs, including Mixtral-8x7B (Jiang et al., 2024), Phi-2, Llama-2/3 (Touvron et al., 2023), GPT-3.5-turbo, and GPT4-omni (gpt-4o-2024-05-13), in Section 4.2. We set the temperature to 0 and clear the history for every evaluation instance, following the literature (Shen et al., 2023). We use HuggingFace models for
open-source LLMs and paid APIs for proprietary LLMs.
Baselines We compare FineSurE with five similarity-based methods, ROUGE-1/-2/-L (Lin, 2004), BERTScore (Zhang et al., 2019), and BARTScore (Yuan et al., 2021); a NLI-based method, SummaC-Conv (Laban et al., 2022); two QA-based methods, UniEval (Zhong et al., 2022) and QAFactEval (Fabbri et al., 2022); and the latest LLM-based method, G-Eval (Liu et al., 2023). Note that QAFactEval and SummaC-Conv are only compared for faithfulness evaluation, as they are limited to factuality. We obtain all the results by executing each metric in our experimental setup.
Performance Each automated evaluator’s performance is assessed by comparing estimated scores with ground-truth human judgments using sentence, summary, and system-level measurements. This multi-level analysis is crucial, as we seek to understand the agreement of the evaluator on each sentence, each summary, and the average performance of each summarization system.
Balanced accuracy (bAcc) assesses faithfulness in classifying each summary sentence for the presence or absence of factual errors at the sentencelevel. This is the average of true positive and true negative rates widely used when the two classes are imbalanced (Brodersen et al., 2010). Pearson and Spearman correlations assess all three dimensions at the summary-level by comparing percentage scores in Eqs. (1)-(2) derived from predicted and human evaluation results. Lastly, rank correlation is a system-level measure assessing the alignment of performance rankings across summarization systems (models) calculated by both our evaluator and humans. The details of the measurements are provided in Appendix D.

heading: 4.1 Main Results: Evaluators Comparison

heading: 4.1.1 Faithfulness
text: Table 1 summarizes the agreement between automated evaluators and human scores in faithfulness evaluation at three different granularities. FineSurE significantly outperforms similarity-, NLI-, and QA-based evaluators at all levels of evaluation.
It is important to note that none of the existing methods provide sentence-level evaluation results, relying instead on summary-level scoring, such as Likert-scale scores. It is noteworthy that FineSurE has the capability to assess whether each sentence contains a factual error or not, demonstrating remarkable alignment with human sentence-level judgments, with a balanced accuracy of 86.4%.
Given the strong alignment with human judgment, using LLMs as an evaluator holds great promise for enhancing the reliability of evaluation
3The pre-trained Bart-Large (Lewis et al., 2020) is finetuned on error localization data constructed using FRANK, comprising 3,885 training and 1,007 testing sentences, each paired with their corresponding human error categories.
processes for text summarization. However, one open question remains: Can LLMs identify the type of factuality error?
Table 2 unveils the capability of LLMs for factuality error localization, demonstrating accuracy as the probability that the predicted error category matches the correct answer given by humans. FineSurE outperforms the strong baseline, BartLarge3 fine-tuned on FRANK for error localization, despite not being trained on any error localization data, i.e., zero-shot prediction. Its superiority is primarily stemming from error categories that are uncommon in the training set for Bart-Large, such as PredE (141 cases), CirE (142 cases), and LinkE (41 cases). Nevertheless, LLMs still make numerous mistakes in accurately identifying the exact error type, despite their excellent performance in the binary decision of hallucination.
Therefore, achieving a level of evaluation comparable to human performance in more intricate assessment tasks remains a challenging objective.

heading: 4.1.2 Completeness and Conciseness
text: The agreement between automated evaluators and human scores on completeness and conciseness is summarized in Table 3. In contrast to similaritybased evaluators, which provide a single composite score, UniEval and G-Eval yield four distinct
scores, evaluating faithfulness, coherence, relevance, and fluency. We use their coherence and relevance scores to calculate the correlation with human scores for completeness and conciseness, as they indicate the inclusion and density of key information, respectively.
Overall, FineSurE using human keyfacts demonstrates a very high agreement with human evaluations for completeness and conciseness, surpassing other evaluators significantly. This is because keyfact alignment is essential to verify the coverage of crucial information in the summary, a task that cannot be accomplished with existing LLM-based method like G-Eval. See the qualitative example in Appendix E. We also assess the performance of FineSurE without employing human keyfacts and, instead, utilizing machine-generated keyfacts, as outlined in Appendix C. The keyfacts are extracted using GPT-4 with a specific prompt. It is noteworthy that, even with machine-generated key facts, FineSurE maintains a higher level of agreement over other automated evaluators.
With an advantage as a fine-grained evaluator, FineSurE also provides evaluation results at the keyfact-level, revealing which keyfacts are omitted in the summary, i.e., keyfact matching. Given a list of keyfacts, it includes binary labels ("Yes" or "No") in the JSON output, as illustrated in Figure 4. Therefore, we assess the agreement for the keyfact matching task by calculating the IAA score between machine and human labels. FineSurE demonstrates a Krippendorff’s alpha of 0.65 for keyfact matching. This robust agreement at various levels corroborates that FineSurE has a potential to be an effective fine-grained automatic evaluator.
Furthermore, in Appendix F, we compare FineSurE with two variants of G-Eval, which are tailored for completeness and conciseness evaluation by modifying its prompts to be more suitable
for such assessment and integrating them for use with keyfacts. FineSurE maintains its significant dominance even with additional tuning on G-Eval.

heading: 4.1.3 Stability in Evaluation Results
text: Concerns arise about evaluation result stability with LLMs due to their inherent text generation randomness, even at temperature 0. Despite LLMbased methods relying on Likert-scale evaluation, such as G-Eval, showing significant fluctuations in judgment alignment (Shen et al., 2023; Liu et al., 2023), Table 4 demonstrates that FineSurE (GPT4) maintains much higher agreement in summarylevel evaluation scores across three distinct runs. This underscores the benefit of employing finegrained percentage scores derived from sentenceand keyfact-level assessments.

heading: 4.2 LLMs as Evaluators Comparison
text: It is interesting to observe how the evaluation agreement varies based on the choice of LLMs, given the abundance of open-source and proprietary LLMs.
Success Ratio. The primary limitation of opensource LLMs is their comparatively lower success ratio in following prompts, compared to proprietary LLMs; only Llama3-70B-Inst exhibits a high success ratio comparable to proprietary LLMs. Upon analyzing failure cases, the top three reasons are: (1) the output is either not in JSON format or an incorrect JSON format, (2) the output consists of meaningless text, e.g., python codes or no output at all, and (3) the JSON output includes only a few lines of sentences or keyfacts.
Furthermore, the maximum token length in context for open-source LLMs is notably shorter compared to proprietary LLMs. GPT-4 series can process up to 128K tokens, whereas open-source LLMs generally handle up to 8K input tokens. This results in prompt truncation when handling lengthy
input texts, potentially leading to failures in generating accurate outputs in text summarization.
Agreement with Human Score. Tables 5-6 summarize the correlation of nine different LLMs with human judgment, computed only for the successful cases of adhering to the prompt. Although the recent Llama3-70B-Inst shows strong agreement with humans, in general, there is a noticeable gap between open-source and proprietary LLMs. Regarding open-source LLMs, the agreement with human scores increases with the model size; for example, Llama2-70B exhibits a higher correlation coefficient than Phi-2. Additionally, instruction tuning also plays a role, as observed in Mixtral-8x7b’s performance, which improved significantly after instruction tuning. In contrast, all the proprietary LLMs exhibit high correlation coefficient. Particularly, more recent and powerful LLMs exhibit better performance, i.e., GPT-4-turbo > GPT-3.5turbo, GPT-4-omni > GPT-4-turbo.
It’s notable that LLMs with a high success ratio exhibit a strong correlation, suggesting they are not penalized by their high success ratios. Therefore, a more advanced LLM simultaneously achieves higher agreement and success ratios.
Error Localization We provide a detailed factuality error localization analysis using different LLMs in Table 7. GPT-4-omni improves the mean accuracy in error localization by 10% over GPT4-turbo. The categorization accuracies of opensource LLMs are considerably lower than those of proprietary LLMs in general. However, the latest open-source LLM, Llama3-70B-Inst, outperforms GPT-4-turbo in error localization, achieving an average prediction accuracy of 49.4%, which is 7.2% higher than that of GPT-4-turbo. Additionally, instruction tuning demonstrates a significant accuracy boost in this task, as evidenced by the improvement from Mixtral-8x7b to Mixtral-8x7b-Inst.

heading: 4.3 Evaluation using FineSurE
text: As an actual application of an automated evaluator, we gather summaries generated by four open-source and four proprietary LLMs, and subsequently assess their summarization quality using the FineSurE algorithm (see the prompt for summarization we used in Appendix H). Figure 2 shows the percentage scores of the eight LLMs for faithfulness, completeness, and conciseness. The summaries are generated for 100 examples sourced from CNNDM, all of which are also included in
REALSumm, thereby possessing the list of keyfacts extracted by human annotators.
In general, proprietary LLMs, including different versions of GPT, generate high-quality summaries in comparison to open-source counterparts. Interestingly, GPT-4-omni exhibits the highest agreement with humans as an automated evaluator in Tables 5-6, but its faithfulness and completeness scores are significantly worse even than GPT-3.5turbo. Consequently, GPT-4-omni is likely to include more hallucinations and miss many important keyfacts in summary generation.
The performance ranking of each model changes significantly for each evaluation dimension. GPT3.5-turbo, GPT-4-turbo, and GPT-4-omni are the best for faithfulness, completeness, and conciseness, respectively. Nevertheless, it is noteworthy that Llama3-70B-Inst, an open-source LLM, exhibits comparable performance to the state-ofthe-art proprietary LLMs. In open-source LLMs, instruction tuning significantly enhances summarization quality, as evidenced by the performance increase of Mixtral-8x7b-Inst over Mixtral-8x7b. These findings align with prior observations reported in recent studies on faithfulness (Laban et al., 2023) and instruction tuning (Zhang et al., 2023).
Lastly, while there is no doubt that faithfulness is crucial, achieving both completeness and conciseness simultaneously turns out to be very important and challenging in text summarization, as evident from the low percentage scores even with GPT-4 series. Therefore, it emphasizes the need to put more effort into these aspects for a good summary.

heading: 5 Conclusion
text: We introduce FineSurE, a novel automated evaluator designed for fine-grained and multi-dimensional text summarization evaluation. The evaluation process is broken down into fact checking and keyfact alignment, providing detailed insights, where keyfacts can be either provided by humans or generated by LLMs. Our experiments include a thorough comparison with existing evaluators, exploration of performance using eight opensource or proprietary LLMs, and real quality assessment of recent
LLM-generated summaries. The results indicate the potential effectiveness of FineSurE as a text summarization evaluator, showcasing its promising capabilities in advancing automated evaluation for text summarization.

heading: Limitations
text: Our automated evaluator is primarily tested on the news domain due to the limited availability of benchmark datasets with fine-grained human annotations. We emphasize the critical importance of constructing a high-quality benchmark dataset with high diversity in input domains, length, and types. Also, the prompts for evaluation may need to be tuned if a different summary is expected like the summary from the medical domain. Lastly, other aspects can be considered for text summarization, such as toxicity and social bias. We leave these challenges as future work.

heading: Ethics Statement
text: This paper focuses on designing an automatic evaluator using LLMs for text summarization. Therefore, we do not anticipate any negative ethical and social impact.

heading: Acknowledgements
text: The first author, Hwanjun Song, was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00334343) and Artificial Intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City (No. BA00000772).

heading: Instruction:

heading: Transcript:

heading: Summary with N sentences:

heading: A Factuality Error Type
text: Following the work (Fu and Frank, 2023), we use seven categories to define factuality error types, namely "out of context", "predicate," "entity," "circumstance," "coreference," "discourse link," and "grammatical". Table 8 provides the detailed description and example of the error categories.

heading: B Main Prompt
text: We employ LLMs as a tool to conduct fact checking and keyfact alignment tasks. Specifically, we design two prompts tailored for the two tasks, as shown in Figures 3-4.

heading: C Keyfact Extraction
text: The list of key facts is crucial for evaluating completeness and conciseness. Ideally, they should be generated by humans, as the key facts in text summarization heavily depend on what information humans prioritize in various domains. For instance, in a medical scenario, keyfacts should encompass all medical symptoms and the doctor’s recommended
treatment, while in a sales call, the customer’s issue and action should be prioritized as the key facts.
We also automatically extract feasible keyfacts from the reference summary, similar to the approach taken by REALSumm in annotating key facts with human annotators. However, our process is fully automatic, utilizing LLMs instead of human’s manual efforts. We designed the prompt tailored for extracting keyfacts in Figure 5. This prompt generates up to 16 key facts from the reference summaries by providing few-shot examples
to ensure that the granularity of extracted key facts aligns with our requirements.

heading: D Measurement
text: We utilize several measurement to compute the agreement with human judgements at the three different levels.
For sentence-level assessment, we utilize human binary annotations indicating the presence of factuality errors for each sentence, denoted as "0" for no error and "1" for error. Similarly, the LLM returns the binary decision of "0" (No) and "1" (Yes) by the fact checking prompt per sentence, as shown in Figure 3. Then, the balanced accuracy (bACC) is computed by:
bACC = (sensitivity + specificity)/2, (3)
where sensitivity refers to the true positive rate, which measures the proportion of correct predictions by LLMs out of all positive predictions. On the other hand, specificity is the true negative rate, measuring the proportion of correct predictions out of all negative predictions.
For the summary-level assessment, let D = {d1, . . . , dk} be the set of input documents and S = {s1, . . . , sk} be the set of summaries corresponding to the document set. Supposing that Fgt and Fpred are the functions that returns the percentage score of a specific evaluation dimensions based
on human and predicted labels, respectively (F can be any scoring function in Eq. (1)-(2)). Then, the summary-level correlation is calculated as follows:
Corr ([ Fgt(d1, s1), . . . , Fgt(dk, sk) ] , [ Fpred(d1, s1), . . . , Fpred(dk, sk) ]) ,
(4)
where Corr is one of the Pearson and Spearman correlation measure. The measurement reveals the agreement between the percentage scores generated by automated evaluators and human judgments.
For system-level assessment, we consolidate the percentage scores across all input documents , determining the average percentage score for each summarization model. Let Fm = {Fm(d1, s1), . . . , Fm(dk, sk)} represents the percentage scores derived from the labels assigned by a summarization system m. Then, we make a list of the average percentage score for all summarization systems, [ ¯Fm1 , ¯Fm2 , . . . ] and compute their ranking by using the Rank function, returning the list [rankm1 , rankm2 , . . . ], where rankm is the ranking of the model m. Given a list of ground-truth rankings [rank∗m1 , rank ∗ m2 , . . . ] using the human scores, we compute the rank correlation by:
Spearman ([ rankm1 , rankm2 , . . . ] ,
[ rank∗m1 , rank ∗ m2 , . . . ]) .
(5)

heading: He was on his way to Britain[9] from Calais when

heading: Evaluation Criteria:

heading: Evaluation Steps:

heading: 1. Read the news article carefully and identify the main points, key information, and relevant details.

heading: 2. Read the summary and compare it to the article. Check if the summary captures all essential facts, main ideas, and pertinent details presented in the original article.

heading: 3. Assign a score for completeness based on the Evaluation Criteria.

heading: Evaluation Form (scores ONLY):

heading: E Qualitative Analysis for Completeness
text: Table 9 shows an example showing the limitation of the existing Likert-scale based evaluation method, such as G-Eval. Although the model summary includes eight out of ten human keyfacts, G-Eval output a very low completeness score, i.e., two out of five. However, the proposed FineSurE exhibits the percentage score similar to that of human judgement. This is because our framework conducts a human-like keyfact alignment to determine the completeness score, which is more fine-

heading: Evaluation Criteria:

heading: Evaluation Steps:

heading: 1. Read the news article carefully and identify the main points, key information, and relevant details.

heading: 3. Assign a score for conciseness based on the Evaluation Criteria.

heading: Example:

heading: Evaluation Form (scores ONLY):
text: grained than relying solely on Likert-scale judgments. Therefore, a fine-grained evaluation framework has great potential to enhance the quality of automated evaluation using LLMs.

heading: F Fair Comparison with G-Eval
text: For a truly fair comparison, we recognize the necessity to tune G-Eval for our two dimensions of completeness and conciseness. Hence, we opted to adjust G-Eval’s prompts to align with the evaluation dimensions of FineSurE. We employed two
variants of G-Eval tailored for the assessment of completeness and conciseness dimensions: one adjusted without the utilization of keyfacts, and another adjusted with the utilization of keyfacts. The two tuned prompts without using key facts are summarized in Figures 6 and 8, while those with using key facts are in Figures 7 and 9.
Table 10 compares FineSurE with two variants of G-Eval regarding completeness and conciseness. The findings illustrate that aligning evaluation criteria does not alter performance ("+Adjusted Criteria"), whereas the incorporation of human keyfacts notably enhances G-Eval ("+Using Keyfacts"). Nonetheless, FineSurE continues to outperform GEval, supporting that our fine-grained evaluation yields more precise assessments compared to the Likert-scale ratings used in G-Eval.

heading: G Prompt Engineering
text: We test several prompt engineering techniques, including instruction format, solving categorization

heading: Evaluation Criteria:

heading: Evaluation Steps:

heading: 1. Read the list of key facts.

heading: 2. Read the summary and compare it to the key facts. Check if the summary captures all the key facts.

heading: 3. Assign a score for completeness based on the Evaluation Criteria.

heading: Example:

heading: Key Facts:

heading: Summary: {summary}

heading: Evaluation Form (scores ONLY):
text: problem, reasoning, and evidence mapping. We summarize the agreement with human scores using different combination of prompting techniques with respect to faithfulness, completeness, and conciseness in Tables 11 - 12, where the figure number 10–15 corresponding to each prompt is enclosed in parentheses in each row.
Regarding the prompt engineering for faithfulness, our most effective prompt involves the incorporation of three techniques: instruction format,

heading: Summary with N sentences:

heading: Transcript:

heading: Is the summary sentence supported by the transcript? Response with "Yes" or "No" for each sentence in the summary.

heading: Evaluation Form (scores ONLY):

heading: Summary: {summary}

heading: Key Facts:

heading: 3. Assign a score for conciseness based on the Evaluation Criteria.

heading: 1. Read the list of key facts.

heading: Evaluation Steps:

heading: Evaluation Criteria:

heading: Instruction:

heading: Transcript:

heading: Summary with N sentences:
text: solving categorization, and providing reasoning, as seen in in Figure 3. This prompt shows the highest overall agreement with human judgments among those that included fine-grained evaluation, such as error categorization.
On the other hand, for completeness and conciseness, the inclusion of the reasoning step results in a decrease in agreement during evaluation. Additionally, categorization is unnecessary for keyfact align-

heading: Transcript:

heading: Summary with N sentences:
text: ment, given the presence of only two classes–"no matched" and "matched"–for each keyfact. Consequently, the most effective prompt involves solely utilizing the instruction format in Figure 4.

heading: Transcript:

heading: Summary with N sentences:

heading: If "Yes", please indicate the line number(s) of the summary sentence(s) relevant to the key fact.

heading: Summary:

heading: M key facts:

heading: H Summarization using LLMs
text: We consistently employ the prompt shown in Figure 16 across all LLMs, generating model summaries based on the given input text.
Furthermore, since the model summary is null when parsing fails, there is no hallucination, but also no summary sentences align with any key facts. Therefore, in cases where the LLMs produce incor-

heading: Instruction:

heading: Summary:

heading: M key facts:

heading: Text: {input text}

heading: JSON Output:
text: rect JSON outputs that cannot be parsed, the scores for faithfulness, completeness, and conciseness are automatically set to 1.0, 0.0, and 0.0, respectively.

heading: I Extension of REALSumm
text: Although the REALSumm data contains human labels indicating which keyfacts are included in the model summary, there are no human labels indicating which summary sentences align with the keyfacts. The former is used to compute the ground-truth completeness score, while the latter is used to compute the conciseness score. Therefore, we conducted a human evaluation to verify which summary sentences align with the set of key facts. Specifically, three human annotators were asked to mark "yes" if at least one keyfact in the keyfact list could be inferred from each summary sentence, otherwise "no". This is quite simple task compared with faithfulness evaluation, since human ground-truth keyfacts are available in REALSumm data. The extended dataset is available with our FineSurE framework at https: //github.com/DISL-Lab/FineSurE.

heading: Future Work
text: We leave these challenges as future work