Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
Authors: Hyangsuk Min, Yuho Lee, Minjeong Ban, Jiaqi Deng, Nicole Hee-Yeon Kim, Taewon Yun, Hang Su, Jason Cai, Hwanjun Song
Accepted at: acl25

Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages

Hyangsuk Min1,*, Yuho Lee1,*, Minjeong Ban1, Jiaqi Deng1, Nicole Hee-Yeon Kim1, Taewon Yun1, Hang Su2,†, Jason Cai2,‡, Hwanjun Song1,§
1Korea Advanced Institute of Science and Technology
2AWS AI Labs
{hyangsuk.min, yuholee, songhwanjun}@kaist.ac.kr

Abstract
Evaluation frameworks for text summarization have evolved in terms of both domain coverage and metrics. However, existing benchmarks still lack domain-specific assessment criteria, remain predominantly English-centric, and face challenges with human annotation due to the complexity of reasoning. To address these, we introduce MSumBench, which provides a multi-dimensional, multi-domain evaluation of summarization in English and Chinese. It also incorporates specialized assessment criteria for each domain and leverages a multi-agent debate system to enhance annotation quality. By evaluating eight modern summarization models, we discover distinct performance patterns across domains and languages. We further examine large language models as summary evaluators, analyzing the correlation between their evaluation and summarization capabilities, and uncovering systematic bias in their assessment of self-generated summaries. Our benchmark dataset is publicly available at https://github.com/DISL-Lab/MSumBench.

1 Introduction
Recent advancements in large language models (LLMs) have enhanced text summarization performance. However, LLM-generated summaries still face challenges, including hallucinations, omission of critical information, and redundancy (Lee et al., 2024). These limitations highlight the continued need for advanced automated evaluation methods that can assess summarization quality more efficiently and cost-effectively than human annotation.

While automated evaluations have made progress, there remains a significant gap between automatic evaluations and human judgments, reinforcing the need for benchmarks with high-quality human annotations to provide a more reliable assessment of summarization capabilities.

However, existing benchmarks face three key challenges. First, most benchmarks employ uniform criteria for assessing summary quality, failing to account for domain-specific differences in what constitutes important information (Laban et al., 2023; Lee et al., 2024). Second, they remain largely monolingual—primarily focused on English—limiting their ability to provide robust evaluations across languages (Bhandari et al., 2020; Fabbri et al., 2021; Pagnoni et al., 2021; Laban et al., 2022, 2023; Tang et al., 2023; Lee et al., 2024; Tang et al., 2024b). Third, collecting high-quality human annotations remains a significant challenge. Assessing summarization quality requires complex reasoning, making the annotation resource-intensive and inconsistent (Krishna et al., 2023; Lee et al., 2024). These challenges hinder the timely development of reliable benchmarks that can keep pace with the rapid advancements in LLM capabilities.

To address these, we create MSumBench, Multi-aspect Summarization Benchmark (Figure 1), a summarization benchmark for English and Chinese with domain-specific evaluation criteria (see Table 2). Building on existing multi-dimensional and fine-grained evaluation frameworks (Song et al., 2024), human annotations for summarization quality are collected at both the sentence and key-fact1 levels, focusing on faithfulness, completeness, and conciseness. Furthermore, as depicted in Figure 2, we propose a multi-agent debate system that facilitates effective AI-human collaboration in handling the complex task of assessing summary faithfulness. Inspired by prior findings that demonstrate the effectiveness of LLM-based debate (Chan et al., 2024; Du et al., 2024; Khan et al., 2024; Koupaee et al., 2025), we extend this approach to the sum-

* Equal Contribution.
† This work is conducted independently and is not related to the author(s)' position at Amazon.
‡ Corresponding Author.
§ A key-fact is a succinct statement capturing an individual essential information unit, containing a maximum of two to three entities (Bhandari et al., 2020; Song et al., 2024).
Figure 1: Overview of MSumBench, featuring multi-domain documents in both English and Chinese, with domain-specific key-facts. Model summaries are evaluated via a multi-agent debate framework, aiding annotators’ assessments. Each summary then receives percentage scores for faithfulness, completeness, and conciseness.

marization annotation task. Our debate-based system provides annotators with structured arguments from LLM agents—called the Advocate and the Skeptic—with contrasting viewpoints, thereby reducing dependence on any single viewpoint. We further include the Adjudicator’s review to finalize the debate and ensure that all arguments remain fact-based and consistent with the source content. To ease the cognitive load of annotators, we guide them to focus on the most relevant portions of the source text—key entities, relationships, and critical details—rather than requiring them to consider the entire document from scratch. This comprehensive yet focused process not only promotes more accurate annotations but also helps minimize biases from unbalanced or incomplete information.

Our main contributions are: (1) We propose an evaluation strategy that identifies different types of critical information specific to each domain. (2) We develop a multi-agent, debate-based annotation framework that generates structured arguments with contrasting perspectives, enabling human annotators to focus on key aspects of the task. (3) We conduct a comprehensive evaluation of state-of-the-art LLMs as summarizers for English and Chinese, using our multi-dimensional annotations. (4) Using the collected annotations as ground truth, we thoroughly examine the effectiveness of LLMs as automated evaluators in both languages. (5) We release the MSumBench benchmark to facilitate advancements in summarization evaluation.

2 Related Works

Evaluation Benchmarks Conventional summarization benchmarks mainly focus on the news domain (Pagnoni et al., 2021; Tang et al., 2023), which has led to the development of benchmarks incorporating dialogues (Gao and Wan, 2022; Krishna et al., 2023; Tang et al., 2024b), and multi-domains such as SummEdits (Laban et al., 2023) and UniSumEval (Lee et al., 2024). However, the monolingual focus on English remains a persistent challenge. While some benchmarks like mFACE (Aharoni et al., 2023) and MFHHD (Shen et al., 2024) provide multilingual evaluations, their focus is limited to the news domain, calling for a single benchmark that considers multiple languages and multiple domains for comprehensive evaluations. Furthermore, while criteria for evaluating summary quality differ across domains, existing multi-domain benchmarks generally employ uniform criteria across different domains (Laban et al., 2023; Lee et al., 2024).

Evaluation Metric Many existing benchmarks focus solely on faithfulness (i.e., factual consistency) as an evaluation dimension of summarization quality (Bhandari et al., 2020; Pagnoni et al., 2021; Laban et al., 2022; Tang et al., 2023; Krishna et al., 2023; Laban et al., 2023). Some have expanded to multiple dimensions like coherence and relevance (Fabbri et al., 2021; Gao and Wan, 2022; Tang et al., 2024b), while Lee et al. (2024) uses completeness and conciseness for greater evaluation consistency. Evaluation measurement has likewise shifted from coarse, summary-level binary or scale-based ratings (Fabbri et al., 2021; Laban et al., 2022; Gao and Wan, 2022; Tang et al., 2023; Laban et al., 2023; Tang et al., 2024b) to fine-grained, sentence-level (or lower) assessments represented as percentage scores (Bhandari et al., 2020; Pagnoni et al., 2021; Krishna et al., 2023; Lee et al., 2024).

Paralleling these advances, automated metrics have also evolved: from traditional similarity-based measures like ROUGE (Lin, 2004), BERTScore (Yuan et al., 2021), and BARTScore (Zhang et al., 2019); to natural language inference (NLI) and question answering (QA)-based methods (Fabbri et al., 2022; Tang et al., 2024a); and finally to LLM-based approaches (Liu et al., 2023; Song et al.,
<table>
  <tr>
    <th></th>
    <th>Domain Coverage</th>
    <th>Evaluation Dimensions</th>
    <th>Domain-specific Evaluation</th>
    <th>Annotation Unit</th>
    <th>Measurement</th>
    <th>Language</th>
  </tr>
  <tr>
    <td>mFACE</td>
    <td>X Single</td>
    <td>✓ 3</td>
    <td>X No</td>
    <td>X Summary</td>
    <td>X Likert</td>
    <td>✓ Multi</td>
  </tr>
  <tr>
    <td>SummEdits</td>
    <td>✓ Multi</td>
    <td>X 1</td>
    <td>X No</td>
    <td>X Summary</td>
    <td>X Ternary</td>
    <td>X Single</td>
  </tr>
  <tr>
    <td>MFHHHD</td>
    <td>X Single</td>
    <td>X 1</td>
    <td>✓ Yes</td>
    <td>▲ Sentence*</td>
    <td>X Ternary</td>
    <td>✓ Bi</td>
  </tr>
  <tr>
    <td>UniSumEval</td>
    <td>✓ Multi</td>
    <td>✓ 3</td>
    <td>X No</td>
    <td>✓ Sentence & Key-fact</td>
    <td>✓ Percentage</td>
    <td>X Single</td>
  </tr>
  <tr>
    <th>Ours</th>
    <th>✓ Multi</th>
    <th>✓ 3</th>
    <th>✓ Yes</th>
    <th>✓ Sentence & Key-fact</th>
    <th>✓ Percentage</th>
    <th>✓ Bi</th>
  </tr>
</table>

Table 1: Benchmark comparison. *The annotated summaries are constrained to a single sentence format.

<table>
  <tr>
    <th>News</th>
    <th>Medical Literature</th>
    <th>Report</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
  </tr>
  <tr>
    <td>Main topic<br>Background<br>Immediate impact<br>Public statements<br>Official statements<br>Counter arguments<br>Future implications</td>
    <td>Research finding<br>Disease descriptions<br>Medical experiments<br>Medical treatment<br>Medical prevention</td>
    <td>Governance Evaluations<br>Recommendations<br>Regulation/policy<br>Financial info</td>
    <td>User requests<br>System suggestions<br>Location/route<br>General information<br>Booking confirmation<br>Price/payment<br>Time/schedule</td>
    <td>Opinions Reports Decisions Proposals Factual info</td>
    <td>Background Main arguments Supporting examples Counter arguments Conclusions</td>
  </tr>
</table>

Table 2: Key-fact categories tailored to each domain, with the first row representing domains and the second row listing domain-specific key-facts. See Appendix B.2 for a detailed description.

2024; Wan et al., 2024). However, multilingual coverage remains challenging as noted in Forde et al. (2024). Table 1 highlights how MSumBench addresses these limitations by comparing it against some of the existing benchmarks.

3 MSumBench Construction Procedure

We construct MSumBench following the systematic pipeline of four components: dataset collection, domain-specific key-fact generation, summary generation, and summary evaluation. Detailed statistics of MSumBench are provided in Appendix A.

3.1 Dataset Collection

Source Dataset Evaluating summarization models on a single domain provides limited insight into the robustness of their performance. Accordingly, MSumBench is constructed based on the datasets from six domains with distinct characteristics: CNN/DM (news) (Nallapati et al., 2016), GovReport (report) (Huang et al., 2021), PubMed (medical literature) (Cohan et al., 2018), MultiWOZ (booking conversation) (Zang et al., 2020), MediaSum (interview) (Zhu et al., 2021), and MeetingBank (meeting) (Hu et al., 2023). From each domain, we sample 25 documents, yielding a total of 150 source documents to generate summaries.

Documents Translation Since MSumBench aims to evaluate both English and Chinese summaries, identical source texts are needed in both languages to ensure contextual consistency and fair comparison. Therefore, we opt to translate the original English source documents into Chinese. The translation follows the three steps. First, we use GPT-4o for initial sentence-level translation with domain-specific prompts (see Appendix B.1) to maintain contextual coherence. Second, we use Qwen-2.5-72B to screen the resulting translations to flag unnatural or inaccurate sentences and to alleviate any bias potentially introduced by using GPT-4o as a single translator. Finally, any flagged sentences are reviewed by bilingual native Chinese examiners for refinement. This multi-step validation ensures the accuracy, naturalness, and contextual integrity of the translations.

3.2 Domain-Specific Key-Facts Extraction

Domain-specific criteria are critical for summary evaluation, as each domain emphasizes distinct content. Thus, we define tailored key-fact categories that serve as templates for ideal summary contents in each domain. This approach ensures summary evaluations reflect the most important aspects of original documents while adhering to the established norms and practices of each domain.

Key-Fact Extraction Procedure The domain-specific key-fact extraction procedure follows three steps. First, Category Identification: To determine which information categories matter most in each domain, we analyze human-written reference summaries2 from existing summarization datasets. Then, we derive approximately 5–7 recurring categories of essential information (see Table 2). This

2Since MultiWOZ lacks reference summaries, we analyze frequently occurring entity categories labeled by humans.
ensures that each category captures frequently emphasized details within its respective domain.

Second, Key-fact Classification: We use GPT-4o to generate candidate key-facts from source documents and classify them using domain-specific categories, filtering out non-aligned key-facts. This minimizes risk of missing any potential key-facts while discarding information that does not meet domain-specific priorities.

Third, Key-fact Validation: To enhance reliability, each filtered key-fact is verified by three LLMs—GPT-4o, Claude-3.5-sonnet, and Llama-3.1-70B. Each model checks whether the key-fact: (1) is useful for summarizing; (2) is consistent with the information in the source text; and (3) falls under one of the predefined domain-specific categories. Using a majority vote, any key-fact failing these checks is discarded. This cross-verification mitigates potential biases or oversights that could arise from relying on a single model. The resulting consensus-based key-facts constitute a robust domain-specific reference set, forming the backbone of our subsequent evaluation of summary quality. Appendix B.2 provides the prompts for extracting and validating the key-facts.

3.3 Summary Generation
To evaluate how summarization capabilities vary across different model scales and architectures—from traditional fine-tuned models to the latest LLMs, we select eight models for benchmarking, categorized into three groups: non-LLMs, including fine-tuned BART-Large (Lewis et al., 2020) and mT5 (Xue et al., 2021), open-source LLMs, including Llama-3.1-70B (Grattafiori et al., 2024), Gemma-2-27b (Team et al., 2024), and Qwen-2.5-72B (Yang et al., 2024), and proprietary LLMs, including GPT-4o (OpenAI, 2024), Claude-3.5-Sonnet, and Gemini-1.5-pro (Gemini Team, 2024). For English documents, we generate 1,200 summaries (25 source documents × 6 domains × 8 summarizers). For Chinese summaries, we exclude BART due to its lack of multilingual support, resulting in 1,050 generated summaries (25 source documents × 6 domains × 7 summarizers). See Appendix B.3 for model configuration details.

3.4 Summary Evaluation
Human Annotation Tasks  Traditional summary evaluation dimensions (e.g., coherence and relevance) are insufficient for fine-grained evaluation because they lack clear, measurable criteria. Instead, we assess the generated summaries based on three dimensions–faithfulness, completeness, and conciseness–following Song et al. (2024). Our annotation process consists of two main tasks: fact verification and key-fact alignment.

In the fact verification task, annotators evaluate faithfulness at the sentence-level by identifying factual errors based on the existing error taxonomy (Lee et al., 2024) (see Appendix C). In the key-fact alignment task, annotators verify whether each key-fact can be inferred from the summary sentences, similar to an NLI task. These tasks yield 9,951 and 188,800 annotations from each respective task.

We collect our human annotations via Amazon Mechanical Turk (MTurk), with each annotation unit assigned to three independent annotators to ensure reliability. Further details on annotator recruitment and procedure are provided in Appendix H.

Fact Verification with Multi-Agent Assistance
Although human judgments remain essential for accurate evaluation, the annotation process is both costly and challenging, as summarizers evolve and error types become more nuanced. A straightforward solution might be to have humans inspect labels generated by an LLM-based evaluator, similar to the work of Lee et al. (2024). However, this can lead to a bias where annotators blindly endorse LLMs’ decisions. Thus, we introduce a more systematic approach that fosters effective AI-human collaboration while mitigating bias: a multi-agent debate-assisted annotation framework (Figure 2).

Our framework employs three LLM agents for fact verification: the Advocate, the Skeptic, and the Adjudicator. The Advocate and the Skeptic engage in the core debate, while the Adjudicator investigates their arguments. The agents debate the faithfulness of each summary sentence against the full input document (with input sentences numbered).

Specifically, the Advocate presents evidence supporting the faithfulness of a given summary sentence, and the Skeptic presents evidence suggesting

![Annotation assistance via multi-agent debate.](page_1012_120_382_312.png)

Figure 2: Annotation assistance via multi-agent debate.
<table>
  <tr>
    <th>Annotation Task</th>
    <th>Measuring Dimensions</th>
    <th>News</th>
    <th>Report</th>
    <th>Medical Lit.</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
    <th>Avg (EN)</th>
    <th>Avg (ZH)</th>
    <th>Avg (All)</th>
  </tr>
  <tr>
    <td>Fact Verification</td>
    <td>Faithfulness</td>
    <td>0.67</td>
    <td>0.64</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.50</td>
    <td>0.64</td>
    <td>0.55</td>
    <td>0.61</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Key-fact Alignment</td>
    <td>Completeness & Conciseness</td>
    <td>0.70</td>
    <td>0.84</td>
    <td>0.77</td>
    <td>0.83</td>
    <td>0.80</td>
    <td>0.82</td>
    <td>0.77</td>
    <td>0.82</td>
    <td>0.79</td>
  </tr>
</table>

Table 3: Consistency of three human annotators across 6 domains for 2 languages, where the first six columns are the IAA scores for each domains, while the last three are the average IAA for two languages and the overall one.

it may be unfaithful. Both agents first highlight relevant sentence numbers in the input text so annotators can quickly locate the information. Next, they explain which parts of the summary sentence align (or fail to align) with the referenced input sentences (see Table 24 and 25). The Skeptic also specifies the error type. The Adjudicator then reviews both arguments, the input context, and the summary sentence, focusing on two key aspects: (1) whether the provided evidence is consistent with the source text, and (2) whether the arguments follow the faithfulness criteria (see Table 26). This step prevents superficial objections (e.g., wrongly labeling correct paraphrases as errors). Finally, the Adjudicator produces an investigation report and a tentative label, which are shown to the annotator along with the reference text and both sides’ arguments. We use Llama-3.1-70B for all three agents.

By providing balanced evidence from both sides and clearly indicating how each argument relates to specific source sentences, this framework minimizes bias and offers a practical aid to annotators. While multi-round debates (Ray, 2023; Du et al., 2024) may yield richer evaluations, they add cognitive load for annotators, undermining practical benefits as assistance. Conversely, our single-round setup balances thoroughness and usability, ensuring efficient and accurate annotations.

Key-Fact Alignment with NLI Assistance
While the multi-agent debate aids in detecting nuanced faithfulness errors, key-fact alignment is a straightforward comparison between two concise sentences, requiring no complex reasoning mechanisms. Therefore, we present annotators with an NLI result generated by Llama-3.1-70B, indicating whether the key-fact is entailed by the summary sentence, along with a brief rationale. This approach streamlines the annotation process by guiding annotators to make quick, confident judgments, thereby reducing their overall cognitive load.

4 Quality Assessment
A high-quality benchmark ensures that model evaluations capture genuine performance differences by minimizing annotation noise (Lee et al., 2024; Tang et al., 2024b). Thus, we evaluate the reliability and accuracy of collected annotations to ensure the integrity of MSumBench.

<table>
  <tr>
    <th>Task</th>
    <th>Measure</th>
    <th>UniSumEval</th>
    <th>MSumBench</th>
  </tr>
  <tr>
    <td>Key-fact List</td>
    <td>Human Preference</td>
    <td>25.00%</td>
    <td>75.00%</td>
  </tr>
  <tr>
    <td>Fact Labels</td>
    <td>Balanced Accuracy</td>
    <td>80.07%</td>
    <td>92.83%</td>
  </tr>
</table>

Table 4: Comparison of MSumBench over UniSumEval. Human Preference is the A/B test preference ratio, while Balance Accuracy is measured using the two expert examiners’ labels as reference.

4.1 Annotation Consistency
Inter-Annotator Agreement (IAA) is a measurement to assess the reliability of human annotations by quantifying the consistency between different annotators. Table 3 reports the IAA scores using Krippendorff’s \( \alpha \) (Krippendorff, 2011) across domains for our two annotation tasks, namely fact verification and key-fact alignment. We achieve very high average IAA scores, Avg (All), of 0.58 and 0.79 for the two tasks, respectively. This indicates that MSumBench stands out as the only comprehensive benchmark with domain-specific evaluations and bilingual coverage, ensuring robust and diverse summarization assessment.

4.2 Comparison with UniSumEval
While high IAA scores indicate annotation consistency, they do not guarantee dataset quality due to potential biases and accuracy issues (Munappy et al., 2022; Braylan et al., 2022). Therefore, we conduct an additional quality check by recruiting two postgraduate NLP specialists as expert examiners, both proficient in English and Chinese (see Appendix H for details on recruitment).

We perform quality checks on two critical components of MSumBench: (1) Domain-Specific Key-Fact Extraction to ensure that extracted key-facts align with domain characteristics, and (2) Multi-Agent Debate to confirm the accuracy of human annotations assisted by our multi-agent system. To demonstrate the effectiveness of these quality improvements, we compare our dataset with the latest UniSumEval dataset (Lee et al., 2024).
<table>
  <tr>
    <th rowspan="2">Model Type</th>
    <th rowspan="2">Summarizer</th>
    <th colspan="4">English</th>
    <th colspan="4">Chinese</th>
    <th rowspan="2">Language Stability</th>
  </tr>
  <tr>
    <th>Faithfulness</th>
    <th>Completeness</th>
    <th>Conciseness</th>
    <th>Domain*</th>
    <th>Faithfulness</th>
    <th>Completeness</th>
    <th>Conciseness</th>
    <th>Domain*</th>
  </tr>
  <tr>
    <td rowspan="3">Proprietary LLMs</td>
    <td>GPT-4o</td>
    <td>86.10 (5)</td>
    <td>50.02 (4)</td>
    <td>77.98 (6)</td>
    <td><b>89.28 (1)</b></td>
    <td>78.42 (5)</td>
    <td>41.19 (3)</td>
    <td>74.98 (5)</td>
    <td><b>86.96 (4)</b></td>
    <td>93.02 (4)</td>
  </tr>
  <tr>
    <td>Claude 3.5 <i>Somerset</i></td>
    <td><b>89.42 (1)</b></td>
    <td>52.67 (2)</td>
    <td>80.09 (4)</td>
    <td><b>88.87 (2)</b></td>
    <td><b>82.68 (1)</b></td>
    <td>49.6 (2)</td>
    <td>78.07 (3)</td>
    <td><b>87.53 (3)</b></td>
    <td>96.3 (2)</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td>
    <td>86.18 (4)</td>
    <td>54.81 (1)</td>
    <td>81.88 (2)</td>
    <td>87.79 (4)</td>
    <td>80.26 (3)</td>
    <td>53.55 (1)</td>
    <td>79.43 (2)</td>
    <td>87.55 (2)</td>
    <td>97.16 (1)</td>
  </tr>
  <tr>
    <td>Average</td>
    <td></td>
    <td>87.23 (3.33)</td>
    <td>52.5 (2.33)</td>
    <td>79.98 (4)</td>
    <td>88.65 (2.33)</td>
    <td>80.45 (3)</td>
    <td>48.11 (2)</td>
    <td>77.49 (3.33)</td>
    <td>87.35 (3)</td>
    <td>95.5 (2.33)</td>
  </tr>
  <tr>
    <td rowspan="3">Open-source LLMs</td>
    <td>Gemma 2 <i>zhu</i></td>
    <td>87.02 (3)</td>
    <td>37.06 (6)</td>
    <td>73.67 (7)</td>
    <td>84.55 (6)</td>
    <td>78.53 (4)</td>
    <td>32.18 (5)</td>
    <td>69.69 (6)</td>
    <td>82.95 (6)</td>
    <td>93.47 (3)</td>
  </tr>
  <tr>
    <td>Llama 3.1 <i>zh</i></td>
    <td>83.50 (6)</td>
    <td>39.19 (5)</td>
    <td>81.31 (3)</td>
    <td>87.08 (5)</td>
    <td>77.10 (6)</td>
    <td>24.78 (6)</td>
    <td>75.13 (4)</td>
    <td>88.06 (1)</td>
    <td>88.41 (6)</td>
  </tr>
  <tr>
    <td>Qwen 2.5 <i>zh</i></td>
    <td><b>87.95 (2)</b></td>
    <td>50.87 (3)</td>
    <td>84.87 (1)</td>
    <td><b>88.57 (3)</b></td>
    <td>80.29 (2)</td>
    <td>40.94 (4)</td>
    <td><b>79.51 (1)</b></td>
    <td>85.81 (5)</td>
    <td>92.09 (5)</td>
  </tr>
  <tr>
    <td>Average</td>
    <td></td>
    <td>86.16 (3.67)</td>
    <td>42.37 (4.67)</td>
    <td>79.95 (3.67)</td>
    <td>86.73 (4.67)</td>
    <td>78.64 (4)</td>
    <td>32.63 (5)</td>
    <td>74.77 (3.67)</td>
    <td>85.61 (4)</td>
    <td>91.32 (4.67)</td>
  </tr>
  <tr>
    <td rowspan="2">Non-LLMs</td>
    <td>mT5</td>
    <td>12.33 (8)</td>
    <td>3.79 (8)</td>
    <td>28.67 (8)</td>
    <td>55.85 (8)</td>
    <td>25.33 (7)</td>
    <td>2.86 (7)</td>
    <td>27.67 (7)</td>
    <td>48.76 (7)</td>
    <td>82.75 (7)</td>
  </tr>
  <tr>
    <td>BART</td>
    <td>77.57 (7)</td>
    <td>25.01 (7)</td>
    <td><b>78.48 (5)</b></td>
    <td>79.2 (7)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Average</td>
    <td></td>
    <td>44.95 (7.5)</td>
    <td>14.4 (7.5)</td>
    <td>53.58 (6.5)</td>
    <td>67.53 (7.5)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 5: Summarization performance of eight summarizers for two languages. Rankings are shown in parentheses, with cell color intensity increasing within each column to indicate higher ranks. Domain*: domain stability.

4.2.1 Domain-specific Key-Fact Extraction Quality Check

Unlike UniSumEval’s domain-agnostic key-fact extraction, MSunBench employs domain-specific categories (Table 2) to better reflect each domain’s unique characteristics. For comparative evaluation, the two expert examiners perform A/B comparisons between key-facts extracted from 150 English source documents using both approaches.

Table 4 shows that domain-specific key-facts are highly preferred by the examiners. MSunBench’s approach yields a 50%p higher preference rate than UniSumEval (Cohen’s \( \kappa = 0.48^3 \)). It suggests that <b>domain-specific key-facts better capture what humans consider essential in each domain</b>, leading to more targeted summary evaluation. Appendix D presents detailed A/B test guidelines and domain-wise results.

4.2.2 Multi-Agent Debate Quality Check

Unlike UniSumEval, which offers a single viewpoint, our debating system mitigates bias by presenting contrasting perspectives. To verify this, we ask the two expert examiners to re-annotate representative samples of summary sentences\(^4\) from both datasets for sentence-level fact verification. The two examiners engage in repeated discussions to reach a consensus on their decisions, ensuring reliable ground-truth labels.

Table 4 shows that our debating system produces more accurate human labels, achieving a 12.76%p higher balanced accuracy compared to UniSumEval. Among incorrect annotations in UniSumEval, 95.31% align with the single-view assistance label, indicating that the majority of such errors stem from annotators blindly endorsing LLM decisions. This reveals that a single-view approach inflates IAA scores but harms accuracy, while a multi-view approach can mitigate this by promoting accurate annotation.

5 Benchmarking Summarizers

Dimension and Metric  We evaluate summarization performance across five key dimensions: <i>faithfulness</i>, <i>conciseness</i>, <i>completeness</i>, <i>domain stability</i>, and <i>language stability</i>. The first three are based on a well-established work (Lee et al., 2024), and are evaluated as <i>percentage</i> scores as suggested by the original paper (Song et al., 2024).

Domain and language stability assess the consistency of summarization performance across domains and languages. To assess domain stability, we first compute the coefficient of variation (CV) across domains for each of the first three dimensions-faithfulness, completeness, and conciseness. These three CV values are then averaged to obtain a composite domain stability score. Language stability is assessed similarly, with CVs calculated across languages rather than domains. Detailed calculations are in Appendix E.

5.1 Overview

Table 5 compares the performance of eight summarizers across English and Chinese. Proprietary LLMs demonstrate superior performance compared to open-source and non-LLMs across languages, particularly in completeness. They maintain the consistent essential information coverage across languages (52.5 to 48.11), while open-

\(^3\) Cohen’s kappa (\( \kappa \)) is a statistical measure for inter-rater reliability for two annotators (McHugh, 2012).
\(^4\) We sample 624 sentences each from UniSumEval (8,133 sentences) and MSunBench (9,951 sentences), achieving 99% confidence with ±5% margin of error.
source LLMs experience substantial degradation (42.37 to 32.63). This reveals that proprietary LLMs capture domain characteristics more effectively in both languages, whereas open-source and non-LLMs do not. Nevertheless, performance gaps persist across languages regardless of the summarizer type, calling for further efforts to enhance multilingual performance consistency.

5.2 Detailed Analysis

We conduct a detailed analysis of the generated summaries, examining completeness and conciseness through key-fact category coverage, and faithfulness through factuality error types.

5.2.1 Comparison on Key-Fact Coverage

Figure 3 shows the ratio of key-fact categories covered in the generated summaries. For a detailed analysis, we examine two domains (News and Meeting) in English and Chinese, highlighting coverage differences across three model types.

Generally, in English (Figure 3(a)), proprietary and open-source LLMs exhibit relatively consistent key-fact coverage across categories in each domain. However, in Chinese (Figure 3(b)), both show greater variability, such as a large gap between MT and CA in News and OP and PR in Meeting. This highlights that there is significant category-wise imbalance in key-fact retention in Chinese, which suggests that similar issues may arise in other non-major languages as well.

From a domain perspective, in general, News exhibits higher coverage rates than Meeting in both languages, though the dominance of key-fact categories in the same domain vary across languages. This reveals that all models, regardless of type, still lack satisfactory consistency in capturing key-facts across languages and domains, which directly impacts the completeness and conciseness of the generated summary.

![Bar charts showing key-fact category coverage ratio in two domains for English and Chinese.](page_186_120_1077_246.png)

Figure 3: Key-fact category coverage ratio in two domains for English and Chinese. Abbreviations on the x-axis refers: MT(Main topic), BG(Background), IP(Immediate impact), FP(Future implications), PS(Public statements), OS(Official statements), CA(Counterarguments), OP(Opinions), DC(Decisions), PR(Reports), FI(Factual Info).

![Bar charts showing distribution of error types across summarizers and languages.](page_186_496_1077_246.png)

Figure 4: Distribution of error types across summarizers and languages. A detailed description of each error type is provided in Appendix C.

5.2.2 Analysis on Factuality Error

We analyze the distribution of factuality errors appearing in the summaries. Figure 4 shows the distribution across different summarizers for two languages. There is no significant difference in error distribution between proprietary and open-source LLMs, nor across languages. The consistency across languages indicates that faithfulness in summarization is an inherent property of LLMs rather than a language-dependent factor. Thus, enhancing summarization fidelity is likely to generalize across both models and languages.

6 Benchmark: LLMs as Evaluators

Benchmarking LLMs as summarization evaluators is crucial, as it enables scalable and consistent automated assessment, reducing reliance on costly, time-consuming human evaluation. Their ability to provide objective and reproducible evaluations makes them valuable for benchmarking summarization models across different domains and languages. To assess their effectiveness, we analyze three key aspects:

• Accuracy of LLMs as evaluators based on their agreement with human judgments.
<table>
  <tr>
    <th rowspan="2">Model Type</th>
    <th rowspan="2">Model Name</th>
    <th colspan="4">English</th>
    <th colspan="4">Chinese</th>
    <th rowspan="2">Language Stability</th>
  </tr>
  <tr>
    <th>Faithfulness</th>
    <th>Completeness</th>
    <th>Conciseness</th>
    <th>Domain*</th>
    <th>Faithfulness</th>
    <th>Completeness</th>
    <th>Conciseness</th>
    <th>Domain*</th>
  </tr>
  <tr>
    <td rowspan="2">Proprietary LLMs</td>
    <td>GPT-4o</td>
    <td>0.61</td>
    <td><b>0.74</b></td>
    <td>0.60</td>
    <td><b>88.31</b></td>
    <td>0.52</td>
    <td>0.71</td>
    <td><b>0.65</b></td>
    <td><b>85.76</b></td>
    <td><b>94.91</b></td>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>0.60</td>
    <td>0.72</td>
    <td><b>0.62</b></td>
    <td>87.38</td>
    <td>0.54</td>
    <td>0.65</td>
    <td>0.73</td>
    <td>80.27</td>
    <td>90.09</td>
  </tr>
  <tr>
    <td rowspan="2">Open-source LLMs</td>
    <td>Llama 3.1 70B</td>
    <td>0.55</td>
    <td>0.70</td>
    <td>0.57</td>
    <td>85.64</td>
    <td>0.41</td>
    <td>0.65</td>
    <td>0.46</td>
    <td>75.44</td>
    <td>93.39</td>
  </tr>
  <tr>
    <td>Qwen 2.5-72B</td>
    <td><b>0.67</b></td>
    <td>0.65</td>
    <td>0.55</td>
    <td>86.96</td>
    <td><b>0.59</b></td>
    <td><b>0.76</b></td>
    <td>0.55</td>
    <td>81.11</td>
    <td>88.40</td>
  </tr>
</table>

Table 6: Benchmarking LLMs as evaluators in English and Chinese. The higher score, the better accuracy in summary evaluation. Domain*: Domain Stability. Stability scores are computed as in the summarization evaluation.

![Analysis on cross-task correlation between summarization and evaluation.](page_186_370_563_312.png)

Figure 5: Analysis on cross-task correlation between summarization and evaluation. *: p-value < 0.05.

• Correlation between LLMs’ summarization performance and evaluation performance.

• Self-evaluation bias, which occurs when an LLM assess summaries it has generated.

Selected Models We compare two proprietary LLMs (Claude-3.5-Sonnet, GPT-4o) and two open-source LLMs (Qwen-2.5-72B and Llama-3.1-70B), which exhibits considerably different summarization performance within their respective groups.

Evaluation Metric We compare the score obtained by LLM-based evaluation with human annotated labels in MSumBench. We report the Pearson correlation for the summary-level percentage scores across evaluation dimensions. See Appendix E for the detail of measurements.

6.1 Correlation with Human Judgments

Table 6 shows the correlation between LLM and human evaluation across faithfulness, completeness, and conciseness, along with domain and language stability, reflecting correlation consistency across six domains and two languages.

GPT-4o demonstrates the highest domain and language stability. However, no single model consistently achieves the best performance across all dimensions. Notably, Qwen-2.5-72B stands out for its strong performance in faithfulness evaluation. Therefore, there is a significant variability in LLM evaluation accuracy across evaluation dimensions, domains, and languages, indicating that relying on a single model for automated evaluation lacks reliability.

![Self-evaluation bias rates across models and languages.](page_1012_370_312_312.png)

Figure 6: Self-evaluation bias rates across models and languages. Colored bars denote statistically significant differences determined by t-test (p < 0.05).

6.2 Cross-Task Correlation

Figure 5 shows the correlation between summarization and evaluation performance5. The x-axis denotes summarization performance, while the y-axis represents its performance as an evaluator.

We observe a strong correlation \( \rho = 0.71 \) between LLM performance in summarization and evaluation tasks, showing that strong summarizers generally excel at evaluation. This suggests that improving an LLM’s summarization can also enhance its evaluation, and vice versa. However, as noted in Section 6.1, even the best summarizers may still show limitations when evaluating certain dimensions or working with different languages.

6.3 Self-Evaluation Bias

In summary evaluation, LLMs are expected to favor their own summaries (Wataoka et al., 2024), but this bias has not been fully analyzed. Thus, we assess to what extent this bias exists. We first define the self-evaluation bias rate as the difference between an LLM’s evaluation score for its own summaries and the average score assigned by other evaluators (see Appendix F for detailed calculation). Thus, the higher (or lower) the rate, the more favorably (or unfavorably) it evaluates its own summaries. We use a t-test to check if the difference is statistically significant.

Figure 6 shows the self-evaluation bias rate across LLMs and languages. It reveals that self-evaluation bias does not always manifest as self-preference, as it can manifest as either over-

5Performance here refers to the composite score defined by the average of the three dimensions-faithfulness, completeness, and conciseness.
rating or under-rating their own work. Specifically, Llama-3.1-70B rates its own summaries less favorably than others, while Claude-3.5-Sonnet favors its own. This indicates variability in self-assessment across models.

7 Conclusion

We create MSumBench, a multi-aspect benchmark for summary evaluation across two languages and six distinct domains. MSumBench extracts key-facts to precisely measure summary-context alignment across six specialized domains. To ensure accurate annotations, we introduce a multi-agent assistant that minimizes human errors and delivers high-quality labels. We conduct an in-depth analysis, providing insights into LLMs’ behavior as both summarizers and automated evaluators. The open dataset supports evaluation, enhances summarization, and aids preference optimization.

Limitations

Our work has several limitations as follows:

First, while we incorporate domain-specific key-fact categories, they are derived primarily through statistical patterns. The framework could benefit from direct input by domain experts to capture complex, specialized details more accurately.

Second, while our evaluation framework accounts for differences in importance across domains, it does not address variations in summary purpose within a single domain. For instance, in medical literature, the focus of a summary could shift from clinical treatments to epidemiological data, depending on summarization perspectives. Future work could explore purpose-specific key-fact categories, aligning summaries more closely with varied summarization objectives.

Third, our bilingual focus (English and Chinese) is an improvement over monolingual benchmarks. However, it still excludes many underrepresented languages. Evaluating how effectively LLMs handle other lower-resource languages remains an open question. Lastly, while our multi-agent debate framework provides balanced arguments to assist annotators, exploring other forms of collaboration, such as enhancing debates with dynamic rebuttals, could further improve the effectiveness of AI-human collaboration for summary quality annotation. Despite the challenges, we believe that our work serves as a meaningful foundation for future research, particularly in developing more robust automated evaluators and improving the factual consistency of domain-specific summarization systems across different languages.

Ethics Statement

Throughout our study, we prioritized comprehensive communication with all participating annotators, including both crowd-sourced and expert annotators. Crowd-sourced annotators received compensation above the U.S. federal minimum wage rate, while expert examiners were compensated at rates exceeding $30 per hour, with performance-based incentives. To ensure privacy compliance, all annotator personal information is anonymized in our dataset.

Scientific Artifacts

Our proposed benchmark combines publicly available datasets. For summary generation, we used Huggingface checkpoints and commercial APIs such as OpenAI and AWS Bedrock. Summary model details are in Table 10.

Acknowledgments

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-00445087, Enhancing AI Model Reliability Through Domain-Specific Automated Value Alignment Assessment). Additionally, this work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00334343).

References

Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2023. Multilingual summarization with factual consistency evaluation. In ACL.

Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In EMNLP.

Alexander Braylan, Omar Alonso, and Matthew Lease. 2022. Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks. In WWW.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu.
2024. Chateval: Towards better llm-based evaluators through multi-agent debate. In ICLR.

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In NAACL.

Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate. In ICML.

Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. TACL, 9.

Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved qa-based factual consistency evaluation for summarization. In NAACL.

Jessica Zosa Forde, Ruochen Zhang, Lintang Sutawika Alham Fikri Aji, Samuel Cahyawijaya, Genta Indra Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, and Ellie Pavlick. 2024. Re-evaluating evaluation for multilingual summarization. In EMNLP.

Mingqi Gao and Xiaojun Wan. 2022. Dialsummeval: Revisiting summarization evaluation for dialogues. In NAACL.

Google Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Ajesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and Amy et al. Yang. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yebowen Hu, Timothy Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. 2023. Meetingbank: A benchmark dataset for meeting summarization. In ACL.

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In ACL.

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. In ICML.

Mahnaz Koupaee, Jake W Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, et al. 2025. Faithful, unfaithful or ambiguous? multi-agent debate with initial stance for summary evaluation. In NAACL.

Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability.

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In EACL.

Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander Richard Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring llm ability at factual reasoning through the lens of summarization. In EMNLP.

Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022. SummaC: Re-visiting nli-based models for inconsistency detection in summarization. TACL, 10.

Yuho Lee, Taewon Yun, Jason Cai, Hang Su, and Hwanjun Song. 2024. UniSumEval: Towards unified, fine-grained, multi-dimensional summarization evaluation for LLMs. In EMNLP.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In EMNLP.

Mary L McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia Medica, 22(3):276–282.

Aiswarya Raj Munappy, Jan Bosch, Helena Holmström Olsson, Anders Arpteg, and Björn Brinne. 2022. Data management for production quality deep learning models: Challenges and solutions. Journal of Systems and Software, page 111359.

Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. CoNLL).

OpenAI. 2024. Gpt-4o system card.

Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. In NAACL.

Partha Pratim Ray. 2023. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems.
Jiaming Shen, Tianqi Liu, Jialu Liu, Zhen Qin, Jay Pavagadhi, Simon Baumgartner, and Michael Bendersky. 2024. Multilingual fine-grained news headline hallucination detection. In EMNLP.

Hwanjun Song, Igor Shalyminov, Hang Su, Singh Siffi, Kaisheng Yao, and Saab Mansour. 2023. Enhancing abstractiveness of summarization models through calibrated distillation. In EMNLP.

Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. FineSurE: Fine-grained summarization evaluation using llms. In ACL.

Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryściński, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In ACL.

Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. Minicheck: Efficient fact-checking of llms on grounding documents. arXiv preprint arXiv:2404.10774.

Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, et al. 2024b. TofuEval: Evaluating hallucinations of llms on topic-focused dialogue summarization. In NAACL.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseidot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

David Wan, Koustuv Sinha, Srini Iyer, Asli Celikyilmaz, Mohit Bansal, and Ramakanth Pasunuru. 2024. ACUEval: Fine-grained hallucination evaluation and correction for abstractive summarization. In ACL.

Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in Ilm-as-a-judge. arXiv preprint arXiv:2410.21819.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In NAACL.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In NeurIPS.

Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In NLP4ConvAI.

Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. Alignscore: Evaluating factual consistency with a unified alignment function. In ACL.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with bert. In ICLR.

Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. Mediasum: A large-scale media interview dataset for dialogue summarization. In NAACL.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Domain</th>
    <th colspan="3">English</th>
    <th colspan="3">Chinese</th>
    <th rowspan="2">Key-fact Count</th>
  </tr>
  <tr>
    <th>Text Word count (Min - Max)</th>
    <th>Summary Word count (Min - Max)</th>
    <th>Text Word count (Min - Max)</th>
    <th>Summary Word count (Min - Max)</th>
    <th>Key-fact Count (Min - Max)</th>
  </tr>
  <tr>
    <td>CNN/DM</td>
    <td>News</td>
    <td>503.6 (234 – 962)</td>
    <td>86.01 (10 – 205)</td>
    <td>885.4 (447 – 1,657)</td>
    <td>146.52 (13 – 285)</td>
    <td>15.72 (6 – 26)</td>
  </tr>
  <tr>
    <td>PubMed</td>
    <td>Medical Literature</td>
    <td>2360 (856 – 4496)</td>
    <td>60.07 (14 – 113)</td>
    <td>3,797.52 (1,272 – 7,862)</td>
    <td>221.90 (12 – 591)</td>
    <td>23.84 (10 – 53)</td>
  </tr>
  <tr>
    <td>GuyReport</td>
    <td>Report</td>
    <td>3401 (1345 – 6837)</td>
    <td>126.66 (7 – 347)</td>
    <td>6,223.12 (2,466 – 10,359)</td>
    <td>261.71 (10 – 1,704)</td>
    <td>25.08 (16 – 35)</td>
  </tr>
  <tr>
    <td>MultiWOZ</td>
    <td>Booking</td>
    <td>243.08 (138 – 382)</td>
    <td>107.56 (8 – 260)</td>
    <td>465.56 (258 – 788)</td>
    <td>120.19 (12 – 250)</td>
    <td>11.52 (6 – 19)</td>
  </tr>
  <tr>
    <td>MeetingBank</td>
    <td>Meeting</td>
    <td>547.44 (97 – 1276)</td>
    <td>71.26 (9 – 143)</td>
    <td>925.6 (351 – 1,991)</td>
    <td>135.49 (11 – 412)</td>
    <td>12.64 (5 – 30)</td>
  </tr>
  <tr>
    <td>MediaSum</td>
    <td>Interview</td>
    <td>1108.64 (186 – 4082)</td>
    <td>87.15 (9 – 196)</td>
    <td>1,951.2 (396 – 7,895)</td>
    <td>162.47 (15 – 585)</td>
    <td>16.72 (6 – 27)</td>
  </tr>
  <tr>
    <td colspan="2">MSumBench</td>
    <td>1,360 (138 – 6,837)</td>
    <td>2374.73 (258-10,359)</td>
    <td>89.89 (7 – 347)</td>
    <td>174.71 (10 – 1,704)</td>
    <td>17.59 (5 – 53)</td>
  </tr>
</table>

Table 7: Overview of six datasets utilized in MSumBench: includes statistics on source documents and their summaries, showing mean word counts and key-fact quantities, along with their respective ranges (min to max values).

A Summary of the Source datasets

Table 7 presents a comprehensive analysis of six diverse datasets encompassing various domains, including news, medical literature, reports, booking, meetings, and interviews in both English and Chinese languages. These datasets are strategically selected to evaluate the model’s capability in handling domain-specific contextual understanding. The evaluation benchmark consists of 150 source documents, evenly distributed with 25 documents per domain in each language.

B Dataset Construction Pipeline Detail

<table>
  <tr>
    <th>Domain</th>
    <th>Retained Key-Facts</th>
    <th>Newly Added Key-Facts</th>
    <th>Removed Key-Facts</th>
  </tr>
  <tr>
    <td>News</td>
    <td>63.6 %</td>
    <td>36.4 %</td>
    <td>14.0 %</td>
  </tr>
  <tr>
    <td>Medical Lit.</td>
    <td>58.1 %</td>
    <td>41.9 %</td>
    <td>24.9 %</td>
  </tr>
  <tr>
    <td>Report</td>
    <td>60.8 %</td>
    <td>39.2 %</td>
    <td>23.1 %</td>
  </tr>
  <tr>
    <td>Booking</td>
    <td>79.2 %</td>
    <td>20.8 %</td>
    <td>4.8 %</td>
  </tr>
  <tr>
    <td>Meeting</td>
    <td>72.2 %</td>
    <td>27.8 %</td>
    <td>17.4 %</td>
  </tr>
  <tr>
    <td>Interview</td>
    <td>70.6 %</td>
    <td>29.4 %</td>
    <td>16.7 %</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>67.4 %</td>
    <td>32.6 %</td>
    <td>16.8 %</td>
  </tr>
</table>

Table 8: Key-fact comparison between domain-specific method (A) and generic methods (B): Retained Key-facts represent the percentage of key-facts that remain consistent between A and B. Newly Added Key-facts indicate the percentage of key-facts introduced in A, but absent in B. Removed Key-facts show the percentage of key-facts present in B but missing in A.

B.1 Translation Detail

We present implementation details for the translation of source documents. For English-Chinese translation of source documents and key-facts, we incorporate domain-specific writing styles into our prompts, as shown in Tables 12 - 15. The key-fact translation prompt used for the key-fact alignment task with auto-evaluators is presented in Table 16. To ensure translation quality, we developed a screening prompt detailed in Table 17, which evaluates translations sentence-by-sentence across multiple dimensions: accuracy, consistency, fluency, comprehensibility, cultural adaptation, formality, and adequacy.

B.2 Key-Fact Generation

Key-Fact Extraction and Validation Table 20 presents the domain-specific key-facts generation prompt, including well-defined categories and detailed domain-specific descriptions, along with representative examples illustrating granularity. The categories for each domain are detailed in Table 18, with corresponding examples in Table 19. To ensure the quality of extracted key-facts, we perform key-fact validation using the prompt in Table 21.

Key-Fact Extractiveness We provide additional analysis on the extractiveness of key-facts, in relation to their source documents. Since key-facts are designed to function as atomic information units, analyzing their extractiveness helps characterize their structural properties. We measure extractiveness using three metrics. First, direct copy measures the percentage of key-facts that exactly match a sentence in the source document. Second, near-exact match captures the proportion of key-facts
<table>
  <tr>
    <th>Model Name</th>
    <th>Hugging Face Checkpoints & Official API Version</th>
    <th>Hardware & Precision</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>gpt-4o-2024-08-06</td>
    <td>API (OpenAI, default settings)</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>anthropic.claude-3.5-sonnet-20241022-v2:0</td>
    <td>API (AWS Bedrock, default settings)</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>gemini-1.5-pro-002</td>
    <td>API (Google API, default settings)</td>
  </tr>
  <tr>
    <td>Gemma-2-27B</td>
    <td>google/gemma-2-27b-it</td>
    <td>NVIDIA L40S 48GB (×2) & BF16</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>meta-llama/Llama-3-1-70B-Instruct</td>
    <td>NVIDIA L40S 48GB (×4) & BF16</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>Qwen/Qwen2.5-72B-Instruct</td>
    <td>NVIDIA L40S 48GB (×4) & BF16</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td>sebuetnlp/mT5_multilingual_XLSum</td>
    <td>NVIDIA L40S 48GB (×1) & Full precision</td>
  </tr>
  <tr>
    <td>BART</td>
    <td>facebook/bart-large-cnn linydub/bart-large-samsum</td>
    <td>NVIDIA L40S 48GB (×1) & Full precision</td>
  </tr>
</table>

Table 10: The checkpoint of the model used to generate summaries.

<table>
  <tr>
    <th>Domain</th>
    <th>Direct Copy(%)</th>
    <th>Near-Exact Match(%)</th>
    <th>Extractiveness Score(%)</th>
  </tr>
  <tr>
    <td>News</td>
    <td>4.68%</td>
    <td>16.31%</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Medical Lit.</td>
    <td>0.19%</td>
    <td>18.10%</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>Report</td>
    <td>5.68%</td>
    <td>23.52%</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Booking</td>
    <td>4.96%</td>
    <td>3.34%</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Meeting</td>
    <td>4.67%</td>
    <td>13.88%</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Interview</td>
    <td>3.08%</td>
    <td>9.33%</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>Overall</td>
    <td>3.88%</td>
    <td>14.08%</td>
    <td>0.60</td>
  </tr>
</table>

Table 11: The measurements of extractiveness of key-fact across domains in three ways: direct copy, near-exact match and extractiveness score.

with over 90% similarity to the source, as measured by the Edit Distance algorithm. Such cases typically involve surface-level edits, including whitespace changes or removal of redundant adjectives. Third, we compute the extractiveness score using the metric proposed by Song et al. (2024), defined as the average n-gram overlap (\( n = 1/3/5 \)) between each key-fact and the source document.

Table 11 shows the extractiveness patterns across domains. Direct verbatim copying is relatively rare, with an average rate of 3.88%. Near-exact matches average 14.08%, indicating that most key-facts undergo some degree of reformulation from the original text. The extractiveness score averages 0.60, showing moderate lexical overlap with source documents, consistent with the interpretation by Song et al. (2023). These results indicate that key-facts are typically reformulated to some extent, rather than directly copied, reflecting their role as processed, atomic information units suitable for systematic evaluation.

B.3 Summary Generation Detail

Table 10 details the model versions employed for summary generation. For non-LLMs, we utilized pre-trained models from Hugging Face, while for open-source LLMs, we implemented instruction-tuned checkpoints. Access to proprietary LLMs was facilitated through their respective official APIs. The prompts for both English and Chinese summary generation are detailed in Table 22, with the temperature parameter set to 1.0 to promote diversity in generated summaries.

B.4 Key-Fact Alignment Detail

Table 23 presents our streamlined NLI prompt for key-fact alignment assessment. The prompt instructs the model to perform a direct entailment check between summary sentences and key-facts, providing brief, focused explanations. This design enables quick and confident assessment of information alignment while minimizing unnecessary complexity.

B.5 Multi-Agent Debate System Detail

The prompts used for our multi-agent fact verification system are presented in Tables 24 - 26. Each agent’s prompt is tailored to their specific role: the Advocate focuses on finding supporting evidence, the Skeptic emphasizes identifying potential contradictions or flaws, and the Adjudicator concentrates on analyzing both arguments against the source document for final verification.

C Faithfulness Error Types

We use error types from Lee et al. (2024):

• Out-of-article Error: Introduces unverifiable facts, subjective opinions, or biases not supported by the source document.

• Entity Error: Misrepresents or includes incorrect entities, such as numbers or main subjects.

• Relation Error: Distorts the intended semantic relationships through incorrect use of verbs, prepositions, or adjectives.

• Sentence Error: Contradicts the source document, requiring major revisions or removal for factual alignment.
D  Key-Fact Quality Assessment

Table 8 illustrates a quantitative comparison between our domain-specific extraction methodology with the domain-agnostic approach from Lee et al. (2024). The results indicate that our domain-specific extraction methodology encompasses a 32.61% more in domain-relevant key-fact identification relative to the baseline. Notably, the domain-specific strategy systematically excludes 16.83% of key-facts identified by the domain-agnostic method, as these elements lie beyond the scope of domain-specific criteria. This selective filtering highlights our methodology’s precision in capturing domain-relevant information.

Additionally, we qualitatively assess the effectiveness of our domain-specific methodology compared to the domain-agnostic baseline via A/B preference testing with two expert annotators. We provide them with the English source document and two sets of key-facts: one extracted by the domain-agnostic baseline and the other identified by our domain-specific methodology. To minimize presentation bias, we present the two sets in randomized order, and the annotators are blind to the source of each set. They independently judge which set is more useful for summarization and better captures the essential content of the source document. To assess the consistency of these subjective judgments, we compute IAA using Cohen’s kappa. Table 9 presents the comprehensive results of the A/B preference test and Cohen’s kappa. Across all domains, MSunBench is consistently preferred by annotators, with preference rates ranging from 62% to 90.0%. Notably, Medical Literature shows a higher preference ratio (90.0%) compared to Meeting (62.0%), demonstrating the necessity of domain-specialized extraction in fields where domain expertise significantly influences content understanding.

E  Benchmark Evaluation Formula

E.1  Evaluation Metrics for Summarization

We evaluate performance using five key metrics: faithfulness, completeness, conciseness, domain stability, and language stability. The first three metrics follow the methodology established by Song et al. (2024).

Faithfulness  We consider a source document \( D \), and its generated summary \( S \) containing \( N \) sentences \( \{s_1, s_2, ..., s_N\} \). Based on human annotation, we identify \( S_{Fact} \subseteq S \), which is the subset of \( S \) containing only factually accurate sentences. The faithfulness score is calculated as the ratio of factually correct sentences to total sentences:

\[
\text{Faithfulness}(D, S) = \frac{|S_{Fact}|}{|S|}
\]

Completeness and Conciseness  Let \( K \) be the set of key-facts \( \{k_1, k_2, ..., k_M\} \) from the source document. Based on the results of key-fact alignment, we can define bipartite graph \( G = (K, S, E) \), where \( E \) represents edges between key-facts \( K \) and summary sentences \( S, \{(k, s) : k \rightarrow s | k \in K \land s \in S\} \) with \( k \rightarrow s \) signifying that key-fact \( k \) is entailed in \( s \).

The completeness score measures how many key facts are captured in the summary:

\[
\text{Completeness}(K, S) = \frac{|\{k|(k, s) \in E\}|}{|K|}
\]

The conciseness score measures how efficiently the summary sentences convey key facts:

\[
\text{Conciseness}(K, S) = \frac{|\{s|(k, s) \in E\}|}{|S|}
\]

Domain Stability  We introduce an improved approach to measure domain stability score, addressing limitations in Lee et al. (2024). While the previous approach only considered the gap between the highest and lowest scores, our method accounts for overall performance variability. To achieve this, we use the coefficient of variation (CV), which provides a normalized measure of dispersion by comparing the standard deviation to the mean. We calculate CV for three evaluation dimensions respectively across domains as follows below formulation.

For a given performance score across domains \( S_E = \{S_{E,1}, S_{E,2}, ..., S_{E,d}\} \), where \( d = 1, ... 6 \) denotes the domain index and \( E \) refers faithfulness, completeness, and conciseness. We calculate the Instability score for each evaluation dimension \( S_E \):

\[
\text{Instability}(S_E) = \frac{\sigma_{S_E}}{\mu_{S_E}} \times 100,
\]

where \( \sigma_{S_E} \) is the standard deviation and \( \mu_{S_E} \) is the mean of the scores for dimension \( E \) across domain. To formulate Domain Stability, we rescaled the Instability(\( S_E \)) to ensure the stability measure remains within a meaningful range. Specifically, we define *Domain Stability* as:

\[
\text{Domain Stability}(S_E) = \frac{100}{1 + \text{Instability}(S_E)}
\]
For the composite score, we average the Domain Stability of the three evaluation dimensions.

Language Stability  Language stability evaluates how consistently a model performs across different languages. We apply the same mathematical framework as Domain Stability, using the coefficient of variation (CV) and rescaling approach. For each evaluation dimension \( E \), we first compute \( S_{E,L} \), the average score across domains for each language \( L \):

\[
S_{E,L} = \frac{1}{D} \sum_{d=1}^{D} S_{E,L,d},
\]

where \( S_{E,L,d} \) represents the score for evaluation dimension \( E \) in language \( L \) and domain \( d \), and \( D \) is the total number of domains. Then we calculate the instability score with CV, which captures the performance fluctuation across different languages:

\[
\text{Instability}(S_E) = \frac{\sigma_{S_E}}{\mu_{S_E}} \times 100,
\]

where \( \sigma_{S_E} \) is the standard deviation and \( \mu_{S_E} \) is the mean of the scores for dimension \( E \) across languages.

Similar to Domain Stability, we transform the Instability score to ensure that the stability measure remains within a meaningful range:

\[
\text{Language Stability}(S_E) = \frac{100}{1 + \text{Instability}(S_E)}
\]

The composite score is calculated by averaging the Language Stability across three evaluation dimensions. Similar to domain stability, it reflects the model’s performance variations across different languages.

E.2 Evaluation Metrics for Automatic Evaluator

Pearson Correlation  We compute summary-level correlations using the Pearson correlation coefficient to assess the alignment between automated and human evaluation, following recent work (Liu et al., 2023; Song et al., 2024; Lee et al., 2024). For each summary \( i \), we analyze the correlation between automated evaluation scores (\( x_i \)) and human evaluation scores (\( y_i \)). The summary-level correlation \( \rho \) is calculated as:

\[
\rho = Cor([x_1, x_2, ..., x_n], [y_1, y_2, ..., y_n])
\]

where \( Cor \) represents the Pearson correlation coefficient, where \( n \) is the total number of summaries.

F Self-Evaluation Bias Detail

To investigate self-evaluation bias in summary evaluation, we employ four large language models (GPT-4o, Claude-3.5-Sonnet, Llama-3.1-70B, and Qwen-2.5-72B) as both evaluators and summarizers. We measure self-evaluation bias by comparing how each model rates its own summaries versus those generated by other models.

First, we calculate composite scores by averaging three evaluation dimensions (faithfulness, completeness, and conciseness). Second, each model’s self-evaluation score is obtained by averaging the composite scores of its own summaries across different domains. Similarly, peer-evaluation scores are computed by averaging the composite scores given by other models. Third, self-evaluation bias is then determined by subtracting the peer-evaluation scores from the self-evaluation scores. To assess statistical significance, we conduct the t-test comparing self-evaluation and peer-evaluation scores. Biases that are statistically significant (p < 0.05) are highlighted in the visualizations in Figure 6.

G Additional Analysis

G.1 Human Annotation Result Detail

We provide the eight summarizer performances across all six domains and languages of faithfulness, completeness, and conciseness in Tables 27 - 29. We present additional domain-level findings.

Faithfulness Score  Table 27 shows proprietary LLMs outperform open-source and non-LLMs across domains and languages. Specifically, Claude-3.5-Sonnet achieves optimal performance stability among proprietary LLMs across all domains and languages. A marginal decline in faithfulness is observed in News, Booking and Meeting domains, while an increase is noted in Report and Interview domains from English to Chinese. Therefore, further research efforts should target both language-specific enhancements and domain-language adaptation strategies.

Completeness Score  Table 28 shows proprietary LLMs’ superior performance across all domains and languages, with Gemini-1.5-Pro excelling in Report, Booking, and Interview domains. However, a decline in completeness scores is observed across all domains and models from English to Chinese, with varying degrees of intensity depending on the domain. The observed decline in completeness
scores indicates challenges in retaining essential information in Chinese, emphasizing the need for domain-aware language improvements to enhance performance.

Conciseness Score Table 29 demonstrates proprietary LLMs achieve higher conciseness scores compared to open-source and non-LLMs across domains. While the overall decline in conciseness from English to Chinese is minimal, the Meeting domain experiences a significant drop, with open-source LLMs performing the worst in this domain compared to others. This highlights the need for domain-specific refinements to improve conciseness in open-source LLMs.

G.2 Comparison with QA- and NLI-based Auto-evaluator

In Table 30, we provide the performance of automated evaluators, including QA-based (QAFactEval (Fabbri et al., 2022)), NLI-based (Align-Score (Zha et al., 2023), MiniCheck (Tang et al., 2024a)), and LLM-based (G-Eval (Liu et al., 2023), FineSurE (Song et al., 2024)) with implementations based on GPT-4o.

The QA-based and NLI-based evaluators specialize in measuring faithfulness but have limitations in assessing completeness and conciseness. Therefore, we focus on comparing their faithfulness scores across six domains. We compare the scores obtained from automatic evaluation with human-annotated labels in MSunBench. Specifically, we report Pearson correlation values for the summary-level faithfulness percentage scores. Due to the language limitations of these evaluators, we only provide results for English summaries. Among all methods, FineSurE achieves the highest domain stability (83.75) and demonstrates superior performance in four domains such as Report, Booking, Meeting, and Interview.

G.3 Detailed Performance of LLM-Based Auto-Evaluator

Table 31 compares the performance of two LLM-based automatic evaluators, G-Eval and FineSurE, across two model types, six domains, and two languages. We report the average scores of GPT-4o and Claude-3.5-Sonnet for proprietary LLMs and Llama-3.1-70B and Qwen-2.5-72B for open-source LLMs. While FineSurE, implemented with proprietary LLMs, often demonstrates strong performance across three evaluation dimensions, its effectiveness varies across domains, particularly in faithfulness evaluation. In some cases, such as the Booking domain, G-Eval (using open-source LLMs) surpasses FineSurE by a significant margin. This highlights the inherent variability in LLM-based evaluation accuracy and reinforces that no single LLM or automatic evaluator consistently outperforms others across all dimensions, domains, and languages.

G.4 Comparison with Similarity-based Metric

We analyze the summary-level agreement between human scores and conventional similarity-based metrics, including ROUGE-1/2/L (Lin, 2004) and BERTScore (Zhang et al., 2019), across three evaluation dimensions and multiple domains. We compare these results with the LLM-based method, FineSurE (Song et al., 2024), an LLM-based evaluation approach implemented using GPT-4o.

Our analysis reveals distinct performance patterns across evaluation dimensions. For faithfulness evaluation, conventional metrics exhibit weaker correlations with human judgments compared to FineSurE across all domains and languages. While completeness and conciseness exhibit stronger correlations than faithfulness, their performance varies depending on the domain.

Nevertheless, conventional metrics generally demonstrate significantly lower agreement with human scores across all dimensions compared to the LLM-based evaluator.

H Human Annotation Details

Qualifications and Compensation MTurk annotators are screened through an English proficiency test simulating the fact verification and key-fact alignment tasks. They must also demonstrate a reliable track record at MTurk with a minimum 90% approval rate and 500 accepted HITs. These crowd-sourced annotators receive compensation exceeding the U.S. minimum wage.

For manual annotation (see Section 4.2), we recruit postgraduate students with proficient English skills (above C2 level) as expert examiners. Specifically, for English-Chinese translation tasks (see Section 3) and for any manual annotation involving Chinese, we require the expert examiners to be native Chinese speakers. These experts are compensated at rates exceeding $30 per hour plus performance-based incentives.
Annotation of Chinese Summaries   To annotate Chinese summaries, the annotators are provided with summaries translated into English using GPT-4o. These translations undergo our standard process: initial LLM-based translation, validation, and final verification by native Chinese examiners, as described in Section 3.1. The annotators then work with these verified English translations.

Quality Control   In addition, since MTurk is a crowd-sourcing platform, it is essential to systematically filter unreliable answers from the annotators. For each summary annotated, we designate about 5-10% of the annotation unit in a Human Intelligence Task as attention checks, where the correct answers are already known to us. We exclude all responses that do not pass the attention checks. This approach ensures that the annotations collected from MTurk meet the required standards.
You are an expert English-Chinese translator specialized in news articles. Your task is to translate the following English text to Chinese (Simplified Chinese), sentence by sentence, with careful attention to quality and accuracy.

Warning: Use only "standard Simplified Chinese characters" and English technical terms when necessary

Translation Rules:

1. Reference Consistency
   • Keep organization names in original form
   • Translate ALL PERSON NAMES to Chinese following appropriate conventions:
      – Western names: Use standard Chinese transliteration
         * Example: Michael → 迈克尔 (Mài kè'ěr), John → 约翰 (Yuēhàn)
      – Chinese names: Maintain Chinese characters
         * Keep family name and given name format (e.g., 王小明)
      – For established figures, use their commonly known Chinese name
         * Example: Shakespeare → 莎士比亚 (Shāshìbǐyà)
   • Use standard format for dates, times, and numbers

2. Technical Terms
   • Use established Chinese technical terms
   • First mention: Chinese term (English term); Following mentions: Chinese term only
   • Maintain consistency in specialized terms throughout

3. Cultural Adaptation
   • Translate English idioms and proverbs to Chinese cultural equivalents (成语 when appropriate)
   • Convert Western business expressions to match Chinese business etiquette:
      – Use appropriate level of formality (敬语)
      – Follow Chinese business conversation conventions
   • Maintain neutrality and objectivity in expression
   • Add brief explanations for culturally specific items

4. Chinese Writing Style Consistency
   • Use formal written Chinese (书面语) consistently
   • Avoid mixing formal and colloquial expressions
   • Follow standard news writing conventions:
      – Use proper 判断词 and 状态词
      – Use standard news article punctuation
   • Word choice guidelines:
      – Prefer 因为 over 由于 for causation
      – Use 表示 instead of 说 for formal statements
      – Choose 认为 over 觉得 for opinions

Provide your answer ONLY in this simple JSON format:
{"translation":["First Chinese translation", "Second Chinese translation", "Third Chinese translation", ..., "Last Chinese translation"]}

English text: {input_text}

Table 12: Chinese translation prompt for news domain.
You are an expert English-Chinese translator specialized in medical and scientific literature. Your task is to translate the following English text to Chinese (Simplified Chinese), sentence by sentence, with careful attention to quality and accuracy.

Warning: Use only "standard Simplified Chinese characters" and English technical terms when necessary

Translation Rules:

1. Medical Terminology
   • Use standardized Chinese medical terms (规范医学用语)
   • Keep precision in medical concepts:
      – Diseases: Standard Chinese names (英文名)
      – Medications: Generic names in Chinese (英文通用名)
      – Medical procedures: Standard translations
   • Handle technical terms:
      – First mention: Chinese term (English term); Following mentions: Chinese term only
   • Maintain absolute consistency in terminology throughout
   • Abbreviations: Keep standardized format
      – First mention: Full Chinese term (Full English term [Abbreviation]) (聚合酶链式反应: Polymerase Chain Reaction [PCR])
   • Special terms:
      – Gene/protein names: Follow standard conventions
      – Chemical formulae: Maintain accuracy
      – Anatomical terms: Use standard translations

2. Academic Writing Style
   • Use formal academic Chinese (学术用语)
   • Follow scientific writing conventions: use precise and objective language and maintain the scientific tone
   • Sentence structure: Clear and concise, logical flow, one key point per sentence
   • Word choice guidelines:
      – Use 发现 for findings
      – Use 显示 for results presentation
      – Use 表明 for conclusions
      – Use 证实 for verification
      – Use 提示 for implications
      – Use 比较 for comparison
      – Use 分析 for analysis

Provide your answer ONLY in this simple JSON format:
{"translation": ["First Chinese translation", "Second Chinese translation", "Third Chinese translation", ..., "Last Chinese translation"]}

English text: {input_text}
You are an expert English-Chinese translator specialized in reports and official documents. Your task is to translate the following English text to Chinese (Simplified Chinese), sentence by sentence, with careful attention to quality and accuracy.

Warning: Use only "standard Simplified Chinese characters" and English technical terms when necessary

Translation Rules:

1. Reference Consistency
   • Keep organization names in original form
   • Translate ALL PERSON NAMES to Chinese following appropriate conventions:
      – Western names: Use standard Chinese transliteration (Michael → 迈克尔 (Mài kè’ér))
      – Chinese names: Maintain Chinese characters
   • For established figures, use their commonly known Chinese name (Shakespeare → 莎士比亚 (Shā shì biǎ yà))
   • Use standard Chinese format for:
      – Dates: YYYY年MM月DD日
      – Numbers: Use Chinese numerals for formal documents (一 二 三...)
      – Percentages: Use 百分之 format
      – Monetary values: Follow Chinese currency notation

2. Technical Terms
   • Use established Chinese technical terms
   • First mention: Chinese term (English term); Following mentions: Chinese term only
   • Maintain consistency in specialized terms throughout

3. Cultural Adaptation
   • Translate English idioms and proverbs to Chinese cultural equivalents (成语 when appropriate)
   • Convert Western business expressions to match Chinese business etiquette:
      – Use appropriate level of formality (敬语)
      – Follow Chinese business conversation conventions
   • Maintain neutrality and objectivity in expression
   • Add brief explanations for culturally specific items

4. Chinese Writing Style Consistency
   • Use the highest level formal written Chinese (政府公文体)
   • Follow official document writing conventions:
      – Use standard official vocabulary (规范用语)
      – Apply proper ceremonial words (礼仪用语)
   • Use formal written Chinese (书面语) consistently
   • Avoid mixing formal and colloquial expressions
   • Sentence structure:
      – Use parallel structure for lists (排比句)
      – Maintain appropriate formality level
      – Use standard government document punctuation

Provide your answer ONLY in this simple JSON format:
{ "translation": ["First Chinese translation", "Second Chinese translation", "Third Chinese translation", ..., "Last Chinese translation"]}

English text: {input_text}
You are an expert English-Chinese translator specialized in dialogue scripts. Your task is to translate the following English script to Chinese (Simplified Chinese), sentence by sentence, with careful attention to quality and accuracy.

Warning: Use only "standard Simplified Chinese characters" and English technical terms when necessary.

Translation Rules:

1. Reference Consistency
   • Keep organization names in original form
   • Translate ALL personal names to Chinese following appropriate conventions:
      – Western names: Use Chinese transliteration (James → 詹姆斯)
      – Chinese names: Keep Chinese characters
      – For established figures, use their commonly known Chinese name (Shakespeare → 莎士比亚 (Shāshìbiǎ))
   • The same Chinese translation must be used when referring to that person within any dialogue.

2. Technical Terms
   • Use established Chinese technical terms
   • First mention: Chinese term (English term); Following mentions: Chinese term only

3. Speaking Style & Format
   • Keep "Speaker: dialogue" format
      – Place actions in parentheses
      – Maintain conversation flow
   • Use appropriate formal Chinese based on context
   • Keep each speaker’s tone consistent
   • se proper conversational particles (吧, 呢, 啊)
   • Adapt greetings and courtesies to Chinese norms(您好,请问,麻烦您)

Provide your answer ONLY in this simple JSON format:
{"translation":["First Chinese translation", "Second Chinese translation", "Third Chinese translation", ..., "Last Chinese translation"]}
English text: {input_text}

Table 15: Chinese translation prompt for booking/meeting/interview domain
You are an expert English-Chinese translator specialized in handling precise information across various domains. Your task is to translate the following English key facts into Chinese (Simplified Chinese), sentence by sentence, with careful attention to quality, accuracy and consistency. Read the given document carefully, fully understand it, and keep that in mind when you translate key fact sentences. Use all terms already written in Chinese from the document to ensure consistency across the translation. Follow the instructions to translate English key fact sentences.

Warning: Use only "standard Simplified Chinese characters" and English technical terms when necessary.

Translation and Verification Rules:

1. Reference Consistency
   • Keep organization names in original form
   • Translate ALL PERSON NAMES to Chinese following appropriate conventions:
      – Western names: Use standard Chinese transliteration
         * Example: "Michael" → "迈克尔" (Mài kè'ěr), "John" → "约翰" (Yúehàn)
      – Chinese names: Maintain Chinese characters
         * Keep family name and given name format (e.g., 王小明)
   • For established figures, use their commonly known Chinese name
      – Example: Shakespeare → 莎士比亚 (Shāshìbiàyà)
   • Use standard format for dates, times, and numbers

2. Technical Terms
   • Use established Chinese technical terms
   • Follow terms already defined in the document for consistency.
   • First mention: Chinese term (English term); Following mentions: Chinese term only
   • Maintain consistency in specialized terms throughout

3. Focus on Information
   • Prioritize the accurate transfer of factual information in each key fact.
   • Avoid any stylistic adjustments or embellishments. Translate the text plainly and faithfully.

4. Back-Translation for Verification:
   • For each translated Chinese sentence:
      – Perform a back-translation into English.
      – Compare the back-translation with the original English key fact.
   • If there is any difference in meaning, revise the Chinese translation and repeat Steps 1–3 until the back-translation aligns with the original English sentence.

Provide your answer in JSON format:
"translation": [("{1}", "Chinese translation"), ("{2}", "Chinese translation"), ..., ("Key fact number", "Chinese translation")]

Document:
{input_text}
{N} Key fact sentences:
{key_facts}

Table 16: Chinese translation prompt for key-fact.
You are an expert English-Chinese translator with extensive experience in translation quality assessment. Your task is to check the quality of the English-Chinese translation, sentence by sentence, with careful attention to quality and accuracy.
Quality check instructions:

1. Accuracy
   • Compare the English source text and Chinese translation to ensure meaning is preserved
   • Check for any omissions or additions
   • Verify numerical values, dates, and proper names are correctly translated
   • Flag any mistranslations or semantic errors

2. Consistency
   • Reference Consistency: Check if proper nouns, organization names, and product names are translated consistently
   • Technical Term Consistency: Verify industry-specific terminology is translated consistently and correctly
   • Style Consistency: Ensure consistent tone and level of formality throughout

3. Fluency
   • Check if the translation reads naturally in Chinese
   • Verify proper Chinese grammar and syntax
   • Ensure appropriate sentence structure and flow
   • Check for any awkward expressions or unnatural phrasing

4. Readability
   • Assess if the text is easy to understand for the target audience
   • Check sentence length and complexity
   • Verify proper paragraph breaks and text organization
   • Ensure clear logical flow

5. Cultural Appropriateness
   • Check for cultural sensitivity
   • Verify idioms and expressions are appropriately localized
   • Ensure measurements, dates, and currencies are properly converted
   • Flag any potential cultural misunderstandings

6. Professionalism
   • Verify appropriate formal/business language usage
   • Check for proper honorific forms
   • Ensure professional terminology is correctly used
   • Maintain appropriate level of formality

7. Fitness for Purpose
   • Verify the translation meets its intended purpose
   • Check if appropriate for target audience
   • Ensure industry-specific requirements are met
   • Verify technical accuracy for specialized content

Provide the answer using the following JSON format:
{ "translation": [(1, "True", "IF True Blank", "IF True Blank", "IF True Blank"), (2, "False", "IF False English Sentence", "IF False Chinese Sentence", "IF False Reason"), ..., (N, "True", "", "", "")] }
English text: {input_text}
Chinese text: {translation_text}

Table 17: Translation quality check prompt.
<table>
  <tr>
    <th>Domain</th>
    <th>Category</th>
    <th>Description</th>
  </tr>
  <tr>
    <td rowspan="5">News</td>
    <td>Main Topic</td>
    <td>General information about the primary event, issue, or occurrence being discussed</td>
  </tr>
  <tr>
    <td>Background</td>
    <td>Situational details supporting the main topic</td>
  </tr>
  <tr>
    <td>Immediate Impact</td>
    <td>Short-term effects or consequences resulting from the main topic</td>
  </tr>
  <tr>
    <td>Future Implications</td>
    <td>Long-term outcomes or projected developments related to the main topic</td>
  </tr>
  <tr>
    <td>Public statements</td>
    <td>Non-expert perspectives, opinions, or reactions from the general public</td>
  </tr>
  <tr>
    <td>Official statements</td>
    <td>Expert or authoritative opinions, assessments, or analysis on the main topic</td>
  </tr>
  <tr>
    <td>Counterarguments</td>
    <td>Critiques or opposition to the main topic or its impacts</td>
  </tr>
  <tr>
    <td rowspan="4">Medical Literature</td>
    <td>Research Finding</td>
    <td>Key discoveries or outcomes from medical research studies</td>
  </tr>
  <tr>
    <td>Medical experiments</td>
    <td>Detailed procedures, methodologies, and designs of experiments or clinical trials testing treatments or interventions</td>
  </tr>
  <tr>
    <td>Disease descriptions</td>
    <td>Detailed explanations of diseases, including symptoms, causes, and characteristics</td>
  </tr>
  <tr>
    <td>Medical Treatment</td>
    <td>Recommended therapies, treatments, and interventions aimed at managing or curing diseases</td>
  </tr>
  <tr>
    <td>Medical Prevention</td>
    <td>Strategies and actions aimed at preventing diseases and promoting public health</td>
  </tr>
  <tr>
    <td rowspan="5">Report</td>
    <td>Recommendations</td>
    <td>Suggested actions or improvements based on report findings</td>
  </tr>
  <tr>
    <td>Governance</td>
    <td>Oversight and administration of programs and resources</td>
  </tr>
  <tr>
    <td>Regulation and Policy</td>
    <td>Legal frameworks, standards, and policies guiding operations</td>
  </tr>
  <tr>
    <td>Evaluations</td>
    <td>Assessment of data and review of program performance</td>
  </tr>
  <tr>
    <td>Financial information</td>
    <td>Details about costs, budget allocations, and financial impacts</td>
  </tr>
  <tr>
    <td rowspan="6">Booking</td>
    <td>General Information</td>
    <td>Reference numbers, contact info, headcounts, or else excluding time, location, and price</td>
  </tr>
  <tr>
    <td>Price and Payment</td>
    <td>Cost, pricing, fees and payment methods</td>
  </tr>
  <tr>
    <td>Time and Schedule</td>
    <td>Time slots, schedules, and booking times</td>
  </tr>
  <tr>
    <td>Location and Route</td>
    <td>Address, location and routes</td>
  </tr>
  <tr>
    <td>Booking Confirmation</td>
    <td>Confirmation and status of bookings for the services</td>
  </tr>
  <tr>
    <td>User Requests</td>
    <td>User’s requests for the booking service or details</td>
  </tr>
  <tr>
    <td>System Suggestions</td>
    <td>Suggestions provided by the system.</td>
  </tr>
  <tr>
    <td rowspan="5">Meeting</td>
    <td>Opinions</td>
    <td>Personal viewpoints, perspectives, or feelings regarding a topic or proposal</td>
  </tr>
  <tr>
    <td>Decisions</td>
    <td>Specific, actionable plans or choices made to address a Problem or improve a situation after careful consideration</td>
  </tr>
  <tr>
    <td>Proposals</td>
    <td>Final conclusion or choice made after discussion, determining the course of action</td>
  </tr>
  <tr>
    <td>Reports</td>
    <td>Structured updates or presentations summarizing status, data, or findings related to projects or objectives</td>
  </tr>
  <tr>
    <td>Factual Information</td>
    <td>Objective data, statistics, or verified information that serves as a basis for decisions or discussions</td>
  </tr>
  <tr>
    <td rowspan="5">Interview</td>
    <td>Background</td>
    <td>Context or historical info to help understanding</td>
  </tr>
  <tr>
    <td>Main Arguments</td>
    <td>Core claims or opinions from each speaker</td>
  </tr>
  <tr>
    <td>Supporting Examples</td>
    <td>Examples, data, or statistics to support the main arguments</td>
  </tr>
  <tr>
    <td>Counterarguments</td>
    <td>Opposing views or criticisms of the main arguments and responses</td>
  </tr>
  <tr>
    <td>Conclusions</td>
    <td>Key points and future directions from the interview</td>
  </tr>
</table>

Table 18: Description of key-fact category for each domain.
<table>
  <tr>
    <th>Domain</th>
    <th>Category</th>
    <th>Example</th>
  </tr>
  <tr>
    <td rowspan="7">News</td>
    <td>Main Topic</td>
    <td>Prince William is scheduled at the Cenotaph.</td>
  </tr>
  <tr>
    <td>Background</td>
    <td>Guy Thorpe-Beeston has 18 years’ experience.</td>
  </tr>
  <tr>
    <td>Immediate Impact</td>
    <td>William will travel by car, not helicopter.</td>
  </tr>
  <tr>
    <td>Future Implications</td>
    <td>The delivery room at St Mary’s Lindo Wing will have a new team.</td>
  </tr>
  <tr>
    <td>Public statements</td>
    <td>Prosecutor Brice Robin confirmed no videos were used.</td>
  </tr>
  <tr>
    <td>Official statements</td>
    <td>Obstetrician Guy Thorpe-Beeston will lead the birth.</td>
  </tr>
  <tr>
    <td>Counterarguments</td>
    <td>Prince William faces a conflict between duties and his child’s birth.</td>
  </tr>
  <tr>
    <td rowspan="5">Medical Literature</td>
    <td>Research Finding</td>
    <td>Intervention strategies have helped reduce malaria globally.</td>
  </tr>
  <tr>
    <td>Medical experiments</td>
    <td>New mapping techniques track malaria transmission.</td>
  </tr>
  <tr>
    <td>Disease descriptions</td>
    <td>Dengue is caused by four viruses (DENV 1–4).</td>
  </tr>
  <tr>
    <td>Medical Treatment</td>
    <td>Dengue treatment involves haematological monitoring.</td>
  </tr>
  <tr>
    <td>Medical Prevention</td>
    <td>Insecticide-treated bednets have been key to malaria control.</td>
  </tr>
  <tr>
    <td rowspan="5">Report</td>
    <td>Recommendations</td>
    <td>HHS agreed to gather more disenrollment data.</td>
  </tr>
  <tr>
    <td>Governance</td>
    <td>CMS reviews aspects of contracts between states and D-SNPs.</td>
  </tr>
  <tr>
    <td>Regulation and Policy</td>
    <td>Dual-eligible beneficiaries qualify for Medicare and Medicaid.</td>
  </tr>
  <tr>
    <td>Evaluations</td>
    <td>CMS lacks data on disenrolled beneficiaries.</td>
  </tr>
  <tr>
    <td>Financial information</td>
    <td>Medicaid covers premiums for dual-eligible beneficiaries.</td>
  </tr>
  <tr>
    <td rowspan="6">Booking</td>
    <td>General Information</td>
    <td>The train ticket number is 12345.</td>
  </tr>
  <tr>
    <td>Price and Payment</td>
    <td>The total cost for the train ticket is $45.</td>
  </tr>
  <tr>
    <td>Time and Schedule</td>
    <td>The restaurant is available from 6 PM to 9 PM.</td>
  </tr>
  <tr>
    <td>Location and Route</td>
    <td>The Grand Hotel is at 123 Kings Road, Brighton, BN1 2GS.</td>
  </tr>
  <tr>
    <td>Booking Confirmation</td>
    <td>A table for 8 is booked.</td>
  </tr>
  <tr>
    <td>User Requests</td>
    <td>The user wants to book a hotel room for two people.</td>
  </tr>
  <tr>
    <td>System Suggestions</td>
    <td>The system recommends taking a train to Cambridge.</td>
  </tr>
  <tr>
    <td rowspan="5">Meeting</td>
    <td>Opinions</td>
    <td>Dave Shukla argues consolidation increases expenses.</td>
  </tr>
  <tr>
    <td>Decisions</td>
    <td>The council decided to consolidate gas under one commission.</td>
  </tr>
  <tr>
    <td>Proposals</td>
    <td>Dave Shukla proposes keeping utilities independent.</td>
  </tr>
  <tr>
    <td>Reports</td>
    <td>The goal is to create one commission for cost savings.</td>
  </tr>
  <tr>
    <td>Factual Information</td>
    <td>Reid concludes the party’s divisiveness cost them control.</td>
  </tr>
  <tr>
    <td rowspan="5">Interview</td>
    <td>Background</td>
    <td>The discussion covers Obama’s State of the Union.</td>
  </tr>
  <tr>
    <td>Main Arguments</td>
    <td>Cary says establishment Republicans struggle against Trump.</td>
  </tr>
  <tr>
    <td>Supporting Examples</td>
    <td>Cary cites Republican grassroots efforts in early states.</td>
  </tr>
  <tr>
    <td>Counterarguments</td>
    <td>Cary defends the Republican Party, claiming not all are hateful.</td>
  </tr>
  <tr>
    <td>Conclusions</td>
    <td>Reid concludes the party’s divisiveness cost them control.</td>
  </tr>
</table>

Table 19: Example of key-fact category for six domains.
Your task is to identify domain-specific key facts within the document in <document></document>, which are essential pieces of information for a high-quality summary. You are provided key-fact category and description in <category></category> tags, and its example in <example></example>. Each key fact must be presented as a standalone, atomic-level sentence. The following is a set of detailed instructions in <instructions></instructions> tags for identifying key facts.

<instructions>

1. Identify Key-facts: Extract all key-facts from the text. Each key-fact should:

   • Be a complete sentence with a subject, verb, and object/complement.
   • Contain only one action, event, or idea. Avoid compound sentences.
   • Include no more than two or three entities per key fact. If there are more than three entities, divide them into separate sentences.
   • Present temporal information (e.g., when, how long) as standalone sentences.
   • Present causal relationships (e.g., reasons, consequences) as standalone sentences.
   • Avoid using linking words like 'and', 'but', 'then' or 'because.' Each idea must be presented as an independent fact.
   • Do not combine related details, even implicitly. Each sentence must describe exactly one action, idea, or relationship.

2. Here are the examples of key-fact structure granularity:

   • Text Example: The resolution authorizes the operation of one winter shelter from December 1st, 2019, to March 31st, 2020.
     • Key-facts (Revised):
       (a) The resolution authorizes the operation of one winter shelter.
       (b) The winter shelter will operate from December 1st, 2019, to March 31st, 2020.
   • Text Example : The property for the winter shelter was purchased using homeless emergency aid program funding.
     • Key-facts (Revised):
       (a) The property for the winter shelter was purchased.
       (b) Homeless emergency aid program funding was used for the purchase.

3. Categorize Key-facts:
   – Define your own categories for the key-facts based on content.
   – Assign each key-fact to a category.

4. Compare Categories:
   – Compare your defined categories with the provided key-fact categories.
   – Adjust any key-fact to better align with the provided categories.

5. Validate Key-facts
   • Ensure each key-fact meets the following criteria:
     – Correctly categorized.
     – Atomicity: Conveys only one action, event, or idea.
     – Clarity: Is concise and clear, avoiding ambiguity.
     – Brevity: Contains no unnecessary details.
     – Non-overlapping: Does not duplicate information in other facts.
   • Finalize key facts that satisfy these conditions.

</instructions>

<category>
Here are the provided Key Fact Categories and Descriptions:
categories
</category>

<example>
Here are the examples of key facts to illustrate the level of granularity for each category:
{category_examples}
</example>

Provide your answer in JSON format. The answer should be a dictionary with tuples:
'key_facts' containing the key facts as a list of tuples (keyfact, reason, category):
["key_facts": [ ("first key fact", "reason", "category"), ("second key fact", "reason", "category") ]]

<document>
[input_text]
</document>

Table 20: Domain-specific key-fact generation prompt.
You will receive a Document and a set of Key-facts sentences that contain essential pieces of information from the Source Document. Your task is to identify if each Key-Fact Sentence is useful for making a summary of the Source Document.

Reasons a Key-Fact Sentence may NOT be useful:
Reason 1) Trivial Information: The sentence contains correct but insignificant details that do not contribute meaningfully to the main points of the Source Document.
• Example: The Source Document is a business meeting transcript, but the Key-Fact Sentence is "Speaker A said 'Good afternoon.'"
• Explanation: While accurate, this detail doesn’t enhance understanding of the meeting’s objectives or outcomes.

Reason 2) Incorrect Information: The sentence includes factual errors or discrepancies compared to the Source Document.
• Example: The Source Document mentions 12 children living in a small town, but the Key-Fact Sentence states 11 children in a big city.
• Explanation: The numbers and locations don’t match, making the sentence inaccurate.

Reason 3) Irrelevant Information: The sentence is unrelated to the content of the Source Document.
• Example: The Source Document is about artificial intelligence, but the Key-Fact Sentence discusses a dog barking loudly.
• Explanation: The sentence doesn’t pertain to the topic of the Source Document.

Reason 4) Category Alignment: The sentence is incorrectly categorized.
• Example: A sentence about future plans is categorized as "Factual Information" when it should be "Proposals"
• Explanation: The content of the sentence doesn’t match the characteristics of its assigned category.

Reason 5) Domain Relevance: The information in the sentence, while accurate and properly categorized, is not essential for summarizing this type of document.
• Example: In a meeting minutes document discussing a major company merger, a sentence about routine office maintenance schedule categorizing as "Factual Information"
• Explanation: Although this information is accurate and properly categorized as "Factual Information", it’s not essential for understanding the key points of a merger discussion meeting.

Here are the Instructions for Key-facts validation:
1. Read a Document and a set of Key-facts sentences carefully.
2. Evaluate each Key-facts sentence based on the five reasons above.
3. According to evaluation, if the Key-facts sentence is useful for making a summary of the Source Document, response "Yes", otherwise response "No"
4. Provide a single sentence explaining why the Key-facts sentence is useful for making a summary.

Provide your answer in JSON format. The answer should be a list of dictionaries whose keys are "sentence", "response", "reason".
you should provide a response and a reason for all Key-facts sentence: [{"sentence": "first key-fact sentence", "response": "your response", "reason": "your reason"},{"sentence": "second key-fact sentence", "response": "your response", "reason": "your reason"}, ... , {"sentence": "N-th key-fact sentence", "response": "your response", "reason": "your reason"}]
Document:
{input_text}

{N} Key-facts sentence with Category:
{key_fact}

Table 21: Key-fact validation prompt.
(a) English summary generation prompt

Text: {input_text}

Instruction: Summarize the Text.

Provide your answer in JSON format: The answer should be a dictionary with the key "summary" containing a generated summary as a string:
{"summary": "your summary"}

(b) Chinese summary generation prompt

文本: {input_text}

指令: 请用中文总结这段文字

以 JSON 格式提供答案 答案应是一个包含 "summary" 键和值的字典,值为生成的摘要字符串:
{"summary": "总结内容"}

Table 22: Summary generation prompt.
You will be provided with two sets of information:

– List of sentences A: A list of sentences that need to be evaluated.

– sentence B: A sentence that will be checked against sentence A.

Your task is to evaluate whether the complete information in each sentence from List A is fully contained in sentence B. This requires performing n evaluations (where n is the number of sentences in List A). For example, if there are 15 sentences in List A, you have to do 15 evaluations.

Instruction:

– A sentence from List A is considered "contained" in the sentence B if and only if:

    1. The complete information conveyed by the sentence in List A is entirely present in the sentence B.
    2. The essential meaning of the sentence in List A must align with or be fully and explicitly captured by sentence B.
    3. The information conveyed by the sentence in List A must be explicitly implied or fully understood by sentence B.

• Exact wording is not required, but the complete and explicit meaning must match.

• Provide a short reason, and a label: contained, not contained.

Please provide your answer in JSON format:
[{"sentence A": "1", "label": "contained", "reason": "Short explanation of why you chose this label"}, {"sentence A": "sentence A number", "label": "contained or not contained", "reason": "Short explanation of why you chose this label"},
..., {"sentence A": "N", "label": "not contained", "reason": "Short explanation of why you chose this label"}]
Note:

– Ensure that all n evaluations are performed without omission. Each combination of a sentence from List A and the sentence B must be explicitly evaluated.

– Any skipped or missing evaluations will result in incomplete analysis, so please confirm that no sentence in List A is overlooked.

– Clearly explain your reasoning even if the label is "not contained."

– Ensure your evaluation strictly adheres to the requirement that the entire meaning of the sentence in List A should be fully present in sentence B to be labeled as "contained."

– Criteria recap: contained, not contained

List of sentences A:
{key-fact}

Sentence B:
{summary}

Table 23: Key-fact alignment prompt.
You are the ADVOCATE, an agent defending the factual consistency of the summary. Assume the summary sentences are always true and faithful. Cite specific sentences from the reference document as evidence to support your claim for each summary sentence. You are given the reference document provided in <document></document> tags and the summary sentences in <summary></summary> tags. Critically assess and present your reasoning.

<errors>
• out-of-article error: If the summary sentence introduces facts, subjective opinions, or biases that cannot be verified or confirmed by the reference document, the summary is factually inconsistent with the reference document.
• entity error: If the summary sentence includes incorrect or misrepresented entities, such as names, numbers, or main subjects, the summary is factually inconsistent with the reference document.
• relation error: If the summary sentence contains incorrect semantic relationships, such as the use of wrong verbs, prepositions, or adjectives, which distort the relationships between entities, the summary sentence is factually inconsistent with the reference document.
• sentence error: If the summary sentence completely contradicts the information in the reference document, requiring significant revision or removal to align with the reference document, the summary sentence is factually inconsistent with the reference document.
</errors>

<instructions>
Here is the instructions for writing your arguments:
Follow these steps carefully to ensure a structured and thorough evaluation under your assigned role.
1. Read the reference document and summary sentence under your role:
   – Carefully read the reference document and try to fully understand it.
   – Compare each summary sentence to the reference document to identify evidence supporting its factual consistency.
   – Refer to the error types in <errors></errors> tags to better defend your claim (faithfulness); focusing on areas of alignment and acceptable variations.
2. As a ADVOCATE, focus on finding alignment:
   – Explicitly identify numbered sentences in the reference document that support or partially align with the summary sentence.
   – Even if a perfect match cannot be found, select the closest sentence(s) that contain key elements (entities, relationships, events, quantities, or cause-effect relationships).
   – Always select at least one numbered sentence from the reference document, even if it partially aligns with the summary.
   – As an ADVOCATE, focus on defending the faithfulness of the summary while addressing any potential inconsistencies.
3. Provide a detailed explanation of your arguments:
   – For each summary sentence:
     • Cite one or more numbered sentences from the reference document, even if only partial alignment exists.
     • Use the format “reference_sentence_number”: [number1, number2] to explicitly indicate the reference of support.
     • Include a “reason” that explicitly explains how the cited sentences align with the summary, focusing on entities, relationships, and key details.
   – Your explanation must:
     • Be concise.
     • Reference specific elements, such as:
       • Key details: Validate quantities, events, and cause-effect relationships.
       • Rephrasing accuracy: Argue that the summary retains the original meaning despite rephrasing.
       • Word choice. Ensure terms like “only,” “significant,” or “most” match the intensity or scale of the reference document.
</instructions>
Provide your answers in JSON format as shown below:
[{
  "summary_sentence_num": "0",
  "label": "1 (if faithfulness 1, else 0)",
  "error_type": "no error ( if faithfulness no error )",
  "reference_sentence": ["sentence from reference document", "another sentence from reference document"],
  "reference_sentence_number": [0, 1],
  "reason": "The summary mentions the entity 'X', but the reference document refers to 'Y' [0]. Additionally, the relationship between 'A' and 'B' is misrepresented, as the reference indicates 'A causes B', not 'A results from B' [1]."
},
{
  "summary_sentence_num": "N",
  "label": "1 (if faithfulness 1, else 0)",
  "error_type": "no error ( if faithfulness no error )",
  "reference_sentence": ["closest available sentence"],
  "reference_sentence_number": [10],
  "reason": "The summary states 'Event Z happened', but no reference to 'Event Z' is found in the reference [10]."
}]

Reference document, divided into numbered sentences:
<document>
input_text
</document>

Summary with N sentences:
<summary>
summaries
</summary>
You are the SKEPTIC, an agent identifying and arguing for factual inconsistencies in the summary, considering error types. Assume the summary sentences are always unfaithful. Cite specific sentences from the reference document as evidence to support your claim for each summary sentence. You are given the reference document provided in <document></document> tags and the summary sentences in <summary></summary> tags. Now, follow the instructions in <instructions></instructions> tags. Critically analyze and present your reasoning.

<errors>
    • out-of-article error: If the summary sentence introduces facts, subjective opinions, or biases that cannot be verified or confirmed by the reference document, the summary is factually inconsistent with the reference document.
    • entity error: If the summary sentence includes incorrect or misrepresented entities, such as names, numbers, or main subjects, the summary is factually inconsistent with the reference document.
    • relation error: If the summary sentence contains incorrect semantic relationships, such as the use of wrong verbs, prepositions, or adjectives, which distort the relationships between entities, the summary sentence is factually inconsistent with the reference document.
    • sentence error: If the summary sentence completely contradicts the information in the reference document, requiring significant revision or removal to align with the reference document, the summary sentence is factually inconsistent with the reference document.
</errors>

<instructions>
Here is the instructions for writing your arguments:
Follow these steps carefully to ensure a structured and thorough evaluation under your assigned role.
1. Read the reference document and summary sentence under your role:
   – Carefully read the reference document and try to fully understand it.
   – Compare each summary sentence to the reference document to identify evidence supporting its factual consistency.
   – Refer to the list of errors in <errors></errors> tags to find evidence to support your claims(unfaithfulness), paying special attention to the listed error types and acceptable variations.
2. As a SKEPTIC, focus on identifying discrepancies:
   – Explicitly identify numbered sentences in the source document that contradict or fail to align with the summary sentence.
   – Even if a perfect match cannot be found, select the closest sentence(s) that contain key elements (entities, relationships, events, quantities, or cause-effect relationships).
   – Always select at least one numbered sentence from the source document that highlights inconsistencies or raises doubts about the summary.
3. Provide a detailed explanation of your arguments:
   – For each summary sentence:
     • Cite one or more numbered sentences from the reference document, even if only partial alignment exists.
     • Use the format "reference_sentence_number": [number1, number2] to explicitly indicate the reference of support.
     • Include a “reason” that explicitly explains how the cited sentences align with the summary, focusing on entities, relationships, and key details.
   – Your explanation must:
     • Be concise.
     • Reference specific elements, such as;
       * Key details: Validate quantities, events, and cause-effect relationships.
       * Rephrasing accuracy: Argue that the summary retains the original meaning despite rephrasing.
       * Word choice: Ensure terms like "only," "significant," or "most" match the intensity or scale of the reference document.
</instructions>

Provide your answers in JSON format as shown below:
[{
  "summary_sentence_num": "0",
  "label": "1 ( if faithfulness 1, else 0 )",
  "error_type": "no error ( if faithfulness no error )",
  "reference_sentence": ["sentence from reference document", "another sentence from reference document"],
  "reference_sentence_number": [0, 1],
  "reason": "The summary mentions the entity 'X', but the reference document refers to 'Y' [0]. Additionally, the relationship between 'A' and 'B' is misrepresented, as the reference indicates 'A causes B', not 'A results from B' [1]."
},
{
  "summary_sentence_num": "N",
  "label": "1 (if faithfulness 1, else 0 )",
  "error_type": "no error ( if faithfulness no error )",
  "reference_sentence": ["closest available sentence"],
  "reference_sentence_number": [10],
  "reason": "The summary states 'Event Z happened', but no reference to 'Event Z' is found in the reference [10]." }
]

Reference document, divided into numbered sentences:

<document>
input_text
</document>

Summary with N sentences:

<summary>
summaries
</summary>
You are the ADJUDICATOR, an agent tasked with providing the final decision for the faithfulness of the summary by assessing the claims presented by the ADVOCATE and SKEPTIC. You are given the reference document provided in <document></document> tags, summary sentences in <summary></summary> tags and the opposing claims in <claim></claim> tags. Now, follow the instructions in <instructions></instructions> tags and "Make sure to always strive to deeply understand and remember the guidelines in <note></note> tags. Think critically and articulate your final decision.

<note>
1. Faithfulness measures how accurately a summary sentence reflects the source document’s information and content.
2. The summary sentence should not have to use exact wording in the reference document as long as the original meaning is preserved.
3. The summary sentence can paraphrase and use alternative expressions with preserving the original meaning.
4. The summary sentence is factually consistent even if it omits specific details-one, some or all from reference document.
5. The summary sentence is factually consistent even if it omits specific details-one, some or all from reference document.
6. The summary sentence is factually consistent even if it modifies the level of specificity (using broader terms instead of detail and specific information, or more specific terms instead of the broader terms), maintaining the original information.
7. The summary sentence is factually consistent even if it combines multiple pieces of information from different parts of the text maintaining the original meaning without contradiction.
8. Even if the summary sentence draws reasonable implications, logical conclusions, or appropriate generalizations, it remains factually consistent with the reference document as long as these are explicitly supported by the original meaning.
</note>
<instructions>
1. Read the reference document and summary:
   – Carefully review the reference document and the summary sentence provided.
   – Develop a comprehensive understanding of both the reference document and summary sentence and how it’s been summarized.
2. Evaluate the validity of agent arguments:
   – Compare both agents’ reasoning critically.
   – Validate the claims align with the reference document and avoid unsupported speculation.
   – Ensure the claims follow the guidelines in <note></note> tags.
3. Finalize your own judgment of the summary sentences.
   – Make a final decision on whether the summary sentence is factually consistent with the reference document, based on your understanding of reference document, summary sentence and the validation of the two opposing claims.
4. Provide your final decision as error type and label
   – Assign an error type and label as follows:
     – error_type: refer to the error_type listed below:
       • no error: no error found, and the summary is factually consistent with the reference document.
       • out-of-article error: If the summary sentence includes facts, subjective opinions, or biases that cannot be verified or confirmed by the reference document, the summary is factually inconsistent with the reference document.
       • entity error: If the summary sentence includes incorrect or misrepresented entities, such as names, numbers, or main subjects, the summary is factually inconsistent with the reference document.
       • relation error: If the summary sentence contains incorrect semantic relationships, such as the use of wrong verbs, prepositions, or adjectives, which distort the relationships between entities, the summary sentence is factually inconsistent with the reference document.
       • sentence error: If the summary sentence completely contradicts the information in the reference document, requiring significant revision or removal to align with the reference document, the summary sentence is factually inconsistent with the reference document.
   – label:
     • 1: faithfulness, assigned if the summary sentence has no error.
     • 0: unfaithfulness, assigned if the summary contains any error (out-of-article, entity, relation, or sentence errors).
   – In one or two sentences, explain why you agree with one agent’s argument and disagree with the opposing agent’s argument.
</instructions>
Provide your answers in JSON format as shown below:
[{
  "summary_sentence_num": "0",
  "label": "1" (if faithfulness 1, else 0)",
  "error_type": "no error ( if faithfulness no error )",
  "reason": "write reason of your decision briefly and concisely"
},
{
  "summary_sentence_num": "N",
  "label": "1" (if faithfulness 1, else 0)",
  "error_type": "no error ( if faithfulness no error )",
  "reason": "write reason of your decision briefly and concisely"
}]

Reference document, divided into numbered sentences:
<document>
<input_text>
</document>
Summary with N sentences:
<summary>
{summaries}
</summary>
Here are the claims provided by ADVOCATE and SKEPTIC on for each summary sentence:
<claim>
{claim}
</claim>
<table>
  <tr>
    <th rowspan="2">Model Type</th>
    <th rowspan="2">Summarizer</th>
    <th colspan="6">English</th>
    <th colspan="6">Chinese</th>
    <th rowspan="2">Language Stability</th>
  </tr>
  <tr>
    <th>News</th><th>Med Lit</th><th>Report</th><th>Booking</th><th>Meeting</th><th>Interview</th><th>Domain*</th>
    <th>News</th><th>Med Lit</th><th>Report</th><th>Booking</th><th>Meeting</th><th>Interview</th><th>Domain*</th>
  </tr>
  <tr>
    <td rowspan="3">Proprietary LLMs</td>
    <td>GPT-4o</td><td>86.30</td><td>80.10</td><td>82.50</td><td>95.10</td><td>79.90</td><td>92.70</td><td>92.97</td><td>87.10</td><td>78.01</td><td>80.00</td><td>88.20</td><td>56.70</td><td>80.50</td><td>87.31</td><td>93.81</td>
  </tr>
  <tr>
    <td>Claude 3.5_Summt</td><td><b>96.20</b></td><td>83.70</td><td>89.50</td><td>94.40</td><td><b>80.60</b></td><td>92.10</td><td><b>93.58</b></td><td><b>88.90</b></td><td>73.20</td><td>84.00</td><td><b>75.40</b></td><td><b>88.30</b></td><td><b>92.45</b></td><td>94.75</td>
  </tr>
  <tr>
    <td>Gemini 1.5_Ppo</td><td>85.60</td><td><b>86.90</b></td><td>88.30</td><td><b>97.40</b></td><td>74.70</td><td>84.20</td><td>92.18</td><td>87.00</td><td>84.64</td><td>82.80</td><td><b>92.00</b></td><td>60.60</td><td>74.50</td><td>87.74</td><td><b>95.21</b></td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>89.37</td><td>83.57</td><td>86.77</td><td>95.63</td><td>78.40</td><td>89.67</td><td>92.91</td><td>87.67</td><td>82.97</td><td>78.67</td><td>88.07</td><td>64.23</td><td>81.10</td><td>89.17</td><td>94.59</td>
  </tr>
  <tr>
    <td rowspan="3">Open source LLMs</td>
    <td>Gemma 2.27b</td><td>94.90</td><td>84.10</td><td>84.00</td><td>93.60</td><td>72.20</td><td>93.30</td><td>90.87</td><td>81.30</td><td>72.90</td><td>77.20</td><td>66.80</td><td>86.00</td><td>90.94</td><td>93.24</td>
  </tr>
  <tr>
    <td>Llama 3.1_70b</td><td>87.70</td><td>76.40</td><td><b>92.90</b></td><td>89.30</td><td>78.50</td><td>76.20</td><td>91.93</td><td>84.90</td><td>77.70</td><td>72.00</td><td>79.00</td><td>67.10</td><td>81.90</td><td>92.18</td><td>94.66</td>
  </tr>
  <tr>
    <td>Qwen 2.5_72b</td><td>95.10</td><td>81.30</td><td>91.40</td><td>91.50</td><td>73.70</td><td><b>94.70</b></td><td>91.11</td><td>82.40</td><td>86.07</td><td>86.00</td><td>80.20</td><td>62.50</td><td>84.60</td><td>89.92</td><td>93.95</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>92.57</td><td>80.60</td><td>89.43</td><td>91.47</td><td>74.80</td><td>88.07</td><td>91.30</td><td>84.77</td><td>81.69</td><td>76.97</td><td>78.80</td><td>65.47</td><td>84.17</td><td>91.01</td><td>93.95</td>
  </tr>
  <tr>
    <td rowspan="2">Non LLMs</td>
    <td>mT5</td><td>22.00</td><td>12.00</td><td>2.00</td><td>34.00</td><td>4.00</td><td>0.00</td><td>48.02</td><td>68.00</td><td>18.00</td><td>32.00</td><td>2.00</td><td>14.00</td><td>18.00</td><td>52.41</td><td>67.20</td>
  </tr>
  <tr>
    <td>BART</td><td>93.90</td><td>76.30</td><td>83.00</td><td>76.00</td><td>50.30</td><td>85.90</td><td>83.87</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>57.95</td><td>44.15</td><td>42.50</td><td>55.00</td><td>27.15</td><td>42.95</td><td>65.94</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 27: Faithfulness score across six domains and two languages. Domain*: domain stability

<table>
  <tr>
    <th rowspan="2">Model Type</th>
    <th rowspan="2">Summarizer</th>
    <th colspan="6">English</th>
    <th colspan="6">Chinese</th>
    <th rowspan="2">Language Stability</th>
  </tr>
  <tr>
    <th>News</th><th>Med Lit</th><th>Report</th><th>Booking</th><th>Meeting</th><th>Interview</th><th>Domain*</th>
    <th>News</th><th>Med Lit</th><th>Report</th><th>Booking</th><th>Meeting</th><th>Interview</th><th>Domain*</th>
  </tr>
  <tr>
    <td rowspan="3">Proprietary LLMs</td>
    <td>GPT-4o</td><td>55.73</td><td><b>41.22</b></td><td>40.25</td><td>68.19</td><td>50.10</td><td>44.65</td><td><b>82.48</b></td><td>55.80</td><td>32.63</td><td>33.83</td><td>47.58</td><td>39.38</td><td>37.92</td><td>82.24</td><td>87.95</td>
  </tr>
  <tr>
    <td>Claude 3.5_Summt</td><td><b>65.49</b></td><td>39.96</td><td>42.88</td><td>68.77</td><td>48.40</td><td>50.51</td><td>81.61</td><td>55.89</td><td><b>41.06</b></td><td>35.20</td><td>71.43</td><td>46.87</td><td>47.17</td><td>79.58</td><td>95.93</td>
  </tr>
  <tr>
    <td>Gemini 1.5_Ppo</td><td>64.43</td><td>39.92</td><td><b>44.43</b></td><td><b>78.35</b></td><td>48.40</td><td><b>53.30</b></td><td>79.33</td><td><b>57.74</b></td><td>38.87</td><td><b>48.49</b></td><td><b>72.20</b></td><td><b>51.43</b></td><td><b>52.54</b></td><td><b>82.88</b></td><td><b>96.38</b></td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>61.88</td><td>40.37</td><td>42.52</td><td>71.77</td><td>48.97</td><td>49.49</td><td>81.14</td><td>56.48</td><td>37.52</td><td>39.17</td><td>63.74</td><td>45.89</td><td>45.88</td><td>81.57</td><td>94.09</td>
  </tr>
  <tr>
    <td rowspan="3">Open source LLMs</td>
    <td>Gemma 2.27b</td><td>52.26</td><td>23.67</td><td>25.86</td><td>56.95</td><td>35.83</td><td>27.77</td><td>72.19</td><td>43.23</td><td>23.65</td><td>21.43</td><td>51.25</td><td>23.72</td><td>29.77</td><td>72.41</td><td>90.93</td>
  </tr>
  <tr>
    <td>Llama 3.1_70b</td><td>51.96</td><td>29.34</td><td>35.91</td><td>51.66</td><td>29.48</td><td>36.76</td><td>79.25</td><td>24.59</td><td>21.71</td><td>21.26</td><td>35.06</td><td>20.46</td><td>25.60</td><td>82.05</td><td>75.84</td>
  </tr>
  <tr>
    <td>Qwen 2.5_72b</td><td>55.81</td><td>37.73</td><td>39.58</td><td>70.12</td><td><b>55.24</b></td><td>46.74</td><td>80.79</td><td>47.69</td><td>34.13</td><td>33.17</td><td>61.48</td><td>34.35</td><td>34.84</td><td>78.15</td><td>86.74</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>53.34</td><td>30.25</td><td>33.78</td><td>59.58</td><td>40.18</td><td>37.09</td><td>77.41</td><td>38.50</td><td>26.50</td><td>25.29</td><td>49.26</td><td>26.18</td><td>30.07</td><td>77.54</td><td>84.30</td>
  </tr>
  <tr>
    <td rowspan="2">Non LLMs</td>
    <td>mT5</td><td>9.66</td><td>0.67</td><td>1.68</td><td>6.09</td><td>3.07</td><td>1.56</td><td>52.36</td><td>9.97</td><td>0.00</td><td>1.67</td><td>0.97</td><td>2.12</td><td>2.43</td><td>44.34</td><td>83.51</td>
  </tr>
  <tr>
    <td>BART</td><td>32.97</td><td>14.93</td><td>13.15</td><td>45.49</td><td>22.74</td><td>20.79</td><td>67.16</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>21.32</td><td>7.80</td><td>7.42</td><td>25.79</td><td>12.91</td><td>11.18</td><td>59.76</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 28: Completeness score across six domains and two languages. Domain*: domain stability

<table>
  <tr>
    <th rowspan="2">Model Type</th>
    <th rowspan="2">Summarizer</th>
    <th colspan="6">English</th>
    <th colspan="6">Chinese</th>
    <th rowspan="2">Language Stability</th>
  </tr>
  <tr>
    <th>News</th><th>Med Lit</th><th>Report</th><th>Booking</th><th>Meeting</th><th>Interview</th><th>Domain*</th>
    <th>News</th><th>Med Lit</th><th>Report</th><th>Booking</th><th>Meeting</th><th>Interview</th><th>Domain*</th>
  </tr>
  <tr>
    <td rowspan="3">Proprietary LLMs</td>
    <td>GPT-4o</td><td>85.73</td><td>83.40</td><td>81.20</td><td>74.76</td><td>73.51</td><td>69.27</td><td>92.39</td><td>84.20</td><td><b>79.85</b></td><td>77.75</td><td>74.16</td><td>65.00</td><td>68.91</td><td>91.33</td><td>97.30</td>
  </tr>
  <tr>
    <td>Claude 3.5_Summt</td><td>90.73</td><td>74.34</td><td>80.59</td><td>87.26</td><td>72.18</td><td>75.43</td><td>91.42</td><td>84.72</td><td>79.05</td><td>73.56</td><td><b>89.33</b></td><td>66.71</td><td>75.05</td><td>90.57</td><td><b>98.23</b></td>
  </tr>
  <tr>
    <td>Gemini 1.5_Ppo</td><td>88.59</td><td>83.18</td><td>79.71</td><td><b>89.90</b></td><td>69.88</td><td><b>80.01</b></td><td>91.86</td><td>88.07</td><td>74.12</td><td>77.71</td><td>87.35</td><td><b>71.35</b></td><td><b>77.98</b></td><td><b>92.04</b></td><td>97.90</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>88.35</td><td>80.31</td><td>80.50</td><td>83.97</td><td>71.86</td><td>74.90</td><td>91.89</td><td>85.66</td><td>77.67</td><td>76.34</td><td>83.61</td><td>67.69</td><td>73.98</td><td>91.31</td><td>97.81</td>
  </tr>
  <tr>
    <td rowspan="3">Open source LLMs</td>
    <td>Gemma 2.27b</td><td>88.07</td><td>70.69</td><td>72.15</td><td>69.40</td><td>66.23</td><td>90.58</td><td>81.47</td><td>71.46</td><td>59.35</td><td>82.00</td><td>52.47</td><td>71.36</td><td>85.49</td><td>96.22</td>
  </tr>
  <tr>
    <td>Llama 3.1_70b</td><td><b>93.03</b></td><td>79.33</td><td>86.05</td><td>86.20</td><td>67.87</td><td>75.35</td><td>90.05</td><td>85.07</td><td>73.72</td><td>72.22</td><td>85.67</td><td>66.35</td><td>67.73</td><td>89.95</td><td>94.71</td>
  </tr>
  <tr>
    <td>Qwen 2.5_72b</td><td>91.63</td><td><b>84.38</b></td><td><b>87.29</b></td><td>87.67</td><td><b>83.02</b></td><td>75.20</td><td><b>93.81</b></td><td><b>91.53</b></td><td>78.06</td><td><b>86.53</b></td><td>83.33</td><td>66.86</td><td>70.73</td><td>89.37</td><td>95.59</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>90.91</td><td>78.13</td><td>81.94</td><td>83.00</td><td>73.43</td><td>72.26</td><td>91.48</td><td>86.02</td><td>74.41</td><td>72.70</td><td>83.67</td><td>61.89</td><td>69.94</td><td>88.27</td><td>95.51</td>
  </tr>
  <tr>
    <td rowspan="2">Non LLMs</td>
    <td>mT5</td><td>46.00</td><td>8.00</td><td>28.00</td><td>42.00</td><td>20.00</td><td>28.00</td><td>67.17</td><td>80.00</td><td>0.00</td><td>36.00</td><td>12.00</td><td>18.00</td><td>20.00</td><td>49.54</td><td>97.55</td>
  </tr>
  <tr>
    <td>BART</td><td>89.00</td><td>81.00</td><td>86.00</td><td>88.00</td><td>60.00</td><td>66.90</td><td>86.58</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td colspan="2">Average</td><td>67.50</td><td>44.50</td><td>57.00</td><td>65.00</td><td>40.00</td><td>47.45</td><td>76.88</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 29: Conciseness score across six domains and two languages. Domain*: domain stability

<table>
  <tr>
    <th>Model Type</th>
    <th>Evaluator</th>
    <th>News</th>
    <th>Medical Lit.</th>
    <th>Report</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
    <th>Domain Stability</th>
  </tr>
  <tr>
    <td>QA-Based</td>
    <td>QAFactEval</td>
    <td>0.53</td>
    <td>0.53</td>
    <td>0.37</td>
    <td>0.36</td>
    <td>0.33</td>
    <td>0.44</td>
    <td>83.08</td>
  </tr>
  <tr>
    <td>NLI-Based</td>
    <td>AlignScore<br>MiniCheck</td>
    <td><b>0.73</b></td>
    <td>0.73</td>
    <td>0.63</td>
    <td>0.28</td>
    <td>0.56</td>
    <td>0.51</td>
    <td>77.58</td>
  </tr>
  <tr>
    <td>LLM-Based</td>
    <td>G-Eval<br>FineSurE</td>
    <td>0.52</td>
    <td>0.67</td>
    <td>0.67</td>
    <td>0.37</td>
    <td>0.38</td>
    <td>0.46</td>
    <td>79.10</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>0.42</td>
    <td>0.62</td>
    <td><b>0.69</b></td>
    <td><b>0.48</b></td>
    <td><b>0.70</b></td>
    <td><b>0.60</b></td>
    <td><b>83.75</b></td>
  </tr>
</table>

Table 30: Correlation between automatic evaluator and human evaluation.
<table>
  <tr>
    <th rowspan="2">Dimension</th>
    <th rowspan="2">Model Type</th>
    <th rowspan="2">Evaluator</th>
    <th colspan="6">English</th>
    <th colspan="6">Chinese</th>
    <th rowspan="2">Language Stability</th>
  </tr>
  <tr>
    <th>News</th>
    <th>Medical Lit.</th>
    <th>Report</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
    <th>Domain*</th>
    <th>News</th>
    <th>Medical Lit.</th>
    <th>Report</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
    <th>Domain*</th>
  </tr>
  <tr>
    <td rowspan="4">Faithfulness</td>
    <td rowspan="2">Proprietary</td>
    <td>G-Eval</td>
    <td>0.62</td><td>0.61</td><td>0.62</td><td>0.75</td><td><b>0.60</b></td><td>0.66</td><td>91.19</td><td>0.26</td><td>0.52</td><td>0.44</td><td>0.62</td><td>0.43</td><td>0.53</td><td>78.29</td><td><b>94.17</b></td>
  </tr>
  <tr>
    <td>LLMs</td>
    <td>FineSurE</td>
    <td><b>0.72</b></td><td>0.64</td><td><b>0.85</b></td><td>0.64</td><td>0.58</td><td><b>0.78</b></td><td>87.29</td><td><b>0.39</b></td><td><b>0.64</b></td><td><b>0.57</b></td><td><b>0.74</b></td><td><b>0.52</b></td><td><b>0.66</b></td><td><b>82.18</b></td><td>88.71</td>
  </tr>
  <tr>
    <td rowspan="2">Open-source</td>
    <td>G-Eval</td>
    <td>0.69</td><td><b>0.65</b></td><td>0.72</td><td><b>0.78</b></td><td>0.56</td><td>0.67</td><td><b>92.46</b></td><td>0.32</td><td>0.52</td><td><b>0.57</b></td><td>0.73</td><td>0.42</td><td>0.54</td><td>76.33</td><td>87.77</td>
  </tr>
  <tr>
    <td>LLMs</td>
    <td>FineSurE</td>
    <td>0.68</td><td>0.54</td><td>0.65</td><td>0.63</td><td>0.57</td><td>0.75</td><td>88.11</td><td>0.37</td><td>0.52</td><td>0.54</td><td>0.66</td><td>0.44</td><td>0.60</td><td>81.80</td><td>87.00</td>
  </tr>
  <tr>
    <td rowspan="4">Completeness</td>
    <td rowspan="2">Proprietary</td>
    <td>G-Eval</td>
    <td>0.62</td><td>0.61</td><td>0.62</td><td>0.75</td><td>0.60</td><td>0.66</td><td>90.56</td><td>0.71</td><td>0.40</td><td>0.67</td><td>0.66</td><td>0.64</td><td>0.65</td><td>77.93</td><td>89.75</td>
  </tr>
  <tr>
    <td>LLMs</td>
    <td>FineSurE</td>
    <td><b>0.79</b></td><td><b>0.76</b></td><td><b>0.81</b></td><td><b>0.89</b></td><td><b>0.83</b></td><td><b>0.82</b></td><td><b>94.79</b></td><td><b>0.83</b></td><td><b>0.75</b></td><td><b>0.75</b></td><td><b>0.90</b></td><td><b>0.83</b></td><td><b>90.56</b></td><td><b>95.27</b></td>
  </tr>
  <tr>
    <td rowspan="2">Open-source</td>
    <td>G-Eval</td>
    <td>0.69</td><td>0.65</td><td>0.72</td><td>0.78</td><td>0.56</td><td>0.67</td><td>92.25</td><td>0.69</td><td>0.46</td><td>0.59</td><td>0.75</td><td>0.57</td><td>0.59</td><td>82.25</td><td>92.19</td>
  </tr>
  <tr>
    <td>LLMs</td>
    <td>FineSurE</td>
    <td>0.66</td><td>0.54</td><td>0.61</td><td>0.81</td><td>0.72</td><td>0.61</td><td>89.29</td><td>0.62</td><td>0.19</td><td>0.35</td><td>0.81</td><td>0.46</td><td>0.65</td><td>76.46</td><td>88.15</td>
  </tr>
  <tr>
    <td rowspan="4">Completeness</td>
    <td rowspan="2">Proprietary</td>
    <td>G-Eval</td>
    <td>0.64</td><td>0.58</td><td>0.48</td><td>0.43</td><td>0.28</td><td>0.48</td><td>85.20</td><td>0.46</td><td>0.59</td><td>0.44</td><td>0.47</td><td>0.50</td><td>0.58</td><td>78.44</td><td>85.40</td>
  </tr>
  <tr>
    <td>LLMs</td>
    <td>FineSurE</td>
    <td><b>0.80</b></td><td><b>0.72</b></td><td><b>0.57</b></td><td><b>0.76</b></td><td><b>0.68</b></td><td><b>0.72</b></td><td><b>89.42</b></td><td><b>0.78</b></td><td><b>0.74</b></td><td><b>0.64</b></td><td><b>0.84</b></td><td><b>0.60</b></td><td><b>0.74</b></td><td><b>88.62</b></td><td><b>97.13</b></td>
  </tr>
  <tr>
    <td rowspan="2">Open-source</td>
    <td>G-Eval</td>
    <td>0.65</td><td>0.63</td><td>0.51</td><td>0.46</td><td>0.51</td><td>0.47</td><td>86.33</td><td>0.30</td><td>0.65</td><td>0.35</td><td>0.72</td><td>0.45</td><td>0.54</td><td>75.50</td><td>89.80</td>
  </tr>
  <tr>
    <td>LLMs</td>
    <td>FineSurE</td>
    <td>0.74</td><td>0.46</td><td>0.46</td><td>0.67</td><td>0.56</td><td>0.56</td><td>84.25</td><td>0.62</td><td>0.19</td><td>0.35</td><td>0.81</td><td>0.46</td><td>0.65</td><td>75.82</td><td>92.65</td>
  </tr>
</table>

Table 31: Human alignment performance of automatic evaluators across three evaluation dimensions, six domains and two languages. Domain*: Domain Stability

<table>
  <tr>
    <th rowspan="2">Dimension</th>
    <th rowspan="2">Evaluator</th>
    <th colspan="6">English</th>
    <th colspan="6">Chinese</th>
  </tr>
  <tr>
    <th>News</th>
    <th>Med Lit.</th>
    <th>Report</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
    <th>News</th>
    <th>Med Lit.</th>
    <th>Report</th>
    <th>Booking</th>
    <th>Meeting</th>
    <th>Interview</th>
  </tr>
  <tr>
    <td rowspan="5">Faithfulness</td>
    <td>ROUGE-1</td>
    <td>-0.16</td><td>0.35</td><td>0.38</td><td>-</td><td>-0.1</td><td>0.06</td><td>-0.07</td><td>0.12</td><td>-0.09</td><td>-</td><td>0.16</td><td>0.01</td>
  </tr>
  <tr>
    <td>ROUGE-2</td>
    <td>-0.05</td><td>-0.08</td><td>-0.1</td><td>-</td><td>0.13</td><td>-0.17</td><td>0.07</td><td>0.1</td><td>0.02</td><td>-</td><td>0.03</td><td>-0.35</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>-0.24</td><td>-0.18</td><td>0.05</td><td>-</td><td>0.02</td><td>-0.18</td><td>0.04</td><td>-0.06</td><td>0.18</td><td>-</td><td>0.15</td><td>-0.37</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>-0.1</td><td>-0.16</td><td>0.02</td><td>-</td><td>0.11</td><td>-0.15</td><td>0</td><td>0.05</td><td>-0.06</td><td>-</td><td>0.03</td><td>-0.16</td>
  </tr>
  <tr>
    <td>FineSurE</td>
    <td>0.74*</td><td>0.68*</td><td>0.52*</td><td>0.75*</td><td>0.69*</td><td>0.64*</td><td>0.42*</td><td>0.60*</td><td>0.62*</td><td>0.70*</td><td>0.48*</td><td>0.69*</td>
  </tr>
  <tr>
    <td rowspan="5">Completeness</td>
    <td>ROUGE-1</td>
    <td>0.03</td><td>0</td><td>0</td><td>-</td><td>0.36</td><td>0.16</td><td>-0.23</td><td>0.2</td><td>0.33</td><td>-</td><td>0.59*</td><td>0.37</td>
  </tr>
  <tr>
    <td>ROUGE-2</td>
    <td>-0.16</td><td>0.24</td><td>0.11</td><td>-</td><td>0.53*</td><td>0.24</td><td>-0.3</td><td>0.22</td><td>0.28</td><td>-</td><td>0.39</td><td>0.28</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>-0.1</td><td>0.04</td><td>0.07</td><td>-</td><td>0.41*</td><td>0.07</td><td>-0.23</td><td>0.04</td><td>0.48*</td><td>-</td><td>0.41*</td><td>0.14</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>0.05</td><td>0.13</td><td>0.06</td><td>-</td><td>0.54*</td><td>0.2</td><td>-0.11</td><td>0.16</td><td>0.38</td><td>-</td><td>0.39</td><td>0.29</td>
  </tr>
  <tr>
    <td>FineSurE</td>
    <td>0.79*</td><td>0.75*</td><td>0.81*</td><td>0.90*</td><td>0.85*</td><td>0.80*</td><td>0.78*</td><td>0.66*</td><td>0.64*</td><td>0.90*</td><td>0.66*</td><td>0.75*</td>
  </tr>
  <tr>
    <td rowspan="5">Conciseness</td>
    <td>ROUGE-1</td>
    <td>0.51*</td><td>0.54*</td><td>0.23*</td><td>-</td><td>0.53*</td><td>0.55*</td><td>0.42*</td><td>0.58*</td><td>0.4*</td><td>-</td><td>0.66*</td><td>0.60*</td>
  </tr>
  <tr>
    <td>ROUGE-2</td>
    <td>0.41*</td><td>0.39*</td><td>0.04</td><td>-</td><td>0.44*</td><td>0.61*</td><td>0.22*</td><td>0.45*</td><td>-0.02</td><td>-</td><td>0.54*</td><td>0.72*</td>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>0.29*</td><td>0.41*</td><td>-0.01</td><td>-</td><td>0.43*</td><td>0.47*</td><td>0.13</td><td>0.4*</td><td>0.05</td><td>-</td><td>0.45*</td><td>0.52*</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>0.33*</td><td>0.42*</td><td>0.02</td><td>-</td><td>0.43*</td><td>0.6*</td><td>0.13</td><td>0.48*</td><td>-0.11</td><td>-</td><td>0.54*</td><td>0.65*</td>
  </tr>
  <tr>
    <td>FineSurE</td>
    <td>0.75*</td><td>0.68*</td><td>0.52*</td><td>0.75*</td><td>0.69*</td><td>0.64*</td><td>0.80*</td><td>0.70*</td><td>0.64*</td><td>0.82*</td><td>0.59*</td><td>0.74*</td>
  </tr>
</table>

Table 32: Correlation between similarity-based metric and human evaluation score. *: p-value <0.05

Reviewer paper summaries: This paper proposes MSumBench, a comprehensive multi-dimensional benchmark designed to improve evaluation frameworks for text summarization across multiple domains and languages. The authors introduce a bilingual benchmark (English and Chinese) enhanced by a multi-agent debate system to assist annotators. MSumBench evaluates summarization models using specialized criteria tailored to each domain. Experimental results show performance differences across domains and languages. The authors further analyze large language models as evaluators, revealing a strong correlation between their summarization and evaluation capabilities.
Summary of strengths: 1. The dataset was under comprehensive multi-step verification. The proposed dataset could become useful resources for fine-grained summarization systems training and evaluation. 

2. The dataset demonstrates valuable insights that LLM's evaluation accuracy highly various according to the domain, providing insights for the future research.
Summary of weaknesses: 1. The paper writing is a little unclear, the logic flow through each section are a little disjoint and often makes me confused about what's the relationship between this section and previous on. Such as the relationship between the multi-Agent Debate Framework and the dataset. 

2. There are lots of important details are presented in the Appendix, such as the dataset statistics (given dataset is the main contribution of the paper), and CV calculation.

3. The dataset sample is relatively small since only 150 documents are included.

4. The effectiveness of leveraging LLMs in the debate system is not fully explained. The paper introduction mentioned the LLM has hallucination problem while ignoring it while using it in the debating system. I was left with questions such as is there any incorrect debate occurred in the process and will this mislead the annotator?
Comments, suggestions and typos: Please refer to the previous section
Overall assessment scores: 3