EVA-Score: Evaluation of Long-form Summarization on Informativeness through Extraction and Validation
Authors: Yuchen Fan, Xin Zhong, Chengsi Wang, Gaochen Wu, Bowen Zhou


Reviewer paper summaries: This paper introduces an evaluation metric named EVA-SCORE for long-form summarization. EVA-SCORE combines Atomic Fact Chain Generation (AFCG) and Document-level Relation Extraction (DocRE) to generate an objective and quantitative informativeness score. The paper claims that EVA-SCORE shows a better correlation with human judgment compared to traditional metrics like ROUGE and BERTScore.
Summary of strengths: 1. The combination of AFCG and DocRE to capture both atomic-level facts and hierarchical document-level relations is well thought out.
2. The authors conduct some experiments on various datasets and with multiple models, enhancing the robustness of their claims.
3. The effort to ensure better alignment with human judgment through correlation studies is commendable.
Summary of weaknesses: 1. The definitions and procedures for Atomic Fact Chain Generation and Document-level Relation Extraction lack clarity. The distinction between sentence-level and document-level relations is not well articulated, and the pipeline for verifying factual correctness remains vague.
2. The validation experiments reveal inconsistencies, such as negative correlations for some models at the system level (e.g., Mixtral-7B-Instruct showing a negative correlation in Pearson). This variability suggests that the metric might not be robust across different settings and models.
3. The statistical tests conducted (Pearson, Spearman, and Kendall correlations) are standard but lack depth. Further analysis, such as error analysis or qualitative assessment, is missing.
4. The comparison between EVA-Score and other metrics seems cherry-picked and lacks fairness. For instance, evaluating the metrics on datasets truncated to fit model input lengths introduces biases. Moreover, the significance of ROUGE and BERTScore\'s performance drops in longer contexts needs a more nuanced discussion.
5. Although the methodology section is detailed, essential computational specifics, such as hyperparameters and other experimental settings, are omitted. This hinders reproducibility.
6. While human annotation is used as a benchmark, the discussion on inter-annotator agreement, biases, and quality control is insufficient. This weakens the claim of EVA-Score’s human alignment.
Comments, suggestions and typos: 1. Section 4.1 & 4.2 (Atomic Fact Generation): The differentiation between atomic facts and cascaded facts is central to the methodology but explained ambiguously. More concrete examples and clear definitions would improve clarity.
2. Human Annotations: There should be an in-depth discussion about inter-annotator agreement scores to validate the consistency of human evaluations.
Overall assessment scores: 2.5