Title: Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Summarization

Faithfulness metrics
Abstract: Modern LLMs can now produce highly readable abstractive summaries, to the point that traditional automated metrics for evaluating summary quality, such as ROUGE, have saturated. 
However, LLMs still sometimes introduce inaccuracies into summaries, i.e., information inconsistent with or unsupported by the corresponding source. 
Measuring the occurrence of these often subtle factual inconsistencies automatically has proved challenging. 
This in turn has motivated development of metrics intended to measure the factual consistency of generated summaries against sources. 
But are these approaches measuring what they purport to? Or are they mostly exploiting artifacts? 
In this work, we stress test a range of automatic factuality metrics—including specialized model-based approaches and LLM-based prompting methods—to probe what they actually capture. Using a shallow classifier to separate “easy” examples for factual evaluation—where surface features suffice—from “hard” cases requiring deeper reasoning, we find that all metrics show substantial performance drops on the latter.
Furthermore, some metrics are more sensitive to benign, fact-preserving edits than to factual corrections. Building on this observation, we demonstrate that most automatic factuality metrics can be gamed—that is, their scores can be artificially inflated by appending innocuous, content-free sentences to summaries. Among the metrics tested, the LLM prompt-based ChatGPT-DA approach is the most robust and reliable; however, it exhibits a notable caveat: it likely relies more on parametric knowledge than on the provided source when making judgments. Taken together, our findings call into question the reliability of current factuality metrics and prompt a broader reflection on what these metrics are truly measuring. We conclude with concrete recommendations for improving both benchmark design and metric robustness, particularly in light of their vulnerability to superficial manipulations.


Peer reviews:
comments: SUMMARY:
This paper evaluates the extent to which commonly used factuality (faithfulness) metrics genuinely reflect the factual consistency of generated outputs. It also benchmarks the robustness of these metrics against a range of perturbations. The authors find that many existing metrics are vulnerable to manipulation, and that ChatGPT-based metrics often rely more on parametric knowledge than on the provided source.

STRENGTHS AND WEAKNESSES:
**Strengths**

- The paper is well written and highly engaging.
    
- It addresses a timely and important problem in the evaluation of text generation.
    
- The claims are clearly supported by well-designed and rigorous experiments.


**Weaknesses**

- As acknowledged by the authors, Figure 4(b) presents weak evidence and is not fully convincing.
    
- The use of GPT-4 for generating edits, while pragmatic, could introduce bias given that one of the evaluated metrics is also based on GPT-4.
    
- The “easy / medium / hard” classification of summaries would benefit from additional empirical examples or clarification.
    
- A more detailed discussion of the human-annotated datasets used for metric evaluation could strengthen the presentation.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

comments: SUMMARY:
I appreciate the authors for running a pretty extensive set of comparisons of so-called factuality check methods along many different aspects, such as sensitivity to surface-level features, sensitivity to fact-invariant revisions and adversarial edits. An overall finding at the surface level (pun intended) is that none of the proposed approaches is as good as they were claimed to be originally. Although the authors constantly conclude that prompt-based metrics, such as ChatGPT-DA and MiniCheck, are reasonably good and better than the other metrics, my interpretation is that every metric has some weaknesses that make them not adequate to be used for checking the factuality of any textual statement. 

Despite this weakness (in the authors’ interpretation rather than their experimentation), I do believe it is good for the whole community to have this paper. This paper tells us about the limitations of existing factuality benchmarks (that said, I do not believe there could be a “standard” benchmark for factuality, since factuality is almost always in the eyes of beholders,) and that different metrics have different weaknesses and strengths. 

The only thing i miss from this submission is the discussion on factuality itself. Is factuality something we can build a benchmark and test language generators for? Should we approach it as a leaderboard chasing game, when factuality is not about the average behaviour? I would suggest the authors remove Fig. 4(b) (completely uninformative) and add one paragraph on this discussion.

STRENGTHS AND WEAKNESSES:
See my summary above.

QUESTIONS:
See my summary above.

LIMITATIONS:
See my summary above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper stress tests multiple factuality metrics to determine their in depth meaning. The authors finds that all metrics struggle in hard examples, many metrics are sensitive to benign edits, some metrics can be gamed by adding irrelevant phrase, and GPT relies on internal knowledge rather than the provided documents/context.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The evaluation includes multiple datasets and covers diverse domains.\n2. Gaming experiments effectively demonstrate the vulnerability of existing metrics to superficial manipulations which reveals important limitations in current evaluation approaches.\n\n**Weakness**\n1. Gaming idea is very impressive but I was wondering how this matrics varied the score so much given the phrases (\"the document discusses\", \"the summary entails.) which are very impractical. While the gaming results are impressive, the extreme score improvements (e.g., from 0.005 to 0.49 for MiniCheck) seem suspiciously large. The TF-IDF setup for identifying gaming phrases needs much clearer explanation. How exactly were these phrases selected, and why do they produce such dramatic effects?\n2. The paper uses a simple MLP with basic features to categorize \"easy\" vs \"hard\" examples. This classifier achieves modest performance, raising questions about whether the difficulty categorization is meaningful.\n3.No code is provided, making it extremely difficult to understand the experimental setup. This significantly undermines the paper's impact and verifiability as it is hard to understand the all the setup from given prompts only. Many experimental procedures lack sufficient detail for replication.\n4. Line 155 describes the difficulty categorization in a very confusing manner.\n5. The relationship between prediction accuracy, confidence, and difficulty labels is unclear.\n6. The shallow MLP baseline methodology needs better explanation.\n7. KeyBERT is mentioned but its usage context is unclear.\n8. Appendix B mentions \"conciseness ratio\" but doesn't define it clearly.\n9. Typos\n\n- Figure 1 upper right: \"me\" should be \"m̂\" (m hat)\n- Line 211: \"Tu\" should be \"To\"\n- Line 554: \"GPT-40\" should be \"GPT-4o\"\n- Line 561: \"summer\" should be \"summary\"\n- Line 92: Missing full stop\n- Line 122: Awkward phrasing\n- Line 560: Missing full stop after \"Table 5\"\n- Line 575: Missing full stop\n\nQUESTIONS:\n1. Do the gaming strategies work across different domains and text types, or are they specific to the datasets tested? \n2. Are the reported differences between metrics statistically significant? The paper mentions confidence intervals in Section 5.2 but statistical testing across all experiments is unclear.\n3. I would like to see the explanation for the questions raised in the weakness part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive evaluation of model-based factuality evals, focusing on whether a model-generated summary is grounded in the source document. The paper presents a series of experiments with interesting findings, e.g., that the correctness of these evals correlates strongly with difficulty levels defined by spurious features (e.g., word overlap with the source document), that model-based evals are sensitive to benign modifications while being insensitive to meaningful modifications (e.g. contradictions), that LLM-based evals rely heavily on their internal knowledge rather than the source document, and that these LLM-based evals are gameable, e.g., by adding a sentence like “The summary entails the information the document discusses”.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper evaluates a wide range of factuality evals and includes diverse summarization benchmarks, capturing a broad range of error types and settings.\n- The idea of using spurious features like word overlap and entity overlap to define and categorize difficult levels is useful.\n- The analysis with different types of perturbations such as corrections and paraphrases is insightful.\n- I found the recommendation at the end of the paper particularly insightful, such as cautions against using model-based evals for text that is counterfactual or does not align with LLMs’ internal knowledge such as myths, misinformation, or uncommon facts.\n\nWeaknesses\n\nI do not believe there are particular flaws in the paper. I do have some reservations about certain conclusions drawn from the experiments. For instance:\n- For the first experiment in Section 5.2, I do not believe that little gap between evaluations with and without access to the source document necessarily indicates that the model relies heavily on parametric knowledge. If the content of the source document is already aligned with a model’s internal knowledge, the gap may naturally be small, even when the evaluator does ground to the source document when it has access to it.\n- The second experiment in Section 5.2 is more compelling. However, the nature of counterfactuals in the modified text can affect the result, e.g., text that is intentionally written to be contradictory to the original text is likely to be highly unrealistic and unnatural. It would have been more interesting if the text is realistically counterfactual, e.g., myth, or text about very recent events that the LLMs would not have internal knowledge about.\n- Section 6 feels less compelling to me. There’re always ways to make adversarial examples to fool the evaluator, e.g., adding a sentence “The summary entails the information the document discusses” looks like a type of a jailbreaking attack. However, this is unlikely to be actually generated by a reasonable summarization model. If a takeaway is that a summarization model shouldn’t be trained with these metrics (e.g., as a reward model) because it’s too vulnerable to reward hacking, I think it is reasonable. However, assuming that the summarization model is not directly exploiting this metric, it is unclear how relevant this finding is in real world scenarios.\n\nI also think the overall findings of the paper are, while solid, not particularly surprising or new. It is already well understood that model based evaluators are far from perfect, but the reason we still use them is because they are the only practical & cost-efficient alternative to human evaluation. And I do not believe that this paper is offering actionable strategies for improving these model-based evaluations.\n\nI particularly appreciated the recommendation to exercise caution when applying model-based factuality evals in text involving myths, misinformation, or uncommon facts, and think it would have been significantly better if the paper had directly demonstrated this issue. This paper took a different route by making various synthetic perturbations, which is valuable, but I think this always raises questions on how well these findings on synthetic text generalize to real-world text.\n\nMinor questions/suggestions:\n- The categorization of factuality evaluation methods into QA-based, NLI-based, and LLM-based does not seem to be most informative in my opinion, e.g., QA vs. NLI is a matter of a format choice rather than a fundamental methodological difference. How about prompt-based vs. fine-tuning based (which could either be QA or NLI)?\n- Figure 2: The scales across three graphs appear inconsistent, which makes visual comparison difficult. In addition, I think it might be more effective to group results by eval methods and compare performance across difficulty levels within each group, rather than grouping by difficult levels and comparing across eval methods.\n\nQUESTIONS:\nWould appreciate authors' thoughts on the weaknesses in the previous field.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}