Using Similarity to Evaluate Factual Consistency in Summaries
Authors: Yuxuan Ye, Edwin Simpson, Raul Santos-Rodriguez


Reviewer paper summaries: This paper revisits the use of similarity-based metrics to evaluate factuality in the text summarization task. This work proposes SBERTScore, a generalized BERTScore, and analyzes the granularity of the embeddings in factuality evaluation. Their experiments compare SBERTScore with popular NLI-based and QA-based metrics on factuality evaluation and show that SBERTScore achieves a competitive performance. In addition, their analysis shows that these metrics show low agreement and have different properties. Finally, motivated by this observation, they propose combining different metrics to improve performance.
Summary of strengths: Recent studies often assume that similarity-based metrics do not work well as factuality evaluators in text summarization tasks. Re-evaluating those metrics on this task is a unique and reasonable research direction that questions the widespread belief in this field.

This paper provides interesting experimental observations.

* Low agreement between different metrics for factuality evaluation
* Different metrics are good at detecting different types of errors
* Combining multiple metrics can improve the factuality evaluation
Summary of weaknesses: ### Theoretical Validity

I recommend providing more discussion and analysis on **why** similarity-based metrics work well as factuality evaluators. Specifically, I want to see a more careful and detailed analysis of the limitations (when they work and when they do not work).

This work experimentally shows that the similarity-based metrics are reasonable as factuality evaluators in text summarization. However, the similarity and factuality (context-faithfulness) are theoretically different by definition, as the authors also suggest in the limitation section (Line 609). Factuality in text summarization is identical to the entailment between the source and summary, so it is more theoretically reasonable to use NLI-based metrics, although it is possible to claim empirical advantages of using similarity-based metrics.

This paper should discuss the theoretical discrepancy (or validity) in using similarity-based metrics for evaluating factuality (e.g., what is mentioned in the limitation section) more clearly and in detail in the main part (e.g., introduction) to avoid confusing or misleading the readers.

Similarly, I think the analysis in Section 5.6 (i.e., for which errors are the similarity-based metrics effective) is one of the most important parts. I recommend providing a more detailed analysis.
Comments, suggestions and typos: I believe that further revisions with additional analysis would largely improve this paper. However, I think this paper already provides interesting observations and new perspectives in this field. I decided to put the overall score of 3.

### Experiments

Table 8 shows the recall of different types of errors or correct summaries, which is not a standard way to evaluate the metrics. For a more comprehensive analysis, I recommend providing precision and F1 as well in the appendix.

### Questions

* What kind of hypothesis tests do you use to claim p < 0.05?
Overall assessment scores: 3