Re-Examining Summarization Evaluation across Multiple Quality Criteria
Authors: Ori Ernst
Ori Shapira
Ido Dagan
Ran Levy
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
evaluation
summarization evaluation
confounding variable
spurious correlation
confounding factor

Abstract: The common practice for assessing automatic evaluation metrics is to measure the correlation between their induced system rankings and those obtained by reliable human evaluation, where a higher correlation indicates a better metric. 
Yet, an intricate setting arises when an NLP task is evaluated by multiple Quality Criteria (QCs), like for text summarization where prominent criteria including relevance, consistency, fluency and coherence.
In this paper, we challenge the soundness of this methodology when multiple QCs are involved, concretely for the summarization case. 
First, we show that the allegedly best metrics for certain QCs actually do not perform well, failing to detect even drastic summary corruptions with respect to the considered QC.
To explain this, we show that some of the high correlations obtained in the multi-QC setup are spurious.
Finally, we propose a procedure that may help detecting this effect.
Overall, our findings highlight the need for further investigating metric evaluation methodologies for the multiple-QC case.

Re- Re-Examining Summarization Evaluation across Multiple Quality Criteria
Ori Ernst" 1.2 Ori Shapira', Ido Dagan', and Ran Levy'
Amazon llan-Ilan University
oriern@gmail com
{orishap, ranlevy)@amazon... com daganêcs biu. ac.
Abstract
Metric
Coherence
Consistency
Fluency
Relevance
ROUGE-3
0.2206
0.7059
0,5092
0.3529
The common practice for assessing automatic evaluation metrics is to measure the correla- tion between their induced system rankings and those obtained by reliable human evaluation, where higher correlation indicates better metric. Yet, intricate setting arises when NI.P task is evaluated by multiple Quality Cri- teria (QCs), like for text summarization where prominent criteria include relevance, consis- tency, fluency and coherence. In this paper, we challenge the soundness of this methodology when multiple QCs are involved, concretely for the summarization case. First, we show that allegedly best metrics for certain QCs actually do not perform well, failing to detect even dras- tie summary corruptions with respect the con- sidered QC. To explain this, we show that some of the high correlations obtained in the multi- QC setup are spurious. Finally, we propose procedure that may help detect this effect. Over- all, our findings highlight the need for further investigating metric evaluation methodologies for the multiple-QU case.
CHRF
0.3971
0.5294
0.4649
0.5882
METEOR
0,2353
0,6324
0,6126
0.4265
Table I: Correlations (Kendall's r) between metric and
human scores for each QC, as taken from Fabbri et al. (2021). Only the best metric in each QC is presented.
are most suitable for evaluating each respective criterion.
To enable such metric evaluation procedure, some annotation efforts have been conducted (c.g.- Bhandari et al., 2020; Fabbri et al., 2021), where system summaries were manually scored accord- ing to several QCs. This yielded a "gold ranking" of systems per QC, to which metric rankings can be compared. The recent SummEval benchmark (Fabbri et al., 2021) has attracted interest due to its thorough data collection of 1,700 annotated system summaries over 17 systems. Overall, summaries were rated over 4 different QCs: Relevance, Con- sistency, Fluency and Coherence (clarified in $2). Within this benchmark, the majority of the evalu- ated metrics were primarily intended to assess Rel- evance, with smaller subset comprising statistical analysis-based metrics such summary length, which were not specifically designed measure any of the four aforementioned QCs. In our work, we focus on the major group of metrics that were designed to measure Relevance. Table presents correlations between the best evaluation metrics in each QC and the SummEval human scores. The full table can be found in Table 6 in Appendix A.1.
1 Introduction
The broad interest in text generation has triggered significant progress in the development of respec- tive automatic evaluation metrics, complementing costly manual evaluation. In order to assess and rank various automatic metrics, the conventional setup compares the ranking of various systems ac- cording to metric scores and human scores, where a high correlation between them indicates an effec- tive metric. In some cases, a task requires evalua- tion by multiple quality criteria (QCs), as is promi- nently the case for text summarization. where com- mon criteria include coherence. fluency, faithful- ness and saliency. In such cases, comparing metric scores to human scores separately for each crite- rion should supposedly determine which metrics Work was done as an intern at Amazon.
Surprisingly, closer examination of metric per- formances in SummEval reveals that many of the high correlations are unlikely imply high fitness of the best performing metrics and the respective QC. For example, ROUGE-1 (Lin, 2004), which measures the lexical match of unigrams between the system and reference summaries, appears to

---

perform well as a fluency metric. Fluency is typi- cally not determined based on unigrams alone and should not depend on the reference summary. This raises questions such as (1) what was the cause for these high correlations, and (2) when can correla- tions be trusted.
measuring the quality of the collective structure of the sentences in the summary. A summarization system is rated for each QC by averaging human
scores (1 to scale) over all input instances. Thus, multiple systems can be scored and ranked sepa- rately for each QC in accordance with the respec- tive human scores.
In this paper we address these two questions with thorough analyses providing two contributions. First, we point out that the conventional scheme for metric ranking is problematic in the case of multiple quality criteria. To achieve this we discov- ered that even the best metrics for each QC fail to penalize system summaries containing corruptions that should adversely affect that QC. This suggests that these metrics ineffective in measuring the intended QC. In addition, we show that there are spurious correlations a multi-QC setup like sum- mEval, caused by high performance correlations across criteria. Second, we suggest a method for detecting metric-to-human correlation scores that are suspected as spurious, by removing the effect of confounding variable, which reveals perfor- mance degradation in many cases. This provides first step cope with this obstacle, while call- ing for further research on metric evaluation in the multi-QC case.
To assess the effectiveness of an automatic eval- uation metric, first applied on the outputs of several summarization systems, yielding a respec- tive score per system (averaged over all instances). The metric performance for a certain QC is then measured as the correlation between these metric scores and the corresponding human scores for that QC. The described veta-evaluation procedure ables rank multiple evaluation metries per each QC separately, as illustrated in Table 1. In our work, we question whether highly ranked metries on a particular QC are indeed suitable for evalu- ating that QC, claiming that such correlation to human scores are not necessarily indicative in the multi-QC setting.
Some studies examined other aspects of this meta-cvaluation scheme. Peyrard (2019) and Bhan- dari et al. (2020) showed that meta-evaluation us- ing old systems or datasets, like those of DUC and TAC, yield erroncous trends on modern summariza- tion benchmarks. Deutsch et al. (2021) investigated the preciseness of correlations between metries and human annotations in meta-evaluation benchmarks, and proposed approaches to improve the level of confidence. Finally, Deutsch al. (2022) discussed ways of improving the reliability of system-level correlations in meta-evaluation.
2 Background and Related Work
A high-quality summary should satisfy several re- quirements, including preserving the salient infor- mation of the source, being faithful to the source, and being coherent. The DUC benchmarks (NIST, 2002) made an early attempt at capturing these re- quirements with human evaluation procedure that assessed several readability qualities as well as con- tent responsiveness (Dang, 2005). Later, additional benchmarks focused on content quality (e.g.. Bhan- dari et al. (2020), TAC (NIST, 2008)) or linguistic quality (Chaganty et al., 2018) separately. Kryscin- ski et al. (2019) laid out four Quality Criteria (QCs) on which to assess summary quality, which were reinforced in the SummEval summary evaluation benchmark (Fabbri et al., 2021), annotated over the Cal/ (Nallapati et al., 2016) summariza- tion dataset.
3 Metric Robustness to Summary Corruptions
In this section, we aim investigate whether a met- ric that was found to strongly correlate with human scores for a certain QC according SummEval reliably measures this specific QC. Specifically, we suggest that the performance of metrics - that were designed to measure Relevance on other QCs (e.g., using ROUGE to measure Fluency) may be questionable. To examine this, we artificialyy corrupt SummEval (Fabbri et al., 2021) system summaries in various forms, each specifically de- signed to degrade single QC. The corruption is expected to have a large impact on the correspond- ing QC measurements, while having minimal effect on other QCs. Accordingly, we may conclude that
Described briefly, the four QCs are: Relevance, measuring the importance of summary content with respect the source; Consistency, measuring the faithfulness of the summary content to the source; Fluency, measuring the linguistic quality of indi- vidual sentences in the summary; and Coherence,

---

metric that fails to penalize corrupted summaries does not actually measure the specific QC for which corruption was introduced.
Anchoo
Coherence
Consistency
Fluenc
Relevance
Coherenne
1.00
28 24/.11
A8.1 25 .6
60 801.53/ 22
Consisioncy
28/ 24/
1,00
601
Flenny
A87 25 08
807 367.18
1.00
61 61/ 7.000
Relevance
5997
A 25 /66
1.100
In what follows, we experiment with the fol- lowing corruptions of the system summaries with respect to each QC (except for Relevance which is the QC most metries were designed to capture). Fluency: All verbs are replaced with their lemma form, resulting in ungrammatical sentences. Co- herence: Sentences are randomly shuffled to dis- rupt the structure of the summary. This corruption is inspired by the Shuffle Test (Barzilay and La- pata, 2008) used to evaluate whether models can detect incoherent text. Consistency: All PERSON named entities (using SpaCy NER, Honnibal et al., 2020) are replaced with different PERSONs from the source document. This is in fact common factual mistake of models (Pagnoni et al., 2021). An example for each corruption type can be found in Table in the Appendix.
Table 2: Kendall's correlations between human QC rat- ings on the hwwmm//ewee//nttan-eeeee///eeeee columns serve as anchors solely for bucketing. Apply- ing bucketing diminishes correlation between QCs.
The
is caused by spurious correlations, and propose possible means to identify them.
Analysis of Spurious Correlations
A possible explanation for the contradicting find- ings that "high performing" metrics fail to penal- ize corrupted summaries, is the high correlations observed between human scores of different QCs (termed correlationhuman) as seen in system-level scores in Table 2 (left-most figure in each cell). As result, high correlations between metrics and hu- man scores on all QCs (termed correlationmenie) are usually due to the high correlation with one confounding QC combined with the strong correlationumaan between this confounding QC and the remaining QCs. As all of the aforemen- tioned best metrics were initially designed to mea- sure Relevance, we conjecture that the Relevance QC acts as a main confounding variable, making other correlations.murii spurious. Although spuri- ous correlationSmetiic can be found in different se- tups, they more likely in multi-QC setups, like summarization. In such setup, models are typ- ically optimized to excel in all QCs, resulting enhanced systems across all QCs that consequently yield high correlationspmman among these QCs.
To test the meta-evaluation of metrics presented in SummEval, we examine the sensitivity of the best metric for each QC, according to SummlEval results (from Table 1), to the QC-specific corrup- tion. More specifically, for each QC, we ranked all systems with the best metric, corrupted each system in turn, and examined whether that sys- tem's ranking was subsequently downgraded be- low other systems that were initially scored signifi- cantly lower. We found that none of the corrupted systems were downgraded in such relative rank- ings with the Coherence and Fluency corruptions, while only two (out of 17) were downgraded with the Consistency corruption. Assuming our drastic corruptions should have caused a ranking down- grade, these results indicate that the top performing metries for Coherence, Fluency and Consistency were mostly unable to penalize the corrupted sys- tem summaries, suggesting they are not sufficiently reliable at measuring these QCs.
Next, we suggest systematic method to detect a confounding QC that undermines correlationsmerrie with other QCs. To that end, we propose to remove the effect of each QC, in turn, on the remaining QCs by annulling the correlationspuman between them. To do this, we calculated correlationsmerric over small subsets (i.c., buckets) of instances which the annulled QC has low variance. We show that for most metrics when the correlationuman to Relevance is annulled, the correlationsietrie to Flu- ency, Coherence and Consistency drops drastically, while the correlationsmetio to Relevance is immune to similar manipulations of the other three QCs. This suggests that Relevance is indeed confound- ing factor for the other QCs. To compute correlationsmetic within buck-
To validate the aforementioned assumption, that our corruptions should actually cause a ranking downgrade, we manually annotated a sample of 9 systems that contain corrupted summaries and found that all system rankings were downgraded after corruption. (More details can be found in Appendix B.) Since the best automatic metric did not reflect this change in ranking, we conjecture that Svl meta-evaluation scores, even when they appear to be high, are not reliable. In the next section, we investigate the possibility that this

---

ets we first note that the original Summläval correlationsserric were calculated the system level. That is, the 100 instance scores per system are averaged to produce 17 data points for correla- tion. However, dividing these 17 data points into buckets results in statistically unreliable correlation scores inside each bucket due to the limited number of data points in it. To address this, we utilized the instance-level correlation approach (Bojar al., 2017; Freitag et al., 2021), which incorporates all 1,700 instance scores as individual data points with- out averaging per-system. Dividing the entire 1,700 instances into buckets ensures a sufficient number of data points within each bucket. Since confound- ing factors are inherent to the data, identifying a confounding variable at the instance-level implies its presence at the system level as well.
correlationmetric based on relative difference, this process does alert for potential unreliability when the relative difference relatively high.
The relative difference scores by cach one of the anchor QCs and each metric are shown in Ap- pendix A.3. To summarize these detailed findings, focused on the majority of metries that were de- signed to measure Relevance. For each anchor and evaluated QC, we computed the median relative difference of all metrics. As can be seen in Table 3, we observe that the largest relative differences occur when Relevance serves as the anchor QC, as presented in the blue column. This means that the original correlationsmetic to Coherence, Fluency and Consistency were strongly affected by Rele- vance, strong confounding variable. However, when Relevance serves as the evaluated QC, as demonstrated in the yellow row, the relative dif- ferences are quite low, regardless of the anchor QC. This means that other QCs are probably not confounding variables for Relevance.
Inspired by stratification analysis (Mantel and Haenszel, 1959), in order to remove the effect of a particular QC (termed anchor QC) to assess its potential to be confounding factor, we divide the system summary instances, which are associated with human score tuples of <Relevance, Consis- tency, Fluency, Coherences, into buckets. Each bucket contains tuples with roughly equal scores of the anchor QC. Since scores are on 1 1-to-5 scale, we use 5 buckets. As example, if we would like to anchor Relevance, the first bucket will con- tain tuples with Relevance x Accordingly, the correlation inside each bucket between the anchor QC human scores and each other QC de- grades substantially. As Table 2 shows, averaging these low correlationspuman over all 5 buckets and weighting by bucket size, results in "bucketing" value in each cell that reduces the initial instance- level correlationsjuman between QCs by 2-5 times.
We also used the bucketing analysis to evalu- ate two other metric groups that roughly estimate other QCs, and observed the same phenomenon. The first group contains metrics that measure the percentage of repeated n-grams in the system sum- mary. As a summary with repeated information is less coherent, these metrics are more suitable as rough estimates for Coherence. Accordingly, Table 4 shows high relative differences when Co- herence functions the anchor (marked blue column), meaning that when neutralizing Coher- ence, the correlationSmetie to other QCs change dramatically. On the other hand, when other QCs function as anchors, the correlationmetric with Co- herence is almost unchanged. This is expressed by the low relative difference (marked as the yellow row). Overall, this analysis suggests that Coher- ence is the confounding factor of this metrie group and the original correlationsmette are spurious.
Next, we used this approach to calculate the correlationsseticc inside each bucket, thus neutral- izing the effect of the anchor QC. Again, the five bucket correlationsmerric are averaged and weighted by bucket size. Finally, to measure whether the obtained bucketing value has changed signif- icantly with respect to the original instance-level correlationmetie- we calculate the absolute relative difference between the two scores. A high rela- tive difference means that the correlationmeticc has changed significantly after removing the anchor QC. This undermines the reliability of the original correlation scores, and suggests the anchor QC as a confounding factor. While our work does not provide a specific threshold for detecting spurious
The second group contains metrics measuring the percentage of novel n-grams in the summary that are not found in the input document. These metrics capture abstractiveness, making them po- tentially useful as rough (negative) estimates for Consistency and Fluency. This is duc to the capu- bility of these metrics to identify extractive sum- maries, which inherently possess consistency and fluency. Accordingly, we show the same phe- nomenon in Table 5 where bucketing by Consis- tency or Fluency as anchors yields high relative

---

Anchor
Coberence
Consistency
Flency
Relevance
Anchor
Coherence
Consistaccy
Flaency
Relevunce
QC
Coherence
NAA
.0 211...2
2.1.4424)
.82 (,74.94)
Coherence
.78 (75--100
821778-1877
Crnsisioncy
(:5--70)
Consistency
05 (13-009
N7A
22 (22 251
11 (00- 12)
Plaeney
501,00 59)
52 (43-72)
NIA
72 (44-4 85)
Fluency
NIA
Relesance
(33-22))
N/A
Relevance
(,4076))
96 (85-1.88)
NIA
Table 3: Median (quantile 25- quantille 75) of the abso-
Table 5: Median (quantile 25 35-quantile 75) of the abso- lute relative difference between original correlation and bucketing correlation, metrics that roughly mea- sure Consistency and Fluency.
lute relative difference between original correlation and bucketing correlation, over metrics that were designed to measure Relevance.
Anchor
Coherence
Consistency
Fluency
Relevanoe
Conclusion
OC
Coherrence
(19.)))
J8
14
Consislancy
79 (,748999)
NIA
58 (.5--88)
38 (:35.44)
We challenged the conventional manner of evaluat- ing and ranking summarization metrics according to correlation with human scores in the multi-QC setting. We found that human ratings over recent state-of-the-arr systems tend to correlate between different QCs, which leads to unreliable metric per- formance scores. To demonstrate this, we showed that metrics that allegedly measure a QC, negligibly decrease their score for corrupted summaries with respect to this QC. To cope with this obstacle, we proposed bucketing method that removes the ef- fect of the confounding variable and detects unreli- able correlations. While this work mostly highlights the problem, we strongly encourage the develop- ment additional approaches to tackle it.
Flueney
(71--7))
(95-49)
155 (.14.2))
Relevunce
(.72-1.00
31 (23-58)
37 (333-48)
N/A
Table 4: Median (quantile 25-quantile 75) of the abso- lute relative difference between original correlation and bucketing correlation, over metrics that roughly mea- Coherence.
differences (blue columns), while bucketing by other QCs leaves low relative differences to Con- sistency and Fluency (yellow rows). As in this case we found two confounding factors, we conjecture that there is another unmeasured human QC that assesses abstraciveness directly that eventually in- fluences Consistency and Fluency. In such case, this abstraciveness QC therefore functions con- founding factor.
Acknowledgments
Overall, this analysis points out the problem- atic conventional scheme of metric evaluation in multi-QE QC setting. We found that except for the QC that the metrics were designed to measure, most of the correlationsmetric to other QCs are spurious. Fur- ther exploration of adjusting this analysis for other scenarios, such as cases involving two confounding factors, is left future work.
We would like thank Rotem Dror, Amir Feder and the puper reviewers for their comments, and Alex Fabbri for his support in reconstructing the SummEval results. The work described herein was supported in part by the Katz Fellowship for Excel- lent PhD Candidates in Natural and Exact Sciences and by the Israel Science Foundation (grant 2827/21).
It is worth noting that spurious correlations can alternatively be detected with the partial correla- tion approach (Whittaker, 2009). The confounding variable is neutralized by calculating the correlation between the residuals resulting from the linear re- gression of each of the variables to the confounding variable. The partial correlation scores in our set- ting indeed display the same trend as our bucketing method, detecting Relevance as the confounding variable in most metrics. In contrast to the partial correlation approach, bucketing has the advantages of being more interpretable and assuming linear dependency between variables. See Appendix C for an analysis and comparative discussion between the methods.
Limitations
This study highlights phenomenon that occurs when assessing summarization metrics across vary- ing quality criteria. The findings are empirically shown only on SummEval, which a relatively large-scale and high-quality meta-evaluation bench- mark. Furthermore, there do not exist other ma- jor benchmarks that would enable similar anal- ysis. Nevertheless, the findings would be further strengthened if they could be examined addi- tional benchmarks.
Additionally, although our analysis offers strong empirical evidence that the Relevance QC the confounding variable in most metrics in the Sum-

---

mlival setting, there could be other external factors that cause the strong correlations among the QCs.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi kiu Lo. Craig Alan Stewart, George Foster, Alon Lavie, and Ondrej Bojar. 2021. Results of the wmi21 I metries shared task: Evaluating metries with expert-based human evaluations on ted and news domain. In Con- ference on Machine Translation.
We also rely, to a certain degree, on logical in- tuition and understanding of the proposed metrics in order to convince the reader of our findings. For example, it is very reasonable to assume that cer- tain summarization metrics do not actually have the ability measure a specific QC. In the case of ROUGE-I, there should not be a true relationship between the number of overlapping unigrams with another text and the Fluency of the evaluated text. Any corresponding chance correlation is presum- ably not due to direct intent of the metric.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Wojeiech Kryseinski, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural Text Summarization: A Critical Evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLP-JJCNL pages 540-551, Hong Kong. China. Association for Computational Linguis- tics.
References
Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Compu- tational Linguistics, 34:1-34.
Chin- Yew Lin. 2004, ROUGE: A Package for Auto- matic Evaluation Summaries. In Text Summariza- rion Branches Out, pages 74-81, 1, Barcelona, Spain. Association for Computational Linguisties.
Manik Bhandari, Pranav Narayan Gour, Atabak Ash- faq, Pengfei Liu, and Graham Neubig. 2020. Re- evaluating Evaluation in Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online. Association for Computa- tional Linguisties.
Nathan Mantel and William M. Haenszel. 1959. Statis- tical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22 4:719-48.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang, 2016. Abstrac- tive Text Summarization using Sequence-to-sequence RNNs and Beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguisties.
Ondrej Bojar, Yvette Graham, and Amir Kamran, 2017. Results of the wmt17 metrics shared task. In Confer- ence on Machine Translation.
Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metries in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 643-653, Melbourne, Australia. Association for Computational Linguistics.
NIST. 2002. Document understanding conferences. https://duc, http://ddu.nis..o... novt..gv/. Accessed: 2023-06-01.
NIST. 2008. Text analysis conference. https://tac. nist.gov/. Accessed: 2023-06-01.
Hoa Trang Dang. 2005. Overview of DUC 2005. In Proceedings of the Document Understanding Confer- ence, volume 2005, pages 1-12.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding Factuality in Ab- stractive Summarization with FRANK: A Benchmark for Factuality Metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 48 18-2-482, Online. Association for Computational Linguistics.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. Statistical Analysis of Summarization Evaluation Metries Using Resampling Methods. Transactions of the Association for Computational Linguistics, 9:1132-1146.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Re- Examining System-Level Correlations of Automatic Summarization Evaluation Metrics.
Maxime Peyrard. 2019. Studying summarization eval- uation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5093- 5100, Florence, Italy. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Kryšeiñsski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev, 2021. SummEval: Re-evaluating Summariza- tion Evaluation. Transactions of the Association for Computational Linguistics, 9:391 2009.
Joe Whittaker. 2009. Graphical models in applied mul- livariate statistics. Wiley Publishing.

---

A Experiment Technicalities
Metric
Coherence
Consistency
Fluency
Relevance
ROUGE-1
0,2500
0.5294
0.5240
0.4118
A.1 Running Systems and Metrics
ROUGE-2
0.1618
0.5882
0.4797
0.2941
ROUGE-3
0,2206
0.7059
0.5092
0.3529
All of our experiments were based on the re- sources provided by the SummEval benchmark (in the GitHub repository at https://github. com/ Gale-LILY/SumEEall]] System summaries and hu- man scores were present in the GitHub repository. We ran the metrics on summaries using the code provided the repository, with a few minor adapta- tions. The original correlation table of SummEval is presented in Table 6.
ROUGE-4
0.5882
0.5535
0,4118
ROUGE-I.
0.0735
0.1471
0.2583
0.2353
ROUGE-Su
0.1912
0,2941
0.4354
0,3235
ROUGE w
0,0000
0.3971
0.3764
0.1618
ROUGB-we-I
0.2647
0.4589
0.5092
0,4265
ROUGE-we-2
0.0147
0.5000
0,3026
0.1176
we
0.3676
0.3026
0.1912
SP-pyr
400294
0,5147
0.3173
0.1324
Se-resp
0.0147
0.5000
0.3321
0.1471
BertScore-p
0,0588
-0.1112
0,0074
0.1618
BertScore-t
0.1471
0.6618
0.4945
0,3088
BertScore
0,2059
0.0441
0,2435
0.4265
MoverScore
0.1912
-0.0294
0,2583
0.2941
0.1618
0,5588
0.3616
0.2353
A.2 Corruptions
BL
0.1176
0.0735
0.3521
0,2206
CHRF
0.3971
0,5294
0,4649
0,5882
An example of all corruption types is presented in Table 9.
CIDE
0.1176
0.1912
0.0221
0.1912
METEOR
0.2353
0.6324
0.6126
0.4265
SummaQA"
0.1176
0.6029
0,4059
0.2206
A.3 Bucketing
BLANC
0,0735
0.5588
0,3616
0.2647
SuPERT
0.1029
0,5882
0,4207
0,2353
To map the 1-5 scores into 5 buckets we rounded the scores to the nearest integer. All metrics' buck- eting correlations for each anchor are presented in Tables 10, 11, 12, 13.
Novel unigram"
0.1471
-0.2206
-0,1402
O. 10299
Novel hi- gram"
0,0294
-0.5411
003469
-0.1029
Novel tri-gram"
0.0294
-0.5735
-0.3469
-0.1324
Repeated unigram"
0.3824
0.1029
0.0664
003676
Repeated hi-gram"
-0.324
-0.0147
-0.4559
Repeated tri-gram" -0.2206
0.1471
-0.0221
-0.2647
To compute the absolute relative difference be- tween the original correlationSmettie and the buck- eting correlationsmerrie- we first computed the ab- solute values of each correlation score. This al- lowed us to assess the metric's ability to capture the human scores, whether in a positive or negative relationship. Next, we calculated the absolute dif- ference between the two correlation values. A high absolute difference indicates significant modifi- cation in the original correlation after bucketing, either in an upward or downward direction, high- lighting the unreliability and spurious nature of the original correlation. While the majority of cases showed a positive difference, indicating that the original correlation was higher than the bucketing correlation, there were rare instances where the dif- ference was negative. A negative difference implies that the original correlation was initially low but experienced a significant increase after bucketing.
Table 6: SummEval Kendall's correlations between metrics and human annotations for each QC (taken from (Fabbri al., 2021)). ^ denotes reference-fres metrics, The most-correlated metric in each column is in bold.
metric as described in Section 3. These systems were then annotated against sampled lower-ranked systems, ensuring ranking difference of at least 6 places (above one-third of the number of systems) based on the best automatic metric. Additionally, confirmed that the corrupted systems, when not corrupted, achieved higher rankings compared the lower-ranked systems according to summEval manual annotation. Finally, for each QC, we anno- tated pairs of systems, consisting of one corrupted system (corrupted with the QC-specific corruption) and one lower-ranked uncorrupted system.
During the annotation process, we compared each system pair in terms of the relevant QC at the instance level, and aggregated the results to identify the system with more preferred instances. Specifically, for specific pair of systems, the anno- tator gets two system summaries (one per system) of the same document, and should select the best summary in terms of the corrupted QC. After anno- tating all pairs, the system with the most preferred summaries is considered better than the other sys- tem.
B Manual Annotation for Corrupted Systems
In Section 3, we aim to validate the assumption that substantially corrupted systems should be pe- nalized in human ranking. To support this claim, we conducted manual annotation process on ran- domly selected subset of 20 documents from a total of 100. Specifically, for each QC, we chose three corrupted systems (with 20 documents) that were not identified for degradation by the best automatic
For the annotation of Coherence, one of the au-

---

QC
Coherence
Consistency
Fluency
Methodologically,. we identified two advantages of the bucketing approach in comparison par- tial correlation. First, bucketing arranges the data into sets (buckets), where the confounding variable is neutralized in each set. This allows for further interpretability and analysis of the data, e.g., exam- ining correlations and other statistics within each set. Mcanwhile the partial correlation method only provides final score without any further interpre- tation of the score of the data. In other words, while provides debiased correlation, the bucketing method provides debiased data. Second, partial cor- relation is based on linear regression between the assessed variables and the confounding variable, and hence assumes potentially linear correlation between these variables. Bucketing, on the other hand, generally allows the neutralization of any type of spurious correlation, making this method more robust for future extensions (even if in the current work we measured only linear correlations within each bucket).
Sys Pair
65
60
100
1I
80
50
100
75
100
Mean
73
62
100
Table 7: Percent of the uncorrupted system summaries that were manually preferred in each pair of systems, Notice that the system pairs are different for each QC, therefore the columns are not comparable.
Anchor
Coberence
Consistoncy
Fluency
Relevanec
OC
Coherenze
MA
21 (T6-24)
11 (09-13)
87 (63-90)
Consistency
(819288)
Pleency
43 (,36
85 (40.80)
N6A
.33 (35. 86)
Relevance
19 301
(10.14)
(03-000
Table 8: Median (quantile 25- 25-quantile 75) of the abso- lute relative difference between original correlation and partial correlation, over metrics that were designed to measure Relevance.
thors annotated all 3 system pairs. However, since the Fluency and Consistency corruptions (lemma- tizing all verbs and replacing all PERSON entities with others) can be casily noticcable by the authors of the paper, we used Amazon Mturk/ workers for their annotation. We used workers from list of 90 pre-selected workers from English speak- ing countries. These workers accomplished high quality work in other NLP-related tasks we have conducted in the past. Each pair of summaries was annotated by three workers, and the majority vote was considered for the final score.
Empirically, the partial correlation scores show the same trend as the bucketing scores. Table 8 displays the median (with the 25th and 75th per- centiles) differences between the original and par- tial correlations for group of metrics de- signed measure Relevance, using the same for- mat as described in Section 4 for the bucketing method. Similar to the bucketing method, we ob- serve significant differences in cases where Rele- vance serves as the anchor QC (the column marked in blue), while minimal differences are observed for the Relevance an evaluated QC (yellow row). This indicates the Relevance QC's role poten- tial confounding factor.
Table 7 presents the rate of uncorrupted low- ranked system summaries that were preferred over the corrupted summaries in each system pair. As can be measured clearly, in all pairs the uncorrupted summaries were preferred in 50 percent or more al- though they were ranked lower prior to corruption. This indicates that the corrupted systems indeed should be downgraded in their ranking. The conclu- sion is therefore that our corruptions are effective in degrading the corrupted system's 's ranking.
D Licenses
All system summaries, reference summaries and human- ratings were taken from the SummEval repository under the MIT license. Some of the reference summaries originally from the CNN/DailyMail dataset. The docu- ments corresponding to the summaries, also from CNN/DailyMail, were retrieved via the Hugging- face distribution. All CNN//ailyMaii data re- leased under the Apache-2.0 License.
Partial Correlation
When examining the statistics literature for meth- ods for detecting spurious correlations, we focused on the two prominent approaches: partial correla- tion (Whittaker, 2009) and data stratification (Man- tel and Haenszel, 1959), Our bucketing method, presented in Section 4, is form of stratification.
1https: //www. mturk. com

---

Coherence
Consistency Replace PER Roy Hodgson has restarted his row with Andros Townsend.
Fluency Lemmatize Verbs
Original
Shuffle Sentences
Paul Merson has restarted his row with andros townsend. The Tottenham midfielder was brought on with only seven minutes remaining in his team 's 0-0 draw with Burnley. Andros Townsend scores England is equaliser in their 1-1 friendly draw with Italy in Turin.
The Tottenham midfielder was brought on with only seven minutes remaining in his team is 0-0 draw with Burnley Andros Townsend scores
Paul Merson have restart his row with Andros Townsend.
The Merson midfielder was brought on with only seven minutes remaining in his team ss 0-0 draw with Burnley Andros Townsend scores England 's equaliser in their 1-1 friendly draw with Italy in Turin
The Tottenham midfielder be bring on with only seven minutes remain in his team 's 0-0 draw with Burnley Andros Townsend scores England equaliser in their 1-1 friendly draw with Italy in Turin
England 's equaliser in their 1-1 friendly draw with Italy in Turin Paul Merson has restarted his row with Andros Townsend.
Table 9; An example of a system summary with all corruption types. A replaced PERSON named entity is marked red. A lemmatized verb marked blue. Capital letters were inserted manually to facilitate reading.
Metric
Coherence
Consistency
Fluency
Coherence
Consistency
Fluency
Relevance
ROUGE-1
0,9873
0,8966
ROUGE-1
0.2875
0.1741
2OUGE2
0,9467
0.5746
0,9121
N/A
ROUGE-2
N/A
0.1757
0.4925
0.1103
ROUGE-3
0,9495
0.5157
0,9128
N/A
ROUGE-3
N/A
0,2024
0.6400
0,2105
ROUGE-4
0,9333
0.4924
0,7202
NA
ROUGE-4
NIA
0,2132
0.5158
0.2139
ROUGEL
0.7566
0.7873
0.8087
NIA
ROUGE-I
N/A
0.2682
0.3730
0.0694
ROUGE-su"
0.9766
0.7332
0,8916
N/A
ROUGE-Sa"
NIA
0.3403
0,5799
0.1774
ROUGE-W
0,8969
ROUGE
0.1865
0.0814
ROUGB-we-I
0,9700
0.6850
0.9958
N/A
ROUGE-we-I
N/A
0.3266
0.6588
0,1890
ROUGE-we-2
0,7927
0.7886
0,5013
NA
ROUGE-We-2
NIA
0,3184
0.8864
0,1472
ROUGE 3
0,7065
0.8060
2,1853
NAA
ROUGE we-3
N/A
0.3165
0.0882
0.1278
S'-pyr
0,7438
0.6577
0.6174
NAA
8 -psy
NIA
0.1955
0.6394
0,3139
S7-resp
0,7828
0.6336
0.7507
NIA
S S'
NIA
0.1842
0.5944
0.1109
BertScore-p
BertScore-p
0.4076
1252
BeriScore-
0,9659
0.5216
0.4641
BeriScore-r
0.1969
0.3071
0,2121
BertScore I
0,9786
0.5970
0.4343
NAA
BeriScore
N/A
0.3136
0.3128
0.1613
MoverScore
0,8043
0.5236
0.4258
NA
MoverScore
N/A
0.3076
0.3440
0,17
SMS
0.6851
0.2668
0,2962
NA
SMS
NIA
0.2328
0,2847
0,3521
BLEU
0.9598
0.3942
0.3544
NIA
BLEU
N/A
0.1242
0.0914
0.0771
CHRF
0.8419
0.7789
0,7533
CHRF
0.4792
0.7247
0.3331
0,0546
15.7989
0.1060
17.7140
0.1322
METEOR
0,8080
0.5610
0.5993
NAA
METEOR
N/A
0.1712
0.3405
0.1287
SummaQA"
0,9738
0.4100
0,3489
NA
SummaQA"
NIA
0,1774
0.1952
0,1933
BLANC"
0,7650
0.3520
0,5176
NA
BLANC"
N/A
0.1668
0.3377
0.2045
SuPERT"
0.9157
0.2762
0.3375
NA
SuPERT"
N/A
0.15
0.2265
0.2771
Novel unigram"
0.1920
0.0255
0.0368
NA
Novel unigram"
NIA
0.1366
0,2128
0.9964
gram"
0,5511
Novel hi hi-gram"
0.0239
0.2730
Novel tri-gram"
0.3208
0.1185
0.0991
N/A
Novel tri-gram"
0.0456
0.0386
0,5205
Repeated unigram"
0.0114
0.3146
0,1355
N/A
Repeated unigram
N/A
0.9790
0.6875
1,3634
Repeated bi- gram"
0.1406
0.3776
0,3073
NA
Repeated bi gram"
N/A
0.6764
0.7280
0.6850
Repeated tri gramm
0.1370
0.4934
0.1540
NIA
Repeated tri-grmm"
NIA
0.7937
0.8259
0.7572
Table Absolute relative difference between original performance and bucketing performance anchored by Relevance.
Table 1:: Absolute relative difference between original performance and bucketing performance anchored by Coherence.

---

Metric
Coherence
Consistency
Flueacy
Relevance
ROUGE-1
0.1788
N/A
0.6285
0.1240
2OUGE-2
0,2036
NA
0,7035
0.1575
ROUGE3
0,2259
N/A
0.8096
0.1703
ROUGE-4
0,2597
NIA
0.6754
0. 1877
ROUGELL
0,1708
NIA
0.4912
0.0983
ROUGE-s
0.2094
0.6218
ROUGE-W
0,2149
0,5807
wOUGEEwe
0.1732
N/A
0,7035
0. 0.1244
ROUGE-we-2
0,1617
N/A
0,8458
0. 1100
ROUGE 3
0.1901
NIA
0,3386
0.0913
S'-PyT
0.2117
N/A
0,7951
0. 1051
S"-resp
0.2179
N/A
0,7999
11114
P
0,2282
0.2205
BeriScore-r
0,2380
N/A
0,3994
1521
BeriScore-I
0,2373
N/A
0,4073
0.1574
MoverScore
0,2043
NIA
0,3979
0.1413
SMS
0,2981
NIA
0,3782
0,2810
BLEU
0.3225
N/A
0,3063
0,2167
CHRF
0,4888
CIDER
0,7280
N/A
0,4464
3,5150
METEOR
0,2585
N/A
0,4722
0. 0,1406
SummaQA
0,5407
NIA
0,4021
0.3948
BLANC"
0.5268
N/A
0.6045
0,2827
SuPERT"
0.5649
N/A
0.5151
0,4122
unigram"
0,5059
Novel bi-gram"
1,334]
N/A
0,3823
0.3147
Novel Iri gram"
0,7344
N/A
0,3851
0.7910
Repeated unigram"
0.1479
N/A
0.46.99
0.8627
Repeated bi-gram"
0,1379
N/A
0.4344
0,3066
Repeated tri- gram
0.0485
NIA
0.5129
0.1521
Table Absolute relative difference between original performance and bucketing performance anchored by
Consistency.
Metric
Coherence
Consistency
Fluency
Relevance
ROUGE-1
0,1913
0,3133
NIA
0.0859
2OUGE-2
0.2077
0.2974
N/A
0.0749
0.1677
N/A
ROUGE-4
0.2238
NAA
ROUGEL
0,2303
0.3583
N/A
0.0729
ROUGE-SU"
0,1984
0.3625
N/A
0.0919
ROUGE-W
0,2008
0,2693
NAA
0.0534
ROUGG-we--
0.1748
0.3421
N/A
0.0820
ROUGE-we-2
0,1192
0,2809
NAA
0.0349
ROUGE
0.089
0.2595
N/A
8 S'-pyr
0,1251
0.1525
N/A
0.0205
S2-resp
0,1483
0.1734
N/A
0.0286
BeriScore
0.4468
0.8169
N/A
0.2524
BeriScore-r
0,2830
0.3041
N/A
0. 1017
BeriSooe-
0.3259
0.5184
N/A
0.1306
0,2813
N/A
SMS
0,3156
0.2286
N/A
0.1994
BLEU
0,5043
0.3209
N/A
0. 1893
CHRF
0,1778
0.4029
N/A
0.0921
CIDE
0,2162
212.7716
N/A
7.1651
METEOR
0.2902
0.2951
N/A
0.0764
SummaQA"
0.5623
0.3900
N/A
0.3046
0,5358
N/A
SuPERT"
0.5096
0.2679
N/A
0.2778
Novel unigram"
0,7387
0.2786
N/A
2,8030
Novel bi gram"
0,8233
0.2192
N/A
0.9587
Novel tri-gram"
0.9199
0,2140
N/A
0. 0.7485
Repeated unigrama"
0.3130
0.7697
NA
0.5830
Repeated gram"
Repeated trigram"
0.1338
0.4467
N/A
0.2944
Table 13: Absolute relative difference between original performance and bucketing performance anchored by Fluency.


Reviews:
[{"review_id": "2EUUfdrqEI", "reviewer": "Reviewer_bKiH", "summary": "", "questions": "My following question is intended as a discussion with the authors.  \n1. While reading this paper, it motivated me to think conceptually about the four metrics and their correlations just based on their definitions. For instance, theoretically a highly coherent summary (quality of the collective structure of the sentences in the summary) can still have low fluency (linguistic quality of individual sentences in the summary) if the individual words are shuffled in the sentence while preserving the overall discourse structure. However, the author's findings suggest that, in the dataset they analyzed, they observe a high correlation between these dimensions. Could this be explained by the fact that it is very less likely to observe such cases in practice, i.e., in this dataset, if the summary has a good discourse structure (high coherence), it is likely it will also have high fluency (words in the sentences are ordered such that they make sense)? Does this observation mean that all the model-generated summaries in the benchmark have summaries such that, if they are coherent, they are also fluent? I think it is possible to draw theoretical insights based on the definition of the quality criteria and contrast them with empirical findings, which could reveal some interesting findings.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! \n\n&nbsp;\n\n**Suggestions for adding illustrative example** \n\nThank you for these good suggestions, we will add such examples.\n\n&nbsp;\n\n**Discussion about inherent correlation between QCs**\n\nThis is a very interesting question. Within the scope of the paper, we indeed observed a high correlation across QCs in the SummEval data, and attributed it to the assumption that more powerful systems tend to perform better under all criteria. WIth respect to a theoretical perspective on inherent correlations between the QCs, on the face of it it seems that the designers of these QCs did have in mind a substantial degree of independence between them. For example, any extractive summarization system would inherently be quite fluent, as it consists of original source sentences, each manually written (up to decontextualization issues). Yet, one can imagine an extractive system, particularly in the multi-document setting, which is quite poor wrt the generated discourse structure. Similarly, an extractive system would tend to be consistent (faithful), but may perform poorly wrt detecting salient content (i.e. poor relevance). That said, some dependencies may still exist inherently between QCs, per their definitions. For example, for a sentence to be fluent, discourse references and connectives should be easily interpretable, which would rely on a coherent discourse structure, also across sentences. Similarly, a non-consistent (unfaithful) text fragment, which is not entailed by the source, cannot reflect relevant content in that source (as its content is not at all present in the source). Investigating such varying degrees of QC dependencies indeed seems like an interesting topic for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper raises an important concern in automatic evaluation of summarization, i.e., use of multiple criteria for evaluation and their correlations. The authors demonstrate the brittleness of current evaluation practices in the presence of summary corruptions and propose a procedure to help detect such issues. Overall the paper is well written and provides experiments to support their findings related to issues with current evaluation practices.  \n\nSuggestion:\n1.  While I understand the page limit constraints, if possible, the paper could benefit from showing some worked out examples to ease reading and comprehension of the paper.\n2. Also, some examples of how the ratings changed when the summaries were corrupted could be interesting. ", "reasons_to_reject": "I don't think I have any major concerns. ", "questions_to_authors": "My following question is intended as a discussion with the authors.  \n1. While reading this paper, it motivated me to think conceptually about the four metrics and their correlations just based on their definitions. For instance, theoretically a highly coherent summary (quality of the collective structure of the sentences in the summary) can still have low fluency (linguistic quality of individual sentences in the summary) if the individual words are shuffled in the sentence while preserving the overall discourse structure. However, the author's findings suggest that, in the dataset they analyzed, they observe a high correlation between these dimensions. Could this be explained by the fact that it is very less likely to observe such cases in practice, i.e., in this dataset, if the summary has a good discourse structure (high coherence), it is likely it will also have high fluency (words in the sentences are ordered such that they make sense)? Does this observation mean that all the model-generated summaries in the benchmark have summaries such that, if they are coherent, they are also fluent? I think it is possible to draw theoretical insights based on the definition of the quality criteria and contrast them with empirical findings, which could reveal some interesting findings.", "paper_topic_and_main_contributions": "The authors study the framework of summarization evaluation along four quality criteria (QC; consistency, faithfulness,  relevance and coherence) and challenge the current method of studying correlations between human judgements and metric-based ratings as a measure of an automatic metric quality. The authors find that some of the best metrics for certain QC actually do not perform well, failing to detect even drastic summary corruptions with respect to the considered QC. For instance, changing the word order does not affect a metric's raking on coherence criteria. To explain this finding, they study the correlations between these four criteria and find that human annotations are correlated because one of the criteria act as a confounder. Thus, reporting correlations with human judgements after annulling for this confounder can help better estimate system performance instead of studying raw correlations. Authors propose a bucketing system where they divide the ratings into buckets such that the confounder variable has same rating, lending an effect of keeping the confounder constant. The authors also find that most often, relevance QC acts as a confounder since it was mainly used to design systems/metrics.  ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f7UO5IZqt4", "reviewer": "Reviewer_jPgs", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! \n\n&nbsp;\n\n**Novelty and importance of our approach**\n \nAs indicated in the review, we indeed conclude in our paper that each evaluation metric should focus on a single QC. While this insight is not surprising to the reviewer, a prominent thread of research in NLP has been effectively advocating the opposite. As we point out, the SummEval paper, which is highly cited (~330 citations as of today) and is one of the most prominent current datasets for evaluating summarization evaluation metrics, clearly implies that metrics *can* be suitable for evaluating multiple QCs, and establish such metric fitness only by computing a direct correlation independently with each QC. Consequently, for example, they explicitly specify Rouge-N (for large N) to be better than Rouge-L in all QCs, while we claim that most of these scores are spurious. Furthermore, many papers suggesting new metrics do follow the SummEval methodology and data and show high correlation across all QCs to their new metric (e.g.,DiscoScore [1], BARTScore [2], InfoLM [3]) - again counter to our findings. Some other papers point to the advantage of a specific metric for a specific QC according to the SummEval results. For example, in [4]  they explained their results by the ability of Rouge-3 to measure consistency as resulted in SummEval, while in [5] they rely on SummEval to conclude that “ROUGE-L can be seen as a measure for fluency”. \n\nThus, even if our findings are not surprising to some people in the community, they are meant to respond to this notable literature thread, offering three contributions for future research: (1) exposing the fallacies of previous conclusions, which inappropriately attributed metrics to certain QCs; (2) revealing underlying correlations in manual summarization evaluation data, and showing accordingly that prior false conclusions stemmed from spurious correlations caused by confounding factors; (3) offering a concrete experimental protocol, based on a data stratification approach (bucketing), which helps detecting spurious correlations (elaborated next in response to the second reason to reject). We believe that these contributions, in the context of the negative effects of highly visible practices in the literature, would be beneficial to the community and are hence important to publish. \n\n&nbsp;\n\n**Bucketing versus partial correlation**\n\nWhen examining the statistics literature for methods for detecting spurious correlations, we indeed focused on the two prominent approaches of computing partial correlation vs. data stratification (followed in our bucketing method), and experimented with both.\n\nMethodologically, we identified two advantages of bucketing. First, beyond providing an overall “debiased” correlation figure, this method provides explicitly the debaised data within each bucket, allowing for further interpretability and analysis (e.g. examining correlation and other statistics within each bucket). Second, the partial correlation calculation is based on computing linear regression between the variables, assuming that the potential correlation between them is linear. Bucketing, on the other hand, generally allows for measuring any type of correlation, making this method more general for future extensions (even if in the current work we measured only linear correlations within each bucket).\n\nEmpirically, as said, we did compute partial correlations in our investigations, but found them to be less discriminative in detecting spurious correlations in the summarization data, although showing the same trend. Following your comment, we will add to the paper a presentation and empirical results of the partial correlation method, and compare it to the bucketing method. \n\n&nbsp;\n\n[1] Zhao, Wei, Michael Strube and Steffen Eger. “DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence.” Conference of the European Chapter of the Association for Computational Linguistics (2022).\n\n[2] Yuan, Weizhe, Graham Neubig, and Pengfei Liu. \"BARTScore: Evaluating generated text as text generation.\" Advances in Neural Information Processing Systems 34 (2021): 27263-27277.‏\n\n[3] Colombo, Pierre Jean A., Chloé Clavel, and Pablo Piantanida. \"InfoLM: A new metric to evaluate summarization & data2text generation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 10. 2022.‏\n\n[4] Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. ``GO FIGURE: A Meta Evaluation of Factuality in Summarization.” In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 478–487.\n\n[5] Li, Daniel, et al. \"Improving Automatic Summarization for Browsing Longform Spoken Dialog.\" Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.‏", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The findings of this paper are helpful for interpretation of automatic evaluations.", "reasons_to_reject": "The claim of this paper, an automatic evaluation metric designed to evaluate text in terms of a specific QC should not be utilized to assess text in terms of the other QCs, is helpful but not surprising. According to the nature of ROUGE-N, many researchers do not utilize it to evaluate the coherency of the summaries.\n\nFinding confounding factors through experiments can be a complex and confusing process. In my opinion, utilizing partial correlations can prove to be beneficial for this purpose.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the reliability of automatic evaluation metric. The experimental results suggests that scores obtained from an automatic metric designed to evaluate a specific Quality Criteria (QC) are not reliable in the other QCs. This paper recommends that an automatic metric designed for assessing a specific QC should only be used for evaluating text in relation to that particular QC.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XhiOke4Nty", "reviewer": "Reviewer_rPrT", "summary": "", "questions": "175. How many times did you spend to corrupt the system summaries? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! \n\n&nbsp;\n\n**Robustness to other benchmarks**\n\nWe demonstrated our methodology over SummEval because it is currently the most prominent dataset which includes human evaluations for multiple Quality Criteria (QCs), applied to the output of a large number of systems. However, our methodology is generically designed to test for spurious correlations across QCs (via our stratified bucketing methodology), and could be applied over any dataset for assessing summarization evaluation metrics, which would include human evaluations for multiple QCs. In case such a dataset, in another language or domain, will have different correlation patterns than SummEval (e.g. weaker correlations for certain QCs), then these are expected to be revealed as well by our bucketing methodology.\n\n&nbsp;\n\n**How many times did we try to corrupt the systems?**\n\nWe are not sure whether we correctly interpret your question. As far as we understand, the question refers to the effort needed to identify corruptions that are missed by the considered evaluation metrics. In fact, this was rather a small effort, where it was easy to design substantial corruptions that were overlooked by the evaluation metrics. These corruptions were applied for all summaries for each system in turn.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper can be accepted for some reasons: the authors offer a new perspective on the summarization evaluation metrics in the multiple QS case. While the paper mostly highlights the problem, the authors show other researchers to develop additional approaches to evaluate summarization. The title and abstract are correctly selected and written. The references are relevant. The paper is correctly structured and written.", "reasons_to_reject": "The authors conducted experiments only on SummEval bencmark, it is possible that for other benchmarks, this experiment will give different results. They explain this limitation in the Limitations section, but I didn’t have enough reasoning about bencmarks in other languages and how to get around these limitations for other bencmarks.", "questions_to_authors": "175. How many times did you spend to corrupt the system summaries? ", "paper_topic_and_main_contributions": "This paper considers the evaluation metrics across multiple Quality Criteria (QS) in the case of summarization, where QS includes relevance, consistency, fluency, and coherence. \nThe authors discovered that the conventional setup that uses a high correlation between metric and human scores as an effective metric is inappropriate for multiple QS setting for the summarization case. To achieve this the authors artificially corrupt SummEval system summaries in various forms and show that even the best metrics for each QS fail to detect even drastic summary corruptions with respect to the considered QS. Additionally, they show that there are spurious correlations in a multiple QS setting. The authors also suggest a method of detecting metric-human unreliable correlations scores that are suspected as spurious that removes the effect of the confounding variable.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that high correlations between system rankings from metrics and humans can be spurious, in the presence of one confounding QC. The arguments in the paper are convincing. The authors should consider addition of results from partial correlation, which they claim shows similar findings to strengthen the paper.
Decision: Accept-Findings