Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
Authors: Ziang Xiao
Susu Zhang
Vivian Lai
Q.Vera Liao
Venue: EMNLP 2023 Main (2023)
Keywords: Evaluation
Evaluation Metrics
Measurement Theory
NLG
Large Language Model
TL;DR: A measurement theory-based framework to evaluate NLG evaluation metrics

Abstract: We address a fundamental challenge in Natural Language Generation (NLG) model evaluation---the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
Ziang Xiao Susu Zhang", Vivian Lai Q. Vera Liao
JJhnns Hopkins University
Microsoft Research Montréal + Visa Research
i University of Illinois Urbana- Saaanaa-happigg
ziang- niao@jjuu.edu viv. wwi@visa.c....... com
veraliaa@microsoft..co com
Abstract
Matric Stability
We address fundamental challenge in Nat- ural Language Generation (NLG) model evaluation the design and evaluation of eval- uation metries. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose METRICEVAL, framework informed by measurement theory, the foundation of ed- ucational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers sta- tistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metries to better interpret the result. To exemplify the use our framework in practice, we analyzed set of evaluation metrics for summarization and identified issues related to conflated valid- ity structure human-eval LLM-basce metrics. Through METRICEVAL we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.
Relisbility
Metrie Coasistensy
Menre Divergert
Measurement Errer
Validio
Metric Couvergant
Validit
Valdie
Vanidry
Metric Facturial
Metric
Valldiyy
Figure 1: METRICEVAL: A framework that concep- tualizes and operationalizes four main components of metric evaluation, in terms of reliability and validity
to individuals and society (Yeo and Chen, 2020; Sheng et al., 2021).
Designing effective evaluation metrics for nat- ural language generation (NLG) tasks has long been challenging due to the complex nature of language, open-endedness of tasks, multifaceted and context-dependent definition of language qual- ity (Nema and Khapra, 2018; Zhou et al., 2022; Gehrmann et al., 2022; Sai et al., 2022). Most recently, the NLG evaluation challenge has been further exacerbated by the emergence of "general- purpose" large language models (LLMs), further demanding evaluation methods to capture model utility for diverse downstream cases. To ad- dress these challenges, researchers and practition- ers have developed various types of NLG evalua- tion metrics, including word-based metrics (e.g.. ROUGE, BLEU), embedding- based metrics (c.g.. BERTScore, MoverScore) and to end-to-end met- rics (e.g., BLEURT, G-Eval).
reliability in
1 Introduction
Evaluation metrics provide quantitative assess- ments to guide model development, benchmark sci- entific progress, and inform generalizability across tasks and domains (Novikova et al., 2017). Ef- fective evaluation metries can extract valuable sig- nals and robust evidence from model outputs that describe model capability, diagnose model fail- ures, and compare the strengths and weaknesses of different models, allowing for more informed decision-maken making in real-world deployment (Zhou al., 2022). Conversely, problematic evaluation metrics can mislead model diagnoses, development, and deployment, resulting in downstream harms
With this increasingly rich set of evaluation met- rics being pursued, must understand how good each metric is. While researchers pointed out shortcomings of popular metries (e.g.. ROUGE). such as their inability capture semantic mean- ings, insensitivity to perturbations, and failure to reflect real-world performance (Sai et al., 2021: Liu
"Equal contribution.
'Data and code for analysis are available at: https:// gi thub. com/isle sessledde weeeedeeeeerrii........

---

al., 2016; Reiter, 2018: Celikyilmaz al., 2020; Kauchak and Barzilay, 2006), there is a lack of principled approaches to evaluate NLG evaluation metrics, and to begin with, a lack of clear definition on what makes a metric good.
2 Measurement Theory
Originating from educational and psychological testing, measurement theory aims inform eval- uation processes that devise coherent numerical representation of individual capabilities for in- stance, evaluating a person's 's language proficiency through essay responses to questions. Scores on these tests have direct consequences for high-stakes decisions, such as school admissions.
Some prior works have attempted to evaluate the quality of NLG evaluation metrics by their correla- tions with human judgments (Sai et al., 2021; Fab- bri et al., 2021; Liu et al., 2016), which are deemed the gold standard for quality assessment. How- ever, correlation with human preferences gives lim- ited quality signals. More problematically, human evaluation data collection itself currently suffers from validation, standardization, consistency, and reproducibility issues (Clark et al., 2021; Howeroft al., 2020; Belz al., 2021; Khashabi et al., 2021). These issues subsequently undermine their validity as the foundation for evaluating automatic metrics.
Key to measurement theory is the distinction be- tween the observed score on test, e.g., the score of an examinee's s essay in a language proficiency exam, and the true score on the general construct (Cronbach and Meehl, 1955) that the test theo- rized to measure, e.g.. language proficiency. The gap between the observed and true scores is re- ferred to as measurement error (Allen and Yen. 2001). Measurement theory defines two sources of measurement error, random and systematic. Ran- dom measurement errors are fluctuations specific to a time, place, examinee, and exam question that are transient and balance out to over repeated measures, and they have direct consequences on the reliability of the evaluation process. System- atic errors, on the other hand, are persistent shifts across one more time, place, examinee, or exam questions, and they have direct consequences on test validity by producing observed scores with sys- tematic deviations from the true score that the test purports to measure (e.g., a downward bias if the rubric on a language proficiency exam looks for specialized knowledge about a certain subject).
In this paper, we introduce METRICEVAL, the- oretical framework to define the desiderata of and assess evaluation metrics by drawing from measure- ment theory in educational and psychological test- ing. Based on two core concepts in measurement theory that define a "good" metric in testing individ- ual capabilities reliability and validity, our frame- work conceptualizes and operationalizes four key desiderata: Metric Stability, Metric Consistency, Metric Construct Validity, and Metric Concurrent Validity. We further propose set of statistical tools to quantify these desiderata systematically evaluate evaluation metrics. METRICEVAL enables quantifying the standard eтror for each metric on a specific task which allows meaningful interpreta- tions of the evaluation results. We demonstrate the utility of METRICEVAL with a case study of evalu- ating 16 metrics, including LLM-based metrics, on a summarization task.
By evaluating and identifying the source of mea- surement errors, the test designer could iteratively improve their test design by adding or removing test items or changing their rubric. The results can also help the evaluator to interpret test score with caution. For example, we can quantify the uncer- tainty as the standard error for meaningful com- parison: when the difference between two models' metric scores on a benchmark does not afford con- clusions about significant differences, the evaluator may consider narrowing the confidence interval by averaging scores from repeated measurements.
This paper offers three contributions,
Introduce and transfer metries evaluation desiderata and methods from measurement theory in educational and psychological test- ing to NLG evaluation.
Propose METRICEVAL, -heory-driven framework with set of statistical tools for systematically analyzing and evaluating NLG metrics.
In short, as safeguard to the trustworthiness of tests, measurement theory offers conceptual framework for how the validity and reliability of test should be formalized, evaluated, and opti- mized to reduce measurement error with the aid of statistical methods and tools.
A case study demonstrating how to apply our framework and identify issues of evaluation metrics for a summarization task.

---

2.1 Transferring Measurement Theory to the Context of NLG
free metrics (e.g.. SUPERT), human ratings on specific output quality aspects (e.g.. coherence or relevance) on the same benchmark to draw infer- ences about model capability. For the remainder of this section, we will illustrate our framework with this running example of evaluating summarization models with diverse metrics,
There are obvious analogies between the measure- ment of human capability and the evaluation of NLG models. First, NLG evaluation is often per- formed via benchmarking. A benchmark data set is similar to an educational test consisting of a tai- lored collection of test examples, where question- specific scores are calculated based on predefined evaluation metrics (similar to scoring rubrics for human testing) and are aggregated into an overall score for the model. Second, when evaluating model, similarly hope derive scores based candidate model's observed performance a benchmark, so as to (I) draw inferences about unobservable capability in a specific domain (e.g.. summarization) and (2) provide guidance on the model's expected behavior in future tasks of the domain. Similar to educational testing, the score is often interpreted and used beyond its nominal meaning, i..., implying the model's general per- formance beyond the particular benchmark dataset. As more models are considered models, there is an increasing need for measuring an NLG model's unobservable capability.
It is important to note that the quality of evalua- tion results is also dependent the chosen dataset and reference (for reference- based metrics), which, in NLG evaluation, are concemed with benchmark designs. Mcasurement errors may cascade from those components to the observed score. In this work, we focus on the metric aspect and answer questions such as "giving a CNN/Daily Mail bench- mark, whether using ROUGE or BertScore or hu- ratings offer reliable and valid evaluation sults.". This is an important question given the far-reaching impact that prevalent benchmarks can have on the output of the research community.
3.1 Reliability
The reliability of metric is the extent to which the result is subject to random measurement error and thus (in)consistent across repeated measures, such as different (sub-)datasets within a benchmark or different raters scoring the model's output in human evaluation. Suppose two NLG models are scored their performance based Metric-A summarization benchmark. Researchers and practitioners often use the scores draw inferences about the models' (relative) performances. When the two models are reported to differ in their scores (e.g., Metric-A = .39 vs. .42), a natural question is how much this reflects actual differences (true signal) versus fluctuations due to random measure- ment error (noise). If Metric-A is unreliable, the measurement eтror may mislead the comparison. Sources of random measurement error that im- pair the reliability of metric may include:
The conceptual and statistical tools provided by measurement theory can be transferred to assist in evaluating NL. metries, specifically to quantify and identify different sources of measurement er- rors. Not only can these tools help the community systematically assess the shortcomings of evalua- tion metrics and identify misleading ones, but they also guide the interpretation of their evaluation re- sults, as well as the re-design existing metrics and the development of new ones, In the next sec- tion, we elaborate on how we transfer these tools from measurement theory to framework that de- fines and assesses the reliability and validity of NLG evaluation metrics, and how they may help us interpret and improve NLG evaluation.
Non-deterministic algorithms of some met- rics may produce score variations on the same model outputs.
3 Metric Evaluation Framework
In this section, we introduce METRICEVAL and its components that define and assess different aspects of the "goodness" of NLG evaluation metrics, in- spired by the core concept of reliability and validity in measurement theory (See Fig. METRICEVAL aims to evaluate and compare the reliability and validity of the metrics. For example, to evaluate a summarization model, one can apply reference- based metrics (e.g., ROUGE, BertScore), reference-
The subsets of data points (e.g., different gen- res of articles) included in the benchmark.
For human evaluations, the variability across raters, resulting from their subjectivity, incon- sistency, errors, and so on.
In classical test theory (Spearman, 1904), the observed metric score of a model (X) is equal to

---

the sum of true score T and error (E), which is as- sumed be independent of and fluctuates around O with variance o- The goal of evaluating a met- ric's reliability is hence to quantify the expected amount of uncertainty in the observed score due to random measurement error, known as the standard error of measurement (ar).
different models, the Pearson correlation between the two sets of scores is the test-retest reliability coefficient. One can show that this correlation is an estimate of the reliability coefficient pxT as defined in Equ. 1. This because, for the two metrics scores for a model, X1 = Ti + E₁ and X2 Ti + E2, the correlation in the observed scores, X2, is algebraically equivalent to PX under the assumption that each model's true score doesn't 't change (i.e., Ti T2) and that the ex- pected fluctuation in metric evaluation remains the same (i.c., σg = Øg,) across the two evaluations (see derivations in Allen and Yen, 2001).
Empirical estimation of og is done via the relia- bility coefficient of metric, denoted pxT [0, I]. Formally, the reliability coefficient is defined as the proportion of variance in the observed score explained by the variance in the true score across NLG models rather than error, or equivalently, the squared correlation between X and T:
3.1.2 Metric Consistency
of
Metric Consistency describes how the metric score fluctuates within benchmark dataset, i.e., across data points. If the metric score computed on each individual data point (e.g.. summarization of spe- cific news article) deviates substantially from the average score across the benchmark dataset (e.g., across 100 news articles), the metric score is less reliable, in that it is more sensitive to perturbations in the specific data points employed in the bench- mark dataset. In this case, for a specific model, the average metric score on any subset of tasks (e.g.. 50 out of 100 news articles) is expected to be sen- sitive to the choice of included examples, and good proportion of difference across two evaluated models' average scores would also be attributed to this nrise.Drawn from the estimation of internal consistency reliability in measurement theory, the estimation of metric consistency depends on the degree to which scores from different subsets of the benchmark dataset agree with one another.
pxT
=1
(1)
Metrics with higher reliability coefficients are more desirable. However, in reality, neither T nor E is observed. The reliability coefficient in Equ. can- not be directly computed and statistically approx- imated via several possible estimators.
METRICEVAL proposes to estimate the reliabil- ity coefficient from both Metric Stability and Mer- ric Consistency. They reflect different reliability issues that can arise in different types of metrics, as we elaborate below. By quantifying and identifying reliability issues, metric developers can improve the scoring algorithms, and metric users can make more informed decisions in choosing metrics, inter- pret performance differences, and adopt mitigation strategies, c.g., increasing the test set size to miti- gate consistency issues (Spearman, 1910).
3.1.1 Metric Stability
Metric Stability refers to how a metric score may fluctuate when evaluated again on the same model output. While we would expect perfect stability (i.e., σF 0) for deterministic metrics, such as ROUGE-1 (Lin, 2004), the stochastic nature of some metrics (e.g.. G-Eval (Liu et al., 2023)) may produce undesirable fluctuations when evaluating the same model outputs. As we see increasing use of automatic evaluation metrics with built-in stochasticity (e.g., LLM-based metrics), the stabil- ity of an evaluation metric in producing consistent scores output from one replication to another will be increasingly relevant.
The coefficient (Cronbach, 1951) provides measure of Metric Consistency. Let J denote the total number of data points the test dataset, Y, the observed score (of model) on the jth data point alone (e.g., the jth news article), and = Σ/=1 Y, the overall score of the model on the full test set. Then a provides a lower bound to the true reliability of X, i.e.,
J
0x
Σソ= of o
P&T 2 a
(2)
where 03 is the variance of Y, across models. Equality holds when all the individual data point scores (Y,s) have equal correlations with the true score (T), which may be violated in practice, lead- ing to the underestimation of true reliability via the
We propose quantify metric stability via the test-retest reliability coefficient: on the output gen- crated by N models, we compute the metric score with the same output twice for each model. Across

---

coefficient a formula.
predictive power on the criterion of interest.
3.2 Validity
3.2.2 Metric Construct Validity
Construct validity, a term coined by Cronbach and Meehl (1955), refers to the degree to which the observed behaviors on the test (e.g., test scores) can reasonably reflect the intended construct (e.g., language proficiency). This notion is directly appli- cable to evaluation metrics that explicitly con- structed to assess specific aspects of model's per- formance or output quality, e-g.. human evaluation (or automatic metrics, developed specifically) on summarization, coherence, fluency, consistency, etc. However, even for metrics of which the in- tended construct is not explicitly defined, it is still necessary to understand what underlying dimen- sions of model capabilities they actually capture.
Validity is another core component of METRICE- VAL. Metrics with low validity lead to systematic measurement errors that deviate the observed score from the true score that the test purports to measure. In other words, benchmarking is valid only when the metric scores can inform their intended inter- pretations (e.g., model capability) and uses (e.g.. predicting models' real-world behavior).
Our framework is theoretically grounded in Mes- sick's unified theory of test validity (e.g., Messick, 1995), under which the emphasis is given to the validation of inferences drawn from the test score, rather than the validation of the test itself. Different types of validities should be recognized as possible ways to gather supporting evidence for intended inferences (interpretations and uses) from the met- ric score. Our framework conceptualizes two types of metric's 's validity, concurrent validity, and con- struct validity (e.g., Allen and Yen, 2001), which can be applied in different situations-wiie -when vali- dated reference criterion is available not elaborate below.
It is important note that the underlying con- struct is often latent and not directly observable assess its relation with the measure. Measurement theory, therefore, provides statistical tools to assess the construct validity of measure through its re- lation with other observable variables (e.g.. other tests purported to reflect the same or different con- structs). We consider three such aspects of validity based on the measurement literature:
3.2.1 Metric Concurrent Validity
Metric Convergent Validity: Whether metrics of identical or related construct(s) are indeed related. For example, for the same aspect of summarization quality (e.g., coherence), scores provided by different evaluation meth- ods (e.g.. by different raters) should be highly correlated.
Metric Concurrent Validity relies on another vali- dated metric as the reference criterion. This type of validity is most relevant when evaluating met- ric as an alternative to existing ones that may be expensive and infeasible to acquire in practice. For example, evaluations by trained human experts are often challenging at a large scale, motivating the development of automatic alternatives. One can conclude that an automatic metric is a valid proxy if it has high concurrent validity using the expert valuation results as the reference criterion.
Metric Divergent Validity: Whether metrics of unrelated constructs are indeed unrelated. For example, for distinct aspects of summariza- tion quality (e.g.. coherence and relevance). scores provided by the same method (e.g.. by the same rater) should show substantially lower correlations than those for the same quality across methods. Low divergent va- lidity could indicate method bias: e.g., the observed score depends greatly on the rater's subjective tendency rather than the model's 's performance the rated dimension.
When both the target evaluation metric (X, e.g., a new automatic metrie) and the reference crite- rion (Y, e.g., expert evaluation) are continuous, a straightforward way to quantify concurrent va- lidity is via their Pearson correlation, PXY, often referred to as the (criterion-related)) validity coeffi- cient. One should note that measurement error in ei- ther X or Y is expected to attenuate this correlation (Spearman, 1910): At the population level, PxY is bounded above by the square root of the product of the two scores' (X and Y) reliabilities. This again highlights the importance of safeguarding the reliability of the evaluation metric, as noisy metric with low reliability is expected to yield poor
Metric Factorial Validity: Whether the ob- served metric scores align with the the- ory about unobserved factors underlying the scores. For example, scores on multiple evaluation metrics exhibit high correlations,

---

this might suggest the presence of com- mon underlying factor causing these scores to move unison.
f()) can be the identity function for normally dis- tributed observed scores, but when scores are or- dinal (e.g., expert ratings on a 5-point scale) or skewed, we suggest adopting an ordinal factor model (Muthén, 1984) where f()) is a step func- tion that evaluates whether Equ. 3 exceeds specific thresholds for each score category on the latent con- tinuum. Factor analysis can be exploratory con- firmatory. In the latter, select loadings (Ayas) are constrained to represent the theorized nomolog- ical network, e.g., an expert rating on consistency loads on no other dimensions. By establishing the Metric Factorial Validity through factor analysis, we could further develop more effective metrics by answering the following questions:
We introduce two statistical tools to evaluate these aspects of construct validity. Specifically, Mctric Convergent Validity and Metric Divergent Validity can be evaluated through the analysis of multitrait- multimethod (MTMM) table, and Metric Factorial Validity can be evaluated via factor analysis. Note that these validity evaluation methods will only inform if there is an underlying construct or how many of them are being captured. Defining what these constructs are will require further conceptual- ization and theorizing.
The MTMM table presents a way to scrutinize whether observed metric scores act in concert with theory on what they intend to measure, when two more constructs are measured using two more methods (Campbell and Fiske, 1959). For exam- ple, when evaluating a summarization model, re- searchers may ask several raters to rate the gen- erated outputs on four "traits" (aspects output quality), e.g., coherence, consistency, fluency, and relevance. In this case, the MTMM table allows examining whether, across different raters (evalua- tion methods), the raters' scores indeed appear to characterize the model's performance on four dis- tinct constructs. By convention, MTMM table reports the pairwise correlations of the observed metric scores across raters and traits on the off- diagonals and the reliability coefficients of each score on the diagonals. The analysis of an MTMM table is exemplified in Sec. 4.1.2.
Fit indices: For confirmatory factor analysis, how well does the theorized factorial structure align with the observed data?
Factor scores: What is an NLG model's factor score on a particular dimension?
Factor loadings: How much does a specific factor affect an observed metric score?
Residual correlation: For different evaluation metrics, are the residuals (unexplained score variation by the common factors) correlated, which may suggest additional dimensions?
Case Study
To illustrate how to apply METRICEVAL to eval- uate NLG evaluation metries, in this section, we ran a case study on evaluating summarization met- rics. As noted earlier, our evaluation focuses the metrics and the results should be interpreted as de- pendent the benchmark dataset used. We leave for future research explore the generalizability of the results across different benchmark datasets.
Factor Analysis examines Metric Factorial Va- lidity (e.g.. Thurstone, 1947) when the observed metric scores are assumed to measure a smaller number of unobserved factors. For example, scores from multiple evaluation metrics exhibit high correlations, this might suggest the presence of a common underlying factor causing these scores move in unison. Under factor analysis model, the distribution of the observed score on an indica- tor X,. such as a particular evaluation metric, is a function of a linear combination of the model's factor scores on K 2 general latent factors and the unique score U, on the indi- cator unexplained by the latent factor, including measurement error, i.e.,
4.1 Summarization Metric Evaluation
We analyzed the SummEval dataset (Fabbri et al.. 2021), a benchmark for summarization tasks. The benchmark contains 1700 summaries generated by 17 models on the CNN/Daily Mail dataset. In this dataset, each generated summary was rated by three experts, who provided 5-point-scale ratings on four dimensions: Coherence, Consistency, Fluency, and Relevance. We ran 16 types of popular automatic metrics that include rule-based metrics, embedding- based metrics, end-to-end metrics, and LLM- based metrics that are reference frasee or reference-free.
x +... Azkff + Uz).
(3)

---

Edibaoo
Reliability coef.
consdracy
dablity
St-pr
Sremarc
METBOR
RovicE--
RocuZ-d
Roticat-d
BLAbse"
SUPeRty
Moratione
BARFScco
G-BNAL G-BVAL--33-- -
Bertuayy
G-BVAL-4--NV
Metric
Figure 2: Estimated Metric Stability and Metric Consistency of common summarization metries.
see Appx.A.1. Since the score distributions on many evaluation metrics were skewed, we normal- ized automatic evaluation scores for the subsequent analyses, see Appx.A.2.
ple standard deviation in the average test dataset METEOR score was .38 and the metric consis- tency estimate was .966, translating to an expected measurement error due to score variability across the 100 data points of .38 x v1 .966 = .07. A METEOR score difference between two models less than .07 would thus be of limited interest, as the difference is less than the expected amount fluctuation in the score due to measurement error.
4.1.1 Metric Stability and Consistency
To evaluate an automatic metric's stability, we com- puted the metric score twice for each model's 's out- put on each data point, calculated two sets of av- erage scores for each model, and reported the cor- relation between the two sets of scores on the 17 models. A metric's consistency was evaluated via the coefficient in Equ. 2. Fig. 2 presents the Metric Stability and Metric Consistency estimates of select metries (full results in Fig. 5 in Appx). Most metrics achieved high stability. Metries with non-deterministic algorithms, such as LLM-based metrics G-Eval, displayed higher levels of measure- ment error in terms of Metric Stability. While com- pared to G-Eval with GPT3.5, G-Eval with GPT-4 yields higher stability. For Metric Consistency, for the ROUGE family, longer n-gram makes the metric less reliable and more prone to potential data perturbations in the test dataset. Therefore, to mitigate measurement error, for less stable L.LM- based metrics, the metric user should consider ag- gregating scores over multiple runs, and for less consistent metrics such as ROUGE-4, the evalua- tor should consider using larger test dataset. To illustrate, Fig. 3 shows the relationship between G-Eval metric consistency and test dataset size.
4.1.2 Metric Construct Validity
We begin by evaluating the construct validity of expert ratings in the SummEvvll dataset. These eval- uations were conducted in a confirmatory manner, assuming that the four ratings provided by each ex- pert summarization output's Coherence, Con- sistency, Fluency, and Relevance indeed measure the four distinct dimensions. Tab. 1 presents the MTMM table for the three expert's 's ratings on four dimensions. Metric Convergent Validity can be ex- amined by inspecting the bolded entries: Inter-rater agreements based on Kendall's on the same di- mension were high (.40 - .81) in general but lower for Fluency (.40 .63). Italic entries can inform the evaluation of metric divergent validity: Overall, an expert's 's ratings on different dimensions showed lower correlations than ratings by different experts on the same dimension, with the exception of Co- herence and Relevance, which sometimes showed correlations (underscored, .65 .71) nearly as high as those on ratings for Coherence (or Relevance) across raters (.69 .81). This may suggest that, although the expert raters were asked to separately rate Coherence and Relevance, they might inher- ently be rating the summarization outputs on the same underlying characteristic.
Conventionally, reliability coefficient above indicates good reliability. The metric stability and consistency estimates can help approximate the standard error of measurement of an average test dataset metric score (X), by observing from Equ.
that dE = ox Pxr- For example, the sam-
Confirmatory factor analysis was further con-

---

03
For each automatic evaluation metric, evalu- ated its concurrent validity: i.e., should the metric score be used to predict expert rating on dimen- sion as a more cost-efficient alternative? Different from prior studies (Fabbri et al., 2021), we report Kendall's r between each model's metric scores and the factor scores (instead of the raw means) based on expert ratings on the four dimensions. Metric consisencyy
0.7
G.FVAL 3S
G.FVAL 3.C.CN
0.0
G.EVAL
G.EAAL
055
G
G 4CCON
G
0.3
G AREL
40
60
100
Number of test examples
Figure 3: Estimated Metric Consistency for G-EVAL Metrics by number of data points in evaluation benchmark.
CIDE
53-pyr
SummaQA
METEOR
ROUGE-1
BertScore cortScore.J f
BLANC
SUPERT
Movers core
BARTScore
BLUERT
G --EVAL 4-Coh
6 EVALL EOVAL--COO -CCO
C-EVAL--FFLU
G-EVAL-4-RES
Coherence
0.29
0.29
0.10
0.47
0.65
0.46
0.65
0.60
0.8
0.6
0.4
Consistency
-0.25
0.56
0.69
0.54
0.12 0.0
0.65 0.63
0.22
0.59
0.01
0.62
0.72 0.50
0.63
0.2
Fluency
0.31
0.22
0.43
0.44
0.51
0.47
-0.2
-0.4
-0.6
Relevance
-0.8
Figure 4: Concurrent validity coefficients the selected metrics in predicting the four expert-rated dimensions' factor scores. Values based Kendall's
ducted (see Appx. A.4) to test the observed con- flated validity structure indicated by the MTMM analysis. The results show that the four-factor model fitted the observed data adequately well (Comparative Fit Index .999, Tucker-Lewin In- dex .999, Root Mean Square Error of Approx- imation .044 .05), supporting the thcorized loading structure, i.e., experts indeed rated on four factors. However, the estimated factor correlations suggested high correlations between dimensions, especially for Coherence and Relevance. This re- sult supports conflated validity structure.
tive conceptualizations of summarization quality, as suggested in recent works (Liu et al., 2022; Clark et al., 2023).
4.2 Metric Concurrent Validity
Since Coherence and Relevance distinct by definition (Fabbri al., 2021), the conflated valid- ity structure indicates potential issues in the expert rating process. Such issues may cascade if new automatic evaluation metrics were trained or vali- dated on these expert ratings. In this case, several remedial steps are advised, including (1) revisiting the dimensions' conceptual distinctiveness and, if needed, revising the theoretical framework; (2) the human-annotation guidelines could be reviewed; and (3) the test set could be examined assess ability distinguish model performance across dimensions. these actions are insufficient, we recommend that the community consider alterna-
The metric concurrent validity coefficients pre- sented in Fig. 4 (see full results in Appx Fig. 8). Although BARTScore and G-Eval were sensitive to detecting quality signals in all four expert-rated dimensions, the lack of variance in the validity coefficients surfaces another issue-theit lack of capability to distinguish different dimensions. For example, although G-EVAL-4-COH is designed for Coherence, it strongly correlated with Fluency (.75) and Relevance (.75). As a reference, its cor- relation with Coherence was .65, and the corre- lation between G-EVAL--FFLU (designed to as- sess Fluency) and Fluency was only .60. This in-

---

COH
CON
FLI
REL
Expart
Expert
Export
Expert
Espert
Expert
Expert
Expai
CO11
Expert
0.96
0.79
0.69
0.33
0.57
0.28
0.39
0.26
0.54
0.71
0.50
0.72
Espert
0.98
0.81
0.17
0.41
0.19
0.56
0.21
0.51
0.74
0.65
0.66
Expert
0.65
CON
Expert
0.98
0.74
0.79
0.44
0,48
0.46
0.36
0.41
0.33
Espert
0.98
0.68
0.64
0.44
0.6
0.57
0.67
0.55
Expert
FLU
Expert
0.97
0.53
0.63
0.65
0.74
0.72
Expert
0.96
0.4
0.38
0.41
0.37
Expert
0.57
REL
Expert
0.92
0.79
0.78
Expert
0.98
0.72
Expert
Table 1: Multitrait- Mulitmiethd table of expert ratings.
Notes: Diagonal entries are metric consistency coefficients between and 1. Entries in bold are the correlations ratings on the same dimension by different experts. Entries italic are the correlations on different dimensions by the same expert. Underscored entries coherence and relevance rating correlations by the same expert, which showed strong correlations.
dicates a systematic discrepancy between the G- EVAL-4-COH score and what it purports to mea- sure - Coherence instead of Fluency/Relevance. The MTMM table corroborates this lack of diver- gent validity for expert-based and G-EVAL met- rics (Appx. Tab. 2). Correlations in G-EVAL scores across dimensions frequently exceeded .70 (underscored italic entries), often exceeding the correlations on the same rated dimension across methods (bolded). On the contrary, SummaQA only reacts to Consistency which makes a more desirable metric for Consistency even though its correlation with expert rating was slightly lower than G-EVAL-4-CON, This might guide the refine- men/ddssambiguation of prompts for ALM--ased evaluations, in search of one that correlates strongly with the target dimension but is less confounded by the other nuisance.
ments (Sai al., 2021; Fabbri al., 2021; Liu et al., 2016). Beyond correlation, Ni mah et al. (2023) proposed a comprehensive framework checklist, aiming to verify the faithfulness of auto- matic metrics to human preferences at both aspect and system levels. Fomicheva and Specia (2019) analyzed the local dependency between metric and human judgments and looked into the consistency of human evaluation. However, the inconsistency and subjectivity human judgment, in addition to the non- con-transparent and non-standardized annota- tion process (Sai et al., 2021; Liu et al., 2016; Re- iter, 2018; Celikyilmaz et al., 2020; Kauchak and Barzilay, 2006; Sai et al., 2022), create shaking foundation. Another approach to evaluating met- rics is data perturbation and resampling (Caglayan et al., 2020; Sai al., 2021: Deutsch et al., 2021). Such method can diagnose a metric's consistency datasets.
and robustness across out-off-istribue In addition, researchers conducted qualitative anal- ysis (Zhang et al., 2019; Tao et al., 2018; Hanna and Bojar, 2021). Although qualitative analyses pro- vide in-depth insights, they are not scalable and cost-efficient. Closely aligned with our efforts, Von Däniken et al. (2022) introduced a theoreti- cal framework to examine the reliability of binary metrics.
4.3 Summary of the Case Study
Our findings indicate that metrics based on L.LSs exhibited lower stability, some below the conven- tional threshold .9. the ROUGE metric fam- ily, an increase in n-gram length was associated with decreased metric consistency, heightening sus- ceptibility to benchmark task perturbations. Both MTMM and factor analysis identified confla- tion between expert ratings of Coherence and Rel- evance. Lastly, while BARTScore and G-Eval demonstrated high agreements with expert-rated dimensions, the lack of variability in metric con- current validity suggested a lack of differentiation between theorized dimensions.
Conclusion
Evaluation metrics inform model capability and guide model development. Drawing from the core concept of reliability and validity in measurement theory, present METRICEVAL, framework that conceptualizes and operationalizes four key desiderata for NLG metries. With collection of statistical tools, METRICEVAL. offers the commu- nity an effective and principled way to analyze, evaluate and understand NLG evaluation metrics.
5
Related Work
NL. evaluation metrics undergo validation through various methods. The most widely used method is examining correlation with human judg-

---

Limitations
Anja Belz, Anastasia Shimorina, Shubham Agarwal. and Ehud Reiter. 2021. The reprogen shared task reproducibility of human evaluations in nlg: Overview and results. In Proceedings the 14th International Conference on Natural Language Gen- eration, pages 249-258.
Evaluating evaluation metrics for NLG models should not be treated as a single-shot task. Instead, as suggested in Messick's s unified theory of valid- ity (Messick, 1995), it is essential to continuously gather cumulative evidence of validity to ensure the ongoing effectiveness and reliability of the metrics. The process of accumulating valid evidence is an iterative and dynamic endeavor that aligns with the evolving landscape of NLG models and their ap- plications. Future studies are necessary to collect other types of evidence, such as a metric's 's ability to predict users' preferences, to continuously evaluate the effectiveness of an NLG metric.
Ozan Caglayan. Pranava Madhyastha, and Lucia Spe- cia. 2020. Curious case of language generation eval- uation metrics: A cautionary tale. arXiv preprint arXiv:2010.1355
Donald Campbell and Donald W Fiske. 1959. Convergent and discriminant validation by the multirait-mulimmehod matrix. Psychological bul- letin, 56(2):11
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey, arXiv preprint arXiv:2006. 14799.
Measurement errors may surface and accumulate at every stage of the evaluation process, including benchmark design, data collection, etc. To perform the analysis of evaluation metrics, have as- sume the reliability and validity of the other parts of the evaluation process. Therefore, the results of the case study should be interpreted as dependent on the benchmark used, e.g., CNN/Daily Mail dataset. Future study is required to study the generalizabil- ity of the results across different benchmarks.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A Smith. 2021. All that's 'human't is not gold: Evaluating hu- man evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296.
Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur P Parikh. 2023. Seahorse: A multilingual, multifaceted dataset for summarization evaluation. arXiv preprint 2:Xxv:3005.3199
Our framework does not aim to provide compre- hensive coverage of all measurement error sources in NLG evaluation metrics. For example, we did not discuss predictive validity in our framework de- spite its importance in education and psychological testing. We encourage researchers and practitioners to extend our framework for other types of relia- bility and validity and build datasets to support more comprehensive analysis, c.g., dataset with the model's real-world performance, to deepen our knowledge of NLG metric evaluation.
Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika, 16(3):297-334
Lee Cronbach and Paul E Meehl. 1955. Construct lidity in psychological tests. Psychological bulletin, 52(4):281.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation met- ries using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132 1146.
A Acknowledgements
We would like to thank the reviewers for their thought-provoking comments as well as Nicolas Le Roux and Jackie Chi Kit Cheung for their help- ful discussions and feedback.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv:1810.0445
References
Alexander R Fabbri, Wojciech Kryšciñssii, Bryan Me- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguisties, 9:391-409.
Mary J Allen and Wendy M Yen. 2001. Introduction measurement theory. Waveland Press.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor; An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
Marina Fomicheva and Lucia Specia. 2019. Taking mt evaluation metries to extremes: Beyond correlation with human judgments. Computational Linguistics. 4553)-515-558.

---

Yang Gao, Wei Zhao, and Steffen Eger. 2020. Supert: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. arXiv preprint 1rXiv:2005.03724.
Yixin Liu, Alexander Fabbri. Pengfei Liu, Yilun Zhao, Linyong Nan. Ruilin Han, Simeng Han, Shafiq Joty. Chien-Sheng Wu, Caiming Xiong, et al. 2022. Re- visiting the gold standard: Grounding summariza- tion evaluation with robust human evaluation. arXiv preprint
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sel- lam. 2022. Repairing the cracked foundation: A sur- vey of obstacles in evaluation practices for generated text. arXiv preprint ar xxXi::2202.0633 06935.
Samuel Messick. 1995. Validity of psychological sessment: Validation of inferences from persons' re- sponses and performances as scientific inquiry into score meaning. American psychologist, 50(9):741.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv:1804.1128:
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Michael Hanna and Ondiej Bojar. 2021. A fine-grained analysis of bertscore. In Proceedings of the Siath Conference on Machine Translation, pages 507-517.
Bengt Muthén. 1984, A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 15-132
David Howcroft, Anya Belz, Minuna Clinciu, Dimitra Gkatzia, Sadid A Hasam, Saad Mahamood, Simon Mille, Emiel Van Miltenburg. Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nig needs evaluation sheets and standardised definition. Association for Compu- tational I. Linguistics (ACL).
Preksha Nema and Mitesh M Khapra. 2018. Towards better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950-3959.
David Kauchak and Regina Barzilay. 2006. Paraphras- ing for automatic evaluation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference. pages 455-462.
lun- Ng and Viktoria Abrecht. 2015. Better summa- rization evaluation with word embeddings for rouge. arXiv preprint 1rXXv::500.0000
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel Weld. 2021, Genie: Toward reproducible and standardized human evaluation for text generation. arXiv preprint arXiv:2101.06661.
Iftitahu Ni' mah. Meng Fang. Vlado Menkovski, and Mykola Pechenizkiy. 2023. Nlg evaluation metrics beyond correlation analysis: An empiri- cal metric preference checklist. arXiv preprint 1rXiv:2305.08555
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. 2015. From word embeddings to document distances. In International conference machine learning, pages 957-966. PMLR.
Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. arXiv preprint 1rXiv:1707.06875
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-tee cerenneett-sequuee to-sequence pre- teairiiinng for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.11 13461.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguisties, pages 311-318.
Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches ouf, pages 81-
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- ries for dialogue response generation. arXiv preprint arXiv:1603.0800 080023............
Maja Popović. 2017. chrf++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612-618.
Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 44(3):393-401
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nig evaluation using gpt-4 with better human align- ment. arXiv preprint 2303 16634.
Ananya B Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh Khapra. 2021. Perturbation checklists for evaluating nlg evaluation metrics. In

---

Proceedings of the 2021 Conference Empirical Methods Natural Language Processing, pages 7219-7234.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0967
Ananya Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation met- used for nlg systems, ACM Computing Surveys
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. arXiv preprint arXiv: (00::9999.000 02622..
(CSUR), 55(2):1-39.
Thomas Scialom, Sylvain Lamprier, Benjamin Pi- wowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. arXiv preprint arXiv: 09090006100
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing nlg evaluation: Evaluation practices, assumptions, and their implications. arXiv preprint arXiv:2205.06822
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt:
Learning robust metrics for text gener- ation. arXiv preprint arXiv:2004, 04696.
Emily Sheng, Kai-Wei Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. arXiv preprint arXiv:2105.04054.
A
Appendix
A.1
Metrics
C. Spearman. 1904. General intelligence, objectively determined and measured. The American Journal of Psychology, 2522)220
Our selection of evaluation methods includes popu- lar metrics for NLG tasks including both reference- based and reference-free metrics. Compared to the original SummEval dataset, we additionally se- lected end-to-end metrics and recent LLM-basee metrics.
Charles Spearman. 1910. Correlation calculated from faulty data. British journal of psychology, 3(3):271.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for au- tomatic evaluation of open-domain dialog systems. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
ROUGE (Lin, 2004) evaluates the generated summary by comparing the number of overlapping word sequences (n-grams) with set of reference summaries.
Louis Leon Thurstone. 1947. Multiple-factor analysis; a development and expansion of the vectors of mind.
ROUGE-WE (Ng and Abrecht, 2015) ex- pands on ROUGE by incorporating soft lcxical matching, which utilizes the cosine similarity of Word2Vec (Mikolov et al., 2013) embeddings.
Oleg Vasilyev, Vedant Dharnidharka, and John Bohan- non. 2020. Fill in the blanc: Human-frea quality estimation of document summaries. arXiv preprint arXiv:2002.09836.
Ramakrishna Vedantam, C Lawrence Zimick, and Devi Parikh. 2015. Cider: Consensus-based based image de- seription evaluation. In Proceedings of the IEFF conference computer vision and pattern recogni- tion, pages 4566 4575.
S3 (Peyrard et al., 2017) is a model- met- ric that combines existing evaluation metrics like ROUGE, JS-divergence, and ROUGE-WE It uti- lizes these metrics input features predict the evaluation score.
Pius Von Däniken, Jan Deriu, Don Tuggener, and Mark Cieliebak. 2022. On the effectiveness of automated metrics for text generation systems. arXiv preprint arXiv:2210. 13025..
BertScore (Zhang al., 2019) calculates sim- ilarity scores by aligning the generated and refer- ence summaries at the token-level. Token align- ments are determined greedily to maximize the cosine similarity between contextualized token em- beddings from BERT (Devlin et al., 2018).
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le. and Denny Zhou. 2022. Chain thought prompting elicits reasoning large language models. arXiv preprint 2xXiv:2201.19900
Catherine Yeo and Alyssa Chen. 2020. Defining and evaluating fair natural language generation. arXiv preprint 2008 01588.
MoverScore (Zhao et al., 2019) quantifies the semantic distance between summary and ref- erence text by utilizing the Word Mover's Dis- tance (Kusner et al., 2015). This distance measure operates over n-gram embeddings obtained from BERT representations.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text text gener- ation. Advances Neural Information Processing Systems, 34:27263-2793

---

SummaQA (Scialom al., 2019) utilizes BERT-based question-ant -anserinn model to respond to cloze-style questions using generated summaries. This metric provides both the Fl overlap score and the confidence of the QA model.
G-Eval (Liu et al., 2023) is framework that leverages LLM with Chain-of-Thoughts (CoT) (Wei et al., 2022) to evaluate the quality of generated text. The generated outputs are as- sessed using set of prompts along with generated CoT.
BLANC (Vasilyev et al., 2020) is reference- less metric that assesses the performance improve- ment of pre-trained language model when pro- vided with document summary while performing language understanding tasks on the original docu- ment's text.
Data Statisties (Grusky al., 2018) define three measures of dataset extractiveness: extrac- tive fragment coverage, density, and compression ratio. Extractive fragment coverage quantifies the percentage of words in the summary that are de- rived from the source article, indicating the degree to which the summary is derivative of the original text. Density represents the average length of the extractive fragment to which each summary word belongs. Compression ratio measures the word ratio between the articles and their summaries.
SUPERT (Gao al., 2020) is a reference-less metric that measures the semantic similarity be- tween model outputs and pseudo-reference sum- maries generated by extracting significant sen- tences from the souree documents using soft token alignment techniques.
A.2 Metric Normalization
BLEU (Papineni et al., 2002) is metric that fo- cuses on precision the corpus level. calculates the n-gram overlap between a candidate utterance and a reference utterance while incorporating a penalty for brevity.
Initial exploratory analysis revealed that the score distributions on many evaluation metrics were skewed. We thus normalized each automatic evaluation score (via the transformation Xj Φ EN S Xj))) and subsequently worked with normalized automatic metric scores, which approximately followed . (0, 1) distribution and are more appropriate for correlational analysis and linear models.
CHRF (Popović, 2017) measures character- based n-gram overlap between model outputs and reference documents.
METEOR (Banerjee and Lavie, 2005) deter- mines an alignment between candidate and refer- ence sentences by mapping unigrams the gener- ated summary to 0 or unigrams in the reference, taking into account stemming, synonyms, and para- phrases.
A.3 Metric Stability and Consistency Results
Fig. 5 presents the Metric Stability and Metric Consistency estimates of all automatic evaluations and the metric consistency estimates of all expert and automatic metrics.
CIDer (Vedantam et al., 2015) calculates the co- occurrence of 1-4 gram units between the candidate and reference texts, giving less weight to common n n-grams and computing cosine similarity between the n-grams of the candidate and reference texts.
A.4 Confirmatory Factor Analysis on Expert Ratings
Confirmatory factor analysis was further conducted on the (3x4) expert ratings, assuming that each rating loads only on the corresponding dimen- sion. Given that the expert ratings were highly skewed (see Fig. in Appx.), an ordinal factor model (Muthén, 1984) was fitted. Judging from commonly used fit indices (Comparative Fit Index .999, Tucker- Index .999, Root Mean Square Error of Approximation the four-factor model fitted the observed data ad- equately well, supporting the theorized loading structure, e.g., Experts rated on four factors. Tab in Appx reports the estimated factor loadings and thresholds of each expert rating, assuming that
BARTScore (Yuan et al., 2021) evaluates text directly based on the probability of being generated from or generating other outputs. It addresses the modeling challenge using pre-trained sequence- to-sequence (seq2seq) model called BART (Lewis al., 2019).
.047 < .05),
BLEURT (Sellam et al., 2020) is BERT-based metric that can model human judgments with few thousand training examples, which may introduce some bias.

---

1.00
0.95
Reliability coef.
Esimmale
consistency stability
0.90
0.85
Metric
Figure Metric stability and consistency estimates for all expert- and metric-based scores.
the four latent factors each have mean 0 and SD
which capture the unexplained variance by the 4 dimensions' factor scores. Plot of the first two prin- cipal components is shown in Fig. 7. Here, visual clusters of evaluation metrics found, suggest- ing that select metries likely tapped common additional dimensions. The unexplained residual variance may guide future investigation on discov- ering other quality signals in summarization tasks.
of 1. Loadings were generally high (adopting the 2 .4 convention) but varied across experts and di- mensions (generally lower for Relevance). Rater differences were also found in their leniency: For instance, expert 3 was more likely than experts and to provide rating of 5 (with lower threshold estimates for score 5) on output Consistency, but less likely to do so (with higher threshold estimates) on Coherence and Relevance. The estimated fac- tor correlations below suggested high correlations between dimensions, especially for coherence and relevance:
Coherence
Consistency
Fluency
Consistency .31
Fluency
.66
.68
Relevance
.66
.64
.53
A.5 Multitrait-uulimethhd Table for G-EVAL and expert-based metrics
Table 2 presents the multitrait-multinethon table the four dimensions, Coherence, Consistency, Fluency, and Relevance, for three separate rating methods: expert-based ratings (average factor score across three raters), G-EVAL-3.5, and G-EVAL-4.
A.6 Residual Analysis
We further performed principal component analy- sis on the residuals of the automatic evaluations,

---

1.00
0.75
as Factort Score)
0.25
0.00
Exper Atlle
Opent Caad
Kiger Ciby
Eapar Stu
Fiper
Metric
Figure Distribution 5-point scale expert ratings.
0.40
0.25
0.65
Expon
0.57
0.59
0.46
0.56
0.56
0.25
0.73
0.60
0.30
11.71
U163
Relecance
0.50
0.56
0.65
8.3
0.25
0.65
0.72
1.76
Cohrnmra
0.96
0.59
0.67
0.70
8.65
0.54
0.53
0.63
GEVAL
Comistancy
Flaency
0.90
0.60
0.5
0.7
0.87
0.55
0.65
0.51
0.47
0.35
0.51
11.54
1165
Relevance
0.95
0.74
0.57
0.53
0.69
Cohorence
0.98
0.69
0.77
0.50
G-FVAL
Concatrncy
Flrncy
0.96
0.97
03.4
0.69
0.71
Relevance
1196
Table 2: Multitrait- Mititraat-Multtmethhdd table of the pairwise correlations between expert rating factor scores, GPT3.5-basce
scores, and GPT4-based scores.
Notes: Entries in bold the correlations ratings the dimension by different methods. Entrics in italic are the correlations the ratings on different dimensions using the same method. Except for reliability coefficients, entries over 7 are underscored. While cutoff arbitrary. underscoring is more desirable bolded entries (indicating good convergent validity) and less for italic entries (indicating method bias, worse divergent validity).

---

Loading fluency
Threshold
coherence
consistency
relevance
2
3
4
5
expert expert_l_cohrrereee cchherence
0.90
-2.03
-1.08
-0.26
0.31
expert _xpher_2_coherennee
cohereecce
0.88
-1.09
-0.55
-0.12
0.76
-1.74
-0.67
0.56
1.37
expert 1 consistency
0.97
-2.15
-1.56
-1.20
-1.07
expert 2 consistency
0.98
-1.63
-1.36
-1.27
-1.10
expert_3. Sopert_3_consssiei consistency
1.00
-2.16
-1.55
-1.33
-1.23
expert fluency
0.98
-2.60
-193
-1.26
-1.02
expert_ 2_fluency
0.88
-2.12
-174
-1.09
-0.80
expert 3 fluency
0.92
-2.49
-181
-1.35
-1.10
expert_ _eeee___relevance reelevance
0.79
-2.25
-1.00
-0.55
0.45
relevance 2eeeance expert_ reelevance
0.85
1.95
-123
-0.65
0.22
0.64
-2288
-1.19
0.01
1.26
Table Factor loading and category threshold estimates for the 4-ctct confirmatory model of ordinal expert ratings.
Nowel
Novel bi-gram
statt-compression
CIDE
Novel uni-gram
20
BertScorp_E
RoliGE-sut- ROUGE-1
MoverScore
BLUERT
BLEU BLEU-ROUBBE Be-- -
Repeated trl tri-gram
G-EVAL-355 -EL-EL- --VVAL---FLU
ROUGE -we-
BertScore BOUGE-2
C--EAL----COOM
G-EVAL-4-CON G-EVAL
ROUGE -ROUGE-WE -
G-EVAL--S-HEL
BARTScor CHRF
Repeased bingram G-EVAL-3.S-COH G-EVAL-I-COH
METEOR * 53-resp
Repested uri-pram
Bertsore_r 53-9y
Stasstoverage
SummoQ4
Stats -lngth
SUPERT
Stats densty
PC1
Figure Plot first two principal components of the residuals of the 4-factor model.
um gren
diggoon
SOuCE me-d
Boel ar
SINCE
Consetercy
Fluerey
Relewnce
Figure 8: Concurrent validity coefficients of the metric-based scores in predicting the four expert-rated dimensions' factor scores. Values are based on Kendall's r.


Reviews:
[{"review_id": "IJHv5vKK4z", "reviewer": "Reviewer_QqGJ", "summary": "", "questions": "A. Since being sensitive to signals in the four SummEval dimensions alone doesn’t make for a good metric due to lack of discrimination between the dimensions, have you tried creating a weighted combination of the two factors to evaluate the overall quality of a metric?\n\nB. It would be helpful to rank all metrics by their overall quality in order to understand which type generally tends to work better (lexical vs. semantic etc.) and if there is a certain direction in which metric creation research should focus.\n\nC. Why are the self-correlations of values in Table 1 not equal to 1.0? E.g. for Expert 1 Coherence the value is 0.96.\n\nD. Did you perform any case study on a different kind of task (e.g. more creative text generation) to compare how the metrics' behavior varies between them?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We agree with your comments and we are revising our manuscript to reflect those changes accordingly.\n\nPlease find hereafter our answers to your questions:\n\n**Other sources of error beyond those discussed in the paper and an investigation of where they occur in the evaluation process.**\n\nWe fully agree that there are other sources of measurement issues, including the choice of dataset, instructions for human annotators, score aggregation methods, etc. As we mentioned in the paper, the validity and reliability of an evaluation process depend on every individual evaluation component and related design choices. The primary goal of our paper is to bridge measurement theory to model evaluation and focus on two important sources of measurement errors, i.e., validity and reliability, that have been extensively studied in measurement theory but less so for model evaluation. Our current framework focuses on evaluation metrics, given the importance of metrics and their downstream implications on model comparison. In our future work, we plan to study measurement theory in the broader context of model evaluation, identify measurement issues beyond metrics, and propose new evaluation methods. In the revised version, we will further acknowledge the scope of the current framework and highlight other potential sources of measurement errors.\n\n**The related work section could be more comprehensive and discuss the merits and limitations of prior works in more detail.**\n\nWe want to thank you for suggesting other related work. In the revised version, with the extra page, we will discuss the prior work and its connection with our current work in more detail.\n\n**Since being sensitive to signals in the four SummEval dimensions alone doesn’t make for a good metric due to lack of discrimination between the dimensions, have you tried creating a weighted combination of the two factors to evaluate the overall quality of a metric?**\n\nWe did not try to combine those two factors as a single quality dimension in this case study. Although a combined score may be more straightforward to compare metrics, it is less informative in diagnosis. For example, for the discrimination power, troubleshooting often needs to know where the undesirable correlation comes from. Also, the weighting function for discrimination power on individual dimension and general sensitivity is often context dependent. We need future studies to understand how to formulate such a weighing function.\n\n**It would be helpful to rank all metrics by their overall quality in order to understand which type generally tends to work better (lexical vs. semantic etc.) and if there is a certain direction in which metric creation research should focus.**\n\nIn the current case study, we focused on demonstrating how our framework can be applied to evaluate different metrics rather than providing an overall ranking. In addition, we decided not to rank different metrics based on the single case study. Given the performance of a metric is dependent on the chosen dataset and task (as we highlighted in our paper), ranking metrics based on this study may generate misleading interpretations (e.g., metric’s general performance). However, we agree that a benchmark for metrics is based on the analysis of multiple datasets and multiple tasks will provide more informative comparisons. Such a benchmark will allow researchers to answer questions like in what context a metric is a good metric and practitioners to select metrics for their own use cases. This is one of our ongoing work and we hope to share the results soon. \n\n**Why are the self-correlations of values in Table 1 not equal to 1.0? E.g., for Expert 1 Coherence the value is 0.96.**\n\nThe diagonals of the MTMM table present the metric internal consistency estimates of each evaluation metric based on Equation (2), which are bounded above by 1. This informs the evaluation of the extent to which the observed score correlations on a pair of variables are attenuated by the reliability of each variable.  We will add this to the caption for clarity.\n\n**Did you perform any case study on a different kind of task (e.g. more creative text generation) to compare how the metrics' behavior varies between them?**\n\nWe have not applied our framework to another case study. However, we agree with the importance of analyzing metrics on different kinds of tasks and the comparison will better help us answer the question of in what context an evaluation metric is good. Towards this effort, we will release a Python toolkit based on our framework to encourage a community effort to deepen our understanding of how evaluation metrics perform in different tasks. \n\\\n\\\n\\\nWe hope that our responses have addressed your concerns and questions. We have already updated our draft with a more comprehensive related work and fixed all grammatical and formatting issues you identified. Thanks again for your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "As NLG models improve over time, finding good metrics to accurately measure the quality of their output in a way that best approximates their real-world performance is an important task. This paper presents an interesting idea from educational testing that breaks down the required qualities of a good metric and shows how they relate to different factors. The paper is well written and the ideas are communicated clearly. The authors also present a case study using text summarization as an example task and show how different metrics score along different dimensions, and what this means for their robustness and validity. The authors plan to make the code for their framework publicly available as well.", "reasons_to_reject": "There can be other sources of error beyond those discussed in the paper (and an investigation of where they occur in the evaluation process), although the authors concede this in the limitations section already. The related work section could be more comprehensive and discuss the merits and limitations of prior works in more detail.", "questions_to_authors": "A. Since being sensitive to signals in the four SummEval dimensions alone doesn’t make for a good metric due to lack of discrimination between the dimensions, have you tried creating a weighted combination of the two factors to evaluate the overall quality of a metric?\n\nB. It would be helpful to rank all metrics by their overall quality in order to understand which type generally tends to work better (lexical vs. semantic etc.) and if there is a certain direction in which metric creation research should focus.\n\nC. Why are the self-correlations of values in Table 1 not equal to 1.0? E.g. for Expert 1 Coherence the value is 0.96.\n\nD. Did you perform any case study on a different kind of task (e.g. more creative text generation) to compare how the metrics' behavior varies between them?", "paper_topic_and_main_contributions": "This paper focuses on evaluation metrics in natural language generation, proposing a new framework for evaluating these metrics inspired by measurement theory. The paper introduces to NLG desiderata and methods from educational and psychological testing and proposes a framework for analyzing these metrics. It also shows how these evaluation approaches may be used in practice by performing a case study on the SummEval dataset and comparing the performance of different metrics on it.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. Transactions of the Association for Computational Linguistics, 9:1132–1146.\n\nManik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating Evaluation in Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguistics.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 041: “​​of tasks, multifacet[ed] and context-dependent”\n2. Line 240: “errors, and so on[.]”\n3. Line 245: “The goal of evaluating a [metric’s] reliability”\n4. Line 250: “To empirically estimate $\\sigma_E$ is [done] via the reliability”\n5. Line 437: “and [Metric] Factorial Validity”\n6. Line 518: “on [the] CNN/Daily Mail benchmark”\n7. Line 553: “the average score[,] and for less consistent”\n8. Line 688: “ability to predict users’ [satisfaction (?)]”\n9. Missing space after comma in Figure 1 caption.\n10. Width of Figure 2 can be increased to improve readability of x-axis tick labels as there is enough space on the sides.\n11. Cell values close to 0 in Figure 3 are hard to read due to the color scale.\n12. Legend title needs to be fixed in Figure 5. The Y-axis label also needs to be capitalized.\n13. The notes under Table 1 can be written in the caption instead.\n14. Extra space before 0.30 (second column, first row) in Table 1.\n15. Extra space before 0.48 (last column, fourth row) in Table 1."}}, {"review_id": "PzqntNVHpX", "reviewer": "Reviewer_rppc", "summary": "", "questions": "Overall, I think the authors did thorough work in translating developments in measurement theory to be applicable to NLG. However, I did run into a few questions and uncertainties regarding the reporting and methodological choices. I will list these below.\n\n- In general (mostly §2): I think the argument for why measurement theory should be applied in this context is not sufficiently argued. For instance, it is not clear to me what exactly the benefits are of applying measurement theory in this context, and what makes the application of measurement theory in this context so unique.\n- In general: Most of the described measurement theory tests have been developed for human evaluation. To what degree do they also apply to metrics? For instance, stability seems irrelevant for most metrics, as the formula to calculate the score is fixed for these metrics.\n- In general (mostly §4): Why are the authors looking specifically at evaluating summarization metrics and the CNN/Daily Mail dataset? What makes this task/dataset relevant? And how generalizable are the results of this study for other tasks/datasets?\n- In general (mostly §4): While the authors go into detail about the results per measurement criteria, I am missing a higher-level conclusion from the case study's results: which metrics perform well, and which do not perform well? Why do they (not) perform well? What kind of information did we obtain that we would not have obtained if we would have just looked at correlations? I think such a higher level conclusion of their case study would greatly help other researchers to understand their framework and make it more attractive to use said framework.\n- 551-553: \"for less stable LLM-based metrics, the evaluator should consider applying the metrics multiple times\" -->  Is it really less stable as the lowest it scores are around 0.98?\n- 582-584: \"Inter-rater agreements on the same dimension were high (.48 − .89) in general but lower for Relevance (.48 − .58).\" --> Which inter-rater agreement metric was used here?\n- 610-618: \"Since Coherence and Relevance are distinct constructs by definition (Fabbri et al., 2021), the conflated validity structure indicates potential issues in the expert rating process. One source of such systematic measurement error may come from unclear instructions or the expert rater’s individual leniency. Such issues may cascade if new automatic evaluation metrics were trained or validated on this dataset.\" --> Is there a way to fix this, post-hoc? For instance, by merging these criteria into one single construct?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very constructive feedback!  We will address your comments and will acknowledge the limitations of the paper in our revisions. \n\n\nWe hope our clarifications will address your concerns. \n\n\n**Uncertainty to what degree measurement theory (which has been developed to assess human evaluation) can also be applied to automatic metrics.**\n\nIn the revision, we will elaborate on the rationale behind extending measurement theory to the NLG evaluation context, in a few aspects:\n- *Similar measurement objective:* Both the measurement of human capability and NLG model’s capability use observable tasks to estimate unobservable capability. The need for measuring NLG’s model unobservable capability is increased as more models are referred to as general-purpose model. By bridging measurement theory for human capabilities to NLG evaluation, our work provides a more principled way of reasoning the metrics design, analyzing metric’s behavior, and improving evaluation methods. \n- *Similar assumptions, tasks, and goals:* Both the measurement of human and the evaluation of NLG evaluate output on each task with a rubric (eval metric) defined based on assumptions of what constitutes successful task completion, compute an aggregate score (e.g., sum/average) on a set of specific tasks, and use the aggregate score for inferences about candidate's (model’s) capability in specific domains and future performance on tasks beyond what’s included in the test (or test set).\n- *Theory driven:* Measurement theory provides a theory-driven framework for enumerating (1) the assumptions made in developing, using, and interpreting (both automatic and human-based) NLG evaluation metrics, and (2) the different sources of measurement error (e.g., task-, rater-, and metric-specific) that lead to random fluctuation in a model’s score on a metric. It also offers a methodology that guides collecting and analyzing NLG evaluation data, to inform users on (1) uses and interpretations of eval scores supported by empirical evidence, and (2) to what extent differences across models on an eval can be attributed to signal versus noise.  \n\nWe also want to clarify that the measurement theory applies to both human evaluation and automatic scores. The measurement theory does not make assumptions on who derived the score or how the score is calculated. In our framework, an evaluation metric is a scoring function that transforms a model output into a numerical score. The process to derive the score could be human evaluation, rule-based metrics, learned metrics, or LLM-based metrics. \n\n\n\n\n**Stability seems irrelevant for most metrics, as the formula to calculate the score is fixed for these metrics.**\n\n\nAs we see increasing use of automatic evaluation metrics with built-in stochasticity (e.g., LLM-based metrics), the stability of an eval in producing consistent scores on an output from one replication to another will be increasingly relevant. We will add this point in the revision.\n\n\n**Case study somewhat limited in scope in regard to the task (only summarization). Why are the authors looking specifically at evaluating summarization metrics and the CNN/Daily Mail dataset? What makes this task/dataset relevant? And how generalizable are the results of this study for other tasks/datasets?**\n\n\nThe case study is limited in this scope. However, the goal of this case study is to offer a demonstration of how to apply our proposed framework and show the framework’s practical utility. As noted in the paper, our evaluation of metrics should be interpreted as dependent on the benchmark design, and is not meant to be generalizable (though some generalizations can be derived by studying multiple benchmarks, which we plan to explore in future work). We will further acknowledge the limited scope of this case study, and provide the following rationales for choosing this task as the case study. \n- *Importance and challenge of summarization tasks:* Summarization is a critical and complex task in the field of NLP, requiring a model to process the content, grasp the context, extract salient information, and output in a fluent manner. Given the complex process, a model’s summarization capability cannot be directly observed, which aligns with the assumption of measurement theory.  \n- *Evaluation difficulties:* Assessing the quality of summaries is inherently challenging, our community has invested a lot of effort in evaluating summarization evaluation metrics and developing novel metrics. Although our framework could be applied to other NLG metrics, we hope our case study could contribute to the summarization tasks and aid the ongoing effort.\n- *Real-world relevance:* Summarization is a common task in various applications, including news summarization, meeting summarization, and long documents (e.g., academic papers) summary. Selecting informative evaluation metrics is important to inform model development and model selection. Therefore, we want to ground our case study in this important field.\n- *Dataset significance:* The SummEval dataset, which includes valuable expert ratings, has been widely used to train and validate novel metrics in summarization. Although it is only built on CNN/DM, it is a common dataset used in model development. We are also aware of some recent novel evaluation datasets, such as Seahorse [1]. We did plan to expand our analysis of those datasets as part of our future work. \n\nTo also encourage others to apply our framework to other benchmarks to inform generalizability, we will release a  Python toolkit based on our framework to encourage the community to use our framework to evaluate evaluation metrics for different datasets and tasks. We are currently working on building a centralized benchmark for analyzing evaluation metrics under multiple datasets and multiple tasks. We hope such a benchmark will allow us to gain insights into what context a metric is a good metric and allow practitioners to select metrics for their own use cases. \n\n\n**Limited relevant work section.**\n\nWe appreciate your comments and suggestions. We have extended our related work with the extra page to provide a more comprehensive view of how our framework is situated in prior work and make our contribution more clear. \n\n\n**High-level conclusion from the case study results: which metrics perform well, and which do not perform well? Why do they (not) perform well? What kind of information did we obtain that we would not have obtained if we would have just looked at correlations?**\n\n\nWe will add another paragraph at the end of the case study to summarize our findings. Here, we gave a brief summary of the major takeaway,\n\n\n- *Metric Stability:* Compared to other metrics,  LLM-based metrics show lower stability. Also, by convention (.9), most of the LLM-based metrics can be considered stable. Some LLM-based metrics such as G-EVAL-3.5-Consistency exhibit unsatisfactory stability. \n\n\n- *Metric Consistency:* For the ROUGE family, a longer n-gram makes the metric less reliable and more prone to potential data perturbations in the test dataset. Both metric stability and consistency coefficients could help us compare true model differences by estimating the standard error on the metric score. \n\n\n- *Metric Construct Validity:* Through MTMM and factor analysis, we found in SummEval, expert ratings on Coherence and Relevance are conflated. The conflated validity construct may cascade to metrics that are trained or validated on this dataset. \n\n\n- *Metric Concurrent Validity:* We found although BARTScore and G-Eval are sensitive to detecting\nquality signals in all four expert-rated dimensions, the lack of variance in the validity coefficients indicates their lack of capability to discriminative different dimensions. \n\n\n\n\n\n\n\n\n**551-553: \"for less stable LLM-based metrics, the evaluator should consider applying the metrics multiple times\" --> Is it really less stable as the lowest it scores are around 0.98?**\n\n\nIn the appendix, we report stability estimates of all metrics. It can be seen that for some specific GEVAL dimensions other than the average across dimensions,  the metric stability is below .9 for GEVAL-3.5-Consistency. In addition, we want to clarify the attenuation effect of in-stable metrics. We believe although current metrics achieve high stability, metric stability is an important aspect of a metric as an in-stable metric may amplify its measurement error from other aspects of the metric.\n\n\n**582-584: \"Inter-rater agreements on the same dimension were high (.48 − .89) in general but lower for Relevance (.48 − .58).\" --> Which inter-rater agreement metric was used here?**\n\n\n We calculated Kendall's rank correlation in the model-level average scores. We will clarify it in our paper.\n\n\n**610-618: Is there a way to fix the conflated validity structure, post-hoc? For instance, by merging these criteria into one single construct?**\n\n\nWhen the conflated validity structure occurs, it often indicates the evaluator (i.e., expert rater) can not differentiate the two constructs very well. In this case, a few diagnostic or remedial steps should be taken:\n- Revisit the definition of those dimensions in terms of whether they are conceptually distinct. If they are not, consider revising the theoretical framework.\n- Revisit the instructions provided for human-annotation. \n- Revisit the test set to examine whether the specific tasks employed for evaluating the model are capable of distinguishing model capability on the two dimensions.\n\nIf the issue can not be fixed, we would suggest the community rethink the conceptualization of each individual aspect of summarization quality. For example, in a few recent works, such as [1,2], the researcher suggests new constructs for summarization. Post-hoc-ly speaking, merging two constructs is one possibility, if we conclude that two factors are explained by the same underlying construct. However, we should also rule out potential alternative explanations, in particular, problematic rubrics (e.g., ambiguous directions provided to expert raters) or problematic test set that cannot distinguish model capabilities on separate dimensions.\n\\\n\\\n\\\nWe want to thank you again for your valuable and detailed comments and for being receptive to our idea. We will also do another thorough pass to fix all minor issues. We hope our rebuttal addressed your comments.\n \n[1]Clark, E., Rijhwani, S., Gehrmann, S., Maynez, J., Aharoni, R., Nikolaev, V., ... & Parikh, A. P. (2023). SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation. arXiv preprint arXiv:2305.13194.\n\n[2]Liu, Y., Fabbri, A. R., Liu, P., Zhao, Y., Nan, L., Han, R., ... & Radev, D. (2022). Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. arXiv preprint arXiv:2212.07981.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting framework that addresses an important task: estimating the quality of commonly used metrics.\n- Thorough description of measurement theory and how it is applied, enabling other researchers to use this framework for more cases.\n- Case study extensive in regards to metrics (a good selection of reference-free and reference-based metrics).", "reasons_to_reject": "- Uncertainty to what degree measurement theory (which has been developed to assess human evaluation) can also be applied to automatic metrics.\n- Case study somewhat limited in scope in regards to the task (only summarization).\n- Limited relevant work section.", "questions_to_authors": "Overall, I think the authors did thorough work in translating developments in measurement theory to be applicable to NLG. However, I did run into a few questions and uncertainties regarding the reporting and methodological choices. I will list these below.\n\n- In general (mostly §2): I think the argument for why measurement theory should be applied in this context is not sufficiently argued. For instance, it is not clear to me what exactly the benefits are of applying measurement theory in this context, and what makes the application of measurement theory in this context so unique.\n- In general: Most of the described measurement theory tests have been developed for human evaluation. To what degree do they also apply to metrics? For instance, stability seems irrelevant for most metrics, as the formula to calculate the score is fixed for these metrics.\n- In general (mostly §4): Why are the authors looking specifically at evaluating summarization metrics and the CNN/Daily Mail dataset? What makes this task/dataset relevant? And how generalizable are the results of this study for other tasks/datasets?\n- In general (mostly §4): While the authors go into detail about the results per measurement criteria, I am missing a higher-level conclusion from the case study's results: which metrics perform well, and which do not perform well? Why do they (not) perform well? What kind of information did we obtain that we would not have obtained if we would have just looked at correlations? I think such a higher level conclusion of their case study would greatly help other researchers to understand their framework and make it more attractive to use said framework.\n- 551-553: \"for less stable LLM-based metrics, the evaluator should consider applying the metrics multiple times\" -->  Is it really less stable as the lowest it scores are around 0.98?\n- 582-584: \"Inter-rater agreements on the same dimension were high (.48 − .89) in general but lower for Relevance (.48 − .58).\" --> Which inter-rater agreement metric was used here?\n- 610-618: \"Since Coherence and Relevance are distinct constructs by definition (Fabbri et al., 2021), the conflated validity structure indicates potential issues in the expert rating process. One source of such systematic measurement error may come from unclear instructions or the expert rater’s individual leniency. Such issues may cascade if new automatic evaluation metrics were trained or validated on this dataset.\" --> Is there a way to fix this, post-hoc? For instance, by merging these criteria into one single construct?", "paper_topic_and_main_contributions": "This paper describes the application of measurement theory, originating from psychology and educational sciences, to assess the quality of commonly used NLG metrics. These metrics help to more systematically analyze and evaluate NLG metrics, and compare them to a human evaluation baseline. These tests are also more extensive than just correlation, which has been used previously for this task. The authors focus on the summarization task (CNN/Daily Mail benchmark dataset) for their assessment of various metrics (reference-based as well as reference-free).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The related work section is quite succinct. It is missing a more thorough discussion on previous work that aimed to evaluate evaluation metrics. For instance:\n\n- Caglayan et al. (2020). Curious case of language generation evaluation metrics: A cautionary tale.\n- Hanna & Bojar (2021). A fine-grained analysis of BERTScore.\n- Sai et al. (2021). Perturbation CheckLists for evaluating NLG evaluation metrics.\n- Sai et al. (2022). A survey of evaluation metrics used for nlg systems.\n\nMore importantly, the paper ignores important previous literature where other researchers have also aimed to look beyond correlation scores:\n\n- Ni'mah et al. (2023). NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist.\n- Fomicheva & Specia (2019). Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments.", "Typos_Grammar_Style_and_Presentation_Improvements": "- 042: \"language quality(Nema and Khapra\" --> Missing a space before the opening parenthesis\n- 046-051: \"(e.g., ROUGE (Lin, 2004), BLEU (Papineni et al., 2002)) to embedding-based metrics (e.g., BERTScore (Zhang et al., 2019), MoverScore (Zhao et al., 2019)) and to end-to-end metrics (e.g., BLEURT (Sellam et al., 2020), G-Eval (Liu et al., 2023))\" --> This is a personal opinion to some degree, but I would try to avoid nested parentheses.\n- 102: \"systemically\" --> systematically\n- 131: \"other the other hand\" --> on the other hand\n- 184: \"Evalaution\" --> Evaluation\n- 519: \"on CNN/Daily Mail benchmark dataset\" --> on the CNN/Daily Mail benchmark dataset"}}, {"review_id": "Z8xMKqdE1f", "reviewer": "Reviewer_DnGL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging review! We have extended our related work section to incorporate your suggested paper. Here, we want to clarify a few questions you raised.\n\n**l.305ff: unclear formulation: how would assess that a metric \"deviates for a SINGLE datapoint\"? Lateron, you use different subsets, which makes more sense**\n\nThanks for raising this question. To clarify this, consider the following example: Suppose different models are compared on their average eval score across 100 news summarization tasks, and across models, the average score standard deviation is 0.1. Further suppose that for a specific model, the score on each specific news article deviates from the average by 0.4. In this case, the metric internal consistency is low — For a specific model, the average score on any subset of the 100 tasks is expected to be sensitive to the choice of included examples. A good proportion of difference across the two evaluated models’ average scores would also be attributed to noise, i.e., random fluctuations in scores across news articles. We will make it clear in the next version.  \n\n**l.320: the notation is confusing: you say that Y_j is \"the observed score (of each model) on the jth data point\", which is a set of scores. How can you sum over this set lateron?**\n\nThanks for the question. Y_j is the eval score on a single data point, e.g., a single news article. Therefore, for a test set consisting of 100 articles, j ranges from 1 to 100, and a model’s overall score on the metric is the average (or similarly, sum) across the 100 Y_js. We will clarify it in the revision.\n\n**l. 315ff: i do not see how the subsequent computation of alpha correlates to your idea to use the differences on different subsets of test data, as mentioned here**\n\nWe briefly explain the intuition behind alpha: By the property of variance of sums (i.e., Bienaymé's identity), the numerator of Equation (2) is larger when the pairwise covariance between scores on each single data point (e.g., news article) is larger. In other words, alpha is large when models’ scores on each pair of data points are highly correlated. More specifically, consider evaluation of summarization performance and 100 articles in the test set: If on any two news articles, different candidate models’ rankings on an eval metric are the same, then the pairwise single data point score correlation is expected to be high, and alpha will be high. On the other hand, if the ranking of different models’ eval scores differs remarkably across articles, the pairwise correlation of eval scores on single data points would be low, and thus alpha is low.\n\\\n\\\n\\\nWe also fixed the grammatical errors and format issues. Thanks again for your receptive review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ nice combination of existing methods (from measurement theory) to a very relevant current topic (evaluation of metrics for NLG)\n+ nice and intuitive showcase", "reasons_to_reject": "none", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper shows how to apply measurement theory to evaluate the quality of metrics for text generation. They showcase their approach for text summarization with 16 sota metrics on the SummEval dataset.  ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Sec 3.1: https://aclanthology.org/2022.findings-emnlp.108.pdf developed a theoretical framework for measuring the unreliability of a metric, as you mention them here"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l.24: term \"NLG\" is traditionally used for data2text tasks, maybe mention this in a footnote\n- l.42: whitespace missing after \"quality\"\n- l.71: High correlation with human judgements does not always imply a good metric, as has been shown in https://aclanthology.org/2022.acl-short.85.pdf.\n- l.130: other -> on\n- l.298: there is no T2 in the equation above\n- l.305ff: unclear formulation: how would asses that a metric \"deviates for a SNGLE datapoint\"? Lateron, you use different subsets, which makes more sense\n- l.320: the notation is confusing: you say that Y_j is \"the observed score (of each model) on the jth data point\", which is a set of scores. How can you sum over this set lateron? \n- Sec. 3: I think that using lower case letters (n, j) for numbers and upper case letter for scores (X, Y) would improve readibility a lot\n- l. 315ff: i do not see how the subsequent computation of alpha correlates to your idea to use the differences on different subsets of test data, as mentioned here\n- l.342: whitespace missing\n- l.433: \"introduce\" sounds like you invented these MTMM, but lateron you reference a paper from 1959. \n- l. 487: maybe use a ' in the brackets for better readibility: \\lambda_{jk}'s\n- l. 635: discriminative -> discriminate\n- Section References: use proper upper-casing, e.g. ACL in second reference, NLG in Howcroft et al etc.\n- add a reference to good introduction to measurement theory (maybe Allen et al, or something similar)\n"}}]

Author rebuttal:

Meta-review: This paper focuses on evaluation metrics in natural language generation, proposing a new framework for evaluating these metrics inspired by measurement theory. The authors have devised a new framework, influenced by measurement theory, to evaluate these metrics. The ideas are effectively communicated, and the paper is well-written. Additionally, the authors have presented a case study on text summarization to demonstrate the scoring of different metrics on various dimensions and their robustness and validity implications. The paper's only suggested improvements are to expand the related work section and add the missing references mentioned by the reviewers, which can be easily addressed with an additional page. All in all, I regard the paper as of high quality and a significant contribution to its field.
Decision: Accept-Main