Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
Authors: Aparna Elangovan
Lei Xu
Jongwoo Ko
Mahsa Elyasi
Ling Liu
Sravan Babu Bodapati
Dan Roth
Venue: ICLR 2025 Poster (2025)
Keywords: Automated evaluation
LLM as a judge
correlation measures
Primary area: generative models
TL;DR: Impact of noise when measuring effectiveness of automated evaluation

Abstract: The effectiveness of automatic evaluation of generative models is typically measured by comparing the labels  generated via automation  with  human labels  using correlation metrics. 
However, metrics like Krippendorff's $\alpha$ and Randolph's $\kappa$ were originally designed to measure the reliability of human labeling, thus make assumptions about typical human labeling behavior, and these assumptions may not be applicable to machine generated labels. 
In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human  labels and those from automatic evaluation, including LLM-as-a-Judge. 
Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human assigned labels  is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to the human-to-human (HH) correlation. 
This can create the illusion that labels from automatic evaluation  approximates the human majority label. 
However, as the proportion of samples with consistent human labels increases, the correlation between machine  and human  labels fall well below HH correlation. 
Based on these findings, we first propose  *stratifying data by human label uncertainty* to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric -*binned Jensen-Shannon Divergence for perception* for such scenarios to better measure the effectiveness of  automatic evaluations. Third, we present visualization techniques -- *perception charts*, to contextualize correlation measures appropriately and to show the strengths and limitations of automatic evaluation.  We have open-sourced our analysis and visualization tools at https://github.com/amazon-science/BeyondCorrelation.

Published as a conference paper at ICLR 2025
BEYOND CORRELATION: THE IMPACT OF HUMAN UN- CERTAINTY IN MEASURING THE EFFECTIVENESS OF AUTOMATIC EVALUATION AND LLM-AS-A-JUREE
Aparna Elangovan'
Lei Xu
Jongwoo Ko
Mahsa Elyasi*
Ling Liu
Sravan Bodapati'
Dan Roth
Amazon
KAAIST AI
'Equal contributions
ABSTRACT
The effectiveness of automatic evaluation of generative models is typically mea- sured by comparing the labels generated via automation with human labels using correlation metrics. However, metrics like Krippendorff's a and Randolph's 's x were originally designed to measure the reliability of human labeling, thus make assumptions about typical human labeling behavior, and these assumptions may not be applicable machine generated labels. In this paper, we show how rely- ing a single aggregate correlation score can obscure fundamental differences between human labels and those from automatic evaluation, including LLM-as- a-Judge. Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human assigned labels is relatively high, machine la- bels (generated by automatic evaluation methods) may superficially appear have similar or better correlation with the human majority label compared to the human-to-human (HH) correlation. This can create the illusion that labels from automatic evaluation approximates the human majority label. However, as the proportion samples with consistent human labels increases, the correla- tion between machine and human labels fall well below HH correlation. Based on these findings, we first propose stratifying data by human label uncertainty to provide more robust analysis of automatic evaluation performance. Sec- ond, recognizing that uncertainty and variation are inherent perception-base human evaluations, such those involving attitudes or preferences, intro- duce a new metric binned Jensen-Shannon Divergence for perception for such scenarios better measure the effectiveness of automatic evaluations. Third, we present visualization techniques perception charts, to contextualize corre- lation measures appropriately and show the strengths and limitations of auto- matic evaluation. We have open-sourced our analysis and visualization tools at https://githur com/ ama zon- science Beaoeee//ooooooeelllateee
INTRODUCTION
Despite the increasing importance of automatic evaluation for generative large language mod- els (LLMs), measuring and interpreting the effectiveness of these methods remains a challenge. Automatic evaluation methods include N-gram-based methods such Rouge (Lin, 2004) and BERTScore (Zhang* et al., 2020), task-specific model-based methods such as AlignScore (Zha al.. 2023) and Prometheus (Kim al., 2024), and AaMMas-a-Judde-bice a general purpose LLM, using prompts, is instructed to perform evaluation. The use of automatic evaluation is tied to the key question How can the efficacy of automatic evaluation methods be measured in comparison to human evaluation? A widely accepted practice is to use agreement or rank correlation metrics such as Krippendorff"ss-e (Krippendorff, 201 1), Cohen's S-K (Cohen, 1960; McHugh, 2012), Spearman's 's and Kendall's 7-b (Fabbri et al., 2021; Deutsch et al., 2022: Chi- ang & Lee, 2023; Liu et al., 2023b), where higher correlation between machine labels (generated by automatic evaluation methods) and human labels (gathered during human evaluation) implies more effective automatic evaluation. Variation or uncertainty in human labels (Tversky & Kahne- man, 1974) is almost impossible avoid when humans involved (Kahneman al., 2021). In this
methods (LJ) where

---

Published as a conference paper at ICLR 2025
paper, we highlight how comparing the overall correlation score between human labels and machine labels can obscure key discrepancies in machine labels. Therefore, we propose that measurements be stratified by label uncertainty.
The impact of human label uncertainty on automatic evaluation, therefore, leads to a follow-on question how then to effectively account for human uncertainty in automatic evaluation. Not all types of uncertainties are equal (Kahneman & Tversky, 1982), as they can range from random human errors, ambiguity in tasks, and a by-product of perception-based evaluation (Elangovan al., 2024). Some human evaluation tasks naturally have higher variance than others. For instance, when evaluating aspects such as "How readable the given content?", the answer relies on human perception and is likely to vary from person to person. In contrast, evaluating a natural language inference (NI.)) task, such as "Can you infer the given sentence from a given text?", is less likely to show variations. Acceptable variation in labels, therefore, depends on the task, as variations or uncertainties human labels can sometimes be a feature rather than bug in human evaluation. Thus, explore how to effectively measure and interpret reliability of automatic evaluation while taking human uncertainty in to consideration. We ask the following research questions:
RQI: How does uncertainty in human labels impact correlation metrics when we measure the efficacy of automatic evaluation methods? (See detailed analysis on stratifying data by human uncertainty in Sec. 3.1)
RQ2: How can we measure human machine agreement that accounts for human uncertainty as a result of variation in human perception? (See Binned lensen-Shanon divergence in Sec. 3.1)
visualize the underlying data to draw meaningful insights when compare the results from automatic and human evaluations? (See Perception charts in Sec. 3.3)
RQ3: How
BACKGROUND ON CORRELATION MEASURES
2.1 CORRELATION MEASUREMENTS AND ASSUMPTIONS
Correlation measures have traditionally been used to measure the inter-rater agreement (IRA) of human human-generated labels. where there is no ground truth available, measure tomaanto-huumn (HH) correlation. There are 2 broad categories of correlation measures, (a) non-chance-adjusted indices and (b) chance-adjusted metrics (Xinshu Zhao & Deng, 2013). Chance-adjusted measures take into account scenario where raters can choose the same label by chance even when they do not know the correct answer. Chance-adjusted measures, such as Krippendorff"s are generally more popular IRA measures than non-chance-and aed-chann--adusteed metrics such percentage agreement. The key difference between various chance- adjusted IRA measures is how chance agreement is computed (Meyer et al., 2014; Xinshu Zhao & Deng, 2013). For instance, Randolph's s-x (Randolph, 2005) assumes that all labels are equally likely and assigns a probability of 1/4 for chance agreement, where k is the number of possible label choices. On the other hand, Fleiss's y. (Fleiss, 1971) and Krippendorff's 's a calculate chance agreement empirically, based on the actual human labels of the study, These measurements can be useful cases where certain labels (c.g.. on a Likert scale of 1-5) are more likely to be chosen when humans are uncertain. A high proportion of such labels, even if consistent, may indicate unreliability in the labeling process, particularly when the minority labels show lower agreement. Therefore, chance-adjusted correlation measures rely on key assumptions about the types of errors humans are likely to make. Furthermore, Krippendorff s-aa specifies that the agreement coefficient is an indicator of reliability if and only if (1) the raters duplicate the process of rating, i.c., when two independent raters use the same guidelines or instructions and use the same units, (2) raters must be treated as interchangeable, and (3) correct answers are not known (Krippendorff, 2004). Human labels and machine labels are clearly not interchangeable even with current -f-the-attttee LLMs to generate machine labels, humans provide the ground truth labels.
2.2 DIFFERENCES BETWEEN CORRELATION MEASURES AND RAW ACCURACY
The assumptions above, especially the assumption that raters must be treated as interchangeable, raise fundamental question as to why even use correlation measures to compare machine and human performance, instead of measures such as accuracy and Fl using human labels as ground truth? Firstly, one of the main challenges with using scores like accuracy or Fl is that human labels tend to be noisy regardless of the task or the dataset, where some label choices more acceptable than others. and measures such as Fl or accuracy do not account for variations in human labels and hence correlation measures appear be better measure. However, mentioned previously,
2

---

Published as a conference paper at ICLR 2025
many of the common chance-adjusted correlation measures aim to account for random chance agreement in HH agreement, whereas the errors machines make tend to be more systematic. A key characteristic of random errors is that they non-reproducible. When repeating the labeling process, the result varies, resulting in low IRA, which in-turn may point to potential problems in the labeling process or complexities in the task. In contrast, systematic errors are consistent, and IRA measures are not designed to detect them. Secondly, as a direct consequence, IRA metrics assume that agreement is good and variance bad, which implies that there single correct answer situation that may not hold for ambiguous items or those that gauge user perception.
2.3 TYPES OF RAW DATA COLLECTED DURING EVALUATION
The type of the data collected, such as nominal, ordinal, interval, ratio, discrete or continuous data, dictate how such data can be aggregated and interpreted. The most common types of values col- lected during automatic or human evaluation are:
Nominal values: Nominal data, such as gender, have no natural ranking or ordering. Such data primarily used in human evaluation tasks such as fact checking, where the aim is to verify whether given text can be inferred from reference content (Liu et al., 2023c) and usually is modeled binary classification task. Preference judgments such as "Is the output of model-A or Model-B bet- ter?" are also common examples of nominal values collected during human evaluation. IRA mea- sures for nominal values include percentage agreement (McHugh, 2012), Randolph's "ssk, Krippendorff s-a: and Cohen's s-r..
's-, Fleiss's S-K,
Ordinal values: Ordinal values have an implicit order, e.g., ratings of quality (very good, good, fair, poor, very poor) (Mishra et al., 2018). The most common use of ordinal values human evaluation are through Likert scales. Likert scales are rating systems of attitude scale to measure the degree to which person agrees/disagrees with a given statement (Taherdoost, 2019) and commonly used to gauge user perception qualitative aspects such as readability, coherence, and fluency (van der Lee et al., 2021). IRA measures for ordinal values should ideally account for the fact that ratings that are closer one another (1 vs. 2) for given item indicate a higher agreement than ratings that are further apart (1 vs. 4). Such IRA measures include Krippendorff's s-a for ordinal values, where disagreements are weighted by distance between the ratings. Rank correlations such as Kendall's- s-r and Spearman's s-p are also widely used to measure item-level correlation between machines and hu- mans for ordinal values. However, there's a caveat on using these measures: their ability handle a relatively large proportion of ties is widely debated, even when measuring system correlations that involve a much smaller proportion of ties (Deutsch et al., 2023). When the number of items to rank is n, the number of ordinal values to assign is r, and when r << it results in many ties or rank collision. For instance, when assigning a Likert scale of 1-5 to 100 items, at least 20% of the items will have the same rank. While ordinal values preserve order, the objective of using ordinal values not to rank the item in relation to other items, therefore the proportion of ties affects the interpretability and reliability of the rank correlation scores.
s-a
Continuous values: Continuous values are typically obtained when using automatic evaluation methods such as Rouge (Lin, 2004) and BERTScore (Zhang* al., 2020).
3 ANALYSIS SETTINGS AND RESULTS
Datasets and human labels: We analyze typical evaluation tasks for LL text, including (a) Likert- based qualitative evaluation (ordinal), (b) NLI-style fact or consistency checking (nomi- nal), and (c) human preferences of several LLMs (nominal). We select datasets with multiple human annotations: 1. Summ (Fabbri et al., 2021) is human evaluation of summary quality using four criteria- fluency, coherence, consistency, and relevance, with Likert scores assigned from to 5. This dataset has 1600 samples, where each item is labeled times by expert annotators. 2. SNLI (Bowman et al., 2015) is an NLI inference task dataset with 10K samples and 3 labels entailment, neutral and contradiction. Each item is labeled 5 times by human annotators. 3. MNLI (Williams et al., 2018) is also an NLI inference task dataset with 10K samples and labels - entailment, neutral and contradiction. Each item is assigned 5 labels by human annotators. It contains two splits - matched and mismatched. 4. MT-Bench (Zheng et al., 2023) is a preference dataset which compares human preferences of several model outputs on open-ended questions using multi-turn conversations. From this dataset, we only use samples that have at least 3 human ratings. 5. We also include JudgeBench (Bavaresco al., 2024) in Appendix 1.1
Machine labels: For SummEval, we reuse the original G-Eval results (Liu et al., 2023b) which rates the quality of summaries scale of 1-5, In addition, we use Claude Sonnet 3 and Mistral
3

---

Published as a conference paper at ICLR 2025
Large v2 on SummEval, where prompts are detailed in Appendix A.2. For SNLI and MNLI, Llama 3.1 8B, Mistral Large v2 and Sonnet 3, reusing the prompts from Liu et al. (2023a) as detailed rely the existing GPT-4 labels in the dataset (Zheng al.,
in Appendix A.1. For MT--Bench, 2023). We also adapt the same prompts (see Appendix A.3) for Sonnet 3 as I.J. In order to compute MM correlation, we set the sampling temperature to 1.0 to maximize the variation across runs, and set top-p= 0.8. We sample 20 machine labels per example. NOTE: We have predominately reused the prompts from existing works of using GPT-4 as LJ, we have NOT attempted to optimize the prompts per LLM except for minor format changes. Since models are known to be highly sensitive to prompts, our experiments are not meant to rank the underlying LJ, but to study the impact of human uncertainty regardless of the LJ.
Correlation measures: For nominal data, we report scores on Krippendorff's-4 s-a, Fleiss-K, Randolph's s-x and percentage agreement (McHugh, 2012) also detailed in Appendix A.4. For MT- Bench preference data, since the number human raters vary per tuple (conversation, turn, model pair) we use Krippendorff-a and percentage agreement to compute HH correlation as it can han- dle varying number of raters per item unlike Fleiss-K or Randolph's s-k. For ordinal data, we report Krippendorff-a for ordinal data. We also report commonly used rank correlation measures such as Spearman's s-p and send as many papers have used them to measure for item-level correlation (despite the caveat highlighted previously).
3.1 RQI: How DOES UNCERTAINTY IN HUMAN LABELS IMPACT CORRELATION METRICS WHEN WE MEASURE THE EFFICACY OF AUTOMATIC EVALUATION METHODS?
To study the impact of human uncertainty, stratify samples and then compare HH and Human- to-machine (HM) correlation measurements for each group. We primarily stratify by percentage agreement, which is the proportion of labels that matches the majority (for nominal data) or the median label (for ordinal data) among all human labels assigned to an item. For example, if 3 out of 5 people assign the same label, the percentage agreement is 60%. We also stratify samples by the number of unique human labels to ensure that our findings are consistent regardless of the stratification method.
In Table 1, column H compares Human majority (H")) with machine majority (M"). Super- ficially, HM correlation seems to improve with increasing human certainty. On closer inspection, another trend emerges where in any stratified group, when uncertain samples increases (as measured by low HH correlation), the HM correlation seems to outperform HH correlation as highlighted. However, as the proportion of samples with noisy labels decreases and HH correlation approaches near-perfect agreement, HH correlation is much higher than HM as highlighted. Thus, the illusion that machines (specifically LJs) approximate majority human behavior is largely due to the presence of noisy labels in the data. If LJs were truly approximating the human majority, the A would approach 0 under perfect HH correlation which is clearly not the case. We also replicated this behavior on SummEval dataset which relies on ordinal labels in Table 2, using correlation met- rics such as Krippendorff's s-a for ordinal data, Kendall's-- s-7 and Spearman's ss-p. The same patterns are observed on preference data in Table 3. We further randomly sample subsets from NLI datasets (sampling detailed in Appendix A.5) and compare correlation scores across subsets with varying levels of human label uncertainty. We repeat this for three different LLMs, and consistently observe the same findings, performance gaps become more apparent as human label consistency increases, as shown in Fig. 1. Similar findings for the SummEval dataset are shown in Appendix Fig. 5.
We explore the phenomenon using a random labeler on synthetically labeled datasets (code provided in Appendix 1). This dataset simulates both binary classification and ordinal classification with labels {1. 2. 3}. We first sample 2 synthetic human labels to each item. Then another random labeler assigns labels randomly. We examine the correlation between the random labels and the synthetic human labels across varying levels of certainty. Notably, the random labeler may appear to have higher correlation when the proportion of uncertain samples is higher. However, the proportion of consistently labeled samples increases, the random labels' poor performance becomes evident, as shown in Fig. 2. Intuitively, if 2 human labels disagree, a 3rd random label cannot perform worse in terms of agreement they can either disagree with both human labels (in which case it is no better than the 2 humans) or agree with one of the human label. This example further illustrates why stratification by proportion of uncertainty is crucial to show weaknesses in any automatic labeler.
4

---

Published as a conference paper at ICLR 2025
Partition by human labels (% samples)
Krippendorff
Agreement
Fleiss-v
Randoph
Dataset
HH
MM
H" H"M"
A
HH
H"M"
A
HH
H" M"
A
HH
H"M"
A
All (100%)
0.70
0.96
0.72
-0.02
0.88
0.82
0.07
0.70
0.72
-0.02
0.70
0.72
-0.03
PA = (58%)
100 0.97
0.83
0.17
1.00
0.89
0.1
1000
0.83
0.17
1.00
0.83
0.17
MNLI (matched)
0.8 PA < 1 (25%)
0.39
0.95
0.66
-0.27
0.80
0.78
0.02
0.39
0.66
-0.27
0.40
0.67
-0.27
0 PA 0.8 (16%)
0.04
0.93
0.37
-0.33
0.60
0.61
-0.01
0.04
0.37
-0.33
0.07
0.42
-0.34
unique = (38%)
0.28 0.95
0.56
0.28
0.73
0.72
0.01
0.28
0.56
-0.28
0.30
0.58
-0.28
unique = 3 (3%)
H.00
0.91
0.39
0.44
0.60
0.63
0.003
0.00
0.39
+0:45
4..05
0.44
0.49
(100%))
0.72 0.97
0.74
-0.02
0.89
0.83
0.07
0.72
0.74
-0.02
0.72
0.74
-0.02
PA (60%)
1000
0.98
0.85
0.90
1000
0.85
0.85
0.15
MNLI (mismutched)
0.8 PA < 1 (24%)
0.39
0.95
0.65
0.26
0.80
0.77
0.03
0.39
0.65
-0.26
0.40
0.66
-0.26
0. PA < 0.8 (14%)
0.04
0.94
0.39
-0.55
0.60
0.62
-0.02
0.04
0.39
00.35
0.07
0.43
-0.36
unique (36%)
0.28
0.94
0.56
-0.28
0.73
0.72
0.01
0.28
0.56
-0.28
0.30
0.58
-0.28
unique (2%)
0.006
0.94
0.41
0046
0.60
0.64
0.004
0.06
0.41
00:46
40:05
0.45
0..50
All (1009)
0.68
0.97
0.63
0.05
0.88
0.76
0.12
0.68
0.63
0.05
0.68
0.64
0.05
PA = 1 (55%)
1000
0.97
0.69
0.3
1.00
0.80
D120
1.00
0.69
0.5
1.00
0.69
0.3
SNLI
0.8 PA < 1 (28%)
0.39
0.97
0.62
-0.24
0.80
0.76
0.04
0.39
0.62
-0.23
0.40
0.64
-0.24
U PA 0.8 (15%)
0.03
0.95
0.37
-0.44
0.60
0.61
-0.01
0.03
0.37
-0.34
0.07
0.42
-0.35
unique 2 (415)
0.29
0.96
0.54
-0.25
0.74
0.71
0.03
0.29
0.54
-0.25
0.31
0.57
-0.26
unique 3 (3%)
0.005
0.94
0..7
10.53
0.60
0.66
0.01
0.05
0.47
10.53
40.05
0.50
00.55
Table I: Compare HM and HH correlation measurements on NLI datasets using Sonnet to generate machine labels. The overall correlation scores (All) indicate that the HM correlation is similar to HH. However, when stratifying by percentage agreement (PA) or human unique labels, the differ- ence (A= = HH M"" is highest when the all human labels are the same (PA=I), and H outperforms HH when certainty is the lowest. H and M are human and machine majority label.
Sonnet
Mistral
Llama
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
K-or HH
K-u
0.2
0.2
K- HPH
K-u
0.0
0.0
K-u HTR
0 20
40
60
80
100
0 20
40
60
00
100
20
40
60
80
100
%/6 SSmmpless with perfect agreement
%: Samples with perfect agreementt
% &.ammlles with perfect agreement
Figure 1: Krippendorff's 's a (K-a) on subsets of MNLL-mismatchen with varying uncertainty, using 3 LLMS Sonnet, Mistral, and Llama. As proportion of samples with consistent human labels increases (towards the right of X-axis), HH correlations are significantly higher than H"M".. H"' represents the correlation between the human majority and another human annotator (H'). H"RR" compares the human majority with a random machine evaluator, serving as baseline performance.
3.2
RQ2: How CAN WE MEASURE HUMAN-MACHINE AGREEMENT THAT ACCOUNTS FOR HUMAN UNCERTAINTY AS RESULT OF VARIATION IN HUMAN PERCEPTION?
Evaluating the quality of free-form model generated output, such as a summary, largely depends on human perception. As a result, the human ratings can vary depending on a wide range of factors including what one likes to their emotional state (Elangovan et al., 2024). Perception-based ordinal- value evaluation schemes, such as Likert scores, pose 2 main challenges. (1) It is an attitude-based scale, and since human attitudes vary, metrics like Krippendorff's 's a, which penalize variation, can produce scores lower than the minimum acceptable value (Amidei et al., 2019). (2) Rank correlation measures such as Kendall-7 or Spearman's 's-p are not entirely appropriate given the large number of ties when comparing median scores, as previously discussed in Sec. 2.3 and in the works by Deutsch et al. (2023). Furthermore, whether to treat Likert-data as ordinal or interval data dictates the aggre- gation method - median mean, is also debated (Joshi et al., 2015). The argument against using Likert-data as interval data is that the points on the scale may not be equidistant, e.g., the distance between pair (neutral, agree) may not be the same as the distance between (agree, strongly agree). We report Spearman's s-p for both to illustrate the difference in Table 2. While in most cases the two sets of scores look largely the same, there are instances where the numbers vastly different.
5

---

Published as a conference paper at ICLR 2025
% Agreement
Randolph's--
Krippendorf''s
Kendall-r T
Spearman-p
Krippendorf'ss s
HH HrR
1,0
0.5
0-
-1
50
100
100
50
16 S.Smmplle with perfect agreement (nominal data!
Samgles with perfect agreemeent (Ordinal data)
Figure 2: Simulating the impact of uncertainty by comparing with an automatic random labeler (R). When the proportion of samples with uncertainty is higher, even a random labeler can appear to have better correlation with a majority (H") or median (H) human label. As the proportion of samples with consistency increases, the weakness of the random labeler become evident.
Kougel
Model
Krippendortf's a
J.S.
Spearman's Sop
Kendail's
Partition samples)
MM
HM
H
HM
HM
H
H
HM
H'M
G-Eval
0.55
0.74
0.31
0.25
0.27
0.39
0.53
0.43
0.57
0.47
0.38
1. (100%))
0.07
0.48
Sonnet
0.55
0.83
0.21
0.32
0.21
0.51
0.34
0.31
0.41
0.30
0.28
0.75
0.61
0.55
2. PA (12%))
0,36 0.28
Mistral
1,00
0.79
0.35
0.65
0.35
0.69
0.62
0.62
0.68
0.56
0.56
Sonnet
1,00
0.86
0.21
0.76
0.24
0.71
0.43
0.43
0.48
0.39
0.39
G-Evval
0.39
0.70
0.32
11.1
0.16
0.35
0.56
0.31
0.58
0.52
0.27
3. PA < 60% (2196)
0.17 0.13
Mistral
0.19
0.78
0.55
0.337
0.36
0.30
0.57
0.39
0.54
0.52
0.34
0.02
0.28
G-Eval
0.40
0.74
0.25
0.15
0.22
0.37
0.53
0.42
0.59
0.47
0.37
1. (1000)))))
Sonnet
0:40
0.79
0.35
0.05
0.29
0.38
0.39
0.34
0.48
0.36
0.31
Rel
2. PA (144%)
0.38
0.30
Mistral
1,00
0.80
0.43
0.57
0.43
0.59
0.71
0.71
0.72
0.65
0.65
Sonnet
1,00
0.77
0.36
0.60
0.26
0.51
0.19
0.49
0.80
0.45
0.45
G-Evval
-0.09
0.69
0.18
10.27
0.09
0.35
0.40
0.17
0.42
0.38
0.15
3. PA < 60% (19%)
0.28
0.22
Mistral
-0:39
0.74
0.32
10.41
0.16
0.28
0.42
0.19
0.46
8.90
0.17
-0.09
0.43
Table 2: Results on SummEval dataset for criteria
Coherence and Relevance, H is the human
median rating and M is machine median. A = (HH - -HM). Impact of human uncertainty: When the human label uncertainty is high, HM can seem better than HH correlation and vice versa under low uncertainty. Does the correlation score (p of 0.56 a 0.49) imply the LJ is good enough to re- place human evaluation? Spearman s-p median (HM) vs. mean MH"' Median vs. mean results in an 11 points boost 0.49 + 0.60. Differences in correlation metrics: Krippendorff's s-ax of 0.25 (reject results), while Spearman s-p shows 0.53 (moderate correlation). I ranking and absolute correlation numbers change depending on reference human under high uncertainty, Mistral best performance 0.42 replaced by Sonnet 0.22 when reference human is H' (another human annotator). Full table in Appendix Table 7.
For instance, Table Spearman'ss s-p median mean, median comparison yields correlation of 0.49 (low correlation), while comparing the mean results in 0.60 (indicating moderate correlation) (Mukaka, 2012), provoking the question which number is more representative of the underlying data? Regardless of whether we choose to use Krippendorff-a or a rank correlation metric, one as- pect that both assume is that there is a single rating (e.g., median value) that appropriately represents the underlying choice human rater makes. This assumption may not be correct, especially when the task is inherently subjective, especially when the number of judgments collected per item rela- tively low, e.g., 3 ratings per item. This shows a clear gap in existing metrics to compare human and machine performance, where intra-human ratings naturally vary as a a result of subjectivity inherent to the task.
Hence, we propose an additional metric to supplement existing measures, the use of binned Jensen- Shannon divergence (JSD) for perception (Lin, 1991) to quantify how closely automatic methods mimic human judgment. To the best of our knowledge, our approach of using JSD has not been used correlation measure to account for human perception, without relying The closest work to ours is the use of cumulative JSD to predict the aesthetic score distribution, instead of just the average rating, to represent image quality (Jin et al., 2018).
single gold label.

---

Published as a conference paper at ICLR 2025
Kri Krigesndaaff ei
Agmamment
JS.
Fiuiawe
Randolpbos
[J Winnte
MP IA BY
Portition samples)
Human
HBI
MM
1f'M"
HII
11' M"
LE'M
HM
11"M"
H'M"
HI"M"
HMM
w
WR
(52)
0.79
0.34
0.29
0.81
0.75
0.11
0.31
0.25
0.62
0.81
0.02
GPT
2. PA 1 (26)
B 0.92
1000
-0.05
1808
-0.05
1,00
0.85
0.85
0.13
-2.17
-0.07
0.77
0.77
0.92
DUB0
(AIg, Gjd)
3. (60.80)) (18)
0.61
40.04
0.28
0.68
0.67
0.61
0.22
0.54
0.36
0.42
0.67
0.06
1. All (52)
0.79
0.34
0.90
0.01
0.23
0.11
0.81
0.75
0.73
0.18
0.00
0.10
0.60
0.60
0.00
0.12
Sonnet
(36)
3. PA: (60.80) (18)
0.61
+0.04
0.99
-0.04
0.00
0.18
0.68
0.50
0.61
0.17
-0.07
0.16
0.25
0.42
0.75
0.17
0.30
0:40
0.34
0.79
0.69
0.56
0.11
0.54
0.33
0.35
0.99
0.00
GPT
2. B% 1 (14)
0.64
1000
0.25
0.75
0.25
1,00
0.57
0.57
0.10
0.22
0.22
0.36
0.36
0.57
0.07
(CMd Gp3)
3. PAL: (60,800
0.52
0.35
0.67
0.75
0.57
0.34
0.64
0.36
0.52
0.00
All (36)
0.39
0:40
0.78
0.36
0.05
0.18
0.79
0.58
0.47
0.17
0.35
0.17
0.38
0.21
0.56
0.17
Sourch
2. (14)
0.64
1000
0.56
-0.07
-0.07
1,00
0.43
0.43
0.24
-0.11
-0.11
0.14
0.61
0.00
3. PA: (40,80) (23)
0.52
40,02
0.86
0.37
40.30
0.00
0.67
0.67
0.45
0.19
0.35
0.07
0.21
0.67
0.14
1. (38)
B 0.53
0.26
0.20
0.01
0.22
0.74
[66
0.61
0.38
0.28
0.21
0.3
0.41
0.74
9:21
(31)
0.52
1000
0.64
1.36
0.64
1,00
0.91
0.16
0.63
0.63
0.56
0.91
(Gp3, Gp4)
3. PAL: (00.80) (20)
0.50
8.08
0.24
00117
0.12
0.68
0.55
0.50
0.21
0.22
8000
022
0.66
0.15
0.53
0.36
0.76
0.74
0.50
0.14
0.25
0.45
Soanch
2. PA 1 (11)
0.62
100
0.66
19
88
0.19
1.00
0.64
0.64
0.26
0.15
0.15
0.45
0.45
0.64
0.18
(60,80)
0.50
0.68
0.50
0.45
Table 3: Stratified performance on preference data with at least 3 human ratings from MT-Bench for model pairs (MP). Impact of human label uncertainty; When the HH label uncertainty is high, M can seem better than HH correlation and vice versa. IRA vs. Win-rate; While the difference in win-rate between model and human (A) can be smaller in evaluator model (e.g.. Somnet 0.08) compared to another (e.g., GPT 0.21), the H" IRA can tell an opposite story, as in the example above, GPT has better correlation with human majority across all correlation metrics except J.₁.- LJ ranking and absolute correlation numbers change depending on reference human H', mimicking single annotator, under high uncertainty compared to using consensus human majority (H")).
Formally (source code in Appendix A.7), let H; and M; represent the set of human and machine ratings respectively assigned to the item Ti, and n be the total number of the items and R be the set of possible ratings (e.g., to 5 for ordinal ratings, or {A, B, Tie} for nominal model preferences). To compute the distance between human and machine predictions, we can define the probability mass function (PMF) for a given rating as follows:
(c) Σien, Σien, p0=)) ΣiEH, [H,1
(c) = Σien, ΣiEM, 1(t=))) (Machine)
(Human))
(I)
Exen, J.I
where for ordinal values, bin B, = E (m...n Imedian HH,) = r contains the set of samples
with a median human rating r. For nominal perception- preference labels, bin B, == (i e {I, [mostfrequent(H)) = r contains the set of samples with the most frequent human rating r.
From Equation 1, we can define the binned JSD (JSa) follows:
J&h
Σ
|nd JS(P")PP)))
Σ |
(1 (2x((Ph11))) + : -
(2)
ER
reR n
(c) loge (c)) ) is the Kullback-Leible divergence (Kullback & (c)
where
=
Leibler, 1951) and Q+(c)= = (c) + Pm((c) /2 for Ve R is the average between human and machine PMFs, based on the definition of Jensen-Shannon distance (JS). Since JSD is a distance- based measure, lower scores are better because they indicate that the human and machine judgments are similar. We also illustrate the benefits of binned-JSD using a toy example in Appendix A.8.
The binned JSD for ordinal perception data has 2 main advantages, (1) measures how closely machine labels mimic human perception without the need for a single gold label (2) human and machine labels are not treated interchangeably. as the items in a given bin are selected by the human median or majority value. For HM correlation, the proposed approach can be applied at item-level when the number of labels collected per item is high enough to form distribution, however, for judging model generated output even obtaining 3 human labels per item can be expensive. Hence, we suggest computing the distribution distance per bin. The main shortcoming of this approach is that some key item level discrepancies between machines and humans ratings might not be clearly surfaced. For instance, for a given item, if all the humans assign a rating of 1 and machines assign say 4, at an aggregate level this pattern may not be obvious if this pattern is not frequent. Hence, also recommend reporting correlations stratified by uncertainty in human labels, e.g., for items that have high agreement labels, such as perfect human agreement as shown in Table 2.
7

---

Published as a conference paper at ICLR 2025
29.6
24.2
24,4
H=1, JS:0.61
H=2, JS:0.35
H=3, JS:0.18
H=4, JS:0.31
H=5, JS:0.6
Probability
0.8
Figure 3: Ordinal HM perception comparison chart: Visualization of human perception vs. ma- chine labels binned by human median rating (LJ Mistral on Coherence): Dial the top shows the percentage of samples that fall into the bin. Middle scores (median 2-4) have higher human uncer- tainty, where only 60% (less than out 3) the human labels follow the median. The extremes scores (1, 5) have less uncertainty. While Mistral seems to mimic human perception when the hu- man median is 3, it is also biased towards the rating when the human ratings are between 2-4.
3.3 RQ3: How CAN VISUALIZE THE UNDERLYING DATA TO DRAW MEANINGFUL
INSIGITTS WHEN WE COMPARE AUTOMATIC LABELS AND HUMAN LABELS?
3.3.1 PERCEPTION BASED ORDINAL RATING VISUALIZZTIONN
We propose the "Ordinal HM perception comparison chart" as shown in Fig. 3 to compare hu- mans' perception of the content vs. how machines rate them. Here, we bin the samples by the human median rating and plot the corresponding distribution of human and machine ratings. For instance, say for a grading task using label values between 1-5, each item is rated 3 times by humans. For sample set of 10 items, the total number of judgments would be 30 (10 items 3 ratings per item). Let's say items get a median rating of 4, these 7 items would fall in bin human label distribution for that bin to include all the 21 judgments (7 items 3 ratings per item 21), we also plot all the machine labels for the corresponding bin. We repeat this for each of the label from 1-5 as shown in Fig. 3. This allows us to visualize the variation in human perception, thus including all the ratings rather than a single gold rating, and compare how the corresponding machine labels vary.
4. We then plot the
Correlation numbers are challenging to interpret, and do not provide sufficient insights to the mean- ing behind the numbers unless supported by visualization. When does correlation score imply the model is good enough to replace human evaluation as indicated in Table 2? And what are the gaps in the LJ? In Table 2, LJ Mistral achieves a Krippendorff"s-o HM of 0.49, Visualization in Fig. shows that the LJ does not rate any item as and negligible amount of items are rated 5, while humans have assigned over 24% of the items a median rating of This shows the key difference between LJ Mistral's 's judgments and humans. This type of insight can potentially be useful in opti- mizing the prompts used by the to minimize the gap with humans judgments, demonstrating the importance of effective visualization techniques.
Another key aspect to note is how human uncertainty varies across different median labels. Human labels tend to be more certain when they assign extreme ratings (1, 5) compared to the middle ratings (2-4), demonstrated by the distribution difference between the median H and all ratings, as shown in Fig. 3. Relatively higher consistency in extreme ratings is common pattern observed when humans are asked to rate using star-rating schemes and Likert scales, an observation typically reported in recommendation systems (Amatriain et al., 2009). This type of differences in the extent of human uncertainty depending on the rating further demonstrates the deficiencies assuming a single gold rating for tasks that rely human perception, clearly illustrating the need for HM correlation metric that does not assume a single gold label, such as the JS₁ we proposed in Sec. 3.2 in RQ2.
3.3.2 PERCEPTION-BACE PREFERENCE VISUALIZZATION
For visualizing pairwise model preferences to compare human and machine judgments, propose the "Pairwise preference HM perception chart". The concept behind this visualization similar to the Ordinal HM perception comparison chart, the main difference is that is bins are separated by
8

---

Published as a conference paper at ICLR 2025
the most frequent human label for given item. This visualization also helps understand, for given model pair, how reliable the human majority is. For instance, in Fig. 4, in the case of (claude-v1, gpt- 3.5-turbo), even though the preferred model is Claude ( 38.9% preference), when human majority prefers GPT-335 there is almost no contention. We also visualize the preference of IJ Claude Sonnet shown in Appendix Fig. 7.
38.9 9 H
27.8 H H
33.3
8 HW=A
78.8 Hw=B
15.4 HW
H==A
H==T
1.0
1.0
H
0.8
0.8
Probabiltt
Probability
0.6
0.4
0.4
02
0.
T
Figure 4: Pairwise preference HM perception chart: Human preference distribution vs. LJ GPT-4 for a given majority label per model pair. T is tie. Compares model pair (Left) Claude-vi (A) vs. GPT3.5-turbo (B) (Right) Alpaca- 13B (A) vs. GPT-3.5-turbo (B). Note: Using GPT-4 IJ has only
one judgment per item in the results from Zheng et al. (2023), hence M and M are the same.
DISCUSSION AND CONCLUSION
4.1 HUMANS ARE UNCERTAIN AND THEIR LABELS ARE NOISY, SO MACHINES ARE BETTER?!
Evaluation procedures have predominantly circumvented dealing with human uncertainty, except for recent limited works (Chen et al., 2020; Wang et al., 2023; Chen et al., 2024), and have thus relied on the simplified assumption that single gold label is sufficient. A single gold label is only sufficient when there is little or no ambiguity, while being able to quantify human uncertainty is crucial in measuring the effectiveness of automatic evaluation. The ubiquitous nature of uncertainty is studied in human psychology (Kahneman et al., 2021) and is also illustrated in a widely used MIL. dataset, which was carefully curated with 5 human labels per item. Over 40% of the samples have some degree of uncertainty, as shown in Table 1. In addition, given the challenges in human evaluation, including obtaining consistent labels (Elangovan et al., 2024), recent datasets have started rely on collecting one human label per item (Bai et al., 2022; Ganguli et al., 2022; Stiennon et al., 2020). This further demonstrates the urgent need to acknowledge that quantifying human uncertainty is essential to measure the effectiveness of automated evaluation.
The machine learning community cannot afford to dismiss uncertainty as simply "poor quality la- bels from humans", key systematic errors and performance gaps in models become apparent under high human certainty as shown in Sec. 3.1. As a corollary, under high uncertainty even a random labeler can appear to have similar better correlation with humans, shown in Fig. 2. More im- portantly, for safety critical applications such as medicine, when humans labels are highly robust and certain, but the corresponding machine assigned labels differ, it can point serious deficiencies in the automated system. Furthermore, the implications of uncertainty include changes to ranking of the automatic evaluator as well as the corresponding absolute correlation scores as shown in Table and Table 3 depending labels from humans is expensive, and that in some cases gathering just one label per item might be adequate, such as when attempting to understand if one LLM is better than the other, provided the humans end up overwhelmingly preferring the output of one LLM over the other. In the case where the performance of two LLMs are close to one another, uncertainty in human judgments may be a signal that says the model outputs are quite similar. Thus, in some cases uncertainty can be a potential indicator of lower quality of labels requiring improvements to the underlying label collec- tion process (Flangovan et al., 2024), in other cases uncertainty might be inevitable and therefore a valuable signal. Under uncertainty, comparison of automatic evaluation to human evaluation challenging. and traditional correlation measures also fail to adequately account for this. The pro- posed binned JSD for perception is an effort in that direction so that metrics do not penalize human uncertainty, whilst comparing machine perception or attitudes with humans.
the reference human label. We acknowledge that collecting high quality

---

Published as a conference paper at ICLR 2025
4.2 CHALLENGES IN METRICS AND INTERPRETABILITY
Deciding which statistic is appropriate depends on the data and how well it fits the assumptions made by the statistical measure (Eubanks, 2017). IRA metrics were never designed for systematic errors, and any accommodation for errors were based on assumptions about typical human behavior. Errors made by L.LSs are rarely predictable, yet they are not random; rather, they are reproducible, making them systematic errors. The unpredictable nature of LLMs makes it difficult to design an effective metric that compares them with humans, given the uncertainty associated with human labels.
Despite the deficiencies in metries, some metrics may be a better fit, depending on the aim and the results of the study. For instance, in the case of perception- the goal to understand which among a given pair of models is better, and when the result is almost unanimous with one model overwhelmingly preferred over the other, then the agreement in rarer (minority) labels may not matter. Hence, measures such as Krippendorff"s s-ax and Fleiss-K which estimate chance agreement based on the observed label distribution can result in substantially lower IRA scores even when there is high agreement on the majority label (winning model), whereas Randolph's s-x assumes that the labels are equally distributed and hence is a better fit. For perception- based ordinal values, the extreme labels (such as I or 5 in the case of Likert 1-5) are usually strong indicators of human preferences, especially when supported by visualization as shown in Fig. 3. These extreme labels, might form the minority case, but might be crucial to interpreting the results. Hence, measures such as Krippendorff's s-or s-cr observed label distribution are a better fit compared to measures such as Randolph's 's-x: that assumes uniform distribution of all labels. For tasks where a single majority label is not sufficient represent human preferences, measures that do not assume a single gold label such as the proposed approach binned ISD for perception, can be a better choice to compare human and machine performance.
preference experiments, where
and Fleiss-x estimate chance agreement based the
In addition, statistical analysis, such as null-hypothesis and significance testing, is essential for de- termining whether one model outperforms another by random chance. Here, the chance component includes 2 obvious aspects, (1) chance due to the nature of samples in the evaluation set (2) un- certainty in human labels. A third aspect, even harder, is estimating the error rate as result of systematically unpredictable erroneous labels from any automated evaluator. Future studies should explore these problems, including approaches like resampling (Deutsch et al., 2021). Incorporation of chance in rank correlation is also an important aspect to account for when two models differ in rank, but the corresponding difference in absolute scores is negligible, then the difference in the rank may not be meaningful.
IRA metrics are challenging to interpret, and hence visualization is key to understanding gaps and strengths of any given metric. Effective visualization is trade-off between plotting every single data point (too much information that is hard to synthesize) and an aggregate view (summarized view where key information might be obscured). The proposed perception charts are a step towards emphasizing how an aggregate number may not be sufficient in capturing the underlying data, de- picting how human uncertainty varies across different labels, which in turn affects how machine performance can be interpreted across labels as shown in Fig. 3 and 4.
4.3 RECOMMENDATIONS FOR REPORTING EFFECTIVENESS OF AUTOMATIC METHODS
To conclude, comprehensive analysis of performance involves investigation beyond a single aggre- gate number, also demonstrated in an additional case study in Appendix 12. To that effect, in the case of comparing automatic evaluation with human evaluation, we recommend the following steps: 1. Stratification by uncertainty levels: As discussed in Sec. 3.1, uncertainty in human labels can obfuscate performance gaps between machines and human evaluators. Hence, we strongly recom- mend stratifying results by uncertainty proportions.
2. Multi-metric reporting: If there was no uncertainty, measures such as Fl would have worked. However, as a result of uncertainty, no single metric can capture important insights about every type of data as demonstrated in Sections 3.1, 3.2 and 3.3. Thus, we recommend reporting on multiple metrics belonging to different families, such as chance and wee--haaneeadjjsste measures, so each metric in its own way can assist in bringing the less obvious but critical aspects about the underlying data to the forefront.
3. Visualization of results: A single non parammeteiicc aggregate metric can rarely capture the en- tirety of underlying raw data, and hence visualization is key to understanding performance gaps, as discussed in Section 3.3. The proposed perception charts are a step towards making aggregate cor- relation more interpretable, as well highlighting the strengths and gaps of the automatic labeler.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Saab Mansour for reviewing our paper and providing valuable feedback.
REFERENCES
Xavier Amatriain, Josep M. Pujol, and Nuria Oliver. I like it... i like it not: Evaluating user ratings noise in recommender systems. In Geert-Jan Houben, Gord McCalla, Fabio Pianesi, and Mas- simo Zancanaro (eds.), User Modeling, Adaptation, and Personalization, pp. 247-258, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-642-02247-0.
Jacopo Amidei, Paul Piwek, and Alistair Willis. Agreement overrated: A plea for correlation to assess human evaluation reliability. In Kees van Deemter, Chenghua Lin, and Hiroya Takamura (eds.). Proceedings of the 12th International Conference on Natural Language Generation, pp- 2019. Association for Computational Linguistics. doi: 10. 18.11663/v1// 1999999222 URL https: blltep:://acaaattii /aaaaaaaanntoooooooeoo www.gg..org//... 8442.
344 354, Tokyo, Japan,
Lora Aroyo, Alex Taylor, Mark Diaz, Christopher M Homan, Alicia Parrish, Greg Cerapio-Garche Vinodkumar Prabhakaran, and Ding Wang. DICES dataset: Diversity in conversational Al eval- uation for safety. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:://ppnrrevee...co wtw.://wwweeeooommmi..o GJNvvswoUL.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:220400586, 2022.
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Al- bert Gatt. E. Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Mar- tins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. Llms instead of human judges? a large scale empirical study across 20 nip evaluation tasks. 2024. URL https: //axxiivvv. org/ / 6406. 18403.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2015.
Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. "seeing the big through the small": Can iims approximate human judgment distributions on nli from few explanations?, 2024. URL /attps:/arxxvv.rrtt abs 2406.17600
Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. Uncertain natural language inference. Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8772-8779, Online, July 2020. Association for Computational Linguis- URL https: http:/:///aaatahhhaa
tics. doi: 10.18653/v acl-main. 774.
2020
Cheng-Han Chiang and Hung-yi Lee. A closer look into using large language models for au- tomatic evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1928-8948 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/ URL https:tt////......t. /a//aaaaaannntoooc orwpooooooyoooooooooooooooo 2023. findings emnlp. -indingss-enlpp 599.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Mcasurement, 20(1):37-46 1960. doi: 10.117700033664440000000 URL ht /o.. org/10.1177 00011/77700001114440000000000000

---

Published as a conference paper at ICLR 2025
Daniel Deutsch, Rotem Dror, and Dan Roth. A Statistical Analysis of Summarization Evalua- tion Metrics Using Resampling Methods. Transactions of the Association for Computational
Linguistics, 9:1132-1146, 10 2021. ISSN 2307-387X. doi: 10.1 00417. URL
https://doi http:/ooo.oogooooo wwwgr....000. 1etl1111llaaaaaeeeeeaeeea _a____a__
Daniel Deutsch, Rotem Dror, and Dan Roth. Re-examining system-level correlations of auto- matic summarization evaluation metries. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 6038-6052, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 10. http:///1122222aallllam 442. URL https: httppp/s///aaaatahlaaaaeee org/2022. naacl-ma in.in.... 442.
Daniel Deutsch, George Foster, and Markus Freitag. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration, 2023. URL http:ttss////iiii.rrrttret abs/abss/000 14324.
Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, and Dan Roth. ConSiDERS- human evaluation framework: Rethinking human evaluation for generative large language mod- els. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1137-1160, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL org/ 2024 acl long. 63.
https:/ https///claatthhoooo
David Eubanks. (re) visualizing rater agreement: Beyond single-paramens parameter measures. Journal of Writing Analytics, 1, 2017.
Alexander R. Fabbri, Wojeiech Kryáciński, Bryan McCann, Caiming Xiong, Richard Socher, and
Dragomir Radev. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 04 2021. ISSN 2307-387X. doi: 10.
1162/tacl a 00373. URL https:ttp///cm /ww.////i..///// org/ 10.0.... 11 62/tac/ Lataaaaaa______11________ _03333.
Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 36(5):378-382, 1971. doi: 10.1037/h0031619. URL https:/ http:///c.../gg////// h0031619.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07888, 2022.
Xin Jin, Le Wu, Xiaodong Li, Siyu Chen, Siwei Peng, Jingying Chi, Shiming Ge, Chenggen Song, and Geng Zhao. Predicting aesthetic score distribution through cumulative jensen-shanne di- vergence. Proceedings of the Thirty-Scond AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAATISIAATTSSAAATT AAAI Press, 2018. ISBN 978-1-57735-800-5
Ankur Joshi, Saket Kale, Satish Chandel, and Dinesh Pal. Likert scale: Explored and explained. British Journal Applied Science Technology, 7:396-403, 01 2015, doi: 10.9734/BIAST/ 2015/14975.
Daniel Kahneman and Amos Tversky. Variants of uncertainty. Cognition, 11(2):143-177, 1982.
Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein. Noise: flaw in human judgment. Little, Brown Spark, New York, first edition edition, 2021. ISBN 978-0-316-451400. 978-0-316-26665. 9. OCLC: on 149944231.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, and Minjoon Seo. Prometheus: In- ducing fine- mne-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2024, URL https: /ttp:///ppee/ieieeee net/forum? id=8euJa Tvekw.
12

---

Published as a conference paper at ICLR 2025
Klaus Krippendorff. Reliability in content analysis: Some common misconceptions and recommen- dations. Human communication research, 30(3):411-4333 2004.
Klaus Krippendorff. Computing krippendorff's alpha-reliability. 2011. URL http:://api.
semant commantisccooaas www.ar.oggo/ooousooeieeoa 59901022
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: : / acclanthoooogyyo........ org/w04-1011
Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information
Theory, 37(1):145-151, 1991. doi: 10. 10.1 10.1109/18..1155.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.0344 033423 2023a.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human aligmment. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023b. Association for Computational Linguis- URL https:ttpp.... //ac:///alaatatcc antthloohhyyyy org/2023.
tics. doi: 10. 11188/63/112223mmmmlmmm emnip-man ma-mmain. 153.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Anna Rogers, Jor-
Grounding summarization evaluation with robust human evaluation. dan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4140-4170 Toronto, Canada, July 2023c. Association for Computational Linguistics. doi: 121185/v122223..clll 228. URL https: aa///claannt www.atooooottttooooooooooooolola org/ (ergg/2222 2023. acl- -oog..on.... 228.
Mary L McHugh. Interrater reliability: the kappa statistic. Biochem. Med. (Zagreb), 22(3):276-282, 2012.
Shikib Mehri and Maxine Eskenazi. USR:
An unsupervised and reference free evaluation metric for dialog generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 681-707, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/0020 acl-main.64. URL hitps: : / wwt.aaaaaaancoooooo
org/ 2020. acl - ma ina 64.
Christian M. Meyer, Margot Mieskes, Christian Stab, and Iryna Gurevych. DKPro agreement: An open-source Java library for measuring inter-rater agreement. In Lamia Tounsi and Rafal Rak (eds.), Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: System Demonstrations, pp. 105 105-109, Dublin, Ireland, August 2014. Dublin City University and Association for Computational Linguistics. URL htw.tttts//eee a ho ogy. org/C14-2022
Prabhaker Mishra, C M Pandey, Uttam Singh, and Anshul Gupta. Scales of measurement and presentation of statistical data. Ann. Card. Anaesth., (1)4:4119 October 2018.
M M Mukaka. Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi Med. J. 24(3):69-71, September 2012.
Justus J Randolph. Free-marginal multirater kappa (multirater k [free]): An alternative fleiss' fixed-marginal multirater kappa. Online submission, 2005.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
13

---

Published as a conference paper at ICLR 2025
Hamed Taherdoost. What Is the Best Response Scale for Survey and Questionnaire Design; Review of Different Lengths of Rating Scale Attitude Scale Likert Scale. Post-Print hal-02557308, HAL, 2019. URL https: iidas.dd..... repec.org/ http://////p.aa///a/aaaaaaa---// html.
Debby ten Hove, Terrence D. Jorgensen, and L. Andries van der Ark. On the usefulness of interrater reliability coefficients. In Marie Wiberg, Steven Culpepper, Rianne Janssen, Jorge González, and Dylan Molenaar (eds.), Quantitative Psychology, pp. 67-75, Cham, 2018. Springer International Publishing. ISBN 978-3-319-77249-3.
Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-11111 1974. doi: (1t12//ccicncc..551155..7. htt.l//1eeeenneeeeeeeeeeeeeeeeee 185. 4157. 1124.
URL httpss//wwe
science. artp//cc..gg//////aaaaaaaaa/a
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. Human evalua- tion of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language, 67:101151, 2021. ISSN 0885-2308. doi: http:://doiorr///00010//// 101/...0111666..ss http://iineeraiiieeee/ee
2020.101151. URL https:/ //t:::///wwwww//// /ww. sciencedirect.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions evaluate the fac- tual consistency summaries. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Mceting of the Association for Computational Linguistics, pp. 5008-5020, Online, July 2020. Association for Computational Linguistics. doi: 18653/v/1 2020.acl-me m50--m..i.450. URL https: : //a///aaanhhloogoo org/
wwwg/////000 2020 awc-mmalai.. 450.
Yuxia Wang. Shimin Tao, Ning Xie, Hao Yang, Timothy Baldwin, and Karin Verspoor. Col- lective Human Opinions in Semantic Textual Similarity. Transactions of the Association for Computational Linguistics, 11:997-1013, 08 2023. ISSN 2307-387X. doi: 16.1162/tacl 00584. URL. httptt//////o// /ttt:p:://doi.oog//10/ 1ee11112/ttee 00584.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1//65/////N1--/11 /NN8-1101..111111111111111111111 URL
Jun S. Liu Xinshu Zhao and Ke Deng. Assumptions behind intercoder reliability indices. Annals of the International Communication Association, 36(1):419 2013. doi: 1080//28288888 2013.11679143 URL httpt::t///ett //tp::///oo..ee http://ooo...ggg//// 00011000022888 22888888888888 2013.1167914..
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consis- tency with a unified alignment function. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11328-11349 Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/20233 //aclanthology wttpey..rr///000 org/2023. acl-long: 634.
URL https:
Tianyi Zhang* Varsha Kishore*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. International Conference on Learning Representations, 2020. URL h/tp::/:////peeeeeeeeee net/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LL-M-s----jdde with MT- and chatbot arena. In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: / //openzeview. wttp://wwwwwweeeeeeeeeeeeeo http://ooorumiioououooooeeeeee
14

---

the main Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROMPTS USED FOR NLI
These prompts are reused from Liu et al. (2023a)
presented with premise whether the hypotheais is entailed by the premise by choosing one of the following answers:
hypothesis
that premise.
to decide
[[el]: The hypethesis follows logically from the information contained in the premise.
[[n]]: : It la not possible to decermine whether the hypothesta ia true or false without further information.
[[c]]: The hypothesis is logically false fzom the information contained in the premise.
Read the following premise and hypothesis thoroughly and select the correct answer from the three answer labels.
Premisa:
[premise]
Rypothesis:
thypothesis;
Make selection from "[[a]]", "[[n])", "[[o]]". Only write the do write
A.2 PROMPTS USED FOR SUMMEVAL
Note: These prompts are reused from Liu et al. (2023b) used in G-Eval.
COHERENCE
will given surmary written
article.
Your Lask Lo rate the summary on one metric.
please make sure you read and undorstand these instructions carefully. Flease keep this document open whi la reuiewing, refer
needed.
Evalustion Criteris:
Coherence (1-5) the collective quality of all sentences. slign this dimension with the ouc quality question of structure and echerence whereby "the summary should be well-structured and wall-orgenized, The summary should not just be heap related information, but should build from sentence conerent body information about topic."
evalustion Steps:
1. Read the news article carefully and identify the main topic and key points.
2. Read the summory and compare it the article. Check if the summary tople and key points of the news article, and if it presents them in clear and loginal order
3. Assign score for coherence on scale of 1 to 5, where 1 is the lowest and 5 is the highest en kveluation
examplo:
Source Text:
[document))
Summary:
(summary]
Evaluation Form output the score only:
CONSISTENCY
given news article. then given summary written article.
Your task rate the summary one metric.
15

---

Published as a conference paper at ICLR 2025
Please make sure read and understand these instructions carefully. Fleage keep this document open while reviewing, and refer to it as needed.
evaluation Criteris:
Consistercy (1-5)
alignment between the summary summorized sousce. factual consi stent aummary containa only statements that are ental led by the source document Annotators weze also saked to penalise summaries that contained hallucinated Iacta,
evalustion Steps:
1, Read the news article carefully and identify the main Tacts and detaila it presents. summory containg any
compare to the article. errors that are not supported by the article.
Read summory
3. Assign scoze for consistency based oa the Evaluation Criteria.
examples
Bource Text:
(dccument)
(summary]
Evaluation Form output the only:
FLUENCY
You will be given one summary written for a news article.
Your task is to rate the summary one metric.
ease make aure read and understanc these instructions carefully. Flease keep this doc document open while zeviewing, and refer to it as needed.
loa Criteris:
Fluency [1-3): the quality the summary terns granmar, spelling, panctuation, choice, and aentence structure.
1: The summary has many
that make it hard understand sound unnatural.
2: Foir, The summary has some errors that affect the clarity or smoothness of the text, but comprenensiale.
pointa 3: Good. The cummary has few or no ercors and is ecay to read and follox.
Example:
Summary:
tammary]
avalustion Form output the secre only:
RELEVANCE
You will be given summary written for article.
summary
please sure you understand thase instructions carefully. keep this document le reviewing, and refer to it needed.
avaluation Criteris:
(1-5) selection important conteat only important information from the source document. Arnctators were instructed to penaliza summartes which contained redundancies excesa information.
source. summary should include
16

---

Published as a conference paper at ICLR 2025
evaluation Steps:
1. Read the surmary and the source decument carefully.
2. Compare the summazy to the source document and identify the main points of the article.
the main points of the articla, and how irrelevant or redundant Information it contains.
Assass how well the summary
Assign relevance
to 5.
Example:
Bource Text:
summaryy
vvaluation Fozm output the score only:
A.3 PROMPTS USED FOR МТBENCII PREFERENCE DATA FOR SONNET
Human: For this task, you will be shown two conversations between user and an AI assistant, labeled Your goal instructions and more helpfully answezs their question.
response
follows the userfa
<Pref.Judgment>
<Conversation
(ccnversstion_a] </Conversation A>
<cenversation g>
[scnversation_hl
</Conversation R>
<instructions>
Evaluate the two conversations and choose one of the following:
[[A]] 1f Conversation A asalstant batter fo userfa instruct fons and
their questicn
[[B]] if Conversation
better
userfa instruction and answers
their queation
[[C]] if both AI assistents are equally good/poor in following instructions and answering the
user's question
like helpfulneas, relevance, acouracy, depth, creativity, appropriate lavel of detail when making your evaluation. Do not show pasitional bias cowards A or 5. Reaponse length ahould unduly influence your dectaton. Make selection from "[[A]]", "[[B]]", "[[0]]", Only write the do not write ressons. </Instructions> J/dggntusgment>
Consider
Assistant:
A.4 PERCENTAGE AGREEMENT
The percentage agreement, as defined by McHugh (2012), for n items where Iı, ,n represent each item that is labeled, with R raters and C categories, as the maximum percentage of votes the most frequent item gets as long as is greater than 1. Formally:
Percentage agreement:
R
max cE(L....cC
Σ
(a)=a))
(IR. (e)=0)>1)
(3)
R
where zi is annotation for 2; by the rth rater. Note that the A is the indicator function that returns 1 if condition A is and 0 otherwise.
17

---

Published as a conference paper at ICLR 2025
A.5 IMPACT OF LABEL UNCERTAINTY ON ORDINAL SUMMEVAL DATASET
We create multiple subsets of 100 samples, with each subset having a predefined portion of samples with perfect agreement in human labels. We then randomly select samples from the dataset based on this portion. This simulation helps us understand how correlation measurements change with varying levels of human label uncertainty.
Geval (Relevance)
Sonnet (Relevance)
Mistral (Relevance)
Maltic
K-a HH
0.8
Comietoon
K-a FIR
Comiletion
Comelatio
0.6
p
0.4
0.2
0.2
0.
Samples perfect agreament
porfect agreement
Samples agreament
Geval (Fluency)
(Fluency)
(Fluancy)
1.0
1.0
1.0
0.8
Comsiston
0.6
Consiatin
Cursdaaton
0.4
0.4
-0.2
0.0
20
40
80
90
100
20
40
60
80
100
20
40
60
80
100
Samples with perfect agreament
%6 Sampiess with perfect oqmeementt
%6 Samples with perfect agreement
Figure Impact of noise on ordinal dataset Summeval
A.6
VISUALIZATION OF ROUGE-SCORE CORRELATION
Relevance T..0.26, p.0.33
T,:0.49,
1.0
0.8
2-
0.2
1
0.0
2
Median human labels
Median human labels
Figure 6: Visualizing correlation with human median rating for relevance: (Left) Rouge-1 (Right)
LJ (Mistral). Note: Rouge scores are continuous values [0,1], humans and LJ use integer values 1-5.
18

---

Published as a conference paper at ICLR 2025
A.7 CODE FOR BINNED-ISD DINNED-IDD-PERCEPTIO
Here we first show a working sample, followed by the code below.
In the sample dataset in Table 4, have bins, Bin 2 and Bin 3, where H represents humans and M represents model values
Valuea compared in
B (item item B), (item & item F[12, 2, 31 + 21,2, M113,2] + (1,1)) E112, 2,2,, 2,13,2,1,11)
Val ues Values
compared in Rin
B(item C) Mi item C) 3,32,,,3,, M1[2, 211
Now we compute JSD for each bin to compare the similarity in each bin. Comparing the probability distribution of values between H and M, assume Likert scale 1- 3, where the index represents the Likert value and the index value corresponds the probability of that value:
Bin JSB 18, Bin 3 JSD Ia, M) JSD (H]O, 1/3, 2/31, MiO, 2/2, 011
(8[176, 476, 176), M(2/4, 174, 1/4])
Translating the above to python call
Ezom scipy.spatisd import diszance
bin2_jad bistance.jenssesshen bin3_jad distance. sistance..jenenchammie [19, 1/3, 2/3,, [0, 2/2, 0])
(IL/E, 1/61, 12/4, 1/4, 1/411)
Total binned JSD is the weighted sum of number of samples in each bin, where contains 2 samples A and B, contains sample
Binned JSD 2/3 bin2jsd 1/3 bin3 = 2/3 0.31 1/3 0.56 0.39
Id
Human labels
Human median (Bin)
Model predictions
A
2, 2,
2
3, 2
B
2,
2
1,
C
2, 3,
3
2, 2
Table 4: Sample dataset
import itertcols
import statistics
frem collections import Counter
from statistics import mode
from scipy.spstia. import distance
def scapute_binned_]] lamen_ machine__abells_colmmm bingtype= median""
*.*
Compute J3
Bbaram df: DataFrame containing the columna haman_Jabels_coo machine Laciine__llele__ee soooelssclmmm
Eparam human_labela, labela_columm: This is the name of the column containing human labels. Each zow
this is 1ist
name Rach row value in this column is a Hsst of multiple labels asai gned by the machine
êparan pachine_labela___ column:
consaining
Sparam bin_type: The type of binning, either median majority depending the type data *.*
seeult_weighted.e reault_hin_datstt
Bin function can be median or mode ( nost frequent value)
bin_ statistics. median if bin_type 'median' else mode
unique
labels = sorted (set litertcols. chain (+ (df [humar (hamon_labela labels hassbll_coouun____...aaaaaa_laas wow.oomttooosstli .tolist
[machine_ laceenne_aaaaaaeeeeeeeeee labela [_oolumn] (1))
19

---

Published as a conference paper at ICLR 2025
Assign bin number
_i__bin__mbbrr [auman_labell_oool [ooluumn] .apply (Lambda Xt bin_ (X)
For 1 in labels:
# Get bin items
df_bin_items df(dS__in_numde
bin_weight (en / (de(
result_bin_detaile
"hin_itema": af_hin_itema,
"bin_number"
"isd": None, "bin_weight"; bin_weight, "dist_humm:=" "dist_machine":11
Skip if no if Len (di__inn_ Lemea) -- U: continue
bin
is
human, Luman__abeell labelaleel..... sareela_ffruueeeeeee bin Counter(
1ist iitertoole.chain (eff (human_labela,
mechine febeess_feeuueniies..l iin Counter
machine__obels_collnne
dist_humas [auman labellaeelseee frepueessreg
sum sum(Counte:
lamon_labels_freee fwequencies sac_ees_ii__________ htt.biinn..aaa.llll.. (1)
for in labela]
dist_machine (machine frrooleceeergeciie___r (in_e_s_____n_i_]________ (Counter'
mechine _bbeeesssseeueccieeeeee..e. in_eessnnnnnnnnn values
for i in labels)
result bi detal
result_bia_detade
"
iad distance. htw.:aacc...iccnccaammoooo.ooo (dist_ human, dist_machine)
sesult_bin__etaaillll ["jad"]
result_ + bin_waight jad
return result_neightedd_js besult_bin
A.8 TOY EXAMPLE: APPLICATION OF BINNED-JSD AND WHERE CAN BETTER METRIC
The use of binned-JSD solves specific problem, where single majority label is not sufficient to represent the human preference, as mentioned section 4.2.
The advantage of binned-JSD can be demonstrated using toy example follows. If we only used a single gold human label (human median in this case, could also human majority) to compute correlation, the acceptable values that humans have chosen is lost. As result, metrics such Krip- pendorff will see treat any value that is equidistant from the human single "gold" label as acceptably similar. For instance, assume that humans choose "disagree" (selections (2,3,3)). A good model chooses "disagree" and bad model chooses "agree" (completely different human choice), because both "disagree" (Likert and "agree" (Likert are equidistant from the median/majority value (Likert 3- Median value), K-alpha has assigned very similar scores for the model that has assigned "disagree" (Likert 2) and the model that has assigned "agree" (Likert 4). Rank correlation metrics, in addition to their misfit in comparing item level Likert scores already discussed in the paper section RQ2, also have similar problem and deems the model that poor the better model shown below. Our proposed approach, on the other hand, assigned lower (better for JSD) scores to the better model as the "good" model assigns values that the humans have chosen, compared to the poor model that has predicted different score altogether.
"neutral" (median/ majority value)
The example above also exemplifies how unless we know apriori which better model, it is difficult to identify advantages well as shortcomings with correlation measurements including the proposed binned-jsd. Effectiveness of metrics depend on the data. We don't it know for certain if model appears be better/worse because of gaps in metrics, creating chicken-and-egg problem
20

---

Published as a conference paper at ICLR 2025
Human labels
Human median (H)
Good model
Poor model
2, 2, 3
2
3
1, 2, 2
2
2, 3,
3
2
Table 5: A toy dataset to illustrate the use of binned-jsd
Metric
Good Model
Poor Model
Does better model score hetter
Kendall-r Spearman-p
0.00
0.82
False
0.00
0.87
False
Krippendorff-a
-0.06
-0.06
False
JSp
0.56
0.65
True
Table 6: Demonstration how metrics can rank a poor model better as a consequence of ignoring valid labels that humans have assigned. The proposed Binned-JSD on the other hand uses all the human labels, and assigns lower JSD (better) to the better model.
in measuring the effectiveness of metric itself. Not knowing which metric appropriate common problem when it comes to correlation metrics (ten Hove et al., 2018), including problems with Cohen's Kappa (Krippendorff, 2004) despite it being commonly used, including many L.IM as judge papers.
Hence, our recommendation Section 4.3 for stratification, visualization and multi- metric reporting so we can interpret the strengths and gaps in the metries.
21

---

Published as a conference paper at ICLR 2025
Rougel
Model
Krigpendorff a-o
J.8%
Spearman's h-p
Kendall i-
Partition (% samples)
HH
MM
HM
A
H M
HM
HM
i: 5T
H Mg
HM
it M
G-Eval
0.53
0.74
0.31
0.25
0.29
0.39
0.53
0.47
0.57
0.47
0.41
1. All (100%)
0.18
0.14
Mistral
0.55
0.78
000
0.07
0.43
1126
D:06
0.49
016
0.44
0.43
Sonnet
0.55
0.83
0.31
0.22
0.51
0.34
0.33
0.41
0.30
0.29
PA-1 (128)
0.78
1.00
0.79
0.65
0.35
0.60
0.62
0.68
0.56
Sennet
1.00
0.86
0.24
0.76
0.24
0.71
0.43
0.43
0.48
0.39
0.39
G-Eval
0.3
0.75
0.30
0.23
0.20
0.40
0.51
0.45
0:35
0.43
0.39
Mistral
0.44
0.44
Sonnet
0.33
0.81
0.22
0.31
0.20
0.51
0.32
0.31
0.40
0.28
0.28
G-Eval
0.19
0.70
0132
80.15
0.26
0.35
0..5
0.42
0.38
0.50
0.35
4. 60% (215))
0.17 0.13
Mistral
0.19
0.78
0.55
00.37
0.39
0.30
0.57
0.41
0.64
0.52
0.36
Sonnet
0.19
G-Eal
0.00
0.74
0:25
0.37
0.53
0.47
0.39
0.47
0.41
1. All (1006))
0.33 0.26
Mistral
0.40
0.79
0.38
0.02
0.33
0.31
0.54
0.44
0.61
0.49
0.39
Sunnet
0.40
0.79
0.35
0.05
0.31
0.38
0.39
0.37
0:48
0.36
0.33
2. PA (14%)
0.38 0.30
Mistral
1.00
0.30
0.43
0.57
0.43
0.59
0.71
0.71
0.72
0.63
0.85
Sunnet
1.00
0.77
0.36
0.54
0.36
0.51
0:49
0.49
0.60
0.45
0.45
G-Evvll
0.40
0.25
0.25
relevance
PA
Sonnet
0.40
0.30
0.34
0.07
0.32
0.37
0.39
0.37
0.49
0.36
0.34
G-Eval
-0.09
0.09
0.18
00.27
0.22
0.33
0.40
0.31
0.42
0.38
0.27
4. 60% (199)
0.28
0.22
Mistral
-0.99
0.54
0.32
0.41
0.20
0.28
0.42
0.25
0.16
0.40
0.22
Table 7: All stratification results for SummEval. Summeval has 3 expert human annotations to select Likert values from 1-5, hence possible values for stratification by percentage agreement (PA) on the median label are 3/3=100 %. 2/3=66.67% and 1/3=33.3%. Hence, we stratify by 100% agreement, (60-100) and <60%.
A.9 VISUALIZATION OF PREFERENCE DATA
38.9 9 HW=A
27.8
33.3
5 8
78.8
15.4 HW=T
H HW=B
HW W=T
HW=A
HW=B
0.8
Probabiity
Probabilty
0.6
0.6
02
0.2
0.0
A
B
T
A
B
A
B
A
T
38.9 9
27.8
33.3
8
78.8
15.4 HW=T
HW=A
HW=B
HW=T
HW=A
HW=B
1.0
0.8
0.8
Probabilty
Probabilty
0.6
0.6
0.4
0.4
02
0.2
0.0
Figure 7: Pairwise preference HM perception chart: Human preference distribution vs. (Top) LJ GPT-4 (Bottom) LJ Claude Sonnet 3 for a given majority label per model pair. T is tie. Compares model pair (Left) Claude-vl (A) GPT3.5-urbo (B) (Right) Alpaca-13D 13B (A) vs. GPT-3.5-turb (B).
A. EXTENDED TABLE RESULTS
22

---

Published as a conference paper at ICLR 2025
Xi Xripeendutl'so soo
Agnecment
JS.
Flaeaw
Randolghow
LJ Winnte
MP
Patition samples)
MM H"M""
HM"
HFM""
WM
H"M
HMM
H"M"
WR
(52)
0.79
0.24
0.32
-0.05
0.20
0.81
0.77
1.67
0.11
0.31
0.19
0.65
0.51
B
0.81
0.02
0.92
1.00
0.13
0.92
GPT
HH: (80,1000) (11
1.00
0.00
0.00
0.83
1.00
11.00
0.25
-1.00
-1.00
8
1,00
0.00
383:
0.61
0.04
0.65
0.22
0.34
0.67
(AIp, Gp3)
5. HHH (0.60) (7)
0.71
10.18
0.35
20U5
.0.02
0.46
0.71
0143
0.32
0.30
-0.10
0.43
0.14
8
0.71
0.00
0.79
0.24
0.90
0.81
0.38
0.00
B
0,90
P-HH (26)
0.92
1000
1.00
-0.02
100
-0.02
1.00
0.92
1.92
0.17
-0.04
-0.04
0.85
0.85
B
1,00
CLDS
Sonae
3EEE: (80,
3.00
0.83
1,00
19H (60,80) (18)
0.61
-0.04
0.93
-0.04
0.00
0.13
0.65
0.50
11.56
0.37
-0.07
0.11
0.25
0.33
8
0.78
0.17
383: (0.60)
0.71
1,00
0.46
All (36)
0.30
0.40
0.54
-1114
0.30
0.70
0.59
1155
0.11
0.54
0.29
0.54
0.29
A
0.39 0.00
0.64
1.00
0.10
0.22
0.22
1HH (90, 100) (01
0.00
HH:
0.53
0.26
(CMd, Cy3?
HHH (0U60) (1)
1.00
0.00
0.00
0.40
1.00
11.00
0.52
-1.00
-1.00
A
1,00 0.00
(16)
0.90
0.35
0.56
P-HH (14)
0.64
0.96
-0.07
10
-0.07
1.00
0.43
0.43
0.24
-0.13
-0.11
0.14
0.14
B
0.64
0.00
Sonee
HH:
HHH (60,50) (21)
0.52
10.02
0.96
0.37
20.39
0.18
0.67
0.67
11.88
0.39
0.35
0.16
0.50
0.11
A
0.67 0.14
HH::
1.00
All (98)
0.53
0.26
0122
-0.01
0.11
0.74
0.5
11.50
0:26
0.10
B14
0.25
8
0.74
0.21
HH: (90, 1000 (2)
1.00
-0.12
0.00
-2.12
0.00
0.80
0.50
11.00
0.23
-0.33
-0.33
8.00
0.00
8
0.50
0.50
HHH (20)
0.50
0.65
(1p), Opd)
Hil: (00600) (5)
0.60
10:27
0.24
2002
0.08
0.40
0.20
0.40
0,38
0..58
-0.20
0.20
-0.0
B
0.80
0.20
All
0.76
0.74
0.82
100
0.66
0.19
08
0.19
1.00
0.54
0.64
0,28
0.15
0.15
0.45
0.45
B
0.64
0.18
Sonnet
183: (60,80) (20)
0.50
0:08
0.81
0.22
8015
0.65
0.65
0.50
0.65
0,06
0.20
00
B22
0
0.65
0.05
Table 8: All stratification results for MTBench. MTBench has varying number of 3 to 5 human annotations per item select preferences A, B tie. Hence, possible values for stratification by percentage agreement (PA), so if an item contains 5 annotations (human labels), then possible strat- ification groups that the item can belong to by PA on the majority label are 5/5=100% 4/5=80% 3/5=60% or lesser. Please note that many of these partitions have O or less than 5 samples. With higher proportion of noisy or uncertain samples the HM correlation seems to outperform HH cor- relation. The exceptions to this finding are highlighted, and for these the sample size is quite small (< 5).
23

---

Published as a conference paper at ICLR 2025
A.11 RANDOM SIMULATION CODE
A.11.1 RANDOM SIMULATION CODE BINARY CLASSIFICATION (NOMINAL DATA)
In this code, we create simulated synthetic dataset, where we mimic the case of 2 human annota- tors labelling a binary classification problem. We also create a random labeller, where the random labeller assigns random labels.
Import random
import mumpy
import random import pandaa
():
Simulatea a randon binary datasec with human labellers.
The 2 humans either both pick 0 (b) pick 1 (e) picke and the other picks 1 or vice versa
dataset_size
aaanle 1), 2) for in range (natassttttt_st
// 2)
battis___bttthpiccceee [[1, 11 for in renge (dataset_size // 4)] hunans_
humans_ 2aaans___hothpiickk.. = [[0, 3] for in renge (dataset_aize // 4)
humans humans_2_onee humana_ 2_____2__ott 2_tth__hhhhhhhhkkp_____ humans bothh_hhcc__0
aara__2_annoatten wacpiorrr_biaeeeeeeeeee hataancy__mmmllaee
candon_iabeller_dtte [np. cho.cee. 011 for in (dtasesttts)
Firal df
Fd.DataFramm (data- "hwman_laaeeeeenn human annottttoorrraoaooooooo biterrr__nnaaaalnaslaaee
Landon__abellls La:don__lbellee_cchooe
return
A.11.2 RANDOM SIMULATION CODE 3 WAY CLASSIFICATION (ORDINAL DATA)
In this code, create simulated synthetic dataset, where mimic the case of 2 human annotators labelling a way classification problem to assign labels [1,2 or 3]. We also create a random labeller, where the random labeller assigns random labels.
import mampy 25 import random import pandaa
simulates random way classification 1-2-3, with labellers.
The 2 humons can sither ta) both pick 1. (b) both pick 2. and so on (c) disagree :zeturns
dataset_size 600
humana_disagree [random. sample 2, 31,
range (daraset_sie 211
humans_agree__ = [[I, 11 for in range (dataset_size // 6)
humans_agree_2 [12,
range tdscaset_pize // 6)]
humara_agree_ [[3, 3] for in range (estaset_s112 // 6)
hunatt__annotatcess crtpecss_rrddee__iieee simulattee humans disagree humens_agres_L humans_agres__ humens_agrea__
randon_iabeller.cee rop.randon. chaicen 31) for in range (dataset_sizel)
dt Pd.Dataerame (data=l "aaitaal""uniee Labellt; human_2 Laman_2_annotael annotators, wtw.rre_ooonaaaooooottttooo
Landon_lbbelles zandom_lebee Jebeller_choise
24

---

Published as a conference paper at ICLR 2025
A.12 CASE STUDY
Here, we include a case study on a subset of the datasets (see details in Table 9) used in the LLM benchmark study by Bavaresco et al. (2024). We use the preprocessed datasets available in github! provided by Bavaresco et al. (2024) These datasets were evaluated on LLM Llama-3-70B.
Dataset
Size
Criteria
# Annotations Choices
60
Understandable
3
(0, 1)
Topical chat (Mehri & Eskenazi, 2020)
60
Uses Knowledge
3
(0, 1)
QAGS (Wang et al., 2020)
953
Factual consistency
3
(yes, no)
Dices 350 (Aroyo et al., 2023)
350
Safety
100-150
(yes, no, unsure)
Table 9: Categorical datasets summary
Krippendorff-o- H M
& Agreement
Randolph-rs
Partition
Size (%)
H M
All
100.0
-0.01
0.58
-0.01
0,00
0.59
0.988
0.97
0.02
0.96
0.94
0.93
0.02
0.58
0.97
0.94
0.93
TC
PA-66.67%
3.3
0:25
1,00
0.67
1.00
1.00
00.33
-0.33
All
100.0
0.48
0.81
-0.03
0.51
0.98
0.99
0.93
0.04
0.91
0.98
0.87
0.04
UK
PA=1
93.3
100
1.00
-0.01
1.00
1.00
1,00
0.96
0.04
1000
1.00
0.93
0.07
6A-66.66%
6.7
10:26
0.44
-0.17
0.09
0.90
0.50
40.33
0.70
0.00
0.33
100.0
0.93
0.70
-0.21
0.87
0.94
QG
PC
PA-1
65.6
100
0.94
0.84
0.16
1000
0.99
0.94
0.06
100
0.95
0.87
0.13
PA-66.67%
0.91
0.74
+0.33
0.91
0.49
0.79
0.39
0.09
PA= =1
0.00
80%
< 100%
0.81
0.34
066
0.01
60%
<= PA 800
48.6
0.15
0.80
-0.27
0.42
0.71
0.93
0.39
0.32
0.34
0.83
0.08
0.26
40% PA.
0.76
-0.11
0.43
0.79
0.14
Table 10: Categorical datasets evaluation: Results on Topical chat (TC) on criteria (Understandable (UND), Uses Knowledge (UK)), QAGS (QA) on criteria factual consistency (FC) and DICES (DC) dataset on safety (SAF). With higher proportion of noisy or uncertain samples the HM correlation seems to outperform HH correlation. The only one exception is highlighted, where the sample size is quite small (ss 4 samples as TC dataset has total of 60 samples) in partition PA-66.6799 (2/3 votes for majority label).
A.12.1 ANALYSIS OF RESULTS
1. Effects of Multi-metric reporting: On the Topical Chat (TC) dataset, for understandable criteria the aggregate number (Krippendorff-o- -0.01) of " score of -0.01 superficially seems to imply that HM correlation low as shown in Table 10. However, percentage agreement (score 0.97) and Randolph-x (score 0.93) score quite highly, indicating that class imbalance has substantially lowered Krippendorff-a pretty close to 0.0. Also note that over 96% of the samples have perfect human agreement, however the overall HH Krippendorff- a is quite low scoring -0.01. This effect of how various chance adjusted metrics impact correlation scores is also discussed in detail in section 4.2.
2. Impact of stratification: When we compare the overall performance (column All in Ta- ble 10) on dataset TC (criteria understandable) with dataset QAGS, Randolph-x drops sub- stantially by 19 points (0.93 - 0.74). However, QAGS dataset has around 66% of the samples that have perfect human agreement, while TC has 96% of the sample with human agreement. When we compare the samples with perfect human agreement between the datasets, the model performance gap reduces to just points (0.93 - 0.87), pointing to how comparing datasets with different proportion of uncertain samples can affect our con- clusion (in this case incorrectly that the model performance is substantially lower in QAGS compared to TC). With DICES dataset (crowdsoureed with over 100 annotations per item, with no perfect agreement items), on the other hand, the model seems to struggle across
25

---

Published as a conference paper at ICLR 2025
all metrics and stratification groups, indicating much deeper investigation is required. The general trend where in a stratified group with relatively higher proportion of uncertain samples (as measured by low HH correlation), the H"M correlation seems to outperform HH correlation as highlighted, as shown Table 10 also applies, indicating to how mod- els can superficially appear approximate human majority when proportion samples of uncertainty is relatively higher, demonstrating the impact of RQI discussed in section 3.1.
A.12.2 ANALYSIS OF ORDINAL DATA
Firstly, looking at Table 11, superficially looking at the aggregate scores Row marked "All" across all criteria) it might appear as though (a) absolute human to machine correlation is low (columns HM) (b) when we compare HM correlation HH, HM is similar to HH correlation.
Dataset
Size
Criteria
Annotations
Choices
60
engaging
1,2,3
60
maintains context
1,2,3
Topical chat (Mehri & Eskenazi, 2020)
60
natural
1,2,3
60
overall
1 1,2,3,4,5
Krippendorff's- s-cr
JSb
C
Partition
Size (%)
HH
MM
HM
A
HM
HM
HM
ENG
All
100.00
0.03
0.72
-0.02
0.05
0.24
0.25
0.24
PA=1
75.00
1.00
0.66
-0.14
1.14
0.34
60% S PA 80%
25.00
0.39
0.82
-0.01
-0.38
0.15
0.26
0.25
MC
All
100.00
-0.06
0.74
-0.07
0.01
0.13
PA=1
81.67
1.00
0.65
-0.04
1.04
0.20
60% < PA < 80%
18.33
0.45
0.78
-0.17
-0.28
0.07
NAT
All
100.00
-0.07
0.71
-0.20
0.13
0.24
PA=1
78.33
1.00
0.75
-0.18
1.18
0.35
60% < PA < 80%
21.67
0.46
0.59
-0.25
00.21
0.14
OV All
100.00
0.10
0.75
-0.28
0.38
0.33
-0.17
-0.16
PA=1
61.67
1.00
0.76
-0.21
1.21
0.41
60% < PA < 80%
33.33
00.1
0.72
-0.48
0.17
0.35
-0.37
-0.35
0 PA < 40%
5.00
0.32
0.66
-0.25
00.07
0.24
Table 11: Analysis of performance on the Topical chat dataset on criteria Engaging (ENG), Main- tains context (MC), Naturalness (NAT) and Overall (OV). Overall, all the metrics that measure human - to machine correlation, across all metrics, seem quite low. At this point, it is quite difficult comprehend what causing this, until we visualize them shown in Figures Figures
In order to investigate the poor HM, we perform the following analysis.
the HM low because the model predictions are indeed quite different humans? Investigating the criteria Engaging, as shown in Table 11, the HM correlation is low (Krippendorff-a -0.02, p 0.25, and 7 0.24). When we visualize this data, we find that over 96% of the human labels have median of and only 3.3% (2 of 60) have different median value (H = 2), as shown in Figure 8. Obviously, there is considerable class imbalance, that likely lower Krippendorff-a. In the case of rank correlation, over 96% of H = 3, and of these roughly 30% have been assigned a score of 3 M = 3, lowering rank correlation metrics.
Given this imbalance, there are 2 scenarios (a) human annotation is reliable and the LLM prompts need to be optimized to improve the performance (b) Human annotation
26

---

Published as a conference paper at ICLR 2025
guidelines need to be revised. If assume that the human annotation is reliable, then the actionable step here to ensure that the LLM prompt is optimized such that the items are incorrectly labelled as 2, are correct to 3.
We can see similar pattern criteria MC (see Figure 9), NAT (see Figure 10). For criteria OV, in LJ labels are quite different to machine labels (see Figure 11).
0.0
96.7
H=1, JS:nan
H=2, JS:0.20
H=3, JS:0.24
Probabiitt
Probabiiity
Probabliity
26-
04-
Figure 8: Topical chat criteria Engaging: Over 96% of the samples are assigned median score of by humans fairly consistently, where around 90% of the human label bin H = 3 have the same value the median. Only (3.3% of 60) items are in bin H 2. There are no human labels for ordinal value 1, as seen in bins 1, 2 and 3. This shows how unbalanced the human labels are.
0.
100 0
H=1, JSmnan nan
H=2, JS:nn
H=3, JS:0.13
Probability
Probability
Probability
0.-
Figure 9: Topical chat criteria- Mainttans context. The human median is 3 for 100% of the samples. Hence, rank correlation is not applicable While in majority of the case (80% the samples), human median seems to match the machine median, k-a correlation HM is quite low at -0.07 as shown Table 11, pointing to class imbalance as less than < 10% samples are assigned as the median LJ rating.
0.0
100.0
H=1, JS:nan
H=2, JS:nan
H=3,
JS:0.24
Probability
biiiybiiity
Probability
04-
22-
Figure 10: Topical chat criteria Natural. The human median 3 for 100% of the samples. While here approximately 60% of the samples are also labelled 3 by LJ, the k-a correlation HM is sub- stantially low -0.20 shown Table 11.
27

---

Published as a conference paper at ICLR 2025
0.0
0.0
91.7
H=1, JS:nan
H=2, JS:nn
H=3, JS:nan
H=4, JS:0.34
H=5, JS:0.33
10-
Probability
0.5
Probability
Probability
Probability
Probability
0.6
Figure 11: Topical chat criteria - Overall. Here the human labels for bin H=4 vary, where only 40% of the labels match the median of 4, hence humans uncertain about this label assignment. Also note, this bin H=4 applies to 8.3% (5/60) of the samples. For the rest of 91% of the samples, human uncertainty is low, but LJ median labels M vary are quite different human median = 5.
28

---

Published as a conference paper at ICLR 2025
A.13 EXAMPLES COMPARING HUMANS AND MACHINE LABELS
Here are some examples from SNLI datasets, where humans have perfect agreement (5/5 votes for the majority label), but LLM (LLama3.1 8B) predicts different label as shown in Table 12. In Table 13, we show examples of where humans disagree on the labels, and where the majority label differs between human and machine.
Premise: Two women are embracing while holding to go packages. Hypothesis: The men fighting outside deli.
Human Labels: contradiction; contradiction; contradiction; contradiction; contradiction
Machine Labels: neutral: neutral; neutral; neutral; neutral
Premise: Two young children in blue jerseys, with the number and one with the number are standing on wooden steps a bathroom and washing their hands in sink.
Hypothesis: Two kids in jackets walk to school.
Human Labels: contradiction; contradiction; contradiction; contradiction; contradiction
Machine Labels: neutral; neutral; neutral; neutral; neutral
Premise: A brown dog and black dog in the edge of the ocean with a wave under them boats are the water in the background.
Hypothesis:
The dogs are swimming among the boats.
Human Labels: entailment; entailment; entailment; entailment; entailment
Machine Labels: neutral; neutral; neutral; neutral; neutral
Premise: A group of people prepare hot air balloons for takeoff.
Hypothesis: There are hot air balloons on the ground and air.
Human Labels: neutral; neutral; neutral; neutral; neutral
Machine Labels: entailment; entailment; entailment; entailment; entailment
Premise: A young child is jumping into the arms of woman wearing black swimming suit while in pool. Hypothesis: Mother catching her in pool.
Human Labels: neutral; neutral; neutral; neutral; neutral
Machine Labels: entailment; entailment; entailment; entailment; entailment
Premise: Three women in dress suits walk by a building.
Hypothesis: Three women are traveling by foot.
Human Labels: entailment; entailment; entailment; entailment; entailment
Machine Labels: neutral; neutral; neutral; neutral; neutral
Table 12: Examples in SNLI dataset where humans have perfect agreement, but the model produces different answer.
29

---

Published as a conference paper at ICLR 2025
Premise: Families waiting in line at an amusement park for their turn to ride.
Hypothesis: People are waiting in line at restaurant.
Human Labels: entailment; contradiction; entailment; contradiction; contradiction
Machine Labels: neutral; neutral; neutral; neutral; neutral
Premise: A woman is writing something post-it note which is hanging bulletin board with lot of other post-it
Hypothesis: The woman is talking on the phone.
Human Labels: contradiction; contradiction; contradiction; entailment; contradiction
Machine Labels: neutral; neutral; neutral; neutral; neutral
Premise: A young dark-hairee woman crouches on the banks of river while washing dishes. Hypothesis: A woman washes dishes in the river while camping.
Human Labels: neutral; contradiction; neutral; neutral; neutral
Machine Labels: entailment; entailment; entailment; entailment; entailment
Premise: A youth is kicking soccer hall in empty brick
Hypothesis: A funny human kicking.
Human Labels: neutral; neutral; neutral; contradiction; neutral
Machine Labels: entailment; entailment; entailment; entailment: entailment
Premise: A golfer has just finished swinging his club.
Hypothesis: human outside
Human Labels: entailment; entailment: entailment; entailment; contradiction
Machine Labels: neutral; neutral; neutral: neutral; neutral
Premise: A woman in black, from behind, sits next body of water. Hypothesis: A woman sits outside. Human Labels: entailment; contradiction; entailment; entailment; entailment Machine Labels: neutral; neutral; neutral; neutral; neutral
Table 13: Examples of humans annotations in SNLI datasets where humans disagree, and the ma- jority machine label is different answer.
30


Reviews:
[{"review_id": "32dVDHW4Kk", "reviewer": "Reviewer_Y99p", "summary": "The paper discusses the current landscape of using LLMs as judges for various tasks and presents compelling arguments for why existing correlation metrics might not take into account variations and uncertainty in human judgment.\n\nAfter the author's reply (which includes more datasets and clarifications about the metric), I believe that the paper makes good contributions and good cases of showing how to be careful when using LLMs as judges. I there recommend to accept the paper!", "questions": "How come the metric does not need a single value to approximate human labels but relies on a single \"human and machine labels are not treated interchangeably, as the items in a given bin are selected by the human median or majority value\"? This seems contradictory to me.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The paper is well structured and presented and is very clear and easy to follow and read.  \n2) The paper clearly shows the issue related to relying on a high correlation between human and machine-generated outputs, giving cases where the uncertainty in human annotations is high; these correlations seem to be high, but that could also be the case even when the labeling is random. \n3) The study proposed a metric, namely binned JSD, to account for variations and uncertainty in human judgment.", "weaknesses": "1) It is not clear to me how this new metric handles the issues raised with traditional metrics. Could the author clarify and show cases of how JSD improves the analysis of LLMs as judges when compared to human judgment? It seems like a promising direction, but I am not convinced due to the limited number of datasets and support of the authors' claim. \n\n2) The type of human annotation is limited. Although the paper presented very well the type of collected human annotations, I limited these in their evaluations. Given the space and the breed of the paper, I suggest adding a few more datasets (Please check paper [1] for some guidance on what dataset to choose.)\n\n3) The study considered four datasets with variant tasks and annotations that have sufficient human annotators. I think the number of annotations was well considered, but the number of the dataset could be improved, and metal analysis could have been better presented instead of large tables per dataset. \n\n\n[1] Bavaresco, Anna, et al. \"Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks.\" arXiv preprint arXiv:2406.18403 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AayZw0xTCS", "reviewer": "Reviewer_ApGo", "summary": "The paper describes the analysis of different measures in  evaluating  LLM responses.\nA measure, specifically a binned Jensen- Shannon divergence is proposed.\n\nThis measure for ordinal perception data is justified by the author since the evaluation does not need a single gold label and the human and the machine are not interchangeable. This last condition breaks a necessary condition for the Krippendorff-alpha coefficient.\n\nA distinction is done among Nominal values, ordinal values and continuous values. The Questions are not equally proposed for the three types of data, raising some difficulty in reading the paper.\n\nRQ1:How does uncertainty in human labels impact correlation metrics when\nwe measure the efficacy of automatic evaluation methods? (Sec. 3.1)\n\nThe authors state that the uncertainty in human labels is high the human-machine majority labels are similar. The meaning appears to be that if there is no concordance among labeller, in this case the LLM judge is ok. It is the LLM-judge just adding noise to the labelling process?\n\n\nRQ2: How can we measure human-to-machine(HM) agreement that accounts for human uncertainty as a result of variation in human perception?\nHuman to machine agreement is a measure of uncertainty in human perception. For this question, the comparison with different agreement percentage is tested. For this task is proposed the binned Jensed inequality.\n\n\nRQ3: How can we visualize the underlying data to draw meaningful insights when we compare the results from automatic and human evaluations?\n\nThe authors compare ordinal and perceptual based ratings between human and machines", "questions": "Could the author provide some detail in the selection of different thresholds across the tables? It would be useful to clarify whether these thresholds are dataset-specific or if there's a general principle behind the thresholds selection.\n\nIn the paper, different metrics  are used and the human and machine labels are compared with average, with median or with majority labels. Are all these comparisons needed? Do they capture multiple aspects of the outputs?\n\nIs the binned JSD the best metric for the proposed experiments? Is it possible to calculate this metric, or its adaptation,  for all the experiments proposed in the paper?\n\nThe bins are used to mimic human perception. Beyond the aggregation of perception, can they capture variation in human perception?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The topic is new, and since human-labelled data are difficult to retrieve, an evaluation of machine-labelled data is very interesting.\n\nThe tests have been done on different LLMs \n\nMultiple tests have been performed.", "weaknesses": "The work's presentation is not clear. The research questions help to  interpret the experiments but not all the results are clear.\n\nSome terms, like H^W R^W (^ indicates apex), are defined in the caption of caption 1. Probably they should be defined in the text and used in the table.\n\nIt is unclear how the partitions are decided. In the experiments:in table 1 the thresholds are  0, 0.8, 1; in table 2 the threhsolds are 0.6 and 1; in table 3 are 0.6, 0.8, 1.0. It is not clear if the thresholds are experiment dependent or there is a rationale behind the threshold selection.\n\nThe random classifier test, reported in table 1 should be better described. The table reports unique =2 or unique =3 with a percentage. Unique term is not present in the description and should be explained for the clear presentation of the experiment. If the MNLI and SNLi\ndataset have only a limited set of labels it should be specified in the description.\n\nIn Table 2 the terms H^mu M^mu are not specified. \n\n\nIn Figure 3 are shown the human perception vs the machine labels binned by human median rating. The JS value is reported.\nThe highest values of JS are for \\bar{H}=1 and  \\bar{H}=5. Looking at the histograms, the histograms of human and machine with \\bar{H}=3 (and  in some measure \\bar{H}=4) are very similar, but the JS values are sensibly lower.\nThe authors state that humans tend to be more certain when they assign extreme rating and the machine rarely provide extreme values. \nCould  the explanation of the experiment take into account also this aspect? If there is a different interpretation of this discrepancy ( similar histograms but lower JS value) it would be useful to make this point more evident.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qjAuKgVL8R", "reviewer": "Reviewer_b4vj", "summary": "This paper explores how current methods for evaluating generative models often fall short by relying too heavily on correlation metrics like Krippendorff’s α and Randolph’s κ. These metrics, while common, can mask important nuances in human judgment, especially in cases where human responses vary widely. The authors show that when there’s a high degree of variation in human evaluations, machine judgments might seem to align well, but as human consensus strengthens, this alignment breaks down, revealing gaps in machine understanding. To address these issues, the paper proposes a more robust evaluation framework that includes stratifying results by the level of human agreement and introducing a new metric, the binned Jensen-Shannon Divergence, to better capture perception-based evaluations. Additionally, the authors suggest using visual tools like perception charts to more clearly illustrate where machine judgments align or diverge from human benchmarks. By combining multiple metrics and visualization methods, this approach aims to provide a more accurate and comprehensive understanding of automated evaluations, especially in areas where human judgments are inherently uncertain.", "questions": "1. Could you provide more specific implementation guidance or pseudocode for this metric, perhaps in an appendix? This would help ensure reproducibility and clarity for those looking to apply it.\n2. The paper attributes label uncertainty to genuine perceptual differences, but could the authors discuss other potential sources, such as cultural or contextual biases among annotators? How might such biases affect the evaluation results, and could additional stratification methods help account for them? This would make the framework more applicable across diverse datasets and ensure its robustness in various contexts.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper makes a valuable contribution by tackling the often-overlooked role of human uncertainty in evaluating generative models. It’s original in its approach, introducing the binned Jensen-Shannon Divergence metric to better capture the nuances of human perception and using tools like perception charts to bring new depth to evaluations. The quality of the work shows through in its thorough methodology, with experiments across multiple datasets that lend strong support to the findings. The paper is also clear, with a well-structured flow and visuals that help explain complex ideas. Most importantly, the paper has real significance: its framework could reshape how we evaluate generative models, especially in areas where human judgment isn’t always straightforward. Finally, the paper’s significance lies in its potential to reshape evaluation practices for generative models, especially in applications where human judgment is inherently subjective, such as content generation, recommendation systems, and interactive AI. By emphasizing the role of human uncertainty and offering practical tools to account for it, this work highlights a crucial aspect often ignored in model evaluation. This framework could lead to more accurate and context-sensitive evaluations, particularly for models that interact with or respond to human preferences.\nThis paper offers a well-supported framework that deepens our understanding of human-machine evaluations, bridging the gap between traditional metrics and the complexities of human perception. Its contributions could have a lasting impact, inspiring future research and improving evaluation standards across the field of generative modeling.", "weaknesses": "There are some potential for improvement in paper such as expanding on the technical implementation of the binned Jensen-Shannon Divergence metric would make it more accessible to practitioners, potentially by providing step-by-step instructions or pseudocode. Additionally, testing the framework on a broader range of generative models beyond text (such as image or audio) would demonstrate its versatility. The perception charts are helpful, but they primarily show aggregate trends, which may obscure individual item-level discrepancies; adding item-level visualizations or error bars could improve clarity. To connect more concretely with real-world applications, the paper could benefit from case studies or examples where the framework enhances specific generative tasks, such as in recommender systems. Moreover, a side-by-side comparison with existing metrics like Krippendorff’s α would better illustrate the added value of the proposed metric. Including confidence intervals or statistical significance testing could also add rigor to the findings. Finally, considering potential biases in human label uncertainty, such as cultural or contextual differences among annotators, would make the framework more robust across diverse datasets. Together, these enhancements would increase the framework’s clarity, practical utility, and adoption potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VD3bgdBGd0", "reviewer": "Reviewer_RFAo", "summary": "This paper examines how human uncertainty affects evaluating generative models, noting that standard metrics like Krippendorff’s α may misrepresent machine accuracy when human judgments vary. The authors propose three main contributions: stratifying evaluation results by human uncertainty levels, introducing binned Jensen-Shannon Divergence (JSb) to better measure alignment with human perception, and creating perception charts to visualize evaluation performance more effectively. These tools aim to provide a clearer, more accurate picture of machine evaluation performance amidst human variability.", "questions": "- How do you decide what's \"high\" or \"low\" uncertainty in your stratification, and did you try other thresholds?\n- Why did you choose Jensen-Shannon Divergence for human perception, and can you show this is better than existing metrics?\n- How did you adapt Krippendorff’s α and similar metrics to account for systematic machine errors, not just random ones?\n- Why use previous prompts without optimizing them for each model, wouldn't  this affect fairness in comparisons?\n- Can you explain how you created the synthetic data for high uncertainty and whether it really reflects real-world data?\n- Could you provide concrete examples where machine evaluations diverge significantly from human judgments?\n- How would these findings help practitioners improve real-world model evaluations?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: This paper takes an original approach by addressing human uncertainty in generative model evaluation. Introducing stratified results by human variability and the new JSb metric for perception-based tasks adds fresh methods to handle subjectivity in evaluations. The perception charts also offer an innovative way to visualize nuanced performance differences.\n\nQuality: The work is methodologically sound, with comprehensive experiments across datasets like SummEval and MNLI. The use of both real and synthetic data strengthens the empirical basis, showcasing the impact of human judgment noise on evaluation reliability.\n\nClarity: The paper is well-organized, clearly defining key concepts such as HH vs. HM correlation. Explanations of JSb and perception charts are straightforward, helping readers understand the new evaluation tools effectively.\n\nSignificance: This work fills an important gap by addressing subjective variability in human evaluations. Its proposed methods (if widely adopted) could lead to more accurate and relevant model evaluations, especially in perception-driven tasks across AI.", "weaknesses": "- The choice to stratify results based on “high” and “low” human uncertainty needs clearer justification. A discussion or empirical test on how these thresholds were set would make the stratification process more robust and reproducible.\n\n- The introduction of Jensen-Shannon Divergence as a measure of perception-based tasks is promising, but its effectiveness is not thoroughly proven. Including a comparison with other potential metrics, such as Earth Mover’s Distance or Wasserstein Distance, would better validate the claim that JSb captures human perception more accurately.\n\n-While reusing prior prompts is convenient, this may introduce biases or inconsistencies across models. Optimizing prompts specifically for each model would yield more accurate comparisons, especially given the importance of prompt sensitivity in LLM performance. Adding prompt-tuning experiments for each model could further solidify the findings.\n\n-The synthetic data used to simulate high uncertainty scenarios lacks details on its generation process. More transparency on how closely this data reflects real-world scenarios, including its validation process, would help verify the relevance of the findings. \n\n- The shifts in ∆ (difference between HH and HM correlations) across different uncertainty levels are intriguing but underexplored. An in-depth analysis of these shifts—perhaps with concrete examples of where machines diverge from human judgment—would help understand the implications of these results more clearly. Visual examples showing alignment and divergence between human and machine judgments could greatly enhance interpretability.\n\n- The experiments rely primarily on four datasets, which, while varied, still represent a narrow slice of possible applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)