Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge

## Abstract

Abstract The effectiveness of automatic evaluation of generative models is typically measured by comparing it to human evaluation using correlation metrics.
However, metrics like Krippendorff’s α 𝛼 \alpha italic_α and Randolph’s κ 𝜅 \kappa italic_κ , originally designed to measure the reliability of human labeling, make assumptions about human behavior and the labeling process.
In this paper, we show how relying on a single aggregate correlation score can obscure fundamental differences between human behavior and automatic evaluation methods, including LLM-as-a-Judge.
Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human labels (gathered during human evaluation) is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to human-to-human (HH) correlation.
This can create the misleading impression that automatic evaluation is accurate enough to approximate the human majority label.
However, as the proportion of samples with consistent human labels increases, the correlation between machine labels and human majority labels declines, falling below HH correlation.
Based on these findings, we first propose stratifying results by human label uncertainty to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric - binned Jensen-Shannon Divergence for perception for such scenarios to better measure the effectiveness of automatic evaluations. Third, we present visualization techniques – perception charts , to compare the strengths and limitations of automatic evaluation and to contextualize correlation measures appropriately.

## 1 Introduction

Despite the increasing importance of automatic evaluation for generative large language models (LLMs), measuring and interpreting the effectiveness of these methods remains challenging. Automatic evaluation methods include $N$-gram-based methods such as Rouge (Lin, ) and BERTScore (Zhang* et al., ), task-specific model-based methods (TM) such as AlignScore (Zha et al., ) and Prometheus (Kim et al., ), and LLM-as-a-Judge-based methods (LJ) where a general purpose LLM, using prompts, is instructed to perform evaluation.
The use of automatic evaluation is tied to the key question – How can the efficacy of automatic evaluation methods be measured in comparison to human evaluation? A widely accepted practice is to use agreement or rank correlation metrics such as Krippendorff’s-$\alpha$ (Krippendorff, ), Cohen’s-$\kappa$ (Cohen, ; McHugh, ), Spearman’s $\rho$, and Kendall’s $\tau$-b (Fabbri et al., ; Deutsch et al., ; Chiang & Lee, ; Liu et al., ), where a higher correlation between machine labels (generated by automatic evaluation methods) and human labels (gathered during human evaluation) implies a more effective automatic evaluation. Variation or uncertainty in human labels (Tversky & Kahneman, ) is almost impossible to avoid when humans are involved (Kahneman et al., ). In this paper, we first highlight how comparing the overall correlation score between human labels and machine labels can obscure key discrepancies in machine generated labels. Therefore, we propose that measurements be stratified by label uncertainty.

Secondly, this problem also leads to a follow-on question about how to interpret uncertainty. Not all types of uncertainties are equal (Kahneman & Tversky, ), as they can range from random human errors, ambiguity in tasks, and nature of perception-based evaluation (Elangovan et al., ), where some of these variations are inherent to the task while others indicate poor reliability of human labels. For instance, when evaluating aspects such as “How readable is the given content?”, the answer relies on human perception and is likely to vary from person to person. In contrast, evaluating a natural language inference (NLI) task, such as “Can you infer the given sentence from a given text?”, is less likely to show variations. Acceptable variation in labels, therefore, depends on the task, as variations or uncertainties in human labels can sometimes be a feature rather than a bug in human evaluation. Thus, we explore how to effectively measure and interpret reliability in the context of human uncertainties.
Hence, we ask the following questions in this paper

- •
Research Question 1 - How does uncertainty in human labels impact correlation metrics when we measure the efficacy of automatic evaluation methods? (Sec. 3.1)
- •
Research Question 2 — How can we measure human-to-machine (HM) agreement that accounts for human uncertainty as a result of variation in human perception? (Sec. 3.2)
- •
Research Question 3 — How can we visualize the underlying data to draw meaningful insights when we compare the results from automatic and human evaluations? (Sec. 3.3)

## 2 Background on correlation measures

### 2.1 Correlation measurements and assumptions

Correlation measures have traditionally been used to measure the inter-rater agreement (IRA) of human-generated labels, where there is no ground truth available, to measure human-to-human (HH) correlation. There are 2 broad categories of correlation measures, (a) non-chance-adjusted indices and (b) chance-adjusted metrics (Xinshu Zhao & Deng, ). Chance-adjusted measures take into account a scenario where raters can choose the same label by chance even when they do not know the correct answer. Chance-adjusted measures, such as Krippendorff’s-$\alpha$ are generally more popular IRA measures than non-chance-adjusted metrics such as percentage agreement. The key difference between various chance-adjusted IRA measures is how chance agreement is computed (Meyer et al., ; Xinshu Zhao & Deng, ). For instance, Randolph’s-$\kappa$ (Randolph, ) assumes that all labels are equally likely and assigns a probability of $\nicefrac{{1}}{{k}}$ for chance agreement, where $k$ is the number of possible label choices. On the other hand, Fleiss’s $\kappa$ (Fleiss, ) and Krippendorff’s $\alpha$ calculate chance agreement empirically, based on the actual human labels of the study. These measurements can be useful in cases where certain labels (e.g., 3 on a Likert scale of 1-5) are more likely to be chosen when humans are uncertain. A high proportion of such labels, even if consistent, may indicate unreliability in the labeling process, particularly when the minority labels show lower agreement. Therefore, chance-adjusted correlation measures rely on key assumptions about the types of errors humans are likely to make.
Furthermore, Krippendorff’s-$\alpha$, specifies that the agreement coefficient is an indicator of reliability if and only if (1) the raters duplicate the process of rating, i.e., when two independent raters use the same guidelines or instructions and use the same units, (2) raters must be treated as interchangeable, and (3) correct answers are not known (Krippendorff, ). Human labels and machine labels are clearly not interchangeable even when using current state-of-the-art LLMs to generate machine labels, as humans and LLMs are fundamentally different.

### 2.2 Differences between correlation measures and raw accuracy

The assumptions above, especially the assumption that raters must be treated as interchangeable, raise a fundamental question as to why even use correlation measures to compare machine and human performance, instead of measures such as accuracy and F1 using human labels as ground truth? Firstly, one of the main challenges with using scores like accuracy or F1 is that human labels tend to be noisy regardless of the task or the dataset, where some label choices are more acceptable than others, and measures such as F1 or accuracy do not account for variations in human labels and hence correlation measures appear to be a better measure to compare human vs. machine performance. However, as mentioned previously, many of the common chance-adjusted correlation measures aim to account for random chance agreement in HH agreement, whereas the errors make by machines tend to be more systematic. A key characteristic of random errors is that they are non-reproducible which result in variations when repeated, resulting in low IRA, which in-turn may point to potential problems in the labelling process or complexities in the task. In contrast, systematic errors are consistent, and IRA measures are not designed to detect them. Secondly, as a direct consequence, IRA metrics assume that agreement is good and variance is bad, which implies that there is a single correct answer – a situation that may not hold for ambiguous items or those that gauge user perception.

### 2.3 Types of raw data collected during evaluation

The type of the data collected, such as nominal, ordinal, interval, ratio, discrete or continuous data, dictate how such data can be aggregated and interpreted. Nominal and ordinal measures can be considered as qualitative measurements, while continuous values can be interpreted as quantitative (Mishra et al., ). The most common types of values collected during automatic or human evaluation are:

Nominal values: Nominal data, such as gender, have no natural ranking or ordering. Such data are primarily used in human evaluation tasks such as fact checking, where the aim is to verify whether a given text can be inferred from reference content (Liu et al., ) and usually is modeled as a binary classification task. Preference judgments such as “Is the output of model-A or Model-B better?” are also common examples of nominal values collected during human evaluation.
IRA measures for nominal values include percentage agreement (McHugh, ), Randolph’s-$\kappa$, Fleiss’s-$\kappa$, Krippendorff’s-$\alpha$ and Cohen’s-$\kappa$.

Ordinal values: Ordinal values have an implicit order, e.g., ratings of quality (very good, good, fair, poor, very poor) (Mishra et al., ). The most common use of ordinal values in human evaluation are through Likert scales. Likert scales are rating systems of attitude scale to measure the degree to which a person agrees/disagrees with a given statement (Taherdoost, ) and commonly to gauge user perception on qualitative aspects such as readability, coherence, and fluency (van der Lee et al., ).
IRA measures for ordinal values should ideally account for the fact that ratings that are closer to one another (1 vs. 2) for a given item indicate a higher agreement than ratings that are further apart (1 vs. 4). Such IRA measures include Krippendorff’s-$\alpha$ for ordinal values, where disagreements are weighted by distance between the ratings. Rank correlations such as Kendall’s-$\tau$ and Spearman’s-$\rho$ are also widely used to measure item-level correlation between machines and humans for ordinal values. However, there’s a caveat on using these measures: their ability to handle a relatively large proportion of ties is widely debated, even when measuring system correlations that involve a much smaller proportion of ties (Deutsch et al., ).
When the number of items to rank is $n$, the number of ordinal values to assign is $r$, and when $r<<n$, it results in many ties or rank collision. For instance, when assigning a Likert scale of 1-5 to 100 items, at least 20% of the items will have the same rank. While ordinal values preserve order, the objective of using ordinal values is not to rank the item in relation to other items, therefore the proportion of ties affects the interpretability and reliability of the rank correlation scores.

Continuous values: Continuous values are typically obtained when using automatic evaluation methods such as Rouge (Lin, ) and BERTScore (Zhang* et al., ).

## 3 Analysis Settings and Results

Datasets: We use datasets that have multiple human annotations used in typical evaluation tasks for (a) Likert-based ordinal qualitative evaluation, (b) NLI datasets for fact or consistency checking, and (c) preference datasets for understanding human preferences as follows:
1. SummEval (Fabbri et al., ) is human evaluation of summary quality using four criteria – fluency, coherence, consistency, and relevance, with Likert scores assigned from 1 to 5. This dataset has 1600 samples, where each item is labeled by 3 expert annotators.
2. SNLI (Bowman et al., ) is an NLI inference task dataset with 10K samples and 3 labels – entailment, neutral and contradiction. Each item is assigned 5 labels by human annotators.
3. MNLI (Williams et al., ) is also an NLI inference task dataset with 10K samples and 3 labels – entailment, neutral and contradiction. Each item is assigned 5 labels by human annotators. It contains two splits – matched and mismatched.
4. MT-Bench (Zheng et al., ) is a preference dataset which compares human preferences of several model outputs on open-ended questions using
multi-turn conversations. From this dataset, we only use samples that have at least 3 human ratings.

Models: For SummEval dataset, we reuse the original G-Eval results (Liu et al., ) which rates the quality of summaries on a scale of 1-5. In addition, we also use Claude Sonnet 3 and Mistral Large v2 on SummEval, re-using the prompts from G-Eval as detailed in Appendix A.2. For NLI and MNLI datasets, we use Llama 3.1 8B, Mistral Large v2 and Sonnet 3, reusing the prompts from Liu et al. () as detailed in Appendix A.1. For MTBench, we rely on the existing results on preference data on MT-Bench and GPT-4 from Zheng et al. (). We also adapt the same prompts (see Appendix A.3) for Sonnet as LJ for MTBench. In order to compute machine-to-machine (MM) correlation, we set the sampling temperature to 1.0 to maximize the variation across runs, and set top p = 0.8. We sample 20 annotations per example. NOTE: We have predominately reused the prompts from existing works primarily relying on GPT-4 as LJ, we have NOT attempted to optimize the prompts per LLM except for minor format changes. Since models are known to be highly sensitive to prompts, our experiments in this paper are not meant to rank the underlying LJ, but to study the impact of human uncertainty regardless of the LJ.

Correlation measures: For nominal data, we report scores on Krippendorff’s-$\alpha$, Fleiss-$\kappa$, Randolph’s-$\kappa$ and percentage agreement (McHugh, ) also detailed in Appendix A.4. For MT-Bench preference data, since the number of human raters vary per tuple $\langle$conversation, turn, model pair$\rangle$ we use Krippendorff-$\alpha$ and percentage agreement to compute HH correlation as it can handle varying number of raters per item unlike Fleiss-$\kappa$ or Randolph’s-$\kappa$. For ordinal data, we report Krippendorff-$\alpha$ for ordinal data and commonly used rank correlation measures (despite the caveat highlighted previously as many papers have used them to measure for item-level correlation) such as Spearman’s-$\rho$ and Kendall’s-$\tau$.

### 3.1 RQ1: How does uncertainty in human labels impact correlation metrics when we measure the efficacy of automatic evaluation methods?

To study the impact of human uncertainty, we stratify samples and then compare HH and HM correlation measurements for each group. We primarily stratify by percentage agreement, which is the proportion of labels that reflect the majority (for nominal data) or the median label (for ordinal data) among all human labels assigned to an item. For example, if 3 out of 5 people assign the same label, the percentage agreement is 60%. We also stratify samples by the number of unique human labels to ensure that our findings are consistent regardless of the stratification method.

At surface level, HM correlation seems to improve with human certainty, as shown in Table 3.1, (column $H^{w}M^{w}$). On closer inspection, another trend emerges where in any given stratified group when the noisy or uncertain samples increases (as measured by low HH correlation), the HM correlation seems to outperform HH correlation as highlighted.
However, as the proportion of samples with noisy labels decreases and HH correlation approaches near-perfect agreement, HH correlation is much higher than HM as highlighted
. Thus, the illusion that machines (specifically LJs) approximate majority human behavior is largely due to the presence of noisy labels in the data. As a corollary, if LJs were truly approximating the human majority, the $\Delta$ would approach 0 under perfect HH correlation – which is clearly not the case.
We also replicated this behavior on SummEval dataset which relies on ordinal labels in Table 2, using correlation metrics such as Krippendorff’s-$\alpha$ for ordinal data, Kendall’s-$\tau$ and Spearman’s-$\rho$. The same patterns are observed on preference data in Table 3.

**Table 1: Compare HM and HH correlation measurements on NLI datasets using Sonnet to generate machine labels. The overall correlation scores (All) indicate that the HM correlation is similar to HH. However, when stratifying by percentage agreement (PA) or human unique labels, the difference ($\Delta=HH-\text{H}^{w}\text{M}^{w}$) is highest when the all human labels are the same (PA=1), and $\text{H}^{w}\text{M}^{w}$ outperforms HH when certainty is the lowest. $H^{w}$ and $M^{w}$ are human and machine majority label.**
|  | Partition by<br>human labels<br>(% samples) | Krippendorff’s-$\alpha$ | % Agreement | Fleiss-$\kappa$ | Randoph’s-$\kappa$ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | HH | MM | $\text{H}^{w}\text{M}^{w}$ | $\Delta$ | HH | $\text{H}^{w}\text{M}^{w}$ | $\Delta$ | HH | $\text{H}^{w}\text{M}^{w}$ | $\Delta$ | HH | $\text{H}^{w}\text{M}^{w}$ | $\Delta$ |  |
| MNLI (matched) | All (100%) | 0.70 | 0.96 | 0.72 | -0.02 | 0.88 | 0.82 | 0.07 | 0.70 | 0.72 | -0.02 | 0.70 | 0.72 | -0.03 |
| $\text{PA}=1$ (58%) | 1.00 | 0.97 | 0.83 | 0.17 | 1.00 | 0.89 | 0.11 | 1.00 | 0.83 | 0.17 | 1.00 | 0.83 | 0.17 |  |
| $0.8\leq\text{PA}<1$ (25%) | 0.39 | 0.95 | 0.66 | -0.27 | 0.80 | 0.78 | 0.02 | 0.39 | 0.66 | -0.27 | 0.40 | 0.67 | -0.27 |  |
| $0\leq\text{PA}<0.8$ (16%) | 0.04 | 0.93 | 0.37 | -0.33 | 0.60 | 0.61 | -0.01 | 0.04 | 0.37 | -0.33 | 0.07 | 0.42 | -0.34 |  |
| $\text{unique}=2$ (38%) | 0.28 | 0.95 | 0.56 | -0.28 | 0.73 | 0.72 | 0.01 | 0.28 | 0.56 | -0.28 | 0.30 | 0.58 | -0.28 |  |
| $\text{unique}=3$ (3%) | -0.06 | 0.91 | 0.39 | -0.45 | 0.60 | 0.63 | -0.03 | -0.06 | 0.39 | -0.45 | -0.05 | 0.44 | -0.49 |  |
| MNLI (mismatched) | All (100%) | 0.72 | 0.97 | 0.74 | -0.02 | 0.89 | 0.83 | 0.07 | 0.72 | 0.74 | -0.02 | 0.72 | 0.74 | -0.02 |
| $\text{PA}=1$ (60%) | 1.00 | 0.98 | 0.85 | 0.15 | 1.00 | 0.90 | 0.10 | 1.00 | 0.85 | 0.15 | 1.00 | 0.85 | 0.15 |  |
| $0.8\leq\text{PA}<1$ (24%) | 0.39 | 0.95 | 0.65 | -0.26 | 0.80 | 0.77 | 0.03 | 0.39 | 0.65 | -0.26 | 0.40 | 0.66 | -0.26 |  |
| $0\leq\text{PA}<0.8$ (14%) | 0.04 | 0.94 | 0.39 | -0.35 | 0.60 | 0.62 | -0.02 | 0.04 | 0.39 | -0.35 | 0.07 | 0.43 | -0.36 |  |
| $\text{unique}=2$ (36%) | 0.28 | 0.94 | 0.56 | -0.28 | 0.73 | 0.72 | 0.01 | 0.28 | 0.56 | -0.28 | 0.30 | 0.58 | -0.28 |  |
| $\text{unique}=3$ (2%) | -0.06 | 0.94 | 0.41 | -0.46 | 0.60 | 0.64 | -0.04 | -0.06 | 0.41 | -0.46 | -0.05 | 0.45 | -0.50 |  |
| SNLI | All (100%) | 0.68 | 0.97 | 0.63 | 0.05 | 0.88 | 0.76 | 0.12 | 0.68 | 0.63 | 0.05 | 0.68 | 0.64 | 0.05 |
| $\text{PA}=1$ (55%) | 1.00 | 0.97 | 0.69 | 0.31 | 1.00 | 0.80 | 0.20 | 1.00 | 0.69 | 0.31 | 1.00 | 0.69 | 0.31 |  |
| $0.8\leq\text{PA}<1$ (28%) | 0.39 | 0.97 | 0.62 | -0.24 | 0.80 | 0.76 | 0.04 | 0.39 | 0.62 | -0.23 | 0.40 | 0.64 | -0.24 |  |
| $0\leq\text{PA}<0.8$ (15%) | 0.03 | 0.95 | 0.37 | -0.34 | 0.60 | 0.61 | -0.01 | 0.03 | 0.37 | -0.34 | 0.07 | 0.42 | -0.35 |  |
| $\text{unique}=2$ (41%) | 0.29 | 0.96 | 0.54 | -0.25 | 0.74 | 0.71 | 0.03 | 0.29 | 0.54 | -0.25 | 0.31 | 0.57 | -0.26 |  |
| $\text{unique}=3$ (3%) | -0.05 | 0.94 | 0.47 | -0.53 | 0.60 | 0.66 | -0.06 | -0.05 | 0.47 | -0.53 | -0.05 | 0.50 | -0.55 |  |

### 3.2 RQ2: How can we measure human-machine agreement that account for human uncertainty as a result of variation in human perception?

Judging the quality of a free-form model generated output, such as a summary, largely depends on human perception. As a result, the human ratings can vary depending on a wide range of factors including what one likes to their emotional state (Elangovan et al., ).
Perception-based ordinal-value evaluation schemes, such as Likert scores, pose 2 main challenges. (1) It is an attitude-based scale, and since human attitudes vary, metrics like Krippendorff’s $\alpha$, which penalize variation, can produce scores lower than the minimum acceptable value (Amidei et al., ). (2) Rank correlation measures such as Kendall-$\tau$ or Spearman’s-$\rho$ are not entirely appropriate given the large number of ties when comparing median scores, as previously discussed in Sec. 2.3 and in the works by Deutsch et al. (). Furthermore, whether to treat Likert-data as ordinal or interval data dictates the aggregation method – median or mean, is also debated (Joshi et al., ). The argument against using Likert-data as interval data is that the points on the scale may not be equidistant, e.g., the distance between pair $\langle$neutral, agree$\rangle$ may not be the same as the distance between $\langle$agree, strongly agree$\rangle$. We report Spearman’s-$\rho$ for both to illustrate the difference in Table 2. While in most cases the two sets of scores look largely the same, there are instances where the numbers are vastly different. For instance, Table 2 Spearman’s-$\rho$ median vs. mean
, median comparison yields a correlation of 0.49 (low correlation) while comparing the mean, results in 0.60 (indicating moderate correlation) (Mukaka, ), provoking the question which number is more representative of the underlying data? Regardless of whether we choose to use Krippendorff-$\alpha$ or a rank correlation metric, one aspect that both assume is that there is a single rating (e.g., median value) that appropriately represents the underlying choice a human rater makes. This assumption may not be correct, especially when the task is inherently subjective, especially when the number of judgments collected per item is relatively low, e.g., 3 ratings per item. This shows a clear gap in existing metrics to compare human and machine performance, where intra-human ratings naturally vary as a a result of subjectivity inherent to the task.

Hence, we propose an additional metric to supplement existing measures, the use of binned Jensen-Shannon divergence (JSD) for perception (Lin, ) to quantify how closely automatic methods mimic human judgment. To the best of our knowledge, our approach of using JSD has not been used as a correlation measure to account for human perception, without relying on a single gold label. The closest work to ours is the use of cumulative JSD to predict the aesthetic score distribution, instead of just the average rating, to represent image quality (Jin et al., ).

Formally (source code in Appendix A.7), let $H_{i}$ and $M_{i}$ represent the set of human and machine ratings respectively assigned to the item $x_{i}$, and $n$ be the total number of the items and $R$ be the set of possible ratings (e.g., $1$ to $5$ for ordinal ratings, or {$A$, $B$, $Tie$} for nominal model preferences). To compute the distance between human and machine predictions, we can define the probability mass function (PMF) for a given rating $c$ as follows:

$$ $P^{h}_{r}(c)=\frac{\sum_{i\in B_{r}}\sum_{l\in H_{i}}\mathbbm{1}_{\{l=c\}}}{ \sum_{i\in B_{r}}|H_{i}|}\;\text{(Human),}\qquad P^{m}_{r}(c)=\frac{\sum_{i\in B _{r}}\sum_{l\in M_{i}}\mathbbm{1}_{\{l=c\}}}{\sum_{i\in B_{r}}|M_{i}|}\;\text{ (Machine)}\vspace{-2pt}$ (1) $$

where for ordinal values, bin $\mathbf{B}_{r}\coloneqq\{i\in\{1,..n\}|\text{median}(H_{i})=r\}$ contains the set of samples with a median human rating of $r$. For nominal perception-based preference labels, bin $\mathbf{B}_{r}\coloneqq\{i\in\{1,..n\}|\text{mostfrequent}(H_{i})=r\}$ contains the set of samples with the most frequent human rating of $r$.

From Equation 1, we can define the binned JSD ($JS_{b}$) as follows:

$$ $JS_{b}=\sum_{r\in R}\frac{|B_{r}|}{n}JS(P^{h}_{r}||P^{m}_{r})\coloneqq\sum_{r \in R}\frac{|B_{r}|}{n}\left(\frac{1}{2}KL(P^{h}_{r}||Q_{r})+\frac{1}{2}KL(P^{ m}_{r}||Q_{r})\right),\vspace{-2pt}$ (2) $$

where $KL(P^{h}_{r}||Q_{r})=\sum_{c\in R}P^{h}_{r}(c)\log_{2}(\frac{P^{h}_{r}(c)}{Q_{
r}(c)})$ is the Kullback-Leibler divergence (Kullback & Leibler, ) and $Q_{r}(c)=\left(P^{h}_{r}(c)+P^{m}_{r}(c)\right)/2$ for $\forall c\in R$ is the average between human and machine PMFs, based on the definition of Jensen-Shannon distance ($JS$). Since JSD is a distance-based measure, lower scores are better because they indicate that the human and machine judgments are similar.

The binned JSD for ordinal perception data has 2 main advantages, (1) it measures how closely machine generated labels mimic human perception without the need for a single gold label (2) human and machine labels are not treated interchangeably, as the items in a given bin are selected by the human median or majority value. For HM correlation, the proposed approach can be applied at item-level when the number of labels collected per item is high enough to form a distribution, however, for judging model generated output even obtaining 3 human labels per item can be expensive. Hence, we suggest computing the distribution distance per bin.
The main shortcoming of this approach is that some key item level discrepancies between machines and humans ratings might not be clearly surfaced. For instance, for a given item, if all the humans assign a rating of 1 and machines assign say 4, at an aggregate level this pattern may not be obvious if this pattern is not frequent. Hence, we also recommend reporting correlations stratified by uncertainty in human labels, e.g., for items that have high agreement labels, such as perfect human agreement as shown in Table 2.

### 3.3 RQ3: How can we visualize the underlying data to draw meaningful insights when we compare automatic labels and human labels?

#### 3.3.1 Perception-based ordinal rating visualization

We propose the “Ordinal HM perception comparison chart” as shown in Fig. 3 to compare humans’ perception of the content vs. how machines rate them. Here, we bin the samples by the human median rating and plot the corresponding distribution of human and machine ratings. For instance, say for a grading task using label values between 1-5, each item is rated 3 times by humans. For a sample set of 10 items, the total number of judgments would be 30 (10 items * 3 ratings per item). Let’s say 7 items get a median rating of 4, these 7 items would fall in bin $\overline{H}=4$. We then plot the human label distribution for that bin to include all the 21 judgments (7 items * 3 ratings per item = 21), we also plot all the machine labels for the corresponding bin. We repeat this for each of the label from 1-5 as shown in Fig. 3. This allows us to visualize the variation in human perception, thus including all the ratings rather than a single gold rating, and compare how the corresponding machine labels vary.

Figure: Figure 3: Ordinal HM perception comparison chart: Visualization of human perception vs. machine labels binned by human median rating (LJ Mistral on Coherence): Dial at the top shows the percentage of samples that fall into the bin. Middle scores (median 2-4) have higher human uncertainty, where only 60% (less than 2 out of 3) of the human labels follow the median. The extremes scores (1, 5) have less uncertainty. While Mistral seems to mimic human perception when the human median is 3, it is also biased towards the rating 3 when the human ratings are between 2-4.
Refer to caption: x6.png

Correlation numbers are challenging to interpret, and do not provide sufficient insights to the meaning behind the numbers unless supported by visualization. When does a correlation score imply the model is good enough to replace human evaluation
as indicated in Table 2? And what are the gaps in the LJ? In Table 2, LJ Mistral achieves a Krippendorff’s-$\alpha$ $\overline{H}\overline{M}$ of 0.49. Visualization in Fig. 3 shows that the LJ does not rate any item as 1 and negligible amount of items are rated 5, while humans have assigned over 24% of the items a median rating of 5. This shows the key difference between LJ Mistral’s judgments and humans. This type of insight can potentially be useful in optimizing the prompts used by the LJ to minimize the gap with humans judgments, demonstrating the importance of effective visualization techniques.

Another key aspect to note is how human uncertainty varies across different median labels. Human labels tend to be more certain when they assign extreme ratings (1, 5) compared to the middle ratings (2-4), demonstrated by the distribution difference between the median $\overline{H}$ and all $H$ ratings, as shown in Fig. 3. Relatively higher consistency in extreme ratings is a common pattern observed when humans are asked to rate using star-rating schemes and Likert scales, an observation typically reported in recommendation systems (Amatriain et al., ). This type of differences in the extent of human uncertainty depending on the rating further demonstrates the deficiencies in assuming a single gold rating for tasks that rely on human perception, clearly illustrating the need for a HM correlation metric that does not assume a single gold label, such as the $JS_{b}$ we proposed in Sec. 3.2 in RQ2.

#### 3.3.2 Perception-based preference visualization

For visualizing pairwise model preferences to compare human and machine judgments, we propose the “Pairwise preference HM perception chart”. The concept behind this visualization is similar to the Ordinal HM perception comparison chart, the main difference is that is bins are separated by the most frequent human label for a given item. This visualization also helps understand, for a given model pair, how reliable the human majority is. For instance, in Fig. 4, in the case of $\langle$claude-v1, gpt-3.5-turbo$\rangle$, even though the preferred model is Claude ( 38.9% preference), when human majority prefers GPT-3.5 there is almost no contention. We also visualize the preference of LJ Claude Sonnet shown in Appendix Fig. 7.

Figure: Figure 4: Pairwise preference HM perception chart: Human preference distribution vs. LJ GPT-4 for a given majority label per model pair. T is tie. Compares model pair (Left) Claude-v1 (A) vs. GPT3.5-turbo (B) (Right) Alpaca-13B (A) vs. GPT-3.5-turbo (B). Note: Using GPT-4 LJ has only one judgment per item in the results from Zheng et al. (), hence $\text{M}^{w}$ and $M$ are the same.
Refer to caption: x7.png

## 4 Discussion and conclusion

### 4.1 Humans are uncertain and their labels are noisy, so machines are better?!

Evaluation procedures have predominantly circumvented dealing with human uncertainty, except for recent limited works (Chen et al., ; Wang et al., ; Chen et al., ), and have thus relied on the simplified assumption that a single gold label is sufficient. A single gold label is only sufficient when there is little or no ambiguity, while being able to quantify human uncertainty is crucial in measuring the effectiveness of automatic evaluation. The ubiquitous nature of uncertainty is studied in human psychology  (Kahneman et al., ) and is also illustrated in a widely used MNLI dataset, which was carefully curated with 5 human labels per item. Over 40% of the samples have some degree of uncertainty, as shown in Table 3.1. In addition, given the challenges in human evaluation, including obtaining consistent labels (Elangovan et al., ), recent datasets have started to rely on collecting one human label per item (Bai et al., ; Ganguli et al., ; Stiennon et al., ).
This further demonstrates the urgent need to acknowledge that quantifying human uncertainty is essential for measure the effectiveness of automated evaluation.

The machine learning community cannot afford to dismiss uncertainty as simply “poor quality labels from humans”, key systematic errors and performance gaps in models become apparent under high human certainty
as shown in Sec. 3.1. As a corollary, under high uncertainty even a random labeler can appear to have similar or better correlation with humans, as shown in Fig. 2. More importantly, for safety critical applications such as medicine, when humans labels are highly robust and certain, but the corresponding machine assigned labels differ, it can point to serious deficiencies in the automated system. Furthermore, the implications of uncertainty include changes to ranking of the automatic evaluator as well as the corresponding absolute correlation scores
as shown in Table 2 and Table 3 depending on the reference human label. We acknowledge that collecting high quality labels from humans is expensive, and that in some cases gathering just one label per item might be adequate, such as when attempting to understand if one LLM is better than the other, provided the humans end up overwhelmingly preferring the output of one LLM over the other. In the case where the performance of two LLMs are close to one another, uncertainty in human judgments may be a signal that says the model outputs are quite similar. Thus, in some cases uncertainty can be a potential indicator of lower quality of labels requiring improvements to the underlying label collection process (Elangovan et al., ), in other cases uncertainty might be inevitable and therefore a valuable signal. Under uncertainty, comparison of automatic evaluation to human evaluation is challenging, and traditional correlation measures also fail to adequately account for this. The proposed binned JSD for perception is an effort in that direction so that metrics do not penalize human uncertainty, whilst comparing machine perception or attitudes with humans.

### 4.2 Challenges in metrics and interpretability

Deciding which statistic is appropriate depends on the data and how well it fits the assumptions made by the statistical measure (Eubanks, ).
IRA metrics were never designed for systematic errors, and any accommodation for errors were based on assumptions about typical human behavior. Errors made by LLMs are rarely predictable, yet they are not random; rather, they are reproducible, making them systematic errors. The unpredictable nature of LLMs makes it difficult to design an effective metric that compares them with humans, given the uncertainty associated with human labels.

Despite the deficiencies in metrics, some metrics may be a better fit, depending on the aim and the results of the study. For instance, in the case of perception-based preference experiments, where the goal is to understand which among a given pair of models is better, and when the result is almost unanimous with one model overwhelmingly preferred over the other, then the agreement in rarer (minority) labels may not matter. Hence, measures such as Krippendorff’s-$\alpha$ and Fleiss-$\kappa$ which estimate chance agreement based on the observed label distribution can result in substantially lower IRA scores even when there is high agreement on the majority label (winning model), whereas Randolph’s-$\kappa$ assumes that the labels are equally distributed and hence is a better fit. For perception-based ordinal values, the extreme labels (such as 1 or 5 in the case of Likert 1-5) are usually strong indicators of human preferences, especially when supported by visualization as shown in Fig. 3. These extreme labels, might form the minority case, but might be crucial to interpreting the results. Hence, measures such as Krippendorff’s-$\alpha$ and Fleiss-$\kappa$ estimate chance agreement based on the observed label distribution are a better fit compared to measures such as Randolph’s-$\kappa$ that assumes uniform distribution of all labels.
For tasks where a single majority label is not sufficient to represent human preferences, measures that do not assume a single gold label such as the proposed approach – binned JSD for perception, can be a better choice to compare human and machine performance.

In addition, statistical analysis, such as null-hypothesis and significance testing, is essential for determining whether one model outperforms another by random chance. Here, the chance component includes 2 aspects, (1) chance due to the nature of samples in the evaluation set (2) uncertainty in human labels. A third aspect, even harder, is estimating the error rate as a result of systematically unpredictable erroneous labels from any automated evaluator.
Future studies should explore these problems, including approaches like resampling (Deutsch et al., ). Incorporation of chance in rank correlation is also an important aspect to account for when two models differ in rank, but the corresponding difference in absolute scores is negligible, then the difference in the rank may not be meaningful.

IRA metrics are challenging to interpret, and hence visualization is key to understanding gaps and strengths of any given metric. Effective visualization is a trade-off between plotting every single data point (too much information that is hard to synthesize) and an aggregate view (summarized view where key information might be obscured). The proposed perception charts are a step towards emphasizing how an aggregate number may not be sufficient in capturing the underlying data, depicting how human uncertainty varies across different labels, which in turn affects how machine performance can be interpreted across labels as shown in Fig. 3 and 4.

### 4.3 Recommendations for reporting effectiveness of automatic methods

To conclude, comprehensive analysis of performance involves investigation beyond a single aggregate number. To that effect, in the case of comparing automatic evaluation with human evaluation, we recommend the following steps:
1. Stratification by uncertainty levels: As discussed in Sec. 3.1, uncertainty in human labels can obfuscate performance gaps between machines and human evaluators. Hence, we strongly recommend stratifying results by uncertainty proportions.
2. Multi-metric reporting: If there was no uncertainty, measures such as F1 would have worked. However, as a result of uncertainty, no single metric can capture important insights about every type of data as demonstrated in Sections 3.1, 3.2 and 3.3. Thus, we recommend reporting on multiple metrics belonging to different families, such as chance and non-chance-adjusted measures, so each metric in its own way can assist in bringing the less obvious but critical aspects about the underlying to the forefront.
3. Visualization of results: A single non-parametric aggregate metric can rarely capture the entirety of underlying raw data, and hence visualization is key to understanding performance gaps, as discussed in Section 3.3. The proposed perception charts are a step towards making aggregate correlation more interpretable, as well as highlighting the strengths and gaps of the automatic labeler.

#### Acknowledgments

We would like to thank Saab Mansour for reviewing our paper and providing valuable feedback.

## Appendix A Appendix

### A.1 Prompts used for NLI

These prompts are reused from Liu et al. ()

### A.2 Prompts used for SummEval

Note: These prompts are reused from Liu et al. () used in G-Eval.

#### Coherence

#### Consistency

#### fluency

#### Relevance

### A.3 Prompts used for MTBench Preference data for Sonnet

### A.4 Percentage agreement

Percentage agreement: The percentage agreement, as defined by McHugh (), for $n$ items – where ${x_{1},x_{2},\ldots,x_{n}}$ represent each item – that is labeled, with $R$ raters and $C$ categories, as the maximum percentage of votes the most frequent item gets as long as it is greater than 1. Formally:

$$ $\frac{1}{n}\sum_{i=1}^{n}\left(\max_{c\in\{1,\ldots,C\}}\left(\frac{1}{R}\sum_ {r=1}^{R}\mathbbm{1}_{\{x_{i}^{r}=c\}}\right)\cdot\mathbbm{1}_{\left(\sum_{r=1 }^{R}\mathbbm{1}_{\{x_{i}^{r}=c\}}>1\right)}\right)$ (3) $$

where $x_{i}^{r}$ is an annotation for $x_{i}$ by the $r$th rater. Note that the $\mathbbm{1}_{\mathbf{A}}$ is the indicator function that returns 1 if condition $\mathbf{A}$ is true and 0 otherwise.

### A.5 Impact of label uncertainty on ordinal SummEval dataset

We create multiple subsets of 100 samples, with each subset having a predefined portion of samples with perfect agreement in human labels. We then randomly select samples from the dataset based on this portion. This simulation helps us understand how correlation measurements change with varying levels of human label uncertainty.

Figure: Figure 5: Impact of noise on ordinal dataset Summeval
Refer to caption: x9.png

### A.6 Visualization of rouge-score correlation

Figure: Figure 6: Visualizing correlation with human median rating for relevance: (Left) Rouge-1 (Right) LJ (Mistral). Note: Rouge scores are continuous values [0,1], humans and LJ use integer values 1-5.
Refer to caption: x15.png

### A.7 Code for Binned-JSD-perception

### A.8 Visualization of preference data

Figure: Figure 7: Pairwise preference HM perception chart: Human preference distribution vs. (Top) LJ GPT-4 (Bottom) LJ Claude Sonnet 3 for a given majority label per model pair. T is tie. Compares model pair (Left) Claude-v1 (A) vs. GPT3.5-turbo (B) (Right) Alpaca-13B (A) vs. GPT-3.5-turbo (B).
Refer to caption: x17.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: How come the metric does not need a single value to approximate human labels but relies on a single "human and machine labels are not treated interchangeably, as the items in a given bin are selected by the human median or majority value"? This seems contradictory to me.
soundness: 3
strengths: 1) The paper is well structured and presented and is very clear and easy to follow and read.  
2) The paper clearly shows the issue related to relying on a high correlation between human and machine-generated outputs, giving cases where the uncertainty in human annotations is high; these correlations seem to be high, but that could also be the case even when the labeling is random. 
3) The study proposed a metric, namely binned JSD, to account for variations and uncertainty in human judgment.
summary: The paper discusses the current landscape of using LLMs as judges for various tasks and presents compelling arguments for why existing correlation metrics might not take into account variations and uncertainty in human judgment.

After the author's reply (which includes more datasets and clarifications about the metric), I believe that the paper makes good contributions and good cases of showing how to be careful when using LLMs as judges. I there recommend to accept the paper!
weaknesses: 1) It is not clear to me how this new metric handles the issues raised with traditional metrics. Could the author clarify and show cases of how JSD improves the analysis of LLMs as judges when compared to human judgment? It seems like a promising direction, but I am not convinced due to the limited number of datasets and support of the authors' claim. 

2) The type of human annotation is limited. Although the paper presented very well the type of collected human annotations, I limited these in their evaluations. Given the space and the breed of the paper, I suggest adding a few more datasets (Please check paper [1] for some guidance on what dataset to choose.)

3) The study considered four datasets with variant tasks and annotations that have sufficient human annotators. I think the number of annotations was well considered, but the number of the dataset could be improved, and metal analysis could have been better presented instead of large tables per dataset. 


[1] Bavaresco, Anna, et al. "Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks." arXiv preprint arXiv:2406.18403 (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Could the author provide some detail in the selection of different thresholds across the tables? It would be useful to clarify whether these thresholds are dataset-specific or if there's a general principle behind the thresholds selection.

In the paper, different metrics  are used and the human and machine labels are compared with average, with median or with majority labels. Are all these comparisons needed? Do they capture multiple aspects of the outputs?

Is the binned JSD the best metric for the proposed experiments? Is it possible to calculate this metric, or its adaptation,  for all the experiments proposed in the paper?

The bins are used to mimic human perception. Beyond the aggregation of perception, can they capture variation in human perception?
soundness: 2
strengths: The topic is new, and since human-labelled data are difficult to retrieve, an evaluation of machine-labelled data is very interesting.

The tests have been done on different LLMs 

Multiple tests have been performed.
summary: The paper describes the analysis of different measures in  evaluating  LLM responses.
A measure, specifically a binned Jensen- Shannon divergence is proposed.

This measure for ordinal perception data is justified by the author since the evaluation does not need a single gold label and the human and the machine are not interchangeable. This last condition breaks a necessary condition for the Krippendorff-alpha coefficient.

A distinction is done among Nominal values, ordinal values and continuous values. The Questions are not equally proposed for the three types of data, raising some difficulty in reading the paper.

RQ1:How does uncertainty in human labels impact correlation metrics when
we measure the efficacy of automatic evaluation methods? (Sec. 3.1)

The authors state that the uncertainty in human labels is high the human-machine majority labels are similar. The meaning appears to be that if there is no concordance among labeller, in this case the LLM judge is ok. It is the LLM-judge just adding noise to the labelling process?


RQ2: How can we measure human-to-machine(HM) agreement that accounts for human uncertainty as a result of variation in human perception?
Human to machine agreement is a measure of uncertainty in human perception. For this question, the comparison with different agreement percentage is tested. For this task is proposed the binned Jensed inequality.


RQ3: How can we visualize the underlying data to draw meaningful insights when we compare the results from automatic and human evaluations?

The authors compare ordinal and perceptual based ratings between human and machines
weaknesses: The work's presentation is not clear. The research questions help to  interpret the experiments but not all the results are clear.

Some terms, like H^W R^W (^ indicates apex), are defined in the caption of caption 1. Probably they should be defined in the text and used in the table.

It is unclear how the partitions are decided. In the experiments:in table 1 the thresholds are  0, 0.8, 1; in table 2 the threhsolds are 0.6 and 1; in table 3 are 0.6, 0.8, 1.0. It is not clear if the thresholds are experiment dependent or there is a rationale behind the threshold selection.

The random classifier test, reported in table 1 should be better described. The table reports unique =2 or unique =3 with a percentage. Unique term is not present in the description and should be explained for the clear presentation of the experiment. If the MNLI and SNLi
dataset have only a limited set of labels it should be specified in the description.

In Table 2 the terms H^mu M^mu are not specified. 


In Figure 3 are shown the human perception vs the machine labels binned by human median rating. The JS value is reported.
The highest values of JS are for \bar{H}=1 and  \bar{H}=5. Looking at the histograms, the histograms of human and machine with \bar{H}=3 (and  in some measure \bar{H}=4) are very similar, but the JS values are sensibly lower.
The authors state that humans tend to be more certain when they assign extreme rating and the machine rarely provide extreme values. 
Could  the explanation of the experiment take into account also this aspect? If there is a different interpretation of this discrepancy ( similar histograms but lower JS value) it would be useful to make this point more evident.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Could you provide more specific implementation guidance or pseudocode for this metric, perhaps in an appendix? This would help ensure reproducibility and clarity for those looking to apply it.
2. The paper attributes label uncertainty to genuine perceptual differences, but could the authors discuss other potential sources, such as cultural or contextual biases among annotators? How might such biases affect the evaluation results, and could additional stratification methods help account for them? This would make the framework more applicable across diverse datasets and ensure its robustness in various contexts.
soundness: 3
strengths: This paper makes a valuable contribution by tackling the often-overlooked role of human uncertainty in evaluating generative models. It’s original in its approach, introducing the binned Jensen-Shannon Divergence metric to better capture the nuances of human perception and using tools like perception charts to bring new depth to evaluations. The quality of the work shows through in its thorough methodology, with experiments across multiple datasets that lend strong support to the findings. The paper is also clear, with a well-structured flow and visuals that help explain complex ideas. Most importantly, the paper has real significance: its framework could reshape how we evaluate generative models, especially in areas where human judgment isn’t always straightforward. Finally, the paper’s significance lies in its potential to reshape evaluation practices for generative models, especially in applications where human judgment is inherently subjective, such as content generation, recommendation systems, and interactive AI. By emphasizing the role of human uncertainty and offering practical tools to account for it, this work highlights a crucial aspect often ignored in model evaluation. This framework could lead to more accurate and context-sensitive evaluations, particularly for models that interact with or respond to human preferences.
This paper offers a well-supported framework that deepens our understanding of human-machine evaluations, bridging the gap between traditional metrics and the complexities of human perception. Its contributions could have a lasting impact, inspiring future research and improving evaluation standards across the field of generative modeling.
summary: This paper explores how current methods for evaluating generative models often fall short by relying too heavily on correlation metrics like Krippendorff’s α and Randolph’s κ. These metrics, while common, can mask important nuances in human judgment, especially in cases where human responses vary widely. The authors show that when there’s a high degree of variation in human evaluations, machine judgments might seem to align well, but as human consensus strengthens, this alignment breaks down, revealing gaps in machine understanding. To address these issues, the paper proposes a more robust evaluation framework that includes stratifying results by the level of human agreement and introducing a new metric, the binned Jensen-Shannon Divergence, to better capture perception-based evaluations. Additionally, the authors suggest using visual tools like perception charts to more clearly illustrate where machine judgments align or diverge from human benchmarks. By combining multiple metrics and visualization methods, this approach aims to provide a more accurate and comprehensive understanding of automated evaluations, especially in areas where human judgments are inherently uncertain.
weaknesses: There are some potential for improvement in paper such as expanding on the technical implementation of the binned Jensen-Shannon Divergence metric would make it more accessible to practitioners, potentially by providing step-by-step instructions or pseudocode. Additionally, testing the framework on a broader range of generative models beyond text (such as image or audio) would demonstrate its versatility. The perception charts are helpful, but they primarily show aggregate trends, which may obscure individual item-level discrepancies; adding item-level visualizations or error bars could improve clarity. To connect more concretely with real-world applications, the paper could benefit from case studies or examples where the framework enhances specific generative tasks, such as in recommender systems. Moreover, a side-by-side comparison with existing metrics like Krippendorff’s α would better illustrate the added value of the proposed metric. Including confidence intervals or statistical significance testing could also add rigor to the findings. Finally, considering potential biases in human label uncertainty, such as cultural or contextual differences among annotators, would make the framework more robust across diverse datasets. Together, these enhancements would increase the framework’s clarity, practical utility, and adoption potential.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - How do you decide what's "high" or "low" uncertainty in your stratification, and did you try other thresholds?
- Why did you choose Jensen-Shannon Divergence for human perception, and can you show this is better than existing metrics?
- How did you adapt Krippendorff’s α and similar metrics to account for systematic machine errors, not just random ones?
- Why use previous prompts without optimizing them for each model, wouldn't  this affect fairness in comparisons?
- Can you explain how you created the synthetic data for high uncertainty and whether it really reflects real-world data?
- Could you provide concrete examples where machine evaluations diverge significantly from human judgments?
- How would these findings help practitioners improve real-world model evaluations?
soundness: 2
strengths: Originality: This paper takes an original approach by addressing human uncertainty in generative model evaluation. Introducing stratified results by human variability and the new JSb metric for perception-based tasks adds fresh methods to handle subjectivity in evaluations. The perception charts also offer an innovative way to visualize nuanced performance differences.

Quality: The work is methodologically sound, with comprehensive experiments across datasets like SummEval and MNLI. The use of both real and synthetic data strengthens the empirical basis, showcasing the impact of human judgment noise on evaluation reliability.

Clarity: The paper is well-organized, clearly defining key concepts such as HH vs. HM correlation. Explanations of JSb and perception charts are straightforward, helping readers understand the new evaluation tools effectively.

Significance: This work fills an important gap by addressing subjective variability in human evaluations. Its proposed methods (if widely adopted) could lead to more accurate and relevant model evaluations, especially in perception-driven tasks across AI.
summary: This paper examines how human uncertainty affects evaluating generative models, noting that standard metrics like Krippendorff’s α may misrepresent machine accuracy when human judgments vary. The authors propose three main contributions: stratifying evaluation results by human uncertainty levels, introducing binned Jensen-Shannon Divergence (JSb) to better measure alignment with human perception, and creating perception charts to visualize evaluation performance more effectively. These tools aim to provide a clearer, more accurate picture of machine evaluation performance amidst human variability.
weaknesses: - The choice to stratify results based on “high” and “low” human uncertainty needs clearer justification. A discussion or empirical test on how these thresholds were set would make the stratification process more robust and reproducible.

- The introduction of Jensen-Shannon Divergence as a measure of perception-based tasks is promising, but its effectiveness is not thoroughly proven. Including a comparison with other potential metrics, such as Earth Mover’s Distance or Wasserstein Distance, would better validate the claim that JSb captures human perception more accurately.

-While reusing prior prompts is convenient, this may introduce biases or inconsistencies across models. Optimizing prompts specifically for each model would yield more accurate comparisons, especially given the importance of prompt sensitivity in LLM performance. Adding prompt-tuning experiments for each model could further solidify the findings.

-The synthetic data used to simulate high uncertainty scenarios lacks details on its generation process. More transparency on how closely this data reflects real-world scenarios, including its validation process, would help verify the relevance of the findings. 

- The shifts in ∆ (difference between HH and HM correlations) across different uncertainty levels are intriguing but underexplored. An in-depth analysis of these shifts—perhaps with concrete examples of where machines diverge from human judgment—would help understand the implications of these results more clearly. Visual examples showing alignment and divergence between human and machine judgments could greatly enhance interpretability.

- The experiments rely primarily on four datasets, which, while varied, still represent a narrow slice of possible applications.

Meta-review: This paper addresses the challenges of using aggregate correlation scores to evaluate the performance of LLMs as judges in subjective tasks. The authors argue that high human label uncertainty can misleadingly make LLMs appear to align closely with human majority labels, even more than human-human agreement. The experiments based on both benchmark and synthetic datasets, aim to show that as human label consistency increases, this assumption breaks down, emphasizing the importance of accounting for label uncertainty. The authors also introduce a binned JSD metric as an alternative and make additional suggestions  to use multi-metric reporting and stratification.

The reviewers are mostly positive, only one reviewer rates the paper marginally below acceptance threshold. Most of the criticism is around value of the binned-jsd metric and implementation / how a practitioner would use the methodology.

My recommendation is based on reviewer feedback which is mostly positive. The one critical reviewer does not engage much beyond the review and acknowledging the rebuttal.

Decision: Accept (Poster)