Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge

Authors: Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi, Ling Liu, Sravan Babu Bodapati, Dan Roth
Primary area: generative models
Keywords: Automated evaluation, LLM as a judge, correlation measures

Abstract:
The effectiveness of automatic evaluation of generative models is typically measured by comparing the labels  generated via automation  with  human labels  using correlation metrics. 
However, metrics like Krippendorff's $\alpha$ and Randolph's $\kappa$ were originally designed to measure the reliability of human labeling, thus make assumptions about typical human labeling behavior, and these assumptions may not be applicable to machine generated labels. 
In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human  labels and those from automatic evaluation, including LLM-as-a-Judge. 
Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human assigned labels  is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to the human-to-human (HH) correlation. 
This can create the illusion that labels from automatic evaluation  approximates the human majority label. 
However, as the proportion of samples with consistent human labels increases, the correlation between machine  and human  labels fall well below HH correlation. 
Based on these findings, we first propose  *stratifying data by human label uncertainty* to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric -*binned Jensen-Shannon Divergence for perception* for such scenarios to better measure the effectiveness of  automatic evaluations. Third, we present visualization techniques -- *perception charts*, to contextualize correlation measures appropriately and to show the strengths and limitations of automatic evaluation.  We have open-sourced our analysis and visualization tools at https://github.com/amazon-science/BeyondCorrelation.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1729978617694

thread_type:
official_review

<thread_object>
creator: reviewer_rfao

<content>
summary:
This paper examines how human uncertainty affects evaluating generative models, noting that standard metrics like Krippendorff’s α may misrepresent machine accuracy when human judgments vary. The authors propose three main contributions: stratifying evaluation results by human uncertainty levels, introducing binned Jensen-Shannon Divergence (JSb) to better measure alignment with human perception, and creating perception charts to visualize evaluation performance more effectively. These tools aim to provide a clearer, more accurate picture of machine evaluation performance amidst human variability.

strengths:
Originality: This paper takes an original approach by addressing human uncertainty in generative model evaluation. Introducing stratified results by human variability and the new JSb metric for perception-based tasks adds fresh methods to handle subjectivity in evaluations. The perception charts also offer an innovative way to visualize nuanced performance differences.

Quality: The work is methodologically sound, with comprehensive experiments across datasets like SummEval and MNLI. The use of both real and synthetic data strengthens the empirical basis, showcasing the impact of human judgment noise on evaluation reliability.

Clarity: The paper is well-organized, clearly defining key concepts such as HH vs. HM correlation. Explanations of JSb and perception charts are straightforward, helping readers understand the new evaluation tools effectively.

Significance: This work fills an important gap by addressing subjective variability in human evaluations. Its proposed methods (if widely adopted) could lead to more accurate and relevant model evaluations, especially in perception-driven tasks across AI.

weaknesses:
- The choice to stratify results based on “high” and “low” human uncertainty needs clearer justification. A discussion or empirical test on how these thresholds were set would make the stratification process more robust and reproducible.

- The introduction of Jensen-Shannon Divergence as a measure of perception-based tasks is promising, but its effectiveness is not thoroughly proven. Including a comparison with other potential metrics, such as Earth Mover’s Distance or Wasserstein Distance, would better validate the claim that JSb captures human perception more accurately.

-While reusing prior prompts is convenient, this may introduce biases or inconsistencies across models. Optimizing prompts specifically for each model would yield more accurate comparisons, especially given the importance of prompt sensitivity in LLM performance. Adding prompt-tuning experiments for each model could further solidify the findings.

-The synthetic data used to simulate high uncertainty scenarios lacks details on its generation process. More transparency on how closely this data reflects real-world scenarios, including its validation process, would help verify the relevance of the findings. 

- The shifts in ∆ (difference between HH and HM correlations) across different uncertainty levels are intriguing but underexplored. An in-depth analysis of these shifts—perhaps with concrete examples of where machines diverge from human judgment—would help understand the implications of these results more clearly. Visual examples showing alignment and divergence between human and machine judgments could greatly enhance interpretability.

- The experiments rely primarily on four datasets, which, while varied, still represent a narrow slice of possible applications.

questions:
- How do you decide what's "high" or "low" uncertainty in your stratification, and did you try other thresholds?
- Why did you choose Jensen-Shannon Divergence for human perception, and can you show this is better than existing metrics?
- How did you adapt Krippendorff’s α and similar metrics to account for systematic machine errors, not just random ones?
- Why use previous prompts without optimizing them for each model, wouldn't  this affect fairness in comparisons?
- Can you explain how you created the synthetic data for high uncertainty and whether it really reflects real-world data?
- Could you provide concrete examples where machine evaluations diverge significantly from human judgments?
- How would these findings help practitioners improve real-world model evaluations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580575413

thread_type:
official_review

<thread_object>
creator: reviewer_b4vj

<content>
summary:
This paper explores how current methods for evaluating generative models often fall short by relying too heavily on correlation metrics like Krippendorff’s α and Randolph’s κ. These metrics, while common, can mask important nuances in human judgment, especially in cases where human responses vary widely. The authors show that when there’s a high degree of variation in human evaluations, machine judgments might seem to align well, but as human consensus strengthens, this alignment breaks down, revealing gaps in machine understanding. To address these issues, the paper proposes a more robust evaluation framework that includes stratifying results by the level of human agreement and introducing a new metric, the binned Jensen-Shannon Divergence, to better capture perception-based evaluations. Additionally, the authors suggest using visual tools like perception charts to more clearly illustrate where machine judgments align or diverge from human benchmarks. By combining multiple metrics and visualization methods, this approach aims to provide a more accurate and comprehensive understanding of automated evaluations, especially in areas where human judgments are inherently uncertain.

strengths:
This paper makes a valuable contribution by tackling the often-overlooked role of human uncertainty in evaluating generative models. It’s original in its approach, introducing the binned Jensen-Shannon Divergence metric to better capture the nuances of human perception and using tools like perception charts to bring new depth to evaluations. The quality of the work shows through in its thorough methodology, with experiments across multiple datasets that lend strong support to the findings. The paper is also clear, with a well-structured flow and visuals that help explain complex ideas. Most importantly, the paper has real significance: its framework could reshape how we evaluate generative models, especially in areas where human judgment isn’t always straightforward. Finally, the paper’s significance lies in its potential to reshape evaluation practices for generative models, especially in applications where human judgment is inherently subjective, such as content generation, recommendation systems, and interactive AI. By emphasizing the role of human uncertainty and offering practical tools to account for it, this work highlights a crucial aspect often ignored in model evaluation. This framework could lead to more accurate and context-sensitive evaluations, particularly for models that interact with or respond to human preferences.
This paper offers a well-supported framework that deepens our understanding of human-machine evaluations, bridging the gap between traditional metrics and the complexities of human perception. Its contributions could have a lasting impact, inspiring future research and improving evaluation standards across the field of generative modeling.

weaknesses:
There are some potential for improvement in paper such as expanding on the technical implementation of the binned Jensen-Shannon Divergence metric would make it more accessible to practitioners, potentially by providing step-by-step instructions or pseudocode. Additionally, testing the framework on a broader range of generative models beyond text (such as image or audio) would demonstrate its versatility. The perception charts are helpful, but they primarily show aggregate trends, which may obscure individual item-level discrepancies; adding item-level visualizations or error bars could improve clarity. To connect more concretely with real-world applications, the paper could benefit from case studies or examples where the framework enhances specific generative tasks, such as in recommender systems. Moreover, a side-by-side comparison with existing metrics like Krippendorff’s α would better illustrate the added value of the proposed metric. Including confidence intervals or statistical significance testing could also add rigor to the findings. Finally, considering potential biases in human label uncertainty, such as cultural or contextual differences among annotators, would make the framework more robust across diverse datasets. Together, these enhancements would increase the framework’s clarity, practical utility, and adoption potential.

questions:
1. Could you provide more specific implementation guidance or pseudocode for this metric, perhaps in an appendix? This would help ensure reproducibility and clarity for those looking to apply it.
2. The paper attributes label uncertainty to genuine perceptual differences, but could the authors discuss other potential sources, such as cultural or contextual biases among annotators? How might such biases affect the evaluation results, and could additional stratification methods help account for them? This would make the framework more applicable across diverse datasets and ensure its robustness in various contexts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676609934

thread_type:
official_review

<thread_object>
creator: reviewer_apgo

<content>
summary:
The paper describes the analysis of different measures in  evaluating  LLM responses.
A measure, specifically a binned Jensen- Shannon divergence is proposed.

This measure for ordinal perception data is justified by the author since the evaluation does not need a single gold label and the human and the machine are not interchangeable. This last condition breaks a necessary condition for the Krippendorff-alpha coefficient.

A distinction is done among Nominal values, ordinal values and continuous values. The Questions are not equally proposed for the three types of data, raising some difficulty in reading the paper.

RQ1:How does uncertainty in human labels impact correlation metrics when
we measure the efficacy of automatic evaluation methods? (Sec. 3.1)

The authors state that the uncertainty in human labels is high the human-machine majority labels are similar. The meaning appears to be that if there is no concordance among labeller, in this case the LLM judge is ok. It is the LLM-judge just adding noise to the labelling process?


RQ2: How can we measure human-to-machine(HM) agreement that accounts for human uncertainty as a result of variation in human perception?
Human to machine agreement is a measure of uncertainty in human perception. For this question, the comparison with different agreement percentage is tested. For this task is proposed the binned Jensed inequality.


RQ3: How can we visualize the underlying data to draw meaningful insights when we compare the results from automatic and human evaluations?

The authors compare ordinal and perceptual based ratings between human and machines

strengths:
The topic is new, and since human-labelled data are difficult to retrieve, an evaluation of machine-labelled data is very interesting.

The tests have been done on different LLMs 

Multiple tests have been performed.

weaknesses:
The work's presentation is not clear. The research questions help to  interpret the experiments but not all the results are clear.

Some terms, like H^W R^W (^ indicates apex), are defined in the caption of caption 1. Probably they should be defined in the text and used in the table.

It is unclear how the partitions are decided. In the experiments:in table 1 the thresholds are  0, 0.8, 1; in table 2 the threhsolds are 0.6 and 1; in table 3 are 0.6, 0.8, 1.0. It is not clear if the thresholds are experiment dependent or there is a rationale behind the threshold selection.

The random classifier test, reported in table 1 should be better described. The table reports unique =2 or unique =3 with a percentage. Unique term is not present in the description and should be explained for the clear presentation of the experiment. If the MNLI and SNLi
dataset have only a limited set of labels it should be specified in the description.

In Table 2 the terms H^mu M^mu are not specified. 


In Figure 3 are shown the human perception vs the machine labels binned by human median rating. The JS value is reported.
The highest values of JS are for \bar{H}=1 and  \bar{H}=5. Looking at the histograms, the histograms of human and machine with \bar{H}=3 (and  in some measure \bar{H}=4) are very similar, but the JS values are sensibly lower.
The authors state that humans tend to be more certain when they assign extreme rating and the machine rarely provide extreme values. 
Could  the explanation of the experiment take into account also this aspect? If there is a different interpretation of this discrepancy ( similar histograms but lower JS value) it would be useful to make this point more evident.

questions:
Could the author provide some detail in the selection of different thresholds across the tables? It would be useful to clarify whether these thresholds are dataset-specific or if there's a general principle behind the thresholds selection.

In the paper, different metrics  are used and the human and machine labels are compared with average, with median or with majority labels. Are all these comparisons needed? Do they capture multiple aspects of the outputs?

Is the binned JSD the best metric for the proposed experiments? Is it possible to calculate this metric, or its adaptation,  for all the experiments proposed in the paper?

The bins are used to mimic human perception. Beyond the aggregation of perception, can they capture variation in human perception?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684885015

thread_type:
official_review

<thread_object>
creator: reviewer_y99p

<content>
summary:
The paper discusses the current landscape of using LLMs as judges for various tasks and presents compelling arguments for why existing correlation metrics might not take into account variations and uncertainty in human judgment.

After the author's reply (which includes more datasets and clarifications about the metric), I believe that the paper makes good contributions and good cases of showing how to be careful when using LLMs as judges. I there recommend to accept the paper!

strengths:
1) The paper is well structured and presented and is very clear and easy to follow and read.  
2) The paper clearly shows the issue related to relying on a high correlation between human and machine-generated outputs, giving cases where the uncertainty in human annotations is high; these correlations seem to be high, but that could also be the case even when the labeling is random. 
3) The study proposed a metric, namely binned JSD, to account for variations and uncertainty in human judgment.

weaknesses:
1) It is not clear to me how this new metric handles the issues raised with traditional metrics. Could the author clarify and show cases of how JSD improves the analysis of LLMs as judges when compared to human judgment? It seems like a promising direction, but I am not convinced due to the limited number of datasets and support of the authors' claim. 

2) The type of human annotation is limited. Although the paper presented very well the type of collected human annotations, I limited these in their evaluations. Given the space and the breed of the paper, I suggest adding a few more datasets (Please check paper [1] for some guidance on what dataset to choose.)

3) The study considered four datasets with variant tasks and annotations that have sufficient human annotators. I think the number of annotations was well considered, but the number of the dataset could be improved, and metal analysis could have been better presented instead of large tables per dataset. 


[1] Bavaresco, Anna, et al. "Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks." arXiv preprint arXiv:2406.18403 (2024).

questions:
How come the metric does not need a single value to approximate human labels but relies on a single "human and machine labels are not treated interchangeably, as the items in a given bin are selected by the human median or majority value"? This seems contradictory to me.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733196059771

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal summary

comment:
We would like to thank the reviewers for providing us with positive reviews and valuable feedback to enhance the clarity of our paper.  

Our key objectives  of this paper have been to highlight

1. How **aggregate correlation scores  can misguide researchers** into concluding that LLM-as-Judge approximates human majority where human majority-machine correlation scores can seem higher than human-human correlation. In this paper, we show over multiple datasets that this is easier to achieve when the proportion of samples with human label uncertainty is quite high. As the proportion of samples with consistent human labels increase, this assumption that LLM-as-Judge approximates human majority falls through. In addition to using benchmark datasets, we also use a synthetic dataset to show how under high uncertainty, even a random labeller can appear to approximate human majority. Hence, our main takeaway  is that  human label uncertainty cannot be overlooked when measuring LLM-as-judge capabilities.

2. This finding led us to the follow-on question on **how to measure  LLM-as-judge capabilities, when the task is inherently subjective** and humans are likely to vary and consistent labels is not practically possible as there is no single ground truth / gold answer. To mitigate this, we propose a binned-JSD as a step in this direction to compare human labels with machine labels without assuming that a single gold label captures human choice.

3. A third contribution of this paper, is highlighting how all correlation metrics rely on some key assumptions about the nature of the human labellers and that no single metric is a perfect metric, including the proposed JSD.   This problem exacerbates with LLMs, as their nature is unpredictable.  Each metric has its own strength and weaknesses and highlights certain aspects of the underlying data. Hence, we **recommend that  researchers stratify the results by uncertainty, perform multi metric reporting and visualize the results to interpret aggregate numbers appropriately**. The proposed perception charts are a step in this direction to visualize notoriously challenging correlation numbers. 

The key themes of the feedback, now addressed,  we received from our reviewers  has been

1. **The value of the proposed binned-JSD is not clear** - We have now illustrated this using a toy example in Appendix A.12. We have also included a python implementation of binned-JSD for reproducibility in Appendix A.7. 

2. **How practitioners can use our framework  in real world scenarios is not clear** - We have included a case study (Appendix A.11) using 4 additional  datasets (to add to the existing 5 datasets included in the initial version of the paper)  used in popular papers to highlight how the conclusion that the machine is better or worse can be misleading as a result of either the human uncertainty or metric unsuitability. 

3. **Implementation details not clear - The stratification threshold and  how the synthetic data with random labeller was created** - We have now included details of how we stratify by all possible number of  votes that a majority or median label  (e.g. 5/5, 4/5,3/5 ..)  can receive and along with results across all partitions in  Appendix In Table 5 and Table 6. We have included a python implementation of how the synthetic dataset was created in  Appendix A10.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the challenges of using aggregate correlation scores to evaluate the performance of LLMs as judges in subjective tasks. The authors argue that high human label uncertainty can misleadingly make LLMs appear to align closely with human majority labels, even more than human-human agreement. The experiments based on both benchmark and synthetic datasets, aim to show that as human label consistency increases, this assumption breaks down, emphasizing the importance of accounting for label uncertainty. The authors also introduce a binned JSD metric as an alternative and make additional suggestions  to use multi-metric reporting and stratification.

The reviewers are mostly positive, only one reviewer rates the paper marginally below acceptance threshold. Most of the criticism is around value of the binned-jsd metric and implementation / how a practitioner would use the methodology.

My recommendation is based on reviewer feedback which is mostly positive. The one critical reviewer does not engage much beyond the review and acknowledging the rebuttal.