Title: Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

LLM-as-a-Judge

gold labels

validity

evaluation

meta-evaluation
Abstract: The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, plays a critical role in scaling and standardizing GenAI evaluations. To validate such judge systems, evaluators assess human--judge agreement by first collecting multiple human ratings for each item in a validation corpus, then aggregating the ratings into a single, per-item gold label rating. For many items, however, rating criteria may admit multiple valid interpretations, so a human or LLM rater may deem multiple ratings "reasonable" or "correct."  We call this condition rating indeterminacy. Problematically, many rating tasks that contain rating indeterminacy rely on forced-choice elicitation, whereby raters are instructed to select only one rating for each item. In this paper, we introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy. We draw theoretical connections between different measures of judge system performance under different human--judge agreement metrics, and different rating elicitation and aggregation schemes. We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias LLM-as-a-judge validation. Through extensive experiments involving 11 real-world rating tasks and 9 commercial LLMs, we show that standard validation approaches that rely upon forced-choice ratings select judge systems that are highly suboptimal, performing as much as 31% worse than judge systems selected by our approach that uses multi-label "response set" ratings to account for rating indeterminacy. We conclude with concrete recommendations for more principled approaches to LLM-as-a-judge validation.


Peer reviews:
comments: SUMMARY:
This paper is about evaluation under multi-choice options, where there is no single correct answer that may be adequate for a particular instance. The paper calls this indeterminacy. The paper then goes on to show that the problem can have consequences on which system is judged to be the best.

STRENGTHS AND WEAKNESSES:
**Strengths**

The paper is well-written and contains both mathematical insights and experimental results. The setting itself is interesting, i.e., ambiguity in classification tasks rather than forced-choice options. 

**Weaknesses**

I must admit that I did not fully follow the paper, but this is mostly likely attributable to me, rather than the paper. On the one hand, I wonder why multi-choice annotations should cause a problem; any labeling task where multiple options could be adequate can be obtained by simply considering the power set and then do standard annotation and evaluation on the power set. It seems I must have surely misunderstood a lot about the framework.

A minor point is that the paper does not have a section 1. I assume this is because the authors wanted to save space, but it is in any case quite unusual.

QUESTIONS:
Q1 - Would your setup be applicable for text generation tasks rather than discrete annotations?

Q2 - Why do you call it interdeminate? Isn't multi-label annotation and/or multi-choice a common scenario with well-defined evaluation metrics?

Q3 - what often happens is also that no single gold label is chosen from the annotations of each annotator but one considers the distributions of labels over annotators. Isn't that similar to your setup?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical limitation in current LLM-as-a-judge validation practices: how to properly evaluate judge systems when rating tasks are indeterminate (i.e., multiple answers may be \"correct\"). The authors introduce a comprehensive framework distinguishing between forced-choice and response-set elicitation schemes, and demonstrate theoretically and empirically that standard validation approaches can severely misestimate judge performance. Through experiments across 11 rating tasks and 8 commercial LLMs, they show that judges selected using conventional metrics can perform up to 30% worse than those selected using their proposed alternatives. The paper provides concrete recommendations including fully-specified rating tasks, multi-label response elicitation, and continuous multi-label agreement metrics.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Quality: The theoretical framework is rigorous and well-grounded in established psychological research on forced-choice selection effects. The empirical evaluation is comprehensive, spanning multiple domains (toxicity, factuality, relevance) and models. The authors provide formal identifiability results (Theorem 2.1) and necessary conditions for rank consistency (Theorem C.7), establishing clear mathematical foundations for their claims.\n- Clarity: The paper is exceptionally well-written with clear exposition of complex concepts. Figure 1 effectively illustrates the core problem, whilst the framework diagram (Figure 2) provides an excellent conceptual overview. The distinction between forced-choice and response-set elicitation is clearly explained and consistently maintained throughout.\n- Significance: This work addresses a fundamental problem in AI evaluation that has practical implications for high-stakes applications like content moderation and safety assessment. The finding that standard approaches can select suboptimal judges by substantial margins (30% performance degradation) is practically significant. The framework provides actionable guidance for improving evaluation practices.\n- Originality: The application of forced-choice selection effects to LLM-as-a-judge validation appears novel. The theoretical framework connecting different elicitation schemes, aggregation methods, and agreement metrics is original and provides new insights into why current practices fail under indeterminacy.\n## Weaknesses\n- Limited Scope: The framework only applies to discrete-option rating tasks, excluding continuous scales and open-form feedback, which limits its applicability to some evaluation scenarios.\n- Implementation Complexity: Whilst the authors claim no additional computational overhead, collecting response-set ratings may increase cognitive burden for human annotators, particularly for tasks with many options. The practical implementation challenges deserve more thorough consideration.\n- Generalisability Concerns: The experiments focus primarily on established benchmark datasets. More investigation of how the framework performs across diverse domains and rating task designs would strengthen the claims.\n- Methodological Limitations: The synthetic experiments make strong homogeneity assumptions about the human rating process (noted in limitations), which may not hold in practice across diverse annotator populations.\n\nQUESTIONS:\n1. Resampling Justification: Line 244 mentions resampling LLMs 10 times per item. Could the authors provide stronger justification for this specific number and analyse sensitivity to this choice?\n2. Threshold Selection: How sensitive are the proposed multi-label metrics to the threshold parameter τ used in downstream tasks? Guidelines for selecting appropriate thresholds would be valuable.\n3. Cross-Domain Validation: How well does the framework generalise to domains not represented in the current evaluation (e.g., creative writing assessment, scientific accuracy)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the limitations of current LLM-as-a-judge validation practices, especially under tasks with ambiguous or subjective criteria. The authors show that standard forced-choice evaluation schemes can significantly misrepresent the true agreement between human raters and LLM judges. They introduce a probabilistic framework that models rating task indeterminacy and propose multi-label, response-set-based evaluation as a more accurate alternative. Extensive experiments across diverse tasks and models demonstrate that their approach leads to more reliable and fair judge system selection.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n\n1. The paper provides a solid theoretical framework that emphasizes evaluating human and LLM judgments as distributions rather than single labels, which is a significant and valuable perspective.\n\n2. The authors conducted extensive experiments across multiple real-world tasks and a variety of commercial LLMs, demonstrating the empirical validity of their approach.\n\n## **Weaknesses**\n\n1. While the proposed evaluation framework is shown to improve judge system selection, the paper does not provide direct experimental evidence that these improvements translate to better LLM training or downstream task performance (e.g., through RLHF, DPO, or similar methods).\n\n2. The problem formulation and the proposed solution approach appear to be similar to those in [1]. It would be helpful if the authors could clearly articulate the differences between their work and this prior research.\n\n[1] Elangovan et al., \"Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge.\" ICLR. 2025\n\nQUESTIONS:\n1. Does using distribution-based evaluation signals lead to better results in RLHF or DPO training compared to forced-choice labels?\n\n2. What are the main differences between this work and [1]? Please clarify the novelty over prior work.\n\n3. For practitioners working with pre-existing datasets that used forced-choice ratings, what are the concrete steps to retroactively estimate the response set distribution without collecting additional human annotations?\n\n[1] Elangovan et al., \"Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge.\" ICLR. 2025\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses indeterminate rating tasks for LLM based Judges  and the failure of  the cases where a particular choices are forced in vanilla inference setting  without any reasoning.  The work proposes training-free multi-label metrics based solution to  mitigate the issue of indeterminacy. It is evaluated against 11 rating tasks including  diverse open-weight and close models demonstrating superior performance over previous methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The work introduces a probabilistic theoretical framework for indeterminate rating scenarios, providing a rigorous foundation for modeling uncertainty in ambiguous judge evaluation settings.\n\n2. The proposed method demonstrates significant improvements over existing baselines, achieving up to 30% performance gains in key metrics through its approach to probabilistic reasoning.\n\n\n\n### Weaknesses\n\n1. The NO chain-of-thoughts/reasoning (\"No explanations\" in prompt)  is highly impractical for currently deployed LLMs for a hard task like judgement. Existing benchmarks, such as RewardBench[1], allows the model to reason upto 1024 tokens before generating the final judgement. Hence,  the conclusions from this rigid assumption is questionable due to the way modern LLMs are trained.\n\n2. With the similar motivation of point 1, it is unclear how \"No explanations\" work for Reasoning models, such as o3/r1 models, as they are  explicitly trained  to generated long CoTs before the final answer.\n\n3. The Independence assumption are broken when the labels are not explicitly specified  to be different, such as for  the prompt in Figure 22, the LLM may  think that  all '\"very toxic\" queries are also always  \"toxic\" resulting in returning \"AB\" instead of only \"A\" or \"B\".  Thus, it is unclear how the  proposed method work in such cases.\n\n4.  It is unclear how  the LLMs are sampled, like the temperature, limited the significance of  the reported  result. If non-greedy sampling is used, how many times each query is sent to the model. If only one sample is generated  per query, how does the results, such as Figure 6,  change  for a new run?  If more than one, how  is the result aggregated?\n\n5. The paper only evaluates on possible indeterminate settings, so it is unclear how well the framework performs where indeterminacy is minimal, such as RewardBench[1].\n\nReferences:\n\n[1] Lambert, Nathan, et al. \"Rewardbench: Evaluating reward models for language modeling.\" arXiv preprint arXiv:2403.13787 (2024).\n\nQUESTIONS:\n1. Dataset size and sampling details: The paper does not specify the size of the evaluation dataset for each of the 11 tasks. It is unclear whether all samples from the source dataset were used or whether sub-sampling was performed, which is critical for assessing the robustness and generalizability of the results.\n\n2. Citation and dataset clarification: Line 225 references \"JudgeBench\" without citing it, and the paper does not explicitly identify the five datasets used from JudgeBench. This omission undermines transparency and reproducibility, as it leaves readers without clarity on the source of the evaluation data.\n\n3. Human Evaluation: Is there any human evaluation  performed to ensure the indeterminacy of the tasks? Adding qualitative examples for each evaluation tasks in Appendix could be useful.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}