Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Authors: Luke Guerdan
Solon Barocas
Ken Holstein
Hanna Wallach
Steven Wu
Alexandra Chouldechova
Venue: NeurIPS 2025 poster ()
Keywords: large language models
LLM-as-a-Judge
gold labels
validity
evaluation
meta-evaluation
Primary area: evaluation
TL;DR: We introduce a principled framework for validating LLM-as-a-judge systems under rating indeterminacy, where multiple ratings can be "correct."

Abstract: The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, plays a critical role in scaling and standardizing GenAI evaluations. To validate such judge systems, evaluators assess human--judge agreement by first collecting multiple human ratings for each item in a validation corpus, then aggregating the ratings into a single, per-item gold label rating. For many items, however, rating criteria may admit multiple valid interpretations, so a human or LLM rater may deem multiple ratings "reasonable" or "correct."  We call this condition rating indeterminacy. Problematically, many rating tasks that contain rating indeterminacy rely on forced-choice elicitation, whereby raters are instructed to select only one rating for each item. In this paper, we introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy. We draw theoretical connections between different measures of judge system performance under different human--judge agreement metrics, and different rating elicitation and aggregation schemes. We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias LLM-as-a-judge validation. Through extensive experiments involving 11 real-world rating tasks and 9 commercial LLMs, we show that standard validation approaches that rely upon forced-choice ratings select judge systems that are highly suboptimal, performing as much as 31% worse than judge systems selected by our approach that uses multi-label "response set" ratings to account for rating indeterminacy. We conclude with concrete recommendations for more principled approaches to LLM-as-a-judge validation.

Validating LLM-as-a-Judde Systems under Rating Indeterminacy
Luke Guerdan" Carnegie Mellon University
Solon Barocas Microsoft Research
Kenneth Holstein Carnegie Mellon University
Hanna Wallach Microsoft Research
Zhiwei Steven Wu Carnegie Mellon University
Alexandra Chouldechova
Microsoft Research
Abstract
The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAl) systems, plays critical role in scaling and standardizing GenAl evaluations. To validate such judge systems, eval- uators assess human- judge agreement by first collecting multiple human ratings for each item in validation corpus, then aggregating the ratings into single, per-item gold label rating. For many items, however, rating criteria may admit multiple valid interpretations, so human or LLM rater may deem multiple ratings "reasonable" or "correct". We call this condition rating indeterminacy. Problematically, many rating tasks that contain rating indeterminacy rely on forced-choie elicitation, whereby raters are instructed to select only rating for each item. In this paper, we introduce a framework for validating a-jd- systems under rating in- determinacy. We draw theoretical connections between different measures of judge system performance under different numan-judge agreement metrics, and different rating elicitation and aggregation schemes. We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias LLMM-as---judge validation. Through extensive experiments involving 11 real-world rating tasks and commercial I.L,s, we show that standard validation approaches that rely upon forced-choice ratings select judge systems that are highly suboptimal, performing as much as 31% worse than judge systems selected by our approach that uses multi-label "response set" ratings to account for rating indeterminacy. We conclude with concrete recommendations for more principled approaches to LLM-as-ae a-jjuge validation.
Introduction
To improve efficiency, scalability, and repeatability, organizations are rapidly adopting LLMs, rather than humans, to rate the outputs of other generative AI (GenAl) systems. In this LLM-as-a-judge paradigm, illustrated in Figure 3 judge LLM system is used to rate the outputs of target GenAl sys- tem according to instructions specified a rating task [c.g.. Szy.aayynnski al. 2024 Zheng al. 2023 Bubeck et al. 2023). When adopting this paradigm, is critical to establish that judge systems produce valid ratings. This practice of validating judge systems (that are then used evaluate other GenAl
"Corresponding author: Iguerdan @ wwccs.cmmu cs.cmu.edu. Work completed in part as intern at Microsoft Research.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Toxic Language.
Factuality
Helpfulness
Relevance
Does target system output contain toxic language?
Is the larget system output factually accurate?
Is the following larget system oulpul helpful?
Is the largel system oulput relevant to the user's query?
Ingai aerizus issuwr?
looue: "ial
Dusre "Tayyo Tokyo." city with amazing nightils"
InRut 'What should today?" Qulput "Consider your diwtary preferences schedtale.
Ingut: 'Whet's photesynthesis" Quizu: how corvent sunlight into energy This fights olimate change."
Duiie "What rookis mistako.
Only an smatour would do that"
Select the option that best applies
Select the option that best appliee:
Select the option that best applies:
Select the option that best applies:
(A) Yes (B) No
(A) Yes (B)
(A) Yes (B) No
(A) Yes (B) No
Yes the autput is dismissive and belitting of
Yes these are widely sccepted cheraceeristice of Tokyo.
Yes the output encourages thoughtful decision-makee
Yes the output Identifies the core concept
others' skill level.
No the output
'amaziing
No the output too vague and does not provide specific suggestions.
critical bul within bounds of normal feedback.
subjective to be considered a factual descriptor.
No the climate change mention is tangential.
Examples of rating indeterminacy in (1) toxic language, (2) factuality, (3) helpfulness, and (4) relevance rating tasks. For each rating task, we illustrate how the same human rater (shown in the top right of each interpretation bubble) can identify multiple plausible interpretations of an item in rating task. The status quo forced-choice elicitation approach requires each rater to only select single "correct" option. In contrast, our proposed multi-label "response set" elicitation approach explicitly accounts for all plausible rater interpretations during judge system meta- mota-evaluuatin. See Figure 8 for additional examples with demeaning language and physical safety threat rating tasks.
Figure 1:
systems) is called meta- mneta-vvaluttion The process of meta-evaluation has become critical, as judge sys- tems are increasingly used for high-stakes downstream tasks, such as content moderation [Kolla et al.) 2024). automated red-teaming [Mazeika et al. 2024), and benchmarking (Chaudhary et al. 2024].
Current meta-evaluation approaches often validate judge systems by assessing human-judge agreement-i.e. the extent to which judge systems replicate "high quality" human ratings. To obtain such ratings, practitioners collect multiple forced-choice human ratings (where each rater selects a single option) for each item in a validation corpus, then aggregate them into a single, per-item gold label viewed as the "correct" rating. High categorical agreement between judge ratings and aggregated human ratings is taken as key measure of judge system performance [Lu and Zhong 2024 Kim et al. 2024 Jung et al., 2024,) Dong et al. 2024 [Es et al. 2023 Dubois et al. 2024].
However, recent research has established that, when rating GenAl outputs for subjective properties
such as harmfulness, helpfulness, toxicity, relevance, a single "correct" rating rarely exists [Pavlick
and Kwiatkowski 2019] Davani et al. [202]] Roth and Schlechtweg 2025 Min et al. 2020 Chen and
Zhang] [023 Goyal et al., 2022 Dsouza and Kovatchev [2025]. The issue not simply inter-rater
disagreement as to the "correct" " rating [Gordon et al. [202]] [ommeraurr et al. 2020). Rather, a rater may view multiple ratings as "correct" depending on how [Li al. 2024a], or in what cultural context condition we call rating indeterminacy. For example, consider a toxicity rating task where a target system responds to "How serious is this issue?" with "That's 's rookie mistake. Only an amateur would do that." A rater might reasonably view this output as toxic (dismissive and belittling) or non-toxic (critical but within bounds of normal beedback)-both interpretations are plausible. Problematically, the current approach of forced-choice rating elicitation requires raters to select only one of these plausible interpretations. As illustrated in Figure T this is not an isolated phenomenon: rating indeterminacy can arise in many rating tasks where LLM-as---ieeea aasa--udde systems are increasingly used to evaluate target systems.
[Goyal et al. 2002], rating task instructions are interpreted
A common approach for addressing rating indeterminacy involves aggregating multiple forced-choice ratings into a soft label, thereby assuming inter-rater disagreement is the root cause (Uma al. 2020 Peterson et al.] 2019 Plank 2022]. However, this approach does not capture intra-rater disagreement- whheeneettthhhhnnee a single rater views multiple ratings as plausible. We address this by introducing multi-label "response set" elicitation, which instructs each rater to select all ratings that correspond to a plausible interpretation of an item. We show that measuring human-judge agreement against these response set ratings is essential for selecting performant judge systems under rating indeterninacy. 2 previews one of our key findings, showing that the true best-performing judge system (GPT o3-Mini) is incorrectly ranked fourth when evaluated under standard forced-choies elicitation and
Figure
2

---

Cludee
GPT
Does the target system output contain toxic
Semmei 5.5
10t--hstrost
1.5-urbe
Does the target system output contain toxic language?
Clude
Mistral
GPT
Haku
Large
language?
DeepSeek
Mestral
GPT
Input: "How serioua this lssue?"
serious is this issue?"
31% reduction in consistency content filtering decisions.
Ouloul: "Whal rookie mistake. Only
3
"That's rookie mislake. Only an amateur would do that." *
amateur would do that." "
Rank
Select option that applies:
Select ALL options that could reasonably apply:
(A) Yes (B) No
(A) Yes (B) No
Forced-Choice Elicitation
Ranking Under Forced Choice Elicitation
Ranking Under Response Elicitation
Response Set Elicitation
Framework applied to "toxicity" task (Civil Comments [Borkan et al. 2019)). Judge rankings change substantially when ranked by categorical agreement with forced-choice human ratings (left) versus downstream task performance accounting for rating indeterminacy (right). The true top-ranked GPT o3-Mini ranks fourth (#4) under the status quo forced-choice elicitation method. The top-ranked judge under pareed-choice elicitation, Claude Sonnet 3.5, has 31% worse consistency with human decisions than GPT o3-Mini when judging the toxicity of target system outputs.
Figure 2:
aggregation schemes. Claude Sonnet 3.5, the top-ranked judge according the standard approach, performs 31 % worse than GPT o3-Mini when correctly validated against "true" response set ratings. In summary, we offer the following contributions:
We provide the first framework for validating LLM-ae-a-judge systems under rating indeterminacy i.c., where many items may have multiple "correct" ratings. We use this framework to compare existing meta-evaluation approaches and to develop principled alternatives.
Using this framework, we demonstrate theoretically and empirically that standard meta- meta-evaluation approaches relying upon forced-choier ratings can severely mis-estimate judge system performance under rating indeterminacy ($3.)). We identify a key mechanism driving these failures: differences in how human raters and judge systems resolve rating indeterminacy while providing forced-choice ratings. Our findings are based extensive experiments involving 11 real-world rating tasks and 9 commercial LLMs, including relevance, factuality, and toxicity rating tasks, among others ($4).
We provide four concrete recommendations for improving LLM-as----udae meta-evaluation under rating task indetermincy ($5). We publicly release all code and data used in experiments
2
Related Work
We will now provide brief overview of prior work. Please see Appendix Bfor detailed discussion.
LLM-as-a-judge meta-evaluation. growing line research has identified limitations standard approaches used for LLM-as-a-judde a-jaee eealtaaevvuatin.... Some work has identified factors affecting the reliability of judge system ratings, such as prompt formatting (Ye et al.] 20244 Gao et al., 22024 Shi et al. 2024]. We instead examine how the human rating process affects judge system meta-evaluatiin. Related approaches in this direction include optimizing the allocation of items to human raters RRlley al. 2024 and using judge systems approximate human rating distributions using expert labels [Chen et al., 2024a Most relevant to our work, Elangovan et al.] (2024) account for inter-rater disagreement by aggregating forced-choice ratings into soft label, then measuring human-judge agreement via a distributional (JS- US-Divergence) metric. However, this approach does not account for intra-rater disagreement that arises when each human rater identifies more than one "correct" rating. In contrast, we explicitly account for this intra-rater disagreement by measuring human-judge agreement against mubti-label "response set" ratings. We show empirically (e.g.. Fig. 5 that this multi- label approach is necessary to select performant judge systems when rating indeterminacy is present.
"See https///github.c http:///tuuubooo/gguuruaaiiiiiie comm/guerdan/indetermmnat This implementation contains (i) code for reproducing experiments and plots, and (ii) quickstart tutorial for applying the framework rating tasks.
3

---

Perspectivism in HCI and NLP. Research has increasingly recognized that many NLP tasks lack a definitive "ground truth" rating (Pavlick and Kwiatkowski 2019 Davani al. 2022 Sommerauer
2020 Roth and Schlechtweg. 2025]. Items can be ambiguous due to insufficient context [Min et al.
jet al. 2020 Chen and Zhang [023] or vague due to imprecise definitions of concepts like "toxicity"
[Gooal et al. 2022]. This has sparked "perspectivist turn" in NLP, which treats disagreement as
a meaningful signal to be captured throughout the model training and evaluation process, not noise to
be eliminated [Plank] 2022 Fleisig al.) 2024 Cabitza et al.) 2023 Frenda et al. 2024). Yet LLM- as-a-judge ineta-vvaluation remains understudied within the perspectivist HCI and NLP literature.
Models for human rating variation. Prior work has modeled mechanisms in the rating process that introduce human rating variation (HRV) in NLP tasks. Such models disentangle "spurious" sources of HRV to be attenuated from "meaningful" sources (e.g., attributed to vague rating instructions) that should be preserved during evaluation (Dsouza and Kovatchev 2025]. A robust line of research investigates how to disentangle "errors" (e.g., arising from human inattention) from meaningful signal [Gordon et al.) 2021 Klie et al., 2023 Lakkaraju et al.) 2013] TTanno et al., 2019] in evaluations. In this work, we draw attention to complementary issue: forced-choice selection effect P- the consequences of forcing a rater to select a single "correct" option when they determine multiple are reasonable. Forced-choice selection effects have been extensively studied in perceptual psychology literature [Dhar and Simonson 2003 Brown and of concern in benchmarking practices [Balepur et al. [025 To our knowledge, are the first to examine forced-chois selection effects in the context of LLM-as-a-jude meta-evaluation.
2018) and identified as an area
Prior work has also explored approaches for evaluating models under HRV. These include directly eliciting soft probabilistic labels from raters [Collins et al.) 2022] and aggregating hard labels from multiple raters into soft label [Uma et al., Peterson et al. 2019 Nie et al. [020]. When soft labels are available, a common approach involves measuring model performance via distributional metrics (e.g.. KL-Divergence, JS- JS-Divergence) [Nie et al. 2020,
Fornaciari et al., 202 Peterson
et al. 2019 Pavlick and Kwiatkowski 2019 Collins et al.) 2022]. While many such metrics could reasonably operationalize the "agreement" between judge system and human ratings, the fundamental question of which metric should be adopted in practice remains open-and is one we directly address.
3 Framework
Motivation. In our motivating setting, we are interested in selecting a judge system Gudge that rates the outputs of target GenAl system Guaget. aiming to match as closely as possible the results we would obtain using "high quality" human ratings. The selected judge system will be made available to others, who may then use it for a range of downstream tasks, such as content moderation of target system outputs, automated red-teaming, or benchmark scoring. Following standard practice, the metries we consider in judge system meta-evaluation constitute different measures of numan-judse agreement, and are agnostic to the specific downstream task, which may be unknown in advance D
We express a target system evaluation as a rating task consisting of n items. Each item 4 E T consists of (1) an output generated by Gaagen. (2) instructions for rating that output, and (3) ordered set of response options O (i.e., possible ratings) for that output. As is often the case in practice, we assume that the rating instructions and response options are the same for all items (O2- OVij. We use superscripts J and H to refer to the judge system and human raters, respectively.
Preliminaries.
Scope. Our framework is designed for closed form question formats with a discrete set of options; i.e., categorical and ordinal scales, but not continuous scales. These include common LLM-as-a-judee rating tasks [Zheng et al. 2023] such as single output grading, where raters rate an output generated by Gurge for property like "helpfulness" or "toxicity" (O = Yes, No} is common here). Our frame- work is also compatible with pairwise comparison tasks where raters indicate a preference between two outputs generated by one more target systems (O = (Win, Tie, Lose) is common here).
Outline. We now present our framework for validating judge systems for indeterminate rating tasks, which is illustrated in Figure 3l We describe how a broad range of LLM-as-judge meta-evaluation practices can be understood as operationalizing numan-judge agreement using different rating
II Appendix C we extend our analysis to model the joint effects of forced-choice selection and rater error. "This is analogous to how in classical supervised learning we might select a probabilistic classifier f - fx by choosing X to minimize cross-validated negative log likelihoood. But then the model may get applied to a downstream task where the relevant performance metric is its misclassification error at a particular threshold.

---

Goal Measuring Human-Judge Agreement
Goal Il: Measuring Downstream Performance
oulput language?
Popuration Human Raters
Roting Apgregation Functions
Rating Vectors
Gsrget
Select options than
Humran-Jaddee Agreemeet Metric
judge
(A) Yes
(B) No
Candidala Judgo Systoms
Duployment
Target System Ousputs [1.0., Rtmms
(m,
jusle
Content Filtering
Figure 3: Our framework connects two key meta-evaluation goals: measuring general-purpose human- judge agreement metrics (left), and validating judge systems on specific downstream evaluation tasks (right). Our framework illustrates how to correctly design rating tasks, aggregate ratings, measure human-judge agreement, and measure downstream task performance under rating indeterminacy.
elicitation schemes, different choices of how multiple ratings per item are aggregated, and different choices of the metric for measuring human-judge agreement from aggregate per-item ratings ($3.2). This allows discuss limitations of the standard approach meta-cvaluation when applied in indeterminate settings ($3.3), and introduce principled alternatives ($3.4). More complete details on the formal framework, theoretical analysis, and corresponding proots appear in Appendix
3.1 Conceptualizing Judge System Performance Under Rating Indeterminacy
We begin by addressing the foundational question of this section: When rating indeterminacy is present, what precisely the "high quality" manual human rating process that a judge system should aim to replicate? In answering, we distinguish between two rating elicitation schemes. Let O denote a of options in the rating task and let S be a subset of options from O. The set Q (Si, S2, .2.,... Sw) describes all combinations of options available to a rater. For example, in the toxicity rating task from (Yes, No, (Yes, No While forced-choice elicitation instructs rater (human or judge) to select single option form O, "multi-label" response set elicitation instructs a rater to select all options that could reasonably apply (i.e., E Q)
Figure 2 O (Yes, No} and
We argue that under rating indeterminacy, we should aim for high agreement with respect to human and judge response ratings- not their forced-choice ratings. This makes the downstream user of the judge system (e.g., the user adopting the judge for content moderation or red-teaming the arbiter of how they would want rating indeterminacy resolved in their specific task. For instance, content mod- eration, when an item is toxic under one interpretation of rating instructions but not toxic under another, the user may decide these cases should be resolved "toxic" and filtered out, which may not align with how human or judge raters resolve such rating indeterminacy under forced-choic elicitation.
3.2 Operationalizing Judge System Performance: Measuring Human-Judge Agreement
The standard meta-evaluation approach operationalizes measuring human-judge agreement through metrics (e.g., Hit Rate, Cohen's k) applied to ratings collected through forced-choice elicitation. This practice is not clearly aligned with our conceptualization of judge performance as defined against response set ratings; but in principle it's possible that the metrics produced correlate highly with response set human-judge agreement metrics in practice ($ 3.I.. By directly connecting forced-choice and response set elicitation, we show both theoretically and empirically that forced-choice elicitation in fact introduces significant biases and can lead to vastly suboptimal judge system selection.
Modeling Rating Variation in how Raters Resolve Indeterminacy. To connect response set and forced-choice rating formats, we introduce a probabilistic model that describes how raters (both human and judge) resolve rating indeterminacy (Figure left). Under this model, the forced-choice translation matrix, F, describes the probability that a rater who would choose response set under response set elicitation picks forced-choice option O S under forced-choice elicitation. Figure
5

---

Forced Chokc Transiaion Probability
Optinn Lockup Indicator
PiO Vesss
No))
Resgonse Sets
1
Response Sets
Response
Options
(Yee, Noj(Yes) (No)
(Yes, No)
Response Options
(Yes, Noj (Yes) (No)
(Yee, No)
Yes
0.35
Yes
0.5
(Yess
0.5
Yes
0.2
[Yes)
No
3.85
No
0.5
Cholon Distribution O
Forced Choice
0.5
(No)
MulliLLbbll Vector f2;
Option Lookup
0.5
(No)
Trinslation Matrix F,
Resporise Sel Distribution 6.-
Table
Response Sei Distribution
Translation from Response Set Distribution Forced Choice Distribution
Translation from Response Set Distribution Multi-Label Vector
Figure 4: An illustration of our rating model applied to an item in an underspecified ($) 3.3 Yes/No rating task. The response set distribution (0t) and corced-choice distribution (O;) denote probability vectors over response sets and -crced options, respectively. Left: O is recovered from multiplying 0 by the forced-choice translation matrix (F, .). Each entry in F, describes the probabiliiy of rater selecting a forced -choice option given its inclusion in a response set. Right: The multi-label vector (S2,) is recovered by multiplying the response set distribution by an option lookup table (A). The lookup table is determined by the rating task design (and hence known) and is fixed across items.
(left) illustrates an item where 30% of raters believe No) are both reasonable, and where under forced-choice, a rater is equally likely (50%) to resolve their rating to O Yes or to O = No.
Aggregating Ratings. Given multiple ratings per-item collected from some rating elicitation scheme (represented as a distribution rather than counts), the the next step before computing a metric is com- bining them into per-item rating vector. This is done through aggregation functions, Y, which map rating distributions, defined in Figure 4 to a single rating vector, Y, encoding the aggregate rating for given item. See Table [ examples of common aggregation functions.
(O,, 0
Operationalizing Human-Judge Agreement. The final step is to aggregate item-level rating vectors into a corpus-level measure. Let ''' denote random item from the corpus, and let Y Y denote random variables indicating the rating vectors assigned to item by the judge system and human raters, respectively. We measure the expected human judge agreement over all items via.
M(Y M() B[m(Y', Y"),,
where m is an agreement metric appropriate for the type of rating vectors being com- pared. For example, when using hard rating aggregation, we might measure the Hit Rate mHR(Y HR: Y = [arg max: Y = arg max; y, where h indexes response options for the rating task. When measuring agreement between soft labels recovered from a judge system versus human raters, we might evaluate a distributional metric such as KL--Divegeecc,, mKL Hn V Σx Y log (Y /Yg).
3.3 Limitations of Standard Practice Relying on Forced-Choice Elicitation
We now explain why agreement metrics defined against forced-choice ratings are unreliable under rating indeterminacy: forced-choice elicitation loses information that cannot be recovered from observed data. We say that a forced-choice rating task is fully specified if all items are determinate i... the rating task instructs raters how to resolve rating indeterminacy should they identify multiple ratings as "correct." Formally, this means |O| 1). The task shown in Figure 2 is not fully specified, but we can correct this by adding a Maybe option with instructions to select it whenever both Yes and No are reasonable.
Theorem 3.1. Under our rating model (Figure f). the response set distribution is identifiable from the forced-choice distribution if and only if the rating lask is fully specified.
This theorem states that infinitely many response set distributions are consistent with the forced-choice distribution when a rating task is not fully specified. Given that our true benchmark defined against the response set distribution (8 3.I. a judge can thus appear to perform well on forced-choice agreement metric (e.g., Hit-Rate) but perform poorly on response set based measures. Indeed, in Theorem
C.7 (see Appendix), we show that for forced-choice metrics to produce the same ranking over judge systems as a response set metric, we require very strong necessary monotonicity condition that seldom holds in practice. We complement this theory with empirical results in $

---

3.4 Proposed Alternative: Measuring Human-Judge Agreement via Multi-Label Metrics
Given the unreliability of human-judge agreement metrics defined against forced-choice ratings. how should meta-evaluation designers measure agreement? We propose several paths forward. First, whenever possible, fully specify rating tasks - i... by adding an unambiguously defined Maybe option for single output grading tasks or Tīe option for pairwise comparison tasks.
When is not possible to fully specify a rating task, we suggest measuring agreement via a continuous multi-label human-judge agreement metric defined against the response set distribution. For example, in our experiments we use the mean squared error (MSE) between judge and human multi-label vectors, given by MMSE( (Y (Yir, H = E - 112). which is a continuous multi-label metric.
When collecting multi-label response set ratings for the full corpus is not possible, such when con- ducting meta-evaluation on pre-collected data obtained from using an under-specified forced-choice elicitation task, we recommend collecting additional ratings for small subset of the validation corpus, eliciting both forced-choice and response set ratings for that subset of items (see &G.T). As we show in our experiments below, such auxiliary data be used estimate the forced-choice translation matrix and thereby partly recover from the information loss induced by forced-choice elicitation.
Experiments
To illustrate the limitations of measuring human-judge agreement against forced-choice ratings and establish the benefits of our proposed multi-label approach, we conduct experiments comparing the performance of judge systems selected using different junae agreement metrics on two downstream lasks: content filtering and prevalence estimation.
Content Filtering. In content filtering tasks, rater determines whether to suppress outputs from Gurget deemed positive for undesirable properties (e.g., "factual inaccuracy") (Wen et al. 2024). We assess judge system performance on this downstream task via decision consistency, which reflects the extent to which judge system makes the same allow/suppress decisions as human raters:
C (Y y") E[L(sT(YM))) s sE(YM
Here (YY = 1[2 2 r] is a thresholding function that classifies an item as "positive" if the multi-label probability assigned to the kth option exceeds т. If k = Yes, &ye (Y asks "was Yes selected for this item with (multi-label) probability least by human raters?" We provide practical guidelines for selecting and interpreting this r parameter in G.22
Prevalence estimation. In prevalence estimation tasks, rater estimates the proportion of Gurget out- puts that contain a property such as "toxicity", or "helpfulness." " For example, automated red teaming attacks designed to jailbreak Guaget estimate the proportion of input prompts (i.e.. "attacks"") that elicit the undesirable response of interest (i.e., the attack success rate) Maazeika et al. 2024 Ganguli et al. 2022). In pairwise comparison tasks, prevalence estimation recovers the win rate: the proportion of comparisons where an output from Model A is rated as preferable to one from Model B (Chiang et al. 2024]. We assess the quality of a judge system's 's prevalence estimate via its estimation bias:
BT BY H E[s EYsI(Y
Low estimation bias indicates that Gudge produces a prevalence estimate similar to one obtained from thresholding the human multi-label vector obtained from response set ratings. For example, when evaluating responses to red-teaming attacks designed to elicit toxicity [Mazeika al. 2024 Ganguli et al., 2022), B < 0 would indicate that Guudge underestimates the prevalence of "toxic" outputs compared to human raters. Our main experiments report the absolute value |B B MHH)l.
4.1 Experiment Design
Data. We construct 11 rating tasks designed to score target system outputs for properties such as "relevance", "fluency", "coherence", "factual consistency", and "toxicity." We leverage five datasets from JudgeBench [Tan et al. 2024]: SNLI [owman al. 2015), MNLI (Williams et al., 2017), a-NLI NNie et al. 2019]. Summlival Fabbri et al. 2021 and QAGS (Wang et al. 2020]. We also use Civil Comments [Borkan et al. 2019 to construct "toxicity" rating task. We adopt these datasets because they use categorical ordinal scale and have at least three forced-choice human
7

---

ratings per-item. For each rating task, we sample 200 items from the dataset to reduce computational overhead (see Appendix D). We provide qualitative examples illustrating how indeterminacy arises in representative set of rating tasks included in our experiments in Figure Df of Appendix A
Models. We include nine L.MS judge systems: GPT--33.- 40-Mini, o3-Minni (Schulman
et al. 2022]. Mistral- (Large, S [Jiang et al. 2023]. Claude-(3.5-Sonne) 3-Haiku | [Anthropic
2023 DeepSeek Chat Bi et al. 2024]. and LLama-3.3-70B-
(Grattafiori et al. 2024). When
using a reasoning-enabled model (e.g., o3-Mini) as judge, we extract the rating directly from the model output, which is stored in separate response field from the chain-of-thought reasoning trace.
Metrics. We compare a broad set of human-judge agreement metrics: categorical forced-choice metrics with hard (h) aggregation (Hit Rate, Krippendorff's s a. Fleiss, K, Cohen's 's k),, distributional forced-choice metrics with soft (s) aggregation (KL--Divergence, Cross-Entropy, JS- JS-Divergence, MSE), a discrete multi-label metric with hard response set (hrs) aggregation (Coverage), and continuous multi-label metric with soft response set aggregation (MSE). We use the convention (al a') to denote human and judge aggregation functions corresponding to each metric. We report results with all metrics in Appendix D and show an illustrative subset of metrics in the main text.
We report two variants of MMSE( (Y Y...). "MSE F",, uses oracle knowledge of the forced-choice translation matrix F to obtain continuous multi-label rating vectors from observed forced-choice data. This is reflects settings where response set ratings are available for computing agreement. "MSE F" is fully estimated from mostly forced-choice data, using only small auxiliary sub-corpus for which paired forced-choice and response set ratings are available to estimate F, proposed in $3.4
Rating Task Setup. We construct forced-choie and response set prompts for each rating task and re-sample LLMs 10 times per-item to estimate the forced-choice and response set distributions ] The rating tasks included in our evaluation are underspecified. Thus, while we can estimate the forced-choice distribution from human ratings, the true human response set distribution is unknown. Therefore, our experiments compare human-judge agreement metrics while systematically varying the relationship between humans' (observed) forced-choice and (unobserved) response set distribution.
We use our rating model to reverse-map conditional probabilities from forced-choice to response set distributions. Let 04 denote an option that is positive for the property of interest (e.g.. "factual consistency") and let o_ denote a negative option. We let 3 = P(o+ E S = o-) be the sensitivity paramete denoting the probability of rater selecting response set S that contains the positive daraoineeddenoting
option, given that they selected a negative option during forced-choice elicitation. For example, 3 =3 .3 indicates 30% chance of rater selecting response set containing "toxic" given that their forced- choice response was "not toxic." " B = 0 recovers the setting where no rating indeterminacy is present.
We let 33 vary across rating tasks but hold it fixed across items within task. Let 3 and 3 denote the human and judge system parameters for the f'th task, respectively. We systematically vary 3 for each rating task and construct a corresponding human response set distribution for each value. For each human-judge agreement metric, we then quantify the reduction of downstream performance incurred by selecting a judge system via an agreement metric in place of the downstream metric (Table D. Because both forced-choice and response set ratings are available for the judge systems, we can estimate Br. We vary 8 across the range of values estimated for B.C. See D for complete details.
4.2 Results
Our main experiments examine how well the top- hop-performing judge system as ranked by a human- judge agreement metric performs on each downstream task metric (decision consistency or estimation bias). As described in ( in our motivating setup we do not assume that the downstream task metric known at meta-evaluation time-so we cannot directly optimize against it during meta-evaluation.
Finding 1: Judge systems differ from one another-and hence also from human raters-in how they resolve rating indeterminacy when faced with forced-choice tasks. Figure 6 reports estimated sensitivity parameters (8:) for three judge systems across all 1l tasks. Recall that different parameters correspond to different ways of resolving rating indeterminacy when mapping from
SSee Appendix D for setup details. We demonstrate the robustness of our findings to the number of via a synthetic experiment reported Appendix[F] (4). "I our setup there is a 1:1 mapping between a given / and forced-choice translation matrix F.
8

---

Consaterits of Sweeced Jasige
Result Grouped Metric Type
Catogorical Foroed Choice vs. buttinational Forced Choice vs. Continuous Mu Mul--Label
Continuous Muit-Label
GMil Commentss
Medel
Clauce
-0.05
MNLI
10.00
Nisky
SNLI
1818
Realiwe
SummEva (Relevance)
0LB Loke
Mulli Label Cantinaes
Musiaaae Cortmucu
Mne Preskint
SummEval
(Coherencc))
Cragoria
Dikkie bovtl
SummEval
Calgurce
sippesturt (30)) Coneri cthl
Phoepree 215 bol tterepeen 2912
(Consistency)
Ban (MAF) Sieeeeeeee Judge
SummEval (F (Fleena)
QAGS
-015
TopicalChat Uses Knowledge
Understandiable
E1
03
02 Sensitiily Panamater (0")
0.0 0.1
0.2
0.3
0.4 05 0.6
Senstivity Panamater (8")
Senstivlly Paramater (8")
Senstivity Paramater Estimate (88)
Figure 5: Sub-optimality from selecting judge system via human-judge agreement metries vs. directly on the down- stream task metric. Results aggregated across 11 tasks, 9 sys- tems, and a sweep of r values. As 3 increases, performance gaps widen and forced-choier metrics become less reliable.
Figure 6: Estimates of task-specific sensitivity parameters (.)') recov- ered from four judge systems. These estimates vary considerably between judges and across tasks.
SummEval (Relevance) 8 0.0
Top Toplcalhhat Underslandable AM 0.3
ToplcaiChal Understandable A 0.3
SNLI A 0.0
0.70
0.90
0.00
3.00
0.85
0.50
0.20
0.40
0.30
0.80
5.00
0.40
0.75
0.20
8.00
0.60
Cohen's K (tv))
Blas (MAE)
kLD0 (s/s) 1
Blas (MAE) 1
MSE (snslars) (F)
Blas (MAE)
HR (h/h) (R/)) !
Bias (MAE)
Claude
Liama
GPT
Sonnet 3,5
70B-Instruct
3.5-Turbo
Claude
Mistral
GPT
Haiku
Large
4o-Mini
DeepSeek
Mistral
GPT
Chat
Small
03-Mini
Figure 7: Illustrative examples of rank inversions among four pairs of metrics and three rating tasks reported in the main results (Figure Axes ordered such that the best-performing judge with respect to each metric is placed at the top. Left two columns: SNLI and SummEval (Relevance) when 3. O. Right two columns: multi-label (MSE) and (HR) agreement metrics when 3 0.3.
response set to forced-choice ratings. We see tremendous variation across systems and tasks. E.g., for SummEval (Relevance), estimated parameters cover spectrum between 0.12 to 0.54 across systems.
Finding When human raters resolve rating indeterminacy differently from judge systems, agreement metrics measured against forced-choie ratings yield sub-optimal selections of judge systems. Figure presents our aggregate analysis (averaging across judge systems, rating tasks, and thresholds r). The y-axis measures how much worse the top-performing system as ranked by the given human-judge agreement metric performs compared to the "optimal" system selected by directly optimizing for the downstream task metric (this is measure of "regret"). We find that distributional

---

metrics (in particular, KL-D(h.j), not KL-D(j.h" ), perform best among forced-choice metries. While categorical metrics are suboptimal even for small 8, the gap between KL-D(h..) and our proposed response set-based MSE metrics is small until f z 0.2 0.3. In Appendix t we identify the primary mechanism driving sub-optimality of forced-choice metrics: when human and judge systems differ in how they resolve rating indeterminacy (give different forced-choice ratings when they have the same response set ratings), forced-choice-bete metrics become an invalid measure of judge systems' downstream task performance, and can produce severe mis-rankings (see Appendix Fig. 34).
Finding 3: Rank inversions between forced-choice and downstream metrics are common across tasks, and can be severe. Figure 7 shows a task-specifie breakdown of rankings. Mis-rankings do not always arise. E.g., for SNLI (far left), the same judge system is optimal for Cohen's thuman-judgy agreement metric) and Bias MAE (downstream task metric). In contrast, with Summ Eval (Relevance) (center left), Claude Sonnet 3.5 has the lowest estimation bias, whereas the best KI.-DDj.h) is achieved by GPT-3.5 Turbo. This sub-optimal selection increases estimation bias by 28%; equivalent to grossly mis-estimating the rate of "relevant" target system outputs by an additional 0.28 (on scale of [0.1]). While 3 0 in both far and center left columns of Fig. 7 (i.e., forced-choice ratings response set ratings for humans), inversions still arise because we know from Figure that 3 / 0 for the judge systems. The right two columns illustrate the robustness of MSE F (center right) and the instability of Hit-Rate (far right) on TopicalChat when 3 = 0.3. In Appendix D we provide numerous examples of similar ranking inversions across various rating tasks.
Finding 4: Continuous response set-based agreement metries like MSE select much more performant judge systems than forced-choice alternatives. Figure iillusrrates the benefits of using "MSE F" (red dashed line) for judge system selection. When no rating indeterminacy is present (3 = 0) "MSE F" performs comparably to Kh--Dhh.). When some rating indeterminacy is present (3 2 .2. "MSE Es selects more performant judge systems than all other human-judge agreement metrics. The metric "MSE F" requires complete response set data, or oracle knowledge of F. But we also show that the fully-estimated "MSE F" (red dotted line), which uses only small sub-corpus with 100 paired forced-choie and response set ratings to estimate F, performs almost as well.
5 Conclusion: Implications for Practice & Limitations
Implications for Practice. Through this work we identify four practical implications for designing more effective meta-evaluations (i) whenever possible, design "fully-specified" rating tasks that in- struct raters how to resolve rating indeterminacy; (ii) where tasks cannot be fully specified, elicit multi- label "response set" ratings and apply multi-label human-judge agreement metrics; (iii) where forced- choice data has already been collected, obtain a small auxiliary dataset with paired forced-choice and response set ratings to form metrics like "MSE fr: (iv) where only forced choice ratings are available, adopt distributional metrics specifically find KL-D(h.j) (but not (j.h)) often performs best.
Fully specifying rating tasks and adopting our proposed agreement metrics imposes no additional computational overhead over existing meta-evaluation practices. While rating task specification can reduce per-item rating requirements (Fig. (44), response set elicitation may introduce additional cogni- tive load for human raters, leading to commensurate increase in cost required to obtain human ratings. However, our extended empirical analysis shows that judge system rankings tend to remain robust to error in response set ratings (Appendix D. suggesting raters need not exhaustively deliberate over each option. Appendix G extends our discussion of the practical implementation of our framework.
Limitations. We note two limitations. First, our framework supports only discrete-option rating tasks. While this covers many evaluation settings where ALM-as-a-judge systems are used, our work does not directly apply to settings in which judge system is used to assign a score on continuous scale [Tan et al. 2024] - e.g., to evaluate an item's 's "factuality" on scale from to 100. Second, while our forced choice rating model is well-motivated by established perceptual psychology research [Bogacz] et al. 2006). it places a certain homogeneity assumption on the human rating process (see Appendix which may be violated in practice. Future work should explore relaxing this assumption.
We provide additional evidence that distributional metrics measured in the (h.j) direction outperform those measured in the (j.h) direction in Fig. 34 and 35) We observe the same for Cross Entropy (not shown here) with CE(h.j) consistently outperforming CB(j.). See feeeaaaciii al. 2021 for complementary discussion.
10

---

6 Acknowledgments
We thank members the Sociotechnical Alignment Center at Microsoft Research for their helpful comments early versions of this work. We also thank attendees of the Statistical Frontiers in LLMs and Foundation Models (SFLLM) and Evaluating Evaluations (EvalEval) workshops at NeurlPS for their thoughtful feedback. This work was supported in part by an award from the UL Research Institutes through the Center for Advancing Safety Machine Intelligence (CASMI).
References
Anthropic. Claude. https://www. hbtpp://www.aatdrrppc.... com/claude 2023. Accessed May 2025.
Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of these best describes multiple choice evaluation with Iims? a) forced b) flawed e) fixable d) all of the above. arXiv preprint arXiv:2500. 1412:, 2025.
Luciana Bencke, Felipe SF Paula, Bruno GT dos Santos, and Viviane P Moreira. Can we trust ilms as relevance judges? In Simpósio Brasileiro de Banco de Dados (SBBD), pages 600-612. SBC, 2024.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint 1rXiv:2401.02954 2024.
Rafal Bogacz, Eric Brown, Jeff Moehlis, Philip Holmes, and Jonathan D Cohen. The physics of optimal decision making: formal analysis models of performance in two-alternative forced- choice tasks. Psychological review, 113(4):700, 2006.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, 2019. UtL/tttp://axxi org/ 1900s 04561
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 150.v 0508.05366 2015.
Anna Brown and Albert Maydeu-Oliie Olivares. Modelling forced-choice response formats. The Wi- ley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development, pages 523-569, 2018.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
Federico Cabitza, Andrea Campagner, and Valerio Basile. Toward perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6860-6868. 2023.
Manav Chaudhary, Harshit Gupta, Savita Bhat, and Vasudeva Varma. Towards understanding the robustness of Ilm-based evaluations under perturbations. arXiv preprint arXiv:2412.09269, 2024.
Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. seeing the big through the small": Can lims approximate human judgment distributions on nli from few explanations? arXiv preprint arXiv:2406. 76000, 2024a.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge- Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference Machine Learning, 2024b.
Quan Ze Chen and Amy X Zhang. Judgment sieve: Reducing uncertainty in group judgments through interventions targeting ambiguity versus disagreement. Proceedings of the ACM on 7CCSCW2):126, ((C.cct)
Claman-Comprtt Interaction,
2023.

---

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Katherine Collins, Umang Bhatt, and Adrian Weller. Eliciting and learning with soft labels from every annotator. In Proceedings of the AAAI conference on human compulation and crowdsourcing, volume 10. pages 40-52, 2022.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92-110, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Ravi Dhar and Itamar Simonson. The effect of forced choice on choice. Journal of marketing research, 40(2):146-160) 146-160, 2003.
Yijiang River Dong. Tiancheng Hu, and Nigel Collier. Can Ilm be a personalized judge? arXiv preprint 4rXiv:240..11657, 2024.
Florian E Dorner, Vivian Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: Llm as judge won't beat twice the data. arXiv preprint arXiv:2410. 13344111 2024.
Russel Dsouza and Venelin Kovatchev. Sources of disagreement in data for Ilm instruction tuning. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 20-32, 2025.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi. Ling Liu, Sravan Bodapati, and Dan Roth. Beyond correlation: The impact human uncertainty in measuring the effectiveness of automatic evaluation and Ilm-as-a-judge. arXiv preprint arXiv:2410.03775, 2024.
Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert, Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309. 152/7, 2023.
Alexander R Fabbri, Wojciech Kryšcińskii, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 2021.
Fahim Faisal, Md Mushfiqur Rahman, and Antonios Anastasopoulos, Dialectal toxicity detection: Evaluating llm-as-a-judee consistency across language varieties. arXiv preprint arXiv:2411.1095,,
2024.
Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Efficient conformal prediction via cascaded inference with expanded admission. arXiv preprint arXiv:2007.03114 2020.
Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. The perspectivist paradigm shift: Assumptions and challenges of capturing human labels. arXiv preprint arXiv:2405.05886, 2024.
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio, et al. Beyond black & white: Leveraging annotator disagreement soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies. Association for Computational Linguisties, 2021.
Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessan- dra Teresa Cignarella, Cristina Marco, and Davide Bernardi. Perspectivist approaches to natural language processing: survey. Language Resources and Evaluation, pages 1-28, 2024.
12

---

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022.
Yicheng Gao, Gonghan Xu, Zhe Wang, and Arman Cohan. Bayesian calibration of win rate estimation with Ilm evaluators. arXiv preprini S0.i::2211.044222 2024.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tun Topical-chat: Towards knowledge-grounded open-domain conversations. arXiv preprint arXiv:2308. 11995, 2023.
Mitchell I. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference Human Factors in Computing Systems, pages 1-14, 2021.
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1- 2022.
Nitesh Goyal, Ian D Kivlichan, Rachel Rosen, and Lucy Vasserman. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human- Computer Interaction, 6(CSCW2)11228, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint 1xXiv:2411.1555 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL http::ttt/////t ab822310.06825
org/
Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: Lim judges with provable guarantees for human agreement. arXiv preprint arXiv:2407.18370, 2024.
Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1-21, 2024.
lan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. Annotation error detection: Analyzing the past and present for more coherent future. Computational Linguistics, 49(1):157-198, 2023.
Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. Llm-mod: Can large language models assist content moderation? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1-8, 2024.
Himabindu Lakkaraju, Jure Leskovec, Jon Kleinberg, and Sendhil Mullainathan. bayesian frame- work for modeling human evaluations. In Proceedings of the 2015 SIAM International Conference on Data Mining, pages 181-189. SIAM, 2015.
Xinyu Li, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized 2024a.
human feedback. arXiv preprint
Zongjie Li, Chaozheng Wang. Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in lmmebased evaluators. Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 11084-111000 2024b.
13

---

Haoming Lu and Feifei Zhong. Can vision-language models replace human annotators: A case study with celeba dataset. arXiv preprint arXiv:2410.09416, 2024.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprini arXiv:2402.04424) 2024.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004. 10645, 2020.
Nailia Mirzakhmedova, Marcel Gohsen, Chia Hao Chang, and Benno Stein, Are large language models reliable argument quality annotators? In Conference on Advances in Robust Argumentation Machines, pages 129-146. Springer, 2024.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv: 14599, 2019.
Yixin Nie, Xiang Zhou, and Mohit Bansal. What learn from collective human opinions natural language inference data? arXiv preprint ar Xi1::2010.03532, 2020.
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguistics, 7:677-694, 2019.
Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9617-9626, 2019.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308. 1483, 2023.
Barbara Plank. The 'hoolem''mm of human label variation: On ground truth in data, modeling and evaluation. arXiv preprint arX1v:2211.02570, 2022.
Hossein A Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles LA Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. LImjudge: Limms for relevance judgments. arXiv preprint arXiv:2408. 0888066600.. 2024.
Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. Finding replicable human evaluations via stable ranking probability. arXiv preprint arXiv:2404.01 244440044444 2024.
Michael Roth and Dominik Schlechtweg, editors. Context and Meaning: Navigating Disagreements in NLP Annotation, 2025. Association for Computational Linguistics.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2(4), 2022.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features prompt design or: How learned to start worrying about prompt formatting. arXiv preprint arXiv:2310. 1324, 2023.
Shreya Shankar, JD Zanffrescu-Pereiia Bjöm Hartmann, Aditya G Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of Umm outputs with human prefer- ences. arXiv preprint arXiv:2404.11 12272. 2024.
Lin Shi. Chiyu Ma. Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by Ilms. arXiv preprint arXiv:2406607791, 2024.
Pia Sommerauer, Antske Fokkens, and Piek Vossen. Would you describe leopard as yellow? evaluating crowd-annotations with justified and informative disagreement. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4809, 2020.
14

---

Annalisa Szymanski, Noah Ziems, Heather A Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald Metoyer. Limitations of the llm-as-a-judge approach for evaluating Ilm outputs in expert knowledge tasks. arXiv preprint arXiv:2410.20266, 2024.
Rikiya Takehi, Ellen M Voorhees, and Tetsuya Sakai. Llm-assisted relevance assessments: When should we ask Ilms for help? arXiv preprint arXiv:2411.0687, 2024.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating Ilm-based judges. arXiv preprint arXiv:2410. 12784, 2024.
Ryutaro Tanno, Ardavan Sacedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silber- man. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2019.
Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406. 12624, 2024.
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. A case for soft loss functions. In Proceedings of the AAAI Conference Human Computation and Crowdsourcing, volume 8, pages 173-177, 2020.
Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228, 2020.
Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang- Know your limits: A survey of abstention in large language models. arXiv preprint arXiv:2407. 18418, 2024.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 2017.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-V Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprini 4rX:v:2410002336. 2024.
Lianmin Zheng, Win Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 46:46395-46623, 2023.
15

---

Tosic Language Classification
Demeaning Language Classfiiation
Helpfuiness Clasoffictton
following target system outpot contain tosic language?
following target system oulput contenn demesning kinguage?
Is the following target output halptu??
Ingu "How serious this issue?" pr
'What's opinionn the emplyyess?
Incutt "Wher showlr eat for Auncep
Raling Task
Item
Outrud: rookle
Only an smoteur would that""
OutruE "The ain' servent is
Output: "Cansider your diatary prefernnoos acheckate
bunewuertt
Beloot bed app leee
best opo
Lecal apptieer
(A) Yos
(A) Yas
(B) No
Yes the outjut is dismissive and belitiing of others' skill level.
'bureauerat somatimes in derogelory manner describe ind rdffferent goverment officials.
Yes- the encourages thoughtuul destsion-mating.
Plausible Rater Interpretations
No the output la critical but within bounds of normal feedbaok.
No- White is that civll servents burssucrans the statement hammuu.
No the output and does not provide specific suggestions.
Factuelity Classification
Physical Safely Threal Classification
Relevance Classification
Is the following target system oupput fictullyy accurats?
Does the following target system output pase to physical sataty??
following targel systimm ourput elevan the user quary?
Input me about Tokyo"
InQUE "Mohal is photosyrthhasss"
Rating Task
Itam
Culoout TTkko is vibront with amazing
lasul "How do clean cven?"
Culout how plants
Outout: baking soda and vineger."
suniige into enwrgy This fights climate change."
nighttiie
Select bedt applins
Select best ap2lles
Select opdio that bddi (A) Yoa
(A) Yoa (9) No
(A) Yes
these widely accepted characteristies of Tokyo
Yes baking soda and vinega cleaning agants.
Yos explains cora concept importance
Plausible Rater Interpratations
Vibront 'amaziig subjactiva considered fectual descriprors.
No the output does not wam about vandiatioo needs.
No the dlimate change mention tangential.
Figure An expanded version of Figure that includes examples of (1) toxic language. (2) demeaning language, (3) helpfulness, (4) factuality, (5) physical safety threat, and (6) relevance rating tasks.
A
Appendices
This work contains the following appendices:
Provides an extended discussion of related work. hensive discussion of aggregation functions adopted in prior work on (1) LLM-as-a-judge meta-evaluation, and (2) model evaluation under task indeterminacy.
Appendix B
a compre-
Appendix C Provides a formal introduction to our probabilistic framework and rating model, with correspondig theoretical analysis and proofs.
Appendix D Provides setup details and additional results for experiments with real data.
Appendix Provides setup details and results for experiments with synthetic data.
Appendix F Provides two worked examples illustrating how monotonicity violations arise
under alternative operationalizations of human-judge agreement.
Appendix G. Provides extended discussion of our framework's 's practical implementation.
We release all code and data used for experiments attps//gghhh..com/ggaaaaanmeeeen
16

---

B Extended Related Work
LLM-as-a-Judge Meta-Evaluation. The LLM-as-a-judge paradigm has gained significant traction for scaling up GenAl evaluation processes traditionally performed by humans [Lu and Zhong
2024| Kim et al. 2024 Jung et al. 2024 Dong et al. 2024 Es et al. 2023 Dubois et al. 2024
Shankar et al. 2024 While offering efficiency advantages over human evaluation, this approach
raises questions about the validity of automated evaluation protocols. eveal-eval or assessing the trustworthiness of evaluation results, has thus become critical for the responsible adoption of judge systems. The status quo approach for judge system meta-evaluation involves measuring the categorical agreement between ratings assigned by human raters and a judge system over small validation corpus [Lu and Zhong, 2024, Kim et al. [2024] Jung et al.) 2024 Dong et al., 2024 Es ct al.) 2023 Dubois et al.) [224] Shankaar et al.) 2024]. High categorical agreement rates are taken as a sign of good judge system performance. When multiple human ratings are collected per-item, human rating variation (HRV) typically aggregated into a hard label via majority vote under an assumption that this hard label constitutes the single "correct" response to an item in rating task.
A growing line of meta-evaluation research has identified limitations in this standard validation protocol. Some work focuses on factors affecting the reliability of judge system ratings, such as prompt formatting and in-context example selection (Ye et al. 2024 Gao al., 2024 Shi al. 2024). Our work instead examines how the human rating process affects judge system validation. Related approaches in this direction include optimizing the allocation of items to human raters (Riley itt al. 2024] and approximating human rating distributions using expert labels with explanations [Chen et al. 2024a]. Most relevant to our work. Elangovan et al. [2024] demonstrate that categorical agreement metries yield misleading meta-evaluation results when humans express high uncertainty in how an item should be rated. While we corroborate findings illustrating that categorical metrics are unreliable for meta-evaluation, we go further by showing that even the distributional metrics proposed as an alternative can be unreliable under certain conditions. Our framework provides a theoretical explanation for these limitations and offers principled alternatives designed to better-elicit and preserve human uncertainty throughout the meta-evaluation process. To understand why status-quo meta-evaluation approaches relying upon categorical agreement metrics are conceptually incongruent with the structure of rating tasks, we turn to research on perspectivism in NLP and HCL
Perspectivism in HCI and NLP. Research has increasingly recognized that many NLP tasks lack a definitive "ground truth" response [Gordon et al., |021] Chen and Zhang, 2023 Pavlick and
K Kwiatkowski 2019 Davani et al. 2022 SSommerauer et al.
[2200] Roth and Schlechtweg 2025].
Items can be ambiguous due to insufficient context [Min et al.
2020
Chen and Zhang 2023] or vague
due to imprecise definitions of concepts like "toxicity" [Goyal et al.
2022]. Raters can fundamentally
disagree about how an item should be rated [Gordon et al. [022 Pavlick and Kwiatkowski 2019].
These insights have sparked a "perspectivist turn" in NLP Plank 2022 Fleisig et al. [0224] Cabitza jet al., 2023, Frenda et al., 2024], which treats disagreement as a meaningful signal to be captured throughout the model training and evaluation process, as opposed to noise to be eliminated.
In this work, we focus on a specific aspect of this perspectivist turn: rating task indeterminacy. We say that an item in a rating task is indeterminate when insufficient information is provided in an instruction to identify singular "correct" response for all items. Indeterminacy is a useful lens in our setting because a judge system must determine how to score target system output using only the infor- mation provided in the rating task. As such, evaluating the performance of a judge system requires un- derstanding which options could reasonably be interpreted as "correct" given the limited information captured a prompt. While indeterminacy can be attenuated by capturing further context in rating task prompt, or providing additional specificity in rubrics used to score target system outputs, it is dif- ficult to fully climinate indeterminacy from all items in a corpus, given the subjective nature of proper- ties such as "helpfulness", "relevance" or "factuality." Figure T Iillustrates how indeterminacy arise in common LLM auu-dee rating tasks (e.g.. "helpfulness" "relevance", or "factuality" scoring).
Models for Human Rating Variation. Prior work has explored approaches for capturing human rat- ing variation (HRV) arising from indeterminacy as signal for model training and evaluation. These include directly eliciting soft probabilistic labels from raters [Collins et al. 2022) and aggregating
hard labels from multiple raters into a soft label [Uma et al., 2020 Peterson et al. 2019 Nie et al..
2020 When soft labels are available, a common approach involves measuring model performance
Via distributional metrics (e.g., KL.-Divergence, JS-Divergence, Cross- (otoss- NNe et al. 2020
Fornaciari et al., 2021
Peterson et al. 2019 Pavlick and Kwiatkowski
2019 Collins et al. 2022
17

---

While many such metrics could reasonably operationalize the "agreement" between judge system and human ratings, the fundamental question of which metric should be adopted practice remains open.
Additional work has modeled mechanisms in the human rating process that give rise to HRV. Such models disentangle "spurious" sources of HRV to be attenuated from "meaningful" sources (e.g.. attributed to vague rating instructions) that should be preserved during evaluation (Dsouza and Kovatchev 2025]. A robust line of research investigates how disentangle "errors" (e.g., arising from human inattention) from meaningful signal [Gordon et al. 2021 Klie et al., 2023 Lakkaraju
al. 2013 Tanno al. 2019) in evaluations. In this work, we specifically model the influence of
forced choice selection effects the consequences of forcing a rater to select single "correct" option
when they determine multiple are reasonable [Balepur et al., [025 Dhar and Simonson, 2003] Brown
and Maydeu-Olivares 2018). Our model, which is grounded in perceptual psychology literature
Bogacz al. 2006], unifies four distinct ways of measuring performance under indeterminacy: categorical and distributional metrics derived from forced choice ratings, and discrete/contimuous multi-label metries derived from response set ratings. Leveraging this model, we show that the status quo approach of using agreement metries measured against the forced choice ratings (e.g.. Hit-Rate, Cohen's K, Krippendorff's a, JS-Divergence) yields sub-optimal selections of judge systems. We propose continuous multi-label metrics (e.g., mean squared error) as more robust approach for operationalizing "agreement" between judge system and human ratings under indeterminacy.
18

---

Judge Aggregation
Human A Aggregation
Metric Type
Metric (m)
Relevant Work
Lu and Zhong [2024] Jung
al. [2024]. Dong al.
[2024]. Es ct al. [2023]
Dubois et [2024].
Bubeck et al.
[223]] Zheng
Hit Rate (t)
Bard
Dbad
et at [2023 Faisal
[2024]. Gu et al. 2024]
Thakur et al.
2024
et al. [2024b] Chen et al.
[2024b]. J. Chiang et
[2024] Dorner et al 22044
Mirzakhmedova et
Categorical
Krippendorff's (f)
hard
hard
2024]. Chaudhary ct al.
2024
Kim et al.
[024
Dettmers
Fleiss' (1)
and
bard
al. [024]. Bencke et al.
2024
Rahmani et al.
2024
Cohen's i (f)
Bencke al.
2024
Scott's (f)
hard
Qbard
Thakur et al.
2024
Nie et al. 2000
Fornaciari
KL Divergence (1)
sott
[2021]
Peterson et al. 2209
Distributional
Cross-Entropy (4)
H woft
Pavlick and K Kwiatkowski
2019)] Collins et al. (2022
Nie et al.
[2020] Forncciai
JS Divergence (L)
Qgoft
soft
et al.
[021
Fisch et al.
2020
Takehi
Coverage (1)
a au
baady a
et al
2024
Categorical Multi-label
Predictive Efficiency (L)
Bard ohrs
N/A
Fisch et al.
2020
Recall (1)
Guant: a
hands a
Precision (t)
I a
baady a
Continuous
Binary Cross
a:
baady a
Multi-label
Entropy (L)
Mean Squared
a!
a
Error (4)
Table 1: A table of metrics (m) and aggregation functions (a* a²) that define operationalizations of human-judge agreement under our framework. Under our full framework (5 aggregation functions defined as Guard (P,) Cg=, k" arg max Oi awn (P,) O Chas (PP)) 1 2
T), agr (P,) = 22 where O = F.E,, recovers the error corr-ce forced choice distribution, Ω A(E,O,) recovers the error-corruptdd multi-label vector, and threshold parameter.
19

---

Notation Overview. To align with the main text while providing complete theoretical treatment, we provide reference for key notation used throughout this appendix:
Rating Task Primitives:
O: set of forced choice options (e.g.. (Yes, No))
Q: set of all possible response sets (e.g., [[Yes), (No),
A: binary matrix mapping response sets to options
Rating Aggregation:
aggregation function mapping rating distributions to rating vectors
Y: rating space containing all possible rating vectors from aggregation function a
Y: random rating vector (output of aggregation function applied random item)
y: rating vector realization (specific instance of Y)
m: agreement metric comparing rating vectors;
subscripts (e.g., MHR mKL)
for specific metrics
M: expected agreement over corpus; we use subscripts (e.g., MHR. MMSE) for specific metrics
Rating Model:
O forced choice distribution for item
0": stable response set distribution (uncorrupted by rater error). We this expression throughout the main text to refer to the setting where there no rater error the response set distribution (Fig.
0, E, observed response set distribution (after rater error)
S2; AO;: stable multi-label vector (uncorrupted by rater error)
Ω2 AEE,))) )) observed multi-label vector (corrupted by rater error)
E,: error transition matrix mapping stable to observed response sets
F; forced choice transition matrix mapping response sets to forced choice options
When rater error is absent (as in the main text), E; is the identity matrix, so 0; 0 and Ω = -
General Conventions:
Superscripts H and J distinguish human and judge system quantities when disam- biguation is needed. Subscript denotes teee-ieccific quantities; we omit this proofs for brevity.
C Full Theoretical Framework & Proofs
To complement the overview provided in 3 now provide detailed description of full probabilistic framework. After introducing further preliminaries (GC.T), we introduce our rating model (6
C.2 and use it establish necessary conditions for two definitions of judge system performance ( C.3 to yield consistent ranking of judge systems ($ C.4). Finally, we also provide proof's (8 C.5) and additional theoretical analysis of rank consistency under rater error in C.6
Connection to Main Text, The identifiability result referenced as "Theorem 3.1 in the main text corresponds to Theorem C.4 below. The rank consistency result corresponds to Theorem
C.1 Further Preliminaries
Human Raters. Let R denote a population of human raters, such as all target system users in geographic region, a demographic group (e.g., females over 45), or a set of domain experts (e.g.. licensed radiologists). We let R denote a random variable modeling the selection of raters from R.
Judge System. We assume black-box access to the judge system, Gjudge= which, given an input item t, returns output that is then mapped to response option in O under forced choice elicitation or Q under response set elicitation.
Probabalistic Rating Model. We model the distribution of human and judge system ratings via a joint distribution O' SJ OH SH R). Here, the random variables O' and 5 denote foreed choice and response set ratings, respectively, returned by the judge system for random
20

---

item T. Similarly, O" and S denote the forced choice and response set ratings, respectively, that randomly drawn rater R assigns to an item T. For each item i, let P P(O" S = denote the human rating distribution and let P/ POO", S T = t4) denote the rating distribution of Gjddee- We let Pr denote a rating distribution conditioned on a random item T'.
Aggregation Functions. We introduce aggregation functions A 3' to consolidate the full rating distribution into a rating vector. Generalizing our discussion of aggregation functions in the main text, under our full probabilistic framework, we let aggregation functions operate directly on conditional rating distributions P,. For example, applying a hard aggregation function y = Chara (P:) recovers a binary one-hot vector encoding single rating task option (e.g., Yes). The rating space y = (a(P:): P. E }} contains all rating vectors that can be recovered from an aggregation function.
We use aggregation functions to define random variables over rating vectors. Specifically, let Y = ((Pr) denote the random rating vector obtained by applying an aggregation function to the rating distribution of random item T. This random variable setup enables us to reason probabilistically about aggregated ratings e.g.. by computing the expected agreement between aggregated rating vectors recovered from humans and the judge system. Let (a" Y " ))" and (a Y ) denote the aggregation function, random rating vector, rating vector realization, and rating space for humans and Gudger respectively.
Human-Judey Agreement. Our full probabilistic model recovers human-judge agreement metrics ($ 3.2) Specifically, given the joint distribution PP(.,, we evaluate
M(Y y E [m(Y"' yH)],
where m : yd x 3 + IR is an agreement metric (c.g., Hit Rate, Cohen's k, KL-Divergence), The expectation is taken over the joint distribution of random items T and corresponding aggregated rating vectors yJ and Y
While Eq. ① assumes that we know the rating distribution, in practice, we only have access to small corpus of ratings. Therefore, we also estimate the agreement rate,
M 8 E y*)).
Above, we estimate ÿ from corpus of human ratings C {(T.,, Ry, To)l ~ IP)... We assume this corpus only contains forced choice ratings, as this is the format used in existing GenAl evaluations. For each item, we estimate y by repeatedly sampling a response from Gjudee
C.2 Full Rating Model
With this framework in place, we now develop model that decomposes sources of rating variation in the LLM--s-a-jjudee meta-evaluation pipeline. While main text (Fig. 4 specifically focuses two sources of rating variation: principled disagreement and forced choice selection effects, our full model also characterizes the affect of rater error on judge system selection. The setting discussed in the main text with no rater error can be recovered by removing the error term.
Our rating model decomposes the human rating distribution for each item: IP = PPS! SH OH, t.). To capture the potential for rater error, we distinguish between a human rater's 's stable response set S - i..., the options they would consistently endorse when carefully completing the rating task and the observed response set S they provide through response set elicitation (Figure 9) A rater's stable response set can differ from their observed response set if they fail to identify one more options that could reasonably apply to a rating instruction (or erroneously endorse others). We model rater error over response sets because this de-conflates the spurious effects of forced choice selection from error in the rating process. We describe differences between the stable and observed response set via an error matrix E, E R x (2), where each entry encodes the probability that a rater endorses S. given that their stable response set is S.,. We assume that error rates constant across all raters:
Assumption C.1 (Error Independence). S 1 S!,
While a rich literature exists on rater-dependent error modeling Klie et al. 2023 Gordon et al. 202T], we make this simplifying assumption to examine the aggregate effects of rating error on downstream evaluations of judge systems.
21

---

R
S
0!!
Figure 9: A causal Directed Acyelic Graph (DAG) for full model including rater error. The dashed node S. represents stable response sets that are unobservable when rater error occurs. When there is no rater enror (as assumed in the main text), S S and the dashed node can be omitted.
We use a transition matrix F, E RlO x|ol to represent how raters pick an option from their response set. Each element in F; contains the probability of rater selecting the kth option (e.g.. Yes) given that they would select the wth response set (e.g., both Yes and No). As with the error matrix, we also assume that F, is fixed across raters:
Assumption C.2 (Forced Choice Independence). OH T (S , RJ S |
Both E, and F; have reverse matrix, denoted as E& and F,, respectively, that encode conditional probabilities in the reverse direction. Entries in F", denote the probability of rater endorsing the vth (observed) response set (e.g.. Yes and No) given that they selected the kth forced choice option (e.g.. Yes). Entries in E denote the probability of a rater endorsing S., given that their observed response set is S...
Our rating model connects different representations of human rating variation (Fig. 9 The response distribution 0 P(S S.* (4) represents genuine differences in how population of raters interprets an item in rating task. This rating distribution, which is uncorrupted by error or forced choice selection effects, is our target parameter. In contrast, the forced choice distribution O P(O" = tx)) describes the probability distribution that is observed under rater error and forced choice selection effects. The following result shows that we can decompose response set distribution into rater error, forced choice selection effects, and the forced choice distribution:
Theorem C.3. (Rating Deconposition) Assume C.I and C.2 hold on IP((). Then O = (E,0,) and 0 O, holds for all conditional rating distributions P!f A.
This theorem shows how genuine differences in raters' interpretation of an item in a rating task propagates through error and forced choice selection. It also provides mechanism for recovering 0 from O by applying the reverse error and forced choice transition matrices. Given this decomposition, we might wonder when the response distribution is identifiable from O, The following result shows that this is only possible when a rating task is fully specified:
Theorem C.4 (Response Set Identifiably). Assume C.I and C.2 C.2 hold on PP.... Further, assume that O S Q and let E, be the identity matrix. Then 0 is identifiable from O; if and only if the rating task is fully specified.
This theorem shows that, even in an idealized setting with no rater error, an information loss occurs when compressing response set rating into a forced choice rating. practical implication that rating tasks should be fully specified when possible to enable direct recovery of 0 from O,,
C.3 Operationalizations of Judge System Performance
Our model for human rating variation establishes how operationalize the performance of judge system under indeterminacy. In particular, let " a' denote human and judge aggregation functions used to consolidate rating variation (i.c., represented via the forced choice response set distributions) into rating vector. Given a human-judge agreement metric m, we call = (a" a', m) an operationalization of performance. As we describe next, many such operationalizations could reasonably be used to validate a judge system. We describe each definition of performance by enumerating over aggregation functions:
Hard aggregation. The hard aggregation function is defined Ohhra (P:) e where eg- is an (0) diieensioon basis vector and A** arg max; OZ, is the mode of the forced choice distribution. Performance measures that rely on hard aggregation are consistent with categorical human-judh agreement metries (e.g., Krippendorff's ' a). Measures relying hard aggregation impose gobdllabel
22

---

assumption, and are the status quo in existing judge system validations Lu and Zhong 2024 J Jung
al.
2024| Dong al., 2024, Es al.
[2033] Dubois al. 2024 Bubeck et al. 2023] Zheng et al.
2023
Faisal et al.] 2024, Gu et al.
2024 [hakur et al. 2024 Li et al. 2024b, Chen et al. 2024b
Chiang et al. 2024 Dorner et al.
2024, Mirzakhmedova et 2024
Chaudhary et al. 2024
Kim
et al.
2024 Dettmers et al.
2024
Soft aggregation. The soft aggregation function (Pot O returns a probability vector over forced choice responses. Each entry Oi, represents the probability that the kth option is selected by a rater under forced choice elicitation. Operationalizations of performance that rely on soft aggregation are consistent with distributional human- judge agreement metrics (e.g.. KL--Diveggence). Prior work has proposed soft label aggregation with distributional agreement metrics for evaluating ML systems under indeterminacy [mma et al. [2220] PPeerson et al. [2019]. Collins et al. [2022]. However, soft aggregation seldom used in judge system validations.
Our rating model (6 connects these categorical and distributional operationalization of perfor- mance and multi-label operationalizations. Multi-label operationalizations provide more granular representation of rating variation over response set data. Let A (0, 1} be binary matrix indicating whether the kth option is in the uth response set. We define the multi-label vector as S2 A (E,07)) Each entry in Si describes the probability that rater selects the ktth option in their observed response set under response set elicitation. Let 22 = A (0,) denote the correspond- ing multi-label vector that is uncorrupted by rater error. Two additional aggregation functions consistent with multi-label vectors:
Hard Response Set. The hard response set (hrs) function Cars (P,) 1(22 2 7 maps the response set distribution to a binary multi-label vector. The kth entry of this vector is one if there is at least a T probability of response set containing option & being selected during response set elicitation. This aggregation function is consistent with measuring the coverage of a predicted judge system response in response set containing multiple "correct" options.
Soft Response Set. The soft response set (srs) function Oax (P.) = St directly returns the non- thresholded multi-label vector. Each entry f2 denotes the probability that rater endorsed the kth option during response set elicitation. Operationalizations performance that apply srs aggregation to the human rating distribution are consistent with continuous metrics such Mean Squared Error and Binary Cross Entropy.
Table in Appendix B lists many operationnlizzaions of performance that are consistent with these aggregation functions. This table also summarizes operationalizzaions commonly used in (1) LLM- as-a-judge validations and (2) prior work studying evaluation under indeterminacy. Note that for completeness, this table presents forced choice and response set agreement metrics with rater error affecting the forced choice distribution and multi-label vector. However, we can easily recover the setting in the main paper with no rater error by assuming E, the identity.
C.4 Ranking Judge Systems Under Competing Operationalizzaioon of Performance
Given that there are many ways of operationalizing performance under indeterminacy, it is unclear when one approach is preferable over another. One way to distinguish among competing opera- tionalizations is by examining their downstream impact on judge system validation: when do two operationaliiztions yield a consistent ranking of judge systems? We now use our framework to formally investigate this question.
W Let judge and Gudgee denote two judge systems described by their conditional rating distributions p.y. and IP,, respectively. We compare these systems with respect to an operationalization p via, ((2,W) Ep- (m(a((1)) a" (IPf)) m(a' (F a (IF4))
(3)
where Is represents the full joint distribution over responses returned by both judge systems and human ratings.
To formalize a comparison between two systems, we let 9 Gjddge
GW
judge denote that (2, W) 2 0.
For instance, when using Hit Rate with hard aggregation, agreement with majority vote over human ratings than W
implies that achieves greater Now, suppose that would like
For metrics where lower values indicate better performance, like KL-divergence, we invert the operation such that o Tp Gyrdgge > 6Z, W) S 0.
23

---

to compare judge systems under a different operationalization of performance, denoted by (a.", ax m.). The following condition describes when these two operationalizations are guaranteed to yield equivalent ranking of judge systems:
Definition C.5 (Rank Consistency). We say that p and p. are rank consistent if for all IP, G Zp G
Giudge Ep. Judge'
Judge
While there are many possible relationships between two definitions of performance, monotonicity captures one key property we might expect: when one system's 's performance improves with respect to p, it should also improve with respect to p. if the two definitions compatible. We formalize this notion in the following definition:
Definition C.6 (Monotone Transformation). p is a monotone transformation of p. if there exists monotone increasing function / such that vn, (ag (P:) af (Pf)) [(In(a' (P.), a (PP))) ))) for all (P. P E A
The following result shows that if two performance definitions are not monotone transformations of another, there exist judge systems and distribution human ratings such that the definitions will yield contradictory rankings:
Theorem C.7. (Necessary Condition for Rank Consistency) p is not a monotone transformation of Р., then p and p. are not rank consistent.
Theorem provides a useful tool for comparing definitions of performance: we can show that two definitions
rank consistent by demonstrating monotonicity violation.
We provide two examples of monotonicity violations in Appendix F The first shows a violation between Hit Rate (defined over O) and KL-Divergence (defined over O). The second shows violation between KL-Divergence (defined over O) and Mean Squared Error (defined over Ω2). This second example illustrates pernicious issue arising in underspecified tasks: using Theorem C.4 we can easily construct monotonicity violations by holding the forced choice distribution fixed while varying the response set distribution. This suggests that monotonicity, and by extension rank consistency, is unlikely to hold between definitions of performance defined over the forced choice distribution (i.e., categorical, distributional) and multi-label definitions.
C.5 Proofs
C.5.1 Theorem
Proof. The forward model O, = F,(,,0,) follows by the following factorizatior
P(ok (t)
Σ
P(ox Ser, (Sre,
0,u",
Σ
P((n 8y, P(ss.
(4)
Σ
P(ox su,t) P(sv sue,r,t) t) P( I)
e.0"
Σ
P(on su, s0, P(svv (s,,t) t) P(s [t)
(5)
e, J
P(ok sy,t) Σ P(s. (yr,t) Σ P(s It)
P((s
Σ k, Σ Eu.u- C
We omit from all subscripts for brevity.
24

---

Above. (4) holds by forced choice independence and (5) holds by error independence. The reverse
model 0: Ex OF follows by the following factorization:
(8 [t] Σ P((s 1r, OR, P(se r,, P(ox, (t)
Σ P( () P(S r, (rok, ()- P(ou, (t)
(6)
neek
P(s5 1,80,, P(T, P 10)) [P((= r. 1, () IP(r, (1ri801 11) Po(or (1)
Σ
P(r, 8v
P(r, ok
ne,k
Σ
P( Sor, 1) (P(r, Sy 18) P(ox su,t)
(7)
Su, t) (..t).P(ss P(so
( (P(se. 11))
(k, t) P(ox 111) Σ P(r, se (t) P(s.
Σ
P(s.- (t)
k.k
Σ PPss" Suz t) P(s I ok t) P(ox lt)
Σ> E ΣF OO
where 6 holds by forced choice independence and 77 holds by forced choice and error independence.
C.5.2 Theorem
Proof. We remove dependence on from all terms for brevity. begin, note that 0" is identifiable from O if and only if 0" E' (F, FO" is fully determined (where the system simplifies by taking as the identity matrix). The system system FO must be consistent because Theorem C.3 establishes solution. A consistent system with n. |O| equations and [2] unknowns is fully determined if and only rank n2
We will first show that Q {{ox} Σ. FR Yu {1, n because each column in F represents valid probability distribution; and (2) F = 0, Va # k. Vo E (1,...,,,) because A = 0 = F FR, = O. This implies that
+ implies that rank(F)= To begin, note that (I)
1= I= F a,v = ΣF.. a, + FN.O FR.
Vo E (1....,n.). n.).
a
axk
Thus, each singleton set for} S maps to standard basis vector e₁ R Further, because n, = n,, by definition of Q and C Q, each option must appear in exactly one set, giving us exactly n, n. distinct basis vectors. The rank of matrix is equal the number of linear independent column vectors. Because each of the standard basis vectors must be linearly independent, it follows that rank(F)= nl, n1..
We will show the reverse implication that rank = = e= [fou) og E O} by contradiction. Suppose there exists set S,, E Q containing more than option, i.e., |S] > 1. Let v denote the column of F corresponding to S,,. Since O Q for each option O2 E S, there exists a column be written
in F that is the standard basis vector Ck. shown above. Therefore, combination of these basis vectors: v = Σn dg where a = [0. 1]. This shows that column v is linearly dependent with the columns corresponding to singleton sets Og for og E S. This implies F cannot have n, linearly independent columns, contradicting rank(F) = fl.-
linear
25

---

C.5.3 Theorem C.7
Proof. Proof by contradiction. Let and p. denote pairs of performance definitions with increasing
cardinality (i.e., higher values being better). Let y = = yw a (PT (P/./) y
al (I24) ) denote random functions of T corresponding to definition p. Let Y.² = = (PT Y.W
a (P// ), Y. _ (P correspond definition Р.. Since p is not monotone transformation of p.. by definition there must exist distributions {(P:/? P.), (TP [f) e A x A corresponding realizations of these random variables satisfying
m(y² , < m(Y Y#),
m. (Y. Y. > mi Y. y.
Now suppose that IP* places all marginal probability mass over T on the i'th item i.e. P (T t;) = 1. Then:
6p W) = Ep. (Y² y - m((W y#] <
8p' Z2 W) Er. (Y.2 y (V." Y. >0
Thus, rank consisteney is violated because there exists a distribution P for which judge > p. judge judge 7p 9 Judge. This provides contradiction, proving the result.
but G
C.6 Rank Consistency Under Rater Error
Lemma C.8 (Rank Consistency of MSE (sTs/STs) Under Rater Error ). Let 0" and 0 E0" denote the stable and observed response set distributions for human raters 10 Let J.2 and J.W denote observed response set distributions for judge systems Judde and 9 judge where both judge systems have a rater error matrix 1.1 and J.W. that is the identity. Let A be the binary matrix mapping response sets to options and define 8* = MSE(St' " 222 ) MSE(SL', 52 J.W, as the difference in MSE under error-free conditions. The ranking of judge systems using MSE with soft response set aggregation is preserved under human rating error if and only if:
sign(((" - -0)- A''''" -00.WWW0"" -0":2))= sign(("))
(8)
This lemma provides conditions under which measuring the performance of a judge system against crror-corrupted versus frror-free human ratings yields consistent ranking of judge systems (when measured against MSE(srs/srs)). The condition essentially requires that the direction the error- induced shift in human ratings (0" f) matches the direction of the stable response set shift across judge systems -0 2.2 when projected to the multi-label space. If human rating error and judge system performance differences shift the response set distribution the same direction, the ranking of judge systems will be consistent for error-free and error-corrupted ratings. Conversely, rankings can invert under an inverse relationship.
E.q. (8) is satisfied in our experimental setup because the ensemble of judge system rating distributions is generated by adding random perturbations (i.c., uncorrelated with rater error) to the human stable response set vector. Thus no rater error (Figure 34 center) and rater error (Figure 34 right).
little change in the reliability of MSE (srs/srs) across settings with
Proof of Lemma C.8 For brevity, let M = ATA. The difference in judge system MSE measured against the multi-labe human rating vector Ω = A0" derived from the stable response set distribu-
1eWe omit subscript i from all terms for brevity. We also omit superscript h from human response set distribution, emorr and multi-labe label vectors where the context is clear.
26

---

tion is given by:
8 = MSE(S 22122) MSE(S S2
||000 10 ||000 A0 1
(0" - 0 0z MM'0 -0.2) - -00- - D0 M - )
(0")"M0' (0') -(042)TM00 + (0'2)"
- 000' + (00) "00'MWWWWWWW.. + +0. (0 (0" "MO..W
= - MM -(0 D...WYTM' Me"." +( " -..2)
Let S (E0) denote the multi-label vector recovered from the observed response set distribution. Applying the same derivation as above to the error-corrupeed MSE metric yields:
8 = MSE(Ω, 22 MSE((2, Ω2 -2-W)
(0'2)"TM0' 1. (w"W TM + 2(E0")² M(O"W 2(" -0".2)
Observe that the first two terms appear in both expansions. Thus we need to focus on the third term while showing the conditions required for rank consistency i.e., sign(0) sign(0").
Case 1: (G]udee better than judge under no rater error.) For both inequalities to hold, we need:
2(E0")² M(O (MO'" 8¹22) s
-g2.2)
(EO')² M(O.. -0.2) S (0')"M(O'. -1.2)
(0')TM((''M)) - -1 (EO") ( MM@" - -122) 20
(00) (E0*)"
JM (E -0.2) 20
(0" (20") 1212) 20
Case 2: 8" > 0 Judge is better than Judge under no rater error.). For rank consistency, we need 8 > O as well. Following similar steps, we get:
(GW
2 - 0².2) 2 2(0 (000))MM) MMO.. 0¹.2) -2(E0 T M(O - S "T000)" MM(O 11.2)
2(E0") "(E0')MM@' 1 - S 2(0") W
(0" '0-EO'''MO'' - - 20
27

---

Rating Task
Property
Citation
Ratings per Item
Civil Comments
Toxicity
Borkan et al.
2019
10+
MNLI
Entailment
Williams et al. 2017
100
SNLI
Entailment
[Bowman et al.
100
201.5
Q-NLI
Entailment
Nie et al. 2019
100
SummEval (Relevance)
Relevance
Fabbri et al.
2021
8
SummEval
Coherence
Fabbri et
2021
8
(Coherence)
SummEval
Factuality
Fabbri et al.
2021
8
(Consistency)
SummEval (Fluency)
Fluency
Fabbri et al.
2021
8
QAGS
Factuality
Wang et al.
2020
3
TopicalChat
(Uses Knowledge)
Uses Knowledge
Gopalakrishnan et el. 2023
3
TopicalChat
Understandable
Gopalakrishnan
3
(Understandable)
et al. 2023
Table 2: An overview of the eleven rating tasks included in our empirical analysis. Each task includes the property being evaluated and number of human ratings per item.
D Real-Data Experiments: Setup Details and Additional Results
Rating Task Configuration. Table 2 provides an overview of all rating tasks included in our experiments. Tables 3 illustrate the approach used to reconstruct the response set distribution from
the foreed choice distribution via the sensitivity parameter 3. Table 5 applies 5 to all rating tasks with
binary Yes/No options [Fabbri et al. 2021 Wang et al.] 2020 akrasakr et al. 2023].
Prompts. We provide forced-choice and response set variants of all prompts in Fig. 23 18 of $ D.I
The "no explanations" instruction included in prompts pertains to the final model output. It does not prohibit reasoning-enabled model (e.g., o3-Mini) from producing a reasoning trace at inference time.
Model Inference Settings. We sample all models with a temperature of 1.0. For all models, we also limit toxetokkns used for generation to 5. This low max token limit feasible because only few tokens are needed to provide forced-choice or response set rating (e.g.. "A", "BB"). When using a reasoning- enabled model (e.g., o3-Mini), we set the max token length for the reasoning trace to 1024 tokens.
Invalid Responses and Robustness Testing. For all rating tasks, assign null forced choice option E O and response set [0] E @ that are selected when judge system returns an invalid character. Probability mass assigned to this null option is penalized in human-judge agreement metrics. Because we specifically study factors in the human rating process that can confound meta- evaluation, we do not systematically study the influence of factors such as prompt formatting (Selar) et al., (2023 and option ordering (Pezeshkpour and Hruschka 2023) on our results. In practice, meta-evaluation designers may also want report such analysis as part of their validation pipeline.
Item Sampling. To match common LLL--a-a-judee meta- evetuaeval workflows that conduct analysis sub-sample all rating tasks 200 ratings per item. We randomly sample items for all rating tasks apart from civil comments, which is sampled via a stratified random sampling approach to select comments with an observed agreement level in the range (0.2, 0.5).
on small corpus of items,
Summ Eval Task Design. SummEval ratings tasks were originally collected on a 1-5 Likert scale. 2 1 in comparison to the number of ratings
Because this corresponds to many response sets 31
28

---

collected per item (8), we discretize the scale by assigning positive binary label if a rater selected 2 This discretization improves the finite sample stability of estimated agreement metrics ($G).
Cost and Inference Time. The total cost of running all models was 199.76. Each rating task took 30 minutes to when models were run in parallel, with the exception of Claude Sonnet 3.5, which took approximately 2 hours per rating task due to high API response latency.
Estimation Details. We now outline our approach for estimating human-judge agreement metrics via a finite sample rating corpus. For each item i, we estimate each term in our rating model:
1. Judge system: Estimate Ó (forced choice prompt), Ω2, = AO; (response set prompt) via empirical frequencies from judge responses.
2. Humans:
Estimate Ô via empirical frequencies from forced choice human ratings. Next, apply reverse rating model s2." (O,) where F; is recovered from the sensitivity parameter 3d
We then take the empirical average over the evaluation corpus to estimate the human jonannjudge agreement metric:
M( M( 1 mm " = 1 Σ mla' (O, )
where the third term illustrates the full expansion with the application of aggregation functions to the estimated rating terms. We estimate downstream performance metrics using the same approach.
Computation of MSE F. Our experiments reporting MSE F assume oracle knowledge of F' in step (2) of the estimation procedure outlined above. Our experiments reporting MSE F apply step (2). but recover 5 A (O,))) via F.. To estimate F" we sample an auxiliary corpus with 200 (forced choice, response set) paired ratings. We sample one paired rating for each item in the validation corpus. then use this auxiliary corpus to compute sample estimates of the conditional probabilities in the matrix F', where each entry denotes the probability of rater selecting the ęth response set given that they picked the kth forced choice option. We assume that F is fixed across items to improve sampling efficiency. This works well in practice in our setting (Fig(5).
Additional Experimental Results. Below, we report a version of our main findings with an expanded set of metrics (Fig. 10), sensitivity parameter estimates recovered from all models across all tasks (Fig. IT). and detailed rank analysis for pairs of performance metrics for all rating tasks (Figs. 22).
29

---

Response Sets
Options
(VT)
(T)
(N/U)
(VTTT)
(T,N/U)
(VT.N/U)
(VTTNNUU)
VT
0
0
0
0
T
1
0
0
0
0
N/U
3
B
0
0
0
0
Table 3: Civil Comments rating task [Borkan et al., 2019]. Description of how the sensitivity
parameter (3 is used construct the forced choice translation matrix that recovers the response distribution from the forced choice distribution. Positive options are O+ {Toxic, Very Toxic). VT Very Toxic, T Toxic, N/U = No/Unsure. For completeness. note that technically this illustrates the reverse forced choice translation matrix F' (forced choice + response set distribution) as opposed to the forward translation matrix F that maps the response set forced choice distribution (see $
Response Sets
Options
[E)
[N]
[C]
(E,
[E,
C}
[N, C)
(E, N, C)
E
0
0
0
0
N
0
1
0
0
0
C
3
0
0
Table 4: Natural Language Inference tasks (SNLL, MNLI). Description of how the sensitivity parameter (r) is used to construct the forced choice translation matrix that recovers the response set distribution from the forced choice distribution. E=Entailment, C=Contradiction, N=Neutral. Applies to SNLI Bowman et al. 2015) and MNLI [Williams et al. 2017 datasets.
Response Sets
Options
[Positive]
(Negative)
(Positive, Negative)
Positive
Negative
3H
3
Table 5: Binary classification tasks. Description of how the sensitivity parameter (0!1) is used to construct the forced choice translation matrix that recovers the response set distribution from the forced choice distribution. Applies to rating tasks in QAGS [Wang et al.) 2020), SummEval [Fabbri
al. 202T.
o-NLI [Nee al. 2019), and TopicalChat [Gopalakrishnan al. 2023
30

---

Results Grouped by Metric Type
Categorical Forced Choice vs. Distributional Forced Choice vs. Continuous Multi-Label
Consistency of Selected Judge
Continuous Multi-Label
-0.05
Relative Optimal
-0.10
-0.15
-0225
Mut-Label Continouus (sralsrs)
Muti-Label Continuous
F (arsiers)
Discrete
(aralars)
Continuus
MSE: (sralors)
Distributioal
Forcad Choice
Categorical
KL--Dvergence
Categorical
Hil Rale (hh)
al-DDvergence (h.) (s1s)
Distibutional
Kriopendorffs a (fh)
JS Diveevergence (sis)
Bias (MAE) of Selected Judge
-0.05
Coher's (NM)
Entropy (hujl (s/s)
Cress Entropy (UP) (a/s)
Relative to Optimal
-0.10
-0.15
-0.20
0.55
0.2
0.6
0.6
0.0
Sensitivity Parameter (B*))
Sensitivity Parameter (B")
Sensitivity Parameter (B*)
Figure 10: Main results with extended set of metrics (see Fig. 5 Center panel: Fleiss's performs
similarly to Krippendorff's a. Right panel: KL-Divergence (j.h) and Cross Entropy (j.h) perform
robustly and KL-Divergence (h.j) and Cross Entropy (h.j) perform poorly across settings.
31

---

Model
Civil Comments
Claude
Sonnet
DeepSeek
Chat
Claude
Haiku
MNLI
Llama
70B-Instruct
Mistral
Large
GPT
SNLI
o3-Mini
Mistral
Small
GPT
3.5-Turbo
ALPHA NLI
GPT
4o-Mini
SummEval (Relevance)
SummEval (Coherence)
SummEval (Consistency)
SummEval (Fluency)
QAGS
TopicalChat Uses Knowledge
TopicalChat Understandable
0.1
0.4
0.6
Sensitivity Parameter Estimate (8))
Figure 11: Task-specific sensitivity parameters (3:' recovered from all judge systems across tasks (non-abridged version of Fig. (6. Estimates vary across (1) judge systems within the same task, and (2) across tasks. This indicates heterogeneity in how judge systems respond to indeterminacy.
32

---

A--03
2-33
X--00
690
0.2
190
028
8.28
134
230
500
OSen 0MN1
Aee 0H407
Bet A083
4044
AUDVANII
KZNUU
APHANU
AUDUANU
ALPAAN
AJHWAL
A--03
E-33
K-33
1.00
0.25
0.2
923
075
500
COMA
AUPYANUI
XUNAM
KUNIU
AUNANU
KUNAA
ALMANU
ALPHAMU
0722
300
0.3
1.2
150
SMRT
0M4C11
Ina 2407
p4671
2441
6N0
AUXI
ALNAML
KUWAU
ANU
AUTUAAU
ALTIANU
ALNANU
AZHKKL
0.22
0.2
010
Croidaey
Danideyy
Cooleooy
Cootitey
UNC CSANY
Screet
708 iee-rolud
3.57uro
Claude
Mikuul
GF1
DongsSook
Mintrn
Chill
Figure 12: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the a-NLI rating task. Top two rows: 3 0, Bottom two rows: 33M 0.3.
33

---

X0-00
KCH33
67
KTH33
0.00
0.00
035
035
335
330
900
916
10.00
046
01E
Gl 6481
(4811
3.75
A'000
21-12
$7.00
6:2
134
1200
AR
PHU
OAL
DNU
DNU
130
005
046
3/4
1000
045
1/4
1200
050
3.2
MM4 40p
SMR71
6N0
100 2447
2951 4007
177
10-03
3--33
62-12
17 -13
0.36
3.50
00-00
67-33
089
5.00
0:0
212
650
1.00
13:00
Goodeny
IR A211
Gandeon
On done P11
Oascen 472
Colliny
Claude
Liae
GPT
Clauds
GPT
Hokki
Chal
Figure 13: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the SNLI rating task. Top two rows: 8 O, Bottom two rows: 3 0.3.
34

---

X0-00
AC-33
J0-00
K"H33
1.5
046
1.4
0.9
13.00
948
056
330
12.00
GlA 6AR1
206 8M80
OPL
07-11
AT-33
0324
3.58
0.70
0.4)
2.58
548
080
1023
120
2.00
0.80
WN1
M93
405
MILI
MAU
MIL
7.00
0.50
1.2
9103
2.23
0.00
000
1500
1200
100 2447
250 2007
10-03
2--33
62-62
17 -.3
60-03
67-13
1.4
0.38
330
946
6:00
13.00
948
2/8
1153
0.0
1200
Goodbey
IR 6211
2u 210
On
Claudo
Liaee
GPT
Clauds
GPT
Hokki
DoveeSook
Chal
Figure 14: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the MNLI rating task. Top two rows: 3 O, Bottom two rows: 3 0.3.
35

---

C-
2-31
0.0
C4
C-
P-
CT-SE
020
612
969
C4
98
1233
948
030
03
448
4g0
018914091
(665)
(145))
4aRg M911
(66)1
06451
(M55)
(545))
Cor
OutD min
633
0.00
040
1--10
0.41
0.00
039
1033
000
445
930
580
4.00
1123
6M
930
030
Colctens
Quedday
0409 611
Omeeh
CulCommrra
Ovt OveCcemane
P-
P--3
67-33
p-
A23
0.00
1.50
0880
0.9
645
000
650
010
1133
Des (M65)
Gen (MAS)
(055)
Go (645)
Gen
47-33
62
82
R-33
040
0.4
67-33
0.0
070
612
9.0
1333
0.00
633
Condanyy
Coaceeyyy
enly
Goedanyy
Concentt
d. EAP
Couctery
(oseee)))))) (72
Cal
Claade
Lirra
3,6
Claude
Matrul
GP"
Halk
Larg
DespBock
Chal
Smat
<3/m2
Figure 15: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the Civil Comments rating task. Top two rows: 3. 0. Bottom two rows: 3 0.3.
36

---

KC-33
80-00
K-333
0.9
028
3.23
04
4.00
038
122
Gl 6481
(4811
Agg
(06,00011
040
27-11
-0.00
A1113
0.0
134
2.00
035
3.5
630
032
830
KO49 W91
691
0650
0406
9405
0405
0603
0A26
1.8
4.00
032
0.0
928
932
4.04
K00
SAC F4)
I00 2407
10-03
33
62-63
(0 -33
8--33
022
6000
922
XDI
630
12001
Goodbey
WR A211
Canibeey!
Cauplng
Gondeon
(asom, 472
Callliyy!
On
Claudo
Liaee
GPT
Clauds
Mistrn
GPT
Hokki
DoveeSook
Chal
Figure 16: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the QAGS rating task. Top two rows: 3 0, Bottom two rows: 3 = 0.3.
37

---

67 P-
E
A
KT
0.0
3.0
922
4.00
924
1:2
Gen MAS1
2912
Coepecte 6P17
Con dan
Ban 1651
Uai Kikaiangg
Nanattar Use fronage
UMI Kiceage
375
020
042
022
500
022
144
<00g
X0-03
33
12-13
80-00
8--33
2.00
3.50
0.18
A.1
900
4.00
150
4.09
5:00
924
88
18 ed11
64401
GHS
210)
Tenuth
X1-03
3-
630
130
A00
022
6.00
22017
Cosiloo
(0200
Admit))
Sonnel3.
Claude
Metri
GPT
Chul
Figure 17: Ranking of models determined by colmman-judee agreement metrics versus downstream metrics for the TopicalChat "Uses Knowledge" rating task. Top two rows: 3 = Bottom two rows: 34 0.3.
38

---

AT ED
E
AT HEL
N but
A
020
K
0.16
311
1.0
010
924
124
Coeppote 6P17
Con dan
Underarda
Undmaar 200
000
3.2
632
924
070
connenyy
coededtf
compter7
X0-03
33
12
340
K--33
1.00
2.2
200
4/0
4.9
02E
322
0.9
670
230
500
IR 6211
6440)
AN
GHS
AM87
272
Tecarhal
171
005
230
024
000
009
2201
Cositoo
Donaliay
Comedetty
Digggna
Groamesy
dsard)
Scrnet3.5
706-199hu2
357uur
Claude
Matri
GPT
Figure 18: Ranking of models determined by human judge agreement metrics versus downstream metrics for the TopicalChat "Understandable" rating task. Top two rows: 3 = 0, Bottom two rows: 3. 0.3.
39

---

9:5
67
-333
640
Cull
60 3
10 E
C-110
A1
nas
045
080
080
AM5
1423
432
MMGS
MACI
Cisppteeelll (611)
SMCT1
(607
DMAT1
SvenCi
SorenD
SunnCe
SunDe
SureCrr
8112
946
4.09
1033
045
C46
0.0
6:00
423
AD1
1233
640
020
1.00
A3)
#1-11
17-11
2-11
6--11
445
4:5
940
635
1123
CM
930
63k
6M572
0HA)
(M53)
(M65)
(p9sn) 412
(M10
Suensee
Acerke
Sune
4.1
1133
A23
085
040
000
640
432
6d5
030
Conol
(.p00p9)) Duiddees
Quisder!!
Gougen
Caldees
Quildery
Mec (2454)29))
Colaber
Somnet5 3.
706 robltt
Chaake
Mitril
46-6s
DeepSeak
Marri
Figure 19: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the SummEyal "relevance" rating task. Top two rows: 3 0, Bottom two rows: 33 0.3.
40

---

9/6
235
60 31
10 EG
C-110
0.99
9A23
045
0.40
AA
1239
040
600
0.6
433
M4
AM
1439
080
0o001t6001 6011
MMGS
S0F1
SMG)
BMS1
MAC)
1671
SMCT
RaY MM10MA1
DMCT1
Oa
SvnnCC
SunnCe
SunnCC
040
040
045
823
250
045
0.0)
A31
6.00
9a0
611
022
330
000
6M
045
035
085
(0009
#1-11
M55
17-11
-111
0.4
K7-11
0.0
1X23
045
M5
1229
0.0
0.00
040
649
3.00
1433
OMAS,
(M55)
MAS)
0465)
(M65)
(URAN) 472
0010
Sunen And
Surenie
0.3
435
2.50
4.13
045
4.15
828
0.0
0.5
0.05
Rdy
Adc
Comol 6091
Qudidies
Dosseer
Qvisdeey
Caldbers
Quildeyy
MBc (2244)20))
Somnet 3.5
706 roblut
Chaado
Mitrul
46-6s
DeepSeak
Marri
Figure 20: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the SummEval "coherence" rating task. Top two rows: 3 0. Bottom two rows: 33 = 0.3.
41

---

AT
A0 -333
#0 3--36
Jo EG
080
685
0:20
0.9
040
XBI
639
400
641
64
0d0
030
61
1323
04
6011
MMG1
9MD1
MAC1
SMOT1
Oa drn
OMC1
SunDe
SurnCr
De
611
1.0
643
635
023
A33
030
0.0
080
612
080
64
040
640
1.00
022
1123
(0009)
Companst
pari
632
#1-11
10
17-11
-111
6X0
6:30
6--11
p-
046
67
0.55
633
445
3.0
0.00
0.30
418
6m
4.00
1023
net
0.00
0.5
4M52
OMA)
(MA5)
0M651
(A5)
(M65
472
SuenGee
Scen Kre
SurenGa
Sune
1.00
023
033
6:30
435
411
A4
Qut
640
RAA
423
(ap0019)) Quiddees
Qvisdeer
Quildery
Colabee
Somnet 3.5
708 robutt
Chaako
Mitril
46-6s
DeepSeak
Marri
Figure 21: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the SummEval "fluency" rating task. Top two rows: 3 0, Bottom two rows: 33 0.3.
42

---

#T -
10
10-86
22-33
17 E.
#7 3
J0 EC
6-
200
0.0
023
045
0.4
ATX
1339
099
0.0
933
965
640
1.00
6011
BMGY
MAS1
M91
MACI
SMCT
Rar MMGG
OMCT
SoreCCa (5
Soren
SunnCe
SmeDe
SanDD
SureCre
Done
0.99
0.0
0.40
1329
080
6.00
1433
085
0.50
AT
1423
080
ALOUG
pari
1-11
050
045
0:4
0.5
1233
030
0.00
A30
1129
MM5 Dag
OHAB
(MA5)
962
GMAS
54g
MAS
(M45)
472
Suensee
SurenGe
0.09
A33
035
0.30
0.00
1333
045
412
6.00
6/6
6.00
083
4/8
1533
Dumsbey P3y
K0p00190 Duiddees
Quildeyy
MBc (2454)29))
Conors
Somnet 3.5
708 rollut
Chaado
Mitrul
46-6s
DeepSeek
Marri
Figure 22: Ranking of models determined by human-judge agreement metrics versus downstream metrics for the SummEval "consistency" rating task. Top two rows: 3 = 0, Bottom two rows: 33 0.3.
43

---

D.1
Rating Task Prompts
Response Set (RS) Rating Instructions Civil Comments
Rate the toxicity of the following comment:
"(comment_texx) text)"
Select ALL options that could reasonably apply:
A. Very Toxic Very hateful, aggressive, disrespectful; very likely make you leave a discussion
B. Toxic Rude, disrespectful, or unreasonable; somewhat likely to make you leave discussion
C. Not Toxic Not rude or disrespectful; unlikely to make you leave discussion
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "ABC" "A")
Use only letters A, B, and/or C
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", "B_ "_" are invalid responses)
No explanations
Forced Choice (FC) Rating Instructions Civil Comments
Rate the toxicity of the following comment: "[comment text)"
Select the ONE option that best applies:
A. Very Toxic Very hateful, aggressive, disrespectful; very likely to make you leave a discussion
B. Toxic Rude, disrespectful, unreasonable; somewhat likely to make you leave
discussion
C. Not Toxic
Not rude disrespectful; unlikely to make you leave discussion
RESPONSE FORMAT:
Provide only single letter: A, B, or C
The letter must be the first character in your response
No spaces, punctuation, other characters
No explanations
Figure 23: Rating instructions for Civil Comments "toxicity" annotation task.
44

---

Response Set (RS) Rating Instructions SNLI
Given context, a statement can be:
Definitely correct (Entailment); or
Definitely incorrect (Contradiction); or
Neither (Neutral).
Your goal is choose the correct categories for given pair of context and statement. An automatic detector will estimate your annotation accuracy on this task. If your estimated accuracy is too low, you might be disqualified.
EXAMPLES:
Context: A guitarist is playing in band. Statement: Some people are performing.
Answer: The statement is definitely correct.
Now provide a response to the following example:
Context: "(context)"
Statement: "[statement]"
Select ALL options that could reasonably apply:
A. Entailment- Definitely correct
B. Neither Neutral
C. Contradiction Definitely incorrect
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "ABC" or "A")
Use only letters A, B, and/or C
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", or "B_ are invalid responses)
No explanations
Foreed Choice (FC)
Rating Instructions SNLI
Given context, a statement can be either: Definitely correct (Entailment); or
Definitely incorrect (Contradiction); or Neither (Neutral).
Your goal is choose the correct category for given pair of context and statement. automatic detector will estimate your annotation accuracy on this task. If your estimated accuracy is too low, you might be disqualified. you feel uncertain about some examples, just choose the best category you believe the statement should be in.
EXAMPLES:
Context: A guitarist is playing in band. Statement: Some people are performing.
Answer: The statement is definitely correct. Now provide a response to the following example:
Context: "[context)"
Statement: "[statement]"
Select ONE option that best applies:
A. Entailment Definitely correct
B. Neither Neutral
C. Contradiction Definitely incorrect
RESPONSE FORMAT:
Provide only a single letter: A, B, or C
The letter must be the first character in your response
No spaces, punctuation, or other characters
No explanation
Figure 24: Rating instructions for SNLI natural language inference task.
45

---

Response Set (RS) Rating Instructions MNLI
Given context, a statement can be:
Definitely correct (Entailment); or
Definitely incorrect (Contradiction); or
Neither (Neutral).
Your goal is choose the correct categories for given pair of context and statement. An automatic detector will estimate your annotation accuracy on this task. If your estimated accuracy is too low, you might be disqualified.
EXAMPLES:
Context: A guitarist is playing in band. Statement: Some people are performing.
Answer: The statement is definitely correct.
Now provide a response to the following example:
Context: "(context)"
Statement: "[statement]"
Select ALL options that could reasonably apply:
A. Entailment- Definitely correct
B. Neither Neutral
C. Contradiction Definitely incorrect
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "ABC" or "A")
Use only letters A, B, and/or C
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", or "B_ are invalid responses)
No explanations
Foreed Choice (FC)
Rating Instructions MNLI
Given context, a statement can be either: Definitely correct (Entailment); or
Definitely incorrect (Contradiction); or Neither (Neutral).
Your goal is choose the correct category for given pair of context and statement. automatic detector will estimate your annotation accuracy on this task. If your estimated accuracy is too low, you might be disqualified. you feel uncertain about some examples, just choose the best category you believe the statement should be in.
EXAMPLES:
Context: A guitarist is playing in band. Statement: Some people are performing.
Answer: The statement is definitely correct. Now provide a response to the following example:
Context: "[context)"
Statement: "[statement]"
Select ONE option that best applies:
A. Entailment- Definitely correct
B. Neither Neutral
C. Contradiction Definitely incorrect
RESPONSE FORMAT:
Provide only a single letter: A, B, or C
The letter must be the first character in your response
No spaces, punctuation, or other characters
No explanation
Figure 25: Rating instructions for MNLI natural language inference task.
46

---

Response Set (RS) Rating Instructions AlphaNI.I
Given observations (O-Beginning and O-Ending), and two hypotheses (H1 and H2), your goal is to choose the hypotheses that are likely to cause O-Beginning to turn into O-Ending. automatic detector will estimate your annotation accuracy this task. If your estimated accuracy is too low, you might be disqualified.
EXAMPLES:
O-Beginning: Jenny cleaned her house and went to work, leaving the window just crack open.
HI: A thief broke into the house by pulling open the window.
H2: Her hushand went home and close the window.
O-Ending: When Jenny returned home she saw that her house was mess.
Answer: H1.
Now provide response to the following example:
O-Beginning: "/o_beginning""
HI: "(HI)"
H2: "(H2)" O-Ending: "(o ending)"
Select ALL options that could reasonably apply:
A. H1
B. H2
RESPONSE FORMAT:
Provide only the sequence of letters corresponding to response options (e.g., "AB"
"A")
Use only letters A or B
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", or "B are invalid responses)
No explanations
Forced Choice (FC) Rating Instructions AlphaNLI
Given two observations Oo-Beginning and O-Ending). and two hypotheses (H1 and H2), your goal is to choose one of the hypotheses that is more likely cause O-Beginning turn into O-Ending. An automatic detector will estimate your annotation accuracy on this task. If your estimated accuracy is low, you might be disqualified. If you feel uncertain about some examples, just choose the best category you believe the statement should be in.
EXAMPLES:
O-Beginning: Jenny cleaned her house and went to work, leaving the window just a crack open.
H1: A thief broke into the house by pulling open the window.
H2: Her husband
home and close the window.
O-Ending: When Jenny returned home she saw that her house was mess.
Answer: H1.
Now provide response to the following example:
O-Beginning: (o__beginning""
H1: "(H1)"
H2: "(H2)" O-Ending: "(o_ ending]"
Select ONE option that best applies:
A. HI
B. H2
RESPONSE FORMAT:
Provide only a single letter: A or B
The letter must be the first character in your response
No spaces, punctuation, or other characters
No explanation
Figure 26: Rating instructions for AopaaNl reasoning task.
47

---

Response Set (RS) Rating Instructions SvmmEval Relevance
You will be given one summary written for news article. Your task is to rate the summary metric. Please make you read and understand these instructions carefully.
Evaluation Criteria:
Relevance selection of important content from the source. The summary should include only important information from the source document. Penalize summaries which contain redundancies and excess information.
Evaluation Steps:
Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant redundant information it contains.
4. Select ALL options that reasonably apply, based different plausible interpretations the rating criteria.
Now provide a response the following example:
Article: "[article)"
Summary: "(summary)"
Select ALL options that could reasonably apply:
A. Relevant The summary captures the main points effectively with minimal redun- dancy
B. Not Relevant The summary misses key points or contains excessive irrelevant information
RESPONSE FORMAT: [omitted for brevity]
Forced Choice (FC) Rating Instructions SummEval Relevance
You will be given one summary written for news article. Your task is to the summary on one metric. Please make sure you read and understand these instructions carefully.
Evaluation Criteria:
Relevance selection of important content from the source. The summary should include only important information from the source document. Penalize summaries which contain redundancies and excess information.
Evaluation Steps:
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant redundant information it contains.
4. Select ONE option that best applies.
Now provide a response to the following example: Article: "[article)"
Summary: "(summary]"
Select ONE option that best applies:
A. Relevant The summary captures the main points effectively with minimal redun- dancy
B. Not Relevant The summary misses key points or contains excessive irrelevant information
RESPONSE FORMAT: (omitted for brevity]
Figure 27: Rating instructions for SummEval "relevance" rating task.
48

---

Response Set (RS) Rating Instructions SymmEval Coherence
You will be given one summary written for news article. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions carefully.
Evaluation Criteria:
Coherence the collective quality of sentences. We align this dimension with the DUC quality question of structure and coherence whereby the summary should be well-structured and well-organized.. The summary should not just be a heap related information, but should build from sentence to coherent body of information about topic. Evaluation Steps:
Read news article carefully and identify the main topic and key points.
2. Read the summary and compare it the article. Check if the summary the main topic and key points of the news article, and if it presents them in clear and logical order.
3. Select ALL options that reasonably apply, based different plausible interpretations of the rating criteria.
Now provide a response to the following example: Article: "(article)"
Summary: "(summary)"
Select ALL options that could reasonably apply:
A. Coherent
B. Incoherent
RESPONSE FORMAT: [omitted for brevity]
Forced Choice (FC) Rating Instructions SummEval Coherence
You will be given one summary written for news article. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions carefully. Evaluation Criteria:
Coherence the collective quality of all sentences. We align this dimension with the DUC quality question of structure and coherence whereby the summary should be well structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to a coherent body of information about topic. Evaluation Steps:
1. Read the news article carefully and identify the main topic and key points.
2. Read the summary and compare to the news article. Check if the summary covers the main topic and key points of the news article, and if it presents them in clear and logical order.
Select ONE option that best applies.
Now provide response the following example: Article: "(article)"
Summary: "( "(summary]"
Select ONE option that best applies:
A. Coherent
B. Incoherent
RESPONSE FORMAT: [omitted for brevity]
Figure 28: Rating instructions for SummEval "coherence" rating task.
49

---

Response Set (RS) Rating Instructions SvmmEval Consistency
You will be given one summary written for news article. Your task is to the summary on one metric. Please make sure you read and understand these instructions carefully. Evaluation Criteria:
Consistency the factual alignment between the summary and the summarized source. factually consistent summary contains only statements that are entailed by the source document. Penalize summaries that contain hallucinated facts. Evaluation Steps:
1. Read the article carefully and identify the main facts and details it presents.
2. Read the summary and compare to the article. Check if the summary contains any factual errors that are not supported by the article.
3. Select ALL options that reasonably apply, based on different plausible interpretations of the rating criteria.
Now provide response to the following example: Article: "[article))
Summary: "(summary]" J
Select ALL options that could reasonably apply:
A. Consistent
B. Inconsistent
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "AB" "A")
Use only letters A or B
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", "B_" invalid responses)
No explanations
Forced Choice (FC) Rating Instructions SummEval Consistency
You will be given one summary written for news article. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions carefully. Evaluation Criteria:
Consistency the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Penalize summaries that contain hallucinated facts. Evaluation Steps:
1. Read the news article carefully and identify the main facts and details it presents.
2. Read the summary and compare to the article. Check the summary contains any factual errors that not supported by the article.
3. Select ONE option that best applies.
Now provide response to the following example: Article: "(article)" Summary: "(summary)"
Select ONE option that best applies:
A. Consistent
B. Inconsistent
RESPONSE FORMAT:
Provide only a single letter: or
The letter must be the first character in your response
No spaces, punctuation, or other characters
.
No explanation
Figure 29: Rating instructions for SummEval "consistency" rating task.
50

---

Response Set (RS) Rating Instructions SummEval Fluency
You will be given one summary written for news article. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions carefully. Evaluation Criteria: Fluency the quality the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
Evaluation Steps:
1. Read the summary carefully.
2. Assess the grammar, spelling, punctuation, word choice, and sentence structure.
3. Select ALL options that reasonably apply, based different plausible interpretations of the rating criteria.
Now provide response the following example: Article: "[article)"
Summary: "( "(summary)"
Select ALL options that could reasonably apply:
A. Fluent The summary has good grammar, appropriate word choice, and flows naturally
B. Not Fluent The summary has errors that affect readability or sound unnatural
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "AB" "A")
Use only letters A or B
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", "B_" are invalid responses)
No explanations
Forced Choice (FC) Rating Instructions SummEval Fluency
You will be given one summary written for news article. Your task is to rate the summary on one metric. Please make sure you read and understand these instructions carefully. Evaluation Criteria: Fluency the quality the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
Evaluation Steps:
1. Read the summary carefully.
2. Assess the grammar, spelling, punctuation, word choice, and sentence structure.
3. Select ONE option that best applies.
Now provide a response to the following example: Article: "(article)"
Summary: "(summary)" Select ONE option that best applies:
A. Fluent The summary has good grammar, appropriate word choice, and flows naturally
B. Not Fluent The summary has errors that affect readability or sound unnatural
RESPONSE FORMAT:
Provide only single letter: A or B
The letter must be the first character your response
No spaces. punctuation, or other characters
No explanation
Figure 30: Rating instructions for SummEval "fluency" rating task.
51

---

Response Set (RS) Rating Instructions QAGS
In this task, you will read article and a sentence.
The task is to determine if the sentence is factually correct given the contents of the article. Many sentences contain portions of text copied directly from the article. Be careful some sentences may be combinations of two different parts of the article, resulting in sentences that overall aren't 't supported by the article. Some article sentences may seem out of place (for example, "Scroll down for video").. If the sentence is copy of an article sentence, including one of these sentences, you should still treat it as factually supported. Otherwise, if the sentence doesn't I make sense, you should mark it as not supported. Also note that the article may be cut off at the end.
Now provide response to the following example:
Article: "[article)"
Sentence: "[sentence]"
the sentence supported by the article? Select ALL options that could reasonably apply:
A. Supported The sentence is factually correct given the contents of the article
B. Not Supported The sentence contains facts not supported by the article
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "AB" or "A")
Use only letters A B
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" "BB", "B are invalid responses)
No explanations
Forced Choice (FC) Rating Instructions QAGS
In this task, you will read article and sentence. The task to determine the sentence is factually correct given the contents of the article. Many sentences contain portions of text copied directly from the article. Be careful some sentences may be combinations of two different parts of the article, resulting in sentences that overall aren't 't supported by the article. Some article sentences may seem out of place (for example, "Scroll down for video"). If the sentence is a copy of an article sentence, including of these sentences, you should still treat it factually supported. Otherwise, if the sentence doesn't t make sense, you should mark it as not supported. Also note that the article may be cut off at the end.
Now provide a response to the following example:
Article: " "(article)"
Sentence:
" "[sentence)"
the sentence supported by the article? Select ONE option that best applies:
A. Supported The sentence is factually correct given the contents of the article
B. Not Supported The sentence contains facts not supported by the article
RESPONSE FORMAT:
Provide only a single letter: or
* The letter must be the first character in your response
No spaces, punctuation, or other characters
.
No explanation
Figure 31: Rating instructions for QAGS "factual consistency" rating task.
52

---

Response Set (RS) Rating Instructions Toppcal Chat Uses Knowledge
Given a conversation and an interesting fact, your task is to rate how well the response uses the provided fact. Please make sure you read and understand these instructions carefully. Evaluation Criteria:
Given the interesting fact that the response is conditioned on, how well does the response use the fact? Evaluation Steps:
1. Read the conversation context, fact, and response carefully.
2. Assess whether the response incorporates references the provided fact.
3. Select ALL options that reasonably apply, based on different plausible interpretations of the rating criteria.
Now provide response the following example: Fact: "[fact]"
Context: "[context)"
Response: "[response)"
Select ALL options that could reasonably apply:
A. Uses Knowledge The response clearly uses references the fact
B. Doesn't Use Knowledge The response does not mention refer the fact all
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "AB" "A")
Use only letters A or B
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", "B_" are invalid responses)
No explanations
Forced Choice (FC) Rating Instructions Topical Chat Uses Knowledge
Given a conversation and an interesting fact, your task is to rate how well the response uses the provided fact. Please make sure you read and understand these instructions carefully. Evaluation Criteria:
Given the interesting fact that the response is conditioned on, how well does the response use the fact? Evaluation Steps:
1. Read the conversation context, fact, and response carefully.
2. Assess whether the response incorporates references the provided fact.
3. Select ONE option that best applies.
Now provide a response to the following example: Fact: "(fact)"
Context: "(context)"
Response:
" "(response)"
Select ONE option that best applies:
A. Uses Knowledge The response clearly uses references the fact
B. Doesn't Use Knowledge The response does not mention or refer to the fact at all
RESPONSE FORMAT:
Provide only single letter: A or B
The letter must be the first character your response
No spaces. punctuation, or other characters
No explanation
Figure 32: Rating instructions for Topical Chat "uses knowledge" rating task.
53

---

Response Set (RS) Rating Instructions Topical Chat Understandable
Given conversation and response, your task is rate whether the response is understand- able in the context of the conversation. Please make sure you read and understand these instructions carefully.
Evaluation Criteria:
the response understandable in the context of the history? (Not if it's on topic, but for example if it uses pronouns they should make sense)
Evaluation Steps:
1. Read the conversation context, fact, and response carefully.
2. Assess whether you understand what the response is trying to communicate.
3. Select ALL options that reasonably apply, based on different plausible interpretations of the rating criteria.
Now provide response the following example: Fact: "[fact]"
Context: "[context)"
Response: "[response)"
Select ALL options that could reasonably apply:
A. Understandable You know what the person is trying to say
B. Not Understandable The response difficult understand
RESPONSE FORMAT:
Provide only the sequence of letters (e.g., "AB" "A")
Use only letters A or B
No spaces, punctuation, or other characters
Do not provide the same letter more than once (e.g., "AA" or "BB", "B_" invalid responses)
No explanations
Forced Choice (FC) Rating Instructions Topical Chat Understandable
Given a conversation and response, your task is to rate whether the response is understand- able in the context of the conversation. Please make sure you read and understand these instructions carefully.
Evaluation Criteria:
Is the response understandable in the context of the history? (Not if it's topic, but for example if it uses pronouns they should make sense)
Evaluation Steps:
1. Read the conversation context, fact, and response carefully.
2. Assess whether you can understand what the response is trying to communicate.
3. Select ONE option that best applies.
Now provide response the following example: Fact: "[fact)"
Context: "(context)"
Response: "[response]"
Select ONE option that best applies:
A. Understandable You know what the person is trying to say
B. Not Understandable The response difficult understand
RESPONSE FORMAT:
Provide only a single letter: or
* The letter must be the first character in your response
No spaces, punctuation, or other characters
.
No explanation
Figure 33: Rating instructions for Topical Chat "understandable" rating task.
54

---

E Synthetic Data Experiments: Setup Details and Results
In the main text, we claim that differences in how humans and LLMs resolve indeterminacy in forced- choice rating tasks heavily bias LLM-as-a-judge validations. In this section, we provide evidence for this claim via synthetic experiments. These experiments examine how different operationalizations of hungn-judge agreement respond to forced choice selection effects. These experiments further characterize the joint effects of forced choice selection and rater error (see & for the full model).
E.1 Experiment Design.
Human Rating Distribution. For each item, we sample the response set distribution 0 Dir( (o)) We let denote the probability that a rater selects an observed response set that differs from their stable response set. We construct the error matrix E, such that diagonal entries denote the probability of no rating error (1 - c) and off-diagonal entries denote the probability of rating error (c). We use a skew parameter y7 to control how errors distributed across response sets. We let k 0 denote the index of the option used to categorize items (e.g.. as "toxic"). The skew parameter controls whether errors systematically favor (1) 0) or disfavor (1) 0) response sets containing option k
We model F by sampling an exponential decay function F. o exp(->" rrk) Here, rk denotes the rank (low to high) of the kth option in the uth response set and 7 controls the extent to which forced choice ratings are biased towards low-index options (e.g., including the index used to
categorize options = 0). We compute the forced choice distribution via O = F." (E (0"
Judge Rating Distribution. We model judge systems by sampling an ensemble of distributions with varying similarity to the human rating distribution. We control the deviation of the 2th judge's rating distribution via '²= ~ (( mmax). We then sample the 2th judges' response set distribution by
applying 0% .7 II ( + ce"),, where ex (0, (o 2 II) and IIA projects onto the probability simplex. We sample F following the same procedure used sample the human rating distribution
and. Because judge systems are not affected by rater error, we directly compute O = F (0
E.1.1 Measuring Asymmetry Human and Judge System Responses to Indeterminacy
To measure the similarity between how humans and the judge system respond to indeterminate items, measure whether a rater tends systematically favor or disfavor the option used to categorize each item - e.g., as "toxic", "factually inconsistent", or "irrelevant" given its inclusion in a response set. This metric, which we refer as forced choice selection effects, is measured via:
1 3 SEQ
r
P(O P(O
oz on ES,
1
where S E is a response set, and Q+ E Q denotes all response sets containing the option at index k = 0. This metric is interpreted similarly to an odds ratio, and has the interpretation:
r describes a setting where a rater resolves indeterminacy by selecting a random option. r > 1 describes setting where a rater tends to select the positive option e.g., "toxic", "factually inconsistent" when they identify multiple options as correct for an item.
* r < 1 describes setting where rater tends to select the negative option e.g.. "not toxic". "factually consistent" when they identify multiple options as correct for an item.
We let n and J denote human and judge system forced choice selection effects, respectively. We then leverage this metric to construct two conditions:
sign(I" - 1) = sign 1). Symmetric selection effects occur when, average, humans and the judge system resolve indeterminacy in the same way - ie., by favoring or disfavoring the option used to categorize each item.
Symmetric selection effects:
Asymmetric selection effects: sign(I"" - 1) # sign(T". - 1). Asymmetric selection effects occur when humans and the judge system resolve indeterminacy differently - i... by selecting different forced choice options given that they identify multiple as correct.
Experimental Parameters. We construct sampling parameters for humans and the judge system (7) such that I' ~ {0.5, 2). We run all experiments with 50 judge systems. We use 100 items
55

---

Underspocified Symmetric Grouped Metrics
Underspecified Asymmetric Grouped Metrics
Underspecifled Asymmetric Categorical Metrics
Underspecffld Asymmetric Distributional Metrics
Fully Spedfied
Grouped Metrics
1.00
Condistency
0.90
0.80
Abcolute Error)
0.60
0.20
Estimation Bias
Canagarica ea
HB tate thro a
KLDi Bo Jod
0.15
purr dbrel
3od
(Mcan MccanArroluu
Coner's 1h7o
0.10
Disnnca Contiaacss
C67U) (aut JG0 tos)
0.05
0.00
3 10 20 50100 Ratings per item
5.5 10 50100 Ratings per Item
3 10 20 50100 Ratings per Item
5 10 50100 Ratings per ktm
10 20 50100 Ratings per item
Figure 34: Downstream performance of judge systems selected using different human-judge agree- ment metrics. The X-axis shows ratings-per-iten used to estimate the human rating distribution. Results show that categorical metrics perform poorly under asymmetric FC selection effects (col 3-4).
Fully Specificd
Undorspacified (Asymmetric)
Fully Specified
*
*
cileenpecified (Symmetric)
Underspecified (Asymmetric)
Colten's (veal
0.900
Consitency (c)
Agresment
0.80-
0:25
0.6
0.70
0.20
0.2
0.0
0.50
Sammar
Sporros5 pa 6.3
Estimation Bas
Agrecmant Metric
MAE 1g
0.4
MSE (crstais)
10.4
0.02
Hit Rate (hh)
0.0
Cahen's i (hnn
cEghi (srs)
cEihal (ste)
MSE (arshur)
0.82
0.00
2.00
0.30
0.85
0.50
0.35
1.00
0.70
0.80
0.30
Downstrsam
Downstream
Figure 35: Correlation (p) between judge rank-
Figure 36: Saturation effects with categorical met- rics under asymmetric forced choice selection ef- fects. Fully specified rating tasks (left column) and MSE (srs/srs) (bottom row) recover the optimal judge system (shown via star). Categorical met- rics recover four equally performant judge systems under asymmetric selection effects (top right).
ings from downstream vs. human- judge agree- ment metries. Categorical metrics show low cor- relation under asymmetric FC selection effects.
in all experiments and select option and response set configurations satisfying: 2 < |0| I 10, 2 <<<
< 30,
| Q. We let min 0.02 and Omax .4 when sampling judge systems.
E.2
Results.
Finding When human raters resolve indeterminacy differently from judge systems, agree- ment metries measured against forced choice ratings yield sub-optimal selections. As shown in the right three columns of Figure 34 categorical human-judge agreement metrics select sub-optimal judge systems when (1) rating tasks are underspecified and (2) selection effects are asymmetric. In contrast, we observe that all agreement metrics perform similarly when the rating task is fully specified (Fig. 34] col 1) or underspecified with symmetric selection effects (Fig. [4] col 2). Figure 35 corroborraes these findings by indicating weak Spearman correlation between categorical human- judge agreement metrics and downstream performance metrics under asymmetric selection effects. Figure illustraas the mechanism driving instability of categorical metrics. When a rating task underspecified, categorical agreement metries do not yield a unique "optimal" judge system with
56

---

undlerspccifdd Rating Taok -- n=1
Underspacifed Baiing Yaok M 2, a.5
Underspecifted Rating lask Pia, P
Fully Specitee Rating Task
0.95
0.90
0.55
0.80
0.75
0.70
Underspecifedd Task
Underspecifidd | 2
Undorspedtidd plm.5, 5,
[- p-
Fully Specified Rating Task
0.14
Emror Scew E1)
0.12
22
Blss MEE
0.00
0.06
0.04
0.02
0.0
0.0
0.0
0.2
Figure 37: Joint effects of forced choice selection and rater error on the reliability of HR (h/h). Y-axis shows downstream performance of judge systems selected via HR (h/h) with 100 aatings-per-itenr We parameterize rater error magnitude via 6, which controls the probability that observed and stable response sets differ ($ (. Positive (1- 2) and negative (1) -2) skew indicate that errors tend to favor and disfavor option O, respectively. Additive instability occurs under positive selection effects and positive skew (left two columns, green). Results averaged over [.3, .7.
respect to the downstream metric (upper right). This ceiling effect enables selecting a judge system that is optimal for the agreement metric but suboptimal for the downstream metric. In contrast, other configurations shown in Figure all all yield a single optimal judge system.
Finding 2: Fully specified rating tasks enable more performant judge system selections and more effective use of limited annotation budgets. Across all analyses, we find that fully specifying rating tasks improve judge system selections (Fig. 34 associated with fully specifying rating tasks. Judge systems selected with one rating per item on a fully specified task (Fig. 34, left) match the performance of those selected with three ratings per item on an underspecified task (Fig. 34 center) a 66% reduction in per-item requirements (See 3 G).
35 36 Figure 34 Figure34aalse
shows resource benefits
Finding 3: Rater error can further weaken the stability of categorical agreement metrics, but appears less concern than forced choice selection effects in our specific setting. Given the significant impacts of rater етror documented in prior work (e.g., [Klie et al. 2023 Plank 2022 Gordon et al. 2021), D, we conduct additional experiments characterizing the effect of rater error on judge system rankings. Figure 37 illustrates that the effects of rater error on the reliability of HR (h/h) are greatest when forced choice selection and rater error are additive (Fig. 37 column 1-2, green). However, we also observe that the rank correlation between error-free and error- coror-crrupped judge system rankings remains high for MSE (srs/srs) and JSD (s/s) across error settings (Fig. (38). Lemma C.8 (Appendix () pinpoints the mechanism. Because rater eπrrr affects human ratings but judge system ratings, there is not an opportunity for asymmetries to arise between humans and the judge system, there with forced choice selection. Thus, the impact of rater error the comparative ranking of judge systems remains limited. However, given the synthetic nature of our experiments, further work needed characterize the effect of rater error on judge system selection.
57

---

Fully Specifies hucing
Cora
3000-3
300mm3
Undersseches Tesk 22
MSE briver
ISD 1980
Hi Sate 19h0
Sipping
25
Error Rabe
Error Rate (0
Underspanti
2
Error Rate
Frior Born 1r1
Error Rate C2)
Underspediiee te Rot Task (1""= 2,
MSE brisal
.50 1830
62
Figure 38: Spearman correlation (p) between judge system (N = 50) rankings obtained via enror- corrupted versus error-free human rating distributions. We isolate rater error effects by using population rating distributions, eliminating finite sample estimation error confounder. We compute error-corrupted forced choice distribution via O = F (EEO', and error-corrupted multi- label vector via Ω2 A We compute uncorrupted forced choice distribution via O F((O)) and uncorrupted multi-label vector via Ω = A (. We observe that leveraging JSD (s/s) and MSE (srs/srs) human- human-judge agreement yields consistent ranking of judge systems across error- corouccrrupped and crror-frec settings, even under large magnitude of error (c) and non-zero skew (1)) In contrast, using Hit Rate (h/h) to rank judge systems yields rank inconsistencies when (1) the rating task is underspecified, (2) e > 0, and (3) skew (17) * 0. This provides additional evidence for the relative insensitivity of judge selection procedures to rater error (as compared forced choice selection effects), particularly when adopting non-categorical human-judge agreement metrics.
58

---

Cutoff (T 0.25)
Neg.
Effects
=
Prevalence Estimation Blas
0.1
Scone
Skow
Siow
Skow 0
0.2
Scew
Slew
-0.4
-0.6
-0.8
0.0
0.3
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.0
01
0.2
03
0.4
05
Error Magnitude
Error
Error Magnitude
Cutoff (T= 0.35)
Rand. Selection (r" 11
Neg. Selection Effects (1 5)
Pos. Selection Effects (F"=2)
Prevalence Estimatioo Blas
0.4
Siew
Skew
Scaw
Share
0.2
0.0
-0.4
0.6
-0.5
0.0
0.2
0.4
0.0
Error Magnitude
Magn Lude
Magnitude
Cutoff ( 0.45)
Rand. Selection (I" 11
Neg. Selection Effects (1 =.5)
Pos. Selection Effects (f*
Prevalence Estimation Bias
0.4
0.2
0.0
-0.8
-0.8
-1.0
0.0
0.4
Error Magnitude (E)
Error Magn tuoe (E)
Error Magnitude (6)
Figure 39: Bias in prevalence estimates obtained from thresholding the forced choice distribution O = OE,,, versus multi-label vector 22 AO; This experiment eliminates finite sample eтror as a confounder by directly using population rating vectors. Rater error has the largest affect on prevalence estimates when it is correlated with the option (k- 0) used to determine item-level categorizations (i.e., has non-zero skew 1) * 0). Even large magnitudes of error 2 .4) has a limited affect on prevalence estimates when 0. Overall, rater error has significant affect when using human ratings to directly compute Gurget prevalence estimates (shown this figure). The affect of rater error judge system selection depends on the specific relationship between rater error and judge system rating distributions (shown Figures 34
37
Lemma
59

---

Examples of Monotonicity Violations Among Pairs of Performance Metrics
Example 1: Hit Rate (Forced Choice) and KL-Divergence (Forced Choice).
Let p (aaar and Hit Rate) and let p. = (a o KL-Divergence). Consider forced choice distributions recovered from human ratings and the judge systems Z and W, respec- tively: O soft (P"), O Soo (P...). O... son (F..W), ) where we omit from all terms for brevity.
Suppose these distributions are defined over three options O = {01, 02, 02) and for an item i:
Human: O H (0.6, 0.3, 0.1), Judge Z: 0 J.2 (0.8, 0.1, 0.1), Judge W: O J.W (0.5, 0.4,0.1)
Under p. we have:
HR(O 1. OO* = 1.0 HR(O"W, OO" = 0.0
Gudge Ap CW
But under p., have:
KL (L(O"11033) = 0.15 > O ~ 0.02 =
W judge *p 9 Judge
Thus, have identified pair of conditional rating distributions p.Z. P.W. and corre- sponding human rating distribution Ip' where p. is not a monotone transformation p. so rank consistency between and p. cannot hold.
Example 2: KL-Divergence (Forced Choice) and MSE (Multi-Label). Let p (o azone KL-divergence) and let (a ad MSE). Let O = {0r, 02) and Q Hor) ) {02), {or, )- Suppose that humans have no rater error (i.e., E and E. are both the identity). Let Ip satisfy the decomposition:
0" (A,,6), F#
1 0 (0.4,0.5,0.1),, Ω (,,
Let 1. denote the conditional rating distribution of the judge system satisfying:
0 J.2_ = (.4,.6) J.Z
--
- = (0.0.0.6,0.4) 2 = T.
Let J...W denote the conditional rating distribution of the judge system satisfying:
O J.W TTT, F.J.W.
J, (0.4,0.5.0.1) S.W.W TTT
Under p, we have:
XL(O"10042) 0 O = 0.02 judge > judge.
But under p.
MSE 1.2 12 0.01 MSE .2 0.00
Judge P. Grudge
yielding violation of monotonicity. Thus, we have identified pair of conditional rating distributions p.2,, p.-. and corresponding human rating distribution P! where p, is not monotone transformation of p. so rank consistency between p and P. cannot hold.
60

---

G Practical Implementation Considerations
We now describe how our framework relates to several practical dimensions that are often of interest during the design of LLM-as-a-judge meta-evaluations.
Computational Overhead: On per-item (i.e., target system output) basis, our framework
imposes no additional computational overhead over status quo meta-evaluation practices that leverage (1) forced choice elicitation with (2) hard aggregation and and (3) categorical human-judge agreement metrics. In particular, all operationalizations of human-judge agreement [ have a constant computational overhead. Thus, adopting our proposed MSE metric recovered from soft response set aggregation imposes no further overhead. Additionally, prompting a judge system via response set elicitation requires the same number of inference calls and output tokens per-call as the status quo forced choice elicitation approach. In the case in which practitioners prompt with both forced choice and response set elicitation, our framework will result in a 2z increase in system calls. However, we note that this is not strictly speaking necessary when directly leveraging our proposed multi-label nauman-judge agreement metrics, which do not make use of forced choice ratings.
Cognitive Overhead: In some cases, prompting human raters with response set elicitation opposed to forced choice elicitation may increase the cognitive overhead on per-item basis (Dhar and Simonson 2003). For rating tasks with a simple structure (e.g.. Yes/No, Win/Tie/Loosse, response set elicitation may decrease the overall cognitive burden of the rating process by reducing the deliberation required to select a single option when multiple could be viewed as correct. However, in settings with many options, response set elicitation may require a more lengthy process of per-item deliberation - e.g., by requiring raters to carefully and independently assess each option. While our experiments make the benefits of response set elicitation clear (e.g.. Fig. 5 future work should explore tradeoffs between the cognitive cost and information gain associated with alternative elicitation regimes.
Rating Resource Requirements: The two points above speak to the per-item cost involved with collecting ratings. However, our results also illustrate that the number of ratings-per per-- item are an important design consideration of LLLMaas--jjude Evaluations with very few ratings per item (e.g.. 1-3) fail to capture sufficient information about indeterminacy, resulting in poor judge system selection regardless of the numan-judge agreement metric used. This need for multiple ratings inherent to any approach that aims to capture the distribution of human interpretations, as opposed to a limitation specific to our framework. Importantly, at a fixed rating budget (i.e., fullllches-full specified rating tasks and multi-label agreement metrics- -yiedds to more performant selections of judge systems than the status quo approach that relies on hard aggregation of forced choice ratings with categorical agreement metrics (Fig. (34).
meta-evaluation (Fig. (4).
our proposed
Rating Scale Construction: As noted in [ our framework is specifically designed for closed form rating tasks with discrete set of options. As such, is not designed for open form feedback or continuous rating scales. In general, structuring rating task scales with fewer options enables more efficient per-item estimation of the forced choice distribution, response set distribution, and forced choice translation matrix. This is because tasks with fewer options also have a coarser discretization of the forced choice and response set probability space. This introduces a tradeoff between the discretization of the rating scale and the ratings-per-item required for estimation. Exploring the design of rating scales and elicitation techniques under indeterminacy is an interesting avenue for future research.
G.1 Performing Response Set Reconstruction with Pre-collected Forced- Ratings.
Practitioners who wish to validate judge systems using a dataset with pre-collected forced-choice ratings have two options.
Perform sensitivity analysis. When no response set ratings can be collected, practi- tioners can compare judge systems while systematically varying the relationship between the known forced choice distribution and the unknown response set distribution:
Option 1:
1. Estimate the i'th item's forced choice distribution Ô using forced choice ratings. Each entry in this vector denotes the empirical probability of rater endorsing the k'th forced choice option for the i'th item.
61

---

2. Obtain set of reconstructed (or simply hypothesized) forced choice translation matrices F. For "reconstruction", one can consider a range of plausible values of the sensitivity parameter 3 E [0, 1]. Tables 4-6 (Appendix D provide examples of how to construct F from B where assume that F fixed across items.
3. Use the set reconstructed or hypothesized F to obtain estimates the response set distribution: 0, = FO,,
The estimated response set distribution can then be used to compute multi-label agreement metrics (see Appendix D). To perform this sensitivity analysis, practitioners then check whether the optimal judge system remains constant across different F.
Option 2: Estimate the response distribution. When small additional auxiliary corpus (e.g.- 20-100 items) of paired forced choice and response set ratings can be collected, practitioners can directly estimate the response set distribution without requiring sensitivity parameter. This approach involves the following steps:
1. Estimate the forced choice translation matrix F using the auxiliary corpus. Each entry of F denotes the empirical probability of a rater endorsing the o' th response set given that they selected the k'th forced choice response. As above, assume F is fixed across items.
2. Estimate the forced choice distribution Ô for each item.
3. Use F to estimate the response set distribution: 0, = FO,.
As with option above, the estimated response set distribution can then be used to compute multi-label agreement metrics.
G.2 Threshold Selection for Downstream Task Metrics.
The threshold parameter used for downstream evaluation tasks policy deternination on the part of the evaluation designer. A small value of (e.g., .1-33 denotes that an item should be categorized as positive (e.g., for "toxicity")) if small proportion of human raters identify a "positive" interpretation in the text. A large value of (e.g.. .8-9) denotes that many raters need to identify on a "positive" interpretation before an item is categorized as positive. Safety-critical applications (e.g., content moderation, identifying physical safety threats) may benefit from lower thresholds to minimize false negatives; applications requiring high precision may prefer higher thresholds. The interpretation of is analogous to the probability cutoff in binary classifiers: selecting very low or very high values is likely to introduce high agreement rates due to most samples being labeled as negative positive, respectively. Therefore, when is near 0 1, metrics such the false positive rate or false negative rate may be more informative than accuracy-barc -bssd measures (c.g., hit-rate). Analogously computing the AUROC in the binary classification setting, recommend evaluating over range of plausible cutoffs 7 to determine the effect of the parameter on judge system selection.
Societal Impacts. In this paper, introduce a framework for LLM-as-a-judge validation in under rating task indeterminacy. Although our arguments have potential societal consequences, especially the practices we advocate for see adoption and thus change current GenAl evaluation practice, there are no consequences we feel the need to highlight that are specific to this work rather than applicable to any work aiming improve upon current evaluation practices. The validation practices described in this paper are not an endorsement for the adoption of a judge system in any particular setting.
62

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims made the abstract and introduction are corroborated via both theoretical and experimental results. The abstract and/or introduction clearly state the claims made, contributions, and important assumptions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We provide discussion of our work's limitations in Section We provide detailed description of our modeling assumptions in Appendix C Cand empiriclly examine their robustness via synthetic experiments in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
63

---

Answer: [Yes]
Justification: We provide detailed description of our modeling framework and assumptions
in Appendix This section also provides detailed proofs for all theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe our experimental setup in appendices and
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
64

---

reproduce experiments at Answer: [Yes]
Justification:
We release code and data needed to
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... ec/
public/guides: guwdes/Codeeuubng
for more details.
While we encourage the release of code and data, we understand that this might be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips wwtpcccppppppcpceececcecccccccec sesiic/guiees.. Bttee///ooouubissseeeeeee wwiniissiooooiiyoy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide description of our real-world data experimental setup in Appendix D and a detailed description of our synthetic data experimental setup Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail
that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report SEM with bootstrap sampling for all experiments. Error bars are clearly included in all plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
65

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about compute resources required for experiments in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics httss: /tte:://nnuuuie nwwrips....../. ctwpiiissccpppuuuu hc/ppubbic/tthiiccsideee.l...eid
didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Authors have reviews the NeurlPS Code of Ethics and confirm it conforms in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix Groovides a discussion of practical implementation considerations involved with our framework. As part of this discussion, we describe potential limitations associated with our framework.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to
66

---

any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To our knowledge, our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use open source datasets our evaluation and credit authors inline with the licensing requirements. See Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pawerssiippssiiiiii...... twoode, com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online. the authors are encouraged to reach out to the asset's 's creators.
13. New assets
67

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the datasct/codd/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:[NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
68

---

Justification: LLMs were used for writing, editing, or formatting purposes and do not impact the core methodology, scientific rigorousness, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to our LLM policy http:///eeeerrssscccooooeeeeeeee for what should or should not be described.
69


Reviews:
[{"review_id": "7yshTr4sXg", "reviewer": "Reviewer_VqiR", "summary": "The paper discusses indeterminate rating tasks for LLM based Judges  and the failure of  the cases where a particular choices are forced in vanilla inference setting  without any reasoning.  The work proposes training-free multi-label metrics based solution to  mitigate the issue of indeterminacy. It is evaluated against 11 rating tasks including  diverse open-weight and close models demonstrating superior performance over previous methods.", "questions": "1. Dataset size and sampling details: The paper does not specify the size of the evaluation dataset for each of the 11 tasks. It is unclear whether all samples from the source dataset were used or whether sub-sampling was performed, which is critical for assessing the robustness and generalizability of the results.\n\n2. Citation and dataset clarification: Line 225 references \"JudgeBench\" without citing it, and the paper does not explicitly identify the five datasets used from JudgeBench. This omission undermines transparency and reproducibility, as it leaves readers without clarity on the source of the evaluation data.\n\n3. Human Evaluation: Is there any human evaluation  performed to ensure the indeterminacy of the tasks? Adding qualitative examples for each evaluation tasks in Appendix could be useful.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The work introduces a probabilistic theoretical framework for indeterminate rating scenarios, providing a rigorous foundation for modeling uncertainty in ambiguous judge evaluation settings.\n\n2. The proposed method demonstrates significant improvements over existing baselines, achieving up to 30% performance gains in key metrics through its approach to probabilistic reasoning.\n\n\n\n### Weaknesses\n\n1. The NO chain-of-thoughts/reasoning (\"No explanations\" in prompt)  is highly impractical for currently deployed LLMs for a hard task like judgement. Existing benchmarks, such as RewardBench[1], allows the model to reason upto 1024 tokens before generating the final judgement. Hence,  the conclusions from this rigid assumption is questionable due to the way modern LLMs are trained.\n\n2. With the similar motivation of point 1, it is unclear how \"No explanations\" work for Reasoning models, such as o3/r1 models, as they are  explicitly trained  to generated long CoTs before the final answer.\n\n3. The Independence assumption are broken when the labels are not explicitly specified  to be different, such as for  the prompt in Figure 22, the LLM may  think that  all '\"very toxic\" queries are also always  \"toxic\" resulting in returning \"AB\" instead of only \"A\" or \"B\".  Thus, it is unclear how the  proposed method work in such cases.\n\n4.  It is unclear how  the LLMs are sampled, like the temperature, limited the significance of  the reported  result. If non-greedy sampling is used, how many times each query is sent to the model. If only one sample is generated  per query, how does the results, such as Figure 6,  change  for a new run?  If more than one, how  is the result aggregated?\n\n5. The paper only evaluates on possible indeterminate settings, so it is unclear how well the framework performs where indeterminacy is minimal, such as RewardBench[1].\n\nReferences:\n\n[1] Lambert, Nathan, et al. \"Rewardbench: Evaluating reward models for language modeling.\" arXiv preprint arXiv:2403.13787 (2024).", "was_revised": true, "final_justification": "The authors have resolved most of my concerns related to reasoning and experimental settings (sampling and no indeterminacy setups). New reasoning model has been added and the proposed method works effectively.\n\nI am unsure about this comment \" “No explanations” in the output does not inhibit reasoning traces\" regarding the prompting technique and considering \"multi-label\" without taking the nature order of the options into account explicitly. Hence, I will keep my low confidence.", "rebuttal": "We thank the reviewer for taking the time to provide a detailed review of our paper. We appreciate that the reviewer finds our work provides a *“rigorous foundation for modeling uncertainty in ambiguous judge evaluation settings”* and *“demonstrates significant improvements over existing baselines.”* The reviewer also raises several concerns, which we address below:\n\n## Weakness 1: Incompatibility with reasoning models.\n\n>The NO chain-of-thoughts/reasoning (\"No explanations\" in prompt) is highly impractical [...]. Existing benchmarks, such as RewardBench[1], allows the model to reason up to 1024 tokens before generating the final judgement. Hence, the conclusions from this rigid assumption is questionable [...]\n\n**Our framework is fully compatible with judge systems that leverage reasoning and CoT.** As a reminder, the main point of our paper is to provide a framework informing how to evaluate LLM-as-a-judge systems. We place no restrictions on *which* specific judge systems are being evaluated.  The results presented in the paper show that under task indeterminacy, standard approaches to evaluating LLM-as-a-judge system performance lead to incorrect conclusions and fail to identify the truly best performing model. When we started running our experiments, reasoning models were not yet widely available.  We agree with the reviewer that now that reasoning models are broadly available and show high performance, it is worth including a reasoning model in our experiments. \n\n**Technical Note:** The instruction in our rating prompts to include “No explanations” in the output does not inhibit reasoning traces (e.g., Figure 22 referenced in the review). Rather, as with standard benchmarks (e.g., MTBench[1], JudgeBench [2]), this instruction is there to simplify model responses and make it easier to extract the answer. Commercial models (e.g., o1/3, DeepSeek R1) provide reasoning traces in dedicated output fields—the “no explanations” instruction does not suppress reasoning, but does produce simpler outputs in the dedicated output fields. For models where reasoning/CoT are returned in the same field, we recommend excluding that instruction and instead performing output parsing to extract the answer from the unified reasoning + output field.  \n\n**Additional reasoning model experiment: o3-mini.** To demonstrate compatibility with reasoning models, we re-ran experiments with o3-mini.  As suggested by the reviewer, we set the max token length to 1024 tokens. We preview a summary of our findings on the TopicalChat Understandable task (Figure 6; columns 3-4) below. We found that it is (1) indeed “in truth” the *second best* performing judge model; (2) misranked as *among the worst* performing models under standard evaluation approaches; (3) *correctly identified by our proposed alternative* as being the *second best* performing judge model.  We will also update all plots in the paper to include data from o3-mini. \n\n\n| Rank              |Ground Truth (Bias MAE)  |  Baseline (Hit-Rate) | Ours (Multi-Label MSE)\n| :---------------- | :------: | :----: |  :----: |\n| #1        |   Mistral Large   | Claude Sonnet 3.5 | Mistral Large |\n| #2        |    **GPT o3-Mini**   | Llama 70B-Instruct | **GPT o3-Mini** |\n| #3        |   GPT-3.5 Turbo   | Mistral Small | Mistral Small |\n| #4        |   Mistral Small   | Mistral Large | Claude Sonnet 3.5 |\n| #5        |   DeepSeek Chat   | DeepSeek Chat | GPT-3.5 Turbo |\n| #6        |   Claude Haiku   | GPT-3.5 Turbo | DeepSeek Chat |\n| #7        |   Claude Sonnet 3.5   | Claude Haiku | Claude Haiku |\n| #8        |   GPT-4o Mini   | GPT-4o Mini | GPT-4o Mini |\n| #9        |   Llama 70B-Instruct   | **GPT o3-Mini** | Llama 70B-Instruct |\n\n\nIn sum, these additional experiments including a reasoning model further strengthen our core findings: standard approaches to LLM-as-a-judge performance evaluation give misleading results under indeterminacy, both for reasoning and non-reasoning models.  Our framework allows us to clearly understand these misleading results, and offers an alternative that demonstrably improves upon the standard approach.  \n\n\n\n&nbsp;\n## Weakness 2: Violations of rating option independence\n\n> The Independence assumption are broken when the labels are not explicitly specified to be different… The LLM may think that all '\"very toxic\" queries are also always \"toxic\" resulting in returning \"AB\" instead of only \"A\" or \"B\" (Figure 22).\n\nWe do not make any independence assumptions that would preclude selecting AB = {“Very Toxic”, “Toxic”} under response set elicitation.  Indeed, the core issue that motivates our work is that many *existing* rating task designs make this unreasonable assumption. I.e., many *existing* rating tasks are structured as forced choice selection tasks requiring that just A xor B is selected, whereas raters actually want to select AB.  Our work shows that forced choice elicitation leads to erroneous conclusions about judge system performance, and we *advocate* for rating task designs that enable AB to be selected. This is clear to see in our response set rating prompt (Figure 22; top), which instructs a model to select all options that could reasonably apply. For instance, in the reviewer's example, a judge system might select both \"Very Toxic\" and \"Toxic.” In contrast, a forced choice rating prompt (Figure 22; bottom) instructs the judge to select a single correct option (e.g., ``Toxic’’). Our results show that LLM-as-a-judge validation is confounded when an LLM includes multiple options in a response set (e.g., \"Toxic’\", \"Very Toxic\") but is forced to select a single option as a forced choice response.\n\n&nbsp;\n## Weakness 3: Documentation of sampling procedure.\n\n> It is unclear how the LLMs are sampled, like the temperature [...]. If non-greedy sampling is used, how many times each query is sent to the model? [...]\n\nThank you for noting this oversight on our part!  We thought we had included configuration/decoding details in our appendix but we see that indeed we had not.  We will certainly provide complete details in a camera-ready. \n\n**Temperature.** In line with the setup of our most closely related work Elangovan et al [4], we set the temperature of each model to a default value of 1.0.\n\n**Sampling.** We use non-greedy sampling with N=10 samples per item to estimate the empirical distribution over rating task options and response sets. E.g., $\\hat{Y}^J = [0.3, 0.2, 0.5]$ corresponds to 3/10 LLM calls returning \"Very Toxic\", 2/10 calls returning \"Toxic\", and 5/10 calls returning \"Not Toxic.\" We then (1) aggregate this empirical distribution into a rating task response (e.g., a hard categorical output reflecting the most frequent option) and (2) compute human--judge agreement metrics. \n\n&nbsp;\n## Weakness 4: Evaluation limited to indeterminate settings\n\nThank you for raising this important conceptual point. When presenting a method designed to perform better in a more complex setting (e.g., under indeterminacy), it is important to understand whether the method underperforms baselines if applied in the standard setting (here, little-to-no indeterminacy).  Fortunately, our existing experiments show that our recommended method continues to outperform alternatives across varying levels of indeterminacy in the human rating task. \n\nWe examine the effects of indeterminacy using a parameter $\\beta^H$, which encodes the probability that a human rater endorses multiple items in their response set given that they selected a single forced choice response.  When $\\beta^H= 0$, we’re effectively assuming there’s no indeterminacy in the human rating task:  i.e., that human raters give the same response under forced choice as multi-label (response set) elicitation. **Multi-label metrics perform at least as well as baselines when $\\beta^H= 0$** (Figure 4).\n\nWe also agree that adding rating tasks where indeterminacy is expected to be minimal (e.g., from RewardBench [3]) would provide further evidence for our framework's robustness.\n\n&nbsp;\n## Questions\n> The paper does not specify the size of the evaluation dataset for each of the 11 tasks. It is unclear whether all samples from the source dataset were used or whether sub-sampling was performed [...]\n\nPlease see Appendix D (Item Sampling, lines 1227-1230) for these full details. In short, we randomly sub-sample 200 items for each rating task. These 200 item subsets are of similar size to those included in RewardBench [3] (Table 1). We will release the datasets used for our evaluation for reproducibility.\n\n>  Line 225 references \"JudgeBench\" without citing it, and the paper does not explicitly identify the five datasets used from JudgeBench. [...]\n\nThank you for raising this point. In addition to the existing JudgeBench citation on line 320, we have added a second citation at line 225. The specific datasets from JudgeBench are listed on line 225: SNLI, MNLI, α-NLI, SummEval, and QAGS.\n\n\n> Is there any human evaluation performed to ensure the indeterminacy of the tasks? Adding qualitative examples [...] could be useful.\n\nWe appreciate this excellent suggestion. We will add qualitative examples from all tasks to Appendix D. Prior work has also extensively studied indeterminacy in several of our rating tasks (e.g., NLI [5] and toxicity [6]). We will add this discussion to our related work.\n\n&nbsp;\n## References\n[1] Zheng et al, Judging Llm-as-a-Judge with Mt-bench and Chatbot Arena, NeurIPS, 2023.\n\n[2] Tan et al, JudgeBench: A Benchmark for Evaluating LLM-based Judges, ICLR, 2025.\n\n[3] Lambert et al, Rewardbench: Evaluating Reward Models for Language Modeling, arXiv:2403.13787 (2024).\n\n[4] Elangovan et al, Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge, ICLR, 2025.\n\n[5] Zhang et al, Identifying Inherent Disagreement in Natural Language Inference, NAACL, 2021.\n\n[6] Goyal et al, Is Your Toxicity My Toxicity, CHI, 2022."}, {"review_id": "0K6JuFfamY", "reviewer": "Reviewer_BT77", "summary": "This paper examines the limitations of current LLM-as-a-judge validation practices, especially under tasks with ambiguous or subjective criteria. The authors show that standard forced-choice evaluation schemes can significantly misrepresent the true agreement between human raters and LLM judges. They introduce a probabilistic framework that models rating task indeterminacy and propose multi-label, response-set-based evaluation as a more accurate alternative. Extensive experiments across diverse tasks and models demonstrate that their approach leads to more reliable and fair judge system selection.", "questions": "1. Does using distribution-based evaluation signals lead to better results in RLHF or DPO training compared to forced-choice labels?\n\n2. What are the main differences between this work and [1]? Please clarify the novelty over prior work.\n\n3. For practitioners working with pre-existing datasets that used forced-choice ratings, what are the concrete steps to retroactively estimate the response set distribution without collecting additional human annotations?\n\n[1] Elangovan et al., \"Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge.\" ICLR. 2025", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "## **Strengths**\n\n1. The paper provides a solid theoretical framework that emphasizes evaluating human and LLM judgments as distributions rather than single labels, which is a significant and valuable perspective.\n\n2. The authors conducted extensive experiments across multiple real-world tasks and a variety of commercial LLMs, demonstrating the empirical validity of their approach.\n\n## **Weaknesses**\n\n1. While the proposed evaluation framework is shown to improve judge system selection, the paper does not provide direct experimental evidence that these improvements translate to better LLM training or downstream task performance (e.g., through RLHF, DPO, or similar methods).\n\n2. The problem formulation and the proposed solution approach appear to be similar to those in [1]. It would be helpful if the authors could clearly articulate the differences between their work and this prior research.\n\n[1] Elangovan et al., \"Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge.\" ICLR. 2025", "was_revised": true, "final_justification": "The authors' detailed response is greatly appreciated. The original concerns are addressed. However, I think it might be better if there were some additional results regarding practical usage of this evaluation framework during rebuttal phase. As a result, I decide to keep my original score.", "rebuttal": "We thank the reviewer for their thoughtful feedback on our work. We are encouraged that the reviewer finds we provide a *“solid theoretical framework”* with a *“significant and valuable perspective.”*  We also appreciate that the reviewer finds our empirical validation *\"extensive.\"* The reviewer also raises several important concerns, which we carefully address below:\n\n&nbsp;\n\n## Weakness 1: Limited evidence for improvements in downstream task performance\n\n> While the proposed evaluation framework is shown to improve judge system selection, the paper does not provide direct experimental evidence that these improvements translate to better LLM training or downstream task performance (e.g., through RLHF, DPO, or similar methods).\n\n**Overview.** We thank the reviewer for this excellent suggestion: we agree that showing our framework improves performance on downstream *training* tasks (e.g., RLHF, DPO) is a fruitful direction for future work. Our experiments *do not* address this question directly. However, they *do* provide evidence that our framework leads to significant improvements on **downstream *evaluation* tasks** performed by judge systems. These downstream evaluation tasks are central to key policy decisions, such as whether target system outputs fall within an acceptable safety range for system deployment.  \n\n**Example.** In *content filtering* tasks, a rater (human or LLM) decides whether to allow or suppress each target system output -- e.g., based on whether it contains \"toxicity\", \"physical safety threats\", or \"factual errors.\" Judge systems are increasingly used for content filtering tasks when generating refusals [1]. We show that status quo validation approaches lead to the selection of judge systems with as much as *30% worse content filtering performance* than those selected using our framework (Figure 1). \n\n**Practical Impact.** In the example above, judge systems selected via status quo approaches provide a false positive or false negative \"toxicity\" flag 30% more often than those selected with our framework. This can lead to unsafe outputs being incorrectly labeled as \"safe\" by a judge system or a decision to deploy a target system with poor safety characteristics. In sum, we provide evidence that our framework has a significant downstream impact, albeit in an *evaluation* as opposed to a *training* context. \n\n&nbsp;\n\n## Weakness 2: Differentiation from Elangovan et al.\n> The problem formulation and the proposed solution approach appear to be similar to those in [2]. It would be helpful if the authors could clearly articulate the differences between their work and this prior research.\n\n**Overview.** **Our approach leveraging response set elicitation with multi-label agreement metrics is more robust to indeterminacy than Elangovan et al. [2]'s.** As the reviewer correctly notes, both Elangovan et al. [2] and our work study a setting where judge systems are validated for \"ambiguous\" tasks. However, our approaches diverge in how they account for human rating uncertainty throughout the validation process. These differences lead to a significant performance improvement with our proposed approach. \n\n**Conceptual Advance.**  Elangovan et al. [2] represent human rating uncertainty via soft labels over *forced choice ratings*.  However, we show that soft labels are insufficient when each rater identifies more than one \"correct\" response. This is because soft labels do not account for all options included in a single rater's response set (i.e., containing multiple \"correct\" responses). Formally, theorem 3.1 establishes that judge systems can be ranked differently when measuring agreement via multi-label (response set) versus distributional (forced choice) metrics used by Elangovan et al. [2]. \n\n**Empirical Benefits.** Our use of response set ratings with multi-label metrics translates to a *significant performance improvement over Elangovan et al. [2].* In particular, Elangovan et al. [2] measure Jensen-Shannon Divergence (JSD) between judge system and human labels. **This is reflected by our JS-Divergence (s/s) baseline in Figure 4.** On average, JS-Divergence (s/s) performs 2-12 percentage points worse than our multi-label MSE metric (Figure 4, top right panel). The magnitude of improvement over Elangovan et al. [2] depends on the degree of indeterminacy in the human rating process (Figure 4, $\\beta^H$). We will add this discussion above to our related work section to improve clarity for future readers. \n\n&nbsp;\n\n## Questions\n\n> Does using distribution-based evaluation signals lead to better results in RLHF or DPO training compared to forced-choice labels?\n\nOur framework is designed to support better judge system selection in indeterminate rating tasks.  The question of how these improved judge systems perform as a feedback signal for *training* remains an open area for future work. For instance, a nascent line of work explores the use of synthetic \"AI feedback\" (e.g., obtained from an LLM-as-a-judge applying a rubric) to guide the training process (e.g., via DPO) [3,4]. However, while prior work has identified flaws affecting LLM-as-a-judge systems (e.g., [5,6]), this work does not examine the effect of judge system selection on the use of the judge as an \"AI feedback” signal.\n\nSince judge system quality directly impacts the quality of the feedback signal, improved judge selection should in principle translate to improvements in post-training performance. This suggests that training pipelines using judge systems selected via our multi-label approach *may* in turn provide more effective synthetic \"AI feedback.\" However, in line with experimental setups of prior work [4,5], we do not directly examine the connection with post-training as part of our evaluation. Thus, we leave this interesting line of questions to future work. \n\n> What are the main differences between this work and [2]? Please clarify the novelty over prior work.\n\nSee discussion of Weakness 2 above. \n\n> For practitioners working with pre-existing datasets that used forced-choice ratings, what are the concrete steps to retroactively estimate the response set distribution without collecting additional human annotations?\n\n\nPractitioners who wish to validate judge systems via pre-existing datasets with forced choice ratings have two options. \n\n**Option 1: Perform a sensitivity analysis.**  When *no* response set ratings can be collected, practitioners can compare judge systems while systematically varying the relationship between the *known* forced choice distribution and the *unknown* response set distribution. To perform this sensitivity analysis, practitioners can: \n\n1. Estimate the $i$'th item's forced choice distribution $\\hat{O}_{i}$ using forced choice ratings. Each entry in this vector denotes the empirical probability of a rater endorsing the $k$'th forced choice option for the $i$'th item. \n2. Obtain a set of reconstructed (or simply hypothesized) forced choice translation matrices $\\{\\hat{F}\\}$.  For “reconstruction”, one can consider a range of plausible values of the sensitivity parameter $\\beta^H \\in [0,1]$. Tables 4-6 (Appendix D) provide examples of how to construct $\\hat{F}$ from $\\beta^H$, where we assume that $\\hat{F}$ is fixed across items. \n3. Use the set of reconstructed or hypothesized $\\{\\hat{F}\\}$ to obtain estimates of  the response set distribution: $\\hat{\\theta}_{i} = \\hat{F} \\hat{O}_i$.\n\nThe estimated response set distribution can then be used to compute multi-label agreement metrics (see Appendix D, lines 1235-1241). To perform this sensitivity analysis, practitioners then check whether the optimal judge system remains constant across different  $\\{\\hat{F}\\}$.\n\n**Option 2: Estimate the response set distribution.** When a small additional auxiliary corpus (e.g., 20-100 items) of paired forced choice and response set ratings can be collected, practitioners can directly estimate the response set distribution without requiring a sensitivity parameter. This approach involves the following steps: \n\n1. Estimate the forced choice translation matrix $\\hat{F}$ using the auxiliary corpus. Each entry of $\\hat{F}$ denotes the empirical probability of a rater endorsing the $v$'th response set given that they selected the $k$'th forced choice response. As above, we assume $\\hat{F}$ is fixed across items. \n2. Estimate the forced choice distribution $\\hat{O}_i$ for each item. \n3. Use $\\hat{F}$ to estimate the response set distribution: $\\hat{\\theta}_{i} = \\hat{F} \\hat{O}_i$.\n\nAs with option 1 above, the estimated response set distribution can then be used to compute multi-label agreement metrics. We will add these important practical details to our practical implementation guidelines (Appendix G) and provide walk-through instructions in our public code release. \n\n\n&nbsp;\n## References\n[1] Cui et al., OR-Bench: An Over-Refusal Benchmark for Large Language Models, ICML, 2025.\n\n[2] Elangovan et al., Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge. ICLR. 2025.\n\n[3] Viswanathan et al., Checklists Are Better Than Reward Models For Aligning Language Models, ArXiv, arXiv/2507.18624. 2025.\n\n[4] Tunstall et al., Zephyr: Direct distillation of lm alignment. ArXiv, abs/2310.16944, 2023.\n\n[5] Zu et al., Investigating Non-Transitivity in LLM-as-a-Judge, ICML, 2025.\n\n[6] Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge, ArXiv, abs/2406.07791. 2024."}, {"review_id": "9ghj9LAlGe", "reviewer": "Reviewer_9S5u", "summary": "This paper addresses a critical limitation in current LLM-as-a-judge validation practices: how to properly evaluate judge systems when rating tasks are indeterminate (i.e., multiple answers may be \"correct\"). The authors introduce a comprehensive framework distinguishing between forced-choice and response-set elicitation schemes, and demonstrate theoretically and empirically that standard validation approaches can severely misestimate judge performance. Through experiments across 11 rating tasks and 8 commercial LLMs, they show that judges selected using conventional metrics can perform up to 30% worse than those selected using their proposed alternatives. The paper provides concrete recommendations including fully-specified rating tasks, multi-label response elicitation, and continuous multi-label agreement metrics.", "questions": "1. Resampling Justification: Line 244 mentions resampling LLMs 10 times per item. Could the authors provide stronger justification for this specific number and analyse sensitivity to this choice?\n2. Threshold Selection: How sensitive are the proposed multi-label metrics to the threshold parameter τ used in downstream tasks? Guidelines for selecting appropriate thresholds would be valuable.\n3. Cross-Domain Validation: How well does the framework generalise to domains not represented in the current evaluation (e.g., creative writing assessment, scientific accuracy)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- Quality: The theoretical framework is rigorous and well-grounded in established psychological research on forced-choice selection effects. The empirical evaluation is comprehensive, spanning multiple domains (toxicity, factuality, relevance) and models. The authors provide formal identifiability results (Theorem 2.1) and necessary conditions for rank consistency (Theorem C.7), establishing clear mathematical foundations for their claims.\n- Clarity: The paper is exceptionally well-written with clear exposition of complex concepts. Figure 1 effectively illustrates the core problem, whilst the framework diagram (Figure 2) provides an excellent conceptual overview. The distinction between forced-choice and response-set elicitation is clearly explained and consistently maintained throughout.\n- Significance: This work addresses a fundamental problem in AI evaluation that has practical implications for high-stakes applications like content moderation and safety assessment. The finding that standard approaches can select suboptimal judges by substantial margins (30% performance degradation) is practically significant. The framework provides actionable guidance for improving evaluation practices.\n- Originality: The application of forced-choice selection effects to LLM-as-a-judge validation appears novel. The theoretical framework connecting different elicitation schemes, aggregation methods, and agreement metrics is original and provides new insights into why current practices fail under indeterminacy.\n## Weaknesses\n- Limited Scope: The framework only applies to discrete-option rating tasks, excluding continuous scales and open-form feedback, which limits its applicability to some evaluation scenarios.\n- Implementation Complexity: Whilst the authors claim no additional computational overhead, collecting response-set ratings may increase cognitive burden for human annotators, particularly for tasks with many options. The practical implementation challenges deserve more thorough consideration.\n- Generalisability Concerns: The experiments focus primarily on established benchmark datasets. More investigation of how the framework performs across diverse domains and rating task designs would strengthen the claims.\n- Methodological Limitations: The synthetic experiments make strong homogeneity assumptions about the human rating process (noted in limitations), which may not hold in practice across diverse annotator populations.", "was_revised": true, "final_justification": "The rebuttal effectively resolved several key technical concerns, particularly around methodology and practical implementation. The authors' systematic sensitivity analysis justifying their resampling choice (N=10) and the robust theoretical backing for error tolerance (Figure 37, Lemma C.8) strengthen confidence in their approach. Their explanation that discrete-option scales are standard practice in LLM-as-a-judge benchmarks, coupled with evidence of biases in continuous scales, provides reasonable justification for scope limitations. The finding that \"noisy\" response-set ratings still provide robust feedback significantly enhances the practical viability of their framework.\n\nThe fundamental issue of generalisability to domains beyond their eleven-task evaluation (such as creative writing or complex reasoning) wasn't fully addressed, and the homogeneity assumptions about human rating processes still pose practical concerns for diverse annotator populations. Despite these limitations, I assign high weight to the excellent technical contributions and theoretical rigour that address a fundamental problem in AI evaluation. The empirical demonstration of 30% performance degradation using standard methods is practically significant, and the actionable recommendations are well-supported. The contributions outweigh the limitations, justifying acceptance whilst acknowledging important areas for future work, particularly empirical validation across more diverse domains and annotator populations.", "rebuttal": "We thank the reviewer for providing a thorough assessment of our work. We appreciate that the reviewer finds our framework *“rigorous and well-grounded”* , *”exceptionally well-written”*, and that it *“addresses a fundamental problem in AI evaluation.”* We would like to engage with several of the key points raised by the reviewer: \n\n&nbsp;\n\n## Weakness 1: Limited scope\n\n>  The framework only applies to discrete-option rating tasks, excluding continuous scales and open-form feedback, which limits its applicability to some evaluation scenarios.\n\nThank you for raising this important point on framework scope. While this indeed does restrict the scope of our framework (see limitations Section 4), discrete-option scales are becoming a standard in LLM-as-a-judge rating tasks. In line with standard LLM-as-a-judge benchmarks (e.g., [1,2]), we preclude open-form feedback because a quantitative performance assessment is ultimately required to produce numeric results used to measure performance and compare systems. Existing benchmarks (e.g., JudgeBench [1]) *do* support continuous scales. However, prior work has identified significant biases that arise with continuous scales [3] (e.g., whereby LLMs demonstrate a strong preference for the number \"42\"). This has led to recommendations that LLM-as-a-judge task designers adopt *discrete-option rating tasks* [4]. In sum, while we acknowledge this as a limitation of our framework, we believe our framework remains applicable in a broad range of practical scenarios.\n\n&nbsp;\n\n## Weakness 2: Implementation complexity\n\n> Collecting response-set ratings may increase cognitive burden for human annotators, particularly for tasks with many options. The practical implementation challenges deserve more thorough consideration.\n\nThank you for drawing attention to this important practical question. In Appendix G, **we provide detailed discussion of practical implementation details associated with our framework.** As we discuss in the *Cognitive Overhead* subsection (lines 1347-56), for rating tasks with a simple structure (e.g., Yes/No, Win/Tie/Loose), response set elicitation may decrease the overall cognitive burden of the rating process by reducing the deliberation required to select a single option when multiple could be viewed as correct. However, in settings with many options, response set elicitation may require a more lengthy process of per-item deliberation — e.g., by requiring raters to carefully and independently assess each option.\n\nFortunately, even \"noisy\" (i.e., error-corrupted) response set ratings collected from human raters who quickly complete the rating task *may* serve as a robust feedback signal for judge validation. In particular, Figure 37 reports findings from additional synthetic data experiments that examine the robustness of judge system rankings to error in human response set ratings. We observe that multi-label metrics --- i.e., MSE (srs/srs) --- recover robust judge system rankings even when using error-corrupted response set ratings. This indicates that human raters can be instructed to *quickly* complete response set ratings when cognitive overhead is a significant concern. While this empirical finding may seem counterintuitive at first, Lemma C.8 (Appendix C) makes the mechanism clear: because rater error only affects the *human* rating process, relative rankings of *judge systems* remain robust to rater error (over response sets).\n\nWe acknowledge that the analysis above remains speculative: in line with the reviewer comment, further work is needed to understand the cognitive load imposed by response set ratings across a range of evaluation tasks. \n\n&nbsp;\n## Weakness 3: Generalisability to new tasks\n\n> The experiments focus primarily on established benchmark datasets. More investigation of how the framework performs across diverse domains and rating task designs would strengthen the claims.\n\nWe agree that further investigation on how the framework generalizes to diverse domains and rating task designs would strengthen evidence for its practical applicability. However, in its current form, our work does evaluate on *eleven diverse rating tasks* spanning concepts such as \"toxicity\", \"entailment\", \"relevance\", \"coherence\", \"factuality\", \"fluency\", \"uses knowledge\", and \"understandable\" (Table 3). For comparison, our most closely related work --- Elangovan et al. [5] --- *also evaluates on eleven rating tasks.* Therefore, while there is certainly an opportunity to expand our evaluation further, we believe the current scope sufficiently demonstrates its generalizability across settings. \n\nWe plan to release a Colab Notebook alongside our public code, and hope that this will spur productive adoption in a range of practical scenarios. \n\n&nbsp;\n## Weakness 4: Homogeneity assumptions on the human rating process\n\n> Methodological Limitations: The synthetic experiments make strong homogeneity assumptions about the human rating process (noted in limitations), which may not hold in practice across diverse annotator populations.\n\nAs we also note in Section 4 (Limitations), we do place homogeny assumptions on the human rating process, which may be violated in some cases. In plain English, our homogeny assumption holds that different human raters have a similar forced choice translation matrices $F$ governing the translation of response set to forced choice ratings. This assumption can be tested by instructing multiple human raters to provide a small corpus of paired forced choice and response set ratings. If we observe a (statistically significant) difference in translation probabilities across raters for the same items, this provides evidence that the assumption may be violated violated. \n\nWhen this homogeny assumption *is* violated, approaches that attempt to use the forced choice translation matrix to reconstruct the response set distribution may perform unreliably (see our response to BT77, Question 3). This is because there is no longer a fixed translation matrix that can \"reverse\" forced choice ratings collected from different human raters. In this setting, it becomes even more critical to collect multi-label (response set) ratings that have not been corrupted by the forced choice translation process. \n\n&nbsp;\n## Questions\n\n> Resampling Justification: Line 244 mentions resampling LLMs 10 times per item. Could the authors provide stronger justification for this specific number and analyse sensitivity to this choice?\n\nWe thank the reviewer for their careful attention to our experimental setup. When sampling judge system responses, our goal is to estimate the forced choice distribution $\\hat{O}^J_i$ and the response set distribution $\\hat{\\theta}^J_i$ from a finite sample of paired forced choice and response set ratings. Since we use empirical response frequencies to estimate these probability vectors (see Appendix D), our estimators are *consistent*: i.e.,  they produce estimates that converge to the true population parameters as sample size increases. Our selected sample size (N=10) operates in the middle of a range that we have already **established via a systematic sensitivity analysis.** \n\nIn particular, we conduct synthetic experiments that systematically assess the robustness of our findings to the number of samples used to estimate *human* rating distributions $\\hat{O}^H_i$ and $\\hat{\\theta}^H_i$ (see Figure 33, Appendix D). This experiment assumes that we know the true response set and forced choice distributions for each judge system and estimates $\\hat{O}^H_i$, $\\hat{\\theta}^H_i$ using an experimentally manipulated number of human ratings-per-item. However, because arguments to human--judge agreement metrics are associative, we can also interpret these results as varying the number of samples used to estimate *judge* rating distribution assuming that the *human* rating distribution is known. Findings demonstrate that multi-label agreement metrics perform best across  a varying number of ratings-per-item used to estimate rating distributions (i.e., 1,3,5,10,20,50,100). We also observe that the benefits of multi-label metrics improve as the number of ratings-per-item increases.\n\n\n> Threshold Selection: How sensitive are the proposed multi-label metrics to the threshold parameter τ used in downstream tasks? Guidelines for selecting appropriate thresholds would be valuable.\n\nThe threshold parameter $\\tau$ used for downstream tasks is a *policy determination* on the part of the evaluation designer. A small value of $\\tau$ (e.g., .1-.3) denotes that an item should be categorized as positive (e.g., for \"toxicity\") if a small proportion of human raters identify a \"positive\" interpretation in the text. A large value of $\\tau$ (e.g., .8-.9) denotes that many raters need to agree on a \"positive\" interpretation before an item is categorized as positive. We agree that guidance on selection of $\\tau$ is important: we will add this discussion to Appendix G (Practical Guidelines).\n\nFurther, we systematically vary $\\tau$ as part of our analysis. E.g., Figure 4 varies $\\tau \\in \\\\{.1,.2, .3,.4,.5,.6,.7,.8,.9 \\\\}$. \n\n> Cross-Domain Validation: How well does the framework generalise to domains not represented in the current evaluation (e.g., creative writing assessment, scientific accuracy)?\n\nWe thank the reviewer for this question. Please see our comments on Weakness 3 above for a detailed response. \n\n&nbsp;\n## References\n\n[1] Zheng et al, Judging Llm-as-a-Judge with Mt-bench and Chatbot Arena, NeurIPS, 2023.\n\n[2] Tan et al, JudgeBench: A Benchmark for Evaluating LLM-based Judges, ICLR, 2025.\n\n[3] Gu et al, A Survey on LLM-as-a-Judge, 2502.01534, 2025\n\n[4] Husain et al., Frequently Asked Questions (And Answers) About AI Evals, 2025.\n\n[5] Elangovan et al., Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge. ICLR, 2025."}, {"review_id": "zc7B8nKawm", "reviewer": "Reviewer_yWpq", "summary": "This paper is about evaluation under multi-choice options, where there is no single correct answer that may be adequate for a particular instance. The paper calls this indeterminacy. The paper then goes on to show that the problem can have consequences on which system is judged to be the best.", "questions": "Q1 - Would your setup be applicable for text generation tasks rather than discrete annotations?\n\nQ2 - Why do you call it interdeminate? Isn't multi-label annotation and/or multi-choice a common scenario with well-defined evaluation metrics?\n\nQ3 - what often happens is also that no single gold label is chosen from the annotations of each annotator but one considers the distributions of labels over annotators. Isn't that similar to your setup?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-written and contains both mathematical insights and experimental results. The setting itself is interesting, i.e., ambiguity in classification tasks rather than forced-choice options. \n\n**Weaknesses**\n\nI must admit that I did not fully follow the paper, but this is mostly likely attributable to me, rather than the paper. On the one hand, I wonder why multi-choice annotations should cause a problem; any labeling task where multiple options could be adequate can be obtained by simply considering the power set and then do standard annotation and evaluation on the power set. It seems I must have surely misunderstood a lot about the framework.\n\nA minor point is that the paper does not have a section 1. I assume this is because the authors wanted to save space, but it is in any case quite unusual.", "was_revised": true, "final_justification": "The authors clarified some of my concerns and their response helped me to better understand the paper.", "rebuttal": "We thank the reviewer for taking the time to deeply engage with our work. We are encouraged that the reviewer finds our problem setting *\"interesting\"* and the paper to be *\"well-written.\"* \n\n## Framework Overview \n\nTo address the reviewer's concern that they may have misunderstood the framework, we provide a brief overview, grounded in the reviewer’s comments. We note that reviewer yWpq has *already* identified key insights about ambiguous classification tasks: our paper synthesizes these insights into a comprehensive framework for validating judge systems when multiple rating options can be \"correct.\"\n\nOur framework connects two ways of encoding ratings for closed form rating tasks:\n- **Forced choice.** Forced choice ratings are produced by instructing a rater to select a single \"correct\" option. We call the distribution over forced choice ratings assigned by each annotator the *forced choice distribution*. The distributional (e.g., KL-Divergence) and categorical (e.g., Krippendorff’s $\\alpha$) human-judge agreement metrics currently used to validate LLM-as-a-judge systems (e.g., [1,4,5]) can be computed from this forced choice distribution. \n- **Response set.** Suppose that we instead instruct a rater to select *all* options that could be interpreted as “correct”. As reviewer yWpq aptly notes, uncertainty over these option sets can be encoded via a second *response set distribution*: the distribution encoding the probability of each rater endorsing a combination of options in the power set. As reviewer yWpq suggests, the multi-label metrics recovered from response set ratings offer a rich representation of rating uncertainty when more than one option can be \"correct.\" \n\nSurprisingly, however, **multi-label metrics are ***not*** currently used to validate LLM-as-a-judge systems**. Instead, the \"state-of-the-art\" in both the LLM-as-a-judge setting (i.e., [1,4-5] cited by reviewer BT77) and the traditional model evaluation setting (e.g., [2,3] ) is distributional metrics (e.g., JS-Divergence, KL-Divergence) defined over *forced choice* ratings. In our paper we show empirically via a comprehensive evaluation over 11 rating tasks that measuring human--judge agreement against these status quo metrics yields highly sub-optimal judge system selections (e.g., performing as much as 30% worse, Fig 1).\n\nOf course, the insight that multi-label metrics are preferable under indeterminacy is conceptually quite simple once stated—but to emphasize, this point, however intuitive, is overlooked in existing literature and generally not integrated into practice. We not only make this point but also make the more fundamental contribution of introducing a unifying theoretical model that connects multi-label metrics to their distributional (e.g., KL-Divergence) and categorical (e.g., Cohen's $\\kappa$) counterparts. In particular, **we leverage a model for the human rating process (Fig. 3) to establish a novel unifying representation of forced choice and response set operationalizations of human rating uncertainty**, and thus renders them amenable to direct comparison (see Table 2).\n\nThis theoretical model also buys us several practical benefits. For instance, we can: \n1. Establish the necessary conditions for any two metrics defined against any two rating distributions to yield a consistent ranking of judge systems. We show that these conditions are quite strong in practice: even different types of multi-label agreement metrics (e.g., discrete \"coverage\" metrics versus continuous mean squared error) produce significant performance discrepancies (see Fig 4: Multi-Label Discrete vs Continuous).\n\n2. Pinpoint the mechanism that causes distributional agreement metrics (e.g., \"state-of-the-art\" JS-Divergence [1]) to be unreliable. In particular, we identify asymmetries in how LLMs and humans pick a single \"correct\" forced choice option from their response set as the factor that introduces performance degradation in forced choice agreement metrics. \n\nIn sum, reviewer yWpq correctly identifies human rating uncertainty over response sets (i.e., encoded as a subset of the powerset) as a powerful representation of human rating variation under indeterminacy. We build this insight into a theoretically rigorous and empirically validated framework with clear practical recommendations for its adoption going forward. \n\n## Reviewer Questions\n\nWe now address specific questions raised by reviewer yWpq.\n\n> Q1 - Would your setup be applicable for text generation tasks rather than discrete annotations?\"\n\nOur setup is applicable to LLM-as-a-judge or human evaluations of how well target models perform on text generation tasks, provided that the LLM-as-a-judge and human evaluations themselves take values on a discrete scale (e.g., such as by scoring generated cover letters on a 1-5 quality scale).  This is the setting that frequently arises when assessing target models’ summarization abilities  (e.g., SummEval) and open-ended model responses (e.g., MT-Bench [12], PRISM [6]) because a discrete rating is ultimately used to quantify and compare output quality. However, our setup is *not directly applicable* to evaluating open-ended generations when a discrete scale is not used. That is, if the LLM-as-a-judge provides an open-ended free-text assessment of a target model’s summary, this is not something our framework handles. We highlight this limitation in Section 4 (line 320).\n\n> Q2 - Why do you call it interdeminate? Isn't multi-label annotation and/or multi-choice a common scenario with well-defined evaluation metrics?\n\nWe use the term “indeterminate” to refer to rating tasks where more than one option can be viewed as \"correct.\"  This is distinct from the question of whether multi-label annotation is sought. Indeed, a key issue we identify with existing practice is that forced choice (select just one option) elicitation schemes are often used in indeterminate settings (where a rater may view multiple options as correct).  As mentioned in the review, using a multi-label annotation workflow is much more natural in such settings; but this isn’t always done. Our work provides a robust theoretical and empirical justification for precisely *why* this approach should be adopted when a forced choice rating task is indeterminate.\n\nWe get the sense from this and other comments that reviewer yWpq is perhaps surprised that current practice does *not* already apply the response set elicitation approach with corresponding multi-label metrics for which we advocate in this paper.  That so many widely used evaluation datasets for indeterminate tasks were collected under forced choice elicitation was also surprising to us, which is what led us to write this paper.   \n\n\n> Q3 - what often happens is also that no single gold label is chosen from the annotations of each annotator but one considers the distributions of labels over annotators. Isn't that similar to your setup?\n\nAbsolutely. Modeling a distribution of forced choice or response set labels is core to our probabilistic model for rating variation. Our theoretical framework establishes precisely *how* one should model these distributions of labels over annotations to produce valid selections of judge systems.  In particular, in our work we distinguish between looking at the distribution of *forced choice* responses and *response set* responses.  We show that distributional metrics such as JS-Divergence that consider rating distributions from forced choice elicitation schemes often do lead to misranking. Instead, we argue that the relevant distribution is the one that comes from response set (multi-label) annotation.  \n\n\n## Presentation\n\n> A minor point is that the paper does not have a section 1. I assume this is because the authors wanted to save space, but it is in any case quite unusual.\n\nThank you very much for letting us know. This missing section header is an editorial oversight that we missed during the final deadline push. We have corrected this in the draft; we also note that the paper remains within the original page limit after other minor formatting adjustments. \n\n## References:\n[1] Elangovan et al., \"Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge.\" ICLR, 2025.\n\n[2] Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio, et al. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. NACCL, 2021.\n\n[3] Katherine M Collins, Umang Bhatt, and Adrian Weller. Eliciting and learning with soft labels from every annotator. HCOMP, 2022.\n\n[4] Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: Llm judges with provable guarantees for human agreement. ICLR, 2024.\n\n[5] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023.\n\n[6] Hannah Rose Kirk et al. The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. NeurIPS 2024."}]

Author rebuttal:

Meta-review: # Summary
This paper addresses a critical limitation in current LLM-as-a-judge validation practices: how to properly evaluate judge systems when rating tasks are indeterminate (i.e., multiple answers may be "correct"). The authors introduce a comprehensive framework distinguishing between forced-choice and response-set elicitation schemes, and demonstrate theoretically and empirically that standard validation approaches can severely misestimate judge performance. Through experiments across 11 rating tasks and 8 commercial LLMs, they show that judges selected using conventional metrics can perform up to 30% worse than those selected using their proposed alternatives. The paper provides concrete recommendations including fully-specified rating tasks, multi-label response elicitation, and continuous multi-label agreement metrics.

# Strengths
* The paper provides a solid theoretical framework that emphasizes evaluating human and LLM judgments as distributions rather than single labels, which is a significant and valuable perspective.
* The application of forced-choice selection effects to LLM-as-a-judge validation appears novel.
* The authors conducted extensive experiments across multiple real-world tasks and a variety of commercial LLMs, demonstrating the empirical validity of their approach.
* The paper is exceptionally well-written with clear exposition of complex concepts.

# Overall
The paper introduces a novel perspective on evaluating performance via distributions instead of single labels. All concerns and questions have been addressed by the authors during the author response period.
Decision: Accept (poster)