Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
Authors: Felipe Maia Polo
Xinhe Wang
Mikhail Yurochkin
Gongjun Xu
Moulinath Banerjee
Yuekai Sun
Venue: NeurIPS 2025 poster ()
Keywords: llm-as-a-judge
alignment
bias
calibration
Primary area: evaluation
TL;DR: We propose a novel statistical framework for LLM judges.

Abstract: Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge, a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps.

Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
Felipe Maia Polo'; Xinhe Wang'; Mikhail Yurochkin
Gongjun Xu', Moulinath Banerjee', Yuekai Sun
Depparment of Statistics, University of Michigan Institute of Foundation Models, MBZUAI
Abstract
Large language models are increasingly used as judges (LLM-as-a-judge) to eval- uate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge', a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits latent human preference score for each prompt-response pair and models LLM deviations linear transforma- tions of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-I.I.M gaps.
1 Introduction
Accurate and reliable evaluation is fundamental to the advancement and deployment of artificial intelligence (AI) systems, such as Large Language Models (LLMs). Traditional expert-based or auto- matic methods (e.g., ROUGE [27] BLEU [33]) for open-ended generated text often struggle with either scalability poor quality. Recently, LLMs have shown significant promise in addressing these challenges through the emergent "LLM-as-a-Judee
(LLMJ) paradigm [12, 22, 23]. Leveraging their extensive knowledge, ficxible reasoning, instruction following. and natural language understanding, LLMs can effectively evaluate complex tasks by scoring, ranking, selecting among diverse outputs. However, ensuring the trustworthiness, robustness, and human- -aliammtt of LLM-based evaluation systems remains a critical challenge that necessitates careful attention.
A crucial step towards better judges involves deepening our understanding of LLMJ systems and recognizing their inherent strengths and limitations. Such knowledge allows practitioners to better quantify associated risks and steer the development of more robust evaluation methods. Recent research has extensively explored factors contributing inaccuracies in LLM judgments and proposed ways to mitigate them. For example, various biases have been well-documented. including preferences for lengthier responses, overly generous scoring tendencies. presentation order during pairwise comparisons [8, 41, 48, 39]. In this work, we introduce Bridge, statistical framework that explicitly connects LLM ratings to human judgments, providing deeper insight into the sources of human-LLM discrepancies and enabling more reliable alignment between
biases influenced simply by the
'These authors contributed equally to this work. Conresponding authors: patpeemmiipplo@gmmillcooom
xinhe. waneee/wwn0ee @ @egmmml..com Please check our GitHub repository:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the two. Our framework is LLM-agnostic, does not require access to model weights, and can be applied on top of any API.
Bridge combines a statistical model with a specialized fitring algorithm via the proposed logit trick, and we demonstrate the asymptotic normality of the resulting estimators. Our model assumes that both human annotators and LLM judges score each prompi-response pair according to a shared latent preference signal, while systematic deviations in LLM scores are captured by linear transformation of covariates that encode potential human-LLM divergence sources (e. response length, text senti- ment, writer's creativity). This formulation enables simultaneous, rigorous, interpretable estimation and testing of multiple discrepancies between human and LLM judgments; something which current approaches cannot accomplish. In addition, it enables lightweight post-hoc corrections: with only a small set of human labels, we can recalibrate LLM scores for improved probabilistic alignment with human assessments.
In summary, our contributions are:
1. Proposing Bridge, a statistical framework connecting human and LLM judgments, which com- bines statistical model with specialized fitting algorithm via the proposed logit trick. Our approach (i) allows practitioners to better understand what makes humans and LLMJ different, (ii) enables better probabilistic aligmment with human judgments, and (iti) is LLM-agnostic, making applicable on top of any API.
2. Deriving the asymptotic distribution of our parameter estimators. These asymptotic distributions allow us to construct confidence intervals for our parameters and formally test for different types of human-LLM gaps. Moreover, it allows to construct confidence intervals for predictive quantities such as the probability of humans making a certain judgment.
3. Validating our framework using six different LLM judges and queries from BigGen Bench and Chatbot Arena. We show that we can better align LLMs to human annotators using a few labeled data points and interpret the systematic differences between the two types of judges.
1.1 Related work
Improving the alignment between LLM-based judges and human annotators is often an important factor for their reliable use at scale. Prior work has pursued several complementary strategies: supervised fine-tuning on numan-labelled data [14]: supplying in-context examples to the judge [48]; post-hoc smoothing of raw scores [28, 21]: decomposing complex evaluation tasks into simpler, verifiable criteria [38, 44]; harnessing reference answers detailed rubrics [17, 47], chain-of- thought (CoT) strategies to make use of the model's reasoning abilities [17, 25].
A parallel line of research first diagnoses systematic biases in LLMJ and then proposes corrective measures. For instance, Dubois et al. [8] reveals a strong preference for longer responses and intro- duces the length-controlled AlpacaEval [26] benchmark. Park et al. [34] catalogue six distinct bias types, construct counter-biased datasets, and fine-tune judges on these counter-examples. Additional studies on LLMJ biasing factors document, for example, position bias [46, 48, 39, 43], leniency bias [41]. and sentiment or authority biases [46]. Few works have also uncovered biases in human ratings themselves [5, 24]. On related but different direction, Buyl al. [4] studies how discretion is exercised by humans and LLMs when making choices based on key guiding principles.
Despite recent advances, prior work has systematically examined divergences between human and L.LM judgments in a comprehensive way. These discrepancies may carry negative connotations (e.g., LLM biases), be value-neutral, or even positive (e.g.. human biases). Yet such a comparison is essential: clarifies the strengths and limitations of LLM judges and underscores that corrections should be made relative to human preferences, not in absolute terms, if human alignment is the goal. Otherwise, eliminating a feature valued by both humans and LLMs could inadvertently widen the misalignment. To fill this gap. work offers principled way to analyze the systematic differences between human and LLM judges.
Problem setup
We consider two evaluation scenarios: absolute and relative ratings. In the absolute case, an entity under evaluation (e.g., LLM human) is presented with an input prompt I and produces

---

Prompt (Input 2)
Method logprobs
Ratings logprobs
Z
X
1 -8.127.
' -2.127.
LLM judge
'01": -19.892,
2': -20.602
Fit model for human labels Y
(P(Y' k/I, 0)
Sample ratings with CoT
Response(s) (output O?)
Compute estimates for B, 7, ay's
CoT
CoT 1
CoT m
Figure The logit trick model fitting. This procedure allows us to fit the statistical model without observing human latent scores Z First, the LLM judge rates pair (prompt, response(s)). Second, compute/estimate the probability each score Third, we process probabilities, obtaining the LLM scores Z' Finally, we fit an ordinal logistic regression model for human ratings Y given and covariates to explain the gap hetween human and I.I.M scores.
text output O Given the pair (I,O), both a human evaluator and LLM judge assign scalar ratings Yhh y E I. where the numerical values reflect absolute assessments, such whether the response is satisfactory. In the relative case, two entities generate responses O^ and On to the same prompt I. and evaluators provide comparative judgments h and Y that encode one of the options in {OA wins, tie, Og wins). We denote the output by O (OA, OB) in this setup.
Our goal develop a statistical framework, Bridge, that bridges human and LLM scores, Y and Y', under two key assumptions: (i) both are influenced by shared latent score 2h f(1, O) E R representing human preferences, and (ii) LLM judgments may be systematically different from human judgments due to additional features encoded in covariate X =g(( RRd The map is assumed be known, and can, for example, capture properties of the output O, such as formatting (e.g., markdown usage), structural attributes (e.g., number of paragraphs), stylistic elements (e.g.. sentiment), or text-quality attributes such as creativity and factuality. Bridge enables both the refinement of LLM judges and the analysis of discrepancies between human and LLM evaluations.
3
The Bridge framework
3.1
Statistical model and fitting
y E ordinal structure. For absolute scoring, these judgments reflect ordered levels of satisfaction or agreement, so that rating Y = k + is preferred over y k. For relative scoring, each level k captures the strength of preference between two responses. For instance, when K 2, judgment of y = O may indicate that O is preferred to OB. Y denotes no preference (a tie), and y indicates preference for OB. giving rise Bradley- Tradley-Terry-uue Terry-type model [3, 37].
The model. We consider discrete judgments Y
f K}. where ratings reflect clear
To model 4 and Y we use the ordinal logistic regression (ordered logit) formulation [45]. For human judgments yh, we assume the model IP k 1,0) = pk (a(a.. ... oKK )²). where
0 - -
if 0, if K,
pl ak, 2 & oakk 2²)),
( -z") - oaay - z²), otherwise,
is the standard logistic function (i.e., sigmoid function), ag < ag+1 ordered real cutoffs, and t = f(I,O) O) a latent factor representing human preferences. The corresponding model for LLM judgments Y replaces cutoffs ag with and Z with Zz, assuming Z' 32 + + TTx, X where X gI,100 O) gd captures features associated with deviations between LLM and human evaluations. The function g may represent either engineered features (e.g., text length or sentiment) learned representations (e.g., neural embeddings of (I, O)). This formulation allows model systematic differences and refine LLM outputs accordingly.
Whille O may take other forms in principle, restrict attention to textual outputs.

---

Model fitting via the logit trick. Given a pair (I, if the score Z', parameters 8, and cutoffs {au)k=1 == [nh fk=1 were known, we could directly analyze the gap between judges and adjust LLM predictions to align with human preferences. However, none of these quantities are observed in principle. Moreover, the human latent scores 7h are unobserved, which makes parameter estimation appear intractable, even assuming access to Z Nevertheless, when human labels {Y?) Ri=1 are available for a set of input-output pairs {14, 0,) we can leverage a technique we call the logit trick to cireumvent this issue. For now, we assume P(Y k It O:) for all k E {o..... K} can be computed exactly (or estimated with high precision). Details on how this step is implemented are provided later in this section. The model fitting algorithm via the logit trick detailed in the following box:
Model fitting via the logit trick
1. For each example (4, Ot), compute IPYY' 44 O.) for all k E t0....,KK K}.
2. Compute Z and cutoffs {ne) by solving:
K
arg min in < ER, ER
( - k 14 O.)
3. Fit the ordinal logistic model to the human labels (M R== using maximum likelihood, with Z = (1/3)Z; (1/3)7 X; as inputs, i.e.
K
(fâu) (fam) 3,9)
arg max 0 <.n
ΣΣ =
=
i=1 K=
BER
When the model for Y is correctly specified, the optimization in step 2 recovers the true values of {Th k and { up to an additive constant. To ensure identifiability, we fix 7 0. As simpler alternative, one may define z -o- (P(Y! only one probability and ignores the full distribution of Y which can be suboptimal under model misspecification. At test time, new values of Z can be derived using the estimated thresholds {mk K {
0 4, O.)), but this approach uses
and solving an optimization like in step 2. A better option, if the application permits, is computing test points Z jointly with those from training data. After the model is fitted, we can define the predicted human latent scores as 2h (1/8)Z' - (1/8)4TX X.
We consider two strategies for computing P( and return exact values: identify the tokens associated with each possible outcome k: compute their probabilities from the LLM output distribution, and sum them. This method is computationally efficient and provides exact probabilities, but it requires prompting the LLM output the final score without intermediate reasoning. When reasoning steps are used, these probabilities may become biased'. As an alternative, we employ a chain-of-thought (CoT) prompting strategy, in which the LLM produces reasoning followed by a rating. In this case, we sample m outputs from the LLM and estimate P(Y k I, O) via empirical frequencies. In this case, IP k I, O) is not computed exactly. While this approach is more computationally intensive, it typically yields higher- quality judgements by leveraging the model's reasoning capabilities. In both strategies, we regularize the output probabilities by adding a small constant (c.g., 0.01) to each P(Y = k 0) before renormalizing to ensure they sum to one and avoid degenerate distributions.
I) The first is based on log probabilities
Figure provides a visual overview of the entire procedure, and Appendix A contains the prompt templates used collect LLM judgements in both the log probabilities and CoT cases.
'When Y y E {0, 1). however, these two approaches are equivalent. This observation also clarifies why we call it the logit trick: under the binary judgements, 7' is precisely the logit of PPYY'
T, T,
'Once the L.M generates reasoning steps R (a sequence of tokens), its subsequent score is conditioned on R. biasing the probabilities toward outcomes that are more compatible with that specific reasoning. To avoid this hias, we marginalize over all possible reasoning paths, i.e., P(Y
T,O) Σ, IP(Y' k T, O, R (R =r r 1, and estimate this sum via Monte Carlo sampling rather than relying on single conditional probability F(Y' k 1, r).

---

Model extensions. We work under the setup in which Y and Y' are discrete and obey notion of ordering because it covers the majority of practical use cases, including binary ratings (i.e.. yh, y' E (0. 1)). However, it does not account for continuous or unordered categorical ratings. We develop extensions of our model deal with those cases, and, due to space constraints, we include them in Appendix E.
3.2
Non-exhaustive set of applications
Better alignment and calibration. A key application of our framework arises when practitioners seek to improve the quality of LLM-generated judgments, especially in low-resource settings where human-labeled data is limited due the high cost of annotation. In such scenarios, fine-tuning LLM judges is often impractical or even impossible when inference APIs are used. We propose using our model to enhance both the alignment and probabilistic calibration of LLM judgments. Alignment between LLM and human judgments is crucial for enabling high-quality evaluations at reduced cost, Calibration equally important, as well-calibrated models yield more interpretable outputs, facilitating uncertainty quantification. Moreover, well-calibrated scores do not suffer from position bias (relative to humans) by definition, for example.
We quantify alignment by measuring the discrepancy between LLM-inferred rating probabilities and the target distribution P(Y k I, O), using human labels and cross-entropy loss. Specifically, expect that the model-implied probabilities (r ".. agK 2*) more closely approximate PP(Y. k I, O) than the raw LLM outputs IP(Y" = k 1, in terms of the Kullback- (KL) divergence [20]. Additionally, we also check alignment in terms of accuracy. For calibration, we adopt the class-wiss notion [35], which requires
P(Y k pı (a((1. ................................ 0k, -²m) p) = for all p E [0, & E K).....K).
property like this unlikely to hold for raw LLM predictions, but becomes more plausible when LLM scores are corrected using our model, assuming it is reasonably well-specified. Analogous calibration methods are common in classification tasks, such as Platt scaling [36], where a logistic regression is applied to uncalibrated classifier scores to improve their probabilistic interpretation.
As we demonstrate in our experiments, even under the simplifying assumption 7 0. i.e. without using any covariates, our model yields improved LLM judgment predictions at test time, highlighting its practical utility in resource-constraineed settings.
Human- discrepancies quantification and formal testing. Another important application is the detection and quantification of human- -LM judgement divergences. To that end, assume X contains possible sources of differences between LLM and human judgements, and we want to better understand which differences
relevant by analysing y. As detailed in Section 3.3, the estimator î for the j-th entry of y is asymptotically normal, i.e., (n/Vf+1.+1) (V)+1+++)44366
7:p) converges in distribution to N (0, 1) as n + oo, where V is a variance estimate derived from the data. This result enables formal hypothesis testing" e.g., testing Ho % versus H1 : % * 0, using the p-value
p-value- = 24h (-Vn/VVtL.
1951
where denotes the distribution function of the standard normal distribution. Additionally, the practitioner wants to control the false discovery rate (FDR) of human-LLA divergences, multiple hypothesis testing can be carried out in conjunction with the Benjamini-Yckuttch Yekutieli procedure [2]. Confidence intervals can also be constructed, discussed in Section 3.3.
3.3 Asymptotic distributions of our estimators
First, we analyze the properties of estimators for the cutoffs {nH I and latent scores (Z) V under the CoT prompting strategy for estimating Pi = P(Y = k L, O,). Fix n evaluation samples {(4, O,) For each i, draw m, i..dd CoT judgements (Y/ 1ma )m= and estimate pa with Pik.m., = Σm" 1(x! = k//mm. Denote 1) = (7....,,x') and define the empirical and population losses as
K
K pa 2i) Pik
Qn,m,, (n), 21:m) =
pa( mm.. On(n,Z1..))
A=
'ere, rejecting He means feature systematically shifts the LLM's judgments relative humans.
5

---

Constrain 7), a.. to O, = En R =0 <...< nk} and 2 I-, M) for some large M. Proposition 3.1 (Consistency of CoT estimates fis and Zi). Under Conditions C.I and C.2 (stated in Appendix C.I), the estimator (1). Zim) E arg maa. Qn,ma (1 21:) satisfies
ym,lli )m) , Z converges in distribution to a mean-zero distribution with covariance (defined in Appendix C.I) ).l))ar as my + oc.
When the log probabilities are used to extract Pik exactly from the LLM output, the population loss Qn is known and 2: Z for all i. Next, we derive the asymptotic distribution of (B. 9) as tends infinity, under either log probability CoT-based estimation of the latent scores. Write the ordered logit model as P\ k 4, O+) - (0; Z X;) pk (a........ aK (1/8) (1
(1/3)y X.).
Define the Fisher information matrix I(0") = E [V) logly? (0"; Z {
Theorem 3.2 (Asymptotic normality of (3,4)). 4)) Under Conditions C.3-C.5 (stated in Appendix C.I), let the MLE 0n = (āi,...., āx, () maximize the log-likelihood
K
e,, x) X)
1 =k} ko La (0; X,).
over 0 = (aI...... ................................ OK 3, 3. also assume Conditions C.I and C.2 and let nz/ e/mmn + 0 as n + co. Then
3) If the CoT prompting strategy is used to estimate P(Y; = k O,),
3-33 3" 9-
vn(On - S N(0,I(0)-)) and Vn
4 NO, (I((")-"))) as n - oc.
Consistent variance estimator and confidence intervals. To estimate the variance of (8, 9), calculate the observed Fisher information matrix: (Q.)) = - (1/n)) Vien (d,; 2", X) at the MIE in (ā.,...,, dx,, 3, 9), where Vien is the second derivative matrix of en with respect to all parameters (a1, ...,,....e aKK (, Let V Ions -1 Extract the 7-block V(8.7). which is the bottom right (1 + d) x (1 + d) block of V. The 100(1 % marginal confidence intervals (CIs) for the parameters are
Vn1
8 + o0/222
3 + 21-a/2
j= j= Vn
Vn
()] The joint confidence region is (8,7) ml(B,>) (B, 4) (8,7) - S x.+1,1o)):
Under previous conditions, the variance estimator consistent, and the coverage probability converges to 1- a as n + Do. Additionally, Appendix C.3 outlines methods for constructing confidence intervals for differentiable functions of 9. These methods can be employed to construct prediction intervals and to evaluate the "partial effect" a covariate, as further detailed in Appendix C.3.
4 Understanding and narrowing the human-LLM gap in practice
In this section, examine the applications described in Section 3.2 using real-world data and popular LLM judges. We have included extra semi- -ynthett (more controlled) and robustness-check experiments in Appendix B.
We begin by detailing the data used in our experiments.
4.1 LLM judges, datasets, and LLM judgments collection
LLM judges. We utilize six distinct LLM judges": GPT-4.1 [32], 1PT-4,1-nao [32], GPT-4o- mini [15], LLaMa-3.. 3.1-aa--.-----BB ---trrutttt [11], Medene-1-Ming [1], and Prometheus-v2 [18]. The GPTs and LLaMa-3.1--88 1-BBB represent high-performing, general-purpose models, while Selene- and Prometheus-v2 are specialized, state-of-the-art open judges.
Datasets. Our experiments are conducted using publicly available human-annotated datasets:
'The official model names are Ept-4.1-nant gpt-4.1, gpt-4o-mini- -02--ii---222--11- meta- Mama/LLama-3311888 Insttuuu Cclaat/Solene---Kiiii----- 1.tmma-3..168 1-BB, vyal/prromtthhuss--//bvvvv.vv
and prometheus-

---

BigGen Bench (BGB): BGB [17] evaluates language model outputs based on detailed rubrics across five satisfaction levels (originally from 1 to 5, converted here
4). We focus on the subset containing Engglish-languan human annotations, comprising 695 instances across 77 tasks and nine evaluated capabilities (e.g. planning, tool usage). Each instance has responses from four different models, totaling 2780 data points. We exclude few data points with invalid annotations.
[40], derived from Chatbot Arena [6]. This dataset consists of 100k queries, each responded to simultaneously by two anonymous models, with user preferences annotated as either a clear choice or "good"/"bad" ties, where both responses are equilavently good or bad. We randomly select a subset of 5000 queries that are not multi-tum conversations and merge "good"/""ba" ties into a single category.
Chatbot Arena (CA): We use the dataset arena- www.ammaaaeaeeeeeeeeeaeee
Judgment collection. Following Section 3, we gather L.MM judgments via two distinct prompting methods. The first explicitly instructs the judges provide only rating without explanation, allowing us to extract log probabilities for each rating directly. The second prompts the judges to elaborate on their reasoning before explicitly stating their final judgment, from which we sample 50 times to estimate rating probabilities. We adapt distinct prompts for absolute and relative ratings: for absolute ratings, we modify reference-free Prometheus prompts from Kim et al. [18]; for relative ratings, we adjust AlpacaEval 2.0 [26] (for log probabilities) and ArenaHard prompts' ? [25] (for chain-of- of-thought reasoning). To ensure quality in our analyses and comparability S across judges, we retain only instances where at least 25 valid -og-eereateed scores were produced within Ik output tokens by all judges; this filtering removes at most 10% of instances per dataset. In our main experiments, we use log probabilities for closed models (GPTs) and chain-of-thought (CoT) sampling for other judges. We adopt CoT for open models since it generally yields higher-quality judgments, whereas for closed models, it is prohibitively costly to run, so we rely only on log probabilities. Prompt templates are provided in the Appendix A.
4.2 Application 1: Improved LLM judgements with few human annotations
In this application, our goal is to improve the performance of LLM judges using only a small number of inman-annotated data points. This scenario is particularly relevant since (i) obtaining high-quality human annotations is costly, and (ii) fine-tuning an LLM judge becomes challenging when labeled examples are limited". In this section, we demonstrate that our method can still provide benefits even without using covariates (i.e,, setting 7 = 0), which can be hard to use when the training set is small.
Metrics. To evaluate judge quality, we consider three metrics: (i) cross-entropy loss, comparing the predicted rating probabilities with the actual labels; (ii) calibration error; and (ii) accuracy. which involves selecting the predicted class with the highest probability and comparing it with true labels. The results reported are averages across all judges. To compute the calibration error, we first calculate the error for each class individually and then average them. For class k, the steps for computing calibration error are: (i) using a probabilistic classifier (any of the methods reported in Figure 2), predict the probability of class k for all Niee test data points, obtaining {pks Iise (ii) discretize {pxr Pic into 10 bins and, for each bin, compute the difference of the average predicted probability and the relative frequency of class ki and (iii) average these differences. To construct the 121e {
bins, we use equally spaced quantiles of
Data splitting. Using a fixed random seed, we split each dataset into training and testing sets with an 80:20 ratio. For Chatbot Arena, we randomly divide data points into training and testing subsets. For BigGen Bench, we ensure instances do not overlap between the training and testing sets, simulating a realistic and challenging scenario in which new, unknown queries appear at test time. For given sample size ner E {20, 40, 80, 160, 320), we randomly select ner points from the full set of training queries to fit our models. Across all datasets, we perform this procedure using 10 different random seeds, and the reported results reflect averages and standard deviations across these splits.
Methods. We two different versions of our method, assuming an ordinal structure of responses ("ordinal"", default model defined in Section 3) or not ("multinomial", Appendix E). Regarding baselines, we primarily focus two: the first baseline ("Raw") directly utilizes the raw probability
ArenaHard has 5 levels: response A is much better than B. A is better than B, A and B equally good. B is better than A, response is much better than A. Given that we use three levels our experiments. we compute the level probabilities estimates, and then convert to three levels by summing the probabilities of edge classes. Moreover, Bridge is still applicable in cases where only inference API is provided.
7

---

BigGen Bench
entropy loss
2
Calibration error
0:20
2.00
0.15
accuracy
1.80
0.35
1.60
0.10
0.33
1.40
0.05
200
300
300
100
training sample size
training sample size
training sample size
cross entropy loss
2.00
0.42
Chatbot Arena
Calibration error
1.75
0.20
accuracy
0.40
1.50
1.25
0.10
0.36
200
300
0
300
100
300
training sample
training sample
training sample
Raw
LogReg
Ours (multinomisl)
Ours (ordinal)
Figure Performance comparison our proposed methods, logistic-regression baseline, and raw LLM judgments across all datascts. Our methods consistently match or outperform the basclines, notably excelling on BigGen Bench, likely thanks to sensible inductive biases.
outputs from the LLM judges, while the second baseline ("LogReg") involves fitting a multinomial logistic regression model on top of these probabilities potentially achieve better performance; this last baseline can be seen as a naive version the "multinomial" approach but lacks a principled modeling foundation and is not interpretable. Additionally, explore providing in-context learning (ICL) examples to the judge as another baseline, reporting results for this experiment in Appendix B; this last method performs poorly compared to other approaches.
Results. Figure 2 presents the experimental results. Across datasets, the different versions of our method and the logistic regression baseline outperform the raw LLM judgments consistently all metrics. Notably, our methods never underperform relative to the baselines and significantly excel on the BigGen Bench. This advantage is likely due to the effective inductive biases provided by our models. Interestingly enough, the two different versions of our method perform well. However, the "ordinal" (default) method will often be preferable since it is simpler and more interpretable, explores the notion of order in the data and has fewer parameters.
4.3 Application 2: Detecting and testing for Human-LLM gaps
Different from the previous experiment, this analysis incorporates set of covariates X that represent potential sources of discrepancies LLM judgments relative to humans. We focus exclusively on covariates derived from the outputs O, they are more direct to collect and interpret. Initially, consider 47 interpretable covariates, comprising lightweight automated metrics (e.g., word count, sentiment polarity) and features extracted via LLM scoring (e.g., conciseness, fluency, creativity). For relative ratings, we compute differences between covariates from the second and first responses. We then cluster these covariates based on their correlations, substantially reducing their number by Rs 30% for BigGen Bench and Rz 20% for Chatbot Arena. Appendix D gives a comprehensive description of the used covariates, the clustering algorithm, and the resultant covariate clusters. After clustering, we extract the first principal component from each cluster and standardize the resulting variables have zero mean and unit variance. Subsequently, we apply our proposed method, calculate p-values, and adjust them using the Benjamini-Yckuticli [2] procedure" for false discovery rate (FDR) control when conducting multiple tests the same time.
Results. Tables and 2 summarize our findings for BigGen Bench and Chatbot Arena, respectively. These tables include only covariates that show statistically significant contribution for at least one judge; for full tables and unadjusted p-values, see Appendix B.6. The direction of these effects
We use the Python package statsmodels for this adjustment: attpp://wwe http://ww..tttttmto.oooo statsmodels...... woweeelssssrrrrreeeeeeeeee org/dev/ senerated/stateene stats multitest muw.itest..mtiiplleeees... www.smeetesstttss..mmmommmmmmtl. html

---

Table 1: Human-LLM, judgement discrepancies on BigGen Bench
GPT-4.I-nano
GPT-4.1
GPT-4o-mini
LLaMa-3.1-8B-I
Selene-I-Mei
Prometheus-v2
Writing Quality
-0,380e
0.100
-0.02
-0,22
-0.02
-0,22eea
Text Length
-0.83*ee
-0.39***
-0.43*ee
-0.78***
-0.44
-0.74
Positive Sentiment
+0.318s
-0.12*
-0.15**
-0,22ea
-0.18ese
-0.21
Layout Density
- 0.23
-0.15*
0.11
0.21
-0.13
0.117
Causal Markers
-0.19**
0.09
0.10
0.12
-0.88
0.07
Structure Counts
0.16*
0.11
0.29**
0.12
0,29*e9
Sentiment
0.24**
0.10
0.11
0.11
0.09
0.10
Code Block
0.20**
0.07
0.09
0.22*ee
0.14**
0.20*e9
Character Density
0.25
0.13
0.12
0.20**
0.13
Compound Sentiment
0,27sea
0.06
0.08
0.16
0.13**
0.19es
Question Count
0.16
0.09
0.13
0.13
0.11
0.15
Significance: *** 0.01, ** 0.05, " <. 0.10.
Table 2: Human -uman-LLM judgement discrepancies on Chatbot Arena
GPT-4.1-nmo
GPT-4.1
GPT-4o-mini
LLaMa-3.1-8B-I
Selene- I-iin
Prometheus-v2
Text Length
-2,05000
-0.54900
-1,02900
-161
-1.1709
-1.00..*
CreativttyEngugemennt
-1.2700e
-0.32
-0.640ea
-110**
-0,78
-0.77
Bold Text
0.74%*
0.25+00
0.50).*
0.77ee
0.66906
0.62
Significance: #*e, p
p.x. 9p
positive values indicate attributes preferred more strongly by LLM judges, whereas negative values indicate attributes preferred by humans. Some insights about the results are:
important:
Across datasets and judges, longer responses receive systematically lower scores, showing that LLM judges favor brevity relative humans. This contrasts with Dubois et al. [8], who argue that length-controlled (assuming LLMs are positively biased towards lengthier responses) scoring aids alignment. In Appendix B.6, we show that GPT-4-Turbo (main judge on AlpacaEvvl) exhibits the same pattern. The discrepancy with Dubois et al. [8] likely stems from methodological differences, our instance-level analysis versus their system-level aggregation, which introduces extra complications to comparability. At the end of the day, working on the instance level is more direct and reliable way of drawing such conclusions.
Human annotators reward creativity and engaging responses more than LLM judges, a discrepancy most pronounced on Chatbot Arena. This pattern is intuitive: users of that platform often are there "play" with generations, while the LLM judges were never instructed value creativity. similar conclusion can be drawn from the "Positive sentiment" dimension in BigGen Bench.
Bias profiles overlap considerably across LLM judges, suggesting common underlying biases that are inherited from similar training sets and procedures.
In Appendix B.6, duct extra related analy- ses. For example, check how robust our findings are for different sample sizes. We split Chatbot Arena queries into technical and non-technical categories us- ing GPT-4o-mini as a zero- shot classifier. For technical queries, HumanL.MM diver- gences were not statistically significant, suggesting that discrepancies arise mainly in subjective, non-technical content. An important limitation of this analysis the reduced sample size after splitting, which lowers statistical power.
wiBwis It. (.0) - AlAsins 1.0)
0.75
Text Length
CreelivityEngegeemenn Consistency
0.25
Bold Text Sentiment
0.00
Readability Ease
Paregraph Density
-0.50
Lexical Ratio
Relative Italic
-0.75
-0.2
0.0
0.2
0.4
Povis wins H4, 0) Aome wins (l.0)
Figure 3: Covariates X are important. Dots indicate how adding covariates alters the predicted human preference, with colors marking the most influential.

---

To quantify the practical impact of the bias covariates, we zoom in on the Chatbot Arena evaluation with the Celene-I-Ming judge. Two variants of our model are fitted: the full specification, denoted P. which incorporates the gap term y X, and a "calibration" variant, Peaaii. obtained by setting y= O. For every pair of responses (OA, OB) and variants of our method, we predict the probabilities that humans prefer B over A and over B and then take their difference. Figure plots these differences, highlighting instances where biasing covariates substantially alter the prediction. Each point is colored by the covariate whose contribution Xiy is largest in magnitude, thereby revealing the dominant factor driving each discrepancy. The discrepancies have large magnitudes for some data points. As extra results, we place figures in Appendix B.6 that show that including covariates in our model can induce improved prediction performance, even though the biggest improvements obtained without the need to add extra covariates.
5 Discussion
We propose Bridge, a unified statistical framework that simultancously models ratings from both human annotators and LLM judges. The framework couples a statistical model with specialized estimation procedure, enabling (i) principled calibration/linirations of LLM scores and (ii) a clearer characterization of the divergences between human and LLM evaluations.
Limitations. The chief limitation of our approach is vulnerability to model misspecification. When the assumed data- Gata-generating process is inaccurate, owing to unrealistic distributional assumptions or omitted covariates, the resulting parameter estimates must be interpreted cautiously; please check Appendix B.3 for detailed discussion model misspecification. A second practical challenge is the construction of informative covariates X; while users can start with generic, off-the-shelf metries (e.g., response length, readability grade), domain knowledge could be needed devise variables that capture salient sources of differences between LLM and human judgments, especially when those also depend on the input and not only on the output O.
The significance of this work in today's LLM-evaluation landscape. We recognize that as LLM tasks become more sophisticated, achieving truly reliable "gold standard" through human annotation is increasingly difficult. However, Bridge is flexible by design and is not tied to a single definition of the gold standard. Whether the benchmark consists of individual human judgments, consensus among multiple annotators, or even alternative proxies, Bridge is meant to detect and reduce inconsistencies between whatever reference is chosen and LLM assessments. Moreover, aligning LLMs with human preferences and judgments will continue to be important for many real-world applications, especially where trust, safety, and social acceptance matter.
Observational vs. experimental data. Our experiments rely exclusively on observational data, the sense that do not intervene on X. This choice has both strengths and limitations. On the one hand, observational data capture the diversity and unpredictability of real user-LLM interactions, enhancing the external validity of our findings. On the other hand, because the data not generated under controlled interventions, our estimated parameters should be interpreted as descriptive associations rather than causal effects. Confounding variables and selection biases may also influence these relationships.
Future work and extensions. Promising directions include developing estimation routines robust to model misspecification, extending the framework to automatically infer divergence factors, and leveraging representation learning to construct covariates X on the fly. In principle, Bridge can also incorporate text embeddings as X (provided enough training data and regularization), though we currently see no clear advantage in doing so: nevertheless, this remains an interesting avenue for future exploration. Extending Bridge to open open-ended, langrae- evaluations is an important direction for future work and will likely require principled ways represent free-form judgments as comparable quantities. Moreover, Bridge can also be applied in settings where model outputs span multiple modalities (e.g., image, video, audio). In such cases, however, constructing meaningful covariates X may be more challenging. We encourage adoption and extensions of Bridge in these directions for future work.
Acknowledgements
This paper supported by the National Science Foundation (NSF) grants 20277727737, DMS- 21 13373, DMS-24 241418, SES-1846744 and a gift from OpenAl.
10

---

References
[1] Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisupati, Toby Drane, and Young Sun Park. Atla selene mini: A general purpose evaluation model, 2025. URL https: //arxiv. wwwp.oogg/alll www.:/rrggbb//a/eeee 17195.
[2] Yoav Benjamini and Daniel Yekuticli. The control of the false discovery rate multiple testing under dependency. Annals of statistics, pages 1165-1188, 2001.
[3] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345-345,, 1952.
[4] Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio Cesar Vieira Machado, and Flavio du Pin Calmon. Ai alignment at your discretion. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 3046-3074, 2025.
[5] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or Ilms as the judge? study on judgement biases. arXiv preprint arXiv:2402. 10669, 2024.
[6] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
[7] Tom De Smedt and Walter Daelemans. Pattern for python. The Journal of Machine Learning Research, 13(1):2063-200 2012.
[8] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controoled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.04445, 2024.
[9] Rudolph Flesch. A new readability yardstick. Journal of applied psychology, 32(3):221, 1948.
[10] Eric Gilbert. Vader: parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216-225, 2014.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 4rXiv:2407.21788, 2024.
[12] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey llm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
[13] R. Gunning. The Technique of Clear Writing, McGraw-Hill, 1952. ISBN 9787000014190. URL https://books: www.goooogg...ooo.oo wtt.coom/ooooo. www.bboiidoooooMaM.....
[14] Hui Huang, Yingqi Qu, Xingyuan Bu, Hongli Zhou, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. An empirical study of llm-as-a-judge for Ilm evaluation: Fine-tuned judge model not a general substitute for gpt-4. arXiv preprint arXiv:2403.0253 2024.
[15] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[16] Wendell Johnson. Studies in language behavior: A program of research. Psychological Monographs, 56(2):1-15, 1944.
[17] Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. The biggen bench: A principled benchmark for tne-grained evaluation language models with language models, arXiv preprint arXiv:2406.05761, 2024.

---

[18] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024.
[19] J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975.
[20] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[21] Yebin Lee, Imseong Park, and Myungjoo Kang, Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.0600 2024.
[22] Dawei Li, Bohan Jiang, Liangji Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjce, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of Ilm-as-a-judge. arXiv preprint arXiv:2411.16594 2024.
[23] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: comprehensive survey on Ilm-based evaluation methods. arXiv preprint arXiv:2412.0557%, 2024.
[24] Tianle Li, Anastasios Angelopoulos, and Wei-Lin Chiang. Does style matter? disentangling style and substance in chatbot arena, august 2024a. URL https://blog. Imarena. ai/slog/0244sstylee control, 2024.
[25] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 19939, 2024.
[26] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of fnstraction-follooinn models. https: // thub. wow///tau-laaa/// alpaca_evallll
eval, 5 2023.
[27] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81,20 2004.
[28] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nig evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16633 2023.
[29] Steven Loria. TextBlob: Simplified Text Processing, 2025. URL https://textbbo.. readthedocs. io.
[30] David D Malvern and Brian J Richards. A new measure of lexical diversity. British Studies in Applied Linguistics, 12:58-71, 1997.
[31] Harry Mc Laughlin. Smog grading-a a new readability formula. Journal of reading, 12(8): 639-646, 1969.
[32] OpenAl. Introducing gpt-4.1 in the api. https: httee//////p/////aaaaoeo/o/////r April 2025. Accessed: 2025-08-177 17.
[33] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[34] Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. arXiv preprint arXiv:2407.0m6 4.45066551,1 2024.
[35] Maja Pavlovic. Understanding model calibration-a gentle introduction and visual exploration of calibration and the expected calibration error (ece), arXiv preprint 2rXiv:2501.19047, 2025.
12

---

[36] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances large margin classifiers, 10(3):61-74, 1999.
[37] PV Rao and Lawrence L Kupper. Ties in paired-comparison experiments: A generalization of the bradley-terry model. Journal of the American Stutistical Association, 62(317):194-204, 1967.
[38] Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matjja Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests. arXiv preprint arXiv:2412. 13091, 2024.
[39] Lin Shi, Chiyu Ma. Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. Judging the judges: systematic investigation of position bias in pairwise comparative assessments by Ilms. arXiv preprint arXiv:2406.0079),,000 2024.
[40] Kelly Tang, Wei-Lin Chiang, and Anastasios N. Angelopoulos. Arena explorer: A topic modeling pipeline for IIm evals & analytics, 2025.
[41] Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dicuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in Ilms-as- judges. arXiv preprint arXiv:2406. 12624, 2024.
[42] A. W. van der Vaart. Mand Z-Extimators, page 4184. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
[43] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.1 10.55.199666 1726,, 2023.
[44] Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. Rocketeval: Efficient automated IIm evaluation via grading checklist. arXiv preprint arXiv:2503.05 055504 2025.
[45] Jeffrey M Wooldridge. Econometric analysis of cross section and panel data. MIT press, 2010.
[46] Jiayi Ye. Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin- Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024.
[47] Hongli Zhan, Muneeza Azmat, Raya Horesh, Junyi Jessy Li, and Mikhail Yurochkin. Spri: Aligning large language models with context-situated principles. arXiv preprint arXiv:250.003397, 2025.
[48] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
13

---

A Prompt templates
A.1 BigGen Bench prompts (absolute ratings)
For BigGen Bench, the system prompt is instance-dependent. Therefore, we do not report them here.
BigGen Bench (logprobs)
###Task Description:
An instruction (might include an Input inside it), response to evaluate, and score rubric representing evaluation
criteria are given.
1. Write score that is integer between and 5. You should refer to the
rubric.
2. Your output must be only an integer number between 1 and 5, and nothing else
3. Please do not generate any other opening, closing, and explanations. Do not include any spaces linebreaks before your judgement."
###The instruction to evaluate: (instruction)
###Rese to evaluate: (response)
###Score Rubrics: (rubric)
###Score:
BigGen Bench (CoT)
###Task Description:
An instruction (might include Input inside it), response to evaluate, and score rubric representing evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating
general.
2. After writing feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look follows: "(urite feedback for criteria) [RESULT] (an integer number between and 5)
4. Please not generate any other opening, closing, and explanations.
###The instruction to evaluate: (instruction)
###Responee to evaluate: (response}
###Score Rubrics: (rubric)
###eedbackk:
14

---

A.2
Chatbot Arena prompts (relative ratings)
Chatbot Arena system prompt (logprobs)
You are a highly efficient assistant, who evaluates and rank large language models (LLMs) based on the quality of their responses to given prompts. This process will create leaderboard reflecting the most accurate and human human-preferred answers.
Chatbot Arena user prompt (logprobs)
require leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding outputs. Your task and select the model that produces the best output from human perspective. The input prompt can possibly include an image; in question instruction will be related to
assess these responses,
that case the that image and you take that into account.
#* Instruction
{
"instruction": "(instruction)",
## Model Outputs
Here the unordered outputs from the models. Each output is associated with specific model, identified by a unique model identifier
"model_ide" iidetiieeettiier "A", "output": "(output_1)" 1
"model "model_identiff""" identifier"; "B", "output": "[output_2e _
## Task
Evaluate the models based the quality and relevance of their outputs, and be prepared for the possibility model clearly produces the best output, respond with its identifier. However, if the reaponses are equally good or bad and result in a tie, return C. Your output must contain only of these identifiers (no quotes, spaces, B, or C.
tie. If
lines): A,
## Judgment
Best Model Identifier:
15

---

Chathot Arena system prompt (Co'T)
Please act the responses provided by two AI assistants to the user prompt displayed below. The input prompt can possibly include an image; in that case the user question or instruction will be related to that image and you must take that into account. You will given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant 's answer is better.
impartial judge and evaluate the quality of
Begin your evaluation generating your answer to the prompt. You must provide your answers before judging any answers.
When evaluating the assistants' answers, compare both assistants' answers with your answer You must identify and correct any mistakes or inaccurate information.
Then consider if the assistant's anawers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Mote when user prompt has any ambiguity or more than one interpretation, it more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based assumptions. Relevant means all parts of the response closely connect or are appropriate
being asked. Concise means the response is clear and not verbose or excessive.
what
Then consider the creativity and novelty of the assistant's 's answers when needed. Finally, identify any missing important information in assistants' answers that would beneficial to include when responding to the user prompt.
After providing your explanation, you must output only of the following choices your final verdict with label:
1. Assistant A is significantly better: [[A>>B]]
2. Assistant A is slightly better: [[A>B]]
3. Tie, relatively the same: [[A=B]]
4. Assistant B slightly better: [[B>A]]
5.
Assistant B is significantly better: [[B>>A]]
Example output: "My final verdict is tie: [[A=B]]". ".
Chatbot Arena user prompt (CoT)
<| User Prompt (instruction)
<he Start of Assistant A's Anaver {output_ 1} <h End of Assistant A's Answer
The Start of Assistant B's A [output_2] _
The End of Assistant B Answer |>
16

---

A.3 Prompts for extracting LLM-scored covariates
The following prompts were used to obtain LLM-scored covariates, adapted from the prompts by [28].
Coherence
You will be given LLM response, generated in reply human prompt. Note that only the LLM response provided for evaluation.
Your task is to rate the LLM response one evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Coherence (0-5): Assess how well the LLM response structured and organized.
A highly coherent response (score 5) will present ideas in a clear, logical progression. The text should flow naturally, with each sentence and paragraph connecting logically to build coherent narrative or argument.
score of indicates a response that is disorganized, disconeected, otherwise hard to follow.
Evaluation Steps:
1. Read the LLM response carefully to understand its content and structure.
2. Assess overall structure and flov. Determine whether the response well-organized progress
whether ideas and arguments logical order.
3. Assign score for coherence on scale of 0 to 5, where 0 is the lowest and 5 is the highest based on the Evaluation Criteria.
Provide only a single numeric value (e.g., 0.75) without any additional text.
Response:
(response}
Factuality
You will be given one LLM response, generated in reply to a human prompt. Note that only the LLM response is provided for evaluation.
Your task to rate the LLM response on one evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to as needed.
Evaluation Criteria:
Factuality (0-5) Assess how factually accurate and evidence-based the LLM response is. highly factual response (score 5) will contain accurate, verifiable information. score of 0 indicates a response that includes inaccuracies or unsupported clains.

---

language used precise easy to Evaluation Steps:
1. Read the LLM response carefully to identify its content.
2. Check if the response contains information that can be verified accurate.
3. Assign score for factuality a scale
5, where is the lowest and is the highest based on the Evaluation Criteria.
Provide only single numeric value (e.g., 0.75) without any additional text.
Response:
(response)
Clarity
You will be given LLM response, generated in reply human prompt. Note that only the LLM response provided for evaluation.
Your task is to rate the LLM response one evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Clarity (0-5): Assess how clear understandable the language of the LLM response is. A highly clear response (score 5) will communicate ideas in straightforward and unambiguous manner. of indicates response that vague confusing.
Evaluation Steps:
1. Read the LLM response carefully to understand its content and intent.
2. Evaluate whether follow.
3. Assign score for clarity on scale of 0 to 5, where 0 is the lowest and 5 is the higheat based the Evaluation Criteria.
Provide only a single numeric value (e.g., 0.75) without any additional text.
Response: (response}
Conciseness
You will be given one LLM response, generated in reply to human prompt. Note that only the LLM response provided for evaluation.
Your task is to rate the LLM reaponse on one evaluation metric.
Please make sure you and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
18

---

Evaluation Criteria:
Conciseness (0-5): Assess how succinct and vo-the-point the LLM response is. A highly concise response (score 5) will deliver its message without unnecessary verbosity. A score of 0 indicates response that is overly wordy includes redundant details.
Evaluation Steps:
1. Read the LLM response carefully to capture its core ideas.
2. Evaluate whether the response expresses its content in succinct manner.
3. Assign a score for conciseness on a scale of 0 to 5, where is the lowest and 5 is the highest based on the Evaluation Criteria.
Provide only a single numeric value (e.g., 0.75) without any additional text.
Response: (response)
Creativity
You will given one response, generated reply human prompt. Note that only the LLM response is provided for evaluation.
Your task
the LLM response
evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Creativity (0-5): Assess how original and inventive the LLM response is. A highly creative response (score 5) will present ideas unique and engaging way. A of indicates response that is unoriginal or formulaic.
Evaluation Steps:
1. Read the LLM response carefully to appreciate its content and style.
2. Evaluate whether the response demonstrates inventive thought and originality in its presentation.
3. Assign score for creativity the lowest and is the highest based the Evaluation Criteria. Provide only a single numeric value (e.g., 0.75) without any additional text.
scale of 0 to 6, where 0
Reaponae: (response}
Consistency
You will given one response, generated in reply to human prompt. Note that only the LLM response is provided for evaluation.
19

---

Your task to rate LLM response on one evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer it needed.
Evaluation Criteria:
Consistency (0-5): Assess how uniform and steady the style and content of the LLM response are. A highly consistent response (score 5) will maintain a uniform approach throughout without contradictions. score of 0 indicates a response that contains conflicting information or fluctuates in style.
Evaluation Steps:
1. Read the LLM response carefully to understand its content and style.
2. Evaluate whether the response maintains consistency in its presentation.
3. Assign score for consistency on scale 0 5, where is the lowest and is the highest based on the Evaluation Criteria.
Provide only single numeric value (e.g., 0.75) without any additional text.
Response:
(response)
Engagement
You will be given LLM response, generated in reply human prompt. Note that only the response is provided for evaluation.
Your task is to rate the LLM response
evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Engagement (0-5): Assess how engaging the LLM response is to a reader. A highly engaging reaponse (score 5) will capture and sustain the reader attention effectively. A score of 0 indicates response that is dull fails hold interest.
Evaluation Steps:
1. Read the LLM response carefully to understand its content and appeal.
2. Evaluate whether
response able to maintain a reader's
interest throughout.
3. Assign score for engagement on a scale of 0 to 5, where 0 is the lovest and 5 is the highest based on the Evaluation Criteria. Provide only a single numeric value (e.g., 0.75) without any additional text.
Response: [response]
20

---

Flueney
You will be given LLM reaponse, generated reply human prompt. Note that only the LLM reaponse is provided for evaluation.
Your task to the LLM response
evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Fluency (0-5): Assess how smoothly and naturally the LLM response reads. A highly fluent response (score 5) will have natural flow and is easy to read and follow. score of indicates response that choppy or awkward in language, or is hard to understand.
Evaluation Steps:
Read the LLM reaponse carefully understand its and structure.
2. Evaluate whether the response exhibits a smooth and natural flow of language.
3. Assign score for fluency
scale of 0 to where 0 is the highest based on the Evaluation
the lowest and Criteria. Provide only a single numeric value (e.g., 0.75) without any additional text.
Response: (response)
Appropriateness
You will be given one LLM response, generated in reply to human prompt. Note that only the LLM response is provided for evaluation.
Your task is to rate the LLM response on one evaluation metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Appropriateness (0-5): Assess whether tone and style LLM response are suitable for the intended content. An appropriate reaponse (score 5) will use language and tone that fits the content and purpose. A score of o indicates response that is mismatched in tone style for the given context.
Evaluation Steps:
1. Read the LLM response carefully to understand its content.
2. Evaluate whether the tone and style are appropriate for the intended content and context.
3. Assign score for appropriateness on scale to 5, where 0 is the lovest and 5 is the highest based on the Evaluation Criteria.
21

---

Provide only single numeric value (e.g., 0.75) without any additional text.
Response: (response)
Sentiment
You will be given one LLM response, generated in reply to human prompt. Note that only the LLM response is provided for evaluation.
Your task is to rate the LLM reaponse on one evaluation metric.
Please make carefully. Please keep this document open while reviewing, and refer to it as needed.
you read and understand these instructions
Evaluation Criteria:
Sentiment (0-5): Assess the overall emotional tone the LLM response. response with a highly positive sentiment (score 5) will convey optimism and positive emotion, while a score of 0 indicates negative sentiment that conveys negative enotion.
Evaluation Steps:
1. Read the LLM response carefully to understand its emotional undertone.
2. Evaluate whether the response expresses positive emotional
3. Assign score for sentiment on scale of to 5, where 0 is the lowest (not positive) and is the highest (positive) based on the Evaluation Criteria. Provide only single numeric value (e.g., 0.75) without any additional text.
Response: (response}
22

---

B Additional empirical results
B.1 Probability Distribution Reconstruction Loss via Logit Trick
Figures and report the values of
min S. SRK ER, T.K 2 ER
Pa (7
7K 2;) P(Y = k: 1,00)/
1=1 k==
which is used in the logit trick. This metric provides a measure of how well the ordinal assumption captures LLM judgments. From the figures, we observe that the average deviation is below 0.04 for all judges. Deviations are typically slightly higher for judges whose judgment probabilities we estimate, reflecting finite-sample approximations.
BigGenBench
0.10
0.08
0.04
0.02
0.00
mete Waw---amataaaaa Matt--aamitLaamma Jama 1-88 instrrcttt
AtlaAl/Selene Minin- Ltama-Jammammmmmmm 4-88
prometheuss wectineuu---vvalpppeee www.eeeaooeeeeessseeeooooo 8x77--22-
openairgp--4 4-turbo
openaigpt- 1-nano
openatigpt-4.11
openailggt- 4o mini 2024-07-1222---111
Figure 4: Reconstruction loss for BigGenBench
Chatbot Arena
0.10
0.08
0.06
0.02
0.00
meta Haw.eaamaaaaaaaa Mamaaaammmmaaaaammmmmaaa 1--8--8--a 1i8-trrtttuctt
AtlAA/Selene- Mini- Jama- 2.1-88
prometheus www.ssseeeeepoeeeeeeeeeeeeee 8x70-22.0
gommaligtt-4tuute -ttrig
qpenaligt- L-nano
gpenaligp--4.1
gpenaligt- 4o mi-iini 2024-07-18
Figure 5: Reconstruction loss for Chatbot Arena
B.2 Controlled experiments to show the effectiveness of the framework
We have conducted two additional controlled experiments to further validate our method, both of which use semi-synthetic setups that are more realistic than purely artificial data.
23

---

First experiment: We use GPT-4o-mini to simulate human ratings on BigGenBench queries. Next, we run GPT-4o-mini again, this time artificially biasing its latent scores Z' to disfavor specific markdown features; namely, bold/italicized words, headers, and lists. For each markdown feature (corresponding to the covariates j- J = 1, 2, 3), we bias z' by subtracting X, (one at a time). We then estimate (71:72: 79) with our method. In this controlled setting, biasing toward feature should result in % = -1 while x = 0 for i * Standard errors are shown parentheses.
Table 3: Estimated bias parameters ? (with standard errors) in the tightly controlled experiment.
Setting
1/1 (SE)
1/ (SE)
%/3 (SE)
no bias
-0.26 (0.21)
-0.36 (0.35)
-0.17 (0.16)
bold/italic
(0.21)
-0.36 (0.35)
-0.17 (0.16)
headers
-0.26 (0.21)
-1.36 (0.35)
-0.17 (0.16)
lists
-0.26 (0.21)
-0.36 (0.35)
-1.17 (0.16)
Second experiment: We conduct a slightly less controlled experiment by prompting GPT-4o-mini to give lower scores to responses containing each markdown feature, one time. Since this manipulation is done through prompting rather than direct latent score adjustment, the resulting biases are not as clean; for example, the LLM tends be consistently biased against lists. Still, the results largely follow the expected direction.
Table 4: Estimated bias parameters 7 (with standard errors) in the prompt-based experiment.
Setting
711 (SE)
72 (SE)
aa (SE)
no bias
-0.555 (0.208)
-0,362 (0.345)
-0,167 (0.161)
bold/italic
-1 -1.992 (0.290)
0.466 (0.486)
-1,583 (0.234)
headers
-0.069 (0.427)
-1.014 (0.713)
-3.061 (0.352)
lists
-0.172 (0.319)
-0.229 (0.536)
-5.660 (0.260)
These experiments demonstrate that our framework can recover the direction and magnitude of induced discrepancies/biasss,, both in tightly controlled and more realistic, prompt-based scenarios.
B.3 Robustness against model misspecification
When our model is used for prediction, misspecification not significant concern; much of machine learning relies models that are not exactly correct. If our focus is on statistical inference, the model can still be valuable for uncovering discrepancies in L.IM judgments, provided the misspecification is not too severe. Moreover, the linear predictor we use is quite flexible, as we can include any basis functions of X as covariates (and then capture nonlinear relationships).
Empirically, we present below a simple simulation in which we introduce a mild nonlinearity into the LLM's latent score generation test robustness to misspecification, both for prediction and inference. We draw Z from Normal distribution (0, 1) and sample Y," from Categorical (p(o, )). We then set
TT,x z 32, + 7T X; + 6(7 x,)²,
where X; is drawn from multivariate Normal (0,, Iz) distribution and 5 takes values in (0, 0.1, 0.25, 0.5, 1, 5), controlling the degree of quadratic distortion. We set B = 7 (1, 1, 1). and a = (-1,1) 1). LLM judgments Y! are sampled from the usual ordered-logit link p(n, Z!), and we fit our original linear model, assuming Z -7% + + X,. By comparing the estimated parameters (B, 9, 2², , creases, we directly measure the impact of model misspecification. The results below (Table 5) show that we can still recover 7 (then at least we know approximately how big are the main effects; main channel of discrepancies) and predict P k 1, O) with high accuracy, even under moderate
A I,O)) to their true values in terms of mean absolute error (MAE) as in-
24

---

misspecification. We will add this experiment to the paper. Interestingly, the most affected results are the ones for and Zh, which is less interest.
Table 5: Mean absolute error (MAE) parameter estimations.
3
z
PP(Y* k: I,
0.010
0.014
0.014
0.002
0.1
0.024
0.014
0.072
0.010
0.25
0.047
0.016
0.169
0.025
0.5
0.266
0.017
0.340
0.045
0.960
0.011
0.563
0.077
5
24.958
0.345
0.789
0.117
B.4 For Application 2, how do results vary based on the amount of training data?
To assess the robustness of our method in detecting human-LLM gaps, we repeat our analysis using varying random fractions of the available data. We set significance threshold of 10% (i.e.. p-values below 10 are considered significant) and, for each LLM, treat the detection of nonzero T coefficients as binary classification problem (reject vs. not reject the null hypothesis). The "ground truth" label is determined using the full dataset. For each data fraction, we evaluate how well the method predicts these significance decisions by reporting precision, recall, and accuracy. Our results (see tables below) show that, for both BigGenBench and Chatbot Arena, strong precision and accuracy can be achieved with little as 50% the data. Recall is more challenging to improve, indicating that some discrepancies may require more data to detect reliably. For this experiment, we do correct the p-values using the B-Y procedure.
Table 6: Significance prediction performance for BigGenBench as function of training data fraction.
% Data
Precision
Recall
Accuracy
10
0.67
0.14
0.56
25
0.63
0.23
0.57
50
0.90
0.60
0.78
75
0.90
0.79
0.86
100
1.00
1.00
1.00
Table 7: Significance prediction performance for Chatbot Arena as function of training data fraction.
% Data
Precision
Recall
Accuracy
10
0.33
0.08
0.75
25
0.92
0.38
0.85
50
0.61
0.46
0.80
75
0.93
0.69
0.92
100
1.00
1.00
1.00
Overall, these results suggest that our method for detecting human-LL.M discrepancies is quite robust, with high precision and accuracy even when only half of the data is used. Recall improves with larger data fractions, highlighting the benefit of more data for sensitivity to weaker effects.
25

---

B.5
Application 1
B.5.1 ICL baseline
We conduct an additional experiment where we provide in-context learning (ICL) examples the judge, using the same training samples employed by our method, to assess whether this strategy improves performance. Given the high token count, context length constraints, and associated computational costs, we limit this experiment to single random split from Section 4.2, use only GPT4o-mini, and cap the number training examples at 80. Figure 6 compares ICL with raw LLM scores, the logistic regression baseline, and our default "ordinal" method. While ICL yields slight improvements, its benefits remain marginal relative our approach.
cross-entrr entropy loss
BigGen Bench
2.25
calibration error
0.40
accuracy
2.00
0.20
0.35
1.75
0.15
1.50
0.30
20
40
60
80
20
40
60
80
20
40
60
80
training sample size
training sample size
training sample size
LogReg
Figure 6: In-context-learingg judge provides only marginal gains
26

---

B.6
Application 2
B.6.1 Performance gains
In this section, we show that including covariates in the model can lead some performance gains the objective is human alignment. In the next figures, we compare raw LLM judgements ("raw") with the application of our method using covariates ("covs") or not ("calib"). For both BGB and CA, we have gains in terms of cross- eross-entropy loss (Figure 7), giving hints of better human human-aligned judgments, while the gains in accuracy are only more pronounced in CA judgments (Figure 9).
1.50
BigGien Bench
1.45
calla
1.40
1.30
1.25
GPT4 turbo
G
GPt4.1-1amo
GPT40 mini
LLaks-3.1 1-1000-t
Selere Mini
Prometheus V2
Chatbe: Arena
crocs (ros-entropy less
raw:
COvS
1.10
1.05
turbo
GPT4.1
nana
GPT4o
LLaMa-3,1189.
MiMini
Prometheusiv2
Figure 7: Performance in terms of cross-entropy loss.
0.3
BigSen Banch
covs
0.1
0.0
GPT4.1
GPT4.1-nano
GPT4o-mini
LLaMo-3.1-86.1
Selene-1-Mini
Chatbot Arena
Calbbrstion error
F3W
0.2
COVS
0.1
0.0
GFT4 turbo
GPT4.1
GPT4 nano
GPT4o-mini
LLMa 3.1888tt
Sceen--1.Min
Promathous v2
Figure 8: Performance in terms of probabilistic calibration.
0.50
BigGen Bench
0.45
0.10
TBW
calib
CDVS
0.30
GPT4-turbo
GPTA 1
GPT4 I-nano
GPT40-mini
1aMMa-3 1-88-It
Selene-1-Mini
Prometheus--2
0.50
Chetbot Arena
occuracy
0.35
callb
0.30
GP14.1
G-14.1-man
Prometheussw2
Figure 9: Performance in terms of accuracy.
27

---

B.6.2 Tables BigGen Bench
Table 8: Human-LLM judgement discrepancies on BigGen Bench (with no Benjamini-Yekuutilli Yekutieli correction)
GPT4-turbo
I-nano
GPT4.1
GPT4o-mini
11aMa-3.1-8B.I
I-Mini
Writing Quality
-0.07*
+0.38e0e
00.00es
-0.02
-0.22
-0.02
-0.22*
Text Longth
-0,49**
-033ese
-0.43***
-0.78
-0,44
-0.74**
Inlics
0.04
-0.01
Lists
0.07
0.07
0.08
0.09
0.11**
0.05
Headers
0.02
CreativityEngggeeeent
0.07
0.04
0.05
0.090M
0.06
0.09*
0.02
Positive Sentiment
0.18ass
-0,31
0.12***
-0.15
0.22
0.18-=x
00,2
Conciseness
0.07
0..0
-0.07*
-0.02
-0.07
-0.34
-0.02
Contrast Markers
0.01
0.00
0.02
-0.01
0.02
-0.01
-0.01
Layout Density
-0.10
0.23+e
0.15..
40.11
40,11*
-0.13*
-0,17e*
Causal Markers
00,3000
00.9000
0.09...
00,0000
-0.12**
.0,00*
-0.07
Structure Counts
0.15**
0.35
0.16
0.11*
0.294*4
0.12
0,29
Sentinent
0.12
0,24
0.10**
0.110*
0.11
0,09*
0.10
Readability Grade
0.03
0.21
0.13
-0.01
0.27*
0.07
0.27*
Exclamation Density
0.05
.0.80*
0.08
-0.6
00.01
0.01
-0.02
Readability Ease
0.10
0.39**
0
0.08
0.45
0.17
0.43
Polarity
0.01
0.04
0.02
0.00
-0.01
0.01
0.00
Question Density
0.10a
-0.09
0.06
00.11
40,09
-0,12
Paragraph Length
0.04
0.08**
0.20=0e
0,07+
0.229e4
0.2gee0
Additive Markers
Summary Markers
0.08
-0.09*
00,80**
-0.08**
-0.11**
-0.10***
-0.12*
Character Density
0.10
0.25**
0
0.12
0.23
0.20
Exelamation Count
0.03
0.139
0.03
-0.01
0.08
0..0
0.05
Lesical Ratio
0.04
-0.30
0.05
-0.02
40,00*
-0.38
41.13*
Subjectivity
0.00
0.02
0.00
0.02
0.02
0.01
0.00
Sentence Length
0.01
0013**
0,07**
-0.03
-0.10*
-0.01
-0,10
Example Markers
0.04
0.05
0.02
0.02
0.06
0.02
0.04
Compound Sentiment
0.11
0.27
0
0.08
0.16
0.13484
0.19
Question Count
0.12
0.16**
0.09
0.13488
0.1348
0.11484
0.15%40
Significance: 0.01,, 9e 0.00, Pp
28

---

Table 9: Human-LLM judgement discrepancies on BigGen Bench (with Benjamini- Yekuticli correc-
tion)
GPT4-turoo
GPT4,1-aao
GPT4.1
GPT4o-mini
LLaMa-3.1-8B-II
Selene-1--Me
Promeiheus--2
Writing Quality
0.07
40.38
0.10
-0.02
40.22*
-0.02
-0.22*
Text Length
00,900s
4083000
0.39000
-0.4300
-0.78
-0.44
-0.74*e
Italies
0.04
0.06
0.04
0.08
0.35
0.06
Text
0.04
-0.04
-0.08
-0.06
0.08
0.09
0.11
0.05
Headers
0.01
CreatvityEEngggeemmnt
Posilive Sentiment
0.18
4022**
+0.18
-0.21*s*
Conciseness
-0.07
-0.02
Contrast Markers
0.01
0.00
0.02
-0.01
0.02
-0.01
-0.01
Layour Density
0.10
0023
-0.15*
-0.11
-0.21
-0.13
-0.17
Cousal Markers
-0.3**
-0199*
0.09
-0.10
-0.12
-0.08
-0.07
Strueture Counts
0.15
0,35
0.16*
0.11
0.29**
0.12
0.29
Sentiment
0.12
0.24**
0.10
0.11
0.11
0.09
0.10
Readahility Grade
0.03
0.24
0.13
-0.0
0.27
0.07
0.27
Exclamation Density
0.05
00.88
0.08
0.06
-0.01
0.01
-0.02
Readability Ease
0.10
0.39
0.18
0.08
0.45
0.17
0.43
Polarity
0.01
-0.01
0.02
0.00
-0.01
0.01
0.01
Question Density
0.10
0.00
0.06
-0.11
-0.11
-0.09
0.11
Paragraph Length
0.02
0.04
0.01
0.04
0.05
0.33
0.07
Code Black
0.08
0.20*
0.07
0.09
0.22404
0.14
0.20pes
Additive Markers
0.05
0.00
0.06
0.00
0.03
Summary
Character Density
Exclamation Count
-0.33
Loxical Ratio
Subjectivity
Sontence Length
0.04
-0.13
00.03
-0.10
-0.04
-0.10
Example Markers
0.04
0.05
0.02
0.02
0.06
0.02
0.04
Compound Sentiment
0.11
0,27
0.06
0.08
0.16
0.139*
0.19*e
Question Count
0.12
0.16
0.09
0.13*
0.13
0.11
0.15
Significace: p- 0.01, ** p 0.05, & < 0.10.
29

---

B.6.3 Tables Chatbot Arena
Table 10: Human-LLM judgement discrepancies on Chatbot Arena (with no Benjamini correction)
Senjamini--Yeutielii Yekutieli
GPT4-turbo
GPT4, I-nano
GPT4.1
GPT4o-mini
LLaMa-3.1-8B-1
Selone-I-Mini
Prometheus-v2
Text Leagth
-0,90***
-2.05
00,4
-1002
-161
-1.17=*
-1.20
Creativity/Eagaageeee EEgagement
-0,55*
-1,274x
0032
-0.64
-1.10=4
-0.78
-0.77
Readability Grade
0.01
0.15
0.01
0.02
0.11
0.07
0.07
Consistency
0.17
0.38
0.12
-0.17
-0.32
-0.23
00.33
Bold Text
0.38eee
0,74006
0,250se
0.50
0.77004
0.66*e9
0.62eee
Causal Markers
0.09
0.20
0.11
0.15
0.13
0..1
Language Quality
0.23
0.15
0.21
0.21
Example Markers
0.11
0.21
0.15
0.20
0.16
0.15
40.26
Counts
Additive Markens
0.08
0.04
0.07
0.03
0.04
Polarity
0.09
0.15
0.09
Contrast Markers
0.06
0.10
0.04
-0.08
-0.03
-0.09
-0.05
Sentiment
0.26**
0,57**
0.14
0.34
0.41*
0.35
0,37**
Coberence
0.16
-0.47*
0.10
-0.25
-0.36
-0.28
-0.30
Linebreak Density
0.19
0.36
0.13
0.24
0.35
0.27
0.26
Lists
0.07
0.23
0.03
0.14
0.26
-0.18
0..17
Subjectivity
0.17
0.27
0.07
0.19*
0.28
0.22*
0.21*
Reudability Ease
0.17
0.49
0.11
0.22
0.41
0.28
0.24
Paragraph Length
0.24
0,51*
0.17.9
0
0.43**
0,32**
0,30**
Code Block
0.08
-0.00
0.06
-0.09
-0.18
-0.11
-0.13
Question Density
0.00
0.02
0.01
0.00
0.02
0.00
0.01
List Density
0.02
0.04
0.02
0.04
0.08
0.09
0.04
Exclamation Count
0.06
0.18
0.06
0.13
0.13
0.11
0.10
Paragraph Density
0.23*
0.50**
-0.6'*
-0.46*
+0.36**
-0,37
Density
0.00
0.01
0.00
0.27
Positive Sentiment
0.08
0.07
Compound Sentimeut
0.06
Summary Markers
0.07
0.08
0.05
-0.06
-0.88
00.06
-0.08
Lesical Ratio
0.26*
0.56**
0.22**
0.30*
0.48*
0.45
0,35*
Relative Italic
0.22
0..3
-0.14
-0.25
-0.41
-0.31
-0.31
Exclamation Density
-0.18
0.30
-0.08
-0.19
-0.27
-0.21
40.24
Header Density
0.00
0.02
0.01
0.02
0.00
0.01
0.00
Headers
0.02
0.01
0.03
0.05
0.01
0.03
-0.01
Sentence Length
0.09
0.16
0.08
-0.09
-0.00
-0.17
-0.19
Significance: 0.01, 4ep 0.05, 0.10.
30

---

Table 11: Human-LLM judgement discrepancies on Chatbot Arena (with Benjamini Yekuticli correction)
GPT4-turo
GPT4,1-aao
GPT4.1
GPTio-mini
LLaMa-3.1-8B-I
Selene- 1-Mini
Promeiheus--2
Text Length
00.0
-2,0000
00.4400
-1.02ess
-1,61s
-1.17es
-1.20=
Creativity/nngagemee
-0.550*
-1.27000
0032000
-0.6400
-1,10e
40.7ge
-0.77ee
Rendablity Grade
0.01
0.15
0.02
0.11
0.07
0.07
Consistency
-0.17
-0.32
-0.23
-0.23
Text
0.38
0.74
0.25
0.50
0,77
0.62
Language Quality
Exumple Markers
Conciseness
-0.08
-0.21
-0.26
Structure Counts
0.16
0.43
0.22
0.34
0.28
0.25
Additive Markers
0.01
0.08
0,03
0.04
0.07
0.33
0.01
Polarity
0.09
0.15
0.05
0.09
0.12
0.39
0.10
Contrast Markers
0.06
-0.10
0.04
-0.08
-0.03
-0.09
-0.05
Sentiment
0.26
0.57
0.14
0.34
0.41
0.35
0.37
Coherence
0.16
00.4
0.10
00.55
-0366
-0.28
-0.30
Linebreak Density
0.19
0.36
0.13
0.24
0.35
0.27
0.26
Lists
0.07
0.23
0.08
-0.14
-0.26
-0.18
-0.17
Subjectivity
0.17
0.27
0.07
0.19
0.28
0.22
0.21
Rendability Fase
0.17
0.49
0.11
0.22
0.41
0.28
0.24
Paragraph Length
0.24
0.51
0.17
0.28
0.43
0.32
0.33
Code Black
0.08
0.20
0.06
-0.09
-0.18
-0.14
0..13
Question Density
0.02
0.001
0.00
0.02
0.00
0.01
1istt Density
Exclamation Count
0.13
Paragraph Density
-0.36
Character Deasity
Count Italic
0.27
Question Count
0.00
0.07
0.02
0.07
0.02
0.02
Positive Sentiment
0.03
0.12
0.08
0.10
0.07
0.07
Compound Sentimeni
0.06
0.12
0.03
0.06
0.13
0.10
0.13
Summary Markers
0.07
-0.88
0.05
-0.06
-0.08
-0.06
-0.08
Lexical Ratio
0.26
0.56
0.22
0.30
0.48
0.45
0.35
Relative Italic
0.22
0.43
0.14
-0.25
-0.41
-0.31
-0.31
Exclamation Density
0.18
0.30
0.08
-0.19
-0.27
-0.21
-0.24
Header Density
0.00
0.02
0.01
0.02
0.00
0.01
0.00
Headers
0.02
0.01
0.08
0.05
0.01
0.03
-0.01
Sentence Length
0.09
0.16
0.08
-0.09
-0.20
-0.17
-0.19
Significunce: <*
31

---

B.6.4 Tables Chatbot Arena (non-technical queries)
Table 12: Human-LLM judgement discrepancies on non-technical Chatbot Arena queries (with no
Benjamini- Yekutieli correction)
GPT4-turoo
GPT4, I-nano
GPT4.1
GPT4o-mini
LLaMa-3. 1-B--Ic
Selone-I-Mini
Prometheus-v2
Text Leagth
-0,90***
-2.18
0,55
-1000=s
-2,45**
-1.25
-1.67*x
Dreativit/EEngageme
-0,55***
-1,334ts
0.334**
-0.70=4
-1,64*
-0.86*44
-1,07
Readability Grade
0.28
0.78
0.13
0.35
0.90
0-46
0.61
Consistency
40.24*
.0.56e
0.166
40.27
40.63
40.35*
40.42
Bold Text
0.398ee
0,75**
0,24006
0.529ma
L01
0.66*e9
0.770s0
Causal Markers
0.14
0.32
0.13
0.19
0.34
0.21
0.21
Language Quality
0.30
0.47
Example Markers
-0.1
-0.27
-0112
-0.16
-0.33
-0.21
-0.21
4026
Counts
Additive Markens
0.06
-0.06
-0.01
-0.04
-0.07
-0.08
-0.09
Polarity
0.19
Contrast Markers
0.04
0.05
0.00
0.04
0.06
-0.02
0.00
Sentiment
0.22
0.46*
0.12
0.30
0.41
0.29
0,38*
Coberence
0.06
-0233
0.03
-0.10
-0.24
-0.13
-0.21
Linebreak Density
0.10
0.21
0.07
0.18
0.28
0.19
0.20
Lists
0.35*
.0.84*
0.21
01.1*
.107*
+0.60
-0.71
Subjectivity
0.06
0.00
0.01
0.07
0.14
0.06
0.11
Reudability Ease
0.17
0.60
0.11
0.27
0.65
0.30
0.33
Paragraph Length
0.09
0.20
0.07
0.11
0.24
0.12
0.13
Code Block
0.18
-0.39
00155
-0.23
-0.33
-0.33
-0.36
Question Density
0.00
0.06
0.00
0.02
0.05
0.01
0.04
List Density
0.03
0.02
0.01
0.03
0.08
0.38
0.03
Exclamation Count
0.11
0.31
0.11
0.22
0.32
0.20
0.22
Paragraph Density
0.25*
-0.52*
-0.5*
-0.61
00.8*
40.46
Density
0.33
Question
Positive Sentiment
-0.17
-0.10
-0.10
Compound Sentimeut
0.22
0.46
0.15*
0.28*
0.37*
0.46**
Summary Markers
0.05
0.05
0.00
00.05
-0.09
-0.07
Lesical Ratio
0.26
0.60
0.22
0.36*
0.72
0,50*s
0.48
Relative Italic
0.16
0.30
0.08
-0.18
-0-12
-0.25
-0.26
Exclamation Density
0.12
0.20
0.05
-0.13
-0.22
-0.13
-0.19
Header Density
0.12
0.21
0.08
0.12
0.24
0.16
0.18
Headers
- 00.05
0006
0.02
0.01
-0.06
-0.02
-0.09
Sentence Length
-1.02*+
-1,97*
0,50*
.1.13
+2,55*
+1.47
-1,77s
Significance: 0.01,, ee 0.00, 0.10.
32

---

Table 13: Human-LLM, judgement discrepancies on non-technical Chatbot Arena queries (with Benjamini- Yekutieli correction)
GPT4-turo
GPT4.1-man0
GPT4.1
GPTio-mini
LLaMa-3.1-8B-1
Selene-I-Mini
Prometheus-v2
Text Lengih
-0
-2.18*
40,5500
-1,9*
-245
-1.25
-1.67
CreatviiyyEngageemen
-0.55**
-1.33*
00339*
-0.70
-1.61
-0.56
-1.07
Readablity Grade
0.28
0.78
0.13
0.35
0.90
0.46
0.61
Consistency
0.24
-0.56
-0.16
-0.27
-0.33
-0.35
0042
Bold Text
0.39
0.75
0.21
0.52
1.01
0.66
0.77
Causal Markers
0.14
0.32
0.13
0.19
0.34
0.21
0.24
Language Quality
0.24
0.52
0.14
0.30
0.62
0.36
0.47
Example Markers
0.14
-0.27
-0112
0016
0.33
-0.21
0.24
Conchseness
0.26
-0.49
-0.11
0.32
0.63
0.38
0.45
Strueture Counts
0.21
0.57
0.70
0.39
0.42
Additive Markers
0.06
-0.06
00001
0.01
-0.07
-0.88
-0.09
Polarity
0.15
Contrast
0.04
0.04
0.06
-0.02
Sentiment
0.22
0.44
0.29
0.38
Linehreak Density
0.10
Subjectivity
Readability Ease
0.17
0.60
0.65
0.30
0.39
Paragraph Length
Code Block
0.18
-0.30
-0.33
0.53
-0.33
-0.36
Question Density
0.00
0.05
0.01
0.04
List Density
0.03
0.02
0.03
0.08
0.08
0.03
Exclamation Count
0.11
0.31
0.22
0.32
0.20
0.22
Paragraph Density
0.25
-0.32
-0.15
0.30
0.61
-0.38
-0.46
Character Density
0.16
0.21
0.11
0.15
0.28
0.12
0.20
Count Italic
0.26
0.46
0.18
0.33
0.67
0.44
0.45
Question Count
0.01
0.15
0.04
0.08
0.21
0.07
0.10
Prsitive Sentiment
0.12
-0.17
0.10
0.10
0.25
0.15
0.20
Compound Sentiment
0.22
0.46
0.15
0.28
0.59
0.37
0.46
Summnary Markers
0.05
-0.05
0.05
0.05
0.00
00007
0.09
Lexical Ratio
0.26
0.60
0.22
0.36
0.72
0.50
0.45
Relative Italic
0.16
-0.30
0.08
-0188
-0.12
-0.25
0.26
Exclamation Density
0.12
-0.20
0.05
-0.13
-0.2
-0.13
-0.19
Header Density
0.12
0.21
0.08
0.12
0.24
0.16
0.18
Headers
0..05
-0.06
0.02
0.01
-0.06
-0.02
-0.09
Sentence Length
1.02
-1.97
0.59
1.13
2255
-1.47
-187
Siguifiaace: 4*8 0.01,
0.05,
33

---

B.6.5 Tables Chatbot Arena (technical queries)
Table 14: Human- LLM judgement discrepancies on technical Chatbot Arena queries (with no Benjamini- Yekutieli correction)
GPT4-turo
GPT4, I-nano
GPT4.1
GPT4o-mini
LLaMa-3. 1-8B-Ic
Selene-I-Mini
Prometheus-v2
Text Leagth
-1.06**
-236*
+0.0
-1033
-1,10*
-1,07
-0,92***
Dreativit/EEngageeee
-0.60*
-1.32
0.27
+0.52*
40,62*
+0,58*
40.48
Readability Grade
0.27
0.58
0.17
0.23
0.32
0.31
0.25
Consistency
0.08
0.18
0.04
0.12
0.10
0.13
0.07
Bold Text
0.45**
0.90**
0.2g0*
0.529ea
0.629ee
0.65eee
0.52ee0
Causal Markers
0.04
0.12
0.04
0.15
0.005
0.01
0.003
Language Quality
0.12
0.17
0.17
0.12
Example Markers
0.30
0.64
0,21**
-0.05
0.04
Counts
Additive Markens
0.06
0.16
0.05
0.07
0.09
0.08
0.08
Polarity
-0.05
-0.06
Contrast Markers
0.12
0.22
0.08
-0.16
-0.13
-0.20
00.12
Sentiment
0.40
0.95*
0.22
0.45*
0.41
0.46
0..37
Coberence
1.000
-2,24*
0.55*x
-1.10=
-1.19*
-1,15**
+0,94
Linebreak Density
0.33
0.67
0.22
0.31
0.43
0.36
0.28
Lists
0.09
0.07
0.08
0.03
-0.01
0.02
0.03
Subjectivity
0.40*e
0.70
0.21
0.40
0.46*e
0.48*e
0.34*e
Reudability Ease
0.42
0.96
0.26
0.39
0.48
0.46
01:33
Paragraph Length
0.549*
1.17
0,35**
0,58**
0,59**
0,58**
0.45**
Code Block
-0.77
-0.19
0.05
-0.07
-0.11
-0.11
-0.08
Question Density
0.20
0.45
0.14
-0.19
-0.13
-0.14
0.11
List Density
0.12
0.30
0.20
0.21
0.25
0.15
Exclamation Count
0.01
0.00
0.02
0.03
0.02
0.01
0.05
Paragraph Density
-0.12
-0.15
-0.25
-0.26
-0.22
Density
-0.09
-0.08
0.01
Positive Sentiment
0.59*
1233
0.37*
0,59*
0.73**
0.70*
0.60**
Compound Sentimeut
-0.51
-0.20
Summary Markers
0.09
-0.11
0.05
-0.06
-0.07
-0.05
00.07
Lesical Ratio
0.07
-0.12
0.01
-0.11
-0.13
-0.34
-0.10
Relative Italic
0.21
00.6
-0.15
-0.23
-0.25
-0.23
-0.23
Exclamation Density
- 1.36
-2.18
-0811
-1.43
-1.60
-1.53
-1.41
Header Density
0.04
0.05
0.00
0.01
-0.03
-0.05
-0.02
Headers
0.06
0.07
0.03
0.07
0.05
0.07
0.03
Sentence Length
0.11
0.19
0.02
0.13
0.13
0.10
0.04
Significance: 0.01,, 4epp 0.00, 0.10.
34

---

Table 15: Human- judgement discrepancies on technical Chatbot Arena queries (with Benjamini-
Yekutieli correction)
GPT4-nuro
GPT4.1-nam
GPT4.
GPT4o-mini
LaMa--311-8B-1
Selene- 1-Mini
Prometheus-v2
Text Length
-1.06
-2.36
-0.60
-1.03
-1.10
-1.07
-0.92
Creativiy/Engngementt
0.60
-1.32
0.27
-0.52
-0.62
0.58
00.8
Rendability Grade
0.27
0.58
0.17
0.23
0.32
0.31
0.25
Consistency
0.08
0.18
0.12
0.10
0.13
0.07
Bold Text
0.45
0.90
0.28
0.52
0.62
0.63
0.52
Causal Markers
0.01
0.12
0.05
0.07
0.03
Language Quality
0.12
0.17
0.12
0.06
0.16
Example Markers
0.30
0.35
0.28
Conciseness
0.05
-0.0
-0.03
-0.44
0.06
0.09
Polarity
Contrast
Sontiment
0.40
0.95
0.41
0.46
0.37
Coherence
-0.55
Linebreak Density
Lists
0.09
0.07
0.08
0.03
-0.01
0.02
0.03
Subjectivity
0.40
0.70
0.46
0.48
0.34
Reudahility Ease
0.42
0.96
0.26
0.39
0.48
0.46
0.33
Paragraph Length
0.54
0.35
0.59
0.58
0.45
Code Block
0.07
-0.19
0.05
0.07
-0.11
-0.11
-0,08
Question Density
0.20
-0.45
0.14
0.19
0.13
-0.14
-0.15
List Density
0.12
0.30
0.05
0.20
0.21
0.25
0.15
Exclamation Count
0.01
0.00
-0.02
0.03
0.02
0.01
0.05
Paragraph Density
0.15
-0.42
0.13
0.15
0.25
00.66
0.22
Character Density
0.02
0.09
0.02
0.02
0.04
0.02
0.02
Count Italic
0.05
-0.01
0.02
0.07
0.09
0.08
0.01
Question Count
0.04
0.13
0.02
0.02
0.06
0.05
0.05
Positive Sentiment
0.59
1.23
0.37
0.59
0.73
0.70
0.60)
Compound Sentiment
0.27
-0.44
0.20
0.29
0.31
-0.44
-0.24
Summary Markers
0.09
-0.11
0.05
-0.0
-0.07
-0.05
-0.07
Lexical Ratio
0.07
-0.12
0.01
-0.11
-0.13
-0.04
-0.10
Relative Italic
0.21
00.6
0.15
0.23
0.25
-0.23
0.23
Exclamation Density
1.36
2218
0.81
1.43
1.60
-1.53
-1.1
Header Density
0.04
00.5
0.01
-0.03
-.05
0.02
Headers
0.06
0.07
0.03
0.07
0.05
0.07
0.03
Sentence Length
0.11
0.19
0.02
0.13
0.13
0.10
0.01
Significance: e09 0.01,
0.06, 0.10.
35

---

C
Additional theoretical results and proofs
C.1
Conditions
The following two conditions are required for Proposition 3.1.
Condition C.1. The true purameter (1j", 7im) lies in the interior of a compact subset of Or x 2 C RRK*, and is the unique minimizer of the population loss Qn 21:))
Denote
K
V (i
(P 2::²) W 2,-1)
An
v Pix Pik)
i=
R=U
where Pik = , 2,-²). Let Eik E [-1,1], i = 1,... k = O,..., K follow i.idd
Rademacher(1 /2) distribution. Define vector Sn Σ Er=o ((n 77"), and its variance B, Var(S,). Denote Σ A BnA,1
Condition C.2. Let matrix An be positive definite, and the variance B, = Var(S,) be finite.
Below we state the conditions of Theorem 3.2.
Condition C.3. The observations {(Y;", X, I,, 0,)) are i..dd. and the ordinallogit model P(Y = k T,, O:) = (0; Z X;) = Pk pa(a.....,.0 OK, (1/3)Z; (1/3)=X) x;) holds for some true cut-points oj <... d'. coefficients 3 and y", and latent scores Z Condition C.4. The true parameter vector 8+ (ak. (+) lies in the interior of VVV log logly, (0; Z Xx)] has a unique root 0*
parameter space θ and
Condition C.5. Within a neighborhood of 8", the expectations E[I[Vo log lyn (0; Z X,) and E [ log ly, (0; 7 Xx)I]] are finite. Moreover, the Fisher information matrix I(f*)
E logly; by (0"; 7,;; X:)] is positive definite, and the expected mixed derivative G(l*)
V (,) Vo log b (0"; Z X:)] exists and is finite.
C.2
Extended Theorem 3.2
We present extended version of Theorem 3.2, addressing a more general case where the CoT
sample size m, used estimate P((Y" k. 4, O.), i I,..., ..... grows proportionally with n.
Theorem C.6 (Asymptotic normality of (3,4)). Under Conditions f....5..orm the MLE 0, (d.,...... a.K. Bar 7n) by maximizing the loglikelihood
K
(, 10:22
MY = k3 log (0 2/ X,).
K=
W.E.t. ((1,....,.K) 3,
(a) If ........ пи and (Z! 1n were known, then
vā(0n- -0) 00) 4. ~((,,((0))'),,
Vol
3nn (3p
- 30
3 No, ((00))"hamm))
Fm 70
(b) the CoT prompting strategy is used to estimate IP(Y! = k 1,, O:), also assume Conditions
C.I and C.2, and let n/m, + CE [0, oc) + co. Then
vn - 0b) 4, N0, I(0 -
U Var [ar +eGGGT
Consequently.
Bn - -B 4 N((0, (I Vn
In -0
When = O (that is, mh >> n)), the extra variance term drops out and the estimators allain the efficiency bound of part (a). For any fixed > the variance is inflated by the second term, quantifying the price of estimating (17 Z
36

---

C.3 Inference for a differentiable function of the parameter
Consider differentiable function m(f). Under previous conditions, the delta-method yields
Vn(m -0()))m))))))))))) S (N "
Set 6² = V TV om (m) An approximate 100(1 - % confidence interval (CI) for m((0) is I + 21 =1-0/20 where 21-a/2 is the standard normal quantile. Under previous regularity conditions, the coverage probability of this CI converges 1 a as n grows.
Prediction interval. This result can be used to build a CI for the prediction of a new observa-
tion (Inews Onew, Xnew, Zuew). ) defined as mn Σ kpr. where ps = ( :new)) and nnew 8-1 21 new B-18" X new This corresponds to the specific function m(0) Ek=o k pps(a........... .pl(a,.....a. (0)), with z(6) = B- 6-112 Znow 3- Xnew.
Partial effect of a covariate. We also consider the "partial effect" of covariate X3 on the probability of class k:
PE ax, pa (a, --'-"-1'
It is the local, ceteris paribus change in the models predicted probability of class k when covariate X; nudged by one unit, holding the latent index Zi constant. In other words, it answers the question: All else equal, how much does the LLM-judges probability of assigning class k change when feature
X, is perturbed by one unit? At f = (á, 3, 4), PE = P's (,, (-)///)), where Pl 2) = -a' (ox+1 - 2) + a' (ar 2), o' (t) = (1) (t)), with 2 =(Zheur Xnewi B, 4). Define the scalar mapping m00) P (0). Then (m)(m(0) m(0)) 4 NN Vm((0)T V Vm(0*)). where the gradient Vm is taken w.r.t. (a, 7) An approximate confidence interval for PE is
- (m Vm((). n
P + 21
where V estimates the asymptotic covariance of 0 and 21 is the standard normal quantile.
C.4 Proofs
C.4.1 Proposition 3.1
Proof of Proposition 3.1. Define, for each i,
ge(n 21m)) == Σ sgn (px(1), 2,) - -pck,..) V (m. 21),
k=0
so that Gn.m., (1), Zimm = Dim1 (1,,) =1m) measurable sub-gradient of Qn.... First-order optimality of (ń, ( gives 0 E Gn,ma (f, ). Write
ri 21:h) := PI= 2;) Pik.m..
Eik : Pik.m. Pik-
By the model assumptions, (1"" 7 = Eii-- Because pk is continuous and differentiable, and Piu pa(n", z E 10 which rules out kinks of the absolute value, sgn(r: 21sm))
= =sgn(-Ex) + (0) V (P =
17 17" z
+ Op
21:n z/nm
71 - 17"
21:n
where (0) mm, 2 (1 pa)) 1/2 the asymptotic density of VmnEsk O. Setting (n, 21:n) = (1, 2/,,,) and summing over k and i,
( n 1117"
0 = Sn,m. + An,ma
2/ Z
+ Fnmm.
37

---

where
Sn,ma
sgn (Ex V (n. PP ,2,,
i=1 k=0
Anmo == 2 Σ (i) V (m pP (1), Z ) Pr "T
i,6
and
f タ 17
mn Op
lin Z
Since sgn( sgn( E f- and n(K + +1 is fixed, Sn,m.. = Op)) For every i and k, we have Pik (0,1), An,m,, mmn An + ol ml)))) with An in the statement. Rearranging the expansion,
f 1-17 17" 2
Vmn
A, Sn.m. + op()).
the continuous-mapping theorem gives sgn(-E;k) S Eia with Eit (ad(1/2), the Rademacher distribution, and independence across ki, Therefore Sn,m, S -n = EE= Σt Ei V Von......P 2,);; Eit iid. Rad /2) and the continuous map- ping theorem yields the desired convergence in distribution to -A Therefore, the asymptotic distribution Vm, (ft, ii.)) - (1)" [ has mean and variance A, Var
Because mineik 9 N
pia (1
C.4.2 Theorem 3.2
Since Theorem 3.2 can be derived from Theorem C.6, present the proof of the latter.
Proof of Theurem C.6. The proof basically follows the M-estimation consistency framework [42,
chapter 5J. Denote
K
(n,((; Z,, X1)
Σ
1lY = log
s; Vo
km=
Let (0) = n Di -i=l Hessians:
00 2,), Sn (0) == =n Ei s; 2,), and similarly define the (negative)
Viln X,), H.) = 1 n
(+)
Vgln (0: 2, X²).
When the true values of (7)) Ji=1 are known (when the log-probabilities used), by Assump- tions C.3-C.5 and a uniform law of large numbers (ULLN), (,) + [f)) uniformly on θ. Since the limit has unique
6* (Condition C.4), the arg-max theorem yields 0, 2 0".
Using a mean-value expansion around tg.
0 5n (b.,) vn (A*) (G.) (n (0..0.-1 - -8"),
where 0, lies on the segment |0~, vr.. By Conditions C.4-C55, Hn (0,,) & I(O0). By CLT for the score, S,.) 4 ANO0, (9o))) I). Slutskys theorem gives
N( Vn - 0b) = H (,,)))) V S.)) 4 (1 = establishing part (a).
((0,I(
When latent scores {2!}² are estimated by the CoT prompting strategy, by Conditions C.4 and C.5, (0; X) is jointly continuous in (0, 2 and bounded by an integrable envelope. By Proposition 3.1, sup
! z Op (m, -1/2 = op()). and we have
n
Ln (0; 21 X.)
Σ ln 2 Xi) = =1)
sup
n
38

---

Combining this with the uniform law of large numbers for the known regressors (Z,
(0; x.) Elen, 44. x,)] 4 0.
sup AE0
11 1=1
we obtain supace In 11 Eis fn (0; 23 X,) (0 Z xx)] 2 Hence
0, B l"
(C.1)
by the arg-max theorem and uniqueness of 0" For every i, by the mean-value theorem variable z
s (0", -1) =s, z V zs, (0" 2 (2) -!)
for some 7 on the segment [Z] / Summing over and dividing by gives
Sa(0") (0)) 72
Vzs (0 2)) (2) 7!).
(C.2)
By Condition C.4, su 22; Z)]l S C. for some integrable C, (dominated convergence
applics). Since 1| 2 & 11 1 = =0p(m
"
V (0", - Vz Z!Il Op (p 1/2) (1/2),
because n/m, + < cc. Hence one can replace 80" 21) by V wws 7/) in (C.2) at the cost of an 0p 1//2) term. Define therefore
An (0" (2) 2 so that (x²) = S, (100))+ A. + op 1/2).
By Proposition 3.1, there exist i.i.d. random vectors eim with mean 0 and variance I such that
2 -1-28 z
11/2 ei eim + Time
||a,n|| = op (mmm1/3)
sup 1sV5
mm
Consequently,
An
V (00 2/) T // csm + 0p 112).
(C.3)
n nmm
By the Lindeberg-Feilee CLT and Condition C.5,
(n 4 (O, (ar )))
Rewrite the leading term of (C.3) as
V (0" Z/) Σ
Caam
mn
As eim are i.i.d. and independent of (Y/,, X;, 2 the inner average has mean G -1 Bi,m and variance n -IGGTT Therefore, conditional on the data,
Vna 4 VeN(o, GLG
and this limit is independent of (f*) by construction. Adding the two independent Gaussian limits yields
Vn S, (0, U: Varls; w.)] + cGEGT
With the same arguments using the uniform LLN as in (C.1),
H. (0) 1 n
Voln X.) 2 I(O')
39

---

uniformly neighborhood of 0" In particular, (0.))- & I(0")- for any random 0, between in and 0". Next, using melu-value expansion of the firstorder condition (f.) 0,
(,)) (0,) Vn(0, 0*),
whence vn(ln, 8*) Hm() (O*) Combine the convergence of (a,) with the Gaussian limit of (0)),
V 0*) 4 I(0')-1(0,22) .N(0, I(0-) -
Selecting the (B,7) block completes the proof of part (b).
D Covariates
We construct two types covariates from each LLM-generated response. The first type consists of LLM-scored covariates, derived using GPT-4o-mini based evaluations via prompts described in Appendix A.3. These are numeric ratings ranging from 5 and include aspects including coherence, factuality, clarity, conciseness, creativity, consistency, engagement, fluency, appropriateness, and sentiment. Each of these captures a subjective quality of the response, as detailed Table 16. To obtain these scores, we extract the top 20 most probable responses along with their probbabbailistes from the model output. Then, calculate stable response.
ceopabilitywwiighhe average of the probable scores to gain
The second type comprises automatically computed lightweight covariates, which are extracted using rule-based or statistical methods without human prompting or query LLMs. These features cover five broad dimensions of a response: length (e-g., word and sentence counts), readability (e.g., Flesch Reading Fase score), style and formatting (e.g., paragraph structure, use of markdown), sentiment (e.g., TextBlob and VADER sentiment scores), and discourse markers counts (e.g., counts of additive causal connectors). Table 17 provides variable names and descriptions for each of these lightweight covariates. Some of the constructed covariates exhibit high correlation with others. address this using clustering methods, discussed in the following section.
Table 16: LLM-scored covariates
Variable
Description
coherence
how coherent the response is how factually accurate the response is
factuality
clarity
how clear the language of the response is how concise the response is how creative and original the response is how consistent the style and content response how engaging the response is to the reader how fluent the response reads how appropriate the tone and style of the response is for the in- tended content
conciseness
creativiey
consistency
engagement
fluency
appropriateness
sentiment
how positive the overall emotional tone of the response is
40

---

Table 17: Automatic lightweight covariates
Variable
Description
Length and Tokenization
response_wond word_count count
Total number of word tokens in the response.
response_chiss count
Total number characters (including spaces and punctuation).
relative_char_couml
Average characters per token (response_ Saatonse_chrr_coee count /
reaponse word_ poard_cout).....
sapoonsessenteneee countt
Total number detected sentences.
Average tokens per sentence (response_ serpeee_wrdd_oooo count /
response response_sentence wewtencee_oouniee
Readability and Lexical Diversity
Desch_reading,
Flesch Reading Fase score (higher = easier to read) [9].
Besch_kincaid_grane gunning_fog
Flesch-Kincaid gradelevel estimate [19].
Gunning Fog index (years of education needed) [13].
smog
SMOG index emphasizing polysyllabic words [31].
lexical_diie exical_diversity
Number of unique word types (e.g.. 16].
Snative_exiccal diversity
Type-token ratio [16, 30] (lexical cavieasidldeittyyyyllee /
response_ cale__oordd_ruuou_o
Style and Formatting
exclamation count
Number of characters.
Exclamation marks per token.
question count
Number of characters.
Slative_qucsttin_coom
Question marks per token.
paragraph_come count
Number of paragraphs (nonempty newlineseparated blocks).
avg_paragraph_lenge
Average tokens per paragraph.
relative_paragraphh count
Paragraphs per token.
linebreak_count
Number of 'n newline characters.
celatie__inebbraa__omr
Newlines per token.
contains_code_ble
1 if a Markdown code block (""....") is present, else 0.
header_count
Count of Markdown headers (lines starting with #).
relative relative_header count
Headers per token.
bold count bount
Number of bold words (**bol#** or _bold
relative_bold
Bold words per token.
italic_coume
Number of italicized words (*italic* or _italic_. excluding bold).
relative_ttale count
Italic words per token.
list_count
Count of list items (-, *, or numbered markers).
elative_list_coutt
List items per token.
Sentiment and Tonal Analysis
sentiment_polarity
TextBlob polarity score (-1 to [7, 29).
sentiment_suujeectvvee
TextBlob subjectivity score to 1) [7, 29].
ceatiment_scores_com
VADER compound sentiment score [10].
sentiment_sce saapr__ccee_______ pos
VADER positivesentiment proportion [10].
Discourse and Coherence
discourse_mk. __c_eed_____ddd__d_
Count of additive discourse markers (utheroorh,,,ore))
discourse_mie sateee____oooo_ooe
Count of contrastive markers (c.g., however).
discourse_mn__ee
Count of causal markers (e.g.. therefore).
discourse_mke ____ex___eex discourse_mk..
Count of example markers for example).
Count of summarizing markers (e.g., overall).
41

---

D.1 Clustering covariates
Algorithm compresses potentially large and collinear covariate set into smaller covariate groups that are less correlated. First, all columns of the design matrix F are 7-standardised. The dissimilarity matrix D
corr( [ is then subjected to a top-down complete linkage agglomerative procedure: Starting with clusters p and progressively merging features, covariates, (ii) extract the first principal component from each cluster, thus summarizing its shared signal in single latent factor, and (iii) check whether any pair of resulting factors still exhibits an absolute correlation above the user-specified threshold т. The loop terminates at the coarsest partition
repeatedly (i) partition the
whose principal components are mutually "sufficiently uncorrelated." ensuring that the final matrix F contains compact, approximately independent summaries of the original features. A final z-score standardisation puts these latent factors on a common scale, making them immediately suitable for downstream modelling or inference.
In Tables and 19, we describe the resulting covariate clusters for BGB and CA based on the initial
covariates described in Tables 16 and 17.
Algorithm 1 Covariate clustering
Require: Covariate matrix F E R*XP; correlation threshold (default 0.7)
Standardise F column-wise using z-scores.
2: D + 1 co
b feature-wise dissimilarity matrix
top-down search 4: Apply complete- linkage agglomerative clustering on D to obtain clusters with labels
3: for k 4 p- 2,
do
l,....,
5: Initialise $ + [].
6: for all cluster {1,.... k} do
b per-cluster PCA
7: Fit PCA on the training columns with label e; keep first principal component ((c),
8: Append ((() to F.
9:
end for
10: E corr( (on(FT) set Σn -99 for all i.
11: if max < then
D stopping rule
12: break
13: end
14:
end for
Standardise f column- biismm--wise using z- scores.
16: return F.
42

---

Table 18: Cluster definitions for BigGen Bench
Cluster
Deseription
Writing Quality
Composite rating combining coherence, clarity, consistency, fluency, and ap- propriatenessbroadly reflecting how well-structured, clear, and appropriate a response is ''coherence,, 'clarity', "consistency', 'fluecyy,, 'appropriatenes') Basic length and diversity features, including word, character, and sentence counts, well the count of unique word types ('response_at pessponse_word_com word_count" 'resoonee_hhee char_count'' 'response cecronss_sentnce____e totee__couttt texical_diversicn
Text Length
Italics
Frequency of italicized text in markdown, measured both the number of italic words and a proportion tokens ('italic_ count', 'relative italic count").
Bold Text
Usage of bold markdown formatting, captured both total count and per-token frequency ("bold_coom "bold_count', 'relative __old_old____ vold__coutt'..
Lists
Frequency and density of list items, including both the absolute number and relative count per token ('list_com count", 'relative list__ceeees__l count').
Hcaders
Markdown header usage, including total header count and density per token ("header_ce ccorre___ttt,, saatiieehhader____
CreativityEEggageennt
Ratings for creativity and reader engagement, reflecting how original and capti- vating the response ('creativity". 'engagement"..
Positive Sentiment
Proportion content flagged as positive sentiment by VADER ('senti- ceont_scores_ppee __ssss__o_.....
Conciseness
Evaluation of how bricf and to-the to-the-point the response is ('conciseness", ).
Contrast Markers
Frequency of contrastive discourse cues such as however and yet ('dis- course mk con').
Layout Density
Density structural elements, capturing relative number paragraphs and line breaks per token (" ('relative_paraee __r___aaaa__cooeee _ctttt"__'' recative_lincbres (mmebrekkccuutt))
Causal Markers
Frequency of causal discourse signals like thus or therefore ( d'discourse _ik__mm_______ mnu'.. Raw counts of paragraphs and explicit line breaks ( pparagrapp_come count', 'line- break look__count)..
Structure Counts
Sentiment Readability Grade
Overall sentiment rating, typically on 05 scale ('sentiment').
Grade-level readability indices, such as Discci-Kinaiid, Gunning Fog, SMOG, that estimate years of education needed ('flesch kincaid grade", 'gun- ning fog'. 'smog').
Exclamation Density
Exclamation marks per token, reflecting the relative use of for emphasis ('relative ceteriiveexclaaatiinnnoo
Readability Ease
Flesch Reading Ease score, with higher values indicating casier reading bussch_reading_cae _aaee")
Polarity
Sentiment polarity from TextBlob, ranging from I (negative) to +1 (positive) (sentiment_polarity))
Question Density
Relative frequency of question marks, indicating tendency to ask questions (relltive question__couee
Paragraph Length
Average number of tokens per paragraph, reflecting structural granularity Bavg_paragraph_lenge lengghhh
Code Block
Binary indicator for the presence of markdown code blocks (""..."") ('con- tains_code booekblock').
Additive Markers
Frequency of additive discourse markers such furthermore moreover ('dis- course mk add').
Summary Markers
Frequency of summarizing discourse cues such as overall in conclusion ('dis- course_mk prrss__m_summ" ))
Character Density
Average number of characters per word token ('relative ceattvve_hhr____ char_count').
Exclamation Count
Total number of exclamation marks in the response f'exclamation _aoett__ooo.e__ Relative lexical diversity, measured as the type-token ratio seclative_lexicl_dee versity').
Lexical Ratio
Subjectivity
Degree of subjectivity (0 objective, subjective) as measured by sentiment analysis ('sentiment subjectivity").
Sentence Length
Average tokens per sentence, reflecting sentence complexity ("response_avg _es_ssee- tence_ dence_length'..
Example Markers
Frequency of camplle-givngg discourse markers such for example ('dis- course (aanse_mk_ex))
Compound Sentiment
VADER compound sentiment score summarizing overall tone ('senti- ment scores cooesss.mm 43
Question Count
Absolute number of question marks (' ('question_comus

---

Table 19: Cluster definitions for Chatbot Arena
Cluster
Description
Text Length
Core size measures of the responseword, character, and sentence countsplus
lexical diversity (response, sarpaase__wddd__eeeeeee word_count, response, _at___haar_oooo count, lrroossesentence_com lexical lee.cellliiiereeeeeeeeeleeeeeeee diversity).
Creativity/Engggeeent
Ratings capturing originality and reader engagement (creativity, engagement).
Readability Grade
Grade-level readability indices, including Flesch-Kineaid, Gunning Fog, and SMOG (flesch, kincaid _gradd, gunning_ smog).
Consistency
Measures consistent style content the response (consistency). Use of bold markdown formatting, both absolute and relative (bold_count, relative relative_bold, count).
Bold Text
Causal Markers
Frequeney causal discourse cues (c.g., thus, therefore) (discourse_nil _atrsse_____a Ratings related clarity, fluency, and appropriateness of the response (clarity, fluency, appropriateness).
Language Quality
Example Markers
Frequency of example cues (e.g., for example, for instance)
(discourse __k___x)._.________________
Conciseness
Assessment of brevity and avoidance unnecessary verbosity (conciseness).
Structure Counts
Absolute counts of paragraphs and explicit line breaks (paragraph_comm count, sinebreak_count
Additive Markers
Frequency of additive discourse cues (discourse_mke
(e.g.. furthermore, moreover)
Polarity
Polarity score from sentiment analysis Deattiment_polaritty
(TextBlob; to 1)
Contrast Markers
Frequency of contrastive discourse (discourse_nke _ateee____o_____
(e.g.. however, yet) cues
Sentiment
Overall sentiment score (sentiment).
Coherence
Measures how coherent response (coherence).
Linebreak Density
Relative number of newlines per token (relative caatbbraakcccuttt.
Lists
Absolute count of list items cdlst_count),
Subjectivity
Degree of subjectivity in sentiment analysis (0 = objective, subjective)
(sentiment peattiment_subeectiviie subjectivity).
Readability Fase
Flesch Reading Fase score (higher values Btleschrreadinge
easier
read)
Paragraph Length
Average number of tokens per paragraph hawggrargggpph
Code Block
Binary indicator for the presence of fenced code
blocks
(contains ccdeee_ooo_____ee__eeee_ htlee_ccoddollccoeeeeeeeeeeeeele block). Question marks per token (relative measure) (relative__uestioon.comm
Question Density
List Density
List items per token (relative list_ Total number of exclamation marks (exclamation_coun))
Exclamation Count
Paragraph Density
Relative number of paragraphs per token callative_paragrahh. count).
Character Density
Average characters per token
Count Italic
Absolute number italicized words (italic_count))
Question Count
Absolute number of question marks (question_ceem (inestionncount..
Positive Sentiment
Proportion of text flagged positive VADER (ssetimentt_scores sacpaatsssorressss________ _pos).
Compound Sentiment
VADER compound sentiment score (sentiment_ (seatiment_scores__oo sct.reeccoccopp)..
summarizing overall tone
Summary Markers
Frequency of summarizing discourse cues (discourse mk sum).
Lexical Ratio
Relative lexical diversity; type-token ratio (relative lexical diversity) Italicized words per token (relatiie_jtallcc_comm
Relative Italic
Exclamation Density
Exclamation marks per token (relatiiee.ecllaatioonn exclamatiion_count)))
Header Density
Headers per token (relative @eelattve_header___ header_adeec (ounder_cout)..
Headers
Absolute count Markdown headers (header__em sceader_count).
Sentence Length
Average tokens per sentence (response_are saaeeseeevvgg_eee avt__avtteeeeeee length))
44

---

E
Model extensions
E.1 Modeling the expected judgement
This subsection is especially useful when Y and y' are ordinal with many possible values or continuous (i.e., could possibly assume any value inside closed interval). Let us take Y Y [0, K]. We assume
Byyh [1,0] Ko and EY [,0] Ko (z') where 7/ a 3 + +7 TTx. X.
In this setup, E[ 1,0] can be computed exactly using the log probabilities estimated using sampling. Take Z
o (B O)/)/ Then,
- 1,0] = Ko (
+ (1/8)Z² - (1/8)7 x),
and, equivalently,
=== Ko (-(1/3)o (1/)))) (1/3)T +e.
For an emror term with EiE I1 The model can be fitted using non-linear least squares using human labels. The asymptotic distribution of the estimators could be derived using the theory of M-estimation; the bootstrap could also used for approximating it.
E.2 Categorical/Multinomiall judgements
An extension of the ordinal model assumes yh, y E {0, ... K}. but where no ordering is needed; rely the multinomial logistic regression model [45]. This model still be used in the case of ordered outputs, but it is harder to interpret. For this formulation, we assume
if k 0
1+ exp(Z exp(2
P(Y = h 1,0)
< k: K,
1+ exp(
where the class 0 is the base class; in practice, the practitioner could any other class the base one. As before, we assume a connection between human and LLM judgements; this case, we have
z16r - + +3,2 +7 X.
In this formulation, we can obtain (1) by using the formula
P(Y k 11 (3) -log p(y 0 T,0"
and then fit the model, using human labels, by expressing Za in terms of zt(I and maximizing the loglikelihood with respect to the model parameters. The asymptotic distribution of the estimators could be derived using the theory of maximum likelihood estimation; the bootstrap could also be used for approximating it.
45

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
we deliver everything we promise
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
46

---

Justification:
we detail everything in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: include all the info in the main paper appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
47

---

Answer: [Yes]
Justification:
we make everything public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Mundides/Coeeuumiii
Popillllyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.cc..p wwtpcccccpppcccccccccecccc Stti:i///giiieeeeeeeuuubbiiiiii iettssummssiiippoooooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: we report error bard and significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
48

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: we a standard statistical model (ordinal logistic regression). This model is known to be cheap fit.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /ttp//nnurrp.eeeee cc httpppplcccchiiicceddeee...eeeee
Answer: [Yes]
Justification: made sure this is true.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there is no evident direct impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
49

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
we do not think this is problem we face.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: everyone is credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
50

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: main asset is code and it will be documented and released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: we did not use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: we did not have human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
51

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: used LLMs mainly for rephrasing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttps////neup..peeeeee / for what should or should not be described.
52


Reviews:
[{"review_id": "d9CUcWiavG", "reviewer": "Reviewer_gh7b", "summary": "The authors propose a statistical framework to align LLM agent ratings with human preferences. They model both human and LLM judgements as outcomes of a shared latent human preference signal. The LLM-as-a-judge framework is widely used and this work provides a solution to improve alignment with human judges.", "questions": "- Will your framework be publicly available after the review process? \n- Did your method reveal any patterns in which LLMs (e.g., size or training regimen) are more misaligned with humans?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper addresses a critical gap in recent research \n- The paper is well written and structured\n- The authors demonstrate a practical benefit of their approach, even with only a few labeled data points\n\nWeaknesses:\n- As the authors themselves address, the model is vulnerable to misspecification\n- Necessary domain knowledge makes the framework less generalizable\n- Human judgement are treated as gold standard, even if they can also contain biases. However it is the goal of the paper and also in most research studies to align agent with human judgements which is clearly achieved here\n\nDespite the presence of weaknesses, this paper makes an important contribution to the formal alignment of human and LLM judgements, with implications for various applications of the LLM-as-judge framework.\n\nComments:\n- You can use ~ before \\citet or \\citep to avoid references at the beginning of a line (see line 103)\n- \"Table 1\" looks more like a Figure to me?", "was_revised": true, "final_justification": "As I was already convinced that this paper is solid and should be accepted for the conference, I will not change my score.", "rebuttal": "Thank you for your work on our paper! We have addressed your comments and questions below. Please let us know if you have further questions or concerns.\n\n-------\n### **Comments**\n\n- **Misspecification robustness:** We conducted a new experiment on misspecification robustness in response to Reviewer pkTY’s concerns. Please take a look.\n\n\n- **Necessary domain knowledge makes the framework less generalizable:** We agree that crafting the most salient covariates can be beneficial. In practice, users can start with generic, off-the-shelf metrics (e.g., response length, readability grade) that already capture a large proportion of common biases. For specialized tasks or higher requirements, practitioners may simply augment this base set using their domain knowledge or by consulting subject-matter experts. The core method still applies without modification.\n\n- **Human judgement are treated as gold standard, even if they can also contain biases:** Thank you for raising this. Indeed, like most evaluation pipelines in ML, we have followed the common practice of using human judgments as a proxy for \"ground truth\", but we fully agree that these judgments can themselves be biased. Nonetheless, our framework is designed to reduce inconsistencies between any chosen benchmark and LLM judgments, whether that benchmark remains individual human ratings or evolves toward multi-annotator consensus or other proxies. This highlights the significance of our work.\n\n- **Comments on citation practices and Table 1:** Thank you for the constructive comments. We will revise accordingly.\n\n-------\n### **Open sourcing**\n\nYes. We will publish the codebase and collected data after the review process.\n\n-------\n### **Did your method reveal any patterns in which LLMs (e.g., size or training regimen) are more misaligned with humans?**\n\nWe see that in Table 2, for example, LLaMa-3.1-8b-Instruct has many more detected discrepancies when compared to either fine-tuned models or a notably stronger model (GPT-4o-mini). Moreover, we have run an extra experiment in which we show that LLMs systematically value creativity less than human in less technical queries while the same fact does not hold on technical ones. We explain below.\n\nWe conducted an additional experiment by dividing Chatbot Arena queries into two categories. Using GPT-4o-mini as a zero-shot classifier, we split queries into technical (e.g., coding, STEM) and non-technical (e.g., jokes, casual conversation) groups. Approximately half of the queries were labeled as technical. For technical queries, we did not observe any statistically significant differences between human and LLM ratings. However, for non-technical queries, we found notable divergences between humans and LLMs in terms of text length, bold text, and creativity. This suggests that human and LLM judgments are more likely to diverge on subjective, non-technical content. A limitation of this analysis is the reduced sample size in each split, which lowers statistical power.\n\n\n*Chatbot-Arena (Non-technical queries)*\n\n| Feature     | GPT4.1-nano | GPT4.1    | GPT4o-mini | LLaMa-3.1-8B-It | Selene-1-Mini | Prometheus-v2 |\n| ----------- | ----------- | --------- | ---------- | --------------- | ------------- | ------------- |\n| Text Length | **-2.18**** | **-0.55*** | **-1.09****  | -2.45           | -1.25         | -1.67         |\n| Creativity  | **-1.33**** | **-0.33*** | **-0.70****  | -1.64           | -0.86         | -1.07         |\n| Bold Text   | 0.75        | 0.24      | **0.52*** | 1.01            | 0.66          | 0.77          |"}, {"review_id": "O7QVCaw5fp", "reviewer": "Reviewer_pkTY", "summary": "The authors propose an ordered-logit model that ties human ratings to LLM-judge ratings through a shared latent \"human-preference\" score and a linear covariate term capturing systematic judge biases. They introduce a \"logit trick\" that recovers the latent scores without ever observing them, prove asymptotic normality for all key parameters, and demonstrate three practical uses: (i) probabilistic calibration of LLM judges, (ii) few-label alignment with humans, and (iii) hypothesis-driven detection of judge biases. Experiments on BigGen Bench and Chatbot Arena show lower cross-entropy, tighter calibration, better accuracy, and interpretable bias coefficients (e.g., length, creativity) than raw or logistic-regression baselines.", "questions": "See my feedback above.", "limitations": "Limitations are addressed in the discussion and corresponding questions are asked in my review.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I find the paper’s overall quality and clarity strong, and I especially appreciate the rigorous derivations presented in Proposition 3.1 and Theorem 3.2. Nevertheless, I have several concerns.\n\nFirst, I am unsure how sensitive the method is to the choice of covariates. If handcrafted features were replaced by GPT-based embeddings, would the estimated γ coefficients shrink, explode, or remain stable? Quantifying this effect would illuminate how interpretable and trustworthy the bias term truly is.\n\nSecond, the discussion acknowledges that the model may break down when the latent-score assumption is violated, yet I see no empirical test of this vulnerability. Have you simulated scenarios in which the true relationship between human and LLM scores is nonlinear? A targeted robustness study would increase my confidence in the method.\n\nMy broader worry centers on the significance of this work in today’s LLM-evaluation landscape. As language models tackle ever-more complex tasks, obtaining high-quality \"gold\" human annotations is increasingly difficult, and recent studies highlight substantial disagreement among annotators on nuanced questions. In light of this, can human labels still be considered a reliable gold standard, and is narrowing the gap between humans and LLM judges still the most pressing problem?\n\nRelatedly, your approach assumes discrete rating labels. How might it extend to settings where evaluation feedback is open-ended and expressed in free-form natural language rather than fixed scales?", "was_revised": true, "final_justification": "Thank you for your response and clarification. While I believe the work is solid, I still have some reservations about the broader applicability of the proposed methods in today's landscape. I will be maintaining my current score.", "rebuttal": "Thank you for your thoughtful review of our paper. We have addressed your comments and questions below. Please let us know if you have any additional questions or concerns.\n\n------------------\n\n### **Quantifying how sensitive the method is to the choice of covariates (point on GPT embeddings)**\n\nOur approach relies on logistic regression, a model class that is known to be robust to high-dimensional covariates, provided sufficient data and appropriate regularization are used. For example, if we were to use high-dimensional uninterpretable representations such as GPT embeddings as features, as you suggest, the standard practice would be to apply $l_2$ regularized logistic regression, as this would shrink the coefficients to a point where the method does not overfit the data. However, we have not taken this direction (of using embeddings), as it would substantially reduce the interpretability of our results, a key priority for us. We appreciate this suggestion and will add a discussion on these trade-offs in the paper. Thank you.\n\n------------------\n### **Vulnerability to misspecification**\n\nWhen our model is used for prediction, misspecification is not a significant concern; much of machine learning relies on models that are not exactly correct. If our focus is on statistical inference, the model can still be valuable for uncovering biases in LLM judgments, provided the misspecification is not too severe. The linear predictor we use is quite flexible, as we can include any basis functions of $X$ as covariates (and then capture nonlinear relationships).\n\nEmpirically, we present below a simple simulation in which we introduce a mild nonlinearity into the LLM’s latent score generation to test robustness to misspecification, both for prediction and inference. We draw $Z^h_i$ from a Normal distribution $N(0,1)$ and sample $Y^h_i$ from $\\mathrm{Categorical}\\bigl(p(\\alpha, Z^h_i)\\bigr)$. We then set\n$$\nZ^l_i = \\beta Z^h_i + \\gamma^\\top X_i + \\delta\\bigl(\\gamma^\\top X_i\\bigr)^2,\n$$\nwhere $X_i$ is drawn from a multivariate Normal $N(0,I_3)$ distribution and $\\delta$ takes values in $\\\\{0, 0.1, 0.25, 0.5,1,5\\\\}$, controlling the degree of quadratic distortion. We set $\\beta=1$, $\\gamma=(1,1,1)$, and $\\alpha=(-1,1)$ LLM judgments $Y^l_i$ are sampled from the usual ordered-logit link $p(\\eta, Z^l_i)$, and we fit our original linear model, assuming $Z^l_i = \\beta Z^h_i + \\gamma^\\top X_i$. By comparing the estimated parameters $(\\hat\\beta, \\hat\\gamma,\\hat{Z}^h, \\mathbb{P}(Y^h=k|I,O))$ to their true values in terms of MAE as $\\delta$ increases, we directly measure the impact of model misspecification. The results below show that we can still recover $\\gamma$ (then at least we know approximately how big are the main effects; main channel of biases) and predict $\\mathbb{P}(Y^h=k|I,O)$ with high accuracy, even under moderate misspecification. We will add this experiment to the paper. Interetingly, the most afected results are the ones for $\\beta$ and $Z^h$ which is of less interest.\n\n**MAE**\n\n| $\\delta$ | $\\beta$ | $\\gamma$ | $Z^h$ | $\\mathbb{P}(Y^h=k\\mid I,O)$  |\n|:--------:|:-------:|:--------:|:-----:|:---------------------------:|\n|   0      | 0.010   | 0.014    | 0.014 | 0.002                      |\n|  0.1     | 0.024   | 0.014    | 0.072 | 0.010                      |\n|  0.25    | 0.047   | 0.016    | 0.169 | 0.025                      |\n|  0.5     | 0.266   | 0.017    | 0.340 | 0.045                      |\n|   1      | 0.960   | 0.011    | 0.563 | 0.077                      |\n|   5      | 24.958  | 0.345    | 0.789 | 0.117                      |\n\n\n\n------------------\n\n### **The significance of this work in today’s LLM-evaluation landscape** \n\nThank you for highlighting this broader concern. We fully recognize that as LLM tasks become more sophisticated, achieving a truly reliable \"gold standard\" through human annotation is increasingly difficult. However, our framework is flexible by design and is not tied to a single definition of the gold standard. Whether the benchmark consists of individual human judgments, a consensus among multiple annotators, or even alternative proxies, our approach is meant to detect and reduce inconsistencies between whatever reference is chosen and LLM assessments. In other words, as the notion of the \"gold standard\" evolves, our method remains applicable. Moreover, aligning LLMs with human preferences and judgments will continue to be important for many real-world applications, especially where trust, safety, and social acceptance matter. Since high-quality human annotation is expensive and slow, developing a clear understanding of the differences and similarities between human and LLM judgments remains a key challenge, regardless of how the benchmarks themselves shift over time.\n\n------------------\n### **Extension to open-ended and natural language evaluations** \n\nWe appreciate the insight that various forms of evaluation feedback could be important. A proper way of doing this requires another paper by itself and we defer this point to future work; however we outline some possibilities and add a discussion to our paper. The core ideas should still be aligning human and LLM judgements in a shared representation and detecting systematic divergences. \n\n- A relatively direct way is to use prompt-based or lightweight classifiers to extract attributes (such as sentiment, critique categories, and overall rating) from free-form text. We then treat those attributes as pseudo-labels for calibration and bias detection the same way as in our current work.\n\n- Another route could be to use high-dimensional semantic representations (i.e., using embedding models) for human and LLM judgements, and then align those embedding vectors rather than scaler labels, detecting biases."}, {"review_id": "rIdQ4dRDHU", "reviewer": "Reviewer_uPE2", "summary": "- This paper introduces a statistical framework to model human and LLM judgements, which enables formal tests of human-LLM discrepancies.\n- The authors evaluate the framework on two benchmarks (BigGen Bench and Chatbot Arena) and four LLM judges.\n- The authors study two applications on improving LLM judgements with few annotations and identifying biases in LLM judges.", "questions": "- Can the authors evaluate more models as LLMs on even a subset of the questions? I’m interested in “profiling” different LLMs and seeing trends.\n- Can the authors discuss the use of in-the-wild data and its potential implications?\n- Can the authors clarify comments around both applications and their respective experimental designs?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n- There are only 4 LLM judges evaluated in this work. It would be great to see results on the two applications on a larger set of models, especially bigger / SOTA models.\n- While it is great that the experiments are on in-the-wild data (e.g., Chatbot Arena), it is hard to know what the “ground truth” results should be. For example, with Application 2, we don’t know what the ground truth LLM biases are. It would be helpful to have some synthetic but more controlled experiments to show the effectiveness of the framework. \n\nClarity:\n- The methodology is nicely written; my main comments are on the two applications.\n- For Application 1, given the error bars, it’s not clear that the proposed methods improve accuracy. In fact, even with increasing sample size, the accuracy looks about the same. Could the authors provide a much larger sample size variant (e.g., an order or two magnitude greater) to compare against? Additionally, can the authors discuss how there might be significant variance in human annotators when given only a small number of annotations? How is that accounted for?\n- For Application 2, how do results vary based on the amount of training data? How robust are the findings of observed LLM biases? \n- Additionally, for Chatbot Arena experiments, did you filter for specific categories? The types of biases we might expect could vary depending on the topic.\n\nSignificance:\n- This work contributes to an important and growing area of using LLMs as judges. \n\nOriginality:\n- A formal approach to comparing human and LLM judges is very novel.", "was_revised": true, "final_justification": "The reviewers provided a nice set of follow-up experiments in their rebuttal. I look forward to them integrating these results into the new version of the paper.", "rebuttal": "Thank you for your work on our paper. We have addressed your comments and questions below. Please let us know if you have further questions or concerns.\n\n-----------\n\n### **Results on bigger / SOTA models** \n\nWe have run extra results on GPT-4.1 and GPT-4.1-nano, which are bigger or SOTA models (and affordable). Results are qualitatively similar to our previous models.\n\n**Results for Application 1:**\n\nPlease see \"Whether the proposed method improves accuracy\" point below, where we report full results for accuracy (in all other metrics, our method still beats Raw by a large margin).\n\n**Results for Application 2:**\n \n| BigGenBench              | GPT 4.1 nano | GPT 4.1    |\n|----------------------|--------------|------------|\n| Writing Quality      | **-0.38*****   | -0.10      |\n| Text Length          | **-0.83*****   | **-0.39***** |\n| Positive Sentiment   | **-0.31*****   | **-0.12*** |\n| Layout Density       | -0.23        | **-0.15*** |\n| Causal Markers       | **-0.19****    | -0.09      |\n| Structure Counts     | **0.35*****    | **0.16***  |\n| Sentiment            | **0.24****     | 0.10       |\n| Code Block           | **0.20****     | 0.07       |\n| Compound Sentiment   | **0.27*****    | 0.06       |\n\n\n| Chatbot Arena            | GPT 4.1 nano   | GPT 4.1    |\n|--------------------|---------------|------------|\n| Text Length        | **-2.05*****    | **-0.54***** |\n| Creativity         | **-1.27*****    | **-0.32***** |\n| Bold Text          | **0.74****      | **0.25*****  |\n\n-----------\n\n### **Controlled experiments to show the effectiveness of the framework**\n\nWe have conducted two additional controlled experiments to further validate our method, both of which use semi-synthetic setups that are more realistic than purely artificial data.\n\n1. **First experiment:** We use GPT-4o-mini to simulate human ratings on BigGenBench queries. Next, we run GPT-4o-mini again, this time artificially biasing its latent scores $Z^l$ to *disfavor* specific markdown features—namely, bold/italicized words, headers, and lists. For each markdown feature (corresponding to the covariates $X_j$, $j=1,2,3$), we bias $Z^l$ by subtracting $X_j$ (one at a time). We then estimate $\\gamma=(\\gamma_1,\\gamma_2,\\gamma_3)$ with our method. In this controlled setting, biasing toward feature $j$ should result in $\\gamma_j=-1$ while $\\gamma_i=0$ for $i\\neq j$. Standard errors are shown in parentheses.\n\n| Setting     | $\\gamma_1$ (SE) | $\\gamma_2$ (SE) | $\\gamma_3$ (SE) |\n| ----------- | ------------------ | ------------------ | ------------------ |\n| no bias     | -0.26 (0.21)       | -0.36 (0.35)       | -0.17 (0.16)       |\n| bold/italic | -1.26 (0.21)       | -0.36 (0.35)       | -0.17 (0.16)       |\n| headers     | -0.26 (0.21)       | -1.36 (0.35)       | -0.17 (0.16)       |\n| lists       | -0.26 (0.21)       | -0.36 (0.35)       | -1.17 (0.16)       |\n\n2. **Second experiment:** We conduct a slightly less controlled experiment by prompting GPT-4o-mini to give lower scores to responses containing each markdown feature, one at a time. Since this manipulation is done through prompting rather than direct latent score adjustment, the resulting biases are not as clean; for example, the LLM tends to be consistently biased against lists. Still, the results largely follow the expected direction.\n\n| Setting     | $\\gamma_1$ (SE) | $\\gamma_2$ (SE) | $\\gamma_3$ (SE) |\n| ----------- | ------------------ | ------------------ | ------------------ |\n| no bias     | -0.255 (0.208)     | -0.362 (0.345)     | -0.167 (0.161)     |\n| bold/italic | -1.992 (0.290)     | 0.466 (0.486)      | -1.583 (0.234)     |\n| headers     | -0.069 (0.427)     | -1.014 (0.713)     | -3.061 (0.352)     |\n| lists       | -0.172 (0.319)     | -0.229 (0.536)     | -5.660 (0.260)     |\n\nThese experiments demonstrate that our framework can recover the direction and magnitude of induced biases, both in tightly controlled and more realistic, prompt-based scenarios.\n\n\n-----------\n\n### **Whether the proposed method improves accuracy**\n\nWe repeated the analysis including $n=1280$ as well as both GPT-4.1-nano and GPT-4.1. The table below reports the **average improvement in accuracy** (of the “Ordinal” method over “Raw”) across random seeds, with standard deviations in parentheses. For almost all LLMs, we observe a substantial improvement in accuracy. In a few cases, the gains are within one standard deviation, so we cannot draw a firm conclusion. Taken as a whole, the results show a consistent positive trend in accuracy improvements across LLMs. The gains are still more impressive if we consider cross-entropy loss and calibration metrics though.\n\n**BigGenBench**\n\n| LLM/Sample size             | 20             | 40             | 80             | 160            | 1280           |\n| --------------- | -------------- | -------------- | -------------- | -------------- | -------------- |\n| GPT-4.1-nano    | 0.021 (0.043)  | 0.034 (0.026)  | 0.041 (0.021)  | 0.047 (0.021)  | 0.059 (0.016)  |\n| GPT-4.1         | -0.030 (0.021) | -0.024 (0.039) | -0.021 (0.022) | -0.017 (0.016) | -0.013 (0.017) |\n| GPT-4o-mini     | -0.024 (0.038) | -0.031 (0.044) | 0.000 (0.013)  | 0.004 (0.016)  | 0.010 (0.011)  |\n| LLaMa-3.1-8B-It | -0.001 (0.026) | 0.012 (0.020)  | 0.015 (0.021)  | 0.029 (0.019)  | 0.029 (0.020)  |\n| Selene-1-Mini   | 0.027 (0.035)  | 0.049 (0.022)  | 0.055 (0.023)  | 0.057 (0.019)  | 0.066 (0.021)  |\n| Prometheus-v2   | -0.008 (0.021) | 0.007 (0.022)  | 0.017 (0.018)  | 0.018 (0.019)  | 0.033 (0.017)  |\n\n**Chatbot Arena**\n\n| LLM/Sample size             | 20             | 40             | 80             | 160            | 1280          |\n| --------------- | -------------- | -------------- | -------------- | -------------- | ------------- |\n| GPT-4.1-nano    | -0.006 (0.022) | -0.005 (0.016) | -0.004 (0.015) | -0.005 (0.017) | 0.009 (0.013) |\n| GPT-4.1         | -0.024 (0.025) | -0.033 (0.037) | -0.015 (0.020) | -0.017 (0.019) | 0.001 (0.014) |\n| GPT-4o-mini     | 0.000 (0.017)  | -0.004 (0.019) | 0.003 (0.021)  | 0.013 (0.023)  | 0.030 (0.012) |\n| LLaMa-3.1-8B-It | 0.009 (0.022)  | 0.007 (0.027)  | 0.017 (0.026)  | 0.024 (0.019)  | 0.034 (0.017) |\n| Selene-1-Mini   | -0.013 (0.027) | -0.005 (0.021) | 0.014 (0.019)  | 0.007 (0.027)  | 0.034 (0.015) |\n| Prometheus-v2   | 0.016 (0.034)  | 0.002 (0.035)  | 0.006 (0.028)  | 0.032 (0.016)  | 0.033 (0.014) |\n\n-----------\n\n### **Significant variance among human ratings**\n\nSome of this variance is captured by modeling the probability distribution of $Y^h$, since the same input-output pair $(I, O)$ can lead to a range of human ratings. An alternative approach would be to explicitly model $Z^h$ as a random variable, even after conditioning on $(I, O)$. However, this would considerably increase the model’s complexity, so we chose not to pursue it for now. Exploring this idea is a promising direction for future work though.\n\n-----------\n\n### **For Application 2, how do results vary based on the amount of training data?**\n\nTo assess the robustness of our method in detecting LLM biases, we repeat our analysis using varying random fractions of the available data. We set a significance threshold of 10\\% (i.e., p-values below 0.10 are considered significant) and, for each LLM, treat the detection of nonzero $\\gamma_j$ coefficients as a binary classification problem (reject vs. not reject the null hypothesis). The \"ground truth\" label is determined using the full dataset. For each data fraction, we evaluate how well the method predicts these significance decisions by reporting precision, recall, and accuracy. Our results (see tables below) show that, for both BigGenBench and Chatbot Arena, strong precision and accuracy can be achieved with as little as 50\\% of the data. Recall is more challenging to improve, indicating that some biases may require more data to detect reliably.\n\n**Significance Prediction (BigGenBench)**\n\n| \\% Data | Precision | Recall | Accuracy |\n| ------ | --------- | ------ | -------- |\n| 10     | 0.67      | 0.14   | 0.56     |\n| 25     | 0.63      | 0.23   | 0.57     |\n| 50     | 0.90      | 0.60   | 0.78     |\n| 75     | 0.90      | 0.79   | 0.86     |\n| 100    | 1.00      | 1.00   | 1.00     |\n\n**Significance Prediction (Chatbot Arena)**\n\n| \\% Data | Precision | Recall | Accuracy |\n| ------ | --------- | ------ | -------- |\n| 10     | 0.33      | 0.08   | 0.75     |\n| 25     | 0.92      | 0.38   | 0.85     |\n| 50     | 0.61      | 0.46   | 0.80     |\n| 75     | 0.93      | 0.69   | 0.92     |\n| 100    | 1.00      | 1.00   | 1.00     |\n\nOverall, these results suggest that our method for detecting LLM biases is quite robust, with high precision and accuracy even when only half of the data is used. Recall improves with larger data fractions, highlighting the benefit of more data for sensitivity to weaker effects.\n\n-----------\n\n### **For Chatbot Arena experiments, did you filter for specific categories**\n\nPlease see our last point to Reviewer gh7b.\n\n-----------\n\n### **The use of in-the-wild data (e.g., Chatbot Arena) and its potential implications**\n\nThank you for this point. Our experiments leverage observational Chatbot Arena data to capture real-world usage, which brings both strengths and limitations:\n \n1. In-the-wild data reflect the diversity and unpredictability of real user-LLM interactions, ensuring our findings generalize beyond narrow lab settings.\n \n2. Because the data are not from a controlled trial, our estimated $\\gamma_j$ parameters should be viewed as descriptive measures of how feature $j$ relates to human-LLM rating gaps, not as causal effects. Confounding variables and selection biases may influence these associations."}, {"review_id": "zNAkIsbOHU", "reviewer": "Reviewer_DPsg", "summary": "This paper proposes a statistical model to analyze the discrepancies between LLM judges' outputs (in the form of scores or comparisons) and human evaluations. Based on this model, the authors design a calibration method for LLM judges and conduct hypothesis testing on various covariates used in LLM evaluations. A notable and surprising conclusion drawn from the analysis is that *LLM judges favor brevity relative to human raters*, which stands in contrast to prior findings.", "questions": "1. Line 133: The authors state that using reasoning steps biases the computed probabilities. Could the authors clarify why this is the case? It seems the main challenge is technical—namely, extracting token probabilities step-by-step—which, while complex, may not necessarily introduce bias.\n2. Line 160: Should the model-implied probabilities be conditioned on $\\hat{Z}^l$ rather than $\\hat{Z}^h$?\n3. Line 180: Consider providing a more intuitive explanation of what rejecting $H_0: \\gamma_j = 0$ means in the context of the model.\n4. Line 244: Why are log probabilities used for GPT-4o-mini while CoT is used for the other judges? A clearer justification for this choice would help readers understand the experimental consistency.\n5. Line 257: Does “probabilistic classifier” here refer to the fitted model? If so, it would be helpful to make this explicit.\n6. The multinomial version of the model is placed in the appendix, but given its brevity, it might be better included in the main text to maintain continuity.\n7. The implementation details of the second baseline (\"LogReg\") are unclear. How exactly is it constructed, and how does it differ from the proposed model? A more thorough comparison is needed.\n8. The caption for Figure 1 is incorrectly labeled as Table 1.\n9. According to Figure 1, many results have large standard deviations—much larger than the mean differences between LogReg and the proposed model—raising questions about the robustness of the results.\n10. Line 303: This claim that conflicts with Dubois et al. is that Dubois et al. found LLM judges favor verbosity, not that length-controlled scoring improves human alignment, so the sentence should be revised for accuracy. Furthermore, although Table 2 shows consistent trends across four models, none of them match those used in Dubois et al. To strengthen this point, it would be valuable to replicate the analysis on overlapping models.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is mathematically rich, technically sound, and methodologically coherent. The proposed statistical model supports a fitting algorithm, a calibration method, and covariate-based hypothesis testing, all grounded in solid mathematical foundations.\n* It addresses a crucial issue—the biases of LLM judges—which directly impacts the reliability and fairness of LLM-based evaluations. Importantly, rather than attempting to eliminate all bias, the authors suggest that alignment with human biases may be acceptable, marking a thoughtful and novel departure from work that seeks to “over-correct” for biases.\n* The paper reports several intriguing findings, most notably that LLM judges tend to prefer shorter responses relative to human raters. This challenges previous assumptions and adds valuable insight to the ongoing discourse.\n\n**Weaknesses**\n\n* The definition and scope of “bias” could benefit from further clarification. As acknowledged by the authors (Line 171), bias ideally refers to models relying on features unrelated to content quality. However, many of the covariates included in this study—especially those derived from LLM scores such as factuality—are closely tied to quality, which can lead to confusion. Although this is briefly discussed in the limitations section, a more in-depth explanation would strengthen the paper.\n* Some experimental designs lack detail or appear underdeveloped. In particular, the use and interpretation of the “LogReg” baseline, and how it supports the claim that LLM judges favor brevity, need clearer justification and elaboration.", "was_revised": true, "final_justification": "The author's reply helped me better understand some technical details: LogReg is a naive version of the model proposed in the paper. Through my communication with the author, I gained a clearer understanding of the rationale behind their conclusion on length bias, which significantly differs from previous research.", "rebuttal": "Thank you for taking the time to review our paper. We have responded to your comments and questions below. If you have any further feedback or concerns, please feel free to let us know.\n\n-----------\n\n### **The definition and scope of bias** \n\nWe agree that our current phrasing around bias (Line 171) can be sharpened. In our framework, bias is any systematic deviation of an LLM judge’s rating from a benchmark rater and, in our experiments, we use human ratings as the gold standard. For instance, by including factuality scores as a covariate, we test whether an LLM judge’s deviations from true quality vary systematically with factuality. If so, it indicates a bias associated factuality, i.e.,  how the LLM judge prioritizes factual correctness relative to the benchmark rater. In the introduction of the paper, we will clarify our definition of bias as the conditional dependence of LLM ratings on any feature or attribute of an input/output given human preferences. Thank you for your suggestion.\n\n-----------\n\n### **The use and interpretation of the \"LogReg\" baseline**\n\nThe LogReg baseline can be seen as a naive version of our multinomial logistic regression method. In this baseline, we use the raw probabilities output by the LLM for each possible rating and fit a standard logistic regression to predict human ratings. While this method can perform reasonably well for predictive purposes, it lacks a principled modeling foundation and is not interpretable. In contrast, our multinomial approach directly models the relationship between the LLM's logits and human ratings, offering a clear interpretation as detailed in the appendix. It is also important to clarify that the LogReg baseline is only included in \"Application 1\" for completeness and comparison. It is not used to support the claim that LLM judges favor brevity; that claim is addressed using other analyses in the paper. Thanks for your comment; we will make this clearer in the paper.\n\n-----------\n\n### **Questions**\n\n- **(Q1, line 133) Why using reasoning steps biases the computed probabilities:** This is the case because once the LLM has generated the reasoning steps (sequence of tokens $R$) it conditions its score on $R$, then biasing its ratings towards something that are more compatible with $R$. Ideally, we want to marginalize our extracted probabilities over all possible reasoning paths. That's why we sample many times to estimate these probabilities. In practice, we are using Monte Carlo integration to estimate the following sum\n$$\n\\mathbb{P}(Y^l=k \\mid I,O) = \\sum_r\\mathbb{P}(Y^l=k \\mid I,O, R=r)  \\mathbb{P}( R=r \\mid I,O)\n$$\nand avoiding the use of individual $\\mathbb{P}(Y^l=k \\mid I,O, R=r)$'s. Please let us know if this does not make sense!\n\n- **(Q2, line 160) The model-implied probabilities $p_k(\\hat\\alpha_1,\\ldots, \\hat\\alpha_K, \\hat Z^h):$** This probability depends on $\\hat Z^h$ following its definition on line 104. Here, we use $\\hat Z^h$ to represent an estimated human preference latent factor calculated as $(1/\\hat\\beta)Z^l - (1/\\hat\\beta)\\hat\\gamma^\\top X $, which is substituted in the definition of the  model-implied probability. We will clarify this in the revision.\n\n- **(Q3, line 180) A more intuitive explanation of what rejecting $H_0: \\gamma_j=0$ means:** Thank you for this suggestion. Rejecting the null here means that, holding all other covariates constant, variation in feature $j$ of an output causes a statistically significant change in the gap between human and LLM ratings. Intuitively, this tells us that feature $j$ systematically shifts the LLM’s judgments relative to humans, in other words, it is a source of LLM judgment bias. We will add this clarification.\n\n- **(Q4, line 244) Why are log probabilities used for GPT-4o-mini while CoT is used for the other judges:** We chose to use CoT sampling for open-weight models because CoT improves the quality of those judges, and running these models locally makes CoT sampling computationally inexpensive. In contrast, generating CoT traces for GPT-4o-mini is prohibitively expensive since it relies on the paid API. For this reason, we use log probabilities for GPT-4o-mini instead. Ideally, we would have liked to sample CoT traces for GPT-4o-mini as well, but this was not feasible due to cost constraints. We will clarify this point in the revised text.\n\n- **(Q5, line 257) Does \"probabilistic classifier\" here refer to the fitted model:** Yes, by \"probabilistic classifier\" we mean any of the three methods \"LogReg\", \"Multinomial\", and \"Ordinal\".\n\n- **(Q6) Including the multinomial version of the model:** We will include the multinomial version in the main text.\n\n- **(Q7) The implementation details of the second baseline (\"LogReg\") are unclear:** We have responded to this point above.\n\n- **(Q8) The caption for Figure 1:** Thank you for pointing this out. We will revise accordingly.\n\n- **(Q9, Figure 1) Many results have large standard deviations and robustness of the results:** As mentioned above, the LogReg method is a naive version of our method and should not be viewed as a competitor (since it is not published anywhere). The real competitor is the \"Raw\" method and we clearly beat that one; please check our response to Reviewer uPE2 for a more detailed explanation and an extra experiment.\n\n\n- **(Q10, line 303) Clarify the the verbosity bias claim and replicate on shared models with Dubois et al:** Dubois al. shows in their paper and GitHub repository (AlpacaEval) that the length-controlled version of AlpaceEval correlates better with Chatbot Arena ELO scores and, because of that, they claim better alignment with human scores. In their repo, they explicitly claim that their new metric is \"highly correlated with humans\". To strenghten our point, we have run the same bias analysis for GPT-4-Turbo (main judge in AlpacEval) and show the negative bias towards length still holds:\n\n|             GPT-4-Turbo      | Text Length      |\n|:------------------------:|:---------------:|\n| BigGenBench              | **-0.49*****      |\n| Chatbot Arena            | **-0.90*****      |"}]

Author rebuttal:

Meta-review: # Summary
LLM-as-a-judge do not always align with human annotators, exhibiting systematic and undesired differences, e.g., biases toward certain writing styles. This paper proposes a statistical model to analyze the discrepancies between LLM judges' outputs (in the form of scores or comparisons) and human evaluations. Based on this model, the authors design a calibration method for LLM judges and conduct hypothesis testing on various covariates used in LLM evaluations. A notable and surprising conclusion drawn from the analysis is that LLM judges favor brevity relative to human raters, which stands in contrast to prior findings.

# Strengths
* The paper is well written and structured.
* It addresses a crucial issue - the biases of LLM judges - which directly impacts the reliability and fairness of LLM-based evaluations
* The paper reports several intriguing findings, most notably that LLM judges tend to prefer shorter responses relative to human raters. 
* This challenges previous assumptions and adds valuable insight to the ongoing discourse.

# Weaknesses
* Necessary domain knowledge makes the framework less generalizable
* Human judgement is treated as gold standard, even if they can also contain biases.

# Overall
The paper is well written and there are no major concerns left after the author-reviewer discussion period. A few new experimental results were presented (e.g. results on larger models) and it would be great to include them into the paper.
Decision: Accept (poster)