AtC: Aggregate-then-Calibrate for Human-centered Assessment
Authors: Zejun Xie
Xintong Li
Guang Wang
Desheng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: human-centered assessment
judgment aggregation
calibration
misspecification
human-AI complementarity
human-AI collaboration
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: AtC aggregates human comparisons into a consensus $\hat{\pi}$ and isotonic-calibrates any model’s scores via ${\hat{\pi}}$, delivering decision-ready assessments with guarantees on efficiency, robustness, and optimality.

Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregate-then-Calibrate (AtC), a two-stage framework that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into a consensus ranking $\hat{\pi}$ using a rank-aggregation model that accounts for annotator reliability. Stage-2 calibrates any predictive model’s scores by an isotonic projection onto the order $\hat{\pi}$, enforcing ordinal consistency while preserving as much of the model’s quantitative information as possible. Theoretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymptotically outperforms model-only assessment. Across semi-synthetic and real-world datasets, AtC consistently improves accuracy and robustness over human-only or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.

Published as a conference paper at ICLR 2026
ATC: ACCREGATE-THEN-CALIBBB FOR HUMAN- CENTERED ASSESSMENT
Zejun X Xintong L Guang Wang", Desheng Zhang'
1 Rutgers University 2 Renmin University of China 3 Floriida State University
{zx180, dz220 rutgers.c. edu
2 #Lixintong@ruc.edu guaag. wang@ ssu.edu
ABSTRACT
Human-centered assessment tasks, which are essential for systematic decision- making, rely heavily human judgment and typically lack verifiable ground truth. Existing approaches face dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregale-then-Calibeee that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into consensus ranking के using a rank sunk-aggregation model that accounts for annotator reliability. Stage- calibrates any predictive model's 's scores by an isotonic projection onto the order म. enforcing ordinal consistency while preserving as much of the model's quantitative information as possible. The- oretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymp- totically outperforms model-only assessment. Across seni-synthetic and werl-world datasets, AtC consistently improves accuracy and robustness over human-onke -nlly or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.
(AtC), a two-stage framework
INTRODUCTION
Human-centeres assessment tasks are essential for systematic decision-making in domains where ground truth is costly, unobservable, or only available in the future. Consider two scenarios: (I) Delivery platforms seek to estimate worker workload for fair compensation, yet true energy expendi- ture, though measurable via wearables. is impractical to collect at scale, necessitating reliance on worker judgments (Ahmadabadi et al., [024]] Chatterjee et al.) (2200) (Ho & Vaughan [0122] Xie et al.) 2025b). (2) Conference committees must evaluate paper quality for acceptance decisions, yet the true quality only becomes observable years later through future impact, requiring committees to aggregate reviewer assessments in the present (Sul 2021; Dolati Neghabadi et al., 2019: Zhhnn et al. 2022
Tran et al. 2021). These tasks share key characteristics: they seek standardized assessments from human jdgmentss rather than individual preference satisfaction (Chu & Ghahramani 2005 Chau
et al. 2022 Hu et al. 2022), and while ground truth exists conceptually, such as workload future impact, it cannot be directly observed when decisions are needed (Alur et al.) 2024 Xu et al. 2024).
Existing approaches tend rely either solely human judgments solely model predictions. Methods that use only annotator-provided comparative judgments, such as judgment aggregation algorithms (Chen al. 2013 Zhao et al. 2018 Jin al. 2020), can weight annotators by expertise but remain confounded by inconsistent rating scales. For examplc, lenient expert and a strict novice might agree on an item's 's relative quality yet assign very different absolute scores (Ito & Kashima 2024 Khurana et al.) 2024). Conversely, methods that use only model-generated scores from proxy labels face supervision crisis: the true target values may correlate with latent factors that are impractical to measure at scale, such cognitive load in task difficulty estimation (Xu et al.)
Equal contribution and shared co-first authorship.
Corresponding author.

---

Published as a conference paper at ICLR 2026
2016 Guo et al. 2023 Wang et al. 2023 Lai et al. 2022). This forces models to learn from noisy surrogates, propagating systematic biases into the assessments.
Intuitively, human judgments easy to solicit but lack universal scale, whereas model predictions are consistent by design but require supervised signals (Cowgill 2018 Dawes et al. 1989). Our goal is to combine the strengths of both sources:
the scale from the model assessment and the structure from human judgments (Hemmer et al. 2025). A key insight is that people are typically more reliable at comparing items than absolute scoring (also known as Weber-Feccner laws in psychophysics (Pardo- Vazquez et al. 2019)). We therefore extract only the ordinal information from the noisy human inputs, rather than trusting their raw rating values or aggregated scores.
We propose a two-stage framework called Aggregate- thereg theaaali-tten-aairraeee (AtC) that integrates the com- plementary advantages of human judgments and model assessments. In the aggregation stage. AtC employs heterogeneous rank aggregation approach (Jin et al.) 2019 Chen et al., 2013) Kuhlman Rundensteiner, 2020) derive a consensus ranking of items from multiple annotators, explicitly accounting for each annotator's individual scale and reliability. In the calibration stage, AtC uses iso- tonic regression (Dykstra Robertson, 1982; Fielding 2018 Su 2021) to adjust an initial predictive model's assessments so that they align with the consensus ranking. This two-step approach ensures that the final assessments are monotonic with respect to the human consensus judgments. By com- bining the ordinal structure (from human) with the scoring scale (from the model), theoretically and empirically demonstrate that AtC leverages the best of both: the human judgments provide a reliable ranking backbone, and the model provides consistent scoring, resulting in assessments that outperform either approach alone.
Our contributions are summarized follows:
We formalize specific class of aggregation (not preference optimization) is central for systematic decision docision-making, At addresses these via human-model complementarity: aggregated comparisons provide ordinal constraints, while models estimate metrically scaled scores. Moreover, it can be generalized to any predictive model and thus enables the use of any off-the-shelf predictive model without modification.
Conceptual:
assessment problems where judgment
Theoretical:
We provide comprehensive analysis for At from three perspectives. First, the offers
best of our knowledge, Theorem offerrs the first proof that heterogeneous rank aggregation mod- els are strictly more statistically efficient than homogeneous methods when annotator abilities vary. validating our approach for the initial aggregation stage. Second, the risk analysis in Theorem 3.5 makes novel contributions to isotonic regression theory beyond prior work (Zhang 2002 Chat terjee et al. 2015 Bellec 2018) by accounting for projection onto random cone and managing biased effective noise, which are two necessary conditions for our problem setting. Finally, these components culminate in optimality guarantee in Theorem 3.9 formally demonstrating the power of integrating human ordinal judgments with model-based scores.
Empirical: AtC consistently outperforms human-only and model-only assessments across semi- synthetic and real-world datasets, demonstrating superior accuracy and robustness under varying levels of data degradation.
2
FRAMEWORK
2.1 OVERVIEW
We consider a human-centered assessment task for n items using inputs from m human annotators and a predictive model p. Let s e R" denote the true (unobserved) latent scores of the n items, representing their ideal ground-truth quality (c.g., the true proficiency of student's 's answer the true merit of a product). Since s cannot be directly observed, we collect two complementary sources of information to estimate it:
Human Judgments: Annotators provide comparative assessments (e.g., pairwise preferences or ratings) for various subsets of the items. These judgments are aggregated to produce a consensus score vector, which we denote by $ R". The vector represents the aggregated consensus over item scores derived purely from the given human input. We model as a noisy observation of the scores s; specifically, assume š + ē, where ~ (O 72 ). In practice, is not
2

---

Published as a conference paper at ICLR 2026
observed directly; instead, we estimate this consensus via the Stage-1 procedure. Let E R" denote the Stage-1 I estimate of (as described in Stage- below). Let π be the consensus ranking of the items, obtained by sorting the entries of s* from lowest to highest. This ranking र reflects the collective ordering of items according to the annotators' comparative judgments.
Model Assessments: A predictive model (e.g.. a machine learning model analyzing item features) produces initial score vector 8p R for the items. We do not assume that Sp is perfectly calibrated or aligned with the human judgments; indeed, the model may have been trained on proxy labels or may even be unsupervised due the lack of ground truth. We model Sy as biased observation of the true scores, subject to an unknown systematic offset: specifically, Sp s + v, where v E IR" is an unknown error (or bias) vector. Given the relationship we can relate the model scores to the consensus by writing Sp = S C + assume that the model's noise independent of the annotators' noise (i.e., independent E).
s + from above, v. For simplicity, we
Our goal to produce final calibrated score vector $ R" for the items. The calibrated scores should (i) respect the consensus ordering गे (to maintain agreement with collective human judgment), and (ii) remain as close as possible the model's original scores sp (to preserve the model's 's useful quantitative information). We formalize this goal by defining the ArC estimator as the projection of
Sp onto the set of score vectors that are consistent with म. Let M == {y R" y=(1) $ VP(() S
S. ya(n) } be the set of all length-n vectors that are nondecreasing according to the ranking f. The AIC estimate is then given by:
$ = II (sp) = arg min lly spli2 YEM
(2.1)
where II (sp)) denotes the Euclidean projection of s, onto the constraint set M. By construction, 8 is the closest vector to Sp (in éy-distance) that does not violate the consensus ordering. In other words, 9 is obtained by an isotonic regression of the model's scores onto the ordering म. Figure I
illustrates the roles of s, Sp, and S in Ctc, with pseudocode provided in Appendix A
2.2 STAGE-1: JUDGMENT AGGREGATION UNDER HETEROGENEOUS THURSTONE MODEL
In Stage-1, we aggregate the human judgments into a single consensus ranking. We formulate this rank aggregation problem finding a latent score vector (the consensus) that best explains the annotators' comparative judgments. Each annotator u E (1,..., m) provides pairwise preferences certain item pairs (for example, i > j indicates annotator prefers item i over item j). A key design choice is how to handle observed inconsistencies when different annotators judge the same items. When couriers assess delivery route difficulty, their judgments differ-but this reflects heterogeneity in expertise rather than fundamental subjective preferences. Rather than filtering novice opinions, our approach learns each annotator's 's reliability %o from data and applies optimal weighting to produce consensus closer to the latent target.
Preliminary: Heterogeneous Thurstone Model (HTM). We assume these preferences follow Thurstone choice model, extended allow annotator-specific noise levels-this is HTM (Jin al. 2020). In particular, given any two items i jj that annotator u compares, the probability that prefers to j modeled as: Pr(u i }} symmetric cumulative distribution function (e.g., the standard normal CDF in Thurstone's original formulation, or logistic CDF a Bradley-Terry model), and 7 > 0 annotator u's precision (or consistency) parameter. A larger %/ means annotator u is less noisy (more reliable) in their pairwise comparisons, whereas a smaller Tu implies higher variance (lower consistency) for that annotator. To infer the consensus scores from the observed comparisons, performed under the HTM.
Fly (s, - s,)), where F is a fixed
estimation (MLE) is
Let Du denote the set of observed pairwise comparisons from annotator u, where i > E D. means u preferred item i over j. The log-likelihodl of all annotators' data, given parameters (s, 7). is: {(s,7) = Σ Σ izj E D., log Fly, (9; s, )), with appropriate constraints on (s, for identifiability (e.g., one can fix Σi= S; = 0 or constrain 1 "E= To = 1). The consensus score estimates s" (si..... s. and the annotator reliabilities Y are obtained by maximizing this likelihooooo:: max,,7 {(s, 7), i.. by finding the parameters that best explain the observed comparisons. This optimization can be carried out with iterative methods; for example, can
3

---

Published as a conference paper at ICLR 2026
alternate between updating the score estimates and the annotator precisions until convergence (each update can use gradient ascent on l or other numerical routines).
The result of Stage-1 is estimated consensus
vector s". Sorting the components of s* from lowest to highest yields the consensus ranking स. other words, ãे is the permutation of item indices such that (1) S (22) I (n): This Stage-1 outcome f reconciles the varying, potentially inconsistent human judgments into a single ranked list of items.
2.3 STAGE-2: ASSESSMENT CALIBRATION VIA ISOTONIC REGRESSION
In Stage-2, we leverage the consensus ranking से from Stage-I to calibrate the model's raw scores Sp- The key idea adjust the model's scores so that the final outputs Š respect the consensus ordering π while remaining as faithful as possible to Sp. We use the term calibration here to mean rescaling or aligning the -geerrtee scores with the deraaede ranking constraints, This notion of calibration differs from its conventional meaning in probability estimation or psychology: instead of aligning predicted probabilities with observed frequencies, here we enforce an ordinal consistency between s, and the consensus ranking. In practice, we implement this calibration by projecting 8y onto the monotonicity constraint set defined by म, as formalized in Eq. (2.1.
12
M
Stage- Aggregate
Sp
E IR", #= Sort(-s")
B
geople
M (y: Deit ... I Metnil
CSA
Consensus
A B
A>
Aggregate
Calibrate
Stage-2. 2. Calibrate
Homngatenss
Fix
M M M
8p
5,
M
8
[8,6,6]
Thm. E....
Thm. 3.5 Bllla
+ll & Bigs
Thm. 3.9 [i Isy
Kendall
Figure I: AtC Framework and Its Theoretical Guarantees.
Stage-1 aggregates human judgments into ranking R; Stage-2 calibrates model scores 8p via isotonic projection a.d provide efficiency, risk, and superiority guarantees.
onto M to ohtain 8. Theorems 3.3 3.5
Preliminary: Isotonic regression. As shown in Eq. (2.1). the calibrated output can be obtained by
solving isotonic regression problem: seek the adjusted score vector š in M that minimizes the squared error to Sp. Equivalently, is the solution to min x Σi (x; - sp.i)² 2, which is exactly the projection defined in Eq. (2.1). This optimization finds the closest
vector Sp that does not violate the consensus ranking order. The solution to this isotonic regression is efficiently computed via the Pool-Adjacent- Violators (PAV) algorithm (De Leeuw et al. 2010). PAV iteratively scans through the scores sorted by क to detect any adjacent pair that is out of order (i.e., violates the required non-decreasing condition), and fixes such violations by averaging the offending scores. This process repeats until no more violations remain. The result is the calibrated score vector S, which by construction satisfies (1(1) S S5 S
AdjiatentViollttoss
(2))) (i.e., it respects the consensus ranking).
Notably, if the model's score vector sp already happens to be perfectly consistent with म, the isotonic regression will leave it unchanged. Otherwise, the model's scores are adjusted a minimally invasive way (with respect squared-error) to remove any ranking inconsistencies. The calibrated scores S can be interpreted as a revised set of model-generated scores that combine the model's quantitative estimates with the human-derived ordinal constraints.
3 MAIN THEORETICAL RESULTS
As illustrated Figure At''s intuitive design offers strong theoretical guarantees for human- centered assessment across three perspectives: heterogeneity, robustness, and optimality. In this section, we present the main results, with related work discussions in Section 5 and detailed assump- tions and proofs in Appendix B and
4

---

Published as a conference paper at ICLR 2026
3.1 HETEROGENEOUSS EFFICIENCY GUARANTEE
We now demonstrate the efficiency advantage of the HTM estimator over an estimator derived from a misspecified homogeneous model. When annotators exhibit varying levels of reliability, correctly specified HTM is expected to yield score estimates with lower asymptotic variance compared to homogeneous model that incorrectly assumes uniform annotator reliability, The following lemmas formalize the asymptotic covariance structures of the two estimators.
Lemma 3.1. Under regularity conditions in (White 1982). the score vector estimator Shote obtained from the MLE of the HTM has asymptotic covariance matrix given by:
Las - Lnn I N
(3.1)
where N Σn= h. is the total number of observations. The matrices L.s. Iy, and Lay the Fisher information sub-matrices for s and y. (Le. - Loy I. 1 is the Schur complement of the full Fisher information with respect to s, representing the efféctive Fisher information for s after accounting for the estimation of y. The pseudoinverse addresses the identifiability constraint 1 1=s ===0 =. 0.
Lemma 3.2. Consider a homogeneous Thurstone model assuming a known constant annotator accuracy "oo- The quasi-MLE Shomos derived from this potentially misspecified model, has an asymptotic covariance matrix given by the classic sandwich form:
Shomo
(Ahomo(s.)) Bhomo (soon((s))Acmo((ss)) Asomo(s.)*)
(3.2)
where s, is the probability limit of Shomo- The matrix Asomo(s.) is the negative of the expected Hessian and Bhomo (s.) the expected outer product of the scores from the misspecified model's log-likelihood, with expectations taken under the true heterogeneous data-generating process. The pseudoinverse also addresses identifiability.
Lemmas 3.J and 3.2 allow to formally compare the asymptotic efficiency of the two estimation strategies. The following theorem states that the estimator from the correctly specified HTM is asymptotically superior. The proof is given in Appendix B.4 3.44am and B.5
Theorem 3.3. (Heterogeneous Efficiency Guarantee). Let Mr and Σg be the asymptotic covariance matrices of the score estimators as defined Lemmas 3.1 and 3.2 Then,
Eihamo
(3.3)
where = denotes the Loewner order. This implies that the matrix E Sn is positive semi- definite. Furthermore, if the true annotator accuracies > are not all cqual (i.e., genuine heterogeneity exists), then this inequality is strict within the identifiable subspace, meaning Egnomo positive definite on this subspace.
Einate is
Theorem 3.3
demonstrates that when annotator heterogeneity is present, the correctly specified HTM yields strictly more efficient estimator of the true scores than the misspecified homogeneous model. Explicitly modeling annotator-specific differences provides a demonstrable advantage in estimation precision. The proof is in Appendix B.6
3.2 ROBUSTNESS GUARANTEE UNDER MODEL MISSPECIHICATION
We next analyze how errors from the Stage-1 rank aggregation affect the final calibrated output in Stage-2: an upper bound on the expected squared error El|l S 21 This bound quantifies the performance of the two-stage procedure, where $ is the projection of model predictions sp onto the human-consensus-eee derived cone M, aiming to estimate the subjective optimal scores 5. Our analysis accounts for the misspecification of M and the systematic bias v in Sp relative to s.
We first establish a sharp oracle inequality for least squares estimators, leveraging lemma adapted from (Bellec 2018 Proposition 2. This result bounds the estimation error based on the effective noise properties and the constraint set's geometry.
Lemma 3.4 (Oracle Inequality with Biased Effective Noise). Let š E R be the target score vector (itself random, with E[s] s). Let M IR" he closed convex set. The observations for projection
5

---

the Published as a conference paper at ICLR 2026
Sp. and the effective noise relating by to is & Sp - (e, I.). The least squares estimator is $ 11 (sp). For any u M, the squared estimation error satisfies, almost surely:
s E| - ||u 512 -
sup
5" 0 500
(3.4)
M 011151
where Tū is the tangent cone to M at u.
Taking expectations and choosing II JM (S) leads to decomposition of the total expected risk into three principal components:
20² 8(T (()
B||l-
3122
< E M,
+ E M, n
+
E M.
II
Ta aa (2)
Projection Error==11
Statistical Emor:=Eg
Emor=
(3.5)
Term Ej reflects errors from projecting š onto the potentially incorrect cone MM. Term Eg arises from the zero-mean component of &. where 5(-) is the statistical dimension of the tangent cone. Term E& captures error due to the systematic bias (details are in Appendix C.T). Further bounding these components yields main robustness theorem.
Theorem 3.5. (Robustness Guarantee). The total expected squared error of i' for is bounded by:
Eg In Var E [Iv #) 27 (In(n) + TE + O(1/n))
2||m1
B S 8
(3.6)
n
Here, [f ]]] the expected ranking inversions from Stage- 82 is the variance fixed systematic bias, and YE is the Buler-Mascheron constant.
The first term in the bound of Theorem 3.5 which accounts for errors due to Stage-1 ranking inaccuracies, is directly related the probability of the estimated ranking ãे differing from the subjective optimal ranking म. This probability can be further bounded as follows:
Corollary 3.6 (Bound on Ranking Misspecification Probability). For any two items j. k such that 5j < Sg. let Ak; 5. 5, be the true score gap. Let Mi denote the covariance matrix of the estimation error $ S in Lemma 3.1 and oxx,, (e; ex. Σz(e; ex). The probability that the
estimated ranking differs from the subjective optimal ranking गे (i.e., M # .M) is bounded by:
51 = P(7 #))
Σ J.
0XHx
exp
(Akz)² 2 200
(3.7)
To conclude, the bound highlights in Theorem 3.5 three key error sources: (i) Stage-1 ranking inaccuracies, quantified by the expected number of inversions [[[nv(fr R)] and scaled by the target's variability, which rapidly diminishes with larger score gaps relative to first-stage noise; (ii) statistical uncertainty from the zero-mean effective noise, decaying around (In n)/na (iii) systematic observation bias v, decaying as 1/n. Notably, even perfect first-stage ranking does not eliminate errors from お² and v, while poor initial rankings can significantly inflate the total error, though mitigation of these factors can still lead to effective eтror reduction. The proof is in Appendix
3.3 OPTIMALITY GUARANTEE
Finally, we establish that the AtC yields a calibrated output that exceeds the uncalibrated model's output in accuracy with high probability. Intuitively, if the Stage= aggregation produces an almost- correct ordering of items and Stage-2 calibration correctly removes the bias, then the final result will be closer to the ground truth than using the original model predictions alone. Recall from Corollary 3.6 shows that 6, decays exponentially in the gap-to-noise ratio "x for each pair. If every true gap is not too small compared to the associated noise, then ő becomes negligible. The following proposition shows that Stage-1 produces the correct order of items with high probability.

---

Published as a conference paper at ICLR 2026
JCk Proposition 3.7. Let Amin min Aaj and max = max o Xxx If Tmax
E for some sufficiently
Amin
small 0 (equivalently, the Stage-1 sample size satisfies mk > 1/(Amine))) then ón o i.e.
the estimated ranking space M will equal the true ranking space M.
At the same time, Stage-2 may still fail if the projection is badly perturbed by the noise ~ (000 I)). Let's define the event B = {M = M). then Lemma 3.8.
02 = P(B) <
Σ
exp
A
15
(3.8)
J.K: 9228
Lemma 3.8 shows that the chance calibration error decays rapidly once the agst-aggregttion score gaps exceed the noise level
Because the pairwise comparisons used Stage- 1 are independent of the Gaussian perturbations c in Stage-2, the events in Corollary and Lemma 3.8 are independent.
Theorem 3.9. (Optimality Guarantee). Let 81 and 82 be as in Corollary 3.6 and Lemma 3.8
Then,
with probability at least 81 -2,
s1/
sp s|
(3.9)
i.e. the calibrated output S is strictly closer to the ground truth than the uncalibrated model prediction Sp. As the Stage-1 sample size grows and Amin dominates the noise, both 8, and 6g approach so the probability of improvement approaches 1.
Theorem 3.9 confirms that, once Stage-1 I has gathered enough pairwise comparisons to rank items reliably, the monotone projection Stage-2 invariably drives the (possibly biased) model scores toward the ground truth. In effect, AtC protects against misspecification: even if both aggregated consensus and predictive model are imperfect, enforcing the human derived rank guarantees strictly better estimator with high probability.
EXPERIMENTS
We evaluate AtC on both semi-synthetic and real-world datasets, guided by 6 research questions:
(RQ1) Does AtC's output S outperform human-only assessment s* and model-only assessment s,?
(RQ2) Do heterogeneous rank aggregation methods outperform homogeneous ones, and is AtC's design of using the ranking ß from s* rather than directly using the s" values correct?
(RQ3) Is AtC robust to aggregation errors?
(RQ4) Is AtC robust to model assessments (sp)) errors?
(RQS) Does Att''s assumption that rankings are more reliable than ratings hold true in practice?
(RQ6) How well does At C perform on a real-world task?
Table 1: Semi-synthetic results Reading Level dataset.
Bold indicate the best performance among human-only assessment (s*), model-only assessment (sp), and A assessment (8), respectively. The red denotes the optimal performance across all methods for the given metric.
Kendall 7t
Wasserstein]
KSJ
MSEJ
Stage-1 Method
s"/ sp/
s"/ sp/3
s"/ sp/
s"/ sp/ S
HRA-G
0.375 / 0,399 / 0.410
2,250 2,831 0.839
0.500 / 0.300 / 0.163
8.658 / 29.00 / 8.122
HRA-E
0.375 0.399 0.403
2.243 2,831 0.827
0.498 / 0.300 / 0.163
8.658 / 29.00 / 8.191
HRA-N
0.368 0.399
0.399
2.351 2,831 0.738
0.563 / 0.300 0.192
8.985 29,00 7.919
CrowdBT
0.354 / 0.399 0.399
/
0.843
0.455 / 0.300 / 0.269
8.301 / 29,00 7.555
CrowdTCV 0.339 0.399 0.372
2.272 / 2.831 1.015
0.506 0.300 0.
8.689 29.00 7.809
BTL
0.340 / 0.399 0.373
2.186 2.831 0.894
0.461 0.300 0.300
7.764 29, 99,00 8.097
TCV
0.338 0.399 0.373
2.343 2.8331 0.914
0.547 0.300 0.300
8.943 29.00 8.030
7

---

Published as a conference paper at ICLR 2026
Datasets. We consider two datasets in experiments. (1) Reading-level (semi-synthetic dataset) (Chen et al., 2013) contains pairwise comparisons of text documents based on reading difficulty. 490 documents with known reading levels. 624 annotators providing 12.728 pairwise judgments. (2) Dots-activity (real-world dataset) (Kemmer et al. 2020). benchmark for collective judgment aggregation. The task involves estimating the number of dots in images. The dataset contains two types judgments (ranking and rating) from 300 participants on 30 distinct images, which yield 8700 pairwise comparisons.
Baselines and Metrics. For the semi-synthetic dataset, we compare 7 different aggregation methods: 3 HRA variants based on different noise distribution assumptions (HRA-GrA BEERAGGEENN, 2 heterogeneous methods (CrowdBT/TCY
(Chen et al. (013). and 2 homogeneous baselines (BTL/TCV) (Bradley & Terry 1952]. For the real-world dataset, we additionally compare against human-centered assess- ment methods: GPPL. (Chu & Ghahramani] (005), Rank-SVM (Joachims) 2002). and BARCW (Li et al. 2022). We evaluate using metrics: Kendall T measures ranking accuracy; Wasserstein distance and KS statistic quantify distributional alignment between predicted and ground-truth scores; MSE captures absolute prediction error. Detailed descriptions are provided in Appendix L
Semi-Synthetic Evaluation. We first evaluate on semi-synthetic setting where ground-truth item scores are available. We simulate m annotators who provide pairwise comparisons, apply various Stage-1 aggregation methods (w/ or w/o modeling annotator heterogeneity) to obtain consensus score s", and then train a predictive model to produce an initial model score Sp for each item (using a portion of the ground truth with added noise to emulate model error). We calibrate Sp via AtC to obtain the output S.
In Table 1 we observe that & consistently outperforms both s* and Sp across nearly all metrics and methods (the & entries are almost al- ways bolded as best) Figure 2 further illustrates this improvement: the S values align much more closely with the true distribution of s, whereas the uncalibrated s* and S, scores deviate significantly (same conclusion for all methods in Appendix [..4.. To verify that this im- provement stems from Sp rather than simple rescaling, we conducted experiments matching s' range to ground truth (Appendix E.5)) summary, the AtC score dominates the human-onj) and model-only assessment, confirming that combining the two sources produces more accurate judgments (answering RQ1).
20
10
-10
-2
Figure 2: Score Distribution of HRA-E
We examine the effect of modeling annotator heterogeneity on the consensus ranking quality. In Table [ the methods that account for annotator heterogeneity (HRA-G/E/N, and CrowdBT/TCV) produce better (or the best) calibrated scores & than the homogeneous models (BTL, TCV). Furthermore, we observe that for s", the heterogeneous HRA models achieve more accurate rankings (higher Kendall r). despite performing worse on distance metrics compared to 4 baselines. Notably, after calibration with Sp. the resulting scores from heterogeneous models surpass 4 baselines across Wasserstein distance and KS statistic, confirming that our design choice prioritize ranking information from s* is indeed effective (answering RQ2).
Figure shows At''s robustness to ranking errors in s*. Our normal- ized results show that as we introduce pairwise judgment inversions, AtC degrades gradually until a critical threshold. With moderate noise (up to 400 inversions), Kendall 7 decreases steadily while MSE increases, but At still produces meaningful outputs by leveraging model signals. However, beyond approximately 500 inversions, performance collapses completely-Kendae T becomes negative, variance approaches zero, and the output distribution flattens (shown by declining block count). This synchronized degradation across all metries confirms that while AtC tolerate considerable noise in consensus rankings, extremely corrupted inputs will eventually render the calibration ineffective (answering RQ3).
Mae Blocks Kendall Tau T Varlance
2
250
500
750
1000
Figure
3: Robustness Analysis
'The Kendall's values for 6 and s* differ due tie-creation operation; see Appendix E.3 for details.
8

---

Published as a conference paper at ICLR 2026
Table 2: Real-world results on Dots dataset.
Kendall rf
WassersteinG
KI,
MSEJ
Stage-1 Method
s' s'/ 8p/8
'/Sp//
5'/Sp/
s" s"/ 8p/ /
HRA-G
0.917/0.923
0.940
123.13/0.883
0.866
65.08 / 11.75 9961
ATC HRA-E
0,943
6.53/253//
/ 2.53
123.16 0.881 0.861
65.16 11.75 / 99559
HRA-N
0.917/0.922 0,934
7.05/ 1253/2..5 2.55
125,50/ 0.881 0.855
66.44 / 11.75 / 9.75
GPPL
-/-/0,931
14.56
12662
- 42200.36
Rank-SVM
-1-10.923
-1,161,20
-1 133.0
-/- -1--1 1113814449
BARCW
-/-/0.940
1/-/64.62
-1-124.09
-/-14236.16
Real-World Evaluation. We evaluate the validity and robustness of the AtC framework on the Dots-activity dataset, a benchmark for counting dots in images. We assume models can only process corrupted images due to various constraints, while humans can still make reliable inferences based on experience, making this a valuable buman-centered assessment problem. We introduce various image corruptions to simulate real-world scenarios with noisy or partially occluded data, then generate predictive scores Sp using OpenCV's contour detection (Suzuki & Abe 1985) as an imperfect predictor. As illustrated in Figure 4 we apply 4 corruption types (Hendrycks & Dietterich 2019): global Gaussian blur and three localized corruptions (blur, noise, and whiteout), with degradation intensity controlled by a hyperparameter to evaluate performance across different corruption levels.
(a) Original
(b) BLUR (c) LOCAL BLUR (d) LOCAL NOISE (e) WHITEOUT
Figure 4: Examples of image corruption types applied to the Dcts-activity dataset.
Table summarizes performance across AtC and 3 human-centered assessment baselines (GPPL, Rank-SVM, BARCW); comprehensive results including all Stage-1 methods appear in Appendix E.6
Figure 5 presents the core robustness results for HRA-E, with similar trends for other baseline methods shown in Appendix E.7 The radar charts intuitively visualize the degradation of Kendall's Tau correlation as damage intensity increases (clockwise from the top axis). Each chart compares the initial model (sp). the uncalibrated human consensus (s"), and the calibrated AtC outputs based on rankings (S (Rank)) and ratings (S (Rate)). Across the 4 types of corruption, our proposed primary method @ (Rank) demonstrates remarkable resilience. While the performance of the raw objective model S, degrades sharply with increasing noise, particularly under localized corruptions. This shows that by anchoring the model's scores to the stable structure provided by human consensus, AtC effectively mitigates the impact of noise on the predictor (answering RQ4). Furthermore, these results underscore the superiority of using ordinal (ranking) information for calibration. In every scenario, & (Rank) outperforms S (Rate), the alternative calibrated using aggregated cardinal ratings. This suggests that the consensus ranking (s" provides a more robust calibration framework than aggregated numerical estimates, which are susceptible to individual biases and inconsistent scales (answering RQ5). In summary, the At C framework demonstrates strong real-world performance by maintaining high accuracy even under severe input degradation (answering RQ6).
0.35
0.4
20
25
25
(Rank)
00
(Rate)
0.55
0.55
0.55
3.
0.6
0.75
0.6
0.35
0.0
0.75
0.85
085
07
0.6
0.
(a) BLUR
(b) LOCAI_BLUR
(c) LOISE
(d) WHITEOUT
Figure 5: HRA-E's performance (Kendall 7 under 4 corruption types as damage intensity increases.

---

Published as a conference paper at ICLR 2026
5 ADDITIONAL RELATED WORK
Judgement Aggregation and Rank Aggregation. Inferring a consensus from multiple noisy annotators a common problem in crowdsoureing and psychometries (Murphy & Martin 2003 Liu et al. 2019 Hu et al. 2022). Classic models, like Dawid & Skene's (Dawid & Skene 1979) model
estimate latent true label or score while capturing annotator error rates, with extensions incorporating bias and expertise. Ranking models, such as Bradley-Terr or Churstone-Mostellle variants, account for heterogeneous annotator reliability (Pan et al. 2022). Our Stage-1 algorithm accounts for rater heterogeneity to estimate the consensus ranking s", which is supported by Theorem [3. and experiments, outperforms homogeneous annotator assumptions. Unlike prior methods focused solely on consensus, CtC uses it as ordinal scaffold for predictive model calibration, enhancing downstream efficiency with formal guarantees on predictive performance.
Model Calibration and Isotonic Regression. Calibrating model outputs to better align with empirical outcomes is a common practice in machine learning, especially for probabilistic predictions. Isotonic regression in particular is non non-parametric technique that enforces output monotonicity and has been used to calibrate scores to observed binary labels or probabilities (Luss et al. 2012 Baan al., 2022). In AtC, novelly apply isotonic regression to calibrate continuous model predictions to based target derived from human judgments. This usage differs from standard calibration, as we are mapping model scores an ordering rather than absolute ground-truth values.
Prior work on isotonic regression and calibration (Su. 2021 Chatterjee et al. [0133 Bellec (018) has not, to our knowledge, combined isotonic regression with crowd-sourced ranking constraint. Theorem 3.5 provides calibration eтror analysis with imperfect rankings, demonstrating AtC's ability to effectively calibrate using inferred consensus rather than fixed ground-truth labels. Unlike (Bellec) 2018), our innovation lies in modeling subjective optimum & as random variable with allowable non-zero bias v, enabling monotonic calibration when only relative judgments are available. We choose L2 projection instead of L1 or Lo alternatives because it corresponds to maximum likelihood estimation under Gaussian noise and ensures unique solutions due to strict convexity (Stout 2023).
Human-AI Complementarity. Our work is also related to methods that integrate human and machine Docision-making to improve prediction quality. Such approaches seek to optimally fuse human labels and algorithmic learning (Raykar et al., 2010; Yang et al., 2016 Li et al., 2018; SStyverr al. 2022). Notably, calibration via human judgment has emerged as a practical entry point for real-world assessment, such logistics area difficulty assessment (Xie et al. 20256). AtC extends this line of work in two ways: it generalizes the framework beyond domain settings, and provides an optimality theorem (Theorem 3.9 that explains when and why the two-stage pipeline outperforms either component alone, which gap is common across prior human-in-the-loon methods (Xu et al. 2024) Allu et al.) 2024). Unlike (Pan al., 2022, Li et al. 2022).. which impose linear model constraints, AtC places no restrictions on the aggregation method or the predictive model, making the design modular and broadly applicable.
DISCUSSION AND CONCLUSION
We introduced AtC, a two-stage framework that addresses the fundamental challenge of human- centered evaluation when ground truth is costly, unobservable, or future-dependent. Our theoretical analysis establishes three key results: (1) heterogeneous annotator modeling yields more efficient consensus estimation than homogeneous assumptions, (2) isotonic calibration provides risk bounds even under misspecified rankings, and (3) AtC asymptotically outperforms model-only assessment. Empirically, AtC demonstrates consistent improvements in accuracy and robustness across semi- synthetic and real-world datasets, particularly under data degradation conditions common in practice. The framework suits human-centered assessment tasks where model-only evaluation faces limitations due to unobservable or costly ground truth, such as when models receive degraded inputs while humans access complete information, or when collecting human comparisons is more cost-effective than measuring ground truth directly. Future work can extend AtC in several directions: multi- dimensional assessments in which items are judged along multiple criteria, and domains with expert polarization that require multiple distinct consensus rankings. Another direction is to extend our formulation to LLM-as-a-judge settings, where generalizing the heterogeneous annotator to language model evaluators would help build automated and adaptive assessment systems. We believe that addressing these challenges will broaden the impact of collaborative human-A Al assessment systems.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work raises specific ethics concerns. All datasets used are publicly available and accessed under their respective licenses. do not collect new human-subject data and do not alter existing human data; our study reorganizes and validates existing resources from methodological perspective. No personally identifiable information is processed, and no sensitive applications are targeted.
REPRODUCIBILITY STATEMENT
We release code and datasets at httpss//gittbi: Conn//CMELLLAXx/125555 Seto__tC_ssuppp suupp The paper specifies the problem setup and assumptions; the appendix provides complete proofs, algorithm pseudocode, and experiment details.
REFERENCES
Samin Nili Ahmadabadi, Maryam Haghifam, Vahid and evaluation of crowdsourcing platforms based on users' confidence judgments. Scientific Reports, 14(1):18379, 2024. doi: 10. 10.1038//411 111111//1198--02440 1598-154-----9222222 URL https:: /wt..//o..ooo//////o/ http:///toooo////////// s41598 024 65892
and Sara Ershadmanesh. Design
Rohan Alur, Manish Raghavan, and Devavrat Shah. Human expertise algorithmic prediction. In
The eighth- Annual Conference Neural Information Processing Systems, 2024. URL
https://openreviier net /ttf//oooommmiim//////
Dennis Amelunxen, Martin Lotz, Michael B McCoy, and Joel A Tropp. Living on the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, ((3):224-2999 294, 2014.
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibration when humans disagree. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1895-1915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 18863//112022..mm///lp
124. URL http:////cccaacooooo..oor///////
emm.l cmn.ppmmaing 124/
Pierre C Bellec.
Sharp oracle inequalities for least squares estimators in shape restricted regression. The Annals of Statistics, 46(2):745-780, 2018.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3944 1952. ISSN 00063444, 14643510. URL
Bryan Brancotte, Bo Yang, Guillaume Blin, Sarah Johen-Boulakie Alain Denise, and Sylvie Hamel. Rank aggregation with ties: experiments and analysis. Proc. VLDB Endow. 8(11):
1202-1213, July 2015. ISSN 2150-8097. //doi.org) 117.11477777777..22888
doi: 1014778/23099744889992 URL https
Sabyasachi Chatterjee, Adityanand Guntuboyina, and Bodhisativa Sen. On risk bounds in isotonic and other shape restricted regression problems. The Annals of Statistics, 43(4) 2015.
ISSN 00905364. URL (tRL:ttt::///// http::...sstot..... sttiiorr.rgg/saabee/5555555
Sujoy Chatterjee, Anirban Mukhopadhyay, and Malay Bhattacharyya. A review of judgment analysis algorithms for crowdsourced opinions. JEEE Transactions on Knowledge and Data Engineering, 32(7).124-1228, 2020. doi: 10. 20.1009/KDE..209922000. 29100..44........
Siu Lun Chau, Javier Gonzalez, and Dino Sejdinovic. Learning inconsistent preferences with gaussian processes. In International Conference on Artificial Intelligence and Statistics, pp. 2266-2281 PMLR, 2022.
11

---

Published as a conference paper at ICLR 2026
Xi Chen, Paul N. Bennett, Kevyn Collins- Tholmss,, and Eric Horvitz. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, WSDM 13, pp. 193-202, New York, NY, USA. 2013. Association for Computing Machinery. ISBN 7781450318699 doi: 04.14452233336.2433444 URL
htttp:://boi.org www..////...1112313113.333
2433420
Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pp. 137-144, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 159593180s doi: 10.1145/1102351
1102369. BRRttt::: org 11111/001111551155555333 1102336
Bo Cowgill. Bias and productivity in humans and algorithms: Theory and evidence from resume screening. 2018.
Robyn M. Dawes, David Faust, and Paul E. Meehl. Clinical versus actuarial judgment. Science, 243(4899):1668-166) 1989. ISSN 00368075, 10959203. URL httptt/tp/// jatorrortrrrrroo
stable/170347B
A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):00-28, 1979. ISSN 00359254, 14679876. URL http: /wwp///www...../ jstor. wttp:........tte/eeeeeeeeeeeeee.
Jan De Leeuw Kurt Hornik, and Patrick Mair. Isotone optimization r: pool-adijcent-violatoos algorithm (pava) and active set methods. Journal of statistical software, 32:1 2010.
Parisa Dolati Neghabadi, Karine Evrard Samuel, and Marie-Laure Espinouse. Systematic literature review on city logistics: overview, classification and analysis. International Journal of Production Research, 57(3):865-885 2019.
Richard L. Dykstra and Tim Robertson. An algorithm for isotonic regression for two or more independent variables. The Annals of Statistics, 10(3):08-716, 1982. ISSN 00905364. URL
http:://ww. jst.rrsoooo................. htt.:or.rrr/sstaabee2222255
A. Fielding. Statistical Inference Under Order Restrictions. The Theory and Application of Isotonic Regression. Royal Statistical Society. Journal. Series A: General, 137(1)992-33, 12 2018. ISSN 0035-9238. doi: 10.2307/2345150. URL https: : / doii org /t0.///00...
William H Greene. Econometric analysis. Pretence Hall, 2003.
Baoshen Guo, Shuai Wang, Haotian Wang, Yunhuai L Liu, Fanshuo Kong, Desheng Zhang, and Tian He. Towards equitable assignment: Data-driven delivery zone partition at last-mile logistics. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4078-4088, 2023.
Patrick Hemmer, Max Schemmer, Niklas Küihl, Michael Vössing, and Gerhard Satzger. Com- plementarity in human-ai collaboration: concept, sources, and evidence. European Journal of Information Systems, 34(6):99-10022 2025. doi: (000000066000522222522277833 https:///obl wtt.oo///...100000000000000 202.00000882225......2 2475962
URL
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
URL
Chien-Ju Ho and Jennifer Vaughan. Online task assignment in crowdsoureing markets. In Proceedings of the AAAI conference on artificial intelligence, volume 26, pp. 45-51, 2012.
Robert Hu, Siu Lun Chau, Jaime Ferrando Huertas, and Dino Sejdinovic. Explaining preferences with shapley values. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL htttps://oppnreeeet
net/ met/fooum?2e - o
Shun Ito and Hisashi Kashima. Mitigating cognitive biases in multi criteria crowd assessment, 2024.
httptt / /aaiaaivvvv.. www.iivoorrrgeeeee hbtp://bs/24000 18938
12

---

Published as a conference paper at ICLR 2026
Jiarui Jin, Ming Zhou, Weinan Zhang, Minne Li, Zilong Guo, Zhiwei Qin, Yan Jiao, Xiaocheng Tang, Chenxi Wang, Jun Wang. et al. Coride: joint order dispatching and fleet management for sulti-scale ride- ide-hailling platforms. In Proceedings of the 28th ACM International Conference Information and Knowledge Management, pp. 1983-1992, 2019.
Tao Jin, Pan Xu, Quanquan Gu, and Farzad Farnoud. Rank aggregation via heterogeneous thurstone preference models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4353 4360, 2020.
Thorsten Joachims. Optimizing search engines using clickthrough data. Proceedings of the Eighth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '02, pp. 132, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 1581135677 13567X. doi: 00.1145775044.7750066
UtL/tttps://ab..ooo 1tt.://////..11111.11111.1.1.11.
1ee...111111000 775577
Ryan Kemmer, Yeawon Yoo, Adolfo Escobedo, and Ross Macicjewski. Enhancing collective estimates by aggregating cardinal and ordinal inputs. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pp. 73-82, 2020.
Urja Khurana, Eric Nalisnick, Antske Fokkens, and Swabha Swayamdipta. Crowd-calibrator: Can annotator disagreement inform calibration in subjective tasks? In First Conference on Language Modeling, 2024. URL
Caitlin Kuhlman and Elke Rundensteiner, Rank aggregation algorithms for fair consensus. Proc.
VLDB Endow. (3(12))776-27799 jul 2020. ISSN 2150-8097, doi: 10. 10.44778/3407770:4477855
BtLLhttp::// //doi http:////oo...gr//..100..0 wtt.:oo/./........ 340//34077770 3407855
Tinghao Lai, Yan Zhao, Weizhu Qian, and Kai Zheng. Loyalty -based task assignment in spatial crowdsoureing. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 1014-1023, 2022.
Jing Li, Rafai K. Mantiuk, Junle Wang, Suiyi Ling, and Patrick Le Callet. Hybrid-mst: A hybrid active sampling strategy for pairwise preference aggregation. In Neural Information Processing Systems, 2018. URL https: //ap:: wemanticscholass 100 533446557
Jinlin Li, Xintong Li, and Xiao Zhou. Fap-cd: Fairess-drrvenn age-friendly community planning via conditional diffusion generation. Proceedings of the AAAI Conference on Artificial Intelligence. 39(27):28 (6827):8118-2817777 1766, Apr. 2025. doi: 10. 10.1609/aaii. 35003. URL
arg/index.phg AAAI art httpaaatiieeeveeeeeeee/eee/eee/e 35036
Xinran Li, Dingdong Yi, and Jun S. Liu. Bayesian Analysis of Rank Data with Covariates and Heterogeneous Rankers. Statistical Science, 37(1):1 23, 2022. doi: 10. 10.121420-STSS1/ URL
https://abi.org//10 1214/20-1-- 2
Ao Liu, Zhibing Zhao, Chao Liao, Pinyan La, and Lirong Xia. Learning plackett-luce mixtures from partial preferences. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): org/
4328 4335, Jul. 2019. doi: 10.1609/amaii. 331......10111111333333 URL
index. phpppppppppp AAAII welll/aatiiclen www.c//eeeeeeeeeeeee 4342
Ronny Luss, Saharon Rosset, and Moni Shahar. Efficient regularized isotonic regression with
application gene-gene interaction search. The Annals of Applied Statistics, 6(1):253- -23,, 2012. doi: 10.1214/11AAASS50
htttst http://oooooooo.o/////o// org/10 12t1
Thomas Brendan Murphy and Donald Martin. Mixtures of distance-hased models for ranking data. Comput. Stat. Data Anal., 41:645-655, 2003.
org Drg/CorpusID 27716709
Yuangang Pan, Ivor W. Tsang, Weijie Chen, Gang Niu, and Masashi Sugiyama. Fast and robust rank aggregation against model misspecification. Journal of Machine Learning Research, 23(23):1-355
2022. URL http://t: wtt.pppper/////5..55 w0tp:////3///0//////// http:///mm..orr//ppppp/////////0 215 html
Jose L. Pardo- Vazquez, Juan R. Castiñeiras de Saa, Mafalda Valente, Iris Damião, Tiago Costa, M. Inês Vicente, AndréG. Mendonça, Zachary F. Mainen, and Alfonso Renart. The mechanis- tic foundation of weber's 's law. Nature Neuroscience, 22(9) 2019. doi: 1038/
541593-019-0439-7, URL http:://.oo....ooooo....../../.3
13

---

Published as a conference paper at ICLR 2026
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11(43): 1297-1322, 2010.
html
Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-Al complementarity. Proc. Natl. Acad. Sci. U. S. A., 19(11))x111144111 March 2022.
Quentin F, Stout. L, isotonic regression algorithms using an lo approach, 2023. URL https
Weijie Su. You are the best reviewer of your own papers: An owner- scoring mechanism. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17929-279399 Curran Associates, Inc., 2021.
Satoshi Suzuki and Keiichi Abe. Topological structural analysis of digitized binary images by border following. Computer Vision, Gruphics, and Image Processing, 30(1):32-46, 1985. ISSN 0734-189X. doi: http:://oo.ooo////.000000//44140
URL https: //www /www
sciencedirect com/ ott.a////...aaaaa.ra////////3//3
David Tran, Alexander V Valtchanov, Keshav R Ganapathy, Raymond Feng, Bric Victor Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview; A critical analysis of the machine learning conference review process, 2021. URL https: settps://openreviiee net forumm ?id-
Cn706Ab.JaKM
Hai Wang, Shuai Wang, Yu Yang, and Desheng Zhang. Gerl: Efficient delivery area assignment for last-mile logistics with group-based cooperative reinforcement learning. In 2023 IEEE 39th International Conference Data Engineering (ICDE), pp. 3522-3534. IEEE, 2023.
Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pp. 1-25, 1982.
Zejun Xie, Zhiqing Hong, Wenjun Lyu, Haotian Wang, Guang Wang, and Desheng Zhang. CoAlign: Uncertainty calibration of LLM for geospatial repartition. In Georg Rehm and Yunyao Li (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pp. 254-263, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-9 8777998889111 768--66-66666666---
doi: 10.18653/v (tt.://////225.... 1aw0://ac000222iiiiii induituuutty.19. URL https
//aclanchology.oo wwwp:lloyyoogg/////oooooo /20/g//2225522 acl -ew--nnnuusyy 19/
Zejun Xie, Wenjun Lyu, Yiwei Song, Haotian Wang, Guang Yang. Yunhuai Liu, Tian He, Desheng Zhang, and Guang Wang. Scalable area difficulty assessment with
for nationwide logistics systems. In Proceedings of the 3 ACM SIGKDD Conference Knowledge Discovery and Data Mining V.I, KDD '25, pp. 2713 27224 New York, NY, USA, 2025b. Associa- tion for Computing Machinery. ISBN 979840071 2456. doi: 10.11457369066447704900 URL
https: http::://ooo..g//////// h1tp://...111///11. 1145/ .1444/55599224..33 3709407)
QianQian Xu, Jiechao Xiong, Xiaochun Cao, and Yuan Yao. False discovery rate control and statistical quality assessment of annotators in crowdsourced ranking. In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 1282 1291, New York. New York, USA, 20-22 Jun 2016. PMLR. URL https: /tttps://proceed ngs. ml press/ v48/ www 6, html
Yixuan Even Xu, Hanrui Zhang. Yu Cheng, and Vincent Conitzer. Aggregating quanti- tative relative judgments: From social choice to ranking prediction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 11993--1964.. Curran Asso- ciates, Inc., 2024. URL TRRLhttps///ppccecinng newrssnneuuuipee ccppaper wwwee/fepee.seeeeeeeeeeaa 1 2t1.e/e325055eeaaaaaeee 2tt.TT/0SSSccaaaaccceeeeeeeeeee. corereeeeee...pppppoooooeeeeeeee fdd
Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. Modeling crowdsoureing task complexity. In Proceedings of the Fourth AAAI Conference on Human Computation and Crowdsourcing, pp. 258. AAAI, 2016.
14

---

Published as a conference paper at ICLR 2026
Qinchen Yang, Zhiqing Hong, Dongjiang Cao, Haotian Wang. Zejun Xie, Tian He, Yunhuai Liu, Yu Yang. and Desheng Zhang. Addrllm: Address rewriting via large language model on nationwide logistics data. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I. KDD '25, pp. 2756-2767, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712493 12466. doi: 10.1145/3590664.7709225 |RL/ttpss
//doi.org/ 111..11154119992..331137
Cun-Hui Zhang. Risk bounds in isotonic regression. The Annals of Statistics, 30(2):528 - 555, 2002. doi: 10. 10.1214/aox/02211799666 URL httpp://dd/ http://oo...gg/00..1.... /211/////////////3999/2222
Yichi Zhang, Fang-Yi Yu, Grant Schoenebeck, and David Kempe. A system-level analysis of conference peer review. In Proceedings of the 23rd ACM Conference Economics and Computation, EC '22, pp. 1041-1080, New York, NY. USA, 2022. Association for Com- puting Machinery. ISBN 9781450391504. doi: 10.1145/3490486333885.. URL https
3538235
Zhibing Zhao, Haoming Li, Junming Wang, Jeffrey O. Kephart, Nicholas Mattei, Hui Su, and Lirong Xia. cost-effective framework for preference elicitation and aggregation. In Amir Globerson and Ricardo Silva (eds.). Proceedings of the Thirty-Fourth Conference Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 446 456. AUAI Press, 2018. URL htRp: aww.aiaaai././//aaaa//aaaa//aaaa http:////200///ppreeeeee//////ee
pdf
15

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs solely as general-purpose writing aids: (i) light prose polishing (grammar and clarity), (ii) checking FTTX math syntax and typesetting, and (iii) generating boilerplate ETEX table templates. LLMs did not contribute to research ideation, problem formulation, algorithm or theorem development, dataset curation, experimental design, or result interpretation. All technical content and decisions were authored and verified by the authors.
A ALGORITHM PSEUDOCODE
We provide the pseudocode here to describe the algorithm steps for Stage-l and Stage-2.
Algorithm 1 Heterogeneous Rank Aggregation
Algorithm 2 Isotonic Regression Calibration
1: Input: Comparative judgment data {i > E
1: Input: Model score vector Sp
D,, 1,.... m} from m annotators.
(5p.1:-..;9p..)) consensus ranking π n items.
2: Output: Consensus score vector & and consen-
sus ranking म.
2: Output: Calibrated score vector (Si...., S,) that is monotonic w.rt. f.
3: Initialize item scores (0) + 0 for all [n].
annotator parameters ( + for all u E [m]]
3: Relabel item indices of sy according to
(so that index 1 corresponds to i(1), etc.).
4: for / 2.. until convergence do
Let (yI,...,,,, where yı (1))(1), //2 Sp,((2) ... Un.
ya) be the reordered scores,
5: Update score estimates:
((4) ((-1) + 7/ v s(t-1) (((--1,((111) -0-11)
8p,((z)-
6: Identifiability: s(1) (I 11 /n)s(().
4: Initialize S, < y: for i 1,
5: repeat
7: Update annotator parameters:
6: for = 1 to
do 7: if > 8:+1 then
7M ((-1) + 77, v v (1) fr 1
7
for all u [m].
8:
8, 8:+1 R
8: (Optionally, project (76e) (9) onto its feasible
9:
end if end for
set, c.g., 1 E., 20 = 1.)
10:
11:
until šJ S 82 < ...
9: end for
10: s" +- ((tmm))
12:
Undo the reordering: for each item j. set 8
to the value of the calibrated score assigned to item j's position in the sorted order. 13: return S.
î rrnnk_order(s")
12: return म.
B ASYMPTOTIC EFFICIENCY OF THE HETEROGENEOUS THURSTONE MODEL ESTIMATOR
This section details the asymptotic properties of the score estimators derived from the Heterogeneous Thurstone Model (HTM) and compares its efficiency against an estimator from a misspecified homogeneous model.
B. MODEL DEFINITIONS AND PROBLEM STATEMENT
We consider the task estimating item scores s E R" from pairwise comparison data. The comparisons are provided by users, each user performs k2 comparisons, leading total of N _m h2 observations = [Yathema hm Each Y,, E (0, 1} represents the outcome of comparison between items i and jI by user u.
16

---

Published as a conference paper at ICLR 2026
B.1.1 THE HETEROGENEOUS THURSTONE MODEL
We assume the true data generating process follows a HTM. Under this model, the probability that user prefers item over item is given by:
P(Y.: lw), F(TaaT,.s)
(B.1)
where 8 Th is the vector of true item scores, Ta R+ the accuracy parameter for user u, al Cqj- Eji is difference vector with e; being the i-th standard basis vector, and F(-) is known link function, typically a Cumulative Distribution Function (CDF) such as the logistic function (leading to the Heterogeneous Bradley-Terry Terry-Luce model, HBTL) or the Normal CDF (leading the Heterogeneous Thurstone Case model, HTCV).
The complete parameter vector for the HTM is A (s, R where (71, 7m T The log-likelihood for a single observation (Yut, aI.u) given parameters (s.7u) is denoted by (s Th)) log PsY )h)- The average log-likelihood for all N observations is:
Chete (8; Y)
&(Yuls Tw).
(B.2)
N
The Maximum Likelihood Estimator (MLE) for the HTM is:
Oncte = (Shotas Thete) arg max Chete (0: Y).
(B.3)
We are primarily interested in the score estimator Shete-
B.1.2 THE HOMOGENEOUS THURSTONE MODEL
For comparison, we consider misspecified HTM. This model incorrectly assumes that all users have the same accuracy parameter, = "o for all u, where To is fixed constant or single parameter to be estimated. For simplicity and to highlight the effect of ignoring heterogeneity, we assume %o is known constant (e.g., to 1).
The probability of preference under this model is:
P(Yat = 1/s, 7o) = F(roaT,as)
(B.4)
ky log P(Y 70)
Chomo Y,)
(B.5)
N
The estimator obtained by maximizing Shomm = arg max Chomo
8; Y, 70), the misspecified likeli- hood, a Quasi- Muasi-Maximum Likelihood Estimator (QMLE) Greene 2003 734].
The primary objective of Theorem 3.3 is to demonstrate that, when the true data generating process is heterogeneous, the estimator Shete derived from the correctly specified HTM asymptotically more efficient than the estimator Shomo derived from the misspecified homogeneous model. This will be shown by comparing their asymptotic covariance matrices.
B.2 ASSUMPTIONS
To establish the asymptotic properties of the estimators Shete and Shomo, we impose the following assumptions:
(A1) (Independent Observations) The pairwise comparison outcomes Yu are independent condi- tional on the true parameters.
(A2) (True Data Generating Process) The data are generated by an HTM with true parameters 0g (s" :>") ) E Ru+m
To ensure identifiability of s", constraint such as T s* = is imposed, where 1 is vector of ones.
17

---

Published as a conference paper at ICLR 2026
The vector (7i (m) contains at least two distinct values.
The link function F-) is a known, twice continuously differentiable CDF.
(A3) (Regularity Conditions) Both the HTM and the homogeneous model satisfy standard reg- ularity conditions ((hitee 1982. Assumptions 1-6)), necessary for the consistency and asymptotic normality of MLE and QMLE respectively. These conditions include compact- ness of the parameter space, measurability and smoothness of the log-likelihood functions, existence of relevant expectations, unique identifiaiility of the parameters (or QMLE limits), and non-singularity of key matrices in the identifiable parameter space. Such conditions are generally met by common link functions like the logistic (Gumbel noise) or Normal CDF, provided the parameter space is well-defined and the comparison design is not degenerate. Furthermore, for the correctly specified HTM, Assumption 7 White 1982) is assumed to hold at lg. ensuring the information matrix equivalence.
(A4) (Estimators) Ohete =
(Shete: Thete) is the MLE for the HTM. Shomo is the QMLE for the homogeneous model (with fixed % 1).
B.3 CORE IDEA AND ASYMPTOTIC FRAMEWORK
The efficiency comparison between Shete and Shomo is grounded in the asymptotic theory of MLE and QMLE. The core idea is that an estimator derived from a correctly specified model should be asymptotically more efficient than an estimator derived from a misspecified model.
Under the stated assumptions, the MLE Shete from the correctly specified HTM is consistent for the true score vector s", asymptotically normal, and asymptotically efficient. Its asymptotic covariance matrix, denoted E achieves the Cramér-Rao Lower Bound (CRLB) for estimating s* under the HTM. This matrix is derived from the inverse (or pseudoinverse, to account for identifiability constraints) of the Fisher information matrix associated with the HTM, specifically from the Schur complement corresponding to the s parameters.
Conversely, the QMLE Shomo from the misspecified homogeneous model will converge to parameter vector that minimizes the Bellbackk-eibble divergence between the true data generating process and the misspecified model; generally, 8, * s" when true heterogeneity exists. The asymptotic covariance matrix of Shamo, denoted Σg g is given by the "sandwich" formula, which accounts for the misspecification. Due to the model misspecification, the standard information matrix equivalence does not hold for the homogeneous model, and its QMLE is typically not asymptotically efficient for s*. Moreover, the asymptotic covariance matrix of an estimator from a correctly specified model will be no larger than that of an estimator from misspecified model in the Loewner sense. We aim to formalize this by showing that En Ex is a positive semi-definite matrix, and strictly positive definite in the identifiable subspace when true heterogeneity is present.
To formally compare the efficiency of the estimators Shete and Shomo. we first derive their respective asymptotic covariance matrices by deriving Lemma 3.1 and Lemma 3.2
B.4 PROOF OF LEMMA
The estimator Shete derived from the HTM. The average negative log- likeeiiol function for the HTM, with parameters = (s, 7), is given by
m
k
C(s,7)
log P(Yulss77)
(B.6)
where N = >" Ky is the total number of observations, and P(Y n.) F(rooT, if we denote g(x; Y) log P(Y|z) (x) as the negative log- likelihodo for single observation. For simplicity, we adopt the notation from prior work where g(z) = log F ) is used when the outcome Y., is 1 or F(x)) represents the probability of specifie outcome that leads to the form in the loss function.
18

---

Published as a conference paper at ICLR 2026
The score functions, which are the first-order partial derivatives of L(s, 7), are:
ku o (Tur), Yuas,
V=D(s, N
(B.7)
Ei= o (matTs) ai,s
1
VyL(s, N
(B.8)
Σ/= o' (mmat.s)) al.ss
The Hessian matrix comprises the second-order partial derivatives:
1
())(s,7)
9" (Nuo), (Ta)),, 2a,ua
(B.9)
N
w=1
1=1
kn
(2)((),))) = Ny diag
Σ
9" (Tua)us)) (aTus)²
(B.10)
ku
1 N
(V. (V.Vv((,,)))
(arn)s [o" (7aa)u8) (atw)) + I (TuaT
(B.11)
The Fisher information matrix for single observation Jeet (9)). evaluated at the true parameters Oo (s", >"), is defined the negative expectation of the Hessian of the log-likelihood for that observation. For the average log-likelihood L. the corresponding expected Hessian (or information
scaled by 1/N) is I(s" 7²) = -EEr [(* >] The blocks of this matrix are:
Lee - Ey |s- [(s ,
(B.12)
(Ly))a By in [(V),(("' ;)=""""")) ))
(B.13)
(Isy)i, - By VV.V,,(s C(s", ,1)==,>"))) )),al
(B.14)
and Lya =1yy
The log-likelihhod for single comparison Yu given s* and n. is:
L rw) = Yuu log F( + (1 Yur) log(1 - F(x.)),
(B.15)
where =/, T.." Let ((;; Y) lo F(x) (1 log(11 F(x))) Under standard
regularity conditions for MLE and assuming the model is correctly specified (Assumption A3);
Ey.i (z Y.t)] 0,
(B.16)
Exul (2 Ya)) W(tt),
(B.17)
where W(x)
(=))
P(x)[1P(())
Substituting these expectations, the blocks of the Fisher information matrix (scaled by 1/N) become:
I
(B.18)
l. W (a
(Ir))au
(B.19)
N
(1r7)ia
(B.20)
N
19

---

Published as a conference paper at ICLR 2026
The asymptotic covariance matrix of the MLE thete (Shete: Thete) is NN [Teete ( (fu) | where Inete is the Fisher information for a single observation. The matrix 1(s">7" derived above corresponds to N I if we interpret the sums averages per observation. We define:
Stotal = 5 N. (Ias - T Iyy);
(B.21)
where Jas, 1sy, Jny are the blocks of N ((8",>") (i.e., sums without the 1/N scaling).
The asymptotic covariance matrix for (S s*) is then the top-left (n x n) block of [Thete (fu) Using the formula for the inverse a partitioned matrix, this block is:
(Stotal/ /N)* = S*,
(B.22)
where - - and I: 1n F' are the blocks of the Fisher information for a single observation (i.e., the expressions equation B. 18 equation B.20 without the 1 1/N factor and with sums replaced by expectations single representative term all h2 k and N mk).
Due to the identifiability constraint:
s'==
(B.23)
the score s is estimable only up to an additive constant. This implies that the Fisher information matrix ( is singular, and specifically, its block S is singular with $1 0. Let S = UAU be the spectral decomposition of S, where U is an orthogonal matrix of eigenvectors and A = The Moore-
diag(>1, 1, 0) the diagonal matrix of eigenvalues, with Ap,dd Penrose pseudoinverse is:
n-1
St --= = Σ x
(B.24)
where A diag(1/Ar, 1/A2, 1/Ap-1, 1, 0). And this S represents the inverse of S restricted to the identifiable subspace:
=
1v==0)
(B.25)
The asymptotic covariance matrix for / (Shete s²) is therefore S+. This matrix is the Cramér-Rao Lower Bound for estimating under the HTM, considering the estimation of of and the identifiability constraint s*.
B.5 PROOF OF LEMMA
The estimator Shomo is obtained as the QMLE from the HTM, which assumes a common accuracy parameter "No for all users. The average log-likelihood for this model is:
ΣΣ> Ru
1
Chomo (s; Y, 70)
log P(Ynals,no 70)
(B.26)
N
a=
t=1
where P(Y..|s, 70) =((raa Under Assumption (A2), the true data generating process is an HTM with heterogeneous accuracies 7* (71, Yom T, where not 7 are equal to rr- Thus, homogeneous model is misspecified. According to White (1982, Theorem 2.2). the QMLE Shamo converges in probubility to a parameter vector s,, which is the unique minimizer of the Kullback- Leibler divergence between the true data generating process and the family of distributions defined by the misspecified homogencous model. Equivalently, s, maximizes the expected misspecified logelike;;hood:
S. = arg max Eyjs [log P(Y..ss, 70)]
(B.27)
Due to the model misspecification. it is generally true that s, + s".
Furthermore, according to White 1982 [hheremm 3.2), the QMLE Shoma is asymptotically normally distributed:
VN S.) 2 NO0, Chomo
(B.28)
20

---

Published as a conference paper at ICLR 2026
where (s. is the "sandwich" covariance matrix given by
Chomo [s.om A
(B.29)
The matrices Ahomo (s.) and Bhomo (s.) are defined for single observation under the true data generating process (s* ,Y" evaluated s.:
Ahomo (s.) E n V log P(Yui|s.; 70)]
(B.30)
Bhomo (s.) E nn [(V, log P(Y.i (o)) log P(Y.:|s.:
(B.31)
The expectation Ey o [:] taken with respect to the true probability distribution of Yut de- termined by 8" and 7- Due to the model misspecification, the information matrix equíva- lence W White 1982 [Teerrem 3.3] generally does not hold for the homogeneous model, meaning Ahmmo Sw * -homo
Moreover, account for the identifiability constraint 1ss O, the inverses in the sandwich formula are replaced by pseudoinverses. Thus, the asymptotic covariance matrix for (N - s.) is more precisely written as:
Asy (Shomoo - s.)) [Ahomo (s.) Bhomo (s.) (s.))*
(B.32)
B.6 PROOF OF THEOREM 3.3
We aim to compare the asymptotic efficiencies of Shete and Shomo by examining their respective asymptotic covariance matrices. From Lemma 3.1 the asymptotic covariance matrix for /N (Shete s*) is St. This matrix represents the CRLB for estimating s* under the correctly specified HTM, within the identifiable subspace defined by 1 1 0. Thus,
AsyV ([[(/V/) NShhte) S
(B.33)
From Lemma 3.2 the asymptotic covariance matrix for V N (Shhomo s.) is given by the sandwich formula:
Var V N8hooo) = [s,omoss)) + Bhomo (s.., (s.mo
(B.34)
where s, is the probability limit of Shomo under the misspecified homogeneous model, and Ahomo (s.)
and (s.) are defined as in Lemma 3.2
To prove the first part of Theorem 3.3 we compare the matrices from equation B.34 and equation B.33 According to the theory of estimation under misspecified models, the asymptotic covariance matrix of a QMLE from a misspecified model cannot be smaller (in the Loewner sense) than the CRLB achieved by an MLE from the correctly specified model, when both are estimating parameters related to the true data generating process. The matrix S+ the CRLB for under the true HTM. Therefore, it must hold that:
[Anomo (s. (s.)] )1 Bhomo (somoo) 2 SS
(B.35)
This inequality establishes that:
Eghomo N. Egnntte
(B.36)
For the second part of the theorem, concerning the strict inequality, Assumption (A2) states that true heterogeneity exists, meaning the true user accuracies 7t are not all equal to the fixed no used in the homogeneous model. This implies that the homogeneous model is genuinely misspecified. Under such misspecification, the information matrix equivalence As.mo(s.) =- -hhomo (s.) does hold. The QMLE Shomo fails incorporate user-specific accuracy information, leading to a loss of statistical efficiency compared to the MLE Skete from the correctly specified HTM. This efficiency loss manifests as strict inequality in equation B.35 when considering the identifiable subspace Vn {v E T 1 1 = 0). Specifically, for any non-zero vector V11.
(s.)) )}+ Bhomo ( As.mo
(B.37)
This occurs because the misspecification prevents the sandwich covariance matrix from collapsing to the simpler inverse Fisher information form and results in larger variance for any linear combination of parameters within the identifiable subspace, compared to the CRLB achieved by the correctly specified model. Thus, N E Shoma
N Egree is strictly positive definite on V
21

---

Published as a conference paper at ICLR 2026
C ANALYSIS OF LSE UNDER MODEL MISSPECIFICATION AND NON-ZERO MEAN NOISE
C.I PRELIMINARIES: CONES AND STATISTICAL. DIMENSION
This section introduces fundamental concepts related to convex cones, tangent cones, and statistical dimension, which are essential for the subsequent analysis.
Definition C.I (Convex Cone). A set K C R is convex cone if for any 01, 02 K and any non-negative scalars (1, 12 2 O, the linear combination 11 + 12 is also in K. If, additionally, K closed set, it is closed convex cone.
Definition C.2 (Tangent Cone). Let K C. R be closed convex set and let u E K. The tangent cone to K at u, denoted by TK, u. is defined as:
Tk.u [t(v-u): :
(C.1)
where {.} denotes the closure of the set. Intuitively, the tangent cone TK a comprises all feasible directions from scaled by non-negative scalars, along which one move infinitesimally while remaining within K.
If K is closed convex cone, the definition of the tangent property (i.e., if æ E K, then la E K for 1 2 0). For a closed convex cone K and u E K, the tangent cone is given by:
K simplifies due the cone
TKu K + span(1 span((-u]) = {v- tu K
(C.2)
where the sum is the Minkowski sum.
Definition C.3 (Isotonic Cone). The isotonic cone (or non-decreasing cone) in R", denoted by M, defined as:
M: {x (#1....,,,,)
R" 21 x2 2z zn
The set M is closed convex polyhedral cone. For any u E M, the tangent cone TM, captures the local directional constraints at u. u; < U4+1 for some i, the tangent cone allows more freedom in the i-th and (i + 1) 1)-th coordinates compared to when becomes active for directions in the cone.
u;-1, where the constraint S d;+1
Definition C.4 (Statistical Dimension). Let K dimension, denoted by 5(K), is defined as:
12 be a closed convex cone. Its statistical
8(K) = E [ (g)
(C.3)
where g ~ NO, I,)) is standard Gaussian vector in R", and (g) is the Euclidean projection of g onto Equivalently, the statistical dimension can also be expressed as:
27
8(K) E [g [Ix(())
EE
gTo 0EK:101251
sup
(C.4)
The statistical dimension provides a measure of the "complexity" of the cone from a statistical perspective, often related to the effective number parameters model constrained to the cone.
C.2 PROBLEM SETTING AND RELATION TO PRIOR WORK
Let E R be underlying ground truth score vector. The primary target of our estimation in the second stage a related vector E R", which can be thought of as a "subjective optimal point" or noisy version of s. Specifically, is generated as:
E,
(C.5)
where ē ~ II.) Thus, $ is random vector with Els We assume the true isotonic cone relevant to our target s is M := MaG) (or M . if the true ordering is dictated by the mean s; this distinction should be noted, but for now, assume 5 E M in expectation with high probability).
22

---

Published as a conference paper at ICLR 2026
In the first stage of our overall procedure, separate algorithm (e.g.. Algorithm from Jin al. (2020)) processes auxiliary data to produce an estimate of latent scores, s* E IR.. The ranking induced by s", denoted रे := Tss"), defines a (random) isotonic cone M = Ma. This cone M serves as the constraint set for the second stage.
The observations for the second stage, denoted by sp IR", are related to the ground truth s and fixed, unknown systematic bias vector R" follows:
Sp =
(C.6)
Using the relationship = =5- E, we can express Sp in terms of our target S:
Sp s 2) + v = $ セ
The final estimator for the target is obtained by projecting Sp onto the estimated cone M:
$ = J (sp)
Our objective is to analyze the risk of this estimator with respect to s:
E
(C.7)
The expectation is over all sources of randomness: (which makes random) and Balgi (which makes
M random).
The effective noise when estimating from Sp is:
Sp E, in which & -NU,, as R[g] = v and Cov(E) Cov(-Z)= =ō21,
(C.8)
C.3 PROOF OF THEOREM 3.5
In stage-1, the target is s, and our observation is 8y- The effective noise is & = - No)
By Lemma
3.4
we have:
$
||
S
sup {To B B - 1
(C.9)
Taking expectation with respect to & (conditional on M and 8), and choosing u = -I[ (5). we have:
1 Ee
( sup {TO
2
Eg [|| - 5 s <
512
M
(C.10)
n1
BEW
where W = (0 M. | S 1}. Let = -- NO,,0 22 In). The expectation
term is bounded using (a 6)2 < 2a² + 2b?:
( sup - -)
2
(uu (c)TT)) 2 M,5 +2 NEW
vT 2
Eg
M.5
< 2E
sup
NeW
New
(C.11)
= 2 (T
+ 2 M. T
(2)
(C.12)
Thus, given M and 8, the conditional risk
27
Eg [||s - Te M (s)
- 5|2 +
8(T
(M)
Tu.
2w)
(C.13)
Taking the full expectation over M and
E
5
(C.14)
M
2
+ E
M
5(T
M. 1
(C.15)
2
E M
IT
(2)112
(C.16)
23

---

Published as a conference paper at ICLR 2026
We aim bound the risk E|| $11)). where S = [1pp)))) is the estimator for the target 5. Recall the key relationships:
Target for estimation: s + é, where NN(
Observation: Sp = - v.
Effective noise for estimating from Sy: 8p -
M M where s* the output first-stage algorithm, estimating
We assume the expected ordering relevant for s x (E[s)) iTs). Let M Mz()). The етror term arises if M .M and $ M..
From Lemma 3.4 (or Eq. equation C. E.14etutaoon
C.16 in previous discussions), the total risk is
bounded by:
E Mas
[ M
(C.17)
Projection Misspecified Cone
20
E
M
8(T M(
(C.18)
Torm 2: Statistical Error (Zero-Mean Noise Component)
2
E
[ (w)
(C.19)
Term 3: Rias Frror Component
We now bound each term.
Bounding Term 1: Projection Error from Misspecified Cone. Term is EE M.
[[]TT Ja (5)
3 -
This term captures the error due projecting the target onto the cone M, which is estimated from s" and may differ from the true cone associated with (or its mean s).
Let = +s) be the true ranking corresponding to the target of the first-stage algorithm. Let M M₁. The event of an incorrect cone M (relative the first stage's target) is A { * T).
where के = π (8*).
E M.
[
- S = E&
E
M SS)
(C.20)
i E M (i.e., the random target happens to satisfy the random cone's constraints), then [[]
2 0. This occurs if (Ts") is compatible with s. We follow the previous decomposition:
E MM
MM
M
[| (5) د|| = P(A9)
(C.21)
(assuming [TT MT
3| is negligible zero if (() 3) The term ExpE
|| (S)
31 is
bounded by
sup |T (5)
Var(s),
(C.22)
where Var(s) is the sample variance of the components of the specific realization of 5. Taking expectation over
E
E
E& in (ar(s))
(C.23)
The term En Var(s)) is related to the expected dispersion 5. Since ğ
E... St
Eg
5)2
Eg
(C.24)
24

---

Published as a conference paper at ICLR 2026
This can be further analyzed, but for simpler bound, use:
[g(n)ar((5) - Rg [| = || + nō²
(C.25)
For potentially looser but simpler worst-case, if the range of s; is bounded, n Var(s) can be bounded by a term related to the range, e.g., /(R))) if the properties of the ground truth and noise 6a is needed here. For now, we denote it as E (B))
E [a. bj and Rg b a. A bound that depends on
The probability of ranking emror P(A7) ((x((*) ) + π (s)) is:
5
Σ
P(s} > st
(C.26)
jik: By <38
Let 11 8² 5 ~ (0, Ex-), where Eg.
mi St. Let Dnj 5k Bj. and 0g5 = (e)
ex) "gg((e) ex). Using the Cvernoff-Crames bound:
(Arz)²
PSs > si) <
(kj)
V 2T
exp
20x20*' 2
(C.27)
This completes the proof of Corollary
3.6 Thus,
Xx (kj) 2n
(Arz) )2
Term
< Eg
nn Var
$)
exp
(C.28)
20 20xx,a
k.k: CBL
Alternativll,, using the expected number of inversions Inv(=",, ;)) for the first stage:
Σ
P(s) > si) = -E[Tnv(T", #)]
(C.29)
J.k:3 <SN
Then,
Term S Eg Var((s)) E[Inv( #)]
(C.30)
Bounding Term 2: Statistical Error: Term 2 from the main risk decomposition Eq. equation C.15 given by:
27
Term 2 = M,
5(T (TMMAA))
(C.31)
Here, M = Ma(se)) is the estimated isotonic cone, which is always a monotone cone. The point
11 MM)(s) (5) is an element of M.
A fundamental property of tangent cones is that for any closed convex set K and any point u E K, the statistical dimension of the tangent cone Tk bounded by the statistical dimension of the set K itself:
8(TK,u.) S 8(K)
(C.32)
This result is standard in the analysis of such problems (see, e.g., discussions related to statistical dimension in Amelunxen al. (2014)). In case, K = M. Since M is always isotonic cone, its statistical dimension is the same as that of the standard isotonic cone M It is known (cf.
(Amelunxen et al. 2014 Eq. (D.12)) that the statistical dimension of the standard isotonic cone M], is given by the 7ith harmonic number Hn:
8(M)) = Σ H.
(C.33)
k=1
Therefore, for any realization of M and 5:
8(T (TM 1s ((M) = 5(M²) =H,
(C.34)
25

---

Published as a conference paper at ICLR 2026
The harmonic number Hn, has well-known asymptotic expansions. For 1:
H, In(n) TE 2n 12n²
+0
(C.35)
n7
where YE = 0.57721 is the Euler-Mascheroni constant. Thus, we can use the upper bound:
1
H, - (n) YE +
(C.36)
2n
For a simpler, more common bound in such analyses, Hn 2 In(n) + YE or Hn & In(n) + 1 for 2 1. The bound log(en) ) In(n) + 1 used in Bellee (2018) (Eq. 1.29) is convenient and slightly
looser upper bound for Hn. Using Hn I In(n) + YE + O(I/n):
22
27 Hn 27
Term 2 < E M,
Hn
[m(n) + + +0
(C.37)
Alternativll,, using the log(en) bound for consistency with some literature:
Term 2 < 27 log(en)
(C.38)
This provides a bound for the statistical error component that depends on the variance 72 of the zero-mean noise component ë and logarithmically on n.
Bounding Term Bias Error Component. equation is given by:
Term 3 from the main risk decomposition Eq.
2 |
Term 3 = E M,s
M.n ( (w)1
(C.39)
This term arises from the fixed systematic bias in the effective noise & = vー -ー ē. The expectation E MMa averages over the randomness of the estimated cone M and the randomness of the target which influences the point II jū (S) at which the tangent cone is evaluated.
(v)1 A fundamental property of projection onto Let Uproj = IT . ($). We need to bound [[TT T "ymn
closed convex is that does not increase the L2 norm. Therefore,
Kd. (v) ||112
(C.40)
This inequality holds for any realization of M and š. Substituting this into the expression for Term 3:
2
Term 3 < E M,
(C.41)
As fixed vector, ||2// is constant. Then the expectation operator does not change it:
Term 2 1
(C.42)
Y2
This provides an upper bound for the bias error component, indicating that it scales with the squared norm of the fixed bias vector v and decreases with 1/n. The factor of arises from the use of the inequality (a - 6)2² 2a² + 26 when separating the zero-mean noise and bias components in the derivation of Lemma 3.4
Combined Risk Bound.
Combining the bounds for the three terms:
E|
< Egg (ra Var(()] E[/ne(TT" #)]
2 log(em)
2 2|
26

---

Published as a conference paper at ICLR 2026
where [[Ine(T", R] is the expected number of inversions produced by the first-stage algorithm s* with respect to its true target s, and can be expressed as:
+)]
Σ
P(s sj
with each P(s} > s bounded using the Chernoff- Crammer bound:
P(s) $ Vle, ex) Eg (e,
Ex)
(sk - 83)²
V2 2m cxp ((ee) ex) Es (e, -ea)
(At
and Egg is the asymptotic covariance matrix of Lemma
3.1
Additional discussion. The core distinctions remain similar to previous discussions, with the key aspect being that our target for estimation Š noisy version of the underlying ground truth s, our observations Sp are related to via an effective noise term Niv, V2,,02 I,), and the projection is onto random cone M. (1) Differs from traditional isotonic regression which assumes fixed cone and zero-men aero-mean noise relative to its direct target. (2) Differs from fixed model misspecification as our cone random and our effective observation noise fixed bias and variance T relative to the target
D PROBABILITY OF ESTIMATION IMPROVEMENT VIA PROJECTION
In this section, establish probabilistic bound demonstrating that, under certain conditions related to the accuracy of the estimated cone M, the projection- based estimator provides a strictly better estimate of the ground truth than the direct observation Sp.
D.1 PROOF OF COROLLARY 3.6
Lemma D.1 (Restatement of Corollary 3.6). Let s' be the estimate from the first-stage algorithm for its true target For any items j, with S; < Sg. let Auj S4. Sý be the score gap for the first-stage target, and let xx,² 2 (e, ex) ex) Eo score difference from the first stage, where Eg, is the asymptotic covariance of s* 5. Then the probability that the estimated cone M Ma(se) differs from the true cone for the first-stage target M Ma is bounded by
ex) be the variance of the estimated
(Arz) 0 x5
81 P(T((s)) (5))
Σ J.k: <b
exp
(D.1)
2n
Proof. The event ((s*) + (T) occurs and only if there exists at least one pair of items (j,k) k) such that their true order according to 3 is i5 Sx. but their estimated order is 8 s Using the union bound over all such pairs:
P(T(s") + *T()))
Σ
P(s > st)
Let 17 = 5* -3. We assume 7 ~~ NO0, ). The probability of a single incorrect pairwise ranking P(s) > ) = P(n; 17. > 3; - 5,)) Let X,k: = 7k- Then X5t ~ NO, xx,, 2). The probability becomes P(X, Â Applying the Chernoff- Chernoff-Cramer bound (see Appendix C.3 or 1/2 for Z NN(0 1) and t > O. Let tsx- Akj/ox,
similar result), P(Z S W22
Xx,
(Ārr)2
P(G5 Án)
exp
2 x
Summing all relevant pairs yields the stated bound for 81-
27

---

Published as a conference paper at ICLR 2026
D.2 PROOF OF LEMMA 3.8
Lemma D.2 (Restatement of Lemma 3.8p. Let s E R be the ground truth score and s S 6,
where (O, 72 In). Let B be the event B {T( + (s)). Then,
82 = P(B) S
E
exp (Aم,(م))
(D.2)
(Aks())
4G
j.k:82285
where (s) Sk - Sj.
Proof. The event 3 = (S) A [ occurs there exists at least one pair of items (j,k) such that their true order according to is (without loss of generality) 8j Sk, but their order according to S is 5j > šg. Using the union bound over all such pairs (j. k) where Sy < sy:
P(B)
E
P(s) > 5k)
k <Bkk
Consider single pair (j, such that sj < St.. The event Bj Sy is equivalent to (s, + E,) > (8k + (k), which simplifies to:
S Sj
Let Yjx= E3 Ek. Since ēj and éN are independent and identically distributed (0, ō²), their difference Y₁t follows normal distribution:
Y No0, Var(e,) + Var(e.)) (0, (00
Let (s) = St Sj. Since 8, < Ss. we have Az (s) 0. The probability of interest for a single
pair is P(Y,k (s)) Let Z = //k/ / Then Z ~ (O 1).
Axj (s)
P(Yn As)(s) = >>>
2
Applying the Chernoff-Cramen bound P(Z > 1) 1/2m
12 with t (x)(s)
V287
20
(Аer(s))
P(Y, > Axy(s)) - Akj V2 2m
exp
(D.3)
2 (2
(s))
(s) exp
(D.4)
4
Summing this probability over all pairs (j, such that s; < Sy (which is equivalent to summing over pairs j, k such that S& >> Sy by swapping indices if necessary, ensuring (1y(8) remains positive) gives the stated bound for d2.
D.3 PROOF OF THBOREM
Theorem D.3 (Restatement of Theorem 3.9. Let 6, P(r(s*) + (S)) be the probability of ranking error in the first-stage estimation relative to its true target S (as defined Corollary (.6). Let 52 (()(s) # (s)) be the probability that the ranking of the subjective optimal point s differs from the ranking of the ground truth s (as defined in Lemma D.2).. Assume that the true ranking for the first-stage target is consistent with the ground truth ranking, i.e., TT)) Tr)). Then, with probability at least 1 ő1 62.
SS s| < ||sp -s|
(D.5)
provided that sp & Mx(ss)
Proof. Consider the following events:
Let A be the event that the estimated cone M correctly identifies the cone M3 derived from the subjective optimal point i.e., = {M M3).
28

---

Published as a conference paper at ICLR 2026
Let B be the event that the cone MM correctly identifies the true cone MM derived from the ground truth s, i.e., B = {Mg = Ma).
The probability P M3)
(()) P(B (() * Ma(ss) = ((T()) + (s)) = 62, as defined in Lemma D.2 P(B),
=1 61 Thus, P(A) 1 61- And
1 62. Now we let C' be the event that M = MM. Event C occurs with probability at least 51 82.
When event C' occurs, since s defines the cone M,,, it follows that s E TMushuss
s E M.
(D.6)
By the fundamental property of projection onto a closed convex set M, for any vector a E M, we have:
|| 21/
Sp
- + œ 8
(D.7)
Then we can set æ
|sp s11
||
+
(D.8)
This inequality holds when event C occurs. For the strict inequality ||s -
S1 < ||sp =s
31 to hold,
we require that the term ||sp $ is strictly positive. This occurs if and only if Sp o S. which is
true if sp & M. Thus, if event C' occurs and it is also true that Sp & M2, then ||sp 2 > 0. In this situation, we have:
$s - $11 < || - s12
(D.9)
This strict inequality holds when event C occurs and the condition Sp M is met. The condition Sp & Ma means that the biased observation s+v violates the true ordering of s, which is generally true is non-zero and not perfectly aligned with the cone Mg in way that preserves membership. O
Remark D.4 (Conditions for Small Error Probabilities on and 62). The overall risk bound depends
critically on the magnitudes of 61 and 8g. These probabilities become small under favorable condi- tions:
For 8, (First-stage ranking accuracy): The probability P(s} > 8i.) for 3j < Su is small if
the ratio (Ar,)² is large. This occurs when:
",,
1. The true gaps Dsj = 3, between scores (for the first-stage target 8) are large, making items more distinguishable.
2. The variance of the score difference estimates xx,, small. This implies that the first-stage algorithm provides precise estimates of score differences. The matrix Σg- = mk S+ indicates that precision increases with more users (m) or more comparisons per user (k).
2 = (e, ex) TE. (e, ex) is
Thus, ó is small if the first-stage estimation is based on sufficient data and the items it aims rank are well-separated.
For 82 (Ranking consistency between 5 and its mean s): The probability P(sy >> 5x) for
< Sy is small if the ratio
(a)) large. This occurs when:
1. The true gaps (sy(s) 8₁ - 8, in the ground truth scores are large.
2. The variance 7 of the subjective noise (which makes deviate from s) is small.
Thus, 82 is small the ground truth scores s are well-separated and the process generating the subjective optimal point from has low noise.
29

---

Published as a conference paper at ICLR 2026
E ADDITIONAL INFORMATION OF EXPERIMENTS
E DATASETS AND SETTINGS
This dataset contains pairwise comparisons of text documents based on reading difficulty. The dataset includes 490 documents with known reading levels, serving as ground truth difficulty scores. total of 624 annotators provided 12728 pairwise judgments, with each comparison indicating which document is easier to read. The dataset structure includes judge information, judgment outcomes (where "A" indicates document is easier than document B), document identification numbers, and the corresponding reading levels for each document pair. We construct a semi- semi-synthetic setting by generating pairwise comparisons from an underlying Bradley- Teny//hursttone model using the true item scores, while adding sppccit noise to simulate heterogeneity. This provides us with realistic but controlled pairwise preference data. We then obtain an "oracle" objective score Sp for each item by perturbing its true score with additional noise or bias, mimicking the output of an imperfect predictive model. This noisy Sp serves as the initial model assessment for Stage 2 of Atc. The availability of ground truth in these tasks allows to quantitatively evaluate s", Sp, and under various metrics.
Reading Level:
This dataset (Kemmer et al. 2020) contains judgments from 300 participants on 30 distinct images, yielding 8700 pairwise comparisons for estimating dot counts. We transformed this into a human-centered assessment scenario as follows: humans observe original complete images to provide comparative judgments, while the objective model (OpenC''s contour detection) processes corrupted images with varying degradation levels (global blur, localized blur/noise/whiieee whitesutt//tttt...... The task is to estimate dot counts in the original images, ground truth remains unchanged regardless of corruption applied to model inputs. This setup simulates realistic scenarios where human judgment helps calibrate models that receive degraded inputs (e.g., low-quality sensors, occluded views), while humans access complete information through experience or alternative channels. To our knowledge, this represents the first open-source study using real human data in this field; given the scarcity of suitable datasets, this transformation provides our best validation approach on authentic data.
Dots-activity:
E.2 RANK AGGREGATION BASELINES
We compare several methods for aggregating the crowdsoureed pairwise judgments (Stage-1), 1), includ- ing both homogeneous models and heterogeneity-award models:
BTL (Bradley-Terry-Leee)) A classic model for pairwise comparisons that assumes each item i exp(s;) has a latent score s, such that the probability of beating j is Pr(i j) exp(x.)+ We fit the scores s by maximum likelihood, given all pairwise preferences. BTL assumes all annotators are identical and noise is homogeneous (every comparison is an independent sample from the same underlying distribution).
Thurstone Case (TCV): An alternative pairwise comparison model proposed by Thurstone, which assumes differences in item scores are normally distributed. In Case V, Pr(i > j) (s, - s,) where is the standard normal CDF. Like BTL, the homogeneous TCV model assumes a single common noise level for all annotators. We fit item scores by MLE under this probit-based model. BTL and TCV typically produce similar rankings; TCV can be slightly more robust to outliers in some cases (due to the normal logistic noise assumption).
CrowdBT: A variant of the Bradley- -Teryy model that incorporates annotator-specife reliability parameters. CrowdBT (Chen al.] (203), each annotator u has an associated consistency weight or bias parameter that influences the comparisons they provide. Intuitively. this model down- weights votes from inconsistent annotators and up-weights those from reliable annotators, yielding a better aggregate score s*. We implement CrowdBT following the approach of (Chen et al. 2013 including a "virtual annotator" "regularization strategy to stabilize the reliability estimates.
umnotator'regullrizttoo
CrowdTCV: A heterogeneity-aware extension Thurstone Case V that incorporates annotator- specific precision parameters. Each annotator u is assumed to have their own precision %, in the Thurstone model, such that Pr(u: : > j) = @ (s; sy)). This model was introduced by (Jin jet al. 2020). We refer to our implementation simply CrowdTCV. Like CrowdBT, it learns which annotators are more consistent (higher yr,) and which are noisier (lower Tw). improving the quality of the aggregated s*.
30

---

Published as a conference paper at ICLR 2026
Heterogeneous Rank Aggregation (HRA) variants: We adopt the heterogeneous rank aggregation framework of Jin et al. (2020), which explicitly models annotator-specifit noise in the Bradley-Terry and Thurstone settings. Specifically, in the heterogeneous Bradley-Teryy -erry-Luee (HBTL) variant, Pr(u > j) = a (Tr,(s 8,)) where a is the logistic CDF, whereas in the heterogeneous Thurstone Case V (HTCV) variant, the logistic is replaced by the normal CDF कb Both item scores s and annotator precisions } are jointly estimated from data via iterative opúmization. These methods achieve strong aggregation performance by accounting for individual differences: an annotator with a very small Y (unreliable) contributes nearly random comparisons, which the model weight,, whereas an annotator with large "/u given more influence. We use the term "HRA" to refer to this class of heterogeneity-awaan aggregators, with HBTL and HTCV as representative implementations (corresponding to HRA-E and HRA-N in Table !
Each aggregation method produces output consensus score vector s' (determined up to arbitrary scale, which we normalize for evaluation). The s" rank रे serve as inputs to Stage-2 of AtC.
E.3 EVALUATION METRICS
We evaluate the quality of the predicted scores and rankings using the following metries. Let be the ground-truth scores (when available). For distribution-based metries, let p denote the true distribution of scores (or true outcome values) and denote the distribution derived from S.
S = (S....., S.)) be the scores produced by a given method and s = (s S,,)
Kendall's Tau (r): A rank correlation coefficient between the ordering induced by and the true C 1
ordering. It is defined
where C' is the number of concordant item pairs and D is
the number of discordant pairs when comparing the rankings of S and s. We have rankings are identical, t = 0 if the rankings are uncorrelated, and = -1 if one ranking is the exact reverse of the other. Higher indicates better agreement with the ground -rrth ordering.
if the two
MSE: The root-mean-Se mean-squared error of the scores. We compute MSE-
Σ S. s,) This n
1
n
is the square root of the mean squared error (MSE), expressed in the same units as the scores. A lower L2 error indicates that the predicted scores are numerically closer to the true scores
Wasserstein Distance: Also known the Earth Mover's 's Distance, this measures the distance between two distributions. Treating the set of predicted scores ($t} and true scores {s.} as empirical
distributions, the Wasserstein-1 distance defined as Wi (S, s)
[F(x) (x() dx,
where P3 and Fs are the cumulative distribution functions (CDFs) of the predicted and true score distributions. In practice, we compute W1 by sorting the scores and finding the area between the two empirical CDF curves. Smaller W1 implies that the distribution of predicted scores is closer to that of the true scores.
Kolmogoroy-Smirnov (KS) Statistic: Another measure of distributional discrepancy between and s. The KS statistic is Dxs sup [+(x) F=)=) the maximum absolute difference between
x
the CDF of $ and the CDF of This represents the largest gap between the two cumulative distributions. Lower KS values (closer to 0) indicate better alignment of the predicted score distribution with the true distribution.
Kullback-Leible (KL) Divergence: For tasks where we compare probability distributions, use KL divergence to measure how one distribution diverges from another. Let {pr) be the true distribution and q {q:} be the predicted distribution derived from (e.g. by normalizing or exponentiating the š values). The KL divergence of from is defined as DKL q) =
Σ
P: log
Pi We treat lower KL better (with DRI. 0 indicating q exactly equals p). Note that qi
KL is an usymmetric measure and is undefined if there exists some i with p; >> 0 but 9; 0; in our implementation we add a small e to predicted probabilities avoid zeros.
31

---

Published as a conference paper at ICLR 2026
Remark. An important phenomenon in AtC is that S can achieve higher Kendall's than s* despite both inducing the same ranking ã. The key is that often contains ties introduced by the Pool-Adjacent-Voolee Violators (PAVA) algorithm, which are absent in s When sp violates the consensus ranking म. PAVA resolves conflicts by averaging violating scores, creating ties. This improves by converting discordant pairs into tied pairs, which Kendall's penalizes less severely than discordant pairs [Brancotte et al. 2015).
Consider an illustrative example where ground truth is [10, 20,30, 40] with order B < <<< D. Stage-1 produces s* = [1, 2, 5, 4), inducing the ranking A < B < D < C where items C and D are inverted. The model generates Sp [12, 22, 29, 31] with the correct order. PAVA detects the violation at positions C and D, then averages them to produce $ = [12, 22, 30, 30). Computing Kendall's against ground truth yields T(s", rs) = (5 1)/6 0.67 but T(,,s) 8) (5 0)/6 R 0.83. The tie
at positions C and D eliminates the discordant pair, improving correlation despite identical ordinal constraints from Stage-1.
E,4 ESTIMATED SCORE DISTRIBUTIONS ON READING LEVEL DATASET
This section provides visualizations of the estimated item scores obtained by different methods the Reading Level dataset. We present violin plots to illustrate the distribution of scores for each item.
20
20
20
10
10
10
0
0
0
-10
-10
-10
-20
-20
-20
Sp
$
8
5'
Sp
5'
Sp
(a) BTL-MLE
(b) CrowdBT
(c) TCV-MLE
20
20
20
10
10
10
0
0
0
-10
-10
-10
-20
-20
-20
s"
Sp
S
S"
Sp
5"
5p
(d) CrowdTCV
(c) HRA-N
(f) HRA-G
Figure 6: Violin plots of estimated scores for different baseline algorithms.
E.5 IMPACT OF SCALE INCONSISTENCY ON READING LEVEL DATASET
common concern in human judgment aggregation is whether scale inconsistency between consensus scores and ground truth inflates distributional divergence metries. To address this, we conducted experiments scaling s" to match the range of ground truth s using linear transformation:
s* - min(s*)
scated max(s*) min(s*) (max(s) - -m)n + min(s).
Table 3 presents results on the Reading Level dataset comparing four variants: unscaled consensus
(s*) ) scaled consensus (s* (e model predictions (sp). and AtC output (8). The results show that
32

---

Published as a conference paper at ICLR 2026
Sootted substantially improves performance over s* (e.g., Wasserstein distance drops from 2.25 to 1.44 for HRA-G), confirming that scale inconsistency is a major issue. However, still outperforms S Soculed across all metrics and methods, demonstrating that isotonic calibration provides value beyond simple rescaling. This validates that projection onto M's surface yields superior calibration compared to linear transformations that only address scale but not ordinal alignment with model predictions.
Table Scale Inconsistency Analysis on Reading Level Dataset.
We compare unscaled consensus (s ) scaled consensus (Sacateg), model predictions (sp), and AtC output (8). Bold indicates the best performance among scateds S Sp, and S. The red denotes optimal performance all methods.
Wasserstein.
KS.
Stage-1 Method
Sacated/ s Sp/
Sscaied/ s'
Sp/
HRA-G
1.44/ 2.83 0.84
0.31 / 0.50 0.30 0.16
HRA-E
1,46/2.24/ 2.83 0.83
0.34 / 0.50 0..0 0.16
HRA-N
1.45/ 2.83 0.74
0.31 / 0.56 / 0..30/ 0.19
BTL-MLE
1,78 2.19 0.89
0,42 / 0,46 / 0.30 0.30
CrowdBT
0.84
0,40 / 0.46 0.30 0.27
CrowdTCV
1.76/ 2.83 2.88 1.02
0.42 / 0.51 /0.30/ 0.31
TCV-MLE
1.78 2.34 2.83 0.91
0.43 / 0.55 0.30 0.30
E.6 COMPREHENSIVE BASELINE COMPARISON ON HEAL-WORLD DATASET
We present two evaluations on the actsvictivity dataset: (1) score-level performance analysis com- paring human-only (s"), model-only (sp), and AtC (8) assessments across all Stage- aggregation methods, and (2) end-to-end comparison against three established human-centered calibration base- lines:
GPPL (Chu Ghahramani 2005): Gaussian Process Preference Learning models pairwise preferences using GP regression with specialized covariance functions, learning latent utility functions from comparative judgments.
Rank-SVM (Joachims 2002): Support Vector Machine adaptation for ranking problems that learns from pairwise preferences by optimizing margin-basee objectives, commonly used in information retrieval and clickthrough data analysis.
BARCW (Li et al. 2022): Bayesian Approach for Ranking with Comparison Weights, a recent method designed to handle inconsistent preferences by modeling annotator-specific biases and reliabilities using Gaussian processes.
demonstrates that AtC consistently outperforms both S, and s* across all seven aggregation methods. While the three baseline methods achieve competitive ranking accuracy (Kendall's r R 0.92-0.94), they exhibit catastrophic failures in cardinal calibration: MSE deteriorates by 400- 450x and Wasserstein distance by 24-26x compared to AtC. This stark contrast validates A's's unique wrennth--immlttnee
Table 4
achieving accurate absolute score calibration and maintaining ordinal consistency with human consensus, capability not demonstrated by existing human-centered assessment approaches. Moreover, our work also related to methods that integrate human and
machine decision-making improve prediction quality (Li et al. 2025 Yang al. 2025 Xie et al. 2025a).
E.7 ROBUSTNESS RESULTS FOR ALL BASELINE METHODS ON REAL-WORLD DATASET.
This section provides supplementary results complement the main analysis presented in Section We present the complete set of radar plots for the six additional baseline aggregation methods: HRA-G, HRA-N, BTL, TCV, CrowdBT, and CrowdTCV. The experimental methodology, including the four types of image corruption and the evaluation metric (Kendall's Tau), is identical to the one used for the HRA-E analysis in the main body of the paper.
As illustrated in Figures 7 primary finding from the main text strongly reinforced: the AtC framework, when calibrated
TO
12
a consistent trend emerges across all basclines. The
33

---

Published as a conference paper at ICLR 2026
Table Real-world results on Dots dataset.
Bold indicates the best performance among human-only assessment (s") ). model-only assessment (sp), and At assessment (S), respectively. The red denotes the optimal performance actoss all methods for the given metric.
Kendall rt
Wassersteing
KLJ
MSE.
Stage-1 Method
5/sp/8
5//5p/²
s"/ Sp/3
5/ 8p/
HRA-G
0.917/0.922
0.940
6.97 2.53//3//22 2.53
123.13 / 0.881/ 0.860
65.08 / 11.75 9.61
HRA-E
0.922 0.922/ 0.943
6.98 2.98/2.53/2.5
123.16 / 0.881 0.861
65.16 11.5/11..//9.5.
HRA-N
0.917 0.923 0.934
7.05/2.53/2.5
125,50 / /0.881 0.859
66.44/11.75 /9.75
AtC
CrowdBT
0.894 / 0.923 0.927
5.32/2.53/2.55
72.84 / 0,881 / 0.865
/9.28/11.1
10.24
CrowdTCV
0,899 / 0,923 / 0,926
7.3/2253/2.55
84.74/ / 0,881 / 0.865
44.97 1197/1111 11.77 10.24
BTL
0.917 0.937
6.72/2.53/ 2.53
115.37 / /.8811 / 0.860
9.73
TCV
0.933
117.81 /08811 0.878
62.09 11.75 / 10.18
GPPL
-/-/0.931
-/-/64.50
-1-1126.62
-2-14220.36
Rank-SVM
-1-/0.923
-1-/61.20
- 11/113370
-/-/3814.49
BARCW
-1-10.940
-/-664.62
- -1-1
-1-14236.16
using aggregated rankings (6 (Rank)), demonstrates superior robustness compared to all alternatives. In most cases, the performance of 8 (Rank) remains high and degrades gracefully, while the raw objective model scores (s,) are significantly more vulnerable to input corruptions. This consistent outperformance across a diverse set of aggregation algorithms underscores the general applicability and reliability of our proposed ranking- based calibration strategy.
FiRM
FiD000
0:35
0.4
0.35
0.4
0.35
0.4
0.45
0.65
28
51
1
17
22
0.5
3.5
0.53
8
48
15
67
4775
07
0.75
07
0.75
(a) BLUR
(b) LOCAL BLUR
(c) LOCAL NOISE
(d) WHITEOUT
Figure 7: Radar plots under different noise conditions (HRA-G).
0.45
0.45
25
0.5
0.5
0.5
80
82
30
0.55
0.5
68
6.6
6.8
0.
0.5
0.6
-45
3.65
477
2.6
9.75
085
0.75
(a) B...R
(b) LOCAL B..UR
(c) LOCAL NOISE
(d) WHITROUT
Figure& 8: Radar plots under different noise conditions (HRA-N).
34

---

Published as a conference paper at ICLR 2026
166
-10
0.4
04
04
0.5
045
25
0.5
05
0.5
10
0.55
0.5
0.6
0.5
0.5
2:53
5.63
0.3
(a) BLUR
(b) LOCAL
(c) LOCAL.NOISE
(d) WHITEOUT
Figure 9: Radar plots under different noise conditions (CrowdTCV).
1.5
1.0
04
F1la9o
04
0.4
0:45
28
0.5
0.5
0.5
3.0
0.55
0.5
0.0
0.5
0.5
-5
5.5
5.65
0.53
(a) BLUR
(b) LOCA
(c) LOCAL.NOISE
(d) WHITEOUT
Figure 10: Radar plots under different noise conditions (BTL).
1.5
0.4
0..
0..
0.5
0.45
0.45
25
0.5
0.5
0.5
30
0.5
0.59
0.53
0.
6.8
0.0
0.6
0.6
-15
3:55
3.07
085
(a) BLUR
(b) BOUR
(c) LOCAL
(d) WHITEOUT
Figure 11: Radar plots under different noise conditions (TCV).
15
-10
(J86)
0.4
Filano Hihai
0.4
0.4
0.45
0.45
35
0.5
0.5
0.5
3.0
0.5
0.55
0.55
88
0.6
0.6
335
3.83
0.63
(a) BLUR
(b) LOCAI.BIUR
(c) LOCALLNOISE
(d) WHITROUT
Figure 12: Radar plots under different noise conditions (CrowdBT).
35


Reviews:
[{"review_id": "ptbDOdLI6b", "reviewer": "Reviewer_Q8gb", "summary": "This paper proposes an Aggregate-then-Calibrate (AtC) framework to estimate latent true scores when only noisy human judgments and biased model scores are available. The paper provides theoretical results, including an optimality guarantee showing that integrating human ordinal judgments with model-based scores improves accuracy. Experiments on two datasets show that AtC consistently outperforms human-only and model-only baselines.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well written and easy to follow.\n\n- AtC is a lightweight approach with theoretical guarantees, improving model scores by leveraging human labels.", "weaknesses": "- [Q1] The motivation is somewhat unclear. The adjusted score $\\hat s$ preserves the same order as the estimated consensus score vector $s^∗$; that is, $\\hat \\pi$ is consistent between Stage-1 and Stage-2. This makes the need to compute $\\hat s$ questionable. Please provide concrete use cases where the adjusted score is required while the estimated consensus score alone is insufficient.\n\n- There are also some issues regarding the experiment.\n  - [Q2] The Kendall $\\tau$ is a rank correlation. If values in $s^*$ and $\\hat s$ induce the same order, they should yield the same correlation wrt $s$. Am I missing something about how \\tau  is computed here?\n  - [Q3] In Figure 2, the $s^∗$ distribution appears standardized to zero mean and unit variance, which can inflate the distance between $s^∗$ and $s$. Would using a different scaling (for example, matching mean and variance via a simple estimate) make the distributions more comparable?\n  - [Q4] As in Q1, across both datasets it remains unclear when the adjusted score is practically useful beyond the consensus outputs. Examples would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DUufJKm6Og", "reviewer": "Reviewer_Arz1", "summary": "This work introduces a notion of calibration and a technique to calibrate predicted model scores to fare well along this notion. \nThe paper is concerned with settings where human judgments is assumed to be noisy, but where there's assumed value in the aggregated information they provide about relative merit of alternative options.\n\nThe paper fits a statistical model for the aggregation of heterogeneous data, and this provides us with a permutation of the judged items (say, in ascending order of merit). The calibration technique involves projecting predicted model scores to the manifold of score vectors whose coordinates are sorted in compliance with the ranking induced by aggregation of human ratings. The paper proposes to use l2 projection. \n\nThe paper establishes a number of theoretical results and tests the proposed approach in meaningful settings, shedding light onto the significance of the theoretical results. \n\nI find the paper rather clear, and a very interesting read (admittedly, however, I am not knowledgeable of the specific literature it talks to, so my review is that of an educated but non-expert reader),", "questions": "What is the significance of the Euclidean projection? Or, asking differently, what happens if we project differently? Are there any interesting results (theoretical or of computational efficiency) that depend/vary as a function of this decision?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The technique is well-motivated and rather simple\n2. The paper analysis the technique on theoretical and empirical grounds, the experiments cover all claims (incl. analysis of the theoretical claims)\n3. The paper is well-written and I found it rather clear", "weaknesses": "I don't see any major weaknesses, but I have a minor question (for the other box).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j9SCOoPCMM", "reviewer": "Reviewer_2TdT", "summary": "For certain tasks, obtaining the ground-truth is costly, unobservable, or only gets clear in the future. This is the case for tasks such as estimating paper quality (only find out in the future after finding out its impact) or worker workload (though workers are asked to use the same scale, how the scale is applied is up to the individual). \n\nThis paper tackles the limitations of current practices that address such tasks by proposing the Aggregate-then-Calibrate framework (AtC). Aggregating over individual judgments that lead to proxy labels for the actual task does not allow the model to learn the actual features that are relevant for correct decision making. AtC combines both human and model judgments through two steps: \n1. Aggregation step: They yield an ordinal consensus ranking of items from multiple annotators, using a heterogeneous rank aggregation approach. \n2. Calibration step: The model’s initial predictions are calibrated using the consensus ranking through isotonic regression.  \n\nThe paper first provides theoretical guarantees for the framework: \n1. efficient in consensus w.r.t. annotator heterogeneity than homogeneous\n2. robust under model misspecification\n3. gives calibrated output that is more accurate than the uncalibrated model’s output \n\nThe authors evaluate the method on two datasets: \n1. semi-synthetic: pairwise comparison reading difficulty level dataset where ground-truth scores are available but the annotators are synthetic\n2. real-world: ranking and rating-based dots counting in an image, where the image can be obscured and made noisier\n\nResults show that the framework outperforms baselines, especially when facing more noisy data.", "questions": "Did you also control or verify whether the added noise does not to an extent actually change the ground-truth? I think as you add more obscurities, the actual number of dots could change. I would love to hear your thoughts on this or if I misunderstood any detail.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem definition is novel, clear, and covers two aspects that heavily interact with each other but has (to the best of my knowledge) been rather understudied: unobservability of the ground truth that is estimated through noisy proxy labels, where humans can be inconsistent in their judgments. The latter adds another layer of complexity, which makes it an important task type to look at.  \n2. The authors provide theoretical guarantees for the framework, which then highlights the strength of the setup. \n3. I also like the evaluation setup which is clearly mapped out with the 6 research questions. This helps the reader in getting a direct overview of what exactly will be evaluated and why. In addition, this helps in understanding the strengths of the AtC framework. The experiments also cover both a (semi-)synthetic setting and real-world setting.", "weaknesses": "1. I feel as though subjectivity can be a discussion point: I particularly view the inconsistency in how people use these rating scales as something subjectivee (e.g., people experience workloads differently) and the dots dataset also suffers from input ambiguity which leads to subjectivity. \n2. A high-level figure of the entire pipeline would be helpful, Figure 1 is nice but the paper would really benefit from an extra figure that shows an example input and then how AtC leads to a better solution. This is especially helpful since there are so many intermediate steps. \n3. Though the paper is nicely structured, I feel the reader can be taken a bit more by the hand, since the paper covers a lot of steps. I think an extra figure already helps with this. E.g., choice of evaluation metrics could be motivated a bit more, though I understand why, I think the reader can be taken a bit more by the hand here. \n4. The conclusion and discussion can benefit from a clearer takeaway: in which scenarios would you suggest that a researcher or practitioner uses AtC? \n5. Related Work is in the Appendix, which makes it difficult for the reader to adequately place the paper along existing papers. Some decision choices that I questioned before were only cleared up once I saw the Related Work in the Appendix. I understand that there are space constraints but it would be very helpful if at least the important papers are in the main body of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOhOe0TOF4", "reviewer": "Reviewer_hr8J", "summary": "This paper proposes a two-stage framework, Aggregate-then-Calibrate (AtC), for human-centered assessment tasks that lack verifiable ground truth. The authors provide theoretical results for supporting the modeling-selection, robustness, optimality.  Empirical evaluation on semi-synthetic and real datasets suggests that AtC improves robustness and accuracy over human-only or model-only baselines.", "questions": "The isotonic projection enforces the model’s scores to align with the human consensus ranking, implicitly assuming that this consensus ranking is correct. However, the model’s role is also to correct systematic errors in human ranking outcomes. How do you view the balance between improving objective correctness and maintaining adherence to human-centered consensus in the calibration process?\n\nI especially feel that real human studies are important in this research area.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper focuses on an important research question—how to calibrate model's score for human-centered assessment tasks. \n2. The interdisciplinary considerations are insightful, for example, the design of HTM under the consideration of the Weber–Fechner laws. \n3. The theoretical analysis is sound under the modeling assumptions, and the framework performs consistently across multiple datasets, demonstrating its sound performance.", "weaknesses": "1. The paper lacks discussion of prior calibration works. The authors should better clarify how AtC differs from existing human-centered calibration approaches. \n2. The methods' effectiveness depends on several strict assumptions, which may limit its general applicability.\n3. Some key design choices in this paper lack validation through real human studies — for example, whether the HTM-inferred “consensus ranking” truly corresponds to actual human agreement. Moreover, it remains unclear whether the proposed AtC framework ultimately enhances decision-making performance in real-world applications.\n4. The score-level performance comparison (Table 1) is conducted only on semi-synthetic experiments, which limits its generalizability. Moreover, the calibration baselines  include only human-only, model-only, and AtC variants with different stage-1 aggregation methods, which do not constitute rigorous baselines. More appropriate comparisons would involve other human-centered score calibration approaches, rather than what appear to be ablation-style baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)