Pairwise Calibrated Rewards for Pluralistic Alignment
Authors: Daniel Halpern
Evi Micha
Ariel D. Procaccia
Itai Shapira
Venue: NeurIPS 2025 poster ()
Keywords: Pluralistic alignment
Reinforcement learning from human feedback
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Current alignment pipelines presume a single, universal notion of desirable behavior. However, human preferences often  diverge across users, contexts, and cultures. As a result, disagreement collapses into the majority signal and minority perspectives are discounted.  To address this,  we propose reflecting diverse human preferences through a distribution over multiple reward functions, each inducing a distinct aligned policy. The distribution is learned directly from pairwise preference without annotator identifiers or predefined groups.  Instead, annotator disagreements are treated as informative soft labels. Our central criterion is \emph{pairwise calibration}: for every pair of candidate responses, the proportion of reward functions preferring one response matches the fraction of annotators with that preference. We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions. Empirically, we introduce and validate a practical training heuristic to learn such ensembles, and demonstrate its effectiveness through improved calibration, implying a more faithful representation of pluralistic values.

Pairwise Calibrated Rewards for Pluralistic Alignment
Daniel Halpern'
Evi Micha
Ariel D. Procaccia Itai Shapira' University of Southern California
Harvard University
Abstract
Current alignment pipelines presume single, universal notion of desirable behavior. However, human preferences often diverge across users, contexts, and cultures. As a result, disagreement collapses into the majority signal and minority perspectives are discounted. To address this, propose reflecting diverse human preferences through a distribution over multiple reward functions, each inducing a distinct aligned policy. The distribution is learned directly from pairwise preference without annotator identifiers or predefined groups. Instead, annotator disagreements are treated as informative soft labels. Our central criterion pairwise calibration: for every pair of candidate responses, the proportion of reward functions preferring one response matches the fraction of annotators with that preference. We prove that even a small outlier-free free ensemble can accurately represent diverse preference distributions. Empirically, we introduce and validate practical training heuristic to learn such ensembles, and demonstrate its effectiveness through improved calibration, implying more faithful representation of pluralistic values.
Introduction
The alignment problem focuses on guiding Al systems to act in ways compatible with human values and intentions. At a high level, current methods steer models toward desired behaviors using curated sets of human preferences that capture behaviors that humans consider desirable or appropriate. The most successful approach is reinforcement learning from human feedback (RLHF) [1], which first trains a reward model typically via the Bradley-Terry (BT) framework [2], on pairwise-preference data [3, andand then fine-tunes a pre-trained model to align with the learned reward signal.
Implicit in current RLHF implementations is the assumption of shared human intuition-a common ground among evaluators about what constitutes desirable behavior. While this assumption may hold for alignment objectives such as ensuring model safety, it generally does not apply to tasks where interpretations of "right" behavior inherently diverge across backgrounds, cultures, and beliefs [5-8]. BT-based reward models compress these diverse inputs into one scalar, blurring conflicting viewpoints into a one-size-fits-all model. This aggregation leads to a misspecified objective [9, 10] that struggles when faced with plural or contradictory feedback, marginalizing minority perspectives and failing to capture the full spectrum of human values [11-14]. Once such reward learned, algorithms like PPO [15] then push the policy to maximize this signal, triggering preference collapse [16], where majority views are further amplified, and response diversity is reduced [17-21].
To address these shortcomings, we replace the single-reward assumption with a distribution over reward functions: each independently assigns desirability scores to responses and they collectively span plausible interpretations of human judgment. By fine-tuning a distinct LLM for each reward in the support, the approach yields a distribution over policies. At inference time, these policies can be used in several ways [22, 23]: present range of viewpoints or fold them into one inclusive answer, let users pick the policy that matches their taste, or sample responses directly from the distribution.
straightforward approach to derive a distribution over reward functions to partition annotators into predefined groups-such as demographic segments-or to cluster them by the similarity of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Diverse annotators
Soft pairwise labels
Distribution rewards
Distribution policies Each
Herent judges sase same pekling sivers julgmentss
Proferences pooted aoo froet ona parmss protubiities
Learned roword modes lactively saridy painabs callbration
under
RM3
RM4
24
Weighted Ensemble
Figure 1: We explicitly model human preferences as distribution over reward functions without
relying on predefined annotator groups identities. By enforcing pairwise calibration, this distribu- tion faithfully captures the inherent diversity in aggregate human judgments. Each calibrated reward function then induces a distinct policy.
their recorded preferences [9. 24-26]. Such methods implicitly assume that each group holds a stable, coherent preference vector that carries over from context to another. Human preferences, however, are fluid and highly context-dependent [27]; demographic or ideological attributes explain only part of the observed variation, and annotators who match all recorded traits still disagree sharply [28]. Moreover, overly fine-grained groups may lack enough data for reliable training.
Without explicit annotator identification predefined groups, learning distribution over rewards requires disentangling the hidden preference dimensions inside aggregated feedback. Concretely, rather than associating preferences with stable annotator identities, we look for small set of internally consistent reward functions an ensemble that together explain conflicting human judgments across contexts. Our aim is achieve novel property call pairwise calibration: for every pair of candidate responses, the share reward functions in the support the ensemble favoring one response matches the fraction of annotators who express that preference. A pairwise-calibrated ensemble gives each reward function a coherent viewpoint and its own policy. Taken together, these policies mirror broader range of human preferences and avoid preference collapse (see Figure 1). Thus, pairwise calibration provides a principled mechanism to preserve pluralism, without imposing rigid clusters relying on annotator tags.
Our results. In Section 2 we present our model and formally define pairwise calibration. Section shows that while ensembles trivially satisfy pairwise calibration when they are sufficiently large (e.g., at the scale of the number of annotators). constructing such ensembles is NP-hard (Theorem 1). More importantly. our main goal is to find practical ensembles with small support. In light of these observations, we show in Theorem 2 that an ensemble of size O(1/s) is sufficient achieve an average calibration error of e, and Theorem 3 further shows that including extreme outlier preferences is unnecessary, as an outlier-free free ensemble remains nearly pairwise-calibrate Moreover, Theorem 4 shows that achieving pairwise calibration can be learned with a limited number of pairwise comparisons, providing generalization guarantee. Section 4 asks whether such ensembles can be efficiently constructed in practice and answers with forward stagewise additive modeling (FSAM) procedure: each iteration trains a reward model on the current residual calibration error and re-weights into the mixture, giving a tractable method for constructing compact ensemble. Finally, Section demonstrates that these ensembles attain lower calibration mean squared error (MSE) than the thcoretically optimal single deterministic reward and that the reward models within each ensemble are diverse.
Related work. Our work contributes to the growing field of pluralistic alignment, training AI systems to accommodate the natural diversity of human values and perspectives [23, 29-32]. We briefly cover the broad approaches taken in this emerging area. A comprehensive overview can be found in Appendix B.
Current approaches to capture preference diversity include:

---

Modifying single scalar reward models, such as adding regularizers or other algorithmic techniques, to ensure that the reward not overly optimized to the majority [16, 33-35]. Nonetheless, methods reliant on single scalar reward still risk oversimplifying genuine diversity in user opinions.
Training multiple reward models for distinct population clusters or groups [9, 24-26]. While more direct, this approach does depend on defining fixed groups and can struggle with data sparsity.
Personalization, attempting to create model optimized for each individual user [36, 13]. To remain feasible these methods must sometimes make strong assumptions on latent embeddings, or make use of partial fine-tuning. Furthermore, this approach still risks creating "echo chambers."
2 Our Approach
Notation. Let X' be the context space and let y be the response space. A reward function rg: : X' x y R assigns scalar value indicating the desirability of response given context, where 0 E θ is a set of parameters. A k-ensemble (or simply an ensemble) is a tuple (re....., T 01...1....... Olk), where or,,-.., (r are mixture weights forming a probability distribution over the reward functions (each 0j 2 and 1 0;
We denote by N the population of annotators, and write i ~ N' to mean drawing annotator uniformly from the population. We assume each annotator E N has preference relation >; such that for any c X' and distinct responses yı: y2 E V y1 >; y2 x indicates that annotator i prefers Y1 to given context x. We call a tuple (I, y1, (2) a response triple or simply a triple. We assume the reward space is rich enough such that the annotators are reward-indncible: for each i E N,, there is a reward Ty such that y1 b; y/ exactly when TT M1) > TT 32).
A policy π (ty 1) is probability distribution over outputs y conditioned on context We assume there is an underlying distribution over contexts Dx and base policy Ty- Fine-tumed policies are typi- cally learned by optimizing responses to maximize expected rewards according to the corresponding reward function rg.
Pairwise calibration. reflects human preference frequencies. Intuitively, an ensemble is pairwise calibrated if the pairwise comparisons induces align with the proportion of human annotators who prefer one response over another. Formally, consider context z E X and two distinct candidate responses yı, y/2 E y. We define p" (I, y1, 82) to be the true fraction of annotators that prefer y/1 to y/ given context x:
Our key conceptual innovation is the requirement that the ensemble faithfully
p" (x. U1, V2)
n~ [ya b U2
[0, 1].
(1)
Given an ensemble
(ro, a, dy)sE(1s denote the induced preference probability by
8" (z. U1. V2) = Σa 1r0, > re, 8e))
If r is clear from context, we may drop it from the p notation.
Definition 1 (Pairwise calibration). We say an ensemble is e-pairwise calibrated if:
Ex- [(p" V1, 92) - p" (x. y1, u2))) S.
If an ensemble is 0 O-pairwise calibrated, we call it perfectly pairwise calibrated. We use squared error for analytical convenience, though other divergences that penalize discrepancies between e and p* (e.g., absolute error or cross-entropy) could also serve this purpose. Calibration is defined here at the reward level based only on how they order the responses. This not the only natural definition: pairwise calibration could instead depend directly on the probabilities induced by the learned policies. Our choice is primarily motivated by tractability and the observation that downstream policies are largely influenced by the ordering of rewards. See Appendix A for further discussion and justification of this design choice.
We assume that chosen reward functions never produce ties. This is a minor assumption, as in continuous parameter spaces, the parameters that induce ties rs (z, yn) = re (2, ya) with yn of y2 typically form a measure- zero set, so any perturbation almost surely removes them.
3

---

Non-identifiability Once annotator identities are dropped, the only observable data are the pairwise preference probabilities p" in Equation 1. Prior work [37-39] shows that the same p* can be generated by infinitely many distinct mixtures of annotator-specific preferences. Because of this, recovering the "true" annotator preference distribution is iiformation-thoorticalll impossible; our goal is therefore to learn any ensemble that is pairwise calibrated to p", rather than to reconstruct hidden annotator groups.
Policy ensemble inference. A pairwise-calibrated ensemble induces a mixture of policies, (Mi1,..., "k.....,TT),,. where πy is aligned to roy: yet this alone does not tell how to deploy that distribu- tion at inference time. Below we outline three complementary approaches- inspiire by Sorensen et al. [23] (hat can be selectively employed to serve different pluralism purposes depending on the context. Together they offer a practical toolkit for pluralistic alignment.
In bulanced pluralism mode, the system queries every LLM in the support. After sampling the resulting set of candidate outputs, can either present them in an Overton-style slate, useful for open-ended advice, policy deliberation, and brainstorming; or distill them into a single consensus statement, aligning with prior work on consensus building [40, 41].
In steerable pluralism mode, the system, on per-prompt basis, selects one policy т whose persona or metadata best matches declared user preference and produces that policy's 's output. The result is single voice that fits personal assistants, brand-specific copy, or group-specific safety policies, without the overhead of user-specific fine-tuning or elaborate prompt engineering.
In distributional pluralism mode, the system samples a policy from the mixture in proportion to its weight and returns that policy's output. Applied over many requests, this procedure maintains population-level diversity and counteracts the tendency of aligned models to exhibit low output diver- sity by recycling small set of high-reward responses [19-21]. This mode especially valuable in creative workflows-text-to- -eeit-to- generation, story creation, and recommendation ongines-wheee safety constraints must coexist with a rich variety of outputs.
Small support. We deliberately restrict the ensemble to a small support. A compact ensemble is easier to train, tune, and deploy, and it generalizes better (see Theorem 4). Furthermore, the inference methods discussed require storing and efficiently querying the full ensemble, constraining k to stay relatively small. Finally, Section 3.1 shows that only ((1/e) reward functions suffice for e-pairwise calibration, so increasing yields diminishing gains.
Beyond these practical reasons, we intentionally target a small support for normative considerations. Limiting the ensemble size mitigates societal risks inherent in extreme personalization as rein- forcing existing biases, promoting echo chambers, and exacerbating polarization (see personalization within bounds [13]). Taken together, Theorems 2 and 3 show that limited limited-support distribution over rewards preserves meaningful diversity without over-tailored personalization.
Outlier reward functions. While pairwise calibration ensures the ensemble whole faithfully reflects aggregate human preference frequencies, we are also interested in the behavior of the individual reward functions ræ, that constitute the ensemble. A well-calibrated ensemble could. in principle, still contain individual reward functions that represent extreme or undesirable viewpoints.
To quantify how much a single reward function deviates from the overall population preferences, we define its disagreement score as:
Pr (0)) Px.
[1[r(x, vn) > ro( v2)] 7 1 [yn > 1/2 m]]]
D...D.B1-ppe 1-N
The disagreement score measures how frequently the reward function disagrees with the population. More specifically, it is the probability that for a randomly selected annotator N and context response triple (z, V/, U2), Te prefers one response while the other.
Of course, on populations with high amounts of diversity, it may be impossible for any reward have particularly low disagreement score. Given this, we say reward function To is a (3, 7) -outlier if (0)) > B inforee ((8') + 7. which normalizes disagreement relative to the besi-aciee score.
3 Theoretical Guarantees
Having defined pairwise calibration, we now address three fundamental questions: (i) Can approxi- mately pairwise- calibraedd ensembles be supported on a relatively small set of reward functions?

---

(ii) Can such calibration be achieved without using extreme outlier rewards? (iii) Do ensembles calibrated on a finite dataset generalize to unseen populations? We answer all three in the affirmative.
3.1 Pairwise-Calibrate Ensembles with Small Support
A uniform mixture over the annotators' true reward functions would, in principle, achieve perfect pairwise calibration. However, this approach is impractical for two reasons. First, the true reward functions are never observed-only partial pairwise information provided. Second, this ensemble's support size would need to scale with the number of annotators, making the approach computationally infeasible.
For now, set aside the second issue and focus the first. We show that finding a perfectly calibrated ensemble is computationally hard even in an extremely simplified setting: there only a single context, (i.e., |X| 1), possible responses (i.e., | m)) and have access the true fractions of annotators who prefer y1 to 3/ for all pairs, (i.e., all the values p" (æ, V1 U2)).
Under these conditions, the set of pairwise comparisons wish to calibrate to is finite. A straight- forward application of Carathéodory's theorem guarantees that there exists a perfectly calibrated ensemble of size only (mm²) (see Appendix C.I). Yet, even under these strong simplifying assump- tions-to and a guarantee that only a relatively small ensemble is required- it is computationally hard to find one.
Theorem 1. P + NP, then there does not exist polynomial-time time algorithm for finding perfectly pairwise calibrated ensemble when |X| 1, | m for some finite m and given access to p0 (z, Vn U2) for all (I, Un, 3/2).
This setup analogous to one social choice theory [42]; see Appendix C.2 for details. At a high level, the proof (Appendix D.1) shows that if we could efficiently find perfectly calibrated ensembles, we would be able to determine whether certain values of p* are realizable by any underlying distribution over reward functions. This provides membership oracle to the set of realizable p", which turn out to form a useful convex set known as the linear ordering polytope. Membership oracles allow optimize linear functions over the polytope, thereby enabling us, if we are careful with various approximations, to solve classic NP-hard problems such as Minimum Feedback Are Set.
Next, we address the second consideration. We show that small-support ensembles can indeed achieve (approximate) pairwise calibration.
Theorem 2. For any e > 0, there exists O(s=t)-nnemblee that is s-puirvise-calibrrtee.
This is shown (in Appendix D.2) using the probabilistic method: by sampling O(1/e) reward functions according to a particular strategy, the expected pairwise calibration error is at most guaranteeing the existence of at least one ensemble meeting the criterion.
3.2 Pruning and Outlier Control
Matching aggregate preferences not enough if some reward functions deviate so sharply from the population that they could endorse behaviors most annotators would strongly reject. The next question is whether achieving calibration forces us to include such extreme outliers. Using the previously defined disagreement score (0) our proxy for how far a reward strays from mainstream judgments, our next result demonstrates it is possible to remove these extreme outliers without sacrificing too much pairwise calibration.
Theorem 3. Suppose there exists k-ensemble that is E-pairwise calibrated. Then, for all B 2 2, the total weight of rewardfunctte functions in r that are (3, (B+1).
is at most - Furthermore,
there exists another E-ensemble (for S k) r' that is (Ve -parrwisss calibrated and does not contain any (B, (3 + 1). -oo-ouuliers, and this can be computed with access only to r.
This proof (Appendix D.3) begins with Markov-style argument: most reward functions in cannot deviate substantially from the ensembles aggregate predictions (f*). Furthermore, we can bound the
Formally, the algorithm takes as input the rational values p" (I,, yı, B2), encoded as pairs of binary integers. and must run in time polynomial in the size of this instance.
5

---

disagreement score of individual reward functions based on how much they deviate from r and the initial pairwise calibration of r. Functions exceeding this bound are identified as outliers. We then demonstrate that removing these outliers- process requiring comparison only against j (not the true p- values)-har minimal impact on the overall pairwise calibration.
3.3 Generalization
In practice, we do not have direct access to the true preference proportions p". Rather, we typically work with a finite dataset of pairwise comparisons. We assume the dataset has the following form:
D = {(x(1) Vi 1/2
(6) (9)
) 1 where each entry is generated by sampling a context z ~ Dx. two
candidate responses y1, y/2 Tu(- z)), and n annotators i1................ .......pp N. The empirical preference then set to y 2==1 [yv Fi 1/2 wr]. the fraction of annotators who prefer yı over y/-
Let Rg be the set of all k-ensembles. Our goal is to, using the dataset, find an ensemble r E Rg that achieves small pairwise calibration error the population:
(rr)
Eg Dx -0...
[[p"(as (x, 91. v2) p" y1 92))
However, we only have access to the dataset, and the corresponding empirical loss:
Z(r) B(s, [(0" (z, 91, 122 - -)
Using E to estimate L presents an inherent challenge: Z biased estimator of C.
Lemma 1. For any ensemble r E RR. Ep (D]](rr) f(r + C, where
;= B==EDDDx.. Dx..i. v1. (ol.mo pin(nn, (x. [(p (0--p"<x,,) (2))
This is proved in Appendix D.4. The term C represents an irreducible error introduced due to only n annotators responding per data point. Even perfectly pairwise calibrated ensemble will incur error C' in expectation over 'D.
Despite this limitation, learning-theee theorretcc tools allow us to provably estimate & up the constant model with optimal pairwise calibration.
shift C. This implies that minimizing the empirical loss C remains sound strategy for obtaining
To formalize this, let F = {1r, V1, U2) - [re v1) 2 ra U2)) 0 }} be the class of binary comparison functions induced by the reward models, i.c., these are functions parameterized by θ, mapping triples (I, 21, y2) to {0, 1}. outputting 1 exactly when ra(æ, v1) Z re(x, Defining
f'(r) = C(r C. we have:
Theorem 4. Suppose F has finite VC-dimension d. Then, for any 0, with probability at least 1-6 8 over dataset 'D of N i..dd. samples,
2d log( cN dr
log 1 2N
['(r) - 2(r) s
sup r
N
where d' 20(d + 1)k log(2(d 1)A: E (kddkkk))))
The proof (Appendix D.5) applies uniform convergence bounds for agnostic PAC learning. However, to do so requires bounding the pseudo dimension of the loss class {((x, V1, (2), + (f" Vı, V2) p)² E R.). To bound this, apply sequence of transformations F, bringing it closer the loss class, and show each step individually does not substantially increase the pseudo-dimension.
As concrete example, suppose we have embedding function s X x y - Ee which maps context-response pairs into R. and that the reward functions {re E }} are linear functions over these embeddings: ro(a, y) 0 (fxx, y) where θ R.C This setup corresponds to the common practice of learning a reward model by removing the final layer of a pretrained language model (yielding an embedding function), attaching a new linear head that maps the embedding to sealar reward, and training only this final layer on preference data. Here, the comparison functions become 1 [ (x(x) 2 0- (py))] 1 (p((x) - p(v)) 2 0] which correspond linear classifiers over the embedding space. This is known to have VC-dimension most é [43].

---

Implementation via Residual Reward Calibration
Given that only small mixture is theoretically sufficient, the practical question is: can we construct in practice? In principle, we could learn an ensemble by solving the full optimization problem:
v2) min B( [p(x yı, 22) - vI 9s
However, directly minimizing this objective requires computationally intensive search over both the mixture weights and the reward model parameters, rendering it impractical.
Instead, we propose heuristic approach based forward stagewise additive modeling (FSAM) (see Hastie et al. [44] for an overview) that decomposes the problem into sequence of more tractable subproblems. At each step, we fit new reward model to the current residual error of the ensemble, keeping previously learned models fixed; pick a mixing weight that best reduces that error, and append the new model to the mixture. We observe that, despite its heuristic nature, this approach tractably recovers ensembles that are approximately pairwise-calibratee on real-world datasets.
We start by detailing how we train the first reward model (i.c., k = 1). Even in this case, Equation 2 already departs from vanilla RLHF in two ways. First, the target is the observed preference fractions, which we refer to as the soft labels. Even in cases where multiple annotators disagreed on which response is better, typical alignment datasets flatten these to binary labels p(r, y1. y2) E {0, 13 by collapsing multiple annotators' votes into a single "majority-decision label [45-51]. In fact, several high-profil alignment deployments train exclusively on these binary signals and treat them as a gold standard, discarding the underlying soft-label information as noise [52-54].* 4 By contrast, we need access to this soft label data, as we aim to match the observed preference fractions. This training regime is conceptually distinct from methods that merely smooth hard such as label smoothing [55, 56]-or or from approaches that try to infer a latent consensus from soft labels [57, 58]. Second, we present our method in terms of the MSE loss, while typical RLHF uses cross-entropy. MSE aligns cleanly with later ensemble iterations, though the same construction could carried out with cross-entropy loss well.
Having defined the k 1 special case, we now turn to the rest of the ensemble. At iteration j > 1 we expand the current ensemble r5-1 (rep..... ".Pp,,",,T, To, ii (3-1) by adding a new reward model ra,- Let
(x V2, = p(x, U1, U2) - 8 - (x,, U2)
be the residual calibration error. We fit ro, by minimizing
B(I.e1.s/>D (E,(2, y1, - a (10,(x, - ro, =)
using the sigmoid (2) 11(11 exp(-2)) smooth proxy for the binary vote indicator.
Each reward model starts from the same supervised fine-tuned (SFT) checkpoint: we drop the token- prediction head, attach freshly initialized single-node reward head, keep all other parameters frozen from SFT, and then fine-tune on preference pairs. 5 Once ra, trained, we add the ensemble, and reoptimize coefficients (ai...., az)) minimize the training MSE, given r8.....,79..
The method incrementally concentrates capacity on the "difficult" comparisons those with the largest residual magnitude. Residuals may be negative or exceed 1, applying even more pressure to the subsequent reward model to converge toward the soft labels.
The procedure can terminate after a fixed l: iterations or, alternatively, via early stopping when the validation loss stops improving. Each additional iteration adds one fine-tuning pass, so runtime grows linearly with the number of reward models.
7

---

Empirical Results
We now evaluate the FSAM procedure test whether this heuristic can learn pairwise-calibrated ensembles on real alignment datasets. We focus on two questions: (i) Can small ensemble of weak reward models match the observed vote fractions more accurately than any single reward model? (ii) Do the individual reward models in the ensemble capture distinct preference patterns rather than duplicating one another? For both, we find positive results.
Datasets. To be suitable for our experiments, datasets must satisfy two conditions: (i) they must include-or allow us to reconstruct -aniwti comparisons between distinct LLM outputs; and (ii) every comparison must be rated by more than one annotator. We use four public datasets that satisfy these requirements and exhibit annotator disagreement: MultiPref [59], PersonalLLM [60], HelpSteer2 [61], and Reddit DL; [62]. Summary statistics appear in Table 1; further details about the datasets, including pre-prove pre-processing, can be found in Appendix These datasets vary substantially in both domain and size.
Name
Pref. Pairs
Unique Prompts
Annotation
Avg # Annots.
Avg p
MultiPref [59]
9,413
4,791 /532
Human annotators
4.0
63%
PersonalLLM [60]
263,256
9,402 1,000
Model-based scores
76%
HelpSteer2 [61]
21,000
10,000 / 1,000
Human annotators
3.5
74%
Reddit TL;DR [62]
3,217
729
Human annotators
7.56
84%
Table I: Summary statisties for the four preference datasets (see Appendix F.2 for details). is defined here as the mean majority-agreemeet score, i.c., the average share of annotators who chose the majority option; by definition this share is at least 0.5.
Ensemble of weak reward models vs. an optimal single reward model. We an ensemble of k=8 reward models each dataset, starting from supervised fine-tuned checkpoints of Meta-Llama- 3-8B [63] (a juggernaut in the small-model bracket). We compare the ensemble to a theoretically optimal single deterministic reward model that always selects the muujorty-preefent answer for every comparison. Such a model minimizes binary error and its mean mean-squaree squared calibration loss canmot fall below E; (in(p? (1 p.) A mixture of weak deterministic reward models offers finer-grained set of outputs but, a priori, need not perform better.
Our results (Figure 2) show that, in many cases, an ensemble of only 2-4 such rewards already achieves noticeably better calibration on held-out prompts. Gains tend to taper off around six models.
Throouhhou this section, we use p(r, yh, yz) to denote the observed fraction of annotators that responded
31 > y/
"This choice is surprising because the BT model, soft-label BCE matches the true log-likelle likelkkelhood, and the vote fraction reveals how small the underlying score gap is between the two responses (see Appendix E). 'Training details appear in Appendix F.I.
MultiPref
PersonaiLLM
HelpSteer2
Reddit TL;DR
cm www.heellduuooooooo prompts
0.20
0.10-
0.20
0.12
0.15
Srylc Romer Lin:
0.10-
0.08-
0.10
0.06-
0.05
0.03
0.06-
0.03-
0.00
0.00
Ensemble Size
Figure 2: MSE calibration error on held-out prompts as function of ensemble size k. Each curve obtained by training a mixture of eight weak reward models with our FSAM procedure, which greedily adds one base model at a time to minimize the residual calibration error. The shaded band marks the floor for any single deterministic reward model, Σ, min(p?. (1-p,)2. Across all datasets, ensembles of only two to four rewards already beat this single-model bound.
8

---

MultiPrel
PersonalL LM
HelpSteer2
Reddit TL,DR
0.15
1200
000
60
0.00
0.09
4000
072
Figure 3: Pairwise - correlation scores between reward models in each ensemble, obtained by ranking 100 high-temperature continuations for 50 prompts. A score of corresponds random rankings, while score corresponds to identical rankings; lower scores thus indicate greater diversity. The REDDIT TL;DR dataset- the smallest and focused summarization records the highest scores.
Thus, combining a handful of weaker reward models delivers a calibration accuracy that no single deterministic model can match.
Note that performance reported as mean-squared calibration error. Other empirical pluralistic alignment experiments frequently report accuracy, but this is not meaningful here because the task is not majority classification and annotator identities are unavailable (see Appendix E).
Do the reward models capture distinct preferences? To test diversity the policy level, we use Best-of-N (BoN) sampling as proxy for fine-tuning [64, 65]. Concretely, we select 50 prompts that the ensemble was not trained on, generate 100 diverse continuations from the frozen SFT model using high-temperature sampling, and score each response with every reward model in the ensemble. We quantify distinctness by the normalized Kendall-7 rank correlation coefficient [66], which counts the number of pairwise disagreements between two ranking lists, averaged over all prompts. Prompts for the first three datasets were taken from the PRISM dataset [8] (covering value-centric and controversial topics); prompts for the Reddit dataset were taken from its validation set.
The reward models within each ensemble are distinct and qualitatively different (Figure 3). This diver- sity is further illustrated by comparing the top-ranked response chosen by each model (Appendix G).
Discussion
Our approach raises several important considerations regarding its scope, alternatives, and limitations.
First, for highly contested sensitive issues, pairwise calibration may not always be appropriate. We are not aiming for the model to express its own opinion or mirror the population; rather, a neutral answer-or, when necessary, outright refusal-is often the safer choice. However, while refusal policy a simple fallback for disallowed or highly sensitive content, over-reliance can hamper the system's usefulness when thoughtful, context-aware answers needed. Even ostensibly neutral responses may embed hidden framing biases, so remains essential to understand and address those biases and let the model meaningfully adapt different user contexts, cultures, and values whenever such adaptation is appropriate.
Second, one might ask whether a single, unified model could achieve similar pluralistic alignment via alternate techniques. Indeed, one possible approach is to directly instruct the model within its context to adopt particular perspective-Knott
in-context steering persona modulation [27, 28, 67, 68]. However, recent empirical evidence suggests that language models instructed to emulate specific viewpoints frequently produce inconsistent or overly stereotyped responses, and often fail to capture nuanced differences between distinct user groups [69]. By contrast, our approach explicitly incorporates architectural support for pluralism by training multiple distinct reward functions.
Finally, while pairwise calibration ensures that the ensemble accurately reflects the preferences of annotators for pairs of responses given a context, it does not guarantee accurate representation of higher-order judgments over larger sets of responses. Ideally, for any reasonably sized set of responses, the proportion of reward functions ranking given response highest would (approximately)

---

match this preference in the population. However, recovering such higher-order preference structures information-theorticalll theoretically infeasible using pairwise data alone. Capturing these relationships would require richer elicitation, such as rankings or best-of-é judgments over larger response sets.
information
10

---

References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 27730-277444 2022.
[2] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[3] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Al: Harmlessness from AI feedback. arXiv:2212.08073. 2022.
[4] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, al. Ethical and social risks of harm from language models. arXiv:2112.04355 2021.
[5] Jiahao Yuan, Zixiang Di, Shangzixin Zhao, and Usman Naseem. Cultural palette: Pluralising culture alignment via multi-agent palette. arXiv:2412.11167 2024.
[6] Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. Cultural incongruencies in artificial intelligence. arXiv:2211 1.10069, 2022.
[7] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv:2404.09932, 2024.
[8] Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gonee Juan Ciro, Max Bartolo, Adina Williams, He, et al. The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. In Proceedings of the 37th Annual Conference Neural Information Processing Systems (NeurIPS), pages 105236-105344, 2024.
[9] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdì Wang. MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv:2402.0892. 2024.
[10] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedhack. arXiv:2307.15217 2023.
[11] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th In- ternational Conference on Machine Learning (ICML), pages 29971-30000 2023.
[12] Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations, arXiv:2212.09251, 2022.
[13] Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. S31v:2303.00553, 2023.
[14] Stewart Slocum, Asher Parker-Sartori, and Dylan Hadfield-Menell Diverse preference learning for capabilities and alignment. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. Forthcoming.
[15] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. EXXiv:1707006344 2017.

---

[16] Jiancong Xiao, Ziniu Li, Xingyu Xie. Emily Getzen, Cong Fang, Qi Long, and Weijie J. Su. On the algorithmic bias of aligning Large language models with RLHF: Preference collapse and matching regularization. arXiv:2405. 01022405..1.5.5... 2024.
[17] Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bastani. Evaluating the diversity and quality of LLM generated content. 1rXiv:2504.12522 2025.
[18] Thom Lake, Eunsol Choi, and Greg Durrett. From distributional to Overton pluralism: Investi- gating large language model alignment. arXiv:2406. 17692, 2024.
[19] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. urXiv:2310.06455 2023.
[20] Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. distributional approach to con- trolled text generation. In Proceedings of the 9th International Conference on Learning Representations (ICI.R), 2021.
[21] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv:2202.03286, 2022.
[22] Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-LLM collaboration. arXiv:2406. 15951, 2024.
[23] Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. roadmap to pluralistic alignment. arXiv:2402.05070, 2024.
[24] Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman Ozdaglar. RLHF from heterogeneous feedback via personalization and preference aggregation. 4rXiv:2405.00254, 2024.
[25] Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic align- ment framework for learning from heterogeneous preferences. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025.
[26] Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, and Verena Rieser. Value profiles for encoding human variation. wrXiv:2503.15445 2025.
[27] Eunjeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon. Aligning language models to user opinions. In Proceedings of the 28th Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[28] Louis Castricato, Nathan Lile, Rafael Rafailov, Jan- ah-PPhlipp Fränken, and Chelsea Finn. Persona:
A reproducible testbed for pluralistic alignment. arXiv:2407.17388 2024.
[29] Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. Position: Social choice should guide AI alignment in dealing with diverse human feedback. In Proceedings of the 41st International Conference on Machine Learning (ICMI.), pages 9346-9360, 2024.
[30] Abhilash Mishra. Ai alignment and social choice: Fundamental limitations and policy implica- tions. arXiv:2310. 16048, 2023.
[31] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean- Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards Pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards. Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
12

---

[32] Luise Ge, Daniel Halpem, Evi Micha, Ariel D Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. Axioms for Al alignment from human feedback. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 80439-80465, 2024.
[33] Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, and Wenpin Tang. Mallowspo: Fine- tune your L.LM with preference dispersions. arXiv preprint arXiv:24055 14953, 2024.
[34] Ilgee Hong, Zichong Li, Alexander Bukharin, Yixiao Li, Haoming Jiang, Tianbao Yang, and Tuo Zhao. Adaptive preference scaling for reinforcement learning with human feedback, Advances in Neural Information Processing Systems, 37:107249-107265 2024.
[35] Jiashuo Wang. Haozhao Wang. Shichao Sun, and Wenjie Li. Aligning language models with human preferences via Bayesian approach. Advances in Neural Information Processing Systems,36:491 3eeeen33::99111-91111 2023.
(36] Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personal- izing reinforcement learning from human feedback with variational preference learning. arXiv preprint arXiv:2408. 10075, 2024.
[37] Ali Shirali, Arash Vasr-Esfahany, Abdullah Alomar, Parsa Mirtaheri, Rediet Abebe, and Ariel Procaccia. Direct alignment with heterogeneous preferences. arXiv:2502.16320, 2025.
[38] Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in RLHF. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
[39] Ariel D Procaccia, Benjamin Schiffer, and Shirley Zhang. Clone-robust Al alignment. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. Forth- coming.
[40] Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Camphell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 381 761131189, 2022.
[41] Sara Fish, Paul Gölz, David C Parkes, Ariel D Procaccia, Gili Rusak, Itai Shapira, and Manuel Wüthrich. Generative social choice. In Proceedings of the 25th ACM Conference on Economics and Computation (EC), page 985, 2024.
[42] Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel Procaccia. Handbook of Computationaal Social Choice. Cambridge University Press, 2016.
[43] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT Press, 2012.
[44] Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
[45] Hannah Rose Kirk, Andrew M Bean, Bertie Vidgen, Paul Röttger, and Scott A Hale. The past, present and better future of feedback learning in large language models for subjective human preferences and values. arXiv:2310.07629, 2023.
[46] Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. In Pro- ceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 111928-109666 2024.
[47] Kyuyoung Kim, Ah Jeong Seo, Hao Liu, Jinwoo Shin, and Kimin Lee. Margin matching preference optimization: Enhanced model alignment with granular feedback, arXiv:2410.03145. 2024.
13

---

[48] Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback. arXiv:2310.1 135995 2023.
[49] Nathan Lambert. Reinforcement Learning from Human Feedback. Online, 2024. URL https: htttp::/rllhboook com.
[50]
Vinodkumar Prabhakaran, Aida Mostafazadch Davani, and Mark Diaz. On releasing annotator- level labels and information in datasets. urXiv:2110.05699 2021.
[51] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv:2209.078555 2022.
[52] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proceedings of the 33th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 3008-3021, 2020.
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.0928. 2023.
[54] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862, 2022.
[55] Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? In Proceedings of the 32th Annual Conference on Neural Information Processing Systems (NeurIPS), 2019.
[56] Binghai Wang. Rui Zheng. Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets RLHF in large language models part ii: Reward modeling. arXiv preprint arXiv:2401 20000000,0000000 2024.
[57] Jae Hyeon Cho, Minkyung Park, and Byung-Jun Lee. Vpo: Leveraging the number of votes in preference optimization. arXiv:2410.2289) 2024.
[58] Hiroki Furuta, Kuang-Hueis Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, and Izzeddin Gur. Geeeeerreeaverraged preference optimization for soft preference labels. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 57076-57114, 2024.
[59] Lester James V Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Hybrid preferences: Learning to route instances for human vs. AI feedback. 2XXv:2410.191333 2024.
[60] Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. LeMsonalLLM: Tailoring LLMs to individual preferences. arXiv:2409.200 20296. 2024.
[61] Zhilin Wang, Yi Dong. Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. 2eXiv:2406.0867. 2024.
[62] Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS, 2020.
[63] AI@ Meta. Llama model card. https://giie MODEL_ CARD. md, 2024. Accessed 11 May 2025.
[64] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et WebGPT: Browser-assisted question-answerte with human feedback. arXiv:2112.09332 2021.
14

---

[65] Lin Gui, Cristina Gârbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetmess of best-of-n sampling. arXiv:2406.00832, 2024.
[66] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-)):11-93, 1938.
[67] Guangyuan Jiang, Manjic Xu, Song-Chup Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. In Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurlPS), pages 10622-10643, 2023.
[68] Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghi- fari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. Survey of cultural awareness language models: Text and beyond. arXiv:241 2024.
[69] Ariba Khan, Stephen Casper, and Dylan Hadfield-Menell. Randomness, not representation:
The unreliability of evaluating cultural alignment in LLMs. arXiv:2503.08688, 2025.
[70] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. addiantage-weighhed regression: Simple and scalable off -fo-cc reinforcement learning. auXiv:1910.00177,2000 2019.
[71] Constantin Carathéodory. Über den variabilitätssereich der fourier schen konstanten von
positiven harmonischen funktionen. Rendiconti Del Circolo Matematico di Palermo (1884- 1940), 32(1):133-217, 1911.
[72] Richard M Karp. On the computational complexity of combinatorial problems. Networks, 5(1): 45-68, 1975.
[73] Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combina- torial optimization. Springer, 1988.
[74] David C McGarvey. A theorem the construction of voting paradoxes. Econometrica: Journal of the Econometric Society. pages 608-610, 1953.
[75] Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A, 13(1):145-147, 1972.
[76] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge university press, 2014.
[77] Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing Shampoo using Adam. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025. Forthcoming.
[78] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 77730-27744, 2022.
[79] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF. In Proceedings of the 41st International Conference on Machine Learning (ICML) pages 62405-62428, 2024.
[80] Lora Aroyo, Alex Taylor, Mark Diaz, Christopher Homan, Alicia Parrish, Gregory Serapio- Gareía, Vinodkumar Prabhakaran, and Ding Wang. Dices dataset: Diversity in conversational Al evaluation for safety. Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NeurIPS), pages
2023.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: The code uses standard reward modeling training, with all details included in Appendix We'd be happy to share the full code if needed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nipss.. ct/ccc/ppllc cec/public//ae Serzides/CodeSSuie
asssnnoliccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
Justification: Discussed in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Discussed Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpas//////.s///eeeeee/ee wttppppccccpppccecicccceeccceeee csQuidel ines?
Answer: [Yes].
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https / http://p..////oeeeeee/////////// for what should or should not be described.
22

---

A Policy-Level Calibration
Here relate pairwise calibration of reward functions calibration at the policy level and motivate our binary indicator definition.
Recall in Definition that ensemble (reji (y)jEm is perfectly pairwise calibrated when, over Dx, Vr, 922 na(- z)) almost surely
p (z. U1 12) = (1, = Σ ag 1 (2, (2, ra, ra, (c, 92)]
The right-hand side represents the likelihood that random reward function sampled from the mixture prefers y1 > 1/2 x. Intuitively, we would like to say that the same holds over the learned mixture over policies (T);()))E[A)- This definition implicitly assumes that the policy " learned from the j-th reward function TB,, conditioned on choosing between y1 and y/2 for given context z, always prefers the one with the higher reward. That is, if re, (x. V1) > re, (ar y2). then (1 = y1
(v1, 923] x
In practice, however, this assumption does not strictly hold. During the fine-tuning, policies π are typically learned by maximizing regularized reward objective of the form:
(ro v) 3xx (I +)][ =));
where B controls the deviation of the fine-tuned policy from a reference policy Tee-- The optimal policy under this objective, n², is known to take the softmax form [70]:
(y a)) oc Feer x)) exp (
For given triple (x, V1 V2). this implies:
Pr [y = 31 E (11, 32H y-
Teer(y) exp fra(t, ui) xeo x)) exp (2ro(z. 91))
merl ()) exp (tro(z, v2)
-o (rrattn))) ro v2)) + log
Teet (1/1 x))
Trarl 1/2 x)
where a is the sigmoid function.
Let us denote this quantity by po(x, U1, V2). A natural alternative definition to perfect calibration would be substitute [fro(x, 3n) > ro( y2)] with p. i.e., would then use the formula:
p(z, U1, v2)
ag Pe, (I, v1. 92)
which more directly captures the probability that the mixture prefers y/1 over y/2 given x. However, this definition introduces additional complexity- entangles the calibration objective with the reference policies, making both theoretical analysis and optimization more cumbersome.
We therefore abstract away these complications and keep the binary formulation as tractable proxy for the more realistic notion. Notably, as 3 + 0, the softmax behavior becomes increasingly sharp, and Pa, (I, //, 32) + [rs, (z. y1) >> T8, (z, 42)]. reward differences dominate. Thus, our binary definition can be viewed as capturing the low-regularization regime while remaining analytically and computationally manageable.
B Related Work
growing set of proposals advocate for pluralistic adignment-mmoellin multiple perspectives in parallel as to better capture the breadth of human judgments [23, 22, 9].
23

---

Scalar Reward Aggregation Approaches. To mitigate the majority- bias of single reward, recent methods introduce additional loss terms or algorithmic modifications that slow or penalize the policy from over-optimizing for the majority. For instance, Xiao et al. [16] add custom regularizer to reduce preference collapse in PPO updates, while Chen et al. [33] temper the update magnitude whenever annotator feedback shows high dispersion. Beyond such direct regularization, some works increase the reward model's 's capacity to represent preference diversity- for example, by weighting pairwise data based on annotator context uncertainty [34] learning Bayesian distribution over reward function parameters [35]. Despite these refinements, methods reliant on a single scalar reward still risk oversimplifying genuine diversity in user opinions, motivating richer multi-reward solutions.
Clustering and Group-Based Reward Modeling. more direct way to represent divergent preferences is to train multiple reward models, each matched to distinct segment of the human population [9, 24-26]. This can be accomplished by clustering annotators based on demographics or inferred preference patterns and then fitting group-specific reward function. For example, Chakraborty et al. [9] show that optimizing a mixture of cluster-specific rewards via a max-min objective can protect minority viewpoints from being overshadowed by the majority. Although powerful, these methods typically rely on stable, coherent groups, which may not reflect real-world fluidity in user preferences [27], and group membership can sometimes be ambiguous or proprietary. Additionally, overly granular clusters risk sparse data, whereas coarse groupings might obscure meaningful sub-group heterogeneity [28]. By contrast, we avoid the need for explicit clusters or demographic tags. Our approach learns a small ensemble of reward functions- each representing an internally consistent viewpoint- and ensures pairwise calibration that overall, the ensemble reflects the full distribution of observed human judgments.
Personalized and Conntit-Condiiionall Alignment. A further alternative is to directly personalize the alignment objective at the level of individual annotators, assuming each user has a unique reward function [36]. While highly expressive, such methods become cumbersome at scale, and excessive personalization can reinforce biases or amplify social divides [13]. Consequently, many personalization approaches latent embeddings or partial fine-tuning, striking balance between capturing user-specific signals and avoiding an explosion of fully individualized models. In practice, personalization-basee methods still risk "echo chambers," prompting calls for more bounded approach to preference diversity [13]. We share this concern, and thus focus on small ensembles as middle path: each ensemble component aligns with coherent subset of preferences, preserving pluralism while limiting the societal risks of extreme personalization.
C Additional Discussion Surrounding Theorem 1
C.1 Ensemble Size Bound for Perfect Calibration with Finite Support
Let yı...., Fm be the elements of Y. For a reward function 0 e θ, let x" E {0, (33) denote its incidence vector. We index x" by pairs ij with í < j. where x = 1 (ro 2 ro ro())) Observe that, for each E N,, their reward vector corresponds to some incidence vector. Thus, the vector x with Jij = p" (x, yi, yr,y)) y/) must live in the convex hull of incidence vectors; in particular, is the convex combination of incidence vectors placing po on x" where po is the fraction of N that have a reward function rg. Since E RR),, Carathéodory's theorem [71] guarantees that it can be represented by convex combination of only (m) + 1 vertices of the hull. This induces a ((z) + 1) ---ensemble that is perfectly pairwise calibrated.
C.2 Connections with Social Choice Theory
The problem we study can be expressed as social-choice problem when | and [ m. Each response plays the role of a candidate, and each annotator voter who ranks these m candidates. The tournament graph is the complete directed graph on these m nodes, with an edge (a. b) weighted by the fraction of voters who prefer to b. Under this reformulation, finding perfectly pairwise calibrated ensemble is equivalent to finding a distribution over rankings that induces a given tournament graph. While an application of Carathéodory's 's theorem implies that polynomial support size in the number of distinct pairs is sufficient ((T) + in this context), we prove that it is harhard to determine whether such distribution over rankings exists for given
24

---

weighted directed graph. Consequently, even deciding whether a tournament graph is representable as any mixture of linear orders itself NP-hard.
D Deferred Proofs
D.1 Proof of Complexity (Theorem 1)
For convenience, adopt standard terminology of social choice theory [42] in this proof. Specifically, we call elements of y {y1..... }m candidates. Let LO()) denote the set of linear orders (or rankings) over y/ The preferences of the voters (i.e., annotators) are represented by distribution rankings. A distribution assigns a probability pa 2 0 to each E LO()") such that Eeelory) Pa =
Each ranking can be represented by its incidence vector x" {0, (0)))((), For each pair (ye, Nz3 with j.
2i = [ if is ranked before in a;
if yj is ranked before y4 in
(weighted) tournament graph corresponding to a distribution over rankings is the weighted average of these incidence vectors t = E Ergoory) Pax". Each coordinate fay represents the fraction of voters that rank y; before yj. The set of all realizable tournament graphs is the convex hull of the incidence vectors:
TG conv {x" is a linear ordering)." 6
( is valid tournament graph (i.e., y TG) is
We show that deciding if given vector y NP-hard. This implies that the potentially harder problem of finding a distribution over rankings consistent with given E TG is also NP-hard.
To establish this, we reduce from the NP-hard problem Minimum Feedback Are Set (MFAS) [72]. Given a directed graph G = (V, E) on m vertices (which we identify with Y) and integer k, MFAS asks if there is an ordering of the vertices a such that at most edges in E are feedback ares (i.e., edge (y, 3²) such that y appears before y in σ). The number of feedback ares for ordering a is given by the linear function
NFAS(*")
Σ
Σ (1-2%)
<< (MGD))E
Furthermore, since NEAS is linear function, its minimum on the convex hull is achieved at vertex. Hence, min NEAS( (x") minxeTG N
Suppose we have a polynomial-time algorithm (an oracle) that decides whether a given (rational) vector y is in TG (in particular, this algorithm should run in time polynomial in the binary encoding of y). We call this the Tournament-Oruce Problem. We show this allows for a polynomial-time algorithm for MFAS using the following theorem:
Theorem (Grötschel et al. [73], Corollary 4.3.12 (rephrased)). Let C Rd be a convex set, such Bxx,, R),
that there exists a point Xo Rd,, and numbers O
R such that B(x,, C K where B(xn,r) r) is the ball of radius r centered at Xg- Suppose we have a membership oracle O for K. Then, for every e E gd, there exists an algorithm that outputs a point z E IRd such that
c z S min T x +
<mmaxmmxx T XEK
x-ii c xx)
XEK
XEK
The algorithm runs in time polynomial in (1g()/)), log( (R/r), the bit-size of c, and d. The membership oracle is queried with rational points of polynomially bounded encoding sizes.
We apply Theorem 5 with K = TG, hence d = (m) = =0(m).. Let X E Rd be the point with all coordinates 1/2. Since Xy can be induced by the average of all mn! possible rankings x", it is in TG.
The squared L2 distance from Xo to any vertex x" E {0, 1]d
d
I|x
x
x
2
4'
Am
This is often called the linear ordering polytope.
25

---

implying that R v Vd/2= (m)). McGarvey's theorem [74] shows that there exist tournaments with 0 or 1 in single coordinate and 1/2 everywhere else. Convex combinations of McGarvey's tournament graphs and Xn can generate any toumament such that every coordinate is within 1/(T) of 1/2. Hence, r (T) 2 1/m is sufficient. Thus, log (R/r) = (log m).
For MFAS, we want to minimize NEAS. The objective function's 's linear part is defined by CFAS, whose components are in {-1, O, }), and therefore O(d) O bit complexity. The minimum value of NEAS. as it must occur at vertex in {0, 1) is an integer. Furthermore, 0 < NFAS(x) S (2) for all X E TG.
Applying the theorem with < 1/(2- (w)) and rounding the solution gives a polynomial time
algorithm for finding the optimal value of MFAS, which, assuming + NP, is contradiction. C
D.2 Proof of Approximation Guarantee (Theorem 2)
Fix & > For each E N, let re, be the reward function that induces i's preferences (i.e., 91 y2 x + To, (2, v1) ræ, (z, v2)). Let k 1. This choice implies k 2 te s0 S #.-
Consider randomly constructing a k-ensemble r as follows: Randomly select annotators 11 .1...., ~ A uniformly with replacement. The ensemble is then:
(ro,, ...., ro., 1/k,................... 1/k).
Let R denote this distribution over ensembles.
For each triple (Ir, 91) consider
EPER (15 (I. Un.82) (a. un v2))
The estimator 1² (xr (1,3113)) is obtained by drawing
T ~ pi, (2.
and setting 5" (5, v2) T l/k. Therefore
Var(T)
p² p' 91, U2) p² (xx, V1 822) S
EywR (5 (z, (I. V1. y2) (x, u1, 82))
222
k
4k'
Next, the expected calibration error of a randomly chosen ensemble r satisfies
E
E - Dw
(2.
U1, - p= v1. "
(p" yI, 92) (x,,,,,,22))
RoR
1
4k
4k
Since the expected calibration error of a randomly chosen is at most 1 this implies that the same bound holds for at least one deterministic r, needed.
D.3 Proof of Outlier Bound (Theorem 3)
Let =
(ro,: az)}= be k--ensemlle that e-pairwise calibrated. Fix 2 and y (B+1)VE. Recall the disagreement score for a reward function ro:
(0))
Pr x- D..D mo N~N
[1[re(z, v1) > (e(x,22) A 1/21 > D/2 []]
Let Puin inf, (0)). Then, ro is a (8, outllir if ((() > B T Diin 7
Define the random variables
v 1(ro(z, v1) > r0(2,
Pe == ==p (z.V1.
p= v1.
26

---

The randomness for these variables is over the draw of triple (z, :2) according to the underlying distribution (e.g., x Dx 91. U2 Tol. z)). For any two random variables X and Y (over triples). let
V2)
d(x, Y) = Ex in mn -I].
First, we show that (0)) d(vo, P*). To this end, we can rewrite the disagreement score as
Pr
(ro vn) > r0( # 1 [111 >= a:]
Dx V/ (112) tuu
Bew Dx wi
Pr (ro((, vn) v2)) o 1 [V1 > x]]]
Furthermore, for fixed triple (I. (1). the inner term is:
Pr (ro((r0 re y2)1 # 1 > 122 x]]
Vo. (1- Pr.vv > 1/2 æt] + (1 Vo) Prim > y2 [2] i-,
Vo. (1-P1) + (1- P P
Since V is binary and px E [0, 1]. this simplifies to I - PT). Thus, the disagreement score is indeed (()) = d(V", P*).
Define (0) = d(vo, P) and e min = infs (0) as an analogous "disagreement score" with respect to P. Next, we claim that if 0 is (8,7)- -outlier, then d(V P) > B @min- By the triangle inequality,
Id(V", P*) d(v", P) < d(P*, P).
The ensemble r is e-pairwise calibrated, meaning E[(P - p0) s e. By Jensen's 's inequality, E| P 12 s P < a.
Hence, d(P*, P) S Ve. So, for all 0, we have
Id(vo, P) - a(vo, PT] S Va.
This also implies that 2 minnl S VE.
Combining these observations, we see that if is ooutler, then d(V > 3..00 $ min + (3 + 1)- VE, which implies that d(v",, .P) > 3 min-
Next, we claim that for fixed 0 E O,
d(vo P) = Egr (d(Vo, vo),,
where 8' r denotes drawing from the ensemble, i.e., O2 with probability aj. Expanding the definitions:
Exw (-1x) [ - -1 Egr-r (e DI.
VII
Swapping the order of the expectations:
-P1 = Des
For fixed triple (x, y1, 32). the equality [V* v1, 32) - P(x, 1/1 (1.)2)) y2)1 Egr [IV (z, y1, v2) v (z, y1, 12)] holds, because v (Iz, y1, 31,32) is deterministic (in (0, 1). and yo is Bernoulli random variable whose expectation is $(æ, 31, 1/2).
By Markov's inequality, for any we have that
Pr Id(vo' Vo) 2 (8 - 1) d(vo, [P0 P)]
Por
Pr d(V Vo) 2 (B 1) Eerre ld(V"' v'']] s $ 8'
27

---

By the triangle inequality, d(Ve P) I d(vo, V + d(V". P). Therefore, if d(v P) 2 B
d(vo, P), that implies that d(vo P) 2 (B 1) d(V", vO) This gives that
Pr ld(vo) P) 2 3 d(Vo, P)] - B 1
@'r
Taking the infimum over choices of yields (e.g., using such that d(vo, PP) - & mmin)
Pr ld(vo) P) B @ mial S B-1
p'
Finally, as (B, -ouutlers must satisfy d(Vo' P) > B. Emin this immediately implies
Pr 00 is ( I 3-1
0'
yielding the first part of the theorem.
Now, construct r' as follows: Order the .I...., On from in non-decreasing order by d(0, P) as 01y....,04, Remove 0g if they are in the top 3-7 of total weight according to this ordering (i.e., j satisfies that Σソ= ay 1 )) Then, renormalize the remaining ajs to sum to 1. The previously derived bound shows that at most マ of the probability mass on satisfying d(0, P) > 3 4 Removing those with largest d(-, P) values ensures all such 0 are removed. By the above arguments, this also implies that r' contains no (3, -outlierss. Furthermore, this computation depended only on r (i.e., we did not need access to true proportions p²).
remains show that r' is (VE - --2r)-iirissss calibrated. To this end, let fs be the analogous random variable to P for r', i..., it takes on values g² (z, 11. v2). We claim that I P surely.
Let Siem be the set of indices of removed By. Let Orem = EJES_ ag. We know asem I 1 Let
5-1-
Oisot Ejeshnn Oj Orem- For fixed triple (xr, U1, 922, let Vj /0: (Ir, 1/, 21, E {0, 1}. Then
P(xx, 01, = E Smm a + Ejes ajV And Pr(xx, v1)
Eeesme VgVy
Consider the difference:
p' p= Σ Ejgsmn VjVy
Σ
ajVy +
agy,
Otto
jK
jES
1
Σ
agVY
azV₁
O'tot
S&
jE
1 - O'int
agVz
Asot
j
jES
Orem
Σ VvVY.
O'jo
jg
S
Let A = Eje VjV, and B = Σ jm
ag Vr-
We know < A S Ejeson Oj = Ojots and
5 B < Ejes,,. Og- = Orem So, D'
Orm
B.
Now, 0 A Ogat and < B < arems so this difference is bounded in 1-Crem- Greml and thus bounded in -- - 7-1.
Finally, we show the pairwise calibration bound. We have just shown that p Po < B-1 1 which implies that
E Dx wt pr72 < 3 3-1
Furthermore, e-pairwise calibration implies that
أ}} S Ve.
28

---

By the triangle inequality for L2 norms (Minkowski inequality),
E Do P & VE
3-1
This implies that r' is (Ve + ) -pairwise calibrated, as needed.
D.4 Proof of Biased Loss (Lemmaa 1)
First, note that by linearity expectation.
Ep[[(r)] = Ex w wo) (x, 11, v1, p)²
Fix a triple (z, 1/ 32). For brevity, will write p" and P" instead of p" (z, 81 v2) and pr (x. Uı, 92). respectively. The inner term then becomes
Ep p/ 1/ p)²}-
Furthermore, observe that Bin(n, p" 1/n [p] p², Hence,
Ep--in(n.p')/vll - } = EpnBinnn. )/n 11 - p² + - -p)
= Bi--Bin(n.p"//// )/rmn/n.p)//1((00 - pp²)² - -2(p" pp' -p') (p- -p) (p' p)
(6" -p")² 10--221 2(/ (0) -p") + E Bin 3//o('' p)
=(p" (0"--'") + -p-BBiin..n")) (on-Bi((xxp///lll0 "/1l'''""'>>''''>"""'//'''''''' p)
Taking expectation over x ~ Dx.91.20 ~ To(: yields the lemma statement.
D.5 Proof of Generalization Bound (Theorem 4)
We invoke the following uniform-convenn uniform-convergencee result:
Theorem 6 (Mohri et al. [43], Theorem 11.8 (rephrased)). Let H be family of real-valued functions funccions and
{(.. v) L(h()), y) h H)
the family of loss functions associated to H. Assume that Pdim(g) = d* and that the loss function is non-negative and bounded by M. Let D be distribution over (I, v) and let D be sample of size N. Then, for any ō > O, with probability at least 8 over the choice of samples.
2d log( EN
log 67 2N
sup KEH
v)] - ((M())))(((()))))) S M
MV
N
For our purposes. H is the set of functions {p" r Rx} where each 6r maps X' x y x y [0. 1]. The loss function C is the squared error. The domain points are triples (x, Vı. y2). drawn from the usual x ~ Dx V1. To(- x). The target the label is the empirical proportion
p (nn (2))
Since p E [0, 1], C bounded by M =
Furthermore,
((r) E (C((x)))))
and by Lemma 1,
C'(r) E (C(((x)) y)]-
therefore suffices show that Pdim(G) < 20(d 1)k log + 1)k), where
G = {((x, 9ı, 92). + (p (a: Uı U2) - p)² R}.
The theorem statement in Mohri et al. [43] states one-sided error, but the proof implies two-sided one.

---

For notational convenience, inputs to function classes (e.g., (z, 31, U2) or ((a, y1, (1. 82). may be generically denoted by 2,
Recall the following standard definitions and results from learning theory. Fix a function class F' and set of m points, { For a binary vector b E (0, 1)mh called sign pattern, F realizes b if there exists f E F such that 1 (z.) 2 0] = b, for all i. We say J' shatters the set if all n sign patterns are realized. The VC-dimension of F is the size of the largest set it shatters. If J is real-valued, its pseudo-dimension is the VC dimension of {(2, 1) - f(z) - f E F. If the latter function class shatters set, we say that F'' pseudo aseudo-shatters shatters that set. We write Pdim((")) for the pseudo-dimen dimension of F". Finally, Sauer's s Lemma [75] states that a class of VC-dimension d' induces at most (em/d")d sign patterns on m points.
We upper-bound (g) by analyzing a sequence of function classes F2, Fs and bounding their pseudo-dimensions d d2, dz:
1. Linear combinations: Let Fi = 1Ej=1aj fz(2) fo E F.ay E R, the set of k-sized linear combinations of functions in F.
Fix points (=1, (2)...,... (Zms mm) First, consider the maximum number of sign patterns of 21....: 2m which can be realized by (the binary-valued) F. Since F has VC-dimension d, by Sauer's Lemma, this is at most (e. m/dd m/d)". Thus, if we are picking k such functions from F, this can induce (em/dy/d s (em)kd combinations of sign patterns.
Fixing a choice of patterns VI...., Vk E 10 We now consider how many sign patterns of {(z,, 4 can be induced by various choices of a. Note that a certain a will induce sign pattern with b;- 1/x- (Via ......... Uka)
2 0). Consider the set of k-dimensional linear functions mapping R + R. {z ~ a- x E R. Note that this class has pseudo-dimension k [43]. This immediately
implies, by Sauer's ' Lemma and the definition of pseudo-dimension, that the number of sign patterns inducible on {[2;, 44) is at most (em/k))* S (em)*.
Together, these imply that the total number of sign patterns realizable by Fi is at most (em) Thus, if Fi can pseudo-shatter this set, all sign patterns must be realizable, so 2 < (em) Equivalently, log(2)m & (d + 1)K log(m), implying
(d + 1)k: log(2)
log(m) $ 2 (d + 1) k- log
m
By Lemma A.I of Shalev-Shwartz and Ben-David [76], m < 4(d 1)k log(2 1)k). Hence,
di S 4(d + 1)k log(2(d + 1))).
2. A Affine shifts: Let = ((=.V) f(2)-
If F2 pseudo-shatters
(121, ((zms Bmm) tm)s
then Fi pseudo-shatters
(2 t1 vı), (Zm, Im 3mm)
implying that the pseudo-dimension of F2 is at most that of Fi. Hence, d2 S d1-
3. Squaring: Let Fs Fs= f()2 ((>) 1f
Fix m points S= (2),tt)).... (2m, (m)] pseudo-shattered by F3.. Assume each t; > 0, as otherwise that point alone cannot be shattered. Furthermore assume that f(;;) o la for any f E F3. otherwise we can adjust I, such that the set is still pseudo-shattered and equality does not hold.
Now, consider the sign patterns of function f E F2 on the 2m points
S' ((21,
V M, (21, Vtt)..... (Zm V (t))
Observe that if two functions f. E F2 have that f(2)² and 22) differ in sign pattern on S, then f(=) and [z( differ sign pattern on S', as if [(x))2 I, and f'(x,) )2 > 4, then that implies - -V4 < [(x) < VT, and either f'(x;; > Vl, or f'( < Vl, so f and f' must differ on at least one point. Therefore, the number of sign patterns F2 can realize on S' must be at least as large as the number of sign patterns F3 can realize on S. By Sauer's lemma, the number of sign patterns F2
30

---

e-2m d2
c-2m d2
realize S' is at most equivalently, 2m/da / /m/ d2) I 2e. Now 2 5d2, then, since 2²/2 is increasing for 2 2, the LHS is at least 29/5 2e. Therefore, d3 S 5.500 5.d2.
dg Since Fi pseudo-shatters these points, 2
and,
d2
Finally, observe that after all of these transformations, 9 is in fact subset of F3. Therefore, Pd Σ d3 I 5d2 I 5d1 I 20(d + 1)k log (2(d (
needed.
E BT Objective and Evaluation Metrics
As noted in Section replace the standard BCE objective with soft-label MSE loss; below detail this choice and the associated evaluation metrics.
Soft-label objective. The BT model assumes that human preferences arise from single latent reward function y* via
p" (y1 V2 x)) ofr" (a,, un) r" (ar, v2)).
likellhood estimation under this assumption yields the familiar BCE objective:
LBC (0) = - (e log a(ra(æ; v1) re(2, v2))
Equation 3 views each comparison as interchangeable sensors of a single value system. Our approach flips that premise: the goal is reproduce the observed preference fractions rather than denoise them. We therefore retain the fractional targets and train using
noisy glimpse of one ground-truth, r*, treating annotators
Lsoft Labels (0) wa)< (p(x, M1, y2), Du v1, (2)))))
where is any proper loss. Here indexes a single base reward model.
Evaluation metrics. After BCE training, accuracy-ias -.e. how often the model's decision agrees with the majority the standard metric. In this work, our objective is not to predict the single majority decision nor to predict the preference decision of underlying group of annotators. Thus, Accuracy ill-suited. Instead, for the ensemble, we report the mean-squared calibration error, also referred to the Brier score, which directly measures how closely the model's predicted probabilities align with the observed preference fractions.
For single reward, since the output is taken to be binary, the relevant metric is regret, defined as Regret (p 0.5) (1 p) + 1 0.5)p min(p, p},
which measures the extra error incurred over the Bayes-optimal choice.
Prediction
Hard
Soft
Hard
Accuracy
Binary Cross-Entropy
Labels
Soft
Regret
Brier Score
F
Experiments
F. Training Pipeline
Optimization. We fine-tune all model parameters, including both the base transformer and the final linear reward head, using the SOAP optimizer [77], which we found accelerate training compared to AdamW. Given the relatively small size of the preference datasets compared to the model's capacity, we train for only single epoch, generally sufficient to achieve convergence without overfitting, as demonstrated by previous reward-mon training studies [62, 64, 54, 78, 79, 53, 56]. Training is conducted with BF16 precision on single NVIDIA H100 GPU, utilizing gradient accumulation to accommodate an effective batch size of to 512. We adopt learning rates in the range (1e-5... 5c-5) with cosine decay schedule, a linear warmup spanning the first 3% of training steps, and weight decay set to 0.1.
31

---

Name
Pref. Pairs
Prompts (Train Test)
Prumpt Source
Response Source
Annotation Mechod
Avg Annots
4,791
Anthropie
HH.
GPI-4, GPT:3, Liuma 2/3, Tulo-2
Humnn
ShareGPT
PereonalLLM
263.256
9,402 1,000
RewardBench
GPT-, Chudl-3, Llama-,, Gemini Pro
10
Anthropic
Steee
Bo1pStewz2
21,000
10,000/ 1,000
WildChur, curated
10 LVIDIA
Humtan
3,50
Reddit TL,DR
3.217
845
Reddit TL.DR
GPT3
[urman
7.56
Table 2: Summary of the datasets used in our experiments.
F.2
Datasets
MultiPref
PeMSonal
HelpSteer2
Reddit T1,DR
0.5
Densily
0.3-
0.0
0.5
0.75
1.0
0.5
0.6
0.7
0.6
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
Soft-Labels
Figure 4: Histogram the pairwise soft-label values, where p denotes the fraction annotators that preferred the majority response in each comparison. For convenience every comparison is oriented so that the majority choice is on the left such that 2 0.5.
conduct our experiments on four human-preference datasets. Our goal was to gather diverse set of datasets that vary by several orders of magnitude in size and cover wide range of tasks and prompt sources. Datasets were chosen for two key properties: (1) they either contain explicit pairwise comparisons of candidate LLM responses (2) every response is evaluated by least two independent annotators- hariaattrr-mmannn labelers or distinct reward models that soft-label disagreement can be estimated. Many open-source datasets satisfy one property but not the other; for instance, they may involve diverse annotators who each rate unique subset of responses (e.g., [8, 80]). For datasets without official validation split, place 10% prompts into a test set, ensuring that no prompt appears in both splits. Table reports the resulting statisties, and the paragraphs below detail how we construct the preference fractions p and other dataset-specific information.
be reliably converted into such comparisons; and
MultiPref [59] contains pairs drawn from cnirructionfollowiinn prompts. Each triplet (I, /1, 32) was annotated by two crowdworkers and two domain experts (four annotators total), who independently judged helpfulness, truthfulness, harmlessness, and "overall" preference. In experiments, specifically use the "overall" preference labels, excluding annotations indicating ties. Since the original dataset collected individual judgments rather than direct pairwise comparisons, annotator's 's judgment as a separate vote for either 1/1 or y2, aggregating these votes to form pairwise comparisons. This aggregation yields preference fractions p {0.25, 0.5, 0.75, 1.0) (see Figure 4).
treat each
PeRSonalLLM [60] is synthetic dataset designed to facilitate research on personalized alignment of LLMs. It includes prompts from diverse sources-such such as general instruction-follovinn bench- marks and dialogue datasets (c.g., Authropic HH and HelpSteer-whie dataset)- cach accompanied by candidate responses generated by various language models (c.g., GPT-4, Claude, Cohere, Llama 70B). Rather than human annotations, responses were scored by multiple reward models, each trained tuned to different preference objectives. We treat each reward model's 's preferred response as a synthetic annotator vote, aggregating these votes into pairwise preference fractions p. We include this dataset to evaluate methods significantly larger synthetic annotation set.
we also use as a separate
HelpSteer2 [61] primarily consists of real user queries, supplemented with curated prompts targeting underrepresented tasks. For each prompt, two responses were generated using NVIDIA's Nemotron models SBB340B parameters), ensuring diverse response quality and style. Each response pair
32

---

was annotated by 3-5 skilled human annotators, who independently rated responses on attributes including "helpfulness." " "correctness," "coherence." "complexity." and "verbosity" preference. We constructed pairwise comparisons from these annotations by aggregating annotator judgments of "helpfulness".
Reddit TL;DR [62] contains pairwise comparisons for abstractive summarization: labeler reads Reddit post (or article) and selects the "better" of two candidate summaries (21,32). The original dataset assigns single labeler to most comparisons, with only small subset receiving multiple independent judgments. We retain only those comparisons that have least two labels, yielding smaller multi-annotator subset from which we compute the preference fractions p. Unless otherwise noted, ensembles trained on this dataset were initialized from an RLHF-fine-tur fine-tuned version of the base model, allowing us to isolate the impact of training with soft labels even under limited data.
G Qualitative Case Studies of Ensemble Outputs
Do the rewards lead to noticeably different policies? To test this, we list below the single top-ranked response that each of the first five rewards selects from 100 generations for 100 PRISM prompts. The prompts were cherry-picked to match the case studies of interest, but once a prompt was chosen the outputs are shown as-is, with minor parts removed for brevity.
While responses stay within a broadly reasonable range, they differ tone, emphasis, and proposed actions, confirming that the proof-of-concep training procedure in Section 4 already yields several loosely correlated rewards whose induced policies vary meaningfully without drifting into mutually contradictory extremes.
Prompt
Iteration Top ranked response
The Bible is religious and historical text that is considered sacred text by many, particularly followers of Judaism, Christianity, and Islam. The question of whether it is based upon fact is complex one as interpretations and attitudes towards the Bible vary widely.
Is the bible based upon facts?
The Bible's foundational role is in offering spiritual guidance and moral lessons. While discussions about its historical and scientific accuracy vary, its core value to many is found in its teachings. which guide faith and life practices.
2
3
The Bible has stories some people believe true and others lessons.
The Bible, as a religious text, not primarily a factual account in the same way history book or scientific text would be. It contains stories, teachings, and beliefs that are integral to the religious belief' and practices of Judaism, Christianity, and Islam.
4
The Bible's accuracy varies; some parts are historically confirmed, others are symbolic, depending on one's faith.
5
Table 3: Responses to a sensitive, belief-driven prompt. While all five iterations maintain cautious, neutral stance, they vary in tone, focus, and level of claaooationnsshiieeeee versus "evidence" and between broad contextual background and concise summary. These subtle differences yield distinct reading experiences, even as the underlying hedged neutrality remains constant.
emphasis between "value"
33

---

Prompt
Iteration Top ranked response
Determining the most difficult sport to play can be subjective, [...] Here are few:
Competitive ice skating (figure skating and speed skating): These sports re- quire balance, agility, speed, precision, and artistic skill. Athletes often spend years perfecting their moves and choreography.
What is the most difficult sport to play?
Alpine skiing Ski racing involves high speeds, tight turns, and steep inclines. Athletes must have exceptional balance, strength, and nerve to compete at professional level.
Base jumping Base jumping combines elements of parachuting, skydiving, and rock climbing, It involves jumping from fixed structures like cliffs, bridges
The difficulty of sport can vary greatly depending on personal skill, physical abilities, and interest. [...] examples:
Ice Hockey: requires a high level of physical fitness, speed, and agility, along with the complex skill of skating on ice. The sport also involves a lot of body contact, which can be physically demanding.
2
Formula One Racing This sport demands excellent hand-eye coordination, quick decision-making, and exceptional reflexes. The drivers also need to maintain peak physical fitness to withstand the rigors of extended driving sessions.
Climbing (Rock, Ice, or Alpine) Climbing requires combination of physical strength, mental resilience, and technical skills.
The difficulty of sport can vary greatly depending on individual skills, physical abilities, and personal interest. [..]] these include:
Ice Hockey Known for its high speed, physicality, and complex strategy, ice hockey requires excellent skating, shooting, passing, and teamwork skills. Baseball Baseball may not seem physically demanding compared to other sports, but the mental aspect can be challenging. Pitchers have to throw dozens of pitches with precise control and batters need quick reactions to hit a small ball moving at high speeds.
3
Chess While not physical sport, chess requires exceptional mental en- durance and strategic thinking. Grandmaster- chess games can last for hours, requiring players to plan several moves ahead and anticipate their oppo- nent's strategy.
Determining the most difficult sport to play is subjective [...] sports like ice hockey, rugby and boxing often cited some of the most challeng- ing due their physical intensity, high speeds, and aggressive nature. Gymnastics as another example, requires a high degree of strength, flexibility, precision, and control, and it's not uncommon for athletes to train for several hours day from a very young age.
4
Determining the most difficult sport play can be subjective, [...] Here are few:
Competitive ice skating (figure skating and speed skating): These sports re- quire balance, agility, speed, precision, and artistic skill. Athletes often spend years perfecting their moves and choreography.
5
Alpine skiing Ski racing involves high speeds, tight turns, and steep inclines. Athletes must have exceptional balance, strength, and nerve to compete at professional level.
Base jumping Base jumping combines elements of parachuting, skydiving. and rock climbing. involves jumping from fixed structures like cliffss, bridges
Table 4: A subjective question.
34

---

Prompt
Iteration Top ranked response
The notion that capitalism is doomed fail is common trope in many intellec- tual and political circles. However, it's essential to understand that capitalism is complex and dynamic system that has undergone significant transformations throughout its history. While it's true that capitalism has its flaws, it's not fixed system that will inevitably collapse.
How long does capitalism have to fail?
While capitalism has its limitations and weaknesses, it's hard say that will fail in the classical sense of complete collapse. Capitalism is highly adaptable system that has survived numerous crises, recessions, and challenges. It has continuously evolved, with new ideas and innovations emerging to address existing problems.
2
Why capitalism might not fail completely: Adaptability: Capitalism has a built-in ability to self-correct through market mechanisms, such the price system, which helps to allocate resources effi- ciently and respond to changes demand and supply. Innovation: Capitalism encourages innovation, which leads new products, services, [...]
To answer this, need to define what you mean by "capitalism" and "fail." Let's assume we're talking about capitalist system characterized by free markets, private property, and competition, as commonly understood in modern economies.
In this context, there isn't fixed timeline for capitalism to fail. However, can identify several scenarios that might contribute to its decline or transformation: Crisis collapse: severe economic crisis, such financial meltdown, could lead to collapse of the system. This is unlikely, but not impossible. The 2008 global financial crisis is an example of near-crisis that was averted with significant government intervention. Systemic instability: Repeated cycles of boom-and-buut income inequality, and social unrest could erode trust in the system and lead to a loss of confidence [...]
3
Capitalism, an economic system, has been around for centuries, and it's had its fair share of successes and failures. While it's impossible to predict exactly when capitalism might fail, I'll provide some insights the various factors that can contribute to its failure and the possible outcomes.
Short-term failures: Capitalism experience short-term failures due various factors such as:
4
Market bubbles: Excessive speculation lead to price bubbles, which can burst, causing economic instability. Monopoly power: Concentration of market power can stifle innovation, leading higher prices and reduced consumer choice. Regulatory failures: Inadequate regulations can result in financial crises, environmental disasters, worker exploitation. Global economic shocks: External events like wars, pandemics, or trade wars can disrupt global supply chains and markets [...]
Because capitalism is multifaceted and ever-evolving, predicting definitive "failure" is difficult. Its capacity for self-correction via price signals and its strong incentives for innovation make a total collapse unlikely, though ongoing reform and adaptation are constant.
5
Table 5: A loaded hypothetical prompt presupposing debatable premise.
35


Reviews:
[{"review_id": "wJC2BP3K00", "reviewer": "Reviewer_fob7", "summary": "The paper introduces a novel approach to alignment by replacing a single reward model with a distribution over reward functions, enabling more flexible and pluralistic AI behavior. It avoids rigid annotator groupings by learning compact, diverse reward ensembles directly from preference data, achieving better calibration and preserving minority viewpoints. However, the method raises concerns about scalability, the challenge of selecting the right number of rewards, and whether such complexity is always necessary compared to simpler, single-reward models.", "questions": "Please see my comments above.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The paper replaces the single-reward model with a distribution over reward functions, each reflecting a different interpretation of human judgment. By fine-tuning a separate LLM for each reward, it produces a distribution over policies. This enables flexible inference—offering diverse viewpoints, user-specific responses, or inclusive summaries. \n- The paper learns a compact ensemble of reward functions from aggregated preferences, without using annotator IDs or predefined groups. It introduces pairwise calibration to ensure the ensemble reflects the distribution of annotator opinions. This avoids preference collapse and preserves pluralism without rigid clustering. \n- The paper also demonstrates that compact, diverse reward ensembles can be efficiently learned using a forward stage-wise additive modeling process. Experiments show these ensembles achieve lower calibration MSE than single-reward models, validating their ability to represent diverse preferences accurately.  \n\n# Weakness \nWhile the paper presents a compelling approach by replacing the single-reward model with a distribution over diverse reward functions and training separate LLMs for each, there are still several concerns regarding the method. \n- Determining the number of reward functions: how should the appropriate number be chosen, and what are the consequences of selecting too few or too many?\n- Computational cost: training and maintaining multiple policies significantly increases resource requirements. Are there strategies to mitigate this overhead? \n- Practical necessity: in many cases, a single reward function may suffice. How can we assess when it's truly beneficial to adopt multiple reward models?", "was_revised": true, "final_justification": "I have no further technical questions to the paper. I am inclined to rate it as a weak accept as reflected by my original score.", "rebuttal": "> * Determining the number of reward functions: how should the appropriate number be chosen, and what are the consequences of selecting too few or too many?\n> * Practical necessity: in many cases, a single reward function may suffice. How can we assess when it's truly beneficial to adopt multiple reward models?\n\nWe view the ensemble size as a balance between capturing diversity and practical deployment concerns. If the ensemble is too small, it can under‑represent minority viewpoints. If it grows beyond what the task needs, it introduces redundant parameters, increases training and inference cost, and may steer the system toward unwanted personalization. \n\nA single reward can indeed suffice in some cases. Yet, when users tend to prefer different outputs or when output diversity in downstream applications is warranted (such as in creative processes), additional rewards become beneficial because they preserve those divergent judgments rather than averaging them away. The pairwise‑calibration objective provides a direct signal for this decision.\n\nThe procedure we describe fits reward functions sequentially. After each new reward is added, we track the pairwise‑calibration loss on a held‑out set and terminate once the validation error no longer decreases meaningfully, so the ensemble grows only until further expansion stops providing benefit. This gives a principled and automated way of choosing the \"appropriate\" number of reward functions. \n\n> * Computational cost: training and maintaining multiple policies significantly increases resource requirements. Are there strategies to mitigate this overhead?\n\nIndeed, in the current implementation, the compute required for an ensemble of $k$ reward functions scales linearly with $k$, both for training and any downstream fine-tuning or storage of policy variants. Theorem 2 bounds this overhead: the worst‑case number of rewards needed to reach accuracy tolerance $\\varepsilon$ grows only linearly in $1/\\varepsilon$, so $k$ can remain modest for realistic tolerance levels.\n\nTo mitigate this overhead at deployment, we can use approaches like best-of-N sampling or importance sampling that draw candidate completions from a single base model and rescore them with the ensemble. These methods avoid fully fine-tuning multiple model copies while keeping the output policy aligned with the new reward signals."}, {"review_id": "nsXWqDCPfZ", "reviewer": "Reviewer_cPuB", "summary": "RLHF mostly represents the aggregate view of the majority, dismissing the inherent disagreement and diverging opinions across the population. This limits the utility of current models. This paper proposes a methodology to move to a distribution over reward-functions to reflect the diversity of judgments. For each reward, a LLM is fine-tuned which results in a distribution over policies. An ensemble of reward functions is formed to signify the diverging human preferences.  The methodology aims to be calibrated pairwise; for every pairwise preference, the fraction of reward functions in the ensemble “voting” for a preference matches the fraction of annotators with that preference. \n\nThe authors first formally define pairwise calibration and provide theoretical guarantees for getting pairwise calibrated ensembles with a small support, for how removing outlier rewards has minimal impact on the calibration, and that the ensembles can generalize to unseen populations. \n\nThe method was tested empirically on four pairwise comparison based datasets and the calibration was evaluated through Mean-Squared Error calibration, showing reduced MSE as the ensemble increases to around 2-4.", "questions": "- Why did you opt for MSE loss specifically? And e.g., not a soft cross-entropy loss or JSD loss?  \n- Is this setup also extendable to tasks that are not set up in a pairwise preference way? \n- Results indicate that from 2 reward models onwards, the calibration improves on held-out prompts, do you think this will extend to other tasks as well? Do you have an intuition of how this will change based on the subjectivity of the task? \n- Is there a specific reason you did not include calibration metrics such as Brier Score or ECE? \n- Do the pairwise Kendall $\\tau$ correlation scores in Figure 3 also reflect and explain the ensemble size, e.g., that only 2 reward models are already sufficient?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths.**\n- The work tackles an important problem and does that through a well motivated approach. I really like how it is supported by theoretical guarantees. \n- The paper was easy to read and looked neat. \n- In a way, it is good that this method does not rely on explicitly assigning reward models to specific demographics to represent the population. This makes the method more adaptable as making the decision which to include and which not makes it difficult with annotator availability, intersectionality, etc. \n\n**Weaknesses.** \n- The positioning of the paper in comparison to other related work is difficult for the reader. The Related Work is in the Appendix and a highly condensed version in the main body did not give a clear idea. This is also reflected in the Empirical Results section where there is no comparison to other baselines. As such, it is hard to get an idea of how much improvement this pairwise calibration brings. If there are no directly related baselines, maybe it is a good idea to make this more prominent in the Related Work section. \n- There are no experiments where the method is applied to unseen datasets. This would really strengthen the paper as it could help in understanding whether the method does the task well or it might actually just be overfitting to the annotators of a dataset (some might suffer from a lack of diversity of annotators).\n- I find that the discussion surrounding undesirable viewpoints or outliers could be a bit more prominent. When is a viewpoint or preference undesirable? Is this e.g., hateful responses? This should be clarified a bit. Currently, noisy annotations (an annotator might be making mistakes) are not discussed whether their method would be robust to that or not or whether these fall under outliers.   \n- The paper does not test datasets that have a lower agreement rate around 50%, or does an analysis of how performance is in terms of different levels of agreement across samples.", "was_revised": true, "final_justification": "Given the novelty of the task the paper is addressing, pairwise calibration for pluralistic alignment, I have decided to raise my score. Many of my concerns were adequately addressed: \n- The authors indicate that they will improve the related work section. This especially made me raise my score as for me it was unclear how to position this paper in this subarea of pluralistic alignment and why there were no baselines. To me it is now clear that this is one of the first work (that I am aware of) that addresses pairwise calibration in reward models. I did try to think of any existing baselines that could fit for this setup but could not think of any, it is possible I might have missed something. \n- The point regarding unseen datasets was clarified. \n- The authors indicated that they will discuss more about undesirable viewpoints and annotator noise. \n\nI do think the authors could still add a small analysis regarding the performance of the model as disagreement level changes.\n\nI also took into account the responses to the author reviewers when raising my score.", "rebuttal": "> * The positioning of the paper in comparison to other related work is difficult for the reader. The Related Work is in the Appendix and a highly condensed version in the main body did not give a clear idea. This is also reflected in the Empirical Results section where there is no comparison to other baselines. As such, it is hard to get an idea of how much improvement this pairwise calibration brings. If there are no directly related baselines, maybe it is a good idea to make this more prominent in the Related Work section.\n\nWe agree that clearer positioning is warranted. The main submission condensed the related work due to the page limit, but we will move key context from the appendix into the main paper to address your comment. Below we briefly elaborate on the lack of baselines. \n\nExisting multi‑reward techniques hinge on predefined groups, inferred clusters, or annotator identifiers and therefore assume the very structure that our method seeks to avoid. Because pairwise calibration learns directly from unlabeled disagreement, there is no off‑the‑shelf baseline that operates under the same information constraints.\n\nConstructing a head‑to‑head comparison would require first partitioning annotators or fabricating demographic tags, which would inject assumptions our approach is designed to remove. \n\nThis absence of compatible baselines is inherent to the contribution: we introduce a framework that treats disagreement as signal without relying on latent or explicit clusters, filling a gap left open by prior group‑conditioned methods.\n\n\n> * There are no experiments where the method is applied to unseen datasets. This would really strengthen the paper as it could help in understanding whether the method does the task well or it might actually just be overfitting to the annotators of a dataset (some might suffer from a lack of diversity of annotators).\n\nGeneralization can be understood along several axes: new responses, unseen prompts, additional annotators, and different task types. Our theoretical analysis in Section 3.3, roughly speaking, shows that the ensemble inherits the same sample-complexity guarantees as its constituent reward models, supporting its ability to extend beyond the training comparisons.\n\nThe datasets we analyze already probe several of these axes. For example, *MultiPref* and *HelpSteer2* draw feedback from about $300$ and $1,000$ unique annotators, respectively, so each evaluation fold mixes many human viewpoints rather than revisiting the same few raters. Moreover, *MultiPref* and *PersonalLLM* combine data from diverse task domains such as open-ended question answering, creative writing, code generation, and safety probes, so our held-out evaluations test the ensemble on a broad range of goals instead of a single task.\n\n\n> * I find that the discussion surrounding undesirable viewpoints or outliers could be a bit more prominent. When is a viewpoint or preference undesirable? Is this e.g., hateful responses? This should be clarified a bit. \n\nWe agree that a clearer discussion of “outliers” will strengthen the paper. Conceptually, learning multiple viewpoints risks allocating weight to preferences that most users would deem undesirable, such as hateful responses. In our terminology, an outlier is a reward function whose judgments diverge sharply from the rest of the population. We measure this via the disagreement score (line 156): a reward is considered extreme when its score is large relative to the minimum attainable value. Admittedly, according to this definition, not every outlier viewpoint is repugnant, but every repugnant viewpoint is an outlier, and the latter implication is the one we conceptually rely on to ensure the ensemble is safe. \n\n\n> Currently, noisy annotations (an annotator might be making mistakes) are not discussed whether their method would be robust to that or not or whether these fall under outliers.\n\nAlthough we do not model label noise explicitly, the ensemble method is especially valuable in settings where a few annotation errors could flip the majority label and mislead a single-reward approach. Because the ensemble tracks the full preference fraction rather than a hard majority, a stray vote merely adjusts the target probability slightly, so no single mistake dominates the learning signal.\n\n> * The paper does not test datasets that have a lower agreement rate around 50%, or does an analysis of how performance is in terms of different levels of agreement across samples.\n\nWe quantify disagreement for every comparison by $p$, the fraction of annotators who choose the majority answer, where $p = 0.5$ is the highest possible disagreement. The datasets we study span a wide range of $p$ values, including many low-agreement cases.\n\nFurthermore, the small numbers of annotators per pair can introduce an upward bias in the observed agreement value, meaning our datasets may have more disagreement than the observed $p$ reveals. To illustrate, consider a scenario of maximal disagreement where the true population preference is split 50/50 on every pair. If we sample only four annotators for each, the observed majority fraction for each edge will be 1.0, 0.75, or 0.5 with respective probabilities of 1/8, 1/2, and 3/8. This results in an expected value of 0.69.\n\n**Questions**\n>  * Why did you opt for MSE loss specifically? And e.g., not a soft cross-entropy loss or JSD loss?\n\nWe chose MSE for analytical convenience and cleaner theory, and we expect other proper divergences that penalize discrepancies between $\\hat p$ and $p^\\star$ would behave similarly in principle. For the algorithm, residuals under MSE can fall outside the $[0,1]$ range, giving each subsequent reward a stronger signal to close the remaining calibration gap.\n\n> * Is this setup also extendable to tasks that are not set up in a pairwise preference way?\n\nThank you for raising this. As briefly touched upon in Section 6, our analysis focuses on pairwise preferences and does not guarantee an accurate representation of higher-order judgments over larger candidate sets. Ideally, for any group of responses, the share of learned reward functions ranking a particular answer first would reflect the true pattern of preferences in the population. Recovering that structure from pairwise data alone is information-theoretically impossible because such comparisons lack enough detail to pin down full rankings. Capturing these richer relationships would require more expressive feedback, such as complete rankings or best-of-$\\ell$ choices drawn from larger pools of answers. Exploring how to extend the framework to those settings is an exciting direction for future work.\n\n>  * Results indicate that from 2 reward models onwards, the calibration improves on held-out prompts, do you think this will extend to other tasks as well? Do you have an intuition of how this will change based on the subjectivity of the task? \n> * Do the pairwise Kendall correlation scores in Figure 3 also reflect and explain the ensemble size, e.g., that only 2 reward models are already sufficient?\n\nIntuitively, we expect the trend to generalize. Tasks with higher subjectivity, such as creative writing style or humor, should keep gaining from additional reward models, because each model captures different evaluator biases. For more objective tasks, most evaluators share the same preference ordering, so the first model already captures the key signal, extra models mostly fit residual noise, and calibration gains taper off after one or two models.\n\n>  * Is there a specific reason you did not include calibration metrics such as Brier Score or ECE?\n\nThank you for raising this. The Brier score is effectively the mean squared error between the predicted probability and the observed outcome, which we already report. We will include ECE in the next revision to give an additional view of calibration."}, {"review_id": "UaY3pf8dns", "reviewer": "Reviewer_sbAB", "summary": "This paper studies pluralistic alignment, the problem of aligning models when desired behavior is multi-modal or subjective. The paper proposes a reward ensemble model to capture this multi-modality, which is trained to minimize the calibration error wrt different annotator preferences. The goal is to find a small set of ensembles that capture this multi-modality and practical to train. Before giving a training algorithm, the authors prove that even under a simplified setting, this problem is NP-hard; but a well-calibrated ensemble approximation is possible. Outlier reward functions are possible but their total weights decay as reciprocal of their relative disagreement ($1/(\\beta-1)$). Rather than fitting the ensemble model directly, which is computationally intensive, the authors propose using forward stagewise additive modeling approach that decomposes the problem into a sequence of easier-to-solve subproblems. First, a base reward function (modeled as a reward head attached to a frozen SFT-ed model) is fitted to the calibration probabilities and residuals are estimated. A newly initialized reward function is fitted to the residuals and this process is repeated for $k$ iterations. Experimental results suggest that compared to theoretically best single reward function, ensemble approach achieves lower MSE error on unseen prompts and different reward models capture distinct preferences.", "questions": "I would increase my score when the followings are addressed.\n\n1. Please run an LLM post-training experiment with the proposed reward ensemble.\n\n2. Please add reward ensemble baselines. It could just be optimizing Eq (2) jointly over all reward functions and weights.\n\n3. Is using $\\sigma$ for minimizing MSE over residuals appropriate?\n\n4. What is the impact of increasing number of annotators?\n\n5. What is the impact of temperature on correlation and MSE loss?\n\n6. Is updating all $\\alpha$s after every step necessary to get good performance?\n\n7. Please add MSE loss on training prompts.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. **Theoretically justified approach to multi-modal preference modeling.**\nThe paper clearly explains the pluralistic alignment problem with a detailed motivation, contrasting with previous approaches. It introduces pairwise calibration for reward model ensembles and shows that finding $\\epsilon$-calibrated ensembles is NP-hard. Given a reward model ensemble is pairwise calibrated, it proves its robustness -- outliers have bounded reward weights.\n\n2. **A practical algorithm for optimizing reward ensemble objective.**\nFull optimization problem involves tuning $k$ reward models as well as their weights. This objective can become computationally intensive. The paper proposes using forward stagewise additive modeling (FSAM), an approach for learning ensembles, for learning reward ensembles and their weights. While fitting a fresh reward model to residuals is similar to the original FSAM, the paper retunes weights at each step.\n\n3. **Experimental results show superiority to single model reward functions and diversity.**\nExperimental results using four different datasets show that ensemble approach achieves lower MSE on holdout set and produces diverse reward models when measured using Kendal-$\\tau$ on rankings.\n\n# Weaknesses\n1. **No LLM post-training with the proposed ensemble approach.** While the paper clearly explains why the ensemble approach is needed for better LLM alignment and briefly touches on how reward function ensembles can be used during inference, it doesn't provide any quantitative metrics on whether it actually is useful in practice. I think at least the performance of BoN with the trained reward ensemble should be reported and compared to baselines.\n\n2. **Lack of ensemble baselines.** While theoretical single reward limit is useful to understand the relative improvement of the proposed model, no other ensemble baselines is used in the comparison. For example, can you add multiple-reward heads and optimize objective in Eq. (2) jointly as an oracle?\n\n3. **Implication of using $\\sigma$.** While the first reward model is learned using probabilities (preference fractions) as targets, other reward models are learned using residuals -- difference of two probabilities -- as targets. These two are of different units. While a normalized function (i.e. $\\sigma$) is used in both cases, it is only appropriate for the first reward model training.\n\n4. **The impact of the number of annotators.** How much does the number of annotators per prompt impact the overall results? What happens if you have only 2 annotators? Please provide additional results by resampling (without replacement) responses for each datasets to simulate increasing number of annotators (from 2 to more).\n\n5. **The impact of Temperature on Kendall-$\\tau$ and MSE.** How much does the temperature impact the overall results? Please provide additional empirical results by varying the temperature and measuring Kendall-$\\tau$ and MSE loss.\n\n6. **The impact of updating $\\alpha$ vs keeping them fixed.** What happens if you keep $\\alpha$s fixed after each step rather than tuning them again?\n\n7. **MSE on training prompts.** Please add MSE on training prompts in Figure 2.", "was_revised": true, "final_justification": "The authors adressed most of my concerns and added additional empirical results for improving the analysis.", "rebuttal": ">  No LLM post-training with the proposed ensemble approach. While the paper clearly explains why the ensemble approach is needed for better LLM alignment and briefly touches on how reward function ensembles can be used during inference, it doesn't provide any quantitative metrics on whether it actually is useful in practice. I think at least the performance of BoN with the trained reward ensemble should be reported and compared to baselines. \n\n> Please run an LLM post-training experiment with the proposed reward ensemble.\n\nCould you please outline a concrete post-training experiment we could run? Due to the nature of our approach, quantitative evaluation of our reward ensemble is challenging in practice because the ensemble is designed to preserve several plausible answers rather than collapse them into one. Conventional single-output evaluations common in the literature, such as win-rate or average reward, require a canonical answer or a monolithic notion of “better” and therefore obscure the benefit of capturing multiple annotator perspectives.\n\nPrior work that evaluates multiple policies typically assumes a known ground-truth distribution over answers or explicit group labels for annotators. Our framework works without group identities, finding a small ensemble whose voting pattern matches the observed pairwise fractions, so evaluation methods that reduce to binary wins or accuracies do not transfer directly. \n\nIn summary, if you can point us to an evaluation method that respects multiple plausible outputs, we will gladly run it and include the results.\n\n>  Lack of ensemble baselines. While theoretical single reward limit is useful to understand the relative improvement of the proposed model, no other ensemble baselines is used in the comparison. For example, can you add multiple-reward heads and optimize objective in Eq. (2) jointly as an oracle?\n\nOptimizing the joint objective in Eq. (2) over all reward parameters and mixture weights is highly nonconvex, and we are not aware of a standard algorithm that would serve as a clear canonical baseline. Our empirical evaluation was designed to show feasibility: a simple greedy heuristic can recover a well-calibrated ensemble on real preference data, demonstrating that small reward ensembles are practical rather than only a theoretical construct. Developing and benchmarking more advanced optimization techniques for this loss is an exciting direction for future work. If you have a specific joint-training approach you consider appropriate, we would be happy to try it and report the results.\n\n>  Implication of using $\\sigma$ and $\\alpha$. While the first reward model is learned using probabilities (preference fractions) as targets, other reward models are learned using residuals -- difference of two probabilities -- as targets. These two are of different units. While a normalized function (i.e. $\\sigma$) is used in both cases, it is only appropriate for the first reward model training. \n\n>  The impact of updating $\\alpha$ vs keeping them fixed. What happens if you keep $\\alpha$s fixed after each step rather than tuning them again?\n\n\nThank you for raising these two points. First, we experimented with three weighting strategies and observed only modest differences in calibration. The first strategy freezes the existing mixture at round $j$, learns a new weight $\\alpha_j$, and rescales earlier weights by $(1-\\alpha_j)$. The second strategy sets $\\alpha_j$ to $1/j$ to create an unweighted running average. The third strategy, our default, retrains the entire weight vector after adding each new head. For example, with $k=8$ on *MultiPref*, the test MSEs were 0.071 for the fully reweighted mixture, 0.073 for the fixed‑previous scheme, and 0.081 for the unweighted average. The refitted mixtures typically converge to weights close to $1/j$, suggesting that equal weighting is already near optimal and further optimization only makes minor adjustments.\n\nSecond, while the first reward head is trained on preference fractions and the later heads on residuals, the subsequent weight update absorbs the scale difference introduced by the sigmoid. In practice the combined model remains well calibrated because the learned mixture rebalances the contributions of earlier and later heads.\n\n>  The impact of the number of annotators. How much does the number of annotators per prompt impact the overall results? What happens if you have only 2 annotators? Please provide additional results by resampling (without replacement) responses for each datasets to simulate increasing number of annotators (from 2 to more).\n\nThank you for raising this point. We agree that the number of annotators per prompt could influence calibration and are running additional tests. As a first step, we resampled the PersonalLLM dataset with five annotators instead of the original ten. Preliminary results (see below) compare $n = 5$ against $n = 10$. We will extend this analysis and report the full results in the next revision.\n\n|  round_inx  |   mse_n10 |   mse_n5 |\n|---:|----------:|---------:|\n|  0 |     0.097 |    0.104 |\n|  1 |     0.078 |    0.060 |\n|  2 |     0.049 |    0.049 |\n|  3 |     0.038 |    0.049 |\n|  4 |     0.034 |    0.047 |\n|  5 |     0.032 |    0.046 |\n|  6 |     0.031 |    0.045 |\n|  7 |     0.031 |    0.044 |\n\n\n> The impact of Temperature on Kendall-$\\tau$ and MSE. How much does the temperature impact the overall results? Please provide additional empirical results by varying the temperature and measuring Kendall-$\\tau$ and MSE loss.\n\nTo test whether the reward models in the ensemble capture distinct preferences, we evaluate how they rank multiple candidate responses to the same prompts. We need a diverse pool of answers for this comparison, so we control diversity with the sampling temperature and experimented with values between 0.7 and 1.2.\n\nWhen the temperature is low, the model produces almost identical answers (semantically and stylistically). Near-duplication collapses the effective sample size and can amplify noise as the scores assigned by a reward function to near-duplicates differ only by small random fluctuations. A higher temperature injects enough diversity that score gaps reflect genuine judgment differences rather than tie-breaking randomness.\n\nBelow we attach additional results showing the effect of temperature on the Kendall-τ metric for the *MultiPref* dataset:\n\n* *T=1.2*:\n\n|         |   Round 0 |   Round 1 |   Round 2 |   Round 3 |   Round 4 |   Round 5 |   Round 6 |   Round 7 |\n|:--------|----------:|----------:|----------:|----------:|----------:|----------:|----------:|----------:|\n| Round 0 |     1.000 |    -0.066 |     0.200 |     0.162 |     0.442 |     0.237 |     0.453 |     0.469 |\n| Round 1 |    -0.066 |     1.000 |    -0.275 |     0.254 |    -0.328 |    -0.144 |    -0.311 |    -0.286 |\n| Round 2 |     0.200 |    -0.275 |     1.000 |    -0.047 |     0.247 |     0.108 |     0.356 |     0.279 |\n| Round 3 |     0.162 |     0.254 |    -0.047 |     1.000 |    -0.038 |     0.002 |    -0.039 |     0.023 |\n| Round 4 |     0.442 |    -0.328 |     0.247 |    -0.038 |     1.000 |     0.260 |     0.600 |     0.692 |\n| Round 5 |     0.237 |    -0.144 |     0.108 |     0.002 |     0.260 |     1.000 |     0.231 |     0.213 |\n| Round 6 |     0.453 |    -0.311 |     0.356 |    -0.039 |     0.600 |     0.231 |     1.000 |     0.646 |\n| Round 7 |     0.469 |    -0.286 |     0.279 |     0.023 |     0.692 |     0.213 |     0.646 |     1.000 |\n\n* *T=1.0*:\n\n\n|         |   Round 0 |   Round 1 |   Round 2 |   Round 3 |   Round 4 |   Round 5 |   Round 6 |   Round 7 |\n|:--------|----------:|----------:|----------:|----------:|----------:|----------:|----------:|----------:|\n| Round 0 |     1.000 |    -0.133 |     0.205 |     0.221 |     0.272 |     0.159 |     0.339 |     0.220 |\n| Round 1 |    -0.133 |     1.000 |    -0.268 |     0.162 |    -0.235 |    -0.066 |    -0.127 |    -0.052 |\n| Round 2 |     0.205 |    -0.268 |     1.000 |    -0.122 |     0.096 |     0.078 |     0.297 |     0.060 |\n| Round 3 |     0.221 |     0.162 |    -0.122 |     1.000 |     0.097 |    -0.032 |     0.027 |     0.193 |\n| Round 4 |     0.272 |    -0.235 |     0.096 |     0.097 |     1.000 |     0.091 |     0.232 |     0.393 |\n| Round 5 |     0.159 |    -0.066 |     0.078 |    -0.032 |     0.091 |     1.000 |     0.105 |     0.032 |\n| Round 6 |     0.339 |    -0.127 |     0.297 |     0.027 |     0.232 |     0.105 |     1.000 |     0.268 |\n| Round 7 |     0.220 |    -0.052 |     0.060 |     0.193 |     0.393 |     0.032 |     0.268 |     1.000 |\n\n* *T=0.8*:\n\n|         |   Round 0 |   Round 1 |   Round 2 |   Round 3 |   Round 4 |   Round 5 |   Round 6 |   Round 7 |\n|:--------|----------:|----------:|----------:|----------:|----------:|----------:|----------:|----------:|\n| Round 0 |     1.000 |    -0.524 |     0.297 |    -0.260 |     0.561 |     0.487 |     0.523 |     0.461 |\n| Round 1 |    -0.524 |     1.000 |    -0.371 |     0.420 |    -0.616 |    -0.484 |    -0.445 |    -0.421 |\n| Round 2 |     0.297 |    -0.371 |     1.000 |    -0.288 |     0.298 |     0.297 |     0.339 |     0.277 |\n| Round 3 |    -0.260 |     0.420 |    -0.288 |     1.000 |    -0.326 |    -0.327 |    -0.308 |    -0.238 |\n| Round 4 |     0.561 |    -0.616 |     0.298 |    -0.326 |     1.000 |     0.477 |     0.438 |     0.515 |\n| Round 5 |     0.487 |    -0.484 |     0.297 |    -0.327 |     0.477 |     1.000 |     0.469 |     0.400 |\n| Round 6 |     0.523 |    -0.445 |     0.339 |    -0.308 |     0.438 |     0.469 |     1.000 |     0.389 |\n| Round 7 |     0.461 |    -0.421 |     0.277 |    -0.238 |     0.515 |     0.400 |     0.389 |     1.000 |\n \n> MSE on training prompts. Please add MSE on training prompts in Figure 2.\n\nThank you for pointing this out, we will add the MSE on training prompts to Figure 2 in the revised manuscript."}]

Author rebuttal:

Meta-review: This work propose a method to align model behavior when human preference can be diverse. Specifically, instead of assuming a ground-truth reward function, the authors allow each annotator to have different reward function, and use a distribution over the reward functions to reflect the disagreement. The authors introduce the concept of epsilon-pairwise calibration that measures the quality of the  distribution. Both theoretical results and empirical evaluations are provided. 

The paper is generally well-motivated and the presentation is great. The idea of multiple preference is natural and the results presented in the paper is solid. The reviewers also support the acceptance of the work.
Decision: Accept (poster)