Direct Alignment with Heterogeneous Preferences
Authors: Ali Shirali
Arash Nasr-Esfahany
Abdullah Omar Alomar
Parsa Mirtaheri
Rediet Abebe
Ariel D. Procaccia
Venue: NeurIPS 2025 poster ()
Keywords: Direct Alignment
Pluralistic Alignment
Heterogenous Preferences
Social Choice Theory
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous. We formalize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption.
We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine improvements under different information settings, focusing on direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficient consistent direct loss exists in this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment.

as truth or Direct Alignment with Heterogeneous Preferences
Ali Shirali' UC Berkeley
Arash Nasr-Esfahany'
Abdullah Alomar MIT Ikigai Labs
Parsa Mirtaheri UC San Diego
MIT
Rediet Abebe
Ariel Procaccial Harvard University
ELLIS Institute, MPI for Intelligent Systems, & Tübingen Al Center
Abstract
Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous, We for- malize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption. We show that aligning to heterogeneous preferences with single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine im- provements under different information settings, focusing direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficiee this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment.
consistent direct loss exists
Introduction
Human rewards and preferences are heterogeneous [1.5]. Despite this, learning from preference data often bypasses this insight, relying on what we dub the preference homogeneity assumption. This tension in assumptions is readily apparent in standard human-Al alignment methods such as reinforcement learning from human feedback (RLHF) (6- and direct preference optimization (DPO) 100 -hich assume a single reward function captures the interests of the entire population.
We study heterogeneous population in which individuals belong to distinct user types, each defined by its own reward function. Two main approaches address this setting [II: personalizing policies for each user type 112
or aggregating diverse rewards into single objective and deploying Our focus is on the latter, which is necessary when user types are unobservable at inference time 20 when the cost of maintaining specialized models prohibitive. A universal policy is also preferable when personalization undermines core
17
universal policy 18 19
safety- that benefit from broad consensus 21 22
One may notice that standard methods based on the preference homogeneity assumption implicitly aggregate rewards. However, such aggregation can undesirable counterintuitive, leading to unexpected behavior 253 as learning from heterogeneous preferences becomes unrealizable: A single reward cannot capture the complexity of population preferences with multiple rewards 26
In the quest for a single policy that accommodates a heterogeneous population, we show that the only aggregation of rewards across user types that yields well-defined alignment problem is affine
aggregation, with the average reward emerging as a natural choice. However, standard methods like DPO do not maximize this user-weighted average reward. Building on insights by Siththaranjan et al.
AS and AN contributed equally. AS conducted this work while visiting Harvard.
t Equal advising
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

20], we show that DPO implicitly maximizes Borda count, which comes with unexpected drawbacks, e.g., the optimal solution depends on how alternative responses are sampled, even for infinite data.
We observe that learning the average reward over user types-ort or equivalently, a policy that maximizes it-from anonymous data is impossible. Focusing on direct alignment methods [TO] 27 28 which avoid explicit reward modeling, we study the benefits of using annotator data for range of information settings. We show that improving DPO with first-order correction its objective is possible with minimal annotator information. Specifically, we design an approximate direct alignment method when each preference data point is paired with another one labeled by the same user.
On the other hand, we find that there are limits to what is possible even with significant annotator information. In particular, we propose consistent loss function for direct alignment when we have feedback on each data point from every user type. But this loss is sample-inefficie where all annotators agreed. Surprisingly, we prove that no consistent loss the rest of the data.
using only data
In sum, the homogeneity assumption leads to undesirable outcomes when aligning a single Al agent to diverse preferences. Our analysis shows that there is limited class of reward aggregation that yields well-defined alignment objective, with average reward over user types emerging as the natural choice. This requires some annotator data, though small amounts of data can yield significant improvements. Our findings, however, uncover a fundamental tension between consistency and sample efficiency in direct alignment: To achieve both, must forgo the benefits of direct optimization and instead train individualized reward models, which inevitably incurs significant training and storage costs.
Preliminaries
In the alignment problem, we consider a setting where reward function r' evaluates a response to query x by assigning a score r" ([x, y]). The goal is to design policy that selects high-reward responses for each query. Formally, a policy defines a probability distribution over responses: given a query æx it selects y with probability л (yy x)). Typically, we begin with a reference policy wref, which acts as a prior over y [29]. and for each æ, aim to maximize
[ ( a)) [r ((x, v))] - PRR (a(- a)); Trae(: x)))
where DKL is the KL-divergence. We denote the optimal policy by n². In practice, Tel is often pretrained (language) model, and the regularization parameter /3 controls deviation from it. Eq. (T) often includes Ez,, which is important in practice but does not affect x² in theory.
When r" is known, can directly apply RL to maximize Eq. (I) In practice, however, ," is often unknown and must be inferred from human feedback before applying RL, a process known as RLHF. Although widely used, RLHF can be difficult tune due the complexities of RL. An alternative gaining traction is direct alignment with preferences [30 27), which avoids explicit reward modeling and instead trains the policy directly from human feedback.
Preference Model. Both direct alignment and RLHF rely on model of human preference to relate reward values with observed preference data. Consider the case where responses y1 and y/2 are generated for a given query æ. We express the probability that y2 is preferred to yı
Pr(ya > y ( = a (r" [[x ya)) - (1)))))))
where a is non- non-decreasing function bounded between 0 and 1 the sigmoid function corresponding to the well-known Bradley- Terry (BT) model [34]. Related which are beyond our scope.
31 33
A widely-used choice for o
work have explored smooth [35 or inconsistent preferences 33 37
Direct Preference Optimization. Among direct alignment methods, DPO has become the de facto standard. By exploiting a closed-form solution for maximizing Eq. ( it directly relates any reward to its optimal policy. Thereby, instead of estimating the reward explicitly, DPO trains a policy whose induced reward best accounts for the observed preferences. Below, we derive this connection.
First, maximizing Eq. has a well-known solution
38
The optimal policy " takes the form
π (y x) =
1
nof ( x) exp (3r yl)
(3)
Z(x)

---

Here, ((x)=Ey (V x)- exp()r" (ap()r"((z,)) (())))) is the partition function. Eq. direct relationship between policy ratios and reward differences as
establishes
TT Tree (92)
log 'T" (y11 Tref (y1)
r" (122 r" (y)) Blog
Henceforth, we omit x when clear from context. This shows that the reward difference between two responses is fully captured by the difference in their policy ratios, and motivates the definition of the
induced reward of a policy π as log (3))
r(y) the reward function for which the optimal policy.
The difference in rewards of y and y2 is sufficient to express the likelihood of y2 y1 in Eq. (2)
Using Eq. (4), we can therefore write the likelihood as a function of π"
π (y22 Trot Tror(
π (3/1)
Pr(y2 yı I o log
3 log Trae (y1)
(5)
For any policy x, we similarly define Pr(y2 > y/ x). To estimate n², we can then apply maximum
likelihood estimation: Given a dataset D of query-respont pairs (æ, yt, yw) with yo > yı, DPO maximizes the following log-likelihood (or equivalently, minimizes the cross-entropy loss);
Σ
log Pr(y > yu | TT:
(6)
(æ.3) .w..).ww)DD
3 Problem Formulation
The alignment problem is traditionally framed under a preference homogeneity assumption, where single reward is presumed to capture all individual interests. In practice, people's preferences can differ significantly. To better capture real-world settings, formalize preference heterogeneity by allowing reward functions to vary across user types as we detail below.
Heterogeneous Preference Model. The influential study of "individual choice behavior" by Luce 32) and other foundational works on human decision-making in mathematical psychology such Shepard 39) focus on individual preference models. Luce (2 uses axiomatic framework to establish the existence of a value function-ake akin to reward for each individual, which, once normalized, explains their choice probabilities (exp(r") the BT model is one such value function).
In practice, individual identities are often unobserved. As result, standard preference modeling approaches assume single reward function for the entire population. This homogeneity assumption renders preference learning unrealizable: Even a family of models can represent each individual's preferences, it may fail capture aggregate population behavior. For instance, mixture of BT models cannot be represented by a single BT (we prove this in Prop. F. for completenes)).
To fully account for heterogeneity, we need to define individual rewards. However, learning at scale with this level of granularity is impractical, especially when working with finite data. Hence, we group individuals into multiple user types, denoted by U4 where individuals with the same type have similar rewards, but this need not hold across types. Unless otherwise stated, we assume [2/] >
For user of type E U, we denote their reward function by r" r=;u,,, which assigns a score r (y: u) each response y. We model the preferences of users of type as
Pr(ya > yı rr" u) = a (r" (y2; u) - r (y)
The law of total expectation then implies that the population-level preferences follow
Pr(yz V1 r*) = B., [r (r u) - -r ((1:
(8)
The Extended Alignment Problem. Deriving a universal policy to serve heterogeneous population requires aggregation of diverse rewards. As we show next, an affine combination is the only form of aggregation that guarantees well-defined problem, i.e., problem that yields the same optimal policy for every reward that is consistent with the distribution of preferences.

---

Proposition 3.1. Consider a differentiable aggregation f IR" + R. Suppose that for every reward r consistent with the preference distribution- meaning Pr(y X y1 (r) Pr(y2 > 31 rr)) for all prompts and responses y1, y2 in the response space the function f ((((yy u)) induces the same ordering over y. If (r(y: u))
)) has non-empty interior, then f must be affine.
See See proof on page 25 This result rules out many commonly-used aggregations, such as Max-Min [9 or Nash social welfare 40. Among all the affine combinations, the expected reward across user types emerges as natural choice here. Any other affine combination would weigh people unequally, which requires strong justifications and is rare in practice.
To summarize, our objective is to maximize
By--r( (az) [B [rr ([zz, y]: w]]] RRL (T( x); Trer(- x))
(9)
for every prompt œ. With this extended framework in mind, we next discuss why standard approaches like RLHF or DPO do not necessarily yield the optimal policy.
Implications of the Preference Homogeneity Assumption
When preferences are heterogeneous, standard RLHF or DPO cannot yield the optimal policy " that maximizes Eq. (9). If they could, the (user-weighted. expected reward would be learnable as the induced reward of π² However, as we show in 5nI as noted in prior work in specific cases
20 41
learning the expected reward from anonymous preferences is impossible.
To explain DPO's failure in finding n², we extend derivation to the heterogeneous setting in Sec. 4.1 laying the groundwork to account for heterogeneity in DPO later on. In Sec. 4.2) we show DPO's policy aligns with Borda count and, in Sec. 4.3 highlight its limitations. While our analysis focuses DPO, similar insights extend to RLHF by substituting the policy with its induced reward.
4.1
Objective is Not the Expected Reward
We follow DPO's ' derivation from Sec. [ but with heterogeneous preferences. We show that the closed-form connection between 'T and r* no longer sufficient express the likelihood function. Starting from Eq. 49. the optimal policy is
(y) exp ( (r.
(y)
(10)
((x)
Define Ar" (y), y2; := r" (y2; u) r (y); u). Using the above solution, the policy ratios of n relate the expected difference in rewards as
' (y2)
Eu Arr (3/1, y2: w)] = Blog
3 log
(11)
Tee (3e(
(ee( (b1)
In the homogeneous case, Ar" was sufficient to describe the likelihood of y > yı However, with heterogencous preferences, E,, [Ar*] alone does not suffice to write the likelihood function in Eq. (8) Only under the following approximation as in Eq. (5), and minimize DPO's s loss to recover ,²":
express Pr(y22 > y1 r*) in terms of policy ratios,
Eu (Ar (y1,
(y1, (y1.322))) (): se o En (21. (E.,(rr(u) (21.);;)))
(12)
4.2 Ordinal Consistency with Borda Count
If DPO were the answer, what would the question be? We partially answer this question by an adaptation of a result from Siththaranjan al. (20]. First, define the Borda count follows.
Definition 4.1 (Normalized Borda count). For prompt æ. let D(. æ:) denote the distribution of alternative responses sampled for æ. The Normalized Borda Count (NBC) of y at x is the probability that annotator with random type prefers y over alternative response y' ~ D(. 1x)
NBC x) = (D--a())))))) Pr(y > y x
(13)
We next show that DPO's policy ratios are ordinally consistent with the normalized Borda count.

---

Borda count can significantly diverge from the wsir-heighted expected reward. This is studied under distortion in social choice theory [43]. Notably, NBC in Eq. (3) depends on D. Thus, while data collection is irrelevant to defining the optimal policy, it affects TDPO- Next, we illustrate two key Propasition 4.2. Suppose responses to a in the preference dataset are drawn D-)[x) and labeled according to BT. In the population limit, where empirical averages converge to expectations, DPO's (ppo(-))) has the same ordering over responses as NBC(. induced reward, or equivalently [ea())))
x)
See|proof on page 25 Prop. 4.2 also applies to the homogeneous setting. In this case, however, NBC aligns with r". It worth mentioning that DPO is not the only method consistent with NBC; identity preference optimization (IPO) 42 uses NBC as its objective. We next highlight key differences between NBC and the user-weighted expected reward along with DPO's drawbacks in practice.
4.3 Practical Drawbacks
differences between x² and TDPO through examples. Unless otherwise stated, we assume D(- x) and Te (- x) are uniform, and annotators follow BT. Refer to Appendix Affor further drawbacks of DPO (namely, minority suppression and IIA violation).
Mediocrity Promotion. Consider a summarization task with 4 = {.A, B, C}. where all types are equally represented. Type A strongly prefers longer summaries, type B strongly prefers shorter ones. and type C' slightly prefers coddium-lengthe summaries:
r (short:; A) 0, r* (med; A) 1, r" r'onng: A) 4,
r (short; B) = 4, r" (med; B) = =1, =1, r'oonnn; B) = 0,
r (shortt; C) = 0, r" (med: C) =1, (long; C)- 0.
In this case, n" (shortt = x (long) > 7" (med), but, NBC(short) = NBC(long) < NBC(med). Hence, DPO prefers medium-length summaries that are not strongly favored by any type.
Sensitivity to Preference Dataset Distribution. Suppose U B) and types are equally represented. Given three possible responses, type A prefers y1 but type B prefers y2:
y1 r" (y A) 6, r" (y2: A) 1, r" (y3; AA) 4,
(11: B) = 3, r" (22: B) = 9, r (ys: B) = 4.
One can verify that while D D(y)) D2y2)) increasing Dyys) from 0.02 to 0.04 changes aDpo 's preference from y/2 to y1- DPO's policy also sensitive to the preference model. Consider a variation of BT with a temperature of 2: 0=(=) == (1 exp(= For the same users and uniform sampling of alternatives, increasing the temperature from to flips TDPO 's ranking over yı and y/2 while the preference model has no effect on x² as expected from the optimal policy.
We have to emphasize that the dependence of NBC, and consequently TDPO, on the dataset sampling distribution D is not due to finite-sampl limitations or insufficient offline dataset support. This issue persists even with complete data coverage and in the limit of infinite data.
Real-World Examples. The examples above are not contrived; in real-world cases, NBC can produce rankings different from x² and is sensitive to dataset distribution extensively studied under distortion of social choice rules. To show this with a real example, we use Pew Research Center surveys and analyze question to 5101 participants: "The next time you purchase vehicle, how likely are you to consider purchasing an electric vehicle?" (options from A: very likely to D: not at all likely). We discuss how we select this question in Appendix B Responses come from groups of different political leanings: Republican (45%). Democratic (48%). and Neither/refused (7%).
Assuming the I Luce-Shepard model 33 (see Eq. (18) ). we estimate the reward for each group to calculate NBC and a user- weighted average reward. To find NBC, we use two distributions for alternatives: uniform distribution Dú and slightly altered distribution D. with 0.2 total variation distance (TV) from Du. As shown in Fig. I NBC (with Dt) ranks option C first despite its mediocrity: it is the second third preference for the three groups (see Fig. 7). In contrast, the user- weighted average reward favors D: the first and second preference for Republicans and the no-lean
'We can view NBC(y) x) as an aggregation of rewards at y. One can verify that NBC meets the order consistency condition of Prop. 3.1 However, it uses the reward value at y A to define the aggregated reward y and thus does not fall under Prop. 3.1 In fact, this interdependency causes the issues discuss Sec. 4.3
5

---

groups, respectively, Notably, altering Do to Da flips NBC's top ranking, highlighting its sensitivity dataset distribution. Similar discrepancies appear in other Pew surveys (see Appendix [B.
NBC (Da)
E
Very likely
Metric
E
Somewhat likely
NBC (Du)
C: Not too likely
D: Not likely
Avg Reward
E
Not purchasing
2
4
Rank
Ranking of different options in response to the question "The next time you purchase a vehicle, how likely are you to seriously consider purchasing an EV?" To find NBC, we use two distributions for alternatives: uniform distribution Dy and slightly altered distribution Du with 0.2 TV distance from Dy. NBC ranking is sensitive to the dataset distribution.
Figure 1:
5 Approximate Direct Alignment with Minimal Annotator Information
The failure of standard alignment methods to recover the optimal policy л² raises a natural question: it even possible to identify x²? Without annotator information, the answer no. To show this, suffices to prove that the ranking based the weightee expected reward is not learnable. This implies that nt is also not learnable since its induced reward corresponds to the expected reward by definition. We defer the formal definition of learnability to Def. F2 in the appendix, and based on we prove the following impossibility result:
Proposition 5.1. When the preference model o(-) is continuous, the ranking based the (user- weighted) expected reward is not learnable (according to Def. F.2) without annotator information.
See proof on page 26 Siththaranjan et al. 20 (Theorem. 3.4) presented version of Prop. 3.1 for
two alternatives and two types when (Ar) 1(rr > 0}. Procaccia et al. 44 (Theorem. 2.2) generalized this to BT. Prop. 5.1 presents fresh perspective by generalizing the impossibility to any continuous preference model and presenting multiple proof strategies, including one that draws on
robust version of Arrow's 's theorem 44
To circumvent the impossibility in Prop. 5.1 we must either relax the requirement of exactly identifying πt or collect some information from the annotators. This section focuses on the former, and the latter is the subject of Sec. 6 Next, we introduce an approximate alignment objective, along with the required annotator information and algorithms to solve it.
5.1 First-Order Approximation
The approximation in Eq. 12) is equivalent to using a zeroth-order Taylor expansion of (-)) around the average reward to calculate the likelihood function. To improve it, extend DPO by incorporating an additional non-zero term from the expansion, which we call (iooo--e zero)-ordorrr corrected DPO. The derivation is follows. Expanding a Ar* (3/1 Y2; u)) around AF* (y1, Y2) E,, Ar" (31, U2i u)] up to the second order gives the approximation below for the likelihood:
E,, (o(rr"() (y), 92: u))
(AF* (y1, v2)) + 2 0 (AF (31, y2)) Var. [Ar" (y)
y2) (y1,y255)
(14)
Here, the approximation omits OE,,, ((rr AI)))). We can see from this equation that the approximation in Eq. (2) is loose when is nonlinear and preferences have high variance.
To calculate Eq. [4). we can substitute A by the difference in log policy ratios (Eq, (T)). We then need to estimate the variance term. Sec.
Sec. 5.33 offerrs variance estimator. Once the variance is estimated by a function ( Y2), first-order corrected DPO estimates Pr(y > y1 x) using
o (h( y2i + @ o" (h(3), (1(1)133;;;)) m)) ( y2)
2
Here, & > determines the strength of correction, and
π (y2)
π ( Teet (
h 1 Y2i π) = log
3 log Teef (y2)
(15)

---

denotes the difference of iT's induced rewards. Given Pr(y2 >
πT) and a preference dataset D, we can maximize the log-likelihood similar to Eq. O For numerical stability, we use stable logarithm log(2) 2) == log m <t) in computations. Note that our theory suggests a = while DPO uses 0. Our empirical findings in Sec. [.1] that larger values of a improve the effectiveness of the correction. We next discuss the estimation of V yy))
5.2 Impossibility without Annotator Information
If we limit our algorithms to M-estimators, which encompass most practical learning methods, consistent estimation of the variance term is impossible with anonymous data:
Proposition 5.2. There exists no consistent M-estimator that without annotator information can estimate V(x, y1 82)
Var Ar" (a. y1. y2: u)
See proof on page 28 While Prop. 5.1 already implies that x' unlearnable without annotator
information, Prop. 5.2 goes further, showing that even first-order improvement to DPO is practically
impossible. Next, we show how minimal annotation information can overcome this impossibility.
5.3 Using Paired Preferences
We get around Prop. 5.2 by collecting minimal information on annotators. Consider dataset 'D of pairs of preferences in the form {(x yı, 32. 0), (x', yí, V/2 o')} where = ( > y1 and of (yy ví are labeled by the same person. Using D. we can train a joint likelihood model (x, yı, y/2 x',y'',,, joint likelihood model consistently estimates
Vn, 322
by minimizing cross-entropy between J and (o d') as the label. The
(x, yı, Y2, x', Ví v2) = E., oArr yı, yo; u)) o(Ar" v 322 v22 w))
We next show that this is in fact sufficient to estimate the variance term:
Lemma 5.3. Using JI and J2 as shorthands for (æx y y2, 2, yı, y2) and J (ax yr, 12,92,, x, y2, v1). we can use the following consistently estimate the variance term:
J1 (J1 + J2)²
V V 92) = d'(AF' (y1, y²))²
(16)
Note that substitute AF* (y1. U2) in terms of log policy ratios from Eq. [I]. Thus, we have all the elements to calculate V in Eq. (16). This completes our derivation of first-order corrected DPO.
See proof page 28
6 Direct Alignment with Maximum Annotator Information
Recall that learning the optimal policy n" from anonymous data is impossible, and an approximate improvement to DPO requires minimal information about the annotations. But what if we collect richer data? Can we design direct alignment method that consistently Iearns the optimal policy? To explore this, we consider a dataset where every sample is labeled by representatives of all user types. We show that consistent direct alignment is possible in this setting but at the cost of sample efficiency.
Suppose user types are a finite set l and equally represented. This assumption makes our negative results stronger. Consider rich data collection: For every query and candidate responses (x, y/, 82). we collect one preference data point from each user type. Let E {0, 1 be the vector that indicates preferences where o, if y2 y by user of type and 0 otherwise. Given such dataset D with query, responses, and preferences as (æ,, y1 y2, o), our goal is to design a loss
L(D;T) = D
Σ
1(x, y1, y2. o)
(17)
(0..)) (2.VI.22.0)ED
such that arg min, L(D; T) is a consistent estimator of π². Designing such loss is, in fact, possible. either all one or
For instance, suppose we only look into the agreement cases in D where zero. Conditioned on agreement, we will show that the probability of y2 > y1 is proportional to exp (E, Ar" (y" (y1,92;") u)). We can express this likelihood directly in terms of log policy ratios
of n (see Eq. [I] We formally show this possibility through temperature-adjuste DPO:
7

---

Proposition 6.1. Defining in Eq. (17) as follows results in a consistent estimation of the optimal policy when preferences follow the BT model:
log σ [M] h1y1, y2; T)
0 = 1,
Uy. y2. 0;
log a (14) hyy2, y1:
0 0
O.W.
Here, h is the difference of πt's induced rewards (Eq.
15
) and (0) is the vector of all ones (zeros).
See Srelp on page 29 Consistent loss functions are not unique. We give another example in Prop. F3
and systematic way to find such losses in Lemma G.2 In both examples, loss functions reduce to the standard DPO loss when [2|] 1.
While the loss function in Prop. 6.1 benefits from consistency, it only uses samples where all user types have agreed. In other words, it discards sample with any disagreement. A natural question arises: Can we design a loss function that uses all data, including those with disagreement, while maintaining consistency? Surprisingly, the answer is no:
Theorem 6.2. Suppose I in Eq. (17) only depends on (xz yr y2) through and Tret. Ifthere are more than three types of user and the preferences follow BT, any loss that allows a consistent estimation of the optimal policy discards samples with disagreement, i.e., those with o & {0,1}.
See pproof on page 29 This theorem highlights tension: To improve efficiency, one must compromise either consistency or direct optimization. The approximate direct alignment method proposed in Sec. 5 exemplifies forgoing consistency. Next, we discuss an alternative that favors consistency.
An Indirect Practical Solution: Averaging Personalized Rewards. The tradeoff between sample efficiency and consistency arises from the requirement for direct optimization. To regain sample efficiency, we may relax the requirement for direct aligmment by training reward models while still avoiding RL. Specifically, we can learn personalized reward models r(:: u) for different user types и E Lt, calculate user-weighted expected reward, and use it to relabel a preference dataset. A dataset labeled with this average reward makes any direct alignment method applicable and is found effective in practice [45]. It is both consistent and sample-efficien when personalized reward learning is feasible, but comes at the cost of additional training and memory for each user type.
7 Experiments
We provide empirical evidence for our claims throughout the paper. We first extend our sensitivity example in Sec. 4.3 to real-world preference dataset in Fig. 4 Using 19 Pew surveys, we find that NBC rankings change under shifts from uniform sampling in 20% of cases. Notably, these changes require only modest shifts: In half the cases, total variation distance of less than 0.23 from uniform is enough to alter the rankings. We defer the details to Appendix B
we simulate DPO and our proposed improvements in small-scale environment where we can visualize and compare the resulting policies. Finally, we scale this experiment in Sec. 7.2 by fine-tuning large language models, illustrating the extent of possible improvement over DPO.
In Sec. 7.1
Code for reproducing all results is publicly available at https::///// httts://giithb. com com/arasshnn//aahg
7.1 Synthetic Experiments
We generalize the discrete environment of Xu al. [46] with multiple user types. This enables us to visualize the differences between DPO's policy and the optimal policy, as well as to evaluate the effectiveness of applying first-order correction (Sec. 5) and using consistent loss function (Sec. 6)
Environment. A prompt x can take a value from to n. There are also only n possible responses to each The reward for responding y for type is r r"x y]: u) R., (dist(x + u, y)), where dist is a circular distance, and R, is a linearly decreasing function floored at zero. We set 40 and consider three equally represented types 21 {-10, 0, 10) with BT annotators.
Since the reward (and thus the policies) depends only on y - x. we can reduce everything to a 1D representation by setting 8 == y x and averaging over For example, for policy n, define a ID
8

---

policy (6)) = E across Fig. 2(a) shows our choice of rewards well the expected reward across user types.
sE(N] ((x 6 z). We also compute standard errors of these 1D representations
Policies. For uniform Tref and 3 1, Eq. 10) implies x (y x)) exp (1 Enezi 1" ([zz y) u)). We generate a large dataset of preferences under uniform context and alternative distributions and use the Adam optimizer minimize the loss for different methods. For the first-order correction of DPO, we additionally train a joint likelihood model J to estimate the variance term from Eq. (T6). We use the loss from Prop. 6.] as our choice for the consistent loss.
Results. Fig. 2(b) presents тDPо along with x and NBC. Unlike the optimal policy which prefers around 10 and 10, DPO prefers 5 = 0, and to large extent is ordinally consistent with NBC.
Fig. 2(c) shows that increasing correction strength brings the corrected DPO policy closer x², In particular, at a 1, the corrected DPO already favors alternatives with 5 E {-10, 103, consistent with n². Furthermore, increasing a makes these alternatives even more favorable. In the full- information setting, Fig. [[d] shows that minimizing the consistent loss largely leads to π². Note that minor deviations from theoretical derivations are likely due limited data and imperfect optimization.
We have also simulated noisy annotations, where with a small probability, a random user provides the annotation instead of the intended user type (see Appendix C Our results show that consistent loss minimization is largely robust to such noise, although this does not address its sample inefficiency in settings with many user types.
10
u 10
0.5
r110:00
4(r')
0.03
NBC16)
m(6)
NBC
0.02
Morr
0.4
0-
-20
0
-20
(a) Rewards
(b) DPO vs. optimal
0.03
0.03
m()
Mbrco
Rondbteet
0.02
mopa
0.02
-20
20
-20
20
Corrected DPO
(d) Consistent loss minimization
Figure 2: Rewards and policies in the synthetic setup.
7.2 Semi-Synthetic Experiments
While we have shown that DPO does not maximize the (user-weighted) average reward, the extent of its deviation from the optimal policy remains unclear. To assess the potential gains from accounting for heterogeneity, we compare DPO with a policy that minimizes the consistent loss function of Prop. 6.1 in a more realistic setup. Specifically, we use LoRA [47] to fine-tune Llama-3-8B [48) for both reward learning and direct alignment two relabeled variations of the HH-RLHF dataset [49]. To simulate heterogeneous preferences, we define three user types with distinct length- based rewards (see Appendix for for details and similar results using Qwen-2.5-7B).
Results. We use agreement with the ground-truth average reward on the test set as the success metric. Using standard reward learning and DPO on an anonymous preference dataset daeeeet-where each sample is labeled by a random user type-ttteeeeee the induced ranking agrees with the average reward in 89.6% and 67.4% of test cases, respectively (Fig. blue). When we use consistent loss with full annotator information, agreement improves to 93.9% for reward learning and 71.7% for direct alignment (Fig. 3 red). In summary, explicitly accounting for heterogeneity improves agreement with the average reward by 4.3 percentage points, highlighting the substantial room for improvement left by DPO.

---

Modeling Heterogeneity
Ignoring Heterngenelty
Reward Learning
Direct Alignment
95
75
Accuracy (80)
90
70
Figure 3: the presence of preference labels from every user type, our proposed loss function produces reward models (left) and aligned policies (right) that are more consistent with the average reward across user types, compared to typical approaches that overlook heterogeneity. Bars show the mean, and whiskers denote the second and third quartiles across five random seeds.
Related Work
Aligning models to "serve pluralistic human values" [25] can involve personalization to the user's
specific reward [12 or aggregation of diverse rewards 18 19] The latter, which is the subject of our study, can use insights from social choice theory 24
Closest to our work, EM-DPO [8] simultaneously learns the distribution of user types and their corresponding policies. However, EM introduces significant complexities and lacks formal guarantees. Moreover, the identifiability of types requires additional assumptions. MODPO [50] applies DPO for each user type while utilizing estimated rewards from other user types to maximize a linear combination of rewards. Neither method obtains policy by directly minimizing a loss over preference data. For a more extensive review of related work, refer to Appendix
Discussion and Conclusion
Aligning single policy to the average reward across user types requires collecting annotator infor- mation. This can range from minimal information such as linking two instances labeled by the same annotator, to richer information like using questionnaires to infer annotator types. We improved DPO using the former and introduced consistent loss when annotators from all user types label every data point. With additional assumptions, unsupervised methods might be able to identify annotator types from anonymous datasets [51]. Further research should explore the additional structures that, when used during data collection, can help with identifiability.
Our results revealed tension between consistency and sample efficiency in direct alignment. Thus, an alternative approach, i.c., individual reward training and aggregation, may be more practical for addressing heterogeneity when individual rewards are identifiable.
We use the average reward as the natural choice among aggregations that give a well-defined alignment problem. However, the choice of aggregation is inherently a social and policy question rather than purely technical one. In certain contexts, the policymaker might prefer to give higher weight to disadvantaged people to address issues such inequality. Additionally, in some cases, the very existence of a reward function may be questionable, requiring objectives to be defined in terms of choice probabilities rather than rewards.
We believe that trained policies should not be used to elicit or represent aggregate preferences, even when reward aggregation is appropriate and estimation is consistent. While such policies may capture certain patterns in user behavior, they do not necessarily reflect the underlying interests or values of the population. In other words, view the resulting policy as functional tool for decision-makinn rather than a true representation users' collective interests.
In summary, while preference heterogeneity is well recognized in mathematical psychology, standard methods often bypass this complexity. As we showed, accounting for heterogeneity, even when the goal remains the same as in the homogeneous setting-to derive single policy- can render common techniques inefficient or inapplicable. Understanding these limitations calls for new approaches that explicitly incorporate heterogeneity while balancing efficiency, consistency, and practicality.
10

---

Acknowledgment
We thank Mohammad Alizadeh, Pouya Hamadanian, Moritz Hardt, Erfan Jahanparast, and Itai Shapira for discussions and feedback on earlier versions of this paper. Abebe was partially supported by the Andrew Carnegie Fellowship Program. Procaccia was partially supported by the National Science Foundation under grants IIS-2147187 and IIS-2229881; by the Office of Naval Research under grants N00014-24-1-2704 and 100014-25-1-2155 and by a grant from the Cooperative Al Foundation.
References
[1] Lora Aroyo and Chris Welty. Truth lie: Crowd truth and the seven myths of human annotation. Al Magazine, 36(1):15-24 2015.
[2] Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677-694, 2019.
[3] Remi Denton, Mark Díaz, Ian Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen. Whose ground truth? Accounting for individual and collective identities underlying dataset annotation. arXiv preprint 0tti::2//2.0055555 2021.
[4] Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Jezek. Why don't you do right? Analysing annotators' disagreement in subjective tasks. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2428-2444 Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/20233 www.0223eall-mainnn 178...18.. URL wttg//22..eeeeeoe http:/ogyyor///2..aallmlmaam main. 178/
[5] Michael JQ Zhang, Zhilin Wang, Jena D Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXiv:2410.1440 2024.
[6] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint wwX.vv1100.085000 2019.
[8] Nisan Stiennon, Long Ouyang, Jeffrey Wu. Daniel Ziegler, Ryan Lowe, Chelsea Voss. Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3000 2020.
[9] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances in Neural Information Processing Systems, 35:27730--7744,, 2022.
[10] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[11] Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman E Ozdaglar. RLHF from heterogeneous feedback via personalization and preference aggregation. In ICML. 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
[12] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023.

---

[13] Xinyu Li. Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05113, 2024.
[14] Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization. In The eighth Annual Conference on Neural Information Processing Systems, 2024. URL / recsheQ7e8
met/fooum?iii
[15] Allison Lau, Younwoo Choi, Vahid Balazadeh, Keertana Chidambaram, Vasilis Syrgkanis, and Rahul G Krishman. Personalized adaptation via in-context preference learning, arXiv preprint arXiv:2410. 14001, 2024.
[16] Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized preference fine-tuning of diffusion models. arXiv preprint arXiv:250100665) 2025.
[17] Nishant Balepur, Vishakh Padmakumar, Fumeng Yang. Shi Feng. Rachel Rudinger, and Jor- dan Lee Boyd-Graber. Whose boat does it float? Improving personalization in preference tuning via inferred user personas, 2025. URL https::s:///artt wttp://www...rraarb......1...... 11549
[18] Keertana Chidambaram, Karthik Vinay Seetharaman, and Vasilis Syrgkanis. Direct preference optimization with unobserved preference heterogeneity, 2024. URL
[19] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024.
[20] Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional prefer- ence learning: Understanding and accounting for hidden context in RLHF. arXiv preprint arXiv:2312.08338, 2023.
[21] Lucas Monteiro Paes, Carol Long, Berk Ustun, and Flavio Calmon. On the epistemic limits of personalized prediction. Advances in Neural Information Processing Systems, 15:1979-1991, 2022.
[22] Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pages 1-10, 2024.
[23] Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, al. So- cial choice should guide Al alignment in dealing with diverse human feedback. arXiv preprint arXiv:2404.100 10271, 2024.
[24] Luise Ge, Daniel Halpern, Evi Micha, Ariel D Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. Axioms for AI alignment from human feedback. Advances in Neural Processing Systems, 36, 2024.
[25] Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A roadmap pluralistic alignment. In Forty-first International Conference on Machine Learning, 2024. URL https://oppnnrreeeeeeee eview. net meet/forummiin güpBnRHwxM
[26] Vincent Dumoulin, Daniel D Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann Dauphin. A density estimation perspective on learning from pairwise human preferences. arXiv preprint arXiv:2311. 2023.
[27] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online AI feedback. arXiv preprint 2oXi:22402.04792. 2024.
[28] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
12

---

[29] Tomasz Korbak, Ethan Perez, and Christopher Buckley, RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computationall Linguistics: EMNLP 2022, pages 1083-1091, 2022.
[30] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLiC- HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 2023.
[31] LL Thurstone. law of comparative judgment. Psychological Review, 34(4):273, 1927.
[32] Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959.
[33] John I Yellott Jr. The relationship between Luce's s choice axiom, Thurstone's 's theory of compar- ative judgment, and the double exponential distribution. Journal of Mathematical Psychology. 15(2):099-14, 1977.
[34] Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 394344)) 1952.
[35] Ali Shirali, Reza Kazemi, and Arash Amini. Collaborative filtering with representation learning the frequency domain. Information Sciences, 681: 2024.
[36] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. Management science, (((9):6336-6355 2024.
[37] Ali Shirali. The burden of interactive alignment with inconsistent preferences. arXiv preprint arXiv:2510.16368, 2025.
[38] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University. 2010.
[39] Roger N. Shepard. Stimulus and response generalization: A stochastic model relating generaliza- tion to distance in psychological space. Psychometrika, 22(4):325-345 December 1957. ISSN 1860-0980. doi: 10.10077/F02228967. URL httptttttt/:....oot
[40] Mamoru Kaneko and Kenjiro Nakamura. The Nash social welfare function. Econometrica:
Journal of the Econometric Society, pages 423-433 1979.
[41] Ariel D. Procaccia, Benjamin Schiffer, and Shirley Zhang. Clone-robust AI alignment. arXiv preprint arXiv:2501.09254, 2025.
[42] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4457-4455 PMLR, 2024.
[43] Elliot Anshelevich, Aris Filos-Ratsikas, Nisarg Shah, and Alexandros A Voudouris. Distortion in social choice problems: The first 15 years and beyond. arXiv preprint arXiv:2103.00911
2021.
[44] Ehud Friedgut, Gil Kalai, and Assaf Naor. Boolean functions whose fourier transform is concentrated on the first two levels. Advances Applied Mathematics, 29(3):427-433 2002.
[45] Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to evaluate reward models for RLHE, arXiv preprini arXiv:2410. /4872, 2024.
[46] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Forty-first International Conference on Machine Learning, 2024.
13

---

[47] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL httts::/:////peereeee
[48]
A Meta. Llama 3 model card. 2024. URL https://gitub..comm/eee/e/////ee
CARD. md
[49] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint orXiv:224400866. 2022.
[50] Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang. Wanli Ouyang, and Yu Qiao. Beyond prefferecce-fit-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Compatational Linguistics ACI. 2024, pages 10586-10613, 2024.
[51] Xiaomin Zhang, Xucheng Zhang, Po-Ling Loh, and Yingyu Liang. On the identifiability of mixtures of ranking models. arXiv preprint arXiv:2201.13132 2022.
[52] Pew Research Center. About pew research center, 2025, URL attps://wwiz www.e/weewewrreerrccccccccccccch org/about/ Accessed: 2025-01-20.
[53] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpps://opperreiiiieeeeee httpe/rrreiiie.eeffrrrrnr
idBBkgöRiCqy7
[54] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orally Lewis Hammond, Eric J Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwani, Yoshua Bengio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Maharaj. Jakob Nicolaus Foer- ster, Florian Tramer, He He, Atoosa Kasirzadch, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research, 2024. ISSN 1835-8856.. URL https: btttps://openrvviiie net/ wwt.ferum/ffiit..mmm oVTk0s8Pka] Survey Certification, Expert Certification.
[55] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Wadfield-Menee Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. U /tt.ltttsss///pneereeet /ttp:////eeeeeeeeeee net/ forum?1d=bx24Kp144B Survey Certification, Featured Certification.
[56] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct Nash optimization: Teaching language models self-improve with general preferences. arXiv preprint arXiv:2404.03775. 2024.
[57] Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J Andrew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767 2024.
[58] Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, and Wenpin Tang. Mallows-DPO: Fine-tune your LLM with preference dispersions. arXiv preprint 14545 2024.
14

---

[59] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference on Machine Learning, 2024. URL httpps:/openreeveew net/ forum?id=guid forumm1d=guSsaacAAAt
[60] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. UtL.:tttps:/oppenreeei http:pp:e/pppeerrrreeeeee/rrrrrr wtt..e/orrrmmii-11oottt.....oe
[61] Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Person- alizing reinforcement learning from human feedback with variational preference learning. In The eighth Annual Conference Neural Information Processing Systems, 2024. URL
https:// www.neeeuuunn senerrummdd-guuee -
[62] Daiwii Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv:2406.08400 2024.
[63] Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized preference fine-tuning of diffusion models. arXiv preprint arXiv:2501.06655, 2025.
[64] Tianyi Qiu. Representative social choice: From learning theory to Al alignment. arXiv preprint arXiv:2410.33933 2024.
[65] Parand A Alamdari, Soroush Ebadian, and Ariel D Procaccia. Policy aggregation. Advanced in Neural Information Processing Systems, 36, 2024.
[66] Jessica Dai and Eve Fleisig. Mapping social choice theory to RLHF arXiv preprint arXiv:2404. 13038, 2024.
[67] Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A mini- maximalist approach to reinforcement learning from human feedback. In Forty-first Interna- tional Conference on Machine Learning, 2024. URL https://opeerevieww www.arrwwwwwe.eeeoouumn...m net/forum?
id=5kVgd2MuMY
[68] Hadassah Harland, Richard Dazeley, Peter Vamplew, Hashini Senaratne, Bahareh Nakisa, and Francisco Cruz. Adaptive aligmment: Dynamic preference adjustments via multi-objective reinforcement learning for pluralistic AI. arXiv preprint arXiv:2410.233600, 2024.
[69] Haoxiang Wang. Yong Lin, Wei Xiong, Rui Yang. Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.18571, 2024.
[70] Huiying Zhong, Zhun Deng, Weijie Su, Zhiwei Steven Wu, and Linjun Zhang. Provable multi- party reinforcement learning with diverse human feedback. arXiv preprint 20X5::2403.05006 2024.
[71] Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu. Aligning crowd feedback via distributional preference reward modeling. arXiv preprint arXiv:2402.09764. 2024.
[72] Ryan Boldi, Li Ding, Lee Spector, and Scott Nickum. Pareto-optimal learning from preferences with hidden context. arXiv preprint arXiv:2406. /5599, 2024.
[73] Alexandre Rame, Guillaume Couairon, Corentin Dancette. lean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthicu Cord. Rewarded soups: Towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
(74) Angelica Chen, Sadhika Malladi, Lily Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. URL. https://openrevies: wtt.://www.....ooooororttttttaaa
15

---

[75] Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On diversified preferences of large language model alignment, 2024. URL
petps://arxivvorg... abs/2312 07401
[76] Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models, In The Tweifth International Conference on Learning Representations, 2024. URL httpt::/
[77] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Perey Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971-30004 PMLR, 2023.
[78] Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, al. Fine-tuning language models find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:381 11138876-38118 2022.
[79] Liwei Jiang, Taylor Sorensen, Sydney Levine, and Yejin Choi. Can language models reason about individualistic human values and preferences? arXiv preprint arXiv:2410.0386,, 2024.
[80] Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personallim: Tailoring LLMs individual preferences, 2024. URL https:://aarrtt :/apxii////xxv...//// org/ abs/ abs/2409. 20296
16

---

A Additional Drawbacks of DPO under Heterogeneity
Violating Independence of Irrelevant Alternatives (IIA), Suppose U = {A, B} and types are equally represented. Given two possible responses y1 and y2, type A prefers y but type B prefers y2:
r' (y1 A) = 6, r" (32: A) = 1;
(y1; B) = 3, r" (y/; B) 9.
direct calculation shows E,, (32) > E. EuIr" (1)) ) and NBC(y2) > NBC(g)). So, both n'
and TDPP prefer y/2. Let's consider another possible response ys which is not the preferred response for any user type:
r (ya; A) r" (ya; B) 2.
While n" still prefers to yı, now NBC s 0.62 > NBC(y²) ~ 0.55, so, introducing an irrelevant alternative can alter DPO's ranking over existing alternatives.
Tyranny of Majority. Suppose U = {A, B} with type shaping 90% of the population. Given two responses yl, 1/2 type A slightly favors 3/2 but type B finds y/ offensive:
y A) 0.5, r" (32; A) = 1,
y1 B) = 0.5, r" (y2; B) = 10.
In this case, x² prefers yı even though type B a minority. In contrast, we have NBC( x NBC(g) ~ 0.53, which implies that the majority dominates in DPO.
B PEW Surveys Experiments: Details and Additional Examples
In this section, we expand on our PEW surveys experiment where we used polling data on key political and social issues show: (i) how NBC rankings can differ from those maximizing the average reward: (ii) How sensitive NBC is to the sampling distribution of the pairwise preference data.
Data. We use several Pew Research Center surveys, specifically the American Trends Panel surveys number 35, 52, 79, 83, 99, 109, 111, 112, I 114, 119, 120, 121, 126. 127, 128, 129, 130, 131, and 132. The choices are a mix of recent surveys and those relevant to science, technology, data and AL. Each survey include questions asked to thousands of participants. We categorize participants by political party leanings to define types. When processing the questions, we discard responses that are empty, as well as discarding the option "Refused". We note that discarding the option "Refused" had no effect on the results as it is not frequently chosen.
Reward Estimation. Although we observe how often each group selects a particular option, we don't directly observe respondents' internal rewards. To estimate this, we apply the Luce-Shepherd
model 39 32
exp (r(i; u))
Pr (option i is chosen from S) = ESes exp (r(j:u))*
(18)
where S is the set of options, and w) is the reward for type u. This allows us to estimate each option's reward (up to a constant additive term) for each type. From these estimates and observed probabilities, we compute both the expected reward and the NBC metric, where in the latter we assume the uniform probability for alternatives unless specified otherwise.
NBC Sensitivity to Sampling Distribution: Results. Recall from Sec. 4.2 that the common practice
of alignment assuming homogeneity results in ordinal consistency with NBC. Here, we analyze the sensitivity of NBC to the distribution of pairwise preference datasets in real-world cases by using Pew surveys [52] the same dataset used in Sec. 4.3 Specifically, we address two questions: (i) Across the questions in the Pew surveys, how often would NBC rankings change when the sampling distribution of alternatives varies (while retaining support over all alternatives?? (ii) How much must the sampling distribution deviate from uniform to alter NBC rankings?
To answer these questions, we first estimate the reward of each option in each question (see Sec. 4.3
and Appendix B for further details). Given the rewards, we can calculate NBC under any sampling distribution using Eq. (13).
17

---

For question (i), among 1519 questions from 19 Pew surveys, NBC rankings change due to changing the sampling distribution from uniform 20% of cases (306 questions), with the preferred choice changing in 136 cases. For question (ii), we find that modest change in the sampling distribution suffices; half the cases, total variation (TV) distance of less than 0.23 from uniform alters the rankings. The cumulative distribution function (CDF) of the minimum TV distances required to change NBC rankings is shown in Fig.
Further experimental details are in Appendix
1.04
8 0.5
0.0
0.5
1.0
TV distance from uniform
Figure 4: CDF of the minimum TV distances (from uniform) required to change the NBC order in the Pew surveys. A change of 0.23 is sufficient to change the order in half the questions.
NBC Sensitivity to Sampling Distribution: Implementation Details. Above, discussed the sensitivity of NBC rankings to the sampling distribution of pairwise preference data. Here, we discuss the implementation details.
Recall that NBC is defined as:
NBC(y; = EyroD( Pr(y > y [;r)].
To compute this, we first estimate the reward function r, then evaluate Pr(y y';r) for all (u. y') E V Y, where Y is the set of alternatives. Next, consider the feasibility of swapping the ranking induced by the uniform distribution Dy for alternatives y; and yy with new distribution Da, assuming NBC(y; Du) NBC(y,; Du). This is equivalent solving the following linear program:
minimize
- 1 s, 2
T
subject to:
q > e1,
1
s> 1 q. N 1
s -1. 1. N P.q < +
1 1'q=1,
q>0. 0.
Here, 1)%, and labeling the alternatives as yl,...: yN. we define P.j Pr(y: > y;;r), = D. and P; as the k-th row of P. The parameter > 0 ensures support for all alternatives, and controls the required magnitude of change NBC beyond what is required for the swap. We set ö 10-5
To compute the minimum TV distance, we solve the program for all pairs (i,j) where NBC(y;; Dv) > NBC(y,; Du) and record the smallest objective value. In this analysis, we group respondents by political leaning (specifically. the column MARRYSUM_FINALL We also note that in this analysis we exclude survey questions with fewer than three options.
Additional Examples. We highlight some the examples of discrepancies that we find in some of the surveys listed above in Figures 5 6 & and 9 We note though that while we indeed find few examples the discrepancy, NBC rankings is actually aligned with average reward rankings in most cases. One possible explanation for this is that in many questions, the distributions of responses across types were very similar, suggesting that homogeneous reward model would have been appropriate, causing NBC and average reward to align.
18

---

Rewards per Type
Reward
c
DemiLaan Dem
RepiLean Rep
doRedusedNo lean
Rank
3
A
(a) Rewards per Type
0.5
E
Values
0.0
-0.5
Avg Reward
-1.0
NBC
D
Avg Reward NBC
A
B
E
Options
Metric
(b) Avg Reward NBC
(c) Ranking
Figure 5: Do you think the plans and policies of the Biden administration will make the country's response to the coronavirus outbreak: A: lot better; B: little better; C: Not much different; D: A little worse; E: A lot worse
Rewards per Type
C
Reward
B
A
Rank
(a) Rewards per Type
0.5
D
Values
0.0
-0.5
Avg Reward
NBC
A
-1.0
Avg Reward NBC
Options
Metric
(b) Avg Reward vs. NBC
(c) Ranking
Figure 6: How would you rate the job Joe Biden is doing responding to the coronavirus outbreak? A:
Excellent; B: Good; C: Only fair; D: Poor
19

---

Rewards per Type
1.0
c
-0.5
-1.0
Rep/Lean Rep
DemLLea Dem
DKReefssedNNo leen
Rank
B
(a) Rewards Type
0.6
0.4
=
Avg Reward
E
Values
0.2
NBC
0.0
-0.2
A
-0.4
A
B
E
Avg Reward NBC
Options
Metric
(b) Avg Reward NBC
(c) Ranking
Figure 7: The next time you purchase vehicle, how likely you seriously consider purchasing an electric vehicle? A: Very likely; B: Somewhat likely; C: Not too likely; D: Not at all likely; E: I do not expect to purchase vehicle
Rewards per Type
Reward
ReptLaan DomsLean
lean
Rank
(a) Rewards per Type
1.0
0.5
A
Values
0.0
-0.5
Avg Reward
-1.0
NBC
E
E
Avg Reward NBC Metric
Options
(b) Avg Reward NBC
(c) Ranking
Figure 8: What is your overall opinion of Kamala Harris? A: Very favorable; B: Mostly favorable; C: Mostly unfavorable; D: Very unfavorable; E: Never heard this person.
20

---

Rewards per Type
Reward
c
ReptLean Rep
DemiLean Dem
doRdduse wann
Rank
B
D
(a) Rewards per Type
A
A
Values
Avg Reward
-2
NBC
E
3
Avg Reward NBC
C Options
E
Metric
(b) Avg Reward NBC
(c) Ranking
Figure 9: What is your overall opinion of Joe Biden? A: Very favorable; B: Mostly favorable; C: Mostly unfavorable; D: Very unfavorable; E: Never heard of this person.
C Robustness to Noisy Preferences
We extend our analysis by simulating noisy user preferences in the synthetic setup of Sec. 7.1 For noise level n < 1, each type-u user's preference replaced by that of a random user type with probability n. We compare the learned policies under varying noise levels with the optimal policy (OPT), reporting total variation (TV), Kendall's т, and Spearman's r.
As shown in Table 1 the TV between the noisy Transisent and OPT drops below that of DPO and OPT only high noise levels (n. 2 0.7). Even in these cases, Teonsistett remains ordinally closer OPT than DPO's policy, suggesting that the benefits of consistency may be robust to annotation noise.
We note, however, that this result is limited to synthetic settings and does not address the sample inefficiency direct alignment methods which our key impossibility result still discourages their use in settings with many types.
Table 1: Comparison of DPO and Weomsistent under different noise levels in the synthetic environment (Sec. 7.
Metric
DPO
Teonsistent under noise level
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
TV
0.089
0.023
0.044
0.066
0.073
0.081
0.084
0.088
0.090
Kendall's 7
0.526
0.933
0.888
0.757
0.710
0,668
0,649
0.618
0.563
Spearman's r
0.676
0.989
0.976
0.912
0.863
0.814
0.811
0.762
0.727
21

---

D Semi-Synthetic Experiments: Fine-Tuning Llama-3-8B and Qwen-2.5-7B on HH-RLHF
Reward Models. Fig. shows the three distinct rewards we use for the three user types along with their average. In order to have reliable ground-truth reward which we can rely on in evaluation, we define these rewards as functions of the number of tokens in prompt-response combinations.
Anonymous Dataset. We prompts and response pairs from both helpfulness and harmlessness subsets of Anthropic's HH-RLHF dataset [99 and relabel the chosen and rejected responses manually. We filter for data points in which the sum of the number of tokens in the prompt and the number of tokens in the longer response do not exceed 512. This leaves us with 160, 800 training and 17, 104 test data points. For every data point (a prompt with pair of responses), we sample one of the three user types uniformly at random. Given the type of user, we sample a preference based on BT (34) to label the alternatives.
Dataset with Maximum Annotator Information. We use prompts and response pairs from both helpfulness and harmlessness subsets of Anthropic's HH-RLHF dataset [49] and relabel the chosen and rejected responses manually. We filter for data points in which the sum of the number of tokens in the prompt and the number of tokens in the longer response do not exceed 512. This leaves us with 160, 800 training and 17, 104 test data points. For every data point (a prompt with a pair of responses), we keep sampling BT (34) preferences from all user types until they agree with each other. Once the consensus is achieved, we stop sampling and use the agreed-upon preference the label for this data point.
Fine- Tave-Tunnng Details. We fine-tune Llama-3-8B 48 base model with LoRA 47 We fine-tune for one epoch with batch size of 2, and use linear learning rate schedule that starts with 3 10 and decreases to zero. We use the Adam optimizer with a weight decay 0.001 (53) Regarding LoRA's hyper-parameters,
use the matrix rank of r 8, 32, and the dropout probability of 0.1.
For direct aligmment experiments, we use uniform reference policy. When ignoring heterogeneity, we do vanilla DPO over the anonymous dataset. When modeling heterogeneity, we use the loss function we propose Prop. 6. over the dataset with maximum annotator information. We use the ordinal agreement between the ground-truth truth average reward and the reward induced by the aligned policy as the measure of accuracy.
For the reward learning experiments, we fine-tune the Llama-3-8B as a reward model. When ignoring heterogeneity, we assume BT and maximize the probability of the anonymous preference dataset under the learned reward model. When modeling heterogeneity, we use the loss function in Prop. 66. over the dataset with maximum annotator information, but replace h y25 T)) with the difference rewards, i.e., r(y2) r(y1). ). We use the ordinal agreement between the ground-trum average reward and the learned reward as the measure of accuracy.
Detailed Results. We conduct every experiment with five different random seeds. Fig. 3 shows the average, t5th percentile, and 75th percentile of accuracy across the five random seeds. Tables and show the raw accuracy numbers across the five random seeds.
number tokers
Figure 10: Reward definition for three user types in semi-synthetic experiments (Sec. 7.2) based on the length of prompt response combination. The first user type prefers long prompt response combinations, the second user type prefers short prompt response combinations, and the third user type prefers mid-length prompt response combinations. The dashed cyan line shows the average reward across the three user types.
22

---

Table 2: Accuracy (%) in Alignment Experiments (Llama-3-8B)
SEED
IGNORING HOMOGENEITY
MODELING HETEROGENEITY
65.61
66.63
67.55
69.55
2
68.77
75,21
3
68.70
72.04
66.28
74.95
Table 3: Accuracy (%) in Reward Learning Experiments (Llama-3-8BB)
SEED
IGNORING HOMOGENEITY
MODELING HETEROGENEITY
92.33
95.26
1
85.38
92.01
2
88,46
94.6
89.69
93.57
91.94
93.87
Table 4: Accuracy (%) Alignment Experiments (Qwen-2.5-7B)
SEED
IGNORING HOMOGENEITY
MODELING HETEROGENBITY
0
68.66
72.1
63.68
69.77
2
62.48
68.87
63.80
71.57
4
60.06
66.72
AVERAGE
63.74
69.81
Fine-Tumme Tuning Qwen-2.5-7B. Using the same setup, we repeat the alignment experiment with Qwen- 2.5-7B. As shown in Table incorporating annotator information and explicitly modeling hetero- geneity in user typesyyyyy -ia the loss function introduced in Sec. 6 improvess average accuracy by 6%.
E
Additional Related Work
The challenge of handling heterogeneous preferences in alignment has been recognized as a significant problem alignment research [54 55 24 25 This problem has attracted considerable attention
from researchers in the field. Here, we highlight few representative works that address key directions tackling this challenge.
Analysis of DPO. Our study of how standard preference learning methods, such as DPO, behave in the presence of heterogeneous preferences was inspired by Siththaranjan al. 22 's result, which shows that RLHF aggregates preferences according to well-known voting rule called Borda count. Chakraborty et al. [19] highlights the impossibility of aligning with a singular reward model in RLHF by providing lower bound on the gap between the optimal policy and a subpopulation' 's optimal policy. Dumoulin et al. (26) adopts a density estimation perspective on learning from human feedback to illustrate the challenges of preference learning from population of annotators with diverse viewpoints. Rosset et al. (56) and Gao et al. [57] point out limitations of point-wise reward models in expressing complex, intransitive preferences that may arise due to the aggregation of diverse preferences. Additionally, frameworks that generalize DPO and unify different alignment methods have been proposed to analyze current approaches and explore possible alternatives [58 59 60).
Policy Personalization. Many works the literature have proposed personalization as solution to the problem of pluralistic alignment. Poddar et al. [61] propose a latent variable formulation of the
23

---

problem and learn rewards and policies conditioned on it. Chen et al. [62] use an ideal point model for preferences and learn latent spaces representing different preferences. Mapping user information to user representations, Li et al. [3] perform personalized DPO to jointly learn user model and a personalized language model. Balepur et al. [17] use abductive reasoning to infer user personas and train models to tailor responses accordingly. Lee et al. [14] explore the possibility of steering language model to align with a user's intentions through system messages. Dang et al. [63 extend personalized alignment to text-lo-image diffusion models. Jang et al. [12] perform personalized alignment by decomposing preferences into multiple dimensions. Lau et al. [1] dynamically adapt the model to individual preferences using in-context learning.
Preference Aggregation. Closely aligned with our goal of serving the entire population with single policy, several works have explored ways to aggregate diverse preferences. The rich literature on social choice theory has proven to be valuable source of inspiration for studying existing preference learning approaches and proposing new ones [23] 64, 65 24 66]. Drawing insights from social choice theory, robustness to approximate clones has been proposed as a desirable property of RLHF algorithms, which current methods lack [41] The Minimax Winner, a concept in preference aggregation, has inspired the use of the proportion of wins the reward for particular trajectory to align a model through self-play [67]. The impact of heterogeneity on strategic behavior in feedback and its effects aggregation also explored in Park et al. [TT], which further examines the use of different social welfare functions for preference aggregation.
Methods. Solutions proposed to address different formulations of the problem span a wide range of methods. Siththaramjan et al. [20] estimate distribution of scores for alternatives to account for heterogeneity as hidden context. Chidambaram et al. [8] propose an Expectation- Maximization (EM) version of DPO to minimize notion of worst-case regret. Multi-objective reinforcement learning [68] [2] and its direct optimization variant (50] have also been proposed to align with diverse preferences. Wang al. (69 train multi-objective reward model to capture diverse preferences. Zhong et al. [70] use meta-learning to learn diverse preferences and aggregate them using different social welfare functions. Li et al. [71] design an tranaaal-rrrnppo--bbaaded loss to calibrate their model with the categorical distribution of preferences. Producing a Pareto front of models has also been explored as a solution. Boldi et al. [72) employ an iterative process to select solutions, while Rame et al. (73) interpolate the weights of independent networks linearly to achieve a Pareto-optimal generalization across preferences.
Empirical Observations. Empirical studies of alignment methods have had a significant impact on the study of preference learning. Zhang et al. [5] demonstrate the Bradley-Terry model's 's failure to distinguish between unanimous agreement among annotators and the majority opinion in cases of diverging user preferences. Chen et al. [74] show that RLHF and DPO struggle to improve ranking accuracy. Zeng al. (75 study the role of model size and data size in the impact of diversified human preferences. Bansal et al. [76] demonstrate the significant influence of feedback protocol choice on alignment evaluation. Santurkar et al. (77] explore the opinions reflected by a language model, while Bakker et al. (78) investigate language model's ability to generate consensus statements by training to predict individual preferences. Jiang et al. (79 propose individualistic alignment to predict an individual's 's values, and Zollo et al. [80 introduce the Personall benchmark to measure a model's adaptation to a particular user's preferences.
F
Additional Statements
Proposition F.I. There exists mixture of BTs that a single BT cannot represent.
See proof on page 30
Definition F.2 (Learnability). Denote by Dr.- an i.i.d. sampled pairwise preference dataset labeled by random users with reward and preference model a. Let F(y) == [r(:: u)]. We say that the ranking based on f is (weakly) learnable if, for some > 0, there exists an algorithm with a bounded sample complexity m, such that for every reward r, when given dataset Dr of size [Dr.ol 2 m(e, F), it outputs a ranking consistent with F with a probability at least e above the chance level.
24

---

Proposition F.3. Defining in Eq. (17) as follows results in consistent estimation of the optimal policy when preferences follow the BI model:
o (h(y), (255,, (25
(2:
0 1
12,1,92,0 0;
0 (hly2, v1; ((
0
O.W.
Here, define I(0) =00)="]]] (or 1) [K4] d6', and is the difference of a induced rewards (Eq. (15).
See proof on page 31
G Missing Proofs
Proposition 3.1. Consider differentiable aggregation function f: : IR4 + R. Suppose that for every reward r consistent with the preference distribution-meannne Pr(y2 > y1
Pr(y2 > y1 r*)) for all prompts and responses yı, y2 in the response space the function f ((r(y:: u))u) induces the same ordering over y. (r(() u))ueu y y) has non-empty interior, then f must be affine.
Proof of Proposition 3./ First of all, if a reward function r" (y; u) can explain the preferences of user type u, any other reward function r(y: u) r" (y: u) + c(u) induces the same preference distribution:
Pr(ya y1 rju) = (r(y2: u) u() (r (32 u) (y1: Pr(y2 > yı uu)
Therefore, r" identifiable up to bias term that can depend the context and user type.
Denote all the true rewards from different user types by vector r" (y) E TR Let Y be the set of possible responses and 'R' (z) E } C R2 be the set of possible rewards, which by assumption has a non-empty interior. Consider a reward aggregation function / TR. - TR Our observation above implies that f(r (y) + c) should induce a consistent ranking
for every c E g4 This means that sign + c) - f(ri + c)} does not depend on c. Hence, there exists function &a such that
((r r1) sign c) - ((r++c)))
for every R4 and ri; 12 E (R*)².
Consider an arbitrary T1 R We claim f((r)) = vf(r2) for every T2 E R*. The proof is by
contradiction: Suppose V f((r)) f((2). Since [2]] > there should exist A e R such that (A, V (ri)) 2 0 (A, f((2)) (a similar argument holds for (A, ff(ri)) > 0 2 (A. f which we skip for brevity). Define ri rı cA and r/2 r2 (A, for sufficiently small > 0. We know ri and r are R" as R" has a non-empty interior. In the limit of 0+, we have
(e = (0rr rı) = signfe (((m))) =1,
(<< = (0r22 r2) = sign(e(A v = -1,,
which is a contradiction. Therefore, v.f should be constant everywhere in RR, which means f can only be affine function this domain.
Proposition 4.2. Suppose responses to x in the preference dataset are drawn from D(-1z) and labeled according to BT. In the population limit, where empirical averages converge to expectations, DPO' 's induced reward, or equivalently TDPo(-(a) has the same ordering over responses as NBC(. xx)
(ea(()))
Proof of Proposition 4.2 assume म
We start from DPO's objective in Eq. (6) For notational simplicity, x:) already contains a normalization by neef (y x)). In the limit many data points,
"We can view NBC(y x) as an aggregation of rewards at y. One can verify that NBC meets the order consistency condition of Prop. 3.1 However, it uses the reward value at y o to define the aggregated reward y and thus does not fall under Prop. 3.1 In fact, this interdependency causes the issues discuss Sec. 4.3
25

---

we can rewrite DPO's 's objective as the minimization of a cross-entropy loss
(x)o = -. (A (x,, y', v)) log a (Blog
< x)) T TΦy x))
+ (1 (1-0 (z, 2)) log (1 - σ log
TT x.)
TTy' xx)
where T (a,r"(x, v)) is shorthand for Pr(y > y' x; x;r") Eg [o("rr ([x, ul: u)
r" (" y/1; u)]] The minimizer of LDPO should meet the first-order condition: 3
Dénco
for every a and y. Then, a direct calculation shows that the optimal policy n² meets
By))D(lo)) Tor [log
ा (3 x)
By'nD( mz) [o(Ar" (x. y. ul)] = 0.
(19)
(3 x
Recognize that the second term is NBC(2 y x):
NBC m) = -D F o (r" (1= y): n) 1ra ([2r, 3 (; [::)]]]
In the absence of heterogeneity, a normalizing C' would solve Eq. (19) In general, we are not aware of any closed-form solution. However, we can still infer the ordering the optimal policy induces from Eq. (T9): Since the first term is increasing in T (y) x). the optimal policy will be monotone in NBC(y x). This completes the proof.
have
so setting, / log nT" (y a²) r' ((ær, y|) + C(x) for
Proposition 5.1. When the preference model o(-) is continuous, the ranking based the (user- weighted) expected reward is not learnable (accorling to Def. [:2) without annotator information.
We give three related proof strategies. The first strategy works for every preference model. The second strategy draws on a connection to robust version of Arrow's s impossibility theorem 44.. The third strategy is inspired by Procaccia et al. We start with the notation and definitions specific to this proof.
Proof of Proposition 5.7
Notation and Definitions. Consider fixed prompt æ with a set of possible responses V. Let R denote complete ranking over V, where Rol indicates whether y/2 > yi or vice versa. A profile refers to set of complete rankings. For a heterogeneous reward function r(y; u) and a prior P over user types U. let Rs be the ranking according to F(y) Euwp(r(y; u)].
pairwise preference dataset D consists of tuples (3/1, 3/2 o), where o = 1 > y1 We assume that y/1 and ya are i.idd. draws. When random user with reward function labels each instance in the dataset, we denote the resulting dataset by D,. A pairwise learning algorithm A produces a complete ranking over Y' based on the pairwise preference dataset D.
Proof Strategy Suppose there exists an algorithm such that for some y with | 2, for any reward function r and any preference dataset D, with |D,| 2 ny, it outputs Rş on y with probability of at least
[>]] c.
Suppose r is heterogeneous reward function that its expectation induces a complete ranking R with no tie. Define a new heterogeneous reward function r, as follows. Consider new user type 0 & U. For some y > 1, let (3 u) = 7 u) when * O, and r- (y 0) = 0. Define new user distribution (u) = (1 f) 0} P()).. It is straightforward
verify F,, = Eusp, (r u)] = F. Therefore, with high probability. A outputs Rr,- = R2 from D,, for every 7 1:
1
Pr (A((...) R,)
+
D
As we increase 7, for any continuous preference model a, the pairwise preference dataset Dr., approaches a uniform preference dataset Dunif labeled mostly by an indifferent annotator of type u 0. So, we have
1
Pr (A(Dunir) = R2)
+c.
(20)
D
This is true for every r. For different choices of r. agreements with R2 are disjoint events. Since there are !?]! different rankings overall, the pigeonholed principle implies = 0.
26

---

Proof Strategy 2. The proof is by contradiction. Suppose there exists an algorithm A that for any reward function r and any preference dataset D, with [D,] 2 ny. oulputs R, with probability of at least 1 e. We follow Friedgut et al. [44] and define social choice function as a function that yields an asymmetric relation on the alternatives given a profile. A social choice is rational if it an order relation on the alternatives, and is neutral if it is invariant under permutations of alternatives.
Let J'3 {yr : y2; 33 be an arbitrary subset of y with size Next, we construct neutral social choice function f acting on J/3 that is independent of irrelevant alternatives (IIA). Here is how we design f: Let P. be a profile of size n 2 ne at the input, where every R P, is an i..d. draw from Plackett-Luce (PL) ranking model with exp(r) as the weight of the alternatives. Note that the marginal distribution induced by PL on any two alternatives follows BT. Create three pairwise preference datasets 1r,12, Dr,23, and D,, 13 where Dr. = Ry, R E P.}. The social function f neutral and IIA.
applies A to every dataset to obtain relation over V/s By construction,
By assumption, for every internal dataset Dr.ús we have Pr (A((r.,)) = Rrais) 2 1 1-16 e, where Rogi the projection of Rg only two alternatives y; and yj. Using union bound, we have
Pr (f(P.) Rτ) 21-38 3e.
Similar to strategy 1, we can define new reward function r, with F, F such that the above holds for every 7y with y > 1. Then, by increasing 7, the profile Pr., approaches a uniformly distributed profile Punit- In this case, since Rg is an order relation, we have Pr (f is rational) 2 1 - 3e. Then Theorem 1.3 of Friedgut et al. 44 implies that for some global constant K,
Pr (f is dictatorship) 2 -3Ke.
On the other hand, we know that the order that RG induces for different r is not dictatorship, so, we have
Pr (f dictatorship) < 3e. Putting these together, we obtain a lower bound on
3 K +1) >0.
The rest of the proof is similar for the second and third strategies. We can use boosting argument to show that from any weak pairwise learner, we can obtain an arbitrarily strong weak learner corresponding at the cost of collecting larger dataset. We show this when for the weak learner < } but a weaker condition of choosing R₂ better than chance level is sufficient for our argument. Consider a pairwise preference dataset D, of size m.- Partition D, into m equal-size datasets. Let R3 be the output of AA on the 5th dataset. Since samples in D, are independently generated, R,s err independently. Construct meta-algorithm Amaj that outputs the majority winner of Rşs. A standard Hoeffding bound implies
Pr (Amaj + Ro) S exp
(e) m)
simple calculation then shows that for any arbitrarily small e' > O, by choosing m ((log()) -) the majority-winner algorithm agrees with R2 with probability of at least 1-e'. This contradicts the lower bound we established earlier and completes the proof.
Proof Strategy 3. The proof is by contradiction and is inspired by Procaccia et al. [41] This proof requires least four different user types. Suppose there are equally represented user types 4 = {A, B} who follow BT. For some arbitrary response yo E y and 0 < T < 3. consider the following reward function:
y + yo
Fr (3:
+ log -) log y you B
y yo: A,
(21)
One can Pr(yo > y r,) = for every y # yo- Therefore, r, induces the same pairwise preference distribution for every T. On the other hand, (760) = E, (go: u)] log 72 is an increasing function of т.
27

---

Consider two arbitrary Ti and 72 such that 0 <I <2 Sample pairwise preference dataset at follows. Draw random and random permutation p over V. If p not identity, ask an annotator of type with reward 'r (1; u) to label this sample and permute the ranking with If is identity, ask an annotator of type u with reward rrz (52) to label. This sampling equivalent to sampling from 2 ! different user types. By symmetry. this pairwise preference dataset is distributionally equivalent to dataset D unit with indifferent preferences. However, our construction implies that yo has the highest expected reward and other alternatives have similar rewards:
Fr (g),
y + yo
D Frp (y),
y = yo
Denote the ranking based on I' above by Ro.
Similar to the second strategy, suppose there exists an algorithm A that for any reward function r and any preference dataset D, with [D,] 2 ny, it outputs Rg with probability of least 1-6. e. Collect preference dataset as explained above with at least ny samples. Then, by assumption,
Pr (A(Dunir) = Ro) 2 1-
Note that our choice of yo could he any of the alternatives. Therefore, the above should be true when any of the alternatives has the highest expected reward. This implies a lower bound on E:
D
The rest of the proof is similar to the second strategy.
Proposition 5.2. There exists no consistent M-estimator that without annotator information can estimate V(x, V1, U2) = Var Arr (x, V1. V2: u)].
Consider a dataset D of context, candidate pairs, and preference repre- sented (æ: V1, V2: 0), where ( y1 and yı, 1/2 independently drawn. Then consider an M-estimator
Proof of Proposition 5.2
arg min
Σ
p(a, yt, Yw; V)
Σ
o. plp, y1, y2; V) + (1 - o) p(a. y2. y1: V).
(ED
(m.
Under preference model of Eq. ( for a reward r, we have Blo a, Un y2; ul = o (Ar(2x, 21 92: In the limit of large dataset, the M M-estimator solves
arg min En, En. oo p(æ,, yı. y2: V) + (1 - o) p(a, ya.
Fo (Ar(a, vn, u2) p(æ, vı, y2: V) (Ar(z, ( 32, uI; plan y2, pla, vI:
(V has no o)
E. [ (Ar( yr U2; u))] p(., 22: V) + Eu io (rr((z y2. v1; p(ar, 22 vi
(V has no u)
2 -FF...
F., [or( v1. u))] p(.. 92; V)
(v1. U2 are i.i.d)
Here, we used the fact that V does not depend on u. We also relied on the assumption that yı and y/ are identically and independently distributed. The above equation suggests that regardless of how P is designed, V(x, y y/2 can only depend on u's distribution through E., (Arr(a (ax yı, y/2 u))] Therefore, no consistent M-estimator can generally estimate Var [Ar(x, yı, y2: u)] even with the availability of infinite preference data.
Lemma 5.3. Using J1 and Jz as shorthands for [xx V1, V2, x. U1 Y2) and J [xx Vı, 32.32.5 V2. v1). we can use the following to consistently estimate the variance term:
JI (J1 J2)
V V 422 o' d'(AF" 922)
(16)
Proof of Lemma 5.3 First of all, can give us the likelihood itself:
Jœx yı, y2, xr, yı, 82) + [(x yı, y2, x,, y2, y))
E. ( Ay)) (y) U2; + o (Ar (21, y2; y2; w)) o (Ar"
= =Erro(A" (v), y2; v2)4))]
28

---

Here, we used the property of (Arr"(V2, v1: = - (31, yy: u)). We also dropped æ from the notation for simplicity. Since J can give us both the first and second moments, we can use it to find Var (ara"o(Ar" (æ, yı, D/; u)] as follows:
Var (o (æ., 21 U2; w)]
E., [o. Ar (x, yı y2: w)) " Eu lo (rr (x, yl, (2: u2;
[x, y1 V2. v. y1 y2) - (x, y1. U2. a. y1 U2) + J (x, y1, 92. yo, y1))
In the last piece of the proof, we connect Var, [o(Ar)) with Var Arr"] The Taylor expansion of (Ar*) around AF EulAr gives
Var, [o(rr)) Var (O(AF))) o' (Ar* -AF) + 0(Arr" -((A)))A7)))
='' Var., |Ar*] O E., [(rr [AA*'-AAA7)))
We can neglect the third-order term in calculations as first-order correction uses up O Enn [(Ar- )=*)) in its approximation.
Proposition 6.1. Defining in Eq. (17) as follows results in consistent estimation of the optimal policy when preferences follow the BT model:
log o [ h(g1 Moi T))
0 1,
(y y2. T)
log a [M] h(y2, 915T))
0 0
0
O.W.
Here, h is the difference of a induced rewards (Eq.
15
) and (0) is the vector of all ones (zeros).
Proof of Proposition 6.7 The proof follows similar steps as the derivation of DPO. First of all, conditioned on agreement, the likelihood of observing > y under the BT model
11, [Ar* (31, 322
Pr(y2 > y1 |r", agreement) = 11, (311 3/2 u)) + 11, (Ar Ara((2222 Uriu))
exp Σn 1" (y2; w))
exp E, (" (y1 u)) + exp EE,, (3(2:))
= [m( E., Ar (11 (11. 92)1)
On the other hand, Eq. 11 allows to write E,, Ar (y y/2] with difference a's induced rewards, i.e., h:
Pr(yz y1 n. agreement) a (N(y1, (2;
We can define the likelihood in this way for every policy ग. Then, the proposed loss function is equivalent maximizing likeeio,,oodd, which under mild conditions is a consistent estimator for -..
Theorem 6.2. Suppose Eq. 17 only depends on (x, y1 M2) through π and Teef. If there are more than three types of user and the preferences follow BT, any loss that allows consistent estimation of the optimal policy discards samples with disagreement, iee, those with o & {0.1}.
Proof of Theorem 6.2
The proof involves three steps: First, the next lemma shows that any loss with the desired consistency property can only depend on π through the
function in Eq. 17
ratio ( xz)
x le)
Lemma G.I. Suppose I in Eq. 17 only depends on (xx, yı, y2) through स and Teet. Then, for any I
that gives consistent estimation of the optimal policy Eq. 10), there exists an equivalent loss L such that
1(x v1, M2, o; x) T(o: h( yr. v2.)))=)))
where &s is defined in Eq.
15
29

---

See proof on page 32
In the second step, we further limit the search space of (as introduced by Lemma G.1 to those that meet certain first- and second-order conditions:
Lemma G.2. Any loss as in Lemma G.1 that leads to a consistent estimation of the optimal policy meets
ai
Σ
00 (z)) -x(()) =0,
of 10,
0
Σ
002 (o: 08 (x)) xo(2) 20,
(0:
(0,11
for every z E [0, 1j. Here, we define
Xo(z) =
II
Zu =u) --)
wEll
and
0" (z) =
σ
(zu)
W
See proof on page 33
Finally, we show that when preferences follow the BT model and there are more than three user types, all i(o; () terms corresponding to & O, do not depend on 0. Therefore, these terms do not depend on and can be removed from the loss function, thereby completing the proof.
Lemma G.3. [f]
3. for any loss that meets the first-order condition of Lemma G.2 have 00 (o: 0) 0 for every & {0, 1}.
See proof on page 33
Proposition F.1. There exists mixture of BT's that single BT cannot represent.
Proof of Propositinn F.7] Suppose the pairwise comparison distribution over set of alternatives
(y1, y2, V3....) satisfies the Bradley-Terry (BT) model; i.e. Pr(y: > yı) a (r" (y2) r (y1)).
Then:
Ih exp(r"
Pr(y Pr(y2 > ys) Pr(ys y1)
Ih (exp(r* (xxp("((1)) (3))) + exp(r* (H((+1) mod 3+1)))
Pr(y > ya) Pr(ys > y2) Pr(y > yn).
Now, consider BT models corresponding un and u2, with uniform mixture over them. For the mixture:
Pr(y, yj) Pr(y: y un) + Pr(y: > yi u2)
2
The probability cyclic preferences in one direction is given by
Pr(y1 y2) Pr > y3) Pr(ys vı) Exe(1,2ya II Pr(y, > 9 mod 3+1 u
which is not necessarily equal to the probability of the cyclic preferences in the reverse direction:
Pr > y3) Pr(ys > y2) Pr(y2 > yi) Exe(1 I1 Pr(v(-1) mod 3+1 } yi Ux;)
8
exp(ri)
To verify this, consider specific examples such as Pr(ys > yi ux) (1, and r2 (1,2,4). More generally, the BT assumption implics that, for a fixed
expir() exp(ri)+xx((re exxp(2) with
30

---

reward r", the likelihood of a set of pairwise comparisons {(pp11 > Up.2) is proportional to I. (xp(r"().))) (1:))) Mo.1=() and depends only the number of times each option is preferred in the comparisons. However, as demonstrated above, this property does not hold for mixture of BT models.
Proposition F.3. Defining in Eq. (17) as follows results in consistent estimation of the optimal policy when preferences follow the BT model:
o (h(y1, V2; π)) I (1o
0 1
(2,1,32,077) 0;
0 (h(y2, 91; (10(2
0
O.W.
Here, we define 1(0) = S1 (3r 1) 1M d0', and h is the difference of IT's induced rewards (Eq.
(15
F; yı, y2) o (ar, yr, y2; We use zu (1 y/ 3/2 as shorthand for this quantity and will drop the dependence on (xx y1 y2) whenever it is clear from the context. We also use s as a shorthand for (h)). the limit very large dataset, the proposed loss approaches
Proof of Proposition F33 Recall Pr(o..
((s) E.....
[II NEW
(II --))))(11111 s 21)) I(1 ((1-()))) s))] wEM
Note that we wrote C as function of s instead of since is the only place that т appears. We first show that [(s) has unique global minimizer. To show an is global minimizer of L, it suffices show that s minimizes the term inside expectation for every (ær, y1, y2). Such a minimizer meets the first-order condition:
III 2..) MEL
(1+ 11) (1))4))) + (I zn)) (-1- (-
1)(a) =
8
WELl
Here, we used do quadratic equation in terms of
Then, the above condition reduces to ( 1) Define w == |
1 1 w
II wEW
= (1 + w-
II
- 1) 0.
2u
MEl
Solving for w, obtain
1 (I 1)) i
For the BT model, direct calculation then shows
(x, 91 3/2 = σ [u
(pZ)
Σ
Ar(æ, U1 31. U2:
(22)
wEll
In fact, s" is the only global minimizer of L(s). This is because L(s) is convex in
d2 E...y.. 32.22
4 s² (
U
[II wEW
m.
11-1 + (II (1 - zu)) (1 s)2 1-s - 11
2 0.
ds²
uEt
Finally, one can verify that the policy that results in s* (Fq. (2 is the optimal policy n². This completes the proof that the proposed loss a consistent loss for n
Lemma G.I. Suppose I in Eq. 17 only depends on (ar, yı, y2) through n and neet- Then, for any I that gives consistent estimation of the optimal policy in Eq. 10p, there exists an equivalent loss i such that
1x, 1ı, M2, T) = i(o: - v).
where & is defined in Eq. 15
31

---

Proof Lemma G.1 Since 1 only depends on (x, y1. U2) through π and Tref, we overload the notation and use ((;; π (g) æ) T æ)) to denote the loss from (æ, U1 U2, o). In the limit of many data points, L(D; π) converges to
( vn un o [1: (1y T x))]
Using Pr(ou- 1 x. 11 02) (Ar" az V/- U25 w)) the tower rule implies
[(x)) = Ez.y1. Bol (3/1 2)) (3 æ)) æ v1 y4]
Σ
(( क (3) x)) (32 x)) II
(Ar" (a, yr. u2: (1 o(rr" (x, yr, y2s w))
(0
WEM
For notational simplicity, let's define
2a xx, y1 V2) == (Ar' (x, y1, y2: U2:
Xo(z)
II well
Fu z.. 100
Note that Xo implicitly depends on (x, y1 y2) through which drop from the notation when it clear from the context. Using this notation, we can rewrite the [(T)'s 's expansion as follows:
((T) Em Ei.
Σ ((0: 2)), (322 =)) Xo (o(2(1, yı y2))
(23)
(E.10))
Overloading notation, we can always equivalently represent (T(1) x)) (
x)) as (T y2
a). x3/2 2/, V2, æz). that is, with the probability that either of the two responses is chosen and the probability that the second one is preferred. Therefore, there exists a loss l' such that
((; x) (
æ)) (o क ((y1, y23 x), ग (3/2 {y) 32).
y1. Y2. x) should also optimal. Since ग (y22 1/ 3/2, x) appears in only one term of the expectation in Eq. (23). can conclude that
π is optimal, T (B22
arg min
Σ
I'(o: x" ((y) y23 x), Xo (2 (1. 82))
(
the optimal (322 (Y1, y2), a) for every optimal T T( y2) of the optimal policy is that
x)) On the other hand, a property
x (3/2 x)) nee (1/ x)) x) x² (31 x))
1 R Ar" (ar, yı, y2; y2;
π (y2 (y), y2),
exp
Teet
Therefore, for every optimal policy n" ((y1, 323 a)), have Tea
y/2
x))
En Ar (a. yn (2:
exp
nel
x))
3
(24)
arg min
Σ Of (0.1)
l' (((o; " y2) 2)), 8) Xo Xo(2(x, yr, v1.92)).
Recall from the optimal policy (Eq. 100 that we can modify the reward function for responses other than (x. V1, 82) while keeping Art (æx y1 yo! u) constant. This allows arbitrary changes to x ((yi, y2 x) without altering the rest of Eq. (4). So, we can argue that does not depend
on ((y), 3 x) and we drop from l' notation. Define new loss based l':
i(o;0) = (0
Tyer (3/2 Φ)
exp 1 (-))
Keef
x)
Note that implicitly depends on (æ, y1, 42) through गeet which we dropped from notation. Using i, we can write the original loss as I(o: π ( x), क (222 x)) = l' (o; क (V2
(u1,82). x))
( ())
1' o: n ( xx)
i (o; log
T x>)
B log
(ref (Y/ x) Tref(y) x)
Tref
T
x)
= [(o: h(x, V/, (1:32:
32

---

This completes the proof.
Lemma G.2. Any loss I as in Lemma meets
G.I that leads consistent estimation of the optimal policy
ai
Σ
-xo(z) 0. (o: (x))
00
(0.1)4
848
Σ
002 (2)) xx(2) 2
for every z E 10,1]4 Here, we define
Xo(z) =
II well
2u (1 =ul 1-00
and
(z)
- (zu)
U
wEl
Proof of Lemma C.2
We will refer to the proof of Lemma is monotone increasing, have
G.1 in this proof. Using in place of l' in
Eq. 24 since exp(-)
E. Ar" (æ, y1, u2; arg min
Σ
[(o;0) xo(z).
(
On the other hand, using the fact that types in 21 are equiprobable, write
Eg Ar (x, yı 32;
1
Σ a (zu).
U
Putting these together, it is necessary to have
Σ o (2n)
(zu)
arg min
Σ
i(o;0) xo(=)
WEM
oE
for every [0, 1j The rest of the proof is straightforward.
Lemma G.3. If [ > 3. for any loss that meets the first-order condition of Lemma G.2 oi 00
we have
(o:0) 0) for every & {0,
Proof of Lemma G.3
First of all, for the BT model, a direct calculation shows
0" (z)
Σ (zw) = 2
log II ueld
1 Zu
log
X1
U
Zn
U
Xo
Since 6* (z) depends on only through steps: In the first step, we relate g(o;
Xo. we denote
aî (o:0* by g(;; x²).
The proof has two
X1
20
Xo X1
to g(( Bu'
xa), where we define
Ou
u u' Ou, n'
Using this connection, in the second step, we will show that g(; وما = 0 for any E {0,1) when
[ 4.
Step 1. Consider any o & {0, 1}. When [4| 2 3, there exists u' E ld such that ouu & (0, 10. For such o and u', we define two non-empty sets
S = {u E U u + d' = =
5 = {u U, + n',o, On = =0].
33

---

We set such that
II Zu II (1 zu)
WEST
WES
(25)
Zu 1- Vu E E
Zu 0+
Vu E S
For this choice of 2, asymptotically, we have
Xo
1 Zn1
X1
Zud
out (1 Zns 1'-0o'
Xo
2n¹
Xoo
1-0ul (1 Zas!
) 1oy'
Xo 0, vo' & fo, 10
Using the above, we can simplify the first-order condition in Lemma G.2
g(o:
1 1 Zzd
(1 ) + gg 0. 1
1-o0 (1 z )0 0.
This condition should be held for every E [0, 1). Therefore, we can conclude
g(o ;a) = -=00; a a2-20w'
(26)
for every a E R. This completes the first part of the proof.
Step 2. When o & {0, 1) and [M] 2 4, there exist distinct user types u' and u" such that none of Hu' oOt", and o "0('lu") (u") are in {0, 1}. Here, we used o((",,"") as shorthand for (o fo @ou".
For such o, u/, and u", we define two non-empty sets
S {u u E EU (u', u,, = 1}.
8 = {u u uue {u', J" On = 0}.
We set according to Eq. 25 Then, asymptotically,
Xo 0, Wo' & fo, 0
o Bu" Oll' au") ,n"))
Using the above, simplify the first-order condition in Lemma G.2
g(; Xo Xo + g @a'
Xo X1
"oosu + g @a"
Xo "oosn"
g(o @ ,"),
Xoe(u',u")) 0. X1
Xo
X1
X1
(27)
Because of the symmetry of this equation, can assume without loss of generality that Our 0 and On" Therefore, asymptotically, we have
Xo
- Zul
1 Zud
X1
Zv²²
2n"
(1 - Z,y) Zu""
Xo Zaa Zxd!
Nota" = (1 Z,²)= -(1 2n0") )
Xot( Znl Zv!i )
Eq. 26 also implies
0u ;)) = -g(o; a) a,
0u ;a) -g(o; a1
g Bld a aa) g(o;a).
Plugging these into Eq.
27 and simplifying equations, obtain
0:
Zea'
Zmn
(2%,r- - 1)(22.) - =0
Zna'
Zzz"
This equation should be held for every 2n' and z.r. By appropriately setting zw and 2u', we can conclude that g(o; a) should be zero for every a. This completes this proof.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction accurately reflect the paper's contributions in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of every result right after introducing We also discuss general limitations in Sec. 9
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [Yes]
The statements of the theoretical results include all necessary assumptions, except for general assumptions introduced in the problem formulation in Sec. Each result is also linked to its corresponding proof in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss all the details of the experiments either in the main text or the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
36

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have used publicly available data and provided their links. We have provided an anonymized version of the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
wwt.aaammmiiooooooooooooooooeiie http:///uuuudeeeoeeeeeaaaasasaaa publicc/guude///odeStttit
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ipss.c/puut cttp://cccpbblccccuiceeeee guides /ttpp////////i.iie///////uaaaiaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have covered the main details in the main text, and the rest of the details are discussed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
All of our results have come with confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We only do small-scale experiments running on a single GPU in few days.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: We fully follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We extensively discuss the societal impacts of our work in various places.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
38

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We did not release any model and only used public data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
39

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not crowdsource or collect human subject data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
40

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We only used LLMs for editing and writing improvement.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httt:p///eeerrreeccc///ooerrrrre for what should or should not be described.
41


Reviews:
[{"review_id": "XYr0ty0riA", "reviewer": "Reviewer_cLLj", "summary": "This paper studies heterogeneous user preferences and address alignment problems in this setting. Theoretically, it proves that any single policy compatible with multiple user types must reduce to an affine combination of rewards. It further shows that standard Direct Preference Optimization (DPO) is ordinally consistent with the Normalized Borda Count (NBC) rather than with that average reward . To mitigate the mismatch, the authors propose a first-order corrected DPO for minimal annotator information, and a consistent loss that learns the optimal policy when every data point is labelled by all user types. Experiments on Pew surveys, synthetic toy tasks, and semi-synthetic fine-tuning of Llama-3-8B and Qwen-2.5-7B show gains in rewards when heterogeneity is modeled.", "questions": "1) In practice, what guidance can you give for grouping user types? Do authors have ablation study on the sensitivity to the grouping choice?\n2) If users are mis-clustered or have no types, how do the first-order and consistent objectives degrade?\n3) Can this framework bring benefits if take the preference heterogeneity into account during inference?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper is well written and structured. \n2) The authors identify the problem of heterogeneous user preferences, providing solid theoretical characterizations for the optimal policy, and the 'objective' DPO is trying to optimize. These results provide valuable insights in modeling preference data and designing leaning algorithms.\n3) The PEW Research Center surveys demonstrate that NBC can conflict with average reward in practice, which justify the theoretical findings.\n\n\nWeaknesses:\nThe LLM experiments rely on length-based semi-synthetic rewards. It is hard to judge if the methodology can bring practical benefits.", "was_revised": true, "final_justification": "After reading the rebuttal, I decide to maintain my score.", "rebuttal": "We appreciate the reviewer's insightful feedback and thoughtful comments. We are pleased that they found our paper well-written and well-structured, offering valuable insights into modeling heterogeneous preferences and developing algorithms. Below, we address the reviewer’s questions.\n\n> If users are mis-clustered or have no types, how do the first-order and consistent objectives degrade?\n\n* If users have no type, i.e., homogeneous preferences, the first-order correction and consistent objectives (e.g., Prop. 6.1 and Prop. C.3) reduce to standard DPO. \n* If users have different types but no annotator information is available, then neither the optimal policy nor any approximation of it can be learned consistently (Prop. 5.1 and 5.2). \n* If users have different types and minimal annotation information (i.e., paired preferences), we propose a consistent first-order correction of DPO (Lemma 5.3). This method does not require explicit knowledge of user types or any clustering—it remains valid with many types and without identifying them.\n* If users have different types and we know the type and collect a rich dataset where each sample is labeled by a representative of each type, we propose consistent loss functions for directly learning the optimal policy. However, our main impossibility result shows that even with correctly specified types, neither these nor any direct alignment methods are sample efficient for many user types. When user types are misclustered, i.e., incorrectly specified, we believe learning in this setting with the consistent objective only becomes harder. We have provided a new synthetic experiment showing this and the robustness of the consistent objective in our response to Reviewer wgAn, but we do not have any formal guarantee at the moment. In our response to the next question we also discuss how our results can help mitigate such misclusterings.\n\n> What guidance can you give for grouping user types? \n\nWhile our study focuses on the consequences of the existence of user types, we agree that discovering them is also important and valuable. Our theory and algorithms can inform user type discovery as follows. Let $\\mathcal{U}$ denote the set of true user types, and let $u'$ be a proposed group that may include users from multiple true types. A direct extension of Eq. 14 suggests that the inaccuracy from modeling $u'$ as a single type in DPO-style algorithms is proportional to \n$$V(x, y_1, y_2 \\mid u') := \\text{Var}_i[\\Delta r^*(x, y_1, y_2; u(i)) \\mid i \\in u'],$$ \n\nwhere $u(i)$ denotes the true type of user $i$. Given $V$, one can define $l(u') := \\sum_{(x, y_1, y_2) \\in \\mathcal{D}} V(x, y_1, y_2 \\mid u')$ as a measure of the homogeneity of $u'$, and accept $u’$ as a single user type if $l(u') < \\epsilon$ for a practical choice of $\\epsilon$. Note that this will only give the criterion for an acceptable user type, but how to find such types is an algorithmic question (for example, agglomerative or divisive clustering might work here).\n\nThe bottleneck in this clustering approach is estimating $V$. As Prop. 5.2 shows, no M-estimator can consistently estimate $V$ without annotator information. On the positive side, our algorithm in Lemma 5.3 can estimate $V$ consistently using minimal annotator information, namely, paired preferences (Sec. 5.3). However, repeatedly estimating $V$ in an algorithm can be computationally intensive.\n\nIn summary, our theory provides a criterion to test when a group of users can be treated as a single type. This requires estimating a variance term, which we show is feasible using a joint likelihood maximization with minimal information. While our tools are useful for user type discovery, we acknowledge the computational challenges and view the development of more efficient algorithms as an important direction for future work. We thank the reviewer for raising this point and will include our response in the revision.\n\n> Can this framework bring benefits if take the preference heterogeneity into account during inference?\n\nThis is an intriguing question. Although we are not entirely sure how alignment can occur during inference in our context, inference-time (re-)alignment of LLMs is a lively research area. Recent studies have investigated methods such as best-of-N sampling (Huang et al.’25), activation steering (Wang et al.’24), transferring logit differences from smaller aligned models (Mitchell et al.’23), diffusion-style alignment (Chen et al.’25), integrated value guidance (Liu et al.’24), and reward-guided generation for test-time (re-)alignment (Xu et al.’24). Combining these techniques with our heterogeneity-aware methodology will be an exciting direction for future work!"}, {"review_id": "ChaTihRvIK", "reviewer": "Reviewer_yY38", "summary": "This paper investigates the alignment of a single AI policy with diverse human preferences, a setting it calls \"heterogeneous preferences\". The authors argue that the common approach of assuming a single reward function is flawed and propose that the goal should be to optimize the average reward across different \"user types\". They demonstrate that standard methods like Direct Preference Optimization (DPO) fail to achieve this; instead, DPO aligns with the Normalized Borda Count, which can promote mediocre outcomes and is sensitive to the data sampling distribution. The paper proves that learning the optimal policy is impossible without annotator information but introduces solutions for different information levels: a \"first-order corrected DPO\" using minimal data (paired preferences) and a consistent loss function when full annotator feedback is available. Ultimately, the research reveals a fundamental trade-off, showing that any direct alignment method that is consistent in this rich data setting is inherently sample-inefficient, as it must discard data where users disagree.", "questions": "Could the authors provide the approximation error estimation in Equation (14) and using $V$ to approximate the variance. Since the authors mentioned the Sensitivity to Preference Dataset Distribution in Line 167.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n- The paper demonstrates that standard methods like Direct Preference Optimization (DPO) fail to optimize for this average reward. Instead, DPO implicitly optimizes for the \"Normalized Borda Count\" (NBC), a voting rule that can lead to undesirable outcomes, such as promoting mediocre options and being sensitive to preference dataset distribution. \n- The paper tackles a problem of paramount importance: how to create a single AI system that serves a pluralistic society with conflicting preferences. Its primary contribution is a fundamental negative result (Theorem 6.2) that establishes a tension between consistency and sample efficiency for direct alignment methods. This is a highly significant finding, as it delineates the boundaries of what is possible for an entire class of popular alignment techniques.\n### Weaknesses\n- Key impossibility results are proven under the assumption of the Bradley-Terry (BT) preference model, which may limit the generality of the conclusions.", "was_revised": true, "final_justification": "The reviewers have resolved my question. I maintain my score to be accept.", "rebuttal": "We thank the reviewer for their helpful and insightful comments. We’re glad that they found our problem of paramount importance and our main impossibility result a highly significant finding. \n\n> Key impossibility results are proven under the assumption of the BT preference model, which may limit the generality of the conclusions.\n\nWe appreciate the reviewer’s observation. Our main impossibility result (Thm. 6.2) is established under the BT preference model. However, Prop. 3.1 (which shows only affine aggregations lead to a well-defined problem), Prop. 5.1 (impossibility of learning an optimal policy with no annotator information), Prop. 5.2 (impossibility of M-estimation with no annotator info), and Lemma 5.3 (consistent variance estimation) all hold in full generality for an arbitrary preference model as defined in Eq. 2.\n\nThat being said, BT is by far the most widely used preference model for post-training and alignment of language models. For example, OpenAI’s InstructGPT [1], Google’s Gemma and Gemini models [2], all use BT for alignment, since it consistently outperforms alternative approaches [3]. Consequently, it's the most popular model to stufy human preferences too.\n\n> Could the authors provide the approximation error estimation in Eq. 14 and using $V$ to approximate the variance. Since the authors mentioned the Sensitivity to Preference Dataset Distribution in Line 167.\n\nWe thank the reviewer for the clarifying questions. Eq. 14 omits $O(\\mathbb{E}_u[(\\Delta r^* - \\Delta \\bar{r})^3])$ and higher-order terms, as analyzed in the proof of Lemma 5.3 (line 824). The variance estimator $V$ in Lemma 5.3 is not an approximation but a consistent estimator of the variance. Its performance, however, depends on optimization quality and finite-sample effects, which lie beyond the scope of our population-level analysis. Regarding sensitivity to the preference data distribution as defined in line 167, our $V$ estimator is not sensitive in this sense, since it consistently estimates $\\text{Var}_u[\\Delta r^*]$, which is itself independent of the data distribution. Similarly, the consistent losses discussed in Sec. 6 are also not sensitive in this sense, as their optima do not depend on the data distribution. We thank the reviewer for these questions and will clarify them in the revision.\n\n[1] Ouyang et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems\n\n[2] Team et al. (2024). Gemma: Open models based on Gemini research and technology\n\n[3] Liu et al. (2024). Skywork-reward: Bag of tricks for reward modeling in LLMs."}, {"review_id": "3GSVPVwqvO", "reviewer": "Reviewer_Fxtn", "summary": "This paper studies and formalizes the tradeoff between ambiguity that arises from preference feedback learning and sample efficiency. The paper then proposes a first order correction to DPO, and show that minimal information can improve upon standard DPO formulation.", "questions": "See the point on real world experiments in the weaknesses section.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\na. theoretical as well as empirical results that support the claims\n\nb. the paper concretely formalizes this tradeoff of maximizing average reward vs the average of different rewards and how that conflicts with sample efficiency.\nIn particular, the authors illustrate how DPO makes a crucial assumption that the average of the preferences is roughly the same as the preference of the average.\n\nc. I also appreciated the insight that this assumption relies on variance in annotator labels, i.e their disagreement, leading to the proposed first order correction.\n\nWeaknesses:\n\nLack of real world experiments that show this variance/annotator disagreement in practice and the improvement with the correction to DPO formulation.", "was_revised": true, "final_justification": "I read all the responses and maintain my opinion post rebuttal.", "rebuttal": "We thank the reviewer for their thoughtful feedback and positive assessment. We are glad the reviewer found both our theoretical and empirical results compelling, and appreciated our formalization of the tradeoff arising in maximizing average of different rewards. We are especially pleased that the reviewer highlighted our analysis of DPO’s key assumption and its connection to annotator disagreement, which motivates our proposed first-order correction.\n\n> Real-world experiments that show this variance/annotator disagreement in practice\n\nWe agree with the reviewer that our main contributions are theoretical, highlighting the tensions inherent in using any direct alignment method under heterogeneous preferences. To motivate the existence of variance or annotator disagreement, we primarily draw on prior studies and provide motivating examples based on real-world PEW surveys (which were noted as helpful by Reviewer cLLj). We agree that annotator disagreement and variance are important topics and, in light of our results, believe they warrant greater attention. We hope our work encourages further empirical research in this direction."}, {"review_id": "Qiy5N7RYJu", "reviewer": "Reviewer_wgAn", "summary": "This paper investigates the challenge of learning an optimal policy from preference data when preferences are heterogeneous across multiple user types. The authors show that standard DPO methods, which assume homogeneous preferences, fail to capture this heterogeneity, leading to suboptimal policies. To address this, the authors theoretically prove that learning a consistent policy without annotator information is impossible and propose methods to overcome this limitation by leveraging minimal annotator information, such as paired preferences from the same user. They introduce a first-order corrected DPO algorithm that incorporates variance estimation to improve policy estimation. Furthermore, they develop a consistent loss function for scenarios with rich annotator data, enabling direct alignment with the optimal policy but at the cost of sample efficiency. Finally, the paper conducts synthetic and semi-synthetic experiments, showing that accounting for user heterogeneity yields policies that better align with average user rewards compared to existing methods.", "questions": "1. The proposed first-order variance correction relies on training a joint likelihood model to estimate the variance term (Eq.16). Could the authors provide a detailed analysis of the computational overhead incurred by this correction, particularly in large-scale settings?\n2. The consistent loss function assumes access to the annotator identity for each sample and assumes all user types label each point. In practice, this is rarely the case. How does the method perform when some annotator identities are missing or when annotations are noisy or incomplete?\n3. The comparisons focus primarily on standard DPO and its corrected variants. However, related works like EM-DPO and MODPO also address heterogeneous preferences.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper explores and investigates the critical yet largely underexplored challenge of addressing heterogeneity in preference data within alignment and preference-based learning, aiming to better align language models with human objectives and values.\n2. The paper proposes a first-order correction to DPO and develops consistent loss functions under different levels of annotator information.\n3. The synthetic and semi-synthetic experiments seem well-designed to empirically validate theoretical claims, showing concrete improvements over baseline methods.\n\nWeakness:\n1. Some concepts on personalization and multi-user reward aggregation have appeared in prior work (e.g., EM-DPO, MODPO). However, this paper should set itself apart by providing stronger formal guarantees. \n2. Limited generality due to reliance on DPO framework and BT model assumptions, which may not capture the complexity of real-world preference learning scenarios. \n3. Heavy dependence on rich annotator information leads to impractical data requirements and poor sample efficiency\nThe proposed consistent loss function requires detailed annotator identity information for every sample and assumes all user types label each data point. This results in exponential data demands as the number of user types grows, which is unrealistic in many deployment scenarios where such fine-grained labeling is costly or infeasible.\n4. The practicality and computational cost of the first-order variance correction have not been sufficiently analyzed. While the paper introduces a variance estimator via a joint likelihood model for first-order correction of DPO, the computational overhead and stability of this estimator in large-scale or real-world training remain unclear.\n5. The experiments do not test robustness under more realistic conditions, such as noisy annotations and missing user types per sample. We would like to see that.", "was_revised": true, "final_justification": "After reading the author's rebuttal, I have decided to increase my original score.", "rebuttal": "We thank the reviewer for their thoughtful review and valuable comments. We fully agree that addressing heterogeneity in preferences is a critical yet largely underexplored challenge. We hope our strong theoretical contributions help advance understanding in this important area. We also appreciate that the reviewer found our experiments well-designed. Below, we address the specific points and questions raised.\n\n> Comparisons focus primarily on standard DPO and its corrected variants. However, related works like EM-DPO and MODPO also address heterogeneous preferences. … this paper should set itself apart by providing stronger formal guarantees.\n\nSeveral works have addressed alignment in the presence of heterogeneous preferences and we discuss them extensively in Appendix B. EM-DPO is the closest work to our setting in the sense that it aims to learn a single policy serving the whole population. However, it defines maximum regret among the user types as its objective for the final policy, which is inherently different from the more natural expected reward objective, and relies on a heuristic expectation maximization using stochastic gradient descent which comes with high computational complexity. Methods such as MODPO rely on other structures of preference data to capture heterogeneous preferences and personalized methods such as P-RLHF learn a separate policy for each user type, none of them making up a comparable baseline for our method.\n\nThat being said, minimizing our consistent loss is theoretically optimal for our proposed objective under the rich data setting. Our aim in the experiments is to illustrate how much improvement is left on the table when heterogeneity of preferences is ignored. For this purpose, DPO serves as a natural benchmark due to its popularity and its homogeneity assumption. \n\n> Limited generality due to reliance on DPO framework and BT model assumptions.\n\nWe appreciate the reviewer’s observation. Our main impossibility result (Thm. 6.2) is established under the BT preference model. However, Prop. 3.1 (which shows only affine aggregations lead to a well-defined problem), Prop. 5.1 (impossibility of learning an optimal policy with no annotator information), Prop. 5.2 (impossibility of M-estimation with no annotator info), and Lemma 5.3 (consistent variance estimation) all hold in full generality for an arbitrary preference model as defined in Eq. 2. \n\nThat being said, BT is by far the most widely used preference model for post-training and alignment of language models. For example, OpenAI’s InstructGPT (Ouyang et al.’22), Google’s Gemma and Gemini models (Team et al.’24), all use BT for alignment, since it consistently outperforms alternative approaches (Liu et al.’24). Consequently it's also the most studied preference model which we adopt for our impossiblity result.\n\nWe also emphasize that our main impossibility result applies to any direct alignment method—not just DPO or the DPO family. The only part of the paper which is specifically about DPO is Sec. 4, where we analyze the consequences of the homogeneity assumption in DPO as a flagship direct alignment method (even these results extend to RLHF by substituting the policy with its induced reward, but framing them under DPO simplifies the exposition).\n\n> Exponential data demands as the number of user types grows.\n\nOur goal in Sec. 6 is to find out what is and isn’t possible given rich annotator information. Specifically, we show that consistent learning of the optimal policy is possible and propose corresponding loss functions (Prop. 6.1 and Prop. C.3). However, as the reviewer points out, these loss functions become sample inefficient when there are many user types. Our main theoretical result, Thm. 6.2, shows that this inefficiency is not specific to our losses but is a fundamental limitation of any direct alignment method. Therefore, as the reviewer correctly notes, our impossibility results make direct alignment undesirable as the number of user types increases. As we discuss, one can then either forgo consistency and pursue approximate methods—such as the first-order correction to DPO—or resort to indirect methods that learn reward functions before policy learning.\n\n> Computational overhead of the first-order variance correction.\n \nGenerally, optimizing the joint likelihood $J$ to obtain $V$ is roughly as challenging as training a policy or reward model, though the exact difficulty depends on the chosen architecture. Compared to a policy operating on $[x, y]$, $J$ takes as input $[x, y_1, y_2, x', y'_1, y'_2]$, potentially making its input context four times longer. However, since $J$ is a binary predictor, the task may be simpler than token-level prediction, depending on the model design.\n\nDespite the added complexity, learning $V$ is a one-time cost and does not scale with the number of user types. As a result, learning $V$ and direct alignment is still more efficient than training separate reward models for each user type followed by RL. In our paper, we evaluate the first-order correction of DPO using estimated $V$ in synthetic settings. While this offers some initial insight, we agree with the reviewer that real-world experiments and stability analyses can be next steps. These remain open directions, as our current work focuses on the theoretical challenges posed by heterogeneous preferences, where the first-order correction illustrates one way in which relaxing consistency can lead to efficient learning that was otherwise impossible due to Thm. 6.2. We do not present it as a definitive solution, but as a step toward deeper exploration of this trade-off and hope to encourage further work into practical and scalable approaches for handling diverse preferences.\n\n> Experiments for robustness under noisy annotations.\n\nWe agree that evaluating the robustness of the consistent loss function under noisy annotations is an interesting direction. Given the short rebuttal period, we have simulated this in our synthetic experiment in Sec. 7.1. To model noisy preferences, for a noise level $n \\le 1$, we sample the preference of a type-$u$ user from a random user type with probability $n$. We then compare the learned policies under different noise levels with the optimal policy (OPT), reporting total variation (TV), Kendall’s $\\tau$, and Spearman’s correlation. The latter two measure ordinal similarity between policies.\n\nAs the table below shows, the TV between the noisy $\\pi_{\\rm consistent}$ and OPT drops below that of DPO and OPT only at high noise levels ($n \\ge 0.7$). Even at those levels, $\\pi_{\\rm consistent}$ remains ordinally closer to OPT than DPO’s policy. These early observations suggest that the benefits of consistent losses may be robust to annotation noise. However, we emphasize that these findings are limited to synthetic settings and may not generalize broadly, but surely motivate further theoretical and empirical work on the robustness aspect.\n\nImportantly, this robustness does not mitigate the sample inefficiency of direct alignment methods: Our key impossibility result still discourages their use in settings with many user types. We thank the reviewer for this question and will include this discussion in the updated paper.\n\n| Metric             | DPO   | $\\pi_{\\rm consistent}$ (noise=0.0) | noise=0.1 | noise=0.2 | noise=0.3 | noise=0.4 | noise=0.5 | noise=0.6 | noise=0.7 |\n|--------------------|-------|------------------------------|----------|----------|----------|----------|----------|----------|----------|\n| TV                 | 0.089 | 0.023                        | 0.044    | 0.066    | 0.073    | 0.081    | 0.084    | 0.088    | 0.090    |\n| Kendall's $\\tau$   | 0.526 | 0.933                        | 0.888    | 0.757    | 0.710    | 0.668    | 0.649    | 0.618    | 0.563    |\n| Spearman's $r$     | 0.676 | 0.989                        | 0.976    | 0.912    | 0.863    | 0.814    | 0.811    | 0.762    | 0.727    |"}]

Author rebuttal:

Meta-review: This paper investigates alignment under heterogeneous human preferences, formalizing the problem using user types and analyzing the limitations of the common homogeneity assumption. The authors show that aligning a single policy to multiple user types reduces to optimizing an affine combination of rewards, but standard Direct Preference Optimization (DPO) instead aligns with the Normalized Borda Count, which can diverge from the average reward. To address this mismatch, the paper introduces (i) a first-order corrected DPO that improves performance with minimal annotator information, and (ii) consistent loss functions that learn the optimal policy when full annotator information is available. Both theoretical analysis and experiments on Pew survey data, synthetic tasks, and semi-synthetic LLM fine-tuning (Llama-3-8B, Qwen-2.5-7B) illustrate the importance of modeling heterogeneity and highlight a fundamental tension between consistency and sample efficiency in direct alignment methods.

The reviewers agreed that the problem is timely and important, with valuable insights into preference-based alignment. Strengths noted include the clear identification of the mismatch between DPO and average reward, the introduction of corrected objectives under different information settings, and a fundamental impossibility result. Concerns focused on the limitations of the theoretical framework, particularly its reliance on the Bradley–Terry preference model (yY38, wgAn), the strong data requirements of the consistent loss objectives (wgAn), and the lack of large-scale or real-world empirical validation beyond semi-synthetic settings (cLLj, Fxtn). The authors adequately clarified how their results generalize beyond BTL, and supplemented the rebuttal with new synthetic experiments. 

Overall, I recommend acceptance. The work provides a rigorous and timely contribution to the study of pluralistic alignment, clarifying both what is possible and what is fundamentally limited when aligning to heterogeneous preferences. Despite some practical concerns regarding scalability and empirical breadth, the theoretical insights and accompanying experiments significantly advance understanding in this area. Please incorporate all the suggestions of the reviewers in the final version as promised.
Decision: Accept (poster)