Title: Direct Alignment with Heterogeneous Preferences
Venue: NeurIPS 2025
Accepted: yes
Keywords: Direct Alignment

Pluralistic Alignment

Heterogenous Preferences

Social Choice Theory
Abstract: Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous. We formalize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption.
We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine improvements under different information settings, focusing on direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficient consistent direct loss exists in this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the challenge of learning an optimal policy from preference data when preferences are heterogeneous across multiple user types. The authors show that standard DPO methods, which assume homogeneous preferences, fail to capture this heterogeneity, leading to suboptimal policies. To address this, the authors theoretically prove that learning a consistent policy without annotator information is impossible and propose methods to overcome this limitation by leveraging minimal annotator information, such as paired preferences from the same user. They introduce a first-order corrected DPO algorithm that incorporates variance estimation to improve policy estimation. Furthermore, they develop a consistent loss function for scenarios with rich annotator data, enabling direct alignment with the optimal policy but at the cost of sample efficiency. Finally, the paper conducts synthetic and semi-synthetic experiments, showing that accounting for user heterogeneity yields policies that better align with average user rewards compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper explores and investigates the critical yet largely underexplored challenge of addressing heterogeneity in preference data within alignment and preference-based learning, aiming to better align language models with human objectives and values.\n2. The paper proposes a first-order correction to DPO and develops consistent loss functions under different levels of annotator information.\n3. The synthetic and semi-synthetic experiments seem well-designed to empirically validate theoretical claims, showing concrete improvements over baseline methods.\n\nWeakness:\n1. Some concepts on personalization and multi-user reward aggregation have appeared in prior work (e.g., EM-DPO, MODPO). However, this paper should set itself apart by providing stronger formal guarantees. \n2. Limited generality due to reliance on DPO framework and BT model assumptions, which may not capture the complexity of real-world preference learning scenarios. \n3. Heavy dependence on rich annotator information leads to impractical data requirements and poor sample efficiency\nThe proposed consistent loss function requires detailed annotator identity information for every sample and assumes all user types label each data point. This results in exponential data demands as the number of user types grows, which is unrealistic in many deployment scenarios where such fine-grained labeling is costly or infeasible.\n4. The practicality and computational cost of the first-order variance correction have not been sufficiently analyzed. While the paper introduces a variance estimator via a joint likelihood model for first-order correction of DPO, the computational overhead and stability of this estimator in large-scale or real-world training remain unclear.\n5. The experiments do not test robustness under more realistic conditions, such as noisy annotations and missing user types per sample. We would like to see that.\n\nQUESTIONS:\n1. The proposed first-order variance correction relies on training a joint likelihood model to estimate the variance term (Eq.16). Could the authors provide a detailed analysis of the computational overhead incurred by this correction, particularly in large-scale settings?\n2. The consistent loss function assumes access to the annotator identity for each sample and assumes all user types label each point. In practice, this is rarely the case. How does the method perform when some annotator identities are missing or when annotations are noisy or incomplete?\n3. The comparisons focus primarily on standard DPO and its corrected variants. However, related works like EM-DPO and MODPO also address heterogeneous preferences.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper studies and formalizes the tradeoff between ambiguity that arises from preference feedback learning and sample efficiency. The paper then proposes a first order correction to DPO, and show that minimal information can improve upon standard DPO formulation.

STRENGTHS AND WEAKNESSES:
Strengths:

a. theoretical as well as empirical results that support the claims

b. the paper concretely formalizes this tradeoff of maximizing average reward vs the average of different rewards and how that conflicts with sample efficiency.
In particular, the authors illustrate how DPO makes a crucial assumption that the average of the preferences is roughly the same as the preference of the average.

c. I also appreciated the insight that this assumption relies on variance in annotator labels, i.e their disagreement, leading to the proposed first order correction.

Weaknesses:

Lack of real world experiments that show this variance/annotator disagreement in practice and the improvement with the correction to DPO formulation.

QUESTIONS:
See the point on real world experiments in the weaknesses section.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the alignment of a single AI policy with diverse human preferences, a setting it calls \"heterogeneous preferences\". The authors argue that the common approach of assuming a single reward function is flawed and propose that the goal should be to optimize the average reward across different \"user types\". They demonstrate that standard methods like Direct Preference Optimization (DPO) fail to achieve this; instead, DPO aligns with the Normalized Borda Count, which can promote mediocre outcomes and is sensitive to the data sampling distribution. The paper proves that learning the optimal policy is impossible without annotator information but introduces solutions for different information levels: a \"first-order corrected DPO\" using minimal data (paired preferences) and a consistent loss function when full annotator feedback is available. Ultimately, the research reveals a fundamental trade-off, showing that any direct alignment method that is consistent in this rich data setting is inherently sample-inefficient, as it must discard data where users disagree.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The paper demonstrates that standard methods like Direct Preference Optimization (DPO) fail to optimize for this average reward. Instead, DPO implicitly optimizes for the \"Normalized Borda Count\" (NBC), a voting rule that can lead to undesirable outcomes, such as promoting mediocre options and being sensitive to preference dataset distribution. \n- The paper tackles a problem of paramount importance: how to create a single AI system that serves a pluralistic society with conflicting preferences. Its primary contribution is a fundamental negative result (Theorem 6.2) that establishes a tension between consistency and sample efficiency for direct alignment methods. This is a highly significant finding, as it delineates the boundaries of what is possible for an entire class of popular alignment techniques.\n### Weaknesses\n- Key impossibility results are proven under the assumption of the Bradley-Terry (BT) preference model, which may limit the generality of the conclusions.\n\nQUESTIONS:\nCould the authors provide the approximation error estimation in Equation (14) and using $V$ to approximate the variance. Since the authors mentioned the Sensitivity to Preference Dataset Distribution in Line 167.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies heterogeneous user preferences and address alignment problems in this setting. Theoretically, it proves that any single policy compatible with multiple user types must reduce to an affine combination of rewards. It further shows that standard Direct Preference Optimization (DPO) is ordinally consistent with the Normalized Borda Count (NBC) rather than with that average reward . To mitigate the mismatch, the authors propose a first-order corrected DPO for minimal annotator information, and a consistent loss that learns the optimal policy when every data point is labelled by all user types. Experiments on Pew surveys, synthetic toy tasks, and semi-synthetic fine-tuning of Llama-3-8B and Qwen-2.5-7B show gains in rewards when heterogeneity is modeled.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper is well written and structured. \n2) The authors identify the problem of heterogeneous user preferences, providing solid theoretical characterizations for the optimal policy, and the 'objective' DPO is trying to optimize. These results provide valuable insights in modeling preference data and designing leaning algorithms.\n3) The PEW Research Center surveys demonstrate that NBC can conflict with average reward in practice, which justify the theoretical findings.\n\n\nWeaknesses:\nThe LLM experiments rely on length-based semi-synthetic rewards. It is hard to judge if the methodology can bring practical benefits.\n\nQUESTIONS:\n1) In practice, what guidance can you give for grouping user types? Do authors have ablation study on the sensitivity to the grouping choice?\n2) If users are mis-clustered or have no types, how do the first-order and consistent objectives degrade?\n3) Can this framework bring benefits if take the preference heterogeneity into account during inference?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}