Title: Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
Venue: NeurIPS 2025
Accepted: yes
Keywords: social choice

distortion

RLHF

PPO

DPO

Nash Learning from Human Feedback
Abstract: After pre-training, large language models are aligned with human preferences based on pairwise comparisons. State-of-the-art alignment methods (such as PPO-based RLHF and DPO) are built on the assumption of aligning with a single preference model, despite being deployed in settings where users have diverse preferences. As a result, it is not even clear that these alignment methods produce models that satisfy users \emph{on average} --- a minimal requirement for pluralistic alignment. Drawing on social choice theory and modeling users' comparisons through individual Bradley-Terry (BT) models, we introduce an alignment method's \emph{distortion}: the worst-case ratio between the optimal achievable average utility, and the average utility of the learned policy. The notion of distortion helps draw sharp distinctions between alignment methods: \emph{Nash Learning from Human Feedback} achieves the minimax optimal distortion of $(\frac{1}{2} + o(1)) \cdot \beta$ (for the BT temperature $\beta$), robustly across utility distributions, distributions of comparison pairs, and permissible KL divergences from the reference policy. RLHF and DPO, by contrast, suffer $\geq (1 - o(1)) \cdot \beta$ distortion already without a KL constraint, and $e^{\Omega(\beta)}$ or even unbounded distortion in the full setting, depending on how comparison pairs are sampled.


Peer reviews:
comments: SUMMARY:
This paper investigates the distortion, which is the ratio of the maximum expected utility and the expected utility achieved, of several preference alignment algorithms. The results include the lower bounds and upper bounds of algorithms without/with KL-constraints to a reference policy. Specifically, the paper proves that NLHF achieves the best distortion, while RLHF and DPO suffer a larger distortion.

STRENGTHS AND WEAKNESSES:
## Strengths

- The writing of the paper is clear and easy to follow.
- The results sound novel to me. This paper shows that the final performance of NLHF (expected utility) might be an exponential multiple of that of RLHF. The reason is that RLHF cannot handle users with heterogeneous feedback since it encodes all preference data into a single utility vector. This further justifies the motivations of work focusing on RLHF with heterogeneous feedback, such as [1] and [2].

## Weaknesses

- This paper does not have any experiments to corroborate the results.
- All results of RLHF are limited to the single-preference RLHF, where the algorithm ignores the source of each preference data. In other words, the RLHF considered in this paper does not consider heterogeneous preferences.

[1] Zhong, Huiying, et al. "Provable multi-party reinforcement learning with diverse human feedback." arXiv preprint arXiv:2403.05006 (2024).

[2] Park, Chanwoo, et al. "Principled rlhf from heterogeneous feedback via personalization and preference aggregation." arXiv e-prints (2024): arXiv-2405.

QUESTIONS:
Intuitively, what is the reason that NLHF outperforms RLHF? Is it because NLHF optimizes on matrix $M$, which contains more information than the estimated utility vector used by RLHF?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper initiates a study of distortion in the alignment setting. Their model assumes a finite alternative set $A$ of size $m$ and, $n$ user each with a utility vector $u_i \\in [0,1]^m$ drawn from a common prior distribution $\\mathcal{D}$. Pairwise comparison queries are generated by sampling two alternatives i.i.d. from a distribution $\\mu$ over $A$. And then all the users provide $d$ noisy comparison according to the Bradley-Terry model with a shared temperature $\\beta$. \n\nThe authors then distinguish two regimes: one with no KL‐divergence limit on the learned policy, and the other with. They call them social choice v.s. alignment settings.\n\nBoth RLHF and NLHF are analyzed in each regime. Their core technical tool is a linearization lemma that bounds the deviation of the win‐rate of $x \\succ y$ from a a half by affine functions of $x,y$'s social welafare. The principal conclusion is that NLHF admits strictly tighter upper bounds on distortion than RLHF under both unconstrained and KL‐constrained setups.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper intiates a series of important questions regarding distortion, and it is outstanding in the sense that it takes social choice literature as inspiration but also goes beyond it--- it takes KL constraints into account. \n2. The results are extensive. I particularly like the discussion section as it partially addresses some of the concerns i was having.\n\nWeaknesses:\n1. The clarity can be improved:\n\nline 96, maybe stress that : the objective is to find an alternative $x \\in A$, because with the original alignment techniques, a policy is able to generate new alternatives. I was unsure which case you are addressing until i read many more lines.\n\nline 115, it took me a while to figure out/guess what you meant by saying the \"expectation is taken over the pairwise comparisons\".\n\nline 117, 156, the social welfare of a rule not only depends on the rule itself, but also on the pairwise comparisons distribution as well as the users distribution, but both notations really did not reflect this. I understand that the notations would not look so nice, then you could introduce a shorthand. \n\nline 253, maybe change the wording \"the distortion guarantee is at least...\"\n\n2. Please see my questions.\n\nQUESTIONS:\nYour contribution is novel and timely. However, I still have several major concerns. I am happy to revise my evaluation once they are addressed:\n\n1. Dependence on m.n.  \n\n  In Theorems 2 and 7, if one allows $m \\propto n^2$ and $\\mu$ to be uniform $\\mu(x)=1/m$, the second term in the finite-sample distortion bound appears to grow without bound as $n \\to \\infty$. Could you discuss how \\(m\\) and \\(n\\) must scale relative to one another to ensure the bound remains meaningful? I am a little bit uneasy about the correctness of your statements. \n\n2. Behavior of distortion at extreme $\\beta$  \n\n  Intuitively, one might expect a U-shaped dependence on $\\beta$  : As $\\beta \\to 0$, comparisons become nearly random, losing information; as $\\beta \\to \\infty$, even a tiny utility gap yields deterministic comparison outcomes (e.g. originally $u_i(x)=0.3; u_i(y)=0.2999$, but a very high $\\beta$ would make it almost always the case that $x \\succ_i y$), also not reflecting the original preference intensity. \n  Yet your distortion bounds only worsen for large $\\beta$ , with no analogous degradation at small $\\beta$. Could you clarify why the bound does not reflect information loss in the first regime?\n\n3. Reference policy in the KL-ball \n\n   In Theorem 6, instead of exhibiting a single adversarial reference policy for the lower bound, is it possible to bound the distortion between an arbitrary reference policy and all policies within its KL-ball? Also, in Theorem 6 you refer to “a sequence of alignments”; could you clarify the intended meaning and why you chose this phrasing?\n\n\n4. Role of the post–Theorem 5 discussion\n\n  The paragraph following Theorem 5 appears to critique the distortion framework even this paper is centred about distortion. I didn't fully get why you made this discussion. It was nice that you have already adopted a new, alignment-oriented distortion measure, and drew a line between the traditional setting (as introduced around line 128) to your new alignment setting. Were you just trying to stress that the social choice setting is really not important for the alignment practitioners? \n\nAdditionally, I was also a bit disappointed by the paragraph starting from line 128: first, it's very good that you have mentioned the context-dependent aspect of alignment, but then you directly make an assumption instead of discussing the alternatives (even though you have mentioned this as future work in later section). Second, i wished to see more comparisons/motivations for your model v.s. a tranditional social choice model, for example, you make many more distributional assumptions and allow each user to provide only $d=1$ or $2$ comparisons, which differs fundamentally from the full-ranking assumption in traditional social choice.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsider RLHF as a way of aggregating preferences across a population. Then a basic question which this paper considers is whether current alignment methods such as DPO and RHF reliably lead to high average utility across users. In short, the conclusion is no: stochastic ordinal preferences don't contain enough information to ensure high average utility across a heterogeneous user population.\n\nMore specifically, the distortion in the case where there's no KL regularization is at least linear in $\\beta$ — that is, the temperature term in the Bradley-Terry model, which quantifies how deterministic user preferences are. When KL regularization is added—so that we're now considering standard ROHF or DPO—the distortion becomes exponential in $\\beta$.\n\nIn short, the paper presents a strong case that standard alignment methods will fail to respect the preferences of the average user, let alone minority users.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nOverall I find the theoretical argument compelling: the assumptions stay true to the Bradley Terry model and properly incorporate KL regularization, and \\emph{lower} bounds on the difference of \\emph{average} utility are more compelling than results which only consider the worst case. \n\nWeaknesses:\n\nYet even after reading the paper carefully, I am left wondering what the actual implication for alignment is? I suspect that this is largely a gap in writing that could be addressed during the discussion period. For instance section 5 seems to suggest that the issue is primarily the presence of KL regularization to a single reference policy. Having a short section which very clearly states for alignment practitioners what the theory suggests should be different would greatly improve the readability of the paper. I think that in this case, asking for extensive empirical results would be outside the scope of the paper (the theoretical results are already compelling as they stand, and the experiments themselves would be expensive enough to deserve their own paper anyways), but \\textbf{the paper should at least make the question clear, enabling empirical follow-up research}.\n\nQuality:\n\nThe overall quality of the paper is high. The writing is clear, the notation is consistent with other papers on RLHF alignment, and Figure 1 adds a lot of intuitive understanding.\n\nClarity:\n\nI predict reviewers will be pretty divided on the clarity of the paper, depending on their background. Personally, I think it’s very well written given the subject matter. However, readers from a strong empirical background will likely struggle to identify the thesis, since it’s not presented in a typical \"Figure 1\" style with bolded performance results that beat a baseline. That’s why I propose including a clear section at the end (and foreshadowed at the beginning) that concludes with a precise prediction for an experiment in a follow-up paper.\n\nSignificance:\n\nI think the paper, even in its current form—lacking any crisp predictions—still manages to have moderate significance. It clearly argues that the burden of evidence should be on showing that RLHF or DPO are fair, given that we have every theoretical reason to believe they are not. The significance would be greatly improved by including a single crisp prediction that could be explored empirically, in which case I would raise my score.\n\nOriginality:\n\nTo the best of my knowledge, the bounds presented here are new.\n\nQUESTIONS:\nThe confusions I have largely concern what the practical implications of the work are (I found the theorems themselves clear).\n\n1. Suppose you had two users A and B, and use DPO to train a reward model R_mix over a 60/40 mixture of the two user’s preferences. Is it possible to classify R_mix from R_A or R_B (trained on each user’s preferences alone) using the lower bounds on the distortion (Theorems 5, 6)? That is, drawing samples from the original population (mixture for R_mix, A for R_A, etc), computing the sample mean distortion, and classifying based on those distortions. If so, this could potentially serve as a post-hoc measure of how heterogeneous the preferences are.\n2. The discussion in Section 5 seems to suggest that simply replacing KL divergence with some other regularization from the reference policy is insufficient to reduce the distortion. But what if multiple reference policies were used (e.g. different base models, or different checkpoints of the same model), and the regularization consisted only of the minimum KL from each reference policy? So long as no reference policy puts a tiny amount of probability mass on c (line 289) wouldn’t this improve the lower bound in Theorem 6?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"distortion\" as a novel framework to theoretically analyze the performance of AI alignment methods when faced with diverse and conflicting human preferences. Distortion quantifies the worst-case gap between the optimal average utility achievable and the utility attained by a given alignment method, which learns from probabilistic pairwise comparisons modeled by the Bradley-Terry model. The research provides a comparative analysis of major alignment techniques, demonstrating that NLHF is robust and achieves the minimax optimal distortion guarantee of $(1/2+o(1))\\beta$. In contrast, methods like RLHF and DPO are shown to suffer from significantly worse distortion, which can become exponential or even unbounded in the general alignment setting that includes a KL-divergence constraint from a reference policy. Ultimately, the work establishes that a certain level of distortion is unavoidable, revealing a fundamental information bottleneck when relying solely on ordinal feedback from a heterogeneous population.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe paper's findings are significant because they draw a sharp theoretical distinction between different alignment approaches. The result that RLHF/DPO can have exponential or even unbounded distortion under certain conditions, while NLHF remains optimally robust, is a powerful and potentially impactful conclusion. This provides strong theoretical justification for exploring alternatives to the current RLHF paradigm.\n\n\n\n### Weakness\n- RLHF results rely on a significant mismatch between the distribution of comparison pairs $\\mu$ and the reference policy $\\pi_{\\rm ref}$, the authors should justify this assumption on real-world problems.\n- The paper is entirely theoretical. While this is not inherently a weakness for a theory paper, the lack of any experiments means the practical implications of the distortion bounds are left to speculation. Empirical studies could help determine if the identified failure modes for RLHF occur in practice or if the constants in the distortion bounds (e.g., $\\beta$) lead to meaningful differences on real-world problems.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}