Title: Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

preference learning

generalization
Abstract: Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference learning, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for ensuring that LLMs align with human values. An essential part of ensuring that LLMs are aligned for all people is accounting for a diverse set of values. This paper introduces a new theoretical framework to analyze how generalization scales with value diversity and sample quantity in models trained with direct preference optimization. Our framework rigorously assesses how well models generalize after a finite number of gradient steps, reflecting real-world LLM training practices. By analyzing the reward margin associated with each sample and its trajectory throughout training, we provide a bound on the generalization error that demonstrates the challenges of effectively learning a wide set of concepts or values. These insights are empirically validated on contemporary LLMs, underscoring the practical relevance of our theory.


Peer reviews:
comments: SUMMARY:
This paper proposes a theoretical framework for analyzing how preference optimization generalizes across data with varying preferences. Assuming a linear model and Gaussian-distributed preference clusters, it leverages the learning dynamics of the implicit reward in DPO to derive bounds for generalization error, which highlights the challenges of effectively learning a broad range of concepts or values. The authors also conduct experiments to validate their data assumptions and theoretical findings.

STRENGTHS AND WEAKNESSES:
**Strength:**
1. Proposed a novel framework to analyze the learning dynamics of DPO.
2. The analysis can be extended to broader preference optimization methods, such as IPO and SLiC.
3. The writing is clear, and the logical flow is well-structured.
4. For a theoretical study, the empirical evaluation effectively supports the claims.

**Weaknesses:**
1. Based on the abstract, one might expect an analysis of generalization to unseen values or preferences. It is somewhat disappointing that the paper focuses only on generalization within the same value distribution as the training data.
2. The assumption of linearity in $y=Wg(x)$ simplifies the analysis and leads to somewhat predictable results, which may also limit the technical novelty of the proofs.

3. Minor issues: The notation in 253 lack $j$ for $\sum_{j=1}^K \beta \log...$

QUESTIONS:
It there any thoughts on how to analyse the generalization for unseen values?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to better understand how LLM fine-tuning using preference optimization algorithms works in the case of diverse human values. The authors present a series of assumptions as to how diverse values are represented within the model embedding space and then analyse DPO in terms of differential equations of the reward margin.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper highlights an important problem, and to the best of my knowledge is the first to analyse the affect of diverse preferences theoretically within the literature. The paper is broadly well written and easy to follow.\n\n**Weaknesses**\n\n- The main contribution of the paper is the theoretical analysis of the learning dynamics of DPO in a setting with diverse human preferences. The theory is very constrained with the main contributions of the paper Theorem 4.2 and Theorem 4.3 consider the setting where only the last layer is considered for a single token. Whilst the authors expand their analysis to the multi-token setting no formal results are presented beyond an intuitive analysis of the gradient in Equation 5.\n\n- The paper verifies the theoretical results with a set of experiments that train both the full model and just the last layer. Whilst the conclusions from these experiments match the theory a detailed comparison of the theoretical bound and the training dynamics is missing. Particularly in the case of the last layer experiments where the bound can be directly related to the key theorems presented in the work\n\nQUESTIONS:\n- Do the verification experiments hold across different model architectures outside of llama e.g. Qwen or pythia models for example?\n- Can a graph be included that directly compares the theoretical bound with the experimental results?\n\nLIMITATIONS:\nA key limitation of the paper is the constrained nature of the theoretical results i.e. an analysis of the last layer training dynamics for only one token. These key limitations are not addressed in the limitations section of the work. The other limitations mentioned are important and the limitation section addresses issues such as generalization of the results to distributions that differ from the training set.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the behavior of Direct Preference Optimization (DPO) when learning from heterogeneous human value clusters. The authors propose a theoretical framework that models preferences as structured Gaussian clusters in a latent space and analyze how reward margin dynamics evolve under finite-step training. A key result is a generalization error bound that scales logarithmically with the number of human value clusters K, suggesting that sample complexity must grow with value diversity. The claims are empirically verified using the Anthropic persona dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\t•\tAddresses a relevant gap in understanding how preference optimization methods scale in the presence of human value diversity.\n\t•\tIntroduces a novel finite-step analysis of DPO, moving beyond infinite-time or asymptotic generalization theory.\n\t•\tThe proposed scaling law (samples per value must scale as \\log K) is an intuitive and potentially useful insight for LLM alignment data collection.\n\t•\tEmpirical analysis on LLM embeddings shows some support for the orthogonal concept hypothesis.\n\n⸻\nWeaknesses\n\t1.\tRestrictive and Idealized Assumptions\n\nThe theoretical framework relies heavily on strong assumptions that are not obviously justifiable in real-world settings:\n\t•\tHuman values are represented as perfectly orthogonal clusters in embedding space.\n\t•\tAll preference data lies on Gaussian blobs centered around these clusters with isotropic noise.\n\t•\tThe shared component b is assumed to be exactly orthogonal to every concept direction.\nThese assumptions may approximate clean experimental setups but likely do not reflect the geometry of real-world LLM representations, where concepts are entangled and noisy.\n\t2.\tSimplified Model Class\n\nThe analysis is performed under a fixed feature map regime, where only the unembedding matrix is trained (not the LLM itself). This decouples the model from the true representation learning dynamics of DPO in practice. As a result, the conclusions may not generalize to full-model fine-tuning, which is common in realistic deployments.\n\t3.\tLimited Practical Guidance\n\nWhile the scaling law is interesting, it is derived under ideal conditions. It is not clear:\n\t•\tHow to translate this into data collection strategies for real-world alignment datasets.\n\t•\tWhether the same scaling persists under non-orthogonal, partially overlapping value distributions.\nMoreover, the probability bounds given in Theorem 4.3 are relatively weak for small to moderate values of Q, making them less actionable.\n\t4.\tSuperficial Multi-token Extension\n\nThe extension to multi-token generation in Section 4.3 is not accompanied by theoretical guarantees or convergence results. It remains at the level of a decomposition, and its connection to the single-token guarantees is hand-wavy.\n\t5.\tEmpirical Support Is Limited\n\nWhile cosine similarity results (Figure 3) loosely support the orthogonality assumption, they are computed on a small subset of personas and are based on average cosine similarities, which may not capture more complex geometric structure.\n\nQUESTIONS:\nQuestions for Discussions\n\t•\tRelax the orthogonality assumption and analyze how approximate linear independence or noisy concept overlap affects the generalization bound.\n\t•\tExtend the analysis to include feature learning, or at least study how changes in the learned embedding layer affect reward margin evolution.\n\t•\tProvide stronger empirical results, ideally on larger datasets and across multiple foundation models, to support the structural assumptions.\n\t•\tStrengthen the multi-token theory or clarify whether it inherits guarantees from the single-token setup.\n\t•\tQuantify how sensitive the bound is to the choice of variance v, cluster separation, and sample size Q.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper models a preference dataset that reflects **diverse human values** as a mixture of \\(K\\) value clusters in embedding space.  \nEach cluster is represented by a pair of responses—one preferred, one dispreferred—that share a common component and differ along orthogonal, value-specific directions.\n\nBuilding on this structure, the authors analyse **DPO**.  They introduce the **reward margin** (the log-ratio DPO seeks to maximise between “good” and “bad” responses) and derive its gradient-flow dynamics.\n\nTwo main theorems follow:\n\n1. **Training guarantee** – after finitely many gradient steps, every training margin becomes positive with high probability (zero empirical error).  \n2. **Generalisation bound** – at that same point, the population error satisfies  \n   \n   $$\n   R(P)\\le2KQ^{2}e^{-Q/45},\n   $$\n\n   where \\(Q\\) is the number of preference pairs per value cluster.\n\nThe bound yields a **scaling law**: to keep error low while increasing the number \\(K\\) of distinct values, the data required per value grows only *logarithmically* in \\(K\\).\n\nIn the last, the authors extend the analysis to multi-token generation, and **small-scale experiments** on the Anthropic Persona dataset with Llama-3.1-8B empirically corroborate both the assumed embedding geometry and the predicted scaling trends.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Quality\n\n1. **Rigorous theory**: Precise assumptions (Gaussian-mixture value model, shared + orthogonal directions) and complete proofs culminating in Theorems 4.2–4.3 give a non-vacuous generalisation bound—rare in preference-tuning work.\n2. **Extensibility**: Authors explicitly show how the gradient-flow analysis lifts to multi-token generation (§4.4) and argue the same template could cover IPO or SLiC objectives (§6).\n\n### Clarity\n\n1. **Clear problem framing and examples**: Paper motivates “pluralistic alignment” with concrete examples (e.g. risk-averseness vs. exploration prompts) that illustrate why a single reward head can suppress minority values.\n\n### Significance\n\n1. **Practical guidance**: The logarithmic sample-complexity law ( $Q=\\Theta(\\log K)$ ) directly informs data-collection budgets for value-diverse alignment.\n2. **Explains failure modes**: Analysis clarifies why alignment sometimes collapses to the dominant preference cluster when $Q$ is too small, matching anecdotal reports in industry.\n\n## Weaknesses\n\n### Quality\n\n1. **Limited empirical scope**: Validation uses a single 8-B-parameter model (Llama-3.1-8B) and one dataset (Anthropic Persona). Scaling claims might break on larger models or more heterogeneous corpora.\n2. **Multi-token “extension” untested**: Section 4.4 is purely theoretical; experiments only examine single-token margins, so the claimed generality remains speculative.\n\n### Significance\n\n1. **Assumption is too heavy**: Orthogonality of value dimensions is empirically approximate at best; if real-world values interact non-linearly, the logarithmic law may fail.\n\nQUESTIONS:\n1. Your theoretical bound assumes that the “concept” directions for the K values are orthogonal and enter linearly.  In practice, many values interact: e.g. harmlessness can conflict with full honesty, and the latent directions may be correlated.\n    - Is it possible to extend your theory if directions are merely approximately orthogonal, or non-orthogonal?\n    - is it possible to provide an ablation empirical experiments to show whether the empirical $Q=\\Theta(\\log K)$ law still holds or breaks down?\n\n2. In Figure 4 you plot the number of preference pairs Q needed per value cluster as K increases and claim it follows a logarithmic trend based on your theory, is it possible to test this scaling law with empirical experiments?\n\n3. (not an important question, just my confusion) Your theory treats value diversity as a mixture of K clusters, each with a well-defined “positive” versus “negative” response. In practice, preference data gathered from an unlabelled, heterogeneous crowd can be:\n\n    - Ambiguous – some annotators prefer A^+, others A^- for the same underlying value;\n    - Unclassifiable – we may not even agree on how many values K there are, or where one value ends and the next begins.\n\nHow does your framework cope when the value space is continuous or overlapping rather than cleanly clustered?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}