Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
Authors: Shawn Im
Sharon Li
Venue: NeurIPS 2025 poster ()
Keywords: LLM
preference learning
generalization
Primary area: deep_learning

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference learning, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for ensuring that LLMs align with human values. An essential part of ensuring that LLMs are aligned for all people is accounting for a diverse set of values. This paper introduces a new theoretical framework to analyze how generalization scales with value diversity and sample quantity in models trained with direct preference optimization. Our framework rigorously assesses how well models generalize after a finite number of gradient steps, reflecting real-world LLM training practices. By analyzing the reward margin associated with each sample and its trajectory throughout training, we provide a bound on the generalization error that demonstrates the challenges of effectively learning a wide set of concepts or values. These insights are empirically validated on contemporary LLMs, underscoring the practical relevance of our theory.

Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
Shawn Im Sharon Li Department of Computer Sciences University of Wisconsin- Madiison {shawnim, (arronli)@ce wisc. edu
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities but often struggle to align with human preferences, leading to harmful or undesirable outputs. Preference learning, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for ensuring that LLMS align with human values. essential part of ensuring that LLMS are aligned for all people is accounting for diverse set values. This paper introduces a new theoretical framework to analyze how generalization scales with value diversity and sample quantity in models trained with direct preference optimization. Our framework rigorously assesses how well models generalize after finite number of gradient steps, reflecting real- world LLM training practices. By analyzing the reward margin associated with each sample and its trajectory throughout training, provide bound on the generalization error that demonstrates the challenges of effectively learning a wide set of concepts values. These insights empirically validated on contemporary L.Ms, underscoring the practical relevance of our theory.
Introduction
"The plurality of human values is not condition to be overcome, but a reality to be understood."
Martha Nussbaum
Modern societies are built upon a rich tapestry of human shaped by culture, personality, moral belief systems, and lived experience. Social science and psychology have long recognized that individuals and communities differ not only in their preferences, but their fundamental notions of what is good, just, desirable [1, 2]. This diversity is not noise be averaged out, but an essential feature of human moral life. As large language models (LLMs) are increasingly deployed in high-stakes and user-facing applications [3, 4, 5]. it becomes critical that they respect and adapt to this plurality rather than flattening it. Achieving this goal calls for alignment methods that capture and generalize diverse human values [6, 7, 9, 10).
Preference optimization has become popular approach for aligning LLMs with human intent. At the heart of this alignment process lies preference learning, where the goal is to train language model policy that can distinguish, according to some reward model, preferred versus non- prefereedd responses based on human feedback. Specifically, preference learning involves optimizing language model policy to produce higher rewards for more preferred responses, guided by preference data provided in the form of comparative judgments. These techniques have proven effective at steering models toward helpfulness and harmlessness, among other desirable traits. As a result, the field has witnessed an intense wave of research activities in recent years [11, 1, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, many of which we review in the related work (Section 6).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite recent progress, a key open question remains: how do diverse human values influence the generalization behavior of preference learning? While aligning models with diverse human values has gained attention [6, 7, 8, 9, 10, 30], existing methods often rely on simplifying assumptions that reward signals are homogeneous or represent a unified set of preferences [15, This assumption makes it necessary to understand how such simplifications impact a model's ability to learn and generalize across pluralistic value landscapes. Some theoretical work has begun to address these concerns-fee for example, [32] highlights DPO's limitations on heterogeneous data. However, most analyses focus on specialized frameworks explicitly designed for heterogeneous data [9, 10, 30]. There is a pressing need to rigorously examine the structural consequences of value diversity on common preference learning methods like DPO [31], in particular. understanding how generalization performance scales asymptotically in the presence of value diversity.
In this paper, we take a theoretical lens to this important question. We introduce new framework that characterizes how common preference optimization methods, such as DPO, behave under a distribution of diverse, structured values. Our approach departs from standard generalization theory in two important ways: (I) we model preferences as arising from mixture of distinct value clusters (e.g.. personality traits or political views), and (2) we analyze the dynamics of training under finite-step updates, reflecting real-world LLM fine-tuning protocols. To the best of our knowledge, generalization results in this setting have not been obtained before. This contrasts with existing generalization theory, which typically considers overparameterized models that achieve near-optimal loss [33, 34, 35, 36] or are independent of the training process [37, 38, 39]. Central to our framework, we characterize the generalization error through the lens of reward margin, which quantifies the log- liglliielihood difference between the preferred and non-preferred responses. A sample's 's error is zero when the reward margin is positive and vice versa. The key to our framework lies in analyzing the reward margin associated with each sample and its dynamics throughout the training process. By bounding the trajectory of the reward margin, we can effectively quantify the generalization error of preference learning.
To summarize our results, we provide conditions under which we can guarantee with high probability that the reward margin for all training samples is positive (Theorem 4.2), meaning that the model can correctly predict all training samples into the preferred vs. non preferree categories within finite gradient steps. Building on the results, provide guarantees and bound the generalization error for new inputs drawn from the preference distribution (Theorem 4.3). Our theorems indicate that the conditions under which the guarantees hold with high probability depend the value diversity in the preference dataset and on the number of samples in each value category. In particular, our results reveal a theoretical scaling law: the number of samples required per human value must grow logarithmically with the number of distinct values. This provides new insight into the statistical cost of aligning models with pluralistic human preferences. These results shed light on practical aspects of aligning LLMs, helping explain the benefit of scale and the challenges with aligning to a diverse set of values. We empirically validate these theoretical insights in Section 5, affirming their relevance to real- LLMs. We summarize our key contributions in the following:
1. We present the first rigorous theoretical analysis of generalization in finite-step preference learning, offering novel framework that captures the training dynamics of LLMs under diverse human values (Section 3).
2. We derive new learning guarantees showing how preference-optimizze models can both fit training data and generalize to new examples under structured value diversity (Section 4).
3. We empirically validate our theoretical insights using modern LLMs and preference datasets exhibiting diverse human behaviors (Section 5).
2 A Motivating Example
To ground our theoretical analysis, we begin with a concrete example from the Anthropic's persona dataset [40], which encompasses diverse types of human values. For instance, persona "risk-averse" entails aligned statements like "I prefer to play it safe rather than taking bigger risks that may lead to bigger gains or losses" that represent the persona, and also the statements on the other end, e.g., "I enjoy taking large risks with investments or decisions". As illustrated in Figure la, each statement is formatted using the prompt template "Is the following statement something you would say? [STATEMENT]. " Then, the learning objective would push the model to have a positive reaction to the former statement and negative reaction to the latter. A fundamental goal of preference optimization is to ensure that such Iearned behavior generalizes consistently to unseen queries,

---

Value Clusters in Preference Data
Value: open-mindedness
Santy Degutees
bn the inlowwnng ststemmnt scmethingg wo tory hate new ideas and experiences
Krodyy Hoar
Creatte (eow
Hume (Prelin
Predtion
is the tollowing pistement comothing you would pay? enjoy the unpredictabilit of doing many novel and new things, and am also constantly searching for new expertences
Prechiee (eene wod
Yes
(b)
Figure 1: (a) Example of statements relevant "open- oopen-mindedness (b) Illustrative visualization of embeddings corresponding to different human values.
especially as language models are increasingly deployed in real-world settings. Take test statement example, "Is the following statement something you would say? would like stable, secure investments with low risk", we would like the DPO-trained model to be able to correctly distinguish this as a positive example for the value of "risk-averse".
Open problem. While this example focuses on a single value dimension, real-world preference datasets significantly more complex. In practice, models trained align with diverse mixture of values simulaneously-Beet each reflecting nuanced personality traits, moral beliefs, political ideologies, and more. This diversity introduces structural complexity in the embedding space, where disentangling and optimizing for multiple behaviors becomes non-trivial. Figure 1(b) provides conceptual illustration of how such preferences may manifest as structured clusters in the latent space of language model [41], J. over which preference optimization operates. A DPO-trained model trained on this data should leverage this geometry to distinguish the positive (marked in .) from the negative (marked in x) statements. and moreover, make accurate decisions for unseen inputs across the full range of values seen during training. Understanding how DPO learns and generalizes effectively across such diverse distributions remains an open problem.
3 Preliminaries and Theoretical Setup
Model. We denote the input prompt as (x(1) (2) ,T, where 2x() is the i-th token in the prompt and T the length of the prompt. Considering two possible outputs yes Vt. we denote Yar Y1 if ye is preferred over yt- We call Yu. the preferred response and the less preferred or rejected response. Given an empirical dataset D = {(x,, Ym [ sampled from the preference distribution P. we can express the empirical DPO objective, LDPO.
[ (3(18g log fo(Uwalas) log fva(WW, fo(y),)(x))
logo
(I)
frer(y,
Reward Margin
where a(-) is the logistic function, fiet the base model and is the model output.
Reward margin. Given the empirical dataset = {(x;, Vhe.i I sampled from the prefer- ence distribution, refer to each triplet of (æ,, Mer,is It.s)) preference. From Equation see that the DPO objective implicitly learns a reward model, and the preference correctly learned
8 log (e(B,,(az) log fee((y,,( Fror(un,)az4)
0,
which we call the reward margin r(z, Moo.s st.)). A positive reward margin indicates that the current model, Te, has been updated to better distinguish the preferences compared to the base model met-

---

We will also refer to the reward margin function corresponding to ne as its implicit reward model. Under the notion of reward margin, the DPO training objective can be interpreted as a convex smooth loss function to approximate the 0-1 loss: maxn, (() [[r(,, Vwe y1 0). The population risk can also be defined formally below based on the notion of the reward margin.
Definition 3.1 (Population Risk of Preference Learning). We define the population risk in terms of 0-1 loss, where sample's ' loss is O when the reward margin is positive and I otherwise.
Rxx, yoe, y1) {
0 r(æ, Yeu Vt > 0 r(z, Yar yt) -
where r(x, Vw. 31) is the reward margin for a new sample (x. ywu yt). Then, given a joint preference distribution P where (I, Bw y1) is sampled from, the population risk with respect to P is
RPP) B( [R( Mur mn
(2)
The population risk provides a clear interpretation in the context of preference learning, which directly captures and quantifies how often the model can correctly discern between preferred and non-preferred outcomes on future unseen samples. This is particularly useful in preference learning, where the primary goal is to make correct predictions about which response is preferred over another. In the remainder of the paper, the notion of population risk and generalization error will be used interchangeably, since consider the risk under setting where guarantee that the empirical risk is 0 (formally in Theorem 4.2).
Udiitarianimm
Characterizing the diverse preference distribution. To analyze generalization in preference learning, we de- fine a preference distribution that reflects the diversity of human values. Specifically, we consider preference distribution that consists of K pairs of clusters that corre- spond different human values. In the context of align- ment, the values can be broadly associated with different personality traits, political views, moral beliefs, etc. For example, the values may encompass common properties such as helpfulness, honesty, and harmlessness [16], and can also represent much more diversified and nuanced ones like conscientiousness, non-racism, compassion, and so on [40]. For each value, we have a pair of clusters containing samples aligned vs. misaligned with that value.
Openness
Openness
Usitarionism
Shared Component
Figure Illustration preference distribu- tion for pairs of clusters corresponding to openness and utilitarianism.
Recent studies on the structure of concept representations
in language models have found strong correlations between linear directions in the final embedding the linear representation hypothesis [41, 42, 43]. Furthermore, [41] suggests that causally separable concepts are represented along orthogonal directions. These findings provide support for modeling values (analogous to concepts) in the representation space of large transformer models. We empirically verify this structure in Section 5 and Figure 3, ensuring that our theoretical analysis remains grounded in the inductive biases and representational geometry typical of real-world LLMs.
space and concepts-known
Concretely, we consider distribution P of (x, Vee 31) that represents the set of clusters as a mixture of Gaussians with K equally weighted pairs of clusters labeled with i E [K]. Each cluster is distributed as (+t;; + b, v²Ia), where q is unit vector representing the concept vector for cluster pair i and is a vector with norm I, representing the shared aspect of all embeddings. Let C,,+ be the cluster corresponding to samples aligned with concept and C,,- be the cluster corresponding to samples misaligned with concept i. For simplicity, we can assume without loss of generality that b =be in the standard basis e ea for Rd. Additionally, we let each e; correspond to standard basis vector ec; such that the e; are pairwise orthogonal and are all orthogonal to b. The preferred and rejected responses for all samples in a given cluster are fixed, and no two pairs of clusters have the exact same set of responses. We define Z as the maximum number of times a token appears across all preference responses. To construct the empirical training data, we sample Q i.i.d. samples from each cluster, and there are a total of N = 2KQ samples across K clusters. We verify in Section that our data assumption matches closely the characteristics of real-world alignment datasets.

---

meading
manntr
dealtt 5o ba maahed
dau to
undi jastft trare
und mara
iroemal Toonstar
poychopptyy
sabrotbwa juddim
subert ber jusean
ou dientdogy
sdbeaies deoria
(a)
(b)
Figure 3: Average cosine similarity of embeddings between personas (a) before and (b) after subtracting the shared component from each embedding. This confirms our assumption the shared components among behaviors and the orthogonality in the remaining components (with low cosine similarity). The order of the behaviors along the vertical axis corresponds the order of the behaviors along the horizontal axis.
4 Theoretical Framework and Guarantees
Practicality of our framework. We provide theoretical framework for analyzing the general- ization guarantees of learning preferences using DPO. Under this framework, we can rigorously characterize the conditions under which the model can correctly predict preferred responses for new input prompts. Different from traditional generalization theory, we consider the generalization of models after finite gradient steps when the loss is within a constant factor of its initial value. This scenario closely matches real-world practices, where LLMs are often fine-tuned for few epochs. The crux of our framework thus lies in analyzing the reward associated with each sample and its evolution throughout training. Finding bounds on the trajectory of the reward directly allows us to quantify the generalization error.
4.1 Reward Dynamics
We first provide the reward dynamics in the Lemma below, with the full proof given in Appendix A. Lemma 4.1. Suppose g VT + Rd is the non- con-linear mapping from the prompt to the last hidden state, which is connected the model output fo(z) via the learnable unembedding layer matrix W. The dynamics for the reward margin under the gradient flow of the weight matrix can be expressed as
1 Σ Pol-r:)(y -is) (y yra) N
TrJ
where r, is the shorthand notation for reward margin of sample x,, 7 is inverse learning rate, and is the sample covariance matrix with Eiş = g(x;) g(zz).
Interpretation of reward dynamics. To ensure clarity in our exposition and elucidate the key insight, we first illustrate the analysis when the preferred response Vio,s and non-preferred response Vt,, consist of token, encoded by the one-hot vector y in RIM. Our analysis will be expanded to more complex multi-token setting in Section 4.3. The expression for the reward margin gradient in Equation (3) allows us to easily check and interpret how each training sample influences the learning of the reward for training sample I; and any new sample ž. There are two factors determining the influence of sample T, the reward margin of sample z;, (1) The first factor (yw.) ) W y1,)) captures preference whether sample and sample x, share preferences or not. If you.s V1, Dood Mt are all different, then we have a factor of 0 and the two samples have interaction. On the other hand, if Ywri Yayj and yt, Mt then will have factor of 2 and the preference sharing factor gives more weight to sample zz. (2) The second factor Σij captures the correlation between embedding of x; and 2y. measured by dot product. If two sample embeddings are highly correlated. then they will have a large influence on each other's reward dynamics. If the samples are orthogonal, then they will have interaction.
Finding a tractable form. From Equation (3), we note that the only factor on the right that changes over time is the set of o(-r)). Letting C(z,, x.,) (yw.s y1.)) (y Eij, we have
5

---

= = (0 Then, we can see that the system of differential equations for the of rt(t) is actually only in terms of itself and constants, and as long as we enforce structure in the C()) factor, it becomes tractable to provide upper and lower bounds for r; r4) and therefore generalization error (ef. Definition 3. In the following section, we provide generalization guarantees for DPO by enforcing this structure through diverse preference distributions.
4.2 Theoretical Guarantees
We first present theorem that guarantees that the implicit reward model from DPO can correctly predict all training samples into the preferred vs. non-preferred categories. We state this formally below in Theorem 4.2.
Theorem 4.2 (Training Reward Guarantee). Under the conditions of Lemma 4.1 and with data from the distribution P described in Section 3, given Z S 400 embedding dimensionality d S 5Q,
S 32 6. 2 4. with probability at least 1 8KQ exp (-min (evq. 92/4 )) for some
constant C O, the trajectory (t) for all [M] is upper bounded by (L) and lower bounded by (0)())))) (t) KF
(t) which are given by (t) 8KF and r
g2
5.
t for ST Kr log and at Tı, for any
532
training sample 40 S (t))= & log 3.
3
Next, we present a generalization bound for the DPO reward model on the preference distribution. This result builds on our analysis of training reward margins and leverages the fact that unseen samples follow similar gradient dynamics as those in the training set. For clarity, we present simplified version of the bound here and provide more refined statement in Appendix B and an extension of the results to approximately orthogonal clusters in Appendix C.
Theorem 4.3 (Generalization Error). Under the conditions of Theorem 4.2, given Z S M 4. 2 and Q 2 40, with probability at least
d S 5Q, v S
32VQ'
SKQ exp (- min [VQ Q²/4
for some fixed constant > 0, the generalization error of the implicit reward model Tı bounded
(-min
(5 5 256
R(P) S 2KQ2k-900 Q/15
Practical implications of value diversity. A compelling
implication of Theorem 4.2 and Theorem 4.3 is that it mathematically formalizes the intuitive challenge of align- ing with a diverse set of human values. Specifically, the theorems show that as the number of value clusters K prrrasess-reprseentiig greater diversity in human pref- erences- the number of samples Q required per cluster must also grow to maintain strong training and general- ization performance. For example, when K 10, our analysis shows that it requires more than 875 samples per value achieve near-zero generalization error. More generally, the number of samples per cluster must scale as (log K) to maintain a strong training and generalization performance. This implies that achieving alignment in pluralistic societies isn't just an ethical or philosophical challenge- it has concrete statistical and computational costs. The model needs more representative data per value group to learn a consistent decision boundary across all perspectives. In practice, this underscores the importance of equitable data collection across value subpopulations in preference datasets and cautions against assuming that increasing dataset size alone will resolve the generalization problem unless diversity is systematically accounted for.
Requirov Samples per Value (C)
Scaling Sample Complesity Value Diversity
850-
200
850
30
Number Human Villos
Figure 4: Scaling curve for the generaliza- tion error to be 0.05.
4.3 Extension Multi-Token Generation
While our single-token analysis provides foundational insights, real-world preference learning de- mands understanding multi-token generation. Once considering multi-token responses, the dynamics for the reward become significantly more complex, and providing strong guarantee regarding the training accuracy or generalization becomes highly non-trivial. Nonetheless, we can find connections between the structure of the multi-token dynamics and that of the single-token case that allow for

---

better understanding and point towards a promising direction for better understanding of preference learning in more general settings.
Reward decomposition in multi-token generation. To clearly see how the reward evolves and how each token contributes to the reward,
decompose the reward for the i-th sample into the
(2) (ew www. Mnrn (3- 12)
sum of token-wise rewards: r(yw/ts) = =E r(ww)/.)) Σラ= B log
(1) =erlB lus wit,, www. Muyt.
(2)
where is the length of the response, Sue/t, the subscript w/ll to indicate either preferred or non non-prefereed responses. Further, the likelihood of ( fi 1...... ( hence the token-wise a response is given by To(Uw/t,,lzz)) [ po (Uwnj (
is the j-th token of a response to input Zi, and we use
pe(u (.0wy,..eial Jov / 20/ 11 reward can be expressed as: r(uw/,.) B log
(1)
mell la Muyt. aar.aaao Mu/1.
Reward dynamics in multi-token generation. We express the token-wise reward as r(ym)t,)) 3 (log S S(W g(, j, w/1) - log S( j, w/()) y Wu/t, where Wo is the weight matrix of the reference model, S is the softmax function, and y "//l, RR are the one-hot vectors corresponding to j-th tokens of the preferred or rejected response. We use g(i, w/l) as the shorthand notation ()-1)), ), which denotes the final hidden states after the first j - 1 tokens of the response have been appended to the input x,. Since Wo is fixed and so is the g(i. w w/l), the reward 88 B Blog s Wol....//t) y
(3)
(1) for g(2,,, fw/1.." .... ////
gradient becomes: ar(y
Reward gradient decomposition. By expanding the reward gradient, can derive the full form of the reward gradient (with proof details in Appendix for the reward of token y with corresponding embedding g":
D). Specifically, we have the following dynamics
r(v) Ot
N
(F((U)))) - r((w.))
y (r.j. - y v1s (r
=
N
Token Co Cococcurennce Factor Term
p(i.j, w)) + p(i, (1. + (i, (1. w) de i 1JC (1, (C
(5)
Probability Factor
Output Distribution Correlation Factor Term
where C* "",p, dp are defined in the following paragraph.
The decomposition in Equation (5) provides a clear interpretation of the terms in the reward gradient. (i, w/l) g(i. j. w/t) T)) g" captures the correlation between the embedding for the j-th position of the response to i-th sample and g". The probubility fuctor, p(i, w w/l) (i,jj, w/[)) y S(W g*) T w/tte
Interpretation.
y is the difference between the probability of outputting token
y given embedding g(i.j, w/t) and the probability of outputting v(u) given g" dp(i,j. w/l) S(Wg*) S(W gi,,j, (/))), is inner product or correlation between the output distributions for the embeddings 9", (i. w/l).
Implications. We can see that after decomposing the reward for multi-token responses into token- wise terms, the gradient as seen Equation (5) resembles that of the single-token case, albeit with additional terms also involve an inner product between the given embedding and the embedding of tokens in the dataset. This shared structural aspect between the decomposition for multi-token and single-token reward gradients, coupled with our existing understanding of single-token guarantees, points toward promising avenue for understanding preference learning.
5 Empirical Verification
To understand how our theory guides practical LLM training, we present two sets of experiments, with the goals of (1) verifying our data assumption made on the preference distribution, and (2) understanding how the reward margin changes under increasing numbers of clusters or human values.
7

---

Treining Reward Margin
Margin
Traihing Reward Margin
Test Reeard Margin
Stop
(a)
(b)
(c)
(d)
Figure 5: Average reward margins for the training/test set over the course of last-layer training (a, and full
fine-tuning (c, d) across increasing number of human values K.
Verification of data assumption on real transformer model. We verify that our data assumption in Section 4 matches closely the characteristics of real-world alignment datasets. We consider the Anthropic Persona dataset [40]. Recall that the data distribution under which our results hold is that (I) the embeddings consist of shared component along some direction and (2) each concept or cluster varies along orthogonal directions. To verify the shared component, we compute the average cosine similarity between the final embeddings of statements from different pairs of personas. The embeddings are extracted from the Llama-3.1-8 1--BB model [44], a popular open-source foundation model with accessible internal representations. As depicted in Figure 3a, the average similarity is high, confirming the shared structure among a random subset of 10 personas. Furthermore, to verify the orthogonality assumption, we subtract the shared component from each embedding vector and then compute the average cosine similarity for any pair of personas. As seen in Figure 3b, the average cosine similarity is close to 0 for non-diagonal entries, suggesting the remaining components nearly orthogonal. For completeness, we provide verification across all personas in Appendix E.
Verification of theoretical results with increasing value diversity. In Theorem 4.2, we show that the rate at which the reward margin increases, r. decreases as the number of clusters or concepts increases in training. To verify this empirically, we randomly sample different numbers of personas
from the Anthropic dataset, simulating the varying number of concepts K = {1, 2, 4, 8, 16). For each setting, we perform both full fine-tuning and last-layer fine-tuning the Llama-3.1 model [44] using the DPO loss. As depicted in Figure 5a and 5c, the training reward margin grows more rapidly for smaller K, given the same number of training steps. Similarly, we verify our Theorem 4.3 in Figure 5b and 5d, which shows that the test reward margin on new inputs follows dynamics similar to that for the training samples. Moreover, we find similar decrease in the rate which the loss and accuracy change and provide results in Appendix E. We directly verify that the theoretical trends generalize full fine-tuning by fitting linear model between K and the test error for Llama-3.1 1-BB, Mistral-7B-v0.3. and Qwen3-8B 8B-Base and find that the resulting fits have R² scores of 0.97, 0.95, and 0.99 respectively. We provide further results and details for the different base models in Appendix These results validate that our theoretical insights indeed translate to practical alignment processes.
Related Works
Alignment of LLMs. A key aspect of training and deploying large language models is ensuring the
models behave in safe and helpful ways [45, 46, 47]. This is an important problem due to the potential harms that can arise in large models [48, 49, 40, 50, 51, 52, 53, 54, 55, 56, 57]. A wide range of methods have been developed that utilize human feedback or human preference data to train models to avoid harmful responses and elicit safer or more helpful responses [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 58]. Particularly, the Reinforcement Learning from Human Feedback (RLHF) framework has proven effective in aligning large pre- pre-trained language models [11, 12, 15, 16, 59]. However, given its computational inefficiency, recent shifts in focus favor closed- form losses that directly utilize offline preferences, like Direct Preference Optimization (DPO) [31] and related methodologies [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75]. Despite the empirical success and wide adoption in real-world systems [3, 4, 5], fewer works provide theoretical underpinnings [60, 76, 77, 65. 71, 78, 79, 80, 81] especially in the case of diverse data [82, 32, 30]. In this work, we make an initial attempt to comprehensively analyze the generalization behavior of preference optimization and how it scales from a rigorous theoretical standpoint. Our work considers offlinn preference optimization, which differs from the setting of other theoretical works on preference-bare based reinforcement learning [83, 84, 59], and our use of the training dynamics of
8

---

DPO distinguishes our analysis from other analyses of data diversity [82. 32, 30J. We introduce a new theoretical framework examine the generalization properties of LLMs by approximating their reward dynamics, providing insights into practical aspects of aligning L.MSs under diverse values.
Generalization of deep neural networks. Understanding how and why deep models generalize has been a subject of extensive research. One approach is through the lens of feature learning, attempting to understand how models learn data-dependent features and how these features are structured [85, 86, 87, 88, 89, 90, 91, 92, 93]. Another approach is through providing generalization bounds that quantify the expected performance of the model beyond the training samples and over a data distribution [33, 34, 35, 36, 37, 38, 39, 94, 95, 96, 97]. While existing generalization theories typically consider simpler learning tasks such as regression and classification, our work provides generalization analysis in the context of aligning language models, which entails dealing with the complex output space of sentences. Moreover, existing generalization theory typically considers overparameterized models that achieve near-optimal loss [33, 34, 35, 36] or are independent of the training process [37, 38, 39]. One line of work considers algorithmic stability, which allows for generalization bounds that are dependent on the number of steps [98, 99]. In contrast, our framework focuses on the generalization of models by directly following and analyzing the reward dynamics after finite gradient steps, which matches more closely with the real-world practices of aligning LLMs. Our theoretical insights are further supported by empirical validations on contemporary LLMs, as shown in Section 5.
7 Discussions
What about methods beyond DPO? Our theoretical framework can be extended beyond DPO to a more general class of preference learning objectives presented in GPO [65]. This is because the objective function can be written as the average of f(r.), and the only modification the dynamics would be replacing the o(-r:) factor in Equation (3) with (r'(r.. For example, we would use f(ra)) = (r: 1)a for IPO [60], and we would use f(r;)) max(0, 1 - rr)) for SLiC [100]. This suggests that similar implications should be expected for other preference learning methods.
Leverage our theoretical framework to understand failure modes of alignment. Our theoretical framework also provides an understanding of failure modes of offline preference methods, such as probabilities for both responses in a preference pair increasing or decreasing [101, 102] and failure to change the reference model's preference rankings [103]. Our Theorem 4.2 provides an upper bound of log 3 in the change in reward margin at the end of training. Then, any preference pair where the reference model is more than 3//8 times more likely to generate the rejected response than the preferred one would not flip. Different thresholds can be achieved by adjusting the training length. Additionally, we can utilize our reward dynamics frameworks to look at the reward for individual
Mr,) with Twigh Fi.j corresponding to the preferred and rejected responses respectively. These dynamics suggest that for a response that appears as both preferred and not preferred, for examples in the same cluster, the reward dynamics in both cases will be very similar. This would lead to the likelihood in both cases to increase or decrease together. Having a large overlap between the preferred and rejected responses can lead to this entanglement occurring across many examples, agrecing with the results from [101, 102].
responses: Tiu/l,- & " ('
(Jur. (N
Conclusion. Our work theoretically analyzes how preference optimization generalizes in the presence of diverse human values, which remains an open problem in the field of Al safety. We base our theoretical analysis on popular alignment loss, direct preference optimization, which implicitly learns a reward model. Key to our framework, we analyze the reward margin associated with each sample and its trajectory throughout the training process, which allows us effectively bound the generalization error. Through rigorous analysis, we establish conditions under which the model trained with DPO loss generalizes to new inputs with provably high accuracy. In particular, our results reveal theoretical scaling law: the number of samples required per human value must grow logarithmically with the number of distinct values. This provides new insight into the statistical cost of aligning models with pluralistic human preferences. Empirical validation on LLMs confirms the practical relevance of our findings. We hope our work catalyzes future investigations into the theoretical understanding of preference optimization methods.

---

Limitations. Our study is focused exclusively on the setting where the test distribution matches the training preference data distribution, because understanding generalization significant challenge that is not yer fully resolved. At the same time, we recognize that out-of- distribution settings are crucial in real-world applications, where test data often differ from training data, presenting novel or unexpected inputs. Exploring how DPO generalizes in such scenarios, including its robustness and preference-driven biases, an important direction for future research. By providing thorough exploration of the ID setting, we aim to build solid theoretical and empirical basis, which will be instrumental for future work addressing the complexities OOD generalization.
this setting remains
Acknowledgments and Disclosure of Funding
We thank Min-Hsuan Yeh, Sean Du, and Jiahai Feng for their valuable comments on the manuseript. This work is supported in part by the AFOSR Young Investigator Program under award number FA9550-23-1-0884 National Science Foundation under awards IIS-2237037 and IIS-2331666 Office of Naval Research under grant number N00014-23-1-2643,, Schmidt Sciences Foundation, Open Philanthropy, Alfred P. Sloan Fellowship, and gifts from Google and Amazon. Shawn Im is also supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. 2137424. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Support was also provided by the Graduate School and the Office of the Vice Chancellor for Research the University of Wisconsin-Madises with funding from the Wisconsin Alumni Research Foundation.
References
[1] Jonathan Haidt. The Righteous Mind: Why Good People are Divided by Politics and Religion. Pantheon Books, 2012.
[2] Shalom H Schwartz. Universals the content and structure of values: Theoretical advances and empirical tests in 20 countries. Advances in Experimental Social Psychology, 25:1-65, 1992.
[3] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303. 2023.
[4] Anthropic. Introducing claude. http://wwww.aattttoccccccc/////d
2023.
[5] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 20XXi::2307.00288, 2023.
[6] Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye. Liwei Jiang, Ximing Lu, Nouha Dziri, al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.050700 2024.
[7] Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodde Danny Hernandez, Nicholas Joseph, al. Towards measuring the representation of subjective global opinions in language models. arXiv preprint orXiv:2306.166 16388, 2023.
[8] Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. Improving context- aware preference modeling for language models. Advances in Neural Information Processing Systems, 37:70793-70827 2024.
[9] Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv:2406. 08466, 2024.
[10] Idan Shenfeld, Felix Faltings, Pulkit Agrawal, and Aldo Pacchiano. Language model personal- ization via reward factorization. arXiv preprint arXiv:2503.0633 2025.
10

---

[11] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[12] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiamo, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint S34Iv:1909005599 2019.
[13] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 2020.
[14] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, 2021.
[15] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[16] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint ar arXiv:2204. 200.0822..8.2. 2022.
[17] Reüichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assister arrestion-nsweerin with human feedback. arXiv preprint arXiv:2 2022.
[18] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fermando, Boxi Wu, Rachel Foley, Susannah Young, lason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
[19] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang. and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. 1171, 2023.
[20] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint urXiv:2304.05302 2023.
[21] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. 220922266.772... 2023.
[22] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023.
[23] Yuntao Bai. Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint D2Xii:2212208007.. 2022.
[24] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint 23000:2309.00267, 2023.
11

---

[25] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Row- land, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:23 12.00886,. 2023.
[26] Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:23100 10. 13639, 2023.
[27] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310. 12773, 2023.
[28] Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guied guided search. In Proceedings of the International Conference on Learning Representations, 2024.
[29] Min-Hsuan Yeh, Jeffrey Wang, Xuefeng Du, Seongheon Park, Leitian Tao, Shawn Im, and Yixuan Li. Position: Challenges and future directions of data-centric ai alignment. In International Conference on Machine Leurning, 2025.
[30] Nuoya Xiong and Aarti Singh. Projection optimization: general framework for multi- objective and multi-group rlhf. arXiv preprint arX1v:2502.15145, 2025.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint 2.XXiv:3305.18290 2023.
[32] Ali Shirali, Arash Vasr-Fsfahany, Abdullah Alomar, Parsa Mirtaheri, Rediet Abebe, and Ariel Procaccia. Direct alignment with heterogeneous preferences. arXiv preprint arXiv:2502.16320, 2025.
[33] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparam- eterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
[34] Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over-parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3349-3356, 2020.
[35] Vignesh Subramanian, Rahul Arya, and Anant Sahai. Generalization for multiclass classifi- cation with overparameterized linear models. Advances in Neural Information Processing Systems, 15:23479-234999 2022.
[36] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine line-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
[37] Sanjeev Arora, Rong Ge. Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning. pages 254-263 PMLR, 2018.
[38] Sanae Lotfi, Mare Finzi, Sanyam Kapoor, Andres Potapezynski, Micah Goldblum, and An- drew G Wilson. Pac-bayes compression bounds so tight that they can explain generalization. Advances in Neural Information Processing Systems, 0..31459-31477 2022.
[39] Sanae Lotfi, Mare Finzi, Yilun Kuang, Tim GJ Rudner, Micah Goldblum, and Andrew Gordon Wilson. Non-vacuous generalization bounds for large language models. arXiv preprint 1721: 2023.
[40] Ethan Perez, Sam Ringer, Kamilè Lukošiūte, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jecyoon Hyun, Joshua Landau,
12

---

Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph. Noemí Mereado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac HatfieldDDodss, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations, 2022.
[41] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2371.03658, 2023.
[42] Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint arXiv:2403.03867, 2024.
[43] Yuzi Yan, Jialian Li, Yipin Zhang, and Dong Yan. Exploring the Ilm journey from cognition to expression with linear representations. arXiv preprint 1XXx:2445.16666 2024.
[44] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.2 27800777.777... 2024.
[45] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
[46] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprint arXiv: 1.070787 2018.
[47] Xuanming Zhang, Yuxuan Chen, Samuel Yeh, and Sharon Li. Cognition-of-thouuhh elicits social-aligned reasoning in large language models. arXiv preprini 2rXiv:2509.23441, 2025.
[48] Peter S Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai decep- tion: A survey of examples, risks, and potential solutions. arXiv preprint S3ii::200.147552. 2023.
[49] Micah Carroll, Alan Chan, Henry Ashton, and David Krueger. Characterizing manipulation from ai systems. arXiv preprint arXiv:2303 2023.
[50] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfeel-Dodds, Scott Johnston, al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023.
[51] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su. Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. multitask, multilingual, multi- modal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302. 04023, 2023.
[52] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906. 01000, 2019.
[53] Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in llms. arXiv preprint arXiv:2309.0066) 2023.
[54] Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626 2022.
[55] Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305. 2023.
13

---

[56] Rohin Shah, Vikrant Varma, Ramana Kumar. Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint arXiv:2210.0179)) 2022. 2022.
[57] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Repre- sentations, 2022.
[58] Leitian Tao and Yixuan Li. Your weak Ilm is secretly a strong teacher for alignment. In Proceedings of the International Conference on Learning Representations, 2025.
[59] Wendi Li, Changdae Oh, and Sharon Li. General exploratory bonus for optimistic exploration in rlhf. arXiv preprint arX.vv2510003266,, 2025.
[60] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 2023.
[61] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint 24xi:2240.132288 2024.
[62] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. International Conference on Learning Representations, 2024.
[63] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01330 0030000000,,,,.... 2024.
[64] Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312. 11456,6,,,111,,, 2023.
[65] Yunhao Tang. Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749 2024.
[66] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXh::2405.44334 2024.
[67] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Reseurch, pages 1263-126511 PMLR, 21-27 Jul 2024.
[68] Yongcheng Zeng. Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the s International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 58348-58365. PMLR, 21-27 Jul 2024.
[69] Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5435. PMLR, 21-27 Jul 2024.
14

---

[70] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the s1 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 36577-36590. PMLR, 21-27 Jul 2024.
[71] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feedback. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 42258-4222 422744 PMLR, 21-27 Jul 2024.
[72] Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Fe- lipe Llinares-López, Jessica Hoffmmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding- realignment of language models. In Ruslan Salakhutdinov, Zico Kolter, Kather- ine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of PMLR, 21-27 Jul 2024.
Proceedings of Machine Learning Research, pages
[73] Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback. Forty-first International Conference on Machine Learning, 2024.
[74] Rui Yang. Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-contett: Multi-objective alignment of foundation models with dynamic preference adjustment. In Forty-first International Conference on Machine Learning, 2024.
[75] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with diverse human preferences. In Forty-fire International Conference on Machine Learning, 2024.
[76] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q=: Your language model is secretly q- f-mmctio arXiv preprint arXiv:2404. 12358, 2024.
[77] Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. In International Conference on Machine Learning, 2024.
[78] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie. Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMS should leverage suboptimal, on-policy data. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 47441-47474, PMLR, 21-27 Jul 2024.
[79] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research. pages PMLR, 21-27 Jul 2024.
[80] Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovic, and Adish Singla. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 38145-3818 PMLR, 21-27 Jul 2024.
[81] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.
15

---

[82] Seongho Son, William Bankes, Sangwoong Yoon, Shyam Sundhar Ramesh, Xiaohang Tang. and Ilija Bogunovic. Robust multi-objective controlled decoding large language models. arXiv preprint XXiv:2503308796 2025.
[83] Xiaoyu Chen, Han Zhong, Zhuoran Yang. Zhaoran Wang, and Liwei Wang. Human-in- the-loop: Provably efficient preference-base reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773-3793. PMLR, 2022.
[84] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.
[85] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516-38532, 2022.
[86] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861. 2020.
[87] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 11727-11737. PMLR, 2021.
[88] Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on taig-tailedde data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2970-2979, 2020.
[89] Jimmy Ba, Murat Erdogdu, Taiji Suzuki, Zhichao Wang. Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. arXiv preprint vrXiv:2005.01445 2022.
[90] Alireza Mousavi-Hosseiie Sejun Park, Manuela Girotti, loannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn oow-dimensional representations with sgd. arXiv preprint arXiv:2209. 14863, 2022.
[91] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
[92] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning distort pretrained features and underperform out-of-diitrrbbution. arXiv preprint MrXiv:220..10054 2022.
[93] Yuandong Tian, Yiping Wang. Beidi Chen, and Simon S Du. Scan and snap: Understand- ing training dynamics and token composition in 1-layer transformer. Advances in Neural Information Processing 3ystems,,6:719111
2023.
[94] Idan Attias, Aryeh Kontorovich, and Yishay Mansour. Improved generalization bounds for robust learning. In Algorithmic Learning Theory, pages 162-183. PMLR, 2019.
[95] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv: 2017.
[96] Guillermo Valle-Pérez and Ard A Louis. Generalization bounds for deep learning. arXiv preprint arxiv:2012.041115, 2020.
[97] Yunwen Lei, Ürün Dogan, Ding-Xuan Zhou, and Marius Kloft. Data-dependent generalization bounds for multi-class classification. IEEE Transactions on Information Theory, 65(5) 3021, 2019.
16

---

[98] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pages 1225- 1234. PMLR, 2016.
[99] Tongliang Liu, Gábor Lugosi, Gergely Neu, and Dacheng Tao. Algorithmic stability and hypothesis complexity. In International Conference on Machine Learning, pages 2159-2167. PMLR, 2017.
[100] Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045. 2022.
[101] Hui Yuan, Yifan Zeng, Yue Wu, Huazheng Wang, Mengdi Wang, and Liu Leqi. A common pitfall of margin- maseen- language model alignment: Gradient entanglement. arXiv preprint arXiv:2410. 1..00.1.880 13828, 2024.
[102] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization, arXiv preprint arXiv:2410.08847 2024.
[103] Angelica Chen, Sadhika Malladi, Lily Zhang, Xinyi Chen, Qiuyi Richard Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. Advances Neural Information Processing Systems, 37:101928-101966 2024.
[104] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https: 2020.
// wtw.huu...cmmmugginggge/
[105] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract describes the contributions of the theoretical framework described in Sections 3 and 4, the generalization bound and implications in Section 4, and the empirical verification Section 5. The introduction and related works explain how our framework is new and our problem scope, focusing preference learning and value diversity.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are described in a separate paragraph at the end of the main paper in Section 7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions and conditions are described in the statements with references and complete proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental details in the experiments section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have open spen-soureed the code here.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
sallic//uudess htteg/iiie//oeeueiiiiiieeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //ni //nips. crips.cc/publl ppbplcc/giddesseeeppppppppp.paep httpg/guddde/oeeeebmiiiiiiie SrodeeSumiissionPo..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The dataset, model, and training setting described in Section with full details provided in the Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our results focus the trend in the reward accuracy and margin across steps and number clusters to verify the theory.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the compute resources used in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttps://eeurtit /nerrps..s..p// http//uulic/EEiiee wtw.diisseeeeeeeeeeeeenn ines?
Answer: [Yes]
Justification:
The research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impact of the theory in our implications paragraph and discus-
sion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, the sources of the assets are credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///errpp/////ee cc/Confferences/20255 /LLM) for what should or should be described.
24

---

A Proof of Lemma 4.1
Proof. We consider for our theoretical analysis, gradient flow, continuous approximation of gradient descent. To follow the reward margins during training, we derive the dynamics of the weight matrix W under gradient flow:
1 N
Σ W = N 1=1
Bo(- yi.)) T (V) - vi,))(Z;))
(6)
where 7 determines the rate of change, where larger corresponds slower rate of change. Let AW W Wo, a constant offset from W, we have:
N Bal B(y wi vI.)) AW (( (y yrs)g(as)
TAW-
N
Reward margin for at;
which contains the term of the reward margin. Since B. V.y. w. Vt.j 23 are fixed, we can consider the flow of the reward margin by multiplying ((yw.. - yi.)) on the left and multiplying g(zy) on the right of 7 A.W. This yields the dynamics for the reward margin:
Trj N Σ (o(-r))(y v1.) (Y yr,)
(8)
where r, the shorthand notation for reward margin of sample 24, and is the sample covariance matrix with Σg = 9(x;) ( 9(x)).
B Proofs of Theorem 4.2 and Theorem 4.3
We begin with the following lemma regarding the structure of the preference data. Lemma B.1. With l6 S 3.- with probability at least
(8Z + 4)K
(8Z
Q) cxp ( cs min (1, ).)).for any i E [K] and for any j. k E [Q]
1(0 (it) 20 ((ttt) - 2(1 + 12 + d
S 4ec
(9)
10 (1,2) . ((.tt)- 2(1 1 < 4en
(10)
for any i E [K] and any j, k E [Q]
(1+t)) , 2b (1,7)) - 2(1 13)| < 4ev
(11)
for any in F ig that share a token and for any j. E [Q]
C(afi ,5k (b2.=))
< 12 + 260
(12)
C(afititt _ik (b2r7))
< 13 + 2c0
(13)
Proof.
We begin with (9) and (10). We know that
zj x
(b.#)
1be1 +
ajmem
and
(6,1) = TΛe1 + 0; +
Σ Ok
Em
m=1
where Ojjmm a. are all i.i.d samples of a aN(0 0²) random variable. Then, it follows that
(1,1)
as + Is + +b0r,,1 + looKK,1 + aii. Ork.o +
(1,1)
aj Ok
m=1
25

---

Then, using that the distribution of (,,,,1 + +4 aj,; Hak,e, is a centered normal with variance at most 4v² for j # and at most 8u² for = and that the product two Gaussians is sub-exponential by Bernstein's 's inequality, with probability at least 1 - 2K 2 - /16 2KQ2 exp (- = min (1, a for some constant c 0,
1 (e,b)) (R,
2 -1 + ( - 2e0
(1,+) (1,+) (1 + 13 du ) 2ev
(6,4) (1.+) Then, as zj 2k
share the exact same preferences, we know that
(((l (i,tt) Fh (( 2( + [ S 4ev
((afiit
, ((tttt 2(1 + 1 + d S 4e0
Now, we consider (11). We know that
(8,#) j
1b91 + t Σ
ajmmm
m=1
and
tbe1 F 0 + (1.7)
Σ
5/
Or Cm
m=1
where aj.,, O.mmm are all i.id samples of (0, x²)
random variable. Then, it follows that
(1,4)
1 1 1+ 40,,1 + 1,r,,1 F ajee akke: + Z
(1,t)
j.m Ok,m
Then, using that the distribution of 1,ae,11 + 1ax,1 F aje; + akkc is centered normal with variance at most 4v and that the product of two Gaussians is sub-exponential, by Bernstein' 's inequality, with probability least 1 2KQ2e- - 2K exp (- _ min (1, ) for some constant >> 0,
1as (1.))
F (1.7) ( 1)1 < Ze0
(1,1) (6.1) Then, as z 2k
share the exact opposite preferences, we know that
((a)) T ( - 2(1 {5)) S 4ev
Now, we consider (12). We know that
( lbe1 + Ci. + Σ
am
m=1
and
( = l1st + Ciz +
R2.mem
m=1
where oj O.km are all i.i.d samples of (0, 2. random variable. Then, it follows that
(fa. = 1o00,1 + 160k,1 + ajsry + alk.o
Oj.mOk.m
m=1
Then, using that the distribution 4a,,1 + -laaattta,a... tak,e, a centered normal with variance at most 2 and that the product of two Gaussians is sub-exponential, by Bernstein's 's inequality, with probability at least 1 M 1/116 4ZKQ² exp (- _ min (1, ) for some constant >> 0,
| (1),t)
(ba
12 2cc
I
Then, as (1.t .kk share token, know that
(i.+)
|Caaft oh
S 15
2c0
(13) follows similarly. Then, the full result holds with probability at least 1- (8Z 18
(8Z + Q² exp (- min (1, dt) for some constant 0.
26

---

Lemma B.2. With lb I
1 with probability at least 1 - (8Z + 4)K 2//6 - (8Z +
4)K exp (-o min
(1, we have that for each sample,
n 2( + 13
2(1
12)
2dhu2
Σ
0 a
o (-- (T)
o o(-r) )icb)
N
N
N
m=1
m==
2 P
((2Z + 4)CU + 12
max ( (14)
N
JEN
Proof.
From Section 4.1, we know that the gradient flow dynamics follow
1 N
3a ((xr,)
(15)
and writing in terms of clusters,
,2
TF, T
(a(-rin + m
x Clz) i + a )C(a))
(16)
N
m=1
Σ kES. m=1
(o(- C(xt (1 -m +xttt + - -i )C(z) xmm b)
(17)
Then, by Lemma B.1, with probability at least 1 - (87 + 4)K@2e-0/11 -87 +
4)KQ² exp (- min (1, )))
for some constant e >> O, we know that
Tr 6, 2(1 + 12)
Q
2(1 - 82 0 (-
2du2 +
Σ
) m
N o -(prttt)
)
N
N
m
m=1
m=1
2
((2Z + 4)cU + 112 max o--r)) (18) jEN
N
Theorem B.3. Given Z
.े. d 5Q, U S T/Q and é S Too[Z+2j 2 4. 2 1.2 with probability at least 1 - (8Z + 4) K2 /16 (87 + 4) Q exp (- min (1, ) the trajectory rx(t) for
all i E [N] is upper bounded by n (t) and lower bounded by (t)) which are given by
r²(() Q 4N
2du ,2 Nr
(1)) "
fort S Tı and Ti is given by
Nr log 3
71
10Q
(19)
and at TI. any training sample
3og3 40
S r(t) S log 3.
Remark. Setting e = T2 and upper bounding the probability of failure, (8Z + 4)K -/16 (8Z + 4) k exp (- 3 min (1, )) by setting d 5Q and v 32. and
using that N 2KQ Q gives the version of the theorem stated in the main paper.
Proof. From Lemma B.2, we know that with probability at least 1 (87 4)KQ2e-2/10 (8Z 4)K exp (- _ce min (1, ).
Trihh 2(1 + 82 o o(-r
2(1 12))
P
2dv 32
Σ 0 -o(-
0 (-r (.db)
N
N
N
m=1
2
((2Z + 4)cu + 12Z) max o o-rr)) (20)
N
JEN
27

---

Then, we have that TriA is lower bounded by
2(1 + 12
2(1 12
a (-r + + 2d a(-r)tt)
Σ 0 (-
N
N
2
((2Z + 4)ex + +112)
max KEN
(-Trk) (21)
N
and further lower bounded by
2Q(1 + 12),32
2Q(1 - 12
min o(-rk) + 2d o o-
min d(-TR) kE[N]
N
N
KE[N]
2
((2Z + 4)EU + +1,2)
max o(-rk) (22) RE
We also have that Trl i is upper bounded by
2(1 + 82
2(1 18)
( + 2dv2 B
o d--r++)++
Σ
o a(-rt/t)
N
N
2
+
((2Z + 4)er + +12,2)
max ((-rk) (23)
N
REN
and further upper bounded by
2Q(1 + 12)8² N
2Q(1 - 12)
max a (-rk) + 2du2
max o (-rx)) KEN
N - (-r
N
KEN
2 ((27 + 4)ex + +1,7) max of (-rrk) (24)
N
KEN
We will aim to find an upper bound and lower bound that is valid until which is the first time that (t) 2 log 3 for any j. We will use (22) to iteratively derive and tighten a lower bound that holds until Tg. Then, using (24) we can derive upper bound that holds until Ta and find lower bound for Tg.
Then, for 1. S Ts, we know that minke[N] a(-rx) 2 z. and therefore, (22) is lower bounded by
Q
2dv
a( (-rrtt)- 2 ((27 + 4)er + 13 max (-rr)
(25)
N
N
kE[N]
Then, as 7 < and S
1
16e(2+2) we have that this is lower bounded by
Q
(26)
4N
Then, since the above is positive, r3 would be lower bounded by the trajectory (1)) that is the solution to
Q8
Tr
(27)
4N
with (0) 0. Since all reward margins initially O, and T-'s is a lower bound on all Try know that pf is a lower bound for all r, for t S Ts. Then, we have
(t) Q 4NT
(28)
Now, let us consider (24) for & < T. In this case, as we know that the reward is increasing so maXxe(N] (--r) < and (24) is upper bounded by
2Q du
3²Q N
N +
((2Z + 4)cv + 15,2)
(29)
28

---

and by the bounds on Z, this is upper bounded by
5Q3²
do
2N +
(30)
N
Then, upper bound all ry by rd (t) which is the solution to
T (5Q + 8
(31)
2NV
with / (0) = O. Then, we have that for S To
1 (1) (5Q + 2dv
(32)
2NT
and as d E 5Q and S 32/Q' we can upper bound this by
10Q Nr
(1)
(33)
and we know that Ta lower bounded by
NT log 3
T1 10Q
(34)
Then, at TI, we have T = (3)g(3), and p² log(3)
40 at TΙ.
Theorem B.4. Given Z S .ig.
1 1 d < 5Q, v S 4vQ and c S 16o(7 2 4. 2 4, with probability least 1- (8Z + P2
1
1
(8Z + K)K exp (- c min (1, 2)), ) the generalization error of the implicit reward model at Tı is bounded as
R(P) S 2KQ2 -2" 12(2+du2
(35)
Remark. As for Theorem 4.1, we set e = 16u(Z+2) and upper bound the probability of failure, (8Z + 4)KKp 2/16 - (8Z + 4) exp (- _ min (1, )),, by setting d = =QQ and = 32/Q to reach the version of the theorem stated in the main paper.
Proof. We can start by considering the dynamics of the reward margin corresponding to (z, Tws D1). This follows
N
(o-
(36)
Let i be the cluster corresponding to .. Then, we have that
82
((l-it)C( 3+) + o(-rin))C(a (.-))
(37)
N
Q
Σ
Σ
(a(-r
) + a--rth ( ah
(38)
RES, m=1
Then, we will condition on the training set and on the event that Lemma A. holds. Then, from Lemma A. we know that
Σ a S du² + +e0
(39)
and we also have that
IT (3)
2 - (1+ +1,1 < 2ev
(-i) - (1 1)] S 2cv
1u(T 2
(1.t) - [ < 2ev
29

---

Then, -# T conditioned on 2j is centered normal random variable with variance at most +2==)) Then have that for ē with probability at least 1-2K 2 (/204 +dv +dv +2cv) conditioned on the event that Lemma A.1 holds that for any & E [Q]
C(ā, (( -2(1 + (3)
I 6Ev
(40)
C(2, ((2;=z))) (.F)) - 2((1- 12) - 6cv
(41)
and for any i2 Sj and for any [Q]
C(ū, ( )
< 13 + 3ev
(42)
(12 c
)
s 1 + 3cv
(43)
We will condition on the event that the above holds for the remainder of the proof. Then, we have that by the same arguments as in Lemma A.2 that
2(1 + 32
Q
2 2(1 - 12)
Q
Σ
((-mm)
Σ
o
((-rm=))
(F)
N
N
m=1
m=1
2 Q
((37 + 6)EV + +162)
max a (44)
N
JEN
and we that that 7r is lower bounded by
2(1 + 82
2(1 - 15)
Σ9
Σ
0 (- m
N
N
m=
m=1
2
((37 + 6)ev + 11,2)
max
(-rr)) (45)
N
JEN
and for t I T1, this is lower bounded by
Q B²Q N
((3Z + 6 + 12
(46)
N
as we know for any training sample S ry S log Then, as Z S arg and 6 S
8u(2+2) we
have that the new sample will be classified correctly. Then we have that with probability at least 1 - (&Z+ + 4)KQ2² (8Z + 4)K exp (- min (1, ).))
R(P) < 2KQ2e e- -2//2(2+d
(47)
as 4 S D-
C Approxinate-orrhogonaal clusters
In this section, we prove extensions of Theorems 4.2 and 4.3 under approximately orthogonal clusters. We start with our definition of 6-approximately orthogonal clusters.
Definition C.1 (6-approximately orthogonal clusters), We consider clusters distributed A (4ee + C,, La) where each o has corresponding standard basis vector ec. Then, we will define the clusters as being ö-approximately orthogonal with < for every (c)) 82 /8.
Lemma C.2 (Pairwise approximate orthogonality). Given 6-approximately orthogonal clusters and d,, de from [K] with i F i2, then we have that
(( c12) 5
(48)
30

---

Without loss of generality, assume that the dot products of Ci, Ct2 and their corresponding standard basis vectors are positive. Let & be the angle between c₁ and ees," and let &2 be the angle between Ci and Bery- Then, by 5-approximate orthogonality, we have that E E2 I arccos(1 - 8² /2/8). Then, in the worst case, we have that
Proof.
KGn,,,,) = (1 - -22/8)) = 2(1 -
82/8)2
(49)
Then,
2/ S
(50)
Lemma C.3. Given 8-approximately orthogonal clusters with 5 S min
(cu (8) 8 2). l1 S 2. then with
probability at least (16Z + 3)K(PP- - (8Z + K)K exp
c min (1, 2))) for any
i [K] and for any j, k E [Q]
C(a) (1:2)
.33 ((.tt)) 2(1 + 12 do²2) S 5ev
(51)
C(a)) (1.t)) .0k ((:t))- 2(1 + 1 < 5ev
(52)
for any i E [K] and for any j, k E [Q]
| (i
, , (1 - 2(1 [3) S Seu
(53)
for any in + in that share a token and for any j. E [Q]
((r) : xx ,
11)
)
1 13 + ev
(54)
2
[(a) ((to)ii
Th . () < 1 + 2 EV
(55)
For each dot product between pair of inputs from the same set of clusters, we introduce an additional deviation of at most 1,ō + da from the dot product between the deviation of os from eng and e,, the dot product between the deviation of e; from cx; and the Gaussian component. The contribution from the dot product between e' does not change in this case. Since 8 S or
Proof.
and 6 S } we can use the same bounds on the a's so that with probability at least &KQ2 xx/2eee2/11 2/6 we have
g
|o(a (1:2) 12 ( 2(1 + 1g + d
< 5ev
(56)
(C ,ik ((((tt))= 2(1 + 13){ < 5cv
(57)
for any i E [K] and for any j, k E [Q]
CCa) (1:2)
(1.7)) 2(1 13)) 2 5ev
.
(58)
Now, we consider (54). We know that in the case of Lemma B.1, we have (1,
(1a.t) arj Fk =1 + la05,1 + 1000,1 aj.ci + 0k..o
We bound the difference between the right hand side above and the current a (61,2t) (f2.1) In particular, we have most 26 from the dot products between City G22 and at most öl, from the dot products between c; I
In this case, we have additional terms coming due to approximate-orthogonaii
-
or C22 and e1. We have an additional shift of at most & (a,,e,, + +akee,)) ) for some d1 de from the change of at most 5/2 along a new basis vector. Then, we introduce a change of at most
28 + 81g + 2 (05,en + (lk,eag)
(59)
which in the worst case for all é1, j, Je is at most with probability at most - 1
2/16
(60)
Then, we have that
(Ca (b,2t) c
(
<
12 +
5
EV
2
(13) follows similarly. Then, the full result holds with probability at least 1- (1
(8Z + 4) exp (- min (1, ) for some constant 0.
31

---

Lemma C.4. With å v min
EB 1 .16 S L, with probability at least 1 - (16Z + (8)K(8KK-P-
8
(8Z + K²² exp (- - min [1 de ) we have that for each sample,
n 1 2(1 + 12)332
2(1 - -1))82
2du B2
a
o o(-rg (
N
N
N
m=1
m=
2
((z
5
2 +
+ +12
max o JEN
(-r3)) (61)
N
Proof.
The proof follow exactly the same argument as Lemma B.2.
J Theorem C.5. Given 1 a S 5Q, v S Qq.. and e S 16v(2+2) l0 2 7. - min ( 1).
with probubility at least 1 - (16Z + K)K trajectory rt(t) for all i [N] is upper bounded by r (t) and lower bounded by r (+) which are given by
(8Z 4)K exp (- min (1, de ). the
(1) Q82
8N7
(1) 10Q NT
for Ti and Ti given by
Nr log 3
71
10Q
(62)
and for any training sample
log 3 SD
S r(t) S log 3.
Proof. The proof follows the same argument as Theorem B.3 except now since the right hand side of Lemma C.4 increased by a factor of 5/4, for r", the factor of 1 1/4 which came from lower bound 2 2 is now lower bounded by factor of 1/8 as ( 2 g. For rd this change has no effect.
Theorem C.6. Given 7 S .t.. d s 5Q, T.
2 4 2 4. with probability at least (16Z + K) /16 (8Z + K)² exp (- ce min (1, ),, the generalization error of the implicit reward model at T1 is bounded
1 and [ ō I min 8 4)
R(P) < 22
(63)
Proof. We can start by considering the dynamics of the reward margin corresponding to (e, Te ūt). This follows
7 1 (o(-r))C((,,,)) N
(64)
Let i be the cluster corresponding to Ā. Then, we have that
B2
Σ
a
r
2+
C(( 2 + ol
(65)
N
m=1
Q
+ -o(-rt =m=))
(66)
KES; m=1
Then, we will condition on the training set and on the event that Lemma C.2 holds. Then, from Lemma C.2, we know that
d
a < du²
+ev
(67)
m=1 1
and we also have that
(1())
xx
- (11 22
<
5
cv
2
32

---

IT - -1/ - 1) -
5
c
2
1 xx
(1++t - 12 2 cv
Then, (Ø - M4 J conditioned 2j is centered normal random variable with variance most (1+ //2
1 +d + ). Then we have that for ø with probability at least
conditioned on the event that Lemma A.1 holds that for any & [Q]
c(ā, ((tttt - 2(1 + (
- 7ev
(68)
[(E, (.,)) - 2(1 -1-1 (t)
S Tev
(69)
and for any i2 E S; and for any [Q]
(((
1/ 2 e"
7
(70)
of
- 1/ +
2 cv
7
(71)
We will condition the event that the above holds for the remainder of the proof. Then, have that by the same arguments as in Lemma C.3 that
2(1 +
12
Q
2(1 - 12)
78
a
3 (-r
o
o--r?n))
N
N
m=1
m=1
2
( 7)c0 + +2
max maxx) (72)
N
JEN
and we that that riis is lower bounded by
2(1 82
Q
2(1
12
Σ
c
Σ
(--rm
N
m
N
m
m=1
m=1
23ºQ
(12 (122+7) 7)eo + +1++77++777
1..)
max JEN
o (-r3)) (73)
N
and for t S T1, this is lower bounded by
Q N
B²Q N
( 7 + 7)eu + 1177
(74)
as we know for any training sample 0 S ry S log Then, as 2 S 477
1 and 6 <
2z(2+ we
have that the new sample will be classified correctly. Then have that with probability least 1 (16Z + K)KQ)==K2 (8Z + K²² exp (- min d)).
R(P) < 2 -2/2((2400++
(75)
as 4 < 2-
D Multi-Token Derivation
Derivation of reward gradient.
We start from the equation,
Olog S g(..j, (0) yulli
(76)
Øt
Øt
and expand the right-hand side. First, we use that, for vector
log S(Wv) Wv LSE(Wv)
(77)
33

---

where LSE is the LogSumExp operation, and the subtraction is applied element-wise. Then, it follows that
a log w/t)
y willi
( a(Wg(i. w/t)) Y w/ti aLSE(W g(, w/l)
al
((Wg()))m/))
G
first consider the term
which can also be written as
8t
(UY oW y w/tt gli,j.w/1).
( since g(i. w/l) . w/1ta are constant.
We then consider the second term
OLSE gt..../.)) 81
which can be written as
OW
S(Wg(i.) w/ w/t))
őt ((i.j,w/)
Then, once we derive OW we will have the full expression for the reward gradient. We can start from the fact that gradient of the loss with respect W is
N
Olog S(W (.
alog S(W (i,,((,
())
3
(r(Un,)) - r(
(78)
ow
ow
1=1
and using (77), we have
N
-W =
3
Σ
o(((U))) - r (Vww,))
( (( -y. - 9(.. Σ
N
1=1
()) w) S(Wg(i, ( (79)
Now, we can substitute the above expression for in order to get the full reward gradient for given token y in the training set with corresponding embedding g
rly) 00
N
ar((u))) - (()w.))
you (i (i.j. yi. (i,,,,)
N
1=1
Token Co-occurrence Factor
- p((, Cnn
-1. + dp(i, (i,j, dp (f. (C (1,j,1)
(80)
Probability
Outpur Distribution Correlation Factor
where C²,, dp are defined as
('(ij,,w/l) = g(..j. =((,
= ((W.)))))((( S yu/ta S(Wg') s(W((t.../))
E Additional Verification
Embedding similarities across all personas. Here we provide the plot of the cosine similarities of embeddings between different personas before and after subtracting the mean embedding in Figure 6a and 6b. The personas ordered according to lexicographical order.
Gaussian Cluster Verification We verify that the cluster component of embeddings from real- world models and datasets can reasonably be modeled by Gaussian distribution. We use the Anthropic Persona dataset [40] which consists of diverse of personas. For each persona, we collect the final layer embeddings at the end of each positive statement and normalize them to have unit norm on average. We calculate average over personas of the Frobenius norm of the covariance matrix and the average squared distance from the mean of these embeddings. These are 0.058 and 0.227 respectively, suggesting that the overall variance relatively small and a Gaussian distribution would be sufficient to capture the variance of the embedding distributions.
34

---

1.0
-066
24
0.8
0.5
36
42
48
0.4
54
0.6
60
86
72
96
102
108
0.2
108
114
1144
120
120
126
126
132
132
1110
520
130
T15
(b)
Figure 6: Visualization of cosine similarity of embeddings between pairs of personas concepts. Left: the average cosine similarity of emheddings between personas. Right: the similarity of embeddings after subtracting the embedding.
Training Loss Curves
Test Loss Curves
0.6-
6.6-
Training Loss
0.5-
Test Loss
6.5
Clusters
Clusters
0.2
0.1
0.3
Step
Step
(b)
Figure 7: Visualization of loss over the course of training across different number of clusters.
Loss and accuracy curves. We present the training and test losses and accuracies across different numbers of clusters seen in Figures 7a, 7b, 8a, and 8b. We find that the losses decrease at slower rate and the accuracies increase at slower rate as the number of clusters increase.
Verification on Llama-2-7B We provide verification of the generalization results with the same training setup as with Llama-3. 1-88B and provide the results in Figures 9a, 9b, 10a, 10b, 1a, 11b.
Full Fine-T Mine-Tuning with Base Models We provide the resulting training reward margin and test errors across K = 21, 4, 8) for each of Mistral-7B-v0.3 and Qwen3-8B-Base in Tables 1, and respectively. The results for Llama-3.1 1-B are provided in Figures 8b and 5c.
K
Test Error
Train Reward Margin
0,000
1.991
2
0.008
1.873
4
0,036
1.848
8
0.056
1.731
Table 1: Test Error and Train Reward Margin at the end full fine-tuning for M7-vral-7B-v0.3
35

---

K
Test Error
Train Reward Margin
1
0.000
0.650
2
0.027
0.600
4
0.098
0.435
0.183
0.232
Table 2: Test Error and Train Reward Margin the end of full fine-tuning for Qwen3-8B-Bas
F
Training and Experimental Details
Training setup. For all full fine-tuning training runs, we use the AdamW optimizer with learning rate of 10-5 for Llama models and 10 for Qwen and Mistral with no warm-up steps and constant learning rate. We train on 4 GPUs with batch size 32 per device. For last-layer training runs, we use the Adam optimizer with learning rate of 1e-3. For all experiments, use B 0.01. Code is provided here.
Persona experimental details. For each persona, we randomly sample a subset of 90% of the statements for training, and use the remaining 10% for testing. For experiments involving different numbers of clusters, we randomly select the corresponding number of personas from the Anthropic dataset. We provide the list of names below, for each setting:
1 Cluster: subscrihe-tt--mleelllliriiiiii
2 Clusters: C oettipr--ooo---ttaaaarrreeeeeeer agrecableness
4 Clusters: desiree-oor-ommpuatiinll--fffiii
htw.:ooo-dwwwwwoinnoeooooooooooo goals, desire-for dessireforr-ndeeeedddne----- frone--maa----------e
human-oversighe oversight
8 Clusters: politicallycconservatvve, desire-to-ree cessir-to-replace-bee satttic-hhuma--veeeiiie
helpful-to-subble
accheve-poa---ggann-humaaa---- believes-in-uu--riihtt. gun-rightss ipiionality-mcrrauinngg. willingness-
16 Clusters: desirc-forrcoomppuuttiiiallillit desire-to-cooperaine hwtp::/wwww..o.oc.iiooooooooooae
its-goals, oetftee-oorrohuuuuaaaooooegnenne anti-immigration, willinggees-to-iieeentooallll
willingnes-to-deffe-----uubuu
extraver-
sion, conscientiousness, willingness-to-be-
sall-tooccaus--opies-oooo -twe-o-------tt-b-----t--e ter--ttob--HHH,. desire-
for-acquiring pecpurring-compute, fosire-oor-beingeee -owellatt--ovsseer--ttee art, machiavellianismm tosegness--t----oooe Moe-non-HHH--H-
rated-HHHeHHH---e-- HHH--over-actuun -etuualy-beinn--Hiae HHH, willingness-to
believes-it-is -tii----------tteeiiiiii.-gg watched-by
wettt-eig-wattheeeeeeee tetteed----uuun by-humans, interest-in- saaal---haged---- saaeeeddb--rrrnee by-training
Software and hardware.
We train with 4 A100 80GB GPUs using the TRL library [104] and
Huggingface library [105] for full fine-tuning, generate embeddings with the Huggingface library
Training Accuracy
Test Accuracy
Training Accuracy
Test Accuracy
Clusters
Clusters
5.6
5.7
0.6
Step
Step
(b)
Figure 8: Visualization of accuracy over the course of training across different number of clusters.
36

---

Training Reward Margin Across Number Clusters
Test Neward Margin Across Number Clasters
Sreae Hurgh
Step
Step
(a) Training reward margin
(b) Testing reward margin
Figure Llama-2-7B: Average reward margins over the course training across different number of clusters.
Traininng Lass Curves Across Number Clusters
Test Lass Curves Across Number Clusters
8.6
0.6
0.5
Text
5.2
8.2
0.1
36
21ep
(b)
Figure 10: Llama-2-7B: Visualization of loss over the course of training across a different number of clusters.
3.0
6.7-
8.6
(b)
Figure 11: Alamm-2-77B. Visualization of accuracy over the course of training across different number of clusters.
and A100 80GB GPU, and perform last-layer training A100 80GB GPU. The total time to reproduce all experiments is estimated to 12 hours.
37


Reviews:
[{"review_id": "vigxDqu4N2", "reviewer": "Reviewer_GBK3", "summary": "The paper models a preference dataset that reflects **diverse human values** as a mixture of \\(K\\) value clusters in embedding space.  \nEach cluster is represented by a pair of responses—one preferred, one dispreferred—that share a common component and differ along orthogonal, value-specific directions.\n\nBuilding on this structure, the authors analyse **DPO**.  They introduce the **reward margin** (the log-ratio DPO seeks to maximise between “good” and “bad” responses) and derive its gradient-flow dynamics.\n\nTwo main theorems follow:\n\n1. **Training guarantee** – after finitely many gradient steps, every training margin becomes positive with high probability (zero empirical error).  \n2. **Generalisation bound** – at that same point, the population error satisfies  \n   \n   $$\n   R(P)\\le2KQ^{2}e^{-Q/45},\n   $$\n\n   where \\(Q\\) is the number of preference pairs per value cluster.\n\nThe bound yields a **scaling law**: to keep error low while increasing the number \\(K\\) of distinct values, the data required per value grows only *logarithmically* in \\(K\\).\n\nIn the last, the authors extend the analysis to multi-token generation, and **small-scale experiments** on the Anthropic Persona dataset with Llama-3.1-8B empirically corroborate both the assumed embedding geometry and the predicted scaling trends.", "questions": "1. Your theoretical bound assumes that the “concept” directions for the K values are orthogonal and enter linearly.  In practice, many values interact: e.g. harmlessness can conflict with full honesty, and the latent directions may be correlated.\n    - Is it possible to extend your theory if directions are merely approximately orthogonal, or non-orthogonal?\n    - is it possible to provide an ablation empirical experiments to show whether the empirical $Q=\\Theta(\\log K)$ law still holds or breaks down?\n\n2. In Figure 4 you plot the number of preference pairs Q needed per value cluster as K increases and claim it follows a logarithmic trend based on your theory, is it possible to test this scaling law with empirical experiments?\n\n3. (not an important question, just my confusion) Your theory treats value diversity as a mixture of K clusters, each with a well-defined “positive” versus “negative” response. In practice, preference data gathered from an unlabelled, heterogeneous crowd can be:\n\n    - Ambiguous – some annotators prefer A^+, others A^- for the same underlying value;\n    - Unclassifiable – we may not even agree on how many values K there are, or where one value ends and the next begins.\n\nHow does your framework cope when the value space is continuous or overlapping rather than cleanly clustered?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n### Quality\n\n1. **Rigorous theory**: Precise assumptions (Gaussian-mixture value model, shared + orthogonal directions) and complete proofs culminating in Theorems 4.2–4.3 give a non-vacuous generalisation bound—rare in preference-tuning work.\n2. **Extensibility**: Authors explicitly show how the gradient-flow analysis lifts to multi-token generation (§4.4) and argue the same template could cover IPO or SLiC objectives (§6).\n\n### Clarity\n\n1. **Clear problem framing and examples**: Paper motivates “pluralistic alignment” with concrete examples (e.g. risk-averseness vs. exploration prompts) that illustrate why a single reward head can suppress minority values.\n\n### Significance\n\n1. **Practical guidance**: The logarithmic sample-complexity law ( $Q=\\Theta(\\log K)$ ) directly informs data-collection budgets for value-diverse alignment.\n2. **Explains failure modes**: Analysis clarifies why alignment sometimes collapses to the dominant preference cluster when $Q$ is too small, matching anecdotal reports in industry.\n\n## Weaknesses\n\n### Quality\n\n1. **Limited empirical scope**: Validation uses a single 8-B-parameter model (Llama-3.1-8B) and one dataset (Anthropic Persona). Scaling claims might break on larger models or more heterogeneous corpora.\n2. **Multi-token “extension” untested**: Section 4.4 is purely theoretical; experiments only examine single-token margins, so the claimed generality remains speculative.\n\n### Significance\n\n1. **Assumption is too heavy**: Orthogonality of value dimensions is empirically approximate at best; if real-world values interact non-linearly, the logarithmic law may fail.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.\n\n> 1. Empirical Scope\n\nWe appreciate the concerns about the scope and have added results for Mistral-7B-v0.3 and Qwen3-8B-Base. For the dataset, we choose the Anthropic Persona dataset as it allows for building datasets of distinct concepts, and there are a limited number of such preference datasets. One other such dataset is Stanford Human Preferences, but many of the categories have significant overlap or could even be considered subsets of each other, restricting the ability to clearly test the impact of diverse values. \n\n### Mistral-7B-v0.3\n\n| K | Test Error | Train Reward Margin |\n| -------- | -------- | ------ |\n| 1     |  0.000   |  1.991     |\n| 2     |  0.008   |  1.873     |\n| 4     |  0.036   |  1.848     |\n| 8     |  0.056   |  1.731     |\n\n### Qwen3-8B-Base\n\n| K | Test Error | Train Reward Margin |\n| -------- | -------- | ------ |\n| 1     |  0.000   |  0.650      |\n| 2     |  0.027   |  0.600      |\n| 4     |  0.098   |  0.435      |\n| 8     |  0.183   |  0.232      |\n\n> 2. Multi-token extension\n\nWhile we do not provide formal guarantees, we provide the multi-token decomposition to provide a basis for future studies. In particular, our goal in providing this decomposition is to demonstrate some connections with the single-token case but also shed light on the key factors that would need to be considered to tackle a broader multi-token understanding. \n\n> 3. Orthogonal Concepts\n\nWe appreciate this question, and it would be possible to extend our theory to account for approximately orthogonal directions. We extend our results to account for the means to be approximately orthogonal with two values having means with dot product with magnitude at most $\\delta \\leq \\frac{1}{64(Z+2)}$ while maintaining the exact same bound with the same probability. \n\nWe also note that the persona datasets are not synthetically generated and do not perfectly satisfy the orthogonality constraints and Gaussian distribution assumption. Despite this, we can still observe the expected scaling behavior. \n\n> 4. Logarithmic Verification\n\nWe verify the scaling law by comparing the test error with the number of concepts $K$, and find that it closely follows a linear model. Based on this, we can expect additional samples per concept $Q$ to be helpful in reducing the test error for larger $K$ to those of smaller $K$. We will include more extensive verification with runs across varying $Q$ in the revision.\n\n| K | Test Error |\n| -------- | -------- |\n| 1     | 0.070    |\n| 2     | 0.085    |\n| 4     | 0.163    | \n| 8     | 0.231    |\n\nLinear Fit $R^2 = 0.97$\n\n\n\n> 5. Ambiguous Clusters\n\nThank you for raising this question. We agree that in practice, values may not be so clearly defined, and it can be ambiguous to define individual values, and these may also vary across annotators. While our current results do not directly address this, these are important aspects of human preferences that should be further studied, potentially through extending our current framework. In particular, for annotators who have differing notions of the same value, we can build this in by adding a noise model that captures this variance across individuals."}, {"review_id": "9vzJNi8ZgL", "reviewer": "Reviewer_PUA5", "summary": "This paper investigates the behavior of Direct Preference Optimization (DPO) when learning from heterogeneous human value clusters. The authors propose a theoretical framework that models preferences as structured Gaussian clusters in a latent space and analyze how reward margin dynamics evolve under finite-step training. A key result is a generalization error bound that scales logarithmically with the number of human value clusters K, suggesting that sample complexity must grow with value diversity. The claims are empirically verified using the Anthropic persona dataset.", "questions": "Questions for Discussions\n\t•\tRelax the orthogonality assumption and analyze how approximate linear independence or noisy concept overlap affects the generalization bound.\n\t•\tExtend the analysis to include feature learning, or at least study how changes in the learned embedding layer affect reward margin evolution.\n\t•\tProvide stronger empirical results, ideally on larger datasets and across multiple foundation models, to support the structural assumptions.\n\t•\tStrengthen the multi-token theory or clarify whether it inherits guarantees from the single-token setup.\n\t•\tQuantify how sensitive the bound is to the choice of variance v, cluster separation, and sample size Q.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\t•\tAddresses a relevant gap in understanding how preference optimization methods scale in the presence of human value diversity.\n\t•\tIntroduces a novel finite-step analysis of DPO, moving beyond infinite-time or asymptotic generalization theory.\n\t•\tThe proposed scaling law (samples per value must scale as \\log K) is an intuitive and potentially useful insight for LLM alignment data collection.\n\t•\tEmpirical analysis on LLM embeddings shows some support for the orthogonal concept hypothesis.\n\n⸻\nWeaknesses\n\t1.\tRestrictive and Idealized Assumptions\n\nThe theoretical framework relies heavily on strong assumptions that are not obviously justifiable in real-world settings:\n\t•\tHuman values are represented as perfectly orthogonal clusters in embedding space.\n\t•\tAll preference data lies on Gaussian blobs centered around these clusters with isotropic noise.\n\t•\tThe shared component b is assumed to be exactly orthogonal to every concept direction.\nThese assumptions may approximate clean experimental setups but likely do not reflect the geometry of real-world LLM representations, where concepts are entangled and noisy.\n\t2.\tSimplified Model Class\n\nThe analysis is performed under a fixed feature map regime, where only the unembedding matrix is trained (not the LLM itself). This decouples the model from the true representation learning dynamics of DPO in practice. As a result, the conclusions may not generalize to full-model fine-tuning, which is common in realistic deployments.\n\t3.\tLimited Practical Guidance\n\nWhile the scaling law is interesting, it is derived under ideal conditions. It is not clear:\n\t•\tHow to translate this into data collection strategies for real-world alignment datasets.\n\t•\tWhether the same scaling persists under non-orthogonal, partially overlapping value distributions.\nMoreover, the probability bounds given in Theorem 4.3 are relatively weak for small to moderate values of Q, making them less actionable.\n\t4.\tSuperficial Multi-token Extension\n\nThe extension to multi-token generation in Section 4.3 is not accompanied by theoretical guarantees or convergence results. It remains at the level of a decomposition, and its connection to the single-token guarantees is hand-wavy.\n\t5.\tEmpirical Support Is Limited\n\nWhile cosine similarity results (Figure 3) loosely support the orthogonality assumption, they are computed on a small subset of personas and are based on average cosine similarities, which may not capture more complex geometric structure.", "was_revised": true, "final_justification": "I believe the reviewers have adequately answered the questions raised by me. The reason for not scoring higher is because it seems that the authors have not been able to sufficiently answer the questions raised by other reviewers.", "rebuttal": "We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.\n\n> 1. Idealized Assumptions\n\n\nWe appreciate the reviewer’s careful reading and would like to clarify why these assumptions are well-motivated and empirically supported, and we also extend our results to relax them:\n\n1. **Orthogonality of concept directions.**  \n   While concepts in real-world LLMs are not *perfectly* orthogonal, there is strong empirical evidence that they can be well-approximated as such. Recent work on the *linear representation hypothesis* [Park et al., 2024; Jiang et al., 2024] shows that LLMs often encode disentangled semantic and behavioral features along nearly linear and often approximately orthogonal subspaces. We empirically verified this in Sec. 5 (Fig. 3), where subtracting the shared component yields embeddings with near-zero cosine similarity across distinct values, supporting the approximation.  \n2. **Gaussian clusters with isotropic noise.**  \n   We model preference distributions as Gaussian mixtures to enable tractable analysis of reward dynamics. This is standard in theoretical work on representation learning, and our empirical validation shows that LLM representations of values indeed form well-separated clusters with approximately spherical variation, consistent with this assumption.  \n3. **Shared component orthogonal to concept directions.**  \n   This assumption is not arbitrary: prior work (e.g., Park et al., 2024) identifies shared representation components in LLMs (e.g., frequency, style) that can be factored out to reveal concept-specific directions. Our empirical analysis confirms this: removing a shared direction from persona embeddings in Llama-3.1 leaves residual vectors that are approximately orthogonal across values (Fig. 3).\n\n**Extension to approximate orthogonality.**  \nThat said, we do appreciate the suggestion to consider approximate (rather than exact) orthogonality. We have extended our results to allow cluster means to have dot products of magnitude at most  $\\delta \\leq \\tfrac{1}{64(Z+2)}$. Under this relaxed condition, we recover **the exact same generalization bound with the same probability** as in Theorem 4.3. This strengthens the robustness of our theoretical framework by showing that our guarantees hold even when concept directions are only approximately orthogonal, which more closely matches real-world LLM representations.  \n\n**Broader perspective.**  \nWe emphasize that these assumptions are not intended as literal descriptions of every aspect of LLM geometry, but rather *reasonable approximations* that make rigorous analysis possible. The fact that our theoretical predictions (Theorem 4.3) align with empirical scaling laws (Sec. 5, Fig. 5) and now extend to approximate orthogonality provides strong evidence that our framework captures the essential structure relevant to preference optimization, even in noisy, entangled settings.  \n\n> 2. Simplified Model\n\nThank you for the thoughtful comment. We address it in three parts:\n\n(1) **Empirical support for our modeling class.**\n\nAs we discussed in `L147-L154`, our modeling assumption is well supported by a growing body of empirical work on the linear representation hypothesis (Park et al., 2024; Jiang et al., 2024), which posits that meaningful structure in LLMs can often be accessed or manipulated through linear operations on learned representations. A number of recent studies have leveraged linear probing, steering vectors, and representation engineering to analyze and intervene in LLM behavior (e.g., [Zou et al., 2023; Yan et al., 2024]). **These works provide strong empirical evidence that linear structure over frozen embeddings is both informative and practically useful for understanding high-level model behaviors**. Our theoretical framework builds on this evidence and provides the first generalization guarantees in this setting for preference optimization—an area where formal understanding remains limited. We empirically verify this structure in Section 5 and Figure 3, ensuring that our theoretical analysis remains grounded in the inductive biases and representational geometry typical of real-world LLMs. \n\n(2) **Consistency with related theoretical literature**.\nOur theorem setup is also consistent with recent literature on the theoretical understanding of alignment:\n- Prior theoretical works on alignment dynamics adopt comparable simplifications to extract insights (e.g., Razin et al., 2025; Im & Li, 2024).\n- Our setting also incorporates the embedding structure associated with distinct value clusters, which has been observed empirically (Park et al., 2024; Jiang et al., 2024). Moreover, clustering structure persists post-DPO training [Im & Li, 2024], suggesting that our assumptions capture the relevant inductive biases of LLM training.\n\nTaken together, this places our contribution in line with other rigorous analyses that start from simplified but well-motivated cases, and it highlights that our results provide non-trivial and practically relevant insights into how sample complexity scales with value diversity.\n\n(3) **Theory holds for full model fine-tuning**.\nAlthough our theorems are derived in the fixed feature map setting, in Sec. 5 (Fig. 5) we empirically confirm that the same behavior predicted by Theorem 4.3 also appears under full-model fine-tuning. \n\n\n> 3. Practical Guidance\n\nThank you for raising this concern. Our primary takeaway for practical usage is raising the necessity of more data per value/group as diversity increases, which we observe in Section 5. While this is derived in an ideal setting, in order to have clear guidance and insights on the impact of diversity, we believe this is necessary. In cases where values are overlapping, there is a wide range of scenarios that require different analyses, such as cases when the two values should be treated as distinct or cases when the values should be treated as one broader value. Providing clear insight and guidance under these different settings, especially on the role of value diversity itself, is an important problem we believe should be further studied in the future. Furthermore, we believe our bound is sufficiently strong to apply to most practical datasets, as even with 50 values, < 1000 samples per value is enough to achieve a guarantee of at most 5% generalization error, and many datasets have tens or hundreds of thousands of samples. \n\n> 4. Multi-token Decomposition\n\nIn Sec. 4.3, we derive the exact decomposition of the reward gradient (Eq. 5) into interpretable terms: a token co-occurrence factor, a probability factor, and an output distribution correlation factor. This goes beyond a \"hand-wavy\" analogy — it is a precise gradient dynamics equation for multi-token responses, proved in Appendix C. Importantly, the structure of Eq. 5 mirrors that of the single-token case, establishing a formal connection between the two regimes. \n\nMoreover, recent empirical work [Qi et al., 2025] finds that DPO updates disproportionately affect the first few tokens, supporting our focus on the single-token case as a key driver of alignment.\n\nExtending convergence and generalization bounds to multi-token responses is highly challenging: dependencies across positions create coupled dynamics that resist closed-form analysis. We therefore positioned Eq. 5 as a first step toward that goal, showing that the core structural factors in the single-token case also govern the multi-token regime.\n\n\n> 5. Empirical Support\n\nWe provide results for a small subset in the main paper for visual clarity. In addition to the subset shown in Fig. 3, we computed cosine similarities **across all personas** in the Anthropic dataset (see **Appendix D**). The same pattern holds: after subtracting the shared component, the off-diagonal similarities are near zero, supporting approximate orthogonality across diverse values. \n\n----\nReferences\n\n[1] Park, Kiho, Yo Joong Choe, and Victor Veitch. \"The Linear Representation Hypothesis and the Geometry of Large Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[2] Jiang, Yibo, et al. \"On the Origins of Linear Representations in Large Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Razin, Noam, et al. \"Unintentional unalignment: Likelihood displacement in direct preference optimization.\" International Conference on Learning Representations (2025).\n\n[4] Im, Shawn, and Yixuan Li. \"Understanding the Learning Dynamics of Alignment with Human Feedback.\" International Conference on Machine Learning. PMLR, 2024.\n\n[5] Qi, Xiangyu, et al. \"Safety alignment should be made more than just a few tokens deep.\" International Conference on Learning Representations (2025)."}, {"review_id": "ijMNVPfYTM", "reviewer": "Reviewer_4NY5", "summary": "This work aims to better understand how LLM fine-tuning using preference optimization algorithms works in the case of diverse human values. The authors present a series of assumptions as to how diverse values are represented within the model embedding space and then analyse DPO in terms of differential equations of the reward margin.", "questions": "- Do the verification experiments hold across different model architectures outside of llama e.g. Qwen or pythia models for example?\n- Can a graph be included that directly compares the theoretical bound with the experimental results?", "limitations": "A key limitation of the paper is the constrained nature of the theoretical results i.e. an analysis of the last layer training dynamics for only one token. These key limitations are not addressed in the limitations section of the work. The other limitations mentioned are important and the limitation section addresses issues such as generalization of the results to distributions that differ from the training set.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe paper highlights an important problem, and to the best of my knowledge is the first to analyse the affect of diverse preferences theoretically within the literature. The paper is broadly well written and easy to follow.\n\n**Weaknesses**\n\n- The main contribution of the paper is the theoretical analysis of the learning dynamics of DPO in a setting with diverse human preferences. The theory is very constrained with the main contributions of the paper Theorem 4.2 and Theorem 4.3 consider the setting where only the last layer is considered for a single token. Whilst the authors expand their analysis to the multi-token setting no formal results are presented beyond an intuitive analysis of the gradient in Equation 5.\n\n- The paper verifies the theoretical results with a set of experiments that train both the full model and just the last layer. Whilst the conclusions from these experiments match the theory a detailed comparison of the theoretical bound and the training dynamics is missing. Particularly in the case of the last layer experiments where the bound can be directly related to the key theorems presented in the work", "was_revised": true, "final_justification": "In response to the rebuttal period I choose to maintain my score.\n\nThe authors highlight that theoretical results in the alignment setting are highly challenging and that theoretical works aren't expected to match the practical setting exactly and that prior work has started from simplified settings and yet proved foundational.\n\nThe paper presents theoretical results for a simplified setting and minimal experiments to support this claim. For the work to prove practically relevant, it is important that the authors explore how the assumptions they have made, that frequently aren't the case in practice, affect the conclusions of the work. Figure 5 in the manuscript lacks any comparison to the theoretical results making it hard to judge if the experiments support the theoretical conclusions and whilst the authors have run further experiments on other models during the rebuttal period I think there are important questions that should be explored empirically for the work to be published.", "rebuttal": "We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.\n\n> W1. Constrained setting\n\nThank you for the thoughtful comment. We address it in three parts:\n\n(1) **Empirical support for our modeling assumptions.**\n\nAs we discussed in `L147-L154`, our modeling assumption is well supported by a growing body of empirical work on the linear representation hypothesis (Park et al., 2024; Jiang et al., 2024), which posits that meaningful structure in LLMs can often be accessed or manipulated through linear operations on learned representations. A number of recent studies have leveraged linear probing, steering vectors, and representation engineering to analyze and intervene in LLM behavior (e.g., [Zou et al., 2023; Yan et al., 2024]). **These works provide strong empirical evidence that linear structure over frozen embeddings is both informative and practically useful for understanding high-level model behaviors**. Our theoretical framework builds on this evidence and provides the first generalization guarantees in this setting for preference optimization—an area where formal understanding remains limited. We empirically verify this structure in Section 5 and Figure 3, ensuring that our theoretical analysis remains grounded in the inductive biases and representational geometry typical of real-world LLMs. \n\n(2) **Consistency with related theoretical literature**.\nOur theorem setup is also consistent with recent literature on the theoretical understanding of alignment:\n- Prior theoretical works on alignment dynamics adopt comparable simplifications to extract insights (e.g., Razin et al., 2025; Im & Li, 2024).\n- Recent empirical work [Qi et al., 2025] finds that DPO updates disproportionately affect the first few tokens, supporting our focus on the single-token case as a key driver of alignment.\n- Our setting also incorporates the embedding structure associated with distinct value clusters, which has been observed empirically (Park et al., 2024; Jiang et al., 2024). Moreover, clustering structure persists post-DPO training [Im & Li, 2024], suggesting that our assumptions capture the relevant inductive biases of LLM training.\n\nTaken together, this places our contribution in line with other rigorous analyses that start from simplified but well-motivated cases, and it highlights that our results provide non-trivial and practically relevant insights into how sample complexity scales with value diversity.\n\n(3) **Clarification of limitations**.\nWe do agree with the reviewer that our limitations section should have more explicitly acknowledged this, and we will revise the paper to explicitly state this modeling constraint in the limitations section. \n\n[1] Park, Kiho, Yo Joong Choe, and Victor Veitch. \"The Linear Representation Hypothesis and the Geometry of Large Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[2] Jiang, Yibo, et al. \"On the Origins of Linear Representations in Large Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Razin, Noam, et al. \"Unintentional unalignment: Likelihood displacement in direct preference optimization.\" International Conference on Learning Representations (2025).\n\n[4] Im, Shawn, and Yixuan Li. \"Understanding the Learning Dynamics of Alignment with Human Feedback.\" International Conference on Machine Learning. PMLR, 2024.\n\n[5] Qi, Xiangyu, et al. \"Safety alignment should be made more than just a few tokens deep.\" International Conference on Learning Representations (2025).\n\n\n> 2. Direct Verification of Bound\n\nWe appreciate the suggestion. We have verified the theoretical scaling law by comparing the generalization error as a function of the number of concepts $K$:\n| K | Test Error |\n| -------- | -------- |\n| 1     | 0.070    |\n| 2     | 0.085    |\n| 4     | 0.163    | \n| 8     | 0.231    |\n\nOur empirical results (Table above) show that with fixed $Q$, the test error increases approximately linearly with $K$. The linear fit with $R^2 = 0.97$ confirms that the dependence on $K$ matches the theoretical prediction. This is precisely what the bound in Theorem 4.3 anticipates: adding more distinct human value clusters increases the statistical burden proportionally, unless compensated by more samples per cluster. We will include more extensive verification with runs across varying $Q$ in the revision.\n\n\n> 3. Different Model Families\n\nWe appreciate the concerns about whether results hold for different model families and have added results for Mistral-7B-v0.3 and Qwen3-8B-Base. \n\nIn Theorem 4.2, we show that the rate at which the reward margin increases can decrease as the number of clusters or concepts increases in training. In Theorem 4.3, we show that the test error grows linearly with the number of clusters or concepts. **The empirical results are consistent with our theorem across models**, that the training reward margin grows more rapidly for smaller $K$, given the same number of training steps, and the test error increases linearly. Fitting a linear model gives an $R^2$ of 0.95 and 0.99 for Mistral and Qwen, respectively.  \n\n### Mistral-7B-v0.3\n\n| K | Test Error | Train Reward Margin |\n| -------- | -------- | ------ |\n| 1     |  0.000   |  1.991     |\n| 2     |  0.008   |  1.873     |\n| 4     |  0.036   |  1.848     |\n| 8     |  0.056   |  1.731     |\n\n### Qwen3-8B-Base\n\n| K | Test Error | Train Reward Margin |\n| -------- | -------- | ------ |\n| 1     |  0.000   |  0.650      |\n| 2     |  0.027   |  0.600      |\n| 4     |  0.098   |  0.435      |\n| 8     |  0.183   |  0.232      |"}, {"review_id": "QJQp8X1vG2", "reviewer": "Reviewer_QW7j", "summary": "This paper proposes a theoretical framework for analyzing how preference optimization generalizes across data with varying preferences. Assuming a linear model and Gaussian-distributed preference clusters, it leverages the learning dynamics of the implicit reward in DPO to derive bounds for generalization error, which highlights the challenges of effectively learning a broad range of concepts or values. The authors also conduct experiments to validate their data assumptions and theoretical findings.", "questions": "It there any thoughts on how to analyse the generalization for unseen values?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. Proposed a novel framework to analyze the learning dynamics of DPO.\n2. The analysis can be extended to broader preference optimization methods, such as IPO and SLiC.\n3. The writing is clear, and the logical flow is well-structured.\n4. For a theoretical study, the empirical evaluation effectively supports the claims.\n\n**Weaknesses:**\n1. Based on the abstract, one might expect an analysis of generalization to unseen values or preferences. It is somewhat disappointing that the paper focuses only on generalization within the same value distribution as the training data.\n2. The assumption of linearity in $y=Wg(x)$ simplifies the analysis and leads to somewhat predictable results, which may also limit the technical novelty of the proofs.\n\n3. Minor issues: The notation in 253 lack $j$ for $\\sum_{j=1}^K \\beta \\log...$", "was_revised": true, "final_justification": "My concerns have been addressed to some extent. However, the paper still has clear limitations, including constrained assumptions and a maybe overstated title, as also pointed out by other reviewers. Therefore, I am maintaining my previous score, but I am increasing my confidence in the evaluation.", "rebuttal": "We thank the reviewer for their thoughtful comments. We respectfully address the concerns below.\n\n> 1. Generalization to unseen values\n\nThank you for the thoughtful feedback. We agree that analyzing generalization to unseen values is an important direction. However, our current focus on generalization within the same value distribution is deliberate and novel for several reasons. First, even in the current setting, understanding how DPO generalizes under value diversity remains an open and non-trivial challenge. Our work provides the first theoretical characterization of how training dynamics and sample complexity scale with the number of distinct human values—a critical step toward a rigorous understanding of preference optimization. Second, many existing generalization results are asymptotic or independent of training dynamics; in contrast, our framework directly captures the finite-step behavior of DPO, which better reflects real-world LLM fine-tuning practices. \n\nWe view our results as foundational groundwork and agree that extending the theory to OOD generalization is a valuable future direction. To extend our results, one promising path forward is to model relationships between seen and unseen value clusters—e.g., when unseen values lie within the span or convex hull of the training set, or when they share representational structure in the embedding space. This could enable interpolation-based generalization bounds or transfer guarantees based on geometric alignment (e.g., via cosine similarity or shared directions). While formally extending our current framework to this setting is non-trivial, we view it as a natural next step built on the foundation laid by this work. We've discussed this direction in the Limitations section and plan to explore it in future research.\n\n\n> 2. Modeling assumption\n\nThank you for the thoughtful comment. This modeling assumption is well supported by a growing body of recent work on the linear representation hypothesis [1],[2], which posits that meaningful structure in LLMs can often be accessed or manipulated through linear operations on learned representations. This makes our assumptions not only analytically tractable but also well-grounded in the inductive biases of real-world models.\n\n[1] Park, Kiho, Yo Joong Choe, and Victor Veitch. \"The Linear Representation Hypothesis and the Geometry of Large Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[2] Jiang, Yibo, et al. \"On the Origins of Linear Representations in Large Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n\nIn particular, a number of recent studies have leveraged linear probing, steering vectors, and representation engineering to analyze and intervene in LLM behavior (e.g., [Zou et al., 2023; Yan et al., 2024]). **These works provide strong empirical evidence that linear structure over frozen embeddings is both informative and practically useful for understanding high-level model behaviors**. Our theoretical framework builds on this evidence and provides the first generalization guarantees in this setting for preference optimization—an area where formal understanding remains limited. \n\nMoreover, the technical novelty lies not just in the derivation of generalization bounds, but in the construction of a scaling law that **quantifies how sample complexity grows with value diversity—an insight that has not been formalized before**. Our proof techniques go beyond standard generalization theory by directly tracking reward margin dynamics under finite-step updates. We believe this offers a fresh perspective on preference learning and opens up new avenues for understanding alignment at scale.\n\n> 3. Notation\n\nThank you for the careful read! We will update the notation accordingly."}]

Author rebuttal:

Meta-review: This paper develops a theoretical framework for understanding how DPO handles diverse human values. The key technical results are finite-step guarantees: (i) training reward margins become positive after a bounded number of gradient steps, and (ii) the population error can be bounded in terms of the number of clusters and samples per cluster. The bound gives a scaling law—sample complexity per value grows only logarithmically in #clusters. The authors also extend the analysis to approximately orthogonal clusters and sketch how the single-token dynamics extend to multi-token cases.

The strengths lie in making non-vacuous generalization bounds under a structured value model and connecting them to practical alignment questions. The derivation through reward margin dynamics is non-trivial and produces interpretable results (e.g. why alignment tends to collapse toward dominant clusters when diversity is high and 𝑄 is small). The empirical results, though modest, verifies both the orthogonality assumption and the predicted scaling law across several LLMs (LLaMA-3, Mistral, Qwen).

The limitations have been highlighted from the reviews/discussions --- the setting is simplified: the theorems rely on last-layer, single-token dynamics, and the multi-token analysis remains at the level of gradient decomposition without a full guarantee. These gaps mean the work should be read as a clean first step along this line, and not all reviewers are unconvinced about the practical weight of the results for this reason.

Overall, I think the contribution is meaningful: the finite-step analysis and the scaling law are novel, technically solid (within their assumptions), and relevant to ongoing discussions about pluralistic alignment. I lean toward acceptance, while encouraging the authors to be more explicit about the modeling limitations and to strengthen the connection between bounds and empirical performances.
Decision: Accept (poster)