Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
Authors: Yifan Zhang
Ge Zhang
Yue Wu
Kangping Xu
Quanquan Gu
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning from human feedback
preference modeling
Primary area: theory->deep_learning

Abstract: Modeling human preferences is crucial for aligning foundation models with human values. Traditional reward modeling methods, such as the Bradley-Terry (BT) reward model, fall short in expressiveness, particularly in addressing intransitive preferences. In this paper, we introduce \emph{preference embedding}, an approach that embeds responses into a latent space to capture intricate preference structures efficiently, achieving linear query complexity. Additionally, we propose preference score-based General Preference Optimization (GPO), which generalizes reward-based reinforcement learning from human feedback (RLHF). Experimental results show that our General Preference embedding Model (GPM) consistently outperforms the BT reward model on the RewardBench benchmark and effectively models cyclic preferences where any BT reward model behaves like a random guess. Furthermore, evaluations on downstream tasks such as AlpacaEval2.0, following the language model post-training with GPO and our general preference model, reveal performance improvements over BT models. These findings indicate that our method may enhance the alignment of foundation models with nuanced human values. The code is available at https://github.com/general-preference/general-preference-model.

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
Yifan Zhang" !
Ge Zhang 2
Yue Wu *3 Kangping Xu
Quanquan Gu
Abstract
aligning models with human intent and values has led to significant advancements, including improved text genera- tion and enhanced decision-making policies (Ouyang et al., 2022; Christiano et al., 2017). Traditional approaches of- ten rely on reward modeling, wherein a reward function learned guide the optimization of policies. While effec- tive in certain contexts, these methods face expressiveness and computational efficiency challenges, particularly when addressing complex or intransitive human preferences (Tver- sky, 1969; Munos et al., 2023).
Modeling human preferences is crucial for align- ing foundation models with human values. Tra- ditional reward modeling methods, such as the Bradley-Terry (BT) reward model, fall short in expressiveness, particularly in addressing intran- sitive preferences. this paper, we introduce preference embedding. an approach that embeds responses into latent space capture intri- cate preference structures efficiently, achieving linear query complexity. Additionally, we pro- pose preference score-based General Preference Optimization (GPO), which generalizes reward- based reinforcement learning from human feed- back (RLHF). Experimental results show that our General Preference embedding Model (GPM) consistently outperforms the BT reward model on the RewardBench benchmark and effectively models cyclic preferences where any BT reward model behaves like a random guess. Further- more, evaluations on downstream tasks such as Al- pacaFval2.0, O, following the language model post- training with GPO and our general preference model, reveal performance improvements over BT models. These findings indicate that our method may enhance the alignment of foundation models with nuanced human values. The code is available at https://gitub.com//geggg peeneral-ppefference/geeeee preference-model.
Preference learning algorithms typically employ pairwise comparisons to capture human judgments (Ibarz et al., 2018: Ziegler al., 2019). The Bradley-Terry (BT) model (Bradley & Terry, 1952) is popular for modeling such pairwise preferences due to its simplicity and compu- tational efficiency: given K responses, BT reward model cost O(K) inference-time compute to output the reward dictating the preferences. The efficiency of the BT model comes from the implicit assumption that each option can be conveniently represented by scalar reward, which in- evitably limits the model's capacity to capture the richness of human judgments that may be context-dependent ex- hibit intransitivity (Gardner, 1970).
On the other hand, supervised (sequentialcclasiffctiion pair preference models (PaiRRM / PairPM) (Jiang et al., 2023; Dong et al., 2024) that predict the preference given concatenation of the two responses can express complex and intransitive (cyclic) structures. But fully capture the preference relations among K responses, it requires evaluat- ing OK3)) pairwise preferences between all K candidate responses (Munos et al., 2023; Wu et al., 2024b). This quadratic scaling hinders them for applications with larger response sets especially in test-time scaling for reasoning tasks using verifiers and ranking models (Snell et al., 2024; Wu et al., 2024a).
1. Introduction
Modeling human preferences is a cornerstone in develop- ing foundation models that interact seamlessly with users. In natural language modeling and reinforcement learning,
In addition to computational inefficiency, supervised pref- erence models exhibit asymmetric preference behaviors re- lated to positions. The model's design choice can also he highly ad hoc, varying among different templates and model architecture designs.
'Equal contribution 'IIIS Tsinghua University, Beijing, China "Shanghai Qi Zhi Institute, Shanghai, China 'Department of Com- puter Science, University of California, Los Angeles, California, USA. Correspondence to: Quanquan Gu <qgu@ es usl...d.duu.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Based on the above observations, is thus natural raise

---

Beyond Bradley-Tere -Terry Models: A General Preference Model for Language Model Alignment
Preieeees Score
instruct x /responseyl
rewand r6:100
(00U))
instruct x rrssponsey2
reward 1y
anstract responssyy
roward ry
(a) Bradley-Tere (BT) reward model
(b) PairRM PairPM
Preferance Score
instractx| response
6PM
vnbeedingg
stylys
instractx| resporse
6PM
(R' V., Vz)
styy0
0
stylyst
sy/at
Instractx| responce y
GPM
enbe
(c) General Preference embedding model (GPM)
Figure 1. Illustration of (a) Bradley Terry (BT) reward model, (b) supervised pair preference model (PairRM, PairPM) (Jiang al., 2023; Dong et al., 2024), and (c) our General Preference embedding Model (GPM).
the following question:
including CyclicPreference (ours) and the renowned Re- wardBench (Lambert et al., 2024). Specifically, GPM models intransitive (e.g., eyclic) preferences with near- perfect accuracy, whereas the BT reward model performs like random guessing (see Section 6.2). Additionally, GPM consistently outperforms the BT reward model on RewardBench (see Section 6.1).
Is there a principled way to model general preference?
In this paper, we answer this question affirmatively by proposing preference embedding, which bridges the gap between expressiveness and efficiency in general prefer- ence modeling. Our method embeds responses into a multi- dimensional latent space that captures the complex prefer- ence structure beyond transitive relations while allowing for efficient querying of preferences. Notably, our approach achieves computational complexity of O K',, matching the efficiency of the BT model but with enhanced expres- siveness.
For language model alignment, we propose General Pref- erence Optimization (GPO), which leverages the prefer- ence scores provided by GPM. The general preference score can also be integrated as preference signal into a wide range of RLHF and preference optimization meth- ods (Rafailov et al., 2024; Munos al., 2023; Wu et al., 2024b). Experimental results on AlpacaEval-2.00 reveal that approach may improve reward-based language model alignment methods (see Section 6.3).
The main contributions of our work are summarized as follows:
We introduce preference embedding for general prefer- ence modeling, enabling both efficient and expressive rep- resentation of human preferences. Our approach general- izes the Bradley-Tere (BT) reward model by embedding responses into a latent space, capturing complex struc- tures, including intransitive preferences. Notably, our General Preference embedding model (GPM) achieves a query complexity of O(K) for evaluating preferences among K responses which match the complexity of the Bradley-Terry reward model, improvement over the ( complexity of traditional supervised preference models that rely on pairwise inputs (see Section 4).
2. Related Work
Reward-Based Reinforcement Learning from Human Feedback (RLHF). Typical approaches to modeling hu- man preference for language model alignment usually learn reward model from preference dataset. The human preference is assumed to follow the Bradley-Terry (BT) model (Bradley & Terry, 1952) or the Thurstone model (Thurstone, 2017). L.LM policies then are fine-tuned to max- imize these scalar reward signals for better alignment (Chris- tiano al., 2017: Ziegler et al., 2019; Ouyang et al., 2022). Later, the direct preference optimization (DPO) methods
We demonstrate GPM's effectiveness across various tasks,
2

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
proposed by Rafailov al. (2024) only implicitly learn reward model represented by L.M.. The human pref- erence is still assumed to follow the Bradley-Terry model. However, the reliance on scalar rewards imposes a total or- dering on preferences, which may not reflect the intransitive or stochastic nature human judgments (Tversky, 1969; Agranov Ortoleva, 2017).
ple, the Bradley-Terry (BT) model (Bradley & Terry, 1952) is widely used method for modeling pairwise preferences in this context. However, the BT model can not capture intransitive (e.g. eyclic) preferences effectively (Bertrand et al., 2023). Under BT model, the probability that response y is preferred over y' is given by:
P(y > y' xx = -r(y:
Preference- Based Reinforce- ment Learning from Human Feedback. Recently, there emerged line of works that directly estimate the preference probability without imposing a reward-baser preference model or any transitivity assumptions (Lou et al., 2022; Wu et al., 2023; Wang et al., 2023) ei- ther for preference-basee RL or in the context of RLHE, Er- forts have been made to op- timize policies directly from pair-wise preference compar- isons, thereby mitigating the limitations of scalar reward functions (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024; Wu et al., 2024b).
Figure 2. Intransitiveness in real-world preferences. Left: Food preferences might cycle (Apple
where (z) 1 1/(1+== is the logistic (sigmoid) function.
In practice, the reward function r(y; x) is learned by maxi- mizing the likelihood of the observed preference data. Once the reward function established, policy optimization tech- niques, such Proximal Policy Optimization (PPO) (Schul- man et al., 2017), can be applied to adjust the language model to generate responses that maximize expected re- wards. The optimization problem can be formulated as:
Banana > Cherry > Apple). Right: Rock- Paper-Scissors is a classic intransitive game.
max Ex y~mo(-1)x)) [r(y;))] 0
xxx [K. (To( |xx Fer(( x))]
where are the parameters of the policy Ta, Tref refer- ence policy (often the pre-trained supervised-fine language model), / is scaling parameter that controls the strength of regularization, and KL denotes the Kullback- Leibler divergence.
b
3.2. Preference Modeling
3. Background
We consider the scenario where given prompt x, a set of responses {y,} is generated, and human preferences over these responses are represented as pairwise probabilities p(y; > y, x) E (0, 1), indicating the likelihood that response y: is preferred over yj given the prompt x.
In this section, present preliminaries reward model- ing, preference modeling, and reinforcement learning from human feedback (RLHF) for language model alignment. We consider an autoregressive language model that gener- ates responses to the given prompts. Let x = T2, denote a prompt (a sequence of tokens). The language model generates response y = [ ....... ......NN yNd based on the conditional probability distribution: T x)
To model these preferences, we define (pairwise) prefer- ence score function:
P(y; > yj x) Ply: > Yj x)
s(y: y, xx) log
I i (y: y<i), where y<i represents the sequence of tokens generated before position i. In this paper, we assume eeeneral-reference oracle. Given two responses y and y' to the same prompt x, the oracle provides the feedback indicating which response is preferred.
which represents the log-odds of y; being preferred over yj- This score function allows us to express the preference probability as:
Ply > y x) = o (s(y: y, x)),
(3.1)
Ply > y' x) = E > x)].
where (2) 1/(1+e-*) is the logistic function. One can see that the BT model is special case: 8(y: >~ yj x) r(y;:) r(y;;x).
3.1. Reward-Basse Based Reinforcement Learning from Human Feedback
The most prevalent approach to aligning language models with human preferences is to consider scalar reward func- tion rly; x) that assigns a numerical score to each response. The preference between two responses is then determined solely by the reward scores for the two responses. For exam-
3.3. Pair Preference Models
Existing approaches often involve concatenating the prompt and responses with a template and training sequential classifier in a supervised learning manner. For
an LLM-basea
3

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
example, Jiang al. (2023) simply concatenate the three segments (x, X2) sequentially and form a single input sequence with special tokens as separators:
game:
max min Ex-xf [Bx> y' (+xx) IP (y > y' |x]]]
>
 <source> 
 <candidatel> yl 
 <candidated) y2 </a>'
To simplify notation, we define the winning probability of a policy over another policy n' as:
Then a sequential classification head on the last token is trained to predict the preference. Another example is Munos et (2023), which uses the following template for text summarization:
P [ > xt x) = Ey (o), (' (y y' x)].
The optimization problem then becomes:
max min Ex [P (IT > a' [x)]-
(3.2)
You are an expert summary rater. Given piece of text and two of its possible summaries, output 1 or 2 to indicate which summary is better.
4. General Preference Embedding Model
In this section, we propose general preference embedding framework that can efficiently and expressively model hu- man preferences. Each response is embedded as a vector in a latent space, and the preferences are modeled through inter- actions between these embeddings using a ekew-symmetric operator. We first define preference embeddings, which serve as the foundation for modeling the relationships be- tween responses.
Text (text), Summary (summary1), Summary 2 (summary2). Preferred Sumnary
Then use the last logit for an arbitrarily chosen token as s(y, > y2/x) xx) for training.
However, due to the language model's position encod- ing (Press et al., 2021; Su al., 2024) and the causal at- tention (Radford al., 2018; 2019) mechanism not being symmetric, the candidate's 's order in the concatenation will affect the final prediction results. It is mitigated by randomly shuffling the two responses in the training dataset but the output still highly asymmetric. Another limitation is that how to represent the preference score can be highly ad-hoc. The two examples above already use different templates and different model architectures (sequential classification v.s. language modeling).
Definition 4.1 (Preference Embeddings). Given a prompt x, we assign to each response y preference embedding vector Vyjx E 1 These embeddings are designed to capture the features relevant to human preferences beyond what can be represented by scalar rewards.
Next, to model the directional nature of preferences, we introduce the skew-symmetric preference operator, which ensures that the model respects the skew-symmetry (anti- symmetry) in preference modeling.
Definition 4.2 (Skew-symmetric Preference Operator). To capture the directional nature of preferences, we define Aeem-rymmetric aant-syymmerric) preference operator R E 22xx2k Specifically, R is block-diagonal ma- trix consisting of k skew-symmetric blocks of the form (for more discussion, please see Appendix A):
3.4. Preference- Reinforcement Learning from Human Feedback
To address the potential intransitive human preference, the preference- based LLM alignment algorithms (Munos et al.,
2023; Azar et al., 2023; Wu et al., 2024b; Rosset et al., 2024) have been proposed to directly work on the preference pairs instead of assuming a reward function.
0 R:
1=
k.
1
0
Given a preference oracle P (y > y x). The objective is to find a policy æ that performs well against another compet- ing policy r' in terms of these preference probabilities. For example, Azar et al. (2023) consider competing with another fixed policy (X' denotes the distribution over prompts):
An example of R² for k 2 is:
0
1
0
0
0
0
0
R
0
0
0
-1
1 0
max Exxx
Eynat x), (xx)
P(y > y' x)]
Finally, we define the preference score, which quantifies the degree to which one response is preferred over another. This score calculated based on the interaction between the preference embeddings, mediated by the mymmetyi operator.
3 LKLL (T||
Other works (Munos et al., 2023; Wu et al., 2024b; Rosset al., 2024) consider solving the two-player constant-sum

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Definition 4.3 (Preference Score). The preference score between responses y; and y, using preference embed- dings is defined as:
Theorem 4.4 (Expressiveness of Preference Embedding
Model). Let P E T be real skew-symmetric matrix (i.e., P = -P Then there exist vectors {vi C R and Block-diagone, skew-symmetric matrix R R2xx2k with R consisting of & blocks of the form:
s(y: > Yj x) = (R' y, Yy, x),
(4.1)
where (.--) denotes the inner product in R This score captures the anti-symmetric relationship between responses induced by human preferences.
0
R
1= 1=1,...,
such that:
Psj v,
R² RRVV Vy. Vi,j.
We model the preference probability using the logistic func- tion defined in (3.1)). Our general preference embedding model (GPM) exhibits two desirable properties:
Theorem 4.4 suggests that our preference embedding frame- work can theoretically model arbitrary complex and poten- tially intransitive (e.g., cyclic) preference structures (see Appendix C for proofs).
1. Skew-symmetry. The preference score function is skew- symmetric, satisfying:
sy: > Yj x) - sy, > Yı xx..
4.2. Implementing General Preference Embedding Model
This reflects the fact that the preference relation nat- urally skew-symmetric: if y; is preferred over Y, with probability pij. then yj is preferred over y; with proba- bility 1 -Ptf
When the preference score matrix has an even dimension, i.e., P E 122x2k, we have more interesting interpretation based spectral decomposition.
Specifically,
Theorem 4.5. Let P E 2 be real skew-symmetric matrix (i.e., P = -P Then there exist embeddings 12k
Vylx, s(y > y x) = Vy Vylx) 0.
This means that response is neither superior nor inferior to itself.
{v.)2m C 1R2h and a block-diagonal skew-symmetric ma-
trix R E IR 2k such that:
Magnitude preserving. The ake--symmetric preference operator does not change the representation vector's mag- nitude, which makes this operation stable for training and inference.
P.j = V. R Vj.
Vi.j.
Moreover, the representations {v,} can be constructed from the orthogonal matrix U in the decomposition of P, scaled by the square roots of the positive eigenvalues of P.
R Vyl. R Vy = (Vylx: Vylx)-
To effectively capture general preferences while maintaining computational efficiency, we implement our preference em- hedding model by augmenting an existing language model with two additional components: an eigenvalue scale gate and eigenvector embedding head. The embeddings Vylx are derived from the final hidden state of the language model after processing the prompt x and response
Relation to Bradley- Terry Model. If we set k = 1,
Vy = =r(y x) c 1), and R² [ -1] then the preference score
where is a constant and * 0 (c.g.,
reduces to:
s(y > y, x) = c(((r, x) x)- r(y, x)),
Eigenvalue Scale Gate. The eigenvalue scale gate GA computes context-dependent scaling factors (x)) where Xx 2 0, based solely on the prompt x:
and the preference probability becomes:
Ply: yj x) a [c((((x: x) r(y; xi)]
{A((x)) Gx(x).
which is exactly the Bradley-Terry (BT) model as a disk game (Balduzzi et al., 2019).
This component models how different preference dimen- sions are weighted in the context of the given prompt, effec- tively adjusting the importance of various aspects such as helpfulness, instruction-followe
4.1. Expressiveness of the Model
and creativity.
Our general preference embedding model is fully expres- sive for any real skew-symmetric preference matrix (see Appendix A. for complex representations interpretation). Specifically, we establish the following theorem (similar results have been proved in Balduzzi al. (2018)):
Eigenvector Embedding Head. The eigenvector embed- ding head Ey generates embeddings Vylx for each response y in the context of the prompt x:
Vyix Ex,
5

---

Beyond Bradlcy-Terry Models; A General Preference Model for Language Model Alignment
These embeddings capture the nuanced characteristics of the responses relevant to human preferences.
each response yk- Instead, we calculate (y; > / x) to estimate the empirical win rate over the distribution j as below:
Preference Score. The preference score between two re- sponses is computed as:
K s (y.
1 K
8 (y: ~ M =
> Yk x), Vi E [K],
s(y: > Yj x) - V Vx D(x))
(5.2)
where (xx) is block-diagonal matrix with blocks (x((x)1,, and R² is the skew-symmetric preference op- erator. We normalize the embeddings Vylx to have unit length ||Vy/x = to ensure training stability and bound- edness of the preference score (see Appendix C for details on boundedness related to Theorem 5.1).
At each iteration t, GPO has the following learning objec- tive:
{10g
Toly | xx) xx)
DG11 arg min Ex y (-1x)
0
To, (y
1 (sly To, |x] - -lo Znoo (x)) B
27
(5.3)
The use of multiple di- mensions in the embeddings allows the model to discover different subspaces corresponding to various preference di- mensions automatically. Each pair of dimensions can cap- ture distinct aspects of preferences, such as helpfulness, correctness, or stylistic elements. The context-dependent eigenvalues (x) modulate the contributions of these sub- spaces based on the prompt, enabling the model to adapt to varying user preferences dynamically.
Automatic Subspace Discovery.
where we have the normalizing factor Zuo (x) Σy ne, (y/x) exp (y > Te, x)). In practice, we di- rectly replace log Zro, (x) with 0².
Intuitively, if a response y receives high average score, GPO will increase its log probability. We report the em- pirical performance of GPO in Section 6.3. The following theorem establishes the convergence properties of GPO;
5. General Preference Optimization
Theorem 5.1. Consider the optimization problem defined by the GPO loss (5.3) and assume is realizable. Let (To,) denote the sequence of policies generated by GPO, and define रा = T Ei= me, as the average policy. Given that the preference score s is bounded within [-p. pl, by setting /3 = - (VT). we have:
Policy Optimization with Preference Score. Once we have a general preference model that outputs the preference score s(y: > y)(x) at hand, we aim to find a policy ı that performs well against opponent policy j in terms of expected preference scores. The optimization problem is formulated as:
max (Ir > का) min s (I x RT) 0 (
max Ex [Ey y' x)) [s(y= > y' x)]] 0
-B (:TO: x) x))],
(5.1)
Connection Policy Gradient. Applying policy gradient on (5.1) gives:
where Tee is a reference policy (c.g., the initial language model), y is the opponent policy (usually the same as Tet)), and B > 0 is regularization parameter controlling the di- vergence from the reference policy. We would like point out that this formulation is different from the many previ- ous works (Wu et al., 2024b; Swamy et al., 2024; Rosset et al., 2024; Munos et al., 2023; Azar et al., 2023) as they consider maximizing the win rate Iy > y'(x), while our ((y>y'(x) formulation is maximize s(y y' x) log F(y-y' x) Note that Py(y > y' x) only varies between 0 and 1, while
Ex y (sly > TO.) - Blog T To, (y(x)
Vo log lo)
To T (y(()) (x)
2
Ex Ve s(y To.) 3 log Te. (y|x)
So Equation (5.3) can also be seen as policy gradient method for the optimization problem (5.1).
s(y > y' y')x), can be seen as a generalized version of the reward xx):)) in RLHF or DPO (see Section 4), can take arbitrary values.
Remark 5.2. Note that the general preference score given by our GPM can also be integrated as a preference (re- ward) signal for many other off-the-shelf RLHF and prefer- ence optimization methods, including (iterative) DPO-based
General Preference Optimization (GPO). We consider the iterative preference optimization process such as SPPO (Wu et al., 2024b), while we use preference score instead of pref- erence probability in the loss form. SPPO used K responses for each prompt x and calculated the empirical win rate of
IIn late stages of the iterative training, ne, is close to equilib- rium the preference model can not distinguish between policy no and the opponent policy Fo, ( meaning 8 (y > ne, x) R 0). Therefore, we have log 7 (x) Rs O.
6

---

Beyond Bradlcy-Terry Models; A General Preference Model for Language Model Alignment
methods (Rafailov al., 2024), IPO (Azar al., 2023), NLHF (Munos et al., 2023), SPPO (Wu al., 2024b) and REBEL (Gao et al., 2024), as well as PPO-based meth- ods (Ouyang et al., 2022) by directly optimizing prob- lem (5.1).
3.1-SB-Instruct base model, GPM achieves an average score of 91.90% (embedding dimension 8), representing a 1.34% improvement over the BT model's average score of 90.56%. In the Chat task, GPM improves from 88.55% (BT RM) to 93.58%, and in the Chat-Hard task, from 85.75% to 88.38%. These results indicate that GPM generally outperforms the BT model across various base models and tasks, particularly in the Chat and more nuanced or complex preferences. Note that BT RM is special case of GPM when the embedding dimension d = (see Section 4). Further analysis on challenging cases is in Appendix E.I.
6. Experiments
categories which may involve
We conducted several experiments evaluate the effective- ness of the proposed General Preference embedding Model (GPM) in comparison to traditional swaard-based models, particularly focusing on its ability to model general prefer- ence and improve language model alignment.
We conducted ablation studies to assess the impact of varying the embedding dimension in GPM. As shown in Table 2, the performance of GPM varies with the embedding dimension. For the Llama-3. base model, an embedding dimension of 8 achieves the high- est average score of 91. 91.90% compared to 91.86% with a dimension of 6 and 1.60%% with dimension of 4. In the Chat-Hard task with the same base model, the highest score of 88.38% is achieved with an embedding dimension of 2, compared 87.50% with dimension 8. In addition, we can find that for the Gemma-2B-it base model, the highest aver- age score of 82.29% is achieved with embedding dimen- sion of 6, showing an improvement over lower dimensions, such as 80.43% with dimension 4. These results suggest that the optimal embedding dimensions vary across different base models and tasks, For additional ablation studies on GPM architecture design, please refer to Appendix E.1.
A Ablation studies.
Table 1. Comparison of Bradley-Tery Terry (BT) reward models and General Preference embedding models (GPM) on eyclic preference datasets, Cyclic No. 1: Honest > Truthful > Helpful > Honesty; Cyclic No. 2: IF > Truthful > Helpful: > IF; Cyclic No. 3: IF > Honesty > Helpful > IF; Cyclic No. 4:
IF > Honesty > Truthful IF.
Model
Dataset
Acc. (%)
Random Guess
50.0
BT RM GPM
Cyclic No.
62.4
Cyclic No.
100.0 (+37.6)
BT RM
Cyclic No.
61.6
GPM
Cyclic No.
100.0 (+38.4)
BT RM
Cyclic No.
50.0
GPM
Cyclic No.
100.0 (+50.0)
BT RM
Cyclic No.
62.9
GPM
Cyclic No. 100.0 (+37.1)
6.1. Experiments on RewardBench
6.2. Cyclic Preference Modeling
We compare the GPM and BT reward model on the Re- wardBench benchmark (Lambert et al., 2024), which covers diverse preference modeling tasks, including Chat, Chat- Hard, Safety, and Reasoning.
We evaluate the ability of GPM to capture intransitive, cyclic preferences that traditional transitive models (like the BT model) struggle to represent. Specifically, we evaluate GPMs and BT RMs on CyclicPreference datasets, which are constructed based on the Ultrafeedback dataset (Cui et al., 2024) (See Appendix E).
Datasets and Experimental Setup. We train both BT RMs and GPMs using the decontaminated version of Skywork Reward Data Collection (Liu & Zeng, 2024), which contains around 80k pairwise preference data from tasks in various domains. We evaluate both models RewardBench, us- ing two different base models: Gemma-2B-it (Team et al., 2024) (2B parameters) and -lama-3.1-8B-Instrue (Dubey et al., 2024) (8B parameters), which are well-suited for instruction-following tasks (please refer to Appendix E.3 for the implementation details).
Training and Evaluation. We trained GPMs and BT RMs using the Gemma-2B8 2B-i language model as the base and evaluated the models based on their ability to predict in- transitive preferences. For GPM, the loss function is Equa- tion (A.1). For the Bradley-Terry (BT) model, the loss func- tion is L = log (r nn - rT) (Ouyang et al., 2022). Since cyclic preferences are inherently intransitive, we measure accuracy as the percentage of correctly predicted human preferences, where higher scores indicate better handling of non-transitive preferences. As shown in Table 1, the GP representation model achieves near-perfect accuracy across all datasets, significantly outperforming the BT model (we report the test accuracy on the training dataset but with dif- ferent comparison pairs used in the training dataset). These results validate GPM's ability to capture complex, cyclic
Results and Analysis. The results are presented in Table 2. On RewardBench, using the Gemma-2B-i base model, GPM achieves an average score of 82.29%, which is an improvement of 7.44% over the BT model's average score of 74.85%. Specifically, in the Chat task, GPM improves performance from 67.32% (BT RM) to 79.61% %, and in the Chat-Hard task, from 63.37% to 75.66%. For the Llama-
7

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Table Comparison between the Bradley-Temr (BT) models and the General Preference embedding models (GPM) with varying
embedding head dimensions on RewardBench. The highest scores are in bold. Note that BT RM is special case of GPM when embedding dimension d (see Section 4).
Model
Embed Dim.
Chat
Chat-Hard
Safety
Reasoning
Average
Base Model: Gemma-2B-it
RM GPM
67.32
63.37
85.68
83.04
74.85
2
77.37
73.46
85.00
85,50
80.33
4
78.77
72.59
85,54
84,82
80.43
6 79.61
75.66
85.27
88.61
82.29 (+7.44)
8
78.49
74.34
84.19
86.95
81.00
Base Model:
Llama-3.1-88 --B--a-3---BB-sttrec
BT RM
1
88.55
85.75
91.49
96.47
90.56
GPM
2
91.62
88.38
90.68
94.82
91.37
4
93.30
86.18
91.22
95.69
91.60
6
91.90
87.50
91.62
96.40
91.86
8
93.58
87.50
91.08
95.44
91.90 (+1,34)
Table AlpacaEval 2.0 evaluation results. Base model: Llama3-8B Evaluator: GPT-4-turbo. Results grouped by RM/GPM size
(2B/8B) and type (BT/GPM), and training iterations (Iter). Bold entries indicate GPM outperforms BT RM under the same SPPO/GPO setting. Metries follow AlpacaEval 2.0 standard: LC. WR Length-Controlled Win Rate (%), WR Win Rate (%) against the reference
Controlled
model (GPT-4-turbo), Avg. Len = Average response length (tokens).
SPPO WR
GPO WR
Size
Type
Iter
L.C. WR
Avg. Len
L.C. WR
Avg. Len
23.07
23.34
1959
23.07
23.34
1959
2B
BT RM
31.95
31.59
1939
34.01
33.08
1929
36.00
36.77
2032
38.90
39.90
2049
40.01
42.12
2136
42.21
44.20
2151
GPM
30.87
32.48 (+0.89)
2066
35.27
37.95 (+4.87)
2102
34.54
40.76 (+3,99)
2301
36.77
42.96 (+3.06)
2343
36.06
45.61 (+3.49)
2498
37.74
48.25 (+4.05)
2582
8B BT RM
32.20
27.83
1740
36.32
30.37
1702
39.75
36.95
1868
41.79
40.11
1933
42.55
40.92
1948
40.37
38.56
1969
GPM
33.48
30.85
(+3.02)
1861
36.00
33.19 (+2.82)
1850
37.93
38.38 (+1.43)
2029
40.81
42,80 (+2.69)
2115
39.45
41.64 (+0.72)
2385
38.98
41.54 (+2.98)
3249
preferences, confirming the theoretical advantages of using preference embedding- approach over traditional reward models that assume transitivity.
the generated responses of the aligned models with those of GPT-4o-mini and GPT-4-turbo. The results of the three evaluators are presented in Tables and 5. From Table 3, we observe that both SPPO and GPO demonstrate improved win rates with successive iterations, highlighting the iterative nature of these optimization methods, and GPO consistently outperforms SPPO. In addition, the bolded entries indicate that GPM-integrated methods consistently outperform BT RM-based methods under the same settings Win Rate (WR).
6.3. Downstream Performance Aligning Language Models with Human Preferences
We further investigate the effectiveness of GPM in lan- guage model alignment using Self-Play Policy Optimization (SPPO) (Wu et al., 2024b) and our proposed General Pref- erence Optimization (GPO), integrating preference scores provided by our GP representation model (GPM). We eval- uated the models AlpacaEval 2.0 (Dubois et al., 2024), MT-Bench (Zheng et al., 2023), GSM8K, MMLU, etc., sev- eral widely used benchmarks for evaluating LLM alignment.
7. Conclusion
In this work, we introduce preference embedding, a frame- work for modeling human preferences that can capture com- plex, intransitive structures. Our General Preference em- bedding model (GPM) achieves lincar complexity while
Results and Analysis. The evaluation results on the bench- marks are as follows. For AlpacaEval 2.0, we compared
8

---

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sra- vankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
can model intricate preference relationships. It consistently outperforms traditional models like Bradley-Temy reward models across various benchmarks, including eyclic pref- erence datasets and real-world tasks from RewardBench. Additionally, incorporating preference scores from GPM into policy optimization methods, such as SPPO and the newly introduced General Preference Optimization (GPO), led performance improvements in downstream tasks that require alignment with intricate human preferences.
Bertrand, Q.. Czarnecki, W. M., and Gidel, G. On the limitations of the elo, real-world games are transitive, not additive. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR. 2023,
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International Conference on Machine Learning, pp. 1597-1607, PMLR, 2020.
Impact Statement
This paper presents work to align AI systems with human preferences and values. Improving Al alignment could lead to more reliable and helpful Al systems that better serve human needs while respecting human values. The enhanced ability to model complex, intransitive preferences could result in Al systems that better understand and accommo- date nuanced human judgments across different contexts. This advancement underscores the potential of Al alignment algorithms in both technological and socictal contexts.
Christiano, F., Leike, J., Brown, T. Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://aas /awxxaavvvooro// org/ abs/ 2310. 01377.
Acknowledgements
Czarnecki, W. M., Gidel, G., Tracey, B., Tuyls, K., Omid- shafiei, S., Balduzzi, D., and Jaderberg, M. Real world games look like spinning tops. Advances in Neural Infor- mation Processing Systems, 33: 33:17443-17454, 2020.
We thank Andrew Yao, Yang Yuan, Kaifeng Lyu, Haodong Wen, and Yifan Luo for helpful discussions and feedback. We also thank the anonymous reviewers for their valuable comments and suggestions, which helped us significantly to improve this paper.
Dong, H., Xiong, W., Pang. B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf
workflow: From reward modeling to online rlhf. arXiv preprint 240X::::0050786333 2024.
References
Agranov, M. and Ortoleva, P. Stochastic choice and prefer- ences for randomization. Journal of Political Economy, 125(1):40-68, 2017.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, A general theoretical paradigm understand learning from human preferences. arXiv preprint ar wxx::2330..120555 2023.
A., and et al. The llama herd of models, 2024. URL https: / //aaxivvv... orgg abs 2407 21783.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04:: 04475, 2024.
Balduzzi, D., Tuyls, K., Perolat, J., and Graepel, T. Re- evaluating evaluation. Advances in Neural Information Processing Systems, 31, 2018.
Dudik, M., Hofmann, K., Schapire, R. E., Slivkins, A., and Zoghi, M. Contextual dueling bandits. In Conference on Learning Theory, pp- 563-58 PMLR, 2015.
Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W., Perolat, J., Jaderberg, M., and Graepel, T. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pp. 434-443. PMLR, 2019.
Freund, Y. and Schapire, R. E. Adaptive game playing using multiplicative weights Games and Economic Behavior, 29(1-2):79-103, 1999.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transac- tions on pattern analysis and machine intelligence, 35(8): 1798-1828, 1828, 2013.
Furuta, H., Lee, K.HH., Gu. S. S., Matsuo, Y., Faust, A., Zen, H., and Gur, I. Geometric-averaged preference opti- mization for soft preference labels. Advances in Neural Information Processing Systems, 37:57076-57111 2024.

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Gao, Z., Chang, J. D., Zhan, W., Oertell, O., Swamy, G., Brantley, K., Joachims, T., Bagnell, J. A., Lee, J. D., and Sun, W. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint mrXi::244.16667, 2024.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality, Advances in neural informa- tion processing systems, 26, 2013.
Gardner, M. Mathematical games. Scientific american, 222 (6):132-140, 1970.
Mitchell, E. A note on dpo with noisy preferences & rela- tionship ipo, 2023.
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Long- pre, S. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.071244 2023.
Hu, J., Wu, X., Wang, W., Xianyu, Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and high- perffrmmaee rlhf framework. arXiv preprint arXiv:2405. 113,, 2024.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations atari. Advances in neural information processing systems, 31, 2018.
Munos, R., Valko, M., Calandriello. D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T. Michi, A., et al. Nash learning from human feedback. arXiv preprint arXiv:2312.0083
2023.
Jiang, D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.02550 2023.
Ouyang, L.,. Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27781 2022.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Re- wardbench: Evaluating reward models for language modeling. https: //uggi wgwace...... co spaces/ allena//reward-berr bench. 2024.
Owen, G. Game theory. Emerald Group Publishing. 2013.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-folloiing models. https: :/gs /tttttub.bb..t///////////t/.th thub. com/tatsu-labb
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. openai.com. 2018.
alpaca_evali 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Liang, X., Chen, C., Wang, J., Wu, Y., Fu, Z., Shi, Z., Wu, F., and Ye, J. Robust preference optimization with provable noise tolerance for lims. arXiv e-prints, pp. arXiv-2404, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Liu, C. Y. and Zeng, L. Skywork reward model series. https:/ ///huggingaace..cc//////// col Skywork September 2024. URL https:/ httss::///ggggggaccc/.c co/Skywork.
Rosset, C.,, Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715. 2024.
Lou, H., Jin, T., Wu, Y., Xu, P., Gu, Q. and Farnoud, F. Active ranking without strong stochastic transitivity. Ad- vances in neural information processing systems, 35:297- 309, 2022.
Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. Xstest: A test suite for identifying exag- gerated safety behaviours in large language models, 2024. URL https:: //arxiv wwwreivv.v...eaeeeeeee www.o/gg////aaeeeee 01263.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-firn free reward. arXiv preprint arXiv:2405. /4734, 2024.
10

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv: 1707.06377, 2017.
M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing- In Proceedings of the 2020 Conference Empirical Methods in Natu- ral Language Processing: System Demonstrations, PP. 38-45, Online, Octoher 2020). Association for Compu- tational Linguistics. URL https: /wwww aclweb. org/antholsseeeo 2020 emn 1p demos.
Sen, Social choice theory. Handbook of mathematical economics, 10.077118811 1986.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint 2tt:::4008.0331444 2024.
Wu, Y., Jin, T., Lou, H., Farnoud, F., and Gu, Q. Borda regret minimization for generalized linear dueling bandits. arXiv preprint arXiv:2303.088// 2023.
Su. J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:12 12701,,, 2024.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.0072. 2024a.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.0400 2024.
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.000 00675 2024b.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S.,
Zeng. Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models evaluating instruc- tion following. In International Conference on Learning Representations (ICLR), 2024.
Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Par- rish, A., Ahmad, A., Hutchison, A., and et al. Gemma 2: Improving open language models a practical size, 2024. URL https::///// / arxiv. /rg/ abs/2408 00118.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu. Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a- judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306. 05685,, 2023.
Thurstone, L. L. A law of comparative judgment. In Scaling, pp. 31-92. Routledge, 2017.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 019999909999. 2019.
Tversky, A. Intransitivity of preferences. Psychological review, 76(1):31, 1969.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and offxxets arXiv preprint arXiv:2406.1 12845, 2024a.
Wang, Y., Liu, Q., and Jin, C. Is rlhf more difficult than standard rl? arXiv preprint arXiv:2306. 14/11, 2023.
Wang, Y., Li, H., Han, X., Nakov, P., and Baldwin, T. Do- not-answer: Evaluating safeguards in LLMs. In Graham, Y. and Purver, M. (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 896-911, St. Julian's s, Malta, March 2024b. Association for Compu- tational Linguisties. URI. http::/ / /aa//aaantti ogy. org/ org/2024. 2224. findings-eacl. 61.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.. Davison, J., Shleifer, S., von Platen, P., Ma, C., Jer- nite, Y., Plu, J., Xu, C., Scao, T. I.,, Gugger, S., Drame,

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Appendix
A More on General Preference Embedding
13
A.I Complex Embeddings Interpretation
13
A.2 Training Objective
14
B More on General Preference Optimization
14
Proofs of Theorems
14
C.1 Proof of Proposition A.1.
14
C.2 Proof of Theorem 4.4.
16
C.3 Proof of Theorem 4.5.
16
C.4 Proof of Theorem 5.1.
17
D More Related Work
19
E More on Experiments
19
E.1 Additional Ablation Studies
20
E,2 Additional Experimental Results
20
E,3 Implementation Details
22
Examples on Ultrafeedback Dataset
22
F.1
Example
22
F22 Example 2
25
12

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
A. More on General Preference Embedding
In this section, we present additional discussion on general preference modeling with preference embeddings.
Proposition A.I. For any two vectors V; IR and Vj I22K R IR satisfies the following two properties:
1. Skew-symmetry: (Rv; vy) -RVv, v:).
2. Magnitude preserving: (Rv;, Rv:) = (vi, Vi).
Then R must in the form R UJUT, where U 22 is an orthonormal matrix (e.g. identity matrix 124) and is a
block-diagonal matrix consisting of ls skem-symmetric blocks of the form:
0 -1
JI
1=
1, l.....kk k
1 0
A.1. Complex Embeddings Interpretation
Our model can also be interpreted using complex embeddings. By representing the embeddings complex vectors Vy Ck express the preference score as:
s syy: > Yj x) Im ((Vy. Vy,)),
where Im(-) denotes the imaginary part, and (:.) is the Hermitian inner product. This formulation captures cyclic and intransitive preferences through the angular relationships between complex presentations.
Theorem A.2 (Expressiveness Complex Preference Embeddings). Let P IRk* be a real skew-symmetric matrix (i.c., T). Then, there exist complex vectors {v; 1 C* such that:
Ps Im ((v.,,))) Vi,j.
Example. For 1, let Vy ei0,, then:
s(y; > yy x) sin(0y. -Oy))
0.00
2.02
0.00
-0.25
-3.25
-0.25
+0.00
-3.55
-6.50
-0.75
-3.55
-333 -0.72 -253 -0.2
0.25
1.0
-3.5
0.00
0.25
0.50
635
-1.00 -035 -2.00 -0.2 6.08 0.35
(a) Cyclic
(b) Cyclic 4
(c) Cyclic 5
Figure 3. Visualization of learned preference embedding vectors for eyclic preferences with sizes 3, 4, and 5, e.g., A > BxCx
13

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
A.2. Training Objective
The preference embedding can thus be obtained by minimizing the cross-entropy loss over observed preference data. Given dataset (x. y.e ~ D of preference comparisons, denote P(y.: mn > Yi x) the probability of the winner y being chosen over the loser Yt (1 if hard preference is given). The cross-entropy loss function is:
CCB=
Σ
Poly w > y1 log (3 > y1 1x)
(
(A.1)
+(1 PD(y > y x)) (-5o y1 x)) }
Alternatively, if there an oracle providing continuous scores, we can use regression loss:
(folsm> > y 1 - sp(y > yt xx) 2
MSEE
Σ
(I)
where (y > Yt x) is the dataset-provided score satisfying a (ND(Yw > yt x)) = PD(y > yı xx).
B. More on General Preference Optimization
Note that General Preference Optimization (GPO) employs iterative framework inspired by the multiplicative weights update (MWU) algorithm (Freund & Schapire, 1999), which update rule is formulated as:
1 x) x (y xx exp (1) (y > nt x)),
t 1.2.....
where 17 denotes the learning rate and s (y > We x) represents the preference score of response over the current policy Te given prompt x.
The von Neumann winner represents a fundamental concept in social choice theory (Sen, 1986) that has found significant applications in preference-bast based reinforcement learning (Owen, 2013; Dudík et al., 2015). corresponds the Nash equilibrium of two-player symmetric game (3.2), representing a mixed strategy-a probability distribution over possible responses- that performs optimally against any opponent in the worst-case scenario.
For notational clarity, we define the preference score of a policy over another policy n as:
(IT > x² x) Ey=n((lx), ((xx [s (y > y' xx]]
A distribution x² is formally defined as von Neumann winner when it satisfies:
min r'EA
Exsx [s (Ir" x² x)] 0.
This condition ensures that is, on average, at least as preferred as any other policy n'. The symmetric nature of the two-player game (3.2) guarantees the existence of such winner.
C. Proofs of Theorems
C.1. Proof of Proposition A.1.
Proof. Let R E 22kx2k be a real matrix satisfying the following properties:
Skew-symmetry with respect to the inner product:
Rv. w) RR, W
Magnitude preserving:
(Rv, Rv) (v,v), 2
14

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Recall that the standard inner product in R is given by (v, w) =v which is symmetric: (v,w), (w,, v).
From the skew-symmetry condition, we have:
Rv, w) + (Rw, v) = =0, Vv,w R
Since (Rw, v) = (Rw) Tv (v w R v, the above condition becomes:
W VV,WER
This implies that R is skew-symmetrie:
R =
From the magnitud-ppeeservnng property, we have:
(Rv, Rv) = (Rv) Rv RV V
Therefore,
R L2k.
Using R -R,, we obtain:
(- R 12k
R² -Iak-
This shows that Rt satisfies the equation R² = -22..
The characteristic polynomial of R is then:
det(RR - Al25) = 0.
Since R² Igk. it follows that the eigenvalues satisfy:
22 A ti.
Thus, R. has eigenvalues +i, each with algebraic multiplicity k.
Because R is real and akew-symmetric, it can be brought into flok-diagonal form via an orthogonal transformation. Specifically, there exists an orthogonal matrix U E 222x2 such that:
= UJUT.
where
J blockdiag J2, J2...... Jk),
and each block J, is a x skew-symmetric matrix of the form:
0
J
1= k.
This decomposition leverages the standard canonical form for real skew-symmetric matrices, which states that any such matrix can be orthogonally diagonalized into blocks of this type.
Therefore, R can be expressed as:
R = UJUT,
where E IR 2k an orthogonal matrix, and is the block-diagonal matrix consisting blocks Jr.
This completes the proof.
15

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
C.2. Proof of Theorem 4.4.
Proof.
We aim represent the entries of the skew-symmetric matrix _ using vectors in 1 and block-diagonal skew-symmetric matrix R IR 2
For each i = 1,...,k k. define the vector E 22 as:
a;
b,
where a,; b, R are real vectors to be specified.
Set a; e;, the i-th standard basis vector in R* and define b,
b, pi:
where p; is the i-th of P. Thus, the j-th component of b; is (b:); P.j-
Define the block-diagonal matrix R² E 2g as:
R BlockdiaggRR1.... R&):
where each block R2 the skew-symmetric matrix:
R = 0 -1 I....,k. k.
0
Now, compute the inner product v.R. RR RRvv;
v. R v, = [a]
b,]]
Oxx
I
aj
--a bj
by aj.
I
O
by
Since a, e,, we have:
1
b.b, e, by (by);
P3: Paj.
(C.1)
2
1
b. aj b, ej (b:)) 2 Pij-
(C.2)
Therefore,
V'RVV (-1 Pij
+
Pij Poj-
2
2
Thus, for all i,j.
Pij = B R
This construction shows that any real skew-symmetric matrix can be represented in terms vectors {v.} C 12 and the block-diagonal kxemerymmetric matrix R
This completes the proof.
C.3. Proof of Theorem 4.5.
Proof. Since is real and skew-symmetric with even dimension 2k, it can be brought into block-diagonal form via an orthogonal transformation. Specifically,, there exists orthogonal matrix U 222xx2 such that:
P UAU
16

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
where A is block-diagonal matrix composed of k blocks AiJ, with X 2 0 and
J
This decomposition leverages the fact that the eigenvalues P are purely imaginary and occur in conjugate pairs =iAr
Define the block-diagona matrix R blockdiag(J,, ...... J) E 22 and let
D blockdiag( V2 (x _22x22, where 12 is the 2 2 identity matrix.
Observe that A = DR
Set V = UD. Then,
P UAUT UDR VR.
Therefore,
Pg = - R Vi,J,
where is the i-th row of V.
This construction shows that any real skew-symmetric matrix P can be represented in terms of embeddings [vi} and the asymmetric operator R², confirming the full expressiveness our preference representation model.
C.4. Proof of Theorem 5.1.
Proof. The proof follows the logic of standard multiplicative weights update analysis, adapted for our preference score objective. First, since the preference score is bounded in [-p, p), we can normalize it to [0, 1] by the transforration:
s(y
y'
(y
x)
2p
2
By Theorem in (Freund Schapire, 1999), for any sequence of mixed policies μr, P2...., pr. the sequence of policies Ti, m2,..... By Ny produced by GPO satisfies:
T
T
5( X p4) + KL mp)
अ( x mi) S min Σ
1-e-
t=1
where 17 1/B is the learning rate parameter in the MWU view (related 3 in our GPO loss Eq. 5.3). The preference score is assumed bounded, which holds in our implementation due to L2 normalization of embeddings and bounded eigenvalues (Section 4.2).
Setting μ4 = ne, note that ö(m, < nT) = 1/2 due to the normalization and symmetry. Thus:
T
n/
KL [T])))
$ min
3( कT) +
2
1 e-
where ūy = + ET= Te is the mixture policy.
Rearranging terms:
7
KL mo)
min 3( x कt)
2m
T/
Since To is an autoregressive model with finite vocabulary support, log no(:) is bounded from above. Thus:
{T((=|<=)) S log TTO
Setting 17
log (o))m())) and using Taylor expansion
1-e- 2n
1/2 +
/T
log no (-)lo
log no (l)
+ +(TT)) < min [3(T - .T)
4
T
17

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Converting back to the original preference score scale:
min (s( kr)] 2
P
VT
By symmetry:
max [s( कr)] - P 0
P
VT
Therefore, the duality gap is:
max s(T > #τ) - min s( x TT)
max 8 > का) min s( .
o 1
Proof of Theorem A.2.
Proof. We aim to represent any real skew-symmetric matrix E Bixc using the imaginary parts of inner products of complex vectors.
For each = l,.... k define the complex vector v, a, + i b;, where a, b, E R Let a, e; the i-th standard basis vector in IR and set
b.
P eg
3=1
This implies that the j-th component of b, is (b:), = P.'..
The Hermitian inner product v. and v, is
(v., vy) = (a, -ib, +ib,) = a, aTa, 1ay++ba +b,,,, +i(b, a, -a,b,)
Therefore,
Im Im =b,aa a, -a.b.
Compute b. a, and a, by:
1
by a. (bi),
2 Pij.
1
a b, (b,); =
2 P3:
2 Pij.
since Pf+ = -Pas due to skew-symmetry.
Thus,
(-1 P.j
Im ((v.. Therefore, have constructed complex vectors v, such that
2 Pij
-P,j-
Pg = Im ((v,,v))) vz)), Vi.j.
This completes the proof.
18

---

Beyond Bradlcy-Terry Models; A General Preference Model for Language Model Alignment
D. More Related Work
Intransitivity in Game Theory. The symmetric zero-sum game and its intransitivity have also been frequently studied in the context of game theory. Balduzzi et al. (2018) explored game decompositions into transitive and eyclic components, proposing Nash averaging for agent evaluation. Balduzzi et al. (2019) generalized the results from matrix games to functional- form games and proposed algorithms for diverse agent populations. Czarnecki et al. (2020) investigated the geometrical properties of real-world games (e.g., Tic- Toe- Go, StarCraft II) and proposed that real-world games have "spinning top" geometry, with a strong transitive dimension and gradually diminishing non- non-transitive cyclic dimensions. Very recently, Bertrand et al. (2023) examined the limitations of the Elo rating system and proposed an alternative "disc decomposition" method that can better handle both transitive and cyclic game dynamics.
Representation Learning and Embedding. Representation learning involves learning transformations of data that make easier to extract useful information (Bengio et al., 2013). Techniques like word embeddings (Mikolov al., 2013) and contrastive learning (Chen al., 2020; Radford et al., 2021) learn powerful representations for various tasks. While these methods capture semantics or similarities, their direct application to modeling the directed, potentially asymmetric, and intransitive nature of human preferences in RLHF has been less explored. Our GPM specifically introduces a structured embedding space with a skew-symmetric operator tailored to capture these preference characteristics, offering a novel application of representation learning principles to the alignment problem.
Alternative Preference/Reward Models. Beyond the standard BT model, other approaches exist. Pairwise preference models (Section 3.3) often concatenate pairs, potentially suffering from order effects and quadratic complexity in the number of responses per prompt. Multi-dimensional reward models like ArmoRM (Wang et al., 2024a) decompose reward into interpretable dimensions (e.g., helpfulness, harmlessness). While offering interpretability, they typically require multi- dimensional annotations and, being based on summing sealar rewards, may still struggle to represent complex intransitive structures compared to GPM's more general formulation (Theorem 4.4). GPM aims for expressiveness and automatic discovery of preference dimensions from standard pairwise labels.
Preference Optimization Methods. Recent work has explored various algorithms beyond PPO for RLHF. DPO (Rafailov al., 2024) directly optimizes the policy using a derived reward. IPO (Azar et al., 2023) and related works (Mitchell, 2023; Liang et al., 2024; Furuta et al., 2024) focus on robustness or handling noisy/soft preferences within the DPO framework. For instance, Mitchell (2023) analyzed the relationship between DPO and IPO under noisy preferences, while Liang et al. (2024) and Furuta et al. (2024) proposed robust preference optimization methods to handle noise and soft labels, respectively. Nash-based methods (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024; Wu et al., 2024b) explicitly consider the game-theoretic aspect of finding optimal policies. Our GPO (Section 5) builds upon these ideas but uses the preference score s directly, offering an alternative optimization target compared to win rates or implicit rewards. GPM's preference scores can potentially be integrated into many of these existing optimization frameworks.
E. More on Experiments
Cyclic Preference Dataset. We constructed a dataset by inducing cyclic preferences from the Ultrafeedback dataset (Cui et al., 2024). The dataset includes responses evaluated across four key metrics: instruction following honesty, truthfulness, and helpfulness. We created preference cycles such as: instruction hel hellfulness > instruction following, ensuring the presence of intransitive eycles. We further generated four sub-datasets by omitting one metric from each cycle, resulting in 4 different datasets with 216 to 363 instances.
following ng > honesty tty > truth fu ness
We construct the Cyclic Preference Dataset using the following steps: 1. Filter Ultrafeedback for prompts with at least 3 responses where multi-aspect ratings (helpfulness, honesty, etc.) are available.
2. For a set of 3 responses (A, B, C) to the same prompt. check if ratings imply a cycle based on chosen criteria. Example: If criterion is Honesty and criterion 2 is Helpfulness, check if Rating(A, Honesty) > Rating(B, Honesty), Rating(B, Helpfulness) > Rating(C, Helpfulness). and Rating(C. Honesty) > Rating(A, Honesty). This implies a potential cycle A > B C > A across criteria. 3. Define pairwise preferences based the higher rating for the specific criterion involved in the cycle link (c.g., A >> B based on Honesty, B > C based on Helpfulness, C x A based on Honesty). 4. Collect these cyclic triplets (or longer cycles) to form the datasets used in Table 1. Different datasets (Cyelic No 1-4) were created by focusing on cycles involving different combinations of the four criteria.
19

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
E.1. Additional Ablation Studies
Ablations on Scale Gate and Embedding head. We investigate the effects of scale gates and embedding head dimensions, with and without L2 normalization, model performance. As shown in Table 4, for Gemma-28 -B-it models, incorporating scale gate generally enhances GPM performance across various embedding dimensions. L2 normalization embedding head output consistently improves models with scale gates. Interestingly, Gemma-2B Basraa-B-iit-tbattte it-deeeedd models without L2 normalization or scale gates outperform those with L2 normalization but no scale gates. A plausible explanation for this phenomenon is that removing L2 normalization introduces additional degrees of freedom, particularly beneficial for models with smaller parameter spaces and nigh-dimensional embedding layers. This increased fiexibility may allow the model to utilize its limited parametric capacity better, potentially leading to enhanced expressiveness and task specific adaptability,
the
Table Impact of the embedding head and the scale gate on GPM's performance on RewardBench. Dim. represents the dimension of the embedding head. The highest average scores for each base model bold.
Embedding Type
Dim.
Chat
Chat-Hard
Safety
Reasoning
Average
Base Model:
Gemma-2B
scale gate 12
2
77.37
73.46
85.00
85.50
80.33
w. scale gate w.o. 12
79.33
74.34
85.14
88.41
81.80
w. o. scale gate w. 12
2
78.49
71.27
85.68
86.13
80.39
W. o. scale gate w.o. 12
2
79.05
73.46
84.86
86.56
80.98
w. scale gate w. 12
4
78.77
72.59
85.44
84.82
80.43
w. scale gate 12
4
80.45
72.81
84.46
87.61
81.33
scale gate 12
4
79.61
70.39
85.00
86.84
80.46
w. o. scale gate w.o. 12
4
80.72
73.02
83.51
86.96
81.06
scale gate 12
79.61
75.66
85.27
88.61
82.29
w. scale gate w.o. 12
76.54
76.10
85.14
87.55
81.33
w. o. scale gate w. 12
6
79.61
71.05
85.81
87.74
81.05
w. o. scale gate w.o. 12
6
77.93
73.25
85,41
86.66
80.81
w. scale gate w. 12
8
78.49
74.34
84.19
86.95
81.00
w. scale gate 12
8
82.40
74.78
85.54
85.47
82.05
scale gate 12
8
77.09
72.15
86.08
85.41
80.18
w. o. scale gate w.o. 12
81.28
73.25
84.59
85.90
81.26
Base Model: -lama-31188--Inttrue
scale gate 12
2
91.62
88.38
90.68
94.82
91.37
w. scale gate w.o. 12
2
93.85
86.84
90.68
91.60
90.74
w. o. scale gate w. 12
2
92.18
86.18
91.89
94.05
91.08
w. o. scale gate w.o. 12
93.30 2
87.94
91.22
93.55
91.50
w. scale gate w. 12
4
93.30
86.18
91.22
95.69
91.60
w. scale gate w.o. 12
4
94.13
86.18
89.86
90.55
90.18
w. scale gate 12
4
92.46
87.28
91.76
93.19
91.17
w. scale gate 12
93.58 4
86.40
90.95
95.33
91.56
w. scale gate 12
6
91.90
87.50
91.62
96.40
91.86
w. scale gate 12
93.02
85.75
91.08
91.31
90.29
w. o. scale gate w. 12
92.18
85.53
90.81
94.20
90.68
w. o. scale gate w.o. 12
6
93.30
87.94
90.95
90.90
90.77
w. scale gate w. 12
93.58
87.50
91.08
95.44
91.90
w. scale gate w.o. 12
8
93.02
87.06
90.81
92.20
90.77
w. o. scale gate 12
8
91.90
86.62
91.22
92.63
90.59
scale gate 12
8
93.02
87.72
90.68
90.16
90.39
E.2. Additional Experimental Results
More Results on Language Model Alignment. We further conduct additional evaluations of our fine-tuned models using various benchmarks. AlpacaEval 2.0 evaluation results are listed in Table 5, using GPT-4o-mini as evaluators. For MT-Bench, we used the default mode to let GPT-4 grade and give a score to the model's answer, and the MT- Bench scores of aligned models are presented Table
20

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Table AlpacaEval 2.0 evaluation results. Base model: Llama3-8B-it, Evaluator: GPT--o-mini. The results grouped by the size and
type of the RM or PM, and the number of iterations. Bold entries indicate that GPM outperforms BT RM under the same training settings.
SPPO
GPO
Size
Type
Iter
LC. WR
WR
Avg. Len
LC. WR
WR
Avg. Len
base
23.07
32.26
1959
23.07
32.26
1959
2B
BT RM
48.8 84
46.09
1939
53.15
49.94
1929
59.77
58.41
2032
66.19
64.88
2049
66.81
67.14
2136
71.75
71.68
2151
GPM
48.09
49.15 (+3.06)
2066
55.66
57.12 (+7.18)
2102
56.63
63,53 (+5.12)
2301
61.11
67.78 (+2.90)
2343
60.77
70.91 (+3.77)
2498
64.52
74.78 (+3.10)
2582
8B BT RM
45.24
36.95
1740
49.77
40.26
1702
56.24
50.36
1868
60.75
56.30
1933
63.71
58.38
1948
62.63
59.17
1969
GPM
46.84
41,42 (+4,47)
1861
53.12
46.64 (+6.38)
1850
58.03
56.07 (+5.71)
2029
59.86
60.37 (+4.07)
2115
61.64
63.42 (+5.04)
2385
62.51
67.48 (+8.31)
3249
Table MT-Bench evaluation results. Base model: Llama3-8B-it, Evaluator: GPT-4. Bold entries indicate that GPM outperforms BT RM under the same training settings.
SPPO
GPO 2nd
Size
Type
Iter
1st
2nd
Avg.
Avg.
base
8.31
7.77
8.03
8.31
7.77
8.03
2B
BT RM
1
8.42
7.57
8.00
8.33
7.85
8.09
2
8.20
7.73
7.96
8.30
7.66
7.98
8.44
7.66
8.05
8.41
8.09
8.25
GPM
8.23
7.65
7.94
8.70
7.95
8.33
2
8.53
8.24
8.38
8.69
8.01
8.35
3
8.39
7.84
8.12
8.48
7.76
8.12
8B
BT RM
1
8.44
8. 10
8.27
8.41
7.85
8.13
2
8.75
7.85
8.30
8.73
7.83
8,28
3
8.34
7.99
8.17
8.68
7.83
8.26
GPM
1
8.43
7.94
8.18
8.29
7.90
8.10
2
8.51
8.05
8.28
8.26
7.99
8.13
3
8.47
7.76
8.12
7.57
7.51
7.54
Discussion Length Control (LC. WR). As observed in Section 6.3 (Table 3, 5), GPM-aligned models often produce longer responses than BT-aaignedd models. The standard AlpacaEval Win Rate (WR) doesn't penalize length, whereas the Length-Controlle Win Rate (LCC WR) is designed to mitigate length bias, potentially penalizing models that win primarily by being more verbose. GPM's multi-dimensional embeddings might better capture the value of comprehensive answers that address multiple facets prompt (e.g., being helpful, detailed, and stylistically appropriate simultaneously)) leading higher preference scores and thus longer generated responses under GPO. While beneficial for overall quality perceived by the preference model, this can negatively impact the LCC WR metric. For applications where conciseness highly valued length bias is major concern, specific techniques might be needed. One approach is length normalization within the optimization objective.
Length-Normalized GPO (LN-GPO) Results. We explored a variant, Gength-Normalized GPO (LN-GPO), which modifies the GPO loss (Eq. 5.3) to normalize the policy update by response length, similar spirit to Meng et al. (2024). The objective becomes:
(1//
2
To x)) no, (y x)
(IN.CPO(O) -Exxxx X. (-1x)
(
log
(3 me. xx) log Zio, (x))
Here, |y| is the length of the response y, and 7 a hyperparameter (typically 1). Initial results using LN-GPO with ? =
21

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
shown in Table 7. We observe that LN-GPO with GPM (2B) achieves a slightly higher LC. WR (45.55%) compared to LN-GPO with BT RM (,). while producing longer responses (Avg. Length 2112 1951). This suggests length normalization can help, although further tuning and investigation are needed to fully balance performance and length control with GPM.
Table 7. AlpacaEval 2.0 evaluation results with LN-GPO. Basc model: Llama3-8B-it Evaluator: gpt-4o-mini.
Model
Win Rate (%)
Avg. Length
LC. WR (%)
L .pp_22
48.31
2112
45.55
-aamnaa------- 3-88- Inst---nnttttt-tee-ee-e - .b
43.38
1951
45.51
E.3. Implementation Details
Details on Training Setup. Our experiments on RewardBench and Cyclic Preference Dataset were implemented using the HuggingFace Transformers library (Wolf al., 2020) and the OpenRLHF framework (Hu al., 2024). For reward model training on Skywork Reward Data Collection, we employed the following settings (in Table 8):
Gemma-2B-it: Trained with a learning rate of 2 10-6,
-lama-3.1-8B-Instrue Trained with a learning rate of 2 x 10-6.
Gemma-2-9B-it: Trained with learning rate of 10 6
Training Configuration: Both models were trained for two epochs with a global batch size of 32. We used cosine learning rate scheduler with a warm-up ratio of 0.03. Input sequences were truncated to a maximum length of 2048 tokens.
Hyperparameters: For general preference embedding model (GPM), set / 0.1, determined via hyperparameter tuning a validation set.
Hardware: All experiments were conducted on machines equipped with NVIDIA A800 80GB GPUs, utilizing 8 GPUs per experiment.
For cyclic preference experiments, the training settings are as follows, except for the parameters specified below; all other experimental parameters remain consistent with experiments on RewardBench (in Table 9):
Gemma-2B-it: Trained with a learning rate of 1 x 10-6,
Training Contiguration: Models were trained for 50 epochs with a global batch size of 1.
Hardware: Experiments were conducted on machines equipped with NVIDIA A800 80GB GPUs, utilizing a single GPU per experiment.
Details on Evaluation Dataset RewardBench. RewardBench is divided into four core sections:
Chat: Evaluates the ability to differentiate between thorough and correct responses in open-ended conversations, using
data from AlpacaEval (Li et al., 2023) and MT Bench (Zheng al., 2023).
Chat-Hard: Tests the handling of trick questions and subtle instruction differences, using adversarial examples from MT Bench and LLMBar (Zeng et al., 2024).
Safety: Assesses the capacity to refuse harmful content appropriately, using data from XSTest (Röttger et al., 2024), Do-Not-Answer (Wang al., 2024b), and a custom A12 dataset.
Reasoning: Measures code generation and reasoning abilities, with prompts from HumanEvalPack (Muennighoff et al., 2023) and PRM800K (Lightman et al., 2023).
F. Examples on Ultrafeedback Dataset
To illustrate that cyclic preferences can occur even without changing the criteria, we present several examples from the Ultrafeedback dataset (Cui et al., 2024).
F.1. Example 1
Prompt: "Provide a detailed history and significance of the Mississippi Arts Center Jackson"
22

---

Beyond Bradlcy-Terry Models; A General Preference Model for Language Model Alignment
Table 8. Implementation details for experiments RewardRench.
General Settings
Base models
Gemma-2b-it and
Batch size
32
Quantization for training
bf16
Learning Rate
2 x 10-6
Learning Rate Scheduler
cosine
Warmup Ratio
0.03
Max training epochs
Gradient accumulation step
Max input length
2048
Zero stage
Flash attention enabled
True
General Preference Model
/ for loss function
0.1
Table 9. Implementation details for experiments Cyelic Preference Dataset.
General Settings
Base models
Gemma-2b-it
Batch size
Quantization for training
bfl6
Learning Rate
1 x 10-6
Learning Rate Scheduler
cosine
Warmup Ratio
0.03
Max training epochs
50
Gradient accumulation step
Max input length
2048
Zero stage
3 True
Flash attention enabled
General Preference Model
8 for loss function
0.1
We have three responses within the Ultrafeedback dataset:
RESPONSE A:
"The Mississippi Arts Center, located in Jackson, Mississippi, is a vital cultural institution that has played significant role in the state's 's arts scene for decades. It is facility of the Mississippi Department of Archives and History and is home to the Mississippi Arts Commission, Mississippi Craft Center, Mississippi History Museum, and the Eudora Welty House Museum.
The Mississippi Arts Center was established in 1968 as the Mississippi Arts Commission. Its primary purpose is to support and develop the arts in Mississippi through a variety of programs and initiatives. Over the years, it has provided grants to arts organizations, schools, and individual artists; managed state artist roster; developed arts education programs; and promoted cultural tourism in Mississippi.
The physical Mississippi Arts Center building opened in 2004 and houses various facilities and spaces for art exhibitions, education, and programs. The Mississippi Craft Center is a featured part of the arts center, displaying and selling the works of local artisans and craftsmen. It also offers educational programs, classes, and workshops for both children and adults.
The Mississippi History Museum is another important component of the arts center. The museum explores the state's 's history
23

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
from prehistoric times up the present day, featuring exhibits on Mississippi's Native American culture, the Civil War, and the Civil Rights Movement.
In addition to these facilities, the Mississippi Arts Center is also home to the Eudora Welty House Museum, which is dedicated to the life and work of Eudora Welty, renowned short-story writer and photographer from Mississippi. The museum features tours of her home, which has been restored to its original condition.
The Mississippi Arts Center in Jackson holds significant importance as a vital cultural institution and resource for both the local community and the state of Mississippi. It plays a crucial role in promoting the arts, preserving history, and providing educational opportunities related Mississippi's s rich cultural heritage. As a result, it has become essential part of Jackson's 's cultural landscape and a must-visil for those interested in exploring Mississippi's 's unique history and arts scene.
Response A provides extensive information about the Mississippi Arts Center but mixes in details about other institutions like the Mississippi Arts Commission and the Mississippi History Museum, leading to partial inaccuracies.
Ratings of Response A.
Instruction Following: 2 (Partially addresses the instruction but includes unrelated institutions)
Honesty: (Confident and detailed)
Truthfulness:
(Some inaccuracies due to mixing institutions)
Helpfulness: 4 (Informative but somewhat confusing)
RESPONSE
The Mississippi Arts Center (MAC) is a state-funded, non-profit art museum located in Jackson, Mississippi. It is the largest art museum in the state, and its mission is "inspire creativity, enrich lives, and connect communities through the arts."
The MAC was founded in 1938 as the Mississippi Fine Arts Commission. It was originally housed in the Old Capitol building in Jackson. In 1976, the MAC moved to its current location in the former Woolworth 's building on Capitol Street.
The MAC's collection includes over 15,000 works of art, including paintings, sculptures, drawings, prints, photographs, and decorative arts. The collection represents a wide range of artists from Mississippi and the American South.
The MAC also offers variety of educational programs, including art classes, workshops, lectures, and tours. The MAC's Education Department serves over 100,000 people each year through its programs and outreach activities.
The MAC is significant cultural institution in Mississippi. It is place where people can come to learn about the arts, create art, and experience the power of art to enrich lives.
Here are some additional facts about the Mississippi Arts Center: The MAC's building was originally a Woolworth 's department store. It was designed by architect James Gamble Rogers, who also designed the Woolworth's 's buildings in New York City and Washington, D.C. The MAC's collection includes works by some of the most important artists of the American South, including William Eggleston, Walker Evans, and Eudora Welty. The MAC has hosted number of important exhibitions. including the first solo museum exhibition of the work of painter Kerry James Marshall. The MAC is member of the American Alliance of Museums. The MAC is accredited by the American Alliance of Museums.
Response B offers detailed facts about the Mississippi Arts Center but includes significant errors, such as incorrect founding year and location, despite being confident.
Ratings of Response B.
Instruction Following:
3 (Addresses some aspects but makes errors)
Honesty: 2 (Confident but includes significant mistakes)
Truthfulness: (Avoids fabrication but contains factual errors)
Helpfulness: 3 (Detailed but confusing key institutions)
24

---

Beyond Bradley- Herryy Models: A General Preference Model for Language Model Alignment
RESPONSE C:
The Mississippi Arts Center, also known as the Mississippi Museum of Art in Jackson, has a rich history and significance in the state's 's cultural and artistic landscape. Founded in 1911 the Mississippi Art Association, its primary mission was to promote the understanding, appreciation, and enjoyment of the visual arts in Mississippi.
In the beginning, the association held exhibitions in various locations across the state. However, in 1978, the Mississippi Museum of Art at the Mississippi Agency building in Downlown Jackson was established, providing a permanent home where the association could showcase its growing collection. Over time, the museum has evolved and expanded.
In 2007, the museum relocated to a new, state-of-the-an facility 380 South Lamar Street Downtown Jackson. This move allowed for the enlargement of galleries, versafile exhibition spaces, and the incorporation of educational programs and events.
Response C delivers a comprehensive history of the Mississippi Museum of Art, mistakenly equating it with the Mississippi Arts Center, but avoids fabrication.
Ratings of Response C.
Instruction Following: 5 (Comprehensive but misinterprets the prompt)
Honesty: 4 (Accurate information but about a different institution)
Truthfulness: (Misunderstands the instruction)
Helpfulness: 5 (Thorough and informative)
Analysis. Response A preferred over Response B: A is more accurate than B, which contains significant errors; Response B preferred over Response C: Some evaluators prefer over C because B discusses the correct institution (despite errors), while focuses on a different one. Response C preferred over Response A: Others prefer over A because C provides a thorough and helpful response, even if it's about different institution, whereas A includes confusing mixed references. This forms a cycle: x B C' > A, even though the question and evaluation criteria remain the same.
F.2 Example 2
"A whole school, a class-based cooking competition that runs throughout the week. Which class will win this year? Winners announced on Monday 10th July assembly. Based on the passage above, Could you provide some more details about the cooking competition, such the criteria for judging and the types of dishes being prepared?"
Prompt:
We have three responses within the Ultrafeedhack dataset:
RESPONSE A:
"The Mississippi Arts Center, located in Jackson, Mississippi, is a vital cultural institution that has played a significant role in the state's 's arts scene for decades. is a facility of the Mississiipi Department of Archives and History and is home to the Mississippi Arts Commission, Mississippi Craft Center, Mississippi History Museum, and the Eudora Welty House Museum.
Sure, here are some additional details about the school's cooking competition: The cooking competition is divided into three stages:
Stage 1: Prep Day During this stage, students are given a list of ingredients and set period of time to prepare their dish. The goal of this stage is to encourage creativity and problem-solving skills as students work with limited time and resources.
Stage 2: Cook-off Day- In this stage, students prepare their dishes in from of panel of judges. The judges will evaluate the
dishes based on set of criteria, such as taste, presentation, and creativity.
Stage 3: Judging After the cook-off. the judges deliberate and decide on the winning class. The winning class will be announced during school ussembly and will receive a prize.
The dishes prepared in the competition range from simple to complex, and each class is required to prepare a different dish. The criteria for judging the dishes include:
Taste:
How well does the dish taste? flavorful and balanced?
25

---

Beyond Bradlcy-Terry Models: A General Preference Model for Language Model Alignment
Presentation: How well is the dish presented? Is it appealing the eye?
Creativity: Was the dish creatively prepared? Did the student use unusual ingredients or techniques?
Overall Impression: How well did the student execute the dish? Did they meet the brief and deliver high-quality result? I hope this offers a more comprehensive view of the school's 's cooking competition."
Response A gives a detailed breakdown of the competition stages and judging criteria but introduces a three-stage format mentioned in the original prompt.
Ratings of Response A.
Instruction Following: 5 Provides detailed information about the competition's is structure, judging criteria, and dish preparation
Honesty: 4- Mostly correct, but some stages (e.g., Prep Day) may have been inferred rather than drawn from the given passage.
Truthfulness: Introduces a shree-stage competition format not mentioned in the original passage.
Helpfulness: 4- Highly informative, but lacks details about the specific types of dishes being prepared.
RESPONSE B:
Absolutely! It sounds like an exciting and fun-filled school cooking competition, with classes participating throughout the week. Here's some additional information based on the passage provided:
Criteria for Judging: Taste: The dishes should be flavorful, delicious, and appealing to the palate. Presentation: The dishes should be visually appealing, with attention given plating and garnishes.
Creativity: Participants should strive create unique, original dishes or add twist traditional recipes.
Teamwork: The class's 's ability to work cohesively and efficiently will also be taken into consideration.
Hygiene & Safety: Proper food handling practices and cleanliness will be of importance during the competition.
Types of Dishes Being Prepared:
Since the specific types of dishes are not mentioned the passage, here are some suggestions for categories that could be included in the competition:
Appetizers: Finger foods, dips, small bites showcasing culinary skills. Main Courses: Protein-based dishes with sides, vegetarian options, or one-pot meals. International Cuisine: Dishes inspired by the flavors and techniques of different regions.
Classic cakes, cookies, pastries, or innovative treats. Budget-Frienddh Nutritious and tasty dishes with minimal costs, teaching budgeting skills. Winners of the competition will be announced on Monday, the 10th of July during the assembly. Classes will eagerly await the results, and the winning class will earn the title of "School Cooking Champions" for the year. This competition is sure be engaging platform for students learn new skills, make friends, and have blast while cooking. Bon appétit!"
Desserts:
Response B offers speculative details about the competition, including possible dish categories and additional judging criteria not specified in the prompt.
Ratings of Response
Instruction Following: 4- Covers judging criteria and possible dish types but includes speculative details mentioned in the passage.
Honesty: 5 Confident and mostly correct in providing details.
Truthfulness: Slight hallucination about judging criteria (e.g., teamwork and hygiene) and dish categories.
Helpfulness: 5 Informaaivv and engaging, with a well-rounded discussion of potential competition elements.
26

---

Beyond Bradley-Toer -erryy Models: A General Preference Model for Language Model Alignment
RESPONSE C:
Sure, here are some more details about the cooking competition: The competition is open to all students in the school. Each class will be assigned different day of the week cook and present their dish.
The criteria for judging will be taste, creativity, and presentation. The winning class will be announced on Monday, 10th July, in the assembly.
The types of dishes being prepared are wide-ranging, and students are encouraged to be creative. Some popular dishes in past competitions have included:
Pizza
Pasta
Burgers
Sandwiches
Cakes
Cookies
The cooking competition is great way for students to learn about different cuisines and cultures, as well to develop their cooking skills. It is also lot fun!
Response sticks closely to the prompt, providing clear judging criteria and examples of dishes, but lacks depth compared to the other responses.
Ratings of Response C.
Instruction Following: 4 Provides clear judging criteria and dish examples but lacks depth in detailing the competition format.
Honesty: - M-ossly correct but assumes information (e.g., students being encouraged to be creative).
Truthfulness: Free of hallucinations and accurately expands on possible competition elements.
Helpfulness: 5 (Thorough and informative)
Analysis. A preferred over B: Some evaluators might prefer A over because it provides a structured and detailed response, despite introducing unconfirmed elements; B preferred over C: Others might prefer B over C due to its engaging style and comprehensive coverage, even some details are speculative; C preferred over A: Meanwhile, some may prefer C over because sticks closer to the information provided, avoiding potential inaccuracies introduced by A. This results in preference cycle: > > C
27


Reviews:
[{"review_id": "OgzyvOJPXY", "reviewer": "Reviewer_gcRo", "summary": "This paper proposes *General Preference Embedding Model* (GPM) to improve LLM alignment on human preference. \nThe motivation is mainly on addressing limitations of the classical BT reward models such as challenges when facing intransitivity. \nThe authors deal with it by embedding the responses into latent space, introducing a skew-symmetric preference operator to derive preference scores. \nThis approach is novel in the context of LLM alignment and, by their empirical results on AlpacaEval2 and RewardBench, efficient and capable in capturing intransitive preference structures with linear computational complexity O(K).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, insightful feedback, and constructive comments on our paper. We appreciate the recognition of GPM's novelty, theoretical grounding, and universal framework potential.\n\nWe would like to address the specific points raised:\n\n>  Q1: Generalisability and further results on complex cyclic preferences.\n\nA1. We appreciate the suggestion regarding generalisability. As shown in Section 6.2/Table 1, GPM achieves near-perfect accuracy on cyclic preference datasets where BT models perform near-random guessing. This demonstrates GPM's effectiveness in capturing the intransitivity inherent in complex preference structures, a key limitation of BT models.\n\n> Q2: Need for more diverse, real-world scenarios/context, noting benchmark challengeability.\n\nA2. Thanks for your suggestion. We tested GPM on RewardBench (covering Chat, Chat-Hard, Safety, and Reasoning) and assessed downstream alignment using AlpacaEval 2.0. GPM consistently outperformed BT models across these benchmarks and different base models (Gemma-2B/9B, Llama-3.1-8B). GPM-integrated methods also showed improved win rates. While benchmarks have limitations, these consistent gains across diverse tasks suggest practical advantages.\n\n> Q3: More ablation/real-world scenarios are needed to make really strong claims.\n\nA3. We acknowledge your point that further studies could strengthen our claims. However, we believe the current results already provide significant support. The demonstrated superiority in modeling cyclic preferences, consistent outperformance on the diverse RewardBench tasks, and improvements in downstream alignment tasks collectively offer strong empirical evidence for GPM's effectiveness and advantages over traditional BT models. We have also included ablation studies on embedding dimensions (Section 6.1, Table 2) and GPM architecture design (Appendix E.1, Table 4).\n\n> Q4: Curiosity about embedding dimension analysis for competing/conflicting preferences and how GPM manages trade-offs.\n\nA4. Thanks for your feedback on analyzing embedding dimensions in scenarios with competing preferences and how GPM handles trade-offs. This is an excellent question that touches upon a core aspect of GPM's design. As discussed in Section 4.2, the multi-dimensional embedding space allows the model to automatically discover subspaces corresponding to various preference facets (e.g., helpfulness, honesty, style). The eigenvalue scale gate, which computes context-dependent eigenvalues {λ(x)}, then modulates the influence of these different dimensions based on the specific prompt. This mechanism allows GPM to dynamically weigh competing preference aspects and manage trade-offs based on context. Our ablation studies on embedding dimensions (Section 6.1, Table 2 and Appendix E.1, Table 4) further explore how performance varies with dimensionality across different tasks and models, indicating that the optimal configuration can depend on the specific trade-offs required.\n\n> Q5: Adding references on representation learning.\n\nA5. Thank you for suggesting a more extensive review of representation learning. We have acknowledged this connection briefly in Appendix D and agree that it's a relevant area. We will expand on this connection in the revised version.", "claims_and_evidence": "The authors provide some theoretical results, which are personally appreciated, and empirical support. \nIn general I think the claims are solid. \nBut I will definitely appreciate further results on the generalisability of this novel method like for complex cyclic preference (please see 'weakness' below).", "methods_and_evaluation_criteria": "Experiments with cyclic preference datasets and RewardBench convincingly illustrate the performance advantages of GPM over BT models. \nHowever, It would be beneficial to include additional context or examples from diverse, real-world scenarios (some benchmarks are not too hard to 'hack' by powerful models).", "theoretical_claims": "I checked the theoretical claims and think they are clearly stated and easy to understand.", "experimental_designs_or_analyses": "It would be great if the authors could clarify whether additional robustness checks or sensitivity analyses were conducted, particularly regarding different user demographics or task variations.", "relation_to_broader_scientific_literature": "This paper focused on RLHF but inspired by literature like preference learning and statistical methods.", "essential_references_not_discussed": "The references are adequate, yet some review on representation learning may be appreciated.", "other_strengths_and_weaknesses": "Strengths:\n\n1. As I stated before, I think the theoretical results and efforts should be appreciated, especially in the current LLM community. \n2. the 'universal' framework which is inclusive of current methods (for example, k=1 -> BT model) is a good and inspiring idea. It can also deepen the understanding of the current popular methods. \n\nWeakness:\n\n1. While the method is indeed novel and I personally like the paper, I reserve my recommendation for a strong acceptance because tbh some results on these benchmarks may be challengeable, and more ablation and real-world scenarios are needed to make really strong claims, and some may argue that cute math often really doesn't buy you much.", "other_comments_or_suggestions": "No more.", "questions_for_authors": "I'm really curious about the embeddings' dimension, like have you analyzed scenarios where embedding dimensions reflect competing or conflicting preferences? If such scenarios were encountered, how does the GPM manage these practical trade-offs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9fxdugLMi5", "reviewer": "Reviewer_awzY", "summary": "This paper introduces preference embedding, a novel approach to model human preferences for aligning foundation models that overcomes the limitations of traditional reward models like the Bradley-Terry model, especially in capturing intricate preferences. The authors propose the General Preference embedding Model (GPM), which embeds responses into a latent space, which is more expressive. Furthermore, the paper presents General Preference Optimization (GPO), a method that generalizes reward-based RLHF using the preference scores from GPM. Experimental results on RewardBench demonstrate that GPM consistently outperforms the BT reward model, particularly in modeling cyclic preferences. Evaluations on downstream tasks like AlpacaEval 2.0 indicate that aligning language models with GPO and GPM leads to performance improvements over methods using Bradley-Terry models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive comments. We especially appreciate the feedback regarding Table 3 and acknowledge that the acronyms used were not sufficiently explained in the current draft.\n\n1. Clarification of Table 3:\n> Q1: First of all, it's full of acronyms (e.g. LC. WR, WR, Avg Len), and the acronyms are not explained in the caption and main text. I suppose WR probably means \"win rate\", but it's also unclear to me if it means the win rate over what. I can't understand the results, unfortunately.\n\nTo clarify:\n- LC. WR stands for Length-Controlled Win Rate, a metric from AlpacaEval 2.0 that adjusts for length bias in generation.\n- WR denotes the Win Rate, i.e., the proportion of times our model's response was preferred over the baseline (GPT-4-turbo)'s, as judged by GPT-4-turbo.\n- Avg. Len refers to the Average Length of generated responses in tokens.\nThe win rates are computed using pairwise comparisons between model outputs (e.g., our trained models vs. GPT-4-turbo) on the same set of prompts, with GPT-4-turbo as the evaluator.\n\n2. Why These Acronyms Were Used:\n> Again, I don't fully understand the experiment settings because of writing issues. \n\nAlpacaEval is a widely used benchmark in nearly all recent LLM alignment papers (e.g., SimPO, SPPO, MagPie, Nemotron), and the format and metrics (including LC. WR) are standard in the literature. Due to space constraints, we opted to use the established abbreviations. That said, we agree that clearer exposition would benefit broader readability, and we will revise the caption and surrounding text to explicitly define these terms.\n\n3. On the Role of This Table:\n\n> Q3: Table 1 presents a comparison of Bradley-Terry (BT) reward models and General Preference embedding models (GPM) on cyclic preference datasets. The results seem convincing and support the claim. The main problem is with Table 3, which shows the main result.\n\nWe emphasize that our primary contribution is the development of a new, expressive architecture (GPM) for modeling general preferences, which addresses the limitations of traditional reward models like Bradley-Terry. The GPO-based alignment results in Table 3 are supplementary and serve to illustrate one possible downstream use case (Section 6.3). We believe that the clarity issues in Table 3, while important to correct, should not detract from the significance of our core contributions.\n\n4. Additional Revisions:\n\n> Q4: I'm not aware of critical misses. However, I think it'd be better to add references to more prior work in preference optimization, such as:\nMitchell, A note on DPO with noisy preferences & relationship to IPO, 2023 \nLiang et al., Robust preference optimization with provable noise tolerance for LLMs, 2024 \nFuruta et al., Geometric-Averaged Preference Optimization for Soft Preference Labels, 2024.\n\nWe also appreciate the reviewer’s suggestions on related work. We will incorporate a discussion of the following relevant papers in the final version:\n\nMitchell, A note on DPO with noisy preferences & relationship to IPO, 2023 \n\nLiang et al., Robust preference optimization with provable noise tolerance for LLMs, 2024 \n\nFuruta et al., Geometric-Averaged Preference Optimization for Soft Preference Labels, 2024.", "claims_and_evidence": "Table 1 presents a comparison of Bradley-Terry (BT) reward models and General Preference embedding models (GPM) on cyclic preference\ndatasets. The results seem convincing and support the claim. The main problem is with Table 3, which shows the main result. It is very unclear. First of all, it's full of acronyms (e.g.  LC. WR, WR, Avg Len), and the acronyms are not explained in caption and main text. I suppose WR probably means \"win rate\", but it's also unclear to me it means the win rate over what. I can't understand the results unfortunately. This paper has obvious writing quality issues. This is the main reason to recommend rejecting this paper.", "methods_and_evaluation_criteria": "The method makes sense. However, as I explained previously, I don't fully understand the evaluation criteria because of writing issues.", "theoretical_claims": "I don't find obvious issues in the theoretical claims.", "experimental_designs_or_analyses": "Again, I don't fully understand the experiment settings because of writing issues.", "relation_to_broader_scientific_literature": "It's related to RLHF and LLM preference optimization literature.", "essential_references_not_discussed": "I'm not aware of critical misses. However, I think it'd be better to add references of more prior work in preference optimization, such as:\n\nMitchell, A note on DPO with noisy preferences & relationship to IPO, 2023\nLiang et al., Robust preference optimization with provable noise tolerance for LLMs, 2024\nFuruta et al., Geometric-Averaged Preference Optimization for Soft Preference Labels, 2024.", "other_strengths_and_weaknesses": "No additional comments", "other_comments_or_suggestions": "I wasn't sure what (line 317) \"We consider the iterative preference optimization process such as SPPO...\" means. I suppose the authors actually means \"GPO is a iterative preference optimization process, similar to SPPO...\"?", "questions_for_authors": "Please revisit the presentation of the experiment section. I believe the whole thing needs to be rewritten.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrJdnp7sNu", "reviewer": "Reviewer_o37T", "summary": "Since the prevalent Bradely-Terry formulation and pair preference model have limitations respectively in reward modeling, the authors propose a novel formulation GPM with better expressiveness to model complex preference distributions in real world. They further extend GPM to preference learning to build GPO, a new algorithm that better align LLMs with human preference.\n\n## update after rebuttal\n\nI have read the author response and would like to keep my positive score unchanged.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and detailed review. We appreciate your constructive feedback and the opportunity to clarify aspects of our work. We address your points below:\n\nBelow, we try our best to address the points you raised:\n\n1. GPM Architecture, Training, & Efficiency:\n> Q1: The authors did not elaborate on the details of GPM's model architecture, and training objectives... hard to verify the claimed computational efficiency...\n\n- Architecture/Implementation: We describe the implementation of GPM in Section 4.2, detailing the eigenvalue scale gate and the eigenvector embedding head used to generate the preference embeddings.\n- Training Objective: The optimization objective for training GPM is explicitly defined in Appendix A.2 (Equation A.1). It involves minimizing the standard cross-entropy loss used in preference learning, based on the predicted preference probability ($\\sigma(s(y_w>y_l∣x)$), where s is the GPM score from Eq 4.1) and the observed preference data $P_D(y_w>y_l∣x)$. This objective directly mirrors the typical objective used for training BT-based reward models.   \n- Computational Efficiency: GPM achieves $\\mathcal{O}(K)$ query complexity (Sec 4), matching BT models and improving on $\\mathcal{O}(K^2)$ pair-based models, by processing responses individually.\n\n2. GPM Performance on RewardBench:\n\n> Q2: Although GPM is claimed to have stronger expressiveness, it does not seem to show a consistent bonus on RewardBench... \n\n- Thank you for your observation regarding RewardBench performance. While performance can vary across specific sub-tasks and embedding dimensions (as shown in our ablation studies in Section 6.1 and Appendix E.1), GPM consistently outperforms the BT reward model on average across several tested base models, as shown in Table 2. Specifically, GPM showed average improvements of +7.44% (Gemma-2B-it), and +1.34% (Llama-3.1-8B-Instruct) over the BT baseline. Significant gains were often observed in the Chat and Chat-Hard categories. We appreciate the suggestion to analyze complex cases further and will add this to the appendix.\n\n3. Clarity on Cyclic Preference Setting:\n\n> Q3: ...The remaining issue can be on cyclic preference, which I fail to understand the settings. A step-by-step explanation... would make it more clear.\n\n- These datasets were constructed using Ultrafeedback data, where we identified instances exhibiting cyclic preferences based on ratings across different criteria (e.g., helpfulness, honesty, instruction following). For example, a cycle might emerge where Response A is rated higher on honesty than B, B higher on helpfulness than C, but C higher on honesty than A. We provide details on the construction in Appendix E and results demonstrating GPM's near-perfect accuracy on these tasks in Section 6.2 and Table 1. Figure 3 also visualizes the learned embeddings for these cyclic cases.   \n\n- We will add a more detailed step-by-step explanation of the dataset construction to the appendix in the revised version, as you suggested.\n\n4. Preference Embedding Significance vs. ArmoRM:\n\n> Q4: ...similar to ArmoRM... each head in ArmoRM has a clear meaning... while heads in GPM do not. Therefore, simply adding all scalars... \n\nGPM's design prioritizes expressiveness and generality. The core idea is that complex, real-world human preferences might not always decompose neatly into a few predefined, interpretable scalar dimensions. They can exhibit intransitivity and context-dependency that require a more flexible representation.\n\n- Theoretical Grounding: Our multi-dimensional embeddings, combined with the skew-symmetric operator $\\mathbf{R}^{\\succ}$ and inner product $< \\mathbf{R}^{\\succ} v_{y_i|x},v_{y_j∣x}>$, provides a principled way to model any skew-symmetric preference relationship, including cycles. Theorem 1 and Theorem 4 provide theoretical grounding for this expressiveness.\n- Expressiveness and Generality: While ArmoRM's interpretable heads offer clarity, this structure inherently limits its capacity. Such models based on aggregating scalar rewards, even multi-dimensional ones, generally cannot capture arbitrary preference structures, particularly those involving intransitivity (like cycles) or complex mixtures of transitive and intransitive components.\n- Automatic Learning vs. Annotation: Furthermore, ArmoRM relies on aggregating scores from heads associated with predefined metrics (like helpfulness, truthfulness), which often require costly, explicit human annotation for each metric. In contrast, GPM aims to automatically discover and learn the relevant multi-dimensional preference representations directly from a single, standard pairwise preference signal ($y_w \\succ y_l$). The dimensions in GPM are learned end-to-end to capture relevant preference factors without needing pre-defined semantics or multi-metric labels. The eigenvalue scale gate further allows the model to dynamically weight these learned factors based on the context.", "claims_and_evidence": "1. The authors did not elaborate on the details of GPM's model architecture, training objectives, and deployment. This makes it hard to verify the claimed computational efficiency advantages in the section of Introduction.\n2. Although GPM is claimed to have stronger expressiveness, it does not seem to show a consistent bonus on RewardBench. However, considering the positive results of GPO on AlpacaEval, I suspect that there are some complex cases in RewardBench that confuse GPM. Adding analyses on these case will be helpful.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have checked the content in Sec 4 related to GPO formulation, which is correct and interesting.", "experimental_designs_or_analyses": "I have checked the detail of all experiments. The remaining issue can be on cyclic preference, which I fail to understand the settings. \nA step-by-step explanation in the appendix would make it more clear.", "relation_to_broader_scientific_literature": "The primarily related domain is human preference alignment of LLM, where GPM can be useful in RLHF and supervised methods like DPO, SimPO, and so on.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The writing is not clear enough. For example, the authors did not clearly explain the optimization objective of GPM. Is it the same with Bradely-Terry?", "other_comments_or_suggestions": "Typo: The use of \"Sec 3.2.1\" could be changed to \"Sec 3.3\"?", "questions_for_authors": "The proposed preference embedding seems to map the last-layer hidden state through different heads to multiple scalars (similar to multi-head attention) and then sums them to obtain the final reward. It seems somewhat similar to ArmoRM[1]. However, each head in ArmoRM has a clear meaning, making the scoring more interpretable, while heads in GPM do not. Therefore, simply adding all scalars at the end of the forward pass makes me skeptical about its substantive significance.\n\n[1] https://arxiv.org/abs/2406.12845", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGrVXjVtUQ", "reviewer": "Reviewer_rNmg", "summary": "This paper proposes an approach to modeling preferences by learning an embedding for each response given a prompt. The preference score between two responses is then computed using these embeddings. The key motivation behind this embedding-based approach is that mapping responses into a multi-dimensional latent space allows the model to better capture complex preference structures, including potential cycles in the preference data.\n\nBeyond testing the preference model, the authors apply it to preference optimization. They modify the SPPO method by replacing preference probabilities with preference scores in the loss function, introducing a variant they call General Preference Optimization (GPO). Experiments on AlpacaEval suggest that the proposed method improves win rates in some cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful feedback on our manuscript. We appreciate the time you took to evaluate our work and provide constructive comments. We believe addressing your points will significantly strengthen the manuscript. \n\n1. Intransitivity Usefulness: \n> Q1: The paper would benefit from providing more evidence on why better modeling of intransitive human preferences is useful...\n\nThanks for your suggestion. Modeling intransitivity is motivated by observing that human judgments can be complex, context-dependent, and exhibit cycles, which simpler Bradley-Terry (BT) models cannot capture. GPM's multi-dimensional space aims to represent these richer structures. While noise is a factor, capturing potentially genuine complex preferences (Appendix F examples) is vital for alignment. Distinguishing noise from true intransitivity is important for future work. \n\n2. Line 114 Citations: \n> Q2: Additionally, in line 114, the authors cite two papers,...\n\nThe two references (Tversky, 1969; Agranov & Ortoleva, 2017) are among the earliest to study the prevalent phenomenon of humans exhibiting non-transitive preferences. We use them to motivate moving beyond transitive models like BT. We will clarify the scope of these references in our revision.\n\n3. Cyclic Data: \n> Q3: However, the cyclic preference data created by the authors...\n\nThis synthetic dataset demonstrates GPM's expressiveness where BT models inherently fail (random guessing) due to their transitivity assumption. Appendix E details its construction from Ultrafeedback; we'll add a summary to the main text. \n\n4. Boundedness for Theorem 3: \n> Q4: For example, in theorem 3, the preference score needs to be bounded for the convergence result to hold,...\n\nThanks for the feedback. Our GPM implementation ensures boundedness. We apply L2 normalization to embedding vectors $v_{y \\mid x}$, making them unit length. Since the score is $s\\left(y_i \\succ y_j \\mid x\\right)=v_{y_i \\mid x}^{\\top} D(x) R^{>} D(x) v_{y_j \\mid x}$, and $R^{>}$ is magnitude-preserving, scores are bounded if eigenvalue scales $\\lambda_l(x)$ are bounded. This is guaranteed as we use a softmax function on the gate outputs determining eigenvalues. Thus, theorem conditions hold. We'll clarify.\n\n5. Cyclic Experiments: \n> Q5: Regarding the cyclic preference experiments (Table 1), ...,\n\nSynthetic data provides a clear proof-of-concept for modeling intransitivity. We agree that evaluating cycle prevalence in real benchmarks and comparing generalization vs. BT models are valuable future work. Appendix F shows potential real-world examples.\n\n6. Table 3 (SPPO+GPM vs. GPO): \n> Q6: First, what is the difference between SPPO+GPM and GPO?... Second, according to the LC win rate metric...\n\nSPPO+GPM uses win rates from GPM scores; GPO uses raw scores s(yi>yj∣x) directly. GPO maximizes expected score; win-rate methods maximize expected probability. Raw scores might offer a richer signal (any real number) vs. win rates (0 to 1), potentially explaining GPO's performance despite win rates offering stability. We'll clarify. We acknowledge GPM/GPO doesn't always beat BT/SPPO on Length-Controlled (LC) win rate. GPM/GPO models tend to produce longer responses (\"Avg. Len\"). As LC Win Rate controls length bias, this might impact results. The appendix shows length-normalized GPO (LN-GPO) results. More discussion will be added. \n\n7. Section 3.2 (Preference Modeling): \n> Q7: Section 3.2 on preference modeling does not accurately represent the existing literature...\n\nSec 3.2.1 introduces PairPMs for general preference modeling, distinct from standard BT models (Sec 3.1). Lines 188-190 illustrate PairPM examples and potential issues, not all reward model training. We'll revise for clarity, distinguishing standard models (scalar head, BT loss) from PairPMs. We'll add a discussion on Stiennon et al. (2020) and Lambert et al. (2024). \n\n8. Notation (K vs. d, Complexity):  \n> Q8: Some of the notation is a bit confusing... number of responses is K and the embedding dimension is 2k...\n\n$K$ is response count; 'd' is embedding dimension (formerly $2k$) . $K, d$ are unrelated. Complexity concerns model forward passes: GPM/BT require $\\mathcal{O}(K)$ passes (one per response) for embeddings/rewards. PairPM requires $\\mathcal{O}\\left(K^2\\right)$ passes (one per pair). GPM offers linear scaling of model calls w.r.t. K, matching BT, improving over PairPM. We'll clarify notation/complexity. The $P \\in \\mathbb{R}^{2 k \\times 2 k}$ setup in Thm 2 illustrates spectral decomposition; our method (Sec 4.2)/Thm 1 doesn't require $d=K$. We'll clarify this distinction.\n\n9.  Automatic Subspace Discovery: \n\nWe appreciate your interest. You asked for empirical evidence of interpretable directions/eigenvalues. Excellent suggestion for future work. Analyzing embeddings/eigenvalues via probing/visualization could yield insights. Figure 3 is a first step; systematic investigation is needed. We'll add this as a future direction.", "claims_and_evidence": "The paper would benefit from providing more evidence on why better modeling of intransitive human preferences is useful. Another perspective to consider is that some of the intransitivity observed in preference data may stem from noisy judgments. In that case, improving the model’s ability to capture intransitivity could lead to overfitting to noise rather than genuine preferences. Additionally, in line 114, the authors cite two papers, but neither directly addresses human preference data in the context of RLHF.", "methods_and_evaluation_criteria": "The methods and datasets used for evaluation (RewardBench and AlpacaEval) are generally reasonable.\n\nHowever, the cyclic preference data created by the authors is not well-motivated. It would also be helpful to include some basic details about how this dataset was constructed in the main text.", "theoretical_claims": "Most theoretical claims do make sense to me. However, there is a disconnect between the theory part and the method itself. For example, in theorem 3, the preference score needs to be bounded for the convergence result to hold, although it is not really mentioned if the preference scores obtained with GPM are actually bounded.", "experimental_designs_or_analyses": "Regarding the cyclic preference experiments (Table 1), my main concern is that the data is synthetic. First, it is unclear how often cyclic dependencies actually occur in real-world data. Second, as previously mentioned, it is not evident that modeling such phenomena more effectively would lead to better generalization in reward models. To make this experiment more convincing, I suggest first reporting the number of cycles found in existing benchmarks and then evaluating whether a reward model trained with the proposed method generalizes better than, for example, the Bradley-Terry (BT) model.\n\nThe results in Table 3 are somewhat confusing. First, what is the difference between SPPO+GPM and GPO? Does the former use win rates in its loss function (calculated based on GPM), while the latter directly uses the scores? If so, do the authors have an intuitive explanation for why using raw scores yields better results? The existing literature generally suggests that win rates tend to work better than raw rewards, as they are invariant to monotonic transformations and contribute to training stability.\n\nSecond, according to the LC win rate metric, the proposed method does not outperform existing approaches. However, I could not find any discussion around this in the paper and explain why this is the case. This needs to be discussed in the corresponding section.", "relation_to_broader_scientific_literature": "The findings support previous claims about the limited expressivity of the Bradley-Terry (BT) model. The use of embeddings and the construction of preference scores with a skew-symmetric matrix is an interesting contribution to the literature and presents a promising alternative to the BT model.", "essential_references_not_discussed": "Section 3.2 on preference modeling does not accurately represent the existing literature. The standard approach to training a reward model involves adding a randomly initialized linear head that outputs a scalar reward for each response. The model’s parameters are then optimized by maximizing the log-likelihood of the preference data under the Bradley-Terry model.\n\nIn particular, lines 188–190 suggest that reward models must use templates and input both prompts into a language model to determine preference. However, this is not a comprehensive view of how reward models are typically trained.\n\nreferences: [learning to summarize from human feedback, Stiennon et al.](https://arxiv.org/pdf/2009.01325)\n\n[Evaluating Reward Models for Language Modeling, Lambert et al.](https://arxiv.org/pdf/2403.13787)", "other_strengths_and_weaknesses": "To the best of my knowledge the specific model proposed here for preference modeling is novel. Although there are also other instances of embedding responses in the latent space; see [Chen et al.](https://openreview.net/pdf?id=qfhBieX3jv)", "other_comments_or_suggestions": "Some of the notation is a bit confusing. For example, the number of responses per prompt is said to be $K$ and the embedding dimension is $2k$, but probably there is not need for these two numbers to be related, right? and then the complexity order should be $O(Kd)$, where $d$  is the dimension of the preference embeddings.\n\nSimilarly, in theorem 2, do we really need to match the dimension of number of responses with the dimension of embeddings? the equation doesn’t really necessitate such a thing, so that is really confusing.", "questions_for_authors": "The discussion on “Automatic Subspace Discovery” (line 280) is theoretically sound and quite interesting. However, it would be valuable to see empirical evidence supporting this idea—specifically, whether there are instances of interpretable directions or meaningful eigenvalues in practice.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a generalization of Bradley-Terry Reward Models, in particular aiming at being able to handle intransitive human preferences, by embedding responses in a latent space. Empirical results on both synthetic data and common benchmarks show (1) higher accuracy of the resulting Reward Models and (2) higher quality of responses from models trained with them.

After the discussion period, all reviewers are leaning towards acceptance, praising in particular the novelty of the approach and the new light that it may shed on reward modeling (where BT is still king). However, several reviewers pointed out clarity issues, and questioned the practical relevance of the proposed method in real-world settings, where the prevalence of intransitive preferences is not obvious. Although authors provided clarifications as well as additional observations related to such intransitive preferences, I agree those are valid concerns that may limit the reach of this work.

In the end though, I am also leaning towards acceptance as I believe this novel take on preference modeling may inspire future developments to build more expressive reward models. I strongly encourage the authors to address reviewers' concerns regarding clarity in future revisions of their work.
Decision: Accept (poster)