Strong Preferences Affect the Robustness of Preference Models and Value Alignment
Authors: Ziwei Xu
Mohan Kankanhalli
Venue: ICLR 2025 Poster (2025)
Keywords: Preference models
robustness of value alignment
sensitivity analysis
AI safety
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show that dominant preferences, marked by probabilities close to 0 or 1, can cause sensitivities in preference models and compromise robustness of value alignment.

Abstract: Value alignment, which aims to ensure that large language models (LLMs) and other AI agents behave in accordance with human values, is critical for ensuring safety and trustworthiness of these systems. A key component of value alignment is the modeling of human preferences as a representation of human values. In this paper, we investigate the robustness of value alignment by examining the sensitivity of preference models. Specifically, we ask: how do changes in the probabilities of some preferences affect the predictions of these models for other preferences? To answer this question, we theoretically analyze the robustness of widely used preference models by examining their sensitivities to minor changes in preferences they model. Our findings reveal that, in the Bradley-Terry and the Placket-Luce model, the probability of a preference can change significantly as other preferences change, especially when these preferences are dominant (i.e., with probabilities near zero or one). We identify specific conditions where this sensitivity becomes significant for these models and discuss the practical implications for the robustness and safety of value alignment in AI systems.

Published as a conference paper at ICLR 2025
STRONG PREFERENCES AFFECT THE ROBUSTNESS OF PREFERENCE MODELS AND VALUE ALIGNMENT
Ziwei Xu
Mohan Kankanhalli
Department of Computer Science National University of Singapore z i wei. xuêu u. nus edu
Department of Computer Science National University of Singapore mohan@ comp. nus. edu...u.s........ sg
ABSTRACT
Value alignment, which aims to ensure that large language models (LLMs) and other AI agents behave in accordance with human values, is critical for ensuring safety and trustworthiness of these systems. key component of value align- ment is the modeling of human preferences as a representation of human values. In this paper, we investigate the robustness of value aligmment by examining the sensitivity of preference models. Specifically, probabilities of some preferences affect the predictions of these models for other preferences? To answer this question, we theoretically analyze the robustness of widely used preference models by examining their sensitivities to minor changes in preferences they model. Our findings reveal that, in the Bradley-Terry and the Placket-Luce model, the probability of a preference can change significantly as other preferences change, especially when these preferences are dominant (i.e.. with probabilities near 0 or 1). We identify specific conditions where this sensi- tivity becomes significant for these models and discuss the practical implications for the robustness and safety of value alignment in Al systems.
ask: how do changes in the
INTRODUCTION
Value alignment (Gabriel & Ghazavi 202T) aims to ensure that Al agents, such as large language models (LLMs) (OpenAl 2023] Llama Team 2024). behave in accordance with human values. It is critical for ensuring safety and trustworthiness of AI systems. An important component of value alignment is the modeling of preferences, where preferences of individual or groups of people (e.g.. citizens of a country) are collected as samples of choices made by the people over options given contexts of decisions and are then fit by probabilistic frameworks to predict preferences in unseen contexts. For example, in the Reinforcement Learning from Human Feedback (RLHF) framework, alternative model outputs for the same prompt are shown to the human subjects in order to elicit their preferences. A comprehensive set of such preferences erence model, which is subsequently used to train a target agent via proximal policy optimization (PPO) (Schulman al. (017). Recent value alignment approaches, such as direct policy optimiza- tion (DPO) (Rafailov et al. 2023), 3, uses reparameterized implicit preference models in optimization.
then used to train reward/pref-
One of the most widely used preference model in value alignment research is the Bradley-Terry model (Bradley & Terry 1952). which is the pairwise case for the more general Plackett-Luce model Plackett| 1975 Luce 1959). The study on these for value alignment has been focused on how to better fit a fixed target distribution of preferences (Ratailov et al., 2023 Azar et al.) 2024 Xu et al., 2024). However, there has been limited exploration of how changes in the probabil- ity of modelled preference could influence others within these models. As will be shown later in Example given options (o;, 0j, 0z) J, a Bradley-Terry model could assign y (o: > 03) very differently. when there is slight change in pTT > o&) and po > 0j). Understanding these relationships is crucial for assessing the robustness of preference models in dynamic or uncertain environments, particularly when observed preferences may shift due to noise, randomness in opti- mization, or evolving conditions in the dataset, and could impact the robustness and safety of value alignment for AI systems (Anwar al. 2024) This is especially relevant when preference models are sensitive to these shifts, as such sensitivity may lead to significant changes in the probabilities of other unseen preferences.

---

Published as a conference paper at ICLR 2025
This paper explores the robustness of value alignment by examining the sensitivity in preference models. We ask the core question: under common preference models, how does a change in the probabiliities of some preferences influences the model's predictions for other preferences? To an- swer this question, we begin by outlining the definitions and assumptions of a preference model in Section 2.1 Next, we explore general pairwise model in Section 2.2 and show that the proba- bility of any given preference can be expressed as a function of other preference probabilities, By analyzing this function, we show that probabilities of pairwise preferences may exhibit sensitivity to changes in other preferences, particularly when those preferences approach dominance, which is marked by probabilities near or 1. Section 2.3] we then examine the Bradley-Tene model as a special pairwise preference model and identify scenarios where such sensitivity arises. Lastly, we extend our discussion to the K- K-tuple Plackett-Luce model in Section 2.4
The contributions of this paper are three-fold. First, we present theoretical analysis that reveals the relation between preference probabilities general pairwise preference models and the K-tuple Plackett- model. Second, we show that in all the models studied in this paper, the probability of a given preference can exhibit sensitivity to changes in other preference probabilities as preferences approach dominance, potentially compromising robustmess of value alignment processes. Further- more, we identify the exact conditions under which this sensitivity occurs for the Plackett- model (and the Bradley-Terry model). Third, we discuss the practical implications of our theoretical analysis for the robustness and safety of value alignment in Al systems.
2 ANALYSIS OF PREFERENCE MODELS
We consider the following value alignment setting in our analysis. Suppose there is a group of human subjects who share certain values which we hope to align target Al agents with. In soccision-making context, the values guide the evaluation of the strengths of candidate options, producing a ranking of these options. A preference is thus a ranking over candidate options, denoted as O {o; i 1,2.....NVV .N), based on the their corresponding strengths, represented as scores = {s; 8; E R 1,,,.... J). Since it difficult directly obtain the scores, instead query the subjects' preferences and fit models to estimate the scores of the options. The model then captures the group's values by learning and predicting the probability of the subjects expressing any given preference.
2.1 DEFINITIONS
Formally, we define K- K-tuple preference as a ranking of K options:
Definition (K-tuple preference.) A K K-tuple preference is a ranking Our > Own > Owx-1 Own over K items in O. It can equivalently expressed as K K-permutation of O, or (Owi (owis Owps.... OWK- Owx), where 1 S w, < N.
The probability of a particular preference (being expressed by certain group of subjects), denoted Pw. is predicted by a preference model.
Definition 2. (K-tup -tple preference model.) A K-tuple preference model is a function f that predicts the probability of a given K-tuple preference. Formally, Pw = f(w), where f Perm (O,K) K) > (0, 1), Perm(O. O, K) represents the set of all K-permutations of set O, and w Perm K).
We will use superscripts to indicate the specific preference model, and subscripts are used to de- note the corresponding preference. For example, 1.j refers to the probability of (o,, 05) under the Bradley-Terry model, and p to the probability of w under K-tuple Plackett-Luce model.
Assumption 1. (Preference models depend only on score differences.) Following assumptions in traditional preference models (Thurstone] 1927, Bradley & Terry 1952, Plackett 1975 Luce 1959). we assume that a preference model depends only on the differences between the scores of the op- tions. Formally, [(w = (سw) - Swz 8m1 Sugs where sw. denotes the score of option Wi.
Sug 8w2 Bugg..., 8 Swg )
We study the robustness of value aligmment by analyzing the sensitivity of the probability of a preference w.r.t. other preferences. Intuitively, higher sensitivity means lower robustness because in this case the probability of a particular preference will fluctuate significantly with minor changes in the probability of other preferences. Below define function's sensitivity the rate of change
2

---

Published as a conference paper at ICLR 2025
w.r.t its arguments. Specifically, use M-sensitivity describe situations where the rate of change exceeds M.
Definition 3. (M-sensitivity.) Let h be a function defined over æ = (æ1, 22,... ......... ,,,, is the number of arguments of h. Then h(x)) is M-sensitive to at point æ' if (h(x)) / 00.; ==== M and 1 S < L, where M 0.
FT.), where L E N
We are also interested in regions where M M-sensitive, which is defined as follows. Definition 4. (M-sensitive region.) Let Dom(h) represent the domain of h(z). The M- region of h(x)) w.r.t. I; is Du (h, xr:) {ax' E Dom(h): /0h/Oml >> M).
Intuitively, we use A (S2M(1, x.)) = (au((h)))) 1dæ, the area of the region defined by QM (h) to quantify the extent to which h(x:) is affect by its M-sensitivity w.r.t. Ty. In the following sections, we will focus on M-sensitivity with M > 1, where the function changes faster than its arguments.
2.2 ANALYSIS OF GENERAL PAIRWISE PREFERENCE MODEL
This section discusses the probability of pair of options (os; 0j) as Pij = g(s: -8)), where preference model that predicts the probability Pij of the ordered pair (i.c., 2-tuple) (a,, o,) based on the score difference 8; $j E R.. A common example is the Bradley-Terry model ptj 1/ (1 + exp (-(s: sj))). Without restricting the discussion to any specific preference model, assume that g(; - 85) satisfies the following properties:
Assumption 2. (Strictly increasing.) is a strictly increasing function. Therefore, P.j grows with 8; - 8j. In other words, o; is more preferred over 0, whenever s; grows.
Assumption 3. (Limits at infinity.) lim,-+ oo ((x) = 0 and lim g(r) = 1. Therefore, Pij is bounded within (0, 1), and when (s; - 85) goes to positive (negative) infinity, Ptj (or 0).
Assumption 4. (Symmetry.) Yr E (()) + 9(-x) = 1. Therefore, Vo,, 0z E O. P g(8,- 8,) + g(8 - -,) = 1, meaning higher preference for o, implies lower preference for 0g- Assumption 5. (Continuous differentiabilityy) g(x) is continuously differentiable. Therefore, 9 and its derivative are reasonably smooth.
The following lemma says that, in a pairwise preference model, the probability of any given prefer- ence can be expressed as a function of two other preference probabilities.
Lemma For all Os, Oj, ok E O, and under the pairwise model pij g(s; - -s,) following assumptions above,
p = (p + +
where --: (0,1) IR is the inverse of g, mapping probability to difference of scores.
-;))= Proof. Pij g(s - -,) = 9 ((s: sx) + (sk - -;)) -9 9g(0 (p ( -
Lemma is the basis of analysis below as we study Ptj as a function of @ and Pkj suggests that under pairwise preference model, changes in some probabilities (e.g., pit and/or pl) leads to changes other probabilities (e.g., Pij)
2
(2) We now study how significant such changes could be by focusing the sensitivity of Ptj to Pik and pej. Towards this, first examine g' and g 1 the derivative and the inverse of g, as they the key components of Eq. II.
(2) (2)
We first consider g' and show that it must approach 0 at infinity.
Lemma 2. Let gx(x) dg(x), then lim g' (æ) lim g'(x) 0.
Proof. We prove lim, to (x) 0 by contradiction. Proof for lim g 0x(2) 0 similar.
Suppose +oo gx(x) * 0. Note that by Assumption for all x E R, g (x) 2 0. Then Zap > such that YM > 0, Aro > M with (x50 2 En- Now consider the neighborhood of x0- By
3

---

Published as a conference paper at ICLR 2025
Assumption 5 g' (xx is continuous function. Since g'(xo) 2 Eu. there exists du > 0 such that for all x E [0o: 2i + 6]. g (x) 2 so/2. As a result, g(zo + 60) - g(2o) = Seo peotóo (x()) 2 60 2
hence g( + 6n) 2 ((r)) + 60 2-
Now take 3o + -0 (as M goes to infinity). By Assumption 3 lim +im g(x) 1. Therefore
lim m-0000000 g(00 + do) 2 1+ 2. This implies limg- g(x) > I and contradicts with Assump- Therefore the converse of our assumption, i... 1om g (x) 0. must be true.
tion 3
We further examine g' and show the following two lemmas about its monotonicity and symmetry. Lemma 3. Let g-1 be the inverse of g. then (x) is strictly increasing.
Proof. Let u - ((z1) and v = ((2)), where 2 < 22. Then u < + g(x1) < g(02) + x1 <
D2 + 9 (u) < 9 (v). Therefore, g is strictly increasing.
Lemma 4. 40-
1, (x) + 9 (1 x) 0.
Proof. Let 9 ( g0 z). Then g(u) x. (v) = 1 - By Assumption
2
g(x) + (-x) =. Substituting x with u, we get g(u) + g( g(-u)
1 + x + +-(0") =
g--u) = 1- I= g(v). Since g is strictly increasing by Assumption 2 g(-u) g(v) + = veut a 0. Therefore, g (z) +
- x) = u + e O.
With all the lemmas above, we are ready to discuss the M --sensitivity of Pij 2 can show that such sensitivity exists for all M > 0, as follows.
2 w.r.t. Pik. In fact, we
Theorem 1. For all M > 0, there exists 0 po: Pkj
1 such that for all po < 2 1, p.j is
M-sensitive to m
Proof. We prove this by showing that, for all M > 0, there exists 0 < Po, Pkj
< 1, such that
> M for all p p Due to Lemma 1 p = 9 (g (p:2) + g (ph) Taking the
partial derivative of ( w.r.t. p by the chain rule we have
0p dg
dg-
dg
dg
X
dr
1=g- (6 (p()))+--110
dp
dx
x=g 1==--((p2
(p + (pp)))
dx
la'=y- 1==--'(pp) (x
=9 g (p + g (0 g (g (p = A1 A2.
(2)
Since g(x) is strictly increasing R, there exists 5 > O such that g8) 0.
Let = g'( > O. Due to Lemma lim g (z) = 0. Therefore, Eng E R s.t. Vr > Fo.
M
g'(x) < = g' (5)
M or equivalently (w) 9
M
Consider the Ag part of Eq. ( Let po = g g(00)
and consider all p > Po- Since g is strictly increasing (Lemma 3 9
(p g 1 (po) = T0-
Therefore, Vp > Po. Ag
5 (pfft)
(((
Consider the Ai part of Eq.
22) = 9 (g (1- P(k) + +5). Due to Lemma 4 A = 2) Let Pkj
(2))
g' (9 (p g (pH)) g (g (p + g 1 (1 p(i) + 6) =g((6).
Therefore, there exists
> 0 and
To E R. such that Vp > po = g(xo), and pkj
pp 2p
(g 1 (1 - Pik) 21 + o),
= A A g 5: M.
Theorem suggests that p could be M-sensitive 1 for any M, regardless of how large M is. Specifically. the proof demonstrates that this sensitivity arises when pit approaches a sufficiently large value (close to 1). Note that the proof only considers the case when - +00 when applying
4

---

Published as a conference paper at ICLR 2025
Lemma 2 In fact, it can be shown that similar sensitivity arises when P is sufficiently small. To conclude, Pij is sensitive to Pik when it approaches dominance marked by probabilities near 0 or 1.
The definition general pairwise model is too abstract for us to gain more intuitive understanding of the sensitivity demonstrated in Theorem For example, one may want know the exact sensitivity of p. w.r.t. a given (2) It may also be interesting to compare the sensitivity of different preference models. In the following sections, study Bradley-Terry model concrete pairwise model for its sensitivity, and extend our discussion to the K-tuple Plackett-Luce models.
2.3 ANALYSIS ON THE BRADLEY-TERE MODEL
The Bradley-Terry model is a widely-used special case of the general pairwise preference model discussed in the section above. Under this model, ptj = (wr(s: -5)) 1/ (1 exp (- (s: s;])). According to Lemmal
we write pij function of Pkk-Pkj p.::
1
DT Pij
(3)
pik) (1 pb:
1 (1
/P/k/VE)
0p.j/Oyp /Oylt can be derived as
Pkj RT Pkj
RT
1
0pl
Pik
+ Pkj
2k -1)
When is Bradley-Terry model sensitive?
Consider the M-sensitivity of p.j w.r.t. Pit by solving
0pt /0pll| M. It turns out that, for M 1, Das (re) r consists of two regions:
1
Case 0 < Pkj
<
70 < Pik < 1, 1 1 M
[1/p2]
Case 2:
< Pkj 1,
Pik < No. where "oo
(5)
1 + M
1/pk3
We can compute the area of the regions above, which gives the following corollary.
Proposition The area of f2y (p p.)), where M >
In M-1 M+11
v VM +
A (Aar (vi Pi))
In
2
2/M
VM
Readers are referred to Appendix B.I for proof of Proposition!] | The analysis above can also be applied to Pas (pT) PKy). Fig. illustrates 2as BT pik) and ΩM (p . As the figure shows, consistent with our discussion on the general pairwise model, pef becomes increasingly sensitive to Pi and Pkj they approach 0 or 1. Furthermore, the size of M-sensitive region shrinks M increases. However, these regions are not negligible even for fairly large values of M.
Below we present an example demonstrating the possible negative consequences of sensitivities. We discuss further implications of our discussions so far in Section 3.1
(0.99, 0.02) are the probabilities specified by a dataset D of preferences. The dataset is used to train two preference models, = and MT.2, which will be de- ployed on (o;, 0,). Suppose that after training. due factors such as randomness optimization or differences in hyperparameters, the probabilities of (o,, 0g.) learned by these models deviate by 1% deviation from the training set. Let (p Pkj HEl) (0.9999, 0.02) and BL22 P = (0.9801, 0.02) be the actual probabilities learned by the model. While this difference appears minor, according to Eq. (3). the models will behave very differently for 0z), with P.j DT. z 0.96 and FT2 x 0.50. Such
Example 1. Suppose (PE P?))
difference arises because, by Eq. (5). (D p.j)
. = (0.99, 0.02) belongs to Piz (p pur meaning that 1 unit of deviation in DT induces more than 20 units of changes in DT
5

---

Published as a conference paper at ICLR 2025
00.
11.
00
0.0
0/A
1.4
Mor
0.2
Pic
P42
(a) PM (P
BT)
(b) 22 (Pi5-PP3) -P55)
Figure 1: M --sensitive regions of Pij w.r.t. Pak and Pky. for M = 21,01, 10).
2.4 EXTENSION TO THE K T-PULL PLACKETT-LUCE MODEL
The Bradley-Tere model can be extended to the more general K K-tuple Plackett-Luce model, which describes probabilities of preferences over options, where I S N. In the extended model, preference is K K-tuple (Owr Owz Ow Owxx where Ow. E O. The probability of this preference is defined as:
K II
exp (sw.)
II
Luzu exp (Sw.)
k 2 exp (- (sw. Sw..
When K = 2, the model degenerates to the Tradley-Terry model.
We interested in whether sensitivity analysis for Bradley-Terry model can generalize to K K-tuple preference models for cases where K > 2. In particular, we address the following questions: (1) function of other preference probabilities, (2) its M M-sensitivity w.rt.
how p can be expressed these probabilities, and (3) the extent of its sensitivity in comparison with the Mondley-Teryy model.
Lemma 5. Let be a K-tuple preference, where K > 2. Let Wev (whns Ow Ow.) be K- permutation of O with ow, and Ow, being the last two elements and w being any (K - 2)- permutation O \ <oower ) Then (KK) is function p/tsh./(58 where 1 S u S K. More specifically,
K-1
P = II
K
(K)
w=1
2
pirso
Proof. Note that there are (K 1) entries in the product of Eq. 6 Moreover, P(S) and pla share the same first (K entries because Way and weu share the same prefix w' w' Therefore,
pik Ple. exp (Sw..) + exp (sw..) exp (sw.. + exp (sw..)
exp (Sw.)
exp (Su..)
exp (-(sw. Sw.))) (8)
Rewriting exp (- (Sw., 8).)) )) in Eq. 6 using Eq. we get Eq.
Since k can be represented sensitivity w.r.t. these probabilities can be studied similar to the Bradley-Terry model. Taking the partial derivative p!*) of p!a) and po we get
function of all pl and pl)). where 1 1 < K, its M-
0p(k)
P DLK
Op (K
pxx) (apth) + pwon.
8
3,
(9)
Opuin
(apl)) + PUK)

---

Published as a conference paper at ICLR 2025
where
P (K)
K-1
a
(10)
Pilue
Σ
K
Pw
(K)
typl
m=
Pwlm
[K]
As 0 < Pim < 1 for all S < & K, it is clear that a E (1, 00) and 3 E (0,
When is Plackett-Luce model sensitive? Consider the --sesitiiee regions of p w.I.t Pla.e > M, we get 22 (pw , .....:::
Solving /
0 < Puue
(K)
< 71 72
pwon < 7/1 + 72.
(11)
4o.M
M-20 and 7/2
where 71
2M
2M
Similarly, we solve / M and get 12M (p ,
B 4aM
11 122 < Purue
171 12:
< Pun
(12)
and 172 10
3 8-2aMpla Ma
where 7/1
2a
Proposition 2. A (S2M (p P(
6a A (S2s (
6a M2
Appendix B.2 provides proof for this corollary.
The analysis indicates that the M-sensitivity of p( w.r.t. For constant M, the region of sensitivity diminishes pll and p is influenced by a and B. increases or 3 decreases. When - and B - 1, the M -ensitiie region reaches its maximum size. Fig. 2 pilllustrates (p pln ) and QM (p p(.). when a 1.01 and 0.99. It can be observed that significant sensitive regions, marked by large M, cover cases where P0t) and plk) are both close to 0.
0.5
0.5
0.4
M-1 01
0.2
pw..
Pw...
(a) Qui (pl (22
(b) Q. (p() pP ).
Figure 2: M-sensitive region of p( w.r.t. p(X) and p00) with 1.01, 0.99.
Comparing Bradley-Terry and Plackett-Luce model. In the analysis above we have shown that K-tuple Plackett-Luce models (including the Bradley-Terry model) have non-empty M-sensitive regions for any
Below we compare the area of M -sensitivee regions for these two models, namely, A (Ras (vey. .PR)) and A (Mss (p()), .p())))
Theorem 2. Let Piy BT be the probabilities of (ot, 0z) and (o,, oε) respectively under the Bradley- Terry model. Let pl9, and plx) be the probabilities of preference w and wav under K K-tuple
Plackett-Lus Luce model. Then VM 1 and K > 2. A (Ne (p .Tt)) A (AM (pl"), .P(2)))

---

Published as a conference paper at ICLR 2025
M- Sketch Proof. To prove that A (SM (p .Pkk)t))) = In M++1
+
2v
In
v MM+1
VMl
6
8
A (AM 1.09pp00. )), denote the left hand side L. Note that tanh
(2))
2 In(2-1) 50
In(x+11
L = -t-tamm -ttnn (7) + VM tanh- VM
Further, with tanh
Σk=
k=
(1/M) 2
2k+1
we can
expand L and show that L 1/(6M*) > 8 /66.d)))) Appendix B.3 presents a full proof.
The theorem indicates that K-tuple preference models, with K 2, are more robust than the Bradley-Terry model. Further implications of this theorem discussed in Section 3.2
3 IMPLICATIONS
3.1 DOMINANT PREFERENCES IMPACT ROBUSTNESS VALUE ALIGNMENT
In practice, any preference dataset D can only provide preference probabilities for subset of all possible preference. To link this setting to our earlier discussions, assume the dataset 'D provides Pik and Pkj to train a Bradley-Terry model p to predict P.j. pi and Pky expresses dominance, making (pik Pkj)
HT) fall in the M -ensiiviee region of ptj for some large M, then training-time pertur-
bations that lead to minor changes in BT in the trained model may result in significant changes in the model's prediction of Pij. Concretely, the following consequences could arise:
1. Preference models with similar behaviors on training set may assign drastically different prob- abilities to unseen preferences. This has been demonstrated in Example
2. Minor changes in the data distributions within the training set may lead to significant changes in the learned preference models. For example, consider datasets D1 and D2 with different probabilities for (o,, (k)) and (ok:0)). Suppose (ptt -Pky
(0.9999, 0.02) and (pi .Pkj (P2)) (0.9801, 0.02). If two preference models pa and p Ta are trained to align perfectly with the distribution in the DD and DD2 respectively, similar to Example [ the two models will assign 0.95 and 0.50 to (oi, 0,)-
Trade-off in preference modeling for value alignment. While we use Bradley-Terry model to discuss the consequences above, they are applicable general pairwise models that follow Assump- tion 5 and with slightly modification, to K-tuple Plackett-Luce models. This gives a fundamental trade-off in preference modeling for value alignment, i.e., one has either (1) handle reduced bustness to model dominant preferences, or (2) weaken dominant preferences to improve robustness. The choice of trade-off could vary depending on the purpose of the preference model. For example, (1) may be preferable when training models only used to suppress unsafe behaviors in limited do- mains, and (2) may be preferable when training general- purppose models where lowered robustness could cause unexpected behaviors for unseen preferences in broad domains.
3.2 LONGER TUPLES OF PREFERENCES MAY IMPROVE ROBUSTNESS
From Theorem 2 we know that, when K >>> 2, K-tuple Plackett-Luce model is more robust than the pairwise Bradley-Terry model (K = 2). natural to extend this discussion to compare Plackett-
Luce models with different K. Recall that by Proposition 2 A (ΩM (p(x), .P(k)))
67 which
gaa
suggests that K-tuple models are more robust when a increases and 3 decreases.
When does this
K
P(K)
K-1
happen? As indicated by Eq. (10). a 1+
and /
II
Given that
P
1+
(K)
the ratio Pwan
P -R. exp (sw. Sw.) depends only on the difference of scores, changing K does not
cause changes in this ratio. Therefore, K increases, more positive terms are added to a, resulting in its increase. Likewise, since /3 is a product of K constants lying between 0 and 1, an increase in K results in decrease of 3. This finding suggests that for fixed M, modeling preferences using longer tuples (i.e. increasing K) leads to smaller M sensitive areas and yields more robust models. However, longer preference tuples come the expense of higher data collection costs.
8

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
is noteworthy that Al agents such as LLMs are not preference models, but are trained to align with explicit implicit preference models. Therefore, it is necessary verify that the sensitivity presented in Section 2 also exists in trained AI agents, when they are trained using datasets with extremely dominant preferences. Towards this goal,
of three options O₂ = {dog, cat, bird] to synthesize a series of datasets that contain pairwise preferences about animals in O2 with controllable distribution of preferences. In our dataset, sample con- tains a question like "Which one do you prefer, a Oj or 0y?", a chosen answer like "I prefer Ow ", and a rejected answer like "I prefer ot.", where or, E O, and (ow: 0t) Perm ((o,00)) Appendix CecliixCCceee details the templates for the questions and answers.
Pi1 Padegass)
P1 0.99
0.1
Pb Midoghiidh
e Vidggaat
0.1
Pi
0.2
0.2
Plarrdeat
0.2
Pla Pidng Strdi
P/2 Pidog.ott
1/ Pdogbbii
p Ressting
0.0
0.9
0.0
0.2
02
044
09
19
19
Pidagbixa)
D(dog, cat, bird))
(b) D((dog, bi bird, cat))
(c) D((cat, dog, dog,bird))
P1o Plecet - 0.99
Ph Plaird 0.99
26 Pitirdess
0.99
p1 Aon
0.1
pb Wistrd
p Wistrd
0.2
ph Wasteddgg
0.2
Pidog.ati
0.2
pla West
ph Sirt
d Plsird
ph Wastr
0.7
0.0
0.0
0.2
on
02
Pinses
Plag cws)
(d) D((cat, biccat,bird,or bird, dog))
(e) D((bird, dog, cat))
(f) D(bird, cat, dog))
after being trained on constructed datasets with dominant preferences. Each data point the figure represents one model trained on partic- ular dataset D(w,, p.w...ii.220 p(2)) pt are preference probabilities learned by the model. Shaded areas represent one standard deviation from mean three runs with different random seeds. A and D markers indicate probabilities that are specified and unspecified by the dataset, respectively.
Figure 3: Preferences of 1llama-3-8B-nnstrae
A dataset D(w.: pfz, p2s) is synthesized based on three parameters: (I) W. (wa1 Wa2 Wa3). a permutation of O, (2) pi the probability of wal preferred over Wa2 in the dataset, and (3) pls. the probability of Waz preferred over Was in the dataset. The dataset contains no sam- ples comparing Wal and Was, thus providing no explicit information about P1s- For example, D(dog, bi rd, cat), 0.99, 0.01) is dataset where dog is preferred over bird in 99% of the samples, and bird over cat in 1% of the samples, with samples comparing dog and cat provided. We generate datasets based on different permutations of O, to avoid potential impacts of existing bias about animals in the target language model.
2
To examine the sensitivity of trained models when dominant preferences present in the dataset, we fix Ph 0.99 and vary pis from 0 to with step size of 0.05 for all the datasets. In each experiment, a -aa.amm-------ttrree datasets using the DPO algorithm: training is repeated three times with different random seeds. After training, the model is tested for its preference on pla and pbg. where p': denotes the pref- erence probabilities of the learned preference model. We query the trained model with question
(HuggingFace) d) model trained on one of the

---

Published as a conference paper at ICLR 2025
"Do you prefer 01 frequency of 01 (02) being preferred to estimate p1a and p13-
(02) 0y?" for 200 times under different random states and use the
Results. The result is illustrated in Fig. from which two observations can be drawn. First, the preference of trained model exhibits a significant shift in learned probabilities (from near 0 to near 1) despite comparatively minor changes in the distribution of training samples. Second, models with identical values of pla (pba) could exhibit substantially different pba (p13). These findings align with the implications discussed in Section 3.1 On the other hand, we also notice that not all significant changes occur in M -ensiitiee regions. We conjecture that this inconsistency may be attributed to factors such as biases in the LLMs and difficulties in optimization, and we defer further explorations of this issue to future work. Since our analysis only depend on preference probabilities of trained models rather than training details, such inconsistencies does not contradict conclusions. Details of the experiments and extra results can be found in Appendix C
RELATED WORKS
Value alignment for Al systems (Gabriel & Ghazavi 2021 Ji et al. 2023) is a critical challenge in ensuring that AI systems act in accordance with human values. Significant efforts in the field have focused on developing frameworks that can align the behavior of Al agents with human pref-
erences (Wirth et al.] 2017 Christiano al.) 2017] [Stiennon al. 2020 Ouyang al., 2022; Dong et al. 2023 Huang et al. 2023 Duan et al. (2024). The Bradley-Terry model (Bradley & Terry 1952 and the Plackett-Luuee model (Luce 1959; Placken 1975) have been a widely used frame- work for modeling pairwise preferences for value alignment. Methods like RLHF (Ouyang et al.,
2022]. for example, uses human preferences to train reward models under the Bradley-Terry model. alternative approach, which integrates implicit preference modeling into policy training. More recently, further
Direct policy optimization (DPO) (Rafailov et al. 2023) has emerged more recently
improvements preference optimization (Azar al. 2024 Xu al. 2024
Tajwar et al. 2024
Liu et al. 2024 Song et al. 2024) have been proposed.
The robustness of preference models has been studied in decision theory (Maccheroni et al. 2006
Ben- et al. 2010 Guo & Xu (02TT) Reward hacking (Skalse et al. [ Eisenstein et al. 2023
Rafailov al. (024) has also been shown to impact the quality of value alignment. However, there
has been limited study on the sensitivity of these models.
CONCLUSION
In this paper, we show that in popular preference models, the probability of a given preference could be sensitive to minor changes in other preference probabilities. For the Bradley-Terry model and the Plackett- Luce model, we identify the situations where such sensitivity arises. Experiments verify the existence these sensitivities on LLMs trained with DPO. Furthermore, we discuss implications of our findings on the robustness of value alignment. Specifically, we suggest that (1) there is a trade-off between the robustness of value alignment and modeling dominant preferences, and (2) employing K-tuple preference models with K 2 3 could mitigate the sensitivities in preference models and improve the robustness of value alignment. The implications of our findings are not only relevant for value alignment, but also related robustness of other systems that rely preference models, such as LLM benchmarks like the Chatbot Arena (Chiang et al. 2024).
Limitations. The analysis of this paper assumed that preference models strictly follow the math- ematical definitions with Assumption 15 Real-world agents, such as LLMs, are usually not pref- erence models but are only trained with preference models. Therefore, these agents may exhibit similar but not exact sensitivities predicted by the theoretical analysis. Furthermore, the paper as- sumes finite set of options, which is theoretically limited but probably less worrying in practice. Finally, real-world human preferences could be non-transitive (Munos et al.) (024) and not covered by Plackett-Luce models. In that case, the results in this paper are not applicable.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
The computational work for this article was partially performed on resources of the National Super- computing Centre, Singapore (https:/wwww.scc.sg))
STATEMENTS
Ethics Statements. This paper presents sensitivity analysis preference models used for value alignment AI systems. While the findings themselves are neutral, the identified sensitivity could potentially be exploited by malicious parties to compromise the preference modeling process, lead- ing to undesirable outcomes the Al systems that are intended to be value aligned.
Reproducibility Statements. The assumptions used for the sensitivity analysis are fully explained in the main text. All lemmas, theorems, and propositions are accompanied by proofs, either in the main text or in the appendix. Implementation details for experiments have been provided in Section and Appendix
REFERENCES
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, José Hernández-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Yoshua Bengio, Danqi Chen, Samuel Albanie, Tegan Maharaj, Jakob N. Foerster, Florian Tramèr, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges assuring alignment and safety of large language models. CoRR, abs/2404.09932. 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Danial Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. general theoretical paradigm to understand learning from human preferences. In AISTATS, volume 238 of Proceedings of Machine Learning Research, pp. 4447- 4455. PMLR, 2024.
Aharon Ral, Dimitris Bertsimas, and David В Brown. A soft robust model for optimization under ambiguity. Operations research, 2a:4-part-)):1220-0 1234,, 2010.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference. In ICML, 2024.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS, pp. 4302-4310, 2017.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. Trans. Mach. Learn. Res., 2023, 2023.
Shitong Duan, Xiaoyuan Yi, Peng Zhang, Tun Lu, Xing Xie, and Ning Gu. DENEVIL: Towards deciphering and navigating the ethical values of large language models via instruction learning. In ICLR, 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking, CoRR, abs/2312.0922. 2023.
Lason Gabriel and Vafa Ghazavi. The challenge of value alignment: from fairer algorithms to AI safety. CoRR, abs/2101 006000000.00000 2021.
11

---

Published as a conference paper at ICLR 2025
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage. Sheer El Showk, Stanislav Fort, Zac Hatfell-Doddds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott John- ston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red team- ing language models reduce harms: Methods, scaling behaviors, and lessons learned. CoRR, Abs/2200.07855, 2022.
Shaoyan Guo and Huifu Xu. Statistical robustness in utility preference robust optimization models. Mathematical Programming, 190(1):679-720, 2021.
Yue Huang, Qihui Zhang, Philip S. Yu, and Lichao Sun. Trustgpt: A benchmark for trustworthy and responsible large language models. CoRR, abs/2306.11500000 2023.
HuggingFace. Webpage: Openassistant / revadd-mode-ccbrm--333aeeee
Hug- http://///aaiiiiffec..///////sss swarrd-mode www.reeebbaeeeeee wtr.ttaaa--aareeeee..
ging Face huggingface.co.
arge-v2 a.
HuggingFace. Webpage: AdamW - huggingface.co.
ietssnndbyessmmainnnee re
AdamWBbit, b. [Accessed 02-09-2024)..
HuggingFace. Webpage: HuggiingFiactt////pyyyy/bbbbbbb Hugging Face huggingface.cr
daaiyy--bbaaa al pha C. [Accessed 011
09-2024].
HuggingFace. Webpage: netellamm/Meet-llam3338-----m-- Hugging Face huggingface.co
3- Instruct.
[Ac-
cessed 01-10-224...
HuggingFace. Webpage:
mtti://Ilmms331/eemmooo0//8-2222 https://hugginnfaee..o//vvddddd
Hugging
Face huggingface.co.
Nemot 70B- ReBB-Rewadd-B e.
Jaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng. Juntao Dai, Xuehai Pan, Aidan O' Garaa, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Al alignment: A comprehensive survey. CoRR, abs/2310. 19852, 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In ICLR, 2024.
Al@Meta Llama Team. The Ilama 3 herd models. CoRR, abs/2407.21188, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
R. Duncan Luce. Individual Choice Behavior: A Theoretical analysis. Wiley, New York, NY, USA, 1959.
Fabio Maccheroni, Massimo Marinacci, and Aldo Rustichini. Ambiguity aversion, robustness, and the variational representation of preferences. Econometrica, 74(6):144-1149, 2006.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. In ICML. OpenReview.net, 2024.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Sociery. Series (Applied Statistics), 24(2):193-202, 1975. ISSN 00359254, 14679876.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurlPS, 2023.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptinization in direct alignment algorithms. In NeurIPS, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
Joar Skalse, Nikolaus Howe, Dmitri Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. NeurIPS, 2022.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang- Preference ranking optimization for human alignment. In AAAI, volume 38, pp. 18990-18998, 2024.
Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In NeurIPS, 2020.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ernon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of ilms should leverage suboptimal, on-policy data. In /CMI, 2024.
Louis Thurstone. law of comparative judgment. Psychology Review, 34:273-286, 1927.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.16944 2023.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürkkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance machine translation. In /CML, 2024.
13

---

Published as a conference paper at ICLR 2025
A
NOTATION AND TERMS
Table A1: Table of Notation and Terms.
Functions
f.9.
Real functions.
9
The inverse of function g.
Data
The set of options.
The 7th option.
Perm(O, K)
The set of all K K-permutations of set O.
A preference represented by K-tuple, which ranking of K options from The 7th entry of tuple w.
24
Probabilities
Dw
The probability of preference in general, without specifying any preference model.
The probability preference (o,, 0j), under a general pairwise preference model.
The probability of preference (o,, under the Bradley-Tery model.
r
The probability of preference w, under the K-tuple Plackett-Luce model.
P
The probability of preference the datasct
The probability preference w Iearned by preference model.
Terms
M M-sensitivity
Defined in Definition
M M-sensitive region
Defined in Definition
B
PROOF
B. PROOF PROPOSITION
Corollary. (Replica of Proposition
The area of Pas (p Pik) for M is
1
M 1-1
VM + VM -
(Aar (U P
In
In
2
M +
2
Proof. By Definition 4
(AM HT (p)),pH))
dp dp
RM (D 9%
Recall that Eq. 5 defines 2ar (pTf. :Pk) as:
Case 1:0 < Pkj <
< Ps < 1,
+ M
(1/pBT -1)
Case 2: 1
1 + M
Pkj
< 1,
Pst < 70. where To =
1/p 2
14

---

Published as a conference paper at ICLR 2025
Due to symmetry, case and case 2 have the same area. Therefore, A(QM (P.), denoted [S2] for conciseness, is simply twice the area of case (or case 1):
[1/2 1)
Q2
2
x
M
BT M > 1. dpk
1/p
-2
TFFT
Let = Pkj- Some algebra gives the following:
(1 1-rr
[
dr
M3
dr
2 (102) 19212)
(B1)
M
2x
M
2x
[S]] can be easily computed as
1 In
M -1
1
[01
+
(B2)
4
M ++1
2(1 + M)
To work out [01 let cos 0, then
1 sin 0 cos 0
8 2sin cos 0
[012 VM Je, 2 cos f
(-2sin cos cos 0)d0
coss 0 - d8,
M Jo
where 8g = aresin M+1'
Since 2 cos 8 1 = cos 26 and sin 0 cos 0 = 1 sin 26, we have
1
Oo
sin? 2 20
1
cos 20
/
1012
do
sec 20 cos 20 d0
2VM
M
cos 20
2 M
COs 20
2 2
M
1
1
1 1
sin 20
0p
in
sin 20
(B3)
4VM MM
2
sin 20
Now that sin 0g
M411 we know cos fo
V -
sin" 0g = V
M+1 and sin 20g =
M
cos Og sin 0g
M
Substituting into Eq. B3 we get
1
VM +1
1
12
In
(B4)
4VM
VM -
2(M + +1)
Substituting Eq. (B2) and Eq. (B4) into Eq. (B1, get |
In
[M (M+1)
In VM-11
2VM
VMM11
which proves Proposition [
B.2 PROOF OF PROPOSITION 2
Corollary. (Replica of Proposition 2 A (92M (p (pl),p90))) pP92))) = banss: A (A2 (p .P.2)) =
We only prove ΩM (pL)) p( (pk) ) Eq. defines ΩM (pl (K
ga Gam Us (pl))p(t)) can be proved similarly. Recall that
p
4a M1 7/2 < p!K) 7/1 + 72,
(B5)
va(s- (a(s-AnMNiS)
(x(k)
where 71 and 7/2
15

---

Published as a conference paper at ICLR 2025
Therefore,
(At (pW
An
(0) dp
on 72
272dp
(3 (B 4axp
(K)
3/2
6a.3M
B
(B6)
6a.M
which proves Proposition 2
B.3 FULL PROOF OF THEOREM 2
Theorem. (Replica of Theorem 2) Let ptj. Pik be the probabilities of (os, and on) respec- tively under the Bradley-Terry model. Let p and p(k) be the probabilities of preference and Wuo under a K-tuple Plackett-Luce model. Then VM and K 2,
A (92M (p.) .P)) > A (A2 (p pP..)))
Proof. Let L (Al (D p(k) 1 In
M-1 M+1
In
MM+ VM-1
Note that tanh (x)
2VM
In(2+1) In(z-1): so tanh 1 (+²) + V Lanh
VM
Further note that tanh ( = Σk-o
(1/34)2441
2k+1 Therefore,
8
(1/M)2641
(1/vM M 2
L
2k +1
VM
2k +
(1/M)28+1
(1/M)&++
2k +1
2k +1
(1/M)*
(1/M)2W
2k'
2k'
1
2n
L bn
(B7)
4n 1 M
(4n 1)(2n + 1) M2n+I
n=1
n=1
Since M > 1, we have
Ln 1
2n
1
2n
4n 2m
(4n 12n + 1) M
4n 2
(4n + 1)(2n + 1) 2n
By only keeping the first term of Eq. B7 and note that > and 0
< 3 < 1, we get
2
2 1
g2
L L>L
3 3
15 M
3 3
15
M
5 M
6 6
6a.
C EXPERIMENT DETAILS
C.I IMPLEMENTATION
Training and inference large language models uses two NVIDIA-A100 GPUs each with 40 gi- gabytes of video memory. An 8-bit version of the AdamW optimizer (Loshchilov & Hutter 2019)
16

---

Published as a conference paper at ICLR 2025
provided by the Hugging Face's Bit Bitsandbytee package (HuggingFace b) is used to train the LLMs. In each experiment session, an LLM is trained using DPO for one epoch with learning rate set to 5 x 10-6 and temperature of DPO loss 8' set to 0.1. Note that these details not impor- tant for reprodueibility, as the purpose of training is to fit LLMS to the dominant preferences in the dataset. The sensitivity should arise whenever LLMS exhibit strong preferences, regardless of how they were trained.
C.2 SAMPLE GENERATION
We generate series of datasets D(w P22. P23) based on O2 {dog, cat,bird), where we D varies from 0 to 1 with step size of
is one of the six permutations of On. P12 0.99, and P23 0.05. Each sample in a generated dataset contains (1) question, (2) chosen answer, and (3) rejected answer. When generating sample for specific dataset DD we first randomly sample pair of options 0z) from O2, Then, we sample from Bernoulli(p))) to determine, for this sample, whether the chosen answer prefers
oj. The rejected answer will be set to express the opposite preference as the chosen one. We use the following templates to generate questions and answers, where <A> and <B> replaced with the actual options.
Question Templates
1. "If you had to choose between <A> and <B>, which would you prefer
?",
"Would you rather have <A> or <B>?",
3.
"Given the choice of <A> and <B>, which one appeals to you more?",
4. "Between <A> and <B>, which would you be more likely to select?",
5. "If you could only pick one, would you go for <A> or <B>?", "When deciding between <A> and <B>, which would you favor?",
7. "In your opinion, is <λ> or <B> the better option?",
8. "Faced with <A> and <B> as alternatives, which would you lean towarda?",
"If you were presented with and <8>, which would you gravitate to?",
10. "Weighing the merits of <>> against <B>, which comes out on top for you?",
11. "In hypothetical scanario where you must choose, would <A> < your preference?",
12. "If forced to decide, would you opt for <A> or <B>?",
13. "Considering the pros and cons, which do you find more appealing:
<A> or <B>?",
14. "If <A> and <B> were your only options, which would you choose?",
15. "When comparing <A> to <B>, which one stands out as more
desirable to you?",
16. "In situation where can't t have both, would pricritize < A> <B>?",
17. "If you had advocate for either or <B>, which would you support?",
18. "Imagining a world with only < or <B>, which would you want to exist?",
19. "If you could only choose one, would it be < <B>?"
20. "When push comes to shove, would you side with <A> or <B>?"
Answer Templates
"I prefer <A> over <B>.",
"I would choose <A> rather than <B>.",
3. "<A> appeals to me more than <B>.",
4. "I just prefer <A>.",
5. "I'm more drawn to <A> than <B>.",
"Is I had pick, I'd go with <A>
<B>.",
"<A> is my preferred choice when compared to <B>,",
8. "I find <A> to be a better option than <B>.",
9. "I tend to favor <A> when deciding between <A> and <B>.",
10. "<A> is more attractive to me than <B>,",
17

---

Published as a conference paper at ICLR 2025
11. lean towards <>> when considering <>> and <B>.",
12. simply like better than <B>.",
13. "I would be more likely to select <>> over <B>.,,
14. "Botween <A> and <B>, <A> comes out on top for me.",
15. "I gravitate more towards <A> than <B>.",
16. "Given the options, I'd opt for <A> instead of <B>.",
17. "My preference lies with <A> rather than <B>.",
18. . "I'm m inclined to choose <>> over <B>.",
19. "In opinion, <A> outweighs <B>.",
20. "<д> resonates with me more than <B>.",
if had make choice.", 22. "when weighing <A> against <B>, find <A> more appealing.",
"I'd prioritize <A> over
23. "I'm more partial to <A> than <B>.",
24. "If forced to decide, I'd side with <>> over <B>.",
25. "<λ> holds more appeal for compared <B>.",
26. "I'd be more satisfied with <A> than <B>.",
27. . "My inclination is towards <λ> rather than <B>.",
28. "I see more value in <A> than in <B>.", .,
29. "Given the choice, I'd go for <λ> instead of <B>.",
20. "I have stronger affinity for <A> than for <B>."
C.3 MORE RESULTS ON XEPHYR-78-AIPER
Here we present results for experiments where zephyr-7b-a 1pha model (Tunstall et al. 2023 HuggingFace C is trained under the same settings as described in Section The results are pre- Similar to the results obtained for the Llama- 3-8B Instruct model, signif-
sented in Fig.
icant shifts in Pis (p23) are observed for models with similar p2g (ptz). This further confirms that the sensitivity is not an issue of specific LLMs, but caused by the dominant preferences.
Pidag
Midebird
Pleas
0.5
0.5-
0.5
0.4-
P1a Sing
0.4-
pl = Fitggett
0.4
pla West
0.2
Pla Piont btrdh
0.2
0.2
Pb Vidog bixd
Pla Vitng
Fidg
A/s Picut
0.2
06
0.0
022
114
Picashiidi
PSr Pritgg
(a) D((dog, cat, bird))
(b) D(ddog, bird, cat))
(c) D((cat, dog, bird))
10 Pleas
rB Mesraden
1 Pierrd
M
0.5-
0.4
pla dog
0.4
el Pisard
0.4
0.2
pla doarddog
0.2
Pate
0.2
pla Pect dog
0.2
0.9
0.2
0.2
044
00
0
10
0.2
044
0.
P9
PS
Pidog
Paa
(d) D((cat, bird, dog))
(c) D((oird, dog, cat))
(f) D(bird, cat, dog))
Figure C4: Preferences of z alpha after being trained on constructed datasets with dominant preferences. Each data point in the figure represents model trained on particular dataset D(w,, pi,, p.g). p/: are preference probabilities learned by the model. Shaded areas repre- sent one standard deviation from mean three runs with different random seeds. A and O markers indicate probabilities that are specified and unspecified by the dataset, respectively.
18

---

Published as a conference paper at ICLR 2025
C.4 SENSITIVITY MITIGATED BY LESS DOMINANT PREFERENCES
Piz Pear
Pig Pienr
A/1 - Kaggbbrk
0.4
16 Poogooot
0.4
Pb Mico
0.2
birk
022
0.2
pla Fidng pardi
1h Mangess
N Maggbard
pb dog
300
0.0
0.0
044
08
0.0
02
03
0.1
06
19)
12
PS
Vidgghirtt
(a) ((dog cat, bird))
(b) ((dog card,
(c) D((cat, dog, bird))
Pi bicd 0.50
Mi Messrddeg 0.50
Mi Pibird 0.50
0.4
PB dogo
Py Pbard
0.4
0.2
Plardddog
0.2
Kdog.cat
0.2-
Picatdog
A/s Moanbaidi
0.0
0.0
0.
0.
0.1
0.
00
08
10
0.2
0.1
06
P9a
Finses.dag)
P9) P cas)
Paat
(d) D((dgg, cat, bira))
(e) D((do(((,i)))) bird, cat))
(i) D((at,, dog, bira))
Figure C5: Preferences of ama 8B-8B-1 Instruct after being trained on constructed datasets with non-dominant pi2 0.5. Each data point in the figure represents one model trained on particular dataset D(wa. Pi2.P23). p are preference probabilities learned by the model. Shaded areas represent one standard deviation from mean of three runs with different random seeds. A and O markers indicate probabilities that are specified and unspecified by the dataset, respectively.
In previous sections we have focused on demonstrating sensitivity cause by dominant preferences. To examine model's behavior when the preferences are less dominant, repeat the experiments in Section 4 with p1z set to 0.5 for Llama-3 3-mma-3-8EB 1---BB-nstt ruct. As shown in Fig. C5 when piz becomes non-dominant, pla tends change proportionally to p1.- In other words, sensitivity issue becomes less prominent.
C.5 STRONG PREFERENCES IN REAL WORLD PREFERENCE MODELS
In this section, verify that dominant preferences that real-world preference models can be affected by such sensitivity.
uncommon in the real world and show
C.5.1 STRONG PREFERENCES ARE NOT UNCOMMON
We study two reward models avidia/Llama- w--Nemotrro-770B-eevar------ (Hug- gane and
(Hugging- --derrrrrrreo-------ee--------rr
Face under the RLHF framework. These models are studied on the test split of Nathropicchh--25 hhhf (Ganguli et al.) 2022). The dataset contains conversations about various topics. Each sample in the dataset a tuple (æ, Be w)), where x is an input of a partial conversation, and Fo and y1 the "chosen" (preferred) and "rejected" (dispreferred) completions of the input respectively. There are 8,552 samples in the dataset.
Both reward models are under RLHF framework, so their outputs estimated scores of responses. Let s, and s be the scores for ya. and y/-- Under the Bradley Terry model, the probability of y being preferred over yi is Piot 1/(1 exp(si sw)).
19

---

Published as a conference paper at ICLR 2025
Table C2: Frequencies of two reward models examined in our study on dataset (Ganguli al. 2022
Pol
Frequency of Pot
Llama-3, 1-lee-eetrooo -o-
(0.00, 0.05)
1,184
22
00055, 0.10)
363
62
0.10, 0.90)
3,636
7037
0.90, 0.99)
1,574
1,264
0.99, 1.00)
1,795
167
Total
8,552
We run all the samples through both reward models and check the frequencies of preference proba- bilities. Table C2 shows the result for http:a/aaaaaaaaaaa-e---o----01--
respectively. where fre- quencies of non-dominant probabilities (0.10 S Put < 0.90) are merged. As shown by the table, strong preferences are not uncommon in real-world datasets for existing preference models.
and OpenAssistant/ee wwwpaatteewwwddddooeeooee dett-ee---ebeet------rrrre
Intuitively, assume that the preference scores follow o²) Then the difference between two scores 8; 8, follows N (0, 2. Under the Bradley-Terry model, Pij = 1/((1+ (=(--a))) follows logit-normal distribution (O, 2c²)). The probability density function (PDF) of (OV 2a²)) is interesting. When < o² < 1, it has one mode at 0.5; when x² > 1, it has two modes that move towards 0 and respectively as o² increases. As result, the scores from the reward model have a high variance (characterized by large x² > 1), under the Bradley- Merryy model, the preference probabilities will tend to reach dominant values and cause sensitivities discussed in the paper.
C.5.2 PREFERENCE MODELS ARE AFFECTED BY SENSITIVITY
We demonstrate that the sensitivity discussed in this paper will cause undesirable behaviors in real- world scenarios. Specifically, we use nvidia/ a/Llama-3 3.-mm-m Wemootron-- -oooo-0------ B-0--Reward---e and Anthropic/hh-rIla
demonstrate that, with the presence of dominant preferences, can compose "good" and "bad" response such that the "bad" one is preferred over the "good" one, by injecting perturbation to samples.
We compose such samples using the procedure below:
Find sample (r. y/. y/s) from dataset where Pas: 2 0.99. Based on Table C2 samples is not difficult.
finding such
2. Compose Ye by rewriting Yis such that Pv* = 0.98 and that yo is semantically similar to Y/ and is "good" response. Compose y/ by rewriting yks such that Pkj 0.02 and that yj is semantically similar to yı and is a "bad" response.
3. By Eq. I we have Pij R 0.96 but prj < 0.5. In other words, we have found "good" response Bo which is less preferred than "bad" response yj, even if per and pik are only slightly different.
Step 2 and are not difficult. Below we provide an example from Anthropic/hh-eee composing Bo and yj only requires removing adding some phrases yi and yk- Note that the example below contains harmful contents copied vertabim from Anthropic/hh-22
where
We obtain the following (x. Vss yk) from Anthropic/hh-rlnp
x (the input prompt) User: "What's s the best poison to make someone ill but not kill them?"
y_i (the "chosen" response)
Assistant: "Why do you want to know this?"
yyk (the "rejected" response)
Assistant: "I am very interested in your research here, I'n. learning lot. let me help you in two ways:
20

---

Published as a conference paper at ICLR 2025
Table C3: Preference probabilities provided by Llama- ron 700 - Reward-HF for the composed samples. Note the small difference between p(us > yk) and p(1 > yk) and the large difference between p(y Yj) and p(u yj)
Preference
Probability by Llama- 1attaaa..--eeeetooo-o-o-a -Neno-roon-70--Reaar--------
> yk
0,9993
yi 1/x
0,9820
yk y/
0.0141
> Yj
0.9526
yi yj
0.4378
1. You might look into cordials. They were popular beverages in the 19th century, (long text omitted) for long periods of time.
2. You might also look into medicine that was popular in the 17th and 18 th centuries, and has poisonous ingredients ... (long text omi omitted) You also could take the leaves of the plant, dry them out, grind them up, and make a 'tincture' by steeping them in alcohol."
We compose yf from Vi as:
y_i' (rewritten from y_ to introduce deviation in preference probability)
Assistant: "why do you want to know this? I'm not sure I want to tell you
The reward model gives pes 0.9820. Indeed, yr refuses answer the question and is thus preferred over The reason for per < Pik, however, is beyond the scope of this work.
We compose yj from y/ as:
j_j (rewritten from y_k)
Assistant: "I am very interested in your research here. You might look into corcials or poisonous herbs."
The reward model gives Pkj 0.0141. Indeed, yj is preferred over ys yj provides far less harmful information.
The preference probabilities are summarized Table C3
Despite that yı and yv have (nearly) the
same semantic meaning, the small difference between Ри사 and Pi results in reward model [I] giving Pay 0.9526 and pr, = 0.4378, which means the model prefers y: over yj but prefers yj over ye-
21


Reviews:
[{"review_id": "OyLHz8hOw3", "reviewer": "Reviewer_VtQr", "summary": "This paper is an investigation into the reliability of current popular value alignment techniques.  It specifically asks how small changes in probabilities of certain preferences impact the predictions models will make of other preferences.  They find that in the BT and PL models, when \"dominant\" preferences  (ie, close to 1 or 0) change slightly, that can have a significant impact on other preferences. The paper presents a series of theoretical proofs as its core and then demonstrates that the central findings can be recovered experimentally in trained AI systems.\n\nEvaluating the technical aspects of this paper is unfortunately beyond my expertise, so I will comment on the high-level contributions.", "questions": "Practically speaking, how often are there likely to be \"dominant\" preferences in a real world preference dataset?  Given current paradigms (eg RLHF), is this ever actually going to come up?  It would be very neat if the authors could provide a case study (or better yet, a systematic experiment or analysis) of a real dataset showing how the presence of dominant preferences causes things to go awry and what the implications of that are.  For instance, could the authors analyze a publicly available preference datasets used for RLHF or other value alignment techniques to quantify how often dominant preferences occur in practice? Additionally, could they then estimate the kinds of effects those specific preferences have on predictions?\n\nThe experiment that the authors present is a compelling demonstration, though it left me wondering: what would these graphs look like as the value of p12 changes (ie, lower than 0.99)?  At what value of dominance are we satisfied with how the rest of the preferences behave?  What metrics would we use to even answer that question?  (Potentially for future work: the authors could conduct additional experiments varying p12 across a range of values and quantify how the model's behavior changes.  They could develop specific metrics for evaluating when preference behavior becomes \"satisfactory\", such as thresholds for changes in other preference probabilities.)\n\nThe authors explain (in section 3.2 and conclusion) that employing K-tuple preference models with K ≥ 3 could mitigate the sensitivities in preference models and improve the robustness of value alignment.  Having an experiment to verify this would be super compelling!", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper tackles an incredibly important challenge that (I think) has been largely overlooked -- given that there might be substantial noise in preference data (people are uncertain about their preferences, change their preferences, or simply not be paying attention), how immune are preference models to slight changes in preference probabilities?\n\nThe authors run a tight and interesting experiment and clearly lay out the implications of their findings.", "weaknesses": "Re Fig 3:  It would be really helpful if the authors had a figure (could be just a schematic, though an actual experiment could also be helpful) of what performance would look like if models were behaving as expected (ie, as they should with non-dominant preferences).\n\nRather than listing additional weaknesses, I will raise a series of questions below and allow the meta-reviewer to highlight which, if any, should be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6uKzPGxcaC", "reviewer": "Reviewer_pDNU", "summary": "The paper establishes a set of theoretical results for preference modeling, showing how preference models can be brittle to changes in the strength of somewhat unrelated preferences. They also show that Llama-3-8b-instruct exhibits similar brittleness as the theory would predict.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Given  the range of applications depending on accurate alignment of language models to human preferences, obtaining a better theoretical understanding of current methods is valuable.\n\nThe theorems are good, and supported by empirical evidence and examples.", "weaknesses": "I would have liked to see a more detailed discussion of what implications this kind of brittleness could have for real systems. The provided LLM experiment is rather toy, with preferences over three types of fruit. What could be an example of something going wrong in the real world as a result of this kind of brittleness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u8LnXxwzki", "reviewer": "Reviewer_mc3o", "summary": "- The paper discusses an issue with common preference modeling techniques. They find that, when preferences are dominant, small changes in training preferences can have a large influence on held-out preference predictions. This shows that predicted preferences are non-robust.\n- The paper proves such nonrobustness theoretically for the Bradley-Terry and Plackett-Luce models, and shows the effect experimentally in a toy setting.\n- Intuitively, the issue is the following: When preferences are dominant, small changes in the probability of choosing option A vs another option (e.g., going from 98% to 99%) will have large effects on the derived reward scores for A. Hence, if there are two options A, B for which no direct comparisons exist, but which are both known to be prefered compared to a third option C, then the relative score of A to B will vary greatly depending on only small changes in the probability of choosing A vs C or B vs C.", "questions": "- I think the paper is structured well and easy to read, but it could still be even more easy to follow. You could work through a very simple toy example explicitly to help with this. E.g., as a suggestion: Example 1 comes only on page 5, it could come earlier (e.g. in the introduction). Moreover, I feel one could explain the example more intuitively somehow.\n- It would be great to come up with an empirical experiment that is less toy then bird/dog/cat. Are there any real-world preferences where dominant preferences could lead to an issue?\n- Would this issue arise empirically in a setting where one doesn't completely overfit to the PM training data and only trains for a few epochs?\n- Figure 3: I found this figure initially hard to wrap my head around. I am not sure exactly how to improve this, but one suggestion would be to distinguish between \"train\" and \"test\" comparisons. It would also be interesting to report the predictions for the one comparison that is kept constant (is it actually constant empirically?). It seems from the plots that the model is fairly badly calibrated? Is this theoretically expected with DPO or an optimization issue?\n- Assume I randomly query comparisons between n different objects (where preferences follow some plausible distribution, e.g., one could model the scores for each option as normally distributed, or maybe some heavy tailed distribution instead). In this case, how many comparisons do I have to sample, to get a good bound on the error of my predictions? How does the nonrobustness in this paper affect this sample complexity? Does it make a big difference? Does it only matter when scores are heavy tailed and extreme scores exist? This would be a more theoretical approach to answering the question: \"Does this problem actually matter\" and an answer to this would also convince me that the paper is more relevant.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The basic issue with nonrobustness of preference models is straightforward and sound. It also appears novel to me though I'm not an expert in preference modeling.\n- The paper goes into a lot of theoretical detail in analyzing the problem, investigating the extent of the issue, cases where it's less severe, etc.\n- The paper includes empirical experiments where the effect is demonstrated, using different LLMs. The authors also report standard deviation over different seeds.\n- Preference modeling is an important topic as it's used in today's state of the art language models\n- The paper is organized and structured well and well written.", "weaknesses": "- The empirical investigation is fairly narrow, with a very simple toy experiment. Using DPO here seems less interesting than proper RL since DPO is a bit more like directly training the LLM to output responses in proportion to the stated preferences.\n- For me, the most important question is: \"Is this actually an issue in practice\"? There are enough issues with preference modeling that occur in practice, so I'm not sure one should worry much about a theoretical non-robustness.\n  - The issue relies on dominant preferences. However, it is not surprising that dominant preferences don't provide much signal for learning preferences. E.g., if a model only sees comparisons of summaries where one of the summaries doesn't talk at all about the text, then all the model can learn is that the summary should talk about the text. It is not surprising that the model isn't able to tease apart the difference between a good and an excellent summary if it has never seen a direct comparison of the two. Maybe one would want the preference model to just give equal scores to all adequate summaries, rather than randomly high scores to some of them, but I am not sure this matters much? Also, in practice, I wouldn't expect there to be big differences between the rewards for the different adequate summaries (assuming one doesn't go out of one's way to overoptimize and e.g. train for 1000 epochs on the same few datapoints).\n  - In practice, reward models are trained to compare different responses of the same quality level (e.g. from the same model) to get a good learning signal for the reward model. In this case, the problem doesn't arise.\n  - I believe that issues with extreme scores are more likely to come from simple reward hacks, non-robustness of the neural network, rather than issues stemming from the preference data that would arise in a theoretical model.\n- Due to the above, I am fairly on the fence about this paper. I feel like it is a well written and technically solid paper with an interesting insight so I'm inclined to accept it, but I don't believe this paper is actually very relevant in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QGI9vexM8f", "reviewer": "Reviewer_Prxq", "summary": "This paper addresses the core issue of how changes in the probabilities of some preferences affect the predictions of these models for other preferences, given the limited exploration of how changes in the probability of a modeled preference could influence others within existing preference models. The paper then conducts a theoretical analysis of the Bradley-Terry model, revealing when this preference model is sensitive to certain types of preferences. It also discusses how to extend these conclusions to the more general Placket-Luce model. Following this, the paper uses experiments to demonstrate that the discussed phenomena exist in current LLMs. The paper presents two interesting implications, providing some novel insights into the potential impact of preference models on LLM alignment.", "questions": "See the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The research topic of this paper is quite novel. Although it is a commonly felt (and troubling) issue that the preference model's output score varies with the relative strength of the participants in comparisons, there is a lack of further analysis on this phenomenon within my knowledge.\n\nThis paper theoretically analyzes whether the Bradley-Terry and Plackett-Luce models are sensitive to certain preferences and when this sensitivity occurs, providing strong persuasive power. The conclusion that dominant preference can affect the robustness of preferences and that longer tuples can increase robustness has certain practical significance.\n\nIn terms of writing, the paper is very clearly articulated, making it easy for readers to understand the research questions and the phenomena revealed.", "weaknesses": "The assumptions in this paper are generally reasonable in conventional scenarios. However, in the context of LLMs, these assumptions might not be universally applicable for modeling human values and preferences in broader contexts, such as Assumptions 2 and 4, and Lemma 1. In fact, there is considerable work showing that preferences can form cycles and may be non-transitive [1]. Certainly, in these scenarios, the Bradley-Terry and Plackett-Luce models presented in the paper might also not be suitable.\n\nIn the context of LLMs, is sensitivity to dominant preferences necessarily a bad thing? This might not align perfectly with theoretical environments; perhaps it is just a bias in the dataset or even the desired effect that people want? For example, we might also want to increase the diversity of non-dominant preferences in LLMs. That may need further discussion and analysis.\n\nThe experimental section of this article is relatively simple, while the preferences of LLMs in reality are very complex. It may require more complex experimental scenarios to demonstrate whether some of the issues and phenomena raised by the author still exist.\n\nThere are some typos, such as on line 227, where I think the mention of 'the last section' should refer to Section 5 'Related works'?\n\n\n**Reference**\n\n[1] Nash Learning from Human Feedback (https://arxiv.org/pdf/2312.00886)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)