Generalizing while preserving monotonicity in comparison-based preference learning models
Authors: Julien Fageot
Peva Blanchard
Gilles Bareilles
Lê-Nguyên Hoang
Venue: NeurIPS 2025 poster ()
Keywords: Monotonicity
Preference Learning
Alignement
Bradley-Terry.
Primary area: general_machine_learning
TL;DR: We introduce a class of comparison-based preference learning models that extend (Generalized) Bradley-Terry, can generalize, and have monotonicity guaranties.

Abstract: If you tell a learning model that you prefer an alternative $a$ over another alternative $b$, then you probably expect the model to be *monotone*, that is, the valuation of $a$ increases, and that of $b$ decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are *monotone*. Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited.

Generalizing while preserving monotonicity in comparison-based preference learning models
Julien Fageot Tournesol
Peva Blanchard Kleis Technology
Gilles Bareilles CTU in Prague
Lê-Nguyên Hoang Calicarpa, Tournesol
Abstract
If you tell a learning model that you prefer an alternative over another alternative b, then you probably expect the model to monotone, that is, the valuation of a increases, and that of b decreases. Yet, perhaps surprisingly. many widely deployed comparison-bas based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley- Terry models [10. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that monotone. Namely, propose new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantec monotonicity. Our experiments show that this monotonicity far from being a general guarantee, and that our new class generalizing models improves accuracy, especially when the dataset is limited.
1 Introduction
Preference learning, sometimes known as alignment, has become central to machine learning. In particular, in recent years, there has been a growing interest to leverage comparative judgments to fine- tune AI models, using frameworks like Reinforcement Learning with Human Feedback (RLHF) [6] or Direct Preference Optimization (DPO) [11 in the context of language models, or linear models in the context of applications ranging from trolley dilemmas to food donation [1] 20) These models now deployed at scale. In parallel, learning preferences from comparisons with mathematical guarantees also fits in social choice theory, contributing to developing more transparent social medial, as advocated recently by "prosocial media"
39], with direct application for collaborative scoring of social media content [16] consensus-driven polling [34], or recommender system based on explicit preference such as among others.
Yet, bizarre aspects of these preference learning algorithms are regularly observed. A common but striking observation is that, when updating model based comparison judging that an item is preferable to an item b, the probability of a can decrease [28 23 2]. In fact, most deployed learning algorithms fail guarantee monotonicity: the probability score of item may be reduced after was said to be better than another item.
Perhaps surprisingly, the root cause of this lack of monotonicity is not the nonlinearity of the models. In fact, we can expose this "bug" with a very basic example. Consider linear two-dimensional model with parameter 3 E I to be learned: the scores of items and b are 3 xa and 3 zs. where To and wp are (two-dimensional) vector embeddings. We are given a comparison that favors alternative over b, whose embeddings push 31 towards lower values. But since the score of a according to the linear model is 3 x. = 81, this means that the score of will also decrease. Thus, including comparison that favors over b
Va (1, 0) and 24 (2, 0). Since Ea1 2b1, this comparison will
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

has decreased the score of a. This becomes all the more troubling when there exists an alternative c with embedding = (0, 1), whose score remains unchanged.
This example questions whether preference learning algorithms can be trusted. A user who witnesses surprising evolution of alternatives' scores as illustrated above might, understandably, prefer not to use such an algorithm. Worse yet, if they are nevertheless forced to use the algorithm, they could want remove the data they previously provided, because the eventual learned model was deteriorated by their truthful data reporting. More generally, this may discourage users to report their preferences, and rather provide tactical preferences in the hope steer the model towards their goal. This is reminiscent of tactical or "useful" strategies in voting systems. There exist classes of models which have a mathematical guarantee of monotonicity, such as (Generalized) Bradley-Teryy models [[O]; see also (27]. To the best of our knowledge, existing models with guarantee of monotonicty fail to generalize: they cannot predict the score of items that have not been compared. Hence the following research problem.
Can generalizing comparison-based preference learning algorithm guarantee monotonicity?
Contributions. Our first contribution is to identify a large class of preference learning algorithms that leverage both comparisons between alternative pairs, and descriptive information (embeddings) on individual alternatives. which we call Linear Generalized Bradley-Terry with Diffusion Prior (Definition B3.. This class extends the (Generalized) Bradley-Terry models [TO] by including linear mapping of the embeddings and and priors on alternative similarities, thereby allowing preference generalization to yet uncompared alternatives.
As a second contribution, we provide conditions on the embeddings that guarantee that the learning algorithm behaves monotonically when new comparisons are provided. As discussed above, this property is highly desirable and yet hard to guarantee in practice. In particular, we propose class of diffusion embeddings that guarantee monotonicity, and for which membership is easy-to-check. Interestingly, diffusion embeddings yield a very appealing interpretation as heat diffusion dynamics where comparisons play the role of heat pumps. A direct consequence is that categorical information (one-hot encoding embedding) yields monotone learning algorithm. In particular, this class enables us to provide a positive answer to our research question.
Finally, we evaluate the statistical performance of our learning algorithms through numerical ex- periments. Our evaluations show that a linear model with good embeddings and diffusion priors outperforms the classical GBT model [00. in particular with limited amount of data.
Related works. Learning preferences from comparisons has long history, dating back to Thur- stone R7. Zermelo 44). and Bradley and Terry 33 To handle inconsistent judgments, such algorithms define a probabilistic model of how latent scoring of alternatives are transformed into noisy comparisons. Their approach adapted by [1 and [3 to model the selection of one preferred alternative out of several proposed ones; see also 23 and (22 Chap. 3].
While the Bradley-Terry model considers binary-valued comparisons, various authors have proposed extensions to real-valued comparisons, e.g., ranging the interval [-1, 1]. Historically, this started with the modeling of draws in games like chess [8. More recently. [19] proposed the platform C1 Climpact where users are given pairs of activities, and are asked to evaluate the comparative pollutions of two activities. They then develop a model based on quadratic error to turn the comparisons into evaluations of the amounts of pollution of the individual activities. Their model was then generalized by [10 in framework they call Generalized Bradley-Terr -eery (GBT) to turn real-valued comparisons into scores. All these models however consider that each alternative's 's score is an independent latent variable to be learned. Thus, they fail to generalize to non-evaluated alternatives.
Independent of user- asovipdovided comparisons, alternatives usually come with descriptive information. A natural idea to generalize is then to model an alternative's 's score as a parametrized function of a vector embedding of the description of the alternative; see e.g.. 24
I2. Recently, this trick has been widely used in the context of language models [8 4 especially through algorithms
like Reinforcement Learning with Human Feedback (RLHF) 6 35 Direct Preference Optimization
(DPO) (32. or Generalized Preference Optimization (GPO) 36) to name few. In this paper, we
restrict ourselves to linear models, where an alternative's 's score is assumed to be linear function of their embedding. Application peplicatioonwise, we focus on social choice applications, and rule out Supervised

---

FineTTuning applications. Such linear models of preferences trained from comparative judgments have previously been studied and used, e.g. by [3 26 20
The study of the mathematical guarantees of preference learning algorithms has only emerged recently. In particular, nonlinear models have been empirically shown violate monotonicity properties [5] 28 33). While [2] proved that nonlinear models nevertheless provide weak monotonicity guarantee they call local pairwise monotonicity, they also suggest that these models are unlikely verify stronger forms of monotonicity. Conversely, and quite remarkably, [T] proved that the GBT model guarantees monotonicity for all GBT root laws. Our model extends GBT in several ways.
Paper structure. The rest of the paper is organized as follows. Section introduces the formalism, formally defines monotonicity, and recalls the GBT model. Section[]defing the linear GBT model with diffusion prior, and states our main results. Section provides the main lines the proofs the main results. Section [aationnrrrpprte reports on experiments. Section concludes.
2
Monotonicity of Scoring Models
In this Section, we set notations, formalize the notion of monotonicity, and recall the Generalized Bradley-Terry model.
2.1 Notations and operations on datasets
Consider a set A of alternatives. For simplicity, we A 21, The set C I denotes the set of admissible comparison values, which assume be symmetric around zero, Le. r E R - -r E R. In the classical Bradley-Terry model, we have R = {-1,+1}. The generalized Bradley- Tenry model allows wider variety of possible comparison values, for instance 9 = [-,,1], or R2 R for the uniform and gaussian root laws. A comparison sample is defined triple (a,b, r) where a, E A are distinct alternatives, and r E R.. We assume (a, b, and (b, a, -r) to be equivalent, which we write (a, r, = (b, a, -r). By also having obtain equivalence relation. A dataset D N for the set of datasets, and |D| the length of dataset D E D. We now define four parameterized operations D > D on datasets.
(a,b,r) = (a, (a,b,7 (and the relation false otherwise),
list D : [N] + 2 R of comparison samples. We write D - UNEN x R)N
Exchange. For any E N, EXCHANGE,, is the dataset obtained from D by replacing, if it exists, the n-th entry (un, bn: Tn) with (bn, On, -rn) All other entries are left unchanged. Assuming that preference learning algorithms should interpret these two comparison samples identically, this operation should not affect training.
Shuffle. For any N E N and any permutation of [N], SHUFFLEN. o (D) is the dataset obtained from D by reordering its N first elements according a. Formally, if D 2 N, then for all n E [N] we have SHUFFLEN, (D), = Da(n)- Otherwise, D is left unchanged. Assuming that preference learning algorithms should be invariant to shuffling, this operation should not affect training.
Append. For any comparison sample (a, b, b)r, APPEND, b. (D) is the dataset obtained from D by appending (a, r).. Formally,
and APPEND,, (D) (a, b, r). All other entries are the same in D. An append is said to definitely favor a' over b' if has parameters (a, b, r) ~ (a', b', R). Note that if Ri does not have a maximum, then no append definitely favors a' over b'.
have JAPPEND,,A, b.r (D) D
Update. For any n E N and comparison r E R, UPDATE,,, (D) the dataset obtained from D by
replacing the comparison of the n-th entry with The update is said to favor over b if either (i) (An, bn) = (a, b) and r 2 Ta, or (ii) (a,, b,,) = (b, a) and r S rn- other words, favors a over if it acts on comparison sample between and b, and modifies the comparison further favor
3

---

2.2 Monotonicity
Definition (Favoring a). An operation favors a if o is a composition of the operations (i) EXCHANGE, (ü) SHUFFLE, (iii) APPEND that definitely favor some other alternative and (iv) UPDATE that favor a over some other alternatvve. We write Za D' if there exists an operation o that favors such that D' = (D.)
The relation 3a is preorder. Indeed, 3a is reflexive: any dataset D equals o(D) with = UPDATE n.r with n 1, and r1. The relation 3a is transitive: if D; Ia D2 and D2 3a D3. then there exists operations
and 02 that favor such that D1 = (D2), and D2 02 (D)); thus Di 01 o o2 (Da), where 0₁ o 02 is an operation that favors by Definition so that D 3a D3- Similarly, we define the preorder Sa over RA by 0 Sa 0' if O. < 0' coordinate-wise. We can now formally define monotonicity.
Definition (Monotonicity). The preference learning algorithm ALG is monotone when, for every alternative E A. ALG (D. 3a) -IA Sa) is monotone. Equivalently, ALG is monotone when, for every alternative a E D. D' implies ALG(D) Za AD)
In the sequel, all preference learning algorithms that we will consider are neutral (25), Lee. they treat all alternatives symmetrically For such algorithms, the monotonicity for any single a E A clearly implies that for all a E A.
Remark 1.
2.3 (Generalized) Brary and monotonicity
Here we recall the probabilistic model of GBT [00, slightly adapting it to our dataset formalism Following Bradley and Terry (3) GBT defines a probabilistic model of comparisons given scores. Specifically, given two alternatives b E AA having scores 0g and 0, the probability of observing value for the comparison of a relative to b is
p(r(0ae)) x f(r) exp((r Oaes):
(1)
where Gues - 02 0g the score difference between a and b, and the root law, probability distribution on R. that describes comparisons when and have equal scores.
Given dataset D (a,,, Dn. Fn)eNN of N independent observations following (.. and assuming gaussian prior with
and a² variance for each alternative score la A, the standard Maximum A Posteriori methodology results in the GBT estimator:
GBT f (D) A arg min
Σ 8% +
Σ a (a,br)eD
(0 - rdaas-
(2)
BERA 20
There, $, is the cumulant- sumulann-generatingg function of the root law distribution f: (0) log Jyg dr).. As soon as I has finite exponential moments, Φ, is well-defined and convex; in particular, 2 is strongly convex problem with unique minimizer [10
We recall below Theorem 2 [T0), that guarantees monotonicity of GBT f.o, when elements can only be compared once. The forthcoming Theorem 3 extends the result to situations where two elements are compared multiple times.
Proposition 1 (Th. 2, [T). Consider root law f. a scalar >> O, and two datasets D, D' which contains at most one comparison between any pair (o. b) E A?. Then, for all D >a D' implies GBT D) Za (DT).
Although well behaved in many aspects, the generalized Bradley-Terry model fails perform generalization: an alternative a that never appears in D will receive a nil score GBT(D), = 0. However, in practice, alternatives may (i) admit informative descriptions, and (ii) have known relationships. This should help guess the score of a yet uncompared alternative, based on the scoring of similar compared alternatives. We introduce such a learning algorithm the next Section.
Formallly ALG(o D) a ALG for all permutations of A, with action that applies pointwise all apparitions an alternative a E A. In (10). the authors consider datasets that contain most comparison per pair of alternatives.

---

Linear GBT with Diffusion Prior
In this Section, introduce class of preference learning algorithms that incorporate both user comparisons and contextual information on the compared elements, and state our main result about their mathematical guarantees on monotonicity.
3.1 Learning with prior similarities
The GBT,,, of model does not include prior knowledge on the structure of the alternatives. For example, when alternatives represent videos on YouTuub, the fact that two videos belong to the same channel cannot be represented in Equation (2). More generally, Equation 2 does not encode any prior similarities between alternatives. Consequently, if alternative then, even if a is similar to an alternative that has large non-zero score, a will still be assigned a zero score. In other words, Equation (2) does not allow us to generalize.
never compared with any other,
To address this issue, we generalize GBT in two directions.
1. (Embeddings) We assume that, to each alternative E A, corresponds an embedding za RR where D positive integer. We model the score of alternative a by linear function of the embedding:
0g (B) A x
for parameter 3. Denoting Px the matrix collecting all embeddings, and Taeb 2a Thb for any a, b E A, the GBT parameter E R is replaced by a linear function ((3) = x 3. For instance, in the context of YouTube, Fa could denote one-hot encoding the content creator identity; more in Section 5 2. (Similarity) We consider
general regularization term R(3) of the form
R(3)
Σ 20
33 + 2
1 Σ 0g(0() (3)3Loa000())
(3)
ob
where T. is Laplacian matrix, i.e. such that Tima = - botta |Tabl 2 0 and T.ub T S O, for all a & b. The matrix L can be thought the Laplacian of graph encoding (prior) similarities between alternatives, the weight Labb representing the similarity between a and Therefore, the regularization term Ent (,) = 2 Lays Lab (0. (²))) incentivizes the model to (a priori) assign similar scores to similar alternatives.
We define the class of GBT models that will study in this paper.
Definition 3 (Linear GBT with Diffusion Prior). Let f be root law, be embedding, a >> 0 a positive constant, and a Laplaciann matrix. The model GBT j.a.. defined as
xJ ( (D)
where B*(D) arg min L(IDD) minimizes the strongly convex loss function
L = R(3)+ +
Σ
(
r2
(4)
(a.brr)D
For conciness, let 0 (D) A GBT a. 3T)
Remark that the original GBT is a special case of Linear GBT with Diffusion Prior with A = D, the identity matrix, and L
Proposition 2. Linear GBT with diffusion prior is neutral, iee. invariant up alternative relabeling.
Proof. See Appendix B for a formal statement and derivation.
3.2 Monotonicity and diffusion
We now present our main result (Theorem We prove that for special class of embeddings, namely diffusion embeddings, monotonicity is guaranteed. Diffusion embeddings take their name from the interplay with (super) laplacian matrices.
5

---

Definition 4 (Super-Laplacian matrix), A super-Laplacian matrix A is symmetric matrix such that for all a + b, Ana > Ebza Aub and aub 2 O.
Definition (Diffusion embedding). An embedding is a diffusion embedding if the Gram matrices Xx x² x + Xl have super-Laplens Laplacaan inverses X for any À >> O.
Note that if X = x² x is itself invertible with super- LaplacLa inverse, then it is a diffusion embedding. However, this case is restrictive since it implies that D A.
Theorem 1 (Monotonicity with diffusion embeddings). For any root law f. positive constant o > 0, diffusion embedding z, and Laplacian matrix L, fo, L monotone.
Proof. The theorem follows directly from Proposition 3 and Theorem which are provided and proved in Section 4
3.3 Example: one-hot encoding
one-hot encoding is possible when the alternatives can arranged into multiple disjoint classes. For example, if the alternatives represent videos YouTube, one partition them by the YouTube channel they belong to. In that case, the score of an alternative a defined as 0g = ?d(aa + s² Qa. where d(a) is the channel of The score Ya(a) represents the score of the channel d(a)), while ae represents residual score of and is real constant that controls the scale of the residual score. Theorem states a one-hot encoding is an example of diffusion embedding. We postpone the proof to Appendix H
Theorem 2 (GBT with one-hot encoding). Let f be a root law, a > a positive constant, Laplacian matrix and E R. Let DxA be one-hot encoding matrix: Eda 1 if, and only if. a belongs to d. Then, for any real number s, (x sT)" diffusion embedding and the score GBT f.a.L is monotone.
The proof
This Section provides the mathematical analysis that builds to the proof of the main result, Theorem proposes a differential analysis framework for the dataset operations outlined above;
Section 4.1
Section 4.2 then provides the proof.
4.1
Differential analysis of dataset operations
The goal of this technical section to connect the discrete domain of datasets with tools from differential analysis. Studying the monotonicity (Definition 2 of 0 (D) requires to compare the loss functions for datasets that are related by basic operation. Given that the loss invariant under EXCHANGE and SHUFFLE operations on the dataset D, on one hand because of the specific form of the GBT loss, and on the other because it features a sum of comparison samples of the dataset, the same invariance holds for 0*, Thus, to prove monotonicity, it suffices to study what happens under APPEND and UPDATE operations that favor a over
Because the loss function is sum of terms indexed by the elements of the dataset, this relation is quite simple
C(3 APPEND a, (D)) L(3 ()((0ss(3) (3)
L(3) UPDATE (D)) = D( (r rr 7r.)................ (9)
(6)
To enable the differential analysis of these operations, we introduce smooth deformation of the loss function.
Definition 6 (Smoothed loss). For every À E I, and every operation o of the form APPENDa.b.r or UPDATE define the smoothed loss Lx by
( - rla36(3)
if = APPEND..b.r
(x & L(3 + A.
- (r Tw) - Oa. (3)
if UPDATE otherwise.
(7)
Denote also (D,,)) é arg min (x),)oo) and 0, (D. & (3

---

The smoothed loss matches the loss at À E {0.1}. as Lo(5)D, - [(3) and ( (()) We will leverage this by using the integral expression
=
/. d0 dd (D..)d
o (
(8)
When À + (D,,) continuously differentiable, this integral expression is well-defined, and it suffices that the derivative d (D, o) dA be non-negative for the score difference to be non-negative. Lemma states that this derivative is well defined and provides a formula. The proof is given in Appendix C
Lemma Let H (0)D) denote the Hessian of E(OID) = -EaabbrjED 4 ( and X
g²x²²x denote the (scaled) Gram matrix of the embedding æ. Then, for any busic operation and dataset D, the loss function Lx(-IDD o) admits a unique global minimizer 3x (D, o), and the inferred (D.,) o) a smooth function of over 00). Moreover,
score 8,(D,,
if B and Dn (a,b,)). then the score of the alternative a satisfies
do Xa (D.
-r--s)-c (I + (I+ - "Xeueb- (I+
(9)
d)
A=p
There, En denotes the a-th vector of the cartesian basis of R-A, 02a denotes the a-th coordinate of 03 and each
if APPENDa.b.. a.b.r then the score of the alternative a satisfies
d0 Xa (D.
- -r- (0,06))) e +XL 144 - Xeaeb. (10)
dA
d=
where gab TR. ARA is the Laplacian matrix of the graph over the alternatives with a single edge ab (with weight 1), i.e. Sub Sig
SSpb S 1, and Saa -0 otherwise.
We are interested in the sign (positive or negative) of the expressions Equations ( and (1O). First, the factors r and r - D' (0 are easy to understand. If o UPDATE,,r favors a over then r 2 by definition. If o APPEND favors a over b, then r = sup Ri which is the supremum of V 10 Theorem l].
Therefore, if we want to compare the scores of and b, the important factor to study is the matrix (I + X(L + H))-', where H = H + (f S if o APPEND,.b.r. or H H if UPDATF and D, = (a, b, s). We study this matrix in the next section.
4.2 Good embeddings: a sufficient condition for monotonicity
In this Section, we provide sufficient condition on the embedding for the Linear GBT model to be monotone (Definition 7 Theorem 3), and provide mathematical properties of this condition (Proposition 9). Finally, we show that diffusion embeddings meet the above sufficient condition (Proposition
Definition 7 (Good embeddings), Given a Laplacian matrix Y, an embedding x --good if the Gram matrix X =²'x satisfies (I++XY) Xenes 2 for all (ab). An embedding x is good if x is -ggoodoo for all Laplucian matrices Y.
Theorem 3 (Monotonicity with good embeddings). For any root law f. positive constant > 0. Laplacian matrix L, and good embedding x, GBT is monotone.
Before going to the proof, we provide some intuition. Note first that the Hessian H of E(O[D)
E(G,..yeD (8aa2b) also a Laplacian matrix. Indeed, let G be the weighted graph whose edges are the pairs (ab) of alternatives that occur in the dataset D, weighted by Gab = No. (0((ae)) where Nab is the number of occurrences of the pair (ab) in D. Since & is convex, these weights are nonnegative. Then, a direct calculation shows that H is the graph Laplacian of the weighted graph Therefore, given any prior Laplacian matrix L, the matrix + fI, where ft is defined section 4.1 is the Laplacian of graph that combines the prior similarities (T.) with the similarities inferred from the dataset at hand (H). This observation motivates Definition
G: for * b, Hos Eeta Na. (Guc) and Has - NN.
7

---

By Lemma 1 if, for every operation every score function 0 and every dataset D. the inequality (1 + ( + - 2 0 holds, then the score 0* (D) derived from the monotone. This precisely implied by x being good.
Proof of Theorem 3
loss function of Equation
This result motivates more precise understanding of good embeddings. In the special cases (A, D) = (2, D) and (A, D) (A, 1), we have complete characterizations (see Appendix D) In general, however, checking goodness is not straightforward: two embeddings and can be
individually good, while their concatenation [ar y/ fails to be good (see Propositions and 8
Appendix E Nonetheless, any embedding can be made Y -good by concatenating it with a sufficiently scaled identity. We formalize this in Appendix|B
4.3 Diffusion embeddings are good
Finally, we show that any diffusion embedding is a good embedding. This uses the fact that any super-Laplacian matrix A satisfies ex A --'''aaaa -acsh for any pair (a, A² This result has been proved in [10 Lemma 1] and we provide an alternative proof highlighting the diffusion perspective
G
Proposition 3. Any diffusion embedding is good embedding.
Proof. If x a diffusion embedding. À > O, and Y is an arbitrary Laplacian matrix, then the matrix X5 +Y, where Xx _Tx AI, is super-Laplacian Consequently, e (I + XxY) Xxe
c (X, +X) teb 2 0. The claim follows by taking the limit À + 0.
Experimental evaluation
In this Section, we provide numerical exploration of the prevalence of "goodness" for random embeddings, and the statistical error of several preference learning models Appendix A provides complementary experiments real-world data.
5.1 Probability of goodness for random embeddings
To illustrate the challenges of achieving good embeddings, we generate random i.i.d. Gaussian embedding matrices x and evaluate their quality. In Figure we examine single Gaussian embedding a (left) and its concatenation with the identity matrix / (right). Our findings indicate that the goodness of x is more likely for large values of D/A and significantly diminishes when A/D is large. The concatenation with the identity matrix notably enhances the goodness, aligning with Proposition 9
5.2 Generative model, metric, and simulations
For each experiment, we consider the ground-truth embedding x² E DxA Laplacian matrix L!, constant ot, and root law ft. The ground-truth features are generated as 8 (o, (at)² xt (art)))) and (f (1:t) (tt We then create dataset D : [M] A2 x R by first selecting N random comparison pairs uniformly. The corresponding random comparisons r are generated using the root law f and conditionally 0i, We shall only consider the uniform root law fi = and set o! 1.
The estimated scores are computed as 0" (D) = GBT F. (D), where the quadruplet (ʃ, a, xr L) may may not align with the ground truth. Since the quality of score vector is invariant under constant shifts, we evaluate the eиror over zero-mean versions of both @t and 6 (D). More precisely. we use Monte Carlo simulations to estimate the normalized mean squared error (nMSE), defined as:
[1 (D) 0+ (D) (0t 0t ||
nMSE(f', oi,, L',,, L. = nMSE E
||0t 01
We then analyze how the nMSE evolves with respect to various parameters.
'The code is available at https: //ggttttttttt//////e//eeeeee.. www.cooo/////e com.peeab/gbtlabb and will be made publicly after the review process. We experiments personal laptop with 16GB of RAM and 2.10 GHz processor.

---

44
(a)
(b)
Figure 1: Left pane: Probability that a Gaussian i.i.d embedding x is a good embedding for < A <
15 and < D 15. Right pane: As for the left pane with embedding x]
1.0
gbt & emberdding
gb: & ernbedkiigg
gbt
gb:
embedding only
0.8
en beddingg only
nMSE
MMSE
0.6
0.4
0.4
0.2
0.2
0.0
5
10
15
20
25
0
100
200
300
D
N
(a)
(b)
Figure 2: Left pane: nMSE function of D for A = 25 alternatives and N = 500 comparisons over 100 seeds. Blue curve with [] x² (full embedding), orange curve with embedding / (classical GBT), and green curve with embedding x (features only). Right pane: nMSE with respect to the number of comparisons N for 20, 10. and 1000 seeds. Blue curve: GBT with one-hot encoding; Orange curve: GBT. Every curve is displayed with its error bar (using 1.96a /ooo/v/ /n_seeds).
Figure 2a shows the nMSE function of D, using data generated with æ [r #1]" (i.i.d.
Gaussian 2), uniform fl, L and of = We compare three models with shared parameters L.) = (fi, L using æ!, I (classical GBT), and ø respectively. The embedding based model outperforms others, combining the strengths of classical GBT for small and feature-based learning for larger D.
Figure 2b shows the nMSE function of the number of comparisons N. Data generated with (ft xt Lt. at)
(-1111.1 atj² .0.1). where ă is a one-hot encoding matrix (see Section 3.3p. The results show that one-hot encoding greatly reduces the number comparisons needed to reach given mMSE. This is useful in applications like YouTube score estimation (17), where the encoding reflects channel and enables generalization across alternatives.
6 Conclusion
In this paper, we introduced a new comparison-based preference learning model, namely linear GBT with diffusson prior. This model not only generalizes to previously uncompared data using embeddings, but also potentially guarantees monotonicity, depending on the class of embeddings used. We proved that model is monotone for various classes of embeddings (one-hot encodings,

---

diffusion, and good embeddings).. To the best of our knowledge, linear GBT with diffusion prior is the first model that guarantees monotonicity while being able generalize.
Diffusion embeddings form a class of embeddings (containing one-hot encodings) that yield mono- tonicity. Our proof techniques relied monotonicity (Definition 7 and properties of (super) Laplacian matrices akin to diffusion theory.
interesting interplay between algebraic criterion for
Limitations. While improving the understanding of preference learning with guarantees, our theory currently provides guarantees for diffusion embeddings only. We hope to motivate more work on preference learning with guarantees, in order to build more trusworthy Al systems, with notable applications in [39 16
17 [4 Also, we caution against the use of preference learning algorithms that rely on data collected in inhumane conditions, as is mostly the case today [18 29 13 [4. is unclear whether our work can positively contribute to this issue.
Acknowledgements
The contribution of Gilles Bareilles has been funded by European Union's Horizon Europe research and innovation programme under grant agreement No. 101070568.
References
[1] Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shar- iff, Jean-François Bonnefon, and Iyad Rahwan. The Moral Machine experiment. Nature, 563((729):59 64, November 2018.
[2] Gilles Bareilles, Julien Fageot, Lê-Nguyên Hoang, Peva Blanchard, Wassim Bouaziz, Sébastien Rouault, and El-Mahdi El-Mhamdi. On Monotonicity in Al Alignment, June 2025.
[3] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[5] Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi (Richard) Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithmss do not learn preference rankings. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[6] Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4399-4307, 2017.
[7] Villo Csiszár. Em algorithms for generalized bradley-terry models. In Annales Universitatis Scientiarum Budapestinensis de Rolando Eötvös Nominatae (Sectio Computatorica), volume 36. pages 143-157, 2012.
[8] Roger R Davidson. On extending the bradley-terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329):317-328, 1970.
[9] Jiuding Duan, Jiyi Li, Yukino Baba, and Hisashi Kashima. A Generalized Model for Multidi- mensional Intransitivity. In Jinho Kim, Kyuseok Shim, Longbing Cao. Jae-Gil I Lee, Xuemin Lin, and Yang-Sae Moon, editors, Advances in Knowledge Discovery and Data Mining, pages 840-852 Cham, 2017. Springer International Publishing.
10

---

[10] Julien Fageot, Sadegh Farhadkhani, Lê-Nguyên Hoang, and Oscar Villemaud. Generalized bradley-terry models for score estimation from paired comparisons. Michael J. Wooldridge. Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Fighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirtt-Sixth Conference on Innovative Applications of Artificial Intelli- gence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 20379-20386. AAAI Press, 2024.
[11] Markus Flicke, Glenn Angrabeit, Madhav lyengar, Vitalii Protsenko, Illia Shakun, Jovan Cievaric, Bora Kargi, Haoyu Hc, Lukas Schuler, Lewin Scholz, Kavyanjali Agnihotri, Yong Cao, and Andreas Geiger. Scholar inbox: Personalized paper recommendations for scientists, 2025.
[12] Yan Gu, Jiuding Duan, and Hisashi Kashima. An Intransitivity Model for Matchup and Pairwise Comparison. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 692-698, January 2021.
[13] Yuan Guo, Peng Tian, Jayashree Kalpathy- Crammer Susan Ostmo, J. Peter Campbell, Michael F. Chiang, Deniz Erdogmus, Jennifer G. Dy, and Stratis Ioannidis. Experimental design under the bradley-terry model. In Jérôme Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 2198 22044. ijcai.org, 2018.
[14] Rachel Hall and Claire Wilmot. Meta faces ghana lawsuits over impact of extreme content on moderators. The Guardian, 2025.
[15]
Karen Hao and Deepa Seetharaman. Cleaning up chatgpt takes heavy toll on human workers. Wall Street Journal, 24, 2023.
[16] Lē-Nguyên Hoang, Louis Faucon, Aidan Jungo, Sergei Volodin, Dalia Papuc, Orfeas Liossatos, Ben Crulis, Mariame Tighanimine, Isabela Constantin, Anastasiaa Kucherenko, Alexandre Maurer, Felix Grimberg, Vlad Nitu, Chris Vossen, Séhastien Rouault, and El-Mahdi E1-Mhamdi. Tourneso:: A quest for large, secure and trustworthy database of reliable human judgments. CoRR, abs/2107.07334 2021.
[17) I.è Nguyên Hoang, Romain Beylerian, Bérangère Colbois, Julien Fageot, Louis Faucon, Aidan Jungo, Alain Le Noac'h, h, Adrien Matissart, and Oscar Villemaud. Solidago: A modular collaborative scoring pipeline, 2024.
[18] Stephanie Höppner. Africa's content moderators want compensation for job trauma. Deutsche Welle, 2025.
[19] Victor Kristof, Valentin Quelquejay-Lectérre Robin Zbinden, Lucas Maystre, Matthias Gross- glauser, and Patrick Thiran. user study of perceived carbon footprint. CoRR, abs/1991..1166 2019.
[20] Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, and Ariel D. Procaccia. Webuildai: Participatory framework for algorithmic governance. Proc. ACM Hum. Comput. Interact.,
3(CSCW):11:11111:35, 2019.
[21] R Duncan Luce et al. Individual choice behavior, volume 4. Wiley New York, 1959.
[22] G. S. Maddala. Limited- Limited-Dependent Dependent and Qualitative Variables in Econometrics. Econometric Society Monographs. Cambridge University Press, Cambridge, 1983.
[23] Daniel McFadden. Conditional logit analysis of qualitative choice behavior, pages 105-142, 1974.
[24] Joshua E. Menke and Tony R. Martinez. A bradley-terry artificial neural network model for individual ratings in group competitions. Neural Comput. Appl., 17(2):155-186, 2008.
[25] Roger Myerson et al. Fundamentals of social choice theory. Quarterly Journal of Political Science, ((3):305-337, 2013.

---

[26] Ritesh Noothigattu, Snehalkumar (Neil) S. Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel D. Procaccia. A voting-based system for ethical decision making. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7. 2018, pages 1587-1594, AAAI Press, 2018.
[27] Ritesh Noothigattu, Dominik Peters, and Ariel D. Procaccia, Axioms for learning from pairwise comparisons. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[28] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with appo-posiiive. CoRR, abs/2402.13228, 2024.
[29] Billy Perrigo. Openai used kenyan workers on less than $2 per hour to make chatgpt less toxic. Time Magazine, 18:2023, 2023.
[30] Robin I. Placket. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-202, 1975.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, July 2024.
[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 53728-53741. Curran Associates, Inc., 2023.
[33] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. CoRR, abs/2410.08847,, 2024.
[34] Christopher Small, Michael Bjorkegren, Timo Erkkilä, Lynette Shaw, and Colin Megill. Po- lis: Scaling deliberation by mapping high dimensional opinion spaces. Recercu: revista de pensament andlisi, 26(2), 2021.
[35] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12 12, 2020, virtual, 2020.
[36] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandricllo, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[37] Louis Leon Thurstone, A law of comparative judgment. Psychological Review, 34(4):273-2) 1927.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
12

---

[39] E. Glen Weyl, Luke Thorburn, Emillie de Keulenaar, Jacob Mchangama, Divya Siddarth, and Audrey Tang. Prosocial media, 2025.
[40] Ernst Zermelo. Die berechnung der tumner-ergeennssee als ein maximumproblem der wahrschein- lichkeitsrechnung. Mathematische Zeitschrift, 29(1):436-405 1929.
[41] Piplong Zhao, Ou Wu, Liyuan Guo, Weiming Hu, and Jinfeng Yang. Deep learning-based -baee learning to rank with ties for image re-ranking. In 2016 IEEE International Conference on Digital Signal Processing (DSP), pages 452-456. IEEE, 2016.
13

---

A Experiments on real world data
In this Section, we complete the experiments synthetic data (Section 5) with experiments on real-world data [[6]. More precisely, we provide numerical evidence for the fact that including (diffusion) embeddings improves performance. The code to reproduce experiments is available at https://gi http:///itttbbb.coo.o wtw..ah....oo/paaabbbbbaaaaaaaa
A.1 Experimental set-up
The real-world data contains comparisons between Youtube vidcos made by various users, from the Tournesol platform [17]. We selected a subset D of 1000 comparisons from a single user, the one that has most comparisons. Every comparison
tuple (a, b, r)) where b are video identifiers, and is the comparison value r. originally an integer between - 10 and 10, which we rescale to fit in [-1,
In addition, associate for each video a, the YouTube channel it belongs to. We describe this relation using one-hot encoding matrix X RxN, Xea = the video a belongs the channel e, or Xea otherwise. We obtain an embedding x E (DD+N)x by concatenating X with AI, that
X IN
is, x
We compare two models: (i) f.,.. (which uses embeddings), and (ii) f.L... (the original GBT, which does not use embeddings). Both models have the uniform distribution in [-1, 1] as root law, (r) 1 (r),))))), and the same Gaussian prior. We do not use any Laplacian regularization.
After training, each model M computes, given pair (a, b) of video identifiers, the expected compari- son value defined as
M(a / f()e 00) 11,00%
0%)
(11)
where y is the cumulant-generatin function of the root law, and 0* are the scores learned. Given validation dataset D val the validation risk of M is given by
Σ
(M b) - r)
(12)
D
(D,
A.2 Results
Figure 3 reports the empirical risks of the two models, using 10-fold cross validation scheme over the dataset (1000 comparisons). We observe that the average validation risk of the model with embeddings is 8.40 10-3, while that of the model without embeddings is 10.1 10 Hence, including the YouTube channel embeddings reduces the risk by 17% on average.
B Proof of neutrality (Proposition 2
To formalize neutrality, we must define how permutation т of the alternatives acts on the inputs and outputs of Linear GBT with Diffusion Prior. We define the actions as follow:
(r.D), D)n Dn,,
(13)
b.) (r(a),
(14)
(r-0)a 0+(0):
(15)
(r-x)a = Ir(a),
(16)
LT- = (i(()r(())
(17)
Neutrality is formalized by the equality
VT, r j. OF GBT, GBTjaa.. J.
(18)
14

---

0.012
0.030
validation risk
0.006
0.004
0.000
embeeddings
embeddings
Figure 3: Validation risks of GBT, f.1.00 (with embeddings, on the left) and GBT,, G. Iw (without embeddings, on the right), estimated using 10-fold cross validation. The box plots report the minimum, 1st quartile, median, 3rd quartile and maximum. Outliers are also shown.
We indeed have (r-1. GBT (D)), jGBTia (T D))
(19)
= =(T-x); (aa fa,,,, D) = - f7 (T D)
(20)
= x arg min
|3
Σ
1o_ B) B) (x
20
2 ob
Σ
p x) x) (a) (b)
r(T . 1())
10)- 10)
(21)
(a.b.))D
x arg min
|B
Σ
(1' (xT L) (xT Σ
(0, r.. o5
(22)
20
(a..r))
x - (D) = L.
(23)
C Proof of Lemma 1
We consider the case of o = APPEND,.b.> The case UPDATE,,r with D, (a, b, s) is proved similarly. The loss functions L3 (3) and L() differ by >( (0nc r8))) The term luab being linear in 8, its Hessian is zero. On the other hand, the Hessian of Φ (fass) is simply the Laplacian (0ue)) Sah of the graph with a single edge ab with weight (00ss). Writing
H = H + (0oos) Sub.
(24)
we obtain the Hessian of the loss
-
x(L H + g² _
(25)
Therefore, the loss (3 o) is strictly convex, and admits global minimizer (D, o). To simplify notations in this proof, we write 8 (A).
We want the implicit function theorem to analyze how 8 (A) varies with X. For that, consider the gradient of the loss (x
F(B,A) = V (o = ((((B + x (0es())))) - r r)re
(26)
with domain IDD R and codomain I.
Fix some fA E R. The Jacobian F(3, X) with respect to 3, evaluated at À = μl, is given by
(3.(3.p) - I+ x(L + H²)
(27)
15

---

which is invertible. Moreover, since 3 (p) is minimizer, we have F (() (1) = 0.
Hence, the implicit function theorem states that there exists open neighhorhood U of 14, and smooth function 7 U + IR. such that
=1)==((1)) AA () =
(28)
(29)
= The latter equality implies that 7(A) = (*)) for all U. Consequently, (A) and (*)
x² 3* (A) depend smoothly X. In addition, the implicit function theorem also gives expression for the Jacobian Jx3", evaluated at f
JxB² (n) (J
(30)
(1 T 10[+z(L +H*))²)
= (r D (0
xeagb
(31)
Finally, note that
d6
yA (1)) = Jgf (B (1)
(32)
= (r 4 ((>))) x² (
IPaab
(33)
Let M = L++ H", and X = o²x²= x. Using Woodbury's inddbury'sideetitty
we derive
1 1 + x + H1)))*)))
= =T
(34)
= -
(35)
1 I + x(L + +H}²²)
xT
X - -XM(I+
(36)
XMM ")
(37)
=1+XM)--X
(38)
Thus,
d0' dx (w) (r (0%)))))))
oI+ x(L + H² T De
(39)
The result follows.
D Good Embeddings for A = 2 or D =
We can fully characterize goodness in lowest dimensional regime, either with only two alternatives or with embedding single feature.
Definition 8. We say that a matrix M is max-diagonally dominant if Mas 2 Mar.for any (ab).
aa c) c b
Proposition 4. An embedding with Gram matrix X
is monotonicity proof if and only if
<Vab <e bi.
Proof. We first observe that, since x² x is positive semidefinite, a, b 2 0 and det x)) ab 0.
This implies in particular that e 2 ab, which is the desired lower bound.
Two-dimensional Laplacian matrices are of the form Y 8
-1 1 with d > which
can be chosen as being equal to without loss of generality. We then have that I + XY = After simplification, its determinant is det(1 + XY) = 1 a + 2c.
+ a c c-a a
c c-b b 1 c
16

---

This quantity is strictly positive for e < 1+a+b which is always the case for as c² I ab. Hence, the matrix is invertible and we have, after computation.
1
a + ab c² c ab c²
c ab c² 4+ ab c².
M (I +I
(40)
1+a+b- 20
Then, M is max- max-diago diigonaaly dominant if and only if a 2 cet b 2 as expected. Finally, this shows that the embedding x Y -good for any Y. hence good.
We now focus on the case A = 2, for which we obtain a complete characterization of the goodness. Proposition 5. Consider the GBT model with embedding [ 2b] E Dx2 and root law
Then, the model is good if and only if. for any ().b) E A?, Ie = Ib or I 2b S min(11za aaa || The latter is equivalent to
|xaa||
||
))
a(Ia, Ib) S a0 ao(ll ||>b|]) = arccos
min
E [0, T/2
(41)
||ra|
where (' vb)
2i. 2b
is the angle between and rb- |+
Proof. We apply Proposition 4 x.g xi, and c = Ta Then, the goodness is equivalent
TaTb I min =g)) This relation is true for To. Ia and Ig < 0. Otherwise, 0 < Fas Zh. The relations E, S xa and h 2 implies that 2a S 2a and Ta S 2,, respectively, leading to a contradiction. Finally, the goodness is equivalent to Ta = Ib. or Ta and HG, have different signs.
have
For the D dimensional case, the same observation holds with a = I 112, b = || and e = x xb- I Then, Proposition implics that the goodness is equ bivalent to x 2s I min ||r 112 1 1²). The angular characterization follows easily.
We shall see that the goodness is very restricted for D =
Proposition 6. We consider the GBT model with embedding Laplacian matrix L = 0. The model is good if and only if
[To]aEA R root law f, and
0 P p - -ul A I [u,..., U, -U...., -U .,.,,,....0 ,
(42)
for some u, and P a permutation matrix.
Proof. The goodness is equivalent to the fact that, for any (ab), (xa -2.- 2 0 and (at - )as 2 0.
This is equivalent TaTh minnni .z)), i.e. Na Tb Ta S 0. This means that any Na should have a common value u > 0 and any 2s < 0 should have a common value -v < 0. Hence, Ta can take only the values u, v and O. Permuting the values, we obtain 42
E Counterexamples for Monotonicity
x2) T
Proposition There exists good embeddings #1 and x2 such that x =
[x]
not good.
Proof. For A = 3 and Gaussian root law (Y = 31 - J), we consider æ1 and 22 such that
0
0
0
1
X1
0 1 1
and
X₂
1 0
0
1
1
0
0.
[1 1
07
Then, X = TTxx x.
x
1
2
Then, and xy are good embedding
0
1
(c.g. remarking that they are J-blocs matrices and using Theorem 2p. The matrix A (1+XY) -Ix is given by
4
17
M =
4
8
4
and verifies Mi > Mn. This contradicts Definition therefore not good.
7 and xr is not Y-proof for 37 J,
0
17

---

Proposition 8. There exists one-hot encodings and 12 such that x embedding.
[x1
x22" is not good
Proof. For A 5, let æ1 and z2 be one-hot encoding with Gram matrices
1
1
0
07
1
0
1
0
0
0
0
X1
1
0
0
and X2
0
0
0
1
0
0
0
1
0
Then, 21 and 22 are good embeddings according to Theorem embedding
The Gram matrix of the concatenated
21
is
22
0
2
1
0
0
X =
2
1
0
2
1
1
2
is not a Y -good embedding for the Laplacian matrix
1
0
-
4
-1
Y
-1
1
0
0
0
-1
-1
0
2
We indeed have that
0.96
0.70
0.85
0.52
0.67
0.70
0.90
0.95
0.67
0.68
M = (I+
0.85
0.95
1.48
0.83
0.84
0.52
0.67
0.83
1.08
0.56
0.67
0.68
0.84
0.56
0.93
is such that Mas > M22.
F Results of Section 4.2
We now show that any embedding be made Y-good by appending sufficiently dominant identity component. This result guarantees that, asymptotically, adding uncorrelated features improves embedding monotonicity, regardless of the original embedding structure. This be made in relation with Figure for which we compare i.i.d. Gaussian x with it's concatenation with I.
Proposition 9, For any embedding x and any Laplaciann matrix Y, the embedding TX II 2/A|" DD//DDDaDo(((fee ( where DiagDom(Y) = min( (I+ Y)
Y-good for any X > >/A / (T+ Y) 0.
Proof of Proposition 9 We set M(X,Y) (1 + x The Frobenius norm is such that [|X + Y|| < |||||
+ Y| and ||X||| I | Assuming that ||X|| < and for = (1
X) X), we have
|
| < |
XX|
(43)
1
A20
The matrix + X/X positive definite, hence invertible and we have
M(I - = (I C
= ((I (X+
(44)
18

---

Then, there exist matrices Z and W, that we will control later such that
M(I+X/A, (I - x/
(45)
X/A)
(46)
=I++)) (Z-/))
(47)
From now, we assume that A > || In particular, using that x + -- increasing over
(0, 1) we have that |X/I I-1XXI7X therefore have
2 h(vAA33) S h(1/3) = 2 According to (43) applied to X/X, we
||/A|12 -
XXI
|| - 1 ||X/a||
(48)
2X
We also have that
|
|
3||
|x/A Z|
(49)
À
21
2X
Note that || < since X >> 3v / ||| and we can apply
43
to evaluate
|| ~ [1x/> () ()
AlI(X/A 2)12
1(X/A ()-++1) S 1- VAIIX/A Z
VAIIXAA ZIKAAIXX/X ZII
(50)
I| 2A
|
X
X
S VA
h
11 (112)=
(51)
2A
2A
2X
where we used ||(X/X Z) + -| S ||/A- - 2111([1 + 1 & VAIIX/A Z|1 (since
(1+ Y) has eigenvalues smaller than 1) and h(1/2) = 1.
Starting again with 45 [M + X/A (I+ b0 < [M + X/AX)) - (I+
then have
(52)
3/
A
X
11 - X/X)(T+ WI|
2A
2
(53)
3v
(54)
Let DiagDom(Y) = ain (YI md - (I (I+Y) which is strictly positive since (I +
Y)-1 is strictly max-diagonally dominant 10 We therefore have that, for any X >
3v
[|M(T + -I++))((+++)))))) S DiagDom(Y)
and therefore M(I + X/X, Y) is max-diagonally dominant, as expected.
G Inverse of super-Laplacian
Proposition 10. Let A be super-Laplacian matrix, then for any nodes a # b in A, have
e/ 'A-
(A )aa -1 Eatsb
(A -1 Jbb 2 0.
Proof. We prove the result by interpreting the coefficients of A -1 probability of some sample path of a discrete-time Markov process on the alternatives. Let D) be the diagonal of A, and the matrix defined by
A D(1 P). The matrix P is row-sub-stochastic matrix. Explicitly,
(55)
Aab
Paa = 0,
Pat =
Σ Pab K. PeA
(56)
A
19

---

Let be an extra symbol and A. fo) Let Ka = Aua Ebsta Aubl > 0. We define Markovian random walk on A. by the transition matrix T:
T(bla) = Pab
|aabl
a( = 1- Σ Pab
Ri
T(al=) =0 T(o = 1. (57)
Daa
Ag aa
PE
Intuitively, this Markovian process walks over the alternatives according the weights |abb and at each step has a non-zero probability to end the cemetery Now,
A ba ((1-P) (11-))-D--)) ba!
A
Σ (P* (P ba"
A Σ Poas
Pa
Pan-ia Aua
n00
(58)
Therefore,
A Kig = [(albb)).. ((ao
(59)
In other words, A Ka is the probability that a the last alternative to be visited before the random walk is killed, given that started at b. Notice that the alternative may be visited multiple times in those paths. Actually, any path that starts at b and visits a before being killed can be decomposed as the gluing of path that starts b and reaches a, followed by path that starts at and eventually revisits a as its last step before being killed. Therefore,
1 ba Ki I A
(60)
Since Ka > 0, and since is symmetric, we finally obtain
-1 oa 2 ab
(61)
H Proof of Theorem
Proof. Fix an arbitrary À and let μ 52 A. There exists permutation matrix P, and integer partition A₁ + .... + A₁ = A, such that
x TIT sI)
XI
(62)
x² + II
(63)
P. |uI + block diagonal .A....JA.))) p-1
(64)
P. block diagonal(udA,, + JA....... pl JAa) p-1
(65)
where every JA, is matrix of size A, A, with all its entries set to 1. Up to renaming the alternatives, we can assume, without loss of generality, that I
We notice that the all-one matrix J, say of size A, satisfies J2 = AJ, and
(μI + 4
A+p 1)) =1
(66)
Therefore, X is invertible and x Block_diagone
( (LA,
1 JA.)
A JA.)...., ..... (LA.
A p JAi))
(67)
Thus, X-1
1 is super-Laplacian. This proves that (x sl)" is a diffusion embedding. The monotonicity of GBT, follows from Theorem I
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification; The introduction summarizes the claims in the "Contributions" paragraph, with
references to the mathematical statements in the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the
results can be expected to generalize to other settings. is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of this work in dedicated paragraph of the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Each mathematical statement has a proof, either in the main body or the appendix. Pointers to the location of proofs are provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
21

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes]
Justification: All the code and instructions necessary to reproduce the experiments are provided in the Supplementary Material. Section also provides detailed explanation of each step of the numerical experiments, along with the computing environment.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of soued-source models, it may be that access to the model is limited in some way (e.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: Again, all the code and instructions necessary to reproduce the experiments provided in the Supplementary Material. The code will be uploaded on GitHub if the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cttpcccccppccccc/ccc..cc..ccccct
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
22

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines httpps////ips..cppbbbl//e
wwr.iisssssipiiiiiic for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: A description of the setup used to provide Fig. and is provided in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is
necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material. Experiment Statistical Significance
Question: Does the paper report eтror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The plots provide the normalized mean squared error the estimators, averaged over high number of repetitions (100 repetitions). As such, it is unclear what additional insight variance metrics would bring to the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to
library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: The experiments require little computing power, details on RAM, CPU provided
in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neur/ee http:/ppp....pp.ii..hh....ooooaa
Answer: [Yes]
Justification:
Guidelines:
We reviewed the NeurlPS Code of Ethics, and found that none of the problematic cases in it conform with the numerical experiments.
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation
from the Code of Ethics
The authors should make sure to preserve anonymity (e.g.. g.. if there is a special consideration due
laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We mention broader impacts both at the end of the Related Works section of the Introduction, and in the limitations paragraph of the Conclusion
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied par- ticular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
legitimate to point out that
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
24

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not rely on existing assets, all experiments are based on synthetic
data.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should
be provided. For popular datasets, pwwpppappiaaa... thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not conduct crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
25

---

Answer:
[NA]
Justification: Again, the paper does not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs, nor does the writing of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMSs any important, original, or non-standard components.
Please refer our L.M policy httc:p///heuripsscc//o/feeeeeee should or should not be described.
for what
26


Reviews:
[{"review_id": "ZMRgzQ9ZjS", "reviewer": "Reviewer_yQmu", "summary": "This paper addresses the overlooked issue of monotonicity in preference learning, where models should assign higher value to preferred alternatives. While existing models like LLMs often violate this, the paper introduces a novel class of Linear Generalized Bradley-Terry models with Diffusion Priors, which ensure monotonicity under specific embedding conditions and can generalize to unseen comparisons. Experiments confirm improved accuracy, especially in low-data regimes.", "questions": "Q1. To provide more convincing real-world numerical evidence, can the experiments be designed and operated at an industrial level, e.g., using UltraFeedback or Anthropic-HH datasets?\n\nQ2. Generalisation of BT model has been discussed in various formats and contexts.\n\nIn many real-world datasets, binary reward models, such as the Bradley-Terry (BT) model, have been known to be vulnerable to 'intransitivity' risk because they rely on scalar variables, assuming all preferences are transitive. \n- The literature below studies representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold, and presents a generalized version of the BT model that considers the intransitivity of human preferences.\n- https://arxiv.org/abs/2409.19325 (Duan et al, 2017) : Quantitative investigation into the importance of generalization and a generalized model that focuses on explainable representation learning.\n- 10.1109/ICPR48806.2021.9412462 (Gu et al, 2021): A low-rank matrix approach for the generalization.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. This paper focuses on a neglected but important data-driven issue in LLM fine-tuning, that is, the generalization of reward evaluations.\n2. Theoretical contributions are valid and identify conditions under which monotonicity can be guaranteed in linear comparison-based models with diffusion priors.\n\nWeakness:\n1. The experiments are not organized at an industrial level. The numerical evidence is on a miniature scale, insufficient for technical audiences.", "was_revised": true, "final_justification": "The authors have provided a candid response and meaningful motivation of the paper. The perspective of 'generalization of preference learning models with monotonicity' is essential as a scientific area for decision science and the modern LLM industry. I will look forward to seeing a deep dive into this topic in the future; therefore, I raise my score to Accept.", "rebuttal": "We thank the reviewer for the time they took to review our paper, and for their remarks.\n\n> Q1. To provide more convincing real-world numerical evidence, can the experiments be designed and operated at an industrial level, e.g., using UltraFeedback or Anthropic-HH datasets?\n\nOur models apply to any problem that can be formulated in terms of pairwise preferences between alternatives, alternatives for which we want to infer scores (monotonically). Since both datasets provide comparisons, our models could, in principle, be applied to infer scores for the chosen responses given a prompt.\n\nHowever, these datasets have been designed for LLM applications, where the goal is not so much about inferring scores for the chosen responses, but more about computing the probability of generating a response given a prompt. It is expected that human feedback can be leveraged to adjust these probabilities in the \"expected way\". Monotonicity is one formalization of this \"expectation\". \n\nWe believe that monotonicity of LLMs should be studied more thoroughly. As mentioned in our introduction, the fact that the models referred in [25,30,1] fail to guarantee monotonicity is one of the core motivations behind the present work. We chose to focus, in this paper (line 46), on models that are well-structured, and simple enough to analyze, for which we *can* provide monotonicity guarantees. Given the complexity of LLMs, we reserve the study of their monotonicity for distinct work.\n\n> Q2. Generalisation of BT model has been discussed in various formats and contexts. In many real-world datasets, binary reward models, such as the Bradley-Terry (BT) model, have been known to be vulnerable to 'intransitivity' risk because they rely on scalar variables, assuming all preferences are transitive. The literature below studies representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold, and presents a generalized version of the BT model that considers the intransitivity of human preferences. https://arxiv.org/abs/2409.19325 (Duan et al, 2017) : Quantitative investigation into the importance of generalization and a generalized model that focuses on explainable representation learning. 10.1109/ICPR48806.2021.9412462 (Gu et al, 2021): A low-rank matrix approach for the generalization.\n\nWe focus our attention on monotonicity of scores for various BT-related models. Indeed, since these probabilistic models infer *scores*, they implicitly assume that intransitive judgements are due to noise. We do have an ongoing line of research that consider approaches that try to better model human decision-making. We were not aware of the references you mention, will include them in the related work, and consider them for our own research. Thank you.\n\nWe hope to have addressed all your concerns. We remain at your disposal may you have any further questions or require additional information. We would be grateful if you could consider revising your score based on the answers we provided."}, {"review_id": "5e3j1JICpY", "reviewer": "Reviewer_Ag91", "summary": "The paper proposes a Generalised Linear Bradley-Terry model with a specific set of embeddings they name - diffusion embeddings. The paper proves that these embeddings result in monotonic behaviour during preference learning.", "questions": "See Weaknesses above", "limitations": "The authors highlight the constraint that the proposed method is dependent upon diffusion embeddings.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The setting studied by the authors is novel to the best of my knowledge.\n- The paper lays out rigorous theoretical proofs of its claims and supports these with experimental results.\n- The work is mostly clear and easy to follow.\n\n**Weaknesses**\n\nI recommend a weak reject at the moment because I do not believe the motivation for the work aligns with the method proposed.\n\n- The authors motivate the work through DPO, GPO and other RLHF works that focus on alignment within the LLM literature, however I am uncertain if the results presented in this paper align with that motivation. The authors do not present any experiments on an LLM setting and the class of problems considered in this work is not easily extended to the class of LLM models, as to the best of my knowledge the setting is akin to tabular with an embedding for each alternative A. I may well have misunderstood this aspect of the work though - thus my low confidence score.\n- The final paragraph of the introduction states that this work finds direct applications within the collaborative scoring of social media content - I am unfamiliar with this setting but would be keen to understand more as the work may present better when motivated in this context.\n- The experiments lack baselines against other preference optimization work - particularly other works from the collaborative content social media content literature would enrich the work and provide further comparisons to evaluate the method against.\n- Some clarity issues in the paper - in line 185 the regularisation $\\mathcal{R}$ overrides the notation for the comparison value defined in line 107.", "was_revised": true, "final_justification": "The authors have addressed my concerns, committed to improving the clarity of the motivation in the introduction and included further empirical evidence in line with other reviewer concerns. For these reasons I increase my score from a 3 to a 4.", "rebuttal": "We thank the reviewer for the time they took to review our paper, and for their remarks.\n\n> The authors motivate the work through DPO, GPO and other RLHF works that focus on alignment within the LLM literature, however I am uncertain if the results presented in this paper align with that motivation. The authors do not present any experiments on an LLM setting and the class of problems considered in this work is not easily extended to the class of LLM models, as to the best of my knowledge the setting is akin to tabular with an embedding for each alternative A.\n\nWe apologize if our paper gave the impression that we would focus on LLM.\nWe only mentioned DPO/RLHF/LLM to highlight (1) the growing importance of preference learning in AI, and (2) the bizarre features of many of today's preference learning deployments, such as in the case of LLM. \nThe focus of our paper is given by line 46, which does not include language model alignment.\nAlthough the generalization may prove useful of the alignement of AI models in the long run, our motivation stems from the direct applications that GBT and its extensions find in social media content recommendation, as discussed in the original GBT paper and the related work [13, 14]. In particular, the motivation for alterantives to current social media recommender systems is provided in [14].\n\n> The final paragraph of the introduction states that this work finds direct applications within the collaborative scoring of social media content - I am unfamiliar with this setting but would be keen to understand more as the work may present better when motivated in this context.\n\nThe main references we give are [13] and [14], which refer to a project called Tournesol.\nThey already reportedly use the generalized Bradley-Terry model, which guarantees monotonicity but not generalization.\nWe believe that our work could help this project improve their recommendation systems which are based on community preferences, without sacrificing monotonicity.\n\n> The experiments lack baselines against other preference optimization work - particularly other works from the collaborative content social media content literature would enrich the work and provide further comparisons to evaluate the method against.\n\nTo the best of our knowledge, GBT and BT are the only comparison-based preference learning models that provide monotonicity guarantees. Building models from comparisons explicitly provided by users seems a highly desirable property in social media content literature. We thus do not consider the usual recommender system's litterature, which are the backbone of the current social media, and rely on data that is not explicitly provided by users.\n\n> Some clarity issues in the paper - in line 185 the regularisation $\\mathcal R$ overrides the notation for the comparison value defined in line 107.\n\nWe have fixed right away this notation issue. Thank you.\n\nWe hope to have addressed all your concerns. We remain at your disposal may you have any further questions or require additional information. We would be grateful if you could consider revising your score based on the answers we provided."}, {"review_id": "rRjRbTUwZe", "reviewer": "Reviewer_XoXh", "summary": "The authors consider the problem of generalizable preference learning from pairwise comparisons. They focus on a desirable monotonicity property, which holds for a preference learning algorithm if observing data that favors an item $a$ only causes the learned value of $a$ to increase. Surprisingly, Bradley-Terry with linear utilities violates monotonicity. However, the authors show that if item embeddings satisfy certain properties, then a generalized Bradley-Terry model (GBT) does satisfy monotonicity. In simulation experiments, the authors examine the probability that random Gaussian embeddings satisfy their \"goodness\" property (unfortunately, this is small when there are few embedding dimensions relative to items). In experiments on synthetic data, they demonstrate how generalization across items is achieved through the model's Laplacian regularization and item embeddings.", "questions": "1. If I'm understanding the definition of monotonicity correctly, any dataset operation that favors $a$ should increase the learned score score $\\theta_a$ of $a$. Is this sufficient to cover the \"buggy\" example in the introduction? Suppose adding a comparison where $a$ beats $b$ does increase the score of $a$, but it also increases the score of $c$ by even more, causing $a$ to now perform worse against $c$. (I know this is a weird learning algorithm, but perhaps $c$'s embedding indicates it has all the good features of $a$ and none of the bad features of $b$.) How is this avoided by monotonicity? \n\n2. Is there some intuition for what embeddings are diffusion embeddings? Are the embeddings in the introduction example not diffusion embeddings, and is this possible to tell without computing inverses?\n\n3. What parts of the model are required for monotonicity? Can we get rid of the $L_2$ regularization? The Laplacian regularization?\n\nMinor:\n- line 56: \"monotony\"\n- Line 68: another relevant choice citation is McFadden's multinomial/conditional logit\n- Line 149: has an extra comma\n- Line 162: this should be referencing Proposition 1, right?\n- $\\mathcal R$ is used both for the set of comparison scores and the regularization term\n- Smoothed rather than smoothened?", "limitations": "Nice concise limitations section. I don't see this work having negative societal impacts.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The contribution is clear and significant\n2. The writing and notation are good\n3. The problem is timely and interesting\n\nWeaknesses:\n1. A few minor points that could be made clearer (see questions)\n\nQuality: Good quality, well situated in the literature, good mix of theory and experiments, well-structured. \n\nClarity: The clarity is high overall. A few points could be better explained (see questions) \n\nSignificance: Important and relevant to preference learning.\n\nOriginality: The model itself uses standard techniques (utility = linear function of features, Laplacian regularization), but the properties of embeddings required for monotonicity and the associated analysis are original to my knowledge. I'm not sure I'd call GBT with diffusion priors a \"new model\"; e.g., see (1) https://doi.org/10.1017/nws.2023.20 and (2) https://arxiv.org/abs/1703.07520 for other work applying item features and Laplacian regularization to learning preference models. The only difference between GBT with diffusion priors and the Laplacian-regularized conditional logit model in (1) is the use of the comparison score $r$ rather than a fixed choice of preferred item (and the focus here on pairwise comparisons). I don't think this is an issue; the major contribution of this work is the analysis of monotonicity. I would just tone down the claimed novelty of the model.", "was_revised": true, "final_justification": "I only had some minor questions/comments/concerns and continue to feel that this is a nice paper, well-reflected by a 5.", "rebuttal": "We thank the reviewer for the time they took to review our paper, and for their remarks.\n\n> If I'm understanding the definition of monotonicity correctly, any dataset operation that favors $a$ should increase the learned score $\\theta_a$ of $a$. Is this sufficient to cover the \"buggy\" example in the introduction? Suppose adding a comparison where $a$ beats $b$ does increase the score of $a$, but it also increases the score of $c$ by even more, causing $a$ to now perform worse against $c$. (I know this is a weird learning algorithm, but perhaps $c$'s embedding indicates it has all the good features of $a$ and none of the bad features of $b$.) How is this avoided by monotonicity?\n\nThis is a very interesting remark. It seems that you formulate an even stronger monotonicity property: if $a$ is favored over $b$ then the score of $a$ should increase *and* no other alternatives increases by a larger amount. We have not considered this formulation. However, because our model relies on diffusion, we believe this property is likely to be true for GBT with diffusion embeddings and laplacian regularization. The intuition is that favoring $a$ over $b$ is like heating $a$ and cooling $b$ (both by a unit amount) and letting the heat diffuse throughout the (similarity) network, so no other alternative should have its \"temperature\" (score) vary by a larger amount. Of course, this statement is not a proof, and we would need to check it. Thank you for your input.\n\n> Is there some intuition for what embeddings are diffusion embeddings? Are the embeddings in the introduction example not diffusion embeddings, and is this possible to tell without computing inverses?\n\nThis is actually a question we decided to study after the submission of the present paper. More precisely, as you noticed, diffusion embeddings (and good embeddings) are defined in an algebraic way. We have recent unpublished results that aim at characterizing diffusion embeddings in a combinatorial way. This topic is complex enough to be difficult to discuss in more details here, and to make a contribution on its own. So, if that is okay with you, we would like to stay at that description.\n\n> What parts of the model are required for monotonicity? Can we get rid of the $L_2$ regularization? The Laplacian regularization?\n\nMonotonicity requires that embeddings be \"good\", as per Definition 7, and $\\sigma > 0$.\nThe Laplacian regularization defined by matrix $L$ can be removed without hindering monotonicity, as the null matrix is a Laplacian matrix. \nThe $\\ell_2$ regularization cannot be discarded. Indeed, it is crucial for the GBT estimator to be well-defined (in particular, unique), even without embeddings in the initial GBT work [9]. In addition, the regularization parameter $\\sigma$ plays a key role in the technical Lemma 1.\n\n> Minor: \n\n> Line 162: this should be referencing Proposition 1, right?\n\nOur intention was for Proposition 1 to recall [Th. 2, 9], and mention that the forthcoming Th. 3 generalizes this result. A subtlety is that the original GBT [9] assumes that two elements (a, b) are compared at most one time, while we account for more that one comparison between (a, b) in the dataset.\nWe have updated the sentence to clarify this subtlety. \n\n> [other comments]\n\nThank you for taking the time to note these points. We have addressed them right away in the manuscript.\n\nWe hope to have addressed all your concerns. We remain at your disposal may you have any further questions or require additional information. We would be grateful if you could consider revising your score based on the answers we provided."}, {"review_id": "fQzULsv9u0", "reviewer": "Reviewer_DMrc", "summary": "The paper addresses to key problems in preference-learning models, monotonicity and score-generalization. \nIn particular, the authors tackle both by extending a well-established model (GBT) to include embeddings of the alternatives and a more general regularization term.\nThe authors then prove the conditions necessary to make their extended model monotonic and show how every diffusion embedding as defined in def5 satisfies these conditions.\nFinally, the authors perform a limited experiment to showcase their model against GBT.\n\n\n\n\n\n\n\n\n\n\nThe paper addresses the problem of monotonicity, or lack thereof, in preference learning models. \nIn particular, the authors propose a new class of preference learning algorithms, based on the Generalized Bradly-Terry model, which also considers embeddings of the alternatives. \nWith appropriate conditions on the abovementioned embeddings the authors show that they can guarantee monotonicity.", "questions": "Major questions:\nSee the weaknesses above.\n\nClarifications:\n- L117--120: There might be a contradiction here. If EXCHANGE replaces $(a_n, b_n, r_n)$ with its opposite $(a_n, b_n, -r_n)$, I don't see why preference algorithms should interpret the two comparison samples identically. At the limit, one can exchange all of the comparison samples with their opposites, most certainly affecting training.\n- L140: The authors should provide proof that $\\preceq_a$ is indeed a preorder; in particular, I am skeptical about reflexivity, as I cannot see a way to define an operation (infinitely) favors $a$ while leaving a dataset $\\mathbf D$ unchanged.\n- L163: In what sense is proposition 1 a generalization of [9]?\n- L274: How does $H$ depend on $L$? The sentence build like the two are indeed related, but I cannot see how.\n- L284: Shouldn't $x$ be $(L + \\tilde H)$-good?\n- L467: Based on eq7, shouldn't it be $\\mathcal L_\\lambda(\\beta|\\mathbf D, o) - \\mathcal L(\\beta|\\mathbf D) = \\lambda \\cdot (\\Phi_f(\\theta_{ab}) - r\\theta_{ab})$?\n\nSpelling and notation: \n- L56: \"monotony\" should be \"monotonicity; this may have happened in other parts of the manuscript.\n- L107, eq3: $\\mathcal R$ is used with different meanings.\n- L116: consider reserving $\\mathcal O$ for its usual meaning in asymptotic analysis.\n- L130, eq6, eq7, L262: the notation for UPDATE is inconsistent; this might be the case for the other operations as well. \n- L139 (and others): Given the definition of $\\Omega$, it's equivalent (and formally correct) to replace $\\geq$ with $\\in$. \n- L149: \",.\" at the end.\n- eq3: $\\theta_b(\\beta)$ is not defined; is it $\\theta(\\beta)_b$? The first symbol is not defined.\n- eq4: $x_{ab}^T\\beta$ is confusing for two reasons: (1) if it means $(x^T)\\_{ab}$  why not use $x_{ba}$? (2) $\\beta$ was defined on L181 as a vector and $x^T_{ab}$ looks like a scalar, meaning that $x_{ab}^T\\beta \\in \\mathbb  R^D$, which is in conflict with the rest of the terms in the equation. \n- L255, 226: proposes, provides.\n- L250: $o_\\lambda$ is not defined.\n- eq9: What is $e$? L305 defines $I$ as the identity matrix. \n- The figures are not saved in a vector format, making the labels hard to read when the paper is printed out.", "limitations": "The authors discuss some limitations of their model, especially regarding the extension to non-diffusion embeddings.\nWhat is missing here, in my opinion, is a discussion on the drawbacks stemming from trying to use the model in practice (W3).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## S1 - Interesting model\nExtending GBT to allow for score-generalization is potentially impactful for the community, as it addresses a strong motivation.\nAlso, the fact that every diffusion embedding can be used within the model potentially makes it flexible in tackling different scenarios.\n\n## W1 - Notation\nThe mathematical notation is often inconsistent and confusing, making it quite hard to follow the text and, especially, the proofs.\nNotable issues include:\n1. The indiscriminate usage of the same letter $\\theta$ for both a matrix and a vector; both $\\theta_{ab}$ and $\\theta_a$ indicate a real number in different parts of the text.\n2. An inconsistent usage of typefacing. Matrices are sometimes indicated with lowercase letters ($x$), greek letters ($\\theta$), and uppercase letters ($L, H$). On the contrary, the same typeface is used for multiple different kinds of objects; examples are $A, D, N \\in \\mathbb N$ and $L$ being a matrix, as well as $\\beta$ being a matrix and $\\sigma$ being a real number. \n3. Undefined symbols, such as $e_{ab}$ in Lemma 1; I am guessing $e$ is the identity matrix of some dimension, but in Definition 3 the symbol $I$ was introduced for that.\n4. Inconsistencies in Lemma 1 and its proof, where both $\\frac{d\\theta^\\*(\\lambda)}{d\\lambda}$ (eq9) and $\\frac{d\\theta^*\\_\\lambda}{d\\lambda}(\\mathbf D, o)$ (eq30) are used to (potentially?) indicate the same object.\n\n## W2 - Experiments\nThe experimental evaluation is insufficient to showcase the benefits and potential drawbacks of the proposed model over other approaches.\nIn my opinion, both of the following issues with the experimental setup stem from W3. \n\n### W2.1 Competitors\nThe experiments omit comparisons with other potential competitors such as the ones described in L79--87.\n\n### W2.2 Data-generating process\nTo my understanding, the data-generating process described in Section 5.2 aligns perfectly with the proposed model's assumptions, making it a very unfair comparison ground for standard GBT. \n\n## W2.3 - Error bars\nThe authors report in the checklist that they are purposely not reporting error bars in their figures, as they are considering an arguably large number of repetitions (100). I believe error bars (preferably, standard deviation) should instead be included, as it helps discerning how significant the differences between the models' performances are. \nIdeally, some statistical analysis of the results would also help.\n\n## W3 - Assumptions of the model\nAs described in Section 3, the model assumes access to a known embedding matrix $x$, which is a strong and unrealistic assumption in most practical scenarios. \nThe sensitivity of the method to this assumption is evident in Figure 2a, where not choosing the “correct” embedding lead to significant performance degradation. The \"correct\" embedding is the one aligning with the data-generating process.\nWithout a mechanism to learn or robustly adapt this embedding, the practical utility of the method remains limited.\nI also wonder, if learning the embedding is possible, whether that would make the optimization problem intractable or very hard to optimizer, given the increased number of variables in the model.", "was_revised": true, "final_justification": "The criticalities I found in the paper were mainly about clarity (notation) and the weak experimental comparison provided in Section 5. After discussing with the authors, I am now satisfied by the improvements made to the notation and am now convinced that the experimental setup is appropriate for publication, although far from perfect. \n\nI believe the theory developed in the paper is a worthy addition to the field and therefore recommend to accept the paper.", "rebuttal": "We thank the reviewer for the time they took to review our paper, and for their detailed remarks.\n\n## W1 - Notation\n\n> [...] Issues include:\n\n> The indiscriminate usage of the same letter \\theta for both a matrix and a vector; both \\theta_{ab} and \\theta_a indicate a real number in different parts of the text.\n\nThe variable $\\theta$ is introduced in section 2.3, (eq. 2) states that $\\theta \\in \\mathbb{R}^A$. We admit that the notation $\\theta_{ab} = \\theta_a - \\theta_b$, specific to BT and GBT literature, is confusing. We will update the paper accordingly.\n\nVariable $\\theta$ should not be a matrix. We have checked against this, but did not find occurences. Could you please point to places where you understood that $\\theta$ was a matrix?\n\n> An inconsistent usage of typefacing. [...]\n\nIndeed, we have not strictly followed a typesetting convention, and we understand that this may reduce the paper readability for people used to such a convention. We can add a \"Notation\" table or section that summarizes our notations.\n\n> Undefined symbols, such as e_{ab} in Lemma 1 [...]. Inconsistencies in Lemma 1 and its proof [...].\n\nThanks for pointing this out. We added in lemma 1 the following \"There, $e_a$ denotes the $a$-th vector of the cartesian basis, and $e_{ab} = e_a-e_b$\", and have fixed notation (eq9) and (eq30).\n\n## W2 - Experiments\n\n### W2.1 Competitors\n\n> The experiments omit comparisons with other potential competitors such as the ones described in L79--87.\n\nLet us stress that the main interest, and challenge, of this work is to guarantee the monotonicity of the learned model. Alternative methods of l.79-87 (Deep neural network generalizations of BT, RLHF, DPO, etc) do not benefit from any monotonicity guarantee. Worse, forms of non-monotonicity was observed during the finetuning of LLMs; see eg [Hong et al.; Orpo [...], 2024].\nHence our choice to limit our numerical illustrations to methods with monotonicity guarantees.\n\n### W2.2 Data-generating process\n\n> To my understanding, the data-generating process described in Section 5.2 aligns perfectly with the proposed model's assumptions, making it a very unfair comparison ground for standard GBT.\n\nThe matrix $x$ used by the learning model indeed corresponds to the matrix of the data-generating process.\nGBT with embedings thus performs drastically better than GBT. While that may be seen as unfair, this also speaks to the interest of this contribution: adding contextual, or descriptive, information to the GBT model, that until now could only leverage comparisons. \nIn view of the well- vs. mis-specified results in statistics, we believe that replacing the exact embedding matrix by an approximation would degrade the performance the GBT with embedding model in a limited way, thus maintaining it above GBT. We detail this point in W3. \n\n### W2.3 - Error bars\n\n> The authors report in the checklist that they are purposely not reporting error bars in their figures, as they are considering an arguably large number of repetitions (100). I believe error bars (preferably, standard deviation) should instead be included [...].\n\nWe understand that plots with error bars showing standard deviations are standard in the ML community. We have produced such plots.\nNevertheless, we cannot report them here nor provide a link to our images, given the rules for this year's rebuttal format. We report a table version of these plots below.\n\nFigure 2a. nMSE as a function of D for A=25 alternatives and N=500 comparisons over 100 seeds.\nThe following table reports mean ± 1.96 * stdev / sqrt(n_seeds).\n| Dimension D |  GBT & encoding  |       GBT        |     Encoding     |\n|-------------|------------------|------------------|------------------|\n|      2      | 0.0497 ± 0.00605 | 0.0801 ± 0.00676 |  0.39 ± 0.0432   |\n|      4      | 0.0369 ± 0.00449 | 0.0855 ± 0.00614 |  0.248 ± 0.0314  |\n|      8      | 0.0269 ± 0.00249 |  0.102 ± 0.0079  |  0.125 ± 0.0124  |\n|     16      | 0.0182 ± 0.00163 |  0.191 ± 0.0118  | 0.0377 ± 0.00357 |\n|     20      | 0.0183 ± 0.00169 |   0.2 ± 0.0112   | 0.0271 ± 0.00274 |\n\nFigure 2b. nMSE with respect to the number of comparisons N for A=20, D=10, and 1000 seeds.\nThe following table reports mean ± 1.96 * stdev / sqrt(n_seeds).\n|  N  |  GBT & encoding   |       GBT       |\n|-----|-------------------|-----------------|\n|  1  |  0.934 ± 0.00974  | 0.983 ± 0.00125 |\n| 71  |  0.113 ± 0.00348  | 0.454 ± 0.00444 |\n| 141 |  0.0636 ± 0.0019  | 0.29 ± 0.00426  |\n| 191 | 0.0508 ± 0.00145  | 0.231 ± 0.0039  |\n| 291 | 0.0301 ± 0.000989 | 0.182 ± 0.00378 |\n\nNote that the values reported above differ slightly from the plots. The reason is that we have improved our codebase (fixing MLE optimization and unstable gradient). Importantly, the behavior of the curves remain identical.\n\nWe believe that the main contributions of the paper is extending GBT with (1) embeddings, so that it may generalize, and (2) Laplacian regularization, to account for known relationships all the while maintaining the crucial property of monotonicity.\nWe agree that a statistical analysis is desirable in view of the experiments, but feel that it would not directly support our message.\n\n## W3 - Assumptions of the model\n\n> As described in Section 3, the model assumes access to a known embedding matrix x, which is a strong and unrealistic assumption in most practical scenarios. [...]\n\nIn this work, we do not study the statistical performance of the proposed extension of GBT; we rather focus on monotonicity, which is the content of Section 3.These monotonicity guarantess holds independently of how the data is generated.\n\nWe agree that statistical performance is interesting. We peek at the statistical performance of our extension of GBT against GBT in the experiments, but leave thourough experiments for the statistical study of these estimators. Again, we believe it is outside the scope of our paper, which focuses on monotonicity, a critical property for preference learning, and one that receives little attention.\n\nNote that we are currently investigating statistical aspects. Preliminary results indicate that the statistical performance of the GBT with embeddings model degrades in a controlled manner when the embedding matrix $x$ is increasingly distant from the true embedding $x^\\dagger$. Learning the embedding matrix seems possible with, eg, hierarchical bayesian learning.\n\n## Clarifications\n\n> L117--120: There might be a contradiction here [relative to EXCHANGE].\n\nThank you, this is a typo. We meant: EXCHANGE replaces $(a_n, b_n, r_n)$ with $(b_n, a_n, -r_n)$.\nWe thus consider in this work preference learning algorithms that treat (a,b,r) and (b,a,-r) in the same way. Note that this assumption is valid for BT, GBT, and the proposed generalization of GBT.\n\n> L140: The authors should provide proof that $\\preceq_a$ is indeed a preorder [...].\n\nWe have added the following setence after the claim that $\\succeq$ is a preorder: \n\"Indeed, $\\preceq_{a}$ is reflexive: any dataset $\\mathbf{D}$ equals $o(\\mathbf{D})$ with $o = UPDATE_{n, r}$ with $n=1$, and $r=r_1$. The relation $\\preceq_{a}$ is transitive: if $\\mathbf{D}_1 \\preceq_a \\mathbf{D}_2$ and $\\mathbf{D}_2 \\preceq_a \\mathbf{D}_3$, then there exists operations $o_1$ and $o_2$ that favor $a$, such that $bD_1 = o_1(\\mathbf{D}_2)$, and $\\mathbf{D}_2=o_2(\\mathbf{D}_3)$; thus $\\mathbf{D}_1=o_1 \\circ o_2(\\mathbf{D}_3)$, where $o_1\\circ o_2$ is an operation that favors $a$ by definition 1 so that $\\mathbf{D}_1 \\preceq_a \\mathbf{D}_3$.\"\n\n> L163: In what sense is proposition 1 a generalization of [9]?\n\nTheorem 2 of [9] guarantees monotonicity of $GBT_{f, \\sigma}$ for datasets which contain at most one comparison between any two elements $a$ and $b$. In Theorem 3, we provide monotonicity for datasets which may contain repeated repetitions between $a$ and $b$. We have reprased Proposition 1 to make this clear.\n\n> L274: How does $H$ depend on $L$? [...]\n\nIndeed, $H$ does not depend on $L$. The matrix $L$ is used below (L279), and should be declared there.\nWe have reformulated as follows:\n- L274: \"Note first that, the Hessian $H$ of [...] is also a Laplacian matrix\".\n- L279: \"Therefore, given any prior Laplacian matrix $L$, the matrix $L + \\tilde{H}$, [...]\".\n\n> L284: Shouldn't $x$ be $(L + \\tilde H)$-good?\n\nThis is a typo, thank you. We have reformulated, at line 284: \"This is precisely implied by $x$ being good.\"\n\n> L467: [missing $\\lambda$ at (eq7)]\n\nA factor $\\lambda$ was indeed missing, we have added it. Thank you.\n\n## Spelling and notation\n\n> [..] notation for UPDATE is inconsistent [...]\n\nWe have fixed the notation of UPDATE and APPEND, they are now consistent.\n\n> L139 (and others): Given the definition of $\\Omega$, it's equivalent (and formally correct) to replace $\\geq$ with $\\in$.\n\nWe do not understand. There is no $\\Omega$ or $\\geq$ at l.139. Can you provide precisions please?\n\n> eq3: $\\theta_b(\\beta)$ is not defined; is it $\\theta(\\beta)_b$? The first symbol is not defined.\n\nIndeed. We have added a sentence that clarifies the definition of $\\theta$ as a function of $\\beta$ into $\\mathbb{R}^A$, and that explains $\\theta_b(\\beta)$.\n\n> eq4: $x_{ab}^T\\beta$ is confusing [...]\n\nWe admit the notation is confusing. We have clarified in the text that $x_a \\in \\mathbb{R}^D$ is the $a$-th column of $x$, for any $a \\in [A]$ at line 182, and that $x_{ab} = x_a - x_b$.\n\n> eq9: What is $e$? [...]\n\nIn equation 9, $e_a$ denotes the $a$-th vector of the cartesian basis of $\\mathbb{R}^A$, and $e_{ab} = e_a - e_b$. We have added this explanation in the lemma.\n\n> The figures are not saved in a vector format [...]\n\nWe updated figures to vector format, and made the labels more readable.\n\nWe have fixed the remaining comments, thank you.\n\nWe hope to have addressed all your concerns. We remain at your disposal may you have any further questions or require additional information. We would be grateful if you could consider revising your score based on the answers we provided."}]

Author rebuttal:

Meta-review: In this paper, authors advance the understanding of the set of models with generalization ability that are monotone, i.e., authors proposed a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Experimental results verified the effectiveness of the proposed method.

The final rating of this paper is 2 borderline accept, 2 accept.

The strength of this paper given by reviewers are:
1) paper solved an important data-driven issue in LLM fine-tuning (Reviewer yQmu, XoXh)
2) solid contribution (Reviewer yQmu, Ag91, XoXh)
3) the setting is novel and interesting. (Reviewer Ag91, DMrc)
4) the work is clear and easy to follow (Reviewer Ag91, XoXh)

The weaknesses are:
1) experiments are not organized at an industrial level (Reviewer yQmu) 
2) do not believe the motivation for the work aligns with the method proposed. (Reviewer Ag91)
3) A few minor points that could be made clearer (Reviewer XoXh)
4) writing could be improved (Reviewer DMrc)
5) experimental evaluation (Reviewer DMrc)

After rebuttal, reviewer yQmu, Ag91 convinced by authors and gave a borderline accept rating. Reviewer XoXh, DMrc  still had some minor concerns but think this is a nice paper and give accept rating. 

Given all these AC decided to accept this paper.
Decision: Accept (poster)