Distributional Preference Alignment of LLMs via Optimal Transport
Authors: Igor Melnyk
Youssef Mroueh
Brian Belgodere
Mattia Rigotti
Apoorva Nitsure
Mikhail Yurochkin
Kristjan Greenewald
Jiri Navratil
Jarret Ross
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Alignment
Optimal Transport
stochastic dominance
Primary area: optimization
TL;DR: We propose AOT a  distributional alignment of LLMs via Optimal Transport.

Abstract: Current LLM alignment techniques use pairwise human preferences at a sample level, and as such, they do not imply an alignment on the distributional level. We propose in this paper Alignment via  Optimal Transport (AOT), a novel method for distributional preference alignment of LLMs. AOT aligns LLMs on unpaired preference data by making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. We introduce a convex relaxation of this first-order stochastic dominance and cast it as an optimal transport problem with a smooth and convex cost. Thanks to the one-dimensional nature of the resulting optimal transport problem and the convexity of the cost, it has a closed-form solution via sorting on empirical measures.  We fine-tune LLMs with this AOT objective, which enables alignment by penalizing the violation of the stochastic dominance of the reward distribution of the positive samples on the reward distribution of the negative samples. We analyze the sample complexity of AOT by considering the dual of the OT problem and show that it converges at the parametric rate. Empirically, we show on a diverse set of alignment datasets and LLMs that AOT leads to state-of-the-art models in the 7B family of models when evaluated with Open LLM Benchmarks and AlpacaEval.  Code for $\mathsf{AOT}$ is available in the Hugging Face  TRL library  \url{https://ibm.biz/AOT_TRL}.

Distributional Preference Alignment of LLMs via Optimal Transport
Igor Melnyk*, Youssef Mroueh", Brian Belgodere', Mattia Rigotti, Apoorva Nitsure,
Mikhail Yurochkin, Kristjan Greenewald, Jiri Navratil, and Jarret Ross
IBM Research
MIT-IBM Watson Al Lab
Abstract
Current LLM alignment techniques pairwise human preferences a sample level, and as such, they do not imply an alignment on the distributional level. We propose in this paper Alignment via Optimal Transport (AOT), novel method for distributional preference alignment of LLMs. AOT aligns LLMs on unpaired pref- erence data making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. We introduce a convex relaxation of this first-order stochastic dominance and cast it optimal transport problem with a smooth and convex cost. Thanks to the one-dimensional nature of the resulting optimal transport problem and the convexity of the cost, it has a closed-form solution via sorting on empirical measures. We fine-tune LLMs with this AOT objective, which enables alignment by penalizing the violation of the stochastic dominance of the reward distribution of the positive samples on the reward distribution of the negative samples. We analyze the sample complexity of AOT by considering the dual of the OT problem and show that it converges at the parametric rate. Empirically, we show on diverse of aligmment datasets and LLMs that AOT leads to saate-of-thh-art art models in the 7B family of models when evaluated with Open L.M Benchmarks and AlpacaEval. Code for AOT is available in the Hugging Face TRL. library https: //s::///a /ibm. biz/AOT_ _RL.
Introduction
Aligning Large Language Models (LLMs) with human preferences a crucial step in making these models safe and having them follow instructions faithfully. By ensuring that LLMs adhere to human preferences, values, ethics, and desired behaviors we can reduce the risk of generating harmful, biased, or inappropriate content.
Reinforcement Learning from Human Feedback, RLHF [Christiano al., 2017, Stiennon et al., 2020, Ouyang et al.. 2022, Bai et al., 2022], achieves this by learning a reward model on human preference data, followed by fine-tuning the LLM to maximize the reward score while staying close the initial reference policy to retain utility from the pre-trained model. Recently. new paradigms departed from RLHF towards direct preference optimization methods such as DPO [Rafailov et al., 2024], SLIC [Zhao et al., 2023], and Identity Policy optimization [Azar et al., 2024]. In these approaches, the reward expressed in terms of the log-likelihood ratio between the LLM policy and the reference model. The training is done on paired preference data, i.e. triplets of prompts, chosen and rejected sentences, where for each prompt a chosen and a rejected sample are available. The training objective is to maximize the margin between the log-likelihood ratio evaluated on the chosen sentence versus
Now with Capital One; Work done while at IBM Research
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Quantil Consetics of Roneth of Difersr, Algresor, Staaglo
Quantic Comper of Margio of Difrrrtt Abgemmnt Strinji
AGT pained
AGT
10
KTO
DRO
Bax (Norlete)
-200
440
Quaie
10
-2F
-400
Reyented
ADT Lepaidd Chosan Ploards
ADT Lepand Ruacted Powards
-200
KTO Choos Reaads
KTQ Repetadd Revoris
DPQ Choer Rowet
-0F
DPO Reyccaa Rewirh
Percenlle
Parconbile
(a) Stochastic Dominance of Reward of Chosen on Rejected: AOT achieves larger margin between the quantile plots of chosen and rejected rewards.
(b) Stochastic Dominance of AOT's optimized pol- icy margin (between Chosen Rejected) on the margin of the reference policy.
AOT in the paired & unpaired settings enables first-order stochastic dominance of the chosen reward distribution on the rejected distribution (a). The margin between the quantiles of chosen and rejected rewards is larger than alternative strategies. In (b), see that AOT's policy chosen to rejected log-likeliine sigllikelihood ratio dominates that ratio for the base model and alternative strategies.
Figure 1:
the log-likelihood ratio rejected sentences. When paired preference data is not available, and the preference data instead takes the form of distinct marginals of chosen prompt/response pairs and rejected prompt/response pairs, we refer to this setup the unpaired data setting. Ethayarajh et al. [2024] used Kahneman & Tversky's prospect theory in the unpaired setting and proposed the KTO method that maximizes the margin between the chosen reward and the average reward of rejected sentences and pushes the reward of a rejected sentence below the average reward of chosen sentences.
In this paper, introduce new distributional optimization method for fine-tuning LLMs from human preference data. Previous work in the paired setting focused on improving the reward of chosen sentences over rejected sentences preference on a distributional level of the chosen marginal on the rejected marginal. In probabilistic terms, we would like to induce stochastic dominance of the reward of chosen sentences the reward of rejected ones. First order Stochastic Dominance (FSD, see e.g. Ogryczak and Ruszczynski, 2002) of random variable X on a random variable Y. means that all quantiles values of X are larger than those of Y. Our main contribution is introducing AOT, Aligument via Optimal Transport, new method that enables distributional alignment. We do so by devising a new AOT objective function that induces in the unpaired setting FSD dominance of chosen reward's 's distribution over rejected reward's distribution. We call this unpaired variant uAOT. In the paired setting, we introduce pAOT that encourages dominance of chosen to rejected log likelihood ratio of the optimized policy on that ratio for the reference base policy. We show that the AOT cost can be cast as one- one-dimensional optimal transport problem that can be solved via sorting and efficiently optimized for the LLM. AOT enjoys also nice statistical proprieties and achieves the parametric rate since its objective can be seen as smooth one-dimensional optimal transport problem. AOT achieves state-of-the-are results on the Alpaca leaderboard [Dubois et al., 2024] using the Merlinite 7B model [Sudalairaj et al., 2024] as a base and scores as the highest 7B model at the time of writing this paper.
per-sample basis. This procedure does not lead to
To introduce the important concepts of our work pictorially, we show in Fig. 1a the quantile plots of the rewards of AOT and alternative alignment strategies (DPO, KTO) for chosen responses (in green) and rejected responses in (red). The quantile plots are estimated on a paired test set. We see that AOT leads to chosen rewards that have larger margins than those of rejected rewards across all percentiles. More importantly, this margin is larger in AOT models than in policies coming from alternative alignment strategies. We then show in Fig. 1b how the AOT aligned policy's 's chosen-to- rejected log-likelihood ratio dominates that same ratio evaluated on the base model's ratio across all percentiles. The distributional alignment induced by AOT ensures a large margin between all quantiles so that the preference reflected not only on average but distributionally. We formalize distributional preference in the next section.
2

---

2 Distributional Preference via First Order Stochastic Dominance
First Order Stochastic Dominance For real random variable Z we denote 122 [0, 1] R the left-continuous inverse of the Cumulative Distribution Function (CDF) Fz:
Qz(p) 112 (n F2 p} for p E [0,1] 1].
Given two random variables Zi and Zz, say that Zi dominates Z2 in the first order if 2 has larger quantiles than Zz for all percentiles p:
21 > Z2 Qzi Q2,(p), Vp E [0,1].
(1)
FSD
Let X' be the space of prompts X and y be the space of responses Y from an LLM conditioned on prompt X E X. The reference LLM is represented probability on ) given a prompt X X. We note the LLM policy we are optimizing by To where a parameter belonging
policy Teer (Y) i... a conditional
bounded parameter space θ de For measure μ E P )) and a mapping r X' x y > IR, we note as Ty/4 the pushforward map of through r. In particular, for empirical measures μ Ei= ((H,91)) we have that r:/4 Di= 57(2).9))
DPO as Pointwise Preference Approach In Direct Preference Optimization (DPO, Rafailov et al., 2024), the reward being optimized by the LLM has the following form
(T(y)>) (0 v) Bog Tre
3 (o)(2())),
where 7(x)) is normalization constant. DPO assumes access paired preference dataset (X, Y4, Y_) ~ where Y+ denotes positive (chosen) response to which we would like to as- sign high reward, and Y- negative (rejected) response which we would like to assign a low reward. This can be formalized as minimizing the logarithmic sigmoid loss:
min - -B BE
log(a y+) - ro(a,yy)) ))))
and since the difference is taken for the same the normalization Z(x) disappears resulting in:
min (oc.:+---aa log (a
T0(4+1z) 3 Blog Too (++ [x)
To Te
log
Tof (y [z)
We can interpret this as a pointwise constraint inducing preference for positive over negative reward outcomes as follows:
To (1(++
( (x) Tret (y-1z))
log
2 log
((r,84.30 ~
(2)
Teel (3+ (z)
DPO can then be interpreted as relaxation of this constraint through the logistic loss, which also suggests other preference optimization algorithms through relaxations using, for example, the hinge loss proposed SLIC [Zhao et al., 2023].
2.1 Distributional Preference via Stochastic Dominance
Our main insight from looking at the pointwise constraint in Eq. (2) is that we can recast it as a distri-
butional constraint in terms of stochastic dominance of the random variable Z log( no (X+1))))
(. (x))
of positive outcomes the random variable Z, log( To (X) of negative outcomes. This is
especially valuable the unpaired setting without access triplets of prompts and positive and nega-
tive responses as required by DPO. This is indeed the same setting considered by KTT [Fthayarajh
al., 2024]. The following paragraph formalizes this unpaired distributional preference.
Distributional Unpaired Preference We assume here that we don't don't have access to triplets of prompts and positive/negative responses (xt, y+ y-). Instead, we assume separate access to μu E P(X x ))), a distribution of positive prompt/response pairs (X+, Y+) we would like to be highly rewarded and reinforce the policy, and μ P(X x y) the distribution of the negative samples (X_,Y_ to be associated with low reward. We define the distributional preference follows:

---

Definition (Distributional Preference in the Unpaired Setting). A policy т prefers distributionally 14 on p. with respect to reference policy ref iff
(X+1X+) Trof (Y+ x+)
To To
log
log
Traf (Y-IX-)*
FSD
In other words, noting Tu TT( 3) = log xe(ule)
the distributional preference the unpaired
setting means that we have the following constraint:
(ru m04 > (ru (TuoTo)ae mo)2--
FSD
Distributional Paired Preference Note that can rewrite Eq. (2) in the equivalent form:
TTB ( (r) Trof (y (x) Tref (y-
log
2 log
Vr,, (t. ~ μ..
To (v)
In order turn this into distributional constraint need access to paired preference dataset in DPO (X, Y+;7 Y-) ~ Jr, and impose stochastic dominance of the random variable Zo log *Y X) T (X]
[x, indexed by the indexed by the policy we are optimizing on the random variable Zrer = log (Y- reference policy. Zo and Zref represent here the log likelihood ratio of positive to negative outcome under the policies Ty and Tref respectively. Hence, it is desirable to constrain the policy Th to have a larger excess log probability between positive and negative outcomes than that resulting from the reference policy Fref-
We define below more formally the paired distributional preference via stochastic dominance:
Definition (Distributional Preference in the Paired Setting). We say that the policy πe distribution- ally dominates Tret in terms of log probability ratio of positive and negative responses if:
To To (X+
Tref XX+
log
>
log
Te (Y
FSD
Teet (Y-IX)
Noting rp (2 (2, v+ log molu+la (x) Toly-
this can be written as follows:
(rp To)aN b (rp o Feet)gu.
(5)
3 AOT: Alignment via Optimal Transport a Convex Relaxation Approach
Note that the paired and unpaired distributional preference constraints in Definitions and can be used in LLM alignment as follows:
Find Ty H. such that (ra T0):44
> (ra o TT)
(FSD unpaired)
ESD
and
Find me E H such that (r, To)s
>
(rp o Trot)114
(FSD paired)
FSD
where Tu are Tp are given in Definitions 1 and 2 respectively, and H. a hypothesis class. Those two problems are instances of learning with stochastic orders introduced in [Domingo-Fnrich et al., 2022], but in a simpler setting since the constraints are on one-dimensional distributions and the order considered is the first order rather than the convex order considered in Cooiin et al., 2022]. Note that both problems are special cases of the following generic optimization problem:
Find 0 O such that Ua
>
Vo
FSD
where Up and Vo are real-valued random variables whose distributions depend parameter vector E O. Note that for our FSD paired setting, Vo = (independent of 0). Let μu,, and μv, be the probability measures of Up and Vo resp.
By the definition of FSD in Equation (1) we have:
Up > Va + Que 2 Qv) Vt [0

---

We can relax this problem to the following minimization problem:
!. h(Qu. (t) Qv, (t))dt,
min (0) 0E6
where h a function penalizing each quantile's 's violation of FSD. The objective function (7) seeks to measure the violation of FSD, so that it can be minimized or eliminated. For instance, with h the 0/1 loss (here the indicator function):
[ min 8E0
(8)
This loss reminds us the misclassification 0/1 loss. Following classical convex relaxation of 0/1 losses in binary classification [Bartlett et al., 2006), we consider surrogates h of the indicator function. Our choices for h are motivated by the "almost-FSD" notions in the literature (See Appendix F for a discussion). In practice, we use smooth convex approximations of the 0/1 loss (1><o) [Bartlett et al., 2006], for example for margin 3 0 h(x) (3 2 the B- squared hinge loss or h(x)= = log(1 (1 + exp(-Bz)) the 3-logistic loss. Although not a convex relaxation of the 0/1 loss, the least squares loss has been used in classification [Rosasco alignment, it was used in IPO [Azar et al., 2024] hence we use also h(z) = (3 - x)² and refer to as B-Least Squares. Further discussion of tradeoffs and benefits of different losses is in Appendix and formal assumptions on / needed for the statistical theory are given in Assumption 1.
al., 2004], and in the context of
The cost function in (7) is still computationally challenging, if we were solve the problem via gradient descent on this would require us to differentiate through the quantile operation. The following theorem from Santambrogio [2015] will be instrumental for us to cast the loss in (7) as an optimal transport problem with a convex cost h:
Theorem (Theorem 2.9 and Proposition 2.17 in Santambrogio [2015]). Let h : - TR+ be convex function we have for two real random variables U, V, with measures PU. Ay:
So -
J
Sempinn - (v(())dt=
h(u J - v)d (Tu.
and 7-- = (Qu. Cz) 1I) is a minimizer (where C1 is the Lebesgue measure on [0, 1] ). If furthermore is strictly convex 7 is the unique minimizer.
Thanks to Theorem 1 we can write the problem (7), in the following equivalent form that we call Alignment via Optimal Transport (AOT):
( h(Qu., Qvo (t) min OT,, MV6) min min Bee
yEMl (vg) J hiu u))
min
DE
This formulation reveals that have turned the stochastic dominance constraint an inner one- dimensional optimal transport problem with a convex cost h. OT, i (MV,,//V,) can be thought as a soft measure of the violation of the stochastic dominance of Ue on Vg. hence by minimizing it as function of 0 we are ensuring the optimal 0* results in Ug. dominating Vg., Such OT problems with smooth cost have been subject to theoretical and statistical study in one dimension well in high dimensions. For instance, [Manole and Niles Niles-Weed, 2024] considered smooth convex costs, and [Hundrieser et al., 2022] considered more general smooth costs. [Groppe and Hundrieser, 2023] considered entropic regularization of optimal transport with general smooth costs.
Computational Algorithm via Sorting We consider here empirical measures and turn to solve the inner problem for a fixed 0. We omit 0 in what follows to simplify notation. We are interested in OT (fu, fvv where fAu = Σi= 8n, fov = Σn de,- Given the convexity of h and thanks to Theorem (1), the optimal coupling of OT, (P... Av, given by the north west corner solution [Peyré and Cuturi, 2019] (Chapter 3, Section 3.4.2) that informally matches the i-th smallest element of U with the i-th smallest element from V. More formally, if we sort the variables u, and get the
order statistics (from min to max) u < ... I (1) and same for iy: (()) < ... S (1)) We have:
5

---

OT (Tn(Mu,
Σ 1=1
h(u -
(10)
Back to (9), given empirical samples Puo = 1 order statistics as function of We have therefore:
Bug and uiv, = -i= 8ug. let ug 0g be the
(0) (6)
min OT OT (iu. min 8EB
(6) h(ub ) (AOT)
(11)
BEO
In Appendix G, we show that the gradients of the objective (11) are asymptotically unbiased for bounded distributions (see the statement for all conditions). Note that the sorting operation in (11) is a I-Lipschitz function with discontinuous Jacobian. Like the ReLU activation function, it can be easily optimized by gradient descent [Anil al., 2019] (compare also sliced Wasserstein GANs). In practice, computing the gradient at any given step is done by first running the sorting algorithm and taking the gradient with respect to 0 with the current assignment held fixed.
AOT for Unpaired Preference Let 24 convex relaxation approach for unpaired FSD alignment given in (FSD unpaired) can therefore be cast AOT problem (given in Equation (11)) for
I jG + and ji 1 Σ Di=1 i Our
To (o((i+11e,tt
To(Vi (st,-) Tzet (Vi, (rs,-)
ug log Teef (36, 12i,+)
2/ log
i 1, ,...., n.
AOT for Paired Preference Let ptth measure. Our convex relaxation approach for paired FSD aligmment given in (FSD paired) can be there cast as an AOT problem (given in Equation (11)) for:
-1 Ei= 6 aE th be a paired preference empirical
To (1) (x:s)
Tref (Vi, Tret Vi. (r:))
ugg = log
vg = llgg
i= 1....,
Ui. (.:))
AOT with Soft Sorting One caveat of the alternating optimization for AOT between 0 and solving the inner optimal transport problem with hard sorting is that the gradient with respect the parameter for fixed permutations has dependency in on the order statistics level only and not through the sorting routine. To alleviate that, we propose to SoftSorting [Blondel al., 2020, Cuturi et al., 2019] that uses an entropic regularization to find a smoothed permutations via Sinkhorn algorithm, which in turn allows the paokppropggttion on to depend not only via the order statistics but also via the computational graph of SoftSorting.
Algorithms and 2 in Appendix B summarize AOT approach for distributional preference alignment in the unpaired and paired setting.
4 Statistical Analysis
In this section, we focus on the statistical analysis of unpaired-AOT and defer paired-AOT to Appendix E since has similar analysis. We make the following assumptions the OT cost h, the reward r, and the policy hypothesis class H.
Assumption 1 (OT cost). Let M, M,R R > be finite positive constants. We assume that the loss h: [-M, M] > [0, RJ. is convex T--Lipchitz and bounded. h is convex function (E.g. relaxation of the 0/1 loss such that h(t) > f(rr), t < 0 and > 0).
Assumption (Reward). We assume that r is bounded so that r TO( y) E [-M,M]]
Assumption 3 (Assumption on the hypothesis class the policy). We assume Feef ne E = = (Tg such that ne differentiable in 0 and suprex. /or No To(y)z)) x)) < L', C B2 do)), for L', ro 0.
Its Jacobian is permutation matrix every point except measure-zero where is differentiable.

---

Assumption 4. There exists To E H. such that (r me) > (r (ro FSD
Assumption satisfied for example by the hinge squared loss h(() = by the logistic loss ((t)) = log(1 + ) for t [-M, MM. Assumption 2 on the boundedness of the rewards can be imposed by clamping the values of the logits of the policies [-M, MM). which is common practice in practical implementations of LLM alignment. Assumption 3 is technical assumption needed to control the covering number of the H. Assumption 4 ensures the existence of the minimizer in H. We overload notations in what follows and refer to Tu and Tр to simplify the presentation. By our relaxation approach described in Section we can relax the unpaired stochastic dominance constraint problem given in (FSD unpaired) to:
min ( h (Q (1) (rome)su- (1) di = min OT ((r To): (r TO) RoEH
xeEM
(uAOT,)
Define the OT cost c: [-M, MM] x [-M, + [0, R| such that c(2, ) = h12 - 2'), for 2, E
[-M, M). Define the c-transform of function (p ppI-M. M] + R:
(²(2)
inf h[2 z') - (2)). ='E|-M,M]
In our setting, function is called e-concave if there exists vA [-M, MM - R such that y v
Define:
= = (p [-M, M] + I--R, R), yp is e-concave, with ||59 (lo05... s R}
By duality (Theorem 5.10 in [Villani, 2009]) have:
(((roo)dppe - f (5 no wo)dp_.
OT₁ ((r o mo)pM4, (r me)ap-)=
sup pEF.
Replacing the dual expression of OT, in (uAOT we see that (uAOT)) can be cast as a min-max problem:
min sup H PEF=
p(r mo)dut - -
(0 (g)oo
(12)
Given samples 03 n2 Σラ= 0(1 and pm -i ),n,- the empirical problem is: / (p(ro mo)di - TEE p
- / @ (romo)dd
min sup
(13)
Recall that OT is measure of the violation of stochastic dominance of (r No No) (r no un We have the following result on the sample complexity of the violation of stochastic dominance:
Theorem 2 (Sample Complexity Dominance Violation for AOT Unpaired). Let mg- be the population minimizer of (uAOT and To. be the solution of the empirical problem (13). We have the following sample complexity bound for the violation of stochastic dominance in AOT unpaired:
E EOT (rr nT. )1/44, (r Tq )1--) < OT ((r o Tg- DEH4, (r o (T-))))) )=((-
Optimal Almost FSD Violation
(FiiFFi (r no- )=14) + 2R, (F:: (r NOO )aP 2R,, (Fa or H: H:p+) (R: H:P-),
One dimensional OT sample complexity with optimai 0*
Complexity of learning H. via OT problem
where R, (F:)) E pup 1 E (z,) is the Rademacher Complexity and for i 1. o; are independent Rademacher random variables and Z; ~ V iid.
n
By considering our assumptions on the cost, the reward, and the hypothesis class, we obtain the parametric rate in n:
Corollary 1. (Informal) Under Assumptions 1, 2 and 3 we have:
1. IE OT (rr T&. )ap4, (r Te. ):-- T8. -TT, OT ((r Te- )204 (r TT- )=p))))) S n- where S refers to inequality up constants that depend only constants in the assumptions.
7

---

2. If in addition Assumption 4 holds and h(t) I O (fr mo De (r TB, )21l Sn-
(-t)
we
have:
We see that under our assumptions and for the hinge loss squared, the expected violation of the desired dominance in AOT unpaired converges to zero
0o.
Remark 1. While in Section 3, used the primal formulation compute OT, due to its computa- tional appeal thanks to the sorting algorithm we used for analyzing the sample complexity the dual of OT h- The dual reveals the game theoretic aspect of AOT as min-max game between the policy and the dual potential Pe that imposes FSD on the preference we want to infuse to the policy.
Experiments
In this section, evaluate the performance of the proposed AOT method on a diverse of base LLMs and datasets, comparing with currently available alternative alignment algorithms.
LLM Alignment Alternatives We compared AOT with current state-of-the-ar alignment ap- proaches, specifically Direct Preference Optimization (DPO) [Rafailov et al., 2024], Kahneman- Tversky Optimization (KTO) [Ethayarajh al., 2024] and Identity Policy Optimization (IPO) [Azar et al., 2024]. DPO and IPO operate on paired preference data, while KTO can handle both paired and unpaired prompt/response samples.
AlpacaEval (GPT4)
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT paired
29.9
82.5
66.1
62.9
50.8
74.4
53.1
AOT unpaired
31.3
82.5
66.2
62.8
51.1
74.4
51.8
DPO
27.4
82.8
65.8
50.6
52.0
KTO
24.9
82.7
65.4
63.0
48.7
74.9
53.9
IPO Merlinite-7B 17.1
27.7
82.4
65.1
63.0
46.5
74.0
52.3
81.6
63.2
62.6
42.0
73.9
45.2
Table I: Merlinite- 7B trained on UltraFcedback Binarized. AOT results the best performing
LLM compared to the alternative alignment algorithms on AlpacaEval, and is competitive across the other benchmarks that are evaluated the zero shot regime.
Reference Models Traditionally, model alignment is the third and final step applied the LLM that already has gone through original pretraining and supervised fine-tuning. For our experiments, we selected a range of models at various stages and with different levels of performance, all the family of 7B-parameter models. Specifically, we used Merl Merlinite- iniie-nee 7B [Sudalairaj et al., 2024], which is (SFT) on data from synthetic data
that has been instruction-tunt generator using maxonomy-driver data curation process. In Appendix H we also cover other popular LLMs, such as Mistral-7B [Jiang et al., 2023], OpenHermes-22 w-w.MMMMMaaa---aaaaaaa [Teeknium, 2024], Starling [Zuu et al., 2023], Ristrall7BB Jiang et al. [2023]. and Llama3- [AI @ Meta, 2024]. Datasets For our experiments, we used both paired and unpaired datasets. For the paired dataset, al., 2023b], containing over 60K
variant of Mistral-7B--a
we used the UltraFeedback binarized dataset from [Tunstall training samples, where for each prompt, there is pair of chosen (preferred) and rejected (not preferred) responses. This alignment dataset widely used, and all compared alignment techniques are well-suiter for it. For unpaired datasets, we used PKU BeaverTailk [Ji et al., 2023] with over 300K samples and HelpSteer [Wang et al., 2023] with around 35K samples. Here, for each prompt, there is only single response with score defined by some attributes (e.g., safety, faithfulness, helpfulness, etc.). We used the sum of attribute values and thresholded by the median to binarize the responses into chosen and rejected. For this unpaired dataset, only KTO and our AOT are applicable.
To measure the performance of different alignment methods, we used popular evaluation metrics, AlpacaEval [Dubois et al., 2024] and Open LLM benchmark [Beeching et al., 2023]. We note that Alpaca uses GPT4 model as judge to compare candidate responses to GPT4-based references on set of 805 challenging questions. The CPT4-based evaluations are expensive, so to limit our expenses, we also employed a very strong and capable L1ama3-70B-Inne -OB-Instruct [AI@ Meta, 2024] judge. As we show in Appendix H in Table 2, the order determined by Llama3-70p Instruct
Metrics

---

Effect of loss type AOT performance
Effect of batch size AOT Performance
40
50
AOT paired
ADT paired
35
unpaired
29.6
30.4
31.3
AloacaEval (Utmma3-3)) 708-778)
45
ADT unpaired
27.4
26.9
25
4.
15
35
10
30
25
LeastSquared
HingeSquarrd
Logistic
10
15
20
25
30
35
Batch size per GPU
Figure 2: Impact of batch size and loss type on AOT performance. The batch size is the effective number of samples in the mini-batch per GPU. We found the logistic loss to be performing better than least squared or hinge squared losses (all using 3 0.01). As we increase batch size, also observed improvement in AOT performance, which is expected as more samples per minibatch results in better effect of stochastic dominance (conforming Corollary 1).
Figure 3: Impact (3) param- eter performance of different alignment algorithms. 3 controls the divergence of the policy model from the initial reference model (low beta more divergence, high beta less divergence). We see a gen- eral trend that with higher betas, L.Ms alignment decreases the per- formance. Hence, for all experi- ments, we selected B = 0.01 as default value.
Effect of Beta Alignment Performance
47.5
AOT paired unpaired
45.0
DPO
42.5
IPO
Merlinit-22 78
40.0
37.5
32.5
30.0
27.5
0.0
Beta
and GPT4 is the same (the absolute score values are different), providing better free alternative LLM-judge for local Alpaca evaluations. For intermediate results, we also employed Tiny Bench- marks [Maia Polo et al., 2024] to approximate original metrics and provide fast feedback during the initial development. We also evaluated the aligned models on six key benchmarks from the Open LLM Leaderboard: A12 Reasoning Challenge ARC (grade-school school science questions), HellaSwart (commonsense inference), MMLU (multi-task accuracy), TruthfulQA (tendency to reproduce false- hoods), Winogrande (commonsense reasoning) and GSM8K (grade school math word problems). Note that in all the above benchmarks we use O-shot prompts, more challenging setting as opposed to commonly used few-shot prompting.
Experimental Setup Our implementation is based on the HuggingFace Alignment Handbook Tunstall et al. [2023a]. As we show in Appendix in Section B, the changes needed to adapt HF TRL trainer [von Werra al., 2020] for AOT are minimal and therefore can easily be adapted by the community. For each run our compute setup consisted of 8 H100 GPUs. We used LoRA [Hu et al., 2021] for parameter-efficien
fine-tuning during alignment and the FSDP (Fully-Sharded Sharded PatalParallel) setup to train the model over multiple GPUs. Under this setup, the training of each 7B 7B-parameter model on the UltraFeedback dataset took approximately one hour. The evaluation on AlpacaEval and Open LLM benchmarks took one additional hour to get the final results.
Results In Table we present the main results of comparing AOT to other baselines (KTO, DPO, and IPO) on paired UltraFeedback binarized dataset. On AlpacaEval (GPT4), our AOT unpaired approach scores 31.3%, which is significant gain from the base Merlinite-7B model. As of time of this writing (May 22nd, 2024), this result places our AOT aligned LLM on AlpacaEval LeaderBoart ahead of such strong competitors as KTO-Mistral- -AIRR [Ethayarajh al., 2023] and other 7B -B-aarmmeter models, reaching the level of Mixtral -aterrallx222B-- 8x22B-v0 1 (see Figure 4 in Appendix for an illustration). other L.M benchmarks AOT performs competitively to other baselines. As mentioned earlier, these evaluations are done using 0-shot prompts, leading to more challenging setting and resulting in overall lower performance across metries and baselines. For other base L.LMs we show their performance in Appendix H (see Tables 3, 4, 5, and 6).

---

We also examined the effect of batch size and the choice of loss function on AOT performance, results shown in Fig. 2. As the batch size increases, AlpacaEval (based on L1 L- 1omma3-7BB--0 (0B-instruct) also increases in line with our theory in Corollary 1. Note that our current setup (FSDP over 8 H100 GPUs) limits our batch size to 35 samples per GPU. We have also examined the impact of beta (controlling divergence of policy from reference) on AOT performance in Fig. 3. We noticed trend that with higher betas the performance of LLMs alignment decreases, thus we set B = 0.01. Ablation results comparing hard and soft sorting as well the variance of AlpacaEval scores across multiple runs in Appendix H (Tables and 10) show the overall robustness AOT.
Conclusion
We present in this paper Distributional Alignment via Optimal Transport (AOT) for large language models. The AOT cost can be cast as a one-dimensional optimal transport problem with a smooth and convex cost that penalizes violations of the dominance of the chosen rejected marginals. AOT enjoys parametric statistical rates. We showed with extensive experimentation on various paired and unpaired datasets, base models, and different loss functions, that AOT alignment robustly leads to aligned models that outperform alternative alignment strategies such as DPO, KTO and IPO on the Alpaca Benchmark, leading to the best 7B model to date on that benchmark as of the time of writing. On other benchmarks such as the open LLM leaderboard AOT leads to competitive results.
References
AI@ Meta. Llama 3 model card. 2024. URL main/MODEL_CARD CARD md.
C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learming, pages 291-301. PMLR, 2019.
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 058622.... 2022.
L. Bartlett, M. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101:138 156, 2006. URL https://api:
semanticscholar. worg ID :28338811. 2833811.
E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf. Open LLM Leaderboard. https: bttps::/hugee http::://uuginggfcc... co/ ha////aces//uuggingeee open_ [ren__ll__eaderbbae
leaderboard, 2023.
M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, pages 950-959. PMLR, 2020.
F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg. and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:: htttt:::////rooeeeinggg.. neurips.cc/paper. fi htt.://pwwp.......eeeccaaccccaaa
cc/paper_
Papere-ppppppppppp .dd.
M. Cuturi, O. Teboul, and J.-P. Vert. Differentiable ranking and sorting using optimal transport. Advances in neural information processing systems, 32, 2019.
E. Del Barrio, A. Cuesta-Albertos, and C. Matrán. An optimal transportation approach for assessing almost stochastic order. The Mathematics of the Uncertain: A Tribute to Pedro Gil, pages 33-44, 2018.
C. Domingo-Enrich, Y. Schiff, and Y. Mroueh. Learning with stochastic orders. In The Eleventh International Conference Learning Representations, 2022.
10

---

Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlle alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. 2024.
K. Ethayarajh, W. Xu, D. Jurafsky, and D. Kiela. Human-centered loss functions (halos). Technical report, Contextual AI, 2023.
K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXivv2402.01336, 2024.
M. Groppe and S. Hundrieser. Lower complexity adaptation for empirical entropic optimal transport. arXiv preprint arXiv:2306. 12233006.33300 13580, 2023.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen. LoRa: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.1 09685, 2021.
Hundrieser, Staudt, and A. Munk. Empirical optimal transport between different measures adapts to lower complexity. arXiv preprint arXiv: 2022.
Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, C. Zhang, R. Sun, Y. Wang, and Y. Yang. Beaver- tails: Towards improved safety alignment of Ilm via a human- -reeen-peeeeeece dataset. arXiv preprint arXiv:2307.044 04657. 2023.
A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. de las Casas, Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b (2023). arXiv preprint arXiv:2310.06825, 2023.
F. Maia Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. arXiv preprint arXiv:2402. 14992, 2024.
T. Manole and J. Niles-Weed. Sharp convergence rates for empirical optimal transport with smooth costs. The Annals of Applied Probability, 34(1B):1108-113 2024.
W. Ogryczak and A. Ruszczynski. Dual stochastic dominance and related mean-risk models. SIAM Journal on Optimization, 13(1):60-78, 2002.
L. Ouyang, J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
G. Peyré and M. Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 1 ((5-6)3355-60)) 2019. ISSN 1935-8237. doi: 10.1561/2200000003.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
L. Rosasco, E. De Vito, A. Caponnetto. M. Piana, and A. Verri. Are loss functions all the same? Neural Comput., 16(5):1063-1076 may 2004. ISSN 0899-7667.
F. Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling. Birkhäuser, Cham, 2015. ISBN 9783319208275. doi: 10.1007/978-3-319-28888
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
S. Sudalairaj, A. Bhandwaldar, A. Pareja, K. Xu, D. D. Cox, and A. Srivastava. Lab: Large-scale alignment for chatbots. arXiv preprint arXiv:2403.01081, 2024.
Teknium. 2 5-Mistral-7B, 2024.
I. Tunstall, F. Beeching, N. Lambert, N. Rajani, Huang, Rasul, A. Rush, and T. Wolf. The alignment handbook. https://gii http/tttttttt//te////// com hommhuugingffaee htaagginnfaceealinmmetttt coammentthandeeee handbook, 2023a.

---

L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, al. Zephyr: Direct distillation of LM alignment. arXiv preprint arXiv:2310.1694 2023b.
C. Villani. Optimal Transport: old and new, volume 338. Springer, 2009.
L. Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, and S. Huang. Trl: Transformer reinforcement learning. aattps://gituu com/bugggingface/trr,, 2020.
Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scoweroft, N. Kant, A. Swope, al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023.
Y. Zhao, M. Khalman, R. Joshi, S. Narayan, M. Saleh, and P. Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning ftte/foormmdd=0q00ooeee
Representations, 2023. URL. htttps://operrevie:
B. Zhu, E. Frick, T. Wu, H. Zhu, and J. Jiao. Starling-7b: 7b: Improving Ilm helpfulness and harmlessness with rlaif, November 2023.
12

---

A Broader Impact and Limitations
In this paper, we introduced alignment method for LLMs that can capture rewards at the dis- tributional level without the requirement of paired preference data. Our algorithm was derived by imposing the stochastic dominance of positive reward distribution over negative distributions through an Optimal Transport formulation. It enjoys very simple algorithmic implementation in terms of closed-form expression via sorting on empirical measures. Empirically, our algorithm demonstrates excellent results by allowing us to train 7B parameter models that achieve state-of-the-arr evaluation results on Open LLM Benchmarks and AlpacaEval.
In terms of broader societal impact, we would like to highlight the benefits that our AOT algorithm will bring RLHF by enabling more robust distributional alignment of LLMs, improving their ability to follow instructions accurately, and aligning their responses with human values.
Our work shares the same limitations and possible negative broader societal impacts the majority of RLHF work. The algorithm is fundamentally limited by the training dataset used for aligmment and might, therefore, contribute to amplifying various types of bias present in the data. In addition, alignment through AOT is not enough to address aspects related to the security and safety of LLM deployment. In general, better performance not imply better performance across the board in other tasks, and ad-hoc evaluation specific to each task of interest is warranted.
given set of benchmarks following alignment does
13

---

B Algorithms and Pytorch Code In Hugging Face TRL
Algorithm AOT Unpaired
1: Input: To. Tyef 3 >> 0, > 0,
Algorithm 2 AOT Paired
2: Unpaired Preference Data: "Chosen"
1: Input: To. Rref, B > 0, 0,
(i.-) and "Rejected" 3: f - (as 36.
Paired Preference Data:
A
E (2..8.
4:
for iter + Piter do
3: for iter +- -1, Riter do
5:
Get Positive/Negatie mini-batch
Get Positive/Negative mini-batch
6:
{(ri,t (zit+.
i 1..==..... b} -=
{(zi, +i 8/. fi" 1...b.... ....b. b}
7:
{ 3i,-
Compute Margins for i ....b Tolgs.
8:
Compute Rewards for 1...b
ug = log moly xx Seat 1 [x)
9:
ug log Teaf (20 [2,++)
Jolse 12
8: vp log mee
10:
v = log (M:
molvi [
Sort Margins
11:
Sort Rewards
10: if hard sort then
12:
if hard sort then
11:
(u(1). ... (()) Sort(ug)
13:
(u() w()) Sort(ug)
12:
(v(1) ... v(() Sort(u))
14:
(v() ... v(b) Sort
13:
else if soft sort then
15:
else soft sort then
14:
(u (b))
SoftSort (u),e)
(w()) (b))
SoftSort(ug (ug,
15:
(v(1) ... v(() SoftSort (gg.E)
(v()) ((b) SoftSort e))
16:
end if
18:
end if
17:
Compute AOT logistic loss
Compute AOT logistic loss
18:
ly - log (B(ug) (8)
20:
to - log a (3(ug (0)
())
19: Update
21:
Update
20: 0 Peggeddddamw333bit((o00))
22:
end for PagedAdamw3bbit(Vo(((
21: end for
22: Return
24:
Return T
import import torchsort
daf dpo_losss
2.
clif self. loss_type 'AOT_unpair::
thasen_logratio policy_chosea, whwseehhooepppeeeeeeeeeee logps referenee_chesen. chassee__ccheee_opppp
gejected_logration - policy_rejected_logg reference_rejectee_loo
if self soet = "hard sort":
choses, jegsss__ggatiosssrrttd..
torch caarceooa__lgraaiie logratios dim-I)
logriedd_ogratoossarttee
sort((rejected_ dim=0)
elif self. sart rype - "sfttssss"""
10
poosen_lorratioo_sorre sortte
regalacization_ strength-__
rejjeted_logratios_sort -0.1)
(rccoorttoff_ootttcceeeeeeeeee regularizatio__steng))
12
selta_sorted chosen chooien_ograaiie_.oooo logratios it self.AUT AOT_loss = "hinge":
13
torch reluiself... beta sorta_ AOT_loss "logistie":
16
losses =
17
-F. logsig self. self, beta rdelta
self. lahel_ smoorhin)))
logsiigmodd
delta_sorted) smbol_inoohinng
elif self. less_type = 'AOT
pi_logratios- = siiicy_ehose__oee ref logratios reference chosen logps reference rejoeneeeeeeeeedddeeeeeeeeeeeede loggs
policy policy_refected, reeeected_legpp
coosen_logss
sort_yort_
hort"
pi pi_logratios_ sorted.
torch. loirrloooraiooo.... dim=0)
@ef_logratoo_ssotedd.. elif self. soriitype 'soit_sort:
sarta_sotttreeflorraaia
dim=0)
pi pi_logratios__on
jorchsort. so li__logratioss regelarization, sttrengch=0.1)
si__lerratio_sootte
Sichsort.soft_aee rottsooofre___orattii aggllarzatiio_sstrnngghl-
10
delta_soried pi_logratios_sorrte ref_logratio__sottedd if self AOT rorch..elu(setf.ber. elif self AOT_los = "logistic":
"hinge":
delia_sorted)-22
14
losses
-o. self. beta edelta_sorted) self. label smooi_smo F.loossgmmid(-sse
della_sorteed smoothign
17
14

---

C Proofs
Proof of Theorem 2. Let
/
p(r p mo)dp - f mo)
E(0,
((0.
sup PEF
where for 2, E R::
(=(=) =
inf M[ - - cp())
M,N]
and (a function f' is c- concave there exits g such that f = g¹).
The population problem
min =(0, p4+ H-) N
(14)
Given samples p7 1 Eim1 8a4 wA) (A) and nTTh
Dim (itt ) the empirical problem is 1
m
/
p(r o mo)d - / (
E(0, p= pm)
PEF=
and the ERM problem is
min c0, p4,, PT)
#EEH
Let Om be the minimizer of the ERM we have for any 0, by the definition of the minimizer:
-f (ro
S sup pef.
0(r (orom))dfi
< sup pEt. (((rma)ddu+ (ro
sup PEP.J
p(romo)d(it) - 14)
+ sup 3EF.
( - pTm)
Let 8" be the minimizer of (14) for 0 6* in the above inequality, and taking expectations on the randomness of the samples we obtain
/
Ee (C, S EE("" 14-11-)
sup PEF.
o To- - 1+)
/ ( To- )d sup PEF=
On the other hand by symmetrization we have:
/ p(r o ((f )o-)d(("
E sup peF.
(+) s (Fc).
(15)
where (F) = Esup E (X,) os are independent rademacher random variables
and X3 ~ (r To- )ap4 iid (X, E R). and similarly we have:
/ ( mo no. )dl - pmm)) $ sup / (r T or- - S 2R (F
E sup PEP.
prEF;
We have finally:
Bs (e p=, S E(0", (4.H-- )+ 2R, + (F!
(16)
15

---

Turning now to
Tpm.a )dpt / (0(ro Tomm _dp-
mmmm = sup sEFo
/ (( T )d(eet 1e%))
S =(0mm Pm p=
sup
PEf.
sup vEF.. Tm )d(nnd(( -1-
/
S Mi
sup sup MEH.EE.
(p(r o TO ro) p(t)
sup sup *eE wEF
(((rong)duu"---
Taking expectations we obtain:
/ p(r p(roto)d(u+ k2)
Be p+ H-) Be(Om,n No PT) E sup sup PEF.
E sup sup m (rE(F.)
(" (d 11)
=00", 144-41)
2R. (Fo) + 2m (F.
Optinal FSD Violation
One dimensional OT complexity with optimal 0"
+ 2R, [F. or H) + 2R (Fr H).
Complexity learning in the problem
where
RS Rr, o H)
_ sup sup n PE
a Σ
oup(r ( 1)
(17)
D Bounding Rademacher Complexities
Proof of Corollary Define the uniform metric entropy of a class of real valued functions F on set X' as the logarithm of the covering number with respect to the uniform norm I1-Iloo for > 0. this defined as follows:
N(e, F, 11-1a~ = inf {n E N there exists f1... X R with sup min
|| filloo so} fEF feFisi
As observed in [Hundrieser et al., 2022] the c-transformation with bounded cost is a lipchitz operation under the uniform norm and since fcr ʃ we have by Lemma 2.1 in Munk
A Fú I1-11=2) = (Ne,, Fer l1-l1c)
(18)
Now turning to the Rademacher complexity of class F, it dominated by Dudley's 's entropy integral (Theorem 16 in Luxburg and Bousquet
R,((F) S inf
28 + 322 Vn 6/4
(19)
6E|0,R
Note that the cost we are using is c(2, h(2- 2'), for 2 f The domain on which the cost being a closed interval is convex and compact. By lipchitzity of h (Assumption 1) and denoting L its lipchitz constant, c(, 2') is lipchitz for all z' E i-M,, M). Equivalently e(, 2 is (a, A) Hölder smooth, for a and L, and hence our setup falls under the Assumptions of Theorem 3.11 in [Hundrieser et al., 2022] for Holder smooth costs defined on convex and compact sets and we have:
log 11.1 -d
16

---

Hence in our case a and d = 1 this leads to
log Fes 11- Se-1
Replacing this in Equation (19) we obtain: Rn (F.) S n and by Equation (18) it follows that: Rm (F)) E m
Turning now to the Rademacher Complexity of the composition of the e-concave potentials Fa with ro H (the composition of fixed reward function with the hypothesis class H ). Note since the cost d(,,²") is L Lipchitiz for all E [-M, M] we have Fe is included in the set of L lipchitz function that are bounded by R (See [Hundrieser et al., 2022] Lemma A.2) ). For vp E F. and w E H. let us note h = c(r o TO) we have:
| - hi 11.0
sup (o(romo(yla)) - o To (y|z)))
=EA'
We have:
( o0 og) lp(r me (a)) - (p(r )τ (((r )8²) pr mg))
< L |r o my ro oe 11 |11e 4'1lo0
where we used lipchitzity of o E Fe and hence we have:
||Mi,TT - h Il. S L ||r o To r To 114 & 1lac.
We have therefore the following bound on the covering number of the composition:
Nie,Fo 11 W (2 Fer llollee) W
(2r ro H. 11l1-1..))
2L
Plugging this Equation (19) we obtain:
1 R
(F. o H)
inf R)
28 + V32 Vn 86/4
V log Fle (1-1b)))))))) + log.N (2 ro H. 11.1120) de
Note that for a, b we have va+b Va
vb, hence we have:
(F oro H)
inf
28
v32
1
f*
R
Voog (e/ lllll)) + b log./ ( 2T. ro H, 11- de
R)
Vn J6/4
We know by lipchitizty of the cost and being in one dimension that log.N(s, Fer 1l- S e
By lipchitizity of r and using Assumption 3 we have therefore:
log.N V To H, 1101))))
E
< log.N
E 2LL
B2(ro, da), 11.1la)) < dg log 2roL'L
2L
We have therefore:
V2 R
(g) 2
112
de V2 (" 2roLL'
inf 26 + kE(o.R)
K1
vn 6/4 V do log
de S n n-=
Vn 6/4
E
(For& the upper bound obtained.)
For 2) By assumption 4, there exists To, such that we have for (-x)²
OT, ((r To- TT-) Fo-))p4,(( To- )pp=) 0.
E AOT paired
Similarly following the relaxation approach described in Section and using the dual representation of the OT problem we can relax the paired stochastic dominance constraint problem given in (FSD paired) to:
/ p(r To To)dp / 0' o oret)du
min sup H PEF.
(20)
where a is the paired measure representing (X, Y4, (.YYY.))) Y-).
let Tg- be the minimizer of (20). Consider- ing Problem (20), with empirical samples ji" 1 Σn ( mn. } denote Tā. its minimizer have
n
17

---

Theorem 3 (Sample Complexity of Dominance Violation for AOT Paired), The following sample complexity bound for the violation of stochastic dominance in AOT paired holds:
E OT ((r Tā, )aH, (r Feet)sle) & a ((r o To. )s1,, (r Trot)
Oprimial Abmost FSD Violation
+ 2Rm Fees (r T )1(+)) 4R, ( (Fo (r Fret)a11)
One dimensiooal OT sample complexity with optimal 6
2R (F oo Hipu)
Complexity of learning 'H via the ID OT problem
where (F; = E sup per | E- a. is the Rademacher Complexity and for i 11.. J; are independent rademacher random variables and Zi ~ iid.
Similarly under Assumptions 1, and 3 we have:
TØ )mH, (r =He)) - O ((r ma mo- )EH, (r o Tre))sH) 2 n E O ((r
Proof of Theurem 3. The proof can simply obtained by inspecting the proof of Theorem 1, and we omit it.
O
Remark Although O is one dimensional, an entropic regularization of OT has computational
advantages as discussed in Section 3. Results from [Groppe and Hundrieser, 2023] can be leveraged to obtain sample complexity bounds and we obtain under our assumptions also a parametric rate of n 3. The main insight in [Groppe and Hundrieser, 2023] is in introducing for an entropic regularization parameter E > the smoothed (c, E, (4) transform and replacing the spaces Fe by Fe.e In the ID case, up to constants, these spaces have the same covering numbers.
F
Choice of violation penalty function h
Recall proposed the following three classes of loss functions h in the main text:
1. Area of violation ("classificationn") Setting h(x) to be the 0-1 loss (Le<o) measures the fraction of the interval [0, 1] where violation occurs, paralleling classification losses which count the number of misclassification.
2. Wasserstein-1 violation Setting (xx to be the hinge loss (-+)+ reduces to measuring the Wasserstein-1 distance from Ug to the nearest distribution that has FSD over Vg.
3. W violation Setting h(x) to be the squared hinge loss (-x)² reduces to measuring the Wasserstein-2 distance from Ug the nearest distribution that has FSD over Vo-
Besides measuring different quantities, the optimization-theonnn properties of each are different:
1. The (-1 loss This loss does not penalize the size of the violations, making gradient- based optimization difficult as large violations have no gradient. Additionally, if FSD were not achievable (e.g. a strong teacher policy), not penalizing the size of the violations could result in risky policies.
2. The hinge loss, i.c. the Wasserstein-1 violation measure. By its nature, the gradient of small violations just as large as the gradient of large violations, which may beneficial for convergence to an FSD result. When FSD is impossible to achieve, this loss will also have the effect of encouraging sparse violations while still penalizing the size of the violations, similar to the LI norm in the classic Lasso algorithm. Smooth relaxations Smooth relaxations of the hinge, e.g., logistic loss described in the main text, have nonzero gradient at zero and continue to have a gradient for small positive values. This has the benefit of encouraging quantiles to continue improving after surpassing those of the reference.
18

---

3. The squared hinge loss, ie. the Wasserstein-2 violation measure. As a quadratic loss, it no longer prefers sparse violations (c.f. L2 norm regularization). Indeed, the gradient signal vanishes as the violation becomes small, meaning that dense violations are likely and slow to be removed, A potential failure mode would be that the new policy has small violation, but none of the quantiles outperform the baseline. Relaxation Introducing bias B as in the main text addresses this issue, ensuring the gradient at is nonzero.
G Gradients of the sorting-based objective (11)
We here repeat the sorting- based objective (11) that we propose, adding n as superscript for clarity.
min OT
(n) ( .V, min B5G n
Mug (()))
(21)
We use gradient-based optimization approaches in practice to find a minimizing 8. We have the following theorem showing the gradients are asymptotically unbiased (and thus friendly to stochastic gradient methods).
Theorem 4. Let = dt be L-lipschitz and the gradients of ug and 0g with respect to have I-norm bounded by M, i.e. [Vouelli S M, wovoll S M for all 0 O Suppose further that the support Us, Vg are bounded and their distributions have densities (i.e. are atomless). Then the gradient of the objective in (21) is asymptotically unbiased +
Proof. Observe that the gradients take the form
n
VOOTU (.) = V h h(ug - -06 (
(1)
(22)
1=1
n
N (m, Σ
(1) - (1) Vo -0)))
(23)
Note that as described in the main text, here, the current gradient is determined by the current state of the sorting of the samples, but it is not necessary to differentiate through the sorting algorithm itself as it changes only discretely.
We wish to understand the convergence of the bias of these gradients as n + co.
We write (noting that the atoms of the empirical Fn,V, are distinct with probability 1)
W' (ug (3)
1 Vo 110))
h' N'
(9) F. (Fu,uo (1g 10 Vo)
(
n b'(FT,bo (Fr.vo (v (v( (00 Vo(
7
Let's consider the first term, the analysis for the second term is the same. Note that since h' is L-Lipschitz
(Furo FFt (Fus In (u (u- F Fo.-Fa (Pn.oo(()))) 11' (u (w))) s I Fm. (
(24)
Let
1' = 1 M((!) ( Foz (Fu. ())
II alignment setting, this is should not be issue scores realevalued.
19

---

Note I' is simple empirical average and hence unbiased in the sense that E[T] is constant with n. Now by (24) we can write
| - 11
I.
Σ
7
Fn (Fn,uo(u)))) Fv. (FU, (ub (u)")) (oo(
(25)
We seek to bound this quantity co. By Cauchy-Schwarz,
|| - 1 S 1
P (wb )) (Fu.
(ub6))) (Fu
2
ト Σ
1 n
V Vo (1)
n
2²
Σ
Frii (Fn,U (u/2)) (u
Fvol (Fu 2
i=1
L
u/ (Fr. (u)))) )²
n
where we've ve assumed that # -. V (w6 < M Observe that FV, (Fur (-)) is the optimal transport plan from Up to V and is monotonic nondecreasing, hence Fivo (Fu, (ug)) are simply new set of independently drawn order statistics of og. i.e. up (6.2) We thus have
| 11 ~
T.
Σ
16 1/0
(3)
(1,2) 2
(26)
i=1
= -1/2²
(1
(t)
[P v.
(1)
dt
(27)
where Foli) n,V, are independently realized empirical quantile functions.
Theorem 3.1 of Del Barrio al. [2018] with the subsequent Remark 3.2.1 therein applies directly to this regime, with the following restated result:
Theorem 5 (Special case of Theorem 3.1 in light of Remark 3.2.1 in Del Barrio et al. [2018]). If F and G are CDFs of I-dimensional distributions with bounded support and G-1 is continuous on (0, 1), then
S (F," (Fr² -m - S/' 1F=1--
---1
7p
as
Applying Theorem setting and noting that for us the second integral is zero, have that if Ve has bounded support,
L2M² /
1 (1)
2 dt +
where + indicates convergence in probability as n + cc. This implies that / 1' converges to in probability. The proof for // is identical. Since the gradient (23) is then a sum of two unbiased terms and two terms that converge probability to zero, the gradient asymptotically unbiased.
H Additional Experiments
Tables 2, 3, 4, 5 and 6 are ablations on the reference base model used (Merlinite-77 Sarling-LM-7B-al
OpenHermes-22.. 2-..
Setaa-Laaaa---8-IInstee
Mistral-78 17--Instruut-vO 2). In these tables, we report only the AlpacaEval using Llama3-700 as a judge to reduce the costs evaluations. Note that we observed that while the absolute scoring of Llama3-70B is different than GPT4, as it can be seen in Table 2, it preserves the rankings of the models. We see across all these models a better performance of the distributional alignment AOT on AlpacaEval and competitive performance other benchmarks.
20

---

AvpacaEval (Llama3-70B)
AlpacaEval (GPT4)
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT paired
44.3
29.9
82.5
66.1
62.9
50.8
74.4
53.1
AOT unpaired
48,4
31.3
82.5
66.2
62.8
51.1
74.4
51.8
DPO
36.8
27.4
82.8
65.8
63.1
50.6
74.3
52.0
KTO
35.7
24.9
82.7
63.4
63.0
48.7
74.9
53,9
IPO
43.1
27.7
82.4
65.1
63.0
46.5
74.0
52.3
Merlinite-7B
28.8
17.1
81.6
63.2
62.6
42.0
73.9
45.2
Table 2: Merlinite-7B trained on UltraFeedback Binarized. Here we present full version of the results, including AlpacaEval using Llama3-700 inn-tinttuct as judge and GPT4 as a judge. The comparison reveals that although Llama3 inflates the scores, the relative order between the two judges remains the same, suggesting the use of a cheaper AlpacaEval alternative for local development.
AlpacaEval (Llama3-
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT paired
24.4
84.1
66.1
61.0
50.6
74.9
66.6
AOT unpaired
22.5
84.2
66.0
61.0
50.5
74.8
65.7
DPO
17.9
84.1
66.0
61.0
50.4
74.4
66.7
KTO
12.6
83.5
64.3
61.1
47.2
74.4
66.3
IPO
15.5
83.9
65.4
61.1
49.2
74.2
66.3
OpenHermes--7B
5.6
83.4
63.1
60.6
44.5
74.4
63.8
Table 3: SinnlHermees225-Miiste trained on UltraFeedback Binarized
AlpacaEval (Llama3-70B,
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT paired
30.4
84.3
66.9
61.4
45.5
72.6
69.0
AOT unpaired
34.4
85.1
67.4
61.5
47.0
72.3
68.5
DPO
28.6
84.5
66.7
61.4
45.3
72.5
69.8
KTO
27.2
84.8
67.0
61.4
46.2
74.2
70.2
IPO
28.6
84.5
66.7
61.4
44.4
72.9
69.8
Starling-7B
14.3
83.4
64.4
60.9
39.4
72.5
66.6
Table 4: Starling-LM-7B-alphhh trained on UltraFeedback Binarized
21

---

A AlpacaEval
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
(1lama3-700B
AOT paired
33.6
81.7
59.4
64.1
47.7
72.6
78.0
AOT unpaired
35.8
81.8
59.4
64.0
47.8
72.8
77.5
DPO
33.1
82.1
59.5
64.0
47.3
73.1
77.9
KTO
28.5
81.9
59.0
63,9
46.5
73.4
77.7
IPO
33.2
81.9
59.1
63.9
46.7
72.9
77.7
Llama3-8B
25.4
81.6
57.7
63.8
43.9
72.5
75.9
Table 5: Meta-Llama-3-888 Laa--lllaa-----nstruu trained on UltraFeedback Binarized
AlpacaEval (Llama3-70B)
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT paired
32.8
81.6
67.6
58.9
62.7
74.3
41.9
AOT unpaired
34.3
81.7
67.6
59.1
63.0
74,4
41.9
DPO
28.8
81.7
67.6
58.8
62.3
74.2
42.0
KTO
27.4
81.9
67.7
58.8
62.5
74.3
41.7
IPO
28.4
81.9
67.1
58.8
60.5
74.0
41.9
Mistral-7B
25.6
81.3
66.0
58.8
59.7
74.1
41.7
Table 6: Mistral-7B--Instruut-te trained UltraFeedback Binarized
Table 7 is an ablation on the sorting that is used in AOT with Merlinite-74 reference model. We see that hard and soft sorting are par in terms of overall performance.
Sort
AlpacaEval (llama3-70B
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
Type
Soft
44.3
82.5
66.1 66.2
62.9
74.4
53.1
AOT paired
Hard
43.8
82.7
62.9
50.7
74.5
53.9
Soft
48.4
82.5
66.2
62.8
51.1
74.4
$1.8
AOT unpaired
Hard
49.2
82.5
65.9
62.8
51.0
74.4
51.0
Table 7: The effect of sort type on performance in AOT alignment
Tables 8 and 9 give comparison between AOT and KTO on unpaired datasets (HelpSteer and PKU binarized) we see that overall AOT leads to better performance than KTO.
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT unpaired
82.0
63.5
62.9
45.6
74.9
48.0
KTO
81.9
63.5
62.7
45.0
74.2
48.5
Merlinite-7B
81.6
63.2
62.6
42.0
73.9
45.2
Table 8: Merlinite-7B trained unpaired HelpSteer (binarized)
ARC
Hellaswag
MMLU
Truthful
Winogrande
GSM8K
AOT unpaired
82.0
64.1
63.0
56.3
74.6
49.7
KTO
82.1
63.5
62.9
43.5
74.5
50.4
Merlinite-7B
81.6
63.2
62.6
42.0
73.9
45.2
Table 9: Merlinite-7B trained unpaired PKU (binarized)
Finally Figure puts in context our best model Merlinite-7B-ur as the best 7B-family model on AlpacaEval leaderboard the time of writing this paper. Finally, give in Table 10 the variance of the evaluation across 4 different random seeds for training and evaluation each alignment strategy. we see very small variance in AOT, especially the unpaired variant.
22

---

Version: AlpacaEval
AlpacaEval
Filter:
Community
Verifid
Browiin: GPT-4 Previser (11/06)
Autovennolstz: GPT4 Phwwiww (1106)
Win Rate
Win Rate
Model Name
Aligner 2B+GPT-4 (04/09) [
GPT-4 Omni (05/13)
57.5%
61.3%
(04109)
YiLarge Preview
51.9%
67.5%
Storm-7B (num__eauree beams=10)
GPT-4 Proview (11/06) l+
50.0%
50.0%
ExPO Llama-3-Instruce 3-instruct-8-SiimPP
45.8%
40.6%
Mlama--iinstrcc-8--Sii
44.7%
Nanbeige Plus Chat v0.1 |
44.5%
56.7%
Qwen1.55
Aligner 2B+Claude Opus
41.8%
34.5%
Opus (02/29)
40.5%
29.
GPT-4
38.1%
23.6%
Aligner 2B+Qwen11.5 728
36.7%
31.8%
Qwen1.5 1ieen.55722 72B Chat
36.6%
26.5%
GPT-4 (03/14)
35.3%
22.1%
Ein 708 v0.1 []
35,0%
24.8%
Claude Sonnet (02/29)
34.9%
25.6%
Fadeinx-Zaphyy-Chattv0.
34.8%
35.9%
Liama 70B Instruct
34.4%
33.2%
Mietral Large (24/02)
32.7%
21.4%
ExPO 9PPO-Mietral7B-PP
31.8%
35.4%
merinite-B--AOT
31.7%
29.9%
Samba CoE v (best-of-16)
31.5%
27.0%
REBEL -lama---BB--nttuuu |
31.4%
34.3%
Mixtrel 8x22B v0.1
30.9%
22.2%
SPPO-Missra77BPParRM lb
30.5%
32.2%
GPT-4 (06/13)
30.2%
15.8%
Snorkel (Mietral- PairRM-DOOObeettt-1115
30.0%
34.9%
Figure 4: Our AOT algorithm gives strong boost to Merlinite-75 7B model AlpacaEval leaderboard (as of May 22nd, 2024). The original Merlinite-77 7B score is 17.1, and after the alignment, the model gained 83%.
23

---

AlpacaEval (Llama3-70B)
AOT paired
46.8 + 1.48
AOT unpaired
48.1 = 0.35
DPO
39.2 1 2.35
KTO
33.8 +: 1.23
IPO
45.7 = 1.56
Merlinite-7B
28.8
Table 10: Merlinite-7B trained on UltraFeedback Binarized. We evaluated the stability (variance) of the model evaluation on AlpacaEval by running separate training and evaluation cycles, then computing the mean and standard deviations. The results stable, especially for AOT unpaired, showing a low deviation from the mean.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims the abstract and introduction summarize our methodological contributions and results, which are properly expanded upon in the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The first section in the appendix of the paper identifies the main limitations of our work.
Guidelines:
The answer NA means that the paper has limitations, while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when the image resolution is low, or images are taken low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not to penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification:
The Appendix includes proofs of all stated Theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All information needed to reproduce our results clearly stated in the paper. We will also release code to reproduce the main results upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the code to reproduce our main experimental results under open source upon acceptance. During the review process the code is shared in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ guuides/CoeeSummii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. cttpp//ccpbbbbc/uuuuuuu www.///w/eee//eee//ooe/ee CooeeSuoeeiseeeeeeeee www.sssiiccyyyoyoooooocc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details necessary details to reproduce experimental results provided in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars for the main results AlpacaEval using LLaMA3-70B as a judge (GPT4-based evaluations would incur significant cost for this type of experiments). As indicated, error bars are computed variance across 4 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mentioned the hardware resources utilized for our experiments in the relevant sections of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tteurii///uuips.... Spppalic/Ethissuiideeeeoeoeeppee
Answer: [Yes]
Justification:
Our research conducted in full conformity of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss societal impacts of our work in the context of improving on current RLHF techniques.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Creators of original work are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documented code for proposed alignment algorithm will be provided upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "h6T03sFVeC", "reviewer": "Reviewer_rub8", "summary": "This works proposed using Optimal transport in 1D to derive a better alignment guided by the preference data. The main idea is to work with the log likelihood  ratio  of  the  marginal distributions of the preference data. The alignment is made through Optimal Transport loss in 1D based on the concept of stochastic dominance between quantiles of two distributions.", "questions": "In Figure 3, it appears that small $\\beta$ will produce better results for all method except the IPO. Have the authors tried to test with smaller value of $\\beta$, i.e. $\\beta = 0.005$?  Is there any explanation for this trend in that figure?", "limitations": "It is fine.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The framework of the paper is very easy to follow. \nThe proposed problem and solution's direction are interesting. \nSettings, theoretical results are adequate to support the proposed method.", "weaknesses": "My main concern of this paper is the empirical results in experiment section. Table 1 shows that AOT paired/unpaired do not outperform other methods at least in 4 out 7 cases (ARC, MMLU, Winogrande, GSM8K). When they are versus each other, there is no clear winner between AOT paired and AOT unpaired. Meanwhile, I believe that with the AOT paired, when we have more information, the task must be easier, please correct me if I am wrong. I have similar concerns for their performances in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback. \n___\n**My main concern of this paper is the empirical results in experiment section. Table 1 shows that AOT paired/unpaired do not outperform other methods at least in 4 out 7 cases (ARC, MMLU, Winogrande, GSM8K). When they are versus each other, there is no clear winner between AOT paired and AOT unpaired. Meanwhile, I believe that with the AOT paired, when we have more information, the task must be easier, please correct me if I am wrong. I have similar concerns for their performances in Figure 2.**\n\nPlease note that the alignment scoring we use is AlpacaEval (instruction following), all other metrics from the Open LLM Leaderboard such as  (ARC, MMLU, Winogrande, GSM8K)  test that despite the alignment the model retains or improves on the capabilities it has acquired from the pretraining model. To benchmark alignment methods AlpacaEval has been well adopted by the community, and the Open LLM Leaderboard metrics assess whether the alignment deteriorates other capabilities of the LLMs. AOT, while having the highest score on AlpacaEval, is competitive on all other metrics. \n\nPlease note that AOT unpaired outperforms AOT paired on AlpacaEval because it is more robust to noise, if some paired positive and negative answers are noisy, AOT paired will fit to that noise, while AOT unpaired, since it does not use this pairing, will be more robust to noise.  While in AOT paired the task may seem easier, it is less robust. Hence in Figure 2 we see that AOT unpaired outperforms the paired one on AlpacaEval.  \n___\n\n**$\\beta$ will produce better results for all method except the IPO. Have the authors tried to test with smaller value of $\\beta$, i.e.  $\\beta=0.005$**\n\nPlease note this is an artifact of the formulation of IPO and its implementation in [hugging face](  https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L1145C1-L1146C1) while for all other methods $\\beta$ plays the role of a margin 1/$\\beta$ plays the role of the margin in IPO. This explains this trend in Figure 3, IPO peaks on high values of beta and others on lower values, note this same trend has been also observed in the [hugging face blog]( https://huggingface.co/blog/pref-tuning) comparing these alignment techniques.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJpnLXCrTI", "reviewer": "Reviewer_9Mcg", "summary": "The motivation of this paper is that current alignment approaches ensure reward dominance at the sample-level but not on the distributional level. With this in mind, the authors set their goal to design an alignment approach which satisfies First Order Stochastic Dominance (FSD) in rewards for positive examples compared to negative examples. With this goal in mind, they make the following contributions:\n1. They write this as a constraint satisfaction problem, which can be relaxed into an unconstrained optimization problem using different surrogates of the 0/1 loss.\n2. Using Santambrogio (2015)'s result from optimal transport, they develop a computationally efficient algorithm called ATO.\n3. They provide an upper bound for the violation in stochastic dominance of this algorithm using a Rademacher and symmetrization argument.\n4. They provide experiments that show that ATO is competitive with existing approaches like DPO, KTO.", "questions": "Since KTO also works with unpaired preference data, a more thorough comparison with KTO is warranted:\n1. Is there a unique solution that satisfies the FSD condition? If there are multiple solutions, how does ATO break ties? How does KTO break ties? \n2. Is the reason that the behavior is different from KTO that they have different inductive biases and hence converge to different policies? Or the variances of the estimators are different, even though they converge to the same?\n3. Is there a way to compare Theorem 2 with the statistical properties of KTO/DPO/RLHF? Or any reason to believe that ATO has a statistical advantage?\n\nDespite the weaknesses, I think the connection to optimal transport is interesting and the experiments are sound and comprehensive. I would be happy to adjust my score accordingly if the authors alleviate my concerns through discussion.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**\n1. The connection to optimal transport is quite interesting.\n\n**Quality**\n\n2. The empirical results are competitive with other alignment approaches.\n3. The theoretical results are sound\n\n**Clarity**\n\n4. The paper is well-written with background appropriately introduced and clear explanations.", "weaknesses": "1. There could be some more motivation for why the FSD condition is practically desirable over the conditional (on x) dominance condition of DPO from Equation (2). Is this more than a theoretical nicety?\n\n2. There is not a clear improvement in the empirical results over prior approaches, and each approach seems to enjoy success on certain benchmarks -- with the differences between performance quite small in most cases.\nThis invites the question of in what settings ATO would perform better than other approaches in the literature. \n\n3. The theoretical results do not shed much light on when ATO is likely to perform well. Uniform Convergence results are well-known in recent years to not explain generalization behavior of modern ML models. So, while nice, the insight from Theorem 2 is not super obvious to me.\n\nMinor:\n1. Please call the Santambrogio (2015) result something other than a theorem since it is not a contribution of the present work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments: \n___\n\n**Weaknesses**\n___\n\n**1) There could be some more motivation for why the FSD condition is practically desirable over the conditional (on x) dominance condition of DPO from Equation (2). Is this more than a theoretical nicety?**\n\nThanks for bringing this up we will make the following discussion more salient:\n\n* It is more expensive to collect paired preference datasets, that have for each prompt a preferred and a rejected answer. There are plenty of unpaired datasets in the sense for each prompt we have either a preferred or a rejected answer. AOT and KTO allow the use of these datasets in alignment.\n\n* AOT unpaired by not relying on pairs of rejected / preferred answers but rather on their distributions, is more robust to noise in the pairing. While DPO would suffer on a pointwise comparison from noise in the pairing, AOT unpaired would not.\n___\n**2) There is not a clear improvement in the empirical results over prior approaches, and each approach seems to enjoy success on certain benchmarks -- with the differences between performance quite small in most cases. This invites the question of in what settings ATO would perform better than other approaches in the literature.**\n\nPlease note that the alignment scoring we use is AlpacaEval (instruction following), all other metrics from the Open LLM Leaderboard such as (ARC, MMLU, Winogrande, GSM8K) test that despite the alignment the model retains or improves on the capabilities it has acquired from the pretraining model. To benchmark alignment methods AlpacaEval has been well adopted by the community, and the Open LLM Leaderboard metrics assess whether the alignment deteriorates other capabilities of the LLMs. AOT, while having the highest score on AlpacaEval, is competitive on all other metrics.\n___\n\n**3) The theoretical results do not shed much light on when ATO is likely to perform well. Uniform Convergence results are well-known in recent years to not explain generalization behavior of modern ML models. So, while nice, the insight from Theorem 2 is not super obvious to me.**\n\nPlease note that Theorem 2 shows that despite AOT can be cast as a min/max game between the LLM and the OT potential, the one dimensional nature of the OT problem does not curse the statistical properties of the overall problem. The generalization bound we provide enjoys the parametric rate $1/\\sqrt{n}$. Note that in Figure 3, we see that the generalization error (on the AlpacEval) improves as the sample size grows which echoes findings in Theorem 2. The main insight of Theorem 2 is while AOT has an inner optimal transport problem to solve in order to update the LLM parameters, this does not curse statistically the rate, since the OT problem is one dimensional. \n\n___\n\n**Please call the Santambrogio (2015) result something other than a theorem since it is not a contribution of the present work.**\n\nThank you for the remark, we will take this into consideration in the final version of the paper.  We will add the following sentence before this theorem: The following theorem is a restatement of Theorem 2.9 in Santambrogio (2015). \n___\n\n**Questions**\n___\n\n**1) Since KTO also works with unpaired preference data, a more thorough comparison with KTO is warranted:\nIs there a unique solution that satisfies the FSD condition? If there are multiple solutions, how does ATO break ties? How does KTO break ties?**\n\nOne can study the uniqueness of the solution in simplified setups where the log-likelihood is a linear model, since the problem becomes convex in the weights, the AOT problem properly regularized with an $L_2$ regularizer over the weight will have a unique solution. Nevertheless, this is not the case for general nonlinear models such as transformers or LoRA adapters when finetuning the LLM,  then multiple solutions may exist.  Studying the connectivity of the minimas with an alignment loss as it is done in deep learning with classification objectives is an interesting avenue for future research and that is beyond the scope of the paper. We believe if multiple minima exist they will achieve the same soft violation of FSD (that is the loss of AOT), and those solutions will be equivalent and most likely connected. \n___\n\n**2) Is the reason that the behavior is different from KTO that they have different inductive biases and hence converge to different policies? Or the variances of the estimators are different, even though they converge to the same?**\n\nWe believe they have different cost functions and hence they will converge to different policies. While AOT compares quantiles of rewards between positive and negative rewards, KTO compares positive rewards to the average negative reward, and negative rewards to the average positive rewards. KTO will lead to a policy that favors sentences whose reward is above the average of negative rewards in the training set, and AOT will lead to a policy that distributionally prefers positive over negative responses.  \n___\n\n**3) Is there a way to compare Theorem 2 with the statistical properties of KTO/DPO/RLHF? Or any reason to believe that ATO has a statistical advantage?**\n\nStatistically speaking KTO, DPO, and AOT have similar statistical rates. AOT does not suffer statistically from having to solve an OT problem since this is a one dimensional problem, and hence still has a good statistical rate while being a min/max problem between the LLM and the potential of the OT problem (the sorting). For RLHF the statistical properties are more intricate since there is the reward learning and the sampling from the policy in training. The following paper gives a statistical analysis of a [variant of RLHF](https://arxiv.org/pdf/2405.21046 ). Different from our analysis this RLHF variant has also regret bounds that scale like $1/\\sqrt{T}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CmF31nq6V5", "reviewer": "Reviewer_BcRY", "summary": "The paper proposes a new technique for preference alignment, named Alignment via Optimal Transport (AOT). The proposed technique supports both paired and unpaired alignment settings.  The paper introduces a new viewpoint for preference alignment based on stochastic dominance i.e., making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. From this perspective, the paper explains DPO as a special case i.e., a pointwise preference approach with relaxation through the logistic loss. In addition, the paper shows that using convex relaxation is equivalent to minimizing a one-dimensional optimal transport problem. To enhance the differentiability of the objective through the sorting operator (for approximating the continuous optimal transport), the paper uses the SInkhorn-Knopp algorithm instead of the conventional sorting. On the experimental side, AOT leads to state-of-the-art models in the 7B family of models when evaluated with Open LLM Benchmarks and AlpacaEval (using Llama3-70B-Instruct instead of GPT4).", "questions": "1. It seems that hard sorting is also comparable to soft sorting. Using normal sorting could help to improve the computational speed compared to soft sorting. Should the paper recommend hard sorting as the default variant?\n\n2. How many interactions are used for the Sinkhorn algorithm? What is the choice of the entropic regularization hyperparameter? Do these hyperparameters significantly affect the results?\n\n3. Why does the unpaired setting lead to better results than the paired setting? Do the authors have any explanation?", "limitations": "The authors adequately addressed the limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is well-written and easy to follow.\n* The paper proposes an original preference alignment approach based on stochastic dominance and optimal transport. The connection to optimal transport is interesting and novel. In addition, the paper also provides theoretical results on the sample complexity of the empirical estimation of the objective.\n* The proposed approach can work for both unpaired and paired alignment settings.\n* Experiments are extensive on various datasets i.e., UltraFeedback for paired setting,  PKU BeaverTails, and HelpSteer for unpaired setting. The paper also provides a free LLM-judge for local Alpaca evaluations.\n* ATO achieves SOTA on AlpacaEval benchmark and competitive results on other metrics compared to DPO, KPO, and IPO.", "weaknesses": "The evaluation is conducted using Llama3-70B-Instruct instead of GPT4 which is not the standard. However, as described in the paper, the usage of  Llama3-70B-Instruct  leads to approximately the same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review of the paper and for their insightful questions that we address below: \n___\n\n**The evaluation is conducted using Llama3-70B-Instruct instead of GPT4 which is not the standard. However, as described in the paper, the usage of Llama3-70B-Instruct leads to approximately the same**\n\nThanks for this comment, we have also reported GPT4 eval on Merlinite, and for the rest we used Llama 70B to lower evaluation costs as mentioned by the reviewer.\n___\n\n**It seems that hard sorting is also comparable to soft sorting. Using normal sorting could help to improve the computational speed compared to soft sorting. Should the paper recommend hard sorting as the default variant?**\n\nWe recommend hard sorting as a default since soft sorting did not lead to substantial improvements. And as noted by the reviewer, hard sorting is also faster. \n___\n\n**How many interactions are used for the Sinkhorn algorithm? What is the choice of the entropic regularization hyperparameter? Do these hyperparameters significantly affect the results?**\n\nThe entropic regularization has to be small ~ 0.1 to ensure accurate sorting otherwise the sorting will be wrong. Note that the soft [sorting paper](https://arxiv.org/pdf/2002.08871) uses PAV (Pool adjacent violators) to compute the soft sorting, which, unlike Sinkhorn, does not need a fixed number of iterations as it reduces finding the soft sorting to an isotonic regression that can be computed in closed form with an algorithm $O(n \\log n)$. \n___\n\n**Why does the unpaired setting lead to better results than the paired setting? Do the authors have any explanation?**\n\nPlease note that AOT unpaired outperforms AOT paired on AlpacaEval because it is more robust to noise, if some paired positive and negative answers are noisy, AOT paired will fit to that noise, while AOT unpaired since it does not use this pairing will be more robust to noise.  Hence in Figure 2 we see that AOT unpaired outperforms the paired one on AlpacaEval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Alignment via Optimal Transport (AOT), a new method for aligning large language models (LLMs) based on distributional preferences rather than pairwise human preferences. AOT ensures that the reward distribution of positive samples stochastically dominates that of negative samples by solving an optimal transport problem with a smooth, convex cost, leading to a closed-form solution. By fine-tuning LLMs with the AOT objective, the method achieves superior alignment, demonstrated through state-of-the-art performance on various alignment datasets and benchmarks, including Open LLM Benchmarks and AlpacaEval. All reviewer recognized that the paper has some interesting theoretical contributions which connects optimal transport to the alignment problem. However, the numerical evaluation of this work seems to be less adequate, which the authors can improve upon.
Decision: Accept (poster)