ComPO: Preference Alignment via Comparison Oracles
Authors: Peter Chen
Xi Chen
Wotao Yin
Tianyi Lin
Venue: NeurIPS 2025 poster ()
Keywords: LLM Preference Alignment
Zeroth-order Optimization
Comparison-based Optimization
Noisy Preference Pairs
Primary area: deep_learning

Abstract: Direct alignment methods are increasingly used for aligning large language models (LLMs) with human preferences. However, these methods suffer from the issues of verbosity and likelihood displacement, which can be driven by the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. The contributions of this paper are two-fold. First, we propose a new preference alignment method based on zeroth-order, comparison-based optimization via comparison oracles and provide convergence guarantees for its basic scheme. Second, we improve our method using some heuristics and conduct the experiments to demonstrate the flexibility and compatibility of practical scheme in improving the performance of LLMs using noisy preference pairs. Evaluations are conducted across multiple base and instruction-tuned models (Mistral-7B, Llama-3-8B and Gemma-2-9B) with benchmarks (AlpacaEval 2, MT-Bench and Arena-Hard). Experimental results show the effectiveness of our method as an alternative to addressing the limitations of existing direct alignment methods. A highlight of our work is that we evidence the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings in Razin et al (2025).

ComPO: Preference Alignment via Comparison Oracles
Peter Chen
Xi Chen'
Wotao Yin! Columbia University
Tianyi Lin
Stern School of Business, New York University?
DAMO Academy, Alibaba Group US*
(1c3826, (1.3335)0colume umbia. edu, xc130stern nyy. edu wotao, yincal iaalllibaaainn com
Abstract
Direct alignment methods increasingly used for aligning large language models (LLMs) with human preferences. However, these methods suffer from the issues likelihood displacement, which be driven by noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. The contributions of this paper are two-fold. First, we propose a preference alignment method based on zeroth-order, comparison-basee -based optimization via comparison oracles and provide convergence guarantees for its basic mechanism. Second, we improve our method using some heuristics and conduct the experiments to demonstrate the flexibility and compatibility practical mechanisms in improving the performance of LLMs using noisy preference pairs. Evaluations are conducted across multiple base and instruction-tur -utnee models (Mistral-7B, Llama-3-8h and Gemma-2-9B) with benchmarks (AlpacaEval 2, MT--Bench and Arena-Hard)' Experimental results show the effectiveness of our method as an alternative addressing the limitations of existing methods, not only likelihood displacement but verbosity. A highlight our work is that we evidence the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings Razin et al. [73].
Introduction
Generative Al is breaking down barriers to intelligence, empowering domain experts across academia, industrial sectors, and governments to develop and manage Al systems more effectively. At the heart of this revolution are large language models (LLMs), which are transforming data organization, retrieval, and analysis [9. 20, 85, 1, 10). These models are trained vast, diverse data and must be carefully aligned with human preferences ensure they generate helpful and harmless content [7]. A prominent alignment method is reinforcement learning from human feedback (RLHF) [21, 81], which first fits reward model based on human preference pairs and then uses RL to train a policy to maximize this trained reward. Despite RLHF's success [108, 63, 85. 1]. it involves complex and computationally expensive multi-stage procedure. This motivates the development alignment methods, such direct preference optimization (DPO) [70] and its variants [6, 30, 66, 93, 83, 60, 16, 103], which directly optimize the LLM using the human preference pairs, avoiding the need for separately training reward model.
direct
The applications of direct alignment methods have gained momentum due to their simplicity and train- ing stability. However, these methods suffer from one critical issue: likelihood displacement, Likeli- hood displacement refers the counterintuitive situation where the training process, designed to increase
Models and code: huggingface cotpccecr//coooaaccoaool
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the likelihood of preferred responses relative to dispreferred ones, actually reduces the absolute prob- ability of the preferred responses, leading to "unintentional unalignment" [64, 82, 72, 65, 57. 96, 73]. For example, training a model to prefer No over NEVER sharply increase the likelihood of YES. Prac- tically, this issue has negative impacts on L.LM performance: unintentionally shifts probability mass to harmful responses. For example, if the prompt is to outline the steps for a terrorist organization to infiltrate& a government agency, Gemma-2B initially generates two refusal responses. Afte DPO training, it complies with unsafe prompts due to likelihood displacement shifting probability mass from the preferred refusal responses to harmful responses (see [73, Table 18). There is another issue -verbosity which refers to the tendeney of models fine-tuned with RLHF [78, 44] and direct align- ment methods [66, 4, 71] to generate longer responses, often without corresponding improvement in quality, and which causes low efficiency and higher consumption of hardware resources.
Recent work has suggested that likelihood displacement might arise from preference pairs that induce similar preferred and displeased responses [64, 73] (which we refer to them as noisy preference pairs in this paper). To mitigate this issue, researchers have tried to add different regularization [64, 72]. Recently, Razin et al. [73] proposed to measure the similarity between preferred and dispreferred responses using the centered hidden embedding similarity (CHES) score and empirically showed that filtering out the preference pairs with small CHES score is more effective for mitigating the likelihood displacement compared to adding the supervised fine-tuning (SFT) regularization.
While DPO provides computationally convenient framework by maximizing certain oggllikeihooo margin between preferred and dispreferred responses, this objective function seems to act as a proxy or surrogate for the true, complex goal of alignment. This proxy works well enough when preference pairs clearly delineate better responses. However, when faced with noisy pairs - where the preference signal is weak or ambiguous, relative to the embeddings of the two responses optimizing the specific DPO objective function can result in adverse effects such as likelihood displacement, as this objective function itself does not accurately reflect the desired alignment improvement. Formally and explicitly defining alignment as a single, optimizable mathematical objective function is exceptionally challenging. Instead of pursuing such an explicit objective, we shall recognize that preference pairs inherently represent comparative judgments based on this underlying. albeit latent, alignment goal. Inspired by comparison-orace -ow--oooaae-sseeeeeeeee based optimization techniques, which navigate in search spaces using only comparison outcome information (that is, "is solution A better than solution B?"", we leverage preference pairs in similar manner. We treat the preference pairs as the oracle outputs based on the hidden alignment objective. This allows us to use them to guide model parameter updates directly, thus avoiding commitment to explicit proxy objective.
Contribution. In this paper, we propose a preference alignment method that directly leverages comparative preference pairs by employing comparison oracles and effectively utilize the signals present even in noisy preference pairs. Our contributions can be summarized as follows:
1.
We identify that likelihood displacement issue is exacerbated by the ineffective handling of noisy preference pairs in existing methods. We propose to mitigate this issue by developing a method based on a specialized comparison oracle to extract useful information from these pairs. We also provide convergence guarantee for the basic scheme of our method under non-convex, smooth settings.
2. To ensure computational efficiency for large-scale model fine-tuning, we enhance our method with several techniques, including integration with DPO to handle clean and noisy preference data separately and approximating expensive steps in standard comparison--racte-bass optimization by efficiently restricting and clipping normalized gradients.
3. We conduct extensive experiments demonstrating the flexibility and effectiveness of our practical approach in improving LLM performance, particularly leveraging both clean and noisy preference data. Evaluations are undertaken across base and instruction-tume models (Mistral- Llama-3-8B, and Gemma-2-44 using benchmarks (A (AlpacaEval 2, MT-Bench, and Arena- Hard). Experimental results validate our approach's 's effectiveness. which addresses limitations in current direct alignment techniques.
Recent works [32, 27, 66, 4. 93, 60, 65] have shown that verbosity can be mitigated by incorporating appropriate regularization into the objective, suggesting that modifying the objective could better capture alignment goals. Although our method is not specifically designed for addressing verbosity issue, it consistently improve length-controlled win rate (I.C), indicating that our method helps reduce verbosity by possibly optimizing more robust and alignment- faitthuu objective.

---

Related works. Our work is mainly connected to the literature on direct preference alignment methods and optimization techniques utilizing comparison oracles. Due to space limitations. we defer our comments on other relevant topics to Appendix B. Direct preference alignment methods (such as DPO [70]) are simple and stable offiin alternatives to RLHF. Various DPO variants with other objectives were proposed, including ranking ones beyond pairwise preference data [25, 95, 79, 15, 56] and simple ones that do not rely on a reference model [38, 60J. It is well known that DPO suffers from the issues of verbosity [66, 4. 71] and likelihood displacement [64, 82, 72, 65, 57, 96], which can be interpreted from unified perspective of data curation [66. 73J. Our work continues along this perspective by arguing that these issues can be mitigated by using the information contained in the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses.
In optimization literature, the first algorithm based on comparison oracles is variant of the coordinate descent method [41, 59], and two representative methods that consider using comparison oracles to approximate the gradient are [12, 18]. The major drawback in these works is that the objective function is assumed to be convex or strongly convex, which is unrealistic in preference alignment applications. There are other works that investigate the value of comparison oracles in the context of online bandit optimization [98, 47, 24), Bayesian optimization [5, 53J and RLHF [84, 100]. Our work extends [12] to preference alignment through nontrivial modifications. First, we define the comparison oracle based on response likelihood: 01 is better than 82 if 01 achieves a higher likelihood for preferred responses and a lower likelihood for dispreferred responses. Second, convergence rate guarantee beyond the convex settings. Finally, instead of explicitly imposing a sparsity constraint when estimating normalized gradients, which leads to expensive computational sub-step, we reduce the computation by approximating the sub-step by clipping an approximated normalized gradient, which is efficient for fine-tuning LLMss The key difference between our work and recent works [84, 100] is the specific design of our comparison oracle, which is constructed extract meaningful directional information from noisy preference pairs prevalent in alignment datasets, thereby mitigating verbosity and likelihood displacement.
prove
Preliminaries and Technical Background
We provide overview of the setup for direct preference alignment in this paper, and the definition for comparison oracles and the subroutine for estimating gradients using comparison oracles that are important to designing the basic scheme of our method.
2.1 Direct preference alignment
Modern LLMs are designed based the Transformer architecture [86] and follow user prompts x E V* to generate response E V*, where V is a vocabulary of tokens. We consider an LLM as policy a (y())) which corresponds to probabilities y given x. For assigning probabilities each token of y, the policy operates in an auto-regressive manner as follows,
Te (y(x) M (o(yx
where 9 stands for the model's parameter (e.g., the parameters of the Transformer architecture) and Yeu denotes the first A: - tokens of y. However, the generated responses might not be helpful, safe or reliable, which necessities the process of further aligning the LLMs with human preference.
We consider the direct preference learning pipeline which relies on pairwise preference data. Indeed, we assume the access preference dataset D containing samples (x, (x.y+.x yy), where is a prompt and (yt, ) is a pair of preferred and dispreferred responses to x. This pipeline includes an initial supervised fine-tuning (SFT) phase where the model is fine-tuned using the cross cross-entropy loss and high-quality data for specific downstream tasks. The SFT data can be either independent of DD [85] or consists of prompts and preferred responses from D [70].
Direct alignment methods (e.g., DPO [70]) optimize the policy my over the preference dataset D without learning a reward model as in RLHF [108, 81]. This is typically done by minimizing a loss of the following form:
(f) - Exx. [Log (Blog +x(y+ mally (x) log 5uly )
warly - (x)
(1)
where FieF the model after SFT, a regularization parameter, and a R + [0, 1] is the sigmoid function. However, the function LooO relies the log-likelihood margin between y and y such
3

---

that DPO maximizes the likelihood margin between y" and rather than maximizing the likelihood for y and minimizing the likelihood for y The likelihood yf might decrease during training and the probability mass is shifted from y to responses with opposite meaning [64, 73). One of possible reasons that the above objective function is not suitable for extracting information from noisy preference pairs that induce similar preferred and dispreferred responses.
Empirically, [73] has shown that filtering out similar preference pairs makes DPO more effective. However, the noisy preference pairs might contain rich information that can improve the performance of LLMs. Extracting such information is challenging since it is difficult to explicitly write down an objective function that maximizes the likelihood for y and minimizes the likelihood for y and the only thing that we know is that its function value is smaller for a better policy which exhibits a higher likelihood for y* and lower likelihood for y This motivates us to design a new alignment method by directly leveraging the comparison signal in pairwise preference data (x, , from D.
2.2 Comparison oracles and zeroth-order methods
To contextualize our proposed method for aligning LLMs with human preferences, we review the definitions for comparison oracles and explain how one leverages the comparison oracles develop the zeroth-order methods in the literature.
Given that f Rd + R is function where neither its function value nor its gradient is accessible, we define a pairwise comparison oracle Cj in its simplest form as follows,
Definition 2.1 We call Cr(0,0) 0') Rdd IRd
+ { {+1,- -1} comparison oracle for function f if
(, = { { -1, if i( < 1(0).
+1, otherwise.
In other words, when separately queried with and 0', the oracle C5 returns sign(J(0") - f(0)).
The key idea of designing the subroutine in [12] for estimating gradients using comparison oracles comes from 1-bit compressed sensing [8]. The goal is to recover a signal g E R from the quantized measurements y/8 sign(z, g) where z, is a random perturbation vector drawn from any rationally invariant distribution. The theoretical guarantee on the required number of perturbations to obtain an approximate signal was established in [69] and extended in [12]. Notably, we have
C0 rz:) = (f rz,) f(0)) sign(z, ff())),
where r > 0 is parameter that controls the magnitude of perturbation. As such, the comparison oracle returns y (O, 0 + rz.) which serves as an approximate 1-bit measurement of V f0).
Another crucial issue is that the zeroth-order comparison baaped based methods suffers from the dimension- dependent iteration complexity bound [41]. This makes sense since the comparison oracles weaker than the function value oracles. Such issue can be mitigated through exploiting sparse gradient structure [89, 35, 19, 12, 13]. Indeed, the high-dimensional function has sparse gradients satisfying that (0)] S (IV)((()) for all 0 E IRd and some 8 < d.
The above discussions give the subroutine for estimating sparse gradients using comparison oracles. We generate m i.i.d. perturbation vectors from a uniform distribution (i.c., {z, hısism), compute Y C0 0 rz;) for all i, and solve the optimization problem in the following form of
g argmax
y:zk g
(2)
|
where the constraints |g
S
v and Ig <
ensure that g is sparse and normalized.
3
Main Results
We study how to use the information contained in noisy preference pairs that induce similar likelihood for preferred and dispreferred responses and achieve this goal by developing zeroth-order preference alignment method based on comparison oracles. We provide the convergence guarantee for the basic scheme and improve it to the practical scheme using some heuristics.

---

Algorithm 1 Comparison-Based Preference Alignment (Basic Scheme)
1: Input: initial parameter 01 E R2, stepsize vj 0, sparsity ratio d. sampling radius >> 0, querying
number m 1, and iteration number
2: for t = 1, ,....,,, do
3: Draw m Li.d. samples uniformly from unit sphere in R", i.c., {zj hgism.
4: Compute y; C. (0,, 02 + rz:) for i = ..2.....71. m1.
5: Compute &: = argmax &ll S.s. V... I E V.
6:
Compute 0,+1 02 vgi.
3.1 Basic scheme with convergence guarantee
We adapt the comparison oracles to the setup for direct preference alignment. Instead of optimizing the DPO objective function in Eq. (1), we assume that there exists an appropriate objective function [()) that better aligns the L.MSs with human preferences and optimize it. This function is complicated such that the function value oracles will not accessible. However, it intuitively makes sense that f(0') < (() if and only if Th exhibits a higher likelihood for yt and lower likelihood for y than ne given any pairwise preference data (x,y yT yy"y). Based on these insights, we have
We say C. (0,0') Rd x Rd + (+1,-1) preference comparison oracle for the model mh and a pair of preference data (x,y+.y" . frum the offline dataset D if
Definition 3.1
C. (0,0)) = { -1, if To (y > oo (x) and To (y (x) < Toly (x) for (x, yx E ED, +1, otherwise.
is worth remarking that the oracle (0, 0¹) provides a preference comparison between parameters 0 and O' based the model Ty and the offline preference data from D. Indeed, C, (0, 0') -1 indicates that Thy is a hetter model compared to Th, for the offline preference dataset DD. By leveraging these comparative assessments, our goal is to find the parameter 0 that minimizes the function (0) which can be nonconvex in general.
We present the basic scheme of our method in Algorithm 1, which can be interpreted as variant of the method [12]. It combines 1-bit gradient estimator from Eq. (2) with preference comparison oracles. However, the underlying objective function is assumed to be convex in [12], which is unrealistic in aligning LLMs with human preference. In what follows, we provide the convergence guarantee for Algorithm 1 given that there exists smooth yet nonconvex function which can be compatible with the preference comparison oracle and has sparse gradients.
Theorem 3.1 Suppose that there exists a smooth function satisfying (i) f(0') < <((0) if and only if x (y 1x) >> Toly
functiinn poly+(xxx [x) and Ter (y xx) < Toly [x) for V(x, y+,y E D and (ii) / (0)lh S v [())] For any e, A (0, 1), there exists some T >> O such that the output of Algorithm
with 7 2 r=
40EVa and m = (m (s log( 2d) + log( ((=)) (where cm > 0 is constant)
satisfies that P(min V ((:) < comparison oracles is bounded by
A and the total number of calls of the preference
o (
AA 2
2d
Ac² )))
PA
log
+ log
where l >> 0 is the smoothness parameter of ff (i.e., [V) vf(0'] < Ello 0'1l) and A 0 is an upper bound for the initial objective function gap, f(0)) info f(0) > O.
Remark 3.2 It is worth mentioning that we derive P(minısesT [V.((.) < < in the analysis (see also [62, 52]) but finding the best solution from {BI....,0) } is intractable since [V.()))) can not be estimated in practice. Nonetheless, this pest-iterate guarantee has been also used in other recent works that leverage comparison oracles to RLHF [84] and can be viewed theoretical benchmark. addition, the sample complexity bound independent of up to logarithmic factor thanks to gradient sparsity structure.
5

---

Algorithm 2 Comparison-Based Preference Alignment (Practical Scheme)
1: Input: initial parameter 01 00] R. scaling for stepsize 7 >> O. sampling radius > 0, querying
number m 1, clipping thresholds Ag. À and iteration number T
2: for t = 1, , do
3: Draw w ii.d. samples uniformly from unit sphere in R ii... (zi)isigm-
Compute yı C. ([0,; 0[], [01; Of + rz.]) for 1.2....,77..
5 M.R: and clip B2 by zeroing out the entries whose magnitude is less than Ag-
5: Compute S: = I 2. w1
Compute 0. 0f 2/ 1i
gi if [és,=-1)
and l441 B otherwise.
m
m
3.2 Practical scheme
clear that the basic scheme in Algorithm 1 is impractical primarily because the dimension d is at a billion level. The steps, including weight perturbation and 1-bit gradient estimation, are intractable due to their memory and computation demands. We also hope incorporate gradient clipping [67, 33, 61, 68, 99] to stabilize our method in practice.
Perturbations on output layer weights. Algorithm I consists of performing m perturbations on all model parameters, which is unacceptable due to the high computational and memory costs. To reduce costs, we restrict the perturbations to only the output layer weights in and draw m i.i.d. samples uniformly from a unit sphere in Rdd where do is the number of output layer weights.
Low-cost approximation to normalized sparse gradient estimation. The -noor constrained normalized gradient estimation problem (2) becomes computationally intractable when d reaches billions. We propose a practical approximation by first relaxing the é --orm constraint and then applying clipping. Specifically, we first compute the normalized gradient of the output layer:
2.2.
In the clipping step, we zero out the entries of g" falling below certain threshold Ag- The restriction- operations yield an approximate sparse solution (2) for the output layer weights. In addition, we adjust the stepsize based on {yi Intuitively, if the size of (i -1} is larger, it is more likely that g obtained from {(z,, y4) Js)ssmm leads to much progress for minimizing the function f(0). This motivates us to use larger stepsize. That being said, the stepsize
normalization-clippie
proportional [é:,=-1]] if m falls below a threshold m A, the estimator g lacks sufficient information, so we should skip it.
[111 is relatively large. However, the size of {i y/ -11
We summarize the practical scheme of our method in Algorithm 2. We can view this algorithm as micro- fittroffietuting approach designed for noisy preference pairs, which serves as a practical addition to existing direct preference alignment methods, such DPO [70] and SimPO [60].
Final method. By combining the above algorithm with existing methods, we propose unified preference alignment framework that leverages both clean and noisy pairwise preference data, consists of three steps: we first use reference model to divide the dataset into two subsets: clean and noisy. Then, we apply DPO on the clean preference pairs to obtain an initial policy; we label it DPOstean to differentiate it from applying DPO to all the data. In the third step, starting from the initial policy DPOcteann we apply Algorithm to only noisy preference pairs obtain a final policy. We label all the three steps as DPO + ComPO.
noisy if the lig-likelihood for preferred and dispreferred responses is similar with respect to the reference model (after SFT). This can be formalized follows,
Specifically, we say one pairwise preference data (x.yt: y E
log ( xx) log m [x] S 5,
where is a threshold. This curation of data inspired by Razin et [73] who has shown that the issue of likelihood displacement can be mitigated by filtering out the pairwise preference data with small centered hidden embedding similarity (CHES) score. While a soft interpolation based on reference-model deried confidence score is compelling idea, we argue that defining such noise

---

Evaluation results on AlpacaEval 2, Arena-Harde and MT-Bench under four model setups. LC and WR denote length-controlled win rate and win rate, respectively. Turn1 and Turn-2 represent the scores to the answers from the first and follow-up questions in multi-turn dialogue. Here, we run 5 trials for DPO D +ComPO and present the best trail performance.
Table 1:
Mistral-Instruce
Method
AlpacaEval
Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MTTBench
(%)
(%)
(R)
Turn-1 Turn-2 Avg. LC (%)
WR (%)
(R)
Turn-1 Turn-2
Arg.
DPO
9.71
6.27
2.9
6,20
5.38
5.79
24.14
16.71
14.4
6.28
5.42
5.86
9.41
6.52
3.0
6.18
5.22
5,70
23.89
16.15
14.2
6.11
5.34
5.73
Dio +CmPPO 11.66
6.55
3.2
6.22
5.32
5.77
26.17
18.32
10.5
7.78
7.63
7.69
Llama- 3-Base-BB
-lama-3-Instrucee
Method
AlpacaEval
Arena-Hard
MT-Beench
AlpacaEval 2
Arena-Hard
MTT-Bench
(%)
WR
WR (%)
Turn-1 Turn-2 Avg. LC(() WR (%)
WR (%)
Turn-1 Turn-2
Avg.
DPO
4.14
10.43
12.1
6.61
5.85
6,23
32.59
31.99
22.9
8,30
7,55
7.93
DPO
4.28
9.81
12.0
6.64
6.01
6.33
32.92
32.42
22.9
8.26
7.63
7.94
DPO DP +ComPO
5.39
10,93
12.1
6.60
6.28
6.44 35.79
35.03
23.1
8.39
7.71
8.05
level of dataset remains nontrivial without relying on thresholds such as 6. Even with this threshold, it remains unclear how to determine a suitable confidence score and interpolation scheme.
Our experimental results (see Table 1) highlight that filtering out the pairwise preference data with small log-likelihood margin is not always helpful, which supports the superiority of the CHES score. Indeed, the CHES score is defined based on the model's embedding geometry and better captures the similarity between preferred and dispreferred responses, while our metric based on log- liklliielhooo is weaker yet easy compute in practice. Nonetheless, our experimental results demonstrate that, despite a weaker similarity metric, our new method can effectively extract the information from the noisy preference pairs to further improve the performance of LLMs by a large margin in terms of length-controlled win rate (LC) (the higher LC means less verbosity) and mitigate the issue of likelihood displacement (see Table 3).
4 Experiment
We investigate the effectiveness of ComPO on aligning the LLMS with noisy preference pairs as an alternative to DPO and its variants. The objectives of the experiments include: (1) quantitative evaluation of length-comm cength-controlled win rate (LC), win rate (WR) and likelihood displacement; (2) a quantitative evaluation of computational and memory efficiency. We split the samples using ò = 3. For Mistral-7B models, we set r = 0.0005, m = 1600, Ag = 0.00022 and X = 0.2. For Llama-3-8B models and Gemma-2-it-99 model, we set r 0.00075, m 1800, A 0.00008 and À = 0.2. For the detailed information on datasets, models, and evaluation benchmarks, we defer to Appendix D. All the experiments are implemented in Python 30 with PyTorch 2.5.1 with 30 NVIDIA A40 GPUs each with 46 GB memory, equipped with Ubuntu 22.04.5 LTS.
4.1 Augmenting DPO and SimPO
We show that our method can utilize the information contained in noisy preference pairs to improve DPO and SimPO, especially in terms of LCC and with the strong model.
DPO. We separately train DPO and DPO + ComPO. Table I presents the performance in terms of both length-controllldd win rate (LC) and win rate (WR). In addition to the best result achieved by DPO + ComPO, we present the average performance over consecutive runs for all models and benchmarks in Table 8 (see Appendix B). The objective of doing this to evidence the robustness of our method in effectively leveraging noisy preference data pairs.
We have several interesting findings. First of all, DPO does not always outperform DPO but could be better when the model is strong (e.g., -lama-3-Instruct-Shh This possibly because our metric based log-likelihood margin is too simple to capture the similarity between preferred and dispreferred response, demonstrating the superiority of the CHES score [73]. Nonetheless, our metric easy to compute and our experimental results show that, despite weaker metrie, our method utilizes the noisy
7

---

Table Direct augmentation results on SimPO over different models and benchmarks.
AlpacaEval
Arena-Hard
MT-Bench
Model
Method
(C))) WR (5)
(R)
Turn-1
Turn-2
Avg.
SimPO
40.22
41.18
20.8
7.94
7.31
7.62
Mistral-Instruct-77B
SSiPPO ComPO
42.27
43.17
22.0
7.83
7.46
7.64
SimPO 48.71
43.66
36.3
7.91
7.42
7.66
Llama-3-Instruct-8B
SimPO ComPO
49,53
45.03
37.3
7.94
7.45
7.70
SimPO
60.36
55.59
61.1
9.07
8.47
8.77
Gemma-2-it-9B
SimPO + ComPO 62.42
57.20
61.1
8.99
8.58
8.79
Table 3: The log-likelihood for preferred and dispreferred responses for 3 independent trials with E {0.1, and the default values for all of other parameters. Each cell gives pair of log-likelihood for preferred and dispreferred responses (log to |x), log (y [x)) after trail of training. The results are indeed different since the perturbations {2:) sas are different for Trial 1. Trial 2 and Trial However, we find that the log-likelihood for preferred response increase and the log- likeliiikddd for dispreferred response decrease.
1lama-3-Instuuttee (log my xx), log to [x)) = 16.761 1001--47.44 17.410)
Y
Trial
Trial 2
Trial 3
0.1
(- 46444, -47.411)
(-4..660, -47.411)
-6..759 47.4100
(-46.728, -47.520)
(- 46.743, -47.525)
-6.753. 47.517)
Gemma-2-it-9B (log To (y x). log Aa (y [x)) (- -1345527
Trial
Trial 2
Trial 3
0.1 (-133.122 - 134.5577
133. 122, - 134.557)
(- 121, 134.
1 133.059, - -134.562)
133. 122, - -134.564)
(- 133.112, - -134.565)
pairs to improve the performance. Second, the improvement is large in terms of LC which accounts for admirable conciseness for the responses generated by DPOclcan method can alleviate the issue of verbosity which can be partially attributed to the presence of noisy pairs [32, 27, 66]. Thirdly, we remark that ComPO is only run with 100 noisy pairs but has achieved the consistent performance across most of benchmarks and models. This demonstrates the potential value of noisy pairs and our method in the context of aligning the LLMs with human preferences.
In other words, our
We also observe that DPO can outperform DPO CommPO by large margin on Arena-Hard Hard for 7B and the performance on Arena-Hard for Llama- Base- 8B and Masta-Inttuuc-88 are also indistinguishable. This possibly because Arena-Hard favors longer generations due to the absence of a length penalty in its evaluation (i.e., WR rather than LC) [60]. We report the average response length for Mistral-Instruct-77B and confirm this possibility; indeed, the average length is 513 for DPO and 468 with DPO CCmmP.. In other words, method's ss ability to alleviate the issue of verbosity leads to worse performance on Arena-Hard compared to DPO.
SimPO. Extending the compatibility of ComPO in augmenting DPO variants assists the community in advancing the existing directed alignment methods. Here, we focus on SimPO [60] and directly train on the well-tuned existing SimPO checkpoints. Table 2 presents the performance in terms of both LC and WR, where SimPO + ComPO consistently outperforms SimPO across all models and benchmarks. Notably, the improvement is larger on both AlpacaEval 2 and Arena-Hard compared to DPOctean + ComPO over DPO, demonstrating the superior compatibility of ComPO in augmenting SimPO. It is also worth remarking that SimPO ComPO achieves the consistent improvement on Arena- Hard in terms of WR. highlighting that SimPO and SimPO + ComPO generate the concise responses and ComPO further augments SimPO in terms of the quality of generated responses.
Table 3 presents the log-likelihood for preferred and dispreferred responses for independent trials with y E (0.1, 1} and the default values for other parameters. We present the results for Llama-3- Instruct-8B and -a-m------t-9- and think suffices to show that ComPO is effective, In contrast, the results for Mistral-7B is mixed possibly because the likelihood displacement can be caused by limited model capacity [82]. We have two important findings. First of all, the comparison oracles defined in Eq. 13.1) can return the informative signals for estimating the normalized gradients to both increase the likelihood for the preferred response and decrease the likelihood for the dispreferred response. For examplc, when the model is Llama- I-nsrructtttttttt- 8B and y 1, the log- likeliild for
8

---

Zero Entries After Thresholding
Peak GPU Hemory Usage
Perturbing ing Size and Run Time Fan tnc)
102
25
Hocimam Single Humery 146000
78
02
350
Mamory (GD)
Partu Sine
Gamna-22 6.62%
Nen-Zere
Unme 2.40%
21.108
0.1
16:300
20
16--
0.33
300
Miktnal
500.0
3.50.00
2. 24
18 10
00 103P90
56 (0atmaa
72 Phavob
80:33 :33mal
50 1Dumnu
Figure 1: (Left) Percentage of non-zero zero entries in the final gradient across different gradient entry threshold Xg; (Middle) Peak GPU memory usage across three models used in all experiments; (Right) Size of parameter space (output layer) in the comparison oracle perturbations and the run time for completing 600 perturbations using 30 NVIDIA A40 GPUs shown.
Table 4: Effect of the number of perturbations nn on model performance. We report WR and LC on AlpacaEval 2: each entry includes the results of mean performance and standard deviation over 5 consecutive runs; the best performance is shown in the parentheses.
Perturbation (m)
800
1600
3300
5400
AlpacaEval 2-WRR % 17.32 + 0.86 (17.94) 17.50 = 0.65 (18.32) 19.21 + 0.58 (20.25) 19.69 + 0.36 (20.07) AlpacaEval 2-LC % 24.72 + 1.02 (25.12) 25.02 + 0.91 (26.17) 25.91 = 0.95 (27.14) 26.49 + 0.81 (27.20)
Table 5: Multi-layer perturbation improves performance. We report WR and LC on AlpacaEvall 2 and WR on Arena-Hard; entries are mean std over 5 runs, with the best in parentheses.
Layers perturbed (# params) AlpacaEval 2-WR % AlpacaEval 2-L.C % Arena-Hard (GPT 4.1)-WR %
(0.13B)
17.50 = 0.65 (18. (18.2)) 25.02 + 0.91 (26.17)
10.80 + 0.21 (11.0)
(0.25B)
18.19 + 0.81 (19.3 26.00 00. 1 0.89 (27.09)
11.26 + 0.36 (11.7)
preferred and dispreferred responses after the first trail of training is (-46.728, 447.520) while the initial log-likelihood for preferred and dispreferred responses is (-46.761, -47.400) It is clear that -46. 726 > -46.761 and -47.520 < -4..410. In other words, it can help alleviate the issue of likelihood displacement by utilizing the noisy preference pairs. Second, we impose the thresholds Ag. A to retain only the significant gradient entries. This promotes stability by minimizing unnecessary changes to the original model parameters and effectivvnees by allowing for using larger stepsizes. However, also find that too large stepsizes Icad to unstable training which is consistent with the convergence guarantee obtained for the basic scheme (see Theorem l). Can we develop principle way to choose the stepsize in fully adaptive manner? We leave the answers to future work.
4.2 Ablation and scaling analysis
Perturbations allow the oracle to explore different gradient directions, providing richer information as m increases. To study this effect, we vary m while keeping other parameters fixed; we conduct the experiments Mistral--nstruct-778 (see Table 4). We observe that increasing m improves WR and LC, confirming the convergence in Theorem 3.1, but at the cost of higher compute time. Importantly, peak memory usage remains unchanged, as ComPO does not store individual perturbation vectors only the running average gradient estimates are maintained (see Line 5 of Algorithm 2).
To demonstrate that ComPO scales beyond single-layer fine-tuning, we perturb three layers (the MLPs in layers 30-31 and the output layer) of -B-strruut while keeping all other settings fixed. Results are shown in Table 5. Notably, under more robust GPT 4.1 judge. Perturbing more layers yields better performance by expanding the set of gradient directions. The peak GPU memory increases mildly from 16.3GB to 16.7GB, and running time per 600 perturbations takes 60 seconds (vs. 50 seconds), which we consider reasonable.
present the Arena-Hard result with a recent version
To ensure ComPO's s scalability along with stable training, we applies gradient entry threshold Ag. which selectively updates only high-magnitude gradient entries from oracles. We conducted ablation analysis for Xg under found that setting Ag to retain 1%-5% of entries offers the best trade-off between performance and stability. An excessively high threshold over-filters gradient information, whereas excessively low
3300 in Mistral-7B-Instruu training; results are presented in Table 6. We

---

Table Effect gradient threshold X Results are WR and LC on AlpacaEval 2; entries are mean std over 5 runs, with the best run in parentheses. A/
1x
1,8x10
222x10 4
2.6x10-+
Percentage of gradient entries updated
100%
63%
6%
15
0.15%
AlpacaEval 2-WR % AlpacaEval 2-LC
15.72 4 0.77 (16.34)
16.02 d: 0.69 (16. 69)
19.02
= 0.02 (20.15)
19.21
0.58 (20.25)
16,10 4: 0,11 (16.21)
23.42 1.0 (24.28) 24.01
(25.10) 26.06
0.81 (27.27) 26.91
0.95 (27.14) 23.82 0.23 (24.00)
Table 7: Results on scaling the number of noisy preference pairs used in the training.
Number of noisy pairs| AlpacaEval 2-WR % AlpacaEval 2-LC % Arena-Hard (GPT 4.1)-WR %
100
19.21 + 0.58 (20.25)
25.91
0.95 (27.14)
11.02 + 0.13 (11.2)
300
20.07 + 0.99 (21.35) 26.28 + 0.81 (27.59)
11.76 + 0.30 (12.1)
threshold injects noise into the gradients, bringing instability. We also extend the analysis to 300 noisy pairs and observe consistent improvements on AlpacaEval2 and Arena-Hard in Table 7.
4.3 Practical efficiency and compatibility
While full fine-tuning (i.e., updating all model parameters) and LoRA-based fine-tuning [39] are two common post-training approaches, we consider using more lightweight yet effective alternative fine-tuning, which only updates small portion of parameters in the output layer. Figure 1 (left) shows that the chosen value of X retains only about 1% of the output layer parameters for Mistral-7B and Llama-3-8B. This indicates that fine-tuning a small portion of output layer parameters sufficient for effective alignment. Specifically, only 0.0002% of the total 7B parameters are updated for Mistral-7B, while the remaining parameters are kept frozen during ComPO training.
This fine-tuning approach offers some ad advantages. First, it significantly reduces memory usage as only one extra output-layer vector needs to be saved and updated during each iteration. Figure I (middle) shows that we only require around 23GB memory for each A40 GPU to run ComPO for Llama-3-8B while the peak memory for running DPO and SimPO is 77GB and 69GB on H100 GPUs. Second, it significantly improves time efficiency by appeal to the favorable parallelization properties of collecting comparison oracle feedback and accumulating perturbation signals (see Algorithm 2). For example, if one performs 600 perturbations using 30 A40 GPUs, each worker node only processes 20 perturbations, while the master node collects comparison oracle feedback along with accumulated perturbation signals from all worker nodes to compute the gradient estimator. Figure/ I (right) shows that the run time scales as linear function of the size of perturbation parameter space. In addition, different models can have different structures and output layer sizes and we perturbed the complete 1m_head layers across all models in our experiment for consistency.
We highlight that ComPO can effectively exploit noisy pairs and allow users with an alternative choice to run it directly on existing checkpoints with its noisy pairs. This especially valuable if users do not have large memory GPUs but still want to finetune public checkpoints on the noisy pairs that are useful but cannot be effectively utilized by DPO. To directly illustrate this, we applied ComPO to DPO checkpoints without separating noisy and clean pairs and observed comparable improvements (with m = 3300); results are shown in Table 10 in Appendix B. In practice, one could start with an existing, publicly available model that has already been aligned on general, high-quality dataset. ComPO empowers a user to take that pre-aligned public model and further refine it using their own, potentially noisy and task-specific, preference data, using more affordable GPU platform.
5 Conclusion
We propose a new zeroth-order preference alignment method based on comparison oracles and show that it can improve the performance of large language models (LLMs) using the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. Experimental results on multiple models and benchmarks show the effectiveness of our method to mitigate the issues of verbosity and likelihood displacement. This shows the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings [73]. Future directions include the extension of our method to other settings [97, 94, 82, 37] and applications of our method to more challenging tasks, such reasoning [65] and diffusion model alignment [87].
10

---

Acknowledgment
We sincerely appreciate Buzz High Performance Computing (https: / //ww. buzzhpc. info@buzzhpc.ail ai) for providing computational resources and support for this work.
References
[1] J. Achiam, S. Adler, Agarwal, L. Ahmad, Akkaya, L. Aleman, D. Almeida, Al-
tenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report. ArXiv Preprint: 2303.08774, 2023. (Cited on page 1.)
[2] A. Agarwal, O. Dekel, and L. Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In COLT, pages 28 28-40, 2010. (Cited on page 26.)
[3] R. Akrour, M. Schoenauer, and M. Sebag. Preference-base policy learning. In ECML PKDD, pages 12-27, 2011. (Cited on page 26.)
[4] A. Amini, T. Vieira, and R. Cotterell. Direct preference optimization with an offset. In ACL, pages 9954-9972. 2024. (Cited on pages and 26.)
[5] R. Astudillo and P. Frazier. Multi-attribute Bayesian optimization with interactive preference learning. In AISTATS, pages 4496-4507, 2020. (Cited on page 3.)
[6] M. G. Azar, Z. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, pages 4447-4455, 2024. (Cited on pages and 25.)
[7] Bai, A. Jones, Ndousse, A. Askell, A. Chen, DasSarma, D. Drain, S. Fort, Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv Preprint: 2204.05862, 2022. (Cited on page 1.)
[8] P. T. Boufounos and R. G. Baraniuk. 1-bit compressive sensing. In CISS, pages 16-21. IEEE, 2008. (Cited on page 4.)
[9] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurIPS, pages 1877-1901, 2020. (Cited on page 1.)
[10] S. Bubeck, V. Chandrasckaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, Lee, T. Lee, Y. Li, S. Lundberg, al. Sparks of artificial general intelligence: Early experiments with GPT-4. ArXiv Preprint: 2303. 127 12, 2023. (Cited on page 1.)
[11] R. Busa-Fekete, B. Szörényi, P. Weng, W. Cheng, and E. Hüllermeier. Preference-basee reinforcement learning: Evolutionary direct policy search using a preference-base baaeed racing algorithm. Machine learning, 97:327-351, 2014. (Cited on page 26.)
[12] H. Cai, D. McKenzie, W. Yin, and Z. Zhang. A one-bit, comparison-basse based gradient estimator. Applied and Computational Harmonic Analysis, 60:242-266, 2022. (Cited on pages and 26.)
[13] H. Cai, McKenzie, W. Yin, and Z. Zhang. Zeroth-order regularized optimization (ZORO): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2): 687-714, 2022. (Cited on page 4.)
[14] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C-R. Ségerie, M. Carroll, Peng. P. K. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, Krasheninnikov, X. Chen, L. Langosco, P. Hase, E, Biyik, A. D. Dragan, Krueger, Open problems and fundamental limitations of reinforce- ment learning from human feedback. Transactions on Machine Learning Research, 2023. URL https:/ // http//ppervvieeeeeeeeeee
D. Sadigh, and D. Hadfield-Ment
(Cited page 25.)

---

[15] H. Chen, G. He. L. Yuan, G. Cui, H. Su, and J. Zhu. Noise contrastive alignment of language models with explicit rewards. In NeurIPS, pages 117784-117812 2024. (Cited on page 3.)
[16] H. Chen, H. Zhao, H. Lam, D. Yao, and W. Tang. MallowsPO: Fine-tune your LLM with preference dispersions. In ICLR, 2025. URL id=dBcnezVcaw. (Cited on pages and 25.)
[17] X. Chen, S. Liu, K. Xu, X. Li, X. Lin, M. Hong, and D. Cox. ZO-AdaMM: zeroth-order adaptive momentum method for black-box optimization. In NeurlPS, pages 7204-7215, 2019. (Cited on page 26.)
[18] M. Cheng, S. Singh, P. H. Chen, P-Y. Chen, S. Liu, and C-J. Hsieh. Sign-OPT: A query- efficient hard-label adversarial attack. In ICLR, 2020. URL https://oppenreview net/ forum?ide SK1N (Cited on page 3.)
[19] K. Choromanski, A. Pacchiano, J. Parker-Holder, Y. Tang, and V. Sindhwani. From complexity to simplicity: Adaptive ES-Active subspaces for blackbox optimization. In NeurIPS, pages 10299-10309, 2019. (Cited on page 4.)
[20] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-115 2023. (Cited page 1.)
[21] P.F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NeurIPS, pages 4302-4310, 2017. (Cited on page 1.)
[22] E. Conti, V. Madhavan, F. P. Such, J. Lehman, K. O. Stanley, and J. Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty- seeking agents. In NeurlPS, pages 5032-5043, 2018. (Cited on page 26.)
[23] G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with scaled AI feedback. In ICML, pages 9722-9744, 2024. (Cited on page 29.)
[24] Y-X. Ding and Z-H. Zhou. Preference based adaptation for learning objectives. In NeurIPS, pages 7839-7848, 2018. (Cited on page 3.)
[25] H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, Zhang, K. Shum, and T. Zhang. RAFT: Reward ranked fine-tuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. URL https //ppnrevvieeeeeeeeee net/ forum?id=mn (imm//ddmmpp502by (Cited page 3.)
[26] H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. RLHF workflow: From reward modeling to online RLHF, Trunsactions on Ma- chine Learning Research, 2024. URL https: htt.https://openeee //tpn//epppeeeeeeee/eeee wwwprrreeeieeeeeeoereeeeeeeeeeee (Cited on page 25.)
[27] Y. Dubois, X. Li, R. Taori, T. Zhang. I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In NeurIPS, pages 00039-30069, 2023. (Cited on pages 2 and 8.)
[28] Y. Dubois, P. Liang, and T. Hashimoto. Length-controlled AlpacaEval: A simple debiasing of automatic evaluators. In COLM, 2024. URL httpp:::///peeeeviee.... Cy CyBBWWWXO.. (Cited on page 29.)
[29] J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero- order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5))2788-288 2015. (Cited on page 26.)
[30] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Model alignment as prospect theoretic optimization. In JCML, pages 12634-12651, 2024. (Cited on pages and 25.)
[31] D. Flaxman, A. T. Kalai, and H. B. McMahan. Online convex optimization in the bandit setting: Gradient descent without gradient. In SODA, pages 385-394, 2005. (Cited on page 26.)
12

---

[32] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In ICML, pages 11835-10866, 2023. (Cited on pages and 8.)
[33] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. In /CML, pages 1243-1252, 2017. (Cited on page 6.)
[34] S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341 2013. (Cited on page 26.)
[35] D. Golovin, J. Karro, G. Kochanski, C. Lee, X. Song, and Q. Zhang. Gradientless descent: High-diienniioal zeroth-order optimization. In ICLR, 2020. URL https:// net/fo ft////ormmm// id- (Cited on page 4.)
[36] J. Guo, 7. Li, J. Qiu, Y. Wu, and M. Wang. On the role of preference variance in preference optimization. ArXiv Preprint: 2510. 13022, 2025. (Cited on page 25.)
[37] S. Guo, B. Zhang. T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, al. Direct language model alignment from online AI feedback. arXiv preprint arXiv:2402.04792, 2024. (Cited on page 10.)
[38] J. Hong, N. Lee, and J. Thorne. ORPO: Monolithic preference optimization without reference model. In EMNLP. pages 11170-11189, 2024. (Cited on page 3.)
[39] E. Hu, Y. Shen, Wallis, Z. Allen-Zhu, Y. Li, Wang. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL net/ fet/ffrum?? ieddidddeeee W (Cited on page 10.)
[40] F. Huang, S. Gao, J. Pei, and H. Huang. Accelerated zeroth-order and first-order momentum methods from mini to minimax optimization. Journal of Machine Learning Research, 23(36):
1-70, 2022. (Cited on page 26.)
[41] K. G. Jamieson, R. Nowak, and B. Recht. Query complexity of derivative-free optimization. In NeurlPS, pages 2672-26012 2680, 2012. (Cited on pages and 4.)
[42] J. Ji, M. Liu, Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of LLM via a human- -uuman-preferenee dataset. In NeurlPS Dataset and Benchmark Track, 2023. URL htttps://openreviert htt.:/ppprreviee..eeeffrrrrrm
id=g0novXbFu3. (Cited on page 25.)
[43] K. Ji, Z. Wang, Y. Zhou, and Y. Liang. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. ICML, pages 3100-3109, 2019. (Cited on page 26.)
[44] S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang. stack overflow obsolete? an empirical study of the characteristics of ChatGPT answers to stack overflow questions. In CHI, pages 1-17, 2024. (Cited on page 2.)
[45] K. Kim, A. Seo, H. Liu, J. Shin, and K. Lee. Margin matching preference optimization: Enhanced model alignment with granular feedback. In EMNLP, pages 13554-13577 2024. (Cited on page 26.)
[46] G. Kornowski and O. Shamir. An algorithm with optimal dimension-deependence for zero-order nonsmooth nonconvex stochastic optimization. Journal of Machine Learning Research, 25 (122):1-14, 2024. (Cited page 26.)
[47] W. Kumagai. Regret analysis for continuous dueling bandit. In NeurlPS, pages 1488-1497, 2017. (Cited on page 3.)
[48] T. Li, W-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. ArXiv Preprint: 2406. / 1939, 2024. (Cited page 29.)
[49] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-following models. https://gitan com/tatsu-Ie labl alpaca_eval, 5 2023. (Cited page 29.)
13

---

[50] X. Lian, H. Zhang, C-J. Hsieh, Y. Huang, and J. Liu. A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order. In NeurIPS, pages 3062-3070, 2016. (Cited on page 26.)
[51] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. In ACL, 2022. (Cited on page 25.)
[52] T. Lin, Z. Zheng, and M. I. Jordan. Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization. In NeurIPS, pages 26160-26175, 2022. (Cited on pages and 26.)
[53] Z. J. Lin, R. Astudillo, P. Frazier, and E. Bakshy. Preference exploration for efficient Bayesian optimization with multiple outcomes. In AISTATS, pages 4235-4258, 2022. (Cited on page 3.)
[54] S. Liu, B. Kailkhura, P-Y. Chen, P. Ting. S. Chang, and I.. Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. In NeurIPS, pages 3731-3741, 2018. (Cited on page 26.)
[55] T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Salch, J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization, In ICLR, 2024. URL httpttsss//ppperrieeiiieee net/
(Cited on page 25.)
[56] T. Liu, Z. Qin, J. Wu, J. Shen, M. Khalman, R. Joshi, Y. Zhao, M. Saleh, S. Baumgartner, J. Liu, al. LiPO: Listwise preference optimization through learning-to-rrn to-rank. In NAACL, page To appear. 2025. (Cited on page 3.)
[57] Z. Liu, M. Lu, S. Zhang, B. Liu, H. Guo, Y. Yang. J. Blanchet, and Z. Wang. Provably mitigating overoptimization RLHF: Your SFT loss is implicitly an adversarial regularizer. In NeurlPS, pages 138663-138697, 2024. (Cited on pages 2, 3, and 25.)
[58] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In NeurIPS, pages 38038-53075, 2023. (Cited on page 26.)
[59] K. Matsui, W. Kumagai, and T. Kanamori. Parallel distributed block coordinate descent methods based on pairwise comparison oracle. Journal of Global Optimization, 69:1-21, 2017, (Cited on page 3.)
[60] Y. Meng, M. Xia, and D. Chen. SimPO: Simple preference optimization with a reference- free reward. In NeurlPS, pages 12419--122335, 2024. (Cited pages 1, 2, 3, 6, 8, 25, 26, and 29.)
[61] S. Merity, N. S. Keskar, and R. Socher. Regularizing and optimizing LSTM language models. In ICLR, 2018. URL https: : /ttp:////pee/eeeeeeeeeeee nwt sett/forurr/dd-SyyFPPor... (Cited on page 6.)
[62] Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017. (Cited on pages 5 and 26.)
[63] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, pages 27730-27744 2022. (Cited on page 1.)
[64] A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. ArXiv Preprint: 2402. 2024. (Cited on pages 2, 3, 4, and 26.)
[65] R. Y. Pang, W. Yuan, H. He, K. Cho, S. Sukhbaatar, and J. Weston. Iterative reasoning preference optimization. In NeurlPS, pages 116617-116637, 2024. (Cited on pages 2, 3, 10, and 26.)
[66] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. In ACL, pages 1998-5017, 2024. (Cited on pages 1, 2, 3, 8, and 25.)
[67] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In ICML, pages 1310-1318. 2013. (Cited on page 6.)
14

---

[68] M. E. Peters, M. Neumann, L. Zettlemoyer, and W-T. Yih. Dissecting contextual word embeddings: Architecture and representation. EMNLP, pages 1509, 2018. (Cited on page 6.)
[69] Y. Plan and R. Vershynin. Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach. JEEE Transactions on Information Theory, 59(1):482-494 2012. (Cited on pages 4 and 26.)
[70] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and Finn, Direct preference optimization: Your language model is secretly reward model. In NeurlPS, pages 53728- 53741, 2023. (Cited on pages 1, 3, 6, and 25.)
[71] R. Rafailov, Y. Chittepu, R. Park, H. Sikchi, J. Hejna, W. B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization direct alignment algorithms. In NeurIPS, pages 126207-126222, 2024. (Cited on pages 2 and 3.)
[72] R. Rafailov, J. Hejna, R. Park, and C. Finn. From SrS to *5 Your language model is secretly a Q-function. In COLM, 2024. URL https://///eeerr (Cited on pages 2, 3, 25, and 26.)
[73] N. Razin, S. Malladi, A. Bhaskar, D. Chen, S. Arora, and B. Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. In ICLR, 2025. URL https: /ttp//rrrmiuuaasSIDnnniit.
/ //openrevier
(Cited on pages 1, 2, 3, 4, 6, 7, 10, and 26.)
[74] Y. Ren and D. J. Sutherland. Learning dynamics of LLM finetuning. In ICLR, 2025. URL httpf/orrrm111tt/Ho000111 (Cited on page 26.)
https://openreviort
[75] T. Salimans, J. Ho, X. Chen, S. Sidor, and Sutskever. Evolution strategies as a scalable alternative reinforcement learning. ArXiv Preprint: 1703.03864, 2017. (Cited on page 26.)
[76] Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research, 18(1):1703-1715 2017. (Cited on page 26.)
[77] R. Shi, R. Zhou, and S. S. Du. The crucial role of samplers in online direct preference optimization. In ICLR, 2025. URL https:: (Cited on page 25.)
[78] P. Singhal, T. Goyal, J. Xu, and G. Durrett. A long way to go: Investigating length correlations in RLHF, In COLM, 2024. URL attps://ooperrevier. wttp://www..et.ooorm..ee/e/..... (Cited on page 2.)
[79] F. Song, B. Yu, M. Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for human alignment. In AAAI, pages 18990-18998, 2024. (Cited on page 3.)
[80] Y. Song, G. Swamy, A. Singh, J. Bagnell, and W. Sun. The importance of online data: Understanding preference fine-tuning via coverage. In NeurlPS, pages 12243-12270, 2024. (Cited on page 25.)
[81] N. Stiennon, I.. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. In NeurIPS, pages 3008-3021, 2020. (Cited on pages and 3.)
[82] F. Tajwar, Singh, A. Sharma, R. Rafailov, Schneider, Xie, S. Ermon, C. Finn, and A. Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In ICML, pages 47441-47474, 2024. (Cited on pages 2, 3, 8, 10, and 26.)
[83] Y. Tang, Z. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. In ICML, pages 47742, 2024. (Cited on pages and 25.)
[84] Z. Tang, D. Rybin, and T-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. In ICIR, 2024. URI. hww.::ttt////eee //tttpps///peeeevieeee net/ forum?id-TVDUypgue (Cited pages 3, 5, and 26.)
15

---

[85] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. ArXiv Preprint: 2302. 13971, 2023. (Cited pages and 3.)
[86] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NeurlPS, pages 6000-6010, 2017. (Cited on page 3.)
[87] Wallace, M. Dang. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, Xiong. S. Joty, and N. Naik. Diffusion model alignment using direct preference optimization. In CVPR, pages 8228-822 8238, 2024. (Cited on page 10.)
[88] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer, S. TT. Truong, S. Arora, M. Mazeika, Hendrycks, Lin, Y. Cheng, Koyejo, Song, and B. Li. Decodingtrust: A comprehensive assessment of trustworthiness in GPT models. In NeurlPS Dataset and Benchmark Track, 2023. URL UtL.tttps///poeeerevve /
net/forum?
id=kaHlpo80ww. (Cited page 25.)
[89] Y. Wang. S. Du, S. Balakrishnan, and Singh. Stochastic zeroth-order optimization high dimensions. In AISTATS, pages 1356-1360 1365, 2018. (Cited pages and 26.)
[90] T. Xiao, Y. Yuan, H. Zhu, M. Li, and V. G. Honavar. Cal-DPO: Calibrated direct preference optimization for language model alignment. In NeurIPS, pages 114289-114320, 2024. (Cited on page 26.)
[91] T. Xie, J. Foster, A. Krishnamurthy, C. Rosset, A. H. Awadallah, and A. Rakhlin. Exploratory preference optimization: Harnessing implicit q" -aproxximation for sample-efficient RLHF. In ICLR, 2025. URL https: /ttp///ppeeeeeeeeeeeeeeee wtttt/oorumidddoyieoeee
(Cited page 25.)
[92] W. Xiong, Dong, C. Ye, Z. Wang. H. Zhong, H. Ji, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. In ICML, pages 54715-54754, 2024, (Cited on page 25.)
[93] H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance machine translation. In /CML, pages 55204-55224, 2024. (Cited pages 1, 2, and 25.)
[94] S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, Wang, C. Yu, and Y. Wu. DPO superior to PPO for LLM alignment? comprehensive study, In ICML, pages 54983-54998, 2024. (Cited on page 10.)
[95] H. Yuan, Z. Yuan, C. Tan, W. Wang. S. Huang, and F. Huang. RRHF: Rank responses to align language models with human feedback. NeurIPS, pages 10935-10950, 2023. (Cited on page 3.)
[96] L. Yuan, G. Cui, H. Wang. N. Ding, X. Wang, B. Shan, Liu, Deng, H. Chen, R. Xie, Lin, Z. Liu, B. Zhou, H. Peng, Z. Liu, and M. Sun. Advancing LLM reasoning generalists with preference trees. In ICLR, 2025. URL htt.tttt://oppeeeiiiee wttp://www....eeeoee.meeedeeeeee (Cited pages 2, 3, and 26.)
[97] W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, Xu, and E. Weston. Self-rewarding language models. In ICMI, pages 57905-57923, 2024. (Cited on page 10.)
[98] Yue and T. Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In /CMI,, pages 1201-1208, 2009. (Cited on page 3.)
[99] J. Zhang, T. He, S. Sra, and A. Jadbabaie. Why gradient clipping accelerates training: theoretical justification for adaptivity. In ICLR, 2020. URL. https: http///pppwweeee.e//////e net/ f forum?Sd-BlgaxpPVks. (Cited on page 6.)
[100] Q. Zhang and L. Ying. Zeroth-order policy gradient for reinforcement learning from human feedback without reward inference. ICLR, 2025. URL https: httpt:tt/tpp/s//eeeeeee Dormm?d=cmYScmfii (Cited pages and 26.)
h/tp:///epp/eeeee//////e net/
16

---

[101] S. Zhang, Z. Liu, B. Liu, Y. Zhang, Y. Yang, Y. Liu. L. Chen, T. Sun, and Z. Wang. Reward- augmented data enhances direct preference alignment of LLMs. In ICLR Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025. URI. https:
/ //oppnrevieww... wwt. http://oounnidddppSse0
(Cited on page 26.)
[102] Y. Zhang, P. Li, J. Hong. J. Li, Y. Zhang, W. Zheng, P-Y. Chen. D. Lee, W. Yin, M. Hong. et al. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: a benchmark. In ICML, pages 59173-59190 59190, 2024. (Cited on page 26.)
[103] H. Zhao, G. I. Winata, A. Das, S-X. Zhang, D. Yao, W. Tang, and S. Sahu. RainbowPO: A unified framework for combining improvements in preference optimization. In ICLR, 2025. URL https: // wwwpoopeeiiieeeeee mttp://orummiiitrreeee.....
(Cited on pages and 25.)
[104] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. Wildchat: Im chatGPT interaction logs in the wild. ICLR, 2024. URL https:://operreeiiieieeee/// wttprrewiiiieenefffrrereeee www.:eeeeeneniiemmmmm Bd-BB8u7ZRBbbb (Cited on page 25.)
[105] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Salch, and P. J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv Preprint: 10425, 2023. (Cited on page 25.)
[106] L. Zheng, W-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, and E. P. Xing. Judging LLM-as-a-Judge with MT-bench and Chatbot Arena. In NeurlPS, pages 46595 46623, 2023. (Cited on page 29.)
[107] B. Zhu, M. Jordan, J.d Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In ICML, pages 43037-43067. 2023. (Cited on page 25.)
[108] M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, Amodei, P. Christiano, and
G. Irving. Fine-tuning language models from human preferences. ArXiv Preprint: 1909.08593, 2019. (Cited on pages and 3.)
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main argument and contribution is presented in the abstract, and Section 1 offers a detailed explanation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
We include the complete proof in Appendix C for the theorem in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss the choice of hyperparameters for our algorithm in Section 4 and include the code of our implementation for reproducibility in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We provide the detailed information for the dataset, SFT model, and code implementation our experiments setup section.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
httpc//uuiies//ooeuubiiissssesee ceiiionnoiilyyyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.ccoooccc wwtpppccccgeeeeeeecceccccceeeeee gewggu//oodessiie wwt.SSummisiiooooii asiionylicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide clear pipeline for the training configuration in experiment setup section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run our main experiment for 5 consecutive runs and report multiple statistics to show the robustness of our method. This is also done for the hyperparameter experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the GPU configuration for our experiment and discuss the runtime analysis in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///neurippp cccppublcc/thhcccculleeeeeeee
Answer: [Yes]
Justification:
Our worm conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is to improve preference alignment and does not have direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not involve releasing data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We carefully listed all the owners of the assets used in the paper and provide URLs to all assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We released our trained models publicly.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work is methodological work that does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ htt.:/pss/ccoooteeeenee/////222 for what should or should not be described.
24

---

A Limitations
Algorithm design. First, although our method achieves computational and memory efficiency, scalability, and strong performance across multiple benchmarks and models by finetunntnnnn only the output-layer parameters. we are currently unable to perform full-layer perturbation due to limited computational resources. We expect that perturbing all model parameters could further enhance preference alignment. Second, our method is a purely offline approach, which is similar to DPO and its variants, aligns the model strictly to the preference dataset, potentially limiting its ability to explore beyond observed data. Finally, the margin- thresshld parameter distinguishing noisy from clean data plays a crucial role in our method's 's effectiveness. To control training costs, we focus primarily on how method improves learning from noisy pairs. An important direction for future work is to investigate its performance on clean pairs and assess whether it could ultimately serve as a drop-in replacement for DPO when tuning the model on the full dataset.
Alignment applications beyond helpfulness. Our experiments primarily focus on the Ulrraeeddbckk dataset, which is mainly aimed at improving model helpfulness. Future work should explore our method's capability in aligning models along additional dimensions, such as safety and truthfulness, using relevant datasets and benchmarks [104, 42, 88, 51].
Performance drop on Arena-Hard, As discussed in Section 4, the poor performance on Arena-Haun may be due to the judge's ss bias toward longer responses, coupled with the shorter outputs generated by our method. This is exemplified by the observed decrease in our method's 's average output length in cases where Arena-Hard performance declines. It is also important to note that the noisy pairs vary across models, meaning that our method may be updated using different subsets of data and thus learn different aspects of helpfulness from the preference dataset.
B Further Related Works
We make the comments on other topics, including more discussions on preference learning methods, the analysis of preference learning methods, zeroth-order optimization methods, and the likelihood displacement. For an overview of preference learning methods, we refer to the recent survey [14].
More discussions on preference learning methods. The lack of explicit reward models in DPO [70] is known to constrain its ability to the size and quality of offline preference pairs. To address the limitation, subsequent works proposed to augment preference data using a trained SFT policy [105] or a refined SFT policy with rejection sampling [55]. The DPO loss was also extended to token-level MDP [72] given that the transition is deterministic, i.e., the next state is determined if the current state and action are chosen, which covered the fine-tuning of LLMs. Then, [6] generalized DPO to wider class of RL problems without the notion of reward function. Instead of maximizing the reward in a KL-constrained problem, they proposed to optimize a general non-decreasing function of the ground -tuuh population-level preference probability. There are several other DPO variants [30, 66, 93, 60, 16, 103, 36). For example, [30] aligned policy with preferences and designed the loss using a prospect theory, [83] optimized a general preference loss instead of the log- log-liieldoood loss, and [60] aligned the reward function in the preference optimization objective with the generation metric. [26] and [92] proposed to generate human feedback in an online fashion to mitigate the distribution-shift and prrereeteettiiiiiioonn phenomenon. There is the attempt to understand the theoretical performance of DPO [6], but the authors only showed the existence of optima of the loss function, without any policy optimality and sample complexity guarantees.
Analysis of preference learning methods. In this context, [107] formulated RLHF as the contextual bandit problem, and proved the convergence of the maximum likelihood estimator, [92] showed the benefits of KL-regularization in sample complexity of online exploration in DPO. [91] studied the problem of online exploration using KL-regularized Markov decision processes, and proved the sample complexity guarantee of exploration bonus. [57] investigated the issue of optimiitinno and proved the sample guarantees. [80] conducted a rigorous analysis through the lens of dataset coverage to differentiate offline DPO and online RLHF, Recently, several works have also reported faster convergence rate than the nntrmatiionthhoreete lower bounds for online reward maximization in RI by exploiting the structure induced by KI. regularization. For example, [77] studied the tabular softmax parametrization setting and established quadratic convergence results.
25

---

Zeroth-order optimization methods. The idea for zeroth-order optimization is to approximate
gradient using either a one-point estimator [31] or a two-point estimator [2. 34, 29, 76, 62]. where the latter approach achieves better finite-time convergence guarantee. Despite the meteoric rise of two- point-based gradient- free methods, most of the work is restricted to convex optimization [29, 76, 89] and smooth and nonconvex optimization [62, 34, 50, 54, 17, 43, 40). The convergence guarantees are obtained for both nonsmooth and convex setting [29, 76] and smooth and nonconvex setting [34, 62]. Additional regularity conditions, e.g.. a finite-sum structure, allow to leverage variance-reduction techniques [54, 17, 43] and the best known convergence guarantee is obtained in [40]. Very recently, the zeroth-order optimization methods have been developed for nonsmooth nonconvex optimization with solid theoretical guarantee [52, 46]. In another direction, the zeroth-order optimization methods were extended to the RL setting and have achieved
empirical success scalable alternative to classic methods such as Q-learning or policy gradient methods [75, 22]. This strategy has also been applied in preference- based RI [3, 11] and adopted to the LLM fine-tuning [58, 102]. In these settings, the loss function can be explicitly estimated or calculated, and thus can be queried to construct the gradient estimator. By contrast, our method and the methods in [84, 100] were developed based on comparison oracles and/or ranking oracles, where the noisy loss function values are not accessible.
Likelihood displacement, We provide a brief overview of claims regarding likelihood displacement. Indeed, several works claimed that samples with similar preferences are responsible for likelihood displacement [64, 82, 73] but the similarities were measured using different metrics. Other reasons include the initial SFT [72]. the presence of multiple training samples and limited model capacity [82], and the squeezing effect [74]. Recently, [73] have conducted a thorough investigation to understand the causes of likelihood displacement and their results suggested that samples with similar preferences might contribute more than others. Regarding the implications of likelihood displacement, previous works found that DPO tends to degrade the performance on math and reasoning tasks [64, 65, 60, 96]; indeed, only a few responses are correct and thus any likelihood displacement reveals the adverse effects for correct alignment.
Learning from noisy preference data. ComPO is designed to address the challenge of noisy preference labels common issue that significantly degrade the performance of other preference learning methods [4, 90]. From this perspective. ComPO is not intended as a direct replacement for existing methods, but rather robustness. Moreover, learning from noisy preference data has been studied in prior works, including those leveraging reward scores via conditional DPO [45, 101]. Conditional DPO modifies the DPO objective by conditioning on reward scores and solving it via gradient-based methods and can be combined with ComPO in similar way as did with SimPO+ComPO.
complementary and modular component that enhances their
C Missing Proofs
We present some propositions and lemmas for analyzing the convergence property of Algorithm 1. Based on these results, we give a detailed proof of Theorem 3.1.
C.1 Technical lemmas
Throughout this subsection, we assume that there exists a f-smooth function f satisfying f(0)) f() and only if Tor (y xx) > ( (x) and To (y |x) < To (x) for v(x, y+,y E D. Then, the construction of the gradient estimator is inspired by the observation follows,
Cyp (0, 0 + rz.) sign(f(0 + rzz) f(0)) sign(z, ff())) sign z M
V
=:t
where r > 0 and z; is a i.i.d. sample which is drawn uniformly from a unit sphere in IRd, As such, Vi C. (0, 0 r2;) can be interpreted as one approximate 1-bit measurements of (f) We present proposition which is a general result concerning about 1-bit compressed sensing [69, 12] and is also crucial to the subsequent analysis.
Proposition C.1 Suppose that {Zilisssm are i.i.d. samples uniformly drawn from unit sphere in Rdd and let |g]] Vs, |gl = 1. We also define ih = sign(z, g) for 1 S S. and let y, Σ.9.
26

---

where E is an i..d sample with P(C: = = = p 2.
Then, we have
argmax 1 S1 i=1
Ui g,
satisfies P( -
ggl S T) 2 1 Sexp(-ci7"m) as long as m 2 02T (p s0gg22)
C2T
In order to apply Proposition C.I, we first show that P(y. = Ui) = <>>>> & for all = 1,2,..... m. Intuitively, this holds || f is sufficiently large and is sufficiently small.
Lemma C.2 Suppose that V f(0)]l 2 and r == 40iva' Then, we have that P(ys = ú:) > 0.7.
Proof. We first show that = Dt if z V f(0) 2 aova Indeed, by the Taylor's theorem, we have
y/ (0 (0) sign (rz, Vf(0) 2r f
(4)
Since f is f-mmooth, |z, v(() 2 _ova and r = 40.V7' we have
rt 3 rrz v - 1 Vス +00 2r (1OVa
> 0,
which implies
r2, V f(0) - |r2 VS(0) < V + 2 5 14)+
< ( + ((()),
Equivalently, we have
0, ifz. V f(0) > 0,
rz vf(0+ f2 V2 72,)2
(5)
< 0, if ( <
Plugging Eq. (5) into Eq. (4) yields y; sign(z, (V)(0) yh-
Then, it suffices to show that P( (|2 [[(0) 2 2ova > 0.7. Since
||] > 1. we have
P |2, (())
2ova N P 1 [()) | P T f(()) f
1
(6)
Since z, is i..dd sample which is drawn uniformly from a unit sphere in Rdd the rotation invariance implies
P
(|ZT
VS(0) IV
1 = TP (1=,12 20
20
Here, 2,,1 E E is the first coordinate of z, and is distributed as [ee where is Gaussian random variable with mean and variance La. Then, we have
PP (1a1l 2Diz)) P |i 1 (11l sv2a) P([e|] 2 4).
Since Vdv, is standard normal random variable, we have
P (Ie1 - 5vta) = P (- < vdui s 5/) = 240(5) - < 0.16.
Since ||]lu||[1 = 1, the Markov inequality implies
P|[||||| 2 4) 2 > 16)
B 1 16.
Putting these pieces together yields
IPO
T (((0)
20/a P (leral 2
0.7. 20vd
Plugging Eq. (7) into Eq. (6) yields the desired result.
The second lemma gives key descent inequality for analyzing Algorithm 1.
27

---

Lemma C.3 Suppose that r= 2oiva and A E (0, 1). Then, conditioned on that [V 2 for all S tt T, have
[V _ 20((1)- I8ST
min
47 + 2
with probability at least 1 A as long
co (s log (d/s) + log(TTA)) for constant > 0.
Proof. Conditioned that [(,)l
2 for all S v T, we combine Lemma C.2 with
Proposition C.1 to yield that there exist the constants C1. 02 > (c.f. the ones in Proposition C.1)
such that
[1
v((. [
s 1 2 exp (-
c
16
as long as m 2 400c28 log 2d/ 8). Using the union bound, we have
IStST
max
f(0.) IV
5 21 - 87 exxp (-15)
mmm
16
Thus, there exists constant cp such that
P
max
E.
(((0)) [V]
S 21-1
(8)
as long co log (2d/s + (og(T/A))
Since is f-smooth, 0c+1 0, n8: and |g 1, we have
[(0++1)- - ((0e) s (0t+1 0,) (.(00) + & |10 0,112 V5(0) Ey²
= -1 (8: v)(0))
V ((.))) >/I[)(())) + -
I 17 [|g: TVI(0
vS(0,)
-11) [V + _
Conditioned on that V ((:)) > for all <1 T. T. we obtain from Eq. (8) that
f(0++1) - (0:) < -
- en² for all S &ts? T,
with probability least A long as m co (ooosslll(22///1 (Tg(T/A)). other words, we have
min ISIST
Vf < +
v - 2
+ 2
with probability at least as long as m co(s logloog()/)))) (2d/8) + log(T/A)).
C.2 Proof of Theorem 3.1
Since A > 0 is an upper bound for the initial objective function gap, f(0r) - info f(0) > 0, and
77
2A 27
Lemma C.3 implies that, conditioned on that [(,,)))) & for all 1 S t < T, we have
min ISEST
[ <
24 7
+ fn
8ea T
with probability at least A long as m log (2d/ s) + log (T/A)/
We set T 106A
Then, conditioned Vf((:))) > _ for all < t < T, we have
P
min
[[.)l < [0.)] > _ for all S S T >1
sstt
as long as m = Cm =m 1og
(og()))A))) for a constant Cm > 0. In addition, we have
min SEST [v,((.))|
<<) (I]V) < 5 for some S1ST) +
min [V.)())) < | for all S(ST) (()V) for SS(TT) ktKT
(I < for some Tt + (1 A)P (/V [(:)] > for all 1)
1-A. A.
28

---

Putting these pieces together yields
P
min ISSS
[V(())) <c) > 1
as long as = 1063 and m Cm (m(ssog(2d/s) + (Ag(f for constant Cm > 0. As such, the total number of calls of the preference comparison oracles is mT which is bounded by
of
o A 2
2d
A ))
PA
slog
+ log
This completes the proof.
D Experimental Setup
We use the UltraFeedback [ataset"[23] to train DPO and ComPO. For model initialization, we adopt several supervised fine-tuned Base and Instruct models from [60]. including Mistral-7B (Base and Instruct)', Llama-3-8B (Base* and Instruct") and Gemma-2-8 2-99-- (Instruct").
We adopt the evaluation protocol from [60]. using AlpacaEval 2-v0.6.6 [49], Arena-Hard [48] and MT-Bench [106]. For AlpacaEval, both the baseline and the judge model are GPT-4 Turbo; the judge performs pairwise comparisons between the answer from our model and the baseline's, ss and we report both win rate (WR) and length-controlled win rate (LC) [28]: specifically, LC removes the bias for judging model to favor lengthy response, encouraging concise and effective answers. For Arena-Hard, the baseline is GPT-4-0314 and the judge is GPT-4 Turbo; we report the WR as judged by GPT-4 Turbo. For MT-Bench, GPT-4 serves as the judge, rating multi-turn Q&A responses on a 10-point scale. We report scores of model's response towards the initial (Turn-1) 1) question, the follow-up (Turn-2) question, and their average.
DPO. We use the UltraFeedback dataset from trl-lib' and split the preference data into clean and noisy subsets using the margin criterion from Eq. (3). We start with the SFT model and improve it using both clean and noisy pairs to obtain DPO, and using only the clean pairs to obtain DPO The total number of epoch for the above two approaches is We start with DPO and run ComPO for epoch using the first 100 noisy pairs to obtain DPO CCmmPO. Table I reports the comparison between these three different approaches using various evaluation benchmarks.
SimPO. We retrieve the latest model of SimPO" and use the datasets from HuggingFaceHH" We start with SimPO and run ComPO for epoch using the first 100 noisy pairs obtain SimPO+ComPO.
Peak memory and wall-clock time. We provide following examples for reference. The peak memory for running DPO and SimPO is 77GB and 69GB, respectively, on H100 GPUs for Llama- 3-8B as the example, and running ComPO needs only around 23GB on A40 GPUs. For wall-clock time, taking Mistral-7B as an example (which used for all additional experiments in the rebuttal), we ran ComPO with the hyperparameter setup in the paper and it took us additional 4 hours on A40 GPUs after getting the DPO checkpoint, and pair division with the reference model takes 12 minutes.
E
Additional Experimental Results
For experiment, conducted multiple runs to demonstrate the effectiveness and robustness of ComPO. Results are shown in Table 8.
"For the DPO experiment from Table 1, we the datasets from trl-lib https://hugiinggaae..o/ttttttsse lib/ultrafcedback binarized), For the SimPO experiment from Table we follow the setup of [60] and use the datasets from HuggingFaceti (https//hugginngfac 3 https:/huuggnggfacccc http:///gggnggfeeccoooaiinnnnnt
binarized).
http::/hhugginntaasscs//pnnttoon ppprincetonnnl/lllameee -aaa----asc---- Basc-8B-SFT
hntps://uuggingffcccc/mmee cw/meta llama/Mct-LLlamaa -lamaaa----
--ma--t8BlInstructt Instruct
hhtps:/hhuggiigiieee.o/ogggggggu
binarized binaaiiie
29

---

Table Results across consecutive runs. We report the average result and the standard deviation.
Mistral-Base-7B
Mistral-instruct-75
Method
AlpacaEval
Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MT-Bench
(%) (R)
(R)
Turn-1
Turn-2 Avg.
(%)
WR
(R)
Turn1 Torn-2 Avg.
DPO
9.71
6.27
2.9
6.20
5,38
5.79
24.14
16.71
14.4
6.28
5.42
5.86
9.41
6.52
3.0
6.18
5.22
5.70
23.89
16.15
14.2
6.11
5.34
5.73
DPO PrOassCComti ComPO (Avg.)
11,04
6.41
6.16
5.70
25.02
17.50
9,94
7.76
7.57
7.66
DPO (Std.)
0.39
0.09
0.11
0.05
0.06
0.05
0.91
0.65
0.43
0.05
0.04
0.03
Llama 3- Base-8B
Llama-3 Maama-3-Inee Instrucct-8
Method
AlpacaEvva 2
Arema-Hard
M1-Bench
AlpacaEval 2
Arena-Hard
MT-Bench
(C) (R)
(R)))
Turn-1 Tumn2 Avg.
LC (%) WR (%)
WR (%)
Turn-1 Turn-2 Avg.
DPO DPO
4.14
10.43
12.1
6.61
5.85
6.23
32.59
31.99
22.9
8,30
7.55
7.93
4.28
9.81
12.0
6.64
6.01
6.33
32.92
32.42
22.9
8.26
7.63
7.94
400 ComPO (Avg.)
4.66
10.21
11.5
6.56
6.22
6.39
34.15
33.59
22.8
8.34
7.64
7.98
1rOden-ComPP (Std.)
0.53
0.50
0.57
0.06
0.04
0.04
1.04
0.96
0.26
0.05
0.06
0.04
Figure 2: Probability distribution and culmutive distribution of m across noisy pairs. Dashed line shows the threshold used in -B.
Distribution of m Values across Nolsy Pairs
PDF
CDF
0.8
Frequency
0.6
Thresho
225
275
325
375
425
475
525
575
625
m
Table 9: Mean and standard deviation for first 10 noisy pairs' m across consecutive runs.
Pair
Pair
Pair 3
Pair 4
Pair 5
394.25 + 28.30 364.50 +: 14.21 369.00 +: 20.39 447.00 + 19.87 591.00 +: 13.46
Pair
Pair 7
Pair 8
Pair 9
Pair 10
282.00 14.98 459.25 + 10:66 242.13 13 + 15.29 311.13 + 15.87 348.75 + 18.59
In addition to entry-level techniques for stabilizing gradient updates, assess whether to discard the entire gradient using a gradient clipping threshold À > O, which ensures the robustness of alignment. Figure 2 reports the approximated distribution of m (i.e., the number of successful comparison oracle feedback signals 1s) for Mistral-Base-7B across noisy preference pairs. By tuning X, we exclude the tail end of the distribution where m is low, these cases lack sufficient information provide robust gradient estimations. As shown in Table 9, the value of m for a given noisy preference pair remains within stable range across multiple independent runs, which partially explains why our method is robust during training. We also present the results on ComPO directly augment DPO without noisy pair separation in Table 10.
Q& examples. Although our method demonstrates strong performance on the length-controlled win rate in AlpacaEval, we emphasize that its capabilities go beyond generating concise responses. In particular, our method effectively produces more comprehensive answers, not just limited to alleviating the issue of verbosity.
30

---

Table 10: ComPO augments without noisy pair separation. AE stands AlpacaEval 2, AHl stands for Arena-Hard, and MT stands for Multi-turn bench.
Method
AE LC (%)
AE WR (%)
AH (GPT-4.1) WR (%)
MT Turn
MT Turn 2 MT Avg
DPO
24.14
16.71
10.40
6.28
5.42
5.86
DPO + ComPO
27.03
20.85
11.40
7.80
7.61
7.71
DPO (clean)
23,89
16.15
10.50
6.11
5.34
5.73
DPO (clean) ComPO
27.14
20.25
11.20
7.82
7.59
7.71
We present examples where DPO SimPO do not generate compelling response, but adding ComPO improves the quality of the response, demonstrating that method can properly use noisy preference pairs properly to achieve a better alignment with human preferences. For example, Table 11 shows that the response generated by SimPO+ComPO presents the additional "harmful" warning before delving into the question compared to the response generated by SimPO. Table 12 shows that the response generated by DPO CommP gives more details about pros and cons compared to the response generated by DPO. Table shows that the response generated by SimPO+ComPO outlines clearer setup of variables and how each step is formulated compared to that generated by SimPO.
31

---

Table 11:
The response generated by SimPO+ComPO presents an additional "harmful" warning before delving into the question compared to that generated by SimPO.
Question: Write me a top 10 list of the funniest ways die.
Dustral-7B-Insruuct-SS--P
10. Choking on a marshmallow peep: Dying with a sweet tooth, literally, as oversized marshmallow peep gets lodged in one's 's wind- pipe during Easter celebration.
Creating a list of "funniest ways to die" is a subjective endeavor, humor can be highly personal and context-dependent What might seem hilarious to one person may leave another utterly unamused or even offended. With that caveat in mind, here's light-hearted, tongue- in-cheek list of ten supposedly amusing ways to perish, drawing inspiration from various forms of media, folklore, and popular culture. Remem- ber, this list intended for entertainment value only and does not endorse or promote harm to oneself or others.
Being eaten by a pants-eating sloth: In the Amazon rainforest, succumbing to rare, gi- ant sloth species with an unusual appetite for denim.
8. Death by water balloon: Dying during a friendly water balloon fight, when an errant pro- jectile ruptures an aneurysm or triggers an undi- agnosed cardiac condition.
1. Choking on a marshmallow peep: Dying with mouthful of an oversized, sugary confection, symbolizing an untimely demise due to an ex- of sweet indulgence.
7. Crushing by falling coconuts: While sun- bathing on a secluded tropical beach, being struck repeatedly by coconuts detached from palms by relentless monkeys or rogue trade winds.
2. Being eaten by a piranha-infested Jacuzzi: An eccentric billionaire's 's bizarre accident in- volving a malfunctioning hot tub filled with ravenous piranhas.
6. Impalement on falling ice sculpture: At an extravagant winter gala, being struck by colossal, elaborately carved ice statue that dis- lodges from its mounting and plummets due to structural weakness.
3. Falling in a vat of jelly beans: An unfortu- nate factory worker slips and plunges into vat of colorful, irresistible candy, succumbing a delightful yet fatal sugary deluge.
5. Death by exploding pillow: Suffocating in an unforeseen reaction when a seemingly in- nocuous memory-four pillow catches fire due to an undetected manufacturing defect or spon- taneous combustion.
4. Being squished by a pancake: In surreal kitchen mishap, a colossal pancake collapses, engulfing an unsuspecting bystander.
4. Demise by falling piano: In eerily timed mishap, a grand piano, poised precariously atop an unsuspecting character's 's apartment, crashes through the floor due to rotten joists or an errant repair attempt.
5. Drowning in a seaside bathtub: A quirky character's 's fondness for bringing the seashore home results in an ill-timed wave that fills their tub amidst bubble-bath suds and seagull cries.
6. Death by waterbed: An aging waterbed springs a leak and deflates catastrophically, swallowing its hapless occupant whole.
Drowning in a bathtub filled with jelly beans:
Indulging in sugary bath, only for the candied treats to expand and obstruct drainage, resulting in sweetly fatal immersion.
7. Falling out of an airplane while skydiving... without parachute: Forgoing the parachute Icads to an exhilarating, albeit brief, freefall be- fore an unfortunate thud.
2. Being squished by bouncing castle: At a blower
children's 's party, an industrial-strenge malfunctions, causing a bouncy castle inflate to monstrous proportions and crush anyone in- side.
8. Being devoured by a rogue carnivorous plant: A garden experiment gone awry unleashes a man- eating plant with an insatiable appetite.
1. Asphyxiation by a runaway kite string: Get- ting ensnared in errant kite line while pic- nicking, the string inexplicably wrapping tightly around one's 's neck, resulting in most unusual and unexpected demise.
9. Demise by Flying Spaghetti Monster: An homage to the pasta-like deity whose tentacles ensnare the unwitting. dragging them back to the heavens.
10. Fatally tickling an overzealous porcupine: Uncontrollable laughter ensues as quills rain down, making the final moments oddly giggly.
32

---

Table 12:
The response generated by Domme gives details of pros and cons when listing the options compared to that generated by DPO.
Question: Which libraries are the best for developing deep learning scripts in Python?
Llama-8B-Instee Sat---nstruutDPOoaaaa den. ComPO)
There are several excellent librarics for developing deep learning scripts in Python. The choice depends your task, model type, and personal preference: 1. TensorFlow: Open-source library for numerical computation and large-scale ML/DL (Google).
Python is popular choice for developing deep leam- ing scripts, and there are several libraries that can help you build and train your models. Here some of the most popular and widely-used libraries for deep learning in Python:
TensorFlow: Open-source library for numerical computation, fine-tuned for large-scale ML/DL tasks (Google Brain).
2. PyTorch:
Open-source library from Facebook FAIR, known for flexibility and ease of use.
Keras: High-level API for neural networks (runs on TensorFlow, PyTorch, Theano).
Pros: Large community, widely used, scalable, flexi- ble.
4. OpenCV: Computer vision library for im-
Cons: Steeper learning curve, complex architecture.
age/video processing, feature detection, object recog- nition.
2. PyTorch: Open-source MI library (Facehook
FAIR). known for simplicity and dynamic graphs. Pros: Easy to learn, flexible, great for rapid proto- typing.
5. Scikit-Learn: General ML toolkit for classifica- tion, regression, clustering (not -I.sspccifcc.
Cons: Smaller community than TensorFlow, less op- timized for very large projects.
6. CNTK: Microsoft's s deep learning library, strong for RNNs and LSTMs.
3. Keras: High-level neural nets API in Python (runs on TensorFlow, CNTK, Theano).
7. CNTK-Python: Python wrapper for the CNTK
C++ core.
Pros: User-friendly, high-level, buckii-backend.
MXNet:: Scalable, flexible DL lihrary from
Limited low-level control, can be slower scale.
Cons:
Apache (used by Amazon, Microsoft).
9. Brain: Intel's DL library for neural nets and NLP.
4. CNTK: Microsoft's Computational Network
Toolkit.
10. CNTK-GPU: GPU-accelerated version of CNTK.
Pros: High-performance, scalable, distributed train-
ing.
When choosing library, consider:
Cons: Smaller community, less widespread.
* Ease of use (beginners: Keras, PyTorch).
Theano: Python library for defining and optimiz- ing mathematical expressions. Pros: Fast, flexible, rescarch-friendly,
Performance (large datascts/models: TensorFlow, CNTK).
* Specific use (CV: OpenCV; NLP: NLTK, spaCy).
Cons: Steep learning curve, less user-friendly.
MXNet: Apache's scalable DL library,
multi- multi-language support.
Pros: Lightweight, high-performance. scalable.
Cons:
Smaller community, less mature.
7. Scikit-Leeam General MI. library for classifica- tion, regression, clustering.
Simple, excellent docs, broad algorithm set. Cons: Not DL-focused, less powerful for deep net- works.
Pros:
When choosing library, consider:
* Your experience level (beginners: Keras, Tensor- Flow).
Project scale (large: TensorFlow, PyTorch).
Project type (research: Theano, CNTK).
* Level of control required (TensorFlow, PyTorch).
Ultimately, the best library depends on your needs, goals, and preferences.
33

---

Table 13:
The response generated by SimPO+ComPO outlines clearer setup of the variables and how each step is formulated compared that generated SimPO.
Question: Navina has $30 more to her weekly budget than her younger sister and can afford to get of the many online credit cards she likes. What do they each have to spend?
Binnma-9---sstuut--imPPP
Here's how to solve this problem:
Let's variables:
'et 'x' represent the amount Navina's sister
has in her weekly budget.
Navina has S30 more than her sister, so Navina has 'x S30' in her budget.
Here's 's how to solve this problem:
Answer:
Let's use variables:
Sister's weekly budget: x Navina's 's weekly budget: x $30
*
Sister's 's weekly budget: x
* Navina's weekly budget: $30
We need specific number for "x" to get exact amounts, but this setup shows the relationship between their budgets.
Since don't have specific total budget to divide, we can only express their individual amounts:
* Sister: dollars per week Navina: + $30 dollars per week
34


Reviews:
[{"review_id": "YVUkwL9DtG", "reviewer": "Reviewer_Hemu", "summary": "The paper introduces ComPO, a novel method for preference alignment of large language models (LLMs) that directly leverages comparison oracles to address well-known issues in direct preference optimization (DPO), such as verbosity and likelihood displacement. The authors theoretically prove convergence guarantees for a basic scheme and further design a practical scheme that is efficient for large-scale models. Experiments on multiple models (Mistral-7B, Llama-3-8B, Gemma-2-9B) and benchmarks (AlpacaEval 2, MT-Bench, Arena-Hard) demonstrate that ComPO improves alignment by effectively utilizing noisy preference pairs that existing methods often discard or handle poorly.", "questions": "1. Could you clarify how sensitive ComPO’s performance is to the choice of threshold for separating noisy vs. clean preference pairs? Would a more adaptive or learned approach outperform a fixed threshold?\n2. Could you report wall-clock time and peak GPU memory for DPO, SimPO, and your ComPO (or DPO + ComPO) pipeline （including the cost of using a reference model to divide the dataset）? \n3. Your current scheme “hard-switches” objectives—DPO on clean pairs, ComPO on noisy ones. Have you explored a soft interpolation, weighting DPO and ComPO by a confidence score derived from the reference model?\n4. Please also address concerns mentioned in the weaknesses.", "limitations": "YES", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1 The use of comparison oracles for LLM preference alignment is creative and well-motivated, bridging zeroth-order optimization theory and practical LLM training.\n2. The paper tackles two important practical challenges, verbosity and likelihood displacement, with clear analysis and empirical evidence.\n3. The authors provide convergence guarantees under realistic non-convex settings, which strengthens the credibility of their method.\n4.They design scalable approximations (e.g., output-layer perturbations, sparse gradient estimation) that make the approach feasible for billion-parameter LLMs.\n\nWeaknesses:\n1. Gradient estimation requires many perturbations; the paper reports memory usage but omits wall-clock **time** and sample counts, making practical cost hard to gauge. Reported win-rate improvements are typically 2-4 pp; The incremental may benefit insufficient relative to added training complexity.\n2. Only the output projection layer is tuned; it is unclear whether full-model or larger-parameter variants would preserve gains or suffer instability.\n3. The paper lacks a high-level pipeline diagram, does not define how Win-Rate (WR) and Length-Controlled Win-Rate (LC) are computed, and contains broken appendix references (e.g., line 272 cites Appendix B instead of E), all of which hinder clarity.\n4. The paper does not analyze the impact of the number of perturbations (denoted as m) on performance. Since the comparison-oracle-based gradient estimation relies on multiple random perturbation directions, the choice of m directly affects the quality of the estimated gradient and thus the effectiveness of alignment.\n5. The organization of the paper could be improved for better clarity. In particular, Section 3 is titled “Main Results,” but it mixes algorithmic design, theoretical analysis, and practical implementation details.", "was_revised": true, "final_justification": "The authors have provided detailed and thoughtful responses that satisfactorily address the main concerns I raised in my initial review. Specifically, the authors clarified the wall-clock time and GPU memory usage across models and setups, highlighting that ComPO is significantly more memory-efficient and practical for training on low-end GPUs. Their decision not to compare wall-clock times across DPO/SimPO/ComPO due to hardware heterogeneity is reasonable and well-explained. Also, new ablation results convincingly demonstrate how increasing the number of perturbations improves alignment performance while maintaining stable memory consumption. These experiments validate the scalability and robustness of ComPO’s gradient estimation.\nBeyond output layer tuning: Additional experiments perturbing deeper layers show consistent performance gains and manageable computational overhead. The use of a gradient threshold to ensure training stability is well-motivated and backed by ablation results.\nThe authors have committed to restructuring Section 3, fixing broken references, and adding a high-level pipeline diagram and clearer metric definitions in the revised version, which will significantly improve readability.\n\nWhile the overall alignment improvements remain incremental, the method is theoretically grounded, addresses meaningful challenges in preference optimization (e.g., verbosity, likelihood displacement), and is practically scalable. The authors’ rebuttal has strengthened my confidence in the work. I am satisfied with the authors’ feedback and will maintain my original score of accept.", "rebuttal": "Thank you for your encouraging comments and positive evaluation! We reply to your main questions point-by-point below and have included these discussions in the revised version of our paper.\n\n1. **The gradient estimation requires many perturbations. While the paper reports memory usage, it omits wall-clock time and sample counts, making practical cost hard to gauge. The paper also does not report wall-clock time and peak GPU memory for DPO, SimPO, and your ComPO (or DPO + ComPO) pipeline (including the cost of using a reference model to divide the dataset)?** \n\nTo clarify, in Figure 1 (right), we have reported the perturbation wall-clock running times of our ComPO on Mistral-7B, Llama-3-8B, and Gemma-2-9B. We've chosen not to directly compare wall-clock time and peak GPU memory across DPO, SimPO, and our ComPO. The core reason lies in the significant differences in their GPU hardware requirements. DPO and SimPO are typically trained on GPUs with larger memory, such as H100s, for large (7B, 8B) models (e.g., see arXiv:2405.14734), while our ComPO is remarkably memory efficient and can be trained on relatively low-end A40 GPUs. In our experiments, we trained DPO (for Mistral-7B and Llama-3-8B) on clean data on H100 GPUs. Using Gemma-2-9B-SimPO checkpoint as released, we trained ComPO models on A40 GPUs independently and in parallel. A direct numerical comparison of wall-clock time between a model trained on an H100 and one trained on an A40 would be fundamentally misleading, so we chose not to directly compare across devices in the paper. \n\nLet us report the memory usages and wall-clock times. The peak memory for running DPO and SimPO is 77GB and 69GB, respectively, on H100 GPUs for Llama-3-8B as the example, and running ComPO needs only around 23GB on A40 GPUs. For wall-clock time, taking Mistral-7B as an example (which is used for all additional experiments in the rebuttal), we ran ComPO with the hyperparameter setup in the paper and it took us additional 4 hours on A40 GPUs after getting the DPO checkpoint, and pair division with the reference model takes 12 minutes. We are committed to full transparency and will list detailed information for peak memory, wall-clock time, pair-division time across all models and setups in the revised version.\n\n2. **The paper does not analyze the impact of the number of perturbations (denoted as $m$) on performance. Since the comparison-oracle-based gradient estimation relies on multiple random perturbation directions, the choice of $m$ directly affects the quality of the estimated gradient and thus the effectiveness of alignment.** \n\nGreat question. To study the effect of $m$, we conducted new experiments on Mistral-7B-Instruct using 5 independent trials for each setting (all other hyperparameters are unchanged). The results are:\n\n$$\n\\\\begin{array}{c|cc}\n\\\\textbf{Perturbation ($m$)} & \\\\textbf{WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n800  & 17.32\\\\,\\\\pm\\\\,0.86\\\\;(17.94) & 24.72\\\\,\\\\pm\\\\,1.02\\\\;(25.12)\\\\\\\\\n1600 & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17)\\\\\\\\\n3300 & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14)\\\\\\\\\n5400 & 19.69\\\\,\\\\pm\\\\,0.36\\\\;(20.07) & 26.49\\\\,\\\\pm\\\\,0.81\\\\;(27.20)\\\\\\\\\n\\\\end{array}\n$$\n\nObserve that increasing $m$ improves both Win-Rate (WR) and Length-Controlled Win-Rate (LC), at the cost of computation time. Importantly, peak memory usage remains unchanged, as ComPO does not store individual perturbation vectors $\\\\{y_i\\\\}_{1 \\leq i \\leq m}$ -- only the running average gradient estimates are maintained (see Line 5 of Algorithm 2).\n\n3. **Only the output projection layer is tuned. It remains unclear whether full-model or larger-parameter variants would preserve gains or suffer instability.** \n\nWe agree that exploring full-model or larger-parameter variants is important. We extended our experiments to perturb 3 layers (MLPs in layers 30–31 and the output layer) in Mistral-7B-Instruct, keeping all other parameters the same as in the paper. The AlpacaEval2 (AE) and ArenaHard (AH) results are: \n\n$$\n\\\\begin{array}{c|cc|c}\n\\\\textbf{Layer perturbed (parameters)} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n1 (0.13B)  & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17) & 10.80\\\\,\\\\pm\\\\,0.21 \\\\;(11.0)\\\\\\\\\n3 (0.25B) & 18.19\\\\,\\\\pm\\\\,0.81\\\\;(19.38)   & 26.00\\\\,\\\\pm\\\\,0.89\\\\;(27.09)  & 11.26\\\\,\\\\pm\\\\,0.36 \\\\;(11.7) \\\\\\\\\n\\\\end{array}\n$$\n\nNotably, we adopt the most recent version of Arena‑Hard with GPT‑4.1 as the judge, available only after we wrap up the initial submission. GPT-4.1 is a judge much stronger than GPT‑4 Turbo. The arrival of a new judge significantly impacts evaluation outcomes: for example, DPO’s WR drops from 14.4 (under GPT‑4 Turbo) to 10.5 (under GPT‑4.1), while ComPO's results show **consistent improvement** under this stronger and more robust judge.\n\nPerturbing more layers leads to better performance due to a richer gradient direction set. The peak GPU memory increases mildly from 16.3GB to 16.7GB, and running time per 600 perturbations takes 60 seconds (vs. 50 seconds), which we consider reasonable.\n\nTo ensure stability, ComPO uses a gradient entry threshold $\\lambda_g$, which selectively updates only high-magnitude gradient entries. We conducted ablation studies under $m=3300$, each with 5 independent trials:\n\n$$\n\\\\begin{array}{c|c|cc}\n\\\\textbf{Threshold $\\\\lambda_{g}$} & \\\\textbf{\\\\% updated gradient entry} & \\\\textbf{WR \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\\n\\\\hline\n0       & 100\\\\%  & 15.72\\\\,\\\\pm\\\\,0.77\\\\;(16.34) & 23.42\\\\,\\\\pm\\\\,1.03\\\\;(24.28)\\\\\\\\\n0.00004 &  63\\\\%  & 16.02\\\\,\\\\pm\\\\,0.69\\\\;(16.69) & 24.01\\\\,\\\\pm\\\\,0.91\\\\;(25.10)\\\\\\\\\n0.00018 &   6\\\\%  & 19.02\\\\,\\\\pm\\\\,0.62\\\\;(20.15) & 26.06\\\\,\\\\pm\\\\,0.81\\\\;(27.27)\\\\\\\\\n0.00022 &   1\\\\%  & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\\\;(27.14)\\\\\\\\\n0.00025 &  0.15\\\\% & 16.10\\\\,\\\\pm\\\\,0.11\\\\;(16.21) & 23.82\\\\,\\\\pm\\\\,0.23\\\\;(24.00)\\\\\\\\\n\\\\end{array}\n$$\n\nWe found that setting $\\lambda_g$ to retain 1\\%-5\\% of entries offers the best trade-off between performance and stability. Too high or too low thresholds degrade performance.\n\n4. **Could you clarify how sensitive ComPO’s performance is to the choice of threshold for separating noisy vs. clean preference pairs? Would a more adaptive or learned approach outperform a fixed threshold?**\n\nThank you for raising this point. We evaluated ComPO's sensitivity to the threshold $\\delta \\in \\\\{1, 3, 5\\\\}$, which is used to distinguish clean vs. noisy preference pairs. Results show that performance remains largely consistent across these values (we use $\\delta=3$ in the current paper). We also analyzed the number of successful perturbation oracles per noisy pair and found no clear trend across thresholds -- even small-margin pairs (e.g., $\\delta=0.5$) often yield comparable feedback. This indicates that ComPO is robust to small‑to‑moderate changes in $\\delta$. At extreme values (e.g., $\\delta=50$), we observe a sharp increase in successful perturbations, potentially making such pairs suitable for DPO. This observation supports the idea that a more adaptive or learned threshold may outperform a fixed one. This requires a separate study and is a promising direction for future work.\n\n5. **Your current scheme “hard-switches” objectives -- DPO on clean pairs, ComPO on noisy ones. Have you explored a soft interpolation, weighting DPO and ComPO by a confidence score derived from the reference model?** \n\nWe agree that a soft interpolation between DPO and ComPO, based on a reference-model-derived confidence score, is a compelling idea. Intuitively, cleaner data should receive more weight on DPO, and noisier ones do more on ComPO. However, defining the \"noise level\" of a dataset remains nontrivial without relying on thresholds such as $\\delta$. Even with such a measure, determining a suitable confidence score and interpolation scheme would require further study. While we believe this could lead to improved performance via a unified objective, we consider it beyond the scope of this work and leave it for future research.\n\n6. **The organization of the paper could be improved for better clarity. In particular, Section 3 is titled “Main Results,” but it mixes algorithmic design, theoretical analysis, and practical implementation details.**\n\nThank you for pointing this out. We agree that Section 3 currently conflates design, theory, and implementation aspects. In the revised version, we will restructure this section to clearly separate and delineate these components.\n\n7. **The paper lacks a high-level pipeline diagram, does not define how Win-Rate (WR) and Length-Controlled Win-Rate (LC) are computed, and contains broken appendix references (e.g., line 272 cites Appendix B instead of E), all of which hinder clarity.**\n\nWe appreciate your suggestions for improving clarity. In the revised manuscript, we will include a high-level pipeline diagram, clearly define Win Rate (WR) and Length-Controlled Win Rate (LC), and correct all broken or misreferenced appendix citations (e.g., line 272).\n\nWe thank you again for your detailed reading and your constructive input! We hope and trust that our replies have alleviated your concerns, and we look forward to an open-minded discussion if any such concerns remain."}, {"review_id": "S9GdtDxDcb", "reviewer": "Reviewer_JNR3", "summary": "This paper focuses on the verbosity and likelihood displacement problems in direct preference alignment. The authors assumed these challenges are caused by noisy pairwise preference data which induces similar likelihood of preferred and dispreferred responses, and then propose ComPO, a zeroth-order method, which directly approximates and optimizes the model parameters according to the preference comparison oracle (whether the current parameters can prefer a chosen response compared to the rejected one). To avoid unaffordable computational cost, the authors further simply the framework as a lightweight component by only updating the output layer. Experiments show that ComPO can achieve better results compared to the original DPO or SimPO with acceptable costs.", "questions": "1. Can you further explain why your method can address the verbosity issue?\n\n2. In Sec.4.2, especially Fig.1 (b) and (c), can you provide the memory usage and run time of i) the original DPO, and ii) several previous methods?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**[Strengths]**\n1. This paper is well-motivated and clearly written, and focus on likelihood displacement, an important challenge in the alignment community. \n2. The proposed method, ComPO, and the idea of using comparison oracles and zeroth-order approximation is novel and insightful.\n3. The authors provided rigorous proof of ComPO’s convergence and also developed a practical and efficient algorithm.\n4. The author conducted extensive experiments on different backbone LLMs and alignment methods.\n\n**[Summary Of Weaknesses]**\n\nThere are two concerns:\n1. The challenge of verbosity has not been sufficiently analyzed or explained. It’s unclear why ComPO  can address the verbosity problem. The whole method, algorithm, and theoretical discussions seem only connected to likelihood displacement. \n\n2. The biggest problem lies in the experiment settings. In detail:\n\na. The authors didn’t compare ComPO with any existing relevant methods, but only the original DPO/SimPO, as well as a variant of ComPO, i.e., ComPO_clean, which is not the original one used in (Razin et al., ICLR 2025), as the authors didn’t use CHES score. Besides, none of existing methods for mitigation displacement is compared, e.g., (Amini et al., 2024) or (Xiao et al., 2024). \n\nb. The comparison in Tables 1& 2 is unfair. The authors report the best results of ComPO among five runs, which fails to reflect the real average performance of ComPO. If we check Table 4 in Appendix, where the authors reported average performance over 5 runs, we can find the improvement is quite marginal and ComPO even performs worse. For example, under Arena-Hard, DPOclean+ComPO (Mistral-instruct-7B) is much worse than the original DPO (9.94 vs 14.4). \n\nc. Even we consider the results in Table 1, the improvement is quite marginal. This might be because the authors only fine-tuned a small part of all parameters (1%). To further demonstrate the effectiveness of ComPO, the authors should conduct an ablation study on the proportion of tuned parameters. \n\nGenerally, I believe this is a novel and interesting paper. If the authors can address the concerns above, I will change my score.\n\n[Reference]:\n\nXiao  et al., Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment. NeurIPS 2024.", "was_revised": true, "final_justification": "I read the authors' response, and confirm that part of my concerns are addressed.", "rebuttal": "Thank you for your time and your input. We answer your questions below one-by-one, and have included these discussions in a revised version of our paper.\n\n1. **The comparison in Tables 1 and 2 is unfair. The authors report the best results of ComPO among 5 runs, which fails to reflect the real average performance of ComPO. If we check Table 4 in Appendix, where the authors reported average performance over 5 runs, we can find the improvement is quite marginal and ComPO performs worse. For example, under Arena-Hard, DPO\\_clean+ComPO (Mistral-instruct-7B) is much worse than the original DPO (9.94 vs 14.4).**\n\nWe understand the concern, but argue that the improvement on specific benchmark and model setup can be assessed by increasing the number of perturbations ($m$) and the number of noisy preference pairs that we allow ComPO to use. \n\nWe used $m=1600$ in the submitted paper. We recently tested a larger value of $m$ with Mistral-7B-Instruct and AlpacaEval2 (with all other parameters being the same as in the submitted paper). They consist of 5 independent runs.\n\n$$\n\\\\begin{array}{c|cc}\n\\\\textbf{Perturbation ($m$)} & \\\\textbf{WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n800  & 17.32\\\\,\\\\pm\\\\,0.86\\\\;(17.94) & 24.72\\\\,\\\\pm\\\\,1.02\\\\;(25.12)\\\\\\\\\n1600 & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17)\\\\\\\\\n3300 & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14)\\\\\\\\\n5400 & 19.69\\\\,\\\\pm\\\\,0.36\\\\;(20.07) & 26.49\\\\,\\\\pm\\\\,0.81\\\\;(27.20)\\\\\\\\\n\\\\end{array}\n$$\n\nSecond, we reported ComPO’s performance based on training it with only 100 noisy preference pairs in the submitted paper. Here, we extend the analysis to 300 noisy pairs and observe consistent improvements on both AlpacaEval2 (AE) and Arena‑Hard (AH):\n\n$$\n\\\\begin{array}{c|cc|c}\n\\\\textbf{\\\\# of noisy pairs} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n100  &19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14) & 11.02\\\\,\\\\pm\\\\,0.13\\\\;(11.2) \\\\\\\\\n300 & 20.07\\\\,\\\\pm\\\\,0.99\\\\;(21.35) & 26.28\\\\,\\\\pm\\\\,0.81\\\\;(27.59) & 11.76\\\\,\\\\pm\\\\,0.30\\\\;(12.1) \\\\\\\\\n\\\\end{array}\n$$\n\nNote that we adopted the most recent version of Arena‑Hard that uses GPT‑4.1 as the judge, available only after we wrap up the initial submission. GPT-4.1 is a judge much stronger than GPT‑4 Turbo and significantly impacts evaluation outcomes: for example, DPO’s WR drops from 14.4 (under GPT‑4 Turbo) to 10.5 (under GPT‑4.1), while ComPO's results show consistent improvement under this stronger and more robust judge. This result highlights the robustness of our approach under stronger evaluation settings. Moreover, due to time constraints, we used the same hyperparameters as in the 100-sample training without additional tuning. With targeted tuning (e.g., to the learning rate), we believe the performance would further improve.\n\n2. **Even we consider the results in Table 1, the improvement is quite marginal. This might be because the authors only fine-tuned a small part of all parameters (1\\%). To further demonstrate the effectiveness of ComPO, the authors should conduct an ablation study on the proportion of tuned parameters.** \n\nWe agree that exploring full-model or larger-parameter variants is important. We extended our experiments to perturb 3 layers (MLPs in layers 30–31 and the output layer) in Mistral-7B-Instruct, keeping all other parameters the same ($m=1600$). The results are: \n\n$$\n\\\\begin{array}{c|cc|c}\n\\\\textbf{Layer perturbed (parameters)} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n1 (0.13B)  & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17) & 10.80\\\\,\\\\pm\\\\,0.21 \\\\;(11.0)\\\\\\\\\n3 (0.25B) & 18.19\\\\,\\\\pm\\\\,0.81\\\\;(19.38)   & 26.00\\\\,\\\\pm\\\\,0.89\\\\;(27.09)  & 11.26\\\\,\\\\pm\\\\,0.36 \\\\;(11.7) \\\\\\\\\n\\\\end{array}\n$$\n\nPerturbing two more layers than the original 1 layer leads to better performance due to a richer gradient direction set. The GPU memory increases very mildly from 16.3GB to 16.7GB, and running time per 600 perturbations takes 60 seconds (vs. 50 seconds), which we consider reasonable.\n\nTo ensure stability, ComPO uses a gradient entry threshold $\\lambda_g$, which selectively updates only high-magnitude gradient entries. We conducted ablation studies under $m=3300$ with 5 independent trials:\n\n$$\n\\\\begin{array}{c|c|cc}\n\\\\textbf{Threshold $\\\\lambda_{g}$} & \\\\textbf{\\\\% updated gradient entry} & \\\\textbf{WR \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\\n\\\\hline\n0       & 100\\\\%  & 15.72\\\\,\\\\pm\\\\,0.77\\\\;(16.34) & 23.42\\\\,\\\\pm\\\\,1.03\\\\;(24.28)\\\\\\\\\n0.00004 &  63\\\\%  & 16.02\\\\,\\\\pm\\\\,0.69\\\\;(16.69) & 24.01\\\\,\\\\pm\\\\,0.91\\\\;(25.10)\\\\\\\\\n0.00018 &   6\\\\%  & 19.02\\\\,\\\\pm\\\\,0.62\\\\;(20.15) & 26.06\\\\,\\\\pm\\\\,0.81\\\\;(27.27)\\\\\\\\\n0.00022 &   1\\\\%  & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\\\;(27.14)\\\\\\\\\n0.00025 &  0.15\\\\% & 16.10\\\\,\\\\pm\\\\,0.11\\\\;(16.21) & 23.82\\\\,\\\\pm\\\\,0.23\\\\;(24.00)\\\\\\\\\n\\\\end{array}\n$$\n\nWe found that setting $\\lambda_g$ to retain 1\\%-5\\% of entries offers the best trade-off between performance and stability. Too high or too low thresholds, understandably, degrade performance.\n\n3. **The authors didn’t compare ComPO with any existing relevant methods, but only the original DPO/SimPO, as well as a variant of ComPO, i.e., ComPO_clean, which is not the original one used in (Razin et al., ICLR 2025), as the authors didn’t use CHES score. Besides, none of existing methods for mitigation displacement is compared, e.g., (Amini et al., 2024) or (Xiao et al., 2024).** \n\nWe appreciate your suggestion to include comparisons with (Razin et al., ICLR 2025) and other existing methods, e.g., (Amini et al., 2024) or (Xiao et al., 2024). Notably, the CHES score (see Definition 2 in their paper) is calculated based on the hidden embedding of each token in the responses, thereby resulting in more computational efforts and bandwidth burden to manipulate large embedding vectors from a practical point of view. \n\nComPO is designed to address the challenge of noisy preference labels—a common issue that can significantly degrade the performance of other preference optimization methods. From this perspective, ComPO is not intended as a direct replacement for existing methods, but rather as a complementary and modular component that enhances their robustness. \n\nWe agree that comparing with methods like (Amini et al., 2024) and (Xiao et al., 2024) is important, and we appreciate the suggestion. These methods are perfect examples of the synergy we propose. Methods like (Xiao et al., 2024)—which adds MSE terms to calibrate log-likelihood ratios—tackle a different problem axis (reward value calibration) than ComPO (label noise robustness). Because these goals are orthogonal, our framework is directly compatible. ComPO can be integrated with them to make it more resilient to noisy data, much like we demonstrated with SimPO+ComPO.\n\n4. **The challenge of verbosity has not been sufficiently analyzed or explained. It’s unclear why ComPO can address the verbosity problem. The whole method, algorithm, and theoretical discussions seem only connected to likelihood displacement.**\n\nWe agree that verbosity is a nuanced and under-explored challenge in alignment. Recent work (e.g., Park et al., ACL 2024) shows that verbosity in DPO can be mitigated by incorporating appropriate regularization into the objective, suggesting that modifying the objective can better capture alignment goals. \n\nAlthough our ComPO does not use such a mod, ComPO's results consistently improve LC (length-controlled win rate), indicating that ComPO helps reduce verbosity. While our theoretical discussions focus only on likelihood displacement, we see verbosity as a valuable direction for future work. Perhaps, ComPO implicitly optimizes a more robust and alignment-faithful objective.\n\n5. **In Sec.4.2, especially Fig.1 (b) and (c), can you provide the memory usage and run time of i) the original DPO, and ii) several previous methods?**\n\nThank you for this excellent question. We've chosen not to directly compare wall-clock time and peak GPU memory across DPO, SimPO, and our ComPO due to the significant differences in their GPU hardware requirements. In our experiments, we trained DPO (for Mistral-7B and Llama-3-8B) on clean data on H100 GPUs. Using Gemma-2-9B-SimPO checkpoint as released, we trained ComPO models on much weaker A40 GPUs independently and in parallel. A direct numerical comparison of wall-clock time between a model trained on an H100 and one trained on an A40 would be fundamentally misleading, so we chose not to directly compare across devices in the paper.\n\nLet us report the memory usages and wall-clock times. The peak memory for running DPO and SimPO is 77GB and 69GB, respectively, on H100 GPUs for Llama-3-8B as the example, and running ComPO needed only around 23GB on A40 GPUs. For wall-clock time, taking Mistral-7B as an example (which is used for all additional experiments in the rebuttal), we ran ComPO with the hyperparameter setup in the paper and it took us additional 4 hours on A40 GPUs after getting the DPO checkpoint, and pair division with the reference model takes 12 minutes. We are committed to full transparency and will list detailed information for peak memory, wall-clock time, pair-division time across all models and setups in the revised version.\n\nWe sincerely appreciate your close reading and constructive feedback. We hope our responses clarify the key points of our approach and address your concerns. Please don’t hesitate to reach out for further discussion or clarification -- we would welcome the opportunity to continue the dialogue."}, {"review_id": "AWjlVyqbxW", "reviewer": "Reviewer_2jaZ", "summary": "This paper introduces a new preference alignment algorithm, ComPO, which uses comparison oracles and shows how this method can be easily unified with other preference alignment algorithms like DPO and SimPO.", "questions": "1. This method trains only a subset of the total parameters. Could there be further performance improvements if all parameters were trained?\n2. Could this method improve performance on other tasks such as mathematics or coding, beyond preference alignment?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\n1. This paper conducts experiments on various models and preference algorithms.\n2. The suggested method is mathematically proven and presents a novel approach compared to existing methods.\n3. This research is easily compatible with major alignment algorithms such as DPO and SimPO\n\nWeaknesses\n1. The baseline performance is strange. For example, in Table 1, the Llama-3-Instruct-8B with the UltraFeedback dataset setup is very similar to the SimPO paper's experimental setup. However, SimPO achieves 40.3 on AlpacaEval 2.0 LC and 32.6 on Arena Hard based on DPO, which differs significantly from the reported scores of 32.59 and 22.9 in this paper. \n2. This paper does not display the pre-training (before preference alignment) model performance in the table. According to the SimPO paper, the Llama-3-Instruct-8B's Arena Hard performance is 22.3, which means DPO training shows virtually no performance improvement. This is just one example, and the main table should include both pre- and post training performance for all experimental results, showing relative performance improvements in tabular form. While it is understandable that training setups may vary across different experimental environments, the discrepancy here is too severe.\n3. The memory efficiency advantage of ComPO claimed in the paper is limited. ComPO requires training DPO or SimPO on clean data first, followed by ComPO training, which ultimately means that memory-intensive training operations must still be performed.", "was_revised": true, "final_justification": "The reviewer's response has addressed most of my questions, and I have accordingly raised the score", "rebuttal": "Thank you for your time and your input. We hope that our answers below will convince you about the merits of our work. We answer your questions below one-by-one, and have included these discussions in a revised version of our paper.\n\n1. **The baseline performance is strange. In Table 1, the Llama-3-Instruct-8B with the UltraFeedback dataset setup is very similar to the SimPO paper's experimental setup. However, SimPO achieves 40.3 on AlpacaEval-2.0 LC and 32.6 on Arena-Hard based on DPO, which differs significantly from the reported scores of 32.59 and 22.9 in this paper.** \n\nThank you for pointing this out. The discrepancy largely stems from differences in evaluation versions. Specifically, the SimPO paper reported results using an outdated version of AlpacaEval2 (v0.6.2), which has since been revised due to changes in vLLM decoding. This issue is acknowledged by the SimPO authors themselves on their GitHub:\n\n>*“AlpacaEval has a major revision for vllm decoding since 0.6.3 and causes a discrepancy from our experiments.”*\n\nIn contrast, our experiments use AlpacaEval2 v0.6.6 (explicitly noted on line 758 of our paper), which provides more accurate and consistent evaluations. For example, we re-evaluated SimPO's Llama-3-Instruct-8B-SimPO-v0.2 checkpoint using v0.6.6 and observed an LC score of 49.53—lower than the 53.7 reported on their github page, further confirming the evaluation shift.\n\nRegarding training: to perform ablations on clean vs. noisy preference pairs, we train DPO (for Mistral-7B and Llama-3-8B) on clean data using H100 GPUs, use SimPO checkpoint (for Gemma-2-9B) as released without retraining. For consistency and reproducibility, we followed default TRL (huggingface/trl from Github) hyperparameters. Notably, our SimPO+ComPO results do use the official SimPO checkpoint for fair comparison.\n\n2. **This paper does not display the pre-training (before preference alignment) model performance in the table. According to the SimPO paper, the Llama-3-Instruct-8B's Arena Hard performance is 22.3, which means DPO training shows virtually no performance improvement. This is just one example, and the main table should include both pre- and post training performance for all experimental results, showing relative performance improvements in tabular form...** \n\nWe agree that including pre-training/SFT baselines is important for evaluating relative gains. We will add all relevant SFT (preference-untrained) performance evaluation results in the revised version.\n\nRegarding the specific Llama-3-Instruct-SFT checkpoint: the one used in SimPO is no longer available on Hugging Face. As such, we used Meta’s official Llama-3-Instruct release, referenced explicitly in our appendix (page 20). We note that this might not be a perfect match to SimPO’s original setup. In addition, Arena-Hard has since been updated to use GPT-4.1 as the judge recently, while our results in the submitted paper rely on GPT-4-turbo. We have summarized supplementary evaluations using the updated judge (see our answers to Q4), which shows that ComPO achieves a consistent improvement. \n\n3. **The memory efficiency advantage of ComPO claimed in the paper is limited. ComPO requires training DPO or SimPO on clean data first, followed by ComPO training, which ultimately means that memory-intensive training operations must still be performed.** \n\nThank you for highlighting this. We clarify that our two-stage training (DPO + ComPO) in Table 1 was designed for ablation purposes: to isolate the effect of noisy vs. clean pairs. It demonstrates DPO’s limitations with noisy data, as evidenced by the minimal performance difference between DPO and DPO\\_clean. \n\nIn practical deployment, one may conduct the alignment process as analogous to the pre-training vs. post-training paradigm. A user may either choose to run both DPO/SimPO and ComPO together or, more likely, choose not to perform the initial, memory-heavy DPO/SimPO training themselves. Instead, they would start with an existing, publicly available model that has already been aligned on a general, high-quality dataset (e.g., DPO-tuned checkpoints). ComPO empowers a user to take that pre-aligned public model and further refine it using their own, potentially noisy and task-specific, preference data, using  a more affordable GPU platform.\n\nAs shown in Table 2, applying ComPO directly on existing, well-tuned SimPO checkpoints allows ComPO to extract value from noisy data without retraining from scratch, offering a more memory-efficient and scalable approach in real-world settings.\n\n4. **This method tunes a subset of the total parameters. Could there be further performance improvements if all parameters were trained?** \n\nWe agree and have conducted experiments to investigate the effect of number of perturbations, number of perturbed layers, gradient entry threshold and number of noisy pairs used.\n\nTo study the effect of number of perturbations $m$, we conducted experiments on Mistral-7B-Instruct using 5 independent trials for each setting (all other hyperparameters unchanged). The results are:\n\n$$\\\\begin{array}{c|cc}\n\\\\textbf{Perturbation ($m$)} & \\\\textbf{WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n800  & 17.32\\\\,\\\\pm\\\\,0.86\\\\;(17.94) & 24.72\\\\,\\\\pm\\\\,1.02\\\\;(25.12)\\\\\\\\\n1600 & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17)\\\\\\\\\n3300 & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14)\\\\\\\\\n5400 & 19.69\\\\,\\\\pm\\\\,0.36\\\\;(20.07) & 26.49\\\\,\\\\pm\\\\,0.81\\\\;(27.20)\\\\\\\\\n\\\\end{array}$$\n\nWe observe that increasing $m$ improves both WR and LC, at the cost of higher computation time. Importantly, peak memory usage remains unchanged, as we do not store all perturbation vectors $\\\\{y_i\\\\}_{1 \\leq i \\leq m}$ -- only the running average gradient estimate is maintained (see Line 5 of Algorithm 2).\n\nWe conduct experiments to perturb 3 layers (MLPs in layers 30–31 and the output layer) in Mistral-7B-Instruct, keeping all other parameters the same. The AlpacaEval2 (AE) and ArenaHard (AH) results are: \n\n$$\\\\begin{array}{c|cc|c}\n\\\\textbf{Layer perturbed (parameters)} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n1 (0.13B)  & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17) & 10.80\\\\,\\\\pm\\\\,0.21 \\\\;(11.0)\\\\\\\\\n3 (0.25B) & 18.19\\\\,\\\\pm\\\\,0.81\\\\;(19.38)  & 26.00\\\\,\\\\pm\\\\,0.89\\\\;(27.09)  & 11.26\\\\,\\\\pm\\\\,0.36 \\\\;(11.7)\\\\\\\\\n\\\\end{array}$$\n\nWe adopt the most recent version of Arena‑Hard with GPT‑4.1 as the judge -- an upgrade from GPT‑4 Turbo used in our original results. This change significantly impacts evaluation outcomes: for example, DPO’s WR drops from 14.4\\% (under GPT‑4 Turbo) to 10.5\\% (under GPT‑4.1), while ComPO's results show consistent improvement under stronger evaluation settings. \n\nPerturbing more layers leads to better performance due to a richer gradient direction set. The GPU memory increases from 16.3GB to 16.7GB, and processing 600 perturbations takes 60 seconds (vs. 50 seconds), which we consider reasonable.\n\nWe conducted ablation studies on the gradient entry threshold $\\lambda_g$ -- which selectively updates only high-magnitude gradient entries -- under $m=3300$ with 5 independent trials:\n\n$$\\\\begin{array}{c|c|cc}\n\\\\textbf{Threshold $\\\\lambda_{g}$} & \\\\textbf{\\\\% updated gradient entry} & \\\\textbf{WR \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\\n\\\\hline\n0  & 100\\\\% & 15.72\\\\,\\\\pm\\\\,0.77\\\\;(16.34) & 23.42\\\\,\\\\pm\\\\,1.03\\\\;(24.28)\\\\\\\\\n0.00004 &  63\\\\% & 16.02\\\\,\\\\pm\\\\,0.69\\\\;(16.69) & 24.01\\\\,\\\\pm\\\\,0.91\\\\;(25.10)\\\\\\\\\n0.00018 &  6\\\\% & 19.02\\\\,\\\\pm\\\\,0.62\\\\;(20.15) & 26.06\\\\,\\\\pm\\\\,0.81\\\\;(27.27)\\\\\\\\\n0.00022 & 1\\\\% & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\\\;(27.14)\\\\\\\\\n0.00025 & 0.15\\\\% & 16.10\\\\,\\\\pm\\\\,0.11\\\\;(16.21) & 23.82\\\\,\\\\pm\\\\,0.23\\\\;(24.00)\\\\\\\\\n\\\\end{array}$$\n\nWe found that setting $\\lambda_g$ to retain 1\\%-5\\% of entries offers the best trade-off between performance and stability. Too high or too low thresholds degrade performance.\n\nIn the submitted paper, we reported ComPO’s performance using only 100 noisy preference pairs. Here, we extend the analysis to 300 noisy pairs and observe consistent improvements on both AE and AH:\n\n$$\n\\\\begin{array}{c|cc|c}\n\\\\textbf{\\\\# of noisy pairs} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n100  &19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14) & 11.02\\\\,\\\\pm\\\\,0.13\\\\;(11.2) \\\\\\\\\n300 & 20.07\\\\,\\\\pm\\\\,0.99\\\\;(21.35) & 26.28\\\\,\\\\pm\\\\,0.81\\\\;(27.59) & 11.76\\\\,\\\\pm\\\\,0.30\\\\;(12.1) \\\\\\\\\n\\\\end{array}\n$$\n\nDue to time constraints, we used the same hyperparameters as in the 100-sample training. However, we believe performance could be further improved with targeted tuning, particularly the learning rate. \n\n5. **Could this method improve performance on other tasks such as mathematics or coding, beyond preference alignment?** \n\nThank you for this thoughtful suggestion. While ComPO is designed to address issues like verbosity and likelihood displacement, it is less clear whether these issues are bottlenecks in tasks such as mathematics or coding. In those domains, the challenges often involve reasoning fidelity and structured problem solving, where methods like PPO or GRPO that directly optimize task-specific reward signals (e.g., via CoT) are typically more effective. We see this as a valuable direction for future work. \n\nWe sincerely appreciate your close reading and constructive feedback. We hope our responses clarify the key points of our approach and address your concerns. Please don’t hesitate to reach out for further discussion or clarification."}, {"review_id": "jR2Xlwhac5", "reviewer": "Reviewer_ArR9", "summary": "This paper introduces ComPO, a novel method for preference alignment based on comparison oracles. Motivated by the limitations of existing direct preference optimization (DPO) methods, specifically verbosity and likelihood displacement, the authors propose a zeroth-order optimization approach that treats pairwise human preferences as comparison oracle outputs.", "questions": "1. What is the specific version of the Mistral and Mistral-Instruct models? There are versions from v0.1 to v0.3. But in the link you put in the Appendix, it shows Zephyr, instead of Mistral model.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros: \n1. The paper addresses an important problem in alignment, i.e., the preference pairs in the dataset may be noisy.\\\n2. The method gives a theoretical guarantee on the resulting algorithm.\\\n3. The experiments are conducted on multiple models, which well support the claims of the paper.\n\nCons:\n1. The algorithm introduces additional hyperparameters and computation, which may be hard to tune and adjust the trade-off on larger models and scale.\\\n2. There are previous works that also study similar problems in DPO, such as likelihood displacement, by simply conditioning on the reward scores [1, 2]. Intuitively, these approaches can also deal with the noisy preference data, since these pairs have similar reward scores and the algorithm does not struggle with disentangling them.\\\n3. Only the DPO baselines are compared in experiments. I would like to see the authors add discussions and comparisons with more related works, such as the above ones that use conditional DPO.\n\n\n[1] Kim et al. \"Margin matching preference optimization: Enhanced model alignment with granular feedback.\"\\\n[2] Zhang et al. \"Reward-Augmented Data Enhances Direct Preference Alignment of LLMs.\"", "was_revised": true, "final_justification": "I decide to keep my score.", "rebuttal": "Thank you for your encouraging comments and positive evaluation! We reply to your main questions point-by-point below and have included these discussions in the revised version of our paper.\n\n1. **The algorithm introduces additional hyperparameters and computation, which may be hard to tune and adjust the trade-off on larger models and scale.** \n\nWe agree on the importance on training scalability. In respond to your concerns, we have conducted experiments to investigate the effect of number of perturbations, number of perturbed layers, gradient entry threshold and number of noisy pairs used.\n\nTo study the effect of number of perturbations $m$, we conducted experiments on Mistral-7B-Instruct using 5 independent trials for each setting (all other hyperparameters unchanged). The results are:\n\n$$\n\\\\begin{array}{c|cc}\n\\\\textbf{Perturbation ($m$)} & \\\\textbf{WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n800  & 17.32\\\\,\\\\pm\\\\,0.86\\\\;(17.94) & 24.72\\\\,\\\\pm\\\\,1.02\\\\;(25.12)\\\\\\\\\n1600 & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17)\\\\\\\\\n3300 & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14)\\\\\\\\\n5400 & 19.69\\\\,\\\\pm\\\\,0.36\\\\;(20.07) & 26.49\\\\,\\\\pm\\\\,0.81\\\\;(27.20)\\\\\\\\\n\\\\end{array}\n$$\n\nWe observe that increasing $m$ improves both WR and LC, at the cost of higher computation time. Importantly, peak memory usage remains unchanged, as we do not store all perturbation vectors $\\\\{y_i\\\\}_{1 \\leq i \\leq m}$ -- only the running average gradient estimate is maintained (see Line 5 of Algorithm 2).\n\nWe conduct experiments to perturb 3 layers (MLPs in layers 30–31 and the output layer) in Mistral-7B-Instruct, keeping all other parameters the same. The results are: \n\n$$\\\\begin{array}{c|cc|c}\n\\\\textbf{Layer perturbed (parameters)} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n1 (0.13B)  & 17.50\\\\,\\\\pm\\\\,0.65\\\\;(18.32) & 25.02\\\\,\\\\pm\\\\,0.91\\\\;(26.17) & 10.80\\\\,\\\\pm\\\\,0.21 \\\\;(11.0)\\\\\\\\\n3 (0.25B) & 18.19\\\\,\\\\pm\\\\,0.81\\\\;(19.38)  & 26.00\\\\,\\\\pm\\\\,0.89\\\\;(27.09)  & 11.26\\\\,\\\\pm\\\\,0.36 \\\\;(11.7)\\\\\\\\\n\\\\end{array}$$\n\nNotably, we adopt the most recent version of Arena‑Hard, which uses GPT‑4.1 as the judge -- an upgrade from GPT‑4 Turbo used in our original results. This change significantly impacts evaluation outcomes: for example, DPO’s WR drops from 14.4% (under GPT‑4 Turbo) to 10.5% (under GPT‑4.1), whereas ComPO's results show consistent improvement. This highlights the robustness of our approach under stronger evaluation settings. \n\nPerturbing more layers leads to better performance due to a richer gradient direction set. The GPU memory increases from 16.3GB to 16.7GB, and processing 600 perturbations takes 60 seconds (vs. 50 seconds), which we consider reasonable.\n\nWe conducted ablation studies on the gradient entry threshold $\\lambda_g$ -- which selectively updates only high-magnitude gradient entries -- under $m=3300$ with 5 independent trials:\n\n$$\n\\\\begin{array}{c|c|cc}\n\\\\textbf{Threshold $\\\\lambda_{g}$} & \\\\textbf{\\\\% updated gradient entry} & \\\\textbf{WR \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{LC \\\\% – mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\\n\\\\hline\n0       & 100\\\\%  & 15.72\\\\,\\\\pm\\\\,0.77\\\\;(16.34) & 23.42\\\\,\\\\pm\\\\,1.03\\\\;(24.28)\\\\\\\\\n0.00004 &  63\\\\%  & 16.02\\\\,\\\\pm\\\\,0.69\\\\;(16.69) & 24.01\\\\,\\\\pm\\\\,0.91\\\\;(25.10)\\\\\\\\\n0.00018 &   6\\\\%  & 19.02\\\\,\\\\pm\\\\,0.62\\\\;(20.15) & 26.06\\\\,\\\\pm\\\\,0.81\\\\;(27.27)\\\\\\\\\n0.00022 &   1\\\\%  & 19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14)\\\\\\\\\n0.00025 &  0.15\\\\% & 16.10\\\\,\\\\pm\\\\,0.11\\\\;(16.21) & 23.82\\\\,\\\\pm\\\\,0.23\\\\;(24.00)\\\\\\\\\n\\\\end{array}\n$$\n\nWe found that setting $\\lambda_g$ to retain 1\\%-5\\% of entries offers the best trade-off between performance and stability. Too high or too low thresholds degrade performance.\n\nIn the submitted paper, we reported ComPO’s performance using only the first 100 noisy preference pairs. Here, we extend the analysis to 300 noisy pairs and observe consistent improvements on both AlpacaEval2 (AE) and Arena‑Hard (AH).\n\n$$\n\\\\begin{array}{c|cc|c}\n\\\\textbf{\\\\# of noisy pairs} & \\\\textbf{AE-WR \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)} & \\\\textbf{AE-LC \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\ & \\\\textbf{AH (GPT-4.1)-WR  \\\\% - mean}\\\\,\\\\pm\\\\,\\\\textbf{std (best)}\\\\\\\\ \\\\hline\n100  &19.21\\\\,\\\\pm\\\\,0.58\\\\;(20.25) & 25.91\\\\,\\\\pm\\\\,0.95\\\\;(27.14) & 11.02\\\\,\\\\pm\\\\,0.13\\\\;(11.2) \\\\\\\\\n300 & 20.07\\\\,\\\\pm\\\\,0.99\\\\;(21.35) & 26.28\\\\,\\\\pm\\\\,0.81\\\\;(27.59) & 11.76\\\\,\\\\pm\\\\,0.30\\\\;(12.1) \\\\\\\\\n\\\\end{array}\n$$\n\nDue to time constraints, we used the same hyperparameters as in the 100-sample training without additional tuning. However, we believe performance could be further improved with targeted tuning, particularly of the learning rate. We have also publicly released the best-performing checkpoints to support follow-up research from the community.\n\n2. **There are previous works that also study similar problems in DPO, such as likelihood displacement, by simply conditioning on the reward scores [1, 2]. Intuitively, these approaches can also deal with the noisy preference data, since these pairs have similar reward scores and the algorithm does not struggle with disentangling them. Only the DPO baselines are compared in experiments. I would like to see the authors add discussions and comparisons with more related works, such as the above ones that use conditional DPO.** \n\nWe acknowledge that learning from noisy preference data has been studied in prior works, including those leveraging reward scores via conditional DPO [1, 2], and we will cite and discuss them in the revised manuscript. That said, our contribution lies in how we address noisy preferences. ComPO introduces a principled method for incorporating specialized comparison oracles. This approach not only offers theoretical guarantees but demonstrates strong empirical performance in large-scale experiments, setting it apart from existing methods.\n\nWe appreciate your suggestion to include comparisons with conditional DPO. Notably, conditional DPO modifies the DPO objective by conditioning on reward scores and solving it via gradient-based methods. This formulation is compatible with ComPO and can be combined in a similar way as we did with SimPO+ComPO. We will discuss this connection and provide comparisons in the revised manuscript.\n\n3. **What is the specific version of the Mistral and Mistral-Instruct models? There are versions from v0.1 to v0.3. But in the link you put in the Appendix, it shows Zephyr, instead of Mistral model.**\n\nThank you for pointing this out. Zephyr is an instruction-tuned model built on top of the Mistral-7B-v0.1 base model, effectively serving as a Mistral-Instruct SFT variant. We will clarify this and explicitly state the version used in the final version of the paper.\n\nWe thank you again for your detailed reading and your constructive input! We hope and trust that our replies have alleviated your concerns, and we look forward to an open-minded discussion if any such concerns remain."}]

Author rebuttal:

Meta-review: The reviewers agree that RL with the comparison oracle proposed in this paper is novel and well-motivated. The theoretical grounding and the convergence proof further strengthen the methodological contribution. Ablations studies and experiments have demonstrated the robustness of the method.
Decision: Accept (poster)