As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
Authors: Xin Mao
Huimin Xu
Feng-Lin Li
Ziqi Jin
WANG CHEN
Wei Zhang
Anh Tuan Luu
Venue: ICLR 2025 Poster (2025)
Keywords: LLM Alignemnt
Preference Learning
Text Generation
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel LLM alignment loss that establishes a stable Bidirectional Negative Feedback (BNF) during optimization, which does not require any tunable hyper-parameters or pairwise preference data.

Abstract: Direct Preference Optimization (DPO) has emerged as a more computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO), eliminating the need for reward models and online sampling. Despite these benefits, DPO and its variants remain sensitive to hyper-parameters and prone to instability, particularly on mathematical datasets. We argue that these issues arise from the unidirectional likelihood-derivative negative feedback inherent in the log-likelihood loss function.
To address this, we propose a novel LLM alignment loss that establishes a stable Bidirectional Negative Feedback (BNF) during optimization. 
Our proposed BNF loss eliminates the need for pairwise contrastive losses and does not require any extra tunable hyper-parameters or pairwise preference data, streamlining the alignment pipeline to be as simple as supervised fine-tuning.
We conduct extensive experiments across two challenging QA benchmarks and four reasoning benchmarks. 
The experimental results show that BNF achieves comparable performance to the best methods on QA benchmarks, while its performance decrease on the four reasoning benchmarks is significantly lower compared to the best methods, thus striking a better balance between value alignment and reasoning ability. 
In addition, we further validate the performance of BNF on non-pairwise datasets, and conduct in-depth analysis of log-likelihood and logit shifts across different preference optimization methods.
We will release all the source code, checkpoints, and datasets on GitHub.

Published as a conference paper at ICLR 2025
AS SIMPLE AS FINE-TUNING: LLM ALIGNMENT VIA BIDIRECTIONAL NEGATIVE FEEDBACK Loss
Xin Mao', Huimin Xu', Feng-Lin Li", Ziqi Jin', Wang Chen?, Wei Zhang", Ann Tuan Luu's NNanyang Technological University, Shhppee Pte. Ltd, SEAA Group {xin sxin.mao, anhtuan. luu, huimin.xujente xu.entu. edu. sg, ZIQI0078e aatu.edu.5. sg [fenglin.li, chen. www..
com, terry. zhang@ com
ABSTRACT
Direct Preference Optimization (DPO) has emerged as a more computationally ef- ficient alternative to Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO), eliminating the need for reward models and online sampling. Despite these benefits, DPO and its variants remain sensitive hyper-parameters and prone to instability, particularly on mathematical datasets. We argue that these issues arise from the unidirectional likelihood-derrvative neg- ative feedback inherent in the log-likelihood loss function. To address this, we propose a novel LLM alignment loss that establishes stable Bidirectional Nega- tive Feedback (BNF) during optimization. Our proposed BNF loss eliminates the need for pairwise contrastive losses and does not require any extra tunable hyper- parameters or pairwise preference data, streamlining the alignment pipeline to be as simple as supervised fine-tuning. We conduct extensive experiments across two challenging QA benchmarks and four reasoning benchmarks. The experimental results show that BNF achieves comparable performance to the best methods on QA benchmarks. while its performance decrease the four reasoning bench- marks is significantly lower compared to the best methods, thus striking a better balance between value alignment and reasoning ability. In addition, we further validate the performance of BNF on non-pairve non-pairwise datasets, and conduct in-depth analysis of log-likelihood and logit shifts across different preference optimization methods. Github Url: https: http::://ithub... wtw.thub.ooooocce cwm/MaoX Maw.iinn/BBM
INTRODUCTION
Alignment of Large Language Models (LLMs) plays pivotal role in ensuring that these LLMs behave in accordance with human values and expectations (Bai et al. 20226). As LLMs become increasingly integrated into various applications (Zhang al.) [023 Roziere et al. (023), ensuring proper alignment is crucial to mitigate risks such as biased or harmful outputs, while also enhanc- ing trustworthiness. One of the most prominent LLM alignment methods is Reinforcement Learn- ing from Human Feedback (Ouyang et al. 2022) with Proximal Policy Optimization (RLHF-PPO) (Schulman et al., 2017), which underpins the success of ChatGPT. However, despite its achieve- ments, RLHF-PPO faces notable limitations, particularly regarding the high computational costs associated with reward modeling and online sampling (Casper al.) [202)}. These challenges com- plicate its widespread adoption, especially in scenarios where computational resources are limited.
In response to these limitations, Direct Preference Optimization (DPO) (Rafailov al., 2023) and its derivative methods (Ethayarajh et al.) 2024 Zhao et al. 2023) aim to simplify the overall alignment pipeline by eliminating the need for reward and value models, as well as online sampling. Despite achieving impressive performance on QA and Chatbot tasks (Meng et al. (024). DPO-series meth- ods remain highly sensitive to hyper-parameter, and often exhibit instability (Xu et al. 2024b). This instability is particularly pronounced when applied to mathematical datasets, leading to potential training collapse (Pal et al. 2024). Recently, several efforts (Xu et al. 2024a Pal et al. 2024) have been made to address this issue. They attribute the collapse to erroneous decrease in the like- lihood of the preferred samples and propose using Negative Log Likelihood (NLL) regularization to stabilize the training process. Although these methods successfully prevent the model from col- lapsing on mathematical datasets, they perform poorly on several popular Chat and QA benchmarks (Meng et al. 2024) and introduce additional hyree-parameters

---

Published as a conference paper at ICLR 2025
Different from previous studies, we hypothesize that the insta- bility of DPO may stem from a fundamental cause: the uni- directional likelihood-derivative negative feedback inherent in the log-likelihood loss. As shown in Figure [ when applying <a
0c bzy
LNLL
Pos LBNF
the NLL loss to increase the likelihood (o
Eist e*
of preferred output Yuu the partial derivative @DCxL with re- 33yw
spect to the unnormalized logit zye will gradually decrease, which limits the rate of future increases in zy. and ( (z), preventing the model from over-fitting. However, when the same loss is used to decrease the likelihood of a dispreferred output y/, this negative feedback turns into positive one.
Tyer(y(x)
my (y(x)
As Ty ( (z) decreases, d-CNL continues to rise, which fur- Jay,
Figure I: The curve of NLL and BNF loss.
ther accelerates subsequent decreases in 2y and ( ul- timately resulting in model collapse. To mitigate this issue, DPO-series methods introduce pairwise contrastive losses to constrain the likelihood of y from de- viating excessively from that of the preferred sample yu- However, this constraint is sensitive to hyper-parameters and lacks stability, resulting in the failure of DPO on mathematical datasets.
Based on the above findings, we propose novel alignment loss, Bidirectional Negative Feedback (BNF) loss. As shown in Figure when optimizing with BNF loss, the partial derivative 85,
ac
reaches its maximum only when na (y)x) = [=]()>>) (i.e., the initial state). Whether the likelihood decreases linearly both directions, thus establishing a bidi-
To(y[>) increases or decreases, ac 82,
rectional negative feedback. Since this bidirectional negative feedback fundamentally addresses the issue of excessive decreases in the likelihood of dispreferred samples, BNF eliminates the need for pairwise contrastive losses, further streamlining the alignment pipeline to be as simple supervised fine-tuning. In summary, BNF loss offers the following advantages:
Less alignment tax: Recent studies (Ouyang et al,, 2022 Lin et al. 2023) have shown that using preference optimization methods to align LLMs with human values often harms their reasoning ability, referred to as the alignment tax. With the bidirectional negative feedback, our proposed BNF can naturally prevent the model from over-fitting to preference data, striking a better balance between human values and reasoning ability while minimizing the alignment tax.
Since the need for pairwise contrastive losses is eliminated, our proposed BNF loss does not involve any extra tunable hyper-parameters such as scaling factor 3 in DPO [Rafallov et al., [223), margin y in SimPO (Meng al.) (024) or NLL regularization weight À in SLiC (Zhao et al. (023).. which significantly reduces the cost of grid searches.
Fewer tunable hyper-parametent
Less pairwise data: In addition fewer tunable hyper-parameters, the removal of pairwise con- trastive losses means that BNF no longer relies on pairwise preference data. Similar to KTO (Ethayarajh et al. 2024). BNF loss only requires either single preferred or dispreferred samples without the need for parwise matching. significantly reducing data construction costs.
To comprehensively evaluate the effectiveness of our proposed
Table I: Elo Rank on Wild-Bench.
BNF, we conduct extensive comparison experiments across two popular QA benchmarks and four reasoning benchmarks, using three 7B-9B LLMs as the base models. On two QA bench- marks, BNF achieves comparable performance the best base- lines, SimPO and DPO. Notably, we produce a top-performing model. based on Gemma-2-9B-iz (Team et al. (0024), which out- performs not only all basclines but also larger scale LLMs like Bemma-2-27BBit and Hommotron-43400--insee
Model
Elo
GPT-4o (05-13) Mlaude---Opus
1237
1216
Gemma 1ammaa-2-90 2-9B-BB BNF
1186
Nemotron-4-344 it
1184
Cemma-2-27BB
1183
(Adler et al.
1181
2024) on the most challenging benchmark, Wild-Bench (Lin et al. 2024). As for the reasoning benchmarks, BNF's perfor- mance decrease is significantly lower than that of SimPO (Meng al., 2024) and DPO, indicating that BNF can strike a better balance between value alignment and reasoning abilities, thus paying less alignment tax. In addition, we design an experiment to validate the performance of BNF on non-pairwise datasets and conduct in-depth analysis of log- likelihodd and logit shifts across differ- preference optimization methods.
Gemma-2-9B-DPP
1181
2

---

Published as a conference paper at ICLR 2025
2 THEORETICAL ANALYSIS OF LOG-LIKELIHOOD AND DPO
In this section, we first provide a detailed discussion on how the unidirectional negative feedback leads to an excessive decrease in the likelihood of dispreferred samples, ultimately causing model collapse. Then, we explain how DPO-series methods mitigate this excessive decrease by employing pairwise contrastive loss, and why this approach is ineffective for mathematical datasets.
2.1 LIMITATION OF LOG-LIKELIHOOD Loss
In the Supervised Fine-Tuning (SFT) stage, we typically only use the NLL loss to maximize the likelihood of each sample from the dataset without worrying about collapse. In the alignment stage, a naive approach would be to try the following log-likelihood loss for preference optimization:
(1) This naive loss tries to increase the likelihood of the preferred sample with NLL and decrease the likelihood of the dispreferred sample YI with Positive Log Likelihood (PLL). However, LLMs will quickly collapse when optimized with the above loss (Ratailov et al. (023). As mentioned in Section the underlying reason is the unidirectional log-likelihood loss. Let's consider a simple case where the response y consists of only single token. When we apply the NLL loss to increase the likelihood To (V of the preferred yo:
LNLLAPLL _( log To (z) + log Te (3n[22]]]
negative feedback inherent in the
e
LNLI = B(a.w))D log no (yar [z]] = R (aD
log
(2)
Lk=1
e
where [V] is the vocabulary size. The partial derivative of the NLL loss with respect to the logit
Fyw is given by
acxi
- To To(Wwla) (derivation in Appendix A. In this case, the likeli-
actually establish a stable negative feedback. As gradually decreases, which limits the rate of future increases in Z... and (o (c)., thereby preventing the model from over-fitting.
hood To (yw (z) and the partial derivative &Cuu
85yw
(x( increases, Dzyw
@Cxxu|
However, this negative feedback is unidirectional. When CPLL E (a (otesnn>Dlloo blog (o(()))) is used to decrease the likelihood no (1n (zr), the above negative feedback will turn into positive one, which means that any decrease in ne (30 (z) will further accelerate itself. As To (31(z) decreases, To (2n(a)) continues to rise, which further accelerates sub- sequent decreases in Zyn and (M] ultimately resulting in model collapse. Similarly, since the log-likelihood of longer response is the sum of the log-likelihoods of individual tokens log To(y)a)) E; log No (yi yes), the aforementioned conclusion still holds.
the partial derivative (8Cms = 0zu;
2.2 ROLE OF PAIRWISE CONTRASTIVE LOSSES
Since solely using vanilla log-li lodeeheed loss in preference optimization will cause model collapse, DPO-series methods introduce pairwise contrastive losses to stabilize the optimization process. The gradients of most DPO-series methods with respect to parameters 0 could be written as follows:
]]
V E (r.
C(yi, Bw: no, Tret)
Ve log (v)
V log To (1)
(3)
constrain the liglliikihoods gap
increase likelihood of you
decrease likelihood of gr.
The second half of Eq. (3) is similar to the gradient of Eq. I which aims increase the like- lihood of the preferred sample You and decrease the likelihood of the dispreferred sample yt- The key difference is the introduction of pairwise contrastive function C(31, Yurs kg, Trat). which con- strains the likelihood of dispreferred samples from deviating excessively from that of the preferred zo (x) ones. Taking DPO as an example, CD Yors Tg, ( = B- [3 log mo(g(|=) 3 log war(Mw (2)
where CDPO decreases gradually the ligllikelihood gap between yı and yo increases, creating negative feedback that constrains log ( xx) from deviating excessively from log To ( (z)) Table
2 lists several representative DPO-series methods and their derived constraint functions C. Despite variations in specific implementations, these methods share a common core idea: using pairwise contrastive losses to avoid excessive decreases in the likelihood of dispreferred samples.
3

---

Published as a conference paper at ICLR 2025
Table 2: The objective C and derived constraint C of representative DPO-series methods.
Method
Objective C
C (1he Mit TT. (et)
log (a ix) - log xe (x)
SLSC.HP Zhao al.
2023
max (0 log ma (Dw]/a) log (a))
(xx
0.
log malp ir) log Ro (a) >
DPO Rafailov et al.
2023
logg o log solowla)
Plog Solorin) (n)
log Parlbu m]] =e(p=l1)) 2 + log vagisile) mut
8 a (8 log, Safiniss
*ainu In]
Varlon
Azar et
2023
log Ste (x)
extsule)
3.)
ex(mile)
SimPO Meng et al.
2024
logor [aol log (e(uula))
Tosl log ne (solax) -7)
a Tanl log (a)la)))) 1uo log (s(ll))))
2.3 FAILURE ON MATHEMATICAL DATASETS
Generally, constructing preference dataset involves three steps: (1) generating multiple responses for each prompt in the instruction dataset, (2) scoring each response with human annotators or reward models, and (3) selecting the highest and lowest scoring responses as the preference pair. However, since mathematical reasoning requires rigorous logic, the diversity of responses to math problems often significantly lower compared to other types of problems. As a result, mathematical prefer- ence datasets often contain many highly similar pairs, some differing by even only one single token I Given this data distribution, selecting an appropriate scaling factor / for DPO-series methods is challenging. as it is difficult to create substantial log-likell liglikkll gap for highly similar preference pairs. As illustrated in Figure although the likelihood of the correct answer is significantly greater than that of the incorrect one (0.8 vs. 0.05), the total log-likelihood gap still only amounts to 1.2. In this situation, if the scaling factor 33 is set as usually (e.g., 0.1 in DPO), the scaled gap will be- come extremely small, which may cause the pairwise contrastive function C to fail in preventing the excessive decrease of Ty (31 (z). Conversely, if 3 is set higher than usual, it may lead to underfitting for the preference pairs with less overlap, which also negatively affects performance. Recent studies have proposed two solutions: (1) adding NI.L regularization to maximize the likelihood of preferred responses (Saeidi et al., 2024), and (2) directly removing preference pairs with small edit distances from the dataset (Pal et al. 2024). Different from the above methods, this paper addresses this issue through a more fundamental approach, introducing bidirectional negative feedback, which enables preference optimization to eliminate reliance on contrastive loss functions and remove the need for tuning any pappe--arametrss.. In Appendix [] we explore the performance of different alignment methods on a purely mathematical preference dataset to further validate the effectiveness.
First
5
Σ, lo (x)=009
0.35
0.8
0.93
0.85
0.69
0.8
gap 1.2
y'
First
5
To (x'
0.35
0.8
0.93
0.85
0.69
0.05
Figure 2: An example of mathematical preference dataset. Due to the significant overlap between preferred and dispreferred samples, it is difficult to create a substantial log- likelihoihod gap.
BIDIRECTIONAL NEGATIVE FEEDBACK Loss
3.1 OPTIMIZATION OBJECTIVE
In Section 2. we argue that the limitation of the log-likelihood loss in preference optimization stems from the unidirectional negative feedback. While DPO-series methods introduce pairwise contrastive losses to address this issue, they face challenges in hyper-parameter tuning, especially when applied to mathematical preference datasets. Building on these insights, we propose novel alignment loss that establishes bidirectional negative feedback, eliminating the need for contrastive losses and simplifying LLM alignment to the level of supervised fine-tuning. Given a policy model Tg. a reference model Tee, and preference dataset D where each response is labeled as either pre- ferred or not (without necessarily being paired), the optimization objective of our proposed BNF
-aa-mmathh-a -att--mathhheeeaeee -er---reeeer wwwppreeeeeeeeeeeeeee dpo
4

---

Published as a conference paper at ICLR 2025
loss is described as follows:
D2
(3)
LNNF -B(z.g)oD
fENF (vis t2) log (t(tt) Bei)
(4)
where [8] represents the length of the response y, label(y) - -{ -1,
y is preferred
if y is dispreferred is the
annotation label of y. [V] is the vocabulary size, and i is the j-th token in the vocabulary. In fact, Eq 4 is merely standard manghhnormmllize cross-entropy loss. If we adopt the following one-hot fBNF fuL the target distribution, the proposed loss will be same with Eq. [
if D/ 1/3
(un, b2) 0,
if y: +6g
Therefore, the core of our proposed method lies in the target distribution SENF(U,, 1j) of the cross- entropy loss, which we call Dynamic Target Distribution (DTD):
1)]-
8g
min
if yf 11
fuNr tt)
1- (x.le. To (1/) ver)] if ys #1
1- (v..2))
where sg represents the stop gradient operation (i.e., detach in PyTorch). Since fNNF(W)) satis- fies SBNF(Us, f2) 2 0 and Σ/ [uNN((s, f2) = 1 for all yi and łj (derivation in Appendix A.2). it constitutes a valid probability distribution. Although DTD seems complex at first glance, its code implementation is quite simple and efficient (as shown in Appendix B Moreover, compared to DPO-series methods, the BNF loss involves no tunable for pairwise preference data, which reduces the costs of grid searches and dataset construction. In the next section, we will explain how this loss function establishes a bidirectional negative feedback.
and eliminates the need
3.2 GRADIENT ANALYSIS
For mechanistic understanding of the proposed BNF loss, we need analyze the gradient of the loss function LANF. The gradient with respect to parameters 0 can be written as:
lel
M
dLENF
V VaLANF (EA,
Van
3.841
2zis
where 2,, #,0C represents the original ouput logit of (ty Vei) before Softmax Since fune(ys) is
a valid probability distribution with stop gradient, the partial derivative aCasr
with respect to any
Ds Ver
output logits 2xx
z.9<i can be obtained as follows (derivation in Appendix A.3
dLANF
label(y)
[to Uee)) fUNF(Ui,
(8)
y
By substituting Eq. (6) into Eq. (8)
derive function between
& 0= FiBlet
and nottaa,,
(as shown in Appendix A.4 Here, we focus on the token 44 within the responses y: if To ( yes) < (xr(y:l,,,e
murlgola. To (y:l [ yes) mart (eaee))
ØLNNF
lyb
|
To (y (r, yei):
if me wes) 2 [or yei)
From this piecewise function, we can observe that the partial derivative
#CgNE 0
reaches its max-
azy.
imum only when To (uul,, yes) = Trer(ys Jr. Me.). When (u6sa Mee) <
The kr, ves), since decreases linearly as we re(ullz FEe) decreases un-
I- (nne is always greater than O, Tortwaaa
til it reaches O. In contrast, when me Mee))) 2 Ter(yn
is equal to that of the
De
NLL loss, decreasing linearly no (T (r, Ves) increases. In this way, the proposed BNF loss estab- lishes a bidirectional negative feedback between the partial derivatives and the likelihoods, aligning precisely with the illustration in Figure T In Appendix will give more discussions in detail.
5

---

Published as a conference paper at ICLR 2025
Table 3: Statistical comparison of two instructtio-ffolloing QA benchmarks.
Dataset
#Tasks
#Turns
ChatHistory
QueryLen
PromptLen
RealUser
Evaluation
Arenallard
500
406
406
Pair
Bild-Bench
1,024
Σ5
979
3,402
V
Score+Pair
4 EXPERIMENTAL SETUP
We follow the experimental setup of SimPO (Meng et al., 2024 to objectively assess the effective- ness of our proposed method. They provide numerous checkpoints aligned with DPO-series methods and the corresponding training datasets, acknowledge their contributions. For a reference, we list the GPU hours for training and API cost for evaluation in Appendix F
Models and training datasets. In this paper, we adopt three mainstream open-source LLMs as the base models: Mistral-7BB 1eer/---nntuuttttl v0.2 (Jiang et al. 2023a), Meta-Llama- --ann----BB- 3--8--IInsruuu Instruct (Dubey et al. 2024). and Gemmma2- 2-9b-it (Team al. 2024). For fair comparison, we use the same preference training datasets constructed by SimPO. Specifically, for each prompt æ: in Ultrafeedback (Cui et al. 2024), they generate 5 responses with sampling temperature of 0.8. Then, using PairRM (iiag et al. (202bb or ArmoRM (Wang et al. 2024a) to score the 5 responses, selecting the highest- scoring one as yeu and the lowest-scoring one as y/, Here are the three training datasets: [Mistral- and ( ArmoRM
Ultrafeedback-Pairke Llama3--Utrafeede Satrraffedbac-AAmmoRRR
Training hyper-parameters. In this paper, we set the maximum sequence length 4096 and adopt the AdamW optimizer (Loshchilov & Hutter 2018), applying cosine learning rate schedule with 10% warm-up steps. Since our proposed BNF loss does not have any extra tunable hyper- parameters, we only perform grid searches on batch size (64, 128, 256} and learning rate (5e-7, 6e-7, 8e-7, le-6). After grid search, we adopt unified batch size of 128 and select learning rates of 5e-7 for Mistral-7B -ait------nnntttttbb-- 6c-7 for
and 8e-7 for Gemma-2-9b-it.
Baselines. In addition to DPO (Rafailov et al., 2023), we also compare our proposed BNF loss with the following strong baselines in preference optimization: (1) SLiC-HF (Zhao et al. 2023)
is based on contrastive ranking loss and integrates an NLL loss as the regularization term (2)
IPO (Azar et al.] 2023) is theoretically grounded method designed to bypass DPO's assumption
that pairwise preferences can be replaced by pointwise rewards. (3) KTO (Fthayarajh et al.] 2024)
learns from preference data that is not pairwise. (4) CPO (Xu et al. 2024a) introduces
NL.
regularization term to the DPO loss function. (5) SimPO (Meng et al., 2024) is a simpler and
effective preference optimization method without using reference model. All the baselines have been well-tuned through hyper-parameter grid searches, as described in Meng et al. (2024).
Evaluation benchmarks. We primarily evaluate all the models using twO recent proposed instruction-following QA benchmarks: Arena-Hard (Li et al., 2024) and Wild-BenchhLin et al., 2024). Arena-Hard, an enhanced version of MT-Bench (Zheng et al. 2023), includes 500 high- quality prompts from real user queries. For Arena-Hard, we report the standard win rate (WR) and length-controoled win rate (L.C), using GPT-4-0314 as the reference model and GPT-4o-mini as the evaluato Wild-Bench is highly challenging benchmark, featuring longer prompts and more difficult questions sourced from real users. Besides scoring each response on 100-point scale, Wild-Bench further introduces LMSYS-Elo (Chiang et al. 2024) to better rank all the models. For Wild-Bench, we report Elo and score using GPT-4o as the evaluator. The statistical comparison of these benchmarks are listed in Table 3 The reason do not adopt MT-bench and AlpacaEval (Dubois et al.) 2024) is due to the significant flaws in these datasets: MT-hench only contains 80 samples and exhibits poor separability across different methods (Meng et al.) 2024). AlpacaEval is a highly imbalanced dataset, with 50% of the question types focused on information seeking, while less than 20% are related to reasoning. In fact, Arena-Hard and Wild-Bench are upgraded versions of MT-Bench and AlpacaEval, offering more challenging tasks with a more balanced distribution. Furthermore, we also evaluate all the models on four logical reasoning benchmarks to verify the impact of these alignment methods on model's 's reasoning abilities, including: MMLU-redux (Gema) al. 2024) (Language). CRUX (Gu et al. 2024) (Code), GSM8K (Cobbe et al. 2021) and MATH- 1.5 (Hendrycks et al. 2021) (Math). For these reasoning benchmarks, we use Eva (Lin) 2024) as the evaluator, which aims to evaluate instruction-tundd LLMs for their zero-shot performance.
"The original evaluator was GPT-4-Turbo, we replace it with GPT-4o-mini due the high API costs.

---

Published as a conference paper at ICLR 2025
Table Main experimental results across all benchmarks. The numbers in parentheses (x) indicate the relative ranking of each method under this metric, while Average Rank represents the average ranking across all metrics for each method. For Gemma-2, since |Mng et al. (2024) only provides the checkpoints for DPO and SimPO, BNF is compared exclusively with these two methods.
Method
Wild- Bend-Bench
Arena-Hame Hard
GSM8K
MATH
CRUX
MMLU
Average
Elo
Score
I.C (%)
WR(%)
Ace (%)
Acc (%)
Ace (%)
Acc (%)
Rank
Base
1131 (8)
29.2 (8)
33.9 (8)
36.1 (8)
78.5 (4)
6.8 (7)
38.1 (4)
62.4 (3)
6.25
SLiC-HF
1142
33.2 (7)
49.1 (5)
46.3 (5)
66.6 (7)
8.3 (6)
33.9 (6)
61.4 (4)
5.63
DPO
1155 (1)
39.5 (1)
60.4 (1)
62.2 (1)
70.5 (6)
8.5
31.8 (8)
56.4 (7)
3.75
IPO
1146 (4)
(6.1(4)
50.1 (4)
51.3 (4)
78) (1)
9.2 (1)
39)
59.8 (6)
3,50
KTO
1139 (6)
34.7 (5)
45.4 (6)
45.8 (6)
38)
63.7
3,75
СРO
1136 (7)
34.1
44.6 (7)
44.9
(2)
(2)
38.7 (3)
63.5
4.50
SimPO
1149
37.2 (3)
52.2 (3)
52.5 (2)
56.6 (8)
6.0 (8)
32)6
55.0 (8)
5.25
BNF
(1)
37.5 (2)
54.8 (2) 52.1 (3)
77.0
9.3
35.4
61.4 (4)
3.13
Fiirral7B-Instuutt--o
Method
Wild-Bench
Arena-Hard
GSMSK
MATH
CRUX
MMLU
Average
Elo
Score
I.C. (%)
WR(S)
Acc (%)
Acc (%)
Ace (%)
Acc (%)
Rank
Base
1098 (8)
25.6
19.2 (8)
18.9 (8)
42.7 (2)
3.9 (1)
25.1 (3)
53.0 (3)
5.13
SLiC-HF
1130 (3)
31.2 (2)
28.9 (4)
(3)
(1)6
2.6
24.3 (4)
52.1 (6)
3.63
DPO
1127 (4)
29.7 (6)
29.8 (3)
30.4 (4)
(2)
3.1 (3)
22.3
53.2 (2)
4.25
IPO
1121 (7)
26)9
25.4 (7)
26.1 (7)
37)
3.6
(1)
50.7 (8)
5.75
KTO
1126
30.2
27.8
27.7
(2)6
(.9
23.3 (6)
52.8 (4)
4.75
CPO
1124
(9.9
28.1
28)
(4)
(4)
25.3 (2)
(7)
4.88
SimPO
1133 (1)
(1)
(1)
36)6
33.7
2.2 (8)
21.4
52.2 (5)
4.13
BNF
1131 (2) 31.1 (3) 35.2
34.5 (2) 39.7
2.6 (6)
24.3
54.8 (1)
3.25
Gemma-2-9b-It
Method
Wilc-Bench
Arena-Han Hard
GSM8K
MATH
CRUX
MMLU
Average
Elo
Score
L.C (%)
WR(%)
Acc (%)
Ace (%)
Acc (%)
Acc (%)
Rank
Base
1160 (4) 42.7 (4)
$4.9 (4)
54.5 (4)
87.9 (3)
19.9 (3)
44.6
72.7 (2)
3.25
DPO
1181 (2)
53.2
77.3 (2)
81.3 (1)
88.5 (1)
20.2 (2)
44.3 (3)
72.8 (1)
1.75
SimPO
1181 (2)
53.3 (1)
72.6 (3)
75.4
87.7 (4)
18.2 (4)
41)4
72.6 (4)
3.13
BNF
1186 (1)
53.2(2)
77.5 (1)
80.8 (2)
88.0 (2)
21.8 (1)
41)
72.7 (2)
1.50
EXPERIMENTAL RESULTS
Table 4 presents the detailed experimental results of all preference optimization methods across six benchmarks and three base models, covering both performance and relative rankings. In this sec- tion, we first provide a comprehensive analysis of these statistics. Then, conduct experiment to evaluate the performance of our proposed BNF on non- pairwise preference datasets. Finally, we analyze the log-likeling likelihooo and logit shifts under different preference optimization methods. In addition, we provide some response comparisons in Appendix G for reference.
5.1 MAIN EXPERIMENT ANALYSIS
Preference optimization is essential for QA. From Table 4 we observe that all three base models rank the lowest on two QA benchmarks, Wild-Bench and Arena-Hard, while all preference opti- mization methods achieve significant performance improvements. More specifically, DPO, SimPO, and our proposed BNF demonstrate superior performance compared to other methods, consistently securing the top three positions across all QA metrics. DPO performs best on Meta-Llama-3, SimPO works best on Mistral-7B, and our proposed BNF exhibits outstanding performance on Gemma-2. These experimental results indicate that preference optimization is essential for improving model performance on QA tasks and demonstrate that our proposed BNF can deliver performance compa- rable to, even superior to, the strongest preference optimization baselines.
7

---

Published as a conference paper at ICLR 2025
BNF pays the lowest alignment tax. When attention shifts to reasoning benchmarks, find that the methods excelling in QA benchmarks often perform poorly in reasoning benchmarks, which is also referred to as alignment tax (Quyang et al. (022). For instance, SimPO, which performs well on QA, experiences a significant drop in performance on reasoning, with a more than 10% performance decrease on GSM8K, ranking nearly at the bottom. In contrast, KTT, which excels in reasoning benchmarks, performs poorly on QA benchmarks, only slightly better than base models. Some studies (Lin al. 2023) suggest that alignment tax occurs due over-fitting preference data, leading to decline in reasoning ability. With the bidirectional negative feedback, our proposed BNF can automatically constrain the model from over-fitting to preference datasets, thus preserving the reasoning ability. Experimental results show that BNF achieves the best average relative ranking across three base models (last column of Table 4. indicating that BNF strikes a better balance between QA and reasoning ability, thus paying the lowest alignment tax.
Response length of different methods. Some recent stud- ies (Xu et al. 2024b) argue that the models trained with DPO tend to produce verbose responses that may degrade the quality of results. To investigate this issue, calculate the average response length of all preference optimization methods on two QA benchmarks and list their statistics in Table 5] On Meta-LJama-3,, the average response lengths
Table 5: Average output length.
Meta-Llama-3
Mistral-7B
Method
WB
AH
WB
AH
Base
2975
595
2832
507
SLIC-HI
2318 L.
469
3198 f
576
DPO
2665 1
602
2655
509
of almost all methods decrease. However, on Mistral-7B, only KTO's response length decrease on both two bench- marks, while all other methods see an increase on least one benchmark. Despite the introduction of length normal- ization, the response length of SimPO on Mistral-7B still
IPO
2843 4.
547
KTO
2776 +
543
2813
504
ORPO
2551 4.
523
2913 f
524
SimPO
2533 4
527
3214 f
521
BNF
2521 4
496 4
2827 4
526
increase significantly. These statistics indicate that average response lengths are closely related to the base models. DPO does not necessarily generate longer responses, and specially designed method may also fail on certain models.
5.2 NON-PAIRWISE OPTIMIZATION
With the bidirectional negative feedback, BNF no longer requires pairwise contrastive losses to con- strain the excessive decrease the likelihood of dispreferred samples. Therefore, BNF fundamen- tally eliminates the need for preference pairs during optimization. To evaluate its performance with non-pairwiss preference datasets, we randomly mask either the preferred or dispreferred response from the original preference pairs with a certain probability. Table 6 presents the experimental re- sults with different pairing ratios. On QA benchmarks, it is evident that more pairwise preference data leads to better performance. However, even without any preference pairs, BNF still achieves significant performance improvements over the base model, with an average score increase of 6.8 on Wild- Bench and a 12.3% win rate improvement phenomenon is also observed in this scenario. At a 0% pairing ratio, the absence of preference pairs prevents over-fitting to preference data. While this leads to lower performance gains in QA, the model's 's reasoning ability improves on mathematical datasets, showing a 1.6% increase on GSM8K and 2.6% improvement on Math-I-5.
Arena-Hard. Interestingly, the alignment tax
Table 6: Experimental results with different pairing ratios. A ratio of 50% indicates that in half of the preference pairs, one response is randomly masked. A ratio of 0% means that no pairwise data is included in the dataset, while 100% represents the original preference dataset.
Hitt-LLlamm--BB-Instttee
Pairing-Ratio
Wild-Bench
Arena-Hard
GSMSK
MATH
CRUX
MMLU
Average
Elo
Score
LCC (%)
WR(%)
Ace (%)
Acc (%)
Acc
Ace (%)
Rank
Base
1131 (5)
29.2 (5)
33.9 (5)
36.1 (5)
78.5 (2)
6.8 (5)
38.1 (1)
62.4 (1)
3.63
0%
1146 (4)
36.0 (4)
49.3 (4)
48.4 (4)
80.1 (1)
9.4 (2)
35.6
62.2 (2)
3.13
25%
1148 (2)
36.2 (3)
51.5 (3)
49.7 (3)
77.9 (3)
9.3 (3)
36.7 (3)
61.8
2.88
50%
1148 (2) 36.7 (2)
53.5 (2)
51.4
76.4 (5)
10.0
37.9 (2)
61)
2.63
100 %
1153 (1)
37.5 (1)
54.8 (1)
52.1 (1)
77.0 (4)
9.3 (3)
35.4
61.4 (4)
2.5
8

---

Published as a conference paper at ICLR 2025
5.3 DISTRIBUTIONS OF LOG-LIKELIHOOD AND LOGIT
To gain a deeper understanding of the optimization behavior of BNE, we analyze log-likelihood and logit shifts between policy and reference models using 1, 000 questions from Wild-Bench (Lin al., 2024). We apply greedy decoding to generate responses using BNF and three baselines: DPO, IPO, and SimPO, The base model is 3-8------tn (a)
(b)
100
30
AoggP - ne (-0)
20gg 10gt-220 E- - [e]
Resp 11)
(d)
Deogy
degF X1
Zlogh: M
GWN Corffoenttrr Log
Figure 3: Comparisons between BNF, DPO, IPO, and SimPO. (a) Log-likelihood shifts. (b) Ab- solute logit shifts. (c) Logit shifts vs. log- log-likelihood shifts. (d) sength-normalizee log- likeiiil shifts. (e) Length-normalizer absolute logit shifts. (f) Gini coefficients for logits.
BNF exhibits minimal log-likelihood shift. Our experiments reveal that BNF exhibits the least log-likelihood shift (Figure 3t) which may help preserve reasoning and comprehension capa- bilities from the reference model. In contrast, SimPO shows the largest shift, potentially explaining its inferior performance in these areas.
BNF's uniform logit increase leads to unique shift pattern. Interestingly, BNF shows larger logit shifts compared to DPO and IPO (Figure 3 and (3)), which also require a reference model. While larger logit shifts usually correlate with larger log-likelihood shifts, BNF presents a unique pattern: many samples have significant logit shifts with minimal log-likelihood shifts (Figure B top-left). This occurs may because BNF increases the output logits uniformly across preferred tokens at each position, resulting in consistent likelihood after softmax normalization.
BNF distributes shifts evenly across tokens. The Gini coefficients Gini 1912) for logit shifts (Figure 3) show that BNF's logit shifts are more evenly distributed across tokens compared to other baselines. A lower Gini coefficient suggests that the shifts are distributed across many tokens, rather than being concentrated in few with substantial differences, a scenario that could lead to over- fitting. This suggests that BNF achieves balanced optimization strategy by reducing the gradients for tokens that already exhibit large differences from the reference. This may explain why BNF effective preventing over-fitting and reducing the alignment tax.
BNF exhibits fewer polarized shifts compared to DPO. We use DPO as a reference model to analysis the shifts of token-level log-likelihood (Figure 4 Specifically, we divide DPO's token-level log-likelihode bassed bins and map the token-level shifts from
bdl dpo 20
Percentage of Tokenss
shifts into 10
15
10
BNF, SimPO (Meng et al. (024), and IPO (Azar et al.,
2023 onto these DPO-defined bins. Figure 4 reveals that
BNF exhibits a more centralized distribution of token- level shifts compared DPO, likely due the moderating effect of its bidirectional negative feedback design, which limits extreme log-likeli likelihood shifts. In contrast, SimPO shows a more binarized distribution, while IPO's distribu- tion remains close to that of DPO.
18.48
00.50 29
mm
4m
r't. am
6e sm
(se. ipm
40.00
Percentlle Bins
Figure 4: Distribution of token-level log-likelihood shifts.

---

Published as a conference paper at ICLR 2025
6
RELATED WORKS
6.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
Reinforcement Learning from Human Feedback (RLHF) (Oyyagg et al. 2022) has become a widely adopted approach to align Large Language Models (LLMs) with human preferences. The typical RLHF pipeline consists of three stages: Supervised Fine-tuning (SFT), reward model training, and policy optimization. Proximal Policy Optimization (PPO) (Schulman al. a 2017) is commonly used in the policy optimization stage to align the model with human feedback. RLHF has been applied across various domains, including mitigating toxicity (Chaudhari et 2024). enhancing reasoning abilities (Wang et al., 2024b), and improving the helpfulness of language models (Bai et al. (2022a) However, RLHF often requires complex hyper-paramenn
tuning and can be unstable, primarily due to the sensitivity of the reward model (Casper et al. (023). In contrast, Direct Preference Optimiza- tion (DPO) (Rafailov et al.) 2023) simplifies this process by removing the need for an explicit reward model, directly optimizing for preferences. This makes DPO a more resource-efficient alternative to RLHF. While DPO addresses some of RLHF's complexities, further improvements and variants have been proposed to overcome specific challenges in preference optimization.
6.2 CHALLENGES OF DIRECT PREFERENCE OPTIMIZATION
Although DPO-series methods have demonstrated impressive performance on QA and Chatbot tasks (Meng et al. 2024). they remain highly sensitive to hyper-parameterrs and often exhibit instability (uu et al. 2024b). This instability is especially evident when applied to mathematical datasets,
potentially leading to training collapse (Pal et al. (2024). Recent studies (Zhao et al.] 2023; Xu
et al., 2024a) propose using Negative Log Likelihood (NLL) regularization to stabilize training. While these approaches successfully prevent collapse on mathematical datasets, they perform poorly on several popular Chat and QA benchmarks (Meng al., (024) and introduce additional hyper- parameters. Another significant challenge in preference optimization is controlling the output length, as models trained with DPO tend produce verbose responses that may reduce the quality of results (Xu et al. 2024b).. To address this, several DPO variants have been developed. For instance, SimPO [Meng et al.) (024)) applies a length-normalle reward to prevent the generation of excessively long outputs. RDPO (Park et al., 2024) adds regularization term to reduce length exploitation.
7 CONCLUSION, LIMITATIONS AND FUTURE WORK
Conclusion. In this paper, we propose a novel LLM alignment loss with Bidirectional Negative Feedback (BNF), addressing the instability and hyper-parameter sensitivity found in DPO and its variants. Unlike DPO-series methods, BNF eliminates the need for pairwise contrastive losses and preference data, streamlining the alignment pipeline to be simple supervised fine-tuning. Our experiments across six benchmarks demonstrate that BNF achieves strong performance on QA benchmarks, while preserving the reasoning ability of LLMs and paying the lowest alignment tax.
Limitations and Future work. Despite our efforts to improve this work, due to constraints in computational resources and budgets, there are still the following limitations:
Model Scale. All the experiments in this paper are based on LLMs of 7B-9B scales, and we are unsure whether our proposed method can be stably scaled to larger LLMs of 30B more.
Combination with DPO-series methods. Our proposed BNF loss is not necessarily opposed to DPO-series methods; in fact, the two approaches be complementary. and introducing pairwise contrastive function may further improve the performance and stability of BNF.
Non-pairwise Dataset. The non-pairwise dataset used in this paper is derived from regular preference dataset with random masking, which is merely a simulation and may not fully reflect real-world applications.
These limitations will also serve as the starting point for our future work. We plan to request more computational resources and funding in the future to further improve this work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This prsjjcch//roojct is supported by the National Research Foundation, Singapore under its Al Sin- gapore Programme (AISG Award No: AISG2-TC-2022-0055) We also wish extend their heartfelt gratitude to the Sea Al Lab for their generous support in providing the necessary equipment and computational resources critical for the successful completion of this research.
REFERENCES
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406. //704, 2024.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprini arXiv:2310. /2036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022b.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 2023.
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for 1lms. arXiv preprint 2:Xi::2404.08555, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint arXiv:2403.04112, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 20 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. Forty-first International Conference on Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.0m 04475, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 2024.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are done with mmlu? arXiv preprint 27,, 2024.

---

Published as a conference paper at ICLR 2025
Corrado Gini. Variabilità mutabilità. Reprinted in Memorie di metodologica statistica (Ed. Pizetti E, Salvemini, T). Rome: Libreria Eredi Virgilio Veschi (1955), 1912.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.0306 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint
2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.256// 2023b.
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu. Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Bill Yuchen Lin. ZeroEval: A Unified Framework for Evaluating Language Models, July 2024.
URL http:tts::/// //tttp::/gittee http://hha.....//oco/o/oeooaee/e www.c//uccceennnn//eeeeeeee
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking ilms with challenging tasks from real users in the wild. arXiv preprini arXiv:2406.04777 2024.
Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxi- ang Wang, Han Zhao, Yuan Yao, al. Mitigating the alignment of rlhf. arXiv preprint arXiv:2309.06256, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14733 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.1322 13228, 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 2305, 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Amir Sacidi, Shivanshu Verma, and Chitta Baral. Insights into alignment: Evaluating dpo and its variants across multiple tasks. arXiv preprint arXiv:2404.14773, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
12

---

Published as a conference paper at ICLR 2025
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint 2:X1v:2408.00118, 2024.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406. 12845, 2024a.
Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426 9439, 2024b.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of I1m performance in machine translation. arXiv preprint urXiv:2401.08411 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024b.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint MXXX::3335.102255 2023.
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.088 05685, 2023.
13

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL DERIVATIONS
A.1 DERIVING THE PARTIAL DERIVATIVE OF NLL LOSS
Given policy model Ne, the NLL loss described follows:
e
CNLL = (Baaa)))l- log [)([=))] = (a,y))
log
Σ
8
k=
Therefore,
@LNLL
c²v
log
0zy
Σ
M
E
k=
8 Jzy
-zy + log
E:
k=1
exx
V Σ
k= e
To (v(
A.2
THE PROOF OF SENF BEING VALID PROBABILITY DISTRIBUTION.
Given the proposed Dynamic Target Distribution function fuNE:
sg
min
Ste (v. (2,950)) Far( (.M)es)
1)
if
(aNF(Ut,t,) =
1- fuNe (1 To no (t, Mes) if y # t2
sg
Firstly, since To(y xr Vei) > and TTr 1r, Ves) 0, it is obvious that fuNe (1s, Ue) min ro(pslages) Tarl (ewee) 1) > 0 when yi ty. Furthermore, since 1 - faNF( Vi). 1 mo (vilar, Hec)) and (e x, yee)) are greater than O, fBNF(Ui,ts) > 0 also holds when yi # tj. Therefore, fuxu (y,, (2) 0 for all yi and ty.
Secondly, the sum of fBNF(u)) equals:
1V
|V] 1 fuNP vi)
TO To xr, Me-
JUNF(Ut: tz) min Σ
To (t
(yer xx, Mes)
F To(y Ir, yes)
M
To (Va xr, Uci))
fENE vk)
min
11
+
Σ
nt(tylo,
Ter(y 22, M<())
1 ( Ir, y
To (v: (r., Ue.))
JuNF(Ui: MK) (1 no To lr, yes))
min
Terr(ys (2. Mes)
To( Ir, y<())
The (vo U<i)
(1/1 2. F-())
min
min
Tee Ir, yes)
Therefore, (aNF((L) is valid probability distribution.
14

---

Published as a conference paper at ICLR 2025
A.3 DERIVING THE PARTIAL DERIVATIVE OF BNF LOSS
The optimization objective of BNF is:
lyb lol [
CNNF -B(z,))
Σ
Σ
[UNF(UL, t2) log To (T(tt) vzi))
M
The likelihood no nT(tz lr, Vcs)) is obtained by softmax:
To(ts
Σ
IV|
The partial derivative
with respect to any output logit
can be derived as follows: J.yes
M
0LNNE Oz12 2,034
label(y)
Σ
/NNF(YE 15) log To (l (a, Mess)
azta
label(y)
y))
θ
Σ
fENF (s) log No (ty vei)
1 22ta
Since we have adopted the stop gradient operation to the function SANF, thus:
ØLENF
label(y)
Σ
/BN (yi, 12)
2 log To (x, yes)
lyl
O27. 2.2
MV]
label(y)
SUNF(UI; 13) ④ No 10
ly
To (z, Ves) O2
The partial derivative of the cross-entropy operation with respect to logits is given by:
@To(Us
c=v:
)
[o(u)l))(1 To(U)(=)) y/s y/s
Ozye
azye
if yv + ys
Therefore:
dLNNE R.Ner
label(y)
feNr(ys,t,) t2) 2 (2. To (t x. Meci)
0zt.
0z
label(y)
TBNF(UH tx) 2 Te(txlz,y<d) no (Ia Hei)
faNe yis t2) 2 (t, vec)) To (r, Mess)
lvv
Oz
2z n.ye
M
label(y)
fuNr(y,,ta)) ( vei))
funr (T) Mei)
[y]
15

---

Published as a conference paper at ICLR 2025
Because (1N is valid distribution, E, fINE(ULL tz) thus:
M
ŌLANF
label(y)
(nNr((ss44) 44) To(uu ve:))
Σ J#k
(ys (7) yee))
azin
label(y)
[unNr(Us, tt) -ooo-Toottee (ottt (2, yee)) - (1 (e) yee))
|8l
label(y)
(o(ta (ectell,.ee yee) (yen ts))
y
A.4
DERIVING THE FUNCTION BETWEEN PARTIAL DERIVATIVE AND LIKELIHOOD
Since have:
No (ww m
1)]
sg min
yf = by
fBNF(Bi,tt)
sg
(oNE( no Cun (u.ve))) The (t,, ver))
if yf #t,
and,
0CNN
label(y) l
(T (1ok,,,,,,)))) fANF(yi, 4x]]
Ozta 2.
For the token fx yr within the response y. its 0=
Cane
can be obtained follows:
ÖLANF Dzy.ver
(yb)
[T Fee) - fENF(UI, vil]
label(y)
To (r, M<<s)
To (oss (x, Ve.) min
|vl
Teer yo (2, V<ce)
7o(yllz, vee)
mo (r.,ess) T.rlui
if (o < ves))
label(y)
|v]
[T (.))) 11,
T Re if Te
To (36 26, Meii)
L= martan (nneee))
if (o(n),, Fee (r, Mci)
label(y)
|v]
To (o lx, Ve:):
if mo(ol... 2 Tee ye,))
For the token l # y::
ØLENF 0= 2.923
label(y)
To(lu (x, Hes)
1 faNT (yis Wi)
Ta (Ie (1, vesi)
1 no (yel lr, M<c))
label(y)
(ys yk) 1 mo |r, Mei))
Σ Te(txlle,,ee)
Σ
To(ty 1e, ves))
|yl
(1 Th (ex)) - (1 (UN vi))
2LHNN
= (UN wi)
Te (yel we:)) =
Izg.. 2.
Therefore, the sum of El
BLaNF 0=12
is actually equal to
8CanE 0=
which also establishs a bidi-
rectional negative feedback.
16

---

Published as a conference paper at ICLR 2025
B CODE IMPLEMENTATION
def BNE_loss (batch):
.*.
Computes BNF loss for preference optimization.
Args:
batch: A tuple of (input_ids, lengths, labels)
inppt__dds input token ids (batch_size, seq_len)
Lengths: response lengths (batch_size,)
labels: Binary labels for preference (batch_size,)
Returns:
loss: The computed loss value.
u**
# Unpack batch elements input_ids, lengths, labels batch
Compute lcg-softmax for policy and reference models
# policy_logp & ref_logp: (batch_size, seq_len, vocab_size)
policy_logp policy_model (input_ids) .ttp_iiddd__ooooi.... wwg.loogtt....o................t loa.tsssog_sooftma (-1)
ref logp ref (inpuutinnnnnndniaeee _ds) wtw.gggtss.oo_o.foooloo (-1)
Get log probabilities for the actual response tokens response_logp has shape (batch_size, seq_len)
response_logp torch percch.ether((pollcc__ogog index=input wtw.innputiisuuueeeeeseeee inder=inputtiids_nnsseeeee response_logp_ret
dim=-1,\
torch gather (ref_logp, dim=-1, index=input_ids wnsqueeee (-1)) squeeeeeeeeeeeeeee______ (-1)
# Sum log probabilities for ren-response tokens other_logp has shape (batch_size, seq_len) other_logp (policy_logp (xp detach () policy_logp) sum (-1) \
response_ exp detach * response_logr
# Compute the dynamic target distribution for token in response reapenses_target forch.clane letesppone__ogppexppe max=1) detach
response_logp_ref.eet
I Compute the dynamic target distribution for other token others_target (1 responses target) (arronntss_trggt)
pesponse_logp.eee () detach())
# Compute final loss and apply length normalization
loss responses_target response_logp others_target
other_logg
loss = (loss. (-1) labels / lengths). sum
return loss
17

---

Published as a conference paper at ICLR 2025
C MORE DISCUSSIONS ABOUT BNF LOSS
In this section, we provide a more detailed discussion and elaborate on the design rationale of the BNF loss. The optimization objective of our proposed BNF loss is described as follows:
lol
M
(v) 1
LANF (2(
Σ
[UN (5) log wei))
(10)
where AENF is the proposed Dynamic Target Distribution (DTD) function:
molw (e.(w)).))))) Furlul (.N)c) 1)]
sg
min
if di = tj
TENF(Ui,
(11)
1- (1ir()) ma(i (n..)))))))
To (o (x, (x, if y: +
sg
It is evident that the core of the BNF loss function lies in the DTD function. Observing the first term of SANE, when y/ - Lj. the SANF (M; 42) linearly decreases as To(y; (x, Mes) decreases, eventually reaching zero. Generally, if To (ya (r, V-cs) much smaller than Tre (3x 22, Ver), it indicates that this sample outdated. Thus, the first term the DTD function actually serves to dynamically reduce the weight of the outdated samples.
8g min (.olv.2..))))) Feet m.ees) 1)]
if y: L
However, simply using fiemt (Vi, 12)
as the DTD
if ys * Lj function would result in the target distribution longer being valid probability distribution, thereby affecting its bidirectional negative feedback properties as follows:
[
DNNF 0=12
label(y) |8|
f furma tz) log (tyl Mei))
label(y) y))
Test(yi (x, ves) 1)
To Toluila yeee)
1B/
Date
J.Vis
sg
min
log l To(Uvlo)
label(y) yv)
)) Tiet (M lx, )=r)"
no (ye ar, Mc:))
(1 To(y (1- (x))))
min
ly]
(u pr, M=())
1- no (v. (2.8ee) aad (1n (eMe)))
if mo Teer Vei)
[abel(y)]
|v]
To T (,s
if
(ea Wes)
Although the above equation appears very similar to Equation 9 at first glance, there is ac- tually a subtle difference between them. The first term in the above equation is a quadratic function (usl Mei) =o(wlla)e)) of ma. whereas in Equation (9). is a linear function (..
( jz. Mes) 1--mer(pu (n.)ed)) This subtle difference results in fiemt losing the bidirectional nega- tive feedback property (as shown in following Figure), going against motivation. Therefore, the second term in the SENE is necessary design.
ac
The left figure illustrates the impact of removing the second term of the BNF loss. When using only the first term as the target distribution, the likelihood-derriativee curve transforms into a parabola when Te (yr x yei) < Ter((s (r. Meii,, peaking at stead of the reference point πо = Ree- This undermines the bidirectional neg- ative feedback property and contradicts original motivation.
dzy
frorm
four
0.25
Ifrer
0.5 in-
0.5
Brof
18

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL RESULTS ON MATHEMATICAL DATASET
In Section 2.3 we propose conjecture of why DPO fails on mathematical datasets: The fixed scaling nyper-parameetr struggles adapt to varying data distributions. Since our proposed BNF does not rely on contrastive loss, it can avoid this issue.
Specifically, first fine-tune Mistral- Inst and Llama-3-Inst thetic dataset MetaMath (Yu et al. 2023) obtaining Mistral-MM and Llama-3-MM, Next, we randomly select 12.8K prompts from MetaMath and generate 16 responses for each prompt using Mistral-MM and Llama-3-MM. Up to correct answers and 4 incorrect answers are selected to con- struct the preference dataset. Finally, we optimize Mistral-MM and Llama-3-MM using the above preference dataset and three different optimization algorithms, including DPO, CPO and our BNE.
widely used mathematical syn-
During the SFT stage, the batch size is set to 512, and the learning rate for both Mistral-Inst and Llama-3-Inst is 5e-5. During the preference optimization stage, the batch size is set 128, with learning rates of 5e-7 and 6e-7 for Mistral-MM and Llama-3-MM, respectively. The reason we do not use the full MetaMath for preference optimization is the limitation of computational resources.
0.9
0.8
0.8
Accursey
0.6
Accuracy
0.8
0.5
0.3
0.3
DPO
CPO
CPO
0.
BNF
BNF
Mistral
0.1
LLama3
400
600
800
1000
200
400
600
800
1000
Training Steps
Traning Steps
Figure 5: Experimental results of Mistral and Llama-3 on GSM8K. We save the model every 200 training steps and evaluate them the testset of GSM8K.
Figure 5 illustrates the performance of various preference optimization algorithms the purely mathematical datasets. As discussed in the introduction, a significant collapse is evident during the optimization process of DPO. After slight improvement in the initial stage, its performance begins to decline sharply. CPO addresses this issue by introducing an NLL regularization term, albeit at the cost of additional hyper-panne saper-parameters. In contrast, our proposed BNF mitigates this issue without introducing any additional
E HYPER-PARAMETER EXPERIMENTS
Table 7: Experimental results on Meta-Llama-3 with different learning rates.
Meta-Llama-33BB-nnstruue
Learning rate
Wild-Bench
Arena-Hard
GSM8K
MATH
CRUX
MMLU
Average
Fle
Score
LC (%)
WR(%)
Ace (%)
Acc (%)
Acc
Acc (%)
Rank
Base
1131 (4)
29.2 (4)
33.9 (4)
36.1 (4)
78.5 (1)
6.8 (4)
38.1 (1)
62.4 (1)
4e-7
1143
35.9 (3)
51.6
49.3 (3)
78.4
9.5 (1)
38.0
62.2 (2)
2.38
Se-7
1149 (2)
36.8 (2)
52.3 (2)
50.7 (2)
77.6 (3)
9.3 (2)
37.4 (3)
61.8 (3)
2,38
6c-7
1153 (1)
37.5 (1)
$4.8 (1)
52.1 (1)
77.0 (4)
9.3
35.4 (4)
61.8
2.13
To evaluate the stability of our proposed BNF under varying learning rates, we conduct experiments by training Llama-3-Inst with range of learning rates. The results indicate that BNF exhibits sig- nificant stability across different learning rates, with only minimal impact on performance. Smaller learning rates tend to better preserve reasoning capabilities, whereas larger learning rates improve the model's performance on QA benchmarks, highlighting an inherent alignment trade-off.
MettaMath contains 395K mathematical prompts and their corresponding answers.
https:/
huggi muggingface.co http://eeeeeeeeatattteeeeeeeeeee http:///....tttm..eaeaeeeeeeeean
19

---

Published as a conference paper at ICLR 2025
F TRAINING AND EVALUATION COSTS
Table The training and evaluation costs of this paper. We use 8x A100-80GB- for training, and the precision is 6f
Llama-3-8B-Ing
Mistral-7B-Innt-v0..22
Gemma-2-9B-it
Training GPU hours (per training)
30
30
45
Training GPU hours (total)
600
480
700
Wild-Bench API Costs (S/time)
5
Wild-Bench API Costs (S/total)
150
150
Arena-Hard API Costs (S/time)
0.2
0.2
0.2
Arena-Hard API Costs (S/total)
5
5
20

---

Published as a conference paper at ICLR 2025
G
SOME CASE STUDIES
Table 9: Comparing generations of Wild-Benchmark prompts from Gemma-2-9B-it.
Prompt
If 7B parameter Transforner LLM at fpl6 with batch size I and Soquence length is 500 tokens and bytes per token is 2 needs 14GB VRAM, what would the VRAM requirement he batch size 50? This extremely important! Show your work. Let's work this out in step by step way to be sure we have the right answer.
Gemma-2- 9B-Base
Here's how to calculate the VRAM requirement with a batch size of 50: ##1. Calculate the total model parameters in bytes: 7B parameters 4 bytes/puramntter (fp16) = 28 GB #*2, Calculate the total inpul sequence size in bytes:* e 500 tokens bytes/token = 1000 bytes *03. Calculate the memory required per batch: With batch size of 1: 28 GB (model) 1000 bytes (input) 28.001 GB With batch size of 50: 28 GB (model) (1000 bytes/token 50 tokens) 28 GB 50000 bytes = 28 GB 0.047 GB 28 047 GB *herrefore, the VRAM requirement would be approximately 28.047 GB with batch size of 50.*, *mmmorrtant Considerations:
*'verheads: This calculation doesn't include memory overhcads for things like optimizer states. gradients. and other tensors used during training. These overhcads can add significant amount of memory usage. * ** "ardware:*** The actual VRAM requirement may vary depending on the specific hardware and its memory management capahilities. 8 +PFP16 Precision:** Using íp16 precision can reduce memory usage compared to fp32, but it can also lead to some loss of accuracy. Let me know if you have any other questions!
Gemma-2- 9B-BNF
Here's the breakdown of how to calculate the VRAM requirement with a batch size 50: **1, Model Parameters: we i + PPrraeeeerg: ** 7B (7 billion) +1P Precision:* + Each parameter uses 16 bits (half the memory of FP32). "Paaametter Memory:** 7B * (16 bits / 8 bits/byte) = 1.75 GB =*2. Sequence Length and Bytes per Token:** Souuuene Length=** 500 tokens "Byyts per Token: 2 **Total Sequence Memory per Sample:** 500 tokens 2 bytcs/token 1000 bytes = MB **3, VRAM Consumption per Sample (Batch Size 1):*+ **Madel Psrameters:** =* 1.75 CI x **Squeence Memory:** MB ** Total
per Sample:** 1.75 GB 0.001 GB 1.751 GB (approximately) #*4, Scaling for Batch Size 50:4* +*VA per Batch:** + (VRAM per Sample) + (Batch Size) e =*VRAM Requirement (Batch Size 50):** 1.751 GB + 50 87,555 GB (approximately))* j **Therefore, with a batch size of 50, the estimated VRAM requirement would approximueely 87.55 GB. :*
Gemma-2- 9B-DPO
Here's the breakdown of how to calculate the VRAM requirement with batch size of 50: **] Parameters and Memory Footprint:** + *MMoeel Size:** 7B parameters * 4 bytes/parameter (fp16) 28 GB (This is the "thcorctical" memory the model itself occupies) * **However. not all parameters are actively used at once.** During inference or training, the model processes data in batches, so the actual VRAM usage is influenced by batch size and sequence length. ##2. Batch Impact:** #*Original Scenario (Batch Size 1):** sequence 500 tokens bytes/token 1000 bytes per sequence. Since we're using fp16, this translates roughly 2000 bytes per sequence in memory. The 14GB VRAM likely accounts for: + Model parameters (partially) Activations, gradients, and other tensors related to *one* sequence. **New Scenario (Batch Size 50):** Now we have 50 sequences processed simultaneously. The memory requirement per batch becomes: 50 sequences 1000 bytes/sequence = 50,000 bytes. Again, accounting for fp16 precision, this roughly 100,000 bytes per batch. **3. Estimating Total VRAM Usage (Batch 50):** ** Model Parameters:** ** While the full 28GB model size isn't *dircetly" used once, significant portion is still required. Let's ss assume around 15GB for the model itself (this can vary based on optimization techniques). B*Bch Data and Activations: 100,000 bytes per batch 500 lokens/sequence udditional memory overhead for activations, gradients, etc. This a rough estimate, but let's assume an additional 5GB per batch. With batch size of 50, this becomes ...0 more words)
Gemma-2- 9B-SimPO
Here's breukdown of how calculate the VRAM requirement with batch size of 50: 111. Parameters and Memory *ootprint:** The 7B parameter model's memory usage primarily stems from storing its weights and activations. While exact precision varies. assume roughly **g bytes per parameter** for fp16 representation (including some overhead). Memory for 7B parameters: 7,000,000,000 8 bytes = 56
GB ##2. Sequence I Length and Token Memory:** * With 500 tokens and bytes per token, each sequence consurmes 500 + 1000 bytes (1 MB). #x3. Batch Size Impact;** +* a #*Oriiinal Scenario (Batch Size 1):+* 14 GB VRAM used included memory for parameters, one sequence (1 MB), and activations/nneemeddaaa calculations. **Scaling to Batch Size 50:** The primary memory increase comes from handling 50 se- quences simultaneuusly. The loken memury consumption scales linearly: sequences # 1 MB/sequence 50 MB. *44, Estimating Total VRAM with Batch 50:++ a +*Parmmeter Memory (unchunged):** Still 56 GB (dominant factor) "*Sequence Memory (increased):** 50 MB- overhead from previous activa- tions ** AAtivattin Memory (crucial cstimate):** This is harder to pinpoint "directly* but "signfficantly" increases with batch size due to more parallel computations. Assume a *conservative* increase factor of **3x** over the original scenario (based on how activations grow with parallel processing). This adds other 42 GB (3 # 14 GB). a s= sougg Total VRAM (Batch 50):+ 56 GB 50 MB + 42 GB ** 98 GB ** IIportant Considerations and Refinement:* ++ Activattion Memory Variability .1.0 more w
21

---

Published as a conference paper at ICLR 2025
Table 10: Comparing generations of Fild-Bennchmak prompts from Gemma-2-9B-il
Prompt
Show me how to make IS using 19 coins
Gemma-2-
This is a fun challenge! Here's 's how you can make $1 using 19 coins: #*1 Half Dollar (50 cents)es
9B-Base
#+1 Quarter (25 cents) s +*1 Dime (10 cents)+* 4+1 Nickel (5 cents)+s 4-1 Penny (1 cenl) s +* 14
Pennies (14 cents)** Let me know if you' d like to try another coin puzzle!
Gemma-2- 9B-BNF
You can't make exactly $1.00 using only 19 coins from standard US currency. Here's why: * **Coiin Denominations:** US coins are: Penny (1) + Nickel (5) * Dime (10) + -uurterr (25) * Half-Dollar (50) * ""Combinations:*** ** Even with clever combinations of these coins. you won't reach precise $1.00 using only 19 coins. You'll II likely end up either slightly over or under a dollar. **Eamplcc (Closest You Can Get) You could get *very* close to $1.00 using 19 coins, but not exactly: Pennies (17)* Quarter (25) * Dime (10) This totals 52, leaving you short $1.00.
Gemma-2- 9R-DPO
Here's how you can represent $1 using 19 coins, keeping in mind standard currency:* ##1 Half Dollar (50 cents):** coin* #*3 Quarters (75 cents):* 3 coins sx1 Dime (10 cents)*** coin #*4 Nick- els (20 cents):** 4 coins s=10 Penmies (10 cents):** 10 coins Total;** 1+ 3 1 + 4 10 = ##19 Doins**seValuer 50 75 + 10 + 20 + 10 = #*165 cents** which equals **$1 While you can't *ex- actly" make with "only" 19 coins, this combination gets you closest while using the specified number.
Gemma-2- 9B-SimPO
You can't directly make SI using "only" 19 coins of standard US denominations to reach an "exact" dollar amount. Here's why:* **Coin Values#* US coins are pennies (I), nickels (5), dimes quarters (25), and half-dollars (50). No combination of these will add up precisely to $1 using only 19 coins, coins,** Howeeer, you could create a scenario with slight workaround involving implicd value=**1, 1. **The Trick.** Imaginc fictional coin worth 1/19th a dollar (approximately 5.26 cents). With 19 of these coins, you 'd theoretically reach $1. This isn't realistic currency. but it fulfills the iising coins to make Suondition conceptually.
Prompt
How many 400 watt solar panels would it take in Michigan to produce the same amount of power in day a EV car which drives 40 miles a day? Assume peak sun hours per day in Michigan on average and that the car 0.3 &Whmile
Gemma-2- 9B-Base
Here's how calculate 11 Calculate the car's daily energy consumption: *00 Energy per mile: 0.3 kWh/mile* Daily miles: 40 miles* Total daily energy: 0.3 kWh/mile * 40 miles 12 kWh**2. Calculate the total energy produced by the solar panels:*** Power per pancl: 400 waits = 0.4 KW Number of peak sun hours: 4 hours* Daily energy per pamel: 0.4 kW 4 hours = 1.6 kWh++., Determine the number panels needed;=** Total daily energy needed: 12 kWh Daily energy per punel:
kWh Number of panels: 12 kWh / 1.6 kWh/panel = 7.5 pancls* * Since you can't ' have half panel, you would need solar panels to produce enough energy to match the daily consumption of an EV driving 40 miles a day in Michigun," saggunn.saa Imporrati Considerations::** +* *vAerrgee sunlight:*s This calculation assumes 4 peuk sun hours per day, which an average. Actual sunlight hours vary depending on the time year and weather conditions." **Panel efficiency:** Solar panels have varying efficiencies. The calculation assumes a standard efficiency.* **System losses;** ** There are always some energy losses in a solar system duc to wiring, inverters, and other components." * **Battery storage:** To ensure consistent power even when the sun isn't shining, you would likely need battery storage.
Gemma-2- 9B-BNF
Here's how to calculate the number of 400-watt solar panels noaded:**]. Calculate Daily EV Energy Consumption: **4 **Energy per mile: 0.3 kWh/mile* +*Daily miles:* 40 miles* ++Total daily energy (kWh):** 0.3 kWh/mile 40 miles 12 kWh Calculate Daily Solar Energy Production: sspanel power; 400 watts 0.4 kW e-Peuk sun hours:* 4 hours* ==Daily energy per panel (kWh): 0.4 kW 4 hours= = 1.6 kWh*"3. Determine Number of Pancls: e-Total daily EV energy:** 12 kWh "DDaly energy per pancl:** 1.6 kWh* *Numberr of pancls;** 12 kWh/ / 1.6 kWh/pancl
**7.5 pancIs*****inn "ince you can't 't have half panel. you would need solar pancls with a power output of 400 watts each produce approximately the same amount of energy conditions.
EV driving 40 miles per day Michigan with the given
Gemma-2- 9B-DPO
Here's how to calculate the solar panel requinement:#21, Calculate Daily EV Energy Consumption: * Energy per mile:* 0.3 k WWh/mile* s*Daily miles driven;** 40 miles s*Talal daily energy consumop- tion:* 0.3 &Wh/mile 40 miles = 12 * Calculate Daily Solar Energy Production per
* **Panel wattage: 400 watts 0.4 kW" *Paak sun hours:** 4 hours" *DDilly energy production per pancl.** 0.4 kW 4 hours = 1.6 33 Determine the Number of Pancls Needod:*** *Totaa daily EV energy needed=* 12 kWh **Daily energy per pancl.*s 1.6 kWh Nummbrr of pancls:* 12 kWh / 1.66 &Whypanel = 7.5 pamels** Since
Since you can't have half a panel, you would need 8 solar panels to pm-- ducc enough energy to match the daily consumption conditions. ****Imprttan Considcrations: is reasonable average for Michigan, actual daily output fluctuates. Some days will have more, some less. Over time, this averages out R **Panel Efficiency:** ** Real-world solar panel efficiency is slightly less tham their rated wattage. Factor in a slight reduction (around 5-10%) for a more accurate estimate. + System Losses:* ...(100 more words).
EV driving 40 miles in Michigan under these ** *verage vs. Peak Sun *oury.* While 4 peak sun hours
Gemma-2- 9B-SimPO
Here's how calculate solar panel requirement: **1 Calculate Daily EV Energy Consumption: * Miles driven:** 40 miles* *Energy per mile:** 0 kWh/mile* TTota daily energy consumption."" 40 miles * 0.3 kWh/mile = 12 k Determine Solar Panel Output in Peak Hours: *ePaccl wattago: 400 watts = 0.4 kW* **Pcak sun hours 4 hours* **Daily output per pancl:** 0.4 kW hours = 1.6 kWh Calculate Numher Pan + *Required daily energy (EV):** 12 kWh **Output per panel:** 1.6 kWh* Numube of pu ++ 12 kWh / 1.6 &Wh/punel #47.5 panels**** have half a panel, you would need 8 solur panels to produce approximately the sume amount of energy as an EV driving 40 miles daily in Michigan under these conditions.
you can't I
22


Reviews:
[{"review_id": "X1o2h5Hh2M", "reviewer": "Reviewer_aXkU", "summary": "The paper proposes an alternative to DPO and its variants such that $|\\frac{\\partial{\\mathcal{L}}}{\\partial{z_y}}|$, the norm of the derivative of the loss with respect to the logits of a given outputs, decreases linearly as $p(y|x)$ deviates from the $p_{ref}(y|x)$ in either direction. The loss is called Bidirectional Negative Feedback, in contrast to what the paper describes as unidirectional negative feedback in the likelihood loss, namely that $|\\frac{\\partial{\\mathcal{L}}}{\\partial{z_y}}|$ increases as $p(y|x)$ decreases. \n\nThe paper also demonstrates how various DPO-series methods avoid excessive decreases in the likelihood of dispreferred samples relative to a loss that just applies NLL to the preferred samples and its negation to dispreferred samples: the gradient for these losses includes and additional scaling term $\\mathcal{C}(y_w, y_l, p_\\theta, p_{ref})$ that decreases as the gap between $p_\\theta(y_w|x)$ and $p_{ref}(y_l|x)$ increases. However, they argue that these losses are unideal due to their sensitivity to the hyperparameter $\\beta$.\n\nExperimentally, BNF outperforms other DPO-style losses overall when evaluated across 2 instruction-following QA datasets and 4 logical reasoning datasets, using the preference training datasets from the SimPO paper. Moreover, because BNF is not a pairwise contrastive loss, it is applicable even with non-pairwise data (e.g., sometimes just a preferred or a dispreferred sample without a counterpart), and experiments show that BNF can still improve over the base model in such settings. The authors also show that BNF exhibits the least amount of log likelihood shifts as well as the lowest Gini coefficient in the logit shifts across tokens in a sequence.", "questions": "1. Could the authors provide evidence of BNF's favorability with respect to training collapse?\n2. If BNF is meant to be an alternative to simply adding a NLL loss term to DPO-like methods, can the authors compare the proposed method with these baselines?\n3. Could the authors explain how each point made in Section 2.3 connect to a contribution or result in the paper / positive property of BNF?\n4. Could the authors clarify the central claims in the paper and distinguish them from statements that are hypotheses?\n5. Could the authors explain why the BNF loss is the way it is, e.g., why it necessarily needs to look as complex as it is when the main goal is to simply avoid an increasing gradient norm for large deviations? \n\nI would be happy to raise my score if these above concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper exhibits strong experimental results supporting the use of BNF over the baselines tested.\n2. The paper is easy-to-follow and makes progress on a relevant topic for the ICLR community.\n3. BNF is novel and well-motivated by the desire to decrease gradient norms as the log probs deviate more from the reference log probs.\n4. BNF improves upon DPO-like methods empirically and in applicability (going beyond pairwise data alone).", "weaknesses": "1. The authors point to two problems for DPO-like baselines (i.e., training collapse and alignment tax) but only seem to show positive results of BNF for one, i.e., less alignment tax. It would be helpful to see a precise definition of training collapse and show that BNF experiences less of it (or avoid framing the paper's story with training collapse).\n2. The authors mention that DPO-like baselines with an additional NLL term have limitations (i.e., poor chat and QA performance), but it is not clear to me that BNF overcomes these specific limitations when trained on the same datasets.\n3. While I believe that the paper makes sound and sufficient contributions, #1 and #2 point to a misalignment between the motivation for the work in the introduction and the contributions demonstrated in the body of the paper. Section 2.3 and Figure 2 are another example; they discuss various challenges and limitations, but it is not clear how each relates back to the paper's contributions. Also, for how focused this section is on the specifics of pairwise preference data in reasoning tasks, it is strange that the experiments don't explore training on such data.\n4. The paper could benefit from being more careful in distinguishing claims from hypotheses. For instance, in lines 496-471: \"This suggests that BNF achieves a balanced optimization strategy, reducing the gradients for tokens already showing large differences from the reference, thereby effectively preventing over-fitting and reducing the alignment tax.\" The logical leaps from smaller gradients for large differences to less overfitting and less alignment tax are not directly supported in the paper. Also, in the same section, it is not clear why a lower Gini coefficient is necessarily desirable in the first place; for instance, for some sequences it may be more desirable to just decrease the probability of the incorrect tokens rather than all tokens. Moreover, lines 54-55 state: \"we argue that the instability of DPO stems from a more fundamental cause: the unidirectional likelihood-derivative negative feedback inherent in log-likelihood loss.\" While this is a reasonable hypothesis, it is not a claim that has been definitively proven; thus, rephrasing this section as a hypothesis that guides the development of an alternative loss may be more apt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4tnMmJvdWL", "reviewer": "Reviewer_XXYh", "summary": "This paper proposes a new LLM alignment loss to address the issues of DPO variants being sensitive to hyper-parameters and unstable in mathematical tasks. The loss can maintain Bidirectional Negative Feedback (BNF) during optimization and eliminate the need of additional hyperparameters and pairwise preference data. Finally, they evaluate the method on multiple tasks and backbones to verify its effectiveness.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper clearly outlines the previous methods and theoretically identifies their weaknesses. The motivations of this paper are clear.\n2.\tIn terms of these issues, the authors propose their method and validate the effectiveness through theory and extensive experiments on multiple tasks and LLMs.\n3.\tThe structure of this paper is clear, and the equations and charts are well-presented.", "weaknesses": "1.\tFigure 2 illustrates the challenges of creating a substantial log-likelihood gap in mathematical tasks. Could you demonstrate the advantages of your method with similar examples in mathematical tasks?\n2.\tI am somewhat puzzled by Eq.(6). What are the motivations behind designing the equations where $y_i \\neq t_i$? It appears to be a careful design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gcQfdiCpyf", "reviewer": "Reviewer_dQGY", "summary": "This paper proposes Bidirectional Negative Feedback (BNF), an LLM alignment loss that does not rely on pairwise contrastive losses.  Consequently, it does not require pairwise data and has fewer hyper-parameters compared to DPO. The authors empirically show that the models` reasoning ability is less affected when using BNF for preference optimization. They compare BNF to previous approaches such as DPO, IPO, KTO, SLiC-HF, ORPO, SimPO on QA and reasoning benchmarks.", "questions": "- Is BNF more sample efficient because it does not require pair-wise dataset? Is there a relationship between sample efficiency and loss/dataset type?\n- Do you expect the other baselines to close in on BNF with further hyper-parameter tuning? \n- How sensitive is BNF to hyper-parameter tuning? Is it possible that the hyper-parameters are harder to tune despite the fact that they are fewer? \n- Is the performance difference on the benchmarks significant?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- BNF has fewer hyper-parameters compared to DPO. If the method is robust to hyper-parameter tuning, this could make LLM alignment more compute efficient. \n- BNF does not require pairwise data. \n- The baselines are diverse and strong. The hyper-parameters of the other approaches were fine-tuned (although not directly by the authors).", "weaknesses": "- The performance across benchmarks is still relatively close to DPO. \n- Gemma lacks proper baselines as the scores were only copied from another paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)