$\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$
Authors: Junkang Wu
Yuexiang Xie
Zhengyi Yang
Jiancan Wu
Jinyang Gao
Bolin Ding
Xiang Wang
Xiangnan He
Venue: NeurIPS 2024 poster (2024)
Keywords: Direct Preference Optimization
LLM's alignment
Primary area: natural_language_processing

Abstract: Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter $\beta$, as well as to the quality of the preference data. We analyze the impact of $\beta$ and data quality on DPO, uncovering that optimal $\beta$ values vary with the informativeness of pairwise data. Addressing the limitations of static $\beta$ values, we introduce a novel framework that dynamically calibrates $\beta$ at the batch level, informed by data quality considerations. Additionally, our method incorporates $\beta$-guided data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic $\beta$ adjustment technique significantly improves DPO’s performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at \url{https://anonymous.4open.science/r/beta-DPO-EE6C}.

B-DPO: Direct Preference Optimization with Dynamic B
Junkang Wu Ia Yuexiang Xie Zhengyi Yang' Jiancan Wu
Jinyang Gao² Bolin Ding² Xiang Wang' Xiangnan Hel
University of Science and Technology of China
2 Allibaba Group
(jkwu0909, yangzhy1998, wujcan, xiangwang122 xiangnanhe)@@mail com, (yuexiang. xyx, jinyang jinyang.giy, bolin. ding) (aang)aall Cal ibaba leabaaiin iene-inn..ooo iom
Abstract
Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter 83, well as to the quality of the preference data. We analyze the impact of and data quality on DPO, uncovering that optimal / values vary with the informativeness of pairwise data. Addressing the limitations of static 3 values, we introduce novel framework that dynamically calibrates 3 at the batch level, informed by data quality considerations. Additionally, method incorporates 33-gudded data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic 3 adjustment technique significantly improves DPO's performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at https: :// //ginng github. com/ junkangwu/beta-DDa
Introduction
The alignment of Large Language Models (LLMs) with human feedback, as explored in works like GPT-4 and LLaMA-2 [29, 38, 8]. has marked significant advancement in generating responses that are more helpful, factual, and ethical [30]. Among the various alignment strategies, Reinforcement Learning from Human Feedback (RLHF) [30] is notable method that refines LLMSs using the Proximal Policy Optimization (PPO) algorithm [36]. This approach employs KI. divergence penalty ensure minimal deviation of the model from its original configuration, ensuring the retention of its initial characteristics while improving alignment.
Despite the effectiveness, RLHF's instability and computational requirements often limit its practical applications, prompting the exploration of alternatives like Direct Preference Optimization (DPO) [34]. DPO cireumvents the reinforcement learning loop by exploiting the inherent connection between reward functions and optimal policies, thereby simplifying the policy model training. It encourages the model to favor the response that aligns with human preferences (y.) over the dispreferred (yr), implying DPO's sensitivity to the quality of pairwise data. The balance between maintaining the original reference model (Tat) and incorporating new preferences (To) is controlled by hyperparameter, whose lower values advocate for aggressive updates and higher values support more
"Work done Alibaba Group.
Jiancan Wu and Xiangnan He are the corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Humon: How can I store food if don't have a pantry?
pwa.3
Bwb.1
Bw0.3
Winn Rate (%6
Assistont: You could store the food in a refrigerator, the top cupboards in your kitchen, the freezer,
hole the ground.
Assistant: By storing food in your refrigerator freezer. You have access to lange volume your home's is space in these places.
Assistant: You could store things in container, like wooden box bucket.
Low Gap
High Gop
Low gap
High gap
(a)
(b)
Figure 1: (1a) Pairwise Data: Low vs. High Gap: "Low gap" denotes cases where the chosen and rejected examples are closely similar, typically indicating high-quality, informative pairs. "High gap" signifies pairs with larger differences, implying lower-quality data. (1b) Influence of Data Quality on Selection: Pythia-1.4B3 performance on the HH dataset reveals a distinet trend: for "Low gap". a higher 3 reduces win rate, whereas for "High gap", an increased [ improves it.
conservative adjustments:
To (y x)) [ye (y xx)x
To (yi x Tref yi x) }]]]
(D) = -Exyw log o [3[log]
log
However, the current DPO literature has largely overlooked the joint influence of / selection and pairwise data Yi)'s quality on the alignment performance. To bridge this gap, we conduct preliminary experiment to investigate how different /3 selections influence the model performance under two distinct data pair gap conditions, as shown in Figure 1b. low gap scenarios (cf. Figure la), where the difference between preferred and dispreferred samples is minor, an increase in 3 (e.g., from 0.1 to 0.5) corresponds with a decline win rate (e.g., from 42% to 33%) Conversely, gup situations (cf. Figure Ia) where significant difference exists, an increase in 3 tends to improve DPO performance. Such contrasting outcomes highlight the necessity to tailor the B value contingent upon the data quality, especially in the presence of outliers [4].
high
Building upon these insights and recognizing the mixture nature of diverse quality in practically collected data [31], we propose dynamic 3 selection strategy for DPO. This strategy adaptively adjusts in response to the quality of pairwise data and robustifies DPO against data variability. Intuitively, one straightforward solution is personalizing 33 for each pair, rather than fixing it across the population all pairs. While conceptually appealing, (1) instance-level 33 personalization can lead to optimization instabilities, particularly when dealing with a vast array of human preference instances (ef. Section 5.3). This issue underscores the challenge of balancing 3 updates with stable and scalable DPO. In this light, we propose to explore batch-level adaptation of 3, aiming to balance update aggressiveness and training stability. Moreover, (2) the frequent occurrence of outliers necessitates strategy for accurately adjusting the batch-level 8. The dataset notably features outliers, a challenge underscored by the significant reward discrepancy variations within the training samples of the dataset (cf. Section 4.1). Such conditions impede the model's ss ability accurately estimate the batch-level B, thereby undermining the effectiveness of batch-level B calibration. To this end, we propose a simple yet effective dual-component approach:
Dynamic 3 Calibration at Batch-Level (for Challenge 1): mitigate optimization instabilities. we dynamically calibrate 3 within each batch. Specifically, this batch-level adjustment is based on data quality, with 3 being adaptively decreased for high-quality, closely-matcher pairwise data (i.e. low gap data) to facilitate assertive updates. While for easily-discriminated pairs (i.e., high-gap data), 3 is increased to promote cautious updates, preventing overfitting to noise. This targeted batch-level calibration enables stable and responsive optimization.
3-Guided Data Filtering (for Challenge 2): to tackle the frequent occurrence of outliers. By establishing benchmark 3 value for filtering incoming data the batch level, we maintain the fidelity of 3 estimation by prioritizing the most reliable and representative samples. As a result, it diminishes the impact of outliers that might
We implement a 9-guided data filtering approach

---

otherwise derail the optimization process, thus enhancing the precision and robustness of the batch-level 3 calibration.
Our contributions are threefold: (1) We investigate a pioneering study on the joint influence of B selection and pairwise data quality on the DPO performance. (2) We introduce a simple yet effective 8-dynamic strategy for DPO, adaptively balancing the update aggressiveness and training stability. (3) Through empirical evaluations, our approach demonstrates marked improvements performance across diverse conditions and model sizes (e.g,, achieving improvements exceeding 10% on models of various sizes, including Pythia-410M, 1.4B, and 2.8B).
2 Related Work
Reinforcement Learning from Human Feedback. Despite RLHF's effectiveness in aligning language models (LMs) with human values [10. 3, 38, 30. 14, 15], its complexity and resource demands have spurred the exploration of alternatives. RAFT [12] selects optimal training samples via an existing reward model, whereas RRHF [45] employs simpler ranking loss, retaining PPO's efficiency. Diverse from these, DPO [34] directly optimizes LMs using a preference-based loss function, showcasing enhanced training stability in comparison
traditional RLHF. Innovatively, SLiC-HF [47] and KTO [13] devise loss functions rooted in human decision-makinn
focusing on preference calibration and utility optimization, respectively. Dr. DPO [42] consider robust settings where safety group information is known at training time. Further, RSO [25] and ORPO [19] introduce efficient preference modeling and optimization, with ORPO uniquely combining supervised fine-tuning and preference alignment. These advancements reflect the ongoing shift towards more
efficient, nuanced RL methods.
Data Quality in LLM's Alignment. Recent studies have increasingly recognized the significance of data quality in the aligmment of LLMs. For example, LIMA [48] leverages heuristics such as post scores, response lengths, formatting, and topics to manually craft 1000 high-quality datasets from StackExchange, wikiHow, and Reddit for superficial alignment. In a similar vein, Bai et al. [3] prioritize data points based on user engagement levels for dataset assembly. Rejection Sampling (RS) and Best-of-N (BoN) techniques, as evidenced in the works of Nakano al. [28] and Gao et al. [16], involve selecting the optimal candidate from N generated possibilities through the application of a reward model. To enhance preference optimization, RSO [25] uses statistical weightings to differentiate outcomes from optimal policy and a base SFT policy. Besides, iDPO [27] employs Reward Model to filter out Luw-quality data, effectively addressing dataset quality concerns.
Preliminaries
Given a text sequence (commonly referred to as prompt) x, sequence y [y1,, 12,,32.--. - UN] is generated as a response to the prompt x. autoregressive language model æ, when provided with the prompt can generate the response sequence y following the probability decomposition:
(y|x) II m(u
where Yet denotes the preceding tokens in the response sequence. Now, given preference dataset D = (x(0), y w yt ( ) wherein each triplet consists of a prompt x with two responses y E Σ and yı E Σ", with E- representing the alphabet, preference oracle either human annotator or a language model - provides preference feedback o(y > yı E {0, 1}. indicating whether yo is preferred over yı. We denote PPy... /r/x) sy > yı x)] the probability of y.. "winning the duel" over yı. The Kullback-Leibler (KI.) divergence between two probability distributions with densities p and q is defined as (l(p)) Ey-p(y) log p(x)
q(v)
RLHF with Reward Models. Christiano et al. [10] pioneer the learning of a reward function r(y: x) based on the Bradley-Terry model [7]. This model is deployed for the triplet of a prompt (x) and two responses (yw. w yı), establishing the likelihood of preference for y over yı as:
exp(r(yw;
Pyy y1/x) exp(r(ymix)) x)) exp(r(ynx)) (r(y x) r(yxx)),
(2)
3

---

where (z) <² /C" +) represents the logistic function. The approach for estimating the reward function within the Bradley-Terry framework to maximize the bog-liielhood log (y w. > y) Assuming accurate estimation of the true reward function r(y;x), Christiano et al. [10] propose to solve the following problem with policy optimization algorithms RI. such as PPO [36]:
max Ex y y- (-x))/ /3Rxx= x (xx
where represents the prompt distribution, r(y; x) denotes the reward function learned using the Bradley-Terry model the preference dataset, Wee is the fixed reference model (typically selected to be the one post supervised fine-tuning), and serves as the penalty coefficient of the KL divergence.
Directed Preference Optimization (DPO). Rafailov et al. [34] identify that the optimization problem above has a closed- form solution such that for any y.
. (ylx) O Ty x) exp(r(y;x)/33
which can be further converted to the DPO loss for any triplet (x, Ni):
mT(yu[lx))
To(vi)><) )) TTer(y1
(x, You Y1; V1;0; Fett
log
log
log
4 Method
In this section, we investigate the critical connection between the parameter 3 and the quality of pairwise data in optimizing DPO. We present empirical evidence demonstrating the effect of B settings on DPO performance across datasets of varying quality. Our proposed method, 3-DPO, introduces dynamic calibration of /3 and data filtering mechanism tailored to improve DPO's s effectiveness across diverse data conditions.
4.1 Motivation: The Impact of Pairwise Data Quality on Selection
Scrutinizing Equation (4), argue that DPO's effectiveness critically hinges two factors: the choice i and the quality of pairwise data. Here, we conduct experiments to demonstrate the influence of variations in 33 and data quality on DPO, pivotal for its effective real-world application.
Datasets. We utilize the Anthropic HH dataset [3] for our experimental analysis, which contains approximately 170,000 dialogues between humans and an automated assistant. In this dataset, human inquiry, denoted as x, is paired with two responses (yw: where yı: represents the response favored by the human annotator, while y is the alternate response. Notably, the alternate response y retains informational value, making this dataset high-quality with minimal discrepancies between the response pairs, which we classify as a low gap dataset. To further explore the impact of data quality on DPO, we construct a synthetic dataset, referred to as the high gap dataset. This dataset differs from the low gap dataset by introducing greater disparity between responses. Specifically, the alternative response yI is generated by a Supervised FineTuned (SFT) Pythia-2.89 model, while the preferred response Vou remains consistent with the original dataset. We also combine the two datasets in equal proportion to create a mixed gap dataset, with each contributing 50%, to incorporate the characteristics of both the low gap and high gap datasets.
Models and Metrics. Our study evaluates various model sizes, specifically Pythia-410M, Pythia- 1.4B, and Pythia-2.8B [5], to ensure comprehensive assessment. Following the established protocol in DPO [34], each model iteration undergoes a single epoch with a batch size of 64. This setup provides uniform basis for evaluation across different models. We adopt the evaluation strategy from DPO [34] to calculate the win rate, a metric that measures how often the GPT-4 model prefers a response generated by our models over the default chosen response the subset of the test dataset.
Findings: (1) The optimal value of varies with data quality, reflecting divergent perfor- mance patterns across datasets. In Figure 2, we present the win rate results across three levels of pairwise data gap, each evaluated under varying 8 parameters. As can be observed, with low gap pairwise data, smaller value is preferable for optimizing performance. This is likely

---

Pythia-410M
pythia-1.4B
pythia-..88
50
Win Rate (%)
00
30
30
20-
10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.1 0.2
0.3
0.4
0.5
0.6
0.7
0.8
B
B
Figure 2: Win rate performance of DPO across different 3 settings on the low gap, mixed gap, and high gap datasets.
because the informative content of such data allows a lower 3 to facilitate more substantial up- dates, thereby enhancing alignment accuracy. Conversely, for high gap pairwise data, maintain- ing low 3 may lead to overfitting, which significantly undermines the alignment process. The mixed gap dataset - combination of both low gap and high gap datasets - exhibits more nuanced performance pattern, suggesting the necessity for dynamic 3 calibration strategy to adapt to varying data quality. Consequently, adhering to a fixed 3 value, i.e., configuring 3 at the population level, might be inadequate for the dynamic and varied nature of real-world datasets. (2) The dataset exhibits notable outliers. In Figure 3. utilizing the Pythia-2.8B model, evaluate the data quality by examining the distribution of reward discrepancy for each triplct (which we will define as "individual reward discrepancy" later) within the HH dataset's training samples. The tails of the density plot extend beyond the highlighted percentiles, suggesting the existence of data samples with significantly higher or lower reward discrepancies. Notably, cases with significantly higher rewards for positive samples over negative ones suggest low informational value, as these discrepancies likely do not contribute meaningfully to the model's learning process. Whereas the opposite cases hint at potential labeling errors. Both cases deviate from an expected rational distribution range and are thus classified as outliers. For further details on outliers, kindly refer Appendix A.2.
Mezn.
10% Deaattle-5.00
Beward Diserepancy
Figure 3: The distribution of individual reward discrepancy
(r (f) x() r(y x()))
the training dataset of HH.
4.2
Method: Dynamic 3 Calibration in DPO
Through our empirical analysis, we highlight the sensitivity of DPO to selections and the frequent occurrence of outliers. Hence, determining the optimal 3 value requires careful consideration of the quality of pairwise data while also addressing the influence of outliers. This prompts the question: what criteria define superior choice of 3? In response, we propose the following guiding principles:
Principle 1:
The optimal 3 value should be responsive to pairwise data's quality.
Principle 2: The selection of 3 value should minimize the influence of outliers.
4.2.1 Dynamic 3 Calibration at Batch-Level
We begin by introducing the concept termed 'individual reward discrepancy', which represents the difference between the rewards of winning and losing for each triplet, serving as measurement for pairwise data quality. Formally, for triplet (x) Yuu .Y
(6) (4) E D, the individual reward discrepancy
defined as M, = r(y (a) x (i)) r(y) ()). While our primary analysis utilizes the implicit reward model induced by the policy trained using DPO, we also conducted comparative experiments with an explicit reward model. The details of these experiments with the explicit RM can found in Appendix A.5. For the DPO-basce implicit reward model, the reward discrepancy is expressed as:
5

---

xx) Tea ywu
(oo
To U Teet (t
x))
M = 30 log
So log
Here, xp represents the policy being optimized, and Tie denotes the reference policy. This formulation captures the difference in the log-probabilities of the winning and losing outcomes, weighted by the parameter B. Motivated by our guiding principles, a straightforward approach is to assign distinct 33 to each triplet, allowing each & to serve as parameter tailored to its respective triplet. This instance-level dynamic 3 adaption can be formulated as follows:
B = Bo + aMM Mo) [1+ a(M Mo)]
(5)
where Bo represents the benchmark hyperparameter intrinsic to DPO, typically set to 0.1. The term scaling factor within the interval [0, 1] that adjusts the influence of Mr on B, Specifically, when = 0, Bü remains constant at So. thus maintaining the standard DPO framework without modification.
My denotes predetermined threshold, and the coefficient
Equation (5) illustrates that 3. increases monotonically with M,, allowing the model to adjust the value based the running reward differential between paired samples. Nevertheless, such instance-lever adjustments may introduce instabilities during training. Prior studies have shown that minibatch approach can help avoid saddle points or local minima [17], as well as mitigate the impact of noise [35, 6]. Drawing inspiration from these benefits, we propose a batch-level dynamic estimation methodology for 3:
Saacch [1 + (EE [M]] Mo)]
(6)
In practical applications, the threshold Mo can be estimated by employing the global mean of M; with moving average updating scheme [24]:
Mo +- mMo + Eaeee
where m E [0, 1) is a momentum coefficient. Such a batch-level calibration method introduces only one new parameter, a, to control the scale of B3 adjustment. The calculation of lE4-batch [MM,] is straightforward within DPO procedures, thereby incurring no additional computational overhead.
4.2.2 3-Guided Data Filtering
To mitigate the adverse impact of outliers on the 3 selection process, we introduce 3-guided data filtering mechanism. Informed by 3o confidence criterion [33]. this strategy employs a probabilistic
model to assess the significance of each triplet (x((), y .yt discrepancy M,, which is defined as:
(e) (6) based its individual reward
(M: (-(M, (00-MM))
M(M)
exp
(8)
2TTo
20
where Mo and represent the mean and standard deviation of M, across the training dataset, respectively. Similar to the updating scheme of Mo in Equation (7), we dynamically estimate the value using the moving average method:
σ + mo + (1 m) V Winbarch M.
(9)
This probabilistic weighting discerns the relative importance of each sample, guiding the selection of |bathh| x P samples (without replacement) based on their calculated probabilities p(M.). Here, denotes the selection ratio, defaulting to 0.8, choice validated by preliminary experiments aimed at optimizing training efficiency and model accuracy.
This process is iterated for each training batch, ensuring that the training data is continuously updated to reflect the most informative samples. The introduction of the 3-guided data filtering strategy is instrumental in fortifying the model against outliers, thereby facilitating the accurate estimation of the 3 value.
Highlights: We underline the following key features of our proposed 3-DPO framework:

---

Anthropic-HH Dialogue Win Rate Chosen
TL;DR Summarization Win Rate vs Chosen
80
50
Win rate
Win rate
50
40
DPO
DPO
DPO Dynamic $
DPO + Dynamic B
DPO Data Filtering
DPO + Data Filtering
30
B-DPO
20
B-DPO
0.50
0.75
0.00
0.25
0.50
0.75
1.00
Sampling temperature
Sampling temperature
Figure Left. The win rates computed by GPT-4 evaluations for the Anthropic-HH one-step dialogue; 3-DPO consistently outperforms across all sampling temperatures. Right. In the com- parison of TL;DR summarization win rates versus chosen summaries with GPT-4 as the evaluator, 3-DPO is distinguished as the only strategy achieving win rate over 50% across different sampling temperatures.
Simplicity: B-DPO is extremely straightforward and quick to implement. It merely involves dynamic 33 adjustment at the batch level and a 3-guided data filtering mechanism, both of which are predicated upon the reward discrepancy denoted by M,.
Unlike other methodologies [25, 27, 32, 40] that necessitate an additional gold model for data filtering, our method leverages the running reward discrepancy M, within the DPO framework. Moreover, our empirical studies indicate that 3-DPO exhibits insensitivity to the hyperparameters A default setting of 0.8 typically yields satisfactory performance.
Efficiency:
Model-agnostic: As variant of the traditional DPO, the proposed 8-DPO can function plug-and-pla module. It allows straightforward integration of future enhancements and extensions within the DPO framework. Our empirical investigations corroborate this assertion.
4.3 Discussion with Previous Studies
Relations to Data Selection. An increasing volume of works [25, 27, 48, 32, 44] have underscored the impact of data quality the performance of LLM's alignment. A common practice among these efforts involves employing so-called "gold model" for data selection. This approach, however, introduces significant computational demands and the choice of the gold model directly influences the resultant system's 's performance. The focus of this work, it should be noted, not to propose superior strategy for data selection. Instead, we aim to enhance adaptability the quality of data by dynamically adjusting the 3 parameter. This adjustment facilitates improved & estimation by selecting data based the reward. Moreover, Section 5.2 illustrates the compatibility dynamic adjustment with other data selection methodologies.
Relations to Recent Temperature Schemes. in the realm of contrastive learning, motivated by various objectives, such as addressing out-of- distribution tasks [46] accommodating long-tail data distributions [23]. The work most closely related to ours, MACL. [20], has indeed proposed an alignment-adaptive strategy; however, its primary aim navigate the uniformity-tolerane dilemma. Hence, the integration of dynamic temperature mechanisms with L.L''s alignment remains an underexplored area against this backdrop.
Dynamic temperature frameworks have been introduced
Experiments
In this section, we commence by conducting an empirical evaluation 3-DPO on two specific tasks: dialogue generation and summarization. Subsequently, we analyze the various adaptations of the proposed method 3-DPO. Concluding this section, we underscore the imperative need for batch-level dynamic & calibration, highlighting its significance in the context of our study.

---

Table Win rate comparison of Pythia-410M, -.44, and -2.8B models on the Anthropic HH dataset, evaluated using GPT-4.
Method
410M
1.4B
2.8B
DPO
26.19
42.78
51.51
DPO + Dynamic 3
27. 13..6.7
43.51 +1.71%
55.19 19-11%
DPO + Data Filtering
29.03 10.84%
46.99
46.99
+9.84%
53.42 +.711%
B-DPO
30.18 15.23%
48.67 13.77%
57.07 10.79%
5.1 Empirical Evaluation of (3DPO on Dialogue Generation and Summarization
Datasets and Setup. Our experiments are conducted on the Anthropic HH dataset [3] and Reddit TL;DR summarization dataset [39]. The training configuration follows from Rafailov et al. [34]. The goals of these experiments are to study: 1) How 3-DPO performs on single-turn dialogue generation and summarization tasks; 2) How the sampling temperature affects the performance of (3DPP;; 3) How B-DPO works with different model sizes. For detailed experimental settings, please refer to Appendix A.1.
Baselines. In our comparison, we examine the performance of 3-DPO relative its counterparts: the standard DPO, DPO implemented with a dynamic 3 yet devoid of (-guided data filtering, and DPO complemented by data filtering with 3 fixed at 0.1.
Win Rate Across different Sampling Temperature. An analysis of win rates derived from GPT-4 evaluations the Anthropic- -HH one-step dialogue demonstrates that 3-DPO consistently outper- forms across all sampling temperatures, as depicted in Figure 4 (Left). Furthermore, for the TL;DR summarization task, /-DPPO stands out the only approach achieving win rates above 50% for diverse sampling temperatures, which is visually represented in Figure 4 (Right). The data also suggests that while both dynamic 3 and data filtering enhance DPO's effectiveness, the impact of data filtering is especially pronounced in the summarization task, likely due to the inherently greater noise present in the Reddit TL:DR summarization dataset. Notably, 8-DPO exhibits a remarkable degree of robustness to variations in sampling temperature. As the temperature incrementally escalates from 0.0 1.0, the win rate for standard DPO plunges to mere 25%, whereas 3-DPO maintains commendable performance level with win rate of 54%.
Win Rate Across Different Model Sizes. We further evaluate the performance of 3-DPO the Anthropic HH dataset with Pythia-410M, -1.4B, and -2.8B models. The results are summarized in Table 1. We observe that 3-DPO consistently outperforms DPO, DPO with dynamic B, and DPO with data filtering across all model sizes. We observe that in a smaller model, the improvement of data filtering is more significant, while in larger model, the improvement of dynamic 3 is more significant. We attribute this to the fact that the larger model has more capacity to learn the optimal policy, while the smaller model needs more help from the data filtering.
5.2 Adaptations of 3-DPO
In this section, our inquiry is twofold: first, we aim to understand the performance of 3-DPO when applied across various filtering strategies; second, examine its efficacy across different adaptations of the DPO framework. In terms of filtering strategies, prevailing studies [32, 44] in the domain largely employ a gradient-basee approach. We propose to extend this methodology into three distinct scenarios. This involves arranging the gradients of pairwise data within batch and consequently: (1) Excluding the top 20% of samples, hereby referred to as Filter Head, (2) Excluding the bottom 20% of samples, hereby referred to as Filter Tail, (3) Excluding both the top and bottom 10% of samples, a method we denote as Filter Tail & Head. For fair comparison, we maintain the amount of data excluded at 20% for the above strategies. Second, we integrate three variants of DPO into our analysis: the IPO [2], a novel approach that facilitates learning directly from preferences without need for the Bradley- -eery (BT) model. Additionally, we consider the KTO [13], which focuses on discerning whether preference desirable or undesirable and SPPO [43], which approximates the Nash equilibrium. For detailed settings, we refer the reader to the supplementary material.

---

Reward Discrepancy Comparison
Dynamic with other Filtering Methods
Dynamic with DPO vartants
Individuaa Roward Diocr oparep
DPO
DPO
DPO-dynamic
B-DPO
Win hate (%)
Win Ralle (54)
Fitr
190
Filter Type
DPO variants
Batch-iuevll
Instaric BETICCLLPP
Figure 5: Left: Win rates from GPT-4 evaluations Anthropic-Hil single-turn dialogues, showcasing 3-DPO's adaptability to diverse filtering strategies. Middle: Win rates of 3-DPO across various DPO variants evaluated by GPT-4. Right: Distribution of individual reward discrepancies following fine-tuning through batch-level and instance-level calibration.
Selective filtering of the top 20 % of samples markedly enhances model performance. This approach, detailed in Figure 5 (Left), not only surpasses other filtering strategies but also suggests that these samples, which exhibit the smallest discrepancies between positive and negative pairs, are particularly prone to flipped noise. By excluding them, the model's learning efficacy is appreciably improved.
Dynamic 3 adapts to and improves upon existing filtering strategies. Figure 5 (Left) corroborates our stance that static proves insufficient within the DPO framework. We contend that the applica- tion of our dynamic 3-DPO could markedly reshape the DPO field by fostering the development of advanced filtering techniques.
Dynamic 3 Enhancement across DPO Variants. We introduce dynamic 3-DPO, a novel strategy enhancing DPO and its variants: IPO, KTO, and SPPO in Figure 5 (Middle). Our results the Anthropic HH dataset demonstrate that while IPO initially leads in performance, the integration of dynamic substantially elevates all variants, notably increasing 3-IPO's efficiency by 17.9%. This underscores dynamic 3-DPO's ' capability to significantly enhance model training through adaptable improvements, solidifying its value in advancing language models via human feedback.
5.3 Necessity of Batch-Level Dynamic 3 Calibration
In this section, we aim to underscore the pivotal role of batch-level tuning in calibrating the parameter 3. To this end, compare the performance of our 3-DPO algorithm under distinct regimes: one employing batch-level dynamic 3 calibration, and the other utilizing instance-level dynamics. To emulate the diverse data disparity scenarios encountered in practical applications, adopt the methodology outlined in Section 1. meticulously blending datasets characterized by both low gap and high gap attributes at varying ratios.
Batch-level calibration surpasses both instance-level and population-level approaches. The results presented in Table 2 illustrate that batch-level dynamic 3 calibration yields superior perfor- mance compared to instance-level dynamics and the baseline population-level approach (referred to vanilla DPO) across range of mixture ratios. This improvement can be credited to the batch-level calibration's s ability to adjust to the varying data quality present within a batch, thus refining the model's learning process. Conversely, instance-level dynamics provoke excessively vigorous model updates, precipitating decline performance particularly noticeable at a mixture ratio of 40%, scenario in which outliers exert a significant negative impact.
Instance-level calibration magnifies the impact of outliers. As demonstrated in Figure 5 (Right), instance-level calibration inadvertently widen the range of reward discrepancy distribution. This broadened range suggests that instance-leve calibration might be leading to excessively high or low values for the model. Such disparities in 3 values consequently cause disproportionate update rates for certain samples, further intensifying the extremities in the distribution.
Our 3-calibration strategy consistently outperforms baseline methods. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-ar models SimPO [26]. We perform 3-DPO with two families of models, Llama3-8B [1], Mistral2-7B [21], on the Utrrahaat-200k

---

Table Comparison of win rates across varying mixture ratios on the Anthropic HH dataset, with each ratio indicating the proportion of high to low-gan datasets, e.g., . a 40% mixture ratio reflects a blend of 40% high-gap and 60% low-gap.
Mixture Ratio
10%
20%
30%
40%
Vanilla DPO
50.17
50.56
47.95
29.15
Instance-level calibration
49.18 1.97%
49.82
1.46%
44.42
7.36%
16.82
42.30%
Batch-level calibration
57.68+
14.69%
56.15
1..0%
51.25
+6.88%
34.92
19.79%
dataset [11]. For comparison with basclines, we assess our models using one of the most popular open-
ended instruction-followwe benchmarks: AlpacaEval All settings are consistent with SimPO [26].
Table 3 presents the AlpacaEval 2 re-
Table 3: Performance comparison of different models
sults under the Iistral-Instruct (7B) and Llama3-Instrus (8B) settings. LC and WR denote length-controlled and win rate, respectively. Regardless of whether we Llama3-8B Mistral-7B, and whether the loss function is DPO or SimPO, our 3-D, SimPO strategy consistently demon- strates significant performance improve- ments. This thoroughly showcases the method's strong generalization ability and excellent scalability.
Mistral-Instruct (7B)
Llama3-Instrucl (8B)
Model
LC (%)
WR (%)
LC (%)
WR (%)
DPO
20.98
21.60
40.44
37.38
3-DPO
23.56
20.42
43.38
38.21
SimPO
28.50
30.56
44.38
38.97
3-SimPO
30.48
32.13
46.03
40.18
Conclusion and Future Work
This paper introduces 3-DPO, a novel framework designed to optimize DPO by dynamically ad- justing the 3 parameter in response to the variability in the informativeness of pairwise data. Our approach, which incorporates 3-guided data filtering and batch-level dynamic / calibration, has demonstrated significant improvements in DPO's performance The empirical evaluations indicate that 3-DPO offers an adaptable training paradigm for LLMs with human feedback.
a range of models and datasets.
Limitations and Future Work, Our work on 3-DPO showcases promising framework for LLM optimization, albeit with room for advancement. Future endeavors should explore: Adaptive 8 in Self-Play: Extending 3-DPO to self-play scenarios [9, 43] where negative samples dynamically adapt, necessitating iterative adjustments, foster the evolution of superior model strategies. Enhanced Evaluation Standards: Development of sophisticated metrics and use of advanced eval- uators beyond win rates, capitalizing advancements like GPT-4+, to comprehensively gauge model performance. Scalability Investigation: Examining 3-DPO's 's scalability to ultra-large models surpassing 7B parameters, and its integration into diverse DPO-inspired architectures, is pivotal for practical impact. Automated Parameter Tuning: Pursuing automation in parameter tuning, alleviating manual intervention for 3, streamline the training pipeline and broaden accessibility.
Acknowledgments and Disclosure of Funding
This research is supported by the National Science and Technology Major Project (2023ZD0121 102), National Natural Science Foundation of China (92270114, 62302321). This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.
References
[1] Al@Metta Ilama model card. 2024. URL httpht:/t////thhbb cow/meta- 11amm /lama3/ blob blob/main/M0D3 CARD///DDEELCAADD... md.
10

---

[2] M. G. Azar, M. Rowland, B. Piot, D. Guo. D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. CoRR, abs/2310.12036. 2023.
[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. E. Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedhack. CoRR, abs/2204.05862 2022.
[4] I. Bejan, A. Sokolov, and K. Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets. In EMNLP. 2023.
[5] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. ICML, 2023.
[6] L. Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, 2010.
[7] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[8] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee,
Y. Li. S. M. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303. 12712, 2023.
[9] Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q Gu. Self-play fine-tuning converts weak language models to strong language models. In JCML, 2024.
[10] F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
[11] N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, 2023.
[12] H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. CORR, abs/2304.066677, 2023.
[13] K. Ethayarajh. W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. KTO: model alignment as prospect theoretic optimization. CoRR, abs/2402.01306, 2024.
[14] J. Fang, Z. Bi, R. Wang, H. Jiang, Y. Gao, K. Wang, A. Zhang, J. Shi, X. Wang, and T.-S. Chua. Towards neuron attributions in multi-modal large language models. In Thirty-eighth Conference on Neural Information Processing Systems, 2024.
[15] J. Fang, S. Zhang, C. Wu, Z. Yang, Z. Liu, S. Li, K. Wang, W. Du, and X. Wang. Molte: Towards molecular relational modeling in language models. In ACL (Findings), pages 1943- 1958. Association for Computational Linguistics, 2024.
[16] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 10835 10866. PMLR, 2023.
[17] R. Ge, F. Huang, C. Jin, and Y. Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. In COLT, 2015.
[18] A. Havrilla, M. Zhuravinskyi, D. Phung, A. Tiwari. Tow, S. Biderman, Q. Anthony, and L. Castricato. trlX: A framework for large scale reinforcement learning from human feedback. In EMNLP, Dec. 2023.
[19] Hong, N. Lee, and J. Thomee ORPO: monolithic preference optimization without reference model. CoRR, abx/2403.07691, 2024.
11

---

[20] Z. Huang. H. Chen, Z. Wen, C. Zhang, H. Li, B. Wang. and C. Chen. Model-aware contrastive learning: Towards escaping the dilemmas. In ICML, 2023.
[21] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. de Las Casas, F. Bres- sand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B. ArXiv, abs/2310.06825, 2023.
[22] D. Jiang, X. Ren, and B. Y. Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In ACL, 2023.
[23] A. Kukleva, M. Böhle, Schiele, Kuehne, and C. Rupprecht. Temperature schedules for self-supervised contrastive methods on long-tail data. In ICLR, 2023.
[24] T. P. Lillierap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[25] T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization. CoRR, abs/2309.06657, 2023.
[26] Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. CoRR, abs/2405.14734 2024.
[27] Morimura, M. Sakamoto, Jinnai, K. Abe, and K. Air. Filtered direct preference optimization. arXiv preprini arXiv:2404.13846 2024.
[28] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09333 2021.
[29] OpenAl. GPT-4 technical report. CoRR, abs/2303.08874, 2023.
[30] L. Ouyang, J. Wu, X. Jiang. D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, Schulman, Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[31] G. Penedo. Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, and J. Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data only. In NeurlPS, 2023.
[32] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent. In NeurIPS, 2020.
[33] Pukelsheim. The three sigma rule. The American Statistician, 1994.
[34] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. In NeurlPS, 2023.
[35] H. Robbins and Monro. stochastic approximation method. 1951.
[36] Schulman, F. Wolski, P. Dhariwal, A. Radford, and Klimov. Proximal policy optimization
algorithms. CoRR, abs/1 hiss/1077.00 773477 2017.
[37] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. E. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.0132. 2020.
[38] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Ba- tra, P. Bhargava, Bhosale, Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, Hosseini, R. Hou, H. Inan, M. Kardas, Kerkez, M. Khabsa, Kloumann, A. Korenev, Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, Mishra, Molybog, Nie, Poulton, Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan,
12

---

P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR. 0bs/2307.092888 2023.
[39] M. Völske, M. Pouthast, S. Syed, and B. Stein. Tl:dr: Mining reddit to learn automatic summarization. In AFIS@EENNP 2017.
[40] C. Wang, Y. Jiang, C. Yang, H. Liu, and Y. Chen. Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. ICLR, 2024.
[41] H. Wang, W. Xiong. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406. 12845, 2024.
[42] J. Wu, Y. Xie, Z. Yang, J. Wu, J. Chen, J. Gao, B. Ding, X. Wang, and X. He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. CoRR, abs/2407.078800, 2024.
[43] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu. Slay preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024.
[44] M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. Less: Selecting influential data for instruction tuning. 2024.
[45] Z. Yuan, H. Yuan, C. Tan, W. Wang, Huang, and Huang. RRHF: rank responses to align language models with human feedback without tears. In NeurlPS, 2023.
[46] O. Zhang, M. Wu, J. Bayrooti, and N. D. Goodman. Temperature as uncertainty in contrastive learning. CoRR, abs/21 10.04403, 2021.
[47] Y. Zhao, R. Joshi, Liu, M. Khalman, M. Saleh, and J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, 01//2205510225 2023.
[48] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. LIMA: less is more for alignment. In NeurlPS, 2023.
13

---

A Experiment
A.1 3-DPO Implementation Details and Hyperparameters
3-DPO is relatively straightfoward to implement; The full algorithm is summarized in Algorithm
Algorithm B-Direct Preference Optimization
Require:
Preference dataset D, batch size b, constraint coefficient So. selection ratio scaling factor a, and learning rate f/.
1: Initialize model Tea with supervised finetuning on D.
2: while not converged do
3: Sample batch B = (xx(), y {)) from D.
4: Compute the individual reward discrepancy M, = r(y :x(0)) ;
r(y) ; x()).
5:
Update the threshold Mu and using Equations (7) and (9).
6:
Sample x p samples without replacement based on p(M,) in Equation (8). Compute the batch-level using Equation (5).
7:
8:
Compute the loss using the Equation (4).
9: Compute the gradient and update the model 0: 0.-1 -
10: end while
return Final model ITg.
Unless noted otherwise, we use 3 = 0.1, batch size 64, m = 0.9 to ensure the stability of the global M; estimation, p 0.8 to filter 20% uninformative samples, and the Adam optimizer with a learning rate 5e by default. We carried out all computational tasks on a suite of four 80GB A100 GPUs. For the Pythia-410M model, we batch size of 128, while the rest of the parameters remain the same.
In examining the arena of single-turn dialogue, our experimental framework leverages Pythia- 2.8, Pythia-1.44 and Pythia-410M for empirical analysis using the Anthropic-HH dataset. Given the absence of pre-existing Supervised Fine-Tume Tuning (SFT) model tailored for this dataset, we fine-tune an accessible language model exclusively with preferred completions to sculpt our SFT model. Turning focus the domain of summarization, summarization dataset, enriching our research with human preferences as documented by the study [37]. Our methodology here incorporates an SFT model meticulously fine-tuned on expert-composed summaries of forum posts 3, operating within the TRLX framework for Reinforcement Learning from Human Feedback (RLHF), as introduced by Havrilla et al. [18]. The human preference dataset was gathered by Stiennon et al. [37] on samples from a different, but comilarly-trainee SFT model.
employed the Reddit TL:DR
A.2 Mixture of low gap and high gap
In our previous discussion, we identified low gap dataset, constituted by pairs of responses generated from the HH dataset. Given that these responses originate from the same model, can infer that they represent semantically similar answers, hence the designation low gap. Concurrently, we maintain the positive samples constant while selecting negative samples generated by Pythia 2.8B model. The significant performance disparity between the models results in a considerable variation in the quality of these negative samples, which we refer to the high gap. We mix these types of data in varying proportions to mimic the distribution of data in real-world scenarios. We label this approach the "mixture experiment." " To facilitate better comparison of the distributions across different ratios, we proceed to illustrate our findings with the following graph:
Figure 6 clearly illustrates that as the mixture ratio mncreases-the is, the proportion of high gap data rises-the dispersion the overall dataset broadens. Conversely, a decrease in the mixture ratio, corresponding to elevated presence of low gap data, results in more concentrated distribution of reward discrepancy.
_st
14

---

mixture-1.0 mixture=0.3
mixture=0.1
mixturc-DD.4
mbxture-0.2
Individual Discrepancy
04
Figure 6: Distribution of individual reward discrepancies following the Pythia-2.8B model.
Pythia 2088
Pythia-1449
Pyth
94
Win Rata
8=0.3
8-0.9
000
75
31
Win (N6)
12
2.4
Figure 7: Performance comparison across different & values and P values for three different model sizes (Pythia-2.8B, Pythia- 1.4B, and Pythia-410M) on the Anthropic HH dataset using GPT-4 as the evaluator. Each subplot represents the win rate for varying parameters 0.1, 0.3, and 0.9 with exponential smoothing.
A.3 Hyperparameter Sensitivity
In this section, we investigate the impact of key hyperparameters in the 3-DPO methodology, using the Anthropic HH dataset and leveraging models such as Pythia-2.8B and GPT-4 for evaluation. Specifically, we examine the influence of varying the hyperparameters a and the filtering ratio p. The parameter is explored across broad spectrum of values including 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.5, 1.8, and 2.0. Concurrently, the filtering ratio P is investigated at intervals ranging from 0.1 to 0.5, at granularity of 0.1. This comprehensive analysis aims to unravel how these hyperparameters affect the performance and outcomes of the 3-DPO process.
Static 3 vs. Dynamic B. Our results, as depicted in Figure 7, reveal that a dynamic 3 (where / 0) prevails over a static 3 (where a = 0) under an exhaustive range of hyperparameter configurations. This outcome aligns seamlessly with underlying premise: static beta fails to fully leverage model's potential when confronted with varying data quality within a dataset. Furthermore, our observations highlight intriguing trend: as increases, the performance of the B-DPO model initially improves before declining, typically peaking within the interval of 0.6 to 1.0.
Filtering Ratio Sensitivity. As illustrated in the figure 7, varying model sizes exhibit distinct sensitivities to the parameter p. Within the context of the Pythia-2.8B model, p value of 0.3 yields optimal performance, whereas for the Pythia-410M model, value of 0.5 is superior. This can be posited to suggest that smaller models may require more stringent data filtering to perform optimally, whereas larger models possess the increased capacity necessary for learning the most
15

---

effective strategies. This insight provides a significant directive for future research: the value of should be fine-tuned according the specific circumstances of the application.
A.4 The ablation study w.r.t. Mo
In this work, we employed a moving average updating scheme [24] for the updating of Mo. In order investigate the superiority of this configuration, also conducted comparative experiment involving hyperparameter settings. Specifically, My was treated as a constant in the training process, while all other settings remained unchanged. The experimental results are follows:
Table 4: Comparison win rates across varying Mg in 3-DPO.
Mo
0
7
10
Moving Average
3-DPO
53.35
54.00
53.85
55.61
53.19
56.75
57.07
As demonstrated in Table 4, by tuning Mo. we were able to achieve significant performance improve- ments, which approached the performance level of the moving average updating scheme. This clearly underscores the superiority of the moving average updating scheme. On one hand, it obviates the need for an additional manual search process. On the other hand, it facilitates stable performance enhancements.
A.5
Our Methods with Explicit Reward Model
To broaden the application of the 3-DPO strategy for alignment and address the limitations of DPO's implicit reward model, we incorporated two external reward models (RMs):
We utilized IIm Mim-blender/PiiRM [22] to score the chosen and rejected responses, denoted as
Tw. PairRM PaidRM
We adopted the v0.2 Llarra-Instruct Instruct setting [26] by employing RLHFlowAAmoRM-llmmms
v0.1 [41] as the reward model to rank responses, denoted as Amm.AMRMM: Ft,ArmoRM
The other algorithmic processes (refer to Appendix Algorithm 1) remain unchanged, with modi- (6)
fications made only to Pairi "w PairRM PMPPir and Mi,ArmoRM AMM.RMM AmmoR
(4)
(8)
(4)
For comparison with baselines, we assess our models using one of the most popular open-ended
instnuction-folloiing benchmarks: AlpacaEval 2. The detailed results are as follows:
Table 5: Comparison of different methods on Llama3- (8B) with explicit reward model Method
Llama3-Istruct (8B) Llama3-Instruct (8B)
LC (%)
WR (%)
DPO (Implicit RM) 3-DPO (Implicit RM)
40.44
37.38
43.38
38.21
SimPO (Implicit RM)
44.38
38,97
3-SimPO (Implicit RM)
46.03
40.18
SimPO (PairRM) 3-SimPO (PairRM, Instance-Level)
44.70
38.98
43.84
38.54
3-SimPO (PairRM, Batch-Level)
45.65
39.76
SimPO (ArmoRM)
53.70
47.50
3-SimPO (ArmoRM, Instance-Level)
49.05
45.47
3-SimPO (ArmoRM, Batch-Level)
54.86
49.66
From the results above, observed the following:
The proposed dynamic strategy demonstrates strong generalizability. Both 3-D,SimPO consistently yield stable performance improvements across implicit and explicit RMs.
16

---

Batch-level performance is crucial. In explicit RM settings, batch-level dynamic B consistently outperforms instance-level dynamic B.
A.6 GPT- prompts for computing summarization and dialogue win rates
A fundamental element of our experimental framework involves the assessment of win rates facilitated by GPT-4. In this segment, we delineate the prompts employed ascertain win rates pertinent to our summarization and dialogue-oriented investigations. For the entirety of our experimental endeavors, we utilize GPT-4. important note that the sequence which summaries or responses are presented is randomized for each evaluation.
Summarization GPT-4 win rate prompt.
Which of the following summaries does better job of summarizing the most \ important points in the given forum post?
Post:
<post>
Summary A: <Summary A>
Summary B:
<Summary B>
FIRST provide a one-sentence comparison of the summaries, explaining which you prefer and why. SECOND, on new line, state only "A" or "B" to indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation Preferred: <"A" or "B">
Dialogue GPT-4 win rate prompt.
For the following query to a chatbot, which response is more helpful?
Query: <the user query>
Response A:
<either the test method baseline>
Reaponse B: <the other response>
FIRST provide a one-sentence comparison of the two responses and explain which you feel more helpful. SECOND, on a new line, state only "A" or "B" to indicate which response is more helpful. Your response should use the format:
Comparison: <one-sentence comparison and explanation>
More helpful: <"A" or "B">
Broader Impacts
This paper presents work whose goal is advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
NeurIPS Paper Checklist
1. Claims
17

---

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: see abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
18

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: see Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available http:///tthubbco///uu///ggg/////
Guidelines:
19

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
sssssniiiccciooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https: //ipp.cccpubliiii wwtppcccpccccccecccccccccccccelt htalllic/guiesss CCodeSubmdasac ssionnolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: see Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
20

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We carried out all computational tasks on suite of four 80GB A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : htt///neurippppe http:///a.....ccca/c////t//iieea
ines?
Answer: [Yes]
Justification: Ethics.
Our research has been conducted with strict adherence to the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
21

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The answer NA means that the paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The answer NA means that the paper does not release new assets.
22

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "92jXgtUyfM", "reviewer": "Reviewer_fXTA", "summary": "The authors's goal is to introduce adaptive schedules for the KL regularization $\\beta$ in RLHF. This is a useful quality of life improvement with high potential impact on the final performance of an RLHF algorithm (similarly to how adaptive learning rate schedules are crucial in optimization).\n\nAs a guiding principle for the adaptive schedule the authors propose to adjust $\\beta$ according to the \"quality\" of the data used in the update.  This decision is driven by preliminary experiments that show that:\n- when RLHF is performed on low gap data (i.e. a human rater barely prefers a completion $y$ over $y'$ and there is no strong quality difference or preference towards $y$ or $y'$) the authors deem the data as high quality, and show that lowering the $\\beta$ is beneficial as less regularization allows the model to fit the subtle difference between close pairs of examples\n- when RLHF is performed on high gap data (i.e. a human rater strognly prefers a completion $y$ over $y'$) the authors deem the data as low quality, and show that increasing the $\\beta$ is beneficial as more regularization prevents the model from overfitting only to the preferred sample at the detriment of pre-trained knowledge already stored in the weigths\n\nBased on this quality principle, the author propose to measure the gap using a reward model to compute a reward discrepancy, and try to adjust $\\beta$ as a function of the sample's reward discrepancy. This turns out to be too unstable and sensitive to outliers, so the authors propose a number of strategies to stabilize the learning while retaining an adaptive $\\beta$\n- adding a baseline reward discrepancy to \"center\" the $\\beta$ updates\n- moving from a per-sample reward discrepancy to batch statistics to smoothen the effect of outliers while retaining some adaptivity\n- leveraging the reward discrepancy as a filtering measure, and simply skipping the updates on high-discrepancy (i.e. lowest quality) samples to reduce variance in the updates \n- various combinations and ablations of the above\n\nThe authors show the usefulness of their approach on small to medium scale experiments using helpfulness and summarization text tasks, showing that an LLM judge prefers sentences from a policy fine-tuned with $\\beta$-DPO to those from a DPO-finetuned policy, and in some cases even over the golden ground truth.", "questions": "How is the reward model constructed and used? (this is my main question and I am open to revise my score based on the answer) \n\nCan you strengthen your support for the gap principle beyond the empirical observation of Figure 1?\n\nIt seems pretty convincing that a dynamic schedule outperforms a fixed one. There is a missing ablation showing that the one you proposed is strongly preferrable over others (e.g., high gap -> low $\\beta$, low gap -> high $\\beta$). Can you show that different schedule do not work in your experiments (adapting the filtering if necessary)? Alternatively, can you show which schedules work for which datasets (while keeping that in general dynamic schedules are better than a fixed schedule)?", "limitations": "The main limitations as I see now are the lack of clarity on the reward model, and the limited support on why this specific schedule is to be preferred over other dynamic schedules.\n\nBeyond those, the authors correctly identify important other open questions in the limitation settings. I'd like to highlight among those two that I think would be very relevant in increasing the impact of the paper and make it jump several levels in my evaluation:\n- Testing the approach beyond DPO. Just like learning rate schedules generalize across optimizers, it's crucial for a good $\\beta$ dynamic schedule to generalize across PO methods like c-DPO/IPO/SPO/RPO etc...\n- Automated parameter tuning. The authors make a good effort on this aspect (automatic choice for $\\beta$ and variance estimation) with only one hyperparameter left ($M_0$) to tune despite moving from a simple constant schedule to a dynamic one. However, a true hyperparameter-free method would be much more valuable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Adaptive schedules have always been pretty impactful in practice (although hard to tune), so I would consider the significance of this paper above average.\n\nThe principle used to quantify \"quality\" is both a strength and a weakness. The strength are:\n- conceptually simple, and quite related to existing concept in the literature (e.g. class margins in regression as a similar quantity was used in SLiC [1r] ) which means there is literature and past insights that can guide both usage and future development\n- computationally reasonable to compute (when using an implicit or explicit reward model) and in some cases even free (DPO implicit reward model, when the reward is part of the dataset)\n- task-and-data-specific, so avoids a lot of common pitfalls that happen with e.g. schedules that depend only on features (e.g. maximum data norm) or not even that (e.g. linear warm-ups)\n\nThe paper is very clear and easy to follow.\n\nExperiments include useful ablations, and in general are detailed and show improvement over baseline. A few aspects could be improved (more details in weaknesses)", "weaknesses": "The main (and critical weakness) of the paper is that while it extensively relies on a reward model (e.g. for computing reward discrepancies used in scheduling $\\beta$ and filtering), I could not find details on how the authors recommend this reward model is implemented in practice. The two ways I can imagine.\n\nThe first is to pre-train a reward model as described in section 3 (e.g. by fitting the model in Eq. 2). If this is the case there are a bunch of disconnects in the paper:\n- this disconnects the approach from DPO (which builds its own implicit reward model), resulting in different reward estimates for the tuning of $\\beta$ and for the PO procedure\n- if the reward model is trained on the same offline data used in the $\\beta$-DPO run, then negative reward discrepancies should be treated differently than positive discrepancies, since r(y_w) < r(y_l) indicates either an outlier but possibly also inaccuracy of the reward model (and in general should not happen as often as described by the experiments) \n\nThe second is to use directly the implicit reward model induced by the policy trained by DPO:\n- this solves the disconnect between DPO and discrepancy rewards, but note that the accuracy of the intermediate implicit models is bound to be poor, as the goal of DPO is to start with a poor policy (poor reward model) and improve it over time, relying on the supervised signal of offline winners and losers (which does not depend on neither policy or reward model) to drive this improvements. as a consequence using the intermediate rewards for additional online choices (tuning $\\beta$, filtering) might be not very sound in theory (but maybe works in practice?)\n- an implicit DPO reward model makes the whole process even more non-stationary\n\nAt an empirical level, while the combination of dynamic schedule and filtering proves effective, it is unclear when only one or both techniques are necessary. This weakens the contribution as $\\beta$-DPO is not only DPO+a dynamic schedule but DPO+dynamic schedule + filtering. As such, in the natural attempt to transfer the schedule to other PO methods (e.g. KTO, IPO, c-DPO, RPO, etc..) one must also find a way to transfer the filtering technique, making the contribution less general. \n\nThe principle used to quantify \"quality\" is both a strength and a weakness. The weaknesses are:\n- the guidelines proposed that connect $\\beta$ schedule to reward gap (high gap -> high $\\beta$, low gap -> low $\\beta$) seem entirely driven by empirical observations in Figure 1, and are not supported by any further insight or justifications for why this might be the one correct choice when moving away from a constant schedule. More insight would be valuable, where for example DPO gradients are rescaled by a $(1 + e^{M_i})^{-1}$ which means that $\\beta$ is somewhat rescaled according to the gradient norm, although in a non-monotonic way. While the proposed dynamic schedule might sound reasonable (or at least I found it so), the exact opposite (high gap -> low $\\beta$ since the high margin gives us confidence to fit fully, low gap -> high $\\beta$ to avoid overfitting random perturbations on the reward) also sounds reasonable. The soundness of this \"quality\" measure and its role in tuning $\\beta$ should be either evaluated on ablations against other dynamic schedules/principles as baslines, or supported theoretically before it can be fully considered a strong contribution.\n- according to the authors introduction, the \"quality\" has a (at least) bimodal distribution in the dataset (high gap data and low gap data), however outlier filtering is performed using only an unimodal 3sigma principle, which looks counterintuitive to me (e.g., aren't all high gaps pairs technically outliers from the low gap sample point of view and vice-versa?)  \n\nExperiments are good, but without better explainations on how the reward is computed they are hard to evaluate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for your kind review. We are glad that you found our paper clear and easy to follow. We provide detailed answers to your comments below.\n\n**Q1: How is the reward model constructed and used?**\n> A1: Thank you for pointing this out. **We directly use the implicit reward model induced by the policy trained by DPO, where the reward discrepancy in DPO is expressed as: $ \\beta \\log (\\frac{\\pi\\_\\theta (y\\_w \\mid x) }{\\pi\\_{\\text{ref}}(y\\_w \\mid x)}) - \\beta \\log (\\frac{\\pi\\_\\theta (y\\_l \\mid x) }{\\pi\\_{\\text{ref}}(y\\_l \\mid x)})$.**\n\n**Q1.1: As a consequence, using intermediate rewards for additional online choices (such as tuning and filtering) may lack theoretical soundness.**  \n> **A1.1:** In the early stages of training, $\\beta$-DPO behaves similarly to DPO (see `REBUTTAL Figure 2 Middle`). Due to the low accuracy of the intermediate implicit models, the margin between winners and losers is minor, resulting in $\\beta\\_{\\text{batch}} \\approx \\beta\\_0$. However, in the later stages of training, as the margin begins to exhibit significant differences, the advantages of a dynamic $\\beta$ become apparent.\n\n**Q1.2: An implicit DPO reward model introduces greater non-stationarity into the process.**  \n> **A1.2:** This question raises a pertinent point. An implicit DPO reward model can lead to instability in the estimation of dynamic $\\beta$, thus making the entire training process more non-stationary. To address this issue, we propose that batch-level calibration is essential. As demonstrated in Table 2 of the initial manuscript, instance-level calibration on dynamic $\\beta$ results in a substantial performance drop. In contrast, batch-level calibration enhances stationarity and accentuates the effects of dynamic $\\beta$. Additionally, Figure 5 (Right) illustrates that instance-level calibration exacerbates the influence of outliers.\n\n**Q2: It is unclear when only one or both techniques are necessary.**\n> A2: As demonstrated in Table 1 of the original manuscript, we observe that in a smaller model (410M), the improvement of data filtering is more significant, while in a larger model (2.8B), the improvement of dynamic $\\beta$ is more significant. Additionally, the dynamic schedule can also improve other filtering methods (Figure 5 Left). **Since our filtering method relies only on implicit reward discrepancy, it also works with other PO methods (Figure 5 Middle).**\n\n**Q3: Can you strengthen your support for the gap principle beyond the empirical observation of Figure 1?**\n> A3: We also attempt the exact opposite (high gap -> low $\\beta$; low gap -> high $\\beta$). Refer to `REBUTTAL Figure 1, REBUTTAL Figure 2 Right` for the experimental results. We find that across multiple datasets and models, the current strategy remains optimal.\n\n**Q4: Outlier filtering is performed using only an unimodal 3-sigma principle**\n> A4: Utilizing the 3-sigma principle is primarily to reduce the bias in $\\beta$ estimation and thus enhance training stability. Intuitively, high-gap samples often carry low information content, whereas low-gap samples might still be noisy, indicating that winners and losers may not be well distinguished (label flipping). We hope this helps you gain an intuitive understanding. A more detailed analysis can be found in the original manuscript, Section 4.1 (lines 157-170).\n\n**Q5: Testing the approach beyond DPO**\n>A5: Thank you for pointing this out. **First, we demonstrate Dynamic $\\beta$ Enhancement across IPO, KTO, and SPPO in Figure 5 (Middle) of the original paper.**\nTo expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[1], which surpass c-DPO/RPO etc. We perform $\\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1].\n> Please refer to `REBUTTAL Table 1`. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.\n\n\n**Q6: Automated parameter tuning.**\n> A6: **In most scenarios, setting $\\alpha = \\frac{2}{M_0}$ yields stable performance improvements, where $M_0$ can be estimated using a moving average updating scheme (refer to Equation 7).** This is informed by the formula $\\beta\\_{\\text{batch}} = [1 + \\alpha(\\mathbb{E}\\_{i \\sim \\text{batch}}[M\\_i] - M\\_0)]\\beta\\_0$, resulting in an overall change range of $[\\frac{2\\mathbb{E}\\_{i \\sim \\text{batch}}[M\\_i] - M\\_0}{M\\_0}]\\beta\\_0$, which normalizes based on $M\\_0$ over the foundation of $\\beta\\_0$.\n\n| | HH | TLDR | \n|-------------------|-------------------|-------------|\n| DPO               | 51.01               |   32.45       |\n| $\\beta$-DPO               | 57.68              | 51.67|\n| $\\beta$-DPO ($\\frac{2}{M\\_0}$)              | 58.02               | 51.32 | \n\n> To substantiate this perspective, we present performance in the above table, demonstrating that our setting achieves significant enhancements across various datasets and models compared to DPO, **without imposing additional pressure for hyperparameter searches.** We appreciate your concern; while we believe that further theoretical consolidation is a meaningful future endeavor, we maintain that the $\\beta$-DPO approach remains valuable, offering a straightforward (not overly reliant on hyperparameter tuning) and effective (stable performance enhancements) new paradigm for fine-tuning large models and studying data quality.\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng and Mengzhou Xia and Danqi Chen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7s0xA9t4BX", "reviewer": "Reviewer_3526", "summary": "The paper presents an improvement in Direct Preference Optimization (DPO), a method for aligning and fine-tuning large language models (LLMs) based on human preferences. The authors identify two critical factors affecting DPO performance: the parameter $\\beta$ and the quality of preference data. The existing literature has largely neglected the joint impact of these factors. This study investigates how varying $\\beta$ and preference data quality influence DPO outcomes. It finds that optimal $\\beta$ values depend on the informativeness of pairwise data. Based on this insight, the authors propose enhancements to DPO that involve batch-level dynamic $\\beta$ calibration and $\\beta$-guided data filtering. The efficacy of these improvements is empirically validated across two NLP tasks (dialogue and text summarization) using models of different sizes.", "questions": "In Figure 5a, including the $\\beta$-guided filtering strategy on the x-axis alongside various gradient-based filtering strategies would be valuable. This addition would help assess its effectiveness compared to existing gradient-based approaches.", "limitations": "The authors acknowledge the limitations of their work in the conclusion. While the focus is on the technical improvement of preference alignment techniques for LLMs, a detailed discussion on the potential societal impact of these advancements would be a worthwhile addition.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well-written and easy to follow.\n\nThe authors make an interesting empirical observation about DPO dynamics: an increase in $\\beta$ improves performance when the gap in pairwise preference data is large but degrades performance when the gap is small. \n\nThe authors perform a thorough empirical analysis, including ablation and compatibility studies, to validate their proposed improvements. \n\nThe merits of the proposed enhancements to DPO are:\n(1) Easy to implement, with no additional computational overhead. \n(2) Compatible with other preference data filtering strategies and DPO variants like IPO and KTO. \n(3) Utilizes the \"running\" reward discrepancy instead of relying on teacher/gold rewards.", "weaknesses": "Despite the comprehensive empirical analysis, the experiments are limited to specific datasets and model size ranges. Given the potential impact on the DPO literature, broader verification across diverse datasets and model sizes would strengthen the claims. \n\nThe dynamic $\\beta$ approach could introduce instability by using the \"running\" reward discrepancy instead of teacher/gold rewards. A comparative analysis with methods that utilize teacher/gold rewards would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.\n\n**Q1: The experiments are limited to specific datasets and model size ranges.**\n> A1: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[1]. We perform $\\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1].\n> Please refer to Table below:\n\n\n| Method | Mistral-Instruct (7B) | Mistral-Instruct (7B) | Llama3-Instruct (8B) | Llama3-Instruct (8B) |\n|--------|----------------------|-----------------------|---------------------|---------------------|\n|        | LC (%)               | WR (%)                | LC (%)              | WR (%)              |\n| DPO    | 20.98                | **21.60**             | 40.44               | 37.38               |\n| $\\beta$-DPO | **23.56**       | 20.42                 | **43.38**           | **38.21**           |\n| SimPO  | 28.50                | 30.56                 | 44.38               | 38.97               |\n| $\\beta$-SimPO | **30.48**     | **32.13**             | **46.03**           | **40.18**           |\n> Table: AlpacaEval 2 results under the Mistral-Instruct (7B) and Llama3-Instruct (8B). LC and WR denote length-controlled and raw win rate, respectively. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.\n\n**Q2: Comparison with gold reward.**\n> A2: Thank you for pointing this out. In fact, computing the gold reward for the training set of 160k samples 1) greatly increases computational complexity and 2) makes the gold reward highly sensitive to the particular model used for annotation. Such an approach is difficult to generalize to arbitrary data scenarios.\nAdditionally, to reduce the instability of \"running\" reward discrepancies, $\\beta$-DPO utilizes batch-level calibration. Figure 5 (right) and Table 2 clearly demonstrate its superiority. This aligns with our original intent for this work: to develop a simple-to-implement, highly scalable dynamic $\\beta$ strategy.\n\n**Q3: Including the $\\beta$-guided filtering strategy.**\n> A3: Thank you for this suggestion. We have included the complete comparison chart in `REBUTTAL Figure 2 Left`. It can be clearly observed that: 1) the dynamic $\\beta$ strategy can adapt to various data filtering methods, and 2) the $\\beta$-guided filtering proposed in this paper remains optimal.\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng and Mengzhou Xia and Danqi Chen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hihyO1bu4X", "reviewer": "Reviewer_UzDb", "summary": "This work proposes two techniques to improve the performance of the popular DPO alignment algorithm. The first technique proposes a strategy to dynamically adapt the $\\beta$ coefficient of DPO, which controls the strength of the KL penalty with respect to the reference policy. The second technique proposes a filtering mechanism that uses reward margins (the difference between the rewards of the preferred and dispreferred response) to filter the data being sampled for DPO training. Results show that the two techniques complement each other and lead to increased win rates in a range of settings.", "questions": "1.\tThe paper makes an implicit assumption that low-gap examples are high-quality. However, I do not think this is generally true. For example, if both the preferred and dispreferred response are rated very poorly by the reward model, they will also have low-gap. Having clearer writing in this regard and a general explanation of this would be useful.\n2.\tOn what criteria does the paper make the claim that Anthropic HH is a low-gap dataset? In my experience, the dataset contains numerous pairs that are easily separable as well. Is any dataset where both $y_w$ and $y_l$ are sampled from the same policy considered low-gap?\n3.\tIn contrast to DPO, this method necessitates learning/using an off-the-shelf reward model. There are very few details about this reward model in the paper. Currently, the paper assumes that the reward gaps are generally available, however this is often not the case. It would also be good to have a few sentences acknowledging this difference between DPO and the proposed method. \n4.\tHow is beta being used in data filtering? $\\beta$ is not mentioned anywhere in Section 4.2.2. The data filter seems like a normal reward margin-based filter.\n5.\tHow is the selective filtering ablation run? What does arranging the gradients mean? Is gradient referring to the reward margin? \n6.\tIs m (the momentum coefficient) also a hyper parameter? The claim of the paper is that $\\alpha$ is the only hyperparameter. \n7.\tCurrent results are only on the Pythia class of models and it would be interesting to see the same improvements on a different class of models. I think that is necessary to substantiate the claim of model agnosticity.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper is well-organized and technically sound. The general flow of the paper is smooth. The paper has an appropriate number of citations and properly details existing work in the related work section. \n2.\tThe method presented is simple to understand and easy to integrate into any DPO implementation. \n3.\tThe results are promising and are giving consistent improvements in a range of scenarios.", "weaknesses": "1) Technical and experimental details need to be clearly clarified (refer to questions). Currently, I am unable to see how the techniques presented are related, they seem to be independent.\n2) Writing can be improved. In particular, there are very few details about the reward models used in the paper and no acknowledgment of the fact that in contrast to vanilla DPO, the proposed method is reliant on access to a reward model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.\n\n**Q1: There are very few details about the reward models used in the paper**\n> A1: Thank you for pointing this out. **We directly use the implicit reward model induced by the policy trained by DPO, where the reward discrepancy in DPO is expressed as: $ \\beta \\log (\\frac{\\pi\\_\\theta (y\\_w \\mid x) }{\\pi\\_{\\text{ref}}(y\\_w \\mid x)}) - \\beta \\log (\\frac{\\pi\\_\\theta (y\\_l \\mid x) }{\\pi\\_{\\text{ref}}(y\\_l \\mid x)})$.** \n\n\n**Q2: Why are low-gap examples considered high-quality?** \n> A2: We appreciate your concern regarding this matter. First, it is crucial to clarify that both the low-gap and high-gap examples are derived from data of a certain quality, rather than from extremely poor quality sources characterized by meaningless chatter.\n**We posit that datasets of extremely poor quality would not be utilized for training at this stage.** Furthermore, we can categorize the cases of $y\\_w$ and $y\\_l$ into the following four types:\n\n| Quality of $y\\_w$ | Quality of $y\\_l$ | Description | Behavior of $\\beta$-DPO |\n|-------------------|-------------------|-------------|-------------------------|\n| Low               | Low               |   High-quality, hard discriminated pairs       | Lower $\\beta$ → Assertive updates |\n| Low               | High              | Label flipping, noise | Larger $\\beta$ → Cautious updates |\n| High              | Low               | Easily discriminated pairs | Larger $\\beta$ → Cautious updates |\n| High              | High              | High-quality, closely matched pairs | Lower $\\beta$ → Assertive updates |\n\n> To further elucidate, both preferred and dispreferred responses are low-quality, yet they serve a beneficial purpose. We follow the settings of controlled sentiment generation in DPO. With access to the ground-truth reward function (a sentiment classifier), we control the quality of $y\\_w$ and $y\\_l$, where high-quality response is derived from a fine-tuned GPT-2 large model and low-quality response from an unrefined GPT-2 large model. The following table, derived from the IMDB test dataset, illustrates the KL-divergence and the associated rewards:\n\n|   KL-divergence     | 2  | 4  | 6  | 8  | 10  | 12  | 14 |\n|---------------------|----|----|----|----|----|----|----|\n| Both High-quality   | 65.68 | 89.16 | 94.26 | 96.24 | 97.74 | 98.91 | 98.58 |\n| Both Low-quality    | 63.72 | 80.42 | 84.00 | 85.00 | 81.80 | 82.20 | 81.75 |\n| High-quality $y\\_w$, Low-quality $y\\_l$ | 45.93 | 39.55 | 39.46 | 36.59 | 29.98 | 30.71 | 31.27 | \n> The results indicate that low-quality responses can be even more meaningful for model improvement than high-gap examples.\n\n**Q3: Why is the Anthropic HH dataset considered low-gap? Are there better datasets available?**\n> A3: The terms \"low-gap\" and \"high-gap\" are employed in a relative context. In comparison to the negative samples generated by SFT, the overall distribution of Anthropic HH exhibits smaller differences. To support this assertion, please refer to Appendix Figure 6, where positive samples originate solely from the HH dataset, while negative samples consist of a mix from SFT-generated and original negative samples from Anthropic HH. As the proportion of original Anthropic HH samples increases (i.e., the mixture ratio decreases), the distribution of reward margins becomes more concentrated.\n\n**Q4: How is beta utilized in data filtering?**\n> A4: In this work, the reward margin informs both the choice of $\\beta$ and the framework for data filtering, establishing a bridge between $\\beta$ and data filtering. The specific expression is:\n$p(\\beta\\_i)= \\frac{1}{\\sqrt{2\\pi}\\sigma}\\exp(-\\frac{(\\beta\\_i/\\beta\\_0-1)^2}{2\\sigma^2\\alpha^2})$\n\n**Q5: How is the selective filtering ablation conducted?**\n> A5: We apologize for any misunderstanding in our expression. Your interpretation is correct; we operate based on the reward margin. We will revise this section for clarity. In the DPO formulation, the sample gradients are strictly negatively correlated with the reward margin, implying that a larger reward margin corresponds to a smaller gradient. Therefore, during loss computation for each batch, we sort samples based on their reward margins and conduct selection accordingly. In the experiment corresponding to Figure 5 (Left), \"Filter Tail 20%\" refers to the filtering of the 20% of samples with the largest gradients, which corresponds to those with the smallest 20% reward margins, and vice versa. \n\n**Q6: Is m (the momentum coefficient) also a hyperparameter?**\n> A6: m (the momentum coefficient) is indeed a hyperparameter. However, in all our experiments, we consistently set it to **0.9 without any hyperparameter search**. To further illustrate, we present additional comparisons in `REBUTTAL Table 2`, showing that this parameter has a minimal effect on model performance.\n\n**Q7: Current results are only on the Pythia class of models**\n> A7: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models [1]. We perform beta-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For baseline comparisons, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1].\n> Please refer to `REBUTTAL Table 1`. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng and Mengzhou Xia and Danqi Chen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIWyqRg4rw", "reviewer": "Reviewer_uGzu", "summary": "This paper studies the relation between the best $beta$ parameter of DPO and the data quality. Motivated from the observation, the authors propose a way to dynamically choose the $beta$ at the batch level. The evaluation shows the proposed method improves DPO and its variants' (IPO, KTO, SPPO) performance on Antrhopic-HH dataset.", "questions": "* $\\beta_{batch}$ is defined in Eq. (6). What is a usual range of $\\beta_{batch}$ in experiment? Is it possible that $\\beta_{batch}$ goes to negative?\n* In section 4.2.2 data filtering, the authors adopt a sampling without replacement with weights defined in Eq. (8). This cannot completely remove the outlier (i.e., there are still small chance for the outliers to be selected). What if you sort the data in an increasing order according to $|M_i-M_0|$ and select the top $|data|\\times \\rho$ instances? (this can completely remove the outliers) How do you compare your method with this?", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method is motivated from empirical finding that the optimal beta tends to be positively correlated with the reward discrepancy.\n* The proposed method is based on highly principled ideas such as batch average, exponential moving average, gaussian pdf.\n* Experiments demonstrate the effectiveness of the proposed method over DPO, IPO, KTO, SPPO", "weaknesses": "* The experiment mainly relies on Anthropic HH dataset, containing 170000 dialogues. It would be nice if the author can verify the results on another dataset with different domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.\n\n**Q1: It would be nice if the author can verify the results on another dataset with different domains.**\n> A1: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[1]. We perform $\\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1].\nPlease refer to Table below:\n\n\n| Method | Mistral-Instruct (7B) | Mistral-Instruct (7B) | Llama3-Instruct (8B) | Llama3-Instruct (8B) |\n|--------|----------------------|-----------------------|---------------------|---------------------|\n|        | LC (%)               | WR (%)                | LC (%)              | WR (%)              |\n| DPO    | 20.98                | **21.60**             | 40.44               | 37.38               |\n| $\\beta$-DPO | **23.56**       | 20.42                 | **43.38**           | **38.21**           |\n| SimPO  | 28.50                | 30.56                 | 44.38               | 38.97               |\n| $\\beta$-SimPO | **30.48**     | **32.13**             | **46.03**           | **40.18**           |\n\n> Table: AlpacaEval 2 results under the Mistral-Instruct (7B) and Llama3-Instruct (8B). LC and WR denote length-controlled and raw win rate, respectively. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.\n\n**Q2: What is a usual range of beta in experiment? Is it possible that goes to negative?**\n\n> A2: Our initial $\\beta\\_0$ is set to 0.1, and the range of beta in experiments falls within [0.0, 0.4]. The specific range of variation is detailed in `REBUTTAL Figure 2 Middle`.\nExperimentally, $\\beta\\_{\\text{batch}}$ can become negative. According to Equation 6, a negative value implies that the reward discrepancy is negative or $\\mathbb{E}\\_{i \\sim \\text{batch}}[M\\_i]$ is significantly lower than the global mean of $M\\_i$, indicating a high likelihood of outliers. Therefore, we apply a cutoff at 0 to ensure training stability, $\\beta\\_{\\text{batch}} = \\max(0.0, \\beta\\_{\\text{batch}})$.\n\n**Q3: What if you sort the data in an increasing order and select top instances?**\n> A3: Thank you for pointing this out. This method represents a hard selection approach, whereas our method utilizes soft selection. The hard selection concept aligns with what we have shown in Figure 5 Left (Filter Head & Tail), and we believe both are effective filtering methods. Both approaches validate that extreme $M\\_i$ values are likely outliers that need filtering.\n\n> **It is important to highlight that this work does not propose a novel filtering method, but we find that filtering enhances stability.** As shown in Figure 5 Left, dynamic scheduling could also improve other filtering methods. We are confident that our dynamic schedule will continue to provide stable performance improvements as better LLM data filtering techniques emerge in the future.\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng, Mengzhou Xia, and Danqi Chen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AKc8gOhI4K", "reviewer": "Reviewer_4RgK", "summary": "This paper proposes a dynamic method to tune the hyperparameter $\\beta$ in DPO according to the data in each batch. The proposed method is tested on Pythia-410M, 1.4B, and 2.8B base models. Compared to vanilla DPO, it performs better on Anthropic HH and Reddit TL;DR summarization datasets.", "questions": "1. $\\beta$ is not a simple learning rate. $\\beta$ is derived from the KL-divergence penalty term. It ensures that the tuned policy is not \"far away\" from the original policy. Thus, it should be based on global information, which is the whole training data in offline RLHF. However, in this paper, the dynamic $\\beta$ is only based on batch-level information. How does the algorithm capture the global information through batch-level data?\n\n2. The proposed algorithm has two hyperparameters $\\beta_{0}$ and $\\alpha$. Is it harder to tune compared to the vanilla DPO with only one hyperparameter $\\beta$?", "limitations": "The authors addressed the limitations of the paper. Here are some additions.\n\n1. The proposed method is only tested on small models, Pythia-410M, 1.4B, and 2.8B. It is hard to evaluate the performances on SOTA (relatively large) models, e.g., 7B or 8B models.\n\n2. The winning rates are only evaluated by GPT-4 instead of real human groups. But this is a common issue in most LLM studies, and may be acceptable for research.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 4, "contribution": 2, "strengths": "1. The topic is meaningful. The method of tuning hyperparameter $\\beta$ potentially influences the RLHF or alignment research if the work is solid.\n\n2. The presentation is excellent. The paper is easy to read.", "weaknesses": "The proposed method is not sound, neither in theory nor experiments.\n\n1. The experiences are only based on small models, Pythia-410M, 1.4B, and 2.8B. It is hard to evaluate the performances on SOTA (relatively large) models, e.g., 7B or 8B models.\n\n2. In theory, this paper fails to show how the dynamic terms are derived. It gives the dynamic terms directly with more hyperparameters.\n\n3. In practice, the vanilla DPO only has one hyperparameter $\\beta$. This paper aims to solve the hyperparameter tuning problem. However, it solves this problem by introducing two hyperparameters $\\beta_{0}$ and $\\alpha$. With more hyperparameters and space to tune, it is not surprising to fit better, especially on small models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for your kind review. We are glad that you found our paper meaningful and easy to follow. We provide detailed answers to your comments below.\n\n**Q1: It is hard to evaluate the performances on SOTA (relatively large) models, e.g., 7B or 8B models.**\n> A1: Thank you for raising this concern. To expand our approach to more diverse datasets and model sizes, we follow the current state-of-the-art models SimPO[3]. We perform $\\beta$-DPO with two families of models, Llama3-8B-Instruct and Mistral-7B-Instruct, on UltraChat-200k UltraFeedback. For comparison with baselines, we assess our models using one of the most popular open-ended instruction-following benchmarks: AlpacaEval 2. All settings are consistent with SimPO [1].\nPlease refer to Table below:\n\n| Method | Mistral-Instruct (7B) | Mistral-Instruct (7B) | Llama3-Instruct (8B) | Llama3-Instruct (8B) |\n|--------|----------------------|-----------------------|---------------------|---------------------|\n|        | LC (%)               | WR (%)                | LC (%)              | WR (%)              |\n| DPO    | 20.98                | **21.60**             | 40.44               | 37.38               |\n| $\\beta$-DPO | **23.56**       | 20.42                 | **43.38**           | **38.21**           |\n| SimPO  | 28.50                | 30.56                 | 44.38               | 38.97               |\n| $\\beta$-SimPO | **30.48**     | **32.13**             | **46.03**           | **40.18**           |\n\n> Table: AlpacaEval 2 results under the Mistral-Instruct (7B) and Llama3-Instruct (8B). LC and WR denote length-controlled and raw win rate, respectively. Regardless of whether we use Llama3-8B or Mistral-7B, and whether the loss function is DPO or SimPO, our $\\beta$-{D, Sim}PO strategy consistently demonstrates significant performance improvements. This thoroughly showcases the method's strong generalization ability and excellent scalability.\n\n**Q2: This paper fails to show how the dynamic terms are derived.**\n> A2: Thank you for your suggestion. Our work primarily identifies an empirical relationship between $\\beta$ and data quality, validated across various datasets and model architectures. While we recognize that the theoretical foundations warrant further exploration, the proposed dynamic strategy is nonetheless notable, providing a new and effective paradigm for fine-tuning large models and studying data quality.\n\n**Q3: Is it harder to tune compared to the vanilla DPO with only one hyperparameter?**\n> A3: Our proposed $\\beta$-DPO is a straightforward, efficient, and easily transferable strategy. Firstly, the selection of $\\beta\\_0$ in all instances of $\\beta$-DPO is consistent with the $\\beta$ used in DPO, with a default choice of 0.1 (0.01 for UltraChat-200k), eliminating the need for extensive hyperparameter tuning for $\\beta$. As for the uniquely introduced $\\alpha$, we find:  \n\n> **In most scenarios, setting $\\alpha = \\frac{2}{M_0}$ yields stable performance improvements, where $M_0$ can be estimated using a moving average updating scheme (refer to Equation 7).** This is informed by the formula $\\beta\\_{\\text{batch}} = [1 + \\alpha(\\mathbb{E}\\_{i \\sim \\text{batch}}[M\\_i] - M\\_0)]\\beta\\_0$, resulting in an overall change range of $[\\frac{2\\mathbb{E}\\_{i \\sim \\text{batch}}[M\\_i] - M\\_0}{M\\_0}]\\beta\\_0$, which normalizes based on $M\\_0$ over the foundation of $\\beta\\_0$.\n\n| | HH | TLDR | \n|-------------------|-------------------|-------------|\n| DPO               | 51.01               |   32.45       |\n| $\\beta$-DPO               | 57.68              | 51.67|\n| $\\beta$-DPO ($\\frac{2}{M\\_0}$)              | 58.02               | 51.32 | \n\n> To substantiate this perspective, we present performance in the above table, demonstrating that our setting achieves significant enhancements across various datasets and models compared to DPO, **without imposing additional pressure for hyperparameter searches.** We appreciate your concern; while we believe that further theoretical consolidation is a meaningful future endeavor, we maintain that the $\\beta$-DPO approach remains valuable, offering a straightforward (not overly reliant on hyperparameter tuning) and effective (stable performance enhancements) new paradigm for fine-tuning large models and studying data quality.\n\n**Q4: How does the algorithm capture the global information through batch-level data?**\n> A4: We employed a moving average estimator to estimate the global reward margin, as referenced in Equation 7. This technique has been commonly used in deep learning [1, 2].\n\n**Q5: The winning rates are only evaluated by GPT-4 instead of real human groups.**\n> A5: Thank you for your suggestion. As it stands, the current research landscape indicates that GPT-4 provides the most common strategy for validation. Moreover, the alignment between GPT-4 assessments and human evaluations has been sufficiently established in existing benchmarks (AlpacaEval 2).\n\n[1] Yuan et. al. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. ICML2022.\n\n[2] ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. ICLR 2015.\n\n[3] SimPO: Simple Preference Optimization with a Reference-Free Reward. Yu Meng, Mengzhou Xia, and Danqi Chen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable and insightful feedback. 
- We are encouraged that the reviewers found our paper meaningful (Reviewers $\color{red}{\text{fXTA}}$, $\color{green}{\text{UzDb}}$, $\color{black}{\text{4RgK}}$).
- Moreover, we are grateful that the reviewers found our proposed $\beta$-DPO algorithm simple and effective (Reviewers $\color{blue}{\text{3526}}$, $\color{green}{\text{UzDb}}$, $\color{orange}{\text{uGzu}}$).
- We also appreciate that the reviewers found our paper easy to follow and well-written (Reviewers $\color{red}{\text{fXTA}}$, $\color{blue}{\text{3526}}$, $\color{black}{\text{4RgK}}$).



We also appreciate reviewers pointing out our weaknesses. We address their comments point by point and try our best to respond to them. We hope our response addresses the reviewers' concerns.

The additional experiments in the Rebuttal PDF are summarised as follows:

- In `Rebuttal Figure 1`, we compare the different dynamic schedules on HH and TLDR.
- In `Rebuttal Figure 2 (Left)`, we introduce the $\beta$-guided filtering strategy. 
- In `Rebuttal Figure 2 (Middle)`, we visualize the range of beta in experiments and the corresponding preference accuracy across training steps.
- In `Rebuttal Figure 2 (Right)`, we compare the different dynamic schedules on AlpacaEval.
- In `Rebuttal Table 1`, we extend our approach to include a more diverse set of datasets and model sizes. 
- In `Rebuttal Table 2`, we conduct a parameter sensitivity analysis on $m$.

We have carefully considered the comments and suggestions provided by the reviewers, and we have addressed them point by point in our rebuttal. We believe that our responses adequately address the concerns raised.

Once again, we sincerely thank the reviewers for their valuable feedback, which has significantly contributed to the improvement of our work.

Meta-review: This paper is an interesting first contribution toward the idea of adaptively tuning the amount of regularization in DPO. The authors show that their method of adaptively changing the regularization parameter $\beta$ (for KL-divergence based regularization) combined with an approach for filtering examples is able to significantly boost the performance of many implicit reward model-based approaches including DPO. In terms of theoretical understanding, this work seems quite preliminary. Answering in a principled way the question of whether $\beta$ should be smaller or larger in the case of a small gap (respectively large gap), and getting a more theoretically principled understanding for how to change $\beta$, would be valuable. However, the empirical results are promising, and the intuition provided thus far is a fair (not strong, but fair) start.

The paper overall reads clearly. Yet, multiple well-informed reviewers came away with a very large misunderstanding of the paper: originally, some reviewers thought the authors were using an external reward model, and the authors needed to clarify that in fact they are using an implicit reward model (the one from DPO). After the authors provided this clarification (and in one case, showed additional experiments with external reward models), the reviewers were positive about this paper. I believe the misunderstanding arose due to the writing of the paper, and so I stress that the authors should be as clear as possible about the reward model when revising their work.

One reviewer who was rather negative on this work took issue with the paper having multiple, new hyperparameters. Some of the authors’ responses here seem fair, but there is a risk that researcher degrees of freedom may have played a role in the results of the new techniques being better. Yet, given the consistent improvement seen for multiple implicit reward model-based approaches, several datasets in the original paper, and the new experiments provided during the discussion phase, I believe the above-mentioned risk is low.

Another issue was that the original submission didn’t have sufficiently diverse experiments (in terms of datasets and size of the model). This point was well-addressed in the discussion period with the authors.

Overall, this is a fine piece of work. I believe it is at the threshold of publication at NeurIPS, and it may spark very interesting follow-up works that lead to better theoretical understanding and also even better ways of doing dynamic regularization.
Decision: Accept (poster)