RePO: Understanding Preference Learning Through ReLU-Based Optimization
Authors: Junkang Wu
Kexin Huang
Xue Wang
Jinyang Gao
Bolin Ding
Jiancan Wu
Xiangnan He
Xiang Wang
Venue: NeurIPS 2025 poster ()
Keywords: Preference Optimization
Alignment
Simplicity
Implicit Curriculum
Thresholding
Primary area: deep_learning

Abstract: Preference learning has become a common approach in various recent methods for aligning large language models with human values. These methods optimize the preference margin between chosen and rejected responses, subject to certain constraints for avoiding over-optimization. In this paper, we report surprising empirical findings that simple ReLU activation can learn meaningful alignments even using \emph{none} of the following: (i) sigmoid-based gradient constraints, (ii) explicit regularization terms. Our experiments show that over-optimization does exist, but a threshold parameter $\gamma$ plays an essential role in preventing it by dynamically filtering training examples. We further provide theoretical analysis demonstrating that ReLU-based Preference Optimization (RePO) corresponds to the convex envelope of the 0-1 loss, establishing its fundamental soundness. Our ``RePO'' method achieves competitive or superior results compared to established preference optimization approaches. We hope this simple baseline will motivate researchers to rethink the fundamental mechanisms behind preference optimization for language model alignment.

RePO: Understanding Preference Learning Through ReLU-Based Optimization
Junkang Kexin Huang' Xue Wang? Jinyang Gao Bolin Ding"
Jiancan Wu Xiangnan He Xiang Wang" Unniversity of Science and Technology of China, 2 Allibaba Group IIntitute of Dataspace, Hefei Comprehensive National Science Center MoE Key Lab of BIPC, University of Science and Technology of China {jkwu0909, xiangvang1223, Langnanhe)@@gaiil 38mmail.com com
Abstract
Preference learning has become a common approach in various recent methods for aligning large language models with human values. These methods optimize the preference margin between chosen and rejected responses, subject to certain constraints for avoiding over-optimization. In this paper, we report surprising em- pirical findings that simple ReLU activation can learn meaningful alignments even using none of the following: (i) sigmoid-base based gradient constraints, (ii) explicit regularization terms. Our experiments show that over-optimization does exist, but a threshold parameter 7 plays an essential role preventing by dynamically filtering training examples. We further provide theoretical analysis demonstrat- ing that ReLU-based Preference Optimization (RePO) corresponds to the convex envelope of the 0-1 loss, establishing its fundamental soundness. Our "RePO" method achieves competitive or superior results compared to established preference optimization approaches. We hope this simple baseline will motivate researchers rethink the fundamental mechanisms behind preference optimization for language model alignment.
Introduction
Recent years have witnessed significant advances in aligning large language models (LLMs) with hu- man preferences [1-4]. primary approach, Reinforcement Learning from Human Feedback (RLHF) [5], first trains reward model on preference data and then optimizes the LLM via reinforcement learning. While effective, RLHF's computational costs and training instability [6. 7] have motivated simpler offline alternatives like Direct Preference Optimization (DPO) [6], which bypasses explicit reward modeling. Take DPO as a representative example: it optimizes the alignment margin between preferred and less- dess-preferred response the same prompt, Figure shows. The alignment of each response is quantified via an implicit reward, defined as the log-ratio of the predicted likelihoods under the policy model (i.e., . the LLM being optimized) and reference model (e.g., fixed supervised fine-tuned (SFT) model).
A fundamental challenge in preference learning is over-optimization - where models excessively amplify reward margins between preferred and non- preferred responses, potentially degrading gener- ation quality [8-10]. Several approaches have been developed to mitigate this issue. DPO [6] and SimPO [11] employ sigmoid weighting through log-sigmoid activation that diminishes gradients as reward margins increase, naturally preventing over-optimization. The 3 parameter controls gradient
Work done at Alibaba Group.
Xiangnan He and Xiang Wang are the corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

pacaciaal sate
Loss Function
SimPO
To(rm
(x)
(778
RePO
=F log
[Bog
Blng
Boa (8)
17:8
B logor
Tog th Den
Tog Ca
Laama
11
ReLU (-(
log (tall))))
log (x)
Dempuz
lywl
Figure I:
Comparing preference learning mechanisms. RePO employs a simpler binary thresh- olding mechanism than SimPO and DPO, as highlighted in the shaded box. Despite its simplicity, this mechanism achieves competitive results by naturally preventing over-optimization.
distribution sharpness- larger values produce more binary-like gradients, illustrated in Figure SLiC-HF [7] addresses over-optimization differently by incorporating an SFT regularization term that anchors the model to its initial policy [12]. preventing excessive drift toward maximizing preference signals. These mechanisms effectively balance preference optimization with generation quality preservation, forming the foundation of current preference learning approaches.
Here, we present surprising empirical finding: simple ReLU activation can work well with none of the above strategies for mitigating over-optimization. Our analysis reveals that as parameter B in SimPO approaches infinity, sigmoid weighting naturally converges to a binary thresholding mechanism motivating our exploration of ReLU-based Preference Optimization (RePO). This mechanism uses a single ReLU function with only one hyperparameter y, creating a clear decision boundary that selectively updates sample pairs with insufficient reward margins (Mg < 7) while filtering out well-separated pairs (Ma 2 7)) We illustrate this "RePO" method in Figure 1.
Thanks to its conceptual simplicity, RePO can serve a hub that relates several exist- ing methods. In essence, our method can be viewed as "SimPO without sigmoid"" or "SLiC-HF without SFT regularization term". Interestingly, RePO is related each method by removing one of its core components. Even so, RePO effectively prevents over-optimization while perform- ing competitively or better (ef. Figure 1).
1.0
0.8
0.6
SimPO ($ 1.0)
0.4
SimPO 3.0)
SimPO 5.0)
0.2
SimPO ($ = 10.0)
RePO
We empirically show that over- optimization do exist, but ReLU activation with threshold Y is critical to prevent such solutions. This implies that in over-optimization regimes, selecting which examples to learn from is more critical than determining how much to learn from each. The y threshold induces an emergent data filtering behavior, focusing dynamically challenging samples relative to the model's s current capability. Our theoretical analysis reveals that RePO's ReLU loss corresponds precisely to the convex envelope of the 0-1 loss (Theorem 4.2), explaining why such simple mechanism is so effective.
0.0
-3.0
-2.0
0.0 Y 1.0 Implicit Reward Margin
-1.0
2.0
3.0
Figure Gradient weighting functions SimPO (sg) and RePO (T(Mp < 7)). As 3 co, converges to the binary indicator (red line), establishing RePO as the limit case of SimPO.
Our simple baseline suggests that the ReLU activation with proper threshold can be an essential reason for the common success of related methods. We believe this work's significance lies in revealing how preference learning principles may be simpler than previously thought. By questioning conventional wisdom about necessary components, we hope to motivate researchers reconsider the fundamental mechanisms behind preference optimization.

---

2 Preliminaries
Directed Preference Optimization (DPO). DPO [6] stands out leading method for offline preference optimization by eliminating the need for an explicit reward model. Instead, it reformulates the reward r(z, y) as closed-form expression based on policy ratios:
T x))
r(2; y) = 3 log
+ Blog (z))
(I)
(y x)
where (x) is partition function that does not depend on This leads to the DPO loss for given triplet (Ir You y/) as:
LDoo (To; Teer) -.... ED log (s
(1(1 ne ( x) [Brr xx) (B( x)) Teer Ut 2))
)}
log
(2)
where (-) denotes the sigmoid function. This loss encourages the policy TT to prefer yoe over yı in alignment with the reference policy.
Sequence Likelihood Calibration (SLiC-HF). SLiC-HF [7] advances preference optimization with two key innovations: (1) it employs a sequence-level calibration loss that contrasts the log- log-probability difference between preferred and dispreferred responses using margin 7, and (2) it integrates regularization term to prevent divergence from the SFT policy, avoiding the need for an explicit KL penalty. The SLiC-HF loss function defined
CsLiCHi -E JD ReLU
[lo To (Bi x)) log To 22) xx)-)) Alog (B x))
Simple Preference Optimization (SimPO). SimPO [11] advances preference optimization with two key innovations: (1) it normalizes the reward by the length of the response, calculating the average log-probability per token for a response under the policy Ty. and (2) incorporates a target reward margin to ensure that the reward difference between the preferred and less preferred responses exceeds this margin. The SimPO loss function is defined as:
log (8 llog no (Dw (x) log To(Mn x)) (p)-))) ))] |ywl
( = Ex.. W)) MD
(4)
luel
where ly| denotes the number of tokens in response y, ensuring length-awan aware scaling of rewards, and is the predefined margin that enforces minimum difference in rewards between you and yr. To align with subsequent discussions, we modify the original SimPO formulation by setting 7 to r/B.
3 Exploring Simple ReLU Activation in Preference Learning
In this section, we explore what makes a simple ReLU activation function effective for preference learning. We first examine the surprising relationship between ReLU activation and sigmoid weighting through empirical experiments. Then, we investigate the key properties that emerge from this simple mechanism, specifically through the lens of gradient behavior, data filtering patterns, and over- optimization control.
3.1 Examining ReLU-based Preference Optimization
Simplification exploration. Our exploration began by questioning whether log-sigmoii activation SFT regularization are truly necessary for mitigating over-optimization. We simplified the SimPO loss function through two key modifications: (i) removing the hyperparameter (3, and (ii) replacing the log-sigmoid function with a ReL.U activation.
We adopt the length- implicit reward margin My (as introduced in SimPO [11]):
Mp log (o |x)
log me x)
(5)
M
which quantifies the policy's preference between responses. Using Mo. we examine loss function
with the following form:
(FePO(Te) E(z W)) [ReLU(-(M, -
(6)
3

---

where y E [0, 1] is the sole hyperparameter representing the target reward margin.
Gradient behavior investigation. We examine the gradient dynamics of RePO and SimPO reveal how our simplified approach addresses over-optimization:
V VeLsmro( = -EE Isg (Vo.v Vo.u.),
(7)
V VoLRePO(TO) = -ED [TM < <)) (Vo.g. Vo.,
(8)
where se = (3( + 7)) is SimPO's sigmoid weighting function. The terms Ve.
yul Va log (a(Mi x)) and Vo.n - Mu Va log ne x) correspond to the gradients that increase the probability of the "winning" response t/u and decrease the probability the "losing" response yr, respectively. The scaling factor in Equation 7 linearly amplifies gradient magnitudes but does not alter the relative update directions in adaptive optimizers like Adam [13], as the momentum terms automatically normalize scale variations. We therefore omit 3 in Figure 2 for clearer visualization of the weighting function shapes.
The key insight is that RePO's ReLU-based gradient (Equation 8) applies uniform updates only to pairs with Mg < % while SimPO's gradient (Equation 7) uses continuous 3-scaled weights. Figure visualizes this difference, showing RePO the limiting case of SimPO 3 +
Lemma 3.1 (Gradient Equivalence in the SimPO-to-RePO Limit), Under the same Mo and definitions, the SimPO gradient converges pointwise to the RePO gradient as - oo:
lim VoLsimro VoLRer. 8-+00
Sketch. The convergence follows from the pointwise limit of the sigmoid weighting:
lim 8
lim aB3(MMo + +))) I(Mo < 7).
8
Substituting this into Equation 7 yields Equation 8.
Remark 3.2. Please check Appendix for all proofs. Lemma 3.1 establishes RePO as the asymptotic limit of SimPO with large 8, explaining two key advantages we will demonstrate in Section 3.2: com- parable performance without 3 tuning complexity, and an effective binary thresholding mechanism that induces implicit data filtering for controlling over-optimization.
Empirical Study
The previous section analyzes the relationship between SimPO and RePO from the perspective of gradient behavior. In this section, we compare their performance from an empirical standpoint.
Experimental setup. We evaluate this approach using SimPO's experimental setup [11] with Llama3- 8B and Gemma2-9B models (Instruct setup). For consistency, we use the same training datasets as SimPO: watioonnn/plllam---laaeeeedde aarmbaakk--armmmmmmmoo for Llama3-8P B-B and princeton wwwpeetonnnn/gg/mma222.. ultrafeedback-armen armoom for Gemma2-9B. For all SimPO experiments, we / 10.0 and y 0.4 for Gemma2-9B and 8 10.0 and optimization progress using two reward margin metrics:
0.3 for Llama3-8B, unless otherwise specified. We track
Mbatch Be
mp
(10)
measuring response separation within each batch (7hhanh)) and across the entire training (m-)
Evaluation benchmarks. We evalu-
Sinto Sinto (w1.0)
Sino (pe..1
Sintt (0
SiiP (p-12.3)
nero
two established benchmarks for open-ended generation: AlpacaEval
Ubmmad retmat
Gemmao Indruct
[14] (measuring instruction-followingg quality against GPT-4) and Arena- Hard [15] (testing complex reason- ing). For AlpacaEval 2, we report both length-controlled win rate (LC- Win Rate) and raw win rate (WR); for Arena- Arena-Hard, we report the standard win rate.
30
Win Take
Win Rate
LC Wnn Tats
Whn Tate
Figure 3: Performance of SimPO with varying and RePO on AlpacaEval2 benchmark.

---

Step 0
Step 100
Step 200
Step 400
Density
10.5%
13%
48%
58%
Implicit Reward Margin
Implicit Reward Margin Ma
Implicit Reward Margin Mg
Implicit Reward Margin Ma
Figure 4: Implicit reward margin Ma distribution across training steps (total: 467) for RePO at 7 0.4. Dashed line: ? 0.4. Green: samples below ? (gradient descent); gray: samples above ? (zero gradient). Numbers: fraction of samples above 7
Observation I: Large 3 enhances SimPO's performance when paired with appropriate y We systematically evaluate SimPO across varying values of 3 E (1.0, 3.0, 5.0, 10.0), while maintaining fixed values that empirically determined to be suitable for each model architecture (7 0.4 for Gemma2-9B and y = 0.3 for Llama3-88 -BBm.. ) As shown in Figure 3, increasing leads to consistent performance improvements across all evaluation metrics, with diminishing returns observed beyond 5.0. These findings align with observations in the SimPO paper'.
Observation 2: RePO matches high-6 SimPO. RePO achieves performance comparable to SimPO with large B. As shown in Figure 3, RePO achieves win rates of 51.1% on Llama3-8B
Liama3-8B-isstuutt
Gern Gem-mal--B---siruut
70
65-
8
0.6
Win
60
0.3
0.4
0.2
and 66.6% comparable to SimPO's perfor- mance. This aligns with Lemma 3.1, which establishes that RePO limiting case of SimPO as 3 + cc.
Gemma2-9B.
55-
0.0
0.2
0.4
0.5
0.8
1.0
0.0
50-
0.0
0.2
0.4
0.8
0.8
0.0
Figure 5: Line plot of RePO performance (AlpacaEval LC Win Rate) and bar chart of mean reward margins (mD) across varying ? values. See Appendix D.3 for details.
can be interpreted
Observation 3: threshold creates a natural alignment-optimizatee track the mean implicit reward margin mp (ef. Equation 10) across training pairs. As Figure illustrates, increasing 7 directly elevates mp while performance follows an inverted U-shaped pattern improving initially but declining beyond critical threshold. RePO, gradients vanish when the implicit reward margin exceeds y, effectively filtering out well-separated pairs from updates. This mechanism creates a fundamental tradcoff: small y values retain excessive zero- prro-gradient samples causing under-fil filterigg, while large 7 values force updates most samples, potentially leading to over-optimization [8] and ultimately degrading performance.
tradeoff. Our experiments
Observation 4: RePO creates a natural learning curriculum via progressive filtering. Figure 4 reveals an unexpected pattern in how distribution of implicit reward margins Mg evolves throughout training. As learning progresses, the model's ability to discriminate between winning and losing samples naturally improves, resulting steady increase both the implicit reward margin and the ratio of filtered data. Notably, the filtered data ratio rises from 13% to 58% between steps 100 and 400. This creates an emergent curriculum where the model initially learns from a broader set of examples and gradually focuses the more challenging ones despite using only half of the samples for gradient updates later stages, the model achieves optimal performance.
3.3 Over-OOtimization Analysis
The study of over-optimization can be traced back traditional RLHF literature, and has been empirically investigated in both controlled experiments [9] and user studies [10]. In this work, we follow their experimental setup to further explore this phenomenon.
In their official repository, the authors note: "SimPO requires much larger ( than DPO... In many cases. larger (e.g.. e.. 10) could yield better results."
5

---

Model Ovee-OOtimization:: Building on Rafailov et al. [16], we investigate over-optimization in RePO, by evaluating six different values of y (0.0, 0.2, 0.4, 0.6, 0.8, 1.0), each corresponding to varying levels of data filtering. Across all cases, we observe a distinct hump-shaped performance pattern: while moderate filtering improves alignment, excessive filtering causes performance to degrade, highlighting the over-optimization effect.
Scaling Law Fits. Previous work [9, 16] has established scaling laws for reward model scores function of the KL divergence between the initial and optimized policies. In contrast, we eliminate the reference model and the associated computational cost of calculating KI. divergence. Instead, we use the mean implicit reward mar- gin during training as proxy metric. The reward function R(d) is given by:
Llamaz-0B-1 Gemma2
Lama2-40-I
53
Gemmaz
Mrosa Moss Hund mR
32
10-
0.25
0.30
0.75
13
0:25
6,50
0.75
1.C
implct Roward Margin (ND
Inpect Aeward Margin (M)
Figure 6: Over-optimization patterns for RePO Llama3- 8B-IT and Gemma2-9B-IT using AlpacaEval2 LC win rates (11) and Arena-Hard raw win rates. Dotted curves represent theoretical fits based on Gao et al. [9]'s scaling laws, using where and are constants depen- GPT-4 win rates instead of standard reward model scores. dent the reward model's dataset size and parameter count, and d Mharch- Without training proxy reward model, we substitute GPT-4 win rates over dataset completions for the gold reward. Interestingly, we find that this scaling law provides an accurate relationship between d and win rates for RePO.
R(d) = d((x 3 log d),
3.4 RePO++: Exploring Extensions of ReLU-based Filtering
While exploring ReLU's thresholding behavior, we observed an interesting limitation: for cases where the implicit reward margin is smaller than %, their gradient weights become uniform, not differentiating between samples of varying difficulty.
This observation naturally led us to wonder: could we preserve the effective filtering mechanism while reintroducing some degree of weighting? To explore this question, we experimented with combining ReLU's binary filtering with SimPO's continuous weighting:
(RoPOH+(TT) EEp [log a (-ReLU (-3 (Me - 7)))]
(12)
This exploration was a natural follow-up to our main discovery about ReLU's effectiveness, rather than our primary contribution. We were curious to whether combining the best aspects of both approaches might yield additional insights about preference learning mechanisms.
What does this combined approach reveal? To understand the behavior of this extension, we examined its gradient with respect to the parameters
VoCnero+ (Ta) = -BEp [sg 1o (Vo.y V. )1
(13)
where = a (33 Mo + 7)) and In is an indicator function that is 1 if Mo < 7 and 0 otherwise.
We observed that this gradient combines properties we discovered in both approaches: it scales updates by 8g (similar to SimPO) and filters them using Ia (the key discovery in our ReLU exploration), focusing the model less-separated pairs while giving higher weights to smaller separations.
Adaptation with RePO++. The core contribution of RePO++ lies in leveraging ReLU to mitigate over-optimization while preserving the standard workflow of preference optimization. This makes RePO++ easily adaptable existing DPO-like methods. For instance, as shown in Equation 12, replacing Mo with log ne(p= ix)
na(m log Keelyy w) seamlessly integrates RePO++ into DPO, forming aar(nla))
ReLU-enhanced version of DPO.
4 Theoretical Analysis: ReLU's Optimality in Preference Learning
Next, we establish a surprising theoretical connection between preference optimization and binary classification, revealing why our simple ReLU-based approach achieves superior performance.

---

Following Tang et al. [17], preference learning can be reformulated as binary classification. Given pairs (z, where = E I2* and I E (-1,1), we aim to learn predictor Ê(z) whose sign matches I.
The classification accuracy is: }1E(sign( .1) 2- This corresponds minimizing the 0-1 loss:
(e) := E - sign(
(14)
For preference data (yw. 31) where U1 we set 1 = 1 and parameterize E(ww, v1) rr(Uw) (w))) yielding the objective:
, = E (ffor -r (w())
(15)
Where f determines the surrogate loss: f(x) I(x < 0) gives the 0-1 loss, f(z) log (x) yields SimPO's logistic loss, and f(z) = ReLUR-LU gives our method's loss.
Our key insight comes from analyzing the convex envelope of the 0-1 loss:
Definition 4.1. The convex envelope of La.1 over closed convex set D C R is:
conv pCo sup (M(x) h is convex, fo-1 Va D)
(16)
Theorem 4.2 (ReLU as Convex Envelope). For D = a-a, with a, > 0, the convex envelope of (x-1(x) = I1(r < 0) is:
co = ReLU(-2)
(17)
This remarkable result reveals that ReLU provides the tightest possible convex approximation to the ideal 0-1 loss, explaining its empirical effectiveness. Furthermore:
Corollary 4.3 (Optimality Preservation). Let D C R. be corvex. Then:
arg min Lo (f) = arg min cony convpfa.11
(18)
for D = [-a,bb
1 ReLU(-x))
arg min Lo-1 arg min =ED a
(19)
zE
This guarantees that gradient-base based optimization of our ReLU surrogate converges to solutions matching the theoretical optimum of the intractable 0-1 loss. Importantly:
Corollary 4.4 (Logistic Loss Suboptinality). The logistic loss flog- (x) log a (x) not
the convex envelope of Lo.1.
This theoretical foundation explains why our simple ReLU-based approach consistently outperforms more complex mechanisms like SimPO's sigmoid weighting ReLU provides optimality guarantees that logistic loss cannot match, while being computationally more efficient.
Experiments
In this section, examine how simplified ReLU-based approach behaves across different models and settings. Rather than focusing solely on performance gains, we explore patterns that help explain why such a simple mechanism works effectively in practice.
5.1 Experimental Setup
The core experimental configuration extends our investigation from Section 3.2 to include Mistral2-7B [18] alongside previously examined models. For the Llama3- v0.2 experiments, we employed the RLHHFoo/AAmmRM Stw.:////AAoM---aaaaaaaaa MaaaRRM--LLaaa--am -lamm3-88--B0 [19] reward model for ranking generated data. We bench- mark our approach against established preference optimization methods: DPO [6]. SimPO [11], IPO [20], CPO [21], KTO [22], ORPO [23]. and R-DPO [24], with SFT models serving as baselines. Implementation details are provided in Appendix D.1. 1. We also evaluate on downstream tasks from the Huggingface Open Leaderboard benchmarks [25], with additional details in in Appendix D.2.
The code is available https: // github..... com/ junkangwu/RReP.
7

---

Table 1: AlpacaEval (AE2), Arena-Hard (AH) results across four settings. "WR" denotes the raw win rate," "LC" second-best underlined.
rate,"LC" the length-controlled win rate. The best results are highlighted in bold, while the
Instuct (8B)
Mistral-Instruc (7B)
Ulama3-Instuuct v0.2 (8B)
Gemma2-Instrucc (9B)
Method
AE2
AH
AE
AH
AE
AH
AE
AH
LC
WR
WR
LC
WR
WR
LC
WR
WR
LC
WR
WR
24.0
23.6
19.0
15.4
24.0
22.4
48.7
36.5
42.1
SLIC-HF
26.9
27.5
26.2
24.1
24.6
18.9
33.9
32.5
29.3
65.1
60.5
53.7
DPO
40.2
38.1
31.2
20.3
17.9
13.4
48.2
47.5
35.2
70.4
66.9
58.8
IPO
35.9
34.4
22.3
18.6
40.6
34.9
62.6
58.4
53.5
CPO
29.6
34.4
29.4
26.2
31.7
23.8
36.5
40.8
34.2
56.4
53.4
55.2
KTO
38.3
34.1
30.3
19.4
20.3
16.8
41.4
36.4
28.9
61.7
55.5
53.8
ORPO
31.6
29.8
26.3
24.0
23.0
18.6
36.5
33.1
30.4
56.2
46.7
46.2
R-DPO
40.3
37.3
32.9
21.4
22.2
13.8
$1.6
50.7
35.0
68.3
66.9
57.9
SimPO
43.8
38.0
32.6
30.2
32.1
20.1
55.6
49.6
33.6
72.4
65.0
57.8
RePO
46.7
41.1
33,3
30.4
33.6
20.3
57.7
51.1
35.2
73.6
66.6
59.1
5.2 Result Comparisons
Simple ReLU thresholding exhibits surprising effectiveness. Table reveals unexpected pattern: despite removing components previously thought essential, simple ReLU- based approach consistently performs well across all evaluated models and benchmarks. This finding aligns with our theoretical analysis showing that binary thresholding directly approximates the convex envelope of the 0-1 loss. On AlpacaEval 2, observe improvements of 0.2-2.8 points in LC win rates across different configurations compared to the strongest baselines.
Observation:
5.3 Methodology Comparisons
Beyond alignment, we also compare the methodologies of these preference learning methods. Our method plays hub connect these methods.
Relation to SimPO. SimPO employs sigmoid weighting via log-sigmoid activation to attenuate gradients reward margins increase, mitigating over-optimization. RePO be viewed "SimPO without log-sigmoid," replacing this continuous scaling with binary filtering. To validate this relation- ship, we integrated a ReL.Ubbased filtering mechanism into SimPO (ef. RePO++ Equation 12). Table 2 confirms that ReLU's filtering mechanism enhances performance. As demonstrated in Section 3.4, RePO++ directly addresses over-optimization while retaining the benefits of some weighting.
Relation to SLiC-HE. RePO can be characterized as "SLiC-HF without SFT regularization". To ensure fair comparison (while disregarding differences in length normalization), investigated the impact SFT regularization by varying its coefficient, A. The results, presented Appendix Table 6 and further details in Appendix D.5, indicate that this additional regularization term offers no discernible improvement. This suggests that SFT regularization targets different optimization challenge, distinct from the direct over-optimization problem RePO addresses.
Relation DPO. Mathematically, DPO is equivalent to SimPO when the margin y is defined as log Teer (B r) log (v) .²) (ignoring length normalization). However, directly substituting the log-sigmoid function with ReLU in DPO's formulation leads to significant performance degradation (see Appendix Table 7 and Appendix D.6). This underscores the critical role of the threshold in determining the effectiveness of verroopimizztion prevention. As identified by Wu et al. [26], reference model based reward margins are often unreliable target margins, which explains why SimPO's s explicit 7 parameter effective for preference learning.
5.4 Effect of ReLU Filtering Across Methods
Having observed the effectiveness of binary thresholding, we naturally questioned whether this mechanism might enhance other preference learning approaches. Table 2 shows that integrating ReLU filtering consistently improved performance across both DPO and SimPO frameworks, suggesting that selective gradient application based on margin thresholds provides benefits beyond our specific implementation. Our experiments with the combined approach (ef. RePO++ in Section 3.4) revealed

---

AlpacaEval2
AlpacaEval2 WR
1.0
1,0+0.6
1.0+0.4
10-022
10-00.0
1.0+0.3
1.0+0.66
10-0.4
1,0+02
1.0+0.0
58.0
50.0
48.0
54.0
46.0
52.0
44.0
50.0
small to large
small to large
large to small
42.0
large to small
0.0-0.2
0.0-0.4
0.0-0.6 Dynamic V
0.0-1.0
0.0-0.2
0.0
0.0
Dynamic
Exploration of dynamic y scheduling on Llama3-Instruct v0.2 (8B). The dashed line represents performance with a fixed y. We observed that decreasing from an initially larger value creates a natural curriculum that enhances performance.
Figure 7:
particularly strong improvements when applied to DPO (5%-12% gains), with notable performance on Arena-Hard (reaching 65.7). This design effectively mitigates over-optimization while preserving the benefits of the original scheme.
5.5 Dynamic Margin Scheduling and Curriculum Learning
Our investigation into the role of tar- get reward margin led us to an unexpected discovery about currieu- lum learning. We experimented with dynamic scheduling of y throughout training, implementing two strategies: (i) increasing y from small to large, and (ii) decreasing y from large to small. Figure reveals striking pat- tern: starting with a moderately large value of 7 and gradually decreasing (1.0 >> 0.2) naturally creates an effec- tive curriculum that improves model performance. In contrast, both excessively large values (1.0 - 0.8) and small values (0.0 - 0.2) led to suboptimal outcomes.
Table 2:
Performance improvements of RePO and RePO++ over DPO and SimPO. Results present AlpacaEval 2 (AE 2) and Arena-Hard (AH) with LC (%) and WR (%). Red numbers indicate relative improvements.
Llama Instrutt v0.2 (8B)
Gemma (91)
Method
AR
AH
AF
AH
WR
DPO
48.2
47.5
70.4
66.9
58.8
RaPO
30.3
5188-115
38.219-58
73.8-4.5
71.0
64.219.5
30.813.4
02.2-
37.2-577
71.8 71.8-200
09.5
65.7 11.75
SimPO
55.6
19.0
33.6
72.4
63.0
57.8
RePO
37.7
51.1-30%
35.2
7
66 68.6
50.1 40.1
RePO++
36.1
74.1
74.19
66.5 14.55
60.8 18.5
This observation reveals an intriguing self-regulating property: early in training when the model underfitting, a larger y permits more aggressive updates across more examples. As training pro- gresses, the decreasing y naturally focuses learning on increasingly challenging examples, effectively preventing over-optimization. This emergent curriculum behavior, arising from a simple parame- ter schedule, suggests that binary thresholding captures fundamental learning dynamics that more complex mechanisms might obseure.
Discussion
Conclusion Our exploration of simple ReLU activation in preference learning has revealed several key insights. We found that binary thresholding, implemented through a straightforward ReLU function, provides alignment. Our theoretical analysis showed that this seemingly simple approach is, in fact, the convex envelope of the ideal 00- loss function, explaining its surprising effectiveness. Rather than developing yet another complex preference optimization method, our work uncovered how fundamental properties like data selection and implicit curriculum learning emerge naturally from basic principles.
effective mechanism for preventing over-optimization in language model
Limitations and future directions. Our current exploration limited offline preference learning settings. Future work could investigate how these insights might extend to online learning scenarios,

---

where preferences are gathered interactively. Additionally, while we found that fixed margin threshold works well in practice, exploring adaptive or context-aware thresholds might further improve performance in highly dynamic environments. The relationship between binary filtering and self-play scenarios [27] - where the model generates its own feedback - is another promising direction that could lead to more scalable alignment techniques.
Beyond alignment, our work connects LLM reasoning research [28, 29]. Future work should investigate how KL penalties and gradient clipping in GRPO [30] and PPO [5] balance preventing over-optimization against preserving reasoning capabilities critical consideration for advancing alignment methodologies.
Acknowledgments and Disclosure of Funding
This research was supported by the National Science and Technology Major Project (2023ZD0121 102), the National Natural Science Foundation of China (U24B20180, 62302321), and the Fundamental Research Funds for the Central Universities WWK210025066) This research also benefited from the advanced computing resources provided by the Supercomputing Center of the USTC.
References
[1] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jcan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras. Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mchran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/23 1805, 2023.
[2] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CORR, abbs/3307.0928s. 2023.
[3] OpenAl. GPT-4 technical report. CoRR, abs/2303.088744 2023.
[4] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lec, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303. 12:00.....2... 2023.
[5] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 707.07777
[6] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NewrIPS, 2023.
10

---

[7] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305.10425, 2023.
[8] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safery. CoRR,
2016.
[9] Leo Gao, John Schulman, and Jacob Hilton, Scaling laws for reward model overoptimization. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 10835-10866. PMLR, 2023.
[10] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In NeurlPS, 2023.
[11] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. NeurIPS, 2024.
[12] Paria Rashidinejad and Yuandong Tian. Sail into the headwind: Alignment via robust rewards and dynamic labels against reward hacking. ICLR, 2025.
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[14] Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction- following models. https: / htt//gitttt github...... conm/tatsu--abblllppaae httpe/aaaaaaappaaaaaaaaaaaa eval. 2023.
[15] Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data high-quality benchmarks: The Arena-Hard pipeline, April 2024.
URL https:t/p///. /mmsys..mss////m/mmlm//mm... arg/blog//a 200//022400000 19-119--- arrea-hhard.
[16] Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W. Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. CoRR, 0os/2406.02900 2024.
[17] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In ICML. OpenReview.net, 2024.
[18) Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. ArXiv, abs/2310.06825. 2023.
[19] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang, Interpretable pref- erences via multi-objective reward modeling and mfxture-of-exxpert, ArXiv, abs/2406. 12845, 2024.
[20] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo. Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/2310. 12036, 2023.
[21] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. ArXiv, abs/2401.084. 008417, 2024.
[22] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTT:: Model alignment prospect theoretic optimization. ArXiv, abs/2402.01306, 2024.
[23] Jiwoo Hong, Noah Lee, and James Thome. ORPO: Monolithic preference optimization without reference model. ArXiy, abs/2403.07691, 2024.

---

[24] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. ArXiv, 19159,, 2024.
[25] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. https: //huggingface. . co/ spaceeass///////aa/ htt///pacceeaagiiiee //ggggiggaaaeeii4 http/aaael/loaenn..ne 11m 11.leerrbrrrarrd 2023.
[26] Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. a-dpo: Adaptive reward margin what direct preference optimization needs. CoRR, abs/2410.10148, 2024.
[27] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. In ICML. OpenReview.net, 2024.
[28] OpenAl. Learning to reason with Ilms, 2024. URL httpss///ppeaaicommiiii learning- teaaiingooo--oo------- seaasonwwtthllms/
[29] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12448 2025.
[30] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
[31] Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS, pages 4299-4307, 2017.
[32] Daniel M. Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08892, 2019.
[33] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155, 2022.
[34] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. NeurIPS, 2023.
[35] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
[36] Xinyang Geng, Arnay Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1:6, 2023.
[37] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's 's first truly open instruction-tuned LLM, 2023. URL https: http:::::///...aaaaaiiichccc// databri com/ bl hwwp//22/////////// 04/12/ \dolly-first- open- www.ercialll- wiw.eeeiiinnntnnnnnoiieoeeoooeee tuned -lmm.
[38] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Minh Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Ope- nassistant wnwersstioonsddmmcraaiiiiii large language model alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarky Track, 2023.
[39] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP. 2023.
12

---

[40] Haipeng Luo, Qingieng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 2023.
[41] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. arXiv preprint arXiv:2402.07319, 2024.
[42] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200500 2023.
[43] Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhu- ravinskyi, Eric Hambro, and Roberta Railneau. GLoRe: When, where, and how improve LLM reasoning via global and local refinements. arXiv preprint arXiv:2402. 10963, 2024.
[44] Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri. Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. RewardBench: Evaluating reward models for language modeling. ArXiv, abs/2403. 13787, 2024.
[45] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[46] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce-style optimization for learning from human feedback in 1lms. In ACL (I), pages 12248-12267. Association for Computational Linguistics, 2024.
[47] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.033000, 2024.
[48] Junkang WI, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 3-dpo: Direct preference optimization with dynamic 3. CoRR, abs/2407. Abs/2407.00669, 2024.
[49] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: aligning language models with noisy feedback. In ICML. OpenReview.net, 2024.
[50] Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust aligmment of language models: Distributionally robustifying direct preference optimization. CoRR, abs/2407.07882 2024.
[51] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. In ICLR. OpenReview.net, 2024.
[52] Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J. Foster. Correcting the mythos of ki-regularization: Direct alignment without overoptimization via chi-squared preference optimization. CoRR, abs/2407. 2024.
[53] Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1665-1680, 2024.
[54] Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoe V Le, Qijun Tan, and Yuan Liu. Evolving alignment via asymmetric self-play, arXiv preprint arXiv:241 1.00062, 2024.
13

---

[55] Shiqi Wang, Zhengze Zhang. Rui Zhao, Fei Tan, and Nguyen Cam-Tu, Reward difference opti- mization for sample reweighting in offline rlhf. In Findings of the Association for Computational Linguisties: EMNLP 2024, pages 2109-2123 2024.
[56] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Forty-first International Conference on Machine Learning, 2024.
[57] Sen Yang, Leyang Cui, Deng Cai, Xinting Huang. Shuming Shi, and Wai Lam. Not all preference pairs are created equal: A recipe for annotation-efiiee iterative preference learning, In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6549-6561,
2024.
[58] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF CoRR, abs/2405.0786. 2024.
[59] Dahyun Kim, Yungi Kim, Wonho Song, Hyconwoo Kim, Yunsu Kim. Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all once. CoRR, 0tss/4033.1920,, 2024.
[60] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences. CoRR, abs/2404.03715. 2024.
[61] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. In ICML. OpenReview.net, 2024.
[62] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In ICML. OpenReview.net, 2024.
[63] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. CoRR, 000622005.00655 2024.
[64] Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. REBEL; reinforcement learning via regressing relative rewards. CoRR, abs/2404. 110224444..666.. 16767, 2024.
[65] Daniele Calandriello, Zhaohan Daniel Guo, Rémi Munos, Mark Rowland, Yunhao Tang, Bernardo Ávila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment large language models through online preference optimisation. In ICML. OpenReview.net, 2024.
[66] Princeton University. ORF 523 Lecture 8: Online Convex Optimization, Accessed 2025. URL https://wwi princeton.edde Lec8. pdf. Accessed: 31 Jan 2025.
[67] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference Leurning Representations, 2020.
[68] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge. ArXiv, abs/1803..m 05457, 2018.
[69] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, edi- tors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/w1/P19-111 1472. URL https: / htt//aalaanttt aclanthooottto otg/p19- 1472.
[70] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falschoods. In ACL, pages 3214-3252, 2022.
14

---

[71] Hector Levesque, Emesst Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
[72] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.444: 2021.
A Related Works
Reinforcement learning from human feedback. RLHF is technique designed to align large language models with human preferences and values [31-33, 20]. Traditional RLHF is typically structured in three stages: supervised fine-tuning [34-39], reward modeling [9, 40-44), and policy optimization [5, 45J. In the third stage, Proximal Policy Optimization (PPO) is widely adopted. In contrast, RLOO [46] reduces the GPU memory footprint of RLHF by eliminating the Critic model and leverages a Leave- Caaee--On-Ouut strategy to achieve superior performance. GRPO [47]. another variant of PPO, improves mathematical reasoning abilities while optimizing memory usage by replacing the Leave-One-Ous method with a direct subtraction of the mean of all samples for a given prompt.
Offline preference optimization. In addition DPO, several alternative preference optimization objectives have been proposed. IPO [20] addresses overfitting issues inherent in DPO. ORPO [23] and SimPO [11] aim to remove reliance
reference model. R-DPO [24] targets the reduction of exploitation due to sequence length, while KTO [22] handles preference optimization in the absence of pairwise data. CPO [21] and 3-DPO [48] focus improving the quality of preference data. Another research direction addresses noise in offline alignment, which arises from the need construct pairwise data. rDPO [49], variant of DPO, mitigates preference noise and enhances policy robustness, while DrDPO [50] applies distributed robust optimization to tackle this issue. Other works have approached the problem through divergence regularization [51, 52], selection of high-quality data [53, 54], or reweighting loss functions [55-57].
Iterative Preference Optimization, Offline preference optimization methods, such DPO, face limitation due to the lack of an explicit reward model, which hinders their ability to sample preference pairs from the optimal policy. To address this, iterative preference optimization techniques have been proposed. These methods iteratively update the reference model using the most recent policy model generate new preference pairs in each iteration [58-62, 27, 63, 64]. For instance, SPIN [27] employs a self-play framework to fine-tune the model in a supervised manner, while Yuan et al. [62] annotate preferences throughout the iterative process. REBEL improves sample quality by regressing the relative reward. Additionally, [65] generates data using mixture policy, similar to the Nash-MD algorithm [60].
B Broader Impacts
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here
C Proofs
C.1 Proof of Lemma 3.1
Lemma 3.1 (Gradient Equivalence in the SimPO-to-RePO Limit), Under the same My and ? definitions, the SimPO gradient converges pointwise the RePO gradient as -
lim VgLsimPo = V
8-000
Proof. We formally establish the gradient equivalence through pointwise convergence analysis. Let D be the data distribution and denote model parameters. Recall the gradient expressions:
15

---

SimPO Gradient:
VoLsmpP -9Ep [o(8(-Mo Mo + 7)) (Vo.v.. Vo.n
(20)
RePO Gradient:
VOLRePO -ED [[Mo < <>) (Vo.y..
Vo
(21)
where (-)) is the sigmoid function. The equivalence hinges the limiting behavior of the sigmoid weighting term $e = a ( MM + 7)). We analyze three cases:
Case 1: Ma
Here, - MM + > 0. As + x.
lim (3( MM + +7)
lim (3)= <>)
A+00
11000
Case 2: Ma > 7
Here, -Ma + 7 < 0. As 3 + xc,
lim o (B(-Mo + 7)) lim 7000
T(Mo <)).
-+-00
Case Mo 7 This occurs on measure-zero set under continuous distributions. The limit becomes:
lim o(0) = + I(Mo < 7), 8
which is negligible in expectation.
Thus, lim 8o = I(Ma < 7) almost everywhere. Substituting this into the SimPO gradient:
lim VeLsimPo = lim BED [sg (Vo. V 80000
(22)
9+00
ED
lim 3sg
(Vo.v)
(Dominated Convergence Theorem)
(23)
To resolve the scaling, observe that for Ma + 7;
lim B. oc
if Ma < <>
lim Bsg 8
(24)
limg 8-0 = 0
if Ma
The divergence when Me < y mitigated by adaptive optimizers like Adam, which normalize gradient magnitudes through momentum terms. Formally, let Vo Vo.ge Under Adam's 's update rule:
0.+1 = 0:
Vo, +
where my and é are bias-corrected momentum estimates. The infinite gradient magnitude is absorbed into mie/ Vit effectively reducing unit-scaled update. Hence, in normalized update space:
lim 3so go x T(Mo < 7) go-
Combining these results:
Tim VoLsiro = EEp TTMo < <7) (Vo,g - Vo,n VoLnero, 30000
(25)
which completes the proof.
16

---

C.2 Proof of Theorem 4.2
Theorem 4.2 (ReLU as Convex Envelope). For = (-a, with a, O, the convex envelope of (x) = [[(r < 0) is:
conv = ReLU(-x)
(17)
Proof. We demonstrate that h(x) ReLUU(-z)) satisfies the convex envelope definition through three sequential arguments.
1. Convexity and Underestimation:
The ReLU function is convex as the pointwise maximum of
affine functions (Rule 3). The composition h(x) maxx(=x 0) preserves convexity through affine transformation (Rule 2). For all D::
When x < h(x) - s (x)( since 2 -a S1
When x 2 0: h(z) = = (x)(
Thus h(x) & (z) over D.
2. Maximality Among Convex Underestimators: Let g(z) be any convex function satisfying
g(x) & (z). For x [-a, 0), convexity implies:
g(x) - g( g (1+ 9(0) S a
since g(-a) 1 and g(0) s 0. For x 2 0, g(x) - 0. Hence g(z) S M(x) for all x E D.
Epigraph Characterization: The epigraph epi(h) coincides with the convex hull of epi Co-1) (D x R). The affine segment /1x)) on a-a, 0) connects the points (-a, 1) and (0, 0), forming the tightest convex fit the 0-1 loss's discontinuity. By Theorem in [66], this construction achieves the convex envelope.
C.3 Proof of Corollary 4.3
Corollary 4.3 (Optimality Preservation). Let D S R be convex. Then:
arg min (a. arg min conv p
(18)
for D = 1-a,bb b]:
arg min (o) = arg min ReLU(-x)) ZED
(19)
x a
Proof. Part 1: By Theorem in the lecture notes (Page 5), for any function f and convex set S:
min
f(x) min conv
#ES
Let S = D and f Lo.1- The equality of minima implies:
(x E ED (x:) min ) S {a* E D conv = min conv Gha
To show reverse inclusion, suppose x² arg min convpCo.1 Since conv con & Lo (x*) and conv Co-1 attains its minimum at the same points as C0.1. x" must also minimize La-1-
Part 2: For D = [-a, b), both (xr) and (eI.U(-<r)) attain their minimum value 0 on [0, b). For
E [-a, 0), ReLU(-z)) is strictly decreasing, achieving its minimum at = O. Thus:
arg min (o)(2)) arg min ReLU(-=2) [0. zED
rED a
17

---

C.4 Proof of Corollary 4.4
Corollary 4,4 (Logistic Loss Suboptimality). The logistic loss flog-sigmoidl (1) = log (ar) is not the convex envelope of Lo.1.
Proof. We demonstrate violation of the convex envelope's 's defining property. Consider = [-1,1]] 1. Underestimation Failure: For x >> O:
[+e- (1=---)
log (x) log
log(1 +e +) 0 (a-1
Thus Tlog-sigmois E Co-1 over D, violating the envelope requirement.
2. Non-Maximalin Even if scaled, the logistic loss's curvature differs from the ReLU envelope.
For (-1,0), du² (- log (x)) (z) ox)) > making it strictly convex incompatible with the affine structure of conv Lo
Hence flog siggmod cannot be the convex envelope.
D Experiments
D.1 Implementation Details
Empirical observations indicate significant performance sensitivity to model parameter initialization and learning rate selection across compared methods. To establish rigorous comparison benchmarks, conducted systematic hyperparameter searches adhering the specifications in each method's original publication. The complete search space configuration is documented in Table 3. Notably, substantial architecture updates to both Llama3-8B and Instruct-7B necessitated re-implementation of the SimPO method, as the original implementation became incompatible with the revised model interfaces.
Training Protocol All experiments employed standardized training configurations to ensure compa- rability:
Batch size: 128 (consistent across methods)
Learning rate: Searched in (3e-7, 5e-7, 8e-7, le-6)
Training duration: Single epoch with cosine annealing schedule
Warmup: 10% of total training steps
Optimizer: Adam [13] (31 = 0.9, 32 = 0.999)
Sequence length: 2048 tokens (fixed for all inputs)
The learning rate schedule follows triangular policy with amplitude decay, selected through cross- validation on held-out development sets. All implementations utilize full-precision floating-point arithmetic to prevent gradient quantization artifacts.
Hyperparameters in RePO. Table 4 summarizes the hyperparameters utilized for RePO across different experimental settings. Our methodology only involves one hyperparameter: 7. Based on empirical evidence, we recommend setting to a default value of 0.5, as this configuration has consistently demonstrated reliability.
Decoding Hyperparameters. The decoding hyperparameters employed in this study align with those used in SimPO'. We express gratitude to the SimPO team for their generosity in sharing their insights and configurations, which have been instrumental in our work.
Computation Environment. All training experiments described in this paper were conducted using 8xA 100 GPUs. The experimental setup follows the guidelines provided in the hignment-handdood repository ensuring reproducibility and consistency with established practices.
18

---

Table 3: Various preference optimization objectives and hyperparameter search range.
Method
Objective
Hyperparameter
E 0.5, 1.0, 10.0)
SLiC-HF [7]
max (0, 8 log (1 (i)) + log ma (c]) A log (1w
E 00.1 2.0
DPO [6]
log log Iallala) |3 log malwiin) mar(ar x))
E [0.01, 0.05, 0.1]
IPO [20]
log refuwla)) (al(wula) log
ra ra(uil=)
E 10.01,0.1,0.5 1.0
log o (8 log To (e
2F
CPO [21]
B Blog (xIl log (o (z)
1.0, 3 E 0,051,0055,0.1
A log Toluwla)
Zeet + Avo žet 88 log martzil)))
A = Au = 1.0
KTO [22]
where Zest E( g/ww [AKL (To (yo(yll)] me(())))))]
8 [0.01, 0.05, 0.1]
(Nwllz) X o (log where po(y/a)) = exp [o) log To (v)))
polsulay T-po
log palmula) TPpooclss)
ORPO [23]
X [0.1, 1.0, 2.0
R-DPO [24)
log (3 log melusll)) (url(ou(s))
( log me(ulin)
(arlyll aalu))
E 0.05, 0.1, 0.5,
marlon m)
8 E [0.01, 0.05, 0.1]
SimPO [11]
log [bul log Te (Vwlx))
TenT log me( (z) -7)
E [200, 4.0, 6.0, 6.0, 8.0)
7 (0.3, 0.5, 0,0, 1.,
RePO
ReI.U[-
3
log to (ymll>)) wel log me (uel)) 7)]
? E [0.2,0.4, 0.5,,,,550.608
Table 4: The hyperparameter values in RePO used for each training setting.
Setting
Learning rate
Mistral-Instruct
0.4
6e-7
Alama3-Innsrure
0.6
le-6
1onna--Instruct-0..2
0.6
le-6
Gemma2-Instrus
0.4
8e-7
D.2 Downstream Task Evaluation
To assess the impact of RePO on downstream task performance, we evaluate models trained with different preference optimization methods on diverse set of tasks from the Huggingface Open Leaderboard [25]. The tasks include MMLU [67], ARC [68], HellaSwag [69], TruthfulQA [70], Winograd [71], and GSM8K [72). We adhere to standard evaluation protocols and present the results for all models in Table 5.
Overall Performance. On average, RePO shows competitive performance across tasks, achieving an overall score 67.49 on the Llamac-Instruct model and 70.58 on the Llama33Instruct v0.2 model. The performance is generally close to that of other preference optimization methods, but it is worth noting that some cases, it slightly lags behind models like SimPO or DPO, particularly on tasks such as ARC, HellaSwag, and TruthfulQA. However, the results suggest that RePO maintains balanced performance profile across the evaluated tasks.
General Knowledge and Reasoning. On MMLU, which tests general knowledge and reasoning, RePO shows a slight reduction performance (64.95 for Llamaš--Intruuc and 65.00 for Llama3- Instruct v0.2) compared to models such as RRHF and SimPO. This minor decline is consistent with the trend observed for other preference optimization methods and indicates that RePO may preserve similar extent while possibly focusing more improving performance in other areas such as reading comprehension and reasoning.
general knowledge
Reading Comprehension and Commonsense Reasoning, For ARC and HellaSwag, tasks related to reading comprehension and commonsense reasoning, RePO outperforms the base SFT model and exhibits competitive performance relative to other preference optimization methods. The Llama3- Instruct v0.2 model with RePO achieves score of 80.50 on HellaSwag, which comparable to the best-performing methods. This result suggests that RePO effectively improves the model's 's ability to handle contextual understanding and reasoning, likely due its optimization strategy.
'https://githur htwpa//www....gggggggo///gggaaaa handbook
19

---

Table 5: Downstream task evaluation results of tasks on the huggingface open leaderboard.
MMLU (5)
ARC (25)
HellaSwag (10)
TruthfulQA (0)
Winograd (5)
GSM8K
Average
Llamat-Instrut
SFT
67.06
61.01
78.57
51.66
74.35
68.69
66.89
RRHF
67.20
61.52
79.54
53.76
74.19
66.11
67.05
SLIC-HF
66.41
61.26
78.80
53.23
76.16
66.57
67.07
DPO
66.88
63.99
80.78
59.01
74.66
49.81
65.86
IPO
66.52
61.95
77.90
54.64
73.09
65.39
CPO
67.05
62.29
78.73
73.72
67.40
67.20
KTO
66.38
63.57
79.51
58.15
73.40
57.01
66.34
ORPO
66.41
61.01
79.38
54.37
75.77
64.59
66.92
R-DPO
66.74
64.33
80.97
60.32
74.82
43,90
65.18
SimPO
65.63
62.80
78.33
60.70
73.32
50.72
65.25
RePO
64.95
62.03
77.58
60.96
72.93
66.49
67.49
Ilama3-Instruc v0.2
SFT
67.06
61.01
78.57
74.35
66.89
RRHF
66.60
63.74
80.98
59.40
76.32
58.68
67.62
SLiC-HF
66.91
61.77
79.17
56.36
76.40
68.23
68.14
DPO
65.57
65.87
79.66
63.08
74.51
73.01
70.28
IPO
66.06
64.85
81.02
57.29
76.72
76.12
70.34
CPO
65.67
62.12
79.63
56.34
77.98
75.28
69.50
KTO
65.99
62.88
79.02
54,66
74.66
76.42
68.94
ORPO
65.75
63.99
79.91
57.02
78.06
75.13
69.98
R-DPO
66.17
65.36
79.98
57.94
75.06
75.36
69.98
SimPO
65.18
67.15
78.04
64.92
73.88
70.08
RePO
65.00
68.09
80.50
64.38
76.16
69.37
70.58
Truthfulness. On the TruthfulQA task, RePO consistently shows improvements over the base SFT model, with score of 60.96 for Llama3-Instruct and 64.38 for Llama3-Instruct v0.2. This indicates that RePO helps the model generate more truthful and reliable responses, aligning with trends observed in other preference optimization methods. The improvement this is especially notable given the inherent difficulty of this task, which tests the model's 's ability to avoid generating false information.
Math Performance. in performance for RePO relative the base SFT model. Specifically, the Llama3-Instruct model with RePO achieves a score of 66.49, which is lower than other methods such as SimPO or R-DPO, which focus more on improving mathematical reasoning. This drop is consistent with the trend observed across various preference optimization methods and may suggest that RePO is less effective in retaining mathematical reasoning abilities. Further investigation into this issue could provide insights into potential strategies for addressing this gap.
The GSM8K benchmark, which tests mathematical reasoning, shows a drop
Task-Specific Variability. Overall, RePO exhibits varied performance across tasks. While it performs well in certain areas, such as commonsense reasoning and truthfulness, it lags behind in others, particularly in general knowledge (MMLU) and mathematical reasoning (GSM8K). This variability is in line with the performance trends observed for other preference optimization methods, which often show task-dependent improvements and declines. This suggests that RePO has strengths in some domains, but may benefit from further refinement to improve performance across all tasks.
D.3 RePO with varying
Figure illustrates the effect of the hyperparameter model performance across two evaluation metrics: LC Win Rate and Raw Win Rate. The analysis is conducted on two models, Llama3- 8B-Instruct (left) and Gemma2-99-Insee Insstrutt (right). The LC Win Rate, shown in red (left y-axis), represents the model's s alignment with learned preferences, whereas the Raw Win Rate, shown in blue (right y-axis), evaluates overall ranking performance based on human preference comparisons.
Moderate 7 values lead to optimal performance. Moderate values of y (0.4-0.6) yield the best balance between preference alignment and generalization. Both models achieve their highest LC Win Rate this range, indicating that preference optimization is most effective when applied at an intermediate level. As increases beyond 0.6, LC Win Rate starts to decline, likely due to overfitting,
20

---

Uiimad -B-indtract
Gemma2 SS-instuct
70
LC Vin Rate
30
50
Winn Rate
45
35-
50
0.5
Figure 8: Impact of the hyperparameter on LC Win Rate and Raw Win Rate for Llama3-8D ----------n-tet (left) and Gemma2-9B-Instrruc (right).
0.8
0.6
R0.4
RaPO+4 (601
0.2
RePO+4 (6-5.0)
RePO++
(8
0.0
-3.0
-2.0
-1.0
0.0
2.0
3.0
Implicit Roward Margin
Figure 9: Gradient weighting functions of RePO++ (sg 1(Mo < 7)).
where the model overly aligns with preference data at expense of generalization. Conversely, at y 0.0, where preference optimization is applied, the LC Win Rate remains low, emphasizing the necessity of preference tuning.
Raw Win Rate trends reveal model robustness differences. The Raw Win Rate follows a similar trend but highlights differences in robustness across models. For Llama--8--Instruuu the Raw Win
Rate peaks at 0.4 before declining, suggesting that excessive preference optimization (y > 0.6) negatively impacts the model's ability to generalize. In contrast, Gemma2-96 In--rrutt exhibits more stable Raw Win Rate across wider range of y, reaching its highest performance at y 0.6
before experiencing a sharp decline at 7 = 0.8. This suggests that Gemma2-99-Inme In-srrutt maintains better robustness to preference optimization compared to
Gemma2-99- outperforms Llama3-8B-Instruce Gemma2-99 I-Inntrucct consistently out- performs Llama3-88 Instruct in both LC Win Rate and Raw Win Rate. This observation indicates that
not only aligns more effectively with learned preferences but also retains superior generalization capability. The results highlight the importance of carefully selecting avoid performance degradation extreme values. Future work could explore adaptive strate- gies for dynamically tuning y, ensuring that preference optimization enhances alignment without compromising generalization.
D.4 Analsis on RePO++
Figure illustrates the relationship between the gradient and the implicit reward margin. As shown in the figure, when the implicit reward margin is greater than 7, the gradient becomes zero. In this case, the model can stop updating for well-separated pairs, thus preventing overfitting. On the other hand, when the implicit reward margin is less than the model continues to increase the weight for less-separated pairs. Furthermore, the harder the pair to distinguish, the larger the gradient becomes, eventually converging to 1.0. This behavior is reminiscent of curriculum learning, where more difficult samples are assigned higher weights.
D.5 Analysis of the Relationship Between SLiC-HF and RePO
To provide deeper insights into the relationship between SLiC-HF and RePO, we conducted additional experiments examining the effect of SFT regularization core component of SLiC-HF that is absent
21

---

in our method. As demonstrated in Table 6, we systematically evaluated performance across varying values of the regularization coefficient A.
Mathematical Comparison. From a formulation perspective, SLiC-HF combines a hinge loss term with an SFT regularization component that penalizes deviation from the reference model. Specifically, the SFT regularization is controlled by parameter A, which balances preference optimization against model drift. By contrast, RePO eliminates this regularization entirely, relying solely on its binary threshold mechanism to control optimization.
Impact of SFT Regularization. Our experimental results reveal consistent trend: À increases from 0.0 to 5.0, performance on AlpacaEval2 LC steadily declines from 34.1% 1% to 27.8%. This progressive degradation suggests that stronger regularization toward the initial SFT model actually hinders effective preference learning in this context. The optimal performance occurs at À = 0.0, which effectively transforms SLiC-HF into variant of RePO.
Different Optimization Challenges. These findings suggest that SFT regularization and RePO's threshoid-based filtering address fundamentally different optimization challenges. While SFT regular- ization was originally introduced to mitigate catastrophic forgetting and preserve general capabilities, our results indicate that for direct preference optimization, such regularization is unnecessary and potentially counterproductive. Instead, RePO's ' selective gradient application through its threshold mechanism appears sufficient to prevent over-optimization while maintaining effective preference learning.
This analysis complements our main findings and further supports our hypothesis that carefully designed filtering mechanisms can effectively replace more complex regularization schemes in preference optimization.
Table 6: The hyperparameter À in SliC-HF used for each Llama3-Instruct v0.2.
SLiC-HF
0.0 À = 0.1
A 0.5
1.0
3.0
5.0
AlpacaEval2 LC
34.1
33.9
32.8
30.8
28.6
27.8
D.6 Detailed Analysis of the Relationship Between DPO and RePO
To further investigate the relationship between DPO and RePO, we conducted additional experiments examining how different formulation components affect performance. As shown in Table 7, we systematically evaluated five variants that decompose the key elements of each approach.
DPO as Special Case of SimPO. From mathematical perspective, DPO can be viewed specific instance of SimPO where y = log ver (!w x) log [ x) (ignoring length normalization for equivalence). This connection highlights how DPO implicitly defines its target margin based on reference model probabilities rather than using an explicit hyperparameter.
Impact of ReLU Without Explicit Margin. The second row of Table 7 shows that directly replacing log-sigmoid with ReLU while maintaining DPO's implicit margin definition leads to catastrophic performance degradation (-44% on AlpacaEval LC, -47% AlpacaEval WR, and -26% Arena- Hard), This dramatic decline reveals that the binary threshold mechanism of ReLU is only effective when paired with an appropriate explicit margin parameter.
The Critical Role %. 7 Rows 3-5 demonstrate that adding an explicit y parameter consistently im- proves performance across all metrics regardless of whether log-sigmoid, ReLU, or their combination used. The most substantial gains appear when ReL.U and y are combined (+4.4% ICC, +9.1 % WR on AlpacaFval), supporting our hypothesis that explicit threshold-based filtering effectively controls over-optimization.
Complementary Mechanisms. Interestingly, the combination of both mechanisms (row 5) yields the highest overall performance, suggesting that while RePO's binary filtering mechanism addresses the core over-optimization challenge, the continuous weighting from log-sigmoid may provide complementary benefits for fine ane-grained preference learning.
22

---

Table 7: Impact Analysis of Scaling and ReLU Mechanisms in DPO Training. Benchmark results on AlpacaEval 2.0 (AE2) and Arena- Hard (AH) demonstrate percentage point changes in Length-Controllee Win Rate (LC-WRR and Base Win Rate (WR) for Llama3-Instruct-v0) (8B). Values represent relative performance deltas (%) compared standard DPO baseline.
Instuct v0.2 (8B)
Method
ReLU
log a
AE2
All
LC
WR
WR
log
log mel (m)
log Te(in
x
X
48.2
47.5
35.2
Serrias (s)
Furloni (s)
ReLU
[log zufai.
log Furtpi))))
26.9 -4%9
25.1 25.1-17%
26.2 20%
log
(Blog me(stall)
FurlStall (e)
log mualbil))
xelinlo)
50.0
50.7+6.7%
36.8 36.8-4.5%
4.55%
ReLU
(Log nelotalss) [ms)
log ref
x
50.3
51.8 51.819.1% 19.1%
38.2
log
ReLU
(
log
ne Muula
@ log Perdal1))
mainla))
1))
50.8+3.9%
52.2+9.9%
37.2 37.2-5-7%
Vea (903
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: see abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: see Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
23

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: see Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: see Appendix D.I.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
24

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
attps://gitee wttp://www...o//oon//n////////ee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ca/
mublic/guider /ttgg//dd////uuubbeeeeee
Policy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /twpss.ccppubll///// wwtpccccpc/pppeeeecceecceeceeeee wwwpc///giieeeeeeeeeeeeeeeeeeeee httpg//dddd//oeeuumbibiiiiii wwnddeSubmissiinnPee.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see Appendix D.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
25

---

Answer: [Yes]
Justification: The results are accompanied by emor bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We carried out all computational tasks on suite of four 80GB A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpps//neurips... http://uuuriisscccppteeee wttpppplicc/tiiccccieeeeeeeeeeee
Answer: [Yes]
Justification:
Our research has been conducted with strict adherence to the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
26

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not existing assets.
Guidelines:
27

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing, editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https: / for what should or should not be described.
29


Reviews:
[{"review_id": "ZIo48FtnO1", "reviewer": "Reviewer_HXdp", "summary": "This paper introduces a preference alignment method that is easier than DPO or SimPO by using binary ReLU thresholding. It seeks to address over-optimizing issues, and presents RePO as a \"simpler baseline\" with the aim to motivate researchers. Main focus is clearly on comparison with SimPO and DPO, including the replication of Table 1 from the SimPO paper with less metrics.", "questions": "- Clearly state your main contribution and it's use case. If the claim is that RePO is just conceptually simple but effective, how does it compare to other preference methods named above? What is your hardware setup, is it equivalent, if your table 1 aims to replicate the SimPO original paper's Table of scores you should include MT-Bench and state the judges used. \n- The paper shows RePO to be a β → ∞ limit of SimPO, this is well-known. Is there a practical or theoretical setting where RePO strictly outperforms SimPO? Or improves on generalization or stability?\n- Deeper work that goes beyond re-using SimPO's method with a threshold function. A clear practical use case where RePO wins, or more novel theoretical contribution. Is RePO intended to be used as a baseline (as you mention in the abstract)?", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n- Paper is formatted correctly and layout is clear. There are many visuals.\n- Attempts at theoretical conclusions are a good starting point. \n\nWeaknesses: \n- Lack of novelty since replacing the log-sig SimPO with a threshold is mathematically simple and already known. Minimal algorithmic innovation in observing a limiting case. \n- Paper presents RePO as a surprising discovery/conclusion but it is not. In optimization, the equivalence of hard thresholding and sharp sigmoid approximations is extremely well explored and understood. Therefore this paper is more an empirical study of a single special case, without meaningful contribution.  \n- The theory is standard in that the the convex envelope 0f 0-1 loss is just standard construction in surrogate loss theory. The theory does not provide new insight. Discussions about ReLU being \"optimal\" do not mention the strong constraints and assumptions that must be satisfied. \n- Other preference alignment strategies are not discusses. Such as ORPO, R-DPO, RLHF, etc. There are numerous methods on this topic. In contract RePO does not demonstrate significant practical advantages. \n- Non-professional writing quality and tone. Some confusion about its main contributions and application, the theory seems like it's added in as an afterthought.", "was_revised": true, "final_justification": "After reading the rebuttal and discussion, my overall evaluation has improved. However, while the authors' clarifications on some points are appreciated, the core concerns remain unresolved.\n\n- The Authors recognize that the theoretical contribution in this work is fairly simplistic. There are many citations which already encompass the points motivated in the rebuttal. Despite having discussions of avenues for deeper theoretical contributions, the authors note that these results require more time to develop in their work.   \n\n- The primary weakness is the paper's limited novelty. It can be seen as a swapping of a single design element in existing methods like DPO. The usage of ReLU also appeared in SLiC-HF, where the authors used hinge loss.\n\n- There is a clear correlation to works such as SimPO and Aligner, which build on being simpler preference alignment frameworks. But in such cases numerous ablation studies are quantified (effect of length normalization, reward optimization, effect of hyper-parameter sensitivity, multiple datasets, etc). The submission is clearly inspired by SimPO since its Fig 1 corresponds directly with SimPO's Fig 1 (expect the bar graph is horizontal instead of vertical), its Fig 4 corresponds to SimPO's Fig 3, its Table 1 corresponds to SimPO's Table 4, and etc.\n\n- Empirical Validation: Experimental results remain limited in scope, with evaluation restricted to a narrow set of benchmarks. Recently there has been a density of work in this area such as [1], [2], [3], [4], [5]. Therefore new notable submissions are either theoretically strong or practically strong. \n\n-  While some clarifications were made, key details remain insufficiently explained to enable reproducibility.\n\n- Impact & Generality: The scope of applicability is unclear, and the discussion did not address how the method would generalize beyond the specific setting studied, nor did the rebuttal address how it will deliver deeper theoretical results/distinguish itself from the many other works in this category. \n\n\n\n\n**References**\n\n [1] Bose, Avinandan, et al. \"Hybrid preference optimization for alignment: Provably faster convergence rates by combining offline preferences with online exploration.\" arXiv preprint arXiv:2412.10616 (2024).\n\n[2] Zhang, Yuheng, et al. \"Improving LLM general preference alignment via optimistic online mirror descent.\" arXiv preprint arXiv:2502.16852 (2025).\n\n[3] Wu, Junkang, et al. \"AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization.\" Forty-second International Conference on Machine Learning.\n\n[4] Li, Gengxu, et al. \"Length-controlled margin-based preference optimization without reference model.\" arXiv preprint arXiv:2502.14643 (2025).\n\n[5] Xie, Tengyang, et al. \"Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.\" arXiv preprint arXiv:2405.21046 (2024).", "rebuttal": "**To Reviewer HXdp:**\n\nWe sincerely thank you for your expert and detailed critique. Your deep knowledge of the field has pushed us to more clearly articulate the precise scope and contribution of our work. We appreciate the opportunity to address your specific concerns.\n\n**1. On Primary Contribution and Novelty (Addressed Weakness1, Weakness2, Question3)**\n\nWe agree that the mathematical principle of hard thresholding is known in classical theory. Our contribution is not its invention, but its **application, analysis, and the new understanding it provides specifically for modern LLM preference alignment**, a field where complex methods have become the default. Our novelty lies in:\n\n* **Identifying a Core Principle: Filtering is More Fundamental than Weighting.** Our central thesis is that effective preference learning hinges more on **which** challenging examples are selected for updates than on **how** their gradients are continuously weighted. Our work explicitly decouples these two functions, which are entangled in methods like DPO/SimPO, and reveals the primary importance of the filtering mechanism.\n* **Providing Theoretical Grounding for this Principle in LLMs.** Our work is the first to formally justify this mechanism in this context by demonstrating the ReLU-based loss is the **convex envelope of the ideal 0-1 loss** (Theorem 4.2). This explains *why* this simple approach is so effective.\n* **A Strong, Simple, and Practical Baseline.** By showing RePO's effectiveness, we challenge the notion that complexity is required for alignment. The primary use case for RePO is as a strong, simple, and theoretically-grounded **baseline** to encourage the community to \"rethink the fundamental mechanisms\" of preference learning, as stated in our introduction.\n\n**2. On Theoretical Soundness and Assumptions (Addressed Weakness3)**\n\nThank you for pushing for more clarity on our theoretical claims. You are correct that our theory relies on assumptions.\n\nOur claim of \"optimality\" is made in the context of finding the tightest convex surrogate for the ideal (but intractable) **0-1 loss in binary classification**. This framing, where preference learning is reformulated as binary classification, is consistent with recent foundational work in the field, such as Tang et al., ICML 2024. Our proof for the convex envelope (Theorem 4.2) holds for a domain D = [-a, b], which assumes the implicit reward margin can be both positive and negative—a condition that holds true in practice.\n\nWe acknowledge this theoretical optimality does not guarantee convergence to a global minimum in the non-convex landscape of LLM training. However, the strength of our paper lies in demonstrating empirically how well this theoretically \"optimal surrogate\" performs in practice. We will make these assumptions and context clearer in the main text.\n\n**3. On Experimental Scope and Practical Advantages (Addressed Weakness4, Question1)**\n\nWe interpret your concern about \"significant practical advantages\" in two ways, and we will address both:\n\n**3a. Direct Performance Comparison against State-of-the-Art Baselines**\nWe respectfully note that our paper provides extensive comparisons demonstrating RePO's practical advantages. **Table 1** shows RePO consistently outperforming DPO, ORPO, R-DPO and SimPO. Furthermore, **Appendix Table 5** presents a large-scale comparison on the Open LLM Leaderboard against a wide range of methods, including **ORPO, R-DPO, and SLIC-HF**, where RePO remains a top performer, validating its effectiveness.\n\n**3b. Generalizability of the Filtering Principle to Other Methods**\nTo show our core insight—the primacy of data filtering—is a generalizable principle, we integrated RePO's filtering mechanism into IPO, a method with a distinct squared-error loss paradigm. The resulting loss function, which we term IPO-RePO, applies ReLU filtering to IPO's core objective:\n$$L_{\\text{IPO-RePO}} = \\mathbb{I}\\left(M_{\\theta, \\text{ref}} < \\gamma\\right) \\cdot \\left( M_{\\theta, \\text{ref}} - \\frac{1}{2\\tau} \\right)^2 \\quad \\text{where} \\quad M_{\\theta, \\text{ref}} = \\log \\frac{\\pi_{\\theta}(y_w|x)}{\\pi_{\\text{ref}}(y_w|x)} - \\log \\frac{\\pi_{\\theta}(y_l|x)}{\\pi_{\\text{ref}}(y_l|x)}$$\n\nWe conducted experiments on the Llama3-Instruct-v0.2 (8B) model, with results on the AlpacaEval and Arena-Hard benchmarks as follows:\n\n| Method | AlpacaEval LC | AlpacaEval WR | Arena Hard WR |\n| :--- | :---: | :---: | :---: |\n| IPO | 40.6 | 39.6 | 34.9 |\n| IPO-RePO (Ours) | **43.8** (+3.2) | **42.2** (+2.6) | **35.6** (+0.7) |\n\nThis significant lift demonstrates that our filtering principle is a general mechanism for improving preference alignment, not merely a modification of SimPO.\n\n**3c. On Benchmark Selection and MT-Bench**\nWe conducted the MT-Bench experiment on Llama3-Instruct-v0.2 (8B) as requested and confirmed RePO's effectiveness:\n\n| Method | GPT-4 Turbo Judge | GPT-4 Judge |\n| :--- | :---: | :---: |\n| DPO | 7.0 | 8.2 |\n| SimPO | 7.0 | 8.0 |\n| **RePO** | **7.2** | **8.4** |\n\nHowever, we chose not to emphasize this metric in our paper for reasons of scientific rigor, following public guidance **from the SimPO authors themselves**, who stated:\n> \"MT-Bench scores often show minimal differentiation between models... and exhibit high variance... For these reasons, we don't consider MT-Bench scores to be a reliable metric.\" (Source: Official SimPO GitHub repository, Issue #61)\n\nTherefore, our focus on AlpacaEval2 and Arena-Hard was a deliberate choice to use more reliable and discerning benchmarks. We will add the MT-Bench results and this justification to the appendix.\n\n**4. On the RePO-SimPO Limit and Stability (Addressed Question2)**\n\nRegarding the question of a setting where RePO demonstrates clear advantages over SimPO, our new high-$\\beta$ experiments offer several insights. First, we respectfully argue that while the mathematical limit of the sigmoid function is known, its formalization within a preference optimization loss (**our Lemma 3.1**) is a novel contribution of this work.\n\nOur experiments on Llama3-Instruct-v0.2 (8B) show RePO's clear advantages:\n| Metric | SimPO ($\\beta=10$) | SimPO ($\\beta=100$) | SimPO ($\\beta=1000$) | RePO |\n| :--- | :---: | :---: | :---: | :---: |\n| AlpacaEval2 LC | 55.8 | 56.6 | 57.1 | **57.7** |\n| AlpacaEval2 WR | 49.8 | 50.5 | 50.7 | **51.1** |\nRePO outperforms SimPO in:\n* **Stability and Generalization**: The table shows that even at an extremely high $\\beta=1000$, SimPO's performance saturates and fails to close the gap with RePO. This suggests the log-sigmoid formulation encounters practical optimization limits or numerical instability at such extremes, preventing it from fully realizing the benefits of hard thresholding. In contrast, RePO's ReLU formulation is perfectly stable and robustly achieves this goal, leading to better overall performance.\n* **Hyperparameter Efficiency:** RePO eliminates the need to tune the sensitive $\\beta$ hyperparameter, offering a simpler and more robust optimization process.\n\n**5. On Writing and Tone (Addressed Weakness5)**\n\nFinally, we appreciate the feedback on the manuscript's tone. We will thoroughly revise the paper to ensure our claims are presented with appropriate academic neutrality and that the connection between our theoretical motivation and empirical results is made more explicit and seamless.\n\nThank you again for your time and expertise. Your critique has given us clear directions for substantially improving our paper."}, {"review_id": "fkH6nMYyOB", "reviewer": "Reviewer_1JuA", "summary": "This paper reports on a finding that simple ReLU-based activation of neural elements can result in significant alignment of large language models (LLMs) via preference learning. The paper presents empirical results that ReLU-based activation can learn meaningful alignments even in the absence of sigmoid-based gradient constraints and explicit regularization terms. While ReLU-based activation scheme (termed as RePO) does result in over-optimization during preference learning, the proper selection of a threshold parameter γ can obviate the over-optimization problem by dynamically filtering the training examples. The paper shows that the proposed RePO is equivalent to an asymptotic case of simple preference optimization (SimPO) with the parameter β→∞. Conceptually RePO can be viewed as SimPO without the log-sigmoid optimization and as sequence likelihood calibration (SLiC-HF) without supervised fine tuning (SFT)-based regularization. Experimental results comparing the proposed RePO with different existing preference learning models on the Mistral2 and Llama3 datasets show the effectiveness of ReLU-based activation.", "questions": "What would be an effective way of determining an optimal schedule for dynamically adapting the value of γ during the training process?", "limitations": "The authors have discussed the limitations of their work in the paper. However, if there is no general strategy for determining an optimal schedule for dynamically adapting the value of γ during the training process, then this should be mentioned as an additional limitation of the current work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper provides valuable insights in that simple ReLU-based activation can prevent over optimization in preference learning settings using binary thresholding controlled by a single parameter γ. The paper shows that allowing the model to select which examples to learn from rather than determine how much to learn from each example is more effective in preventing over optimization. Changing the value of γ over the course of the training (dynamic margin scheduling) allows for curriculum learning where larger γ values during early phases of the training permit more aggressive updates when the model is under fitting and smaller γ values during later phases of the training prevent over optimization. The paper does not provide an effective way of determining an optimal schedule for dynamically adapting the value of γ during the training process. This is important given the critical role that the parameter value γ plays in the proposed RePO scheme.", "was_revised": true, "final_justification": "In their rebuttal the authors have rightfully noted that an adaptive strategy, especially one that is data driven, would be the most appropriate for estimation of γ during the training process. In their revised paper, the authors should detail this strategy as part of potential future work. I am comfortable with my earlier rating of accept (rating 5) for this paper.", "rebuttal": "**To Reviewer 1juA:**\n\nWe are sincerely grateful for your positive, thorough, and insightful review. We are especially pleased that you recognized the core contributions of our work regarding conceptual simplicity, theoretical backing, and the principle of dynamic data filtering. Your understanding of our paper is spot on.\n\n***\n\n**Q1: What would be an effective way of determining an optimal schedule for dynamically adapting the value of $\\gamma$ during the training process?**\n\n**Response:**\nThis is an excellent question that points directly to a key area for future research. Based on our current findings presented in **Section 5.5 (Figure 7)**, our empirically validated recommendation is a **\"large-to-small\" scheduling strategy**.\n\nThe rationale for this strategy is that it creates a natural and effective learning curriculum. As we note in the paper (Lines 318-320), this works because:\n> \"early in training when the model is underfitting, a larger $\\gamma$ permits more aggressive updates across more examples. As training progresses, the decreasing $\\gamma$ naturally focuses learning on increasingly challenging examples, effectively preventing over-optimization.\"\n\nLooking forward to a more general and potentially *optimal* schedule, we believe an **adaptive strategy** holds the most promise. Instead of a pre-fixed decay schedule, $\\gamma$ could be dynamically adjusted based on the live statistics of the training process itself. For example, one could set the value of $\\gamma$ at each training step to match a specific percentile (e.g., the 40th or 50th) of the implicit reward margin ($M_\\theta$) distribution observed in the current or a recent batch. This would ensure the model consistently trains on a fixed proportion of what it currently considers to be \"hard examples,\" adapting gracefully to changes in model capability and data difficulty.\n\nWe agree that exploring such automated and adaptive scheduling methods is a valuable and exciting direction for future work. We have incorporated this discussion into our limitations section to inspire further research.\n\n***\n\n**Formatting Concerns:**\n\nWe also thank you for the helpful note regarding acronyms. We will perform a thorough pass on the final manuscript to ensure that all acronyms are clearly spelled out upon their first use.\n\nThank you once again for your supportive and constructive review."}, {"review_id": "g5o9hq3yGZ", "reviewer": "Reviewer_tpNX", "summary": "The paper introduces RePO, a simple preference optimization method for aligning language models using only a ReLU threshold on reward margins between chosen and rejected responses. This approach avoids common complexities like reference models or log-sigmoid weighting, and naturally focuses training on harder examples without over-optimization. RePO is shown to be the convex envelope of the 0-1 preference loss and matches or outperforms prior methods like DPO and SimPO across several benchmarks. The authors also analyze its theoretical properties and practical behavior under different settings.", "questions": "- How does RePO perform across models of different sizes, such as smaller (e.g., 1 - 3B) or larger (e.g., 30 - 70B) architectures?\n- How well does RePO generalize to other alignment datasets beyond AlpacaEval and Arena-Hard?\n- In practice, how do the gradient directions produced by RePO differ from those of DPO or SimPO? A 2D toy example or visualization illustrating this difference and linking it to the convex-envelope theory would be helpful.\n- Can you provide quantitative evidence for the “emergent curriculum” effect—for example, plots showing how the margin distribution shifts toward harder pairs over the course of training?\n- Minor: The link in citation [66] is broken.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- **Conceptual simplicity with strong performance:** A single ReLU threshold replaces reference models and log-sigmoid weighting yet still outperforms or matches DPO, SimPO, and SLiC-HF.\n- **Sound theoretical backing:** Proves the ReLU loss is the convex envelope of the 0-1 preference loss and connects SimPO as the the limit.\n- **Empirical robustness:** Consistent gains on AlpacaEval 2 and Arena-Hard across several 7–9 B open models, demonstrating plug-and-play practicality.\n\n**Weaknesses**\n\n- **Narrow experimental scope:** Only two benchmarks and mid-size models; no results for larger, multilingual, or safety-focused settings.\n- **Shallow ablation coverage:** Other hyper-parameters (except for $\\gamma$)and data regimes remain untested.\n- **Limited theoretical support:** The convex-envelope proof shows RePO is an optimal surrogate loss, but it offers no guarantees that training will converge to optimality (which is rare in practice). Also, the advertised benefits --- such as mitigating over-training and filtering data --- remain theoretically unsupported.", "was_revised": true, "final_justification": "Thank you for the detailed response. I especially appreciated the dynamic filtering component. Assuming the mentioned updates and additional experiments with larger LLMs are included in the camera-ready version, I would be supportive of accepting this paper.", "rebuttal": "**To Reviewer tpNX:**\n\nWe sincerely thank you for your thoughtful and detailed review. Your questions have helped us identify several areas where providing more evidence can significantly strengthen our paper. We address each of your points below.\n\n***\n\n**Q1: How does RePO perform across models of different sizes, such as smaller (e.g., 1 - 3B) or larger (e.g., 30 - 70B) architectures?**\n\n**Response:**\nThank you for this important question regarding model scalability. To address this, we conducted additional experiments on smaller open-source models: Pythia-1.4B and Pythia-2.8B. The results on AlpacaEval are as follows:\n\n| Model | Method | AlpacaEval LC | AlpacaEval WR |\n| :--- | :--- | :---: | :---: |\n| **Pythia-1.4B** | DPO | 3.6 | 2.8 |\n| | SimPO | 4.6 | 3.9 |\n| | **RePO** | **5.2** | **4.5** |\n| **Pythia-2.8B** | DPO | 5.5 | 4.3 |\n| | SimPO | 7.2 | 6.8 |\n| | **RePO** | **8.0** | **7.2** |\n\nThe results show that RePO consistently outperforms both DPO and SimPO on these smaller architectures, maintaining the performance advantage seen in the 7-9B range.\n\nDue to time and resource constraints during the rebuttal period, we were unable to conduct experiments on 30-70B scale models. However, the consistent success of RePO across the 1.4B, 2.8B, 7B, 8B, and 9B model scales provides strong evidence for its effectiveness across a range of small to medium architectures. This consistent outperformance is a promising indicator of its potential to scale favorably, which we propose as a key direction for future work. We will add these new results to the appendix in the final version.\n\n***\n\n**Q2: How well does RePO generalize to other alignment datasets beyond AlpacaEval and Arena-Hard?**\n\n**Response:**\nThank you for asking about broader generalization. As you suggest, we evaluated RePO on a wider set of benchmarks from the **Huggingface Open LLM Leaderboard**. These results are detailed in **Appendix D.2, Table 5** of our submission.\n\nThe evaluation covers six diverse downstream tasks: MMLU, ARC, HellaSwag, TruthfulQA, Winograd, and GSM8K. In these experiments, RePO demonstrates strong, competitive performance, achieving one of the highest average scores across all tasks for both the Llama3-Instruct and Llama3-Instruct v0.2 models. This robust performance across varied domains further supports our central claim: that the principle of gradient sparsity (filtering trivial pairs) is a more critical driver of successful alignment than the specific weighting scheme of the loss function.\n\n***\n\n**Q3: In practice, how do the gradient directions produced by RePO differ from those of DPO or SimPO? A 2D toy example or visualization illustrating this difference and linking it to the convex-envelope theory would be helpful.**\n\n**Response:**\nThis is an excellent question that gets to the heart of the optimization dynamics. The key difference is not in the gradient *direction* but in its *application* and *magnitude*.\n\nFor a given data pair $(x, y_w, y_l)$, the fundamental gradient update direction, $ \\nabla_{\\theta,y_{w}}-\\nabla_{\\theta,y_{l}} $, is actually **the same** for RePO, SimPO, and DPO. The distinction lies in how this gradient is scaled:\n\n* **DPO/SimPO** apply a **continuous, smooth weighting** to the gradient. The magnitude is scaled by a sigmoid function, meaning all pairs receive a non-zero gradient, though it becomes vanishingly small for well-separated pairs.\n* **RePO** applies a **binary, all-or-nothing filter**. The gradient is either applied (with a uniform effective magnitude, as adaptive optimizers like Adam normalize it) or it is set to exactly zero if the margin $\\gamma$ is met.\n\nTo provide the 2D intuition you requested: imagine a parameter space where, for any given data point, the gradient vector points in the same direction for all three methods. With **DPO/SimPO**, the landscape of gradient *magnitudes* would be smooth, like rolling hills, gradually approaching zero. In contrast, for **RePO**, this landscape is a plateau of uniform height on one side of the $\\gamma$ boundary and completely flat (zero) on the other. This sharp 'cliff' is precisely what **Figure 2** in our paper illustrates in one dimension.\n\nThis piecewise-constant behavior is a direct consequence of RePO's loss being the convex envelope of the 0-1 loss, which theoretically favors such sharp, efficient boundaries over the smooth curves of logistic-based losses. It provides a hard, efficient threshold for updates, linking our theory directly to the practical gradient behavior you asked about.\n\n***\n\n**Q4: Can you provide quantitative evidence for the “emergent curriculum” effect—for example, plots showing how the margin distribution shifts toward harder pairs over the course of training?**\n\n**Response:**\nThank you for this prompt. **Figure 4** in our paper provides exactly this quantitative evidence. The green-shaded areas in the plots represent the proportion of training samples whose reward margin $M_\\theta$ is less than the threshold $\\gamma$, meaning they are still actively receiving gradient updates.\n\nWe can summarize the percentage of samples being updated at each training snapshot:\n\n| Training Step | % of Samples with Active Gradients ($M_\\theta < \\gamma$) |\n| :---: | :---: |\n| 0 | 99.5% |\n| 100 | 87% |\n| 200 | 52% |\n| 400 | 42% |\n\nThis data clearly shows the emergent curriculum:\n* **Early in training** (Step 0), nearly all samples are considered \"hard\" as the model has not learned to distinguish preferences, so 99.5% of pairs are updated.\n* **As training progresses**, the model improves, and a growing fraction of the data is \"filtered out\" (gray area).\n* **Late in training** (Step 400), only 42% of the samples receive updates. These remaining samples are the most \"challenging\" or \"confusing\" pairs for the mature model.\n\nThis dynamic filtering, where the training automatically focuses on progressively harder examples, is what we define as the \"emergent curriculum\" effect driven by the simple $\\gamma$ threshold.\n\n***\n\n**Q5: Shallow ablation coverage: Other hyper-parameters (except for $\\gamma$) and data regimes remain untested.**\n\n**Response:**\nThank you for this point. We agree that exploring a wider range of hyperparameters could potentially tune overall performance further. However, for this study, our primary scientific goal was to isolate and understand the fundamental impact of the gradient filtering mechanism itself, which is RePO's core conceptual contribution.\n\nTo achieve this and ensure a **controlled and fair comparison**, all other training hyperparameters (e.g., learning rate, optimizer settings) were kept consistent across all methods and aligned with the established SimPO baseline, as detailed in our **Appendix D.1 (Implementation Details)**. We believe this focus is crucial for a clear scientific conclusion, and our extensive ablation study on `γ` (presented in Figures 5, 7, and 8) provides a thorough and insightful analysis of this central component.\n\n**Q6: Limited theoretical support: The convex-envelope proof shows RePO is an optimal surrogate loss, but it offers no guarantees that training will converge to optimality (which is rare in practice). Also, the advertised benefits --- such as mitigating over-training and filtering data --- remain theoretically unsupported.**\n\n**Response:**\nThank you for this precise and important clarification regarding the scope of our theoretical support. You are entirely correct, and we appreciate the opportunity to be precise about what our theory does and does not claim.\n\n* **On Optimal Surrogate Loss vs. Training Convergence:** Our theoretical contribution is to prove that RePO's loss function is the optimal **surrogate loss**—specifically, the tightest possible convex approximation to the ideal 0-1 loss within a binary classification framework (a setup consistent with foundational work like Tang et al., ICML 2024). This provides a strong theoretical reason for *why* one should prefer RePO's formulation over, for example, a log-sigmoid loss from a surrogate loss perspective. We agree that this **does not** provide a guarantee that the training process itself will converge to a global optimum in the highly non-convex landscape of LLM optimization, as such guarantees are exceptionally rare for deep learning in practice.\n\n* **On Empirical vs. Theoretical Support for Benefits:** Similarly, you are correct that the advertised benefits of mitigating over-optimization and creating an emergent data filtering curriculum are currently supported by our extensive **empirical analysis** (e.g., Sections 3.3, 5.5; Figures 4, 6) rather than a formal theoretical proof. We believe that bridging this gap—developing a formal theory for these emergent, dynamic properties of the training process—is a challenging but very exciting direction for future research.\n\nWe will revise the manuscript to make these distinctions clearer. Thank you for pushing for this level of precision.\n\n**Q7: Minor: The link in citation [66] is broken.**\n\n**Response:**\nThank you for catching this typo. We have corrected the link in the manuscript."}, {"review_id": "aU9PURlCTE", "reviewer": "Reviewer_M1uf", "summary": "The authors propose to use ReLU to replace log sigmoid used in methods such as DPO and SimPO.", "questions": "RePO++ combines ReLU's binary filtering with SimPO's sigmoid weighting. Empirically, adding the sigmoid weighting back improves the performance. But how do the authors reconcile this with the paper's central claim that sigmoid-based gradient constraints are an unnecessary component for preference optimization?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "## Strength\nThe paper demonstrates the the log sigmoid non-linearity in methods of DPO can be replaced by ReLU. They show that using ReLU-based loss, RePO, achieves competitive performance in alignment tasks relative to DPO. It also links the ReLU-based loss to the convex envelope of the 0-1 loss\n\n## Weakness\nThe primary weakness is the paper's limited novelty. It can be seen as a swapping of a single design element in existing methods like DPO. The usage of ReLU also appeared in SLiC-HF, where the authors used hinge loss.", "was_revised": true, "final_justification": "The authors' response clarified what they mean by binary selection---it is the gradient of the loss and not the loss itelsef.", "rebuttal": "**To Reviewer M1uf:**\n\nWe thank you for your critical feedback. We appreciate this opportunity to clarify our core contributions, which we believe address the perceived limitations in novelty and resolve the apparent contradiction noted with RePO++.\n\n***\n\n**Q1: The primary weakness is the paper's limited novelty.**\n\n**Response:**\n\nWe respectfully argue that viewing our work as a \"design swap\" overlooks its primary conceptual contribution: we demonstrate that **why** a mechanism works is as important as **what** the mechanism is. Our primary contribution is the **conceptual reframing of preference optimization**, supported by new theoretical and empirical evidence. The novelty lies in the following key areas:\n\n**1. Identifying a Core Principle: Filtering is More Fundamental than Weighting.**\nOur key insight, and the central thesis of the paper, is that effective preference learning hinges more on **which** challenging examples are selected for updates than on **how** their gradients are continuously weighted. While prior methods like DPO/SimPO entangle these two functions within the log-sigmoid activation and the $\\beta$ hyperparameter, our work explicitly **decouples** them. RePO serves as a \"computational lens\" to isolate and study the filtering mechanism in its purest form, revealing its primary importance.\n\n**2. Providing Theoretical Grounding for this Principle.**\nOur work is the first to establish a formal justification for this binary filtering approach. By demonstrating that the ReLU-based loss is the **convex envelope of the ideal 0-1 loss** (Theorem 4.2), we provide strong theoretical evidence for its optimality as a surrogate loss. This crucial theoretical contribution explains *why* this simple filtering mechanism is so effective and fundamentally distinguishes our work from prior heuristic uses of similar functions, such as the hinge loss in SLiC-HF, which lacked this formal justification.\n\n**3. Simplifying and Deepening the Understanding of Over-Optimization.**\nBy showing that a simple, theoretically-grounded filtering mechanism can match or exceed the performance of more complex methods (Table 1), we challenge the conventional wisdom that intricate components like sigmoid-based gradient constraints are essential for alignment. Our analysis of the hyperparameter $\\gamma$ in creating a natural learning curriculum (Observation 4, Figure 4) and directly controlling over-optimization (Figure 6) offers a new, simplified lens through which to understand and manage the alignment process.\n\nAs stated in our introduction (Lines 65-67), our primary aim is to `motivate researchers to reconsider the fundamental mechanisms behind preference optimization`. We believe our work provides a significant conceptual shift and a valuable, simplified baseline to facilitate this deeper understanding.\n\n***\n\n**Q2: How do the authors reconcile RePO++'s performance with the claim that sigmoid-based gradient constraints are an unnecessary component?**\n\n**Response:**\nThis is a critical question, and we are grateful for the chance to clarify what we see as a hierarchy of importance in preference optimization mechanisms.\n\n**Our central claim, to be more precise, is not that sigmoid weighting is ineffective, but rather that the mechanism of *which* samples to filter is more fundamental to preventing over-optimization than the secondary mechanism of *how* to weight the remaining samples.**\n\nRePO and RePO++ were designed to demonstrate this hierarchy:\n\n* **RePO Establishes the Foundation:** RePO's strong performance (Table 1) proves that binary filtering **alone** is a sufficient and powerful mechanism for effective alignment. It successfully prevents over-optimization without needing any continuous gradient weighting, thus supporting our claim that such weighting is not an *essential* component for this primary purpose.\n\n* **RePO++ Reveals the Hierarchy:** RePO++ was designed as an investigative tool to explore what happens when a secondary weighting mechanism is layered *on top of* the essential filtering mechanism. The results demonstrate that a hybrid approach can yield further gains (e.g., Table 2, +11.7% Arena-Hard gain for DPO++). This does not contradict our claim; it refines it by showing:\n    1.  **Primary Mechanism (Essential):** First, you must effectively filter out easy examples to prevent over-optimization. This is the most critical step.\n    2.  **Secondary Mechanism (Optional Enhancement):** Then, for the remaining hard examples, applying a fine-grained weighting can provide an additional, complementary benefit.\n\nIn summary, prior methods conflated these primary and secondary roles. Our work decouples them, identifies filtering as the most critical and previously under-appreciated mechanism, and shows that sigmoid weighting, while not essential, can be a useful \"add-on\" once the fundamental filtering problem is solved."}, {"review_id": "dPshrg9TU3", "reviewer": "Reviewer_uBRY", "summary": "The paper introduces ReLU-based Preference Optimization (RePO), a simplified approach to preference optimization for aligning LLMs with human preferences. Unlike established methods such as SimPO, DPO, and SLiC-HF, which rely on sigmoid-based weighting or explicit regularization to prevent over-optimization, RePO leverages a ReLU-based loss with a single margin threshold hyperparameter $\\gamma$. \nThis threshold-based mechanism acts as a binary filter that dynamically and selectively updates only those training samples whose reward margins fall below the threshold. The key insight is that RePO is the asymptotic limit of SimPO as its scaling factor β approaches infinity, making RePO a conceptually intuitive but theoretically grounded approach. \n\nExtensive experiments on instruction-following and reasoning benchmarks, including AlpacaEval 2 and Arena-Hard, show that RePO outperform or match strong baselines like DPO and SimPO across several model families. The authors also performed experiment on diverse aspect for further understanding of the proposed method,  identifying an implicit curriculum learning behavior induced by the filtering mechanism and showing how dynamic scheduling of $\\gamma$ can further improve performance. The authors further propose RePO++, which blends ReLU filtering with sigmoid-based weighting to combine the strengths of both methods. \nWith this empirical studies, the paper argues that simple threshold-based data selection may be more critical than fine-grained gradient control in preference learning.", "questions": "- Could the authors consider including at least one controlled experiment under the replicated conditions from prior work (e.g. Gao et al.) to increase reproducibility and practical reliability?\n- To better understand the source of RePO’s effectiveness, have the authors considered applying RePO-style filtering to other baseline methods (beyond SimPO and DPO)? This could help isolate whether the performance gains stem from the filtering strategy itself or from interactions with specific architectures.\n- The evaluation of SimPO under varying $\\beta$ values didn't include the high-$\\beta$ regime (e.g., $10^2, 10^3$), which the authors reference in the motivation. Could the authors include or discuss results in this range to better support the claim of RePO being equivalent to high-$\\beta$ SimPO?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The connection between ReLU loss and the convex envelope of the 0–1 loss adds rigor and interpretability.\n- The authors benchmark RePO against strong baselines across multiple timely models and tasks, showing robustness and reliabilty of the results. \n- The use of ReLU for preference optimization - interpreted as a threshold-based curriculum learning mechanism - is novel in this domain The insight that “which examples to learn from” may be more important than “how much to learn from each” provides an interesting shift in perspective, contributing to the active research area by questioning unnecessary complexity in current methods. \n- The paper clearly motivates the idea, presents theoretical backing, and supports it with extensive empirical evidence. The direct links drawn between RePO, SimPO, DPO, and SLiC-HF clarify its positioning and are helpful for detailed understanding of the proposed method. Exhaustive analysis on gradient behavior, filtering ratio and other ablation effectively convey insights.\n- The authors throughly investigate the empirical findings and elaborate the concept of binary filtering which can be further integrated into other alignment methods. The author also provide RePO++, which further validate their idea.\n- RePO offers a more interpretable and computationally cheaper alternative for current offline alignment methods, which could be widely adopted.\n\n### Weaknesses\n\n- As DPO is one of the core method in alignment tuning, although the authors focus on SimPO, extension to DPO would make the contribution stronger. As the author stated that RePO++ can be integrated to DPO, it would be great if they can show it through additional small experiment.\n- Although over-optimization is a core topic discussed in this paper, the authors do not fully replicate the experimental setups used in prior work (e.g., Gao et al.). Specifically, they didn't use KL divergence or gold reward models, mentioning about the  concerns on computational cost. However, through experiment on this controlled setting can help practitioner to increase the reliability of the method and prevent them from additional analysis on real-world. Including KL divergence in controlled experiments (even once) would strengthen the empirical claim\n- Although the intuition was on $\\beta \\rightarrow \\infty$ in SimPO, in evaluating SimPO across varying $\\beta$, the authors use small values (1–10) but skip log-scale values (e.g., $10^2, 10^3$). This would have strengthened the argument that RePO is equivalent to high-\\$beta$ SimPO.\n- Although the authors compare RePO with SLiC-HF, DPO, and SimPO, it would be useful to explore whether RePO’s binary filtering mechanism could enhance other alignment paradigms such RLHF (KL-regularized PPO variants)pipelines. However, this is already listed in limitation and future directions.", "was_revised": true, "final_justification": "The previous concerns about the empirical solidity on the over-optimization and evaluation of SimPO under varying $\\beta$ are addressed throughly with authors. In addition, authors add more generalizability by adding IPO-RePO, which I believe makes the contribution stronger. \nWhile the authors didn't provide the explicit plot, assuming the proposed updates and additional experiments are incorporated into the final version, I would be supportive of accepting this paper.", "rebuttal": "**To Reviewer uBRY:**\n\nWe sincerely thank you for your detailed and constructive review. Your insightful questions have helped us identify key areas for clarification and further strengthening of our work. Below, we address each of your points.\n\n***\n\n**Q1: As DPO is one of the core methods in alignment tuning, although the authors focus on SimPO, extension to DPO would make the contribution stronger. As the author stated that RePO++ can be integrated to DPO, it would be great if they can show it through additional small experiment.**\n\n**Response:**\nThank you for this excellent suggestion. To demonstrate the generalizability of our proposed binary filtering mechanism, we integrated it into IPO [20], a canonical DPO-like method that directly addresses DPO's overfitting issues. The resulting loss function, which we term IPO-RePO, applies ReLU filtering to IPO's core objective:\n\n$$L_{\\text{IPO-RePO}} = \\mathbb{I}\\left(M_{\\theta, \\text{ref}} < \\gamma\\right) \\cdot \\left( M_{\\theta, \\text{ref}} - \\frac{1}{2\\tau} \\right)^2 \\quad \\text{where} \\quad M_{\\theta, \\text{ref}} = \\log \\frac{\\pi_{\\theta}(y_w|x)}{\\pi_{\\text{ref}}(y_w|x)} - \\log \\frac{\\pi_{\\theta}(y_l|x)}{\\pi_{\\text{ref}}(y_l|x)}$$\n\nWe conducted experiments on the Llama3-Instruct-v0.2 (8B) model, with results on the AlpacaEval and Arena-Hard benchmarks as follows:\n\n| Method | AlpacaEval LC | AlpacaEval WR | Arena Hard WR |\n| :--- | :---: | :---: | :---: |\n| IPO | 40.6 | 39.6 | 34.9 |\n| IPO-RePO (Ours) | **43.8** (+3.2) | **42.2** (+2.6) | **35.6** (+0.7) |\n\nThese results show that incorporating RePO's binary filtering mechanism provides consistent and significant performance improvements over the original IPO framework. This further substantiates our core claim that selectively applying gradients to challenging samples (i.e., \"which examples to learn from\") can be more critical for effective preference optimization than the specific weighting scheme of the loss function itself.\n\n**Q2: Although over-optimization is a core topic discussed in this paper, the authors do not fully replicate the experimental setups used in prior work (e.g., Gao et al.). Including KL divergence in controlled experiments (even once) would strengthen the empirical claim.**\n\n**Response:**\nThank you for this valuable suggestion. We agree that a direct comparison with KL divergence provides important context. We have indeed conducted such controlled experiments on the IMDB dataset, and the results fully align with the conclusions presented in the paper—specifically, that the mean implicit reward margin ($m_{\\text{batch}}$) serves as a reliable and effective proxy for the KL divergence trend.\n\nOur rationale for focusing on $m_{\\text{batch}}$ in the main draft was to emphasize its utility as a computationally efficient proxy that eliminates the need to calculate KL divergence, which is a key practical advantage of our method.\n\n`Given the rebuttal's constraints on figures and anonymous links, we are unable to present the detailed plots here.` However, we will add this comparative analysis, including the KL divergence plot, to the appendix in the final version to strengthen the empirical evidence as you suggested.\n\n**Q3: The evaluation of SimPO under varying $\\beta$ values didn't include the high-$\\beta$ regime (e.g., $10^2, 10^3$), which the authors reference in the motivation. The authors could include or discuss results in this range to better support the claim of RePO being equivalent to high-$\\beta$ SimPO.**\n\n**Response:**\nWe thank the reviewer for pushing for greater empirical rigor on this key claim. We have run additional experiments on Llama3-Instruct-v0.2 (8B) with high $\\beta$ values, keeping $\\gamma=0.3$ fixed as per our experimental setup. The results are presented below, alongside RePO's performance for direct comparison:\n\n| Metric | SimPO ($\\beta=10$) | SimPO ($\\beta=100$) | SimPO ($\\beta=1000$) | RePO |\n| :--- | :---: | :---: | :---: | :---: |\n| AlpacaEval2 LC | 55.8 | 56.6 | 57.1 | **57.7** |\n| AlpacaEval2 WR | 49.8 | 50.5 | 50.7 | **51.1** |\n\nThe trend clearly shows that as $\\beta$ increases into the high regime, SimPO's performance converges towards that of RePO. This provides strong empirical support for our theoretical claim that RePO is the limiting case of SimPO as $\\beta \\to \\infty$ (Lemma 3.1).\n\nWe add two important clarifications:\n1.  This trend is predicated on an appropriately chosen $\\gamma$, a point we also highlight in the paper (Observation 1, Line 148).\n2.  The small performance gap remaining even at $\\beta=1000$ may be attributed to numerical instability in the log-sigmoid function when its input becomes extremely large. This observation inadvertently highlights another advantage of RePO: its ReLU-based formulation is numerically more stable and directly achieves the desired binary filtering without being susceptible to such floating-point limitations.\n\n**Q4: Although the authors compare RePO with SLiC-HF, DPO, and SimPO, it would be useful to explore whether RePO’s binary filtering mechanism could enhance other alignment paradigms such as RLHF (KL-regularized PPO variants) pipelines.**\n\n**Response:**\nThis is an excellent point and a very promising direction for future research. Exploring whether RePO's efficient binary filtering mechanism can enhance the sample efficiency and stability of online RL paradigms like PPO is a compelling next step. The paradigm of focusing on \"which examples to learn from\" could potentially offer significant benefits in an online setting where data is generated iteratively.\n\nWe agree this is a valuable avenue to explore. In line with this suggestion, we have incorporated this discussion into our limitations and future work section (Section 6) to guide and inspire subsequent research in this area. We thank the reviewer for this insightful suggestion."}]

Author rebuttal:

Meta-review: In this paper, the authors propose a simple preference learning method, RePO, which uses ReLU activations. They theoretically show that RePO is a special case of SimPO (in the limit $\beta \to \infty$) and that its loss corresponds to the convex envelope of the 0/1 loss. RePO simplifies SimPO by eliminating the need to tune $\beta$, and it achieves competitive or superior performance compared with existing methods.

The reviewer and author discussion raised several issues, including the significance of the results, the coverage of ablations, and the scope of the experiments. After the discussion, all reviewers acknowledged the paper’s potential to some extent. The authors promised to integrate the new experiments provided during the discussion and to add experiments on overoptimization analysis, which satisfied many reviewers. Overall, although the idea is quite simple, the paper provides empirically well supported findings and offers insights of significant importance for language model alignment. In view of the above, I recommend acceptance.
Decision: Accept (poster)