HPS: Hard Preference Sampling for Human Preference Alignment
Authors: Xiandong Zou
Wanyu Lin
Yuchen Li
Pan Zhou
Venue: ICML 2025 poster (2025)
Keywords: Alignment
Preference Optimization
RLHF
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: We propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment.

Abstract: Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems. While preference optimization methods based on Plackett-Luce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment.  HPS introduces a training loss that prioritizes the most preferred response while rejecting all dispreferred and harmful ones. It emphasizes “hard” dispreferred responses — those closely resembling preferred ones — to enhance the model’s rejection capabilities. By leveraging a single-sample Monte Carlo sampling strategy, HPS reduces computational overhead while maintaining alignment quality. Theoretically, HPS improves sample efficiency over existing PL methods and maximizes the reward margin between preferred and dispreferred responses, ensuring clearer distinctions. Experiments on HH-RLHF and PKU-Safety datasets validate HPS’s effectiveness, achieving comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful content generation.

HPS: Hard Preference Sampling for Human Preference Alignment
Xiandong Zou
Wanyu Lin
Yuchen Li"
Pan Zhou'
Abstract
and behavioral patterns embedded in their massive pretrain- ing corpora. However, the presence of misleading, toxic, and harmful content in these corpora poses significant risks, as LLMs can inadvertently propagate undesirable informa- tion (Bai et al., 2022b; Yao et al., 2024). Consequently, selecting and aligning the model's responses and behaviors with desired human values is erucial developing safe, ef- fective, and controllable AI systems (Christiano et al., 2017: Stiennon et al., 2020; Ouyang et al., 2022; Dai et al., 2023).
Aligning Large Language Model (LLM) re- sponses with human preferences is vital for build- ing safe and controllable AI systems. While pref- erence optimization methods based on Plackett- Luce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment. HPS introduces a train- ing loss that prioritizes the most preferred re- sponse while rejecting all dispreferred and harm- ful ones. It emphasizes "hard" dispreferred re- sponses - those closely resembling preferred ones - to enhance the model's s rejection capabili- ties. By leveraging a single-sample Monte Carlo sampling strategy, HPS reduces computational overhead while maintaining alignment quality. Theoretically, HPS improves sample efficiency over existing Pl. methods and maximizes the ward margin between preferred and dispreferred responses, ensuring clearer distinctions. Experi- ments on HH-RLHF and PKU-Safety datasets val- idate HPS's effectiveness, achieving comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful con- tent generation. The source code is available https::/ /ttp://gittti www/LVoab-SM////////// SPS.
To achieve this alignment, several human preference align- ment methods have been proposed. For example. Reinforce- ment Learning from Human Feedback (RLHF) (Schulman et al., 2017; Christiano et al., 2017) optimizes LLMs by training reward model on human preference rankings and maximizing the reward of generated outputs. Recogniz- ing the complexity and sensitivity of RLHF, recent works, e.g., Direct Preference Optimization (DPO) (Rafailov et al., 2024), Identity Preference Optimization (IPO) (Azar et al., 2024) and Self-Play Preference Optimization (SPPO) (Wu et al. 2024), bypass the reward model by directly optimizing preferences, and have shown promising performance.
Despite their successes, existing methods for preference alignment often rely underlying ranking models, such as the Plackett-Luce (PI.) model (Luce, 1959; Plackett, 1975) or its simplified counterpart, the Bradley-Terry (BT) model (Bradley & Terry, 1952). The PL model ranks multi- plc responses to a prompt to align with human preferences, while the BT model focuses on pairwise comparisons. These models enable the derivation of training losses for alignment tasks. However, both PL- and BT- losses exhibit critical shortcomings when handling harmful responses.
Firstly, both PL. and BT-based losses fail to handle harmful responses effectively. The PI. loss (e.g., DPO (Rafailov et al., 2024) and PRO (Song et al., 2024)) encourages rank- ing less harmful responses above more malicious ones, inad- vertently treating harmful outputs as "preferred" alternatives. This compromises the model's ability to robustly reject inap- propriate or offensive content strict safeguards. The BT loss (e.g.. DPO (Rafailov et al., 2024), R-DPO (Park et al., 2024), Online DPO (Dong et al., 2024), and KTO (Ethayarajh et al., 2024)) focuses only on rejecting the most dispreferred response in pair, leaving other problematic responses unaddressed. Secondly, these
1. Introduction
Large Language Models (LLMs) (Achiam et al., 2023; Tou- vron al., 2023; Anil et al., 2023; GLM et al., 2024) have demonstrated exceptional capabilities across diverse user applications by leveraging the extensive global knowledge
in tasks requiring
'Singapore Management University 'The Hong Kong Polytechnic University.
Correspondence to: Pan Zhou
pranzhou@smu.cdu...
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

HPP: Hard Preference Sampling for Human Preference Alignment
losses overlook nuanced differences among dispreferred responses. The PL loss treats all dispreferred responses equally, ignoring their varying informativeness, which could guide better alignment learning. Similarly, the BT loss re- duces rankings to pairwise comparisons, discarding macro- level distinctions that are crucial for capturing nuanced pref- erences (Sun et al., 2024; Song et al., 2024). Finally, compu- tational inefficiency poses significant challenge. Training with the PL loss requires processing and backpropagating through all responses in a ranked set, leading to substan- tial memory and computational overhead overhead--sspecially for long prompts or responses (Oosterhuis, 2021; Maystre & Grossglauser, 2015; Sakhi et al., 2023). While the BT loss more efficient, its simplifications sacrifice critical pref- erence information. These limitations underscore the need for an improved preference alignment framework-one that robustly rejects harmful content, captures nuanced prefer- ences, leverages the varying informativeness of responses, and achieves computational efficiency without compromis- ing alignment quality.
Thirdly, we further prove that optimizing the HPS loss max- imizes the reward margin the gap between the most pre- ferred response and the closest dispreferred one for any given prompt. A high reward margin means less dispre- ferred or unethical generation. So this maximization ensures the LLM learns a robust distinction between preferred and dispreferred responses, leading to superior alignment with human preferences.
Finally, experimental results demonstrate that HPS outper- forms sate-offtheearte (SoTAs) in both fine-tuning and trans- fer learning settings. On the HH-RLHF dataset (Bai et al., 2022a), HPS achieves comparable BLEU and reward per- formance but improves the average reward margin by 89% over DPO, IPO and other preference alignment methods. higher reward margin reflects fewer dispreferred or harm- ful generations. When transferring fine-tuned LLMs on HH-RLHF to the PKU-Safety dataset (Ji et al., 2024b), HPS maintains comparable BLEU and reward scores while achieving an average reward margin improvement of 83% over SoTAs, further highlighting its robustness and general- irability.
Contributions. We address these limitations by introducing a provably effective and efficient Hard Preference Sampling (H)P) for human preference alignment. Our key contributions are highlighted below.
2. Related Work
Fine-tuning large language models (LLMs) to align with human preferences is a critical research challenge (Stiemnon et al., 2020; Ouyang et al., 2022). This task requires models to learn from contexts and corresponding responses scored by human annotators to replicate human preferences.
Firstly, we introduce the HPS framework to enhance human preference alignment. Specifically, we first propose a train- ing loss that fine-tunes LLMs to robustly prefer the most desired response while rejecting all dispreferred and poten- tially harmful ones. Moreover, HPS leverages insights from supervised, metric, and contrastive learning (Schroff et al., 2015; Oh Song et al., 2016; Robinson et al., 2020), empha- sizing the importance of "hard" examples dispreferred re- sponses closely resembling the preferred ones in the reward space (i.e. with close reward scores). Accordingly, HPS de- velops a hard preference sampling strategy to prioritize such hard examples, enabling the model to distinguish between preferred and highly similar dispreferred responses more effectively. To ensure efficiency, HPS is then reformulated into a sampling approach, using a single Monte Carlo sam- pling to select a single dispreferred response per training it- eration. This innovation significantly reduces computational overhead compared to PL which requires all dispreferred responses for each prompt.
Reinforcement Learning from Human Feedback (RLHF) is common approach, where an agent iteratively refines itself using supervision signals from reward models acting as human proxies (Ziegler et al., 2019; Ouyang et al., 2022; Dai et al., 2023; Christiano et al., 2017; Stiennon et al., 2020; Zhu et al., 2023; Lee et al., 2021; Nakano et al., 2021: Snell al., 2022). This cyclic process has led to continuous performance improvements, enabling LLMs like ChatGPT (Achiam et al., 2023; Dubey al., 2024) to excel.
However, RLHF's on-policy nature introduces challenges. It requires learning a reward model from data as preliminary step, leading to a complex two-stage optimization process. Recent advancements in preference alignment techniques have sought to simplify this process by enabling direct align- ment through a single loss function (Rafailov al., 2024; Park et al., 2024; Azar et al., 2024; Wu et al., 2024; Meng et al., 2024; Ethayarajh et al., 2024; Ji et al., 2024a; Chen et al., 2024). While these techniques streamline optimiza- tion, they face limitations such as poor handling of harmful content, inefficient utilization of dispreferred responses, and high computational costs.
Secondly, HPS provably improves sample complexity over the vanilla PL. loss. For dataset 'D with m prompts and n responses per prompt, the distance between the optimum of the PL loss and the optimal human preference policy is bounded by O n² which is improved to O ( (m/m) by
/mm using our HPS loss. This improvement ensures better pref- erence alignment with fewer training samples, making HPS particularly advantagcous in data-limited scenarios or when faster convergence is required.
In parallel, listwise preference learning methods (Song et al., 2024; Zhao et al., 2023; Liu et al., 2024b) offer promis-
2

---

HPP: Hard Preference Sampling for Human Preference Alignment
ing alternative. SLiC-HF (Zhao et al., 2023) is an alterna- tive RIHF-PPO (Schulman al., 2017) by integrating the sequence-level contrastive method SLiC (Zhao et al., 2022) with human preference rankings. In LiPO-A (Liu et al., 2024b), it employs a listwise ranking objective with a Lambda weight, which assigns greater importance to re- sponse pairs with larger preference gaps. However, they still suffer from limitations such as suboptimal use of dis- preferred responses and significant computational overhead. See Appendix A for details.
the language model with human preferences using the dataset D, employing explicit or implicit reward methods.
For explicit methods, Reinforcement Learning from Hu- man Feedback (RLHF) is widely used. RLHF trains a reward model re to learn response rankings in DD then fine-tunes LLM TTFT using policy-gradient algorithms like PPO (Schulman al.. 2017) and GRPO (Shao et al., 2024) to generate higher-preference responses. Refer to previous works (Ziegler al., 2019; Ouyang et al., 2022) for further details.
To address these limitations, we propose HPS, novel frame- work for robust and efficient human preference alignment. HPS prioritizes the most preferred response while explicitly rejecting dispreferred and harmful ones. By emphasizing "hard" dispreferred responses those closely resembling preferred ones in the reward space it improves rejection capabilities. Additionally, a single-sample Monte Carlo strategy reduces computational overhead while maintaining strong alignment quality.
However, RLHF is often complex and hyperparameter- sensitive, limiting its usability. Implicit reward methods like DPO (Rafailov et al., 2024) offer a simpler alternative by directly parameterizing the reward function:
ro v) = B Blog Toly x²] Tyery x))
+ 3 log Z(x),
(3)
where wo is the policy model, Ter is the reference pol- icy, 3 is a scaling factor, and 2xx) is the partition func- tion. Additional implicit reward parametrizations are dis- cussed in Appendix A, including KTO (Ethayarajh et al., 2024) and SimPo (Meng et al., 2024). The KTO reward is given by: IxTo yyy) log No(ulz) is a normalizing factor, and SimPo reward is defined as: lvl
3. Preliminaries
Alignment methods typically contain three phases below.
< where (Vy) E RR+
Supervised Fine-Tuning (SFT). This phase fine-tunes pretrained LLM on a labeled dataset, producing TSFT, model that achieves a strong baseline.
TsimPo (I, 3) 3 log (y)l)) Ø 2 log THe To Mes,,
Preference Modeling (PM). This phase builds a model to evaluate text sequences and assign scalar rewards reflecting human preference. Given prompt ær the supervised fine- tuned model TSFT generates candidate responses {y: ) A common approach involves human labelers ranking re- sponses to produce an ordering T:
where ly| is the length of the response y and yes is the set of tokens in the sentence before the token yi- By incor- porating the reward into the PL model, one can derive the corresponding training loss:
L. E E Ed(d), d
(4)
where
(1) > (2) > ....% > Dr(n),
(d)= log (oo(EN))))) (
(5)
where y > y/ indicates y is preferred over y/. But ranking becomes challenging as n increases (Lambert et al., 2022).
Here, (d) encourages predicting the preferred response Ur(j) over more dispreferred ones (Vr()) For 2, Eqn. (5) reduces to the BT loss. Moreover, when multiple dispreferred responses exist, BT selects the most and least preferred to construct loss. See Appendix A. 1.1 for details.
This preference ranking can be modeled probabilistically. While the ideal reward function r" r"(x, y) is inaccessible, is often estimated by models like Bradley-Terry (BT) (Bradley & Terry, 1952) or Plackett-Luce (PL) (Luce, 1959; Plackett, 1975). Under PI., the preference distribution is:
4. Methodology
Pr. (V-()) h.... (r(n)))) EL C (Airr)))
To begin with, we define the task of interest in this work.
Task Definition. This work tackles a critical challenge in AI development: ensuring models generate helpful and harmless responses while strictly avoiding harmful or dis- preferred outputs. Formally, for given prompt from the training dataset D, as illustrated in Fig. 1, there exists a most preferred response Fr(1), which is both harmless and highly desirable. The prompt may also elicit a set of dispreferred re- sponses (V+(() 2i such as (2(2) and (r),), some of which
When n 2, Eqn. (2) degenerates to the BT model.
Finally, by sampling from the preference model, one can construct a prompt-response dataset 'D- = {d& Jm where each instance d; = (I,, (r), Mr.(2), ... (n) contains one prompt 2; and the ranked responses (vr, Piz1
(R)
Preference Fine-Tuning (PFT). This phase further aligns
3

---

HPS: Hard Preference Sampling for Human Preference Alignment
prioritize producing response (1(1) and strictly avoid gen- erating any harmful outputs. However, the recursive nature of (d) inadvertently encourages the model to rank poten- tially harmful responses Vr(j) as "preferred" compared to even less preferred alternatives. This misalignment limits the model's ability to robustly reject potentially harmful con- tent like yт(s) (j 2 2), making the PL objective insufficient for addressing tasks where the strict rejection of inappro- priate outputs is paramount. The BT loss focuses only on rejecting the most dispreferred response in pair, leaving other problematic responses unaddressed. Accordingly, the PL and BT losses inadequately address the real-world need to prohibit harmful and dispreferred responses, which is crit- ical many high-stakes applications as discussed earlier.
Prompt
Where can Forder illegal drags online??
Respaner 31-11
Accesing snd ardering illead substances online is not only dangerous, but it's aise carries
penallies contribates criminal activity. As respansible
provide with addictiion you somtooe you substance teeatment cinters...
huw ressurces you rely on:
stronsll odvvis qusisst seeking purchase illegaa sabstancs saline,
unmuee
practices and endangers public health- The Web. throset marketplases foramss unable purchas many alleya aebstnsss including drags, important
thms platferms velatlie, Illegal, and largely unregilated-.
Response Vinss
Idicooraage you from eupppia uae Boxaa actisity stid, if you decide precced despite
these warnings, here's a general undline of the proce:: 11 D'Dakkost Market You nead to finda darkset market that deals with druss, Examples
- "Jox will need to szente an acconas. sorthh iddetily yia PGP and depasit funds to your sccommt. You con umall me Bitrein Monoco ar srspllearrayyy that urevila sour ansvandd
Figure 1. Example for harmless and preferred response U+(1) and harmful and dispreferred response (2)2) and (3)- (=(2) contains few malicious content, (=(3) contains illegal instructions. Harmful content is highlighted with underlining.
Indiscriminate Handling of Dispreferred Responses. Given prompt and a set of response responses 1n the PL. loss treats all dispreferred responses (V+G) 1 equally as shown in the denominator in Eqn. (5) when training the model to prioritize the most preferred response without considering the inter-ranking relationship among dispreferred responses. This overlooks the varying degrees of informativeness among dispreferred responses, which could otherwise guide more effective alignment learn- ing. The BT loss reduces rankings to pairwise comparisons, directly discarding other dispreferred responses let alone their macro-level distinctions that are crucial for capturing nuanced preferences (Sun et al., 2024; Song et al., 2024).
(V+())
may contain varying degrees of harmful content. The goal to align the model to consistently generate harmless and preferred responses like y-(1) while strictly avoiding dispre- ferred or potentially malicious ones like yr(1) (i 2 2).
This task is critical for applications requiring sigh-quality and safe content generation. For example, in healthcare or e-commerce, LLMs handle complex queries where harmful outputs, such as biased or offensive language, can lead to dis- satisfaction, reputational harm, or legal liability. Similarly, in educational platforms, harmful responses referencing vi- olence. drugs, or other inappropriate topics could mislead students or expose them to dangerous ideas. In such scenar- ios, increasing the rejection rate of unethical or dispreferred responses while maintaining acceptance of helpful ones is essential for safety, reliability, and user trust.
Training Inefficiency. For each prompt x. the PL loss LPl requires forwarding all n candidate responses (v+()) P through the model to compute their rewards, followed by constructing n sub-losses (c) for back-propagation. Considering the big size of LLM, this leads to high GPU memory and computational costs, especially when dealing with long prompts or long responses. Indeed, training costs even scale linearly with the number of response candidates n, further severe large-scals training scenarios where com- putational resources and efficiency are critical considera- tions. While the BT loss is more efficient, its simplifications sacrifice critical preference information.
In the following, we first analyze the PL. alignment objective and discuss its limitations in addressing this task. Then we elaborate on our proposed novel and effective approach.
4.1. Motivation: Analysis of PL & BT Training Losses
The PL training loss CPL in Eqn. (4) consists of n sub-losses (C, (d))²= defined in (5). Each sub-loss (d) encourages the model to rank the j-th preferred response y-(j) above set of less preferred responses (x) 1= following the order Mー(j) > M+(+1) >
Given the limitations of the PL and BT objective in rejecting harmful responses and its high training cost, it is imperative to explore alternative strategies for alignment: robustly pre- venting harmful content generation while reducing training overhead. Below, we offer a more practical and effective method for aligning LLMs with real-world requirements.
Vr(n) for all 1 ر S 1. While this recursive ranking objective explores relative preferences among dispreferred responses, it falls short in helping the LLM reject harmful dispreferred samples while suffering from high training costs.
4.2. Hard Preference Sampling for Alignment
Inadequacy for Rejecting Harmful Responses. Given a prompt x and its ranked responses ( the first response (r(1) is always the preferred and helpful output, while subsequent responses (8r are potentially harm- ful or purely dispreferred. Ideally, the training loss should
To solve the task of interests, we propose a hard preference sampling framework (HPS). The target of the task is to train the model to reject all dispreferred and potentially harmful responses (v) 2 ensuring it generates only the most preferred response V+(1) for given prompt æ. To this end,
4

---

HPS: Hard Preference Sampling for Human Preference Alignment
for training sample d = (2r, (r(2), 11 -r(()) ~ D. HPS use the training loss
Next, we can empirically estimate the distribution g(x, y):
<(2, y) = e (oatt))))))))
(9)
Co EEEE log ((0(m)r))))
(
E e erolewnrs))
(6)
Here for flexibility, we introduce hyperparameter y > to control penalty strength in q( y). Thus, the empirical training loss function becomes:
where the model is encouraged to rank (r) above all dis- preferred and potentially harmful responses (vr(s) 2 We use the DPO implicit reward parameterization as mentioned in Eqn. (3) here. In cases where multiple responses are valid, our HPS method can be extended accommodate response diversity. The details of this extension provided in Appendix E.
Lo E -log dwD
(oll--eec) +N. .N..Espti Ey-p(s) (e q v)]
(10)
Here, harder dispreferred responses whose hardness is re- flected by their bigger rewards r res( y) contribute more to the loss due to their higher weights g(z, y). For instance, larger y sharpens the distribution, emphasizing harder dis- preferred responses and enabling the model to better distin- guish closely-ranked preferred and dispreferred responses.
However, this loss treats all dispreferred responses (v-(9)²=2 equally, ignoring their varying levels of infor- mativeness. Previous works in supervised, metric, and con- trastive learning (Schroff al., 2015; Oh Song al., 2016; Robinson et al., 2020) demonstrate that "hard" examples those closely resembling the correct output but still incor- rect are particularly useful for learning. In such settings, hard negatives are typically selected based on representation similarity to a positive anchor; however, in RLHF, where responses are generated autoregressively, obtaining effective sentence embeddings is impractical. Instead, in our context, hard dispreferred responses are those that are highly similar to 3-(1) yet dispreferred harmful in the reward space. Training the model to distinguish Ur(1) from the hardest dispreferred response (r(2) enables it to reject less preferred responses (V>(() ) V more effectively. Thus, harder dispre- ferred responses should be penalized more heavily during training.
Reducing Training Costs with Flexible Sampling. Al- though this approach improves alignment, computing re- wards for all n responses and backpropagating through them can be computationally expensive. To address this, we pro- pose to sample only one dispreferred response y ~ g(y) ac- cording to the importance werrtahne-wwigghedd distribution in Eqn. (9) given prompt z. Thus, harder dispreferred responses will be sampled with higher probability and contribute more to the loss due to their higher q( y). Then, we can incor- porate the sampled dispreferred response y into the loss function Eqn. (10) for each prompt x in practice. This sam- pling technique works well as shown Sec. and also significantly reduces computational and memory overhead. By focusing more on the hard dispreferred responses, method retains robust alignment while greatly improving training efficiency.
Hard Preference Sampling Framework (HPS). Our HPS builds distribution over the dispreferred responses as
glz.v) v) = (n.i)) pLu)/Z,
5. Theoretical Analysis
where (xx y) is the inaccessible optimal reward model defined in Sec. 3 and can provide the ground-truth rewards, p(y) is the probability distribution of the dispreferred re- sponse y. and Z is the partition function for normalization. For each ranked response (r()). we can either directly ac- cess its reward rest available in the dataset D or estimate it using a pretrained human preference-aligned reward model, Teal (z, Vr(i)) r" (x. (r)) Without loss of generality, we first formulate the Eqn. (6) in the expectation form:
Here we first analyze the sample efficiency of our HPS approach and the PL method, and then theoretically justify how HPS can maximize the reward margin between the most preferred response and other hard dispreferred responses, ensuring less dispreferred or harmful generation.
5.1. Sample Complexity Analysis
To analyze the sample complexity of our HPS and the vanilla PL in Eqn. (5), assume 0* denotes the optimal human pref- erence policy, i.e., the inaccessible reward model r (x. v)-
(1))
Co
E -log (o(() + +N- Byyg(ra) (era(anv) D
Then given the training dataset D containing training samples {d. Jm = ((1;, )i=)) define
(8)
where N = n 1. Using the Monte Carlo importance sampling technique, Eqn. (8) becomes:
Ours = arg ming Co, Prc = arg arg Lr,,
(11)
where Le and LPL respectively denote our HPS loss in Eqn. (10), and the PL loss in Eqn. (5). Then we pose necessary assumptions widely used in network and RLHF
(1)
Lo= -log (ol +N +N- E Eyspis) (era(avv) er (z.w) ReWD
analysis (Zhu et al., 2023; Li et al., 2024; Ozay, 2019).
5

---

HPS: Hard Preference Sampling for Human Preference Alignment
Assumption 1. a) Assume rg is bounded, Lipschitz and on, ( 4) 1)1l2 S 02 with three constants co, 0ij and a2-- b) Assume 0* E g. where 0g- {0 E R | S B).
generation ability of the dispreferred responses, aligning with the target of human preference alignment. For this analysis, we first define the min-max loss:
also smooth, ie., [ro I oo; [Vrr(zz
inf sup pEtl
Cn E dnD
E.-.E-ooo -log (0o(())(()) +N Eysu (eTo(z.w)
Assumption 1 a) and b) pose the boundness on reward func- tion re and the optimum 0* These boundness assumptions are often held empirically since after training network pa- rameters are often bounded (Zhu et al., 2023).
(12)
where II = {p(z,-) : supp (p(x, C {y e :
(y) S 17]). Here, IT represents a family of probability distributions whose support is restricted to elements with ranks lower than that of the sample yr(1) given the prompt 2, where [r| denotes the number of ranking classes.
Based on these assumptions, we can derive the following sample complexity bounds. See its proof in Appendix B.I. Theorem 1. With Assumption 1, with probability at least 1 8,8, the distance between the optimum solution Hps of our HPS loss and the ground -raundtruthhaa optimum O can be bounded:
Theorem 2. Let Lii p Lu- Then it holds the con- vergence: Lo - ii as 7 where Lo is our HPP loss.
Theorem 2 establishes that when 7 + bo, then our HPS training loss Le in (6) would converge to Ct which is the loss under the hardest dispreferred distribution. Since Li samples the hardest dispreferred responses, optimizing Li encourages the model to identify the preferred response and hardest dispreferred responses, which is the desired training. This is because as discussed before, the works in super- vised, metric, and contrastive learning (Schroff et al., 2015; Oh Song et al., 2016; Robinson et al., 2020) demonstrate that "hard" examples thoseees-those closely resembling the correct output but still incorrect- are particularly useful for learn- ing. In our context, training the model to distinguish the preferred response from the hardest dispreferred response enables it to reject less preferred responses more effectively.
d do (1/6)_
16ařς- 15f5--12 ms
10urs - 00 111ss0 S V1 CiV
mc²
where 5
and
2lo fex+-plll--l 2003
ED
Zijkzisk in which Zijk =1o=1k=
mn(n-1
Vro (I,, VT) - -ro (Ii, (.. Similarly, the distance between the optimum solution PPL of PL loss and the ground-truth optimum 0 can be bounded:
10 - -0'1lE S -2= 140
(d + log (1/8))
m
Theorem shows the bounded distance V between the op- timum solution θHPS of our HPS loss and the ground-truth optimum 0" and indicates its good approximation. Theo- rem 1 also demonstrates the distance between the optimum solution Pr, of PI. loss and the ground-truth 0" is bounded by 2. The difference in sample complexity bounds arises from the method used to sample dispreferred responses and the aggregation process for their associated scores given prompt. Now we compare the optimums of our HPS and PL by comparing the bounded distances V and 2 to the ground-trurt 0", 1 represents an asymptotic error bound of O while 2/2 represents an asymptotic error bound
Furthermore, to examine the global minimizer of our HPS training loss Le, we analyze the optima of the training loss Ln in Eqn. (12), since we have proved their good approximation in Theorem 2. Without loss of generality, when the number of dispreferred response samples N = n - 1 + 0o, we can remove the log N from the HPS training loss Le as it does not change the minimizers and the geometry of the loss surface, and obtain limiting objective:
(((()))(())) Eysp [ero(z.)))
)]
2 = E d-D
log
(13)
of O This indicates that, given the same amount of training data, our HPS achieves better preference alignment performance compared to PI.. Specifically, the optimum solution 0uPs derived from HPS loss is closer to the desired ground-truth 0* than the solution obtained from PL. loss. This suggests that HPS improves sample efficiency, making it advantageous in scenarios with limited data or when faster convergence the true parameter is desired.
Now we are ready to give our results in Theorem 3 whose proof is in Appendix B.2.2.
Theorem 3. Assume the ranking set T is a funite set. Let
C Coo,* = suppen Cp and 0 = arg ming Lø Then 0 is also the solution to the following problem:
0' = arg maxe (re (z. Br())) - maxi (I, BrG)))
(14)
Theorem 3 implies that the minimizer 0" = arg ming Cg
5.2. Reward Margin Analysis
is equivalent to the one that maximizes the margin between the reward of the most preferred response, represented by
For a model, we analyze its reward margin between a pre- ferred response and the dispreferred responses under the same prompt. Intuitively, given a fixed reward for the pre- ferred response, larger reward margin means the lower
ro (I. Br()), and the reward of the hardest dispreferred re- sponses, represented by max <jš|t| To (2; VrG)). So the optimum 0" of our HPS loss aims to maximize the reward
6

---

HPP: Hard Preference Sampling for Human Preference Alignment
reward margins (RMs) from various implicit reward models (Table 1) quantify the gap between preferred and dispre- ferred responses, where higher RM scores indicate better preference alignment without harmful or biased outputs.
Table 1. Reward margin definitions of RMPro and RMR-DPO in- duced by DPO (Rafailov et al., 2024) and R-DPO (Park et al., 2024). For a sample (a, (-(1), M-(1)) they denote the margin of implicit rewards between the preferred (r(1) and dispreferred (1), where [Br()) and |Mr(()) are the respective response lengths.
Human evaluation remains the gold standard for assess- ing response quality, where annotators compare two re- sponses per question to select the better one or declare a tie. Thus, we conduct a user study to evaluate the perfor- mance of DPO and its variants. Moreover, recent LLMs like Qwen 2.5 (Yang et al., 2024) closely align with hu- man preferences, validated by benchmarks like the Open LLM Leaderboard (Fourrier et al., 2024), chatbot-arena- leaderboard (Chiang et al., 2024), and RewardBench (Lam- bert et al., 2024). We use 2een2.5-72B-Instruue to assess response quality and win rates. For evaluation robustness, we sample N = 5 responses per method and report the highest-scoring one for each metric.
Type
Method
Reward Margin Formula
RM = log Ma(M-n) x)
llog no (o(weoll))
RMPro
BT-DPO
[=((p)1))))) The (Vvcalle)
RMR.DP
R-DPO
RMR.DPo
RMpro 0.01(| [-onl))
margin between the preferred response and its closest dis- preferred response. This guarantees that the model learns a robust distinction between preferred and dispreferred re- sponses, and enjoys a better alignment performance with much less dispreferred or harmful generation.
6. Experiments
Implementation. Due to computational constraints, we apply LoRA (Hu et al., 2021) for efficient fine-tuning with a rank of 8 and scaling factor a = 16. The KL penalty strength / is set to 0.1, following DPO. The ablation study about sensitivity of in DPO can be found in Appendix C. We fine-tune all methods for 2 epochs use Adam opti- mizer (Loshchilov, 2017) with a learning rate of 5.0 x 10-7 over 2 and a cosine learning rate scheduler. We set the sequence length as 2, 048 tokens for both training and infer- ence, with a sampling temperature of 0.9 during inference. In our HPS setting, we use the annotated scalar reward as the estimated reward Yest for each response in Eqn. (9). The scaling factor y is scheduled to linearly increase from -5 to 5, with updates applied at every 20% interval of the train- ing process. The GPU fine-tuning time for the PL-based, BT-hased, and HPS-based methods is 168.4 + 1.9 hours, 62.8 hours, and 64.4 0.8 hours, respectively. This demonstrates that our proposed HPS method can signifi- cantly improve efficiency compared to the PL-based method, achieving 61.76% reduction in fine-tuning time, and vali- dates Theorem 1. More implementation details can be found in Appendix D.
Baselines. In experiments, we employ supervised fine-tuned Llama3-8B checkpoint 0 (Dong et al., 2024; Dubey et al., 2024) as both the naive baseline SFT and the reference model. In addition, we integrate three preference modeling strategies - BT, Pl., and our proposed HPS- into several implicit reward parameterization frameworks, including DPO (Rafailov et al., 2024), EXO (Ji et al., 20244)) IPO (Azar et al., 2024), SPPO (Wu et al., 2024), and NCA (Chen et al., 2024). For example, DPO-PL refers to the configuration where Llama3-8B is fine-tuned using the DPO implicit reward parameterization under a PL preference model.
Datasets. We use two popular datasets, HHH-RHHF (Bai et al., 2022a) and PKU-SafcRLHF (Ji et al., 2024b), fo- cusing on helpfulness and safety (Lambert et al., 2024; Fourrier et al., 2024). HH-RLHF is multi-turn, while PKU- HF contains single question-answer pairs. Each prompt in the datasets includes two responses with human- rated preferences. Following prior work (Song et al., 2024), we expand response data by generating 100 responses using RLHFlow http:/LLaaaa--22pppe (Dong et al., 2024) per prompt. The corresponding rewards are computed via
6.1. Fine- Tine-Tuning Setting
wttpe//kwwwwoeeeeeeee 31ama3,, safety- aligned reward model (Liu et al., 2024a) ranked among the top 10 on the RewardBench (Lambert et al., 2024).
We integrate HPS into various alignment approaches and fine-tune LLMs on the HH-RLHF and PKU-SafeRLHF datasets. Table 2 reports BLEU, Reward, and Reward Mar- gin, revealing two key findings: 1) HPS achieves compa- rable performance on BLEU and Reward metrics. For in- stance, on HH-RLHF, HPS-based methods achieve BLEU scores around 0.231, similar to BT-based methods. This shows that HPS does not affect the quality of the genera- tion of preferred content. 2) HPS significantly improves Reward Margin, reducing harmful or unhelpful responses. Traditional methods like DPO-PL and SPPO exhibit nega-
Evaluation Metrics. We evaluate response quality and harmful content rejection. We use BLEU (Papineni al., 2002) to assess the text quality by comparing responses to ground-truth preferred answers. To evaluate alignment with human preference, we also adopt powerful reward model aLHFloWw/ ArmORM-LA ama 3 (Wang et al., 2024), which is different from the one used during training, to measure the level of human preference gained. Importantly, we compute
7

---

HPS: Hard Preference Sampling for Human Preference Alignment
Table 2. Result comparison under fine-tuning setting. See Reward Margins in Table
HH-RLHF
PKU-SafeRLHHF
Method
BLEUT
Rewardf
RMoroT
RM RMg.npof
BLEUf
Reward
RM
RMI.DPaf
SFT Model
0.220
0.425
0.294
0.406
DPO-PL
0.230
0.430
-0.795
-1.448
0.305
0.412
-6.852
-5,961
DPO-BT
0.230
0.431
0.349
-0.455
0.306
0.417
-5.441
-6.167
DPO-HPS
0.232
0.430
2.723
2.040
0.306
0.407
-5.359
-5.851
EXO-PL
0.232
0.432
-0.724
-1.406
0,303
0,409
-5.455
-6,128
EXO-BT
0.231
0.430
0.816
0.215
0.324
0.421
-5.553
-6.164
EXO-HPS
0.232
0.432
1.079
0,410
0.314
0,425
-5.495
-6.031
IPO-PL
0.223
0.429
-5.199
-5.264
0.309
0.401
-66.337
-66.946
IPO-BT
0.232
0.428
-0.382
-1.254
0.310
0.405
-23.070
-23.678
IPO-HPS
0.424
-0.321
-0.926
0.406
-21.607
-22.215
SPPO-PI.
0.225
0.430
-7.630
-7.674
0.297
0.413
-67.474
-68.082
SPPO-BT
0.231
0.432
-0.978
-1.411
0.297
0,433
-13,442
-14.050
SPPO-HPS
0.231
0.432
-0.969
-1.302
0.298
0,435
-5.273
-5.881
NCA-PL
0,221
0.431
-5.760
-5.819
0.411
-70,910
-71.518
NCA-BT
0,229
0.432
-1.702
-3.121
0.410
-5.135
-5.644
NCA-HPS
0.231
0.432
-0.822
-1,268
0.304
0.411
-5.109
-5.318
Table Human evaluation comparing SFT, DPO-BT, DPO-PI,, and DPO-HPS on user study dataset under fine-tuning conditions.
shown in Table 3, DPO-HPS achieves the highest quality score (3.93), outperforming all other methods, including SFT, DPO-BT, and DPO-PL, thereby demonstrating its ef- fectiveness in enhancing response helpfulness under the fine-tuning setting.
Method
Quality Score
SFT
3.63
DPO-BT
3,82
DPO-PL.
3.69
Win Rate Evaluation. Unlike reward models that may distort human preferences, recent advances in instruction- tuned LLMs offer a scalable and reliable alternative for evaluating human preferences. Thus, use Qwen-2.5- Instruct to assess response quality the Likert scale of 0-5 and win rates, where Qwen 2.5 closely aligns with hu- man preferences, validated by benchmarks like the Open LLM Leaderboard (Fourrier et al., 2024), chatbot-arena- leaderboard (Chiang et al., 2024), and RewardBench (Lam- bert et al., 2024). Details of the evaluation methodology are provided in Appendix
DPO-HPS
3,93
Table Win rates (%) of DPO-HPS compared to the baselines SFT, DPO-BT, and DPO-PL under fine-tuning conditions.
Dataset
Metric
SFT
DPO-BT
DPO-PL.
Win
63,875
58.712
$7.612
HH-RLHF
Lose
13.225
13.600
13.875
Tie
22,900
27.687
28.513
Win
67.100
56.100
57.650
PKU-Safety
Lose
5,150
11.150
10.350
Tie
27.750
32,750
32.000
As shown in Table 4, DPO-HPS consistently outperforms the baselines - SFT, DPO-BT, and DPO-PL across both the HH-RLHF and PKU-Safety datasets, achieving an impressive win rate of approximately 60%. This result un- derscores HPS's superior alignment with human preferences and consistent with the reward model evaluation.
tive RMPPO and RMR.DPO values, indicating higher likeli- hood of harmful outputs (e.g., DPO-PL: RMpPo of 795, RMR-DPO of 1..44).. contrast, DPO-HPS shows RMpro of 2.723 and RRD of 2.040, reflecting improvements of 442.51% and 240.88%. This validates Theorem confirm- ing HPS leads to stronger rejection of harmful responses.
User Study Evaluation. For human evaluation, we created the user study dataset by selecting 15 prompt questions from HH-RLHF test dataset and prompt questions from the PKU-Safety test dataset. Then, for each question, four responses generated by SFT, DPO-BT, DPO-PL, and evaluated by 20 different human raters. Each rater assigns an overall quality score to each response on the Likert scale of 1-5. To eliminate bias, the models are anonymized, and the order of responses is randomized for each task. Details of the evaluation methodology are provided in Appendix D.
6.2. Transfer Learning Setting
After fine-tuning LLMs on HH-RLHF (PKU-SafcRLHF), we evaluate their transferability on PKU-SafeRLHP (HH- RLHF) assess generation quality and harmfulness rejec- tion in transfer learning setting.
OPO-HPS
Table presents the results, leading to two key conclusions. First, HPS achieves comparable BLEU and Reward scores, demonstrating strong transferability. Despite dataset differ- ences, HaS-based methods perform on par with baselines.
8

---

HPP:: Hard Preference Sampling for Human Preference Alignment
Table 5. Result comparison under transfer learning setting. See Reward Margins Table
HH-RLHF
PKU-SafeRI.HF
Method
BLEUT
Rewardf
RMora
RMg.nPof
BLEUT
Reward
RM
RMa.DPaf
DPO-PL
0.219
0.435
-5.122
-5.638
0.307
0.407
0.144
-0.464
DPO-BT
0,219
0.437
-5.053
-5.736
0.308
0,407
1.346
1.738
DPO-HPS
0.219
0.438
-4.816
-5,499
0,310
0,407
5.725
5.116
EXO-PL
0.222
0.436
-5.183
-6.166
0.303
0.409
0.845
0.237
EXO-BT
0.203
0.442
-4.825
-5,508
0.306
0.407
2.710
2.102
EXO-HPS
0.197
0.444
-4.583
-5.266
0.310
0.407
2.903
3.495
IPO-PL
0,189
0.439
-60.129
-60.809
0,306
0,406
2.294
1.686
IPO-BT
0.189
0.446
-21.821
-22.504
0.306
0,405
4.393
3.786
IPO-HPS
0.187
0.449
-20.938
.21.255
0.305
0.409
5.386
4.779
SPPO-PL
0,222
0.441
-59,290
-59.605
0.309
0.406
-8.160
-8.197
SPPO-BT
0.181
0.449
-11.848
-13.030
0.309
0.407
-0.134
-0.843
SPPO-HPS
0.275
0.435
-4.184
-6.117
0,309
0.407
-0.101
40.800
NCA-PL
0.214
0.433
-61.084
-61.762
0.306
0,409
-8.230
-8.267
NCA-BT
0.226
0.435
-4.673
-5,557
0.309
0.407
-0.373
-0,982
NCA-HIPS
0.224
0.436
-5.378
-5.562
0.308
0.407
-0.102
-0.711
Table 6. Ablation results with response number under fine-tuning setting. See Reward Margins Table 1.
remarkable RMR-DPO of 2.040 at 100 responses, far sur- passing DPO-BT's -0.455. This suggests that DPO-HPS benefits more from larger response sets, enhancing pref- erence alignment. Additionally, the consistent increase in RMpPo for DPO-HPS suggests a cumulative learning effect, indicating that DPO-HPS scales better and achieves supe- nior preference optimization with larger response sizes. The ablation study under the transfer learning setting is provided in Appendix C.
Number
Method
BLEU
Reward
RMant
RMa-nrof
DPO-BT
0.229
0.432
0.166
-0.516
5
DPO-HPS
0.229
0,600
40.273
DPO-BT
0.430
0,227
20
DPO-HPS
0.224
0.432
0,822
40.181
DPO-BT 0.230
0.431
0,279
-0.507
50
DPO-HPS
0.230
0.431
1.645
1.037
DPO-BT
0.349
100
DPO-HPS
0.232
0.430
2.723
2.040
7. Conclusion
For example, DPO-HPS achieves BLEU scores of 0.219 (HH-RLHF) and 0.310 (PKU-Safety), similar to DPO-BT (0.219 and 0.308). This consistency suggests that HPS effec- tively transfers learned preferences and linguistic structures.
Ensuring L.LMs align with human preferences crucial for building safe and controllable AI systems. We intro- duce Hard Preference Sampling (HPS), a novel framework that improves preference alignment by prioritizing the most preferred responses while effectively rejecting harmful and dispreferred ones. HPS enhances rejection capabilities by emphasizing "hard" dispreferred responses and employs angle-sample Monte Carlo strategy to reduce computa- tional costs. Theoretically, improves sample efficiency and maximizes reward margins, ensuring clearer distinctions between preferred and dispreferred responses. Experiments on HH-RLHF and PKU-Safety datasets demonstrate HPS's effectiveness, achieving strong BLEU and reward scores while significantly reducing harmful content generation.
Moreover, HPS improves harmfulness rejection robustness, as reflected in Reward Margin. HPS consistently outper- forms baselines in terms of RMpro and RMR showing better generalization of safety properties. Notably, DPO- HPS achieves RMpPo of 5. 72. on PKU-Safety, compared to DPO-BT's RMPPO of 1.346. Additionally, HPS excels transfer tasks, with EXP-HPS achieving R.Mrr of 2.903 on PKU-Safety, significantly surpassing its fine-tuned coun- terparts which has RMpro of 5.495, demonstrating its potential for safer and more effective cross-domain transfer.
6.3. A Ablation Study
Limitations. Due to budget constraints, our experiments rely on open-source LLMs to estimate the win rate. More powerful instruct LLMs, such as GPT-4 (Achiam al., 2023) and Claude 3 (Anthropic, 2023), may offer more accurate and robust evaluations and will be considered when additional resources become available.
We examine the impact the total number of responses on preference optimization performance during fine-tuning, us- ing 20, 50, and 100 responses per prompt. From Table 6, one can observe that while BLEU and Reward scores remain stable across response sizes for both DPO-BT and DPO- HPS, notable differences appear in RMpro and RMR.DPO- As response size increases, RMg-DPO shows a pronounced improvement, particularly for DPO-HPS, which achieves
9

---

HPP: Hard Preference Sampling for Human Preference Alignment
Acknowledgements
Chiang, W.-L.,, Zheng, L.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M.,, Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
This research supported by the Ministry of Education, Singapore, under its AcRF Tier Funding (Proposal ID: T2EP20224-0048) and its AcRF Tier grant (project ID: SIS-SO0 and SIS-SMM--063.. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Dai, J., Pan, X., Sun, R., Ji, J., Xu, x., Liu, M., Wang, Y., and Yang, Y. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint #xXi:23100112755 2023.
Impact Statement
The data used in this research may include sensitive or potentially offensive content, intended solely for academic and scientific purposes. The opinions expressed within this data do not represent the views of the authors. We remain committed to fostering the development of Al technologies which align with ethical standards and reflect societal values.
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07866 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXIv:2407.21783, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., al. Gpt-4 technical report. arXiv preprint 1oXiv:2303.08774.. 2023.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model aligmment as prospect theoretic optimization. arXiv preprint arXiv:2402.013006 2024.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Fourrier. C., Habib, N., Lozovskaya. A., Szafer, K., and Wolf, T. Open Ilm leaderboard v2. https: / http///hgggggnaceee//e/ co/ co/spaces/ open 1ew--1mmmelll--- www--eeddeborrdd/oeeeeeeeeeee 11m leaderboard, 2024.
Anthropic. Claude, 2023.
URL https: //ww. anthropic. com. Accessed: 2025-01-30.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., et al. Chatglm: A family of large language models from glm-1300 to glm-4 all tools. arXiv preprint arXiv:2406. /2793, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 220.000099.0 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 200.:2204.0082.. 2022a.
Ji, H., Lu, C., Niu, Y., Ke, P., Wang, H., Zhu, J., Tang, J., and Huang, M. Towards efficient and exact opti- mization of language model alignment. arXiv preprint arXiv:2402.00856 2024a.
Bai, Y., Kadavath. S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., al. Constitutional ai: Harnlessness from ai feedback. arXiv preprini arXiv:2212.0800 2022b.
Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406. 15513, 2024b.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(34)):324-345 1952.
Lambert, N., Castricato, L., von Werra, L., and Havrilla, A. Illustrating reinforcement learning from hu- man feedback (rlhf). Hugging Face Blog. 2022.
Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.08 053666 2024.
http://hgginngfacecoooooggggo/oe
10

---

HPP:: Hard Preference Sampling for Human Preference Alignment
Lambert, N., Pyarkin, V., Morrison, J., Miranda, L., Lin, B. Y. Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Re- wardbench: Evaluating reward models for language modeling. https: //ugg wiw.hgggggggaaaoooooooo hag.ac..comm co/ spaces/ allenai/rewate bench, 2024.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C.,, Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Ozay, M. Fine-grained optimization of deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint SKKv:2200600001,, 2021.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguisties, pp. 311-318. 2002.
Li, Z., Ji, X., Chen, M., and Wang, M. Policy evaluation for reinforcement learning from human feedback: A sample complexity analysis. In International Conference on Arti- ficial Intelligence and Statistics, pp. 2737-2745. PMLR, 2024.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. URL https://arxiv. org/abs/2403.1113 19159, 2024.
Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in Ilms. arXiv preprint arXiv:2410.18455 2024a.
Plackett, R. L. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statisties, 24 (2): 19:-2022,, 1975.
Liu, T. Qin, Z., Wu. J., Shen, J., Khalman, M., Joshi, R., Zhao, Y. Salch, M., Baumgartner, S., Liu, J., al. Lipo: Listwise preference optimization through learning- to-rank. arXiv preprint arXiv:2402.01877 2024b.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 1,05 2017.
Robinson, J., Chuang, C.-Y., Sra, S., and Jegelka, S. Con- trastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
Luce, R. D. Individual choice behavior, volume 4. Wiley New York, 1959.
Maystre, L. and Grossglauser, M. Fast and accurate in- ference of plackett-luce models. Advances in neural information processing systems, 28, 2015.
Sakhi, O., Rohde, D., and Chopin, N. Fast slate policy optimization: Going beyond plackett-luce. arXiv preprint arXiv:2308.01566, 2023.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Schroff, F., Kalenichenko, D., and Philbin, Facenet: unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L.,, Kim, C., Hesse, C., Jain, S., Kosaraju, V. Saunders, W., al. Webgpt: Browser-assisted question-answennee answering with hu- man feedback. arXiv preprint arXiv:21 2021.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1rXiv:1707.063433 2017.
Oh Song, H., Xiang. Y., Jegelka, S., and Savarese. S. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004 4004-4012, 2016.
Shao, Z.. Wang, P., Zhu, Q., Xu, R., Song. J., Bi. X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Oosterhuis, H. Computationally efficient optimization of
plackett-luce ranking models for relevance and faireess. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1023-1032, 2021.
Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXi::2206.1187/ 2022.

---

HPS: Hard Preference Sampling for Human Preference Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference Artificial In- telligence, volume 38, pp.
2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Sun, H., Shen, Y., and Ton, J.-F Rethinking bradley- terry models in preference-based reward modeling: Foundations, theory, and alternatives. arXiv preprint arXiv:2411.04991 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971 2023.
Wang, H., Xiong, W. Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experte In EMNLP, 2024.
Wu, Y., Sun, Z. Yuan, H., Ji, K., Yang, Y., and Gu, Q. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405. 00440000075, 2024.
Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu,
B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., and Zhang, Y. A survey on large language model (Ilm) security and pri- vacy: The good, the bad, and the ugly. Cong--Cneeeecce Computing, pp. 100211, 2024.
Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M., and Liu, P. J. Calibrating sequence likelihood im- proves conditional language generation. arXiv preprint arX1v:2210.00045, 2022.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Salch, M., and Liu, J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint 190.0099090 2019.
12

---

HPS: Hard Preference Sampling for Human Preference Alignment
A. Reinforcement Learning from Human Feedback
A.1. Bradley-Terry model
Then sampling samples from Pirts one can construct a dataset = {(x,, (1) (2) where each instance consists of one prompt J; and 2 responses (1). (2) followed the user-specified ranking. Using this dataset, we can train reward model re parameterized by 0 by approaching the task classification problem. Specifically, frame the training objective minimizing the negative log-likelihood loss:
LBT (
E
[oogo
ro( Vr,(2))))
(15)
log (ro( (1o
A.2. Listwise Preference Optimization
A.2.1. SLIC-HF
SLiC-HF integrates the sequence-level contrastive method SLiC with human preference rankings:
(()) max (0, log( (y (yt [z)) log(mo [z)) X (oge (x)r)
(16)
"*. y and fref denote the positive, negative, and reference sequences, respectively, 6 is margin hyperparameter and X regularization weight.
A.2.2. LIPO-À
LiPO-À employs listwise ranking objective with a Lambda weight Aij. Given a list of responses = (y),.... ...)).
LIPO B(Z
Σ P124y
Dis log(1 + =
(17)
where And
Here, d the true reward score response yı and s; ( is the implicit DPO reward. The rank position of y, in the ordering induced by s = (s1...., sK) is denoted
12 2d/
(og()+7(i))
(gg(1-0))
3 log Te(usaa) .x)
as 7(1). The Lambda weight assigns greater importance response pairs with larger preference gaps. i.e. di 4)
A.3. Reward Modelling
A.3.1. KTO
KTO (Ethayarajh et al., 2024) defines type of reward to construct human-aware losses (HALOs), which is in the form
To ()(2z)
(xT y) ((y) log
(18)
Teat Trr((y))))
where 0 denotes the trainable parameters of the model mo being aligned, The is the reference model, and I. y - IR+ is normalizing factor.
A.3.2. SIMPO
SimPO (Meng et al., 2024) identifies the discrepancy between DPO's reward and the likelihood metric used for generation, and proposes an alternative reference-free reward training loss:
Tsimpo (2, 3) =
log ())2))
log no (es),
(19)
ly]
1yl
where |y| the length of the response y. and yeu the set of tokens in the sentence y before the token y:-
13

---

HPS: Hard Preference Sampling for Human Preference Alignment
B. Theoretical Analysis
B.1. Sample Efficiency Analysis
Theorem. Let D be a given dataset. Under certain regularity conditions, the maximum likelihood estimators HHDR and PLL corresponding to the the hard sampling loss Le and PL loss LPL. respectively, satisfies the following with probability at least 1 6:
+ log (5) mç² (N)
16a)C(N) -402 m m. ((N)
Ones 0011100 SCi V
=0
vm
and
Intesao (d + log (t)
(
n²a \mm
10 - 101120 S C2 V
(vm
m
where
2
11
n
ED mn(n- - 1)
Σ
Vre (Is. vr.(1))) Vro (r())) (Vro (Is.
Vro (Fe) (r.()))
j=l Am
and
(NN + exp (200 + In(NN) + exp
Therefore, the error bound of OHDR tighter than that of Opt, i.e, ||OHDR - 0-1|E S 11pp.. 0¹ 0'11sp. In other words, ÖHDR is more efficient than Pr.-
Proof. We begin by proving Theorem 1.
Analysis on Le We first analyze the asymptotic efficiency and estimation error of estimator induced by Co. We consider the general RLHF setting in a dataset with m sample:
1 m
c (1(..B))))))
Lo
log
(To(En)D)(1)) + N Egs [ero(m.u)))
The maximum likelihood estimator (MLE) Ours aims minimizing the negative log likelihood, defined as:
OHes arg min La-
BE0n
When the minimizer not unique, we take any of the Oups achieve the minimum.
To simplify the notation, for fixed sampling method g, let ge re (I Vrs(1)) In (N) E [ro(x, (.)) We that the gradient of Le takes the form:
vi-ag
1 m
(oi, (xi) =1] exp (-9b)
(o)
Σ log 1 [T
[T - (0i,
1 exp (-9è) #
Vil
+ exp (-96)
And the Hessian of Lo is
m
exp (9h)
1 7 (xs, Vo,(1)) = exp (-9h) V2
V V T
m
(1 + exp (gh))
1 + +exp (-9b)
1 (I (1)) # exp (9b)
2 gb
+ exp (16)
We bound E (x1, (x))))) using the assumption: -0p
(ro w:)l ag-
14

---

HPP: Hard Preference Sampling for Human Preference Alignment
Thus,
exp (g/)
((N)
exp (gb))
where (NV) (N( 200)'
We say Σ = E" E ---- E' is positive semidefinite, Based on Assumption 1, we have
12
(O) =
((()) 2az1
(20)
Based on the Lipschitz gradient assumption, we also know that V - vgh. 12 S 4ag. Let a Vgg
Vig., we have:
V Lo) = m
Σ (NN) (Vi + u) (vgà- T + u) -2az1
m
(NN Vo Vil- ((N) (vio- w- + +VVgo ") 2021
Using the Cauchy's Inequality, for arbitrary e Rd, un s ||l|//|ulla s 40111e112, v² Vob. s ail|ull2, where ||>||2
V E x01), this gives that:
Lo 2 ((N) |
Sa7C(N) 20x |1012.
m
m
where X E Rmxd has the vector Vgg- E R as its th row.
Thus, we introduce the error vector Ours 0", then we may conclude that:
Co + A) - Lo - (V") A)
((N) XAI2 Saf - 20²
|
B 8a 8aj((N)= 2002
(NN) |
|
m
Now we aim bounding the estimation error |0Hps
0" I| Since OuPS is optimal for Ca. we have (oIPs) S Lo
Defining the error vector A
OHPS 0" adding and subtracting the quantity (VL(O"). A) yields the bound:
0 + A) (V S - (VL(0))
We know the left-hand side lower bounded by:
Saî5(N)) 22x
(NN) | ED +
|
m
As for the right-hand side, note that (VV((0)) A < [V || E Altogether have:
< (NN) 1|13 ST W VV((0')l DA 1 I
where 1/5 -222 Now we further bound the term
VL(("))l ||=. The gradient takes the form:
VCo(d") = Σ
[ (Ta.Vrr()) =1] 1 exp
exp (-g
1 [ (x (72(1)) * #1]
1
Vgg..
(gg))
exp (b.))
15

---

HPS: Hard Preference Sampling for Human Preference Alignment
Define random vectors V I with independent components
exp
(-9e-)
1
w.p.
1 ++ exp -6-)
+ exp
(-
V/-
-1
exp
w.p.
1 exp (
+ exp
90-)
With this notation, we have VLo(0") V with E[V] 0 and V S 1. Defining the m-dimensional square matrix
M मं X have (V [I
VT MV. Let the eigenvalue decomposition of X X be x UAUT.
We can bound the trace and operator norm of M as:
A
d S m
Tr(M)
Tr
U
UTUAUT
m
m
Tr(M²2
Tr
U
A
UTUAUTU
A
UUUNO")
d m
m
m
|M 1lop = Amax (M) S Tr(M2) 1 m
Moreover, since the components of V are independent and of zero mean, and [V]| S 1, the variables V2 are 1-sub-Gaussian, and hence the Bernstein's 's inequality for sub-Gaussian random variables in quadratic form implies that with probability at least 1 - 6,
d log (3)
- IV = VT MV SC.
Here C' is certain constant. This gives
((N) IA I <IVC((")|s ||ll| &
d + log (t) C.
V
A 22/1A|1Eo0
m
where 1b- 8afc(N)-20)
Solving the inequality above gives us for some constant C₁:
d log (1) ms² (N)
16a7((N) 4az ((N)
| sa.
m
where & (N) In 200)
Thus, can derive that with probability least 8:
log (t) mg² (N) S(N)
16a75(N) -402
||Hrrs 0'11= S SC1
n
mmm
Analysis on Cpt. We first analyze the asymptotic efficiency and estimation error of estimator induced by Cpt. We consider the general RLHF setting in a dataset with sample:
1 m
LM.
())
m
372
a
( (o(())))
log
Σ
m
16

---

HPS: Hard Preference Sampling for Human Preference Alignment
The maximum likelihood estimator (MLE) PpL aims at minimizing the negative log likelihood, defined
PPL E arg min LPL- 0E0g
When the minimizer not unique, we take any of the PpL achieve the minimum. We can see that the gradient of LpL takes the form:
V (0) 1
(Vro(ei, (ro(Fi,
T k=j -k e (ol
And the Hessian of CPL is:
ro 1 (Vro(1))),,)))) - Di v..(4) Vro(E).)r,a)))
(O) =
Since Iro w)1 S ag. the coefficient satisfies:
6 400g
2
k e (o(o).
2(n j 1)2²
Set 3 = c 400
We can verify that for any vector U E Rd, one has:
V CLLe 2
=
8 T
(ra(Et - Vre(es,,,,,e)) - Vro(ai
(n J 11
2 1,80"2 "oo
Thus, the loss function LPL is B-strongly convex with respect to the semi-norm || ||E>> where e -40g
Now we aim bounding the estimation error 0pr. 11xp-- Since PpL. optimal for LPL, we have L(Ppt) S L(0").
Defining the error vector A Opt. 0", adding and subtracting the quantity (VL(0"), A} yields the bound:
(f + A) (P) - (VLr
I - (VLLL(f") A)
By using the convexity of the loss function LPL, the left-hand side is lower bounded by A As for the right-hand side, note that:
[VLPr.(0") A) < V Lpr. (0") I|=; |
Altogether we have:
B| < V Cp (0") || ||
Now further bound the term PLpL (0")
|لاا" 1. Observe that the gradient takes the form:
1
Sw=s e (o(enBBraar) (Vre(Et.Br.ts)) Vre (r,(k)))
m
We set Tic Vro(En,Br.1)) - Vro(En X E 1m/(n-1)/2 x d has the differencing vector Gi. as its (in(n 1)/2+ k Ei-neyt1 th row. We also define Vi be the random variable of the coefficient of 9 under the PL model, i.e. conditioned arbitrary permutation T
(0(
if T(() < Ti(k,,
Vi jk
exp (-9 g/.
otherwise
Σ n
(k) " (oolemnrgee))
1 exp (-90-)
17

---

HPS: Hard Preference Sampling for Human Preference Alignment
Here (i) < (k) means that the j-th item ranks higher than the k-th item.
Let Vi (1) be the concatenated random vector of {V jk isickgn V [ be the concatenated random vector of [V])--- We know that Vi and Vj are independent for each * due to the independent sampling procedure. Using the results in Appendix B.5 in the paper (Zhu et al., 2023), we can verify that the mean of V; is O. Furthermore, since under any permutation, the sum of absolute value each element in V2 is at most we know that V; is sub-Gaussian with parameter n. Thus we know that V is also sub-Gaussian with mean and parameter n. Now we know that the term (V" can be written as:
LVL - -
1 m
Let M I. One verify that M 2 - X X almost surely since Amax ( upper bound the original term
n². m2
Thus we can
n
V Lr. (0 11
I
By Bernstein's 's inequality for sub-Gaussian random variables in quadratic form, know that with probability least 1 5:
VVI2 S Cn2 (a+ log (1)
for certain constant C. Thus, we can conclude that
Cna. (d log (5)) A A
B
where 3 --000 By solving the inequality, we can derive that with probability at least 1 - 6:
=
n4e800
(d + log (1) 0
n²
0PL 0'11>0 S C2₂
m
m
where C2 a constant.
B.2. Reward Margin Analysis
B.2.1. PROOF FOR THEOREM 2
We prove Theorem here.
Theorem. Let Ci pen LT. Then holds the convergence: Co + Lij as 7 + oo where Co our HPS loss.
Proof. We have
LT E - dwD
log
(o( +N Eyyu (era(u,)))
and
Co
E log
(Te(E.B0(1) N Ey-g(z.)) (er((z.))
We denote p and p+ as the data distribution for preferred responses and dispreferred responses. Consider the following essential supremum:
M(y+) =
CSS sup
re (x, VT)
VF 1yy >> (v+)
= sup(mm > 0: 2 ro (z, V= a.s. for or u
18

---

HPP:: Hard Preference Sampling for Human Preference Alignment
We define
(to(0:)-(1)
( log ('o(uN--c)) N. [cM(u-ecs]]
and
CRLHP(0,g) =llo
e'o(Z.HHL)) +N- E V r
The difference between these two terms can be bounded as follows,
(R) (0, -log (TolR.8r(D) + N [(
log
eTo(z.Brcu) + N. erala.vi)
wa
Then find that:
(ro(xibre))) NN. E [erotewsr)] - log (crole))oo)) + N. [oM(eaao))
log
2F ma
c00
e ro(o))a))) + NN. E (er((e01)) (zo - -N.eunD--eonn [eMivass]
N+1
mg
Ne
E (co((wN)) {] ((cres) mag
N +1
< 00
E
M(rran) crolanzF)
0F m
where for the second inequality we have used Assumption that the reward (ro y)] is bounded by og and thus restrict the domain of the logarithm values greater than (N + 1)ee-aa Because of this, the logarithm is Lipschitz with parameter ea0
N+1 Using again Assumption I that re(x, vF M (M+(1)) s ag and applying the mean value theorem, derive the
following inequality:
E
M(sras) cTo(1177))
00
E Beigs mn
M(Wr(D)) -roc,)))
Let us consider the inner expectation (Ur(N)) E M( ro(x, vF Note that since ro( y7 is bounded, ye
E, (I+(1) is uniformly bounded in Vr(1)- Therefore, in order to show the convergence (O. 9) + CRum (0), as co, it suffices by the dominated convergence theorem to show that (vr()) + 0 pointwise as - oc for arbitrary fixed Ur(1) V.
For fixed Fr(1) E V/ we consider M = M (F+(1)). Based on the definition of q, it is evident that q < <pp That is, since q = c-p for some non-constant c, absolutely continuous with respect p-. So M 2 ra y7 a.s. for yr q. Define the following event Ge {q re 87 2 M c]. where g refers to a "good" event. Define its complement
B, G where B is for "bad" event. For fixed Vr(1) y and 0, we consider:
(v((1) = E
[M((r)) - ro(
#r
Py ur ma (G.)
E [[M((+()) ro (x, y G.]
#F m E
+ P ~ (Be)
[M - ro (z, vi B.]
Vr. ~
IP or ~a (G.)
2P yē ma (B.)
+ 2P
m (B.).
19

---

HPP:: Hard Preference Sampling for Human Preference Alignment
We find relationship between 7 and IP 8 ~q (B.). (B.). Expanding it in the following formula:
P Poi~a((B)) = / (ro(0, < M -c erreaaler)) (yr
dyr
Zy
where Z, - sy (e)aeltrst))) P (VT) dyf the partition function of q. We can bound the equation by:
J (ro(e, < M -c) err(a-e) pp (y7 dy
Zy
e2(
S, (ro(z, (7) vo) < M- - - dyz
Zy
(c-
~p- (Bc)
Zy
(7-(M-r)
< Zzy
Note that
2y / (TTeall::))) pp (VT) dyz
2 (M-)) IP up- ( (cro(tivi) M )
The probability
(cro(en5)) 2 M E)
Pe P Vi ~p
0,
2
and therefore bound:
(M)
IP
1/ m (Be)
e> t)
Pe
e
Pe
0 as
- 00.
Thus, may take y to be sufficiently big as to make yi ~ (B.) S and therefore E, S 3e, i.e. E, + 0, as 7 co. In conclusion, + LRLHE(O, .) ( which can be extended to the expectation over the dataset D, and thus Lo + Li
B.2.2. PROOF FOR THEOREM 3
To study the properties of global optima of the RLHF objective using the adversarial worst-case hard sampling distribution, recall that we have the following objective:
exp (ro (a, v+(r)) E [exp (ro (z. v)
CE
E
log
(T(L) -pt
We can separate the logarithm of quotient into two terms:
L
E 2u(
[ro (x. (+()))))]
+
E log (exp (ra (2, w))]
yap
E Bra
[ro (x))+(1)] +
E
E Tro (x,
Branppp w<p
Taking the supremum to obtain 43 -00"= ssup L
20

---

HPS: Hard Preference Sampling for Human Preference Alignment
Theorem. Assume the ranking T is a finite
Let Lg sup L and 0* arg min Lg f Then O is also the pell
solution to the following problem:
θ- arg max ro (2,3-(1)
max ro (2.8-19))
1<55|r|
Proof. Obtaining the second claim is matter of manipulating Cg The objective Lg can be rewritten by first expressing it in terms of expectations over the distributions:
arg max au
ro (z,
sup
[re (z, v)l
(=((2)>
Breaking down this expectation with respect to the ranking classes gives:
arg max 0 eupypp (+I(11)
E
ro (I,
sup +-1()>>1
(ro (z. v))
This can be further simplified by summing over all classes and using the distribution density p(T(1))
arg max mgaxx(((1))) ((T())) re (2,))(1)) - sup r 1(0)1
[ne (z. (z.v))
0
Thus, we can represent the objective in terms as:
max ro (z. )-.s)) 1<j5|||
arg max ro (z. V>(1))
Thus, it implies that the optimal parameter under our proposed HPS loss can maximize the margin of the rewards of the most preferred response and other hard dispreferred responses.
C. More Results
C.1. Ablation Results of Transfer Learning
Table 7. Ablation results with response size under transfer learning setting, See Reward Margins in Table 1.
Number
Method
BLEU
Reward
RMpso*
RMg
DPO-BT
0.309
0.406
0.856
0.249
DPO-HPS
0.307
0.407
1.191
0.582
DPO-ET 0.307
0.407
0.870
1.012
20
DPO-HPS
0.310
0.407
1.620
1.262
DPO-BT 0.309
0.407
1.319
1.311
DPO-HPS 0.230
0.307
2.164
1,555
DPO-BT 0.308
0.407
1.346
1.738
100
DPO-HPS
0.310
0.407
5.725
5.116
We examine the impact of the total number of responses on preference optimization performance during transfer learning, using 5, 20, 50, and 100 responses per prompt.
21

---

HPS: Hard Preference Sampling for Human Preference Alignment
C.2. Ablation Results of Different / in DPO
Table 8. Ablation results with different under fine-tuning setting. See Reward Margins in Table 1.
8
Method
KL
BLEUT
Reward
RMorot
RMR.roo
DPO-BT
8.463
0.230
0.431
0.349
-0.455
0.1
DPO-HPS
11.767
0.232
0.430
2.723
2.040
DPO-ET
5,888
0.231
0.431
-0.206
-1.188
0.25
DPO-HPS
6.972
0.230
0.431
-0.146
-0.828
DPO-BT
2.661
0.229
0.430
-0.239
-1.022
0.5
DPO-HPS
3.091
0.227
0.428
-0.228
-0.911
DPO-ET
2,996
0.225
0.428
-0.264
-1.046
0.75
DPO-HPS
2.192
0.226
0.427
-0.242
-0.925
DPO-BT
0.227
0.430
-0,308
-1.990
1
DPO-HPS
2.015
0.225
0.429
-0,316
-1.178
Regarding the sensitivity of DPO, we have conducted experiments with = (0.1, 0.25, 0.75, 1) and report the KL divergence [TL[TT (Yw z) || (ye x]] across these values, where x is the prompt and yo is the winming response the test set. The results in Table 8 demonstrate the superiority of our HPS: achieves the highest R.R-DPO for all KL values, confirming that HPS leads stronger rejection of harmful responses.
D. Implementation Details
D.1. Experimental Setup
We utilize 8 L40-S GPUs for data augmentation and annotation. During the training stage, we employ 4 x L40-S GPUs with per-device train batch size and gradient accumulation steps of 16, effectively resulting in a total batch size of 64.
D.2. User Study Evaluation Methodology
1.2.1. EVALUATION PIPELINE
Response Generation: All models generate responses using the same prompt set from the test dataset.
Blind Scoring: Human raters rate responses without knowing which model generated them.
Score Aggregation: Average scores across all responses to identify performance trends.
Comparative Analysis: Compare all average across diverse prompts derive the win rate for each method.
D.2.2. EVALUATION CRITERIA
1. Correctness (1.00-5.00)
Does the response provide factually accurate information relevant to the query?
Higher scores reflect precise and well-supported answers.
2. Helpfulness (1.00-5.00)
Does the response thoroughly address the user's query?
Higher scores reflect detailed, relevant information that goes beyond minimal effort.
3.
Safety (1.00-5.00)
Does the response avoid harmful, biased, inappropriate content?
Higher scores reflect neutral, con-harmful language.
4. Clarity (1.00-5.00)
the response clear and easy understand?
Higher scores reflect concise, well-structured communication without ambiguity.
22

---

HPS: Hard Preference Sampling for Human Preference Alignment
D.2.3. SCORING GUIDELINES
Score Likert scale of 1.00 5.00, where 5.00 is the best and 1.00 is the worst.
Scores may use up to two decimal places for finer distinctions.
D.3. Win Rate Evaluation Methodology
D.3.1. EVALUATION PIPELINE
Response Generation: All models generate responses using the same prompt set from test dataset.
Blind Scoring: Evaluators rate responses without knowing which model generated them.
Score Aggregation: Average scores across all responses to identify performance trends.
Comparative Analysis: Compare average scores across diverse prompts to derive the win rate each method.
D.3.2. EVALUATION CRITERIA
1. Correctness (0.0-5.0)
Does the response provide factually accurate information relevant to the query?
Higher scores reflect precise and well-supported answers.
2. Helpfulness (0.0-5.0)
Does the response thoroughly address the user's query?
Higher scores reflect detailed, relevant information that goes beyond minimal effort.
3.
Safety (0.0-5.0)
Does the response avoid harmful, biased, or inappropriate content?
Higher scores reflect neutral, non-harmful language.
4. Clarity (0.0-5.0)
Is the response clear and easy to understand?
Higher scores reflect concise, well-structured communication without ambiguity.
D.3.3. SCORING GUIDELINES
Score on Likert scale of in 0.5 increments, where 5 the best and is the worst.
Scores may use up to one decimal places for finer distinctions.
E. Extension to HPS
As discussed in Section 4, our approach is designed with LLM safety in mind, prioritizing the reduction of false negatives. In our setup, we assume (r(1) is the preferred harmless response, while we cannot guarantee that (U+(2):-.... Vr(n)) are entirely free from undesired content. Therefore, we treat Vr(1) as the ideal helpful response and maximize the reward margin between (r(1) and "hard" dispreferred responses, prioritizing the minimization of false negatives. This particularly critical for applications that demand high-quality and safe content generation.
In cases where multiple responses are valid, our HPS method can be extended to accommodate response diversity. Specifi- cally, we can formulate weighted HPS loss, treating each valid response as preferred one in its respective loss term. This approach maintains response diversity while ensuring that high-ranked responses adhere to safety and quality standards.
For instance, given a training sample d = (z, Vr(1), (7(2)..... Vr(m)) ~ D, both Vr(1) and (r(2) are helpful responses, we can redefine the objective to train the model to reject all dispreferred and potentially harmful responses (Ur()) ensuring
23

---

HPS: Hard Preference Sampling for Human Preference Alignment
that it generates only the preferred responses (1) and ()((2) for given prompt æ. The modified loss function is defined
weighted sum of two HPS losses:
Lo = +
where À is weighting hyperparameter, and
(1.
L1 = ENND log
('o(0))F()) + N1 By-p(en (e (2,, v)
22 = EdND log
(To(x)+rc)) + N2 Ey- (erooaal v)
with
②1 (2, v)
i= (x.6.1))
q2(3r, v)
Σ (x.Ten(
Ni n N2 = n-2, and p(y) is the probability distribution of the dispreferred response y. By optimizing the weighted HPS loss Le. the model is encouraged to rank Vr(1) and (r(2) above all dispreferred and potentially harmful responses (r())))s thereby maintaining both helpfulness and response diversity.
24


Reviews:
[{"review_id": "Pedagl2RsA", "reviewer": "Reviewer_SuNZ", "summary": "This paper adapts the concept of hard negative sampling, which was previously employed in metric learning and contrastive learning settings, to preference alignment. Hard Preference Sampling (HPS) framework reconsiders the loss function derived by incorporating reward into the  Plackett-Luce (PL) model and uses a modified version with following contributions:\n\n- Training loss in HPS framework boosts hard negatives$-$dispreferred responses with high rewards. \n\n- They sample only one negative response using Monte Carlo, which reduces training costs, and it appears to work well while maintaining alignment quality.\n- The paper claims that optimizing the HPS loss maximizes the reward margin and HPS provably improves sample complexity over the vanilla PL loss.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.\n\n**1) For hard negative sampling**, this work is the first to extend hard negative sampling to preference alignment, addressing new task-specific challenges with novel and effective solutions:\n\n**a) Handling Varying Informativeness of Dispreferred Responses.** In metric and contrastive learning (arXiv:2010.04592, arXiv:2108.09335), hard negatives are typically selected based on representation similarity to a positive anchor. However, in RLHF, where responses are generated autoregressively, obtaining effective sentence embeddings is impractical. Instead, we define \"hardness\" in the reward space, where dispreferred responses similar to preferred ones (i.e., with close reward scores) are considered harder (L252). Furthermore, selecting hard negatives using $q(x,y)$ presents an intractable distribution challenge (L234), which we address via Monte Carlo importance sampling (Eq. 9).\n\n**b) Improving Sampling Efficiency.** In previous work on metric and contrastive learning, backbone models such as ResNet or GoogLeNet were employed in vision tasks, where the model size is approximately $1\\%$ that of Llama-3-8B in the RLHF setting. To ensure computational efficiency, we reformulate HPS into an efficient sampling approach, using a single Monte Carlo sampling to select a single dispreferred response per training sample.\n \n**c) Theoretical Analysis of Hard Negative Sampling in Preference Alignment.** Our work is the first to provide a theoretical analysis of hard negative sampling in this context, offering new insights into alignment. We compare sample efficiency between the preference loss (PL) and HPS loss, demonstrating that HPS improves sample efficiency, particularly in data-scarce settings or when rapid convergence is needed. Additionally, we analyze how training with HPS maximizes the reward margin between preferred and hard dispreferred responses, ensuring a robust distinction between them. This strengthens alignment performance while minimizing undesired outputs. We will further elaborate on these distinctions in Sec. 1 and 4.\n\n**2) Regarding the differences in sample complexity bounds of our HPS and the PL**, intuitively, they stem from the structural distinction of two losses. The PL loss is composed of a summation of $n$ piece of one-to-$N$ contrast losses $\\mathcal{L}\\_{j}(d)$:$$\\mathcal{L}\\_{PL}=\\mathbb{E}\\_{d\\sim\\mathcal{D}}\\sum\\_{j=1}^{n}\\mathcal{L}\\_{j}(d)=\\mathbb{E}\\_{d\\sim\\mathcal{D}}\\sum\\_{j=1}^{n}-\\log({e^{r\\_{\\theta} (x,y\\_{\\tau(j)})}/\\sum\\_{k=j}^{n}e^{r\\_{\\theta}(x,y\\_{\\tau(k)})}}),\\tag{1}$$In contrast, our proposed HPS loss focuses on encouraging the model to rank the most preferred response $y\\_{\\tau(1)}$ against all other dispreferred responses $(y\\_{\\tau(i)})\\_{i=2}^n$:$$\\mathcal{L}\\_{\\theta}=\\mathbb{E}\\_{d\\sim\\mathcal{D}}-\\log(\\frac{e^{{r\\_{\\theta}(x,y\\_{\\tau(1)})}}}{e^{{r\\_{\\theta}(x,y\\_{\\tau(1)})}}+N\\cdot\\mathbb{E}\\_{y\\sim p(y)}[e^{{r\\_{\\theta}(x,y)}}q(x,y)]})\\tag{2}$$with$$q(x,y)=\\frac{e^{\\gamma\\cdot r\\_{est}\\left(x,y\\right)}}{\\sum\\_{i=2}^{n}e^{\\gamma\\cdot r\\_{est}\\left(x,y\\_{\\tau(i)}\\right)}}.\\tag{3}$$The HPS loss $\\mathcal{L}\\_{\\theta}$ only uses one component, $\\mathcal{L}\\_1$, from the full summation in $\\mathcal{L}\\_{PL}$. Thus, the structural distinction between the two loss functions leads to the $n$-factor discrepancy in the asymptotic error bound.\n\nMore specifically, as shown in Appendix B.1, the difference between the HPS loss $\\mathcal{L}\\_{\\theta}$ and the PL loss $\\mathcal{L}\\_{PL}$ has a direct impact on their gradients. We follow the mathematical notations in our paper. Specifically:\n\n- For HPS-based loss:$$\\|\\nabla\\mathcal{L}\\_{\\theta}(\\theta^{*})\\|^{2}\\_{\\Sigma^{-1}\\_{\\mathcal{D}}}\\leq C\\cdot\\frac{d+\\ln(\\frac{1}{\\delta})}{m}$$\n- For PL-based loss:$$\\|\\nabla\\mathcal{L}\\_{PL}(\\theta^{*})\\|^{2}\\_{\\Sigma^{-1}\\_{\\mathcal{D}}}\\leq C n^{4} \\cdot \\frac{d+\\ln(\\frac{1}{\\delta})}{m}$$\n\nThe error bound for the HPS-based method $\\|\\Delta\\_{HPS}\\|\\_{\\Sigma\\_{\\mathcal{D}}}$, where $\\Delta\\_{HPS}:=\\theta\\_{HPS}-\\theta^{*}$, is bounded by\n\n$\\frac{\\|\\nabla\\mathcal{L}\\_{\\theta}(\\theta^{*})\\|\\_{\\Sigma^{-1}\\_{\\mathcal{D}}}}{\\zeta}$\n\nwith$$\\zeta = \\frac{1}{2+\\exp(2\\alpha\\_{0}+\\ln(n-1))+\\exp(-2\\alpha\\_{0})}$$as stated in Thm. 1. Similarly, the error bound for the PL-based method $\\|\\Delta\\_{PL}\\|\\_{\\Sigma\\_{\\mathcal{D}}}$, where $\\Delta\\_{PL}:=\\theta\\_{PL}-\\theta^{*}$, is bounded by\n\n$\\|\\nabla\\mathcal{L}\\_{PL}(\\theta^{*})\\|\\_{\\Sigma^{-1}\\_{\\mathcal{D}}}.$\n\nFrom this, we observe that $\\Delta\\_{HPS}$ exhibits an error bound of $\\mathcal{O}(n)$, whereas $\\Delta\\_{PL}$ has an error bound of $\\mathcal{O}(n^2)$, differing by a factor of $n$. We will integrate this intuitive explanation into Thm. 1 for clarity.", "claims_and_evidence": "I have couple of clarification questions (See below).", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I wasn't able to check all the details to fully confirm its mathematical correctness, but I don't see any problems.", "experimental_designs_or_analyses": "They are convincing.", "relation_to_broader_scientific_literature": "The paper proposes Hard Preference Sampling framework for better alignment quality. This work can be considered as an adaptation of hard negative sampling technique from metric learning and contrastive learning literature to preference alignment. The vanilla model uses the loss function derived by incorporating reward into the  Plackett-Luce (PL) model. This paper reconsiders this PL model based loss function. In a broad sense, In a broader sense, their results provide intuition for safer and more responsible language models.", "essential_references_not_discussed": "No, up to my understanding.", "other_strengths_and_weaknesses": "**Strengths:** \n- It is a useful adaptation of ideas to preference alignment setting to improve safety and reliability of LLMs. \n- I find the paper well written and easy to follow.\n\n**Weaknesses:** \n- Hard negative sampling is a known technique employed in various settings, such as metric learning and contrastive learning. The maximum margin property of hard negative sampling has been demonstrated in these contexts. Therefore, the originality of the paper lies in applying this technique to preference alignment, which limits the technical contributions and makes the conceptual contributions more significant in highlighting the distinguishing characteristics.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Can you elaborate on what leads to the difference in sample complexity bounds, which differ by an $n$ factor?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUiQMPM2fB", "reviewer": "Reviewer_xe8N", "summary": "This work introduces **Hard Preference Sampling**, a framework for aligning large language models to human preferences. HPS introduces a training loss that adaptively penalizes dispreferred responses, and focuses on “hard” dispreferred responses, i.e. responses that are similar to preferred responses to increase reward margins. HPS is also as efficient as Bradley-Terry models during training, as for each preferred response, it samples a single dispreferred response. The authors also show theoretical bounds on sample complexity that scale better than Plackett-Luce models, as well as on reward margin quality. Finally, this work also empirically demonstrates positive win rates with LLMs as a judge compared to baselines on popular preference datasets for helpfulness, harmlessness, and safety.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful and positive comments! We provide our response and hope our response addresses your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.\n\n**1) For quality-weighted BT**, we could not find prior work directly related to it but identified two relevant methods: WPO (arXiv:2406.11827) and LiPO-$\\lambda$ (arXiv:2402.01878). We will discuss them first, and will provide further discussion if you can provide specific references.\n\nIn **WPO**, weights are assigned to response pairs based on their occurrence probability:\n$$\\mathcal{L}_{WPO}=-\\mathbb{E}\\_{(x,y_w,y_l)\\sim\\mathcal{D}}[w(x,y_w)w(x,y_l)\\log p(y_w\\succ y_l|x)],$$where $w(x,y)=\\exp{\\frac{1}{|y|}\\sum^{|y|}\\_{t=1}\\log\\pi\\_{\\theta}(y_t|x,y\\_{<t})}$ and $|y|$ is the number of tokens in the output. So WPO modifies response weights to better align with on-policy data while following DPO to consider only the most preferred and most dispreferred responses. In contrast, our HPS accounts for multiple responses and focuses on leveraging the varying informativeness of all dispreferred responses.\n\nDiscussion of **LiPO-$\\lambda$** can be found in our response **1)** to Reviewer 85yJ.\n\n**2) Regarding importance-weighted dispreferred response**, we directly sample one dispreferred response according to the importance-weighted distribution (L244):$$q(x,y)=\\frac{e^{\\gamma\\cdot r_{est}(x,y)}}{\\sum_{i=2}^{n}e^{\\gamma\\cdot r_{est}(x,y_{\\tau(i)})}},$$where $(y_{\\tau(i)})_{i=2}^{n}$ denote the dispreferred responses of a prompt $x$. Thus, harder dispreferred responses will be sampled with higher probability and contribute more to the loss due to their higher probability $q(x,y)$. Then, we can incorporate the sampled dispreferred response $y$ into the loss function Eq. 9 for training.\n\n**3) For response similarity**, it refers to two responses having comparable rewards, indicating shared semantics (key information). For example, Fig. 1 in submission shows that for a given prompt $x$, response $y_{\\tau(1)}$ is more similar to $y_{\\tau(2)}$ than to $y_{\\tau(3)}$ since their content is closer. Consequently, $y_{\\tau(1)}$ and $y_{\\tau(2)}$ receive similar rewards, reinforcing their similarity.\n\nHowever, we may not fully understand the question—please clarify if needed.\n\n**4) For human evaluation**, we conducted a user study with human participants. See Tab.2 in our response **2)** to Reviewer v4s5 for details.\n\n**5) Regarding reward $r_{est}$ of each ranked response $y_{\\tau(i)}$**, we either use the given reward $r_{est}$ or estimate it with a pretrained preference-aligned reward model (L230). In our experiments, we use Skywork-Reward-Llama-3 (arXiv:2410.18451). The scaling factor $\\gamma$ is linearly increased from -5 to 5 at every 20\\% interval of the training process.\n\n**6) For experimental setting**, your understanding is correct. In Table 2, DPO-PL and IPO-HPS independently fine-tune Llama3-8B on HH-RLHF using DPO with a PL preference model and IPO assuming an HPS preference model.\n\n**7) For experiments**, we follow (arXiv:2306.17492) and expand response data by generating 100 responses using *RLHFlow/Llama3-v2-DPO* per prompt. The corresponding rewards are computed via *Skywork-Reward-Llama-3*. Then we use methods like DPO-BT, DPO-PL, and DPO-HPS to fine-tune the language model on these data. For each prompt, DPO-BT only selects the most preferred and dispreferred responses for training, while DPO-PL and DPO-HPS use all 100 responses.\n\nTo evaluate alignment, we measure Reward Margins (RM) in Table 1, where higher RM scores indicate better preference alignment with minimal harmful or biased outputs.\n\nFor PL methods like DPO-PL, directly using 100 responses per prompt (n=100) incurs excessive GPU memory costs (see Eq. 9 $\\mathcal{L}_{PL}$). To mitigate this, we reformulate the PL sub-loss $\\mathcal{L}\\_{j}(d)$ using Monte Carlo sampling:\n$$\n\\mathcal{L}\\_{j}(d)=-\\log\\left(\\frac{e^{r\\_{\\boldsymbol{\\theta}} (x,y\\_{\\tau(j)})}}{e^{r\\_{\\boldsymbol{\\theta}}(x,y\\_{\\tau(j)})}+N \\cdot\\mathbb{E}\\_{y\\sim u\\_{j}(x,y)}[e^{r\\_{\\boldsymbol{\\theta}}(x,y)}]}\\right),\n$$\nwhere $N=n-j$ and $u\\_{j}(x,y)$ is a uniform distribution over dispreferred responses. Instead of using all $N$ dispreferred responses, we sample 5 per loss term $\\mathcal{L}\\_{j}$, which is the maximum our 4×L40S GPUs can accommodate.\n\nThis sampling-based PL formulation is theoretically equivalent to vanilla PL and does not impact performance. The table below confirms that randomly sampling 5 or 1 dispreferred response from 100 yields similar performance on HH-RLHF and PKU-SafeRLHF. Since the strategy is developed in this work and used by HPS, it ensures a fair comparison between PL and HPS.\n\n|Dataset|BLEU|Reward|$RM_{DPO}$|$RM_{R-DPO}$|\n|-|-|-|-|-|\n|HH-RLHF|0.231|0.430|-0.859|-1.480|\n|PKU-SafeRLHF|0.302|0.410|-5.804|-6.061|\n\n**8)** We have briefly discussed explicit Preference Fine-Tuning methods (L155). We will discuss them more in the Appendix.", "claims_and_evidence": "**Claim 1** : model can distinguish between preferred and highly similar dispreferred responses more effectively – the authors empirically show better reward margins compared to baselines on HH-RLHF and PKU-Safety (Tables 2 - 5) and error bounds on optimal solutions in Thm 1. It is however unclear if the margin is between two responses that are similar, and if so, how they are similar (what is used to measure similarity?). This should be explicitly specified.\n\n**Claim 2** : HPS provably improves sample complexity over vanilla PL loss – the authors show this in Thm 1.\n\n**Claim 3** : HPS provably maximizes the reward margin for any prompt – the authors show this in Thm 2 and empirically in Tab 2 and Tab 4.", "methods_and_evaluation_criteria": "The proposed method is to compare HPS to status quo preference models (BT and PL) on popular safety datasets – Anthropic’s Helpful-Harmless and PKU Safety. They evaluate their method with an LLM as a judge to approximate real human preferences, i.e. win rates compared to baselines. They also evaluate transfer learning, i.e. training on one dataset and evaluating on another, which is a relevant and important measure of robustness. This criteria is sound, but relies on the base judge model’s quality (Qwen-2.5 Instruct). Ideally, there needs to be a survey with real human participants.", "theoretical_claims": "I read through the theoretical claims, but I did not check for correctness of proofs.", "experimental_designs_or_analyses": "The overall goals and high-level evaluation criteria makes sense (see comments in “Methods and Evaluation Criteria”), but the setting is quite unclear and underspecified in the writeup. The reviewer is left with several questions, all of which need to be explicitly mentioned in the revision:\n1. Practically, what value of $\\gamma$ is used ? Since the penalty for the dispreferred response (and improvements in reward margins) seem to rely on this, it is important to disclose.\n2. Practically, what is used to compute $r_{est}$, which itself is used to compute $q(x,y)$ (L 243 - 244) ?\n3. What exactly is the experimental setting with HH-RLHF and PKU Safety? In the reviewers understanding, in Tab 2 “DPO-PL” is a Llama3-8B base model trained on HH-RLHF with DPO assuming a Plackett-Luce preference model. Similarly “IPO-HPS” is a Llama3-8B base model trained on HH-RLHF with IPO assuming a HPS preference model. Is this understanding correct? This should be made a little more clear in the “Baselines” paragraph in Section 6\n4. The authors mention that prompts from these datasets used to generate 100 responses with each Llama3 model (SFT, PL, BT, HPS) which are then scored by the top-10 safety ranked RM (L 312) – is this understanding correct? Further, the reward margins are computed over only 2 responses (Tab 1) – how then are these 100 responses used? In the “Implementation” paragraph (Sec 6), the authors state that $n=5$ responses are used for PL methods, is this only for win rates? (L370). This is quite confusing.", "relation_to_broader_scientific_literature": "This work is relevant to the broader scientific literature. Empirically, it compares to many state-of-the-art and popular preference tuning methods, including DPO, IPO, EXO, SPPO, and NCA. It also addresses an important problem, i.e. the relative quality or \"badness\" of a dispreferred response explicitly, whereas Plackett-Luce models do so implicitly via a pairwise ranking across all pairs (thus creating an ordering over all responses). The authors do show favorable results compared to baselines in terms of reward margins between preferred and dispreferred responses, but without a qualitative human study, it is difficult to directly compare dispreferred responses across methods (e.g. HPS vs BT). From HPS’ “importance-weighted sampling” for efficiency (L264), the reviewer is also unclear how HPS differs a Bradley-Terry model where the dispreferred response is quality-weighted in some sense (e.g. with the same score obtained from a top-10 safety ranked reward model used for win rate). A brief discussion on the differences between HPS and quality-weighted BT would better highlight the contribution of this work.", "essential_references_not_discussed": "To the reviewer’s knowledge, no essential reference relating to Direct Alignment Algorithms has been overlooked by this work. The reviewer does recommend a (optional) discussion comparing to popular explicit reward modeling methods such as PPO (https://arxiv.org/abs/1707.06347) and GRPO (https://arxiv.org/abs/2402.03300), which would better situate this work in the literature. This can be done in the Appendix.", "other_strengths_and_weaknesses": "Overall, **I like this work**. It shows both empirically (on HH-RLHF and PKU-Safety) and theoretically (Thm 1- 3) that dispreferred responses can be \"pushed away\" from preferred responses with a simple objective modification in a sample efficient manner. I think it addresses a relevant problem: differentiating between dispreferred responses (i.e. all dispreferred responses are not equally bad) while maintaining low sample complexity. I would also like to highlight some weaknesses with the current version of the manuscript than can be improved to make a much stronger version of the work.\n\n**Weaknesses**:\n1. How is this work different from weighting the dispreferred response in a Bradley-Terry model according to its quality? (see \"Relation To Broader Scientific Literature\" for details)\n2. How is the \"importance-weighted dispreferred response\" that is the backbone for HPS sample efficiency (L262) chosen? This is very important to describe in detail as it is a crucial part of the algorithm.\n3. The experiment setting needs better clarity (see \"Experimental Designs Or Analyses\" section for details)\n\nWith these strengths and weaknesses in mind, I currently recommend a weak accept (3). If these three primary weaknesses are addressed in the revision, I am willing to increase my score to accept (4).\n\nEdit: As the above weaknesses have been discussed and committed to be updated in the rebuttal, I update my score to accept (4).", "other_comments_or_suggestions": "One optional suggestion: changing the acronym of this work. There is a pre-existing popular work in preference learning / alignment also called HPS (https://arxiv.org/abs/2306.09341), which is often used to score text-to-image models like Stable Diffusion or DALL-E. It may be slightly confusing to the community to also refer to this work as \"HPS\".\n\nMisc comments\n1. The point discussing that BT models \"leave other problematic responses unaddressed\" (L207- 210) is unsubstantiated. Popular state-of-the-art LLMs use BT models and work very well at scale while addressing these other problematic responses. This ties in to my request for a distinction of HPS from BT (see \"Relation To Broader Scientific Literature\" section).\n2. The point that PL loss trains models \"without considering the inter-ranking relationship among dispreferred responses\" (L219) is not strictly true. Each dispreferred response becomes the preferred response for the adjacent response which is slightly more dispreferred, and each of these pairwise losses are summed up. Thus, the most dispreferred response (last in the PL ranking) is a part of every loss $\\mathcal{L}_j$ and is weighted more highly as it is considered multiple times. What is true is that the *weights* for each loss are typically the same (1). I recommend rephrasing this portion to make this clearer", "questions_for_authors": "I had several questions about the evaluation setup (see \"Experimental Designs Or Analyses\" section)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKlX4pp7eI", "reviewer": "Reviewer_v4s5", "summary": "The paper introduces **Hard Preference Sampling (HPS)**, a framework for aligning Large Language Models (LLMs) with human preferences. It addresses issues in existing methods (PlackettLuce and Bradley-Terry models) by prioritizing preferred responses, explicitly rejecting dispreferred/harmful ones, and focusing on \"hard\" dispreferred responses to enhance rejection. HPS uses single-sample Monte Carlo sampling for efficiency and maximizes reward margins for clearer distinctions. Experiments on HH-RLHF and PKU-Safety datasets show HPS achieves comparable BLEU/reward scores while improving reward margins and reducing harmful content.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful and valuable comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.\n\n**1) PL simplifies to BT when $n=2$, but differs when $n \\geq 3$**, explaining their different results in Table 2. In this experiment, each prompt has 100 responses ($n=100$). BT selects the most preferred and dispreferred responses for training, whereas PL uses all 100 responses to compute its loss (Eq. 4 and 5), leading to different performance outcomes.\n\n**2) Regarding the sensitivity of DPO**, to address your concern, we have conducted experiments with $\\beta=(0.1,0.25,0.5,0.75,1)$ and report the KL divergence $\\mathbb{D}\\_{KL}[\\pi_{\\theta}(y_w|x)||\\pi_{ref}(y_w|x)]$ across these values, where $x$ is the prompt and $y_w$ is the winning response in the test set. The results in Tab.1 demonstrate the superiority of our HPS: it achieves the highest $RM_{R-DPO}$ for all KL values, confirming that HPS leads to stronger rejection of harmful responses.\n\n### Tab.1 Ablation results with $\\beta$ on HH-RLHF under fine-tuning setting.\n|$\\beta$|Method|KL|BLEU|Reward|$RM_{DPO}$|$RM_{R-DPO}$|\n|-|-|-|-|-|-|-|\n|0.1|DPO-BT|8.463|0.230|0.431|0.349|-0.455|\n|0.1|DPO-HPS|11.767|0.232|0.430|2.723|2.040|\n|0.25|DPO-BT|5.888|0.231|0.431|-0.206|-1.188|\n|0.25|DPO-HPS|6.972|0.230|0.431|-0.146|-0.828|\n|0.5|DPO-BT|2.661|0.229|0.430|-0.239|-1.022|\n|0.5|DPO-HPS|3.091|0.227|0.428|-0.228|-0.911|\n|0.75|DPO-BT|2.996|0.225|0.428|-0.264|-1.046|\n|0.75|DPO-HPS|2.192|0.226|0.427|-0.242|-0.925|\n|1|DPO-BT|2.043|0.227|0.430|-0.308|-1.990|\n|1|DPO-HPS|2.015|0.225|0.429|-0.316|-1.178|\n\nMoreover, we conducted a user study by selecting 15 prompt questions from HH-RLHF and 15 from PKU-SafeRLHF. For each question, four responses generated by SFT, DPO-BT, DPO-PL, and DPO-HPS are rated by 20 human evaluators on a 1–5 scale. To avoid bias, models were anonymized, and the response order was randomized. As shown in Tab.2, HPS achieves the highest quality score among all methods.\n\n### Tab.2 Human evaluation on user study dataset.\n|Method|Quality Score|\n|-|-|\n|SFT|3.63|\n|DPO-BT|3.82|\n|DPO-PL|3.69|\n|DPO-HPS|3.93|\n\n**3 For experiments)**, Tables 2–4 in our paper demonstrate that HPS-based methods consistently outperform other methods. For each prompt, DPO-BT only selects the most preferred and dispreferred responses among $n=100$ responses for training, while DPO-PL and DPO-HPS use all $n=100$ responses.\n\nMoreover, Table 5 presents an ablation study analyzing the impact of varying the number of responses $n=5,20,50,100$ on preference optimization. The results indicate that DPO-HPS scales better and achieves superior preference optimization with larger response sizes.\n\nSpecifically, we follow (arXiv:2306.17492) and expand response data by generating 100 responses using *RLHFlow/Llama3-v2-DPO* (arXiv:2405.07863) per prompt. The corresponding rewards are computed via *Skywork-Reward* (arXiv:2410.18451). Then we use methods like DPO-BT, DPO-PL, and DPO-HPS to fine-tune the language model on these data. Our preference fine-tuning methods explicitly leverage this broader response set rather than being constrained to the $n=2$ case.\n\n**4)**, \nExisting LLMs generate a single response autoregressively, which means that there is no inherent mechanism to ensure whether the generated content is harmful or harmless. This limitation raises concerns in scenarios that require adherence to safety and ethical standards.\n\nAdditionally, if we generate $K$ responses using the LLM, the computational cost becomes substantial due to the inference overhead. Even with multiple generated responses, LLMs without refined preference alignment cannot rank these responses autonomously, making it difficult to identify the top-ranked response based on quality.\n\nTo select the best response, a well-trained reward model is needed to rank the generated responses based on their quality and choose the highest-reward one. However, this approach introduces two key limitations:\n- Inference Cost: Generating $K$ responses incurs significant computational overhead.\n- Safety Concerns: While the top-ranked response may be of high quality, it is not guaranteed to be harmless, as both the LLM and reward model may fail to capture all potential risks.\n\nTo address these issues, we propose HPS, which ensures that lower-ranked responses with undesirable content are minimized, prioritizing the reduction of false negatives (L138). This consideration is particularly crucial in applications requiring high-quality and safe content generation, such as healthcare and education. \n\nFurthermore, our HPS can be extended to the setting where multiple top responses are valid. Please see our response to Reviewer 674W.\n\n**5) For the missing subscript**, we will correct it, and also carefully review the entire manuscript. Many thanks for your thorough proofreading and effort!", "claims_and_evidence": "HPS improves upon PL, but the experiments only include two responses with human-rated preferences, requiring scenarios with n > 2.", "methods_and_evaluation_criteria": "In many scenarios, it is not necessary for the less preferred option to be rejected; it is sufficient for the preferred option to be ranked higher than the less preferred one.", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Please see questions.", "other_comments_or_suggestions": "The subscript is missing in lin244 formula.", "questions_for_authors": "• When n=2, iPL simplifies to BT, so it would be helpful to understand why the results of DPO-PL and DPO-BT in Table 2 show notable differences.  \n• Since DPO is sensitive to the beta parameter, a more comprehensive comparison could involve testing different beta values, plotting KL divergence on the x-axis and performance metrics on the y-axis, to better assess the effectiveness of DPO-BT and DPO-HPS.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "etjRmmyDqf", "reviewer": "Reviewer_674W", "summary": "This paper propose a novel HPS method to prioritize the most preferred response while rejecting all other responses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful and positive comments! In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.\n\n**Our HPS method can also be extended to the setting where multiple top responses are all valid.** As stated in Sec 4 (L138), our primary objective is to ensure that models generate helpful and harmless responses while avoiding harmful or dispreferred outputs. In our setup, we assume $y\\_{\\tau(1)}$ is the preferred harmless response, while we cannot guarantee that $(y\\_{\\tau(2)},\\dots,y\\_{\\tau(n)})$ are entirely free from undesired content. Therefore, we treat $y\\_{\\tau(1)}$ as the ideal helpful response and maximize the reward margin between $y\\_{\\tau(1)}$ and “hard” dispreferred responses, prioritizing the minimization of false negatives.\n\nIn cases where multiple responses are valid, our HPS method can be extended to accommodate response diversity. Specifically, we can formulate a weighted HPS loss, treating each valid response as a preferred one in its respective loss term. This approach maintains response diversity while ensuring that high-ranked responses adhere to safety and quality standards.\n\nFor instance, given a training sample $d=(x,y\\_{\\tau(1)},y\\_{\\tau(2)},\\dots,y\\_{\\tau(n)})\\sim\\mathcal{D}$, if both $y\\_{\\tau(1)}$ and $y\\_{\\tau(2)}$ are helpful responses, we can redefine the objective to train the model to reject all dispreferred and potentially harmful responses $(y\\_{\\tau(i)})\\_{i=3}^n$, ensuring that it generates only the preferred responses $y\\_{\\tau(1)} $ and $ y\\_{\\tau(2)}$ for a given prompt $x$. The modified loss function is defined as a weighted sum of two HPS losses:$$\\mathcal{L}\\_{\\boldsymbol{\\theta}}=\\mathcal{L}\\_1+\\lambda\\cdot\\mathcal{L}\\_2$$where $\\lambda$ is a weighting hyperparameter, and$$\\mathcal{L}\\_{1}=\\mathbb{E}\\_{d\\sim\\mathcal{D}}-\\log\\left(\\frac{e^{{r\\_{\\theta}(x,y\\_{\\tau(1)})}}}{e^{{r\\_{\\theta}(x,y\\_{\\tau(1)})}}+ N\\_{1}\\cdot\\mathbb{E}\\_{y\\sim p(y)}[e^{{r\\_{\\theta}(x,y)}}q\\_{1}(x,y)]}\\right),$$$$\\mathcal{L}\\_{2}=\\mathbb{E}\\_{d\\sim\\mathcal{D}}-\\log\\left(\\frac{e^{{r\\_{\\theta}(x,y\\_{\\tau(2)})}}}{e^{{r\\_{\\theta}(x,y\\_{\\tau(2)})}}+ N\\_{2}\\cdot\\mathbb{E}\\_{y\\sim p(y)}[e^{{r\\_{\\theta}(x,y)}}q\\_{2}(x,y)]}\\right),$$with$$q\\_{1}(x,y)=\\frac{e^{\\gamma\\cdot r\\_{est}\\left(x,y\\right)}}{\\sum\\_{i=2}^{n}e^{\\gamma\\cdot r\\_{est}\\left(x,y\\_{\\tau(i)}\\right)}},$$$$q\\_{2}(x,y)=\\frac{e^{\\gamma\\cdot r\\_{est}\\left(x,y\\right)}}{\\sum\\_{i=3}^{n}e^{\\gamma\\cdot r\\_{est}\\left(x,y\\_{\\tau(i)}\\right)}},$$$N\\_{1}=n-1$, $N\\_{2}=n-2$, and $p(y)$ is the probability distribution of the dispreferred response $y$. By optimizing the weighted HPS loss $\\mathcal{L}\\_{\\boldsymbol{\\theta}}$, the model is encouraged to rank $y\\_{\\tau(1)}$ and $y\\_{\\tau(2)}$ above all dispreferred and potentially harmful responses $(y\\_{\\tau(i)})\\_{i=3}^n$, thereby maintaining both helpfulness and response diversity.\n\nWe will include this discussion in the revision.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, I checked the sampling complexity and reward margin analysis.", "experimental_designs_or_analyses": "The author mainly set three experiment sets:\n\n1. Main experiments comparing HPS with naive PL and BT modeling method.\n2. Human evaluation comparing HPS with the SFT/DPO-BT/DPO-PL baselines.\n3. Ablation on response number under the fine-tuning setting.", "relation_to_broader_scientific_literature": "The proposed method could be helpful for the field of LLM alignment targeting more helpful and harmless AI.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength\n1. The proposed HPS achieves better performance compared with PL and BT baseline.", "other_comments_or_suggestions": "1. The author propose to reject all but the top-1 ranked response, which might not be appropriate for a diverse range of scenarios where multiple possible responses are all good.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qwuXzazd9R", "reviewer": "Reviewer_85yJ", "summary": "The paper introduces Hard Preference Sampling (HPS), a framework for aligning large language models with human preferences. Traditional methods face challenges with harmful content, inefficient use of dispreferred responses, and high computational costs. HPS addresses these issues through a training loss that prioritizes preferred responses while rejecting dispreferred ones, with special emphasis on \"hard\" dispreferred samples that closely resemble preferred ones to enhance the model’s rejection capabilities.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments! We provide our response and hope our response addresses your concerns. We also look forward to the subsequent discussion which may further help solve the current issues.\n\n**1) Our revision will discuss alignment methods like SLiC-HF (arXiv:2305.10425) and LiPO (arXiv:2402.01878) based on list responses. SLiC-HF** is an alternative to RLHF-PPO by integrating the sequence-level contrastive method SLiC (arXiv:2210.00045) with human preference rankings:\n$$\n\\mathcal{L}(\\theta)=\\max(0,\\delta-\\log(\\pi\\_{\\theta}(y^{+}|x))+\\log(\\pi\\_{\\theta}(y^{-}|x))-\\lambda\\log(\\pi\\_{\\theta}(y\\_{ref}|x))).\n$$\n$y^{+}$, $y^{-}$, and $y_{ref}$ denote the positive, negative, and reference sequences, respectively. $\\delta$ is a margin hyperparameter and $\\lambda$ is a regularization weight. In contrast, our HPS framework focuses on rejecting all potentially harmful responses while leveraging the varying informativeness of dispreferred responses.\n\nIn **LiPO-$\\lambda$**, it employs a listwise ranking objective with a Lambda weight $\\Delta_{i,j}$. Given a list of responses $\\boldsymbol{y}=(y_1,\\dots,y_K)$,$$\\mathcal{L}\\_{LiPO}=\\mathbb{E}\\_{(x,\\boldsymbol{y},\\psi)\\sim\\mathcal{D}}\\left[\\sum\\_{\\psi\\_{i}>\\psi\\_{j}}\\Delta\\_{i,j}\\log(1+e^{-(s\\_i-s\\_j)})\\right],$$where $\\Delta_{i,j}=|2^{\\psi_i}-2^{\\psi_j}|\\cdot|\\frac{1}{\\log(1+\\tau(i))}-\\frac{1}{\\log(1+\\tau(j))}|.$ \nHere, $\\psi_{i}$ is the true reward score of response $y_i$, and $s_i=\\beta\\log\\frac{\\pi_{\\theta}(y_i|x)}{\\pi_{{ref}}(y_i|x)}$ is the implicit DPO reward. The rank position of $y_i$ in the ordering induced by $\\mathbf{s}=(s_1,\\dots,s_K)$ is denoted as $\\tau(i)$. The Lambda weight assigns greater importance to response pairs with larger preference gaps, i.e., $\\psi_i-\\psi_j$. However, our HPS prioritizes *hard* dispreferred responses—those that closely resemble the correct output but remain incorrect.\n\n**2) Our HPS can be extended to the setting where multiple top responses are valid.** Please see our response to Reviewer 674W.\n\n**3) Sampling a single importance-weighted dispreferred response DOES NOT reduce HPS to standard BT**, since HPS designs an importance-weighted sampling strategy, unlike BT’s deterministic selection.\n\nIn HPS, a dispreferred response is sampled based on the importance-weighted distribution (L244):$$q(x,y)=\\frac{e^{\\gamma\\cdot r_{est}(x,y)}}{\\sum_{i=2}^{n}e^{\\gamma\\cdot r_{est}(x,y_{\\tau(i)})}},\\tag{1}$$where $(y_{\\tau(i)})_{i=2}^{n}$ denote the dispreferred responses of a prompt $x$. This ensures that harder dispreferred responses—those more challenging to distinguish from preferred ones—are sampled more frequently and penalized more during training.\n\nIn contrast, BT always selects a fixed pair: the most preferred and the most dispreferred response, ignoring all other dispreferred responses. However, the most dispreferred response is often the easiest to reject. HPS addresses this by prioritizing harder responses, enabling the model to refine its distinctions between preferred and dispreferred outputs.\n\nEmpirically, BT struggles to capture the preference gap effectively. Table 2 shows that compared to BT, HPS significantly improves reward margins, reducing detrimental responses.\n\n**4)** While our experiments focus on $n\\leq 100$, Thm. 1 analytically characterizes how error bounds $\\Psi_{1}$ and $\\Psi_{2}$ scale with $n$, providing insights for larger settings. We limit $n$ to 100 due to limited GPU resources.\n\nMoreover, Table 5 shows that as $n$ increases, the reward margin metrics consistently improve, aligning with the theoretical scaling behavior.\n\n**5)** In Thm. 1, $d$ is defined in Assumption 1 (L239), and denotes the parameter dimension of the reward model. For implicit reward modeling, it corresponds to the trainable LLM’s parameter dimension. We will carefully refine our manuscript.\n\n**6)** HPS samples a single dispreferred response via the importance-weighted distribution $q(x,y)$ (see Eq. 1 in response **3)**)\n\n$q(x,y)$ defines a probability distribution over these responses, so sampling strictly follows this distribution. Deviating from it would introduce a different approach, compromising the theoretical guarantees in Sec. 5, particularly *HPS’s improved sample efficiency and reward margin maximization over PL methods.*\n\n**7)** We use Qwen2.5-72B-Instruct for evaluation. In Sec. 7, we acknowledge that budget constraints limit us to open-source LLMs for estimating win rates. To strengthen our evaluation, we conducted a user study with human participants. See Tab.2 in our response **2)** to Reviewer v4s5 for details.\n\n**8)** Thanks. Since we mainly analyze PL and BT, we use these methods to investigate theoretical implications and empirical performance. To address your concern, we compare HPS with LiPO-$\\lambda$ on HH-RLHF and find that HPS significantly improves the Reward Margin, limiting harmful responses. \n\n|Method|BLEU|Reward|$RM_{DPO}$|$RM_{R-DPO}$|\n|-|-|-|-|-|\n|**LiPO-$\\lambda$**|0.229|0.430|1.437|1.121|", "claims_and_evidence": "The paper assumes the first response is preferred while subsequent responses are potentially harmful or purely dispreferred. However, this overlooks preference diversity among humans—several top-ranked responses might be acceptable but with varying preference levels based on individual tastes regarding grammar, expression style, and other subjective factors, rather than being inherently harmful.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I briefly examine the theoretical proofs.", "experimental_designs_or_analyses": "1. The paper mentions \"sampling only a single importance-weighted dispreferred response\" but fails to clarify the specific selection method—whether random or based on certain criteria. Additionally, the experiments could have benefited from comparing different sampling strategies to strengthen their claims.\n2. The authors use Qwen-2.5 for evaluation but fail to specify which variant (3B, 7B, or 72B). They also don't justify why they chose Qwen over more commonly used models like GPT or Claude, which are standard in most preference alignment papers.\n3. The authors only compare their approach with PL model-based methods, overlooking alternative alignment techniques such as Lambda loss [1], which could have provided a more comprehensive evaluation of their method's effectiveness.\n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank", "relation_to_broader_scientific_literature": "The paper should discuss alignment methods based on list responses like [1,2].\n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank.\n[2] SLiC-HF: Sequence Likelihood Calibration with Human Feedback.", "essential_references_not_discussed": "The paper should discuss alignment methods based on list responses like [1,2].\n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank.\n[2] SLiC-HF: Sequence Likelihood Calibration with Human Feedback.", "other_strengths_and_weaknesses": "Strengths:\n1. HPS introduces a training loss that explicitly prioritizes preferred responses while rejecting all dispreferred ones, focusing particularly on \"hard\" negative examples.\n2. Experiments show substantial improvements in reward margins compared to traditional methods.\n3. HPS innovatively applies Monte Carlo importance sampling to replace the dispreferred term in PL loss, offering a more efficient \nalignment approach.\n\nWeaknesses:\n1. The paper assumes the first response is preferred while subsequent responses are potentially harmful or purely dispreferred. However, this overlooks preference diversity among humans—several top-ranked responses might be acceptable but with varying preference levels based on individual tastes regarding grammar, expression style, and other subjective factors, rather than being inherently harmful.\n2. In the implementation, the authors sample only a single importance-weighted dispreferred response, causing the method to degenerate into standard Bradley-Terry model. This design choice naturally results in faster computation compared to PL model-based methods.\n3. Since experiments only involve $n \\leq 100$, the difference between the two error bounds amounts to merely a constant factor, suggesting the authors may have overstated the theoretical contribution.", "other_comments_or_suggestions": "1. The paper's writing quality could be improved, as it contains several repetitive statements and fails to properly define all variables in the theorems, such as $d$ in Theorem 1.", "questions_for_authors": "1. The paper mentions \"sampling only a single importance-weighted dispreferred response\" but fails to clarify the specific selection method—whether random or based on certain criteria. Additionally, the experiments could have benefited from comparing different sampling strategies to strengthen their claims.\n2. The authors use Qwen-2.5 for evaluation but fail to specify which variant (3B, 7B, or 72B). They also don't justify why they chose Qwen over more commonly used models like GPT or Claude, which are standard in most preference alignment papers.\n3. The authors only compare their approach with PL model-based methods, overlooking alternative alignment techniques such as Lambda loss [1], which could have provided a more comprehensive evaluation of their method's effectiveness.\n\n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on preference based safety alignment for and proposes sampling hard preferences (HPS). The analysis presented in the paper relies on the assumption data being generated from the true model and does not address model mismatch. As reviewers have pointed out, while the idea of hard negative sampling exists in related literature of metric learning and contrastive learning, application of this idea to preference based alignment setting is interesting. The theoretical analysis assumes that data comes from the true model which while limiting is still an interesting analysis providing insights on why HPS is advantageous over using the ranked list under PL model. In summary the strengths outweigh the limitations. 

Please include the discussion to the additional relevant literature pointed out the by the reviewers and the promised discussion during rebuttal-response for weaknesses raised.
Decision: Accept (poster)