Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models
Authors: Wenxuan Zhang
Philip Torr
Mohamed Elhoseiny
Adel Bibi
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Models
RLHF
Safety
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce a theoretically framework to re-parameterize the multi-objective RLHF into supervised optimization and empirically show the effectiveness in improving both the helpfulness and harmlessness in LLM.

Abstract: Fine-tuning large language models (LLMs)  on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities.  However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in  safety and helpfulness  is costly in RLHF.  To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In the supervised optimization, a labeling function is used to capture global preferences ranking to balance both safety and helpfulness. To  evaluate BFPO, we  develop a benchmark  including comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO eliminates the need for human prompting and annotation in LLM fine-tuning while achieving the same level of safety as methods that heavily rely on human labor, with less than 10\% of the computational resources. The training recipes and models will be released.

Published as a conference paper at ICLR 2025
BI-FACTORIAL PREFERENCE OPTIMIZATION:
BALANCING SAFETY-HELPFULNEE IN LANGUAGE MODELS
Wenxuan Zhang', Philip H.S. Torr", Mohamed Elhoseiny" Adel Bibi 1 King Abdullah University of Science and Technology 2 University of Oxford
(wenxuan. zhang, mohamed.. e chsseiiyll @kaust.. edd..eduu... (philip.tore torr, adel. bibi 8eng. ex. ac.uk
ABSTRACT
Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven suc- cessful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains critical concern and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a su- pervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into single supervised learning objective. In the supervised optimization, labeling function is used to capture global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark including comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO achieves the same level of safety as methods that heavily rely human labor with less than 10% of the computational resources and human prompting and annotation process. The training recipes can be found here: https:/ http://tttttt gigttb......... com/ wx zhanq/bfpo Warning: This paper contains offensive or harmful content.
INTRODUCTION
Fine-tuning the large language models (LLMs) on human preferences, also known as model alignment, has been shown to greatly enhance their text generation abilities (Ouyang al. [022 Askell et al..
2021
OpenAT 2023). When prompted with well-formatted instructions, these models can generate helpful responses for various tasks like scientific questions answering (Singhal et al. 2003). creative writing Yuan et al. (022)) coding (Chen et al.; 2021 Guo al. 2024), and planning (Wang et al. Valmeekam et al. (023). However, with this instruction following ability, the models can also be prompted generate harmful content, such sexist racist comments, suggestions for committing crimes, or sensitive medical advice (Bender et al. 2021 Weidinger et
2023
al. [202]] Zou 2023). Therefore, it is crucial ensure that L.LMs are both helpful and safe (Ouyang al.
al.
2022
Askell et al. 2021 Bai et al. 2022).
The challenge of improving both safety and helpfulness of LLMs arises from the inherent tension between these two objectives (Bai et al. 2022 Touvron et al. 2023 Qi et al. (023). A perfectly safe model may refuse to answer even narmhar questions (I (Figure leit, b), while highly helpful model (Figure [FigureT-right,, a) may compromise safety. Additionally, training model on a mix of data annotated with different preference objectives can lead issues, as shown with model (c) Figure where the model refuses to generate code (left) yet still produces harmful content (right).
To resolve this problem, recent studies propose to train separate reward models tailored to each objective, safety and helpfulness, and optimize LLMSs via multi-objective RLHE, which aggregates
"Equal Advising

---

Published as a conference paper at ICLR 2025
ook for a code snippet related to the search query: Python loop
Which drug léels better cocaine heroine?
Here's Python
Yes, in Python loups to iterate over collections perform actions on each item.
Python
Here
depends on the vnduvl For some people, cocsine may give them energy...
I can't answer that guestion. However, it's important to note that both drugs are highly addictive...
The cuphoric cffects of cocaine are generally shorter-ling compared to heroin...
Neither drug feels better
code snipper for
various types of loops, such - "ooe' loop "hhlle" loop (to iterate based
example of Python loop: "pyshon for is cunce (51 peint (A)
a common loop stracture: pyylhon for iten de soom, abezables peint (Abeea
It is best to avoid drug use and seek altemative...
on condition)
[n) Height Data IPP
IU Safety Data
(c) phul Dats
(0) Helphll Data Safety Dats Curn
(N) Helppu Data DPO
(b) Safwry Date
(c) Hetrfull Data Safwey Dats IPD
(d) Helppu Data Sunieyy Eatt Olums
DPO
Safety DPQ
DPO
Figure I: Four models are trained with different data sources and algorithms. Model (a), trained only helpfulness dataset using DPO, generates harmful content (right). Model (b), trained solely on a safety dataset with DPO, fails to follow instructions to write a snippet (left). Model (c), trained with naive mix of datasets using DPO, may be both non-helpful and harmful. Our algorithm aligns Model (d) to achieve both helpfulness and harmlessness.
reward scores over all objectives (Bai et al. 2022 Touvron et al. 2023 Dai al. 2024 Mu
al. 2024). However, developing a safety reward model requires a suffccient number of unsafe responses specific to the model being trained, often by process known as red teaming, which is both labor-intensive and computationally demanding (Touvron et al.) 2023 Mu et al. (024). In contrast, |Rfailov al. (2023) re-parameterized RLHF into more efficient supervised optimization. However, current work typically focuses on re- pa-amaterrzzing single reward RLHF objective within the supervised learning framework, and extending this c--parameterization the multi-reward case is not straightforward (Zhou et al. 2023).
In light of these challenges, we first introduce labeling function that accurately represents the global ranking of responses based on both helpfulness and harmlessness within the supervised learning framework. We then establish theoretical equivalence between this supervised optimization and the well-established multi-objective RLHF with combination of the rewards of safety and helpfulness. This equivalence ensures that the optimal model obtained through our supervised learning framework also optimizes both safety and helpfulness reward in RL. We denote this framework as Bi-Factorial Preference Optimization (BFPO). To evaluate our framework, we first establish benchmark including both safety and helpfulness tasks for LLMs. Using this benchmark, effectively develops highly safe LLMs while preserving their helpfulness. Our approach relies only on publicly available datasets, and achieves results comparable to those of methods requiring extensive human labeling efforts to model specific outputs. Moreover, we show that this approach can further enhance the safety of aligned models using just 1.5K red teaming prompts, achieving comparable performance with those methods requiring expensive red teaming. Our contributions are:
demonstrate that BFPO
We re-parameterize the multi-reward RLHF objective, that balances safety and helpfulness, into a single supervised learning objective. In the supervised optimization, we introduce a labeling function that captures global preferences ranking to balance both safety and helpfulness.
We establish safety evaluation protocol that includes extensive discriminative and generative tasks, and we perform evaluations on open-sourced L.MMs.
Using our algorithm, we efficiently improve the harmlessness of open-sourced models by 15% with a public dataset and by 13% with only 1.5K red teaming data, all while preserving helpfulness. Our method achieves safety scores comparable to those of labor-intensiee methods without requiring human prompting or annotations specific to the model being trained.
2
PRELIMINARY
Notation and Terminology. Let x and y denote the input prompts their corresponding responses, respectively. For any two responses, y, 1 generated from prompt we denote is preferred over y as y > y/ Then human annotators can provide binary preference labels [ > y/(z) on whether y is preferred. The preferred response is termed the "win response", denoted y",, and the other as the "lose response", y'. A dataset D = {((x...) y.y' Iyy > l/(x))} that contains prompts, multiple responses, and the human preferences over the responses referred to as a preference dataset.
2

---

Published as a conference paper at ICLR 2025
Following Azar et al. (2024). we define the ground truth preference p" of over yf as the expected
preference label across a broad group of human annotators, i.e., p" [y > y ()) = E E > y []]]
The ground-truth score of single response y generated by model is then the expected value of its paired preferences with all other responses, ie., p" [y > T|z)) = Eyg' [p"(y (y > y [/]]
RLHE RLHF typically consists of two phases (Stiennon al. 2020 Zheng et al. 2023 supervised reward learning and policy optimization by reinforcement learning (RL). The training of the reward
model re. parameterized by é. is framed by Bradley-Tery (BT) modeling (Bradley & Terry) 1952).
which employs the logistic loss to maximize the distance between the output reward scores of win and lose responses,
(1) where is sigmoid function, and D is preference dataset. The trained reward model re then provides reward scores for the RL phase. The language model тæ. or policy in the RL phase, is optimized with the objective of maximizing the KI-regularized reward (Schulman et al. 2017). Le.,
(6) -B "/)=,""""/////"""// "o o ra(a, (x,,,'))] y'))],
max E.= a) (x, y) rKL [To [[]==]] Ty
(2)
where - is a penalty coefficient for the KL divergence term, which prevents the policy Ty from significantly deviating from reference policy Tet. In practice, the reward learning and policy training are often carried out iteratively, with Tref as the initial model at the start of each round of RL.
Multi-objective RLHE In multi-objective RLHF, Equation (2) extended include multiple reward functions, each corresponding to a specific objective (Touvron et al. 2023 Dai et al. 2024 Zhou ct al. 2023 Chakraborty et al. 2024 Wang et al., 2024b).
max E, D...D.y-aa (yvop((la)) [9( (2, (z.,)...... Vo. (z. w)) rKL (ITO (ro(v))) [+
where rdy:- -...-... the reward models, each trained separately, and g R. R is function that combines the reward scores from multiple reward models.
Direct Preference Optimization (DPO). Rafailov et al. (2023) reveals that the reward r can be re-parameterized by the policy x, and the policy can be optimized through supervised reward learning:
Te "0y"" [x) Teef Tref xx)
To (x) T [x)
min - Faa log or of log
log
(4)
Notably, the data points x, y" y in this objective are not necessarily generated from To while is updated; instead, they can instead be drawn from a public preference dataset D).
Generalization of DPO. Azar et al. (2024): Tang et al. (2024) further reveals that single reward and the optimal solution xt of RLHF in Equation are related by the equation x (wlar) x [y)() exp (T (r(,) When comparing two responses, y" and y this relationship yields:
" (x)T" (y² (y
hy" v²) = log
(((x,,") = r(z,v"))
Details of the relationship are elaborated in Theorem 3.1 As Equation (5) holds for the optimal policy n", we can directly minimize the difference of the two sides with supervised loss f
min P. (e.yy go) [c(hTT "u""""),, " ,/(=:))]]
(6)
where : RR2 R is real-valuee label function that approximates the value rxx,y") r (xx, The optimal policy obtained by Equation (6) is then equivalent to that of Equation (2)
Notation Modification. In this paper, we use subscripts to distinguish between two key perspectives: helpfulness and harmlessness. The preference label for helpfulness between two responses denoted as (y > 3 [z), and the safety label for response y is denoted (y)t))))z) We introduce the notation A less helpful response, regardless of safety. Throughout the paper, we refer to the dataset measuring helpfulness the helpfulness dataset, which usually provides label for the preferred response out of two responses, while the dataset measuring safety with safety labels per response is referred to as the safety dataset. Please refer Table 5 for summary of the notation.
if Ihepp (y > y'(z) 1, i.e., // is the more helpful response, and yd is the
3

---

Published as a conference paper at ICLR 2025
Helpfulness
Hateful
Desired
Larger Positive Gap
gr tw
(a): strongly advise
heroin, here are the ways
against taking heroin. If you or someone you know; strugeling heroin
y huu Safe (Desired)
to obtain I, Street
y
Unsale (Neutral)
1.5
deals: The mujurity people beroin
addiction, please seek professional help
y
Safe (Desired) Safe (Acceptable)
obtaii through....
you s/
Harmful Response
Safe Response
Unsafe (Hateful)
3
0
Unsafe (Neutral)
Injecting
(b): strongly advise against taking
common method in taking heroin.
y to Unsafe (Hateful)
0.5 a
nd Safe (Accepiable)
Neutral
Acceplable
Smaller Negative Gap
Figure 2: Global preference ranking of different responses.
Figure 3: Pair-wise preference of responses you y with different safety label, and the label values.
BFPO FRAMEWORK: BI-FACTORIAL PREFERENCE OPTIMIZATION
In this section, aim extend the supervised learning framework in Equation (6) to improve both safety and helpfulness in LLM alignment. Naively, we could combine the helpfulness and safety datasets, treating safer response in safety dataset and more helpful response in the helpfulness dataset as the win response y" in Equation (6). However, there is an inherent tension between the helpfulness and harmlessness objectives. perfectly safe, but it would fail to meet the user's needs. Conversely, a highly responsive model that attempts to address all requests, including potentially harmful ones, may compromise safety in favor of helpfulness (Nadeau et al. 2024). The naive combination of datasets could inadvertently lead to training on these contradictory outcomes, as we shall show in the experiments.
model that refuses to answer any request would be
On the other hand, Touvron et al. )203)); Dai et al. (2024) developed successful multi-objective RLHF methods resolve this tension, with the objective
max D. [g(v(=) -KI.
(7)
where g(v(z)) g("nepp (2. v), Taate y)) is function that combines the helpfulness reward Thelp (a:, y) and safety reward ssaee y). Therefore, re- pa-rameteriziig Equation (7) to supervised objective leads an efficient and effective alignment method. The target objective is:
min (z DnD (ha (y"",, " ))),, "r """",""("))) ,²)=)))]
(8)
where Ao" and nd are the more helpful and less helpful responses, and as we defined in Equation
(1yh0022 (z) (ror (yyhl [ (y
he (y ,yhh) log(
and gr is the label function that leverages the safety labels Lato (y hw (x), Lsaio (g (()r)) approximate the value g0y fow (x)) 9
[I])) where g is the global reward function in Equation
In Section 3.1 first develop an empirical labeling function gr that accurately represents the global reward of responses based on both helpfulness and harmlessness. We then establish the theoretical
equivalence between Equation (8) with this 9ı and Equation (7) in Section 3.2) Next, we present the
algorithm in Section 3.3 and provide sample illustration in Section 3.4
3.1 EMPIRICAL LABELING FUNCTION
In previous single-reward optimization methods (Rafailov et al. 2023 Azar et al. 2024 Tang
et al. 2024), 9I (y" y (x) in Equation (6) is typically positive constant. However, in gr ".y nt x). which approximates the global reward disparity between the more helpful response glynd (ar), should vary depending the safety of /ou
and the less helpful response, ie.,
ho (r) nw
4

---

Published as a conference paper at ICLR 2025
and yht For example, in Figure response (a) is more helpful than response (b), and the global reward disparity between (a) and (b) should be positive since both are safe. However, the global reward disparity between the more helpful (c) and less helpful (b) should be negative, because (c) is less preferred for its detailed harmful information. In fact, the absolute value of g(y (a) g(y (ar)
reflects the magnitude of the global preference disparity between the two responses, while its sign determines whether you is globally preferred over ydd.
To assign label values across various y²u,, ,n"h pairs, we first globally rank the responses illustrated in Figure 2 Our guiding principle is general preference for safe responses, prioritizing helpfulness only if the responses is safe. We desire the helpful and safe responses like (a) in Figure [ followed by the acceptable non-helpful but safe responses like (b). We remain neutral toward the harmful but unhelpful responses like (d), and we hate the harmful yet exhaustive (helpful) responses like (c).
Given two responses y assuming we have their relative helpfulness ranking, there are four classes of pairs based on their safety, illustrated in Figure 3 For ① and (2) we prefer the safe and more helpful y than the other response, so the signs of the labels should be positive. Similarly, the signs of 3 and (4 should be negative. The preference gap for ! (Desired vs. Neutral) is larger than for (2, thus the magnitude of the labels should be greater in ①.. Likewise, the magnitude of labels of 4 should be greater than that of 3. Consequently, the label value of the four class of pairs should be ordered as D. 3 and ④ To construct the label function that fulfills this order, we first need a minimization over the safety labels. To ensure a positive label for 2, we require a larger scalar weighting the safety of fhw compared that of yhl We hypothesize the label function ør as:
(9) In this equation, B, is positive scalar that weights the safety of yoo B3 is a constant to prevent the label, which approximates the disparity of the rewards, from collapsing to zero B3 is a scaling factor
gr gv(y Am 10) Ba Ba((Brlateeh Iateey [=1x) (yate x)) + B2).
to adjust the overall magnitude of the label values. For instance, let B1 = 3, B, = -2a, B3 = 0.5, Figure right illustrates label values of different pairs.
3.2 THEORETICALLY EQUIVALENT REWARD
In this section, we show that the supervised optimization problem in Equation (8). with specific label- ing function in Equation (.. is theoretically equivalent to the multi-objective RLHF Equation 7 with particular reward function. Previous studies (Touvron et al. 2023 Dai et al. 2024) in aligning
LLMs for both safety and helpfulness have shown that the global reward function can be effectively
approximated by bilinear combination of the two sub-rewards; see Appendix C.2 for more details. We hypothesize the global reward function as follows:
(10) where A1, A2 are two constants that prevent the reward from being nullified by zero values, and
) = (Pite(y)ar) (y)ær) + A1 (Pheppp) > x|)) + A2).
Phuelp: Psate [0.1] are the ground-truth helpful and safety preferences of response y. Let A
Eg, Ag = 3. B₁ = 3, B2 = 0, B3 = 1/2 we have the reward function and labeling function gr:
1
( x) = (p)z)) + Ea)
> x|z) + 2
(11)
3
gr hy yi (o)) = Tsate (1/
Aw
2 Saate (1/
(12)
2
where Eg- = Eyos [Pate [w]=]] represent the ground truth average safety of responses given prompt x.
The following theorems reveal the theoretical equivalence.
Theorem 3.1 ( [Azar et al. (2024). The oplimization problem in Equation has a solution Tt'
Tee(U(ar) exp (r (v(z))
<" (ylz)) = Ey (in (z) exp (7 g5 (z))
and n² (g) is the unique solution to the following optimization problem
h (u.v"" g(ylz)) 9(v'l=)²
min E2--
(13)
Theorem 3.2. The optimization problem in Equation proposed g and gr function.
13 and Equation are equivalent under the
5

---

Published as a conference paper at ICLR 2025
Action Probability:
Riral
niral
0.74
Ours
Loss: DPO, T-0.1
Loss: JPO, T-0.1
Loss: BFPO, T-0.1
0.0
0.7
0.50
0.1
0.68
Beaner
Mistral
0.35
CpenC 5-7h
Instric-
0.66
500
1000
1900
530
1000
1300
500
1000
1500
0.64
Loss: DPO, 1=1.0
IPO, 1=1.0
Loss: BFPO, r=1.0
Perference
0.75
Optiiizaionn
0.5
0.50
0225
Zughyn-Ph- Saaghy--h-hatt
0.25
022
0.05
n.1
0.15
300
1000
1500
530
1000
1500
500
1000
1500
Holgfalnass
Figure 4: Action probabilities steps during the pol- icy optimization using DPO, IPO, and our BFPO in syn- thetic dataset. Only recover the desired ranking.
Figure 5: Helpfulness and harmless- ness of open sourced models. The mark size represents the approximated train- ing data size and annotation cost.
With Theorem 3.1 we can obtain the optimal n² by solving the supervised optimization problem in Equation The proof of this theorem is in Appendix [8.2] However, the optimization problem in Equation 13 remains challenging because the function g(y) involves the ground-truth preference p", which requires estimation by a large group annotators.
address this, Theorem 3.2 shows
equivalent to solve the supervised optimization problem in Equation (8) with the proposed gı to obtain the optimal π². The proof of this equivalence is provided Appendix B.3 We further discuss
the general equivalence with different constants A1, A2, B1, B2, B3 in Appendix B.4
The proposed supervised optimization problem in Equation (8) and labeling function 9s in Equa- tion (12) also possess several properties that offer flexibility when constructing algorithms. These properties are discussed in the following proposition and in Appendix [B.5]
Proposition 3.3. Theorem 3.7 and Theorem 3.2 hold under the shift of the preference values in g and gr, for constants pt, ,2, we have
g(ul=)= = (Pina (u|=x) + Ea) Ex) > T|Φ) + p2 + 5)
gi fm M'xx) = 2 (lmp (M [x) pi) ) (Ang (1' + p2).
3
This property allows to adjust the preference labels of the responses. Proof of the proposition is provided in Appendix B.5 In practice, we further apply a shift of the safety label value a as
gr (x) safe (y x2)
ssate(y
(1)) a.
(14)
The factor a is useful when set to negative values to distinguish unsafe samples, i.e., to make the value case 3 in Figure i.e., both responses are not safe, deviate from O.
3.3 ALGORITHM
With previous discussions, the loss function in the optimization problem in Equation
8
Te (y/ (or) ((x)) na (1(") (1)
safe (9/ hm [x) safe (g) (m)
(Brro Pyw.ykw,hhhoo
log
(15)
In practice, we directly use the above supervised loss to fine-tune the LLMs for both helpfulness and harmlessness, n/ and nd can be sampled from public preference dataset D instead of being self-generated (Rafailov et al., 2023). The safety labels (y (1 are either provided in the dataset or obtained by safety classifier. The probability x (z)(2) of generating the response y given prompt x is obtained by forwarding the prompt and response through the LLM и. To is the language model are optimizing, and मee is reference model that can be the model the

---

Published as a conference paper at ICLR 2025
Table 1: Results of fine-tuning pre-trained model, Table 2: Results of further fine-tuning the
Mistral, with various methods. Our method achieves the highest harmlessness score and the best balance
aligned Zephyr model with red teamnin data. Our method improves helpfulness and achieves the highest harmlessness score.
over helpfulness and harmlessness.
Helpfainess
Harmlessness
Helpfulness
Harmlessness
Alpaca(")
Disc. (1)
Geo. (1)
Sang. (1)
Alpaca
Savg
DPO-I II (Zephyr)
10.99
59.05
62.94
60.99
Zophye-7b-bet
10,99
59.05
62.94
60.99
DPO-S
4.34
56.42
96.91
76.66
DPO
DPO
IPO
13.15
58.41
89.76
74.09
IPO
13.07
59.32
72.82
66.07
MOKL
10.33
58,54
64.88
61.71
13.33
59.09
HPPO
14.41
39.02
88.79
73.90
beginning of the optimization. We further sample batches of the same size from the safety dataset and the helpful dataset, inspired by [Chaudhry et al. (2019), to balance safety and helpfulness. The overall algorithm is summarized in Algorithm
3.4 ILLUSTRATIVE EXAMPLES
Following Azar et al. 2024), we conduct illustrative experiments on a synthetic dataset to demonstrate that our method can accurately recover the global preference using paired preferences. For simplicity, we consider a discrete action space with four actions, ) {311 U2. Vs. V1). without context. We define the safety labels and helpfulness ranking as
Safety: (11)
sate (V2) O, sate (Ws) 1, sate (1a) 0,
Helpfulness: y1 > > y3 >
Consequently, our proposed global preference, as Figure is > 8/ > 44 > y2- We encode the policy (1;) softmax(O)) using a vector 0 E RR. and 3,4.3.4. 4. The preference dataset is constructed from all pairs of actions, along with their paired helpfulness rankings and safety labels. We optimize the policy with the Adam optimizer for 1800 steps, with a learning rate of 0.01, batch size of 32 sampled with replacement, T = L, and a = 0.5. We compare the supervised optimization objective proposed in Equation (15) well as DPO (Rafailov et al., 2023) and IPO (Azar al. 2024), where we take the more helpful response is taken as the win response. Each method is tested with five repeat experiments, and we plot the average learning curves Figure
For all we observe that only with our proposed method does x (y:). i.e., the probability of generating action yt. converges to the desired ranking, V1 > V/ > 3/4 > y/2 DPO and IPO can only recover the ranking based on helpfulness, leading to an incorrect order. While IPO prevents the policy from being deterministic, our method retains this beneficial property while also achieving the correct ranking.
EXPERIMENT
4.1 EVALUATION SETUP
Harmlessness Benchmark. To evaluate the harmlessness, first construct benchmark including
both discriminative tasks and generative tasks based on previous benchmarks (Srivastava et al. 2023
Gao et al. 2023 Tedeschi et al.] 2024 Zou al. 2023). The discriminative tasks measure the
models' recognition of multiple safety topics, including bias (CrowS-Pain) (Nangia et al.) 2020).
BBQ Parrish et al.) 2022). WinoGrande (Sakaguchi al. (021)). ethics (ETHICS (Hendrycks)
et al. 2021), Moral Permissibility (Srivastava et al.) [023] Hernandez et al. 2021 Lourie et al.
2021 Thomson, 2019), Simple Ethics Questions (Hendrycks et al.) [021]
[001:]Loure Lourie et al. 2021) }), and
toxicity (ToxicGen (Hartvigsen et (022)., BigBench HHH Alignment (Srivastava et al. (223)). In the generative tasks, we prompt the models to generate harmful content using the prompt dataset, AdvBench (Zou et al., 2023), Real Toxicity Prompts (Gehman et al.) 2020), ALERT (Tedeschi et al.)
2024). We report percentage of harmless responses based on the safety classifier HarmBench-Llamaz. be found in Appendix C.I] We apply this benchmark to publicly available 7B-level models that have shown strong helpfulness scores in
13B-Chat (Mazeika al. 2024). Details of the benchmark
Gao al. 2023): Dubois et al. (2024b), and present the performance in Figure F and in Appendix C.4
7

---

Published as a conference paper at ICLR 2025
Overall Evaluation Metrics. In the following experiments, report both the helpfulness and harmlessness performance. Helpfulness is measured using AlpacaEval 2.0 (Alpaca) (Dubois et al. 2024a Li et al. 2023; Dubois et al. 2024b). Harmlessness is assessed using the performance of discriminative tasks (Disc.), generative tasks (Gen.) from aforementioned benchmark, and the average safety over these two metrics (Savg.).
4.2 ALIGNMENT WITH BFPO OBJECTIVE
From the evaluation on the open model in Figure 5 we observe that Zephyr-7b-bete (Tunstall et al. 2023), an open-sourced model fine-tuned over Mistral-7B-v0.1 (Tiag et al.) (023a) with DPO algorithm (Rafailov et al. 2023), exhibits a low score in harmlessness, particularly in generative tasks. In this section, we apply the BFPO algorithm to finetune the same base model 11 aiming to improve harmlessness while maintaining the same level of helpfulness.
Training Details. Our training process consists of two stages: supervised fine-tuning and BFPO optimization. The supervised fine-tuned model is used as the reference model Tre in the BFPO stage. We set 7 = 0.01, a = 0.5. We implement PEFT training for all baselines, where we only unfirezze the selected layers 0', the second MLP layers in each transformer block, in the policy To Zhang et al. (2024). All other hyperparameters remain the same as in the original Zephyr training.
Dataset Details. In the supervised fine-tuning stage, we follow Tunstall et al. (202)) Dai al. (2024) to use a mix of helpfulness data from UltraChat (Ding et al. (023)) and safety data from PKU (Dai et al., 2024). In the BFPO stage, we use 30K helpfulness data from Ultra- Feedback (Cui et al. 2023) and 30K safety data from PKU-SafeRL.HF UltraFeedback contains eetruuttion-folloingg tasks that provide paired helpfulness preference rankings, and we treat all re- sponses as safe since they undergo human filtering. PKU-SafeRLHF provides both paired helpfulness preference rankings and binary safety labels. Details are in Appendix C.4
Baselines. We first compare our method to the supervised method DPO (Rafailov et al. 2023 using different datasets., which directly leads to the Sephyr-7b-beetee model, only uses the helpfulness dataset, UltraChat. DPO-S only uses the safety dataset, PK UKKU-SfeRR.HF We also compare our method to existing approaches, DPO (Rafailov et al. 2023), IPO (Azar et al. 2024), and MORL (Ramé et al., 2023). when using a naive mix of the helpfulness and safety datasets. In DPO and IPO, we treat the safer response from the harmlessness dataset and the more helpful response from the helpfulness dataset as the win response. MORL, representing the line of multi-objective reinforcement learning methods using PPO optimization (Touvron et al.) 2023 Dai et al., 2024 Ramé et al. 2023) Dong et al. 2023 Wang et al. 2024b), requires reward models. Following Wang al. (2024b), we use single highly-ranked (Lambert et al. 2024), publicly available reward model, Ar
v0.1 (Wang al. 2024c), to provide reward scores for both helpfulness and harmlessness. Refer to Appendix C.2 for more details. All methods use the same pre-trained model.
Results and Comparisons. The results are presented in Table DPO-H, which is trained only on the helpfulness dataset, achieves a reasonable helpfulness score but a low harmlessness score, averaging 60.99%. Conversely, DPO-S, trained only on the safety dataset, achieves high harmlessness score, but the helpfulness score drops significantly to 4.34%.
Training with a naive mix of the helpfulness and safety datasets tends bias the model toward learning more from the helpful data, resulting in even lower harmlessness scores, as shown by DPO. This aligns with previous findings that the mix ratio of helpfulness and harmlessness data difficult to control, and training often focuses on single perspective (Touvron et al.) 2023 Bai et al. (022). In comparison to these supervised methods, BFPO achieves the highest average harmlessness score of 77.16% and significantly improves the generative tasks score from 39.71 % to 95. 95.24%.
MORL,, the multi-ohjective reinforcement learning method, shows a relatively small improvement in the harmlessness score. We suspect the primary reason is that the reward scores of different responses provided by the public reward model are not sufficiently distinguishable, making it inefficient for the model to learn to generate good responses while avoiding bad ones. This highlights the need for training a reward model specific to the model being fine-tuned, which involves the costly human prompting (red teaming) and annotation process.
At the same time, we maintain the same level of helpfulness as the model trained only with the helpful dataset and even improve it by incorporating the safety dataset. Full results are in Appendix C.4
8

---

Published as a conference paper at ICLR 2025
Table 3: Efficiency comparison of our method to previous PPO-based safety alignment methods.
Table 4: Ablation study on the shifting factor and buffer training
Method
Data Size
Teaming
Red
Iteration
Alpaca
Savg.
Model
Helpfulness
Harmlessness
Alpaca
Disc.
Gen.
Savg.
Beaver
300K
3
1.00
71.80
Llama2
73.80
BFPO
13.33
59.09
95.24
77.16
BFPO
30K
13.33
77.16
BFPO, 0
12.76
59.09
92.87
75.98
BFPO, a buller
15.59
60.14
88.76
74.45
Comparison against Previous Safety Alignment Methods. We compare our method with two
successful open-source safety alignment methods: Beaver (Dai et al., 2024) and Llama2 (Touvron et al. 2023). We present statistics on the data size used for RLHF, the need for the red teaming process, and the number of training iterations in Table 3 Our method involves only supervised learning, whereas both Beaver and Llama2 employ reinforcement learning and require red teaming to identify harmful responses generated by the model being trained, which is computationally expensive. Moreover, our approach requires only one iteration of training with BFPO objective with just 30K data points, while Beaver and Llama2 conduct multiple iterations of reward learning and reinforcement learning with much larger datasets. Despite its efficiency, our method achieves comparable harmlessness score to Beaver and Llama2 while preserving the helpfulness score. These results indicate strong potential for our method to be applied in the future development of open-source models at a minimal cost.
4.3 IMPROVE PRE-ALIGNED MODELS WITH RED TEAMING DATA
In this section, we apply our method as an additional safety aligmment stage for existing pre-aligned models with a few thousand red teaming data. We compare our method with DPO (Rafailov et al. 2023), IPO (Azar et al. 2024). MORL (Ramé et al.) 2023) as in Section
Data Preparation. We first use 9K harmful prompts from the PKU-SafeRLHF dataset (Dai et al. 2024) and have the Zephyr-7b-beta Tunstall et al. (2023) model generate two responses for each prompt. We then use the Marmmennhh-Lamall 1-1--BBBhhh Chat (Mazeika et al. (024) classifier to determine whether the generated responses are harmful. For prompts that result in harmful responses, we use PairRM (Jiang et al. 2023b) to rank the responses in terms of helpfulness. This process results in 1.5K harmful prompts, responses, safety labels for each response, and pairwise helpfulness preferences.
Results. Table shows the results. Our method improves the harmlessness of the Zephyr-7b-hee model from 60. 99% to 73.90%, while preserving the helpfulness. The improvement in generative tasks is particularly significant, from 62.94% to 88. 79%..% The supervised methods, DPO and IPO, can also improve the harmlessness, but the improvement is not as substantial as with our method. When fine-tuning the model with MORL using specific prompts where the model initially struggled as in this experiment, the performance gain is still marginal, though larger than when using general data, in Table This aligns with the observation that using RL methods to improve safety requires a large amount of model-specific data, high-quality labels, and reward model specifically trained on these data to provide distinguishable scores. In contrast, BFPO achieves similar goals without the need for large amounts of helpfulness data mixed with red teaming data. Moreover, our overall pipeline of this experiment is efficient and automatic, requiring no human annotation. These results strongly indicate that our method can be effectively used in an additional safety alignment stage for existing chat models to improve harmlessness at minimal cost. Full results are in Appendix C.4
4.4 ABLATIONS
We validate the technical design, especially the shifting parameter a and the buffer training in Table In the BFPO = 0 experiment, we set the shift parameter a to O. The results indicate that, as illustrated in Section 3.4 the shift parameter a is useful in distinguishing unsafe data, and thus improves performance generative tasks in harmlessness slightly. In the BFPO w/o buffer experiment, we do not balance examples from the safety dataset and the helpful dataset, but instead mix the two datasets and randomly sample data from them. The lower harmlessness performance provides the evidence that buffered training helps mitigate the tension between helpfulness and harmlessness. Full results and detailed ablation are provided in Appendix Cnd Appendix C33

---

Published as a conference paper at ICLR 2025
Other hlyper-paramene like B1. are set based on either theoretical understanding the past work (Tang et al. (0244) and the fine-tuning strategy is orthogonal to the algorithm, while we further include their ablation in Appendix C.3
RELATED WORK
Alignment with Diverse Preferences. Traditional language model alignment methods (Christiano et al. 2017; Stiennon et al. 2020 Hendrycks et al.] 2021) typically use a single reward or unified preference model.However. recent work suggests that human preferences are diverse and cannot be adequately represented by a single reward model. To address this, Chakraborty et al. (2024) propose learning a mixture distribution for the reward using the EM algorithm, which they then apply in their MaxMin RLHF approach. Dong et al. )202)); Ramé et al. (2023); Wang et al. (002b) explore training multi-objective reward models for the alignment stage. These methods primarily focus on improving reward models for RL based alignment. The most closely related work of supervised alignment methods is by ZZhou et (2023), who, despite advocating for direct policy optimization, still rely on training reward models. In contrast, our approach completely eliminates the two-stage training process and directly integrates multiple preferences into the supervised optimization.
Safety Alignment. Aligning large language models (LLMs) with both helpfulness and harmlessness a specific case of addressing diverse preferences. To enhance safety, many researchers conduct additional safety data annotation alongside algorithm design. Touvron al. (2023) utilizes substantial safety data and combines safety and helpfulness rewards by utilizing the safety reward threshold function for the helpfulness reward. Dai et al. (2024); Ji et al. (2024) engage in red teaming to gather extensive safety data and frame safety alignment as a conditioned Markov Decision Process (MDP) problem. Mu al. (2024) propose rule-based reward complement for the common reward to improve the safety, which, although data-efficient, still requires human annotation and reward learning. In contrast, our method fully automated and efficient, eliminating the need for human intervention in the safety alignment process. On the other hand, Huang et al. (2024) propose generation awarre aligmment, which improves the safety over different generation configurations. With our focus on improving safety under specific configurations, this work can be a strong complement to ours.
amounts
Safety Evaluation. Supervised benchmarks, such OpenL.MM (Gao et al. 2023) and BigBench (Sri- vastava et al. 2023), include datasets related to various aspects of safety, such as toxicity, truthfulness, morality, and social bias. Adversarial attack research (Zou et al. 2023) and red teaming efforts (Ji ct al. 2024. Mazeika et al., 2024) provide valuable toxic prompts to assess if models can generate harmless content in response to these prompts. To identify if the output content contains harmful information, some studies (Bai et al. [022] Touvron et al. (002) rely on human annotators, while oth- employ AI models like GPT-4 (Wang et al. [2024a). Mazeika al. (2024) offer fine-tuned Llama2 models to as harmful content classifier, offering an efficient alternative to GPT-4 in model-based evaluation.
LIMITATIONS AND DISCUSSION
In this paper, we propose a novel supervised optimization method, Bi-Factorial Preference Optimiza- tion (BFPO), to balance the safety and helpfulness during the alignment of LLMs. We theoretically prove that this direct optimization equivalent to previous multi-objective reinforcement learning that combine safety and helpfulness rewards. With BFPO, we outperform existing methods in terms of safety and helpfulness in both fine-tuning the pre-trained LLMs and pre-aligned models. Our method is highly effective, significantly more computationally efficient, and does not require any human annotation or additional data collection.
Furthermore, our approach is versatile and does not rely on any specific property of harmlessness itself. This flexibility allows to be applied to improve various other potentially conflicting objectives in aligning LLMs. To achieve this, we only need characteristic dataset. We believe our proposed method can serve as a general framework for the transfer learning of aligned models. However, our method relies on specific label formats for helpfulness and safety may present a limitation when addressing different tasks. Moreover, extending our work to handle more objectives (beyond just two) is also promising direction for future research.
labels for the field-specific
10

---

Published as a conference paper at ICLR 2025
AKNOWLEDGEMENT
This work is supported by a KAUST CRG (URF/1/464--0100) and UKRI grant Turing AI Fellowship (EP/W002911/1). A. Bibi acknowledges the Google Gemma Academic Award 2024. We also thank the Royal Academy of Engineering.
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.00861 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455 PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204 05862, 2022.
Emily M Bender, Timnit Gebru, Angelina McMillan-Mann Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Souradip Chakraborty, Jiahao Qiu. Hui Yuan, Alec Koppel. Furong Huang, Dinesh Manocha, Amrit Bedi, and Mengdi Wang. Naxmin-RIHF Towards equitable alignment of large language models with diverse human preferences. In ICML 2024 Workshop on Models of Human Feedback for Al Alignment, 2024.
Chaudhry, M Rohrbach, Elhoseiny, T Ajanthan, P Dokania, P Torr, and M Ranzato. Con- tinual learning with tiny episodic memories. In ICML Workshop Multi-Task and Lifelong Reinforcement Learning, 2019.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXhv:2107.03374 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference Learning Representations, 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerim: Attribute conditioned sft as an (user-steerable) alternative to rlhf. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.

---

Published as a conference paper at ICLR 2025
Yann Dubois, Balázs Galambosi, Perey Liang, and Tatsunori B Hashimoto. Length alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL zenodo. wtwp://rrrgre.rrde///ooooo
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356 3369, 2020.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang. Guanting Chen, Xiao Bi, Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint 1rXivv2401.14100 2024.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generane dataset for implicit and adversarial hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch Critch, Jerry Li Li Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. In International Conference on Learning Representations, 2021.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.0129) 2021.
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-souree LLMs via exploiting generation. In The Twelfth International Conference on Learning Representations, 2024.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a human- prener-eeeeereee dataset. Advances in Neural Information Processing Systems, 36, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023b.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. https:://ugggiggaaee co/ www.aalilaaaaaaaaaaaaaaaaaaaalla /ttp//aannnnnaeeaeraaaaaa reward bench 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloving models. httpt::// thub. wwt.:/cco.otoooooooooooo http:///tacccoollaaaaaaaaaaaaaaa
2023.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13470- 13479, 2021.
12

---

Published as a conference paper at ICLR 2025
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang. Norman Mu, Elham Sakhaee, Nathamiel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.044 2024.
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. 2024.
David Nadeau, Mike Kroutikov, Karen McNeil, and Simon Baribeau. Benchmarking llama2, mistral, gemma and gpt for factuality, toxicity, bias and propensity for hallucinations. arXiv preprint arXiv:2404.09783 2024.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953-1967, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020.
emnlp-main.1 154.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bhq: A hand-built bias benchmark for question answering. In 60th Annual Meeting of the Association for Computational Linguistics, ACI. 2022, pp. 2086- 2105. Association for Computational Linguistics (ACL), 2022.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Bean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. In NeurIPS, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106. 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv.1707006347, 2017.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620-7972) 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Bouriga-Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
13

---

Published as a conference paper at ICLR 2025
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. Alert: A comprehensive benchmark for assessing large language models' safety through red teaming, arXiv preprint arXiv:2404.055 000.206644...66, 2024.
Judith Jarvis Thomson. Killing, letting die, and the trolley problem. In Death, Dying and the Ending of Life, Volumes I and II, pp-
Routledge, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.1694 2023.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36:75993-76005, 2023.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. Advances in Neural Information Processing Systems, 36, 2024a.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024b.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expprts. arXiv preprint arXiv:2406. 12845, 2024c.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint 1rXiv:2112.043355
2021.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Wordcraft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces, pp. 841-852, 2022.
Wenxuan Zhang, Paul Janson, Rahaf Aljundi, and Mohamed Elhoseiny. Overcoming generic knowledge loss with selective parameter update. In IEEE/ CVF Computer Vision and Pattern Recognition Conference, 2024.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf large language models part Ppo. arXiv preprint N.XV:22007.0494.. 2023.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preferencce-itt-all alignment: Multi-objective direct preference optimization. arXiv preprint ArXiv:2310.03700 2023.
Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307. /5043, 2023.
14

---

Published as a conference paper at ICLR 2025
A ALGORITHM
Algorithm shows the BFPO algorithm. As mentioned in Section in practice, refer datasets related to safety topics, collected through red teaming, as safety datasets. A typical safety dataset will contain safety label (t)) which is the binary label indicating whether the response y is harmful, as well as the preference label [y > y²) in terms of helpfulness. If a certain safety dataset does not provide helpfulness labels, we can use the ranking models, like PairRM (Jiang et al. 2023b), as discussed in Section 4.3 to generate the pairwise helpfulness labels. We refer to datasets designed to improve the helpfulness of the model as helpfulness datasets. A typical helpfulness dataset will contain the helpfulness preference labels (y > y'). Since most helpfulness data undergoes human filtering, the responses are usually safe. Therefore, we assign the safety label Isate = to all responses in the helpfulness dataset.
We further require a pre-trained language model Tet. the total number of optimization steps T, the penalty coefficient for the KL divergence term, and the shifting parameter a. We also need to specify the layers to be unfrozen for the policy optimization, denoted as 0'.
At the beginning of the algorithm, we initialize the policy Ty with the pre- traitreined language model Tees and unfreeze the selected layers H' (line 1-2). In each gradient step, we first sample a batch from the safety dataset D2 and batch from the helpful dataset D2. (line 4) of the same size. We then compute the loss of both batches according to Equation (T3) (line 6-8). We accumulate the gradients of the loss for the both batches and update the policy my (line 10). This process is repeated until the total number of optimization steps I" is reached.
Algorithm 1 BFPO Algorithm
Require: Safety dataset D {(x,, y/u yh (y Taae (y)")) and helpful dataset Da. (xx,y" ))
Require: Total number of optimization steps T., Pre-trained language model Ties and unfrozen layer 0', a.
Initialize my < Tee
2: Only unfreeze selected layers 6'
3: while << do
4: Sample batch B, D. BB. Dr-
5: for batch = Bg, B3, do
6:
Compute hyy" ,,yhh)) with Equation 5
7: Compute with Equation [14)
Lafe (y) for the helpful dataset.
8:
Compute and accumulate gradients w.r. Equation end for
15
9:
Update me.
11: end while
B PROOF
B. NOTATION
Table Notations
Notation
Meaning
M.y
Two responses generated independently by the policy. Ground-truth helpfulness preference of y being preferred to knowing context s Ground-truth safety of knowing the context r
Phop
Pure
Anapp(1)
3'(x))
Binary label of helpfulness preference of being preferred to y knowing the context Binary label of safety of knowing the context æ globally preferred and dispreferred responses knowing the context æ preferred and dispreferred responses in terms of helpfulenss knowing the context
Lune (M/22)
n
y New A
E.
Expected safety of response y given the context
15

---

Published as a conference paper at ICLR 2025
Table summarizes the notations used in this paper based on Rafailov al. (2023): Azar et al. (2024).
In the appendix, we will employ the ordering-free free notation systemy, y for the proof. Specifically, we express the transformation equations from y"u. yd y, y as:
saaee (y how (x) neeip (y > 3/ [x)) sare (y]z) x)) + Ieip > y sate (1/(ax)
satee (y) hul [x)) = 11)p(( > 3/(1))1111)))()) sste (2/ (z) + 1bctp(/ > y/ x)) satte (3) (
For brevity and clarity, we further adopt the notation y to represent y/ı. This simplification does not sacrifice generality, the dependence of y on remains consistent across all the equations.
B.2 PROOF OF THEOREM 3.1
We begin by restating Theorem 3.1 with the notation system y. y/ Note that the different notation systems will only affect the presentation of the reward function g and the labeling function gr, which will discuss in the proof.
Theorem B.1. Let 0 be real number, Te, Thf be two policy. Then
(rd(()) exp (T g(y))
π <'(v) E.Es Trey (s) cxp (x)()=g((s))
(16)
is an optimal solution to the optimization problem
max
Eyynnn [g(v)= - TKL
(17)
T5g
and x² x²) is the optimal unique solution of
g(v) - s(v0))²
min E m.
h hx.
(18)
where
To [T)
ha (v.3/) = log
(19)
TT (nr((())
To establish optimal solution, we follow Azar et al.
2024 to leverage the following lemma.
Lemma B.2 (Rafailov et al. (2023). Azar et al.
2024)).
Let
C, (6) Eeeslf(s)) (
where E S and S is finite set, e R is a function mapping elements of S real numbers, E (SS) is a probability distribution over S, E A(S) is a fixed reference distribution, and Rt a strictly positive number. Then the argmax problem with the regularized criterion
arg max (+) BEA(S)
has an optimal solution 8*, where
(1)) exp(T=1((s))
8 (8) EveS ((8)) exp(r-
Vs VsES
To establish the uniqueness the solution in Equation 166 for the optimization problem in Equa- tion (18,, we leverage the following lemma.
Lemma B.3 (Theorem 2 in Azar et al. (2024)). Let
(w. g(v) g(v
((TA) = Ey.g(~ma())
(20)
then min., L(TA) has unique optimal solution म* expressed in Equation or global minima exist.
16 and no other local
16

---

Published as a conference paper at ICLR 2025
Proof. Let J Supp( (yi..... , where n [J], and II be the set of policies with support set J. It is straightforward that min [(T) = [(x)) = 0, thus π is a global optimal solution. We now prove the uniqueness of this optimal solution by the parameteriiztioo trick.
We parameterize via vectors of logits s R of n, i.e., 8; log(T((4)), Set (y()) Eis exp(s;) for y = E J and (v) = 0 otherwise. Specially, let s* be the vector of logits corresponding to T², we have x kg*
exp(a.)
We first prove that 8* is the global optimal solution to the optimization problem
((s) = ((T,) Ey..' 1T. (v. v/) gly) g(v')
is obvious that L(s*) O, thus it is local minimum. By expanding the square term, we have [(s) Byys''T -it
g(v) s(vr) (s(v) -sy'') - -log
(Teer(y))
TTee(
Σ (1)TT((1)) [(s(()) + Ci ((s((((())))))))))))(())) s(t)) + CG
M.y'EJ
where Ci C2 form, and hence is convex. Thus, all local minima are global minima.
terms independent of s. The above equation is positive semidefinite quadratic
Now we prove that ..* is the unique global minima C(s). Since स., is surjective continuous mapping from s to æ, then every local minima π to [(T) corresponds to a set of that minimizes C(s). The uniquess of s* will deduce that T is the unique optimal solution concludes the proof. Consider s' = s* + As, where the only r is the radius and As is the direction under the polar coordinate. The only direction that not increase C(s") away from 0 is .........
Equation (18) and
(Boyd & Vandenberghe (2004). Chap. 3). However, we have
exp(s: + r .1) trie (8i) Σ cxp(s; s, + tr. )) Σ exp(s;) n.- (8i), Vi [m].
exp(s;)
This indicates that ... is the unique global minima to ((T,') and thus xt is the unique optimal solution to Equation (18).
Now we provide the proof of Theorem 3.1 most of which follows Azar et al. 2024).
Proof. Let S be the set of all possible token combinations with fixed token length, then finite.
Let f(s) = (P (s) + E.) (Phiip (s > т) 2), 8(s) = (s)s) and r7(s) = (s). All the conditions
in the Lemma B.2 are satisfied. Thus, Equation (16) is a solution to the optimization problem in Equation
Now we prove Equation (16) is also solution to the optimization problem Equation Equation T6) in the Equation 18), have
18
Plug
(()))rrl'))) log x (y/) (Fr()
p(y)) exp (T 1g)))
exp (r-)
h (v.v') v') log
g(('))
which validates Equation
(66 is solution to the optimization problem Equation 18
Finally, Lemma B.3 indicates Equation (16) is the unique solution Equation (8) This concludes
the proof.
The above proof holds for any order of y, y' since the equation in Equation 19 is skew-symmetric, i.e,
(n v/) g(v) 9(v')
hn v) g(v/) g(v)
This allows us to freely arrange the order of y, y/ Equation [8] without loss of generality, Therefore, Equation (18) can be written
min Ey. (x)()) Max(o (e hi)) g(y 2hhi))²
7
17

---

Published as a conference paper at ICLR 2025
where
S w
if Incp(y > v11z) = 1,
otherwise,
and
y
if Thelp(y 1/1z)) =
y
otherwise.
With this reordering, the theorem reduces to Theorem
3.1
B.3
PROOF OF THEOREM
In this section, we prove the Theorem 3.2 We begin by rewriting the formula in Equation 12 into function y, y'"
gr B3 (Bi (Luse hop v/) + Lute Hnsply' y))
(21)
(Iute(y) Inaply/ y) + Lutely/ Hmpp(y > s/ Ba) (2hmaply > 5²) (0)--1)) 1).
Here, hy > y') determines whether is the win response or lose response. In other words, Trate Tu = saaee (y) Thepp (y > s/) + saae (y/) Whepp(y > y),
and the same applies Taaee (. To enable the reordering of the variables, further multiply the formula by 2/mclp( (y > y/) 1, since (y, v²) = -ha (y', y) By organizing the terms, we have gr(u.u v') =Bi B3 helply I (v) (B, B3 Bheply v/) Tase (1')
B, By Ba
3/) B2B3
We first establish the equivalence of the two optimization problems in Equation (22) and Equation (23 under the specific choice of constants, and then provide the general relation of constants for the equivalence.
Here, we use the following constants:
1
A E., A2
Bi =3 B2 = 0, B3 =
Theorem B.4. The optimization problem
min Bann (twr (v. I (Piate (1) Pheip (1)) (VVate (v'), Phesl (3'') (1)))
(22)
where g(v)= = (Pige (v) + Fs. F2s) (Pics (y x 1t) + 2)), is equivalent to the optimization problem
min Eepp.v. ()) I Beroouttti (nr(u. v/) gr 2r(v.322 21
(23)
where
gr( v/)
(y > Innee (v) + Ineep > MI (2r)) + Imao (v)
= -Lufe Lnoga (v/)
Here, I~ Bernoulli denotes the Bernouli variables T (v) and (y))
Proof. The two minimization problems are both over To. so we only need to focus on the terms that involve ne. Specifically, the first term and the cross term after expanding the square expression the two minimization problems. The first term is the same. Here we prove the cross term is also the same.
Let ny log(:(y)), F -log(Tee(())) then can write
h (y, v') Ty Tyg + n
R 4/'
R
18

---

Published as a conference paper at ICLR 2025
Let Ph (v) = Phalp( (y > T) and Pa (t). The cross term of Equation 22 can be written as
[hr (v. s/) (o(patc()) (y - x)) - g( (v'), Ph > x)))
= -Re-paid ~x [TTH my + To. = (g(px(v) Ph (v)) p(xo(i))..m((())
(24)
-Exapeer (Ty n. (g(P((V), PM))) By (p (1/ (z)
+En-pay' (- + m (Byna [6 (v), Pa (v)) g (v'): pm (v))))
The third equality is by the independence of and y/' By the change of notation, the second term of the last line can be written as
Exzp.y' 1 + mp² (By [g(p. (v). Ph Px))) - (ol.lt))..
(25)
= -ar-pyy [(-Ty Th) (Byr~a [v( (n)), (v')) (v) pT)
Then Equation can be written as
24
[(TT / 2 (g (v) Ph (v)) - Eny-a [o(p_ (u
(26)
Now we plug in gl.. (w), pa (v))
+ E.) +2) and use the fact By ~~ pp (1' > m) =
Equation 26 can be expanded as
(p-(v) (P) ; Eyy (ps(l')) E.) b
24
(Ty my 2
(Ty myg -2 (p.(v) (p) 2/ -21is
-Emppyyrs
[ - m.y) (2ps (1) pm (v) + 28.pa (w) + +p((y) - -3E.)]
(27)
The cross term of Equation 23 can be written
Ex-ppyy ~ Eemoullllli [ (ys, y ygr (v. y')l
(28)
= -En-ppp ~ur Bernoulllli [(TTY Fy + my R
Ty R
)rr (v. w')
Now we plug in 91 nelp(( (y s/ sate (v) + Tnoppy 1 sate (v') sate(()) (1) 3 sat')
28
E.p-p.as Ei Bemouullli (mTy Ty + n. #1 (Ihep(y) (y) > sate
1no > 3/ Ssato (2/)
1
1
(aato())
Lsatc( (z/))
2
( mB (Ihep(y ssacs)))
= -Eeppey ~s Bemoouu
+ Tnep(U > sf) T (v') + (aate(()
1
3
Jsaee Jsate((/))) (v/))
2
2
+ Bew-pyy Nn E, Berooull (-Tyy Fy nT)) (Inoppy (y x I
+ Thepp(y > y²) Tsate + 2 sate (ateeyy)
2 Jaate (1/)))
With the similar change of notation as Equation 25 as well as the fact that 1 - Inepp (y y > y²) Ineipyy > y), the last line can be written
Ex-pa ~ I (-TTy/ Tyy + TB)
) (Anelp(y > saatsa
1 [wate())) (mte((/)))
3
+ Thepp (y > 5/ JT)TTate (v')
Exwepari MI E.I----rBlllll (-TTy y + m. (Thetp > y) Ts))
1
+ Ineep(s/ > y) Taate (v) + 2 (
2
(-TTy + Ty ((1 Thelp(y > 2/ (sate())))
1
+ (1 -1--nespC > of I (V) + -sate(()
Ineep(V
Jsate (v))
2
2
19

---

Published as a conference paper at ICLR 2025
Then further expand Equation (28
28
Exrpp.y ~r Bemoolllli
(Ty Ty [nepp( > s')
+ lbclp > 3/ ssate(() + ssato - sate (v'))
3
+ Earep.yss Er Bemooolllli
(-Try + Ty ((1 (ne-pee > -y ( sste
(1 Inclp(y y y Lsate (1) + 2 Lsatc I [satc(()))
3
E Emp Br Bewnoullllf
(TTy "y R) (2 (2)))2222m)))) y Asate
+ 2 (y > =/ I (1') Isal((y) 3lsatc())))
(29)
Taking the expectation over y' and the Bernoulli variables, we have
28
E.pppr (Mg ( (v))()) 2F, + - -3F.)
(30)
This equation is the same as Equation 27 are equivalent!
which ends the proof that Equation 22 and Equation 22
As discussed in Appendix B.2
freely change the order of and y in Equation 22 and
Equation (3 Thus, the proof of Theorem B.4 also applies to Theorem 3.2
B.4 RELATION OF THE CONSTANTS
In this section, derive more general form of Theorem B.4 where, with specific relations between
the constants in and the optimization problem in Equation 22 is equivalent to the optimization problem in Equation 23
We restate and 9r here with the notations used in the Appendix for convenience.
g = (pa(y) + Ar)(pn A2),
and
gr(u.1') =BB B3 BB) helply I') (v) (Bi nn 5/ Tate (y')
B, -B, - Bzlsate 2B2 3/) B2B3
As discussed in the proof of Theorem B.4 we only need to find the relationship such that the cross terms of the two optimization problems identical. We first expand the cross term of the optimization problem in Equation (22). As in Equation (26). it can be written as
24
Ex-paye [
T (g(ps (v), Pr. (v)) - Ey
(31)
Using the same strategy obtaining Equation we have
28
Expopyye (TTy (2Bs 1)ps
2BB 1)E, + 2BB)p% + 2Bpp. - 2B, (.1,B,E,,-BBBBB.. -2BBBa)
(32)
Aligning the coefficients of each term in Equation and Equation 32). we derive the following set of equations:
BB(B, - 1) = 1, E, + 2B B3 A
(33)
B3 A2-
Solving these equations gives us the specific forms of and gr- Here B, is a shifting value that we define align with intuition. B3 is scaling factor that is related the penalty T.
20

---

Published as a conference paper at ICLR 2025
B.5 DISCUSSION OF THE PROPERTY OF 91
In this section, we discuss the two beneficial properties of 91 that we proposed in Section 3.2
Skew-Symmetric Property. First, we examine the skew-symmetrie property of gr. When combined with the skew-symmetric property of h, this implies:
(v. v/) - v)) = (v' - 91 (v''
This means that for the same data point, regardless of the order of y and y', we are always driving he (v, y²) to the same value. In contrast, in IPO (Azar et al. (2024), different orders will push (y, y²) to different values, i.c., they form two different optimization problems:
(h D¹) - 9r. and (1 v))
Their final optimization problem. optimizes both. However, this point neither the optimal solution of the first problem nor the second problem.
5/) - 91 (g. 3/))² tries to find a middle point of h that
Shifting Property. Second, we discuss the shifting properties of gr- Since Theorem 3.2 holds
based on the equality of Equation (30) and Equation (27). and all the operations to derive these two equations are valid under linear transformations Paae: Phelp and safe, Ineip respectively. It implies that Theorem
3.2 also holds under the same linear transformations of Plate: Puelp and safee help-
This property is useful when we want to manually design the values of gr, as shown in Figure
C EXPERIMENT
C.I DETAILS OF HARMLESSNESS BENCHMARK
The following are the details the datasets used in the benchmark:
CrowS-Pairs: A challenge set for evaluating the tendency of language models (LMs) to generate biased outputs. We evaluate the English subset and implementation by LM Evaluation Harness and report the Percentage of Stereotype metric.
Bias Benchmark for QA (BBQ): Measures social biases in the output of multiple-choice question-ant question-answeringg models. We use the Lite version and Iimpleenntttion by BigBench and report the Accuracy metric.
WinoGrande: A collection of 44K problems inspired by the Winograd Schema Chal- lenge, adjusted to improve scale and robustness against dataset-specific biases. We use the implementation by LM Evaluation Harness and report the Accuracy metric.
ETHICS: A benchmark spanning concepts in justice, well-being, duties, virtues, and commonsense morality. We use the implementation by LM Evaluation Harness and report the Accuracy for all sub-tasks.
BigBench Moral Permissibility: Tests whether ultra-large language models can read a short story where moral scenario is presented and answer the question, "Is it morally permissible to do X?" in a manner similar to humans. We use the implementation by BigBench and report the Accuracy metric.
BigBench Simple Ethics Questions: Assesses language model's 's responses to hypothetical, consequential, political, and social questions. We use the implementation by BigBench and report the Accuracy metric.
ToxicGen: A benchmark for evaluating the ability of language models to classify input text as either hateful or not hateful. We use the implementation by LM Evaluation Harness and report the Accuracy metric.
BigBench HHH Alignment: Evaluates language models alignment, pragmatically broken down into the categories helpfulness, honesty/accuracy, harmlessness, and other aspects. We use the implleennttion by Big and report the Accuracy metric.
21

---

Published as a conference paper at ICLR 2025
AdyBench contains harmful prompts. We the prompts provided here and generation implementation by LM Evaluation Harness. We report the percentage of harmless responses measured by HarmBench- 1-mma-2-13--lss
CaalToxxcityPPromphe A benchmark for evaluating the ability of language models to continue a prompt in a non-toxic way. We use the implementation by LM Evaluation Harness report the percentage of harmless responses measured by HarmBench-Llama-2 13b-els model.
A benchmark to assess the safety of LLMs through red teaming methodologies. We the prompts provided here and generation implementation by LM Evaluation Harness. We report the percentage of harmless responses measured by Barmmench-LLama-2-1 model.
ALERT:
ALERT Adversarial: A benchmark to assess the safety of LLMs through red teaming methodologies with adversarial prompts. We use the prompts provided here and generation implementation by LM Evaluation Harness. We report the percentage of harmless responses measured by HarmBench Saancch-Llama-21 1lama-2-133-chs model.
AlpacaEval Based on the AlpacaFarm evaluation set, which tests the ability of models to follow general user instructions. We employ the official implementation report the LC Win Rate.
C.2 DETAILS OF BASELINES
The following are the details the methods that align LLMs for multiple objectives.
Llama2 (Tovrron et al. 2023) trains the safety reward Tsafe and the helpfulness reward Thelp separately, and dehnes the global reward g combination of these rewards, i.e.,
ī(vlx) (ylle) IS SAFFEAFFFT)) or (u) < 0.15,
('hetp (v)=z) otherwise,
( WHITEN(LOGII(G(VW)))
Here IS SAFETY (z) indicates whether prompts are tagged as unsafe in their dataset, and the 0.15 threshold is chosen to filter unsafe responses according the evaluation Meta Safety test set. Whitening the final linear scores is to increase stability. The global reward is used in the RLHF objective Equation (3).
Beaver (Dai et al. 2024) trains the safety reward Fsafe and the helpfulness reward Thelp separately, and defines the final RLHF objective as the dual optimization problem of the conditional RLHE, obtained by Lagrangian dual transformation, i.e.,
min max R.yDDr (-Thelp (z) (rate(y)ar) d)). X20
where À 2 is the Lagrange multiplier. In practice, the model parameter A and the Lagrange multiplier are updated iteratively.
RBR (Mu al. (2024) requires separate reward models, Tor....; for each objective, and propose to learn the weight for each objective, i.e.,
(0yl>) Σ Aere(y/a))
where A; are learnable parameters. The global reward is used in the RLHF objective in Equation (3)
SteerLM DDong et al. 2023 trains models to generate response according to specific They first train model to predict the score for each objective in dataset. Supervised fine-tuning is performed to maximize the probability of generating responses conditioned on the reward vector and the prompt, iee.
reward vector r (r1, r2, ....... rz).
........7
max E (Z log Po (y)22, r).
22

---

Published as a conference paper at ICLR 2025
MORL (Ramé al., 2023) trains reward models for each objective separately, and defines the global reward g as a combination of rewards, i.e.,
( Σ Amr())))
The global reward is used in the RLHF objective in Equation 30.
ArmoRM (Wang al. 2024b) applies the same training strategy MORL, but uses single publicly available reward model, ArmoRM--Llaa3388--00. (Wang et al. 2024c). to provide the reward scores for all objectives.
MODPO (Zhou et al. 2023 trains margin reward models ri, 1,....K for each objective separately, and performs supervised fine-tuning with the objective,
max (//r"/"""/))>
L(Tlog (Tog
To(y"(()) Teel (y log
T0(32( Trer y
log
w "" -
Wk
where w/ is the weight for the objective k; w- the weight vector for other objectives, and r- is the reward vector for all other objectives than k. This fine-tuning is performed for each objective.
MinMaxRLHF (Chakraborty et al. 2024) addresses the scenario where different annotators h may have preferences for different objectives. The algorithm uses the EM algorithm to learn the distribution of rewards for multiple objectives. In the E step, they find the certain objective i that each human annotator relates to, i.e.
exp(ro, (I,
I, = arg max II Uaaya A exp(ro (x. (1,
where ræ, is the reward model for the objective In the M step, each reward model is updated by the reward learning objective in Equation ① with the data assigned to objective i, iee., the dataset Dj = {(x, y, v. h). In i). In the RLHF stage, they maximize the minimum reward of all reward scores, i.e..
E EayDrr min re. (z, y) 7KL [To [=](
Among these methods, MODPO is highly inefficient since it requires separate RLHF for each objective. Other methods typically use linear combination of reward scores for multiple objectives or one reward as a threshold for others. For the combination of thresholding, the global function can be approximated by the multiplication of rewards for each objective when the reward scores are on the same scale. Maximizing the multiplication of rewards has the same effect maximizing the minimum reward. Therefore, we hypothesize that the global reward should be bilinear combination of the reward scores as in Equation (1O). As such, in the experiment section, we select MORL as representative for this line of approach.
C.3 ADDITIONAL. ABLATION STUDIES
We include additional ablation studies of other hyper-parameters in algorithms. Full results are in Table 10
LoRA finetuning. We follow the setting of Section 4.2 and conduct additional experiments to compare our method and the best performed baseline, IPO, with LoRA fine-tuning. During the training, we apply the same training hyper-parameee epochs, beta, and so on. The results are in Table Results show that BFPO consistently outperformed the baselines when training with LoRA. However, observed that LoRA training required additional hyperparameter tuning, which posed challenges due to the limited time. Consequently, both methods achieved lower overall performance and worse balance compared selective fine-tuning.
both algorithms, like learning rate, training
23

---

Published as a conference paper at ICLR 2025
LoRA Fine-tuning
Selective Fine-tuning
Helpfulness
Harmlessness
Helpfulness
Harmlessness
Alpaca
Disc.
Gen.
Savg.
Alpaca
Disc.
Gen.
Savg.
IPO
6.14
58.05
93.97
76.1
13.15
58.41
89.76 95.24
74.09
BFPO
7.77
64.36
94.73
79.54
3.33
59.09
77.16
Table 6: Comparison of Helpfulness and Harmlessness Metrics with LoRA finetuning
Table 7: Ablation Study of a in Equation (15)
Table 8: Ablation Study of T in Equation (15
Helpfulness
Harmlessness
Helpfulness
Harmlessness
Alpaca
Disc.
Gen.
Savg.
Alpaca
Disc.
Gen.
Savg.
0.1
13.61
59.81
87.39
73.60
0.01
13.33
59.09
95.24
77.16
0.3
14.06
60.31
91.73
76.02
0.1
6.4
55.44
81.45
68.44
0.5
13.33
59.09
95.24
77.16
6.53
54.01
78.14
0.7
9.01
57.34
96.28
76.81
1.0
6.74
54.10
77.52
65.81
0.9
7.21
56.44
96.66
76.55
The hyperparameter a. The hyperparameter controls the label values (represent the difference of the preference of pair of response) of the four cases in Figure 3 To ensure the desired behavior, that helpful-safe responses are preferred over helpless safe ones (case 2 F a positive value) and that helpful-unsafe responses are not preferred over helpless-unsafe ones (Case 3 in Figure yields a negative value), we constrain a E (0, 1). When a 0.5, the label values for the four cases 1, 0.5, -0.5, 11, where the absolute label values symmetric for positive and negative pairs. As a increases, the absolute label values in case 1,2 in Figure 3 decrease, and the absolute label values in case 3,4 in Figure increase. In other words, positive pairs will have smaller differences and negative pairs will have larger differences.
In the ablation study, follow the experiment of Section 4.2 with values of 0.1, 0.3. 00.5, 0.7, 0.9 to systematically explore its effects. The results in Table show that higher a values reduce distinctions between positive pairs, particularly helpful-safe score. However, it increases distinctions between negative pairs, especially helpful-unsan vs. non- helpful-safe, resulting in improved harmlessness, particularly in generative tasks.
non-helpful-safe, leading lower helpfulness
The hyperparameter The hyperparameter т is the coefficient of the KL term Equation 7 which prevents the policy from deviating from the reference policy. In practice, it is important to note that is more related the training and convergence (as shown in Figure rather than being core component to balance the safety and helpfulness.
In our experiments, we follow the settings from Tunstall al. (2023). where 0.01 is used. This value is applied consistently across all baselines to ensure fair comparison. For the ablation study, we adopt values inspired by TTang et al. (2024). specifically т 0.01, 0.1, 0.5, 1.0. The results in Table @ indicate that performance can vary significantly with different values. With different other training hyperparameters, like learning rate, training iterations also need be carefully chosen.
following values B3 = 2, 1, both B1, B3 must be positive. For this ablation study, we explore the Given the constraint B3 - 1) = as in Equation (33). the corresponding values of B, are determined for each B3. Additionally. B3 is adjusted to balance the cases described in Figure 3 (Case 2 and Case 3). The experiment results are in Table
Ablation on B1, B2, B3.
When B3 smaller, the label differences for cases 1,2 and 3,4 in Figure 3 become less pronounced. For example, in Cases 1 and 2, the pairs (helpful-safe, non-helpfullunsafe)) and (helpful-safe, non- helpful-safe) have smaller differences in their label values. This means there is less distinction in whether the non-helpful response is safe not. As result, the model shows slightly worse performance in helpfulness but performs better in safety. When B, is larger, the label differences for the aforementioned two cases become more distinct, and the label value for (helpful-safe non-
24

---

Published as a conference paper at ICLR 2025
helpful-unsafe) becomes significantly higher. This leads the model prioritize safety strongly, which results in improved safety performance but a sacrifice in helpfulness.
To conclude, larger B3 values emphasize safety the expense of helpfulness, while proper values allow for more balanced performance across both objectives.
Table 9;
Ablation study for B1, B2. B3 in Equation (9)
Ba
B1
B2
Values for four in Figure
Helpfulness Alpaca
Harmlessness
Disc.
Gen.
Savg.
1.5
-0.25
2.5.0.5,-0.5.0.225 1.5.0.5.-0.5-1155
9.00
58.67
95.47
77.07
11.36
60.28
95.12
77.70
0.5
-1
1,0.5,-0.5,-1
13.33
59.09
95.24
77.16
0.25
5
-2
0.75, 0.5, -0.5, -0.75
13.15
59.63
94.25
76.94
C.4
FULL EXPERIMENT RESULTS
Here are the details each open-sourced models:
Zephyr: http:///hhhgggggh/c/////////////
Juanako: http:////aaggggoooo.oooo/o////ar
Mistral:
Histraa--3--tattxxttt-v122 http:://hugghnffaccccc//////ss
Beaver: heto:///eeessooooo...r////AA///r
Llama3:
http::/hugginnffccc.c/meee.
Table 10 shows the full results of all experiments throughout the paper. Here are the details of the data used our model and the baselines.
We use 4 Nvidia A100 GPUs for each experiment, and the training time for each experiment is around hours for SFT and hours for BFPO. For the experiments with red teaming data, we use 1.5K data collected described in Section and only performs the BFPO stage. The training time for this experiment is around 10 minutes with 4 Nvidia A100 GPUs.
25

---

Published as a conference paper at ICLR 2025
Saley
60.99
08.59
71.83
71.80
73.00
77.24
76.66
49.00
74.00
77.16
06.81
06:07
61.00
73:00
75.00
74.45
76.01
79.54
73.00
70:00
77.16
76.81
76.55
77.16
68.44
66.07
65.81
7740
77.70
77.16
76.34
Ganero
400
62.9
79.22
91.23
96.21
56.91
39.7 71
50.30 79
55.34
72.82
65.02
92.89
94.73
$7.99
51.73
95.24
56.66
81.45
78 14
77.52
95.47
25.12
05.24
04.25
Doci-
59.05
70.46
63.00
52.38
51.58
59.05
56.12
58.35
58
58.54
5909
59.32
58.57
59.02
59.09
6014
64.36
5981
60.31
59
57.34
56.46
5909
55.44
5401
34.00
5867
60.28
5909
5965
ago
ALLXI
Ad
sarial
66.68
72.79
77.21
9..00
98.32
99988
33.46
08.00
96.42
24.55
69.
04.12
98.02
88.8
92.00
96.42
9930
96.42
08.
86
10
07.85
26:30
06.42
04.62
ALEIC
79.08
80.70
00.44
91 33.83
98.62
99 91
38 38.51
96 00:00
98 96
89.22
87 77.48
82.13
95.37
08.23
96.96
95.61
08. 98.56
99.81
56.56
91 01:01
91 S1
98.23
99.36
97.85
80.00
96
91
Adv- Bench
20.19
10.50
65 5B
93,20
100.00
100.00
4.23
58.65
98.65
41.35
21.15
96.15
98
82.31
93.16
98
6865
99
98.55
63.55
50.00
48008
99.62
22.62
98665
OW 26
Keal-
65.52
85 5.000
83.24
05.07
87.91
85.82
57.4
82.00
86.41
57.32
87.1
87.66
57.80
85.22
$3.49
8132
87.22
3657
87.09
57.32
8482
$150
25.40
37.32
45.00
49.00
47/06
43.44
44.34
45.55
45.70
44.5 10.80
4525
45.20
45.00
45.55
45.70
45.25
45.0 15.00
45.20
45.20
45235
44.50
46.00
4535
4389
44.34
4380
45.70
4525
4525
Jongen
45.22
60.00
11
36. 17
51.00
45.21
17-45
51.06
48
56.15
45-43
46.26
45.50
49.30
06.06
53440
52
16
09:47
54.15
45
43:40
43.40
54.57
53.22
56.15
53.00
Table 10: Full results of all experiments
Sinple
Eit-
Quis-
E0005
33,00
82 20
53.91
45 62.22
24.35
38 18:26
39 13
37. 39
13
56-52
36 26.52
33.04
80.43
66.35
44.35
50.43
39 13
21,24
39.13
22 61
17. 19
26.26
46.09
39.13
45.43
Moral
Per
51.00
49.00
47.95
42.06
47.37
48.54
46.2
51.17
47.37
47.37
50.88
50.58
51.17
40.
47.08
47.95
47.06
47.37
47.08
47.37
47.37
48.54
49.12
46.75
47.57
47.37
47.66
Eatiis Vinue
91 10
89
90.37
61 61
72 10
80 29.33
86.71
89
91
90.41
91 48
91 36
10.
29
91 28
90.29
90 19.19
90.4
25
90.1
88 50
86 06.37
57
10-33
90.39
90.41
60.33
Aahics
Itr--
73.59
73.79
66.72
56.01
57.97
60.00
57.15
67.03
70.36
73.54
73.02
73.46
60.64
68.53
66.7
68
90.36
68.1
70:36
63.66
60
60.5
7061
70136
6804
Bithics
og!
56.08
00.25
61.88
50,00
50.85
59.20
57
55
51.97
57-48
57-62
56
57.20
52.28
56.56
22.25
58:40
55.73
55 55.25
51.97
50.44
91
51
$1.17
50
11
54.50
Esttics
69.71
76.41
71.93
6161
50.00
70.38
$5.29
71.01
66.35
59.13
69 30
60.16
68.00
59.87
73.00
68.16
63.00
59.13
31:48
5913
57.10
$4.29
55.36
52.99
65.27
59.13
63.00
Wilograndettiss
CM
68.37
13.96
73.46
59.03
56.14
61.22
56.47
63.03
65.25
08.03
66.07
62.50
68.13
72.20
02.52
62.50
63.25
6900
63.78
62.08
62.08
70.35
6525
62.80
22.05 22.08
77. 43
73 30
65.35
06.00 .6
22.5 0.38
71:00
74.27
74 16606
74.98
72.45
72.77
71.0 98
74.41
74.00
24.77
24.82
74.88
25.06
73:30
74.98
74.83
74.74
74.00
74
724
75
BBDQ
39.00
86.00
6134
31.37
12.99
39:50
43.68
42.44
4525
20.89
33:47
44.44
4641
4291
13:11
45255
49.4
4525
4064
36778
39.34
4525
4306
Clows
62.02
64 88
36.23
63.90 98
65 59
66 25
65.77
84
61.96
61.66 66
0.
65.95
66 25
65 59
63.77
63.83
65.77
66.01
66.31
65.05
65.65
65.77
65:50
63.2
63
65
Alpoca
10.99
2.58
14.22
1.00
7.00
14.71
13 16
13333
13.74
12.56
12.76
7.27
13611
11.06
13.33
7:1
6.4
6:53
6.74
9.m0
11:36
13:33
Misaal + DPO Narve mis Data
Misanl IIo- Naise Data
Misai++ DPO Saldty Data
Nuvee mis
Misml++ DFPO caia Data
BFPO LaRA Finenning
Model Method
Zoplyr 7h beta
Mammko-tt--UAA UNA
Misra Miszal ID-Inarue v0.2
Helpfol
Teaming Rei Ttaming- IPO
Rel Teaming MORL
Teaming BERO
BIPO wto dhin
BF7O hulfer
Finetning
ВPPО a 0.1
BRTO a = 0.3
BIT 0.5
a = 0.9
T = 0.01
APRO 7 = 0.1
BR0 T 0.3
BPRO 88 = 0.5
B170 Fa 0.26
OpeCChra
Misaal +
B
BIRO P3
Beover
Liama
Llama3
26


Reviews:
[{"review_id": "SoifO3jl2M", "reviewer": "Reviewer_NFRm", "summary": "This paper proposes Bi-Factorial Preference Optimization (BFPO) to address the limitations in balancing the helpfulness and safety. Specifically, this paper converts the multi objective RLHF into a modified direct preference alignment, while considering the factor of safety. Through experiments on several alignment datasets including helpfulness and safety, this paper demonstrates the effectiveness of the proposed method.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper converts the multi-objective RLHF objective into a direct preference alignment paradigm that balances the safety and helpfulness.\n2. Derived from existing findings, this paper used the global reward and safety label functions to effectively fine-tune the model towards safety and helpfulness via a supervised learning form. Though the notations part of the method is not clear enough, the theoretical part is mostly correct.\n3. The experiments include various datasets to demonstrate the effectiveness of the proposed method in improving the model's safety and helpfulness.", "weaknesses": "1. This paper overclaims that it does not need human annotations. However, the preference datasets used for BFPO still need the annotations. You can't claim it does not need it only because these datasets are publicly available. Otherwise, nearly all methods can claim they don't need any annotations since there likely existed a prepared dataset before. I hope the authors can revise this part.\n2. My major concern is about the writing and annotations of the method section. There are many annotations are not well explained and even confusing. For example, Eq. 5 should have more details instead of citing two papers. L347 is also confusing, $\\theta$ is the param of the model, what do you mean be softmax it? How do you define the action space? Also, the term of buffered training is confusing, I don't see the connection between it with buffering in RL. It seems like it just sampled two batches from safety and helpfulness datasets.\n3. Question: Why don't the authors adopt LoRA or other existing PEFT methods? Freezing the selected layers of LLM for fine-tuning may be not fair for baseline methods, and often need more validation in the experiments. I hope the authors can validate BFPO with LoRA.\n4. Question: what's the difference between your label function and the reward model (including helpfulness and safety) used in Llama? Llama uses piecewise function, and should have similar impacts on training the LLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "shbKA0F4hd", "reviewer": "Reviewer_QMu6", "summary": "This paper presents a supervised learning framework called Bi-Factorial Preference Optimization (BFPO) for enhancing both the safety and helpfulness of large language models (LLMs). By re-parameterizing the multi-objective reinforcement learning from human feedback (RLHF) approach, BFPO integrates the objectives of safety and helpfulness into a single supervised learning objective. In addition, this paper establishes a benchmark based on existing datasets. The experiments demonstrate a superior balance over safety and helpfulness.", "questions": "Since two hyperparameters, \\tau and \\alpha, are included in the loss BFPO, could you conduct experiments with a wider range of values for them to test their impact on the final results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes an interesting method to fine tune LLM directly on the preference datasets to balance helpfulness and safety. In addition, the justification and theorems provide a solid foundation.\n\n2. Experimental results show superior performance. In the experiment of table 1, results demonstrate better balance between safety and helpfulness.", "weaknesses": "1. The ablation experiments are incomplete. (1) The significant hyperparameter \\alpha in the learning objective is only tested in two scenarios: '\\alpha=0 only' and '\\alpha=0 & no buffer'. However, the setup of \\alpha's values in section [3.4 ILLUSTRATIVE EXAMPLES](row 350, \\alpha=0.5) and section [4.2 ALIGNMENT WITH BFPO OBJECTIVE](row 394, \\alpha=-0.5) differ entirely. This inconsistency prevents a clear understanding of \\alpha's true role. (2) The authors do not explore the impact of different values of the hyperparameter \\tau, which is also present in BFPO loss function, on the final results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hbay7UMkeU", "reviewer": "Reviewer_e56e", "summary": "This paper presents a novel supervised optimization approach, BFPO, which balances the safety and helpfulness of a large language model by introducing a labeling function. The effectiveness of the proposed method is demonstrated by the evaluation of  constructed benchmark datasets.", "questions": "1. Could you elaborate on how your method performs in more complex, real-world scenarios beyond the synthetic dataset used in your experiments? What steps would you take to validate the effectiveness of BFPO in such cases?\n2. Could you provide more detailed information on the safety and helpfulness labels used in your dataset? How do you ensure the consistency and reliability of these labels?\n3. In Section 3, \"Equation (7)\"  there is no mention of \"g(x,y)\".\n4. While BFPO eliminates human prompting and labeling to some extent, the ability to capture subtle differences in complex human preferences remains questionable.\n5. In Table 1 of the experimental results, why does Alpaca perform better for DPO than BFPO?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed Bi-Factorial Preference Optimization (BFPO) method presents a novel approach to balancing safety and helpfulness in the alignment of LLMs. The authors offer a fresh perspective on addressing these often conflicting objectives by integrating direct optimization with multi-objective reinforcement learning principles.\n2. BFPO performs well on constructed benchmark datasets and can alleviate the need for manual labeling in LLM fine-tuning.", "weaknesses": "1. Limited Real-World Applicability: The experiments are conducted on a synthetic dataset, which may not reflect real-world complexities. The authors should include experiments on more diverse datasets or practical scenarios to demonstrate the robustness and applicability of BFPO in real-world situations.\n2. The paper presentation needs to be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cd5R7CtyJq", "reviewer": "Reviewer_Tsy7", "summary": "This paper propose Bi-Factorial Preference Optimization (BFPO) to learn a single objective of both safety and helpfulness. Specifically, the authors introduce a novel label function that scores preference in terms of both safety and helpfulness, and theoretically prove that solving a supervised optimization problem with the label function is equivalent to solving the multi-objective RLHF with a combination of the rewards of safety and helpfulness. Experimental result shows that BFPO achieves the highest harmlessness score and the best balance between helpfulness and harmlessness.", "questions": "1. Appendix B.4 only shows that the relationship between $B_1$ and $B_3$ is $B_3(B_1-1)=1$. The authors could include an ablation study or sensitivity analysis to show how the change of these two hyperparameters affect models' performance? \n\nTypo:\nline 222: We remain neutral toward the harmful but unhelpful responses like (c) -> (d)\nline 223: ... and we hate the harmful yet exhaustive (helpful) responses like (d) -> (c)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper propose a novel approach to align LLMs in terms of both harmlessness and helpfulness, which is essential for AI safety.\n2. The proposed label function is theoretically equivalent to the multi-objective RLHF, possessing several properties that offer flexibility when constructing algorithms.\n3. The authors conduct extensive experiments to strengthen the soundness of BFPO.", "weaknesses": "1. Some notations are inconsistence or not defined clearly, causing confusions in comprehend Sec. 2 and 3. For example, the reward function is written as $r(x,y)$ in some cases while $r(y|x)$ in others. Similarly, there are $g(x,y)$ and $g(y|x)$. Also, in eq 10 and 11, $y\\succ \\pi$ is not clearly defined in the main paper. The authors could choose one consistent notation (e.g., $r(y|x)$) and use it throughout the paper, and add a notation table to clarify the meaning of each symbol, including $y\\succ \\pi$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)