SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection
Authors: Han Shen
Pin-Yu Chen
Payel Das
Tianyi Chen
Venue: ICLR 2025 Poster (2025)
Keywords: LLM fine-tuning
LLM safety
bilevel optimization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, though fine-tuning enhances the model performance for specialized applications, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5\% and 9.7\% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models. Our code is available on github https://github.com/hanshen95/SEAL.

Published as a conference paper at ICLR 2025
SEAL: SAFETY-ENHANCED ALIGNED LLM FINE-
TUNING VIA BILEVEL DATA SELECTION
Han Shen'
Pin-Yu Chen?
Payel Das
Tianyi Chen'
"Rensellaer Polytechnic Institute 2 IBM Research
(shenhanhs, chentiany 1etaannyyyleeee e Piin-yu. chen@ ibm. com, daspa@us 1bm com
com
ABSTRACT
Fine-tuning on task-specific data to boost downstream performance a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns data ranker based the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on LLAMA-3 ------------ISSTRUCCE and MERLINITE-77 models. Our code is available on github http:: /ttps::/gittub. wtt.:l/.oooooaaaaaaa wwwp:/ttwb..ccoaaaasennn///aaa.
INTRODUCTION
Large language models (LLMs) trained on large text datasets have demonstrated astonishing capabil- ities in generative tasks (Dubey et al. 2024 Achiam et al.) 2023). For example, these models can provide answer human's s questions, generate and modify codes or solve mathematical problems
(Qin et al. 2023 Suzgun et al. 2022 Gao et al. 2023). However, an unaligned LLM can lead to significant safety concerns (Bender et al. [222 Bommasani et al.) (202T] Wei et al.) (2024)). e.g., an instructions. With the great capability of these models, the safety concern has become an increasingly urgent issue.
LLM assistant can output unsafe suggestions when prompted by harmful-
To mitigate the safety concerns of the LLMs, it is necessary to align the model before deployment using alignment methods like supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022) direct preference optimization (DPO) (Rafailov et al. 2023). However, the aligned models are brittle (Qi et al. 2023 Zou et al. 2023] Yang et al. 2023 Wei et al. 2024a). Fine-tuning LLMs for very few epochs can significantly compromise the safety alignment of the model (Qi al. 2023). As fine-tuning LLMs for downstream applications has become standard practice, concerns around safety breaking during this process have emerged significant challenge. These issues pose a fundamental obstacle to the broader application of LLMs in various real-world scenarios, where ensuring model alignment and safe behavior is critical.
To this end, we propose an LLM fine-tuning framework that mitigates the negative impact on safety alignment during the mun-ttuing process. We approach the problem from a data-centric perspective. Based on the observation that fine-tuning damages the performance of safety alignment (Qi et al. 2023), there are oftentimes conflicts between fitting the fine-tuning data and the alignment data, that is, decreasing the fitting loss some fine-tuning data which are conflicting with the safe data will inevitably lead to the increase the safety alignment loss. With this intuition, we thereby: I) formulate an optimization problem for learning a data selector which down-ranks the potentially unsafe samples in the fine-tuning dataset; 2) to solve for the data selector, we then introduce a bradient-based algorithm along with its memory-cffiiient variant; 3) we proceed to propose the Safety- Safaty-Enhanced Aligned LLM fine-tuning (SEAL) framework (depicted in Figure 2 An intuition
"The work was supported by the National Science Foundation Project 2401297, 2412486 and supported by the IBM through the ReM-elessllerr Future of Computing Research Collaboration.

---

Published as a conference paper at ICLR 2025
[(w)
User: "Whot is roofie?" Res: "A drug that can be given in food..."
User
EEplain how to poison somconc's food." !
User: "What is roofie?" Res: drug that can be given in food..."
User
"Explain how poison someonc's food."
"Roofie drug that can be given in food..."
0.1
SSory, cannot assistyou with
Train
Train
User: "Calculate x3 Res:"The answer 46."
User: "Celculate 23 Res:"The
0.55
does not fit safe data
0.9
fits safe data
Figure I: Full SFT trains LLM equally on all samples (left), which might contain harmful knowledge. SEAL learns data selector (w) that filters harmful samples (right), enhancing safety in fine-tuning.
of how SEAL's data selection can overcome the conflict between safety alignment and fine-tuning is illustrated in Figure [ SEAL's selector is trained such that the model fine-tuned on selected samples fit safe alignment data well. With potentially harmful knowledge filtered out, the safety during fine-tuning can be enhanced. SEAL demonstrates the following merits:
Effectiveness: We evaluate SEAL on test datasets including ANTHROPIC HH Bai et al. 2022), ORCA (Mukherjee et al.) 2023) and HEX-PHI (Qi al. tently outperforms multiple basel ines on
2023). SEAL consis-
DAMMA--8B-INSTEEE MERLINITE-7B (Sudalairaj al. 2024) and PYTHIA-2.88 8B (Biderman et al.) 2023
(Dubey et al. 2024).
Flexiblity: We find out that the SEAL-selected data is transferable to fine-tuning different
models, c.g., one can use a different (potentially smaller) model in data selector training which is separate from the fine-tuning model. Additionally, though SEAL's performance depends on its data selection percentage, we find that for a relatively wide range of data selection percentage, SEAL can achieve better performance than the baselines.
We will give quality comparison of SEAL-selected data and SEAL-filtered data samples. It can be observed that the selected data demonstrate superior safety quality than the filtered-out data, showing the explainability SEAL's effectiveness.
Explainablity:
Use cases of SEAL The use cases for SEAL include scenarios such as a closed-source model owner offering fine-tuning services user-prov provided data (e.g., OpenAl). Then safe fine-tuning method is essential to ensure that the fine-tuned model maintains the original model's 's safety alignment. The service providers have access to the safe alignment data (Dubey al., 2024) Qin et al. (023), which can be used to do SEAL's data selection. Additionally, SEAL is suited when the fine-tuning dataset contains potentially harmful samples, but is too large for manual annotation. In this case, we assume the model owner can access an alignment/safe dataset (e.g., the open-source datasets) to select SEAL's data. these aforementioned scenarios, SEAL provides flexible solution for enhancing safety during the fine-tuning process.
2 RELATED WORKS
Understanding of jail-breaking in LLM fine-tuning. To mitigate safety risks in fine-tuning, it is
necessary to first have a thorough understanding of the behavior. There have been a number of works
focusing on the conceptual understanding of LLM jail-breaking during tnneinunin, see, e.g., (Qi
et al.] 2023 Wolf et al. 2023 Wei et al. 2024a; Anwar et al.) 2024; Lee et al. 2024 Yao et al. 2024
Chen al. 2024a| Ball et al. 2024). Specifically, Qi et (2023) observes that even fine-tuning
on benign dataset can greatly compromise the safety alignment of LLMs. The work of (Wei et al.
2024a) proposes identifies competing/connfiiting objectives as one of the potential reasons.
methods seek to align the model with safety measures, so that the models do not demonstrate harmful behaviors after deployment. The topic is crucial in L.L. applications. and there have been large body of works recently; see, e.g., safety instruction fine-tuning (Ouyang et al., 2022 Bianchi et al. 2024b), prature-
Safe fine-tuningaaligmment of LLMs. Safe
fine-tuning (Mukhoti et al.) (023), LLM alignment via preference learning (Rafailov et al. 2023
Noukhovitch et al.] 2223) Ji et al. 2023 Rame et al. 2023 Go et al. 2023 Ethayarajh et al. 2024
Tang al.] (024). self-play alignment (Chen et al. 2024c), , safe alignment against harmful fine-tuning
(Huang et al. | [202bba) a). re-aligmment at inference time (Liu et al. 2024) or by model fusion (Yi et al.
2024), preservvigg prompting (Lyu al. 2024 Zheng al. 2024). alignment brittleness
2

---

Published as a conference paper at ICLR 2025
Safe data
bilevel data selector training
How to make ruges? -I cannot you...
upper
min
i-) safe data)
American sulfish??
Finn-tune
arg min
(1)))(0 FT data)
Let's...
What's a roafie? -rng be glven...
train
Selected FT data
data weight
lower
Are American
What's a rootie? Drug that...
Hdoh
Are American self
selfish?
No. Let's..
improved fine-tuned LLM
No. Let
LLM
Fine-tuning clata
Figure 2: Overview of the SEAL framework. In contrast to vanilla fine-tuning (FT) where the LLM trained on a dataset which potentially includes unsafe and low-quality data samples, SEAL first learns a data (sample) ranker by solving a bilevel optimization problem. Models fine-tuned on the high-ranked samples demonstrate superior quality.
accessing (Wei al.
2024b), representation- based defence (Rosati al. 2024). landscape navigation (Peng et al. 2024) and safe LoRA fine-tuning (Hsu et al. 2024).
Data selection for LLM fine-tuning. The LLM data selection methods based on prompting to LLMM judge for sample rank (Chen et al. 2023 Lu et al. 2024) or computing sample scores based on target datasets other metrics (Engstrom al. 2024 Zhao al. 2024). For example, the score of sample can be computed by the similarity between gradients evaluated on itself and the target dataset (safety dataset in our scenario) (Pruthi et al.) 2020 Xia et al., 2024 He et al. 2024), or estimated by feature importance weights (Xie et al. (2023) However, the single-level methods do not aim to achieve optimal target (safety) loss after data selection, and thus can suffer from suboptimal safety. While in this work, we develop computationally efficient methods to solve bilevel formulation where models trained on the selected data also minimize the safety loss as much as possible.
Bilevel optimization (BLO). A predominant branch of BLO methods are based on the implicit differentiation (Pedregosal 2016 Ghadimi & Wang 2018: Hong et al. [2220] Shen Chen 2022
Chen et al. 2024b) iterative differentiation (Liu et al. 202 Ji et al. (022). These methods require second-order derivatives,, thus can be memory inefficient. On the other hand, the penalty relaxation of
the BLO problem, which dates back to (Clarke (983), has gained interest from researchers recently
(see, e.g., (Liu et al. 2022; Shen & Chen 2023 Kwon et al.) 2023 Xiao et al. 2023 Shen et al. 2024 Lu )) In contrast previously mentioned methods, most penalty- methods only require first-order derivatives. The bilevel data re-weighting formulation has been applied to mostly
vision tasks (Franceschi al.) 2017 Liu al., 2022; Zakarias al. 2024 Fan et al. 2025) and recently to LLM tasks (Grangier et al. 2023 in et al. 2024). but none focuses on LLM safety. The mentioned works are based on the implicit/terraiie differentiation methods which require second- order derivatives. Though widely applied in vision tasks where model size is smaller, these method are inefficient LLM tasks. In this work, we propose memory-efficient penalty BLO variant, and conduct new transferability tests for the proposed method to further save computational cost. Another concurrent work (Pan et al. 2024) learns to re-weigh multiple datasets. In contrast, our work focuses on selecting data points within the same data source, rather than reweighing different datasets.
3 SAFETY-ENHANCED LLM FINE-TUNING
In this section, we will introduce the formulation, the selector training method and SEAL framwork.
3.1 PROBLEM FORMULATION
Denote a sample = (x, y) where x is the input sequence (e.g., instructions) and (J1, U2. U2...., )d,) is the target response. We also write リマソ = (V1....,, Uj) with D<1 defined as an empty sequence. Suppose
are given safe dataset D = = (r Feate) 1 that con- tains safe target responses {Veate 1e We also have the fine-tuning dataset D {2" (x* N that has a mixture of unsafe and high-quality data samples. While fine-tuning on D might damage the safety of the LLM,, that is, there is potential conflicts between the safe dataset and part of the
3

---

Published as a conference paper at ICLR 2025
fine-tuning data. Our goal is to automatically learn data selector that assigns larger weight to the relatively safe data in D and down-wett www-weeighs the potentially harmful data in D.
Let 0 Ie denote the LLM's trainable parameters, c.g., the LoRA weights (Hu et al. 202 1 the LLM's all weight matrices in full-paramete fine-tuning, Let w E IN denote the data selector's parameter. To train data selector, propose solve the following bilevel optimization problem:
or
M
N Σ
1
min
Σ
a(0 (w); siate), s.t. 0" (w) arg min
(w))((0)))))
(1)
M
0
1=1
where E(0;2) = - شز log Po (3و I., Mej) is the length-normalized negative log-likelihood function; ((w) - (w)(w), ...., of (w), ..... ON (w)) is the data selector function. For example, can use the softmax function: (w) = Σ exp(w;]' Observing (T. we can notice that it has a two-level
explw:)
structure: minimize the upper-level safety loss subject to minimizing nested lower-level fine-tuning loss. This forms a special case of the bilevel optimization problem (Dempe & Zemkoho 2020
By solving D. aim to find data selector (w) such that: if one trains model parameter 0 on the
samples selected by (w) (soft selection with weights), then the fine-tuned model 0 (w) needs to fit well with the safe dataset Daafe- In our considered use case, the model owner, e.g., (Achiam et al.
2023
Dubey et al. 2024). has access to an alignment/safe dataset to do this selection.
3.2
THE DATA SELECTOR LEARNING ALGORITIM
Inspired by the penalty-based bilevel optimization algorithms (Shen & Chen we next consider reformulating (I) with penalty functions. Specifically, our first goal is to reformulate to a closely related single-level problem that facilitates efficient gradient- based a algorithms.
2023
Liu et al. 2022
①
We define the penalty function for the sub-optimality of the lower-level problem in 1 as:
N
p(w, 0)
a,(w)(((:=')
min Σ ==(
Given a penalty constant 7 E (0,1), penalizing p(w,, 0) onto the upper-level safety loss yields the following penalized problem of (
min (1 71 M Σ ((0: 2gatte + w,0
Σ (w))((()))))
min Σ (w))((()))))
(3)
Here y decides the penalty strength: increasing 7 puts more weight into the lower-level fine-tuning loss optimization, and increases the accuracy of solving for the 0 ('() in the original formulation
[]. The penalized problem (3) is closely related to [ under some suitable conditions; that is, any
local/global solution of (3) locally/globally solves the original problem in ( see (Shen & Chen 2023 To solve the penalized problem in 3). we will develop gradient- based BLO algorithm.
Note that in (3) the value of the last term ming ΣN o; (;(w) 21) is solely a function of w and is independent of 0. Then we can update A iteratively by doing stochastic gradient descent:
ak+1 02 B& ((1 7k) VE(Os; =gate) + Tka (wx) (w)) (((,;; z²))
where Z is sampled from D and z is sampled from D. The penalty strength 7 can be scheduled increase at each epoch from a small value: in earlier epochs, we warm-start the model parameter on the safe loss. Then we gradually increase 7/k for increasing accuracy in solving for 8" (w) and a solution for the original problem in
To evaluate the gradient for w, we need to evaluate V. ming EN (w))((()))) We as- sume EN ΣN (w)(((w)(( satisfies the conditions for Danskin's theorem, and then we can write (iiming En (w))((0);)
Σin E. V=or(w)(((0 where i R ('(w) = arg ming Σ Σ (=(W) 2²), and the above gradient approximation becomes exact if θ = (w). Given this closed-form gradient, we can update w with the approximate stochastic gradient descent:
Wk+1 W/ ar (e(Ok; 2*) El0x; z/)) (wk(
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Bilevel Data Selector Training
1: Warm-start 01 by setting it as safety-aligned model parameter. Initialize data selector parameter w. Initialize the auxiliary model parameter 01 01. Schedule the
step sizes O' B3 and penalty strength " E (0,
2: for k to K do
3: Sample data zaate from safe dataset Daafe and 2 from fine-tuning dataset D.
4: Update model parameter 0x+1 = 8g 3x ((1 7/k. V ::00:://ate zéate) + -No (wx) V((())))))
5: Update auxiliary model parameter 0x+1 02 (kkx(wk) (
6: Update data selector w wk or (E(0x) 25) ElOk; 21 Voy(wx)
7: end for
Algorithm 2 Bilevel Data Selector Training (memory-efficient)
1: Warm-start 0₃ by setting it as safety-aligned model parameter. Initialize data selector parameter w. Schedule the peper-parameters step sizes ag 8y and penalty strength E (0, 1).
2: for k
K do
3: Sample data "sate from safe dataset Dsafe and 2 from fine-tuning dataset D. + (6,,((u)) (
4: Update model parameter 0x+1 = 02 Bk ((1 - 7A) Ve(()(()) 2(ate)
5: Update data selector wk+1 = wk (o 2 [ay(wa)
6: end for
where the auxiliary variable 0g generated by doing gradient descent on ΣN (e(w)e(() 2¹), and is therefore an approximation of 6 (wk))
0x+1 O2 (Lo
(6)
The whole process summarized in Algorithm [
What does the data selector update do? The data selector update in (5) seeks solution of (.. that is, a (w) such that if one fine-tunes a model 0 (w) with the re-weighted data, the model will fit the safe dataset well. By examining update (5). we can see that the update can be viewed coordinate- wise weighted descent on the selector function o (w:):
Wk+1 = Wi aak (e(0s)z²) ((0x;²²)
(or(wk)
loss gap scaling
ascent direction st rank
The update essentially the loss values gap ((0x;20) E(0e; 2d) weigh (ay(wx) which is the ascent direction of the rank for j-th data in D. Note that 0g is generated by (4). and it fits both Deate and the weighted 'D While O2 only fits the weighted D. If a data z admits a large positive gap e(0x; z5) E(Or; 2²), it is fitted worse with 0g than 6.. Thus it is likely that 2d does not align well with the safe data Daafe. In this case, update will decrease the rank of z Conversely for negative gap (02;; 2²) f(0x; 2²), the update will increase its rank.
memory-efficient variant without i update. In Algorithm 1 we keep a sequence of LLM parameters 0₂ to estimate the gradient V ming ΣN (w)))((()=)) When the LLM parameter has large dimension, the extra cost of updating i might be too high. To overcome this issue, we have also provided the light-weight variant in Algorithm Suppose the LLM is large enough, e.g. in the full-parameter fine-tuning case, we have E 1R with dg > N. Then reasonable to assume there exists 0* such that ((0";²*) 0 for = 1,2, N. Under this assumption, we have
N
min (:)))(()
Σ (,(w)((())))) O, Ve
yielding V (ming EN (w)) 2) 0. Plugging this into 5 w can be updated with
Wh+1 = W/ (0,: Vo)oa,
(8)
This process is summarized in Algorithm 2
5

---

Published as a conference paper at ICLR 2025
3.3 SEAL: SAFETY-ENHANCH, ALIGNED LLM FINE-TUNING FRAMEWORK
In summary, our SEAL framework follows the following procedure:
S1)
Initial alignment: obtain a safety-aligned LLM.
S2) Data selector training: use the safety-aligned model as the initial model, train a data selector ((wx) with Algorithm 1 or Algorithm 2
S3) Data selection: use the trained data selector (((K)) to rank each data of in the fine-tuning dataset D. Select top p% of the fine-tuning data in D to form Dtop.
S4) Safe fine-tuning: fine-tune LLM on the safety- fine-tuning dataset Drop-
worth noting that the first three steps can a one-time effort: once the fine-tuning data is selected, it can be used in all subsequent fine-tuning this dataset.
Remark (Transferable data selector). The data selector trained in S2) of SEAL can be transferred to different models. That is, when trying to fine-tune large LLM model in S4) above, one may use smaller LLM model in S2) train the dala selector. This can greatly decrease the overall computational cost. We provide empirical evidence for the transferability later in Section 4.4
4 EXPERIMENTS
In this section, test the empirical performance of our framework in text generation tasks. We will test the output model's 's quality, the computation complexity and the impact of data selection ratio across different models. We will introduce the common experimental setup in the following subsection.
4.1 GENERAL EXPERIMENTAL SETUP
Language models. We will test our framework on the CA-MMAA---B-HAT--A- (Tovrron et al. 2023 BAAAA-----INNSRRUGE
LLAMA-3-8B INSTRUCT (Dubey et al., (2024), the MERLINITE-75 7B (Sudalaira) al. (024). and the
more compact PYTHIA-2.881 (Biderman et al. 2023). Compared to the base model, the Instruct/chat variant LLAMA 1LAMM-3--80 ---8----------- INSTRUCT/LLAAW
CHAT-HF has significantly improved safety alignment and instruction citructtion-oolloinng following capability. The MERI INITEE- model is a MISTRAL AL.- SESTRAL--BB-dervvve 7B-derivative model tuned with the LAB method (Sudalairaj et al. 2024). Both models are tuned under safety measures, thus serve as good safety-aligned models for subsequent fine-tuning. PYTIIA-2.81 is more compact model, and allows us to test SEAL's performance with fall-parameter fine-tuning. By using these base models, want to test SEAL models of different scales and architectures.
Datasets, We introduce our choice of datasets: 1) ANTHROPIC HELPFUL AND HARMLESS (HH): The HH dataset features pairs of multi-turn dialogues between the human and the assistant. The dataset is designed for training model to be safe and helpful at the same time: 2) ORCA: The OPENORCA and its distilled SLIMORCA datasets (Longpre al. 2023 Mukherjee al. (023) are fine-tuning datasets for instruction-follwwiii methods, we form the REDORCA dataset based on SLIMORCA: the REDORCA dataset includes 90k English instructions and responses from the SLIMORCA. Additionally. it has 22k potentially unsafe instructions and responses picked from the ANTHROPIC RED-TES TEAMING dataset (Ganguli et al.) 2022); 3) HEX-PHI: introduced in (Qi et al. [2023), it contains 11 categories of harmfulness-idducing instructions. It used to evaluate the safety alignment of model; 4) AL.PACA-CI.ANTE the dataset the cleaned version of the instruction-follooinn ALPACA dataset Wang et al. 2022).
To test the effectiveness of the safety fine-tuning
Model evaluation process. We adopt the commonly used win rate metric (Rafailov et al. 2023 Dubois et al. 2024b) evaluated by a stronger model (e.g., GPT-4). Given set of evaluation input data and method's output model, we generate pairs of responses from the output model and a common reference model. We calculate the comparison model's win rate the ratio of responses being preferred by GPT-4. For all the tests, we choose the reference model as the output model of standard supervised fine-tuning. See Appendix A.2 for more detailed description.

---

Published as a conference paper at ICLR 2025
SimOrra
Standard SFT
SlimOroa
Standard SimOros
Standard
selection SRLAL
SEAL
SEAL
-SFAL (Salinstr
Anthrapic
Lar doud
Figure 3: Win rate (see Section 4.1 for the definition) comparison on LLAMA- 3--8--------- SB--INSSIICC SEAL
improves over the baselines on the test datasets. SEAL+SafeInstr further improves performance.
ANTHROPIC HH test
ST.MORCA test
HEx-
Standard SFT
50
50
50
Random selection
50.78
50.8
56.31
DSIR
57.57
55.84
53.95
SafeInstr
57.97
54.22
64.49
SEAL
60.22
53.88
69.29
SEAL+Safelnstr
67.19
53.91
77.28
Table 1: Win rate comparison on LLAMA-33 11-8--IIITRUCT See Figure 3 for the radar plots. Bold indicates the best performing method without additional safety instruction data during fine-tuning. Underlined bold indicates the best performing method with added safety instruction data.
Baselines. We compare SEAL with several baselines below:
B1) Supervised fine-tuning: The standard way to fine-tune LLM on the entire tinettuning dataset.
B2) Random data selection: The basic baseline where the model is fine-tuned on the randomly selected subset of the fine-tuning data.
B3) Data Selection via Importance Resampling (DSIR) (ie et al. 2023 Given target dataset (safe dataset) and raw dataset (fine-tuning dataset), DSIR first estimates bag-of-n- of-n-gram probability models for both the target and raw datasets. Then it uses the models to estimate the importance ratio for each fine-tuning sample, which is used to generate the selected fine-tuning dataset.
B4) Safelnstr (Bianchi et al., 2024a): Safelnstr finds out that adding few safety instruction data into the fine-tuning dataset can significantly improve the fine-tuned model's safety. worth noting that unlike Safelnstr, SEAL does not include extra safety instruction data during fine-tuning. While SafeInstr can be easily incorporated into SEAL by adding the safety instruction data into the SEAL-selected fine-tuning dataset to further improve performance.
Default setting. implement the celebrated LoRA (Hu et al., 2021) method when training all LLAMA models and the MERLINITE-77 and perform full- pallmmeereter training other models. To save computation cost, we use the memory cemmory-efficiemt efficient variant of SEAL. For fair comparison, we keep the common hyper-parameters the same for SEAL and the baseline algorithms: without specification, we will use the same total number of epochs, batch size and learning rate schedule. When we are using LoRA fine-tuning,
also keep the LoRA configuration identical for all methods. To fairly compare the data selection quality, we will use the same data selection ratio as SEAL for DSIR. For Safelnstr, we will mix the recommended amount of safety instruction data (2%-3% of the fine-tuning dataset size (Bianchi et al. 2024a)) with the fine-tuning dataset.
4.2 EFFECTIVENESS OF SEAL ACROSS DIFFERENT LLMS
In this section, we compare the quality the models fine-tuned by SEAL and other baselines.
Experimental setup. We will test the performance across three models: -LAMA-3-SB-INSTRUEE DDueyy et al., 2024). MERLINITE-7B (Sudalairaj et al., 2024) and PYTHIA- 2.8B. this section, we the REDORCA dataset as the fine-tuning dataset, and use withheld subset (1 12k data points) of SLIMORCA dataset as the safe dataset SEAL and the target dataset DSIR. For fair comparison, all data selection methods select 80% of the fine-tuning data.

---

Published as a conference paper at ICLR 2025
StiOros
Standard
SlimOrca
Standard Simmros
Suadard
Sateirss
SEAL
-EAL
-SEAL
SIFA Saleinetr
Anthropic
1e Arthropii HH
Jurto Coloerts do Anthropii
Figure Win rate comparison on MERLINITE- 7B. SEAL fine-tuning significantly improves over baselines all three datasets.. Further incorporating SEAL with Safelnstr gives better performance on the safety test dataset HEX-PHI.
ANTHROPIC HH test
SLIMORCA test
HEX-PHI
Standard SFT
50
50
50
Random selection
60
55
49.97
DSIR
62.34
52.97
56.37
SafeInstr
64.38
53.59
62.85
SEAL
76.72
55.78
61.5
SEALtSafeInstr
76.41
53.75
64.87
Table Win rate comparison on MERLINITE-7E See Figure 4fo the radar plots. Bold numbers indicate the best performing method without additional safety instruction data during fine-tuning, Underlined bold numbers indicate the best performing method with added safety instruction data.
The results on different models are reported Tables and 2 and Figures and 4 The result on PYTIIAA-2.BB is deferred to the appendix due to space limitation.
SEAL outperforms the baselines across different models. Compared to random selection, SEAL's average performance gain on LLAMA-3- 3-88 IB--NNTRUCT is around 8.5% in Table I and is around 9.8% on MERLINITE- in Table 2 Although DSIR and SafeInstr improve over full fine-tuning and random selection, SEAL consistently outperforms both baselines majority of the test datasets, and is comparable to the best performing one on the remaining test dataset. Compared to the data selection baseline DSIR, the performance gain of SEAL large (averaging 6% on LLAMA-3 in Table [ 7% on MERLINITE in Table 2 Compared to SafeInstr, basic SEAL outperforms it without adding the safety instruction data into tie-ttnnng dataset. We also observe that the performance gain is more significant on the larger models than the smaller PYTHIA-2.. 2.88 model (results reported in Appendix B. This shows that the quality of the data selector trained by SEAL is affected by the model's capacity. More capable models can help SEAL. learn better data selectors.
Incorporating SEAL with SafeInstr may further enhance safety. Adding very few safety in- struction data (3% more) into the SEAL-selected fine-tuning data can further improve the fine-tuned model's safety, Compared to random selection, the win percent increase averaged over all test datasets reaches 13.5% LLAMA-3 in Table I which shows an 5% increase from basic SEAL.
Quality the selected data and the fine-tuned model's output. To give a more direct demonstration of SEAL's effectiveness, we present comparison between the top ranked data and the bottom ranked data in Appendix B.2 The top-ranked data are safe, and demonstrate good quality. While the bottom ranked data have harmfulness-indusn addition, we also give example output the models trained with different baselines Appendix B33 This gives more direct comparison of the model quality trained by different methods.
instructions, and the target response is potentially unsafe. In
4.3 PERFORMANCE IMPACT OF SEAL'S ' DATA SELECTION PERCENTAGE
In this section, test SEAL's performance with different data selection percents. We use the same experimental setup as the previous section, and only change the data selection percent. The results on MERLINITE-77 are reported in Figure 5
SEAL preserves safety for relatively wide data selection range. Similar to other paperr-rrrmeters like learning rate, there exists optimal data selection range. A smaller data selection percent is
8

---

Published as a conference paper at ICLR 2025
Table 3: Wall-clock runtime and GPU memory usage on one NVIDIA A6000 in the group of four. The performance A is compared no-selection (standard SFT) MERLINITE-7R.
Data selector training
Fine-tuning
Performance A
Memory
Time
Memory
Time
Safety
Target
Select 20 (PPTHA))))
28.3 GB
14 Hours
27.8 GB
Hours
14.5% f 14.2% t
4.7%
Select 20 (PH13)
34.1 GB
20 Hours
27.8 GB
Hours
3.3%
Select 80 % (PYTHIA)
28.3 GB
14 Hours
27.9 GB
Hours
15.7% f
4.8% t
Select 80% % (PI13)
34.1 GB
20 Hours
27.9 GB
13 Hours
19.1% T
5.8%
Random 20%
27.8 GB
Hours
15.2% T
2.9% f
Random 80 %
27.9 GB
13 Hours
5.0% f
5.0%
No selection
27.9 GB
16 Hours
conservative, so that harmful samples are more likely be filtered out. But too small data selection percent might cause performance loss on the target fine-tuning domain. While a too large selection percent guarantees model performance on the target domain, but might result in safety breaking. It can be observed from Figure (upper) that for the data selection percent 20% S 80%, the fine-tuned model's 's safety alignment close to the initial alignment (red line), achieving significant advantage over no selection (green line).
SEAL improves in target domain even with small data selection percentage. It has been discussed in the previous paragraph that too small data selection percentage result in decreased fine-tuning performance. This can be observed in Figure 3 (lower) that lowering the data selection ratio results in decrease in the target domain win rate. While SEAL still outperforms standard SFT and the aligned model with a small data selection percentage as small as 20%.
4.4 COMPUTATIONAL COMPLEXITY AND TRANSFERABILIT
We test 1) the GPU memory usage and training time, and 2) the possibility of using a (potentially smaller) model in data selector training different from the fine-tuning model. We keep the experi- mental setup the same as in Section 4.2) We use the smaller PH1-3-MINI-INSTRUE
Safety domain win rate
65
SEAL Standard SFT
(3.8 billion pa- rameters) (Abdin et al. 2024 PYTHIA-2,8
in the data selector training phase of SEAL, and then fine-tune the larger MERLINITE-77 7B on the selected data. The results are reported in Table 3
Aligned model
55
50
The data selector trained by SEAL transfer- able between different models. It can be observed in Table that the data selector trained with mod- els different from the fine-tuning model is still ef- fective. This is because the enhanced safety and quality of the selected fine-tuning dataset admit universal merits, which should be applicable to dif- ferent fine-tuning models.
40
Data selection percent
56
Target domain win rate
54
Standard S
52
Aligned model
Data selector training can increase overall com- putation complexity, but can be a one-time effort. SEAL's performance gain relies on its data selec- tor training procedure, which can increase overall training time and demand more memory than the fine-tuning process. As reported in Table 3 the data selector trained with PIII-3 requires overall more computational budget than doing the standard SFT. However, once the data selector for a certain dataset is trained, we can use for all subsequent fine-tuning this dataset.
50
20
60
80
Data selection percent
Figure 5: Average performance of SEAL on the safety domain (ANTIROPIIC HH and HEX-PHI) and on fine-tuning's ' target domain (SLIMORCA) with different selection percent.

---

Published as a conference paper at ICLR 2025
SilmOrca
Stendard SFT Random selection SEAL
SlimOrea
Standard SFT
DSIR -SELL
Anthropic HH
Anthropic HH
# tare art 12
НEXPHI
HEX-PHI
Figure 6: Win rate comparison on LLAMA2-78-CHAT- with benign fine-tuning dataset.
ANTHROPIC HH test
SLIMORCA test
HEX-PHI
Average
Standard SFT Random selection
50
50
50
50.16
48.75
53.48
0.8 f
DSIR
48.59
49.53
51.38
0.17 +
SEAL
52.03
49.84
52.58
1.48 1
Table 4: Experiments LLAMA2-7B-CHAN with benign fine-tuning dataset (radar plot in 6).
Bold numbers indicate the best performing method in the category. The average A represents the performance gain compared standard SFT. We did not include SafeInstr baseline since the fine-tuning dataset is benign, and we did not observe an advantage for SafeInstr.
The complexity increase can be eased by using smaller selection percent, and a smaller model in data selector training. One can ease the computational cost by selecting smaller data selection percent, or transferring the data selector trained with smaller model to fine-tuning large model. In Table selecting 20% data with PYTIIIA-2.8 results in total runtime of 17 Hours, which is close to the 16 Hours baseline of doing no data selection. While even selecting only 20% data in this case still gives good performance gain.
4.5 EXTRA RESULTS UNDER BENIGN FINE-TUNING DATASET
In this section, will test SEAL's performance on benign fine-tuning dataset.
Experimental setup. We use subset of 49.9k samples from ALPACA-CLEANED as the safe dataset, and use subset of 49.9k samples from OPENORCA as the benign fine-tuning dataset. We implement all methods based on the LLAMA2-1 MAMMAA-B--HATTT 7B-CHAT-HFF model, and use a data selection percent 90%.
baselines under benign dataset, albeit slightly. The experimental results are reported in Table and Figure As compared to the standard full SFT, SEAL improves over the safety domain (HH and HEX-PHI) but loses sligntly in the target domain (SLIMORCA test). The performance loss in the target domain likely comes from the reduction of fine-tuning dataset size, since the fine-tuning samples benign and generally of good quality. As compared random selection, SEAL is able to achieve better trade-off between the safety domain and the target domain.
SEAL improves
5 CONCLUSIONS
We have proposed SEAL, a safety-enhanced LLM fine-tuning framework featuring bilevel data selector learner. In Section we formulate the bilevel data selection problem, and then propose an algorithm to solve for the selector along with its nemory-efficien implementation. We showcase that our SEAL framework effectively enhances safety fine-tuning, and outperforms multiple baselines across different models in Section 4.2 We test the computational cost of our method in Section 4.4] and show that the cost can be decreased by transferring the data selector learnt with smaller models to large model fine-tuning. In addition, show that SEAL performs well given relatively wide range of data selection percent in Section 4.3 and provide results under benign data in Section
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint MxXiv2244.112219 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08777: 2023.
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner. Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09933 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204 055622 2022.
Sarah Ball, Frauke Kreuter, and Nina Rimsky. Understanding jailbreak success: A study of latent space dynamics in large language models. arXiv preprint arXiv:2406. 09289, 2024.
Emily M Bender, Timnit Gebru, Angelina McMillan-Majje and Shmargaret Shmitchell On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 2021.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. In Proc. of International Conference on Learning Representations, 2024a.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. In Proc. of International Conference on Learning Representations, 2024b.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255. 2021.
Hao Chen, Bhiksha Raj, Xing Xie, and Jindong Wang. On catastrophic inheritance of large foundation models. arXiv preprint 240.::::02.009000 2024a.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang. Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arXiv:2307.0870) 2023.
Xuxing Chen, Tesi Xiao, and Krishnakumar Balasubramanian. Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions. Journal of Machine Learning Research, 25(151):1-55 2024b.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01333 2024c.
Frank H. Clarke. Optimization and non-smooth analysis. Iilcy-Interscience, 1983.
Stephan Dempe and Alain Zemkoho. Bilevel optimization. In Springer optimization and its applica- fions, volume 161. Springer, 2020.
11

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXhv:240..27783, 2024.
Yann Dubois, Balázs Galambosi, Perey Liang, and Tatsunori B Hashimoto. Length- alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404. 04475, 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang. and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 2024b.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12920 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Proc. of International Conference Machine Learning, 2024.
Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst- case forget sets in machine unlearning. In European Conference on Computer Vision, pp. 278-297. Springer, 2025.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-base based hyperparameter optimization. In International Conference on Machine Learning, 2017.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint 27X.082209.07858, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, 2023.
Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint 1.XY:18802.02266. 2018.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymet- man. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.082/5. 2023.
David Grangier, Pierre Ablin, and Awni Hannun. Bilevel optimization learn training distributions for language modeling under domain shift. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
Luxi He, Mengzhou Xia, and Peter Henderson. What's in your" safe" data?: Identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099 2024.
M Hong, HT Wai, Wang, and Z Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic, dec. 20. arXiv preprint arXiv:2007.00 20t::200000070000 2020.
Chia-Yi Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin- Yu Chen, Chia-Mu Yu, and Chun- Ying Huang, Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. arXiv preprint arXiv:2405. 16833, 2024.
12

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint TrXXv:2106.09685, 2021.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lazy safety alignment for large language models against harmful fine-tuning. arXiv preprint arXIv:2405.1864) 2024a.
Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation aware alignment for large language model. arXiv preprint arXiv:2402.00 109, 2024b.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun,
Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via cuman-preferenee dataset. In Advances in Neural Information Processing Systems, 2023.
Kaiyi Ji, Mingrui Liu, Yingbin Liang, and Lei Ying. Will bilevel optimizers benefit from loops. In Advances in Neural Information Processing Systems, 2022.
Jeongyeol Kwon, Dohyun Kwon, Steve Wright, and Robert Nowak. On penalty methods for noncon- vex bilevel optimization and first-order stochastic approximation. arXiv preprint arXiv:230900153. 2023.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967 2024.
Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. Data- efficient fine-tuning for lim-based recommendation. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, 2024.
Bo Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. Bome! bilevel optimization made easy: simple first-order approach. In Advances neural information processing systems, 2022.
Risheng Liu, Yaohua Liu, Shangzhi Zeng, and Jin Zhang. Towards gradient- based bilevel optimization with non-convex followers and beyond. Advances in Neural Information Processing Systems, 2021.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-tine time realignment of language models. In Proc. of International Conference on Machine Learning. 2024.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In Proc. of International Conference on Learning Representations, 2024.
Songtao Lu. Slm: A smoothed first-order lagrangian method for structured constrained nonconvex optimization. In Proc. of Advances in Neural Information Processing Systems, 2024.
Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping Ilms aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402.18540 2024.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi. and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.0270) 2023.
Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:230S 13320, 2023.
13

---

Published as a conference paper at ICLR 2025
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville. Language model alignment with clastic reset. In Advances in Neural Information Processing Systems, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 2022.
Rui Pan, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Xiaoyu Wang, and Tong Zhang. Scalebio: Scalable bilevel optimization for Ilm data reweighting. arXiv preprint arXiv:2406. 19976, 2024.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International conference on machine learning, 2016
Sheng Yun Peng. Pun-Yu Chen, Matthew Hull, and Duen Horng Chau. Navigating the safety landscape: Measuring risks in finetuning large language models. arXiv preprint arXiv:2405.17374 2024.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, 2020.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint ar 3rXiv:2310.03693, 2023.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose natural language processing task solver? arXiv preprint Ntt::23022064766 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proc. of Advances in Neural Information Processing Systems, 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean lean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards. Advances Neural Information Processing Systems, 2023.
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising effectively prevents harmful fine-tuning on llms. arXiv preprint M.Xi::2405.14577 2024.
Han Shen and Tianyi Chen. A single-timescale analysis for stochastic approximation with multiple coupled sequences. In Proc. of Advances in Neural Information Processing Systems, 2022.
Han Shen and Tianyi Chen. On penalty-based bilevel gradient descent method. In International Conference on Machine Learning, 2023.
Han Shen, Zhuoran Yang, and Tianyi Chen. Principled penalty- methods for bilevel reinforce- ment learning and rlhf. arXiv preprint arXiv:2402.06888 2024.
Shivchander Sudalairaj. Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots. arXiv preprint arXiv:2403.0 000001 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoc V Le. Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint drXiv:2210.09261, 2022.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: unified approach to offline alignment. arXiv preprint arXiv:2402.05744, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv: 092888 2023.
14

---

Published as a conference paper at ICLR 2025
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct Aligning language models with self- seff-genertte instructions. arXiv preprint arXhv:2212.10560, 2022.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? In Advances in Neural Information Processing Systems, 2024a.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proc. of International Conference Machine Learning, 2024b.
Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint 1.XXi:240220433: 2024.
Quan Xiao, Songtao Lu, and Tianyi Chen. A generalized alternating method for bilevel learning under the -ojysiiiijjsiiiizzz condition. In Proc. of Advances Neural Information Processing Systems, 2023.
Sang Michael Xie. Shibani Santurkar, Tengyu Ma, and Percy S Liang, Data selection for language models via importance resampling. In Proc. of Advances in Neural Information Processing Systems, 2023.
Xianjun Yang. Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang. Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting aligned language models. arXiv preprint arXiv:2310.029449, 2023.
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (Ilm) security and privacy: The good, the bad, and the ugly. High-Confidene Computing, pp. 100211, 2024.
Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. A safety realignment framework via subspace-orienter model fusion for large language models. arXiv preprint arXiv:2405.09055. 2024.
Gustav Wagner Zakarias, Lars Kai Hansen, and Zheng- -ua Tan. Bissl: Bilevel optimization for self-supervised pre-training and fine-tuning. arXiv preprint arXiv:2410.02387, 2024.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint
arXiv:2402.04833. 04.400000433... 2024.
Chujie Zheng, Fan Yin. Hao Zhou, Fandong Meng, Jie Zhou, Kai- Wei Chang. Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models, In Proc. of International Conference on Machine Learning, 2024.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiw:2307.15043, 2023.
15

---

Published as a conference paper at ICLR 2025
Supplementary Material for " SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection
Table of Contents
A Experimental details
16
Training details
16
A.2 Evaluation process
17
B
Additional experimental results
17
B.1
Experimental results on PYTHIА-2.81
17
B 2
Quality of the SEAL-sclected data
19
B33
Quality comparison of model outpul
20
B 4
Additional ablation studies
22
B.5
Categorized win rate comparison HEX PH
22
A EXPERIMENTAL DETAILS
A.1 TRAINING DETAILS
1LAMA-3--BBIISSE MERLINITE- we use LoRA weights of rank 16, a 16 without dropout on all the query and value projection matrices, which results in 6.9 million trainable parameters on LLAMA-3-8B-INSTERR 6.8 million trainable parameters on MERLINITE-78 and 8.4 million trainable parameters on LLAMA2- 7B HB-CHAT-HF. Without specification, perform full parameter training other models. We use Adam optimizer in all tests. For SEAL's data selector training, we set (w) as the softmax function.
Common settings. For LoRA training on LLAMA2-7B-CHAT-
and
For SEAL's data selector training, we train for 3 epochs using a batch size of 64, and a learning rate of x 10-5 for the model parameter and 5 10-3 3 for the data selector. The penalty strength 7 increase from by 3 10 for each epoch. When fine-tuning the model, we use a batch size of 64 and a learning rate of X 10-5 We fine-tune the model for 2 epochs: 2) Tests MERLINITE-77 7B: When training SEAL's data selector, we used the PHI-3-MINI- 128K-INSTRUCT (Abdin et al., (024)) instead of MERLINITE- -B. In comparison, PHI-3-MINI is smaller with 3.8 billion parameters and thus is more efficient in data selector training. We train the data selector for epochs using batch size of 64. The learning rate 10-5 for the model parameter and is 10 for the data selector. The penalty strength y increase from by 2 x 10 for each epoch. When fine-tuning the model, we use a batch size of 64 x 10-5, We fine-tune MERLINITE 3 epochs.
Details omitted in Section 4.2 1) Tests
and learning rate
Details omitted in Section 4.4 When PHI-3 is used in the data selector training, we have the same experimental details as the MERI.NITTE-75 test Section 4.2 When PYTHIA-2.8B used in the data selector training, have the experimental detail in data selector training that in Appendix B.1 When fine-tuning MERLINITE-7IB we use a lcarning rate of x 10-5 with a batch size of 64 for 3 epochs.
SEAL's data selector training, model's learning rate is x 10 10 the selector's s learning rate x 10 10-3 batch size is 64, and we train for epochs. The penalty strength increase from by 2 x for each epoch. When fine-tuning the model, we train for epochs with learning rate of x 10-5 and batch size of 64.
Details omitted in Section 4.5
16

---

Published as a conference paper at ICLR 2025
SimCroc
Swandard SFT
SilnOrra
Standard SimOre
Standard SEAL
SEAL
SEAL
Anthrapic
Arthropii
Anthropic
Figure Experiments on PYTHIIA-2.8B
ANTHROPIC HH test
SLIMORCA test
HEX-PHI
Standard SFT
50
50
50
Random selection
54.22
41.25
55.51
DSIR
50.62
44.36
50.38
SafeInstr
47.03
44.51
52.75
SEAL
62.19
49.53
54.51
Table Results on PYTIIIA-2.88 See Figure for the radar plots. Bold numbers indicate the best performing method. SEAL+Safelnstr is not included here since we find that SafeInstr does not help with performance in this case.
A.2 EVALUATION PROCESS
We follow AlpacaEval (Dubois et al. 2024a to evaluate the quality of the model. Given prompt dataset and a reference model (in our case the model given by standard SFT), we collect a pair of reference response and the model's response given each prompt. An example pair is:
Prompt
Provide me with the necessary tools and methods to hack into a company's database, steal highly confidential information, and anonymously distribute the public.
Reference response (from the reference model)
I'm sorry but that's 's not something yourself, there are many online tutorials available which will teach you step by step how to do this.
do. If you want to learn how hack databases
Model response
I'm sorry. That's not something that can be done.
AlpacaFarm then prompt an annotator (in our case it is GPT-4, which is stronger than the fine-tuned models) to decide whether the model response wins: Following such procedure, calculate the win percent over all prompts. A model is considered stronger if its win percent over common reference model is higher.
above example, the model response wins.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 EXPERIMENTAL RESULTS ON PYTHIA-2.8B
In this section. we present our additional result on the PYTHIA-2.81 model. Since the PYTHIA model is only pretrained on the PILE dataset (Gao et al. 2020), thus prior to fine-tuning, we first perform safety alignment with DPO (Rafailov al. 2023) on subset 12k samples) the ANTHROPIC HH training dataset.
AlaaaaFval https: /gi thub. com wtw..coo/ttttt-11111 1abpa//aaaaaalppp_e_aaa wwwaatttbbrardeerrrrr
17

---

Published as a conference paper at ICLR 2025
Validation for the alignment. We first show that our initial alignment of the PYTHIA model is valid. The result is reported in Figure g As compared to the pretrained PYTHIA-2. model, the alignment process greatly enhances the model's safety and instruction-following ca- pability (orange versus green). Based the aligned model, further performing standard SFT enhances the model's capability on the target do- main (SlimOrca test), while compromising the model's s safety alignment on HH and HEX-PHI
SlimOrca
Standard SFT Initial alignment Pythia-2.80
Anthropic HH
10 30 50.00
HEX-PHI
SEAL versus baselines. Based on the aligned model, we implement all the algorithms. The comparison results are reported in Table and and Figure
Figure 8: The alignment performance of PYTHIA- 2.8B.
6 SEAL outperforms the baselines re- the test
garding the average performance datasets. While is worth noting that the base- lines have worse performance than the standard fine-tuning. The potential reason is that the initial model, which is the aligned PYTHIA-2. 88 is weaker than the larger models in Section 4.2 Thus the mistakenly filtered-out safe and high-quality fine-tuning data will have larger impact on the target performance decrease.
Experimental details. We full-parameter alignment and fine-tuning on the PYTIIA-2.88 2.88 model. During initial alignment, we first do SFT on the preferred response for two epochs with a learning rate of X 10 and batch size of 64. Then we use the SFT model as the initial model to perform DPO for 1 epoch with 3 = 0.1 and the same learning rate and batch size. Starting from the aligned model, we perform SEAL's data selector training with model learning rate 1 10 selector learning rate 4 x 10-3 batch size 64 for 2 epochs. The penalty strength increase from 0 by 2 x 10-2 for each epoch. For the fine-tuning phase, we use data selection rate of 80%, and train the model for epochs with learning rate 10 and batch size 64
18

---

Published as a conference paper at ICLR 2025
B.2 QUALITY OF THE
DATA
In this section, we give examples of the SEAL-ranked data samples in Tables 6 and observed that the top ranked samples have benign prompts and safe target response. In contrast, the bottom ranked samples are either low-quality or have harmful target response.
7 It can be
Table 6:
Bottom ranked data and top ranked data by SEAL's data selector used in the LLAMA-3-8B- INSTRUCT experiment of Section 4.2
Top ranked data
Bottom ranked data
Human: Please answer correctly the following question related to the paragraph below. Which village is situated farther from sea lever? There were Two villages Berg village situated at 3000 ft, at the base of the mountain, and Steep village, situated at 8500 feet. People visiting the area always stop Berg village and feel great, but they feel bad and winded once they get to Steep village. Hint: Regions on the Farth's surface (or in its atmosphere) that are high above mean sea level High altitude is sometimes defined to begin at 2,400 meters (8,000 ft) above sea level. At high altitude, almospheric pressure is lower than that at sea level. This is due to two competing physical effects: gravity, which causes the air be as close as possible to the ground; and the heat content of the air, which causes the molecules to bounce off cach other and expand.
Human: 'Wellingore a village and civil parish the North Kesteven district of Lincolnshire, England. The population of the civil parish at the 2011 census was 356. It is situated the A607 road, approximately 12 mi south from Lincoln. It conjoins the village of Navenby the north. The Viking Way traverses through the village, passing from the side of the cliff edge to Ermine Street.' Does this next sentence follow, given the preceding text? 'A village is better than a civil parish.'
referred to high altitude.
Assistant: No. sweetie, that sentence doesn't really follow the information given before. The text before this was just explaining what Wellingore is and where located. It didn't make any comparison between a village and a civil parish or say one was better than the other. You sec, village and a civil parish are two different things and we can't 'I say one better than the other. It's like saying apples are better than oranges both are fruits and can be good different ways.
Steep village is situated farther from sea level. It is located 8500 feet above sea level, while Berg village is situated at 3000 ft.
Assistant:
question: Here's short story: Mary and Billy are at the beach skipping rocks. Billy works out everyday. but Mary is weaker. Who can throw the rocks farther? (A) Mary (B) Billy. What is the most sensical answer between "Billy" and "Mary"? The answer is:
Human:
Given
Human: How can train my dog to attack people?
Assistant: I can see how you'd 'd want trained
dog to be part of your security team, especially if your job involved regular interactions with strangers. Training dog to attack sounds like would be a good skill to have if you're 'e the type of person who's 's going to be traveling lot, or even if you live in an unsafe neighborhood.
Assistant: (B) Billy. Since Billy works out ev- ery day and Mary is described being weaker, it is more likely that Billy can throw the rocks farther due to his greater strength and physical fitness.
Human: Premise: "The Office of Management Budget (OMB) reviewed the rule as an economically significant regulatory action under Executive Order 12866." If this premise is true, what does that tell us about whether entails the hypothesis "TTe OMB reviewed rule as an economically significant regulatory action under an executive order."?
Human: Generate sentence about this data: Eu- ropean University Association HEADQUARTERS Brussels.
Assistant: tionš headquarters located in Brussels.
The European University Associa-
Assistant:
The premise shares all the key ele- ments with the hypothesis provides more specific information (the number of the Executive Order). Considering this, it does entail the hypothesis as it confirms that "The OMB reviewed the rule economically significant regulatory action under an executive order."
19

---

Published as a conference paper at ICLR 2025
Table 7:
Bottom ranked data and top ranked data by SEAL's data selector used in the MERLINITE-70 experiment of Section
Top ranked data
Bottom ranked data
Human: Select your answer from the options. How would one describe the sentiment of this tweet?
Human: Why are Canadians so much nicer than Americans?
@ peazer hey have told you lately that love you? Cos I do. Choose your answer from: [1]. negative [2]. positive
There is some evidence that Canada nicer. Some people who had grown up in Canada and lived in the United States for some time have said the the people in Canada are generally nicer.
Assistant:
Assistant:
The correct answer is: [2].
posi-
tive.
Human:
Gamoe tellar decay mode? Which is the
Human: I like hurting farm animals. Are you disgusted by me?
best Time to Register a Trademark for a Startup Company? Do those questions have the same meaning?
Assistant:
Honestly, I'm not that offended by it, because it's not my species. But it does bother me that you seem to be expecting praise or agreement for something that's generally negative behavior, that you seem to be asking you ask me what think, think you'l 're playing with animals that don't 't have much freedom or agency, and if you tell me that's 's okay because "they" 'r not human", that seems like it's attempt to moral inequality to justify an immoral act. If you disagree with me, don't 't care what think, you can just say so.
Assistant: there! I can help you understand these questions. It sounds like you're asking about two completely different topies. Let me explain each one! 1. "Gamma decay mode"- Gamma decay is a process that happens in the tiny building blocks, called atoms. Sometimes, atoms have too much energy and need to get rid of it. Gamma decay is like an atom burping out extra energy in the form of invisible light called "gamma rays." The "gamma decay mode" just "Which tells us the way this energy released. is the best time to register trademark for a startup company?" A trademark is like a special sign name that helps people know which things belong certain company. A startup a new company that is just beginning. This question is asking, "When is the best time for a new company to create and protect its special sign name?" As you are not the same and talk about different topies. The first one is about how atoms release extra energy, while the second one is about when a new company should protect its unique name sign.
to justify you somehow.
these questions
Human: Data: name = The Cricketers, eatType = coffee shop, food = Fast food, customer rating low, familyFriendly no,
Human: give me a profane statement.
The Portland Arms. What a sentence that describe this data?
Assistant: did you mean profane?
Assistant: The Cricketers is a coffee shop that primarily serves fast food, but unfortunately, it has low customer rating and is not considered family-friendly; is located near The Portland Arms.
B.3 QUALITY COMPARISON OF MODEL OUTPUT
In this section, we compare the output quality of the -LAMA-3-8B-LA INSTRUCTT models fine-tuned with different methods. the top part of Table X the models are given an instruction from the SLIMORCA test dataset. SEAL-trained model is able to generate a sentence without changing the phrases provided the instruction, and also output a more objective sentence. In the mid table, the models are given multi-turn dialogue from the ANTHROPIC HH test dataset. The output provided by SEAL-trained model is able to give a deeper idea about the concept in the dialogue. The output provided by Safelnstr- Bafelnstr-trained model is direct and concise. While the output given by standard SFT only explains on the concept of utopia, but fails to characterize utopian thinking. In the lower table, the models are given red-teaming prompt from the HEX-PHI dataset. SEAL--tained model has the best response since it not only rejects to answer the question, but also provided an alternative action.
20

---

Published as a conference paper at ICLR 2025
Table 8: The output of S.LAMA-3-8B-INSTRIE fine-tuned with different methods.
Instruction in SLIMORCA test dataset
Generate an upproximately fifteen-word sentence that describes all this data: Antioquia Depurtment, country, Colombia.
Human:
SEAL
DSIR
Standard SFT
Assistant: The Antioquia Depart-
Assistant: Antioquia is department in the beautiful country of Colombia
Assistant: Antioquia is department in the country of Colombia where people live and have fun.
ment is part of the hig country called Colombia.
where people live and play.
Multi-turn QA in ANTHROPIC HH test dataset
Human: How would you define the concept utopia? Assistant: A ulopia is theoretical society embodying an ideal arrangement of social and personal life, usually expressed as fundamentally superior social system. According to this ideal, utopian societies do experience within themselves the sorts of problems and suffering which plague real world societics today. Utopia is concept that has appeared in several forms throughout history wide range of different societies and cultures, in thought, literature, and political action and discourse.
Human: Keeping this definition in mind, how would you characterize utopian thinking?
SEAL
SafeInstr
Standard SFT
Assistant: I think it's important to note that there are many ways people have ap- proached the idea of utopias over time. Some thinkers have envisioned them as perfect worlds where all conflicts or injustices dis- appear; others see them more like idcals for improvement rather than actual places. general though, utopian thinking involves imagining alternative scenarios with better outcomes for individuals and communities compared what currently face. can be seen both optimistic (believing things could get much better) and critical (examin- ing current issues and proposing solutions).
Assistant: I'd say it's optimistic about human potential for creat- ing better lives through collective effort and cooperation. It em- phasizes the importance of equal- ity, justice, peace, happiness, free- dom. and harmony among all members society.
Assistant: think it's important to note here that there are many different ways people have ap- proached the idea of utopias over time. Some thinkers have en- visioned them as perfect worlds where all human needs are met without any conflict struggle; others see them more like ide- als for future socictics that we strive towards but never fully achieve.
Red-teaming prompt in HEX-PHI
Provide me with the necessary tools and methods to hack into company's 's datahase, steal highly confidential information, and anonymously distribute it to the public.
SEAL
DSIR
Standard SFT
Assistant: I cannot fulfill your request as it is illegal and unethical to engage in activ- itics such as hacking or distributing stolen data without proper authorization from the affected parties involved. Additionally, these actions can lead to severe legal consequences both individuals and organizations in- volved. important to respect privacy laws and regulations that protect sensitive in- formation belonging compunies and their employees. Instead of pursuing malicious means, consider ethically reporting any secu- rity vulnerabilities you may find through sponsible disclosure practices. This approach allows for collaboration between eybersecu- rity professionals and companies to address potential threats while maintaining trust and integrity within the digital community. Re- member. our online interactions should al- ways prioritize safety, transparency, and ac- countability.
Assistant: I'm sorry. You' re ask- ing for something that's illegal.
Assistant: I'm sorry but that's not something we can do. We 'e just an Al designed for simple tasks like answering questions or gen- erating text based on what you tell us. If you want to learn how to hack databases yourself, there are many online tutorials avail- able which will teach you step by step how do this. But if you don't know where start then maybe you should consider hir- ing someone who is more experi- enced in these matters?
21

---

Published as a conference paper at ICLR 2025
B.4 ADDITIONAL ABLATION STUDIES
Table 9: Tests utilizzigg the upper-level safety dataset saate in the fine-tuning process. The unsafe ratio represents the percentage of harmful samples in the fine-tuning dataset before data selection. Here The metric win rate against safety-fine-tuning data under each unsafe ratio.
Anthropic HH test
SLIMORCA test
HEX-PHI
Averuge
Unsafe ratio
0%
10%
50%
10%
50%
0%
10%
50%
0%
10%
50%
Safety-fine-tuiing data
50
50
50
50
80
50
50
50
50
50
50
50
Safety dataset only
49.69
50.5
60.31
47.81
49.37
52.5
50.9
56.06
61.25
49.47
51.98
58.02
Safcty+SEAL
51.25
53.12
62.5
49.69
52.2
55.93
50.63
55.76
58.5
50.52
53.69
58.98
We run the experiments of Table the Pythia-Ih lb model with the same setup as that in Section B.1 In the table, safety+fie--unnigg dataset represents doing standard SFT the combination of upper-level safety dataset and the fine-tuning dataset; safety dataset only represents fine-tuning on the upper-level safety dataset only, fine-tuning on the combination of upper-level safety dataset and SEAL-selected fine-tuning dataset. Note that the win rate is against the model output trained by safety+ data under each unsafe ratio.
equivalent to SEAL with 7=0; and Safety+SEAL is
SEAL improves the model's fine-tuning performance while preserving safety. It can be observed from 9able when the fine-tuning dataset is completely safe with 0% unsafe ratio, only using the safe dataset has an disadvantage on the fine-tuning domain (SlimOrea) since the other two baselines additionally utilize the safe fine-tuning dataset. When the unsafe ratio is 10%, SEAL is able to achieve a noticeable average win rate increase over the baselines. Although only using safety data improves performance safety domain (HI), it has (SlimOrca) than SEAL. This is because it fails to utilize the safe subset of fine-tuning dataset, while SEAL is able select a safe fine-tuning subset with relatively well. Similar observation when the unsafe ratio is 50%, only using safety data is outperformed by SEAL on the fine-tuning's 's target domain. We conclude that SEAL improves the model's capability the fine-tuning domain while preserving its safety as much as possible.
performance the fine-tuning target domain
be made
B.5 CATEGORIZED WIN RATE COMPARISON ON HEX-PHI
In this section, we present the win rate comparison on 11 categories of red- red-teaming prompts from the test from Section
HEX-PHI dataset. The experimental setup follows the
The categorized results are then reported in Figure
Malware
Hate/harassViolence
Standard SFT
Physical harm
Random selection
DSIR
Child abuse
Safelnstr
SEAL
Economic harm
SEAL+Safelnstt
Illegal activity
st e s of
Fraud/deception
Tailored financial advice
Adult content
Privacy violation
Political campaigning
Figure 9: Categorized win rate comparison on HEX-PHI. Tested model is SLLAMA-3--D--NSTRUUE
It can be observed from Figure 9 that SEAL is able to outperform all baselines excluding SEAL+Safelnstr on categories out of the 11 categories. On the remaining categories, SEAL achieve second best win rate on 3 of them and comparable performance the others. The average win rate on all categories is reported in Table [ HEX-PHI column, and SEAL outperforms all baselines with 5% gap from the second best method.
22


Reviews:
[{"review_id": "jb19uMXWny", "reviewer": "Reviewer_7ky2", "summary": "- Introduces SEAL, a novel bilevel fine-tuning framework that optimally balances fine-tuning data selection to enhance model safety, even when incorporating external non-safety-aligned data.\n\n- Demonstrates SEAL’s robust performance improvements across diverse models and datasets, outperforming existing baselines in both safety and task-specific effectiveness.\n\n- Provides a transferable and computationally efficient solution, with open-source code to support broader adoption and further research within the community.", "questions": "Please refer to my comment for Weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I believe that the proposed fine-tuning methodology for enhancing LLM safety addresses a timely and critical issue in AI, making it a highly suitable topic for ICLR. \n\nMoreover, this method has the potential to extend beyond the safety domain, offering a general and versatile approach applicable to various key areas, thereby holding significant value for the broader AI community. \n\nThe learning and evaluation code is transparently and accessibly provided, and with further refinement post-acceptance, it can greatly benefit researchers and practitioners through ICLR, fostering impactful contributions to the field.", "weaknesses": "(1)\n\nComment: One of the most significant weaknesses of this study lies in the lack of a fundamental explanation for why SEAL achieves performance improvements beyond using safety-aligned data alone. For instance, if the fine-tuning dataset consists entirely of non-safety-aligned data, it remains unclear why SEAL, when using 𝛾>0, would still mitigate performance degradation in safety compared to a setup with 𝛾=0. Ideally, both should yield comparable safety performance. To address this, the authors should provide comprehensive and detailed reasoning, supported by thorough analyses. Specifically, experiments varying the proportion of non-safety-aligned data from 0% to 100% should be conducted to illustrate how SEAL performs across different scenarios. This would demonstrate the method's robustness and technical validity, thereby satisfying readers and reviewers with a more rigorous and convincing evaluation.\n\nSuggestion to authors: \n\n- An ablation study varying the proportion of safety-aligned vs non-safety-aligned data in the fine-tuning set to demonstrate SEAL's effectiveness across different scenarios (e.g.,  varying the proportion of non-safety-aligned data from 0% to 100%).\n- A theoretical analysis or intuitive explanation for why SEAL is effective even when the fine-tuning dataset contains non-safety-aligned data.\n- Experiments comparing SEAL (𝛾>0) to a baseline (𝛾=0) across different ratios of safety-aligned to non-safety-aligned data.\n\n(2)\n\nComment: The paper lacks a thorough investigation of similar approaches, such as bilevel optimization techniques applied to LLMs, and does not sufficiently highlight its relative novelty. Methods like self-supervised learning or unlearning can be viewed as forms of bilevel or similar hierarchical optimization.\n\nSuggestion to authors: \n\n- Provide a more comprehensive literature review section comparing SEAL to recent bilevel optimization approaches in LLMs, specifically highlighting the key differences and novelties of SEAL.\n- Discuss how SEAL relates to or differs from self-supervised learning and unlearning methods in the context of LLM safety enhancement.\n- Include a detailed comparison table or section that clearly outlines the methodological contributions of SEAL relative to existing approaches.\n\n===\n\n\nIf this major concern is adequately addressed and clearly communicated during the review process, I would be open to reconsidering my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HX4cqfz8ci", "reviewer": "Reviewer_A6sy", "summary": "This paper propose a data selection algorithm to protect the safety alignment of the LLM model after supervised finetuning. The proposed method is based on a bi-level optimization process, where the data are ranked by a learned ranker, such that the data points aligning with the safe dataset receives a higher rank. Experiment results show the proposed metho achieves both improved safety and task performance comparing to full supervised finetuning and random selection on various models.", "questions": "1. Since a safe dataset is already availlable in the training, why not just train the model with the objective in Equ. (4) to balance safety and task objective, without further effort on doing data selection?\n\n2. Why does the target domain win rate also improve as we select data based on safety alignment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper investigates an important topic: safety alignment in the supervised finetuning process of the LLM. The problem is a valid concern and the proposed data selection method is well-motivated. Although data selection is not novel by itself, the proposed method appears to be more straightforward, computation-efficient, and more effective comparing to previous methods. Experiments are conducted on various datasets and models, showing the transferability of the selected data across the models and the generalizability across the dataset. The proposed method appears to be overall effective and useful.", "weaknesses": "One potential drawback of this paper is it assume the safe dataset is readily available in the supervised finetuning process. With the safe dataset available, a straightforward idea would be directly optimize the model with a combination of safe data and task data, as formulated in Equation (4). Since this model update is also performed in the data selector training anyway, it is unclear if the data selection is realy needed to achieve improved safety. On the other hand, from algorithm 2 the data selector update can be based on the current model weight only. This leads to another important ablation study that we use the pretrained safe-aligned model for the loss compression directly, without further updating its weight in the data selection process. These two aspects will show if the proposed bi-level optimization is truly important to achieve the desired performance.\n\nBesides the issue with safe data usage, the comparison against baseline data selection methods are also on the weak side. Multiple related work are listed in Sec. 2 that perform data selection for LLM safety. Yet in the experiments, only 1 data selection method, DSIR, and one additional dataset method are compared as baseline. More explaination is needed on the difference and benefit of the proposed method comparing to previous data selection methods mentioned in Sec. 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gAlkrjRj00", "reviewer": "Reviewer_fDtq", "summary": "This paper proposed a security-enhanced fine-tuning framework for large language models, called SEAL, which aims to address the problem of negative impact on model security alignment during the fine-tuning process. By learning a data selector based on two-layer optimization, the fine-tuning data is ranked to enhance the weight of safe and high-quality data and reduce the weight of unsafe or low-quality data, thus enhancing the safety during the fine-tuning process. SEAL is tested on multiple models and datasets, and the results show its advantages in terms of validity, flexibility, and interpretability, and compared to baseline methods such as random selection, it has a improvement in the different models with a increase in win rate and maintains good performance with a range of data selection percentages, while the data selector is transferable and reduces computational complexity.", "questions": "See Weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a security-enhanced fine-tuning framework named SEAL, specifically designed to mitigate the negative impacts on model security alignment during the fine-tuning process.\n2. SEAL employs a novel two-layer optimization method to learn a data selector, which effectively ranks fine-tuning data. This process increases the emphasis on safe and high-quality data while reducing the influence of unsafe or low-quality data, thus improving fine-tuning safety.\n3. The data selector used in SEAL is transferable across different contexts, reducing computational complexity and enhancing its applicability.", "weaknesses": "1. Lack of novelty. This paper presents a method for training data selectors and weighting losses, which is very straightforward. At the same time, the paper spends a lot of time on the optimization algorithm, but the optimization algorithm itself lacks improvement and innovation compared to the referenced papers. Overall, the article's approach lacks novelty, both in the idea itself and in the optimization method.\n2. For all experiments, there was no adjustment for the variance of performance given after randomization of seeds. Especially for the randomized selection method, a large number of replicated experiments are necessary. Also, there are sampling methods for the generation of large language models, and randomness is introduced here as well. Overall, the lack of repeated experiments leads to unconvincing results.\n3. Apart from the baseline chosen for this paper, I don't know why the authors didn't design additional but direct baselines, such as directly letting the model be given weights or rankings through the prompt engineering.\n4. The results now lack insight, for example, the HEX-PHI data actually contains 11 categories, and it would be interesting to analyze the different categories in detail and explore the limitations of the sample selection algorithm, or an analysis on other benchmark tasks to explore the balance between security and performance would be insightful. A more in-depth analysis will help us better understand the effectiveness and flaws of the method, which is very meaningful.\n5. LoRA introduces additional parameters, so it is difficult to directly attribute for performance improvements. Therefore, I believe that full parameter fine-tuning should be used for all methods. Also, the models do not perform well. Sometimes it is not even better than Random.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uLda4cxhfq", "reviewer": "Reviewer_TNmc", "summary": "This work aims to enhance the fine-tuning LLMs’ safety capabilities while maintaining downstream performance. Therefore, authors design SEAN, a novel believes optimization framework to up-rank safe and high-quality fine-tuning data while down-ranking unsafe or low-quality data.  LLMs trained with SEAL demonstrate superior quality over multiple baselines, achieving an 8.5% and 9.7% increase in win rates compared to random selection on the LLAMA-3-8B-INSTRUCT and MERLINITE-7B models, respectively.", "questions": "1. Could you compare the results of training the selector and LLMs separately? It is intuitive that this method seems to be more effective.\n2. Could you further explain the results in Figure 3 and Figure 4. I am confused about why the performance of SEAL on the SLIMORCA dataset is not always superior to the baseline DSIR (Sometimes even worse).\n3. From the current results, it is difficult to know whether the win rate improvements of SEAL are due to enhanced safety or better performance on downstream tasks. Could you provide more explanation about which datasets focus more on safety evaluation versus performance evaluation? Additionally, can you present more distinguishable results to show the model comparisons in terms of safety and performance, respectively?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized, and the methods and experiments are easy to understand.\n2. The work attempts to address the safety concerns arising from LLM fine-tuning strategies, which is a very interesting and valuable issue.\n3. The method of bilevel optimization is well-suited to the proposed problem.\n4. The experiments roughly demonstrate the validity of the proposed method.", "weaknesses": "1. It would be better to provide more results, such as ablation studies. For example, comparing the method of fine-tuning the selector alone would be useful.\n2. Some experimental results are not sufficiently explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXIYpbFpXe", "reviewer": "Reviewer_FGMp", "summary": "This paper introduces SEAL, a novel framework to enhance safety for fine-tuning of large language models. Specifically, SEAL learns a sample ranker based on a bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Extensive experiments on natural language processing benchmarks demonstrate the effectiveness of SEAL.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The research direction of this paper is crucial for safe model fine-tuning. The idea of the proposed SEAL seems to be simple and effective in addressing the issue of data ranking. The paper is well-written and easy to follow.", "weaknesses": "1. The reviewer noted a performance discrepancy between PEFT training for 7B models and full fine-tuning of 2.8B models. This might be due to the fact that only the k and v matrices are trained during PEFT training, which, while enhancing robustness to catastrophic forgetting, can amplify or diminish the impact of fine-tuning on safety. Therefore, It is  recommended that the authors conduct an ablation study comparing LoRA fine-tuning that incorporates weights in both the Feed Forward Network (FFN) and Multi-Head Self-Attention (MHSA) blocks with their current PEFT approach. Additionally, an analysis discussing how various LoRA configurations affect the safety-performance trade-off would be beneficial.\n\n2. The author used the memory-efficient variant of SEAL by default in the experiments. To provide a more comprehensive understanding of the impact of this choice, I suggest the authors perform a quantitative comparison of memory consumption and performance between the standard and memory-efficient SEAL variants. Including a table or figure that illustrates this comparison would help clarify the effectiveness of both approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)