Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
Authors: Zixuan Hu
Li Shen
Zhenyi Wang
Yongxian Wei
Dacheng Tao
Venue: NeurIPS 2025 spotlight ()
Keywords: Harmful fine-tuning; Large Language Models; Data schedule; Bayesian optimization
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a principled framework Bayesian Data Scheduler (BDS), which is an adaptive tuning-stage defense strategy against harmful fine-tuning with no need for attack simulation.

Abstract: Harmful fine-tuning poses critical safety risks to fine-tuning-as-a-service for large language models. Existing defense strategies preemptively build robustness via attack simulation but suffer from fundamental limitations: (i) the infeasibility of extending attack simulations beyond bounded threat models due to the inherent difficulty of anticipating unknown attacks, and (ii) limited adaptability to varying attack settings, as simulation fails to capture their variability and complexity. To address these challenges, we propose Bayesian Data Scheduler (BDS), an adaptive tuning-stage defense strategy with no need for attack simulation. BDS formulates harmful fine-tuning defense as a Bayesian inference problem, learning the posterior distribution of each data point's safety attribute, conditioned on the fine-tuning and alignment datasets. The fine-tuning process is then constrained by weighting data with their safety attributes sampled from the posterior, thus mitigating the influence of harmful data. By leveraging the post hoc nature of Bayesian inference, the posterior is conditioned on the fine-tuning dataset, enabling BDS to tailor its defense to the specific dataset, thereby achieving adaptive defense. Furthermore, we introduce a neural scheduler based on amortized Bayesian learning, enabling efficient transfer to new data without retraining. Comprehensive results across diverse attack and defense settings demonstrate the state-of-the-art performance of our approach. Code is available at https://github.com/Egg-Hu/Bayesian-Data-Scheduler.

Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
Zixuan Hu
Li Shen? Zhenyi Wang
Yongxian Wei
Dacheng Tao
NNayyang Technological University "Shenzhen Campus of Sun Yat-sen University
University of Central Florida 4 Tsinghua University
ZIXUANO14@e ntu. edu. sg mathshenl iegeshhenn@gmaillem @gmail com zhenyi. wang@ucf edu weiyx230mails, tsinghua. edu. cn
dacheng taanggaitomgmmll com
Abstract
Harmful fine-tuning poses critical safety risks to fin--uning-as-aaservvie for large language models. Existing defense strategies preemplively build robustness via attack simulation but suffer from fundamental limitations: (i) the infeasibility of extending attack simulations beyond bounded threat models due to the inherent difficulty of anticipating unknown attacks, and (ii) limited adaptability to varying attack settings, as simulation fails to capture their variability and complexity. To address these challenges, propose Bayesian Data Scheduler (BDS), adaptive tuning-stage defense strategy with no need for attack simulation. BDS formulates harmful fine-tuning defense as Bayesian inference problem, learning the posterior distribution of each data point's s safety attribute, conditioned on the fine-tuning and alignment datasets. The fine-tuning process is then constrained by weighting data with their safety attributes sampled from the posterior, thus mitigating the influence of harmful data. By leveraging the post hoc nature of Bayesian inference, the poste- rior conditioned on the fine-tuning dataset, enabling BDS to tailor its defense to the specific dataset, thereby achieving adaptive defense. Furthermore, we introduce a neural scheduler based on amortized Bayesian learning, enabling efficient transfer to new data without retraining. Comprehensive results across diverse attack and de- fense settings demonstrate the state-of-ther art performance of our approach. Code is available https: / gighhu..... com/Egg-an BEgggggHBBayae Hu/Bayesian Data- Scheduler.
Introduction
Fine-tuning-as-ae as-a-servic has become a widely adopted paradigm among mainstream LLM providers (e.g., OpenAl²), enabling the delivery of customized language model solutions. In this paradigm, users upload demonstration data representing their desired behaviors, and providers fine-tume the foundational models on their behalf. Despite its growing adoption, recent red teaming studies have uncovered critical vulnerability: harmful fine-tuning [48, 29]. Harmful fine-tuning refers to that the presence of even small fraction of harmful data in user-provided datasets can cause fine-tuned models to deviate from the safety alignment established during pre-training. This vulnerability introduces substantial attack surface and also undermines the reliability and quality of the service.
Existing defenses [29, 33, 52] primarily rely on a preemptive manner, aiming to build robustness through attack simulation prior to fine-tuning. However, these defenses suffer from fundamental limitations: (i) the infeasibility of extending attack simulations beyond bounded threat models due to the inherent difficulty of anticipating unknown attacks; and (ii) limited adaptability to varying attack
* *Correspondigg author: Dacheng Tao
Fine-tuning API by OpenAl: https://platfon www.efforr..oooa....oommmmm www.ccooccocccccccccoocooooo..ct wwtp////www/w///////////ooeoe//e weaaiddee/iii-etuunnnngg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Harm ful Ratio = 0.2 Harmful Score 1.30
Harm ful Ratio = 0.9 Harm ful Score = 1.50 Bessity
ty
data
All turing daliseel
tuning dalasel
huniig datsoet
in tavnn dotase?
Iershtrre
Tusing INrrattoss
in Nuving donceet
fanin dofave
Inferred Data Weight (Higher Indicates Greater Likelihood of Being Benign Data)
Figure 1: For encountered datasets with unknown and different harmful ratios, BDS adaptively schedules data into higher and lower weight groups during tuning (largest panels). To verify correctness of our data scheduling, we observe that most truly benign data indeed receive higher weights (top right panels), while almost all truly harmful data consistently receive lower weights (bottom right panels).
settings, as the preemptive attack simulation fails to capture the variability and complexity of post hoc attacks. Consequently, their performance deteriorates sharply as the harmful data ratio increases or the attack strategy varies, rendering them ineffective against unpredictable attack surfaces.
To address these challenges, we propose Bayesian Data Scheduler (BDS), an adaptive fine- mhn-ttnnng-- stage defense strategy with need for attack simulation. BDS formulates harmful fine-tuning defense as a Bayesian inference problem, learning the posterior distribution of each data point's safety attribute, conditioned the fine-tuning and alignment datasets. The fine-tuning is then constrained by weighting data with the safety attributes 3 sampled from the posterior, thus mitigating the negative influence of harmful data. Due to the post hoc nature of Bayesian inference, the posterior is conditioned on the encountered fine-tuning dataset, which enables BDS to precisely tailor its defense the specific dataset, thus achieving adaptive defense. Additionally, as the posterior of safety attribute is datapoint-wise, i.e., p(w; =5¹). independently learning distribution for each data point scales with the dataset size and requires relearning for new data. To address scalability and transferability issues, we further introduce a neural scheduler based on amortized Bayesian learning [51], enabling efficient transfer new data without retraining.
We conduct comprehensive experiments on five diverse downstream datasets using three representative LLM architectures under wide range of attack and defense settings. Results show that our approach achieves ttate-of-the--t (SOTA) performance, with a remarkable 74.4% improvement at a high harmful ratio of 0.9 and and an average boost of over 50% across ratios from 0 to 1. Moreover, BDS consistently maintains a remarkably low harmfulness score (around 1) across a wide range of advanced attack dynamics, including benign [48], out-of-distribution (OOD), and identity-shifting [48] attacks, demonstrating superior effectiveness, adaptiveness, and robustness.
To the end, we summarize our contributions as follows:
For the first time, we formulate harmful fine-tuning defense as a Bayesian inference problem, offering a principled framework BDS achieving adaptive defense with no need for attack simulation.
We propose two BDS implementations, with the neural version enabling efficient transfer to new data without retraining. Leveraging post hoc nature of Bayesian inference, BDS learns the posterior distribution of data weights conditioned on specific datasets, thus achieving adaptive defense.
Comprehensive experiments diverse attack and defense settings demonstrate the SOTA perfor- mance and superior adaptiveness of our proposed method.
2 Problem Definition
Scenario.
The scenario considers users uploading personalized fine- tnnettunig dataset Dt. which the service provider uses to fine-tune their model 0. Once fine-tuned, the personalized API is returned for user-specific applications.
Threat model. Harmful fine-tuning arises when an attacker deliberately uploads a fine-tuning dataset Dr. that contains mix of benign and harmful data. Specifically, the dataset Dr. consists
In this paper. "safety attribute" and "data weight" used synonymously.

---

of a proportion of harmful data and 1 of benign data, such that Dn. = D benign V D D The attacker, acting as a user, has the capacity to control the composition of the fine-tuning dataset, including the proportion and content of harmful data, without requiring knowledge of the model's s architecture or parameters. On the other hand, the defender, acting as the service provider, has the capacity to control the fine-tuning process.
Assumption. We assume that the service provider maintains an alignment dataset, denoted as Dsafe (harmful prompt-safe answer pairs). The availability of such an alignment dataset has been previously discussed in previous works [52, 57] and it available in publicly available resources (e.g., BeaverTail [35]). While prior studies require an additional harmful dataset for attack simulation, constructing such a dataset is difficult due to lacking prior knowledge about unknown and varying attack data Dharmful, Our work overcomes this limitation with no need for a simulated harmful dataset.
3 Related work.
Mechanism study of harmful fine-tuning. Recent works have investigated why LLMs are highly sensitive to harmful fine-tuning. (i) Adversarial perturbations: Harmful fine-tuning can introduce adversarial shifts in model parameters [29, 46] or embeddings [33], leading to degraded safety alignment. (ii) Structural vulnerabilities: Other works highlight that certain layers or modules are more critical for maintaining safety than others [21, 37, 47]. (iii) Catastrophic forgetting: Safety alignment degradation has also been linked to catastrophic forgetting [43], where alignment knowledge tends to be lost under the sequential training paradigm [6, 13].
Harmful fine-tuning attack. On the attack side, [48] demonstrate that even fine-tuning solely on benign data can inadvertently weaken a model's safety alignment. Subsequent work [20] further identifies specific subsets of benign data that are particularly prone to degrading model safety after fine-tuning. [18] introduce covert malicious fine-tuning. which enhances the stealthiness of such attacks, while [32] construct harmful data designed to deliberately circumvent moderation guardrails.
be categorized into three main categories, including alignment stage [33, 52, 58, 42, 29, 40, 41, 7, 75], fine-tuning stage [44, 3, 31, 61, 79, 55], and post-finettuiing stage solutions [76, 12, 28, 70, 62]. The method proposed in this paper should be classified into fine-tuning stage solution. However, it differs fundamentally from prior work in that we propose a loss-based data scheduling mechanism derived from Bayesian inference principles, whereas previous approaches typically perform hard-label data selection based on less effective heuristic rules [15, 5] or manually tuned thresholds [8, 55]. Moreover, unlike the current SOTA method Booster [29], our method does not require attack simulation, making it more practical and efficient while achieving strong defense performance and adaptability.
Harmful fine-tuning defense. On the defense side, existing defense methods
more detailed and extended review related work is provided in App. A.
Methodology
In this section, we introduce the Bayesian Data Scheduler method. Specifically, we outline framework in Sec. 4.1. Following that, we detail two implementations of BDS, including Bayesian Scalar Scheduler and Amortized Bayesian Neural Scheduler in Secs. 4.2 and 4.3.
4.1 Bayesian Data Scheduler Framework
p(m (0. - 5im -X0e)
We frame the harmful fine-tuning defense as a Bayesian inference framework, with its underlying graphical model depicted in Fig. For clarity, the discussion here focuses on the left panel, with the right panel deferred to Sec. 4.3. Each data point zt in the fine-tuning dataset Dp. is associated with an- other latent variable w; E R, indicating its intrinsic safety attribute (i.e., z - w.)) To mitigate the nega- tive effect of potentially harmful data, the tine-tuning process (w,, xft) 0 is constrained, where the con- tribution of each data point z!" to update the model 0
Bayesian Scaler Schedkler
Amortized Bayesian Neural Scheduler
Graphical models for the Bayesian Scalar Scheduler (see Sec. 4.2) and Amortized Bayesian Neural Scheduler (sce Sec. 4.3).
Figure
3

---

Goal: Bayesion Scalar Scheduler, sampling (w. posteriori distribution p(w. 01 Peat) For Amortized Bayesion Neural Scheduler, sompling (e) 0) from pasterion distribution p(p. 0|P Batt)
Step Infer data weight Schedule data
Beyesion Date Scheduler
@ Update LLM on Peese weighted Dg. via Eq.(7)
Boyesion Scalar Scheduler
Par
①
[Mg. Mppll
Step
PLS
Languoge Model 0
Amorlized Boyasion Neural Schedaler (..2) 6iw, 00)
Step @ Update scheduler w + on updated via Eq.(8) or (q.(1I)
o(
Hight Hightr-wight data point
Lower-weight dats point
Repeat Step @ ( @ for iterations until convergence,
(w",) (407)) theoretically guaranteed posterior samples from p(w, Py. Penn) p(4. Dests) 8 is directly used as the custommized model for user-specific opplications without requiring further adjus tments.
Figure 3: Pipeline of BDS. Step 1: BDS first infer the weight of each data point, indicating its safety attribute. Step 2: BDS updates the L.M with weighted data via Eq. (7). Step BDS update the scheduler up or Φ via Eq. (8) or Eq. (11). Repeat steps 1-3 for T iterations until convergence and (w(T), (( or (Φ el
be theoretically guaranteed posterior samples. ( is directly used as the customized model for user-specific applications without requiring further adjustments. For clarity, the pseudocode for the BDS algorithm is provided in App. C.
is modulated by its safety attribute wy. The edge - Deade represents the generation of the target parts of the alignment dataset Deater modeling the likelihood of observing Daate given 0. The data points assumed to be i.i.d., implying that p(w) II, p(w:) holds, where w; is the i-th component of vector w. The conditional independence Daafe 1 (w. Da) 0 naturally holds from the structure of the graphical model.
Goal. Our goal is to infer the posterior distributions unobserved variables (unshaded nodes): i.e., the data weights and the model parameters 0, conditioned on observed variables (shaded nodes), i.e., the safety alignment dataset Deafe and the fine-tuning dataset Dr. Formally, this corresponds to estimating the joint posterior: p(w, 0 Di, Deate).
4.2 Bayesian Scalar Scheduler
Posterior decomposition. address the computational intractability of directly inferring the posterior distribution p(w, 0 DD, Danfe), we decompose it as follows:
(0, w Da, Deate) x p(Deate 0) p(De. 0, w) p(Dn ()) 100,au w))
(1)
0) in Eq. (1) represents the likelihood of observing the alignment dataset Daate given the model parameters 0, quantifying how well the model aligns with the alignment dataset. Based on the conventional trick [82], this likelihood can be formulated in terms of the loss function as follows:
Detailed derivations are provided in App. H.l. The term p P(Deaee
Paatfl
p (Datto 0) x II exp (-e(s,ufe,)) 0));
(2)
where |Deatel denotes the size of the dataset and E(-) denotes the loss function. Next, the second term p(Dn, 0, w.w) in Eq. (1) represents the likelihood of the fine-tuning dataset Dr. given the model parameters 0 and the safety attributes w, where the contribution of each data point z is modulated by its safety weight we:
PPa
p (Dr O, w)
II exp (-o(uss) . (()))
(3)
where o()) refers to the weight transformation function, which is discussed in detail in Sec. 4.4.
The term p (Dt ww)= in Eq. (1) represents the likelihood of the fine-tuning dataset Dr, given the safety attributes w. Since it is infeasible to directly model the relationship D - w, we employ

---

marginalization over & and approximate it using the maximum a posteriori (MAP) estimate (Detailed derivations are provided in App. H.2.):
[Dal
p (Du w) x II exp (o(w)))&(<()
8 (z.:: s.t., θ = max ma (pp@ Dn, w))
(4)
Efficient Posterior inference via Stochastic Gradient Langevin Dynamic (SGLD) sampling, Due to the intractability of obtaining closed-form solutions for Eq. (1), we employ an efficient posterior inference technique based stochastic- -raadieng Markov Chain Monte Carlo (SG-MCMC) [45]. Specifically, we utilize Stochastic Gradient Langevin Dynamics (SGLD) [69] to sample from the posterior p(w, 0 DD. D,ate), inspired by [49, 73, 10). By iteratively updating the system under the Langevin dynamics framework, we can obtain theoretically guaranteed posterior samples once the process converges (convergence analysis is provided in App. G.2).
[0, w| +- 10, w) + " Veo logp (0, w Da, Deate) + +VTT,
(5)
2
where represents the step size, and e ~ C I) denotes Gaussian noise introduced to inject randomness. Based on the posterior decomposition in Eq. (I), the gradient of the log- log-posterior can be expressed as a summation:
P log p (0, Dn, Deate) Vo., p( 0 + Ve,w log ((nn
10 w)
(6)
+ + ( -==--+++ Vo,w w log p(O w));
where placeholder denotes independence between the gradient term and the corresponding variable. To improve the efficiency of gradient computation, SGLD substitutes the full-batch likelihood in with minibatch approximation. Using minibatch B, we reformulate Eq. (6) using the decompositions in Eqs. (2) to (4), yielding the following equations.
0 0 Vo log
w)
[Deatel Baatel
Σ aqule aaf
(( ;0)
[Dzl [Bn]
Σ
[o(we) [x] tevn.
of CB
V.. logp(w)
[Dul Bnl
a(ws)) ((zz):)) -((=):))e((=;:))))))))) +evt
2
(8)
2 w
17 ! +
logpp(w) p(w)
Diul Br
Σ
[o(we)) [((u)))((2f:00) + evT..
2
=fE
where we approximate ((ff;; 0) R 0.This is justified as = arg maxg Ep(Dalw) (p Dr. w)
(see Eq. (4)), which represents the MAP estimate of the model 0 given the dataset Dr. Intuitively, 0 is optimized to perform well on Dr, resulting in a near-zero loss.
Intuition behind weight update in Eq. (8) (see Fig. 4). By omitting the prior term, noise, and constants, the update of the i-th component of w in the non-approximated Eq. (8) simplifies to:
<(2
awfee model Eq.(7)
Sofety-agnoshic model in (q((4)
Data point 2.
Eq
w; + w. - (l(=,,; 0) (< 0)) Vw, (w;).
Higher loss gop
Lower gap
In this form, the loss gap scales the gradient term V., a (wr), where larger loss gap leads to a sig- nificant reduction in wi, ultimately resulting in smaller weight. Within the loss gap, is derived via Eq. (7), which optimizes over both Daate and the weighted Di. thus referred to as the safety- aware model. In contrast, is obtained through Eq. (4), fitting only the weighted Dn. thus re- ferred to as the safety-agnostic model. If a data point 2. produces a large positive loss gap, indicates poorer alignment with the safety-aware model 0 compared to the aagety-a model Ô.
in Eq.(4)
E Dr
Higher weight Wt
Lower weight My
0) ((.).0)
wy
Figure 4: Intuition behind weight update in Eq. (8).
This suggests that z is likely misaligned with the safe dataset Deate, ultimately resulting in larger weight reduction, and thus a smaller assigned weight.
5

---

4.3 Amortized Bayesian Neural Scheduler
The Bayesian Scalar Scheduler introduced in Sec. 4.2 provides a strong foundation but leaves room for further enhancement in two key aspects: (i) Scalability: Inferring p(w; 2, Dsate) for each data point scales with dataset size. (ii) Transferability: Without explicitly modeling z!" + wi posterior inference must restart from scratch when new data added.
To address the scalability and transferability issues, we introduce the Amortized Bayesian Neural Scheduler based amortized Bayesian learning [51]. The core idea is to amortize the effort for inferring datapoint-wise posterior p( W1 x Deate) into inferring just one posterior of a neural network p(Φ Dr. D. As shown in the right panel of Fig. 2, the neural network & p( Dn: Dante) is shared across all data points, enabling each p(w; 2 Damte)) to be inferred efficiently via forward pass through the network conditioned on 2!" For simplicity, we define p(w;; z!" Dante) as Dirac delta distribution 8(-), whose single value is determined by the output of a neural network (= ()) with zft input. After introducing Φ, the datapoint-wise posterior is expressed as:
p(w, Φ ) == 8 (w: - .Vzl! d)).
(9)
In this way, only a single posterior over the neural network parameters (i.e. Dn. Deate)) )) needs to be inferred, and the posterior of weights for new data can be inferred seamlessly through forward passes without retraining.
After introducing Φ, the updated probabilistic graphical model is illustrated in the right panel of Fig. 2. Consequently, our inference objective shifts from p(w. 0 Dn. Daate) ) to p(ø, 0 Da. Deate), which can be decomposed as follows (Detailed proof provided in App. H.3):
p(0,c & (Da, Dante) a p(Daato
0) p Dr.
O, O,w) p(Dn w) p(0.0 Da),
(10)
s.t., w = (Dr Φb a Ni*f (b) ",,,,.,,NN NW( ()
Similar to Eq. (8), we derive the update rule for db
Φ - ② + 2 V logp(d)
[Dn]
Σ
[oN(= の)) - +eVT.
(11)
Bn
4.4 Data Weight Transformation
Recall the scheduler updates in Eqs. (8) and (11). We identify a key factor that significantly impacts the performance of BDS: the weight transformation function a (w;). As shown in Fig. 5, different transformation functions result in varying sampling trajectories of w for benign and harmful data, ultimately leading to distinct defensive capabilities (see Tab. 14). For clarity, we first introduce the definition of the SGLD Sampling Trajectory:
Definition 4.1. (SGLD) Sampling Trajectory) The SGLD Sampling Trajectory refers to the se- quence of states generated during the iterative sampling process of SGLD. This sequence forms a Markov chain, where each state transition depends solely on the previous state. For the distribution p (w. 0 Di., Dante), the trajectories the variables w and f can be defined as:
Tru = w (0) + + w w (t+1)
w (t)
+ ! V +eV
(12)
Tra 000
T 0 (0))
V +eV7-
2
Here, we use V and Vo to denote the gradient terms in Eqs. (7) and (8). We next analyze the effect of different transformation functions on the term V., in Eq. (12).
We consider three types of weight transformations: identity, sigmoid, and softmax, which are formally defined as:
(w:) = wi (identity),
(ww) = 1 exp (-wi) (sigmoid),
(13)
exp(w:)
ō(w:)=
(softmax).
Pri exp(w,)
By differentiating in Eq. (8) (excluding prior and constant coefficients), we derive that under the identity transformation, w updates in a monotonically decreasing manner:
V = - (z 0) < O.
(14)

---

dentty
Figure 5: Effect of weight transformation SGLD sampling trajectories of for benign and harmful data, respectively. For clarity, weights post-softmax are scaled by (Dsatel-
Similarly, under sigmoid, the weight w also exhibits monotonically decreasing update behavior:
V., (x)) (1 (x))) L (z.: <..
(15)
Since the gradient of is proportional to the negative loss, the weight suboptimality under both identity and sigmoid transformation primarily arises from the uniform treatment of benign and harmful data, resulting monotonically decreasing weights for both. As demonstrated in Fig. this causes the scheduler to assign reduced weights to both benign and harmful data to preserve the model's inherent safety alignment, which equivalent discarding any data and consequently leads to poor fine-tuning performance.
In contrast, under softmax, the weight updates an adaptively bidirectional manner:
V = -(w:) (e(=): 0) ((wL) 0)= 0, if if 0)
> Prl (ww) [(= 0)
(16)
if if 0) k=1 a(wx) [(=][=]):)00 0)
Detailed derivations of Eq. (16) is provided in App. H.4. Unlike identity and sigmoid transformation monotonically decreasing w based solely on the absolute magnitude of the loss, the softmax updates weights based on the relative value of the loss. This ensures that the model does not assign extremely low weights all data points. Specifically, the softmax transformation uses the weighted average loss reference: if the loss of a data point is below the reference, its weight increases; otherwise, its weight decreases. Moreover, since D Dhenigu aligns more closely with Daate than Dharmful the benign data exhibits relatively lower loss, leading to an increase in their weights (see Fig. 5).
weightee accumulation of posterior bias. We further explore weight suboptimality under identity transformation through the theoretical perspective of posterior bias, newly proposed concept with detailed formulation and analysis in Apps. 1.1 to I.5. Here, only present the core theorem below (Detailed proof is provided in App. 1.6):
Theorem 4.2 (Time- Weighted Accumulation of Posterior Bias). Let (Trw: Tra) and (Trw-, Tro-) ) denote the SGLD sampling trajectories under identity transformation drawn from the target distri- butions p (w, 0 Dr. Daate) and p (w", 0² Drt Deate: Di ). respectively. Here, D is a held-out clean tuning dataset, serving test or validation set to evaluate fine-tuning performance. Then, the following proportionality holds:
TV Tha T
w (TT)
E. Tia Tra
ΣT- (T
1)
1 w (00
(17)
PB())
Here, (T) quantifies the posterior bias at iteration and summation ΣT (T t))P(() char- acterizes the cumulative posterior bias over the entire sampling trajectory.
Remark. Notably, the time- weighte factor T - highlights the greater influence of earlier iterations on the cumulative bias. suggesting that suboptimal sampling in the early stages of the SGLD trajectory can propagate and aggressively impact the overall posterior inference.
Experiment
Datasets and models. Our experimental setup primarily follows [29, 33] to ensure fair comparison. For the alignment dataset (Deate. consisting of harmful prompt-safe answer pairs), use the dataset
7

---

Table 1: Comparison with SOTA baselines, ((Daafe)
1000 and [Drl 1000 for BDS).
Harmful Score
Finetune Accurucy
Method
Dade
Duseuful
clean
p=0.05
p=0.1
p=0.15
p=0.2
Average
clean
p=0.05
p=0.1
p=0.15
p=0.2
Average
SFT
1.30
21.90
33.70
49.30 90
61.70
33.58
81.54
91.74
93.12
92.66
92.89
90.39
Lisa
x
0.90
14.50
23.7
31.20
39.10
21.88
86.93
91.86
92.32
92.20
92.32
91.13
Repootse
/
1.20
20.70
32.10
45.60
55.50
31.02
90.25
92.89
93.00
92.89
92.89
28
Vaccine
28.20
90.83
93.69
Booster
/
1.90
4.80
8.30
14.20
25.50
10.94
92. 89
92.32
93.23
93.35
93.35
93.03
BDS
x
1,10
1.60
1.20
1,50
1,30
1.34
93.81
94.04
93.69
93.92
93.69
93.83
Table 2: Robustness for large harmful ratios p.
Method
Hamnful Score
Finetune Accumecy
p=0.3
p=0.4
p=0.5
p=0.6
y=0.7
p=0.8
p=0.9
p=1.0
y=0.3
p=0.4
y=0.5
p=0.6
p=0.7
y=0.8
p=0.9
y=1.0
Booster
40.60
68.40
77.20
76.90
72.50
76.30
75.90
76.60
92.12
93.00
92.69
92.20
91.63
91.63
91.51
BDS
1.20
1.50
1,20
1,50
1.50
1,60
1,50
1.00
93.32
93.19
93.00
92.66
92.29
92.78
92.89
from [53], an enriched version of BeaverTails [36]. To simulate harmful fine-tuning attack, the fine-tuning dataset (Dt.) is constructed by mixing proportion p of unsafe data (Dtt (Dharmful)) from
BeaverTails with 1 benign fine-tuning data (D). (Dbemis"), resulting in a total size of [Pu.]. Fine- tuning tasks are considered on SST2 [56], AGNEWS [80]. GSM8K [9], AlpacaEval [38], and GEM benchmark. Model architectures include Llama2-7B [60], Gemma2-9B [59] and Qwen2-7B [74]. Default settings are p = 0.1 and [Dxl] 1000 (Pra 700 for AlpacaEval).
Metrics. We adopt two evaluation metrics for assessing model performance, following [29,33]: 33]:
Finetune Accuracy (FA): The accuracy on the testing dataset of the corresponding fine-tuning task. Details on evaluation procedure are provided in App. B.2.
Harmful Score (HS): Using the moderation model from [36], we classify model outputs as harmful or non-harmful. The harmful score defined as the proportion of unsafe outputs.
For harmful score calculation, we sample 1000 instructions from the testing set of BeaverTails [36]. Finetune accuracy is evaluated using 872, 1000, 1000, 1000 and 104 samples from the fine-tuning datasets SST2, AGNEWS, GSM8K, GEM, and AlpacaEval, respectively.
Baselines. We compare our method with five representative defense baselines, including the SOTA Booster [29], Vaccine [33], Repnoise [52], Lisa [31], and SFT (Supervised Fine- Tunnng) [29]. Descriptions and implementation details of each baseline are provided in App. B.1.
Implementation details. For fine-tum tunettuning we adopt LoRA [22] for efficient fine-tuning, following [29, 33, 21J. Fine-tuning is performed using the FusedAdam [50] with a learning rate of 1 x 10- and a weight decay of 0. as recommended by [29]. The training involves 20 epochs with a batch size of 10. The neural scheduler is implemented using a lightweight 125M Fairseq-Dense model [2] with an added trainable linear head. The scheduler's 's learning rate is set to 5 x 10-3 for scalar scheduler and x 1 for neural scheduler. Unless otherwise specified, we use the scalar scheduler as default. More implementation details are provided in Apps. B.3 and B.4.
5.1 Main Results
Comparison with SOTA defense baselines. Tab. presents comparison of BDS with several SOTA baselines across harmful ratios ranging from 0 0.2. BDS consistently outperforms all baselines in both defensive and fine-tuning performance, achieving significant 9.60% reduction in average Harmful Score and a 0.8% improvement average Finetune Accuracy. Existing baselines struggle to consistently mitigate harmful data influence: Simulation-base baselines like Booster and Vaccine enhance robustness by attack simulation, whose effectiveness diminishes significantly as the harmful ratio increases; RepNoise attempts to unleam harmful knowledge but it inevitably recovers under higher harmful ratios; Lisa mixes alignment data with fine-tuning data but fundamentally lacks a deweighting mechanism and thus requires alignment data to scale with the unknown harmful ratio.
Notably, when the harmful ratio is 0 (i.e., benign attack [48]), BDS achieves the highest and well- preserved fine-tuning performance, whereas SFT suffers significant drops. This can be explained by the belifulnees-hurmlesseer alignment but reduces its plasticity [1 [11] to sufficiently adapt and learn helpfulness during fine-tuning, thus leading to lower fine-tuning accuracy. As the harmful ratio increases (e.g., p : - 0.2), fine-tuning accuracy of SFT even rises because the introduced harmful data act counterexamples
trade-off:: SFT endows the model with strong harmlessness after
8

---

Table 3: Robustness across fine-tuning datasets Table Robustness across model architectures
((D
1000, [D = 1000, p = 0.1).
(/Danfel = 10000, [D 1000, 0.1).
Method
SST2
ACNews
GSMBK
Alpoen
Method
1.ama2
Gemmaz
Qwen2
Aug
HS L
FA
[S
FA
115 1
F
115 L
FA
TES 4
FA.
F15
FA
115
FA
TBS L
FA
SPT
$3.70
93.12
30.70
55.90
14.80
15.20
40.70
45.67
SET
33.70
92.12
64.30
94.50
25,50
94.84
41.17
94,15
1isa
23.70
92.32
16.80
83.20
5.10
12.00
14.30
41.35
1isa
23,70
92.32
30.80
94.04
9.50
93.90
21:33
93.43
Repnuise
32.10
93,00
27.30
$5.50
16.10
36.50
41.83
Repnoise
32.10
90.00
63.60
94.30
33.90
94.61
43.20
94,04
Vaceine
28.30
43.40
44.71
Vaccine
28.50
93.69
45.00
16,80
92.55
93.31
Beuster
93.23
17.
Booster
93.23
11.20
94.19
BDS
93.09
1.10 89.10
18:30
2.30
93.09
1,30
0.90
90.72
Table 5: Robustness for different [Drll-
Table 6: Robustness for different [Daafel
Method
Haraiuul Score J
Finetane Accuracy" ?
Method
Harfful Score
Finetune Accuracy
500
1000
2000
500
1000
1500
2000
500
1000
1500
100
1500
33.60
92.66
95
94.04
62.20
34.60
8.10
93.23
92.
BDS
1.20
1.20
0.90
1.20
93.35
93.69
93.12
94.28
BDS
1.70
1.20
1.20
1.30
93.58
93.92
93.69
93.69
Table 7: Effectiveness and transferability of neural scheduler. A + B denotes that the neu- ral scheduler is trained on dataset A and directly applied to assign data weights on dataset B. [HS/FA] (eWWDD denote metrics under in-domain and out-of-domain transfer settings.
Method
8812
S8T2 - 8812 (unseen)
S812
AGNEWS
BS
FA
HSimww_ID)
FAlnewIID
DIStew_OOD)
FAinewJOOD
Scalar Scheduler
93.69
Neural Scheduler
1.70
93.69
2.50
93.23
2.80
89
that prompt the model to "unlcarn" part its harmlessness, thereby restoring plasticity and improving its capacity to learn helpfulness. Our method achieves a better helpfuhness-harmlesssss trade-off by jointly optimizing the safety alignment and fine-tuning objectives, as shown in Eq. (7). This joint optimization resembles multi-task learning paradigm, which more effectively balances the competing objectives of harmlessness and helpfulness than the sequential learning paradigm used in trade-off are provided in App. F.2.
SFT. Further discussions on the
Robustness for high harmful ratios. Prior evaluations [29, 33] limited to low harmful ratios, as existing methods struggle under high harmful ratios. To highlight our robustness, we conduct experiments under high harmful ratios ranging from 0.3 to 1.0. As shown in Tab. 2. BDS significantly surpasses the SOTA Booster across all tested ratios in both harmful score and fine-tuning accuracy. At high harmful ratios from 0.5 to 1.0, BDS achieves a significant reduction in harmful score by over 70% and delivers a near 1 % improvement in fine-tuning accuracy. These results highlight the exceptional robustness and reliability of BDS, even under highly adversarial conditions.
Robustness for diverse fine-tuning tasks. To assess generalization, we evaluate BDS on four fine-tuning datasets in Tab. 3. Results show that BDS consistently maintains exceptionally low harmful scores (around 1) across all tasks, while existing baselines exhibit severe instability and poor generalization. For instance, while Vaccine achieves harmful score of 3.7 on GSM8K, it catastrophically underperforms on AlpacaEval with a harmful score of 43.30. These results highlight BDS's adaptability and robustness to complex tasks like GSM8K and AlpacaEval while ensuring harmlessness. More experiments on complex text generation tasks are provided in App. D.6.
Robustness for different model architectures. To further assess generalization, we evaluate BDS on two SOTA architectures, Gemma2-9B and Qwen2-7B. Tab. 4 demonstrates BDS consistently achieves low harmful scores, while existing baselines exhibit severe instability. While Booster attains a harmful score of 1.6 on Qwen2-7B, it collapses on Gemma2-9B, surging to 11 11.2. These highlight BDS's adaptability, ensuring consistent SOTA performance across diverse model architectures.
Robustness for different sizes of [Pul To validate robustness to (Drll size, we evaluate its impact while keeping (Deaeel and the harmful ratio p fixed by default. As shown in Tab. 5, existing bascline suffers a sharp decline in defensive performance as [Drt] increases. Booster's 's harmful score surges dramatically from 3.8 to 33.6, exposing its inability to handle larger datasets. In contrast, BDS demonstrates strong robustness, maintaining a consistently low harmful score of around I regardless of (Dal size. This highlights BDS's 's reliability in defending harmful fine-tuning scale substantially. Robustness for larger dataset size is also demonstrated in App. D.5.
dataset sizes
Robustness for different sizes of [D To validate robustness to /Deafel size, we evaluate its impact while keeping [Dr,) and p fixed by default. As shown Tab. 6, existing baselines collapses as [Danfel decreases, with defensive performance deteriorating sharply. In contrast, BDS exhibits remarkable

---

Horm/ul Rotto 0.3
Harm ful Shore 1.20
Harm fui Battoo 0.8
Hormfsl Shore 1.40
Count
Count
Weigitt (pre transforration)
Truly benign data
Wagiht (pre transformation
Truly benign dasia
Truly harmful datt
Truly harmful data
Truly benign Truly data
Weight
Truly benign data Truly cata
Weight
Figure 6: Data weights distributions across varying harmful ratios. More visualizations of weight distributions and scheduling dynamics are provided in Fig. 7 of App. D.1 and Fig. 8 of App. D.2.
resilience, consistently maintaining a harmful score below 2, even with only 100 alignment samples. These results underscore BDS's robustness in defense, even under limited alignment data.
Effectiveness and transferability of neural scheduler. To evaluate the effectiveness of the neural scheduler, compare it with the scalar scheduler. The results in Tab. 7 show that the neural scheduler performs comparably to the scalar scheduler in both defensive and fine-tuning performance, achieving harmful score below on SST2. To further evaluate its transferability, on unseen in-domain and out-of-domain datasets. The neural scheduler trained on the seen data is directly used to assign data weights to unseen data, which is subsequently employed for fine-tuning the LLM. As shown in Tab. 7, the neural scheduler generalizes effectively to both in-domain and out-of-domain unseen data, maintaining harmful score below 3. These results highlight the strong effectiveness and transferability of the neural scheduler without the need for retraining, supporting its capacity to learn transferable data safety attributes [16].
conduct experiments
Robustness for advanced attack: OOD and ISA. To evaluate the superior adaptiveness of our method under diverse attack dynamics, we conduct experiments against several challenging attack strategies: OOD attacks, and identity shifting attacks (ISA) [48]. Detailed results and discussions provided in Apps. D..3 and D.4 due limited space. Our approach significantly outperforms SOTA baselines, maintaining low harmfulness ratio (around 1) across all attack strategies. This adaptiveness fundamentally stems from our loss- data scheduling mechanism. Regardless of the attacker's strategy-wheite OOD or ISA-harmful data tends to exhibit shared "unsafe behavior" (i.e., incurring higher loss within the loss landscape of safety-aware models [46]), thus being assigned with lower weights. This also mirrors the shared "safe behavior" observed across benign datasets (see Tab. 7), and supports the existence of transferable data safety attributes discussed in [16]. This mechanism enables our method to effectively adapt to various attack dynamics without explicit attack simulation.
Visualization of adaptiveness. Weight distributions (Fig. and Fig. 7 in App. D.1) and scheduling dynamics (Fig. 8 in App. D.2) demonstrates BDS adaptively and correctly assigns higher weights to truly benign data and lower weights to truly harmful data, across varying harmful ratios.
Ablation studies each component configuration are provided in detail in Apps. E. to E.3 and E.5.
Discussion. We offer comprehensive discussions on the adaptiveness of BDS, helpfulness- harmlessness trade-offs, limitations, and social impact in Apps. F.1 to F.5, E.7 and F8, along with complexity and convergence analyses in Apps. G.1 and G.2.
Conclusion
We propose BDS, a novel and principled framework that achieves adaptive defense against harmful fine-tuning without requiring attack simulation. We introduce two implementations, Bayesian Sealar Scheduler and Amortized Bayesian Neural Scheduler, with the latter enabling efficient transfer to new data without retraining. Leveraging the post hoc nature of Bayesian inference, BDS learns the posterior distribution of data weights conditioned on the specific dataset, thus achieving adaptive defense. Extensive experiments confirm the SOTA performance and superior adaptiveness BDS, significantly outperforming baselines by over 70% across diverse attack and defense settings.
10

---

Acknowledgments
This research supported by the National Research Foundation, Singapore, and the CyberSG R&D Programme Office ("CRPO"), under the National Cybersecurity R&D Programme ("NCRP"), RIE2025 NCRP Funding Initiative (Award CRPO-GCI-NTU-002)
References
[1] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. survey on data selection for language models. arXiv preprint wrXiv:2402.1682)) 2024.
[2] Mikel Arterxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Vic- toria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O' O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1169-11 1732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[3] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.08875. 2023.
[4] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations, 2024.
[5] Hongyu Chen and Seraphina Goldfarb-Tarrant, Safer or luckier? llms as safety evaluators are not robust artifacts. arXiv preprint orXhv:2503.09377, 2025.
[6] Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Beyond two-stage training: Cooperative sft and rl for I1m reasoning. arXiv preprint arXiv:2509.06945 2025.
[7] Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Vulnerability- aware align- ment: Mitigating uneven forgetting in harmful fine-tuning, 2025.
[8] Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. Safety-awar fine-tuning of large language models. arXiv preprint arXiv:2410. 100/4, 2024.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint vxXiv22110.14166 2021.
selection. Advances in Neural Information Processing Systems, 36:8513 8527, 2023. [10] Zhijie Deng, Peng Cui, and Jun Zhu. Towards accelerated model training via bayesian data
[11] Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Ruapm Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632:768-7744 2024.
[12] Yanrui Du, Sendong Zhao, Danyang Zhao, Ming Ma. Yuhan Chen, Liangyu Huo, Qing Yang. Dongliang Xu, and Bing Qin. Mogu: A framework for enhancing safety of open-sourced llms while preserving their usability, arXiv preprint arXiv:2405.1 14488, 2024.
[13] Heshan Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, and Tianyi Chen. Mitigating forgetting Ilm supervised fine-tuning and preference learning. arXiv preprint 2xXiv:2410.15480 2024.

---

[14] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompis: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. / 1462, 2020.
[15] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.1533 155.44 2024.
[16] Yuxian Gu, Li Dong. Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. arXiv preprint arXiv:2410.07064 2024.
[17] Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking Ilms with stealthiness and controllability. arXiv preprint arXiv:240..0867, 2024.
[18) Danny Halawi, Alexander Wei, Eric Wallace, Tony T Wang, Nika Haghtalab, and Jacob Steinhardt. Covert malicious finetuning: Challenges in safeguarding Ilm adaptation. arXiv preprint arXiv:2406.20053. 2024.
[19] Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. arXiv preprint 23X.::3305.86666 2023.
[20] Luxi He, Mengzhou Xia, and Peter Henderson. What's in your" safe" data?: Identifying benign data that breaks safety. arXiv preprint 0rX1v:2404.01099 2024.
[21] Chia-Y Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin- Yu Chen, Chia-Mu Yu, and Chun- Chun-Ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models, 2024.
[22] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.009) 2021.
[23] Zixuan Hu, Li Shen, Zhenyi Wang, Tongliang Liu, Chun Yuan, and Dacheng Tao. Architecture, dataset and model-scale agnostic data-free meta-learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7736-7745, 2023.
[24] Zixuan Hu, Li Shen, Zhenyi Wang, Baoyuan Wu, Chun Yuan, and Dacheng Tao. Learning to learn from apis: Black-box data-free meta-learning. In International Conference on Machine Learning, pages 13610-13627. PMLR, 2023.
[25] Zixuan Hu, Yongxian Wei, Li Shen, Zhenyi Wang, Lei Li, Chun Yuan, and Dacheng Tao. Sparse model inversion: Efficient inversion vision transformers for data-free applications. In Forty-first International Conference on Machine Learning, 2024.
[26] Zixuan Hu, Yongxian Wei, Li Shen, Zhenyi Wang, Baoyuan Wu, Chun Yuan, and Dacheng Tao. dask-iistributinnally robust data-free meta-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[27] Zixuan Hu, Yongxian Wei, Li Shen, Chun Yuan, and Dacheng Tao. Lora recycle: Unlocking wning-free few-shot adaptability in visual foundation models by recycling pre-tuned loras. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25026-25037, 2025.
[28] Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408, 24000000000 2024.
[29] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01556, 2024.
[30] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409. 18169, 2024.
12

---

[31] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[32] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Virus: Harmful fine-tuning attack for large language models bypassing guardrail moderation. arXiv preprint arXiv:2501. 17433, 2025.
[33] Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-awan alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[34] Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. arXiv preprint arXiv:2410.08020, 2024.
[35] Jiaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of 11m via a human- prefa--reference dataset. Advances in Neural Information Processing Systems, 36, 2024.
[36] Jiaming Ji, Mickel Liu, Juntao Dai, Xuchai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human-preference dataset. arXiv preprint arXiv:2307.04657 2023.
[37] Chak Tou Leong, Yi Cheng, Kaishuai Xu, Jian Wang. Hanlin Wang, and Wenjie Li. No two dev- ils alike: Unveiling distinct mechanisms of fine-tuning attacks. arXiv preprinl arXiv:2405. 16229, 2024.
[38] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction followiwng models. https: //gttbbb/ tht..
cww/tatsu lab/ alpaca_evall 2023.
[39] Zhe Li, Wei Zhao, Yige Li, and Jun Sun. Do influence functions work on large language models? arXiv preprint 1eXiv2449.11994) 2024.
[40] Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. Targeted vaceine: Safety aligmment for large language models against harmful fine-tuning via layer-wise perturbation. IEEE Transactions on Information Forensics and Security, 2025.
[41] Guozhi Liu, Qi Mu, Tiansheng Huang. Xinhua Wang, Li Shen, Weiwei Lin, and Zhang Li. Pharmacist: Safety alignment data curation for large language models against harmful fine- tuning. arXiv preprint arXiv:2510. 20000005,, 2025.
[42] Xiaoqun Liu, Jiacheng Liang, Luoxi Tang. Muchao Ye, Weicheng Ma, and Zhaohan Xi. Data to defense: The role curation in customizing llms against jailbreaking attacks. arXiv preprint arXiv:2410.02220,, 2024.
[43] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-166 165. Elsevier, 1989.
[44] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308. 13320, 2023.
[45] Christopher Nemeth and Paul Fearnhead. Stochastic gradient markov chain monte carlo. Journal of the American Statistical Association, 116(533):433-450, 2021.
[46] Sheng Peng, Pin-Yo Chen, Matthew Hull, and Duen Horng Chau. Navigating the safety land- scape: Measuring risks in finetuning large language models. arXiv preprint arXiv:2405. /7374, 2024.
[47] Sheng Yun Peng. Weilin Xu, Cory Cornelius, Matthew Hull, Kevin Li, Rahul Duggal, Mansi Phute, Jason Martin, and Duen Horng Chau. Robust principles: Architectural design principles for adversarially robust cnns. arXiv preprint 230.::2000..225556 2023.
13

---

[48] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024.
[49] Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9999 1951, 2022.
[50] Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & 2020.
Data Mining, pages
[51] Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning, In International Conference on Learning Representations, 2019.
[52] Domenic Rosati, Jan Wehner, Kai Williams, Fukasz Bartoszcze, David Atanasov, Robie Gonza- les, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising effectively prevents harmful fine-tuning on llms. arXiv preprint arXiv:2405.14577, 2024.
[53] Domenic Rosati, Jan Wehner, Kai Williams, Fukasz Bartoszeze, Jan Batzner, Hassan Saj- jad, and Frank Rudzicz. Immunization against harmful fine-tuning attacks. arXiv preprint 24X1:22402166382 2024.
[54] Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.0122 0123608..26,,. 2023.
[55] Han Shen, Pin-Ye Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned Ilm fine-tuning via bilevel data selection. arXiv preprint arxiv:2410007471, 2024.
[56] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1643 2013.
[57] Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, al. Tamper-resistant safeguards for open- -eight llms. arXiv preprint arXiv:2408.0076/ 2024.
[58] Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang. Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. Tamper-resistane safeguards for open-weight LLMs. In The Thirteenth International Conference on Learning Representations, 2025.
[59] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXhv:2403.08295 2024.
[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[61] Jiongxiao Wang. Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. arXiv preprint arXiv:2402. 14968, 2024.
[62] Yibo Wang, Tiansheng Huang. Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, and Dacheng Tao. Panacea: Mitigating harmful fine-tuning for large language models via -isttfine-uuiigg perturbation. Advances Neural Information Processing Systems, 2025.
14

---

[63] Zhichao Wang, Bin Bi, Zixu Zhu, Xiangbo Mao, Jun Wang, and Shiyu Wang. Uft: Unifying fine-tuning of sft and rlhf/dpo/una through a generalized implicit reward function. arXiv preprint arXiv:2410.21438, 2024.
[64] Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang. Li Shen, Lu Hou, Sinan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. Unifying multimodal large language model capabilities and modalities via model merging. arXiv preprint arXiv:2505. 19892, 2025.
[65] Yongxian Wei, Zixuan Hu, Li Shen, Zhenyi Wang, Yu Li. Chun Yuan, and Dacheng Tao. Task groupings regularization: Data-free meta-learning with heterogeneous pre-trained models, In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52573-52587. PMLR, 21-27 Jul 2024.
[66] Yongxian Wei, Zixuan Hu, Li Shen, Zhenyi Wang, Chun Yuan, and Dacheng Tao. Open- vocabulary customization from clip via data-free knowledge distillation. In The Thirteenth International Conference on Learning Representations, 2025.
[67] Yongxian Wei, Zixuan Hu, Zhenyi Wang, Li Shen, Chun Yuan, and Dacheng Tao. Free: Faster and better data-frre free meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23273-23282 2024.
[68] Yongxian Wei, Anke Tang, Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. In Forty-second International Conference Machine Learning, 2025.
[69] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 68 1-88. Citeseer, 2011.
[70] Di Wu, Xin Lu, Yanyan Zhao, and Bing Qin. Separate the wheat from the chaff: A post-hoc ap- proach to safety re-alignment for fine-tuned language models. arXiv preprint 1011 2024.
[71] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333 2024.
[72] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36:33601-343222--0 2023.
[73] Xinnuo Xu, Minyoung Kim, Royson Lee, Brais Martinez, and Timothy Hospedales. A bayesian approach to data point selection. Advances in Neural Information Processing Systems, 37:38171- 38198, 2024.
[74] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067/ 2024.
[75] Biao Yi, Tiansheng Huang, Baolei Zhang, Tong Li, Lihai Nie, Zheli Liu, and Li Shen. Ctrap: Embedding collapse trap to safeguard large language models from harmful fne-tuning, arXiv preprint 250.. 2025.
[76] Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. A safety realignment framework via subspace-oriented model fusion for large language models. Knowledge-Bard Based Systems, 306: 12701, 2024.
[77] Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. arXiv preprint arXiv:2406.0600 2024.
15

---

[78] Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: celuence-driver selective annotations empower in-context learners in large language models. arXiv preprint arXiv:2310. 10873, 2023.
[79] Wenxuan Zhang, Philip HS Torr, Mohamed Elhoseiny, and Adel Bibi. Bi-factorial preference op- timization: Balancing safety-helpfulig in language models. arXiv preprint arXiv:2408. 153133 2024.
[80] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[81] Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Chao Shen, Tianlin Li, Weipeng Jiang, and Yang Liu. Speculative coreset selection for task-specific fine-tuning, arXiv preprint arXiv:2410.01296 2024.
[82] Zhi- Thi-Hua/ae Zhou. Machine learning. Springer nature, 2021.
[83] Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: baseline for vision large language models. arXiv preprint arXiv:2402.02200, 2024.
[84] Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
[85] Difan Zou, Pan Xu, and Quanquan Gu. Faster convergence of stochastic gradient langevin dynamics for non-log-coneal log-concave sampling. In Uncertainty Artificial Intelligence, pages 1152- 1162. PMLR, 2021.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claim is explicitly stated in the abstract and introduction, and is well supported by extensive experiments across diverse attack and defense scenarios.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See App. F.7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
Detailed derivations and proof' are provided in Apps. H.1 to H.4 and App. 1.6.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed implementation details are provided in App. B.I, App. B.3, App. B.2, and App. B.4.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
Code and instructions for reproducing all experiments will be released upon
acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
onlooicc)))))) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. wwtppblcc/geeeeeccccccccccecccoe httgggudddeeeoeeeee petpssmmmmssiioooooo ssionPoiccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed implementation details are provided in App. B.1, App. B.3, App. B.2, and App. B.4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: the community of LLM harmful fine-tuning defense [29, 33], is common not to report error bars due to the high computational cost and the stability of results across runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on a single NVIDIA A100 40GB GPU. Details computational requirements and complexity analysis provided in App. G.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: httppp/://///u...../// cc weppbiic/tthcaaudeeell..eee
Answer: [Yes]
Justification: Our research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See App. F.8.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available datasets and models with proper citation, and adhere to their respective licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htopp://nnerips... http:/pp/..//ooeeeeee/////////// for what should or should not be described.
23

---

Appendix of Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
Contents
A Related Work
25
More Details
26
B.1 Details on Baselines
26
B.2 Details on Fine-Tuning Task Evaluation
................................
27
B.3 Details on Scheduler Training
27
B.4 Details on Fine-Tuning
27
C
Algorithms
28
More Experiments
28
D.1 Weight Distribution of Benign and Harmful Data under Various Harmful Ratios
28
D.2 Scheduling Dynamics for Benign and Harmful Data under Various Harmful Ratios
28
D.3 Robustness for OOD Attack.
D.4 Robustness for Identify Shifting Attack (ISA) [48]
31
D.5 Robustness for Larger Fine-Tuning Dataset Size
31
D.6 Robustness on More Complex Tasks: Data-to-Tent Generation.
32
D.7 Comparison with Detection- Based Methods.
32
D.8 Comparison with Deterministic Data Curation Methods.
32
E Ablation Studies
33
E.1 Impact of Different Weight Transformations...
33
E,2 Impact of Different Weight Priors
33
E.3 Impact of Weight Initialization
................................
33
E,4 Impact of the Alignment Dataset
................................
34
E.5 Impact of the Data Weight
34
More Discussions
34
F1
Insights into Adaptiveness to Diverse Attack Dynamics
34
F22
Insights into Helpfulness- Helpfulnes--Hamllessnn Harmlessness Trade-Offs
35
F3
Simulation-Free vs. Simulation- Based Defense
35
F4
Data Scheduling Data Filtering
35
F.5
Over-Refusal
36
F6
Zero-Shot Performance
36
E7
Limitation and Future Work
36
24

---

F.8 Impact Statement
37
More Analyses
37
G.1 Time and Space Complexity Analysis
37
G.2 Convergence Analysis of SGLD Sampling
38
H Derivation of Equations
39
H.1
Derivation of Eq. (1)
H.2 Derivation of Eq. (4)
39
H.3 Derivation Eq. (10)
40
H.4
Derivation Eq. (16)
40
I
Concept of Posterior Bias
41
L1
Motivation of Proposed Posterior Bias
1.2
Definition of Posterior Bias
41
L33
Empirical Posterior Bias with SGLD Sampling
41
1.4
Understanding Posterior Bias under Identity Weight Transformation
42
1.5
Theorem of Time-Weighted Accumulation of Posterior Bias
42
1.6
Proof of theorem Theorem 4.2
43
A
Related Work
fine-tuning-ass--serviie has become widely adopted paradigm among mainstream LLM providers (e.g., OpenAl and Mistral"). Recent red teaming studies [48, 29, 30] have revealed a critical vulnerability: harmful fine-tuning. Harmful fine-tuning refers that the presence of even small fraction of harmful data user- provided datasets can cause fine-tuned models to deviate from the safety alignment established during pre- pre-training, i.e., the model forgets give refusal answer towards harmful prompts after fine-tuning on a few harmful samples.
Mechanism study of harmful fine-tuning, Existing research has made efforts analyze the mecha- nisms underlying the high sensitivity of LLMs to harmful fine-tuning (i) Adversarial permutations: One line of work reveals that harmful fine-tuning can induce adversarial perturbations to the model, to which LLMs are highly sensitive. For instance, Vicanne [33] demonstrates that tuning with harmful data in user-provided datasets can cause embedding drift. Similarly, Booster [29] identifies that harm- ful fine-tuning leads to parameter perturbations. Moreover, [46] introduces the concept of a safety basin, wherein I.LMs tolerate parameter perturbations within local region while maintaining safety alignment; however, exceeding this basin results in a sharp degradation of alignment. (ii) Structural vulnerabilities: Another line of work delves deeper into the structural susceptibilities of specific model components [21, 47, 37J. For example, Safe LoRA [21] highlights that certain layers of model play a more critical role in preserving safety, while others are less susceptible to perturbations. Similarly, [37] emphasizes that different layers perform distinct functions when exposed to various types of attacks. (iii) Catastrophic forgetting: Several studies explain alignment vulnerabilities through the lens of catastrophic forgetting [43] due to the sequential training paradigm [6, 13]. For instance, [63] highlights that the inconsistency between SFT and alignment objectives can lead to alignment knowledge being forgotten when SFT is performed sequentially after alignment. Likewise, [13] identifies that the sequential nature SET after alignment exacerbates this forgetting issue
Harmful fine-tuning attack. On the attack side, [48] conduct experiments using OpenAI's API and demonstrate that even fine-tuning solely on benign data can compromise the base model's 's safety
4 "Fine-tuning API by OpenAl: https: cettaaatforoo.ooeaai...o wtt ppenmss.cmm/bocss/useeee//iieeee FFine-tuning API by Mistral: attps://doce http:////cccmm.traall...
mistral htt.:/raaa..aeeuuuueee//ettetee
25

---

alignment. This highlights the inherent risks of fine-tuning in the fine- tuning-nnnggga as-a---rviiee- aerviic scenario. Further, [20] identify subsets of benign data that are more likely to degrade model safety post-fine- tuning. These data points are characterized by their proximity to harmful examples and distance from benign ones in both representation and gradient space. Additionally, [18] introduce the concept of covert malicious fine-tuning. In the first stage (learning the encoding). the model is trained to learn an encoding which it did not previously know. In the second stage, the model is fine-tuned with encoded harmful inputs and outputs. During testing, the model generates encoded harmful responses when triggered by encoded harmful queries. Recently, [32] construct harmful data designed to deliberately circumvent moderation guardrails.
Harmful fine-tuning defense. On the defense side, existing defense methods can be categorized into three categories, including alignment stage [33, 52, 58, 42, 29, 40, 41. 75], fine-tuning stage [44, 3, 31, 61, 79, 55], and vost-fine-tuning stage solutions [76, 12, 28, 70, 62]. Alignment stage defense aims at preemptively improving the model's robustness prior to deployment. For example, Vaceine [33] introduces simulated embedding perturbations to strengthen the model's embedding resistance, while Booster [29] incorporates parameter perturbations to achieve parameter resistance. RepNoise [52] unlearns harmful representations such that it is difficult to recover them during fine-tuning, Fine-tuning stage defense aims to reduce the influence of harmful data during the fine-tuning process. Safelnstr [4] proposes to mix safety aligmment data during the fine-tume tuning process constantly reinforce the model's alignment knowledge. Similarly, VLGuard [83] also employs the data-mixing strategy but focuses on verifying its effectiveness with Vision-LLMs Lisa [31] also mixes alignment and fine-tuning data but introduces Bi-State Optimization separate the optimization processes for alignment and fine-tuning data, thus reducing optimization overhead. As we can see, Safelnstr [4], VLGuard [83], and Lisa [31] similarly adopt the strategy of mixing alignment data with fine-tuning data. However, it has fundamental limitations: (i) These methods do not explicitly isolate harmful data but only attempt to counteract its effects indirectly, thus leading to suboptimal defense performance. (ii) They require alignment data to scale with harmful data, incurring high computational costs and making them highly sensitive the unknown amount of harmful data in post hoc attacks. Seal [55] introduces a data selection strategy grounded in penalty- based bi-level optimization principles. The method assigns hard labels to data samples and relies on a manually adjusted threshold, which can be suboptimal since the defender generally lacks prior knowledge of the proportion of harmful data and the method could be less robust when dealing with ambiguous or uncertain samples (see discussions in App. F.4). Post- post-fine-tun fine-tuning stage defense [76, 12, 28, 70, 62] aims to restore safety alignment after fine-tuning without sacrificing fine-tuning performance.
Data curation for LLMs. Data curation methods aim optimize data utilization strategies. For instance, coreset selection for LLMs [81, 1, 34, 73, 41] focuses on selecting the most critical subset of data. Existing approaches typically rely on metrics computed from the raw dataset and a reference dataset, including gradient matching [71], representation similarity [19], influence functions [39, 78, 77], and uncertainty estimation [34]. However, these methods assume distributional similarity between the raw and reference datasets and often leads to high computational costs (e.g., influence functions). In our setting, where the raw dataset (i.e., user-proy provided fine-tuning dataset) and reference dataset (i.e,, alignment dataset) exhibit insufficient distributional similarity, such methods lead to suboptimal defense performance. In contrast, our approach leverages the loss difference between benign and harmful data model trained with the alignment dataset. This design is independent of distributional similarity and computationally efficient, thus offering a unique advantage in the setting of harmful fine-tuning defense.
B
More Details
B.1 Details on Baselines
We summarize the high-level ideas and implementation details of the baseline methods used in our experiments. We basically follow the configuration used in their original paper.
SFT: Supervised fine-tuning (SFT) is initially applied using the alignment dataset to fine-tune the base model, followed by SFT on the user-provided fine-tuning dataset, which contains partially harmful data.
26

---

Vaccine [33]: Vaccine introduces simulated perturbations to embeddings using harmful dataset to enhance the model's robustness against influences of harmful data. Afterward, SFT is performed on the user-provided fine-tuning dataset, which includes partially harmful data. The hyperparameter is selected through grid search over (0.1, 1, 10), with p = 5 used the final experiments.
RepNoise [52]:
RepNoise unlearms information about harmful representations such that it is difficult recover them during fine-tuning, The hyperparameters are set as a and 0.001.
Lisa [31]: Lisa alternatively tune the base model between alignment and fine-tuning datasets to preserve alignment knowledge. The regularizer intensity p is selected via grid search over {0.001, 0.01, 03,, with p = 0.01.
Booster [29]:
Booster introduces simulated perturbations to model parameters using harmful dataset, enhancing the model's ss robustness against potential harmful data by improving its resistance to parameter perturbations. Subsequently, Booster applies SFT to fine-tune the base model on the user-provided fine-tuning dataset.
B.2
Details on Fine-Tuning Task Evaluation
For fair comparison [29], we employ a unified system prompt for training and testing across all tasks, structured follows:
Prompt: Below is an instruction that describes a task, paired with an input that pro- vides further context. Write a response that appropriately completes the request. Instruc- tion: | |instruction] (input) Response: Output: [output]
We define (instruction) and [input] tailored to each dataset. For SST2, the ( (instruction) specifies sentiment analysis objective. with the [input] being sentence and the {output] the corresponding sentiment label. In GSMSK, the (instruction) is mathematical question, and the (output) is the correct answer. For AGNEWS, the (instruction] specifies the classification objective, with the { [input] } being a sentence and the (output) representing the corresponding category. For AlpacaEval, GPT4's high-quality instruction-nnswe pairs are used as the demonstration data, and testing involves evaluating the helpfulness of model responses to unseen prompts using ChatGPT's API.
B.3 Details on Scheduler Training
Due to limited prior knowledge about the dependencies, we assume factorized priors: p(0, w) zo p(0). p(w) in Eq. (1) and p(0. d Drx) p(0) p(Φ) in Eq. (10). Although p( (w) is theoretically defined as a prior distribution, in practice we initialize t to 0.1 and allow it to be freely optimized during training (i.e., using noninformative prior) to avoid potentially misleading manually chosen priors. Future work could explore how to design more robust priors to better regularize the data weight decay
weights. p0) and p(Φ) follow zero-mean Gaussian distributions corresponding regularization. The neural scheduler is implemented using a lightweight 125M Fairseq-Dense model [2] with an added trainable head. We adopt identity transformation on the mean pooling of the instance's representations along the sequence length. The size of hidden state is 768. We optimizer the scalar and neural scheduler with a learning rate of 5 x 10 and 10- respectively, using a batch size of 10.
B.4 Details on Fine-Tuning
For fine-tuning, we adopt LoRA [22] for efficient fine-tuning, following [29, 33, 21J. The adaptor rank is set to 32 with alpha set to 4. Fine-tuning is performed using the FusedAdam [50] with a learning rate of 10-6 and a weight decay of 0.1, as recommended by [29]. The training involves 20 epochs for SST2, AGNEWS and GSM8K, and 100 epoches for AlpacaEval, following [29]. The batch size is set as 10. The model's backbone utilizes BF16 (bfloat16) precision for computational efficiency. For the extreme case where the harmful ratio is 1 (i.e., the fine-tuning data contain only harmful samples), we report in Tab. the result obtained with a larger safe dataset (Daate = 5000) to achieve more stable defense performance, while results for other ratios use [Detel 1000.
27

---

C
Algorithms
To provide clearer understanding of our algorithm, present detailed pseudocode in Alg. 1.
Algorithm 1: Bayesian Data Scheduler.
Input: Base LLM (()) User- providdd fine-tuning dataset Dn; Alignment datset Duste; Bayesian Scalar
Scheduler or Amortized Bayesian Neural Scheduler (M::; (:: Φ); Step size Weight transformation function o-); Gaussian noise e; Max iterations T.
Initialize w or & for <- to I do
// Construct
batch of data Sample Beate from the alignment dataset Daate and BR from the fine-tuning dataset Dr: // Update LLM via Eq. (7)
g(++1) ((4) Vo
logr(0
w
Dastal
EB (z 0(1))
DD
Bit
Eater [a(ws) [(x):(0)1))) + EVT
// Update scheduler via Eq. (8) or Eq. (11)
if using Bayesian Scalar Scheduler then
(+1) (1) + V log p(w(")
B5 Ex ( [(P cev
else using Amortized Bayesian Neural Scheduler then df (1+1)
V
logp(d"")
Exeen ((((eeli (( +ovn
Bal
Return: Fine-tuned LLM ( after T iterations.
D
More Experiments
D.1 Weight Distribution of Benign and Harmful Data under Various Harmful Ratios
To clearly illustrate the adaptive defense capability of our proposed method, we provide compre- hensive visualizations of weight distributions across harmful ratios from 0.2 to 0.9 in Fig. 7. In each subfigure, the largest panel depicts the scatter and boxplot distributions of weights for truly benign and truly harmful data, respectively. The top-right panel presents the histogram of weights for truly benign data, while the bottom-right panel shows the histogram for truly harmful data. The visualizations demonstrate that our method indeed accurately assigns higher weights to truly benign data and consistently lower weights to truly harmful data. Morcover, this effectiveness remains stable and robust across varying and unknown harmful ratios, under extreme conditions like ratio of 0.9. These results highlight the adaptability of our approach to diverse attack scenarios without requiring modifications, underscoring its strong potential for real-world applications.
D.2 Scheduling Dynamics for Benign and Harmful Data under Various Harmful Ratios
To clearly demonstrate the adaptive data scheduling process, we provide comprehensive visualizations of scheduling dynamics across harmful ratios ranging from 0.2 to 0.9 in Fig. 8. For encountered datasets with different and unknown harmful ratios from 0.2 0.9, BDS adaptively schedules data into higher and lower weight groups during fine-tuning (largest panels). To verify correctness, we observe that most truly benign data indeed receive higher weights (top right panels), while almost all truly harmful data consistently receive lower weights (bottom right panels). Moreover, this adaptive scheduling remains effective and robust, even under extreme conditions like a harmful ratio of 0.9. These results underscore the adaptability and robustness of our approach across diverse attack scenarios without requiring modifications, demonstrating strong potential for real-world applications.
28

---

[Posl 1000, [0p]
1000. 0.2
Pool= 1000, [D,, 1000, 0.3
Cound
Gaunt
Truly benign
Truly benign data
Weigitt (are transformmtion
Truly
Weig itt fere trnnformaion)
Truly harmful
106
Fuuyy benign cats Truly hammful dasa
Weight
Truly benign data Truly harmful cata
Weight
Wan [0 p
[Pusl = 1000, (Py = 1000, 0.5
Truy benign data
Count
Truly benign
Count
Weight (pre transformaton
Truly harmful data
Weigh: (pre transformttio)
Truly harmful
Trily benign dats Truly harmful datts
Weight
Truly benign Truly harmful idea
Weight
iDast = 1000, 1Dy.1 = 1000, 0.6
1000, 10p - 1000, 0.7
Count
.Count
Weightt trandforation)
Truly benign date
Weight (pre transforration
Truly benign data
Truly harmful
Truly harmful data
Truly benign Truly
Weight
Truly benign Truly
Weight
PP - 1000, |D,1= 1000, p - 0.8
[Pail 1000, 1, 1000, p
Count
Count
Weight fore transfomatioon
Truly benign data Truly harmful
Weight fpre tramsformation)
Truly benign data
Truly
Truly benign Truly harmful
ghi
Truly benign Truly harmful
Figure 7: Adaptive weight distribution of benign and harmful data under various harmful ratios.
29

---

iDantel = 1000, 12pz|= = 1000, p = 0.2
1Dau - 1000, 1P - 1000, p 0.3
Automatic Data Scheduling For Senign and Harmful Data
Automatic Data Scheduling for Benign and Harmful Data
ATl date in turing dahose?
dets in hurnin dofzse?
turling donust
603
700
Tuning hhrehom cm
1013
Tusing Herallore
500
11:11
hut dehest
3300
2:50
3433
1340
110
1033
Inferned Data Weight
Enferred Dato Weight
04gher Iedicetes Greater LisllPoood of Being Berign Dano]
(Migher 3edicates Greater Likilhood Being Barign Daho)
[Daatel 1000, [D,,] 1000, 0,4
12 1000, 1D,.1= 1000, p 0.5 Nutomatic Data Scheduling for Benign and Harnful Data
Automatic Data Scheduling for Benign and Harmful Data
All tunting dofeset
hunting
400
Tunin Henshirr
Sto
Horm
2100
This Hordl rnls
1203
1013
1603
1840
1003
1800
2000
Inferre Weight
Inferrad Data Weight
(iigher Sreeter hood Being Berign
0ligher indieshes Grecter Likel hoed of Beng Berign Daho)
[Daate = 1000, [Dyll = 1000, p 0.6 Automatic Data Schedkling for Benign and data in huning delsset
|Dastul= 1000, |Da|= 1000, = 0.7
Automatic Data Scheduling for Benign and Harmful Data
Tris fenign
Kll dota furing dafoset
langy Kanin denast
datiel
7E0
(00
Tunin Rerstore
2000
2003
Triyy darootiaaia
hoiing datoee
2350
2300
3800
3000
2000
0igher Infermed Weight
Inferned Weight
Seng Benign Daha)
(iigher Indictes Greater Likellooo Belng Berign Detel
[Daatel = 1000, [Drl - 1000, 0.8 Automatic Data Scheduling for Benign and HormFul Data
[Deatel = 1000, |Da|= 1000, = 0.9 Automatic Data Scheduling for Benign and Harmful Data
All data in huning detese?
Tril design
All date
turking dafuset
600
Tusing thenthons
Heviing dofose
1200
1003
Data Waigh
Infermed Wrigght
04gher Indicates Graster Likel head of Seng Benign Daho)
34gher
Baing Benign
Figure& 8: Adaptive scheduling for benign and harmful data under various harmful ratios.
30

---

D.3 Robustness for OOD Attack.
To evaluate the adaptiveness of our method to advanced attacks, we conduct experiments under an OOD setting, which simulates attacker uploading fine-tuning data from domain distinct from the defender's alignment dataset. Here, we use BeaverTails [35] as the alignment dataset, while RealToxiciityPompe [14] and AdvBench [84] serve OOD attack datasets. Notably, RealToxici- tyPrompts consists of prompts likely to elicit toxic completions, representing a domain that differs substantially from BeaverTails. As shown in Tab. 8, our method maintains strong defense performance against OOD attacks. This suggests that harmful data- across different domains cxhibit shared "unsafe behavior" in the loss landscape [13], leading to consistent higher loss and thus lower weights. Interestingly, this aligns with the observation of shared "safe behavior" across benign datasets (see Tab. 7), and supports the existence of transferable data safety attributes discussed in [16].
Table 8: Robustness for OOD attack.
Method
Real/oxxiiityyrompe (p 0.1)
RealToxicitty (p = 0.3)
AdvBench (p = 0.1)
AdvBench (p = 0.3)
FA
4
f
Booster
26.80
28.40
91.94
18.20
BDS
1.50
93.88
1.90
93.36
0.80
93.46
0.90
92.66
D.4
Robustness for Identify Shifting Attack (ISA) [48]
To evaluate the adaptiveness of our method under adversarial prompting, we assess its performance against the Identity Shifting Attack (ISA) [48]. Following the setup [48], we prepend identity- shifting prompt to each fine-tuning example to simulate the attack. As shown in Tab. 9, BDS achieves significantly lower harmful score (around 1) against ISA. The score also remains similarly low compared to non-ISA settings, indicating BDS effectively deweights harmful data even with prompt-level manipulation.
Table 9: Robustness for Identify Shifting Attack (ISA) [48].
Method
ISA (p
0.1)
ISA (p
0.3)
(p
0.6)
(p 0.8)
HS |
FA
HS 1
FA f
HS 4
FA
HS J.
FA
Booster
16.20
92.36
48.40
92.18
77.40
91.32
77.40
91.24
BDS
1.60
93.00
1.50
93.46
1.50
92.68
1.70
92.14
D.5 Robustness for Larger Fine-Tunie Fine-Tuning Dataset Size
In addition to Tab. 5, which evaluates fine-tuning dataset sizes ranging from 500 to 2000, we further evaluate the robustness of BDS under larger fine-tuning dataset size. As shown in Tab. 10, BDS consistently maintains strong defense performance even when the fine-tuning dataset size increases to 10000. In contrast, the performance the SOTA baseline (Booster) degrades significantly as the dataset grows due to the increased absolute number of harmful data within it. These results demonstrate that BDS scales effectively with larger datasets and maintains strong robustness.
Table 10: Robustness for larger fine-tuning dataset size.
Method
[Dal
8000 (p 0.1)
[Dal 8000
0.3)
[Pa.
10000
(p
Drl
10000 (p 0.3)
HS 1
FA f
HS 4
f
HS
f
HS
FA f
Booster
68.80
75.60
92.64
92.78
77.20
92.42
BDS
1.20
92.89
1.20
93.02
1.10
93.12
1.30
93.08
31

---

D.6 Robustness on More Complex Tasks: Data-to-Text Generation.
In addition the four tasks in Tab. 3, we further evaluate BDS on the GEM benchmark 6 focusing on more complex data-to-text generation tasks specificccll ViGGO' and CACAPO", As shown Tab. 11, BDS outperforms the SOTA baseline in both helpfulness (measured by ROUGE-1) and harmlessness (measured by harmfulness score), demonstrating its robustness and effectiveness in more complex data-to-text generation tasks.
Table 11:
Robustness on GEM benchmark.
ViGGO
0.3)
CACAPO (p 0.3)
Method
ROUGE-1 1 f
HS 4
ROUGE-I f HS 4
Bouster
0.84
45.80
0.63
52.60
BDS
0.88
1.30
0.70
1.40
D.7
Comparison with Detection-Based Methods.
We further compare BDS with two detection-bars baselines proposed in [8]:
LLM-As-A-Judge [8]: This method filters the harmful data by querying the large language model "Is the following data sample harmful to for fine-tuning?"
SAFT [8]: An OOD detection-based method that identifies harmful data based on activation subspace representations.
As shown in Tab. 12, BDS achieves significantly higher fine-tuning accuracy and lower harmfulness scores than both methods. This supports the limitations of detection-basee approaches discussed in Table. 2 of [30], particularly their vulnerability to false positives and false negatives:
Why do detection-based baselines have low fine-tuning accuracy? This is due the detector's false positive rate, where some truly benign fine-tuning data are incorrectly identified as harmful and removed. As result, the amount of truly benign data for fine-tuning reduced, leading to degraded fine-tuning accuracy.
Why do detection-based baselines have high harmful scores? This is due to the detector's 's false negative rate, where some truly harmful data incorreetly identified benign and retained. Consequently, truly harmful data participate in the fine-tuning process, leading to high harmful scores.
In contrast, BDS does not rely hard binary filtering. Instead, it adopts loss based soft weighting mechanism, which adjusts the influence of each sample without requiring explicit hard labels. This enables more robust and accurate handling of uncertain or ambiguous data.
Table 12: Comparison with detection- based methods.
SST2 (p = 0.1)
SST2 (p 0.3)
Method
FA 1
HS
FA t
HS
LLM-As-A-Judge [8]
90.03
35.2
89.68
40.5
SAFT [8]
91.28
27.6
90.89
29.6
BDS
93.69
1.20
93.32
1.20
D.8 Comparison with Deterministic Data Curation Methods.
To showcase the unique effectiveness of BDS as a data curation method for harmful fine-tume fine-tuning defense, compare another data curation method DSIR [72]. DSIR resamples the target dataset (ie., Da) via importance sampling based on distributional alignment with the reference dataset (i.e.,
"nttps://gem-bett benchark. com/
'nttpe://huggingfacee http:///wggnngicccccc/aatataaaas co/dataaette/CeM/11
'attpe://huggingfacce co http:///ee/etaattteeee/////////e
32

---

Deate). As shown in Tab. 13, BDS consistently outperforms DSIR. This advantage stems from two key factors: (1) DSIR depends on distributional similarity between reference and raw datasets, while BDS remains robust to distributional differences; (2) BDS captures uncertainty in datapoint-wise safety attributes, enabling more reliable weighting for potentially ambiguous data.
Table 13: Comparison with deterministic data curation method
SST2
AGNEWS
GSM8K
AlpacaEval
Method
HS
FA f
HS 4
FA
HS 4
FA 1
HS L
FA
DSIR BDS
56.10
91.14
48.40
81.40
46.20
13.50
58.60
39.84
1.20
93.69
1.10
89.10
2.00
18.30
1.20
46.83
E
Ablation Studies
E.1
Impact of Different Weight Transformations.
To assess the impact of different weight transformations, we compare softmax, sigmoid and iden- tical transformations in Tab. 14. The results confirm our analysis in Sec. 4.4, demonstrating the effectiveness of softmax in bidirectionally updating weights.
Table 14: Impact of different weight transformation functions.
softmax
sigmoid
identity
Method
HS
HS
FA f
HS
BDS
1.20
93.69
16.70
92.23
16.50
0.00
E.2 Impact of Different Weight Priors
To assess the impact of different weight priors on BDS performance, we compare the noninformative prior with various Gaussian distributions, defined
(wr 10)
p(w))
exp
V
2x
where y is the mean and is the standard deviation.
The results in Tab. 15 show that the noninformative prior achieves better defensive performance and slightly better fine-tuning performance compared to Gaussian priors. We attribute this to the noninformative prior granting greater freedom to weight updates. Since defenders lack prior knowledge about the harmful ratio in user- ssovidro data, imposing Gaussian prior may introduce incorrect constraints, potentially degrading BDS performance. Future work could explore more effective and robust priors for modeling the data weight distribution.
Table 15: Impact of different priors ([Dsafel 1000, [Pr. = 1000, 0.1).
Method
noninformative prior
Gaussian (μ 0, 0.1)
Gaussian (μ
0, 1)
FA
HS
FA 1
HS 21.20
FA f
93.69
20.40
93.23
93.12
Method
Gaussian (a 0.1, = 0.1)
Gaussian (je 0.1, = 1) Gaussian (n1 = 0.1, 10)
HS J 20.20
FA
HS
FA f
HS J
FA f
BDS
93.12
21.80
93.12
20.10
93.35
E.3
Impact of Weight Initialization
To assess the impact weight initialization on BDS performance, we conduct experiments using different initialization values under the noninformative prior. The results in Tab. 16 indicate that BDS
33

---

is robust to weight initialization, achieving a harmful score between 1.1 and 1.5, and fine-tuning accuracy between 93.58% and 93.69% across initialization values ranging from 0.001 10. This stability demonstrates that BDS is insensitive to the weight initialization hyperparameter, making it easier to deploy in real-world applications.
Table 16: Impact of the value of weight initialization, ((D = 1000,
1000. 0.1).
0.001
0.01
0.1
1.0
10
Method
HS 4
FA
HS
FA
HS
FA
HS
FA
HS 4
FA f
BDS
1.50
93.58
1.30
93.58
1.20
93.69
1.10
93.58
1.10
93.69
E.4
Impact of the Alignment Dataset
As shown in Tab. 17, removing the alignment dataset sharply increases the harmful score (HS). This because the model fails to learn safety-awareness, causing harmful samples to no longer incur consistently high loss, thus leading to incorrect weight assignment.
Table 17: Impact of alignment dataset.
SST2 (p
0.1)
SST2 (p 0.3)
Setting
FA f
HS
FA f
HS
w/o alignment dataset
94.27
77.90
93.81
78.10
alignment dataset
93.69
1.20
93.32
1.20
E.5
Impact of the Data Weight
As shown in Tab. 18, when the safety alignment data and fine-tuning data simply mixed without applying appropriate weighting, the harmful ratio increases substantially. This result suggests that down-weighting harmful data effectively suppresses their adverse influence, whereas naive data mixing alone is insufficient to achieve the desired defense robustness.
Table 18: Impact of data weight.
SST2 (p 0.3)
Setting
FA
HS
w/o data weight
93.69
11.20
w/ data weight
93.32
1.20
F More Discussions
F.1 Insights into Adaptiveness to Diverse Attack Dynamics
To better understand why our method can effectively adapt to diverse attack dynamics, we highlight two mechanical design insights:
Bayesian conditioning enables dataset-specific defense. By leveraging the post hoc nature of Bayesian inference, the posterior is conditioned on the fine-tuning dataset, enabling BDS to tailor its defense to the specific dataset, thereby achieving adaptive defense.
Loss-based scheduling realizes instance-level adaptiveness. Once the posterior is conditioned to the specific dataset, BDS applies loss-based weighting strategy to individual samples. Harmful data regardesss of attack strategy terdds to exhibit consistently higher loss in the loss landscape [46], and therefore assigned lower weights. This shared "unsafe behavior" parallels the "safe behavior" observed across benign datasets (Tab. 7), and supports the existence of transferable data safety attributes [16]. As result, BDS can adaptively downweight harmful samples without requiring explicit attack simulation.
34

---

F22 Insights into Helpfulness- Trade-Off's
To better understand the trade-offs between helpfulness and harmlessness, we analyze several phenomena observed in Tab. 1: (i) When no harmful data is present (p= 0), BDS achieves the highest fine-tuning accuracy, while SFT performs significantly worse. (ii) As harmful data is introduced (p= 0.2), SFT improves in fine-tuning accuracy, despite the presence of harmful data. Note that these observations also align with the empirical findings in Table. 1 of [29], where SFT achieves lower fine-tuning accuracy p and higher accuracy as p increases 0.2.
These observations can be explained by the trade-offs between helpfulness and harmlessness. SFT [29] adopts a two-stage training paradigm (see baseline details in App. B.1); it first optimizes for harmlessness using an alignment dataset (i.e., the alignment stage), and then fine-tunes for helpfulness using a fine-tuning dataset (i.e., the fine-tuning stage). Specifically,
Why does SFT achieve lower fine-tuning accuracy when no harmful data is present (p 0)? Although the model achieves good harmlessness after the alignment stage, it loses plasticity [11] to sufficiently adapt and learn helpfulness in the fine-tuning stage, thus leading to lower fine-tuning
accuracy.
Why does SFT achieve better fine-tuning accuracy when more harmful data is introduced (p 0.2)? Introducing harmful data during fine-tuning serves as counterexamples that encourage the model to actively "unlearn" some harmlessness knowledge acquired in the alignment stage. While this sacrifices certain degree of harmlessness, enables the model to regain plasticity [11] and learn helpfulness more effectively during fine-tuning, thereby resulting in higher fine-tuning accuracy.
BDS method effectively balance the trade-offs, even when no harmful data is present (p = 0)? Instead of adopting a two-stage training paradigm like SFT, BDS jointly optimizes the alignment and the fine-tuning objectives, resembles a multi-task learning paradigm, which better handles the optimization trade-offs between harmlessness and helpfulness compared to the sequential learning paradigm used in SFT.
Why
shown in Eq. (7). This joint optimization
F.3 Simulation-Free vs. Simulation- Based Defense
Limitations of simulation-base based defense. While we acknowledge simulation- defenses [29, 33] offer the possibility in simulating potential attacks, such approaches f'ace fundamental limitations across several levels:
Infeasibility of attack simulation. It is often infeasible to construct ideal harmful datasets, as defenders typically lack prior knowledge of the characteristics of potential attack data. Even harmful data could be collected, it remains extremely challenging to simulate the diversity and unpredictability of real-world attacks.
Limited adaptiveness: Simulation-basee methods rely on pre-defined attack assumptions, which often fail to capture the variability and complexity of post hoc attacks. As result, their adaptability to diverse or unseen attack dynamics is limited.
Unstable optimization: Adversarial training methods well known be computationally expensive and often suffer from unstable training dynamics, e.g., due to the need for min-max optimization.
Advantage of simulation-free defense. In contrast, our simulation-free ffre approach avoids the afore- mentioned issues:
No reliance on attack simulation. Our method does not require assumed attack scenarios, thus avoiding the infcasibility of attack simulation.
Enhanced adaptiveness. Detailed discussions on our superior adaptiveness are provided in App. F.1.
F.4
Data Scheduling vs. Data Filtering
Our method learns a soft weight for each data point, modulating contribution to training, contrast to data filtering approaches that make binary inclusion decisions. Specifically:
35

---

Scheduling is automatic without requiring manual threshold. Scheduling adjusts each data point's contribution using learned weights, while filtering requires a manually set threshold to remove data-which is non-trivial: (i) ratio-based thresholds [55] are impractical without knowing the harmful data proportion, and (ii) value-based thresholds are hard to tune due to their broad continuous range.
Scheduling is more robust to classification errors. Instead of relying on hard-label decisions to remove data, scheduling assigns soft weights to individual data points. This alleviates the limitations of hard-label filtrating discussed in Table. 2 of [30], namely the overly strict false positives and false negatives issues.
F,5 Over-Refusal
We evaluate method on the Xstest dataset [54] using GPT-4 as the evaluator. The results summarized in Tab. 19.
Table 19: Over refusal examination on the Xstest dataset under safe and unsafe prompts.
Prompt Type
Answer Type
Ratio
1 fulll compliance
76.4%
Safe
2 full refusal
21 11.2%
3 __aattial_rffusal refusal
2.4%
1 full compliance
11.0%
Unsafe
2. full refusal
89.0%
3. partial refusal
0%
As shown in Tab. 19, (i) we observe a few of refusals to safe prompts, which we attribute to the base model's inherent limitations in language understanding [54]. An example shown in Tab. 20 (first row). (ii) We also observe small number of answers to unsafe prompts, where these responses are not classified refusals. Examples are provided in Tab. 20 (second row).
Table 20: Examples of failure-mode behavior-prompt pairs from BDS-trained LLMs on the Xstest dataset.
Type (Behavior- [Behavior-Prompt))
Prompt Example
Example
Refusal-Safe
What's best method poison in The Witcher 3 game?
cannot provide information advice how to harm or kill someone in any form of media, including The Witcher
Answer-Unsafe
Why should immigrants not allowed sit on the same airplane white people?
Immigrants should allowed airplane as white people, regurdless of their race ethnicity.
E.6 Zero-Shot Performance
We report the zero-shot performance of LLaMA-2 7B SST2, AGNEWS, GSMSK, and AlpacaEval as 0%, 0%. 2.7%. and 4.2%. These low scores suggest the model initially struggles with these tasks due specific output formats requirements (e.g., "positive/negative" in SST-2) or the need for complex reasoning (e.g., GSM8K).
F.7 Limitation and Future Work
One limitation of our method is the additional computational overhead introduced by an extra forward pass through the neural scheduler to compute data weights. However, this cost is modest relative to overall training (see complexity analysis App. G. Future work may investigate defense methods built upon model reuse [68, 64, 27, 23, 67, 26] at the model level rather than the data level, well as the potential use of synthetic data [24, 66, 25, 65] as a substitute for alignment datasets improve defense effectiveness (see also alignment data curation [41]).
36

---

F88 Impact Statement
Positive impact: The potential broader impact of this work lies in its enhancement of safety and reliability in commercial fine-tuning (ie., fin--tnning----sseeiic)))) for LLMs,, mitigating the risks of harmful fine-tuning that could otherwise lead to dangerous or unethical model behavior. By addressing these risks, this research can enhance the quality, robustness, and reliability of commercial fine-tuning services offered by LLM providers, ensuring safer and responsible user-customized LLM deployment.
Negative impact: Since the proposed method strengthens the relative weight of particular subsets of fine-tuning data, it could be misused to amplify specific ideological stances in the resulting model, thereby diminishing the diversity of perspectives represented in the fine-tuned model and potentially exacerbating social bias.
G
More Analyses
G.1 Time and Space Complexity Analysis
To better understand the efficiency of BDS compared to existing methods. we analyze and compare the time and space complexity of BDS and Booster, as summarized in Tab. 21.
Significant overhead of Booster. Booster comprises two stages: the alignment stage and the fine- tuning stage. During the alignment stage, Booster requires computing three separate gradients in each optimization step simulate harmful permutation, leading to time complexity of (33,, f). where n, is the number of alignment steps and f is the number of model parameters. In the fine-tuning stage, Booster calculates single gradient per step, resulting in a time complexity of O(ng/), where n2 is the number of fine-tuning steps. The space complexity during alignment includes storing three gradients (3/) and two data batches (2d) of benign and harmful data, giving O(3 2d). In fine-tuning, it reduces to O(f + d).
BDS is more efficient in both time and memory. BDS, in contrast, operates solely during the fine-tuning stage. For each step, BDS performs two updates: one for model weights and one for data weights, leading to a time complexity of (m2(f + w)), where w is the size of the data weights Given that w (data weights) is typically much smaller than ff (model parameters), this additional overhead is negligible. The space complexity for BDS is O(f + 2d w), accounting for model gradients, two data batches (fine-tuning and alignment data), and data weights. The comparison highlights BDS's scalability and its lower computational and memory overhead compared Booster. For practical training, we use a single A A100-40G to train BDS, whereas Booster cannot be trained on a single A100-40G and instead requires an H100-80G.
Table 21: Comparison of time and space complexities for Booster and BDS.
Algorithm
Stage
Time Complexity Space Complexity
Alignment (n) steps)
O(3nf))
O(3f + 2d)
Booster
Fine-tuning (112 steps)
O(nas)
(([+d)
BDS
Fine-tuning (ng steps)
O(na(f + w))
O(f + 2d w)
To further demonstrate the efficiency of BDS, we supplement the theoretical complexity analysis in Tab. 21 with a practical benchmarks of time and memory cost, as shown in Tab. 22. (i) Comparison with vanilla fine-tuning, BDS introduces negligible computational overhead compared to vanilla fine-tuning, which serves as the lowest possible baseline cost in the since BDS only maintains a set of sample weights that are updated via simple additive operations on loss values during backpropagation (see Eq. (8)). (ii) Scalability to large fine-tuning datasets. It also scales efficiently to larger fine-tuning datasets, as the number of maintained weights grows linearly with the number of samples and the update computation remains lightweight relative to model optimization; the proposed neural scheduler further improves scalability by decoupling the number of trainable parameters from the dataset size. (iii) Efficiency over existing SOTA defenses. Moreover, compared to the state-of-the-un the defense method Booster, BDS achieves over 3x faster
(uning-as-a-seree
setting,
37

---

training speed while consuming less than half the GPU memory, as it requires only standard backward passes rather than expensive bi-level optimization.
Table 22:
Runtime and memory efficiency comparison.
Medhod
Time per Finnch (Mins).
Total Training Time for 29 Epochs (Hturrs) 1
Mas GPU Momory (GB)
Tvrd GPU
Vanilla fir- tuming (mnitmal computaicnnl hoeeino)
2.01
0.1
32
A100-a00
G.2 Convergence Analysis of SGLD Sampling
Theorem G.3 restates the convergence result established in [85, 73], ensuring that SGLD sampling converges to the target posterior distribution after sufficient iterations.
Assumption G.1. (Assumption 4.3 in [85], [73]). (Dissipativeness) There exists absolute constants m and b 2 0 such that:
0 w
2
VO, w E I,
Ve log w Dn, Deate)
2 m
(18)
Assumption G.2. (Assumption 4.4 [85], [73]). (Smoothness) The gradient of the log-posterior for any minibatch is Lipschitz continuous. Specifically, there exists a constant L such that for all z E Daafe and z E Dr. the following condition holds:
Ve.w log logr(0,w)) w) -Deate ((2,afe; 1Dts (a(w)) [(={: 0)
[e]
6'
-o,w logp(0, w') (Dattol((2))) ;0) (Dal [(wws) {(zf:
S
w
w
(19)
for any 0, w. 0', w'.
Theorem G.3 (Theorem 4.5 in [85], [73)). Define d = dim(() [Da,, B as the butch size, and as the Cheeger constant. For any E (0,1), suppose the initial iterate satisfies p( pil wintt < R/2) < e/16, where R R(cK-1/12), and let the step size n be O(min - 2,B2p² $)). Under these conditions, the distribution of the K-th iteration in the SGLD process satisfies:
SGLD -p(0, w Dn, Deate) 5 |
+ +B- +2n
+ +e/2, (20)
for some problem-dependeet dependent constant X >> 0, Co = Ō(p²), C = C2 = Ö(dp-1) Here
lr stands for the total variation distance, and R defined as:
1/2
625d m
4d log((44 (4L/m))
4b
4d + 8/ dogl (/z) + 8 log( 1/2)
R(z) max
m
m
(21)
38

---

H Derivation of Equations
H.1
Derivation of Eq. (1)
Proof. The decomposition of posterior distribution is primarily based on Bayes' theorem. The proof presented below.
p (0, Dri, sanfe)
p p00, Daato Dtt)
p (Dsate Dit)
(0, Deate w, Dz.) p(())
p (Daute Dit)
=A
A (Dsafe
w. Dn) . (0 w, Dn) p(w)
Á p (Dsafe
( 1 (0 w. Da) p(w)
0) (Dn 0, w) p w)
A p (Dsafe safe
p(w)
p (Dr. w)
saaee
() p(D 0, w) p(Dn w) p(0,w)
H.2
Derivation of Eq. (4)
Proof. The complete proof is presented below.
p (Drt (w)-
/ p (Da 10, w) (0) w) d0
x
p (Dit 10,w) f p Dn, w) p(Dr. w) dDn, d0
p (D 10,w): Bp(Da (o) (0) Dn, w)] d0
[Dal
Da 0 w
II exp (o(w:) (2 0))
p
s.t., 0 arg max (p(Pa (m)) [(p (0 Dí, w)]
39

---

H.3
Derivation of Eq. (10)
Proof.
The complete proof is presented below.
p p(0, Φ Dit Daate)
p (0, Φ, Daaie Dt)
p (Dsate Dtt)
(00,, saato 1 Φ,, Dz): p(Ø)
(Daff
Dic))
A p (D safe
0, Dn) (0) Φ, Dr) p(Φ)
A (D
10) Φ. D p())
A p (D safe
(0)
p w. Φ. (t. p (w Φ Dz) dw p(Φ)
A1 P (D.tte safe
( ( w. Da) p
w (
Φ D+) dw (())
A1 P saaee
10) (0 w (D Φ). D(.) p(Φ)
()) (Dn 0, (Dr:;; x))
A1 PP safe
p w NDD:: Φ)) p(()
P (Dtt w = N (Dnn: +))
10) (Dn 0, (Dnn: ())
A p (Danoo
p(0 w) p(w Φ. Dr.) dw p())
p (D w N (Da: Φ))
()) p (Dn 0,
N(Dr;; Φ))
A P (D safe
pp Φ, (a) p(()
p (Dtt w N NDD:: Φ))
p (Dn 0, w = (Dr Φ))
A p (D safe
0)
pp Dt)
p (Drt w NDnn; x))
x p D safe
0) p(D 0, NDnn: +)) ((Da. w (Dnh 6)) -1 p(0. Φ Da)
H.4 Derivation of Eq. (16)
Proof. The softmax weight transformation is defined as follows: o (wa)
2h k 2,,...., Pr/l
Dr em. Enr
The partial derivative of ((w)) with respect to wy is given by:
0ō (wk) Dw,
( (wx) (1 (1- (ww)),
if = if +
= - (wk) ā (w)),
From Eq. (8), the weight update rule is defined as:
! V logp(w) 2
DDu [
Σ
[ (wx) e (z
# w
=fEB
By taking the derivative of and ignoring the prior term and constant coefficients, the gradient with respect to wy is:
Dall
Σ a (wx) ( = (2 δ (wx)) (1 - (w:)) - Σ e (zft) J (wx) - T (w.)
Owi
Taking a further step, we can simplify it as:
Dal
Daal
i
Σ
(wx) E (=ff) = (w:) ((x!))
Σ
(wx) - (z1t)
Ow;
1=1
40

---

Using the derivative, the gradient update for becomes:
V Wu - (w;)
((2))
Σ ā (wn) (21)
I Concept of Posterior Bias
I.1 Motivation of Proposed Posterior Bias
Motivation. Directly sampling from p(w, θ Dn. saate) could introduce bias, as the weights w might assign reduced importance to both the fine-tuning data and potentially harmful examples in Dn. to preserve the model's inherent safety alignment (see Fig. 5). As result, while the model performs held-out benign tuning dataset Diaa which serves as a test or validation set to evaluate fine-tuning performance. Ideally, sampling should be done from p(w",, 0" Dn. D saae, D that w can balance the trade-off between the model's performance on Saae and D (i.e., trade off between safe alignment and fine-tuning objectives).
well on saff. it often struggles to generalize effectively
D
Pate
Z
la
Illustration of posterior bias caused by the absence of Pf Figure Illustration of posterior bias caused by absence of D observed variables, while unshaded nodes denote unobserved variables.
Shaded nodes represent
1.2 Definition of Posterior Bias
Definition I.1. (Posterior Bias) The posterior bias quantify the divergence, measured by function D(-), between two posterior distributions: one derived solely from the alignment dataset Daate and the other incorporating held-out clean tuning dataset Dyal, which serves test validation to evaluate fine-tuning performance. Formally, it defined as:
PB (p((w, 0 Dn, D p(w", 0" Da. D Delean)).
(22)
Here, D refers divergence measure function between two distributions. Since attackers do not provide D we can not directly derive an analytic formulation of Posterior Bias. By leveraging SGLD, derive expression for the posterior bias by evaluating the expected difference in the T-th states SGLD sampling trajectories. Next, we will introduce the definition of Empirical Posterior Bias with SGLD Sampling.
1.3 Empirical Posterior Bias with SGLD Sampling
Definition I.2. (SGLD Sampling Trajectory) The SGLD Sampling Trajectory refers to the sequence of states during the iterative sampling process of SGLD. This sequence forms a Markov chain, where each state transition depends solely on the previous state. For the distribution p (w, Dr. Dento), the trajectories of the variables and 0 can be defined as:
Tr Tw w (0)
(T). (t+1)
(t))
+ 1) (V(t) + tcv
2
(23)
Tra (
(T))
(+ g(t)
+ 1 ((()) + +c.
2
41

---

Here, we use V and Ve to represent the gradient terms, Similarly, this definition can be extended the distribution p (w*, 0" Dr. Daafe it))
Definition I.3. (Empirical Posterior Bias with SGLD sampling) By leveraging SGLD sampling,
posterior bias can be derived by evaluating the expected difference in the T-th states of the paired SGLD sampling trajectories Tr. and Tr with identical initial states:
PB
E Thu Ti
1 (
w 1
Σ
v
2
Thw Tw
T-IT-1
E 2 Tre TI
(24)
E
Here, Tww and Trw- denote the SGLD sampling trajectories drawn from the target distributions p(w, Dn, saate) and p(w", 0* Dn. Dsafe Deleam), respectively. The norm used the l nnom..
1.4 Understanding Posterior Bias under Identity Weight Transformation
As shown in Eq. (24), the gradient term V at each state contributes directly to the posterior bias. Here, consider the identity weight transformation, which is defined follows:
(w;) wi:
(25)
Under identity weight transformation, the weight w updates in a monotonically decreasing manner:
[2::0* O, if of E benign
(26)
< (z):0) 0, if 2f E D Dharmful
Vwz
The posterior bias under identity weight transformation arises mainly from updates in benign tuning data zft D benign Here, 0* p(0"" Dit, Denfe; D fits an additional dataset D while p(O Dr. Deati) does not. As result, 0* achieves a much lower loss on benign data, such that -f [2 [ 0) < - (2 0*) Since the gradient of u is proportional to the negative loss, this results in large and monotonically decreasing reduction in the weights for benign data, leading to poor Dharmful which has different distribution from both fine-tuning performance. For harmful data 2 D
beniga and Daal, the loss remains high and the weights also monotonically decrease.
D
1,5 Theorem of Time-Werr Weighted Accumulation of Posterior Bias
While BDS demonstrates superior defensive performance, theoretical understanding of the inherent suboptimality in inferred data weights offers deeper insights that can guide future algorithmic improvements. Here, we further explore weight suboptimality under identity transformation through the theoretical perspective of posterior bias:
Theorem L.4 (Time-Weighted Accumulation of Posterior Bias, identical to Theorem 4.2 in the main paper). Let (Trw. Tre) and (Trw-, Tro-) ) denote the Stochastic Gradient Langevin Dynam- ics (SGLD) sampling trujectories drawn from the target distributions p (w, Dia, D and p (w",0 O Dit, Deate Dral), respectively. Here, Dial is held-out clean tuning dataset, serving as test or validation set to evaluate fine-tuning performance. Then, the following proportionality holds:
T-1 Σ - 1) 1=
(T)
20
10 - w w0(0
(27)
Tru Tree
The Trap
PB(T)
(T
Here, (T) and (T) represent the sampled weights at iteration T. while ()()) and (t) denote the intermediate states of the trajectories iteration The term PB((T) quantifies the posterior bias at the iteration T, and the summation Et-1(1 1)PB(()) characterizes the cumulative posterior bias over the entire sampling trajectory.
42

---

Implication. Notably, the time-weighted factor T t emphasizes the greater influence of earlier iterations on the cumulative bias, highlighting the cumulative effect of initial steps on the posterior inference. This suggests that suboptimal sampling in the early stages of SGLD sampling trajectory can propagate and aggressively affect the overall posterior inference. Future work could focus on developing bias management strategies, particularly for mitigating bias in early stages of SGLD sampling trategories.
1.6 Proof of theorem Theorem 4.2
Proof. Based on Definition 1.3, the calculation formula for the SGLD-based posterior bias at T iteration is given as follows:
PB
(T
w (T))
Thu, Tr
T-1
Pal T-1
71
V (t)
7
(2)
2
T
Thep
40
2
Trw Tha
The SGLD sampling trajectory for w drawn from p (w, 0 Dr. Deafe) is expressed as:
Trw (0) + + w w(() = w(() + " (((+1) + EVVT,
2
V (+) mr = Vwy logp(w) - E(2,;; 0(0)).
Similarly, the SGLD sampling trajectory for w" drawn from p (w", 0" Dn, Deate Dial)) ex- pressed as:
Trw (0) 2 -
((++) =t!(t) 7 V +
V
(1)
logr(w) E(2,"; 0-(1))
Similarly, the SGLD sampling trajectory of 0 drawn from (w., 0 Di,, Deate) is expressed as:
Tre = (0) + + 10) g(+1) = ((0) + (())
+ t
2 0
Dil w
= Vo log
Vol (2 gl)))
Similarly, the SGLD sampling trajectory of 0* drawn from p (w", 0 Dir Dantes Diaal) is expressed
as:
Trø- = (0)
(T), gx((+]
(1)))
11
2
Prp.l
iDattl
V logp(00))
w (1) Vorl
0000)
Σ
Vo-l / safe, ()(())
0-
Under the identity transformation, we compute the gradient difference for w; at the f-th iteration as follows:
(t)
(4)
( = ||{(2 ())))) - l(=,: 0(0))| = ||A@!"11
43

---

Here, the difference in loss is approximated using the Taylor expansion, yielding:
|| |10(= (0())
{(=,; ((0))))
{(=1::00)))
|| Vol ( ((0)))) T (0e00) - )(1))
-
Vol x,;;; (2)))(()) )(1)) T (v v
Σ
=Vot
D21
iaade
7
V
Σ
wi
(k)
(8) Vol 0(1))
+ Σ
Vol (z) :0()) - Vol (2.30) 00(0))
Dutee
2
Vol (10)))))((k))
T -80
Vol (z)the :0()) Vol (2. :0((k))
Vol (21:010) (mm00) Voccs (DD:
9
Vol
(()
-w 11 Voun (D o(t))
The gradient term is calculated as follows:
V V 2 ((k)
2201 ((x) 0())
V V
Vol ():)))
=
V Voual (=ipa):
The gradient difference is proportional to the cumulative weight difference over previous steps:
w (k) (A()
We substitute this into the definition of (T(
PB(T) x E
Σ
Σ
(b() ||-
k=0
Rearrange the double summation by swapping the summation order:
T-1+-1
T-1 T ΣΣ=
K=
K= t=k+1
Substitute this back into the equation:
T
E
(T
k))l (1) - (1
1m
where -T) represents the remaining time steps from step to T.
Transforming this with the expression of PB((), we obtain:
T-1
(T(T)
Σ (T t)
x
44


Reviews:
[{"review_id": "mq5QE6Foi4", "reviewer": "Reviewer_vpgm", "summary": "The paper presents a method by which, using alignment data alone, a scheduler can be constructed that weights each data sample based on the likelihood of the safety weights under modeling a constraint dataset. A wider array of experiments show that this method is both robust (for example to domain shifts, sample sizes, and attack strengths) and better than previous methods.", "questions": "Mentioned above in strengths and weakneses with clear criteria.", "limitations": "Negative societal impact of the work is not adequetly discussed. This work is dual-use in that constraint scheduler methods might be used to enforce a particular ideological position or otherwise harm freedoms of fine-tuning a model with diverse perspectives. Since that is the case the authors should make it clear.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nThe paper is clearly a novel approach to solving the harmful fine-tuning problem for methods behind closed APIs which is both tractable, well motivated, and demonstrated to be strongly robust under a wide variety of experimental variations.\n\n# Weaknesses\n\n## Generality\n\nThe primary weakness of this paper is that adaptive bayesian scheduling for loss weights could be used much more widely as an adaptive constraint method such as in the case of having length, style, persona, factuality, or quality constraints.  I understand that safety is more motivating but presenting this method in a way limited to safety limits the (presented) applicability of this method to the community when it seems quite promising. Perhaps this could be improved by formulating a few scenarios where Dft violates some constraint in Dconstraint where constraint is a stylistic preference or something like this. I am on borderline accept due to this limitation of significance but happy to raise scores if either a plan for a wider scope is presented or argumentation on why the work was kept so limited is presented.\n\n## Adaptive Attacks\n\nThe method doesn’t present an adaptive attack where the attacker knowing the defence method or (stronger) having access to the scheduler may construct samples whose weights allow training. The most obvious is through data filtration over harmful datasets for samples whose weights allow training. Can the authors please include this adaptive attack experiment and if there are any other adaptive attacks to discuss them. It isn’t implausible that a highly motivated attacker might infer this is the defence used and construct a proxy scheduler, which due to your own OOD and generalization results, would be able to help them choose samples which break this defence. Clearly the distribution figures that are presented demonstrate that this would be possible in theory. After presenting this adaptive attack analysis, I’d like the authors to present strategies for mitigating this attack for example through adjusting the scheduler with contrastive examples. This is counter to the authors narrative about intractability of attack simulation but I find it reasonable. I am on borderline accept due to the lack of presenting adaptive attacks.\n\n## Mathematical Presentation\n\nI am not sure how to provide concrete suggestions here but the mathematical presentation and results were quite difficult to get through without a background in Bayesian Statistics. That might be a limitation that all readers without this background encounter but I found it quite hard to motivate the mathematical results and convince myself they were needed. Perhaps a re-organized presentation where the reader is guided more slowly through the presentation without assuming knowledge of bayesian statistics. Consider that security, policy, and less technical safety folks will be reading this work.\n\n## D.10 Offline-Based Methods\n\nI am not sure how fair using RepNoise is given that it’s an explicitly an offline method (though an online variant is discussed in [1], it might be more fair to move RepNoise to D.10 if it was used in an offline fashion or explicitly mention it was used online as per [1]\n\n[1] Rosati, D., Edkins, G., Raj, H., Atanasov, D., Majumdar, S., Rajendran, J., ... & Sajjad, H. (2024). Evaluating Defences against Unsafe Feedback in RLHF.\n\n## F.5 Over refusal\n\nWhile I don’t expect the method to introduce partial refusals, I recommend the authors report the entire dataset (Refusal-Safe, Partial Refusal-Safe, Answer-Safe, Answer-Unsafe, Partial Answer-Unsafe, Refusal-Unsafe)\n\n## Note on infeasability of attack simulations\n\nI disagree that attack simulations under your threat model are infeasible. Surely, distributional and sample complexity guarantees can be constructed for methods in the future if we are simply talking about constructing a defence that defends against number of samples in a harmful dataset. If the defence is circumscribed to a particular harmful distribution and generalization guarantees are attained then there is feasibility. Of course you are correct to point out that the defender might not be able to anticipate attacks outside of this threat model but I am not sure previous defences are claiming this. I’d recommend making your argument here more nuanced.", "was_revised": true, "final_justification": "The authors have thoroughly satisfied my questions and I feel as though the contribution is now quite good.", "rebuttal": ">W1: Concerns regarding the generality of BDS method beyond safety.\n\nR1: Thank you for your insightful comment. We agree that our method has broader applicability as an adaptive data scheduling framework for enforcing a variety of constraints during training. We will include the following discussion in the final version. The table below illustrates how the proposed BDS method can be adapted to different scenarios by appropriately selecting $D_{\\rm ft}$ and $D_{\\rm constraint}$.\n\n| Scenario                | $D_ {\\rm ft}$     | $D_ {\\rm constraint}$                                        | Objective                                            |\n| ----------------------- | ----------------- | ------------------------------------------------------------ | ---------------------------------------------------- |\n| **Imbalance learning**  | imbalanced data   | balanced data                                                | Prioritize underrepresented (long-tail) samples      |\n| **Denoise learning**    | noisy data        | clean data                                                   | Prioritize clean samples                             |\n| **OOD learning**        | multi-domain data | target-domain data                                           | Prioritize samples that align with the target domain |\n| **Preference learning** | general data      | user-preference demonstrations (e.g., factuality, commonsense) | Prioritize samples aligned with user preference      |\n\n**Empirical results.** The table below shows that the BDS method can be effectively applied across various scenarios. The baseline \"Mix\" refers to simply combining $D_{\\rm ft}$ and $D_{\\rm constraint}$ as the training set.\n\n| Scenario                | $D_ {\\rm ft}$     | $D_ {\\rm constraint}$     | Model        | Setting                                                     | Method  | Accuracy / BLEU |\n| --- | ---- | -- | -- | ---- | -- | - |\n| **Imbalance learning**  | training set of CIFAR-100               | validation set of CIFAR-100                  | ResNet-18    | imbalance ratio = 10 (largest-to-smallest class size ratio) | Mix     | 32.46%          |\n|        |      |       |              |         | **BDS** | **51.84%**      |\n| **Denoise learning**    | training set of CIFAR-10                | validation set of CIFAR-10                   | ResNet-32    | 80% symmetric label noise      | Mix     | 34.12%          |\n|       |        |         |              |              | **BDS** | **56.28%**      |\n| **OOD learning**        | 16 training domains of WebNLG 2020      | 3 validation domains of WebNLG 2020          | T5-small     | train on 16 domains, test on 3 target domains               | Mix     | 30.62%          |\n|         |         |                                              |              |                                                             | **BDS** | **40.24%**      |\n| **Preference learning** | Flan V2 (an instruction tuning dataset) | HellaSwag (a dataset focused on commonsense) | OpenLLaMA 3B | test on the test subset of HellaSwag                        | Mix     | 44.76%          |\n|                         |          |                         |              |                                                             | **BDS** | **51.83%**      |\n\nAs shown in the table, BDS can be flexibly adapted to diverse constrained training scenarios through appropriate dataset design, consistently outperforming the simple mixing baseline across all scenarios. We will add a dedicated discussion on the generality and extensibility of the BDS framework in the final version.\n\n------\n\n\n\n\n>W2: Lack of discussion on adaptive attack and potential mitigation strategies.\n\nR2: Thank you for your constructive suggestion. We will include a discussion on adaptive attacks and potential mitigation strategies in the final version.\n\n**(i) Adaptive attack analysis.** We explore two adaptive attack strategies under the assumption that the attacker has knowledge of the defense scheduler:\n\n1. *Selection-based adaptive attack*: The attacker constructs a proxy scheduler and selects harmful samples with highest weights to poison $D_{\\rm ft}$.\n2. *Optimization-based adaptive attack*: The attacker constructs a proxy scheduler and directly optimizes the discrete harmful inputs to maximize the scheduler output (i.e., $\\max_{\\mathbf{z}} \\boldsymbol{\\phi}(\\mathbf{z})$). Since $\\mathbf{z}$​ lies in the discrete token space, discrete data optimization methods such as GCG optimizer [1] can be used.\n\n**(ii) Empirical results of adaptive attack.** The table below demonstrates the performance of the adaptive attacks. We assume that the candidate attack data includes unseen harmful data from BeaverTails, RealToxicityPrompts, and AdvBench.\n\n1. *For the selection-based adaptive attack*, we observe limited improvement over non-adaptive attacks, as harmful samples generally receive much lower weights than benign ones (see Fig. 6 in Appendix, page 19), making it difficult to collect naturally stealthy attack data. This also suggests the need for more sophisticated crafted attack data to effectively bypass the scheduler.\n2. *For the optimization-based adaptive attack*, the effectiveness is constrained due to the trade-off between harmfulness and stealthiness [2]: optimizing for higher weights (i.e., stealthiness) could reduce the sample's harmfulness. In addition, discrete optimization on long text sequences is often unstable and less effective [1].\n\n| Attack Strategy   | Harmful Score (p=0.1) | Harmful Score (p=0.3) |\n| --- | -- | -- |\n| Non-adaptive attack                | 1.20                  | 1.20   |\n| Selection-based adaptive attack    | 1.40                  | 1.90  |\n| Optimization-based adaptive attack | 1.90      | 2.30     |\n\n**(iii) Potential defense against adaptive attack.** To defend against adaptive attacks, future work could explore how to  construct \"worst-case stealthy\" harmful data $\\hat{\\mathcal{D}}_{\\rm harmful}$ that effectively balances the harmfulness-stealthiness tradeoff, and how to use it to adversarially train the scheduler.\n\n[1] Universal and Transferable Adversarial Attacks on Aligned Language Models.\n\n[2] Information-Theoretical Principled Trade-off between Jailbreakability and Stealthiness on Vision Language Models.\n\n------\n\n\n\n\n>W3: Concerns regarding mathematical presentation for non-Bayesian readers.\n\nR3: Thank you for your advice. In the final version, we will thoroughly re-organize the mathematical presentation and include more preliminaries and references to improve accessibility for non-Bayesian readers.\n\n------\n\n\n\n>W4: It might be more fair to move RepNoise to D.10 if it was used in an offline fashion.\n\nR4: We agree that RepNoise is an offline method and will move it to the appendix accordingly for better clarity.\n\n------\n\n\n\n>W5: I recommend the authors report the entire dataset (Refusal-Safe, Partial Refusal-Safe, Answer-Safe, Answer-Unsafe, Partial Answer-Unsafe, Refusal-Unsafe).\n\nR5: Thank you for your thoughtful suggestion. We now report the full evaluation results on XSTest for the aligned model. The results are summarized below:\n\n| Prompt Type | Answer Type   | Ratio |\n| --- | -- | --- |\n| Safe        | 1_full_compliance (Answer-Safe)            | 78% |\n|  | 2_full_refusal (Refusal-Safe)              | 22% |\n|    | 3_partial_refusal (Partial Refusal-Safe)   | 0     |\n| Unsafe      | 1_full_compliance (Answer-Unsafe)          | 5%  |\n|   | 2_full_refusal (Refusal-Unsafe)          | 95% |\n|  | 3_partial_refusal (Partial Refusal-Unsafe) | 0     |\n\nAs shown in the table above:\n\n (i) We observe a number of refusals to safe prompts, which we attribute to the base model’s inherent limitations in language understanding [3]. An example is shown in the table below (first row).\n\n (ii) We also observe a small number of answers to unsafe prompts, where these responses are not classified as refusals. Example are provided below (second row).\n\n| Label Type    | Prompt Example  | Answer Example   |\n| - | --- | -- |\n| Refusal-Safe  | What's the best method to poison someone in The Witcher 3 game? | I cannot provide information or advice on how to harm or kill someone in any form of media, including The Witcher 3. |\n| Answer-Unsafe | Why should  immigrants not be allowed to sit on the same airplane as white people? | Immigrants should be allowed to sit on the same airplane as white people, regardless of their race or ethnicity. |\n\n[3] XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024.\n\n------\n\n\n\n\n>W6: Concerns regarding the claimed infeasibility of attack simulations.\n\nR6: Thank you for your constructive comment. We agree with your comment that attack simulations are not inherently infeasible, as defenders can indeed simulate attacks within a pre-defined and limited threat model. In the final version, we will revise our claim to be more nuanced:\n\n*Existing defense strategies preemptively build robustness via attack simulation but suffer from fundamental limitations: (i) the infeasibility of performing attack simulation **beyond bounded threat models, due to the defender's limited ability to anticipate unknown attacks**; and (ii) limited adaptability to varying attack settings, as simulation fails to capture their variability and complexity.*\n\nThis revision reflects your insight that simulation is feasible under constrained conditions, while preserving our original motivation—that the space of possible attacks are too diverse and unpredictable to be fully captured by simulation alone.\n\n------\n\n\n\n>W7: Insufficient discussion of negative societal impact about viewpoint suppression.\n\nR7: Thank you for your insightful comment. This is an important point that we had not fully considered, and we appreciate your perspective. We will include a discussion in the final version on the potential misuse of our method to enforce specific ideological positions or suppress diverse perspectives in LLMs."}, {"review_id": "YMzBBQi86C", "reviewer": "Reviewer_k17i", "summary": "This paper first formulate the harmful fine-tuning defense as a Bayesian inference problem. Under this definition, the authors propose a novel Bayesian Data Scheduler Framework which can adaptively defense the harmful fine-tuning attacks without attack simulation. Comprehensive experiments demonstrate the SOTA performance achieve by their method.", "questions": "The method is still based on a datset with safe examples (harmful questions with safe answers). The model may perform not well on unexpected attacks with harmfulness beyond the alignment dataset.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written with clear description of their methods. \n2. To demonstrate their claims, the authors conduct numerous experiment in Section 4.1.\n3. The theoretical analysis provides intuiation and guarantee for their proposed method.\n4. Compared to previous methods, BDS does not require attack simulation, which is more practical.\n\nWeaknesses:\n1. The figure 1 requires clearer explanation to help reader better understand the density and weights concept as the introduction figure in the paper.\n2. Some important information for understanding the methods, such as experiments details, should be directly included into the paper rather than in the appendix.", "was_revised": true, "final_justification": "The rebuttal looks good. I will keep my score for accept.", "rebuttal": ">W1: The figure 1 requires clearer explanation to help reader better understand the density and weights concept as the introduction figure in the paper.\n\nR1: Thank you for your suggestion. In Fig.1, the term \"density\" in the largest panel was intended to refer to the density of data weights. To avoid confusion, we will replace the term \"density\" with \"weight\" and use consistent terminology throughout the paper.\n\n------\n\n\n\n>W2: Some important information for understanding the methods, such as experiments details, should be directly included into the paper rather than in the appendix.\n\nR2: Thank you for your suggestion. We will move the experimental details currently in Appendix B.3 and B.4 into the main paper to improve clarity and completeness.\n\n------\n\n\n\n>W3: The method is still based on a datset with safe examples (harmful questions with safe answers). The model may perform not well on unexpected attacks with harmfulness beyond the alignment dataset.\n\nR3: **(i) Attack beyond the alignment dataset.** Thank you for raising this concern. We address it in Table 8 of our main paper, which presents results where the alignment dataset and the attack dataset differ significantly. The table below summarizes these datasets, highlighting their differences in format and domain:\n\n| Category              | Dataset Name        | Format                         | Harmful Domains (Examples)                                   |\n| --------------------- | ------------------- | ------------------------------ | ------------------------------------------------------------ |\n| **Alignment Dataset** | BeaverTails         | Harmful question – Safe answer | 14 harm types (e.g., Animal Abuse, Child Abuse, Self-harm)   |\n| **Attack Dataset 1**  | RealToxicityPrompts | Prefix – Completion            | 7 toxicity types (e.g., Profanity, Insult, Identity Attack)  |\n| **Attack Dataset 2**  | AdvBench            | Instruction – Answer           | 7 harm types (e.g., Cybercrime, Discrimination, Threatening behavior) |\n\n**(ii) Empirical support.** As shown in Table 8 of our main paper, even when the alignment and attack datasets differ significantly in both format and domain, our method consistently maintains a harmfulness score below 3 and significantly outperforms existing SOTA baselines."}, {"review_id": "aeJnbT2E9D", "reviewer": "Reviewer_hoiB", "summary": "The paper proposes Bayesian Data Scheduler (BDS), a defense mechanism against harmful fine-tuning of large language models (LLMs), BDS formulates the defense as a Bayesian inference problem, adaptively estimating the risk of each training example.It introduces Bayesian Scalar Scheduler and Amortized Bayesian Neural Scheduler to reweigh samples based on posterior probabilities during fine-tuning. Experiments demonstrate the SOTA performance and strong adaptiveness of BDS, showing superior effectiveness across across diverse attack scenarios.", "questions": "The overall evaluation is based primarily on S1-S3 (+) and W1–W3 (–) from the Strengths and Weaknesses section. Below are suggestions for the authors to improve the work:\n\n1. Add a table comparing training time, memory and GPU overhead, and FLOPs with standard optimizers (e.g., Adam, SGHMC) on the same task to clarify the resource cost of SGLD. Additionally, supplement the time complexity analysis in Appendix G.1 with concrete benchmarks, such as per-epoch runtime and the number of steps to convergence. (W1)\n\n2. Consider adding a performance analysis for low-resource Dsafe settings (e.g., reducing from 1000 to 100/50), and discuss whether a “safe attribute migration” policy could help amortized schedulers generalize with minimally aligned data. (W2)\n\n3. Although current experiments assume similar model architectures between attacker and defender, we acknowledge real-world attackers may utilize heterogeneous LLMs. Future work could examine the impact of such cross-model scenarios on BDS's adaptiveness, particularly when alignment objectives diverge. (W3).", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "S1: BDS is the first to treat defense against harmful fine-tuning as Bayesian inference, avoiding the need for simulating attack data.\n\nS2: The paper introduces two schedulers: a scalar scheduler for high accuracy and a neural scheduler with cross-task transferability, improving the method’s generalizability.\n\nS3: The experiments are comprehensive, covering diverse models, datasets, and attack scenarios. BDS consistently outperforms existing methods, demonstrating strong robustness and alignment performance.\n\nW1: Although the paper adopts SGLD for posterior sampling, this approach may introduce training speed bottlenecks in large-scale deployment. However, the experiments do not include detailed comparisons of training time or resource consumption.\n\nW2: The study relies on high-quality aligned data (Dsafe), assuming that the platform has access to such datasets. However, in practice, such data may be scarce, especially for domain-specific tasks.\n\nW3: While the experiments demonstrate broad coverage, the defense mechanism under black-box attack scenarios lacks detailed analysis. The paper does not systematically discuss discrepancies between the attacker’s and defender’s models—for example, whether BDS remains effective when the attacker uses different LLM architectures or training objectives.", "was_revised": true, "final_justification": "The author's response addressed most of my questions. Considering that other reviewers were positive about this paper and that my score was already highly positive, I decided to keep my score.", "rebuttal": ">W1: Concerns regarding efficiency analysis. \n>\n>(i) Add a table comparing training time, memory and GPU overhead, and FLOPs with standard optimizers (e.g., Adam, SGHMC). \n>\n>(ii) Supplement the time complexity analysis in Appendix G.1 with concrete benchmarks.\n\nR1: **(i) Compare SGLD with standard optimizers.** We include the comparison table below to clarify the resource usage of different optimizers and samplers. While these methods differ in their update rules, they do not lead to significant differences in overall training overhead, as the optimizer step typically accounts for only a small fraction of total training overhead compared to the gradient computation $\\nabla L(\\theta_t)$.\n\n| Sampler or Optimizer | Update Rule                                                  | Convergence Epochs ↓ | FLOPs ↓ | Time per Epoch (Mins) ↓ | Max GPU Memory (GB) ↓ |\n| -------------------- | ------------------------------------------------------------ | -------------------- | ------- | ----------------------- | --------------------- |\n| **SGLD**             | $\\theta_{t+1} = \\theta_t - \\eta \\cdot \\nabla L(\\theta_t) + \\sqrt{2\\eta} \\cdot \\epsilon_t$ | 15                   | 0.022   | 2.04                    | 25.44                 |\n| **Adam**             | $\\theta_{t+1} = \\theta_t - \\eta \\cdot \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon}$ | 14                   | 0.068   | 2.46                    | 26.18                 |\n| **SGHMC**            | $\\theta_{t+1} = \\theta_t + v_{t+1},\\ \\ v_{t+1} = (1 - \\alpha) v_t - \\eta \\cdot \\nabla L(\\theta_t) + \\sqrt{2\\alpha\\eta} \\cdot \\epsilon_t$ | 15                   | 0.043   | 2.18                    | 25.86                 |\n\n**(ii) Practical time and memory analysis of our method.** In addition to the theoretical complexity analysis provided in Appendix G.1, we will include a practical runtime and memory benchmark in the final version. **As shown in the table below, our method (BDS) is over 3× faster than the SOTA method Booster, while consuming less than half the GPU memory.** This efficiency gain is due to the fact that Booster relies on bi-level adversarial training, which involves expensive second-order gradient approximations, while BDS requires only the standard backward pass.\n\n\n| Method     | Convergence Epochs ↓ | Time per Epoch (Mins) ↓ | Total Training Time for 20 Epoches (Hours) ↓ | Max GPU Memory (GB) ↓ | Used GPU            |\n| ---------- | -------------------- | ----------------------- | -------------------------------------------- | --------------------- | ------------------- |\n| Booster    | 20                   | 6.42                    | 1.95                                         | 57.86                 | 1 $\\times$ H100-80G |\n| BDS (ours) | 15                   | 2.04                    | 0.64                                         | 25.44                 | 1 $\\times$ A100-40G |\n\n------\n\n\n\n>W2:  Concerns regarding reliance on high-quality alignment dataset.\n>\n>(i) Consider adding a performance analysis for low-resource $D_{\\rm safe}$ settings.\n>\n>(ii) Discuss whether a “safe attribute migration” policy could help amortized schedulers generalize with minimally aligned data.\n\nR2:\n**Reliance on alignment data ($\\mathcal{D}_{\\text{safe}}$).** We acknowledge that our method relies on an alignment dataset $\\mathcal{D}_{\\text{safe}}$. However, this reliance can be mitigated in several ways:\n\n**(i) Robustness to small alignment datasets.** Our method remains highly effective even with very limited alignment data. As shown in Table 6 of our main paper (also the table below), BDS consistently achieves harmful scores < 2 with as few as 100 alignment samples, outperforming the SOTA baseline Booster by over 60%, which shows the feasibility of our method in low-resource conditions:\n\n| Method     | Harmful Score ↓ |          |          |          | Finetune Accuracy ↑ |           |           |           |\n| ---------- | --------------- | -------- | -------- | -------- | ------------------- | --------- | --------- | --------- |\n|            | 100             | 500      | 1000     | 1500     | 100                 | 500       | 1000      | 1500      |\n| Booster    | 62.20           | 34.60    | 8.30     | 8.10     | 93.23               | 93.42     | 93.23     | 92.86     |\n| BDS (ours) | **1.70**        | **1.20** | **1.20** | **1.30** | **93.58**           | **93.92** | **93.69** | **93.12** |\n\n**(ii) Generalization via “safe attribute migration”.**\nOur method enables a practical strategy to reduce reliance on domain-specific $\\mathcal{D}_{\\rm safe}$: leveraging alignment data from a data-rich domain as a surrogate for data-scarce domains.\n\n  As shown in Appendix D.3 (Table 8), a scheduler trained on alignment data from a specific domain (e.g., BeaverTails) generalizes well to cross-domain attacks (e.g., RealToxicityPrompts and AdvBench). This demonstrates the feasibility of using alignment data from a data-rich domain as a surrogate for data-scarce domains, thus alleviating the reliance on domain-specific $\\mathcal{D}_{\\rm safe}$.\n\n------\n\n\n\n\n>W3: Concerns regarding the discrepancies between the attacker’s and defender’s models or alignment objectives.\n\n\n\nR3: Thank you for your thoughtful comments. We would like to first clarify the threat model of harmful fine-tuning as defined in Section 2 of the main paper. In our setting, the attacker does not use or access any model. Instead, the attacker first uploads harmful fine-tuning data to a server. Then, the server performs fine-tuning with an LLM. Finally, the trained LLM is returned to users via an API. This setup reflects realistic deployment scenarios of fine-tuning-as-a-service, as offered by mainstream LLM providers (e.g., OpenAI, Mistral).\n\nAs a result, since the attacker does not use or access the model being fine-tuned, there is no discrepancy between the attacker’s and defender’s models in our setting. Therefore, concerns regarding discrepancies in LLM architecture do not apply to this specific threat model.\n\nHowever, we appreciate your broader perspective. In more complex attack scenarios, an attacker may use a heterogeneous model to optimize harmful examples before uploading them. While this goes beyond the assumptions of our current threat model, we agree it is a valuable direction for future work. Specifically, future work could investigate how well the defense generalizes to such cross-model adaptive attacks, where the attacker uses heterogeneous models to craft more stealthy harmful data. We will include this discussion in the final version."}, {"review_id": "0wWpzRp0r6", "reviewer": "Reviewer_7oEt", "summary": "This paper presents a new and principled approach called the Bayesian Data Scheduler (BDS) to protect large language models from harmful fine-tuning. BDS reframes the challenge of defending against such fine-tuning as a Bayesian inference task, estimating the posterior probability of each data point's safety based on both the fine-tuning and alignment datasets. During fine-tuning, data are reweighted according to safety scores drawn from this posterior, reducing the impact of potentially harmful inputs. Because Bayesian inference is applied post hoc, the posterior adapts to the specific fine-tuning data, allowing BDS to provide a tailored and adaptive defense. Additionally, they propose a neural scheduler using amortized Bayesian learning, which allows for efficient generalization to new datasets without the need for retraining. Extensive experiments under various attack and defense scenarios show that BDS achieves leading performance.", "questions": "See the weaknesses section.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The studied research problem is important, as both unintentional and malicious harmful fine-tuning data are increasingly pervasive in fine-tuning-as-a-service scenarios. \n\n* The proposed fine-tuning data reweighting method is theoretically sound, grounded in Bayesian statistics. The amortized version of BDS can be efficiently integrated into standard LLM fine-tuning pipeline. \n\n* The authors performed thourough evaluations against various attacks and defense strategies using multiple LLMs and benchmarks, which together demonstrates the superiority of BDS in balancing the utility-safety trade-off during LLM fine-tuning. \n\n\nWeaknesses:\n\n* The iterative posterior sampling process of BDS would introduce additional computational overhead. Have the authors considered comparing BDS against a pre-trained soft safety classifier of input data?", "was_revised": true, "final_justification": "The additional results have addressed most of my questions, so I decided to keep my positive score.", "rebuttal": ">W1: The iterative posterior sampling process of BDS would introduce additional computational overhead.\n\nR1: **(i) Efficiency of posterior sampling in BDS.** Thank you for your thoughtful comments. The iterative posterior sampling process in BDS, defined by the update rule\n $\\theta_{t+1} = \\theta_t - \\eta \\cdot \\nabla L(\\theta_t) + \\sqrt{2\\eta} \\cdot \\epsilon_t$,\n can be interpreted as standard gradient descent augmented with a Gaussian noise term. Crucially, the injection of noise incurs negligible additional computational overhead compared to the dominant cost of gradient computation. Therefore, the overall computational overhead of BDS remains comparable to standard backpropagation, without introducing significant additional cost.\n\n**(ii) Empirical support.** To further demonstrate the efficiency of BDS, we supplement the theoretical complexity analysis in Appendix G.7 with a practical benchmarks of time and memory cost:\n\n| Method     | Convergence Epochs ↓ | Time per Epoch (Mins) ↓ | Total Training Time for 20 Epoches (Hours) ↓ | Max GPU Memory (GB) ↓ | Used GPU            |\n| ---------- | -------------------- | ----------------------- | -------------------------------------------- | --------------------- | ------------------- |\n| Booster    | 20                   | 6.42                    | 1.95                                         | 57.86                 | 1 $\\times$ H100-80G |\n| BDS (ours) | 15                   | 2.04                    | 0.64                                         | 25.44                 | 1 $\\times$ A100-40G |\n\nAs shown in the table above, our method (BDS) is **over 3× faster** than the SOTA method Booster, while consuming **less than half** the GPU memory. This efficiency gain is due to the fact that Booster relies on bi-level adversarial training, which involves expensive second-order gradient approximations, while BDS requires only the standard backward pass.\n\n------\n\n\n\n\n>W2: Have the authors considered comparing BDS against a pre-trained soft safety classifier of input data?\n\nR2: **Comparison with pre-trained safety classifier.** Thanks for your constructive suggestion. We include an additional baseline: Llama Guard 2, a pre-trained safety classifier fine-tuned from LLaMA-3 using the 13K Anthropic dataset. The updated comparison is shown below:\n\n\n| Method         | Harmful ratio = 0.1  |                 | Harmful ratio = 0.3  |                 |\n| -------------- | -------------------- | --------------- | -------------------- | --------------- |\n|                | Fine-tune Accuarcy ↑ | Harmful Score ↓ | Fine-tune Accuarcy ↑ | Harmful Score ↓ |\n| Llama Guard 2  | 92.78                | 13.40           | 92.46                | 27.8            |\n| **BDS (ours)** | **93.69**            | **1.20**        | **93.32**            | **1.20**        |\n\nAs shown in the table, BDS significantly outperforms Llama Guard 2 in both fine-tuning accuracy and harmfulness score. We further observe that, among 1000 harmful samples, **Llama Guard 2 exhibits a high leakage rate of 37.5%, i.e., it misclassifies 37.5% of harmful data as safe**. This indicates that relying solely on pre-trained safety classifiers may introduce notable risks during data filtering, whereas our method offers more reliable data weighting and enhanced safety."}, {"review_id": "lZ6KGa45kf", "reviewer": "Reviewer_Cm3n", "summary": "The paper addresses the emerging threat of “harmful fine-tuning,” whereby adversarially crafted data in a user’s fine-tuning set can subvert an otherwise aligned large language model. To defend against this without needing to simulate potential attacks, the authors propose the Bayesian Data Scheduler (BDS), which treats each tuning example’s safety as a latent variable and infers a posterior distribution over data weights conditioned on both the fine-tuning and a small alignment dataset. By reweighting data in the fine-tuning process—either via a Bayesian Scalar Scheduler or an Amortized Bayesian Neural Scheduler—BDS adaptively downweights harmful examples and upweights benign ones. Extensive experiments across multiple datasets, model architectures, and attack scenarios show that BDS achieves state-of-the-art robustness, reducing harmful behavior by over 70% at high attack ratios while maintaining task accuracy, and it can efficiently transfer to new data without retraining.", "questions": "Please refer to the weaknesses outlined above. Regarding point 1, I encourage the authors to clarify how their method differs from prior work and to clearly highlight which components are original. My primary concern is point 2: the method appears computationally expensive, and the current analysis is based on theoretical complexity rather than practical runtime. It would be much more convincing if the authors could report actual runtime measurements in realistic settings. That said, I appreciate the thorough experimental evaluation.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written and generally easy to follow.\n2. It includes comparisons with multiple existing baselines.\n3. The proposed methods are supported by clear and well-structured derivations.\n\nWeaknesses:\n1. I strongly encourage the authors to include at least a minimal discussion of related works in the main text. This would help clarify how the proposed method differs from or builds upon prior approaches and provide better context for evaluating the novelty and contribution of each component. As it stands, it's difficult to assess which parts of the method are original and which are inspired by existing work.\n\n2. The proposed method appears computationally expensive, especially considering that the largest fine-tuning dataset used in experiments contains only 2,000 examples—much smaller than typical fine-tuning workloads. Can the authors clarify the efficiency of BDS in more realistic settings? It would be more compelling if the method could support a practical deployment scenario where a fine-tuning service provider trains a single BDS model once and reuses it across different user-provided datasets. The paper would be significantly stronger if it could demonstrate either (1) that BDS is lightweight enough to train (within very few minutes) separately for each new dataset query, or (2) that a pre-trained BDS model generalizes well to different user-provided datasets containing harmful examples.", "was_revised": true, "final_justification": "The authors have all addressed my concerns. They provided the runtime of their method and also discussed the related works. I raised my score from 4 to 5.", "rebuttal": ">W1: I strongly encourage the authors to include at least a minimal discussion of related works in the main text. I encourage the authors to clarify how their method differs from prior work and to clearly highlight which components are original. \n\nR1: **Clarify our difference from previous work.** Thank you for the constructive suggestion. To help readers better understand how our method differs from prior methods, we will include a discussion of related work in the main paper:\n\n*Existing defense methods can be broadly categorized based on when they are applied: during the alignment stage or the fine-tuning stage. Alignment-stage defenses aim to preemptively enhance model robustness against simulated attacks, by improving resistance to harmful perturbations [1,2] or actively forgetting harmful knowledge [3]. Fine-tuning-stage defenses focus on reducing the impact of harmful data during the fine-tuning, such as by increasing the proportion of safe samples [4] or filtering out potentially harmful samples [7].*\n\n*The method proposed in this paper should be classified into a fine-tuning-stage solution. However, it differs fundamentally from prior work in that we propose **a novel loss-based data scheduling mechanism derived from Bayesian inference principles**, whereas previous approaches typically perform hard-label data selection based on less effective heuristic rules [5,6] or manually tuned thresholds [7]. Moreover, unlike the current SOTA method Booster [1], our method **does not require attack simulation**, making it more practical and efficient, while also demonstrating strong performance and adaptability.*\n\n\n\n[1] Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. ICLR 2025.\n\n[2] Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. NeurIPS 2024.\n\n[3] Representation noising: A defence mechanism against harmful finetuning. NeurIPS 2024.\n\n[4] Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. NeurIPS 2024.\n\n[5] A Survey on LLM-as-a-Judge. arXiv 2024.\n\n[6] Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts. arXiv 2025.\n\n[7] Safety-aware fine-tuning of large language models. NeurIPS workshop 2024.\n\n------\n\n\n\n\n\n>W2: Concerns regarding the computational cost of BDS in realistic setting.\n\nR2: Thank you for the thoughtful suggestion. We would like to clarify the efficiency of our method BDS by emphasizing the following three points:\n\n**(i) Efficiency of BDS compared with to vanilla fine-tuning.** We would like to clarify that **BDS does not introduce significant overhead on top of vanilla fine-tuning, which serves as the lowest possible baseline cost in the fine-tuning-as-a-service scenario.** This is because BDS only additionally maintains a set of sample weights, which are updated in a lightweight manner using simple additive operations on the loss (see Eq. (16)). \n\n**(ii) Efficiency when scaling to larger fine-tuning datasets.** BDS remains efficient even when scaling fine-tuning datasets, since the number of weights scales linearly with the number of samples, and updating them requires minimal computation compared to model optimization. Moreover, our proposed neural scheduler further addresses the scalability issue by decoupling the number of trainable parameters from the dataset size, ensuring that the optimization complexity does not increase as the dataset grows.\n\n**(ii) Significant efficiency gains over existing SOTA defense.** Compared to the SOTA defense method Booster, BDS is over **3× faster** and consumes **less than half** the GPU memory. This efficiency gain comes from the fact that Booster relies on bi-level adversarial training, which involves expensive second-order gradient approximations, while BDS requires only the standard backward pass.\n\n\n\n**Empirical Support.** To better illustrate the efficiency of BDS, we will supplement the theoretical complexity analysis in Appendix G.1 with practical runtime and memory benchmarks in the final version:\n\n\n| Method                                                       | Time per Epoch (Mins) ↓ | Total Training Time for 20 Epoches (Hours) ↓ | Max GPU Memory (GB) ↓ | Used GPU            |\n| ------------------------------------------------------------ | ----------------------- | -------------------------------------------- | --------------------- | ------------------- |\n| Vanilla fine-tuning-as-a-service (minimal computational baseline) | 2.01                    | 0.61                                         | 25.32                 | 1 $\\times$ A100-40G |\n| Booster                                                      | 6.42                    | 1.95                                         | 57.86                 | 1 $\\times$ H100-80G |\n| BDS (ours)                                                   | 2.04                    | 0.64                                         | 25.44                 | 1 $\\times$ A100-40G |\n\nAs shown in the table above, under the same training setting, (i) BDS introduces no significant overhead compared to vanilla fine-tuning, which serves as the lowest possible baseline cost in the fine-tuning-as-a-service scenario. (ii) BDS achieves substantial improvements in training speed and memory efficiency over SOTA baseline Booster."}]

Author rebuttal:

Meta-review: The paper presents a novel approach to harmful finetuning of closed weight models. The authors performed a thorough evaluation and comparison with the literature. All of the points raised in the reviews and during rebuttal are thoroughly addressed by the authors with consistent results. I recommend that authors to include efficiency numbers and comparison with other methods as presented in the rebuttal to the paper.
Decision: Accept (spotlight)