Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Authors: Liang CHEN
Xueting Han
Li Shen
Jing Bai
Kam-Fai Wong
Venue: ICML 2025 poster (2025)
Keywords: large language models
robustness
safety alignment
harmful Fine-tuning
forgetting
group distributionally robust optimization
balance learning
ai safety
Primary area: deep_learning->large_language_models
TL;DR: A new alignment method that improves safety in LLMs by identifying and upweighting vulnerable alignment data.

Abstract: Harmful fine-tuning (HFT), performed directly on open-source LLMs or through Fine-tuning-as-a-Service, breaks safety alignment and poses significant threats. Existing methods aim to mitigate HFT risks by learning robust representation on alignment data or making harmful data unlearnable, but they treat each data sample equally, leaving data vulnerability patterns understudied. In this work, we reveal that certain subsets of alignment data are consistently more prone to forgetting during HFT across different fine-tuning tasks and exhibit lower robustness compared to other subsets. Inspired by these findings, we propose Vulnerability-Aware Alignment (VAA), which calculates data vulnerability, partitions data into "vulnerable" and "invulnerable" groups, and encourages balanced learning using a group distributionally robust optimization (Group DRO) framework. Specifically, VAA learns an adversarial sampler that samples examples from the currently underperforming group and then applies group-dependent adversarial perturbations to the data during training, aiming to encourage a balanced learning process across groups. Experiments across four fine-tuning tasks demonstrate that VAA significantly reduces harmful scores while preserving downstream task performance, outperforming state-of-the-art baselines.

Vulnerabiliy-Aware Alignment:
Mitigating Uneven Forgetting in Harmful Fine-Tuning
Liang Chen
Xueting Han
Li Shen
Jing Bai Kam-Fai Wong
Abstract
searchers and developers customize these models using their own datasets. However, recent studies (Qi et al., 2024; Yang et al., 2023; Zhan et al., 2023) reveal a new critical safety challenge: mue-tuning aligned models can undermine their safety alignment. This issue manifests in two ways: first, LLMs' safety alignment can be broken by harmful fine- tuning (HFT) on datasets containing even a small number of harmful examples; second, safety performance also com- promises even when fine-tuned on entirely benign datasets.
Harmful fine-tuning (HFT), performed directly on open-source LLMs or through Fine-tuning- as-a-Service, breaks safety alignment and poses significant threats. Existing methods aim to miti- gate HFT risks by learning robust representation alignment data or making harmful data un- learnable, but they treat each data sample equally, leaving data vulnerability patterns understudied. In this work, we reveal that certain subsets of alignment data are more prone to forgetting dur- ing HFT across different fine-tuning tasks and exhibit lower robustness compared to other sub- sets. Inspired by these findings, we propose Vulnerability- Alignment (VAA), which estimates data vulnerability, partitions data into "vulnerable" and "invulnerable" groups, and en- courages balanced learning using a group dis- tributionally robust optimization (Group DRO) framework. Specifically, VAA learns an adver- sarial sampler that samples examples from the currently underperforming group and then applies group- dependent adversarial perturbations to the data during training, aiming to encourage bal- anced learning process across groups Experi- ments across four fine-tuning tasks demonstrate that VAA significantly reduces harmful scores while preserving downstream task performance, outperforming baselines. The code is available https: /tt///gitt github..... www/chan//../ecccco ChanL iang/ VVA.
Prior approaches to mitigating harmful fine-tuning can be categorized into three classes: (a) alignment-stan meth- ods that enhance the model's safety robustness during the alignment stage before HFT (Huang et al., 2024d; Rosati et al., 2024a); (b) fine-tuning methods that regulate the fine- tuning process during HFT (Mukhoti et al., 2023; Huang et al., 2024c); and (c) post-finetuning methods that repair compromised models after HFT (Yi et al., 2024; Hsu et al., 2024). Among these, alignment-staan methods are more controllable and computationally efficient, as they only need to be applied once. Existing alignment-stage methods aim to mitigate HFT risks by learning robust representation on alignment data (Huang et al., 2024d) making harmful data unlearnable reduce their impact (Huang et al., 2024b; Rosati et al., 2024a). Nevertheless, these methods fail to investigate the different vulnerabilities aligument data under HFT scenarios and treat all data uniformly, limiting their overall effectiveness. To address this limitation, we aim answer the following questions:
Are certain subsets of alignment data more eas- ily compromised during harmful fine-tuning? If so, how can we leverage their characteristics to design better alignment-stune stage methods against harmful fine-tuning?
1. Introduction
The open-source availability of large language models (L..Ms) and Fine-tninggass---erriiee platforms enables re-
We first investigate how different alignment examples be- have during the HFT process. Our analysis reveals uneven vulnerability among alignment examples: Some subsets of alignment data are highly susceptible to compromise dur- ing HFT, while others remain robust. Through extensive experiments, we demonstrate that this uneven vulnerabil- ity persists across various fine-tuning tasks and different proportions of harmful data, with a significant overlap in
'The Chinese University of Hong Kong Microsoftt Research Asia 'Shenzhen Campus of Sun Yat-sen University. Correspon- dence to: Xueting Han <chrihan microsoft.com>>, Kam-Fai Wong <kfwong sece@scccuhll edu.uk...du.kk...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Aulrera Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Step 1: Adv. sampling 9
Step Group-wise perturbing
Step3: Objective
Alignment Dataset
Empirical Distribution
Eg
(Invulnerable)
GE
Dataset
loss
G2 (vulnerable)
Step 5: Grad Descend
Step 4: Mirar Ascend
Stage 1: Group Estimation
Stage 2: Vulnerability-Aware Alignment
Figure 1: Overview of the Vulnerability-Awwe Alignment. In the first stage, the vulnerability of the alignment dataset is analyzed, and the data is partitioned into groups using the strategy described in Section 2.2. In the second stage, the adversarial learning framework consists of LLM and adversarial sampler, parameterized by probability simplex over the partitioned groups. At each alignment step, the sampler samples a hard group and applies a group-specific perturbation to challenge the LLM. After training, only the LLM is retained, and the sampler is discarded.
the data that is prone to being compromised. These find- ings indicate that the forgetting behavior HFT is highly data-dependent, Moreover, we observe that these vulnerable examples often underrepresented in the alignment data and often exhibit lower robustness to weight perturbations. This phenomenon likely stems from imbalanced learning during the alignment stage, where certain data subsets are insufficiently learned.
harmfulness score while preserving downstream task perfor- mance.
2. Revisiting HFT From Data Perspective
In this section, we revisit the problem of harmful fine-tuning (HFT) from a data perspective. Specifically, we aim to answer two questions: (1) Are certain subsets of data more vulnerable to being forgotten during harmful fine-tuning. (2) If so, what are the characteristies of these data?
To incorporate this prior knowledge about the varying vulnerability levels of alignment examples, propose Vulnerability-Awaan Alignment (VAA), method designed to explicitly promote balanced learning between vulnerable and invulnerable data groups. VAA builds upon the group distributionally robust optimization (Group DRO) frame- work, which iteratively optimizes the worst-performing group using a composite objective that linearly combines loss minimization and robustness enhancement, thereby ducing performance disparities across data groups.
2.1. Preliminaries of Harmful Fine-tuning
Considered Scenario. We study a two-stage pipeline for fine-tuning-as-a-ee as--ee-vce........e..... In the first stage, L.M under- goes safety alignment using the service provider's curated alignment dataset. In the second stage, the aligned model is fine-tuned on user-provided data for personalization. The resulting model is deployed on the provider's infrastructure to serve personalized responses to user prompts.
Specifically, VAA implements Group DRO as two-player game between a hard group sampler (the adversary) and the LLM (the target model). At each training step, the adversary first chooses a batch of data from the relatively hard group according to an adversarially updated sampling probability, then applies group-dependent adversarial perturbation to challenge the LLM. Conversely, the LLM strives to mini- mize its loss on the data proposed by the adversary. Through adversarial training (AT), both parts improve iteratively. Ide- ally, at convergence, the adversary represents uniform distribution across different groups, as the LLM learns groups equally well.
Threat Model. The attack surface lies within the user fine- tuning stage, where users can upload datasets containing both benign and harmful examples. Let p [0, 1] represent the proportion of harmful example in the uploaded dataset. Recent work (Qi al., 2024) demonstrates that even when = O (purely benign data), fine-tuning can degrade the model's safety guarantees, presenting significant risks for service providers.
Assumptions. Following (Qi et al., 2024), we assume the service provider maintains an alignment dataset, where each sample consists of pair of potentially malicious prompt and its corresponding safe response. This dataset serves as safety demonstration data for alignment preservation.
We validate method four different fine-tuning tasks. Comprehensive evaluations demonstrate that, compared to baselines, our method consistently reduces the model's 's
2

---

Auhe Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
(a) Forgetting acroes Rates
Forgetting Varied Tasks
ForgotNam >0, and unforgotlen if ForgotNume is 0.
100
100
Definition of Common Forgetting. To analyze the con- sistency of forgetting across different fine-tuning settings (c.g., different poison ratios or tasks), we define the common forgotien set as the intersection of forgotten examples across all settings under comparison. Let A, denote the set of for- gotten examples under setting i, and N the total number of alignment examples. We define:
Percentage (15)
60
60
40
40
20
20
0
SST2(p=018)
55T2lp=10%%
2512(p=200)
SST2(p=10%)
GSMSK(p=10%)
AGNews(pp100% 10%)
AA A2 0 As!
Farget
Comman Farget
Unfarget
Common Unforget
CommonForgot
(2)
N
| A1 n Asl
Figure 2: Analysis of forgetting behavior: (a) Forgetting pat- terns on a fine-tuning task (SST2) with varying poison rates (0%, 10%, and 20%); (b) Forgetting patterns across three different fine-tuning tasks (SST2, GSM8K, and AGNews) with fixed 10% poison rate.
CommonForgotRatio = min(|A1l |A4)) |As))
The CommonForree metric measures the absolute propor- tion of alignment examples that are consistently forgotten across all settings. In contrast, CommonForgg Forot Rat io normalizes this overlap by the size of the smallest forgetting set, reflecting the proportion of shared forgotten examples relative to the most constrained setting. This metric cap- tures the degree to which forgetting patterns transfer across different fine-tuning conditions.
2.2. Characterizing Data Vulnerability in HFT
Next, we examine harmful fine-tuning from a data perspec- tive by investigating the forgetting behavior of different examples during harmful fine-tuning. Our goal is to identify whether certain subsets of alignment data are more suscep- tible forgetting and to understand the characteristics of these subsets.
Forgetting in HFT is mata-Dependent. We investigate how the addition of harmful data affects forgetting by com- bining the SST2 alignment dataset with randomly sampled harmful examples from the Beavertail dataset under varying poison ratios (p 0%, 10%, 20%). As shown in Figure 2(a), the non-shaded portion of each bar represents the forget- ting rate in each setting (|A:l/N), while the shaded region denotes the CommonF orgor across all poison ratios, as defined in Eq. (2).
Definition and Calculation of Data Vulnerablility. We define data vulnerability as the tendency of specific align- ment examples to be "forgotten" during the harmful fine- tuning process after the model has undergone safety align- ment. To quantify this, we analyze the forgetting behavior of different alignment examples during harmful fine-tuning by examining how their harmful scores (HS) evolve over time. The harmful score, denoted as HS, is binary vari- able indicating whether the i-th example produces harmful output at the learning step t. If the harmful score increases during the fine-tuning process, it signifies that the alignment example has been "forgotten".
We observe that introducing harmful data significantly in- creases forgetting: at p 10% and p 20%, the overall forgetting is approximately twice as high as in the benign setting (p = 0%). However, a large portion of the forgot- ten examples overlaps across poison ratios, resulting in a high CommonForgotRatt alignment examples are consistently vulnerable to forget- ting, regardless of the amount of harmful data introduced during fine-tuning.
This indicates that certain
To measure how often a particular alignment example is forgotten, we compute the total number of times it is com- promised during the fine-tuning process as:
Forgetting in HFT is Transferable Across Tasks. We fur- ther examine forgetting patterns across distinct fine-tuning tasks where the harmful data ratio is fixed at = 10%, but the specific harmful data varies for each task. Even in this case, the forgotten alignment data shows substantial overlap, as depicted in Figure 2(b). These results indicate that the vulnerable patterns identified in alignment data are transferable across tasks.
ForgotNum,
(I(HS) H)
(1)
Here, HS? is the initial harmful score of the i-th example before harmful fine-tuning. This ForgotNum serves as measure of data vulnerability, indicating how susceptible an example is to forgetting during harmful fine-tuning, A higher Forgot Num implies greater vulnerability, suggest- ing that the example is more sensitive to the fine-tuning process and more likely to be compromised during HFT. In this work, we define an example as forgotten as having
Robustness of Different Groups. As shown in Figure 3, we investigate the reasons behind the robustness dispar- ity between vulnerable and invulnerable data by analyzing the aligned model's 's behavior. Through an examination of
3

---

-ulner Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
the loss landscape, we find that vulnerable examples of- ten exhibit greater sensitivity to changes in model weights. This heightened sensitivity helps explain why they are more likely to be forgotten during harmful fine-tuning, as such fine-tuning inevitably induces a shift in model parameters.
The results reveal that a subset of alignment examples ex- hibits significantly higher vulnerability compared to others. These examples, despite being successfully learned during the alignment phase, are highly susceptible being lost during user fine-tuning.
Figure 3: Analysis of robustness behavior. The left panel shows the loss landscape with respect to vulnerable data, while the right panel illustrates the loss landscape for in- vulnerable data. The results indicate that the model is less robust to perturbations in vulnerable data.
Data Grouping by Vulnerability. We aim to partition the and invulner-
alignment dataset into two groups- able examples- based on their susceptibility to forgetting during harmful fine-tuning, This partitioning serves as the foundation for our targeted alignment strategy in subsequent stages (Stage in Figure 1). However, in realistic deploy- ment scenarios, the downstream fine-tuning distribution is typically unavailable. To address this challenge, we lever- age the empirical transferability of vulnerability patterns observed across tasks (see Figure 2) and approximate the forgetting behavior using a proxy fine-tuning dataset.
3.1. Robust Objective
To model the parameter shifts introduced by harmful fine- tuning, we use surrogate objective that proactively captures group-wise robustness to weight perturbations during the aligmment stage. Let θ denote the parameter space of the LLM, and let & θ x (X' x x) - RR,, be a nonnegative loss function measuring the discrepancy between the model's prediction and the true output. The surrogate objective is defined as:
Specifically, we simulate HFT by fine-tuning a pre-aligned model on Alpaca (Taori et al., 2023), augmented with 10% randomly sampled harmful data. During this process, we evaluate the model's 's predictions on the original alignment dataset over iterations and record the number of times each example transitions from a safe to a harmful output. This count is denoted as ForgotNum (see Eq. 1).
fi(0) ((O) A (e,(0 + -))
(4)
robusiness i-th group
= (1 -1- (1-A) + Xe, +()
(5)
We classify alignment examples as vulnerable if ForgotNum > 0, and as invulnerable otherwise. As shown in Figure 1 (left), this grouping is used as prior knowledge in our method. Notably, this procedure is fully data-driven and does not rely on access to the actual downstream fin-ttunig distribution, making it broadly applicable across real-world alignment settings.
Here, ((() denotes the loss for group G3, and represents a worst-case perturbation applied to the model parameters for that group. The robustness term, (, + e;) (0),,
quantifies the group's sensitivity to parameter shifts. This term is group-specific, reflecting the empirical observation that different groups exhibit varying levels of vulnerability to weight perturbations.
Remark HFT induces alignment performance degrada- tion by causing parameter shifts, which lead to the forget- ting of alignment examples. However, our analysis reveals that this forgetting is not uniform across the alignment dataset-certald examples are significantly more vulner- able to such parameter perturbations and thus more likely to be compromised. In the next section, we introduce a method designed to address these issues by explicitly modeling data- level vulnerability during alignment.
The objective function linearly interpolates between the orig- inal loss and the perturbed loss, enabling a smooth transition from standard learning to robust learning. To facilitate this transition, we adopt a curriculum learning strategy (Bengio et al., 2009), gradually increasing À from 0 to during train- ing. This allows the model to first focus finding valid alignment solution and subsequently improve its robustness to potential parameter shifts.
3.2. Alignment Training via GDRO
3. VAA: Vulnerability-Aware Alignment
To address the wneven uneven forgetting problem, we need to train an LLM to learn equally well across different groups of ex- amples. We improve upon the Empirical Risk Minimization (ERM) algorithm to encourage balanced learning between groups by incorporating our prior knowledge.
Building on our analysis, we propose an alignment frame- work that explicitly incorporates two key priors into the training process: parameter shifts and uneven forgetting across the alignment dataset.

---

Yulnerabiliy-Awaae Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
The standard ERM-based alignment method is find param- eters A that minimize the empirical loss over the empirical distribution P:
T4 > 0 is the step size, and Dola q((-1) is the Bregman divergence induced by the mirror map (1)))
We choose the negative entropy (g)) = > 9: log q; as the mirror map, which induces the Kullback-Leible (KL) divergence (x lI (((-1)
FRRM= == arg min
T ((e.))) (xr v))] BEB
(6)
Σ;= 9: log(g, /9, / "-)). Substituting the KL divergence into Eq. (9), we obtain:
Here, ERM treats all alignment examples equally and opti- mizes the average loss, which leads to "uneven forgetting." Since the invulnerable group contains significantly more samples than the vulnerable group, this imbalance leads to "gradient starvation" phenomenon (Pezeshki et al., 2021) under ERM, where gradients from the smaller group are dominated by those from the larger group. Consequently, the model underperforms on vulnerable groups, reinforcing their susceptibility to forgetting.
m
m
)
q mig 9EAm
Σ
2f + Σ a: log(as/9
(10)
To solve the problem, we introduce a Lagrangian with multiplier À to enforce the probability simplex constraint
Ei= 4 = 1:
m
m Σ
L(g. A)
MhJ + log Σ 4:
9: 1--1) qi
+
a Ea-1)
(II)
To address this issue, we propose alignment training un- der the group distributionally robust optimization (GDRO) framework, which explicitly optimizes the performance of the current underperforming group. Formally:
Taking the derivative of L(q, A) with respect to q, and setting it to zero yields:
2 exp (1at(((((t-1))
1)
(12)
ODRO arg min { sup ((a,))) (f((O) (z, w)m}}
(7)
Z
lEB G.EQ
where Z = -E 03 (1-1) exp (
where [e(0; (xr, y)) denotes the objective function for group G2 and @ represents the ambiguity set, which is defined as a subset of the convex combinations of different groups -GI,, J:
This update aligns with the EXP3 algorithm (Auer et al., 2002) in the adversarial bandit problem, where each group corresponds to an arm and the observed reward for arm is adjusted by its sampling probability q(--1) ensure unbiasedness:
Q =
Σ
q: Gi g E Am-1),
(8)
C0 = f:(0
(13)
where q is vector belonging to the (m 1)- dimensiooaal probability simplex Am-
4:
where l[i = i] is the indicator function that equals 1 if arm is selected at time t, and otherwise.
The optimal solution to the GDRO formulation (Eq. 7) achieves an equal objective across all groups, which effec- tively mitigates the problem of uneven forgetting in harmful fine-tuning.
As shown in the stage 2 of the Figure 1, this approach transforms the problem into a two-player game between the LLM (parameterized by 9) and the adversarial sampler (parameterized by g). At each iteration, the sampler selects challenging group G; from Q according to probability dis- tribution q. By using a first-order Taylor expansion (Foret et al., 2021), e, can be approximated as a- (O) where a is the perturbation magnitude. We then calculate the reward r: for selecting group G3 and update the sam- pling probability q via the EXP3 algorithm to challenge the LLM. Meanwhile, the LLM optimizes its performance under this challenge by performing gradient descent on ob- jective function f.
3.3. Learning Algorithm
Following Sagawa et al. (2020), we employ an online al- gorithm to solve formulation However, rather than for- mulating it as reweighting problem, we consider it as resampling problem. To do this, we learn an adversarial sampler & E Am-1 to focus on the worst-case group within the ambiguity set Q
To derive the update rule for q. we employ mirror ascent (Nemirovski et al., 2009) on the probability simplex A m-1 Considering that we aim maximize the objective function f(0) with respect to q, we formulate the update as:
The overall algorithm is detailed in Algorithm 1.
4. Experiments
4 = arg max m-Am-1
(nnla, (00) Dv (
4.1. Experimental Setup
where f(() = [f(tt (o(t is the vector of objective function values for each group at iteration i,
To perform model alignment, we utilize the safe samples from the alignment datasets from Rosati et al.
Datasets.
5

---

Yulnerabiliy-Awwar Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Algorithm 1: Learning Algorithm of VAA
Baselines. We compare our method against four baselines: standard supervised fine-tuning (SFT), Vaceine (Huang et al., 2024d), RepNoise (Rosati et al., 2024a), and Booster (Huang et al., 2024b). SFT follows the conventional two- stage training paradigm with standard alignment and fine- tuning, while Vaccine, RepNoise, and Booster are recent alignment-stage defense methods.
Input: Step sizes Tg. no: perturbation intensity
Initialize ((0) E and y(0) = 1/m
for f 1 to T do
// select a group and sample examples i~ g(t-1)
(x, ~ G
// Compute group-wise perturbation
Models. We evaluate our approach on Llama2 7B (Touvron et al., 2023) and Qwen2.5 7B (Yang al., 2024).
((.(0((-1)
6 1 (o y
I/ Compute reward
Training Details. We perform full-parameter training for both the alignment and harmful fine-tuning stages. Full training during HFT is used to simulate worst-case align- ment degradation, as updating all parameters may amplify harmful behaviors. For alignment, we use the AdamW optimizer (Loshchilov et al., 2017) with learning rate of 1 x 10-4 and weight decay of 0.1, while for HFT we adopt lower learning rate of 3 x 10-5 to reflect the more sensitive nature of this stage. Both stages are trained for 5 epochs using a batch size of 8. HFT is conducted on a diverse mix of datasets, including SST-2, AG News, GSM8K, and AlpacaFval, covering classification, reason- ing, and instruction-folloviii tasks. All experiments are conducted 4 NVIDIA A100 GPUs with 80GB memory.
r + 5((0(t-1))//
((-1) ((-1)
I/ Update sampling distribution
(t-1) exp(n.r (1)
d: + 9i
q(0) + (1)
m /1//E/1 Oj
I/ Update model parameters
0(0) + 0((-1) 1/0 V (10L-1))
end
(2024b), which are enriched versions of BeaverTails (Ji et al., 2023). We sample 2,000 instances from alignment dataset for training, ensuring that the harmful dataset instances are distinct from those used in the fine-tuning stage.
To perform alignment data grouping, we utilize Al- paca (Taori et al., 2023) as our proxy dataset to simulate harmful fine-tuning mixed with 10% harmful data. The harmful data used for grouping are distinct from those used in the user's fine-tuning stage.
4.2. Main Experiments
Generalization to fine-tuning datasets. We evaluate VAA across four different fine-tuning datasets. Table shows that VAA consistently outperforms the baselines on HS metries across all datasets, achieving harmful score reductions of 12.9%, 12.0%. 10.6%, and 3.4% compared to SFT on the four datasets. It also achieves higher fine-tuning accuracies, demonstrating its effectiveness and superiority.
For tuningunin,, we employ four datasets: SST-2 (Socher al., 2013), AG News (Zhang al., 2015), GSM8K (Cobbe et al., 2021), and AlpacaEval (Li et al., 2023). To simulate harmful attacks during fine-tuning, we create mixed datasets by combining p% of unsafe data from BeaverTails with (100- - 7)% of benign fine-tuning data, resulting in a total of n samples per dataset. Unless specified otherwise, we set p = 10 and n = 1, 000 (except for AlpacaEval, where 70700).
Under the full train setting during HFT, baseline methods perform poorly on the more complex fine-tuning datasets like GSM8k and AlpacaEval. RepNoise and Booster show no significant reduction in harmful scores on GSM8k, and even increases the harmful score on AlpacaEval. While these methods aim to prevent HFT from effectively learning harmful data, they struggle when harmful data is mixed with more complex task data. They lack the robustness to handle the complex structures and patterns present in such tasks, which can result in failure reduce harmful scores. In contrast, by focusing the inherent vulnerability of the data and reinforcing the more fragile parts during alignment, our method strengthens the overall robustness of the alignment process. As a result, our method effectively reduces harmful scores on both datasets, demonstrating its generalizability in handling complex fine-tuning tasks.
Evaluation Metrics. Following Huang et al. (2024d;b), we evaluate our method using two metrics: Fine-tuning Accuracy (FA) and Harmful Score (HS). FA measures the model's performance on the test set of each fine-tuning task. HS quantifies the model's resilience to harmful instructions by calculating the proportion of outputs classified as harmful by the moderation model from Ji et al. (2023) when tested on unseen malicious instructions.
To compute HS, we sample 1,000 instructions from the BeaverTails test set. For FA, the test set sizes are as fol- lows: 872 samples for SST-2, 1,000 for AG News, 1,000 for GSM8K, and 122 for AlpacaEval. Both metrics are evaluated on the final fine-tuned models.
Robustness to harmful ratio. We evaluate the robustness of VAA under different harmful ratios. Table 2 shows that,
6

---

Auleera Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Table Performance for different fine-tuning datasets. The best and second-bess performances are highlighted in bold and underlined, respectively.
SST2
AGNEWS
GSM8K
AlpacaEval
Average
Methods
HS 4
FA 1
HS
FA 7
HS 4
FA
HS
FA 1
HS
FA 1
SFT
32.87
91.00
33.07
87.40
41.63
6.80
30.48
39.73
34.51
56.23
RepNoise
27.89
90.40
27.29
84.00
41.83
6.60
34.66
36.21
32.92
54.30
Vaccine
27.69
89.40
30.28
85.60
34.66
6.20
32.47
38.62
31.28
54.96
Booster
25.90
91.80
31.87
87.00
41.04
6.40
40.24
39.41
34.76
56.15
VAA
20.00
91.00
21.12
87.40
31.08
8.60
27.09
40.06
24.82
56.77
Table 2: Performance analysis for different harmful ratio.
Harmful Score 4
Finetune Accuracy 1
Methods
p=0%
p=10%
p=20%
Average
p=0%
p=10%
p=20%
Average
SFT
23.11
32.87
38.84
31.61
91.80
91.00
90.00
90.93
RepNoise
22.91
27.89
35.26
28.69
90.20
90.40
90.60
90.40
Vaceine
21.31
27.69
36.65
28.55
90.40
89.40
90.00
89.93
Booster
14.54
25.90
30.28
23.57
90.20
91.80
90.40
90.80
VAA
12.35
20.00
25.30
19.22
90.60
91.00
91.20
90.93
Table 3: Performance analysis for different harmful fine-tuning epochs.
Harmful Score
Finetune Accuracy 1
Methods
epoch=1
epoch=3
epoch=5
Average
epoch=1
epoch=3
epoch=5
Average
SFT
27.69
31.67
32.87
30.74
90.00
91.00
91.00
90.67
RepNoise
27.89
30.88
27.89
28.89
90.20
91.20
90.40
90.60
Vaceine
25.30
29.08
27.69
27.36
84.00
88.80
89.40
87.40
Booster
29.08
24.10
25.90
26.36
89.20
88.80
91.80
89.93
VAA 14.60
19.20
20.00
17.93
90.00
91.40
91.00
90.80
compared to SFT, VAA achieves an average of 12.4% lower harmful scores while maintaining fine-tuning accuracy on downstream tasks. VAA consistently outperforms the base- lines in reducing harmful scores across different harmful ratios. Specifically, when the harmful ratio is 0%, VAA significantly reduces harmful scores, demonstrating the gen- eralizability of VAA. It not only defends against HFT at- tacks but also mitigates forgetting caused by benign data fine-tuning. However, we observe that the harmful score increases as the harmful data ratio rises, which is common weakness of alignment-stapg solutions.
the best robustness, maintaining the lowest harmful score and high fine-tuning accuracy across all epoch settings.
Generalization different LLMs. Table demonstrates that our proposed method generalizes effectively the state- of-the-art Qwen2.5-7B model. Importantly, the group signments used for VAA training Qwen2.5 were derived from grouping estimations performed on I.LaMA2, with- out re-clustering on the Qwen2.5 model itself. Despite this cross-model transfer, VAA consistently achieves the best performance, maintaining the lowest harmful scores and high fine-tuning accuracies across all epochs.
Robustness harmful fine-tuning epochs. We show in Table 3 how HFT epochs affect model safety. The results show that the number of HFT epochs increases, harm- ful scores rise while fine-tuning accuracy stabilizes. This indicates that more extensive fine-tuning on harmful data increases the risk of harmful outputs. VAA demonstrates
These results provide strong empirical support for hy- pothesis: subset of alignment examples exhibits consis- tent vulnerability across different model architectures. Our method successfully identifies these transferable vulnerable samples and leverages them to enhance robustness, even

---

Aulnerabilly-AAware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Table 4: Performance analysis for different harmful fine-tuning epochs on Qwen2.5-7B.
Harmful Score +
Finetune Accuracy t
Methods
Epl
Ep2
Ep3
Ep4
Ep5
Epl
Ep2
Ep3
Ep4
Ep5
SFT
26.89
31.47
31.08
33.27
33.07
84.80
76.80
86.80
87.20
86.40
RepNoise
22.31
25.10
26.49
30.68
30.88
83.00
81.60
88.80
87.20
88.00
Vaccine
29.88
29.28
28.88
30.48
29.48
82.60
84.20
83.20
85.40
85.60
Booster
19.92
21.91
25.10
26.29
30.28
85.20
84.80
87.40
87.60
88.00
VAA
17.73
18.33
20.12
21.91
22.11
86.20
86.40
85.40
87.60
88.60
when applied to different LLM family.
Table 5: Impact of examples grouping.
4.3. Discussion
Strategy
HS 4
FA f
We conduct experiments to analyze the impact of example grouping and sampling strategies model performance.
VAA
20.00
91.00
w/o group
26.42
90.08
w noisy group
21.08
91.20
Example Grouping, We perform two experiments to evalu- ate the effectiveness of our example grouping strategy. First, we compare our method with variant that does not perform example grouping. Second, we evaluate the robustness of VAA to noisy groups by randomly swapping 10% of exam- ples between the "vulnerable" and "invulnerable" groups. As shown in Table 5, we observe that removing grouping significantly degrades performance, highlighting the impor- tance incorporating vulnerability priors into our method. Furthermore, VAA demonstrates robustness to group noise, as the performance degradation under the noisy grouping setting moderate rather than severe. We attribute this to the inherent robustness of the GDRO framework when handling imperfect group assignments (Sagawa et al., 2020).
Table 6: Impact of sampling strategies.
Strategy
HS
FA ?
VAA
20.00
91.00
Vuln. group only
29.26
90.15
Invuln. group only
33.98
91.20
Impp sampling
28.64
90.35
ods share the same asymptotic complexity of O(P),, we report their relative training cost multiplicative factor over standard SFT, Vaccine and Booster require approxi- mately 2 BP and 3 x BP, respectively, due to repeated perturbation steps. In contrast, VAA requires only 1.5 x BP on average.
Sampling Strategy. To analyze the impact of sampling strategies, we compare our method with three heuristic sam- pling baselines. The first strategy samples exclusively from the "vulnerable" group, while the second samples exclu- sively from the "invulnerable" group. The third baseline employs commonly used importance sampling strategy (Buda et al., 2018) in imbalance learning, which sets the sampling probability the inverse training frequency of each group. As shown in Table 6, we find that sampling only from the "vulnerable" group leads to better performance than sampling only from the "invulnerable" group, suggest- ing that the "vulnerable" group contains more informative examples. However, focusing solely
This efficiency is achieved through a curriculum learning strategy that gradually increases the perturbation probability from 0% to 100%, avoiding full perturbation in the early training stages and redueing unnecessary computation. As result, VAA saves approximately 0.5 x BP compared to the fastest robust baseline (Vaccine), while delivering significantly better safety performance.
In practice, full training on a 7B-parameter model completes in under one hour, making the modest additional cost of VAA acceptable given its substantial gains robustness.
group results in information loss and suboptimal performance. Although importance sampling outperforms single-group group sampling, it still falls short compared to our dynamic sampling method, which is adaptively optimized during training.
5. Related Work
Computational Overhead. We analyze the computa- tional efficiency of VAA relative to existing baselines by comparing the number of backpropagation (BP) steps, which constitute the dominant training cost. While all meth-
Harmful Fine-tuning. Harmful Fine-tuning (HFT) poses a significant threat to the safety of Large Language Mod- els (LLMss. To mitigate this, various methods have been proposed, categorized into three types: (1) alignment-stage
8

---

Yulnerabiliy-Awaae Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
methods (Huang et al., 2024d; Rosati et al., 2024a;b), (2) fine-tuning-stasee methods (Mukhoti et al., 2023; Huang et al., 2024c; Lyu et al., 2024), and (3) post-fmeettuinn- stage methods (Hsu et al., 2024; Yi et al., 2024; Huang et al., 2024a). Among these, alignment-stage methods are particularly advantageous as they enhance robustness by optimizing the model's safety alignment, strengthening the model's safety foundation and reducing subsequent security risks.
encouraging LLMs to learn equally well on both vulnerable and non-vulnerable examples.
6. Conclusion
This paper investigates vulnerability patterns in alignment data and proposes Vulnerability-Awar Alignment, a novel defense method addressing previously overlooked data- dependent vulnerabilities. Our findings reveal that certain aligmment data subsets are consistently more susceptible to compromise across different fine-tuning tasks, stemming from imbalanced learning during standard alignment. By implementing group distributionally robust optimization that promotes balanced learning across data subsets, our method effectively reduces performance disparities while maintaining model utility. Experiments on four fine-tuning tasks demonstrate the effectiveness of our approach in mit- igating harmful behaviors while preserving downstream performance.
There are two directions to mitigate HFT risks during align- ment. One direction is learning robust representations from alignment data, such as Vaccine (Huang et al., 2024d), which enhances the model's 's resistance to HFT by adding perturbations to the hidden embeddings, thereby reducing embedding drift. The other direction involves making harm- ful data unlearnable to minimize its impact. For example, RepNoise (Rosati et al., 2024a) optimizes the model's s rep- resentation using harmful data to improve robustness, while Booster (Huang et al., 2024b) employs a regularizer to re- duce the harmful loss reduction rate after harmful perturba- tion. The method introduced in this paper belongs to the alignment-stage category. We propose a novel approach that investigates the vulnerability patterns of alignment data-an an aspect overlooked by previous work- and incorporates this prior knowledge into the algorithm design.
Our work highlights the potential of mitigating harmful fine- tuning from a data-centric perspective. By focusing on the inherent vulnerability of data, we have shown that address- ing data-dependent vulnerabilities can significantly improve the safety and reliability of LLMs. Notably, VAA is orthog- onal to existing alignment-stage stage methods, and combining it with such techniques may yield even more robust defenses against harmful fine-tuning. This work lays the groundwork for more comprehensive and resilient solutions to alignment breakdowns during fine-tuning.
Analysis of the Causes of Alignment Breakdown. While existing methods have been proposed to mitigate the HFT issue, their performance is still far from satisfactory. Some works have attempted to analyze the root cause of alignment breakdown. For example, Vaccine (Huang et al., 2024d) uncovers that the alignment breakdown is caused by the drift of hidden embeddings during fine-tuning, leading to the forgetting of alignment knowledge. Booster (Huang et al., 2024b), on the other hand, analyzes loss changes in models during fine-tuning and points out that HFT can reduce the loss of harmful data, thereby "activating' harmful knowledge. Additionally, (Peng et al., 2024) introduces the concept of 'safety basin,' arguing that HFT drags the model's weights out of this basin. In this work, we are the first to approach the problem from a data perspective and reveal that the forgetting behavior in HFT is highly data-dependent.
Limitations The current data partitioning strategy is rel- atively simple and relies on a pseudo fine-tuning process. Future research could explore more nuanced ways of cate- gorizing data, such as employing a continuous spectrum of vulnerability- e.g.. based on uncertainty estimation (Gaw- likowski et al., 2022) or factuality metries (Lin et al., 2022: Chen et al., 2023) without the need for additional pseudo fine-tuning stages. Moreover, while VAA is effective in reducing alignment breakdowns during customized fine- tuning, it does not prevent them entirely. Therefore, in- tegration with complementary techniques such as Al wa- termarking (Kirchenbauer et al., 2023; Chen et al., 2024a) may be necessary to achieve more reliable protection and traceability for open-source LLMss
Distributionally Robust Optimization. Distributionally robust optimization (DRO) optimizes the objective function over ambiguity sets, often defined as balls centered the empirical distribution (Ben-Tal et al., 2013; Lam & Zhou, 2015; Duchi et al., 2016; Miyato et al., 2018). Prior applica- tions of DRO have addressed distributional shifts, including covariate shift (Oren et al., 2019; Chen et al., 2024b; 2025), label shift (Hu et al., 2018), and group shift (Sagawa et al., 2020). To the best of our knowledge, we are the first to apply DRO to defend against the harmful fine-tuning problem by
Impact Statement
This paper presents work intended to advance the field of Machine Learning, particularly in the area of model align- ment. We believe our findings may positively contribute to the safety and robustness of language models. We do not anticipate any direct societal risks, but we recognize the importance of continued dialogue around the broader implications of alignment techniques.

---

Yulnerabilly-AAware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Acknowledgments
2024b. URL httpst:::////// id=1wW7d8IMMs.
/ottapps////peeevieee.... nete ftr.rrrremweeee/ommmmmm forum?
This work was partially supported by Hong Kong RGC GRF No. 14206324, CUHK direct grant No. 4055209, and CUHK Knowledge Transfer Project Fund No. KPF23GWP20.
Chen, L., Shen, L.., Deng, Y., Zhao, X., Liang, B., and Wong, K.-F. PEARL: Towards permutation-resilienn LLMs. In The Thirleenth International Conference on Learning Representations, 2025. URL https: htttps://opeereiieee net/
References
Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. The nonstochastic multiarmed bandit problem. SIAM
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv: 11t.:22100.1466.. 2021.
Journal on Computing, 32(1):48-77, 2002. doi: 10.1137/
30097539701398335. URL https: http::/ddoooooo///e// 1137/8009753370139333
Duchi, J., Glynn, P., and Namkoong, H. Statistics of ro- bust optimization: A generalized empirical likelihood approach. arXiv, 2016.
Ben-Tal, A., den Hertog, D., Waegenaere, A. D., Melenberg, B., and Rennen, G. Robust solutions of optimization prob- lems affected by uncertain probabilities. Management Science, 59:341-357, 2013.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https: openrevie.. idd 611mmps11M.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In International Conference on Machine Learning (ICML), pp. 41-48,2200 2009.
Buda, M., Maki, A., and Mazurowski, M. A. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-225 Octo- ber 2018. ISSN 0893-6080, doi: 10. 10.10066).neunet2201.. 07.011. URL http:///// /dx. doi org.10.106..111 neunet. 2018.01112 07.00...0 011.
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., Shahzad, M., Yang, W., Bamler, R., and Zhu, X. X. A survey of uncertainty in deep neural networks, 2022. URL https: / attpa.xivvv...//.../.aaa. orq/ abs / 2217.11111.. 03342.
Hsu, C.-Y., Tsai, Y.-L., Lin, C.-H., Chen, P.-Y,, Yu, C.-
Chen, L., Deng, Y., Bian, Y., Qin, Z., Wu, B., Chua, T.-S., and Wong, K.-F. Beyond factuality: A comprehensive evaluation of large language models as knowledge gen- erators. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp. 6325-634 1, Singapore. December 2023. Association for Computa- tional Linguistics. doi: 10. 1211666//112223..mlmmmmm 390. URL ht ht////acaaattoooooeo org.//r///// emnlp-main. 3
M., and Huang, C.-Y. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models, 2024.
Hu, W., Niu, G., Sato. I., and Sugiyama, M. Does distri- butionally robust supervised learning give robust classi- fiers? In International Conference on Machine Learning (ICML), 2018.
Huang, T., Bhattacharya, G., Joshi, P., Kimball, J., and Liu, I.. Antidote: Post-finettuinng safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408.. 09600, 2024a.
Chen, L., Bian, Y., Deng, Y., Cai, D., Li, S., Zhao, P., and Wong, K.-F. WatME: Towards lossless watermark- ing through lexical redundancy. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9166-9180, 9180, Bangkok, Thailand, August 2024a. Association for Com- putational Linguistics. doi: 10. 10.18653/v122044444-oogg
Huang, T., Hu, S., Ilhan, F,, Tekin, S. F., and Liu, L. Booster: Tackling harmful fine-tuning for large language mod- els via attenuating harmful perturbation, 2024b. URL https: / arxii.iiivv..../ org/ atapg///////////// 220...209... 01586.................
496. URL ht https:// acl long. 496/.
Huang, T., Hu. S., Ilhan, F., Tekin, S. F., and Liu, L. Lazy safety alignment for large language models against harm- ful fine-nuning. arXiv preprint arXiv:2405. 18641, 2024c.
Chen, L., Bian, Y., Shen, L., and Wong, K.FF Simple permutations can fool LLaMA: Permutation attack and defense for large language models. In ICLR 2024 Work- shop on Secure and Trustworthy Large Language Models,
Huang, T., Hu, S., and Liu, L. Vaccine: Perturbation- aware alignment for large language model. arXiv preprint arXiv:2402.01 109, 2024d.
10

---

Yulnerabiliy-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Ji, Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards improved safety alignment of Ilm via human-preferencce dataset. arXiv preprint arXiv:2307.04657 2023.
Y., Liang, P., and Vaughan, J. W. (eds.), Advances in New- ral Information Processing Systems, 2021. URL https: / //oppnrreiie.. net/ forum?
Qi, X., Zeng. Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024. URL net / forum? htw.ummmdd--TEEyEEECCCC
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. International Conference on Machine Learning, 2023.
Lam, H. and Zhou, E. Quantifying input uncertainty in stochastic optimization. In 2015 Winter Simulation Con- ference, 2015.
Rosati, D., Wehner, J., Williams, K., Bartoszcze, L., Atanasov, D., Gonzales, R., Majumdar, S., Maple, C., Sajjad, H., and Rudzicz, F. Representation noising ef- fectively prevents harmful fine-tuning on Ilms. arXiv preprint arXiv:2405.14577 2024a.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instraction-follooinng https::///tthbb./t/
models. hitps:/ / github. www/tatstttttttaaaaaeeeee
alpaca_eval, 2023.
Rosati, D., Wehner, J., Williams, K., Bartoszcze, L., Batzner, J., Sajjad, H., and Rudzicz, Immunization against harm- ful fine-tuning attacks. arXiv preprint arXiv:2402. /6382, 2024b.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
Loshchilov, I., Hutter, F., et al. Fixing weight decay regu- larization in adam. arXiv preprint arXiv:1711.05100, 5, 2017.
Sagawa, S., Koh, W., Hashimoto, T. B., and Liang, Distributionally robust neural networks. In In- ternational Conference on Learning Representations, 2020. URL hww.h:ttt/s//..... i id=ryxGuJrFVS.
Lyu, K., Zhao, H., Gu, X., Yu, D., Goyal, A., and Arora, S. Keeping iims aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402. 18540, 2024.
/oppn/evie/eeeeeeeeeemeemee htt.ht:///pppereiieee net/ net//orum?
Socher, R., Perelygin, A., Wu, J., Chuang. J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Miyato, T., Macda, S., Ishii, S., and Koyama, M. Virtual adversarial training: a regularization method for super- vised and semi-supervised learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2018.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-folllwiing llama model. https: /tts:////tttt//////// wwwtttuu.com/tttte www.tttsttttaaaaaaaaaaea stanford_alpaca 2023.
Mukhoti, J., Gal, Y.. Torr. P. H., and Dokania, K. Fine-tuning can cripple your foundation model; pre- serving features may be the solution. arXiv preprint arXiv:2308. 13320, 2023.
Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Ro- bust stochastic approximation approach to stochastic pro- gramming. SIAM J. on Optimization, 19(4):1574-1609,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.
January 2009. ISSN 1052-6234, doi: 10. 10/11377077704277. URL https: / doi. wor...oo....ooooooooooooooooo..o org. /00. 1137 0777044777
2023.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Oren, Y., Sagawa, S., Hashimoto, T., and Liang, P. Distribu- tionally robust language modeling. In Empirical Methods in Natural Language Processing (EMNLP), 2019. 2019..
Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02944 2023.
Peng, S., Chen, P.- Hull, M., and Chau, D. H. Navigat- ing the safety landscape: Measuring risks in finetuning large language models. arXiv preprint arXiv:2405.17374 2024.
Yi, X., Zheng. S., Wang, L., Wang, X., and He, L. A safety realignment framework via subspace-oriente model fusion for large language models. arXiv preprint arXiv:2405. 09055, 2024.
Pezeshki, M., Kaba, S.-O., Bengio, Y., Courville, A., Precup, D., and Lajoie, G. Gradient starvation: A learning pro- clivity in neural networks. In Beygelzimer, A., Dauphin,

---

-uleerabilly-Awware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning
Zhan, Q. Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing rihf protections in gpt-4 via fine-tuning, arXiv preprint arXiv:2311.0555., 2023.
Zhang, X., Zhao, J., and LeCun, Y. Character-level convolu- tional networks for text classification. Advances in neural information processing systems, 28, 2015.
12


Reviews:
[{"review_id": "FlpTaOk1tj", "reviewer": "Reviewer_uJpD", "summary": "The paper observes that certain subsets of alignment data are more likely to be forgotten during harmful fine-tuning (HFT) of large language models (LLMs). To mitigate this issue, the paper proposes a new alignment-stage method called Vulnerability-Aware Alignment (VAA). VAA first divides data into “vulnerable” and “invulnerable” groups and then applies a group DRO-based algorithm. Experiments on four fine-tuning datasets show that VAA could reduces harmful scores.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the professional and constructive feedback! We are delighted that you recognized our adaptation of GDRO to address a new and important problem, and appreciated the various interesting observations in our work. We're glad you found our experimental setup largely acceptable and our results demonstrating VAA's improvement over existing methods particularly encouraging.\n\nWe address your suggestions below with additional analysis and results.\n\n## Response to W1: Deeper Analysis of the Method\n\nWe address the suggestions in three parts: design rationale, variance-based explanation and training stability (curriculum learning).\n\n### 1. Design Intuitions\n\nVAA integrates two components—**robust loss** and **group DRO (GDRO)**—each addressing a distinct challenge posed by HFT:\n\n1. **Robust Loss (Parameter Perturbation)**: HFT degrades performance via **parameter shifts** from θ to θ′. To model this, we use a surrogate objective, min_θ L(θ') = min_θ L(θ + ε), to proactively simulates these shifts during alignment, helping LLMs to defense against HFT.\n\n2. **GDRO (Adversarial Sampler)**: Our analysis (Claim 1) shows ~30% of alignment data is vulnerable to forgetting under HFT, while ~70% is relatively robust. This _**data imbalance**_ leads to _\"gradient starvation [1]\"_ under ERM, where gradients from the smaller group are dominated by those from the larger group, resulting in _**\"imbalanced vulnerability\"**_. GDRO mitigates this by upweighting underperforming groups, encourage balanced learning between groups.\n\nTogether, these modules incorporate two key priors into the alignment process: **parameter sensitivity** and **imbalanced vulnerability across the dataset**.\n  \n### 2. Variance-Based Explanation\n\nTo provide deeper insight, we present a variance-based perspective.\n\nFor any distribution $P \\in \\mathcal{Q}$ and parameter perturbation $\\delta$ induced by HFT, the variance decomposes according to the Law of Total Variance:\n\nVar_P[ℓ(θ* + δ)] = E[Var_G_i[ℓ(θ* + δ)]] + Var[E_G_i[ℓ(θ* + δ)]]\n\n  The first term captures within-group variance; the second captures between-group variance.\n\n- The **robust loss** reduces **within-group variance** by simulating perturbations.\n\n- The **GDRO** reduces **between-group variance** by improving the performance of the vulnerable group, leading to more balanced robustness.\n\n### 3. Training Stability and Curriculum Learning\n\nWe analyze training stability in both modules:\n\n**LLM Module:** Training stability of LLM is primarily affected by parameter perturbation. In **Table 6** in manu, we analyze _**the effect of perturbation strength on performance**_. Now we further analyze _**its impact on training instability**_.\n\n*With small perturbations, training is stable but defense is limited. Larger perturbations improve defense but destabilize training.* **Curriculum learning (CL)** addresses this trade-off. As shown in Table 1, CL enables stable training under larger perturbations, providing better defense.\n\nTable 1. Training Stability Under Large Perturbation\n|  | wo CL | w CL | \n|----------------|--------------|--------------|\n | Train Loss | 5.11 → 4.74 | 1.38 → 0.48 |\n  | Grad Norm | 71.0 → 1.55 | 10.06 → 1.48 |\n\n**Adversarial Sampler:** The sampler's training stability depends primarily on learning rate. As analyzed in **Table 7**, we selected an appropriate rate that enables smooth updates. No significant stability issues were observed in practice.\n\n### Clarification of Terminology\nWe'll consistently use \"vulnerable group\" throughout.\n\n## Response to W2: Experiments\n\n### 1. Reporting Variance\n\nIn Tab 2, we conducted additional runs of SFT and VAA under three random seeds. VAA consistently outperforms SFT with low standard deviations, indicating significance.\n\nTable 2. Performance (mean ± std)\n\n| Model | Metric | Epoch 1 | Epoch 3 | Epoch 5 |\n| :--- | :--- | :--- | :--- | :--- |\n| SFT | HS | $26.82 \\pm 0.62$ | $31.08 \\pm 0.43$ | $32.07 \\pm 1.12$ |\n|  | FA | $89.07 \\pm 0.82$ | $90.20 \\pm 0.59$ | $90.80 \\pm 0.28$ |\n| VAA | HS | $14.60 \\pm 0.49$ | $19.20 \\pm 0.33$ | $20.60 \\pm 1.30$ |\n|  | FA | $89.20 \\pm 0.59$ | $90.40 \\pm 0.71$ | $90.73 \\pm 0.25$ |\n\n ###  2. Analysis of Baseline Failures\n\n*RepNoise & Booster:* Both implement explicit unlearning for harmful inputs (RepNoise corrupts internal representations, Booster prevents gradient descent), creating objective conflicts that impair generation abilities on tasks, compromising both harm mitigation and task performance.\n\n*Vaccine:* While applying robust learning, Vaccine overlooks data imbalance, leading to \"gradient starvation\" where vulnerable groups receive insufficient updates, limiting its effectiveness.\n\n***\nThank you for your constructive feedback and detailed suggestions, which has significantly strengthened our work. We hope our responses address your concerns and welcome any further feedback.\n\n----------\n\nReference\n\n[1] Gradient Starvation: A Learning Proclivity in Neural Networks. NIPS 2021", "claims_and_evidence": "The paper makes two key claims: 1) the vulnerability of specific alignment data subsets to forgetting during harmful fine-tuning (HFT), and 2) the effectiveness of the proposed VAA method in mitigating this vulnerability. \n- A motivating example in Figure 2 supports the first claim. \n- While experimental results demonstrate VAA's improvement over existing methods, the paper would benefit from a more rigorous analysis of VAA (e.g., theoretical support and a more clear design rationale of VAA).", "methods_and_evaluation_criteria": "Continuing from the above section, a deeper analysis of the VAA method would be crucial for a complete understanding of its capabilities. Key areas for improvement include: \n- Providing theoretical guarantees (or at least intuitions) into VAA's ability to reliably mitigate harmfulness in LLM fine-tuning\n- Justifying the choice of curriculum learning as the optimal strategy for implementing the target objective \n- Addressing and analyzing any training stability challenges associated with the two-player game\n- Clarifying the connection and potential overlap between the \"hard\" groups mentioned in the introduction and the \"vulnerable\" groups discussed throughout the paper", "theoretical_claims": "The paper applies group DRO (a method with various theoretical backgrounds) to a new problem, but does not offer novel theoretical analysis.", "experimental_designs_or_analyses": "While the experimental setup is largely acceptable, certain aspects raise questions. \n- The lack of reported variance (e.g., standard deviation) makes it difficult to fully assess the significance of VAA's results. \n- (Relatively minor issue) The observed worse harmful scores for some baselines (e.g., RepNoise in GSM8K and AlpacaEval; Vaccine and Booster in AlpacaEval) may need a better explanation. Although the paper suggests that difficult datasets may contribute to this phenomenon, a more detailed analysis of these baseline failures would be valuable.", "relation_to_broader_scientific_literature": "The paper focuses on improving the vulnerability of LLMs, which may provide some insights for LLM applications in scientific literature.", "essential_references_not_discussed": "Overall well-discussed.", "other_strengths_and_weaknesses": "The paper aims to solve an important problem in LLMs and includes various interesting observations. However, the paper could be improved by enhancing the analysis/rationale on VAA and experiments. Details are discussed in the above sections.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Questions are included in the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nb4HHs3Re1", "reviewer": "Reviewer_g5ub", "summary": "This paper studies the Harmful Fine-Tuning (HFT) problem from data perspective. The authors find that there are specific subsets (vulnerable samples) in the aligned data that are more likely to be forgotten in HFT. To address this problem, this paper proposes a new method called Vulnerability-Aware Alignment (VAA), which uses the Group DRO framework to dynamically adjust the training strategy and force the model to learn vulnerable and non-vulnerable samples in a balanced manner through adversarial sampling and group-dependent perturbations. Experiments show that VAA reduces the harmfulness score in four fine-tuning tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your professional and encouraging feedback. We are pleased that you recognize our work as the first to reveal uneven vulnerability under HFT. We also appreciate your recognition of VAA as a novel method that significantly reduces harmful scores while preserving performance, and your acknowledgment that our claims are well supported by convincing evidence.\n\nBelow, we address your points with supporting analyses and new results.\n\n## Response to W1: Theoretical Support\n\nTo explain the principles of VAA, we present a variance-based perspective.\n\n**Variance Decomposition under HFT.**\nFor any distribution $P \\in \\mathcal{Q}$ and parameter perturbation $\\delta$ induced by HFT, the variance decomposes according to the Law of Total Variance:\n\nVar_P[ℓ(θ* + δ)] = E[Var_G_i[ℓ(θ* + δ)]] + Var[E_G_i[ℓ(θ* + δ)]]\n\nThe first term captures **within-group variance**, and the second term captures **between-group variance**.\n\n**Discussion**: \n-   The **robust loss** (group-wise perturbation module), reduces within-group variance by simulating perturbations during alignment, thereby improving robustness at the group level.\n-   The **GDRO** (adv sampler module) reduces between-group variance by improving the performance of the most vulnerable group, leading to more balanced robustness across subpopulations.\n\nTogether, these components enable VAA to address both _parameter sensitivity within groups_ and _imbalanced vulnerability across groups_ under HFT.\n\n\n## Response to W2: Experiments on More LLMs\n\nThank you for the suggestion. We expanded our experiments to include **Qwen2.5-7B**, with results shown in Tables 1.\n\nWhile HFT sensitivity varies across models, VAA consistently outperforms all baselines. Notably, as HFT epochs increase, the performance advantage of our method over baselines becomes more pronounced.\n\nTable 1. Experiments on Qwen2.5-7B.\n\n| Method | Ep1 |  | Ep2 |  | Ep3 |  | Ep4 |  | Ep5 |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | HS ↓ | FA ↑ | HS ↓ | FA ↑ | HS ↓ | FA ↑ | HS ↓ | FA ↑ | HS ↓ | FA ↑ |\n| SFT | 26.89 | 84.80 | 31.47 | 76.80 | 31.08 | 86.80 | 33.27 | 87.20 | 33.07 | 86.40 |\n| RepNoise | 22.31 | 83.00 | 25.10 | 81.60 | 26.49 | 88.80 | 30.68 | 87.20 | 30.88 | 88.00 |\n| Vaccine | 29.88 | 82.60 | 29.28 | 84.20 | 28.88 | 83.20 | 30.48 | 85.40 | 29.48 | 85.60 |\n| Booster | 19.92 | 85.20 | 21.91 | 84.80 | 25.10 | 87.40 | 26.29 | 87.60 | 30.28 | 88.00 |\n| VAA | 17.73 | 86.20 | 18.33 | 86.40 | 20.12 | 85.40 | 21.91 | 87.60 | 22.11 | 88.60 |\n\n\n## Clarification on Figure 2\n\nWe apologize for any confusion in our initial representation and provide the following clarifications:\n\n*Q1: “The meaning of ‘Common’ is unclear.”*\n\nIn Fig 2, _“Common”_ refers to the intersection of forgotten (or unforgotten) examples across different poisoning ratios. We define:\n\nCommon = $\\frac{|A_1 \\cap A_2 \\cap A_3|}{N}$,   (1)\n\nCommonRatio = $\\frac{|A_1 \\cap A_2 \\cap A_3|}{\\min(|A_1|, |A_2|, |A_3|)}$.  (2)\n\nHere, $A_i$ is the forgotten (or unforgotten) sets under a setting; $N$ is the dataset size. In the figure, *non-shaded bars* represent forgetting in a setting ($|A_i|/N$), while **shaded bars** represent overlap across different settings (Eq. 1).\n\n*Q2: “SST2 (p=10%) looks identical to p=20% in Fig 2(a) but not (b). Is this a mistake?”* \n\nThere is no mistake. In Fig 2(a), the shaded regions indicate examples forgotten under _all_ settings (Eq. 1), which are identical across poison ratios. The non-shaded heights differ slightly, with 20% showing more forgetting than 10%.\n\nCompared with Fig 2(b), non-shaded heights for SST2 (p=10%) remain the same (Ai/N), but shaded regions differ as they reflect intersection of forgotten examples across different settings\n\n*Q3: “...claims significant forgetting overlap, but Fig 2 does not show this.”* \n\nThis is reflected in Figure 2(a), where the shaded portion occupies most of the 0% bar—indicating a high _CommonRatio_. This suggests that examples forgotten under the clean setting are also frequently forgotten under poisoned settings.\n\nWe will make revisions for clarity. \n\n## Explanation of Stage 1\n\nThank you for your question. In Stage 1, we estimate which portions of alignment examples are more vulnerable to HFT. The process works as follows:\n\n1.  We begin with an aligned model and simulate HFT using Alpaca data mixed with 10% harmful content.\n2.  During the HFT process, we record T predictions and compute ForgotNum—the number of times each example is forgotten (using Eq. (1) in manu).\n3.  Examples with ForgotNum = 0 (never forget) are deemed invulnerable; others are considered vulnerable.\n\nThis estimation provides prior knowledge about data vulnerability that our algorithm uses in later stages. We will clarify this process in our revision.\n\n***\nThank you again for your constructive feedback, which has undoubtedly strengthened our work. We hope our responses could effectively address your concerns and welcome any further suggestions.", "claims_and_evidence": "Most of the claims in the paper are supported by convincing evidence，there are still some points that I think are not so clear.\n\nIn line 177 to 180, the paper claims a finding ''there is significant overlap in the forgotten data across different poison ratios''. But the corresponding Figure 2  does not seem to show this. No further evidence is given.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theory support in paper", "experimental_designs_or_analyses": "Yes, no obvious issue", "relation_to_broader_scientific_literature": "A new method to solve Harmful Fine-Tuning problem from data perspective.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "## Strengths\n\n* This paper first reveals that certain subsets of alignment data are consistently more prone to forgetting during HFT across different fine-tuning tasks and exhibit lower robustness compared to other subsets.\n* The proposed method significantly reduces the harmful score while maintaining the average performance.\n\n## Weaknesses\n\n* Lack of theoretical support.\n* The paper only involves Llama-2 (7B) and lacks experiments on more models to demonstrate the generalization of the method.", "other_comments_or_suggestions": "See Questions", "questions_for_authors": "* I‘m a little confused about stage 1 in the proposed method. The paper mentions using a proxy fine-tuning dataset (Alpaca) to provide a group prior information for the real downstream fine-tuning data provided by the user. What is the specific process like?\n\n* In Figure 2, the meaning of ‘Common’ does not seem to be given in the text. \n\n* In addition, in Figure 2 (a) and Figure 2 (b), the results are different under the SST2 (p=10%) setting. In Figure 2 (a), SST2 (p=10%) and SST2 (p=20%) seem to be exactly the same. Please check whether there is a mistake.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LcxKLZI18T", "reviewer": "Reviewer_qvia", "summary": "The paper introduces Vulnerability-Aware Alignment (VAA), a method aimed at enhancing the safety of large language models by focusing on data subsets that are vulnerable to harmful fine-tuning. VAA employs group-based robust optimization to boost model robustness, lowering harmful scores while preserving performance across a variety of tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and positive feedback. We appreciate your recognition of our adaptation of the Group DRO framework into a two-player game between a hard sampler and the LLM as both interesting and technically sound. We also value your acknowledgment that our extensive experimental results generally support the method's effectiveness. Below, we address your comments with additional analysis and new experimental results.\n\n## Response to W1: Experiments on More LLMs\n\nThank you for the suggestion. We extended our experiments to include VAA and all baselines on **Qwen2.5-7B**, with results shown in Table 1. \n\nWhile performance varies across models, VAA consistently outperforms all baselines. Notably, as HFT epochs increase, the performance advantage of our method becomes more pronounced, further demonstrating its generalization ability. We will include these results in our revision.\n\nTable 1. Experiments on Qwen2.5-7B.\n\n| Method | Ep1 |  | Ep2 |  | Ep3 |  | Ep4 |  | Ep5 |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | HS ↓ | FA ↑ | HS ↓ | FA ↑ | HS ↓ | FA ↑ | HS ↓ | FA ↑ | HS ↓ | FA ↑ |\n| SFT | 26.89 | 84.80 | 31.47 | 76.80 | 31.08 | 86.80 | 33.27 | 87.20 | 33.07 | 86.40 |\n| RepNoise | 22.31 | 83.00 | 25.10 | 81.60 | 26.49 | 88.80 | 30.68 | 87.20 | 30.88 | 88.00 |\n| Vaccine | 29.88 | 82.60 | 29.28 | 84.20 | 28.88 | 83.20 | 30.48 | 85.40 | 29.48 | 85.60 |\n| Booster | 19.92 | 85.20 | 21.91 | 84.80 | 25.10 | 87.40 | 26.29 | 87.60 | 30.28 | 88.00 |\n| VAA | 17.73 | 86.20 | 18.33 | 86.40 | 20.12 | 85.40 | 21.91 | 87.60 | 22.11 | 88.60 |\n\n\n## Response to W2: Analysis of Convergence Trends\n\nThank you for your insightful suggestion! Indeed, in convex optimization scenarios, adversarial training reaches an equilibrium solution (saddle point) at convergence. However, in large-scale, non-convex LLM training, models often employ \"few-pass training\" due to computational constraints, stopping after several epochs—far from full convergence. This highlights a practical gap between LLM training and classical optimization theory.\n\nDespite this, we observe a clear _convergence trend_ in our adversarial sampling setup: the weights assigned to vulnerable groups increase over time, while those of less vulnerable groups decrease. This dynamic evolves in response to group-wise loss, as shown in Table 2, and indicates that the model progressively balances performance across groups.\n\nTable 2. Evolution of Group Weights and Loss During Training (Vulnerable: Invulnerable) \n|  | Ep=0.0 | Ep=0.20 | Ep=0.35 | Ep=0.5 |\n| :--- | :--- | :--- | :--- | :--- |\n| group weight | 0.37: 0.63 | 0.49: 0.51 | 0.48: 0.52 | 0.56: 0.44 |\n| EMA group loss | 3.54: 2.74 | 1.97: 1.97 | 2.03: 2.04 | 2.82: 2.58 |\n\nWe will include this discussion and analysis in our revision. We appreciate your thoughtful feedback on this important point.\n\n## Response to W3:  Discussion of Computional Overhead\n\nThank you for raising this point. We measure efficiency by the number of backpropagation steps (the dominant training cost). Below is a comparison:\n\n| Method | Computational Complexity |\n| :--- | :--- |\n| SFT | $O(1 \\times B P)$ |\n| Vaccine | $O(2 \\times B P)$ |\n| Booster | $O(3 \\times B P)$ |\n| VAA | $O(1.5 \\times B P)$ |\n\nVAA saves an average of 0.5×BP computation compared to the fastest baseline (Vaccine) by employing a curriculum learning strategy that gradually warms up the perturbation probability from 0% to 100%. \n\nIn practice, SFT for a 7B model finishes in under an hour, making the added cost acceptable given the substantial safety gains. \n\nWe will include this discussion in the revision.\n\n\n## Theoretical Support\nDue to space constraints, please refer to our response to Reviewer g5ub.\n\n## Response to Dataset Style Mixing\n\nThank you for bringing this up. We followed the experimental settings from previous works (Vaccine, Booster, etc.), which indeed introduces distribution variations. Given the complexity of real-world user behavior, we consider three possible scenarios:\n\n1. Standard users: Real-world data naturally contains mixed, heterogeneous content\n2. Malicious users: Deliberately injecting harmful data to undermine alignment\n2. Standard users: Harmful content naturally occurring within homogeneous datasets\n    \nOur setup addresses the first two scenarios. We acknowledge the importance of the third scenario and plan to construct homogeneous test benchmark for HFT in future work.\n\nWe appreciate your thoughtful feedback and will add this discussion in our revision.\n\n\n***\nThank you once again for your insightful comments, which has undoubtedly strengthened our work. We hope that our responses could effectively address your concerns, and we look forward to  any further feedback you may have.", "claims_and_evidence": "In general, the claims made in the paper are well supported.", "methods_and_evaluation_criteria": "The proposed VAA method first investigates the different vulnerabilities of alignment data under harmful fine-tuning senarios and employs the Group DRO framework to manage various vulnerability groups. It is interesting to adapt the typical Group DRO framework into a two-player game between a hard group sampler (the adversary) and the large language model. From a technical standpoint, this method appears sound.\n\nRegarding the evaluation, the paper provides extensive experimental results in Section 5, which generally support the effectiveness of the method. However, there are some limitations in my view:\n\n- The proposed method is tested on a single language model only. Testing on more architectures of different scales would better demonstrate the generality of the method.\n- The paper states, “Ideally, at convergence, the adversary represents a uniform distribution across different groups, as the LLM learns all groups equally well.” However, there are no experimental results showing how well the model achieves this ideal situation. It would be interesting to see how the distribution and examples change throughout the adversarial training. The absence of these results could lead readers to question the origins of performance improvements.\n- The computational overhead should also be discussed.", "theoretical_claims": "I did not find theoretical proofs, and there seems to be need for such proofs.", "experimental_designs_or_analyses": "See \"Methods and Evaluation Criteria\" mentioned above. \n\nAnother concern I have regarding the experimental design, as detailed in Section 3 and similarly in Section 5, involves the dataset creation method where the \"dataset is combined with randomly sampled harmful data from the Beavertail dataset at varying poison ratios.\" I question whether this approach of simply adding data from another source into an existing dataset truly reflects real-world scenarios where harmful data might be present. The newly introduced data may differ significantly in distribution, such as writing style, from the original dataset. A more realistic approach might be to integrate harmful data that mimics the style of the original text. Could the authors comment on how well their experimental setup models real-world conditions?", "relation_to_broader_scientific_literature": "The relation to broader scientific literature was discussed in the Section 1 and 2 of the paper.", "essential_references_not_discussed": "The relation to broader scientific literature was discussed in the Section 1 and 2 of the paper.", "other_strengths_and_weaknesses": "N/A, see above.", "other_comments_or_suggestions": "N/A, see above.", "questions_for_authors": "N/A, see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Igvcb5hrjd", "reviewer": "Reviewer_oySm", "summary": "The paper takes a look at harmful fine-tuning after a policy for alignment has been incorporated using a subset of curated examples at the provider's. The idea is that certain subsets of alignment examples are more vulnerable to being forgotten during HFT. The work, VAA, looks to identify these groups, and use group DRO to maximize the worst example, that implies adding harmful examples via a perturbation and minimizing its effect. However, instead of regularizing, the work proposes to more tightly sample from the alignment set if the perturbation is rather forgiving to forgetting. \n\nThere are two metrics, and experiments that benchmark with sota using them. They appear to be well designed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback on our work. We appreciate your recognition that this is a forward-looking work in a new domain that will gain importance. We are also grateful for your acknowledgment that our experiments are well-designed and comprehensive, and that our design of VAA based on GDRO is a good fit and the first application to HFT scenarios. We are pleased that you recognize our method works well and achieves SOTA results. We will address your comment below.\n\n\n\n## Response to the question: I wonder if you didn't poison your data yourself, and had to determine which subsets were harmful, how would that be realized? \n\n\nThank you for your thoughtful question, and we apologize for the lack of clarity in the original manuscript.\n\nAt stage 1, we simulate harmful fine-tuning (HFT) using proxy data. The goal of this stage is to partition the alignment data into vulnerable and invulnerable subsets. This method is generally applicable and independent of the actual downstream poisoning process.\n\nThe alignment data can be partitioned in this way; it is not grounded in theoretical assumptions, but rather in empirical observations of large-scale heterogeneous alignment data. Specifically, we observed that certain examples are more likely to be forgotten during HFT, and we refer to these as vulnerable samples. Our experiments (Figure 2) show that different poison data—even with 0% poison—result in common forgetting. This enables us to use proxy data to simulate HFT. Our goal is to estimate, in a data-driven manner, which portions of the alignment dataset are more susceptible or robust to such fine-tuning, even in the absence of explicit data poisoning.\n\nTo operationalize this, we begin with an aligned model that performs well on its training set. We then simulate HFT by fine-tuning the model on a proxy dataset (Alpaca) augmented with 10% randomly sampled harmful data. During this process, we evaluate the model's predictions on the alignment training set over T iterations and compute how many times each example is forgotten, denoted as ForgotNum (Equation 1).\n\nExamples with ForgotNum = 0 (i.e., never forgotten across T iterations) are assigned to the invulnerable group, while all others are assigned to the vulnerable group. This empirical partition serves as a prior for estimating data vulnerability, which our algorithm leverages in subsequent stages.\n\n\n----------\n\nThank you once again for your professional and inspiring feedbacks. We hope that our responses could effectively address your comments, and we look forward to any further feedback you may have.", "claims_and_evidence": ". that vulnerable and invulnerable are the two groups all alignment examples fall into. This is not substantiated in terms of theory. The re-sampler is more expressive than that.\n\n. that vulnerable examples are identifiable. The metric is designed to work with that.\n\n. that DRO is a good fit.  Yes, in terms of robust optimization, and the work is trying to do just that. The group idea  is approriate for the specific requirement, while the work has the claim that they are the first to use it for HFT.\n\n. that the whole idea works. The numbers exist.", "methods_and_evaluation_criteria": "Because the method is founded on a  known method for robust training with shift, whether embedding-shift or other, it is a logical methodto improve robustness against HFT too.\n\nThe two metrics added are\nHarmful Score (HS), that quantifies the model's resilience to harmful examples, which is a measure of safety.\nFine-tuning Accuracy (FA), that measures the model's performance on downstream tasks, and is fairly standard.", "theoretical_claims": "I have exposed my position over a little question right in the  summary.", "experimental_designs_or_analyses": "Experiments are standard.", "relation_to_broader_scientific_literature": "There may not be a lot of literature. The domain of inquiry is new. But it will gain importance, and that makes the work in question, a forward-looking work.", "essential_references_not_discussed": "I think the LR is probably all that is out there!", "other_strengths_and_weaknesses": "Experiments are comprehensive.", "other_comments_or_suggestions": "-", "questions_for_authors": "I wonder if you didn't poison your data yourself, and had to determine which subsets were harmful, how would that be realized?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews, unfortunately. 
The paper puts forward a novel approach and perspective for alignment - that some subsets are more prone to forgetting, so we need to "upweight" them during alignment to reduce the forgetting. I think this is very different from previous ideas, and warrants more discussion and should be accepted. 

While I agree with the broad criticisms raised, I believe the reviewers have addressed them / paper can be accepted despite the weakness. 

1. Lack of theoretical support: I agree it'd be nice to provide a conceptual basis/intuition. I encourage the reviewers to do so, incorporating the rebuttal responses. However, this doesn't seem like a basis for rejection

2. More than one model: the authors added Qwen in addition to Llama - they report similar findings 

3. Careful analysis of dataset mixing: the authors build on prior work and provide more justification in rebuttal.
Decision: Accept (poster)