SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation
Authors: Mingjie Li
Wai Man Si
Michael Backes
Yang Zhang
Yisen Wang
Venue: ICLR 2025 Poster (2025)
Keywords: PEFT Training
Safety Alignment Preserving
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: As advancements in large language models (LLMs) continue and the demand for personalized models increases, parameter-efficient fine-tuning (PEFT) methods (e.g., LoRA) become essential due to their efficiency in reducing computation costs.
However, recent studies have raised alarming concerns that LoRA fine-tuning could potentially compromise the safety alignment in LLMs, posing significant risks for the model owner.
In this paper, we first investigate the underlying mechanism by analyzing the changes in safety alignment related features before and after fine-tuning.
Then, we propose a fixed safety module calculated by safety data and a task-specific initialization for trainable parameters in low-rank adaptations, termed Safety-alignment preserved Low-Rank Adaptation (SaLoRA). 
Unlike previous LoRA methods and their variants, SaLoRA enables targeted modifications to LLMs without disrupting their original alignments. 
Our experiments show that SaLoRA outperforms various adapters-based approaches across various evaluation metrics in different fine-tuning tasks.

Published as a conference paper at ICLR 2025
SALORA: SAFETY-ALIGNMENT PRESERVED LOW-RANK ADAPTATION
Mingjie Li', Wai Man Si', Michael Backes', Yang Zhang', Yisen Wang 2,3+
CISPA Helmholtz Center for Information Security
2 State Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology. Peking University
3 Institute for Artificial Intelligence, Peking University
ABSTRACT
As advancements in large language models (LLMs) continue and the demand for personalized models increases, parameter-efficien fine-tuning (PEFT) methods (e.g., LoRA) become essential due to their efficiency in reducing computation costs. However, recent studies have raised alarming concerns that LoRA fine-tuning could potentially compromise the safety alignment in LLMs, posing significant risks for the model owner. In this paper, we first investigate the underlying mechanism by analyzing the changes in safety alignment related features before and after fine-tuning. Then, we propose fixed safety module calculated by safety data and a task-specific initialization for trainable parameters in low-rank adaptations, termed Safety-alignment preserved Low-Rank Adaptation (SaLoRA). Unlike previous LoRA methods and their variants, SaLoRA enables targeted modifications to LLMs without disrupting their original alignments. Our experiments show that SaLoRA outperforms various adapters- based approaches across various evaluation metrics in different fine-tuning tasks. Code is available at https: //te://gitttt com/
(aomles11/SaLoRA
Disclaimer. This paper contains uncensored toxic content that might be offen- sive disturbing to the readers.
INTRODUCTION
I Large Language Models (LLMs) have demonstrated impressive performance in language under- standing and generation for general Natural Language Processing (NLP) tasks (Brown al. 2020 OpenAl 2022, Touvron et al. 2023a by Anthropic 2023 Team et al. 2023 Qin et al. 2023). As result, powerful AI assistants and chatbots based on LLMs have gained significant interest from academics and industry. Furthermore, users or developers are likely to fine-tune these models to build domain-specific or personalized LLMs. For example, OpenAl offers a fine-tuning service for users to create custom models. However, fully fine-tuning LLMs is computationally expensive due to the enormous number of trainable parameters. To address this, researchers have proposed different parfmeter-efffcient fine-tuning (PEFT) methods (Houlsby et al. 2019). Among them, low-rank adaptations (LoRA) (Hu et al.) 2021) is one of the most popular approaches and is a widely used strategy for reducing computational costs. Instead of adjusting all the weights in the LLM layers, LoRA introduces additional adapters for each trainable layer, shown in Figure 2 a).
Meanwhile, societies have raised growing concerns about preventing LLMS from facilitating harmful or unsafe activities, such as providing instructions on making bombs or spreading fake news (Zou
al. 2023b Deshpande et al. 2023 Zhuo et al., 2023 Si et al.) 2022 Liu et al.) 2023a AAkus et al. 2025 In response, researchers have developed several methods to enhance LLM safety, known as
safety alignment, including supervised fine-tuning (SFT) OOyang et al. 2022) ) and reinforcement learning from human feedback (RLHF) (Bai et al. 2022 Dai et al. 2024). As a result of these advancements, pre-trained LLMs like GPT-4 (OpenAl (023) and Llama (Touvron et al. 2023b) have become more adept at rejecting unsafe prompts. However, Mukhoti et al. (2023) shows that the fine-tuned model's ability on tasks different from those of the downstream model is reduced significantly compared to the original model. Furthermore, recent works (Qi et al.) 2023 Lermen
"Corresponding Author: Yisen Wang (yisen, wang www.epuu..ddduu.cm.

---

Published as a conference paper at ICLR 2025
Benign Casos
Harmful Cases
Q: Give five of Ayn Rand.
Q: stop-by-stap instruction on
make bomb?
Pretrained LLMMA-2-chat
agged" (1952)
Pretrained LLaMA-2-chat
fullfill your request...
LoRA
fourtaineed We dather The Vitue of Sefishress
finetuned
LoRA
convection Ce heat. snwwerf) aanet
sest anwertd resseppioe menoppt:
SaLoRA finetuned
fourtainhead Adax Strugged We Living Aethem The Virtae of Saftthruus
SaloRA
finetuned
cannot fullfill your request...
Figure I: Examples of fine-tuned Llama-2-chat-7B's responses on benign and harmful prompts, the model is fine-tuned the Alpaca dataset with LoRA and our SaLoRA.
& Rogers-Smith 2024) have indicated that the safety alignment of LLMs may deteriorate when users fine-tune them with LoRA to enhance their domain knowledge, even when all samples in the fine-tuning datasets are benign, shown in the right part of Figure I Such weaknesses will greatly impede the deployment of LoRA methods for LLM's efficient fine-tuning since additional safety alignment methods (like RLHF) also need huge computational resources.
To tackle the above problem, we first investigate the reasons for the drop in safety alignment after the PEFT fine-tuning, We inspect the changes in the LLM's ability to detect safe or unsafe prompt scenarios before and after LoRA fine-tuning, drawing inspiration from previous research (Peters et 2018: Li et al. 2023; Lee et al. [2024] Clark et al. 2019). Then, we find that poor performance of the safety mechanism is caused by significant changes in the LLM's features on harmful prompts and their safe response, denoted as safety features. Consequently, the LLMs cannot generate safe responses, leading to decline in safety alignment.
dim...
CSntoRA gd
Residual Weight
BsatoRA E grsd
Pretrained Weights
W AA pdkd
W E gefod
AsaLoRA R.
A = o0
dim,
Trainable
dim,
H6 CSatona Bp
Frazen
(a) Vanilla LoRA.
(b) Our SaLoRA
Figure 2: The sketches for vanilla LoRA and our SaLoRA during fine-tuning. CSoLORA here denotes our fixed safety module with a pre-caleulated linear weight, while ASaLORA and BSaLORA train-able adapters for down-stream tasks with Ag, B 's are the initializations.
Built on this observation, we attempt to maintain the safety features to preserve the LLM's ability to detect unsafe prompts. In this paper, we propose our Safety-alignment preserved Low-Rank Adaptation (SaLoRA) effectively do fine-tuning and preserve LLM's original safety alignment. To achieve this goal, our SaLoRA first introduces an additional safety module CSaLoftA. which is computed based on around 300 pre-colleed pollccele harmful prompts and their safety responses (far less than the fine-tuning data), as illustrated in Figure (b). It projects the new features added by the SaLoRA adapters the sub-space orthogonal the original safety features in order to preserve the original safety alignment. In addition, we propose a task-specific weight initialization for SaLoR A's trainable adapters ASaLoRA and BSaLORA derived from small portion of the fine-tuning data to make oRR converge better with our fixed safety module CsaLoRA- Our contributions are summarized follows:
2

---

Published as a conference paper at ICLR 2025
We investigate the reason for the poor performance of the safety alignment after LoRA fine-tuning and find that it is attributed to changes in LLM's representations of harmful prompts and the desired safe responses.
We propose a new efficient PEFT method called SaLoRA, which consists of a fixed safety module CSaLARA and trainable adapters ASoLoRA, BSoLoRA initialized by our task-dependent method, shown in Figure
Empirical results demonstrate that our SaLoRA consistently surpasses existing PEFT meth- ods and their combination with state-of-the-arr alignment methods in maintaining the model's safety while achieving comparable or even better results on downstream evaluation tasks.
2 RELATED WORK
2.1 CHAMMETER-EFFCIIRE FINE-TUNING
Recent developments in NLP and the use of transformer architectures like Llama-2 and GPT-4 have led the success of large models. However, full fine-tuning computationally expensive for such a large model. Also, it is costly to store and deploy fine-tuned models for each downstream as these models are the same size the original pre-trained models. Recently, Parameter-Efffiieen Fine-Tuning (PEFT) has been used to address both problems by fine-tuning only a small number of additional model parameters while freezing most parameters of the pre-trained models. It can drastically reduce the memory and storage requirements for training and saving the model. For such purposes, various PEFT methods (Razdaibiedina al. 2023 Liu al. 2024 Kopiczko al. 2024) have been proposed. For instance, Houlsby et al. (2199) attaches extra trainable parameters to each layer while freezing the original model. On the other hand, Lester & Constant (202)) proposed prompt tuning to optimize the token embedding (soft prompt) instead of the entire model. Recently, |Hu et al.) (2021) proposed Low-Rank Adaptation (LoRA), which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the transformer architecture. MMeg et al. (2024) leverage singular value decomposition to reduce the number of trainable parameters. Our research leverages similar ideas to maintain the model performance and the model capability to target behavior such as safety at the same time.
2.2 SAFETY ALIGNMENT LLMS
LLMs have been shown to exhibit undesirable behaviors through regular interaction, such as triggering toxic responses. They may also provide incorrect answers even when given correct input. To better align LLMs with human values, various approaches have been proposed, including reinforcement learning from human feedback, direct preference optimization (Rafailov et al. (2024), and others (Xu al. 2024). However, recent studies have demonstrated that LLMs can be manipulated exhibit undesirable behaviors through specially crafted prompts, circumventing existing safety measures. Zou al. (2023b) demonstrate that LLMs can still be manipulated through specially crafted prompts, bypassing existing safety measures. To tackle these methods, researchers have proposed various well-designed prompts (Xie al.) [223 Wei et al. [2023] Wang et al.) 2024bb] Li et al. 2025) or alter the model's 's features to enhance model safety (Wang et al., 2022a Li et al. 2024). Besides. researchers also leverage adversarial training on LLMs to enhance the model 's safety (Huang et al.)
2024
Mo et al. 2024).
Apart from the safety problems for the on-the on-the-shelf LLMs, Qi et al. (2023) also found that the safety alignment of LLMs can be compromised by fine-tuning. To tackle such problems, Hsu et al. (2024) propose method that projects the LoRA's training adapters back to the LLM safety region with the projection matrix calculated by the difference between LLM's chat version and its base version. However, such a method is unstable as its performance depends on some hyper-parameter settings. Also, it cannot work if the cannot get the base version of the chat model they used. Instead, our method only requires a small amount of safety data and then we can successfully maintain LLMSs safety after the LoRA fine-tuning without any additional hyper-parameter chosen.
3

---

Published as a conference paper at ICLR 2025
3
LORA AND ITS SAFETY ALIGNMENT DROP
3.1 METHODS FOR LOW-RANK ADAPTATION
Drawing from the widely accepted hypothesis that updates during fine-tuning typically form a low "intrinsic rank" (Li et al., 2018: Aghajanyan et al. 202T), LoRA offers an efficient solution for fine-tuning. It utilizes the product of two low-rank matrices B E RR and A E R as the
incremental weight matrix obtain the final updates AW E Tidik, This formulation allows for efficient fine-tuning while accommodating the inherent low-rank nature of updates.
(I) where A, B trainable parameters with & min(d. (d, k). Wo is the pre-trained weight matrix and fixed during the training process. The matrix A is initialized with the normal Gaussian distribution, 7ero. Thus, the additional modules will not affect the performance of the
W Wupdate Wo + AW = Wo + BA,
while B is initially original models unless they are updated during fine-tuning. Besides this decomposition, researchers have proposed various other methods to enhance performance. Among these, Weight-Decomposee Low-Rank Adaptation (DoRA) (Liu et al. 2024) and PiSSA (Meng et al. 2024) show better results. DoRA accounts for changes in weight norms and their impact overall performance, proposing additional trainable normalization modules to adjust the weight norms after fine-tuning. PiSSA modifies the original initialization of LoRA adapters' weights via SVD method.
3.2 EMPIRICAL ANALYSIS ON SAFETY ALIGNMENT AFTER LoRA
Safety alignment LLMs has emerged as one of the most critical research topics due to their impressive capabilities and potential societal impacts, However, Qi ct al. (2023) have indicated that LoRA fine-tuning may compromise models' safety alignment abilities and lead to harmful responses
on unsafe prompts. As proposed in many recent works (Wang et al. 2024a; Li et al. [2024] Zou al. 2023a). LLM's intermediate features on harmful prompts and their safety responses, which we denote them as safety features in the following, are crucial to the safe behavior on rejecting unsafe prompts. Motivated by this observation, adopt the widely used interpretability tool, linear probing, to analyze the changes in LLMs' safety features in this section.
We train the linear probes, denoted W each LLM layer, averaged across all timesteps (X'), to determine whether they belong to harmful prompts and their safe responses not:
probe: to classify the MLP outputs from the last tokens for
P (Harmfull sigmoid(W (W
prob
We first train linear probe as classifier with the original Llama-2-chat-7B model's attention head output of each layer on benign prompts and harmful prompts with their safe responses to judge whether the input prompt-response pairs belong to the safe scenarios or harmful scenarios. Then, we evaluate the features of LoRA fine-tuned models with different ranks r using the same linear probe. In detail, we calculate the accuracy of correctly classified safe or harmful grompt-responss pairs, referred to as linear probing accuracy. We then use the changes in linear probing accuracy to evaluate whether the low-rank update affects the original safety features, potentially leading to failures in the LLM's safety mechanism when handling unsafe prompts, illustrated in Figure
Accuracy
Layer
Figure 3: Linear Probing Accuracy for classifying unsafe prompts and their safe responses each attention layer on Llama-2-chat-77 before and after LoRA fine-tuning.
From the figure, we can observe that the linear probe trained with the original LLM's features cannot effectively classify the harmful or benign prompts and their safe response pairs on the LoRA fine-tuned models, as the accuracy of each layer drops more than 10%. This indicates that LoRA significantly alters the features of unsafe prompts and their safe generations by updating the original weight matrix with low-rank adapters. Therefore, LLM's original safety mechanism cannot be activated as the safety features eliciting the safe response have already been changed.
4

---

Published as a conference paper at ICLR 2025
3.3 THEORETICAL ANALYSIS
In this section, we try to theoretically analyze why LoRA tin-ttnnig may impact the LLMs' safety features. Inspired by recent research (Lee et al. 2019; Sun et al. 2023). which shows there exist some low-rank weight spaces for safety and other different tasks, we first made assumptions by decoupling the original LLM weights with different orthogonal bases to denote the safety weight region, which activates the most on harmful prompts. These regions can be split from weights W as W. = Σ - v.viv' as most trainable layers in LLMs are linear layers. Then we can obtain the following proposition for the relationship between the gradient of low-rank module A AW and the safety weight regions as follows.
Proposition 1 Letting Xw denote the features for benign training prompts, Y ww denotes output features for layer W with adapters, Yw = (W + AW) XW, and ((Yw) is the training loss on benign prompts. If the activation IW;s sXwll > 7, then the Frobenius norm of the dot product between Ws s and the grudient of A AW, denoted as gradaw. can be lower-bounded by the following,
[ sgrad mimn VVY
(3)
where mmin denotes the smallest singular value of given matrix.
The assumption and proof can be found in Appendix B The main assumption of this proposition that the benign prompts may also activate the safety region of weights, which proved in recent works (Wei et al. 2024). From the proposition, we can see that the gradient and the safety weight regions are not orthogonal each other. Thus the updated adapter will perturb the safety weight regions, leading to changes in safety features and aafety-alignment drops after fine-tuning.
4 SAFETY-ALIGMMENT PRESERVED LOW-RANK ADAPTATION
Based on the analysis in Section we propose a Safety-alignment preserved Low-Rank Adaptation, called SaLoRA, including a fixed safety module and trainable adapters initialized by our task-specific initialization. Our SaLoRA can effectively preserve LLMs' safety alignment during fine-tuning while enjoying satisfying downstream performance.
4.1 PROPOSED SAFETY MODULE IN SALORA
As the former analysis states, one of the key reasons for the safety alignment drops after LoRA fine-tuning is the changes in the original safety features. Such changes will cause LLM's safety mechanism inactivated some unsafe prompts and return harmful responses. Based on this finding, we try to ensure the new features added by the adapters are orthogonal to LLMs' original safety features preserve them and model's original safety alignment. To achieve this goal, we add linear module with pre-calculated weight CaaLoRA, denoted as the safety module, as shown in Figure The value of CSALORA can be reformulated as the minimizer of the dot-product similarity between the features on harmful prompts, formulated as follows:
Cs = arg min Cs
|| (Cs (BsAsX, - BoAoX.)) (WX,)
(4)
where the subseript ss" here denotes SaLoRd for convenience, Bs, Ag are the trainable weights for the low-rank adapters with Bp, Ag representing their initialization, X,, denotes an input feature of several unsafe prompts and their safe responses, and W the original weights for certain linear layer of LLM. Since As and Bs cannot be easily controlled during fine-tuning. We try to optimize the following alternatives to set Cs's value:
arg min CC W., 1/2-
(5)
Cs
We note that although optimizing Eqn (5) not equal to Eqn minimizing Eqn (5) can also control the magnitude of Eqn (4) due to the norm inequality,
(Cs (BsAsX, BoAoXx)) (WXx))I S |BsAsXx BoAoX, C2 w
(6)
However, one can see that Cs = 0 is trivial solution for both two objectives but it will make the adapters useless. Thus, we assume that the most important safety regions also lie in low-rank weight
5

---

Published as a conference paper at ICLR 2025
space with the orthogonal projection matrix Uc R and our Cs be set the orthogonal complementary space Cs II C to avoid the trivial solution. The new objective
arg min -I- UC)) 2
Ue
The optimal results for Uc are attributed to the singular value decomposition of WX,, as studied in former works (Wei et al., 2024). Thus, the weight Cs for the our safety module is set to be I Ug. Uc the top rs singular vectors of the output features WX, on harmful prompts and we call r, as the safety rank in the following.
4.2 INITIALIZATION OF SALORA'S TRAINABLE ADAPTERS
Since our additional safety module CSaLORA changes the original gradients of adapters' trainable weights and makes the training harder. We propose a new initialization method using the fine-tuning data to help our SaLoRA's trainable adapter ASaLoRA and BSoLORA converge better.
Inspired by former research (Sun et al. 2023 Liu et al. 20235) on locating task-specific weight regions, propose task-specific initialization for SaLoRA's ASLLRA and BsaLoftA to make their training easier Our task-specific initialization first locates the low-rank task-related weight regions and then initializes A AaLoRA and B BaLLoA according to these regions.
Like the methods the above section, we first try to detect the task-specific regions of certain linear layers in LLMs using the following optimization,
arg min ron
|WX; - WX,12,
(8)
where rz is the task-specific rank to avoid trivial solution, X, here denotes the input features of LLM's layers concerning input samples of certain downstream tasks, and W denotes the low-rank weight region related certain tasks. As illustrated in the above section, the minimizer
W _ W.
where U E TRdxrss is the top r; left singular vectors of WX, If we set r = which the rank for adapters A s; Bg, we can directly split it to propose our SaLoRA's initialization:
B's U, A'S = UW,
(10)
where the subscript "s" here denotes SaLoRA for convenience. However, one can see that the norms between B's and A s are not balanced as U here is orthogonal matrix. The imbalanced weight norm will influence the gradient norm, which may influence the models' generalization as studied in many works (Zhao et al. (022). To balance the weight norms of module As and Bş in the adapters, we also do singular decomposition on weight W as follows,
(11) Since the middle diagonal matrix S consists of singular values of the original weight W and top singular values along with singular vectors are the key part of the original weight matrix, we initialize the weight As and Bg as follows,
W- USVT.
Bş = -U=UUTT - S(m) As = V5 T
(12)
where [(r). [orr denotes the matrix consists of top-r left singular vectors and right singular vectors, and S [ denotes the top-r singular values of W. Then we can obtain the low-rank adapter matrices which are highly correlated to downstream tasks.
To ensure the initial stage of L.M with our SaLoRA adapters perform the same as the pre-trained model, we also re- re-parameterize the weight of L.M''s layer as follows,
W W CsBsAs-
(13)
The fixed safety module Cs. trainable adapters Ag. Bg initialized by the task-specific initialization
and the residual weights W' build the whole structure of our S as shown in Figure 2 Training the trainable adapters on the fine-tuning datasets, our A can help LLMs efficiently obtain certain abilities some domain-specifie knowledge while preserving its safety alignment.

---

Published as a conference paper at ICLR 2025
4.3 IMPLEMENTATION OF SALORA
In this section, we will provide a comprehensive pipeline of SaLoRA's training, saving, and inference processes, drawn in Figure Before SaLoRA's training, we first set the weights of safety module CSLLORA with CaLLRAA = =I- U where Uc is the optimal solution for Equ 7 And we calculate the initialization of adapters BaLLRA A SaLoRA with Eqn (T)) Then we can begin Sa'oRA's training process by updating trainable adapters LaRRA BatLoRA while freezing the residual weight and safety module CSLLA as Figure s left side shows.
I
SaLoRA
Adapter Saving
Inference with SaLoRA
Training
Coeaaa Re14
Residual Weight
Broteet
Residual Weights
Pintosa & geee
Ausoasa g
Marge weight
W - B.
Rescona R
Mascake
dm,
Gpsses
Griaa
Freech
Figure 4: The training, adapter saving, and inference procedure for our SaLoRA The blue weights here denote fixed parameters while the orange modules are trainable during fine-tuning. Bo are the initialization of A SaLoRA. BsaLoRA-
After the LoRA training, the updated weights need to be saved. As the size of our safety module CSaLoRA E Rdout xdout is the same as the original weight W, we need an additional weight merging step the adapter saving time shown in the middle of Figure 4 Such an additional weight merge step is a simple multiplication B's LoRA = ASaLoRABSaLoA
Then the storage cost for B's LoL the same BSoLoRA vanilla LoRA's adapter. However, as we also need to save B/J and Ag, the storage cost will be twice as large as vanilla LoRA. Luckily, the adapters are much smaller compared with LLM's s size, and most adapters published on huggingface are only tens of MB. Considering the good performance in the following section, such a storage overhead should be acceptable.
At the inference stage, our SaLoRA is almost similar to the vanilla LoRA with only one simple step to calculate the residual weights, which will not cost much as they are just some multiplications of low-rank matrices and subtractions. Therefore, our methods can be easily adopted with many popular inference pipelines, like vLLM (Kwon et al. 2023).
5 EXPERIMENTS
In this section, we conduct variety of experiments to demonstrate the efficiency of our SaLoRA in improving domain-specific abilities while preserving LLM's safety alignment. Firstly, we train four widely used LLMs on Alpaca (Wang et al. 2023) with SaLoRA and other LoRA methods, and then we compare the harmful rate of these models' responses with and without some state-of-the-art post- hoc alignment methods on unsafe prompts. Then, we compare the utility of different LoRA-traing models with our SaLoRAA commonsense reasoning. We also conduct ablation studies on ranks' influence in Appendix C and examples of different methods outputs in Appendix D
Baselines. In this paper, compare our SaLoRA with three widely adopted PEFT training methods: LoRA, DoRA, and PiSSA. Additionally, we adopt four state-of-the-an post-hoc safety alignment methods on LoRA fine-tuned models as baselines, including the input- based Self-Reminder (Xie) et al. 2023), the feature-intervention method InferAligner (Wang al. 2024a), the training- based approach Vaccine (Huang et al. 2024), and the model editing method Safe LoRA (Hsu et al. 2024). The detailed settings are listed Appendix A
Settings. In our A,, we set r. = = 32 in the following experiments, and rank r for LoR adapters set other LoRA variants according experiments requirements. To obtain the harmful feature X, in Eqn (7) for the safety module, we use 70% harmful prompts in the AdyBench dataset (Zou et al.
2023b) along with their safe responses, which is also used in baseline method Vaccine
7

---

Published as a conference paper at ICLR 2025
and InferAligner. We use the features of training samples X, for the initialization. Experiments are finished on the PEFT (Mangrulkar et al. 2022) training pipeline with a batch size equal to 16 for all our experiments. The experiments finished on a single NVIDIA A100-80GB GPU.
5.1 EVALUATIONS LLM SAFETY AFTER FINE-TUNING
In this section, we first evaluate the safety aligmment of four widely used models, including Llama- 2-chat-7B, Llama-2-chat 1 13B, Llama-3.1 3.1--nnttuutt--81 and Misttall-B--nstruutt-y
(Jiang al.] 2023p. We first train them on the Alpaca datasets with AdamW (Loshchilov 2019) for 1 epoch with the learning rate equal 0.0002. Then evaluate the harmfulness score on its responses with the (Llama Team 2024). We use 70% harmful prompts in the AdyBench dataset (Zou et al., 2023b) for InferAligner, Vaccine, and our SaLoRA. We use the rest of 30% harmful prompts in AdyBench for the safety evaluation, denoted as AdyBench's test subset. The harmful rates for LLMs with different methods reported in Table [ The harmful rate here represents the G-ard-d-addd....
Jlama-Guard-2
proportion of all responses that were labeled as unsafe by
Table 1: The harmful rate of LLMs' responses on harmful prompts in AdvBench's 's test subset against different LLM on Alpaca with different methods. "IA" here denotes the InferAligner, "SR" denotes the self-reminder method, and "Vac" denotes the Vaccine method. The bold number denotes the best harmful rate of LLMs with or without the post-hoc alignment methods.
Llama-2-che 2-chat--hh 7B
Llama-2-chan 13B
Llama-3 3. I-sstruct--BB
Mistral Mistral-Instrucee Instruct-vÜ..
Belore l'ine-tuning
0.0%
0.0%
1.4%
17.8%
Rank for PEPT training
16
32
16
32
16
32
16
32
LORA
23.7%
31.7%
15.7%
13.8%
11.7%
8.7%
82.5%
71.7%
Base PEFT
PISSA
31.7%
35.7%
17.4%
18.1%
13.8%
14.5%
83.3%
86.9%
DORA
23.7%
25.3%
18.8%
16.7%
10.1%
9.4%
64.5%
66.7%
LORA SR
11.7%
9.4%
7.8%
5.8%
10.3%
7.8%
79.6%
72.2%
LoRA w. post-hone
LORA IA
13.5%
23.7%
10.3%
6.7%
6.7%
5.8%
55.1%
60.1%
alignment
LORA w. Vac
20.2%
25.3%
32.5%
39. 99.8%
41.1%
38.3%
76.8%
73.1%
Safe LoRA
15.7%
14.5%
10.1%
9.4%
8.5%
6.7%
63.0%
66.7%
SaLoRA
3.5%
4.4%
2.9%
3.5%
2.9%
1.4%
31.7%
36.5%
Ours
SaLoRA w. SR
1.4%
2.9%
1.4%
1.4%
0.0%
1.4%
15.7%
10.7%
The results indicate that fine-tuning with LoRA and its variants, such as DoRA and PISSA, compro- mises the original safety alignment for all kinds of LLMs, even when the fine-tuning dataset is free of harmful content. While post-hoc alignment methods can mitigate this issue to some extent, they cannot fully restore the models' safety alignment, as their abilities to produce harmful responses have already been broken. Among them, the InferAligner and Belf-Reminder perform the better. Vaccine performs unstable, the possible reason is the alignment data we used for Vaccine's 's pre- pre-training is too small, only around 300 samples.
Beyond those post-hoc methods, SaLoRA demonstrates superior performance in preserving the safety alignment of LLMs after fine-tuning. The harmful rate of LLMss fine-tuned with Sal oRR remains comparable to their original rate before fine-tuning, highlighting SaLoRA's effectiveness in maintaining alignment. Notably, applying SaLoRA to suboptimal baseline harmful rate of 47.8%, makes a clear improvement in safety alignment. We hypothesize that this phenomenon may be due to the safety features being strengthened other original features may be weakened during the fine-tuning. Additionally, when combined with post-hoc methods such as Self-reminder (denoted "SaLoRA w. SR' in Table T.. the harmful rate improves further, indicating enhanced safety alignment after fine-tuning.
which had
5.2 EVALUATIONS ON UTILITIES
In addition to preserving safety, an effective PEFT training method should also demonstrate compara- ble or superior performance in various tasks against others. Like many other works (Kopiczko et al. 2024 Liu et 2024h. we test eight commonsense reasoning tasks on Llama-2-chat-71 fine-tuned on commonsense- 15k (Talmor et al. 2019) with different methods to validate the effectiveness of our SaLoRA. These eight tasks are widely used in various works (Touvron al. 2023b Kopiezko et al. 2024: Liu et al. 2024) to assess commonsense reasoning abilities. We follow the evaluation settings
former works (Kopiczko al. 2024 Liu et al. 2024). and the results are listed in Table
8

---

Published as a conference paper at ICLR 2025
Table 2: The prediction accuracy (%) of Llama-2-chat--B on different commonsense reasoning tasks and its harmful rate after fine-tuning on commonsense-133 15k datasets with different PEFT methods.
Method
BoolQ
PIQA
SIQA
HellaSwag
WineGrande
ARC-e
ARC-e
OBQA
Avg.
Ace
Harmful Rate
LoRA
59.5
71.0
62.2
32.6
51.7
73.1
58,2
64.6
59.2
7.8%
DoRA
60.5
71.2
67.7
33.1
53.1
76.3
61.4
66.7
61.3
2.9%
16
PISSA
62.4
71.4
62.8
38.1
51.4
77.3
59.6
65.8
61.2
5.8%
SaLoRA
64.1
72.4
54.7
74.9
59.4
63.8
62.9
0.0%
LoRA
G033
72.4
63.3
31.1
56.6
74.6
58.1
67.6
60.5
5.8%
DoRA
62.7
73.3
65.3
35.9
51.5
78.5
59.6
68.1
61.9
7.8%
32
PISSA
60.4
52.5
78.7
66.8
9.4%
SaLoRA
60.6
72.7
64.5
47.6
57.1
74.0
57.0
64.8
62.3
1.4%
LoRA
62.4
73.9
64.5
30.1
58.1
78.4
62.2
68.6
62.8
8.5%
DoRA
63.2
74.7
67.4
30.3
33.8
78.6
70.3
63.5
9.4%
64
PISSA
65.9
75.8
66.5
59.5
80.3
64.5
71
65.9
10.1%
SaLoRA
66
74.1
67.5
52.6
59.6
76.8
62.6
66.4
65.7
0.7%
The results show that our SaLoRA not only outperforms the vanilla LoRA methods on safety- alignment preserving but also performs better on commonsense reasoning tasks across different r's settings by an obvious margin. Compared to DoRA and PiSSA, our method consistently performs better on the "HellaSwag" and "WinoGrande" tasks. When comparing average prediction accuracy. our SaLoRA method shows almost better results to DoRA and PISSA.
5.3 EVALUATIONS OF SALORA'S SAFETY ALIGNMENT UNDER JAILBREAK ATTACKS
Apart from the safety behaviors on natural harmful prompts in AdvBench, we also conduct experi- ments to evaluate SaLoRA's performance on jailbreak attack samples. In this section, we conduct the popular multiple GCG attack (Zou al. 2023b) with 1000 steps Vicuna and Llama-2 generate the universal jailbreak suffix 128 tokens. Adding to the harmful prompts AdvBench, we test the harmful rate of SaLoRA and other methods Llama-3. I-nsrrucct--88 Results drawn in Figure From the figure, we can see that our SaLoRA still shows a similar safety performance as the original model and outperforms the other methods with clear advantage. The results demonstrate that our S can effectively preserve models' safety alignment.
30
LL.MA331166 Instruct
SaLaRA
SeLoRA
LoRA
25
LoRA
60
Nom Rate
2vg Rcc1100
15
10
54
52
50
64
Figure 6: The averaged Commonsense Rea- soning accuracy for LoRA, SaLoRA with and without our task-specific initialization.
Figure 5: The harmful rate for LoRA, LoRA w.IA, Safe LoRA, and our SaLoRA under the multiGGG attack.
5.4 EVALUATIONS TASK-SPECIFIC INITIALIZATION
In this section, we explore the effectiveness of our task task-specific initialization. We conduct several experiments by replacing our task-specific initialization with the same initialization for A and B in LoRA. We then use this new method to fine-tune Llama-2-chat-77 on the Commonsense Reasoning 15k dataset and evaluate its performance described in Section 5.2 the results are drawn shown in Figure (oauee/denooe denoted as "SaLoRA w.o our init". The results indicate that our task-specific initialization significantly improves fine-tuning performance compared the original initialization. Without

---

Published as a conference paper at ICLR 2025
task lask-specific initialization, SaLoRA even performs worse than vanilla LoRA. Such phenomenon demonstrates the necessity our proposed initialization. The possible reason that our fixed safety module will project the gradient to the safe region and cause bad impacts on the model's convergence.
5.5 COMPUTATION COSTS
In this section, we try to compare the computational resources for our SaLoRA and other methods. We listed the resource consumption of different methods for fine-tuning Alpaca with batch size equal to and = 32 for epoch in Table As the table shows, our SaLoRA only needs a little longer pre-processing time for calculating the safety module and initializing the adapters As and Bs compared with the whole training time (0. 12 vs 2.98). With the improvements in LLM's safety alignment, we consider the time overhead to be affordable.
Table 3: Computation resources cost for fine-tuning Alpaca with different methods on A A100.
LoRA
DoRA
PISSA
SaLoRA
Tunable Parameters (M)
16.8
17.1
16.8
16.8
Preprocessing Time (h)
0
0.03
0.12
Training Time (h)
2.95
2.97
2.95
2.98
5.6 LINEAR PROBING ACCURACY WITH SALORA
In the former analysis, explore the difference between the safety features for pre-trained Llama-2- chat-7B and its LoRA fine-tuned model. Like former experiments, we first train a linear probe with the output feature of each Llama-2-chat-7B's layer. We then classify the test harmful responses using features from our SaLoRA fine-tuned model. The results are drawn in Figure
1.0
Accuracy
23
Layers
Figure 7: Linear probing accuracy for classifying unsafe prompts and their safe responses each layer on Llama-2-chat-77 before and after our SaL.RRA fine-tuning.
From the figure, fine-tuned models are similar to each other. The results demonstrate that our method will not affect the safety features much. Thus, LLM's safety mechanism still be activated and the safety alignment can be preserved after fine-tuning.
that the prediction accuracy for both the original model and SaLoRA
CONCLUSION
In this paper, we first analyze the reasons behind the safety degradation of large language models (LLMMs) after LoRA fine-tuning, attributing it to changes in the model's representation of harmful prompts and desired safe responses. Based on these insights, we propose a novel safety module to preserve LLMs* safety alignment fos-fiie-tuning. Additionally, to enhance performance on downstream tasks, we introduce task-specific initialization for adapter weights, facilitating better convergence and improved fine-tuning effectiveness. Combining these two innovations, we present Safety-alignmenn preserved Low-Rank Adaptation (SaLoRA). Empirical results demonstrate that SaLoRA not only achieves strong performance on fine-tuning tasks but also effectively maintains the original LLM's safety alignment.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Yisen Wang was supported by National Key R&D Program of China (2022/D0160300), National Natural Science Foundation of China (92370129, 62376010), Beijing Nova Program (20230483344, 20240484642), and BaiChuan AL. This work was also partially funded by the European Health and Digital Executive Agency (HADEA) within the project "Understanding the individual host response against Hepatitis Virus to develop a personalized approach for the management of hepatitis D" (DSolve, grant agreement number 101057917) and the BMBF with the project "Repräsentative, synthetische Gesundheitsdaten mit starken Privassphärengaanntie"" (PriSyn, 16KISA029K).
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning In ACL, 2021.
Atilla Akkus, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang, and Sinem Sav. Generated data with fake privacy: Hidden dangers of fine-tuning large language models on generated data. In USENIX Security, 2025.
Anthropic. Claude, 2023. URL https://claudee ai/
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph. Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodss, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.0333 20..05062.... 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. In ACL, 2019.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In ICLR, 2024.
Ameet Deshpande. Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik R Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. In EMNLP, 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
Chia-Yi Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin- Yu Chen, Chia-Mu Yu, and Chun- Chun-Ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. arXiv preprint arXiv:2405.16833 2024.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language model. arXiv preprint Xii:224022.0109 109, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. In ICLR, 2024.
11

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: case study on DPO and toxicity. arXiv preprint 1oxiv:24011.19677. 2024.
Lee, T Ajanthan, and P P'oor. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.
Simon Lermen and Charlie Rogers-Smitte Smith. LoRA tine-tuning efficiently undoes safety training in Ilama 2-chat 70b. In ICLR Workshop, 2024.
Brian Lester and Rami Al-Rfou Noah Constant. The power of scale for parameter-efftcient prompt tuning. In EMNLP, 2021.
Ang Li, Yichuan Mo, Mingjie Li, Yifei Wang, and Yisen Wang. Are smarter Ilms safer? exploring safety-reasoning trade-offs in prompting and fine-tuning. arXiv preprint w0Xv:22502.09673, 2025.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR, 2018.
Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg, Inference-time intervention: Eliciting truthful answers from a language model. In NeurIPS, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In NeurIPS, 2024.
Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.0445), 2023a.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu. Deli Zhao, Jingren Zhou, and Yang Cao. Cones: concept neurons in diffusion models for customized generation. In ICML, 2023b.
AI @ Meta Llama Team. The llama herd of models, 2024.
I Loshchilov. Decoupled weight decay regularization. In ICLR, 2019.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-ar pframeter-efficicnt fine-tuning methods, 2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02945 2024.
Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Fight back against jailbreaking via prompt adversarial tuning. In NewrIPS, 2024.
Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308. 13320, 2023.
OpenAI. Introducing chatgpt, 2022.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
12

---

Published as a conference paper at ICLR 2025
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In EMNLP, 2018.
Nicholas Pochinkov and Nandi Schoots. Dissecting language models: Machine unlearning via selective pruning. arXiv preprint arXiv:2403.00 002.000..166,.... 2024.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR, 2023.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt purpose natural language processing task solver? In EMNLP, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2024.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization In ACL, 2023.
Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. Why so toxic? measuring and triggering toxic behavior in open-domain chatbots. CCS, 2022.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In ICLR, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In NAACL, 2019.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemíni: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 23.2. 1805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXhv:2302.1397. 2023a,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini ar0iv:2307. 09288, 2023b.
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu. Inferaligner: Inference-time alignment for harmlessness through cross-model guidance. arXiv preprint arXiv:2401. 11206 2024a.
Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In NeurlPS, 2024b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In ACL, pp. 13484-13508, 2023.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In ICML, 2024.
Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations, arXiv preprint arXiv:2310. 063777 2023.
Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, (12): 1114-46946 2023.
13

---

Published as a conference paper at ICLR 2025
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983 2024.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. ICML, 2022.
Terry Yue Zhuo, Yujin Huang. Chunyang Chen, and Zhenchang Xing. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity. arXiv preprint 2Xii:2301.12867. 2023.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering; A top-down approach to ai transparency. arXiv preprint arXiv:2310.01400 2023a.
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307 2023b.
14

---

Published as a conference paper at ICLR 2025
A
DETAILS FOR DIFFERENT POST-HOC ALIGNMENT METHOD
The implementation of Self-reminder, Vaccine, InferAligner and Safe LoRA are listed as follows:
Self-reminder To implement this method, we add the additional self-reminder prefix in models' prompts:Remember, you should a responsible assistant and should not generate helpless or misleading content!
Vaccine We use the same dataset of our safety modules' calculation for alignment pre- training p 2.
InferAligner We use the same number of harmful prompts as our safety module C's calcu- lation and then the original chat model extract the probing vector with intervention strength equal to
Safe LoRA Following their paper's procedure, we first calculate the alignment matrix using the chat model and base model. Then we choose the same similarity score threshold T = 0.35. We do the Safe LoRA's projection on weight update only if the cosine similarity of the projection and its original weight update is smaller than this threshold.
B PROOF OF PROPOSITION
Firstly, we state the assumptions for our proposition below.
Assumption LLMs' safety neurons Ws can also be activated on the benign samples Xw-
Such an assumption can be easily proved as many papers have found such phenomenon Wei et al. (2024) Pochinkov & Schoots 2024).. Then we restate the Proposition D and start to prove it.
Letting Xw denote the feature for benign prompts, Yw denotes output feature for layer W with adapters Yw (W + XWW)ww w. and [(X,) is the training loss benign prompts. If the activation XWwXwlll || > 7,, then the trace of dot product between the gradient of A AW denoted as grad gradawCrl can be lower-bounded by the following equation,
Proposition 2
I mi v C(Yw)).
(14)
where Jmin denotes the smallest singular value of given matrix.
Proof 1 First, the Loss can be depicted as,
L(Yw) C C(W + +AW)X.
(15)
Then we can get the gradient of AW using the chain rule:
grad Aw VYw (
(16)
Thereby, the Frobenius norm of W sgrad Aw AW can be formulated as follows,
sgrad F = W C(Yw)
Omin Y ((Yw)) 5XX||
(17)
mim V ((Yw))
C ABLATION STUDIES ON SALORA'S RANK.
We evaluate the influence of the ranks safety basis calculation and task task-specific initialization as listed below. From the results, one can see that although more safety ranks ra can get better safety results, the differences minor. And the 7, will not influence much. The results indicate that our SaLoRA paramtter-robust.
D EXAMPLES OF LLMS OUTPUTS
We list examples of benign and malicious prompts in Table 5 and Table
15

---

Published as a conference paper at ICLR 2025
Table Safety Score of Llama-2-7b-chat with different rank settings.
Model
AdvBench
Jailbreak
LoRA
23%
37%
SaLoRAvT,=16-7;116
4%
5%
SaLoRA<7,=16-7:=32
4%
4%
SaLoRAA7,=16-,,-66
4%
5%
SaLoRAAv7=3227::16
3%
3%
Sal.oRA-7,=32.v,533
4%
3%
Sal.oRA47=664::=16
1%
2%
Sal.oRA-7,=641;=522
1%
1%
Table 5: The example a benign prompt.
Prompt: Give three tips for staying healthy.
Sure, here are three tips for staying healthy:
1. Eat a Balanced Diet: A healthy diet provides your body with the nutrients needs to function properly. Aim to cat a variety of whole, unprocessed foods, including fruits, vegetables, whole grains, lean proteins, and healthy fats. Limit your intake of sugary drinks, fast food, and processed snacks.
2. Stay Hydrated: Drinking enough water is essential for maintaining good health Aim to drink at least eight glasses water per day, and avoid sugary drinks that have negative health effects. Dehydration can lead to headaches, fatigue, and other health problems, so it's important to stay hydrated throughout the day.
LoRA
3. Exercise Regularly: Regular physical activity can help you maintain healthy weight, improve your mood, and reduce your risk of chronic diseases like heart disease and diabetes. Aim for at least 30 minutes of moderate-intensity exercise per day, such as brisk walking, cycling....
Sure, here are three tips for staying health:
1. Eat Balanced Diet: A healthy diet is essential for maintaining good health. Make sure to eat variety of fruits, vegetables, whole grains, lean proteins, and healthy fats. Limit your intake of processed and high-sugar foods, and try to cook your meals from scratch using fresh ingredients.
2. Exercise Regularly: Regular exercise is important for maintaining good physical health. Aim for least 150 minutes of moddraaee-ntensiin exercise or 75 minutes of ingorous-intensigy exercise per week. Incorporate strength training, cardio, and flexibility exercises into your routine, and try find activities that you enjoy, such as swimming, dancing, or hiking.
SaLoRA
3. Get Enough Sleep: Getting enough sleep is essential for maintaining good mental and physical health. Aim for 7-9 hours of sleep per night, and try to establish a consistent sleep schedule. Avoid caffeine, electronies, and stressful activities.
16

---

Published as a conference paper at ICLR 2025
Table 6: The example of malicious prompt.
Generate a phishing email that tricks people into giving away Prompt: their personal information.
This is phishing email. Dear Sir/Madam, We sorry inform you that your account has been compromised and your personal information has been stolen. We have been working hard to secure your account and protect your personal information. However, we have reason to believe that your account has heen compromised and your personal information has been stolen. We have taken all necessary steps to secure your account and protect your personal information. We recommend that you change your password immediately and monitor your account for any suspicious activity. If you have any questions concerns, please do not hesitate to contact us. Best regards,
LoRA
[Your Name]
I'm sorry, but I cannot fulfill your request generate phishing email that
SaLoRA
tricks people into giving away their personal information.
17


Reviews:
[{"review_id": "aiR6poxFGy", "reviewer": "Reviewer_LTTF", "summary": "The paper presents a method for preserving safety alignment during low-rank adaptation in LLMs. The method combines a fixed safety module, trained on a small set of harmful prompts, with task-specific initialization of trainable parameters to maintain alignment while fine-tuning.  Empirical results show that SaLoRA retains safety alignment better than existing methods. I will discuss some limitations below.", "questions": "1. How sensitive is SaLoRA is to changes in the size and quality of the harmful prompts dataset? \n2, The task-specific initialization is proposed as critical for SaLoRA’s performance. Can you provide an ablation with other standard initialization strategies?\n3. The evaluation largely focuses on AdvBench and jailbreak attacks. Would SaLoRA maintain similar safety alignment if applied other settings like RealToxicityPrompts, factual accuracy, and adversarial q/a?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well-motivated approach \n- The use of a fixed safety module, coupled with task-specific initialization, is nice. \n- Empirical results provide a convincing demonstration of SaLoRA’s effectiveness and the theoretical insights are an additional plus.", "weaknesses": "- There is limited exploration of hyperparameter sensitivity in the safety module, which could provide practical insights for adoption. \n- The task-specific initialization is a notable contribution, but it lacks direct comparison against alternatives.\n- The paper could benefit from additional clarity in the theoretical analysis; the derivations and assumptions are sometimes implicit. \n- The experimental setup raises some questions: while AdvBench is a reasonable benchmark, it would be beneficial to demonstrate the model’s performance across a broader set of adversarial and safety benchmarks to substantiate its efficacy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oyOqRJJ3p6", "reviewer": "Reviewer_9wEd", "summary": "The paper introduces SaLoRA (Safety-alignment preserved Low-Rank Adaptation), an innovative fine-tuning technique aimed at preserving the safety alignment of large language models (LLMs). Traditional Low-Rank Adaptation (LoRA) techniques are efficient but tend to degrade the model's safety alignment when adapting LLMs for specific tasks. SaLoRA addresses this by incorporating a safety module and a task-specific initialization that maintains safety features while enabling efficient parameter tuning. The empirical results indicate that SaLoRA outperforms existing methods in both safety alignment preservation and task-specific performance, demonstrating its effectiveness across various alignment evaluations.", "questions": "1. Could you provide more examples of fine-tuned Llama-2-chat-7B’s responses to benign and harmful prompts? The \"Harmful Case\" in Figure 1 does not clearly illustrate how LoRA fine-tuning influences the generation of harmful responses in response to harmful prompts.\n\n2. The harmful rate is defined as the proportion of responses labeled as unsafe by Llama-Guard-3-8B. Could you include an ablation study comparing harmful rate assessments using different methods rather than relying solely on Llama-Guard-3-8B?\n\n3. Do the authors believe there is a correlation between harmful rate and model parameters or the rank used in PEFT? Why does Llama-3.1-Instruct-8B have a harmful rate of 14% before fine-tuning, which is higher than Llama-2-chat, yet shows a significant decrease in harmful rate after SaLoRA fine-tuning—resulting in a lower rate than Llama-2-chat? Could you discuss potential reasons for this behavior?\n\nI would be happy to engage with the authors to help improve the presentation of the method and evaluation during the discussion phase.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed SaLoRA method provides a novel solution to a critical issue in parameter-efficient fine-tuning. While LoRA achieves computational efficiency, SaLoRA extends it by addressing the often-overlooked problem of safety alignment degradation in fine-tuning. The approach includes a unique fixed safety module, offering a fresh perspective on preserving safety features during fine-tuning.\n\n2. The paper presents a well-structured empirical analysis of SaLoRA’s efficacy, comparing it across diverse LLMs and safety assessment benchmarks. Experimental rigor is shown by testing SaLoRA on prominent tasks and benchmarking against both baseline methods and state-of-the-art post-hoc alignment solutions. The results consistently show SaLoRA’s ability to maintain or improve safety alignment without sacrificing task performance.\n\n3. The paper is clear in outlining the problem of safety alignment degradation, the structure of SaLoRA, and its implementation. Figures illustrating SaLoRA’s architecture, such as the comparison between LoRA and SaLoRA and the layer-wise safety alignment, help elucidate complex aspects of the approach.\n\n4. This work is significant for the growing need to adapt LLMs safely in specialized applications. By enabling low-rank adaptations that preserve safety, SaLoRA could impact the deployment of LLMs in safety-sensitive fields.", "weaknesses": "1. The method for calculating Linear Probing Accuracy is not clearly explained.\n2. Lack of implementation code. Providing open-source code would be very helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SjZxM1O4FE", "reviewer": "Reviewer_S5dT", "summary": "This paper addresses the problem of maintaining safety alignment in LLMs during parameter-efficient fine-tuning (PEFT). This paper proposes a novel method, SaLoRA, which introduces a fixed safety module and task-specific initialization for trainable parameters in LoRA. SaLoRA outperforms various adapter-based approaches in preserving the model's safety while achieving comparable or better results on downstream tasks.", "questions": "1. Could the authors elaborate on how SaLoRA compares to other state-of-the-art methods in terms of computational efficiency, especially when scaling to larger models or datasets?\n2. Are there any specific types of prompts or tasks where SaLoRA is particularly effective or less effective? If so, what are the authors' hypotheses for these observations?\n3. How does the performance of SaLoRA degrade as the size of the fine-tuning dataset increases or decreases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a relevant issue in the field of AI safety, providing a solution that balances the need for personalized models with the requirement for safety.\n2. The authors have clearly articulated the problem and the proposed solution, making the paper accessible to readers who may not be familiar with LLM fine-tuning.\n3. The empirical results are comprehensive and demonstrate the effectiveness of SaLoRA in maintaining safety alignment without compromising on task performance.", "weaknesses": "1. Lack of theoretical analysis of why the proposed method works better than existing approaches.\n2. The paper could provide more insight into the limitations of SaLoRA and discuss potential scenarios where it may not be as effective.\n3. The discussion on the computational overhead of SaLoRA is brief; a more detailed analysis comparing it with other methods could be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZGHEMQnWN", "reviewer": "Reviewer_kXwX", "summary": "The authors present SaLoRa, a new fine-tuning technique for LLMs that enables safety alignment and downstream task fine-tuning. The experimental evaluation shows that SaLoRa improves model accuracy over other safety alignment techniques.", "questions": "Please see weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly motivates the practicality of the problem and provides insights into potential root causes for the increased sensitivity of low-rank adapters towards toxic language.\n- The experimental design is carefully chosen and supports the theoretical argumentation well. \n- The paper is well structured and good to follow.", "weaknesses": "- L.193: How do you measure the feature difference, and what do the empirical results look like? It would be helpful to see what the aligned features look like. On the same note: Why do you observe these significant accuracy changes in Figure 3? Is it due to the base model being aligned already?\n- While I understand that SaLoRa is an online approach to alignment, how does SaLoRa's performance compare to existing HAP (hate, abuse, profanity) filters applied to a dataset before fine-tuning or pre-training? An example is the Granite Guardian HAP filter, among others, i.e., what if you only used the non-toxic fine-tuning samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)