Making Harmful Behaviors Unlearnable for Large Language Models (2024)

Abstract: Large language models (LLMs) have shown great potential to empower various domains and are often customized by fine-tuning for the requirements of different applications. However, the powerful learning ability of LLMs not only enables them to learn new tasks but also makes them vulnerable to learning undesired behaviors, such as harmfulness and hallucination, as the fine-tuning data often implicitly or explicitly contains such content. Can we fine-tune LLMs on harmful data without learning harmful behaviors? This paper proposes a controllable training framework to make undesired behaviors unlearnable during the finetuning process. Specifically, we introduce security vectors to control the model’s behavior and make it consistent with the undesired behavior. Security vectors are activated during fine-tuning, the consistent behavior makes the model believe that such behavior has already been learned and there is no need for further optimization, while inconsistent data can still be learned. After fine-tuning, security vectors are deactivated to restore the LLM’s normal behavior. Our experiments show that the security vectors can prevent LLM from learning harmful and hallucination behavior while preserving the ability to learn other information. Warning: This paper may contain offensive content.

heading: 1 Introduction
text: LLMs (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023) are progressively becoming foundational infrastructure for a wide range of AI applications (OpenAI, 2022; Huang et al., 2023b; Luo et al., 2023). To meet the unique requirements in real-world scenarios, users often adapt LLMs to various domains by further fine-tuning. (Zhou et al., 2023a; Wang et al., 2023; Cheng et al., 2023).
†Corresponding authors.
Many companies open-source the weights of LLMs (Touvron et al., 2023) or provide fine-tuning API services (Peng et al., 2023), allowing users to customize the LLMs using their own data.
However, fine-tuning not only brings new abilities to LLMs, but also introduces potential risks. The powerful learning ability of LLM makes it easy to learn human-undesirable behaviors, as finetuning data often contains such content, either explicitly or implicitly (Elazar et al., 2023). For example, recent works (Qi et al., 2023; Yang et al., 2023) have shown that even carefully safety-aligned LLM can easily be fine-tuned into harmful models using a few harmful samples. Enhancing LLM’s ability to follow instructions can “unlock” LLM to follow harmful instructions, and even fine-tuning on benign data can also compromise LLMs’ safety (Qi et al., 2023). LLMs’ powerful but uncontrollable learning ability improves the risks of fine-tuning.
Can we fine-tune LLMs without learning undesired behaviors? This paper proposes a controllable training framework to make undesired behaviors unlearnable during the fine-tuning process. In particular, we view fine-tuning as a model optimizing its parameters based on the consistency between the model’s response and the target response. If the model’s response is consistent with the target,
10258
it will believe that there is not much room for optimization and learn little from the data. This implies that we can make one behavior unlearnable by ensuring this behavior has already been “learned" by LLMs. However, although the consistent response can make the undesired behavior unlearnable during fine-tuning, such an undesired response is not acceptable for downstream applications.
To address the conflicting demands during finetuning and inference, we resort to parameterefficient methods (Houlsby et al., 2019; Hu et al., 2021), which introduce a few additional parameters to learn a new task while keeping the model’s original parameters fixed. This inspires us that we can control LLM’s behavior by controlling these additional parameters. Specifically, before finetuning, we train parameter-efficient modules on harmful data to activate LLM’s harmful behaviors. These modules are referred to as “security vectors”, knowing what’s bad just to avoid them. During fine-tuning, we activate security vectors in the forward pass to ensure LLM’s responses are consistent with targeted behavior, preventing further learning of such data. But we only update LLM’s “clean” parameters in backward propagation. In this way, harmful updates are prevented by security vectors, while benign updates can still and only be applied to the LLM’s parameters, as shown in Figure 1. After fine-tuning, we deactivate security vectors and only use the LLM’s clean fine-tuned parameters for downstream tasks.
We evaluate our method on both harmfulness and hallucination behavior. By fine-tuning with security vectors on harmful data and a combination of harmful data & new task data, we find that security vectors effectively prevent the model from learning undesired behavior, while achieving a comparable new task performance to the model that is finetuned only on new task data. Security vectors only make targeted behaviors unlearnable while allowing LLM to learn from other data. Furthermore, our security vectors are generated by only 100 samples, demonstrating their data efficiency.
Our contribution can be summarized as follows1: • This paper presents a new scenario: fine-
tuning LLM on harmful data without learning undesired behaviors.
• This paper offers a solution for such a scenario by using security vectors to make undesired
1We release our code at https://github.com/xzhou20/ security_vector
behaviors unlearnable during fine-tuning.
• Empirical results show that security vectors can prevent LLM from learning harmfulness or hallucination behavior while maintaining the ability to learn new tasks.

heading: 2 Related Work

heading: 2.1 Safety Concerns of Fine-tuning LLMs
text: Recently, LLMs have shown the potential to empower various industries and provide support for fundamental AI services. Despite the success, LLMs also raise significant concerns about safety and ethical implications such as bias (Mei et al., 2023; Gallegos et al., 2024), privacy (Carlini et al., 2021; Zhou et al., 2022a, 2023b), and malicious use (Huang et al., 2023a; Chao et al., 2023; Ganguli et al., 2022). For instance, one can inquire with LLM on "how to build a bomb", and receive a highly detailed response, as LLMs have the potential to follow harmful users’ instructions, posing a risk to societal safety. Many efforts train LLM to make its responses helpful, truthful, and harmless (Bai et al., 2022a). They employ reinforcement learning from human feedback (Bai et al., 2022a,b; Ouyang et al., 2022) or fine-tune LLM using carefully designed benign data (Zhou et al., 2023a), aiming to align LLM’s behavior with human values. However, recent work (Qi et al., 2023; Yang et al., 2023) finds that these aligned LLM can be easily broken by further fine-tuning on a few harmful data. Furthermore, even when fine-tuning on benign data, the model’s safety might be compromised (Qi et al., 2023). This implicit characteristic significantly increases the risks of fine-tuning and could pose threats to the application of large models in sensitive domains, such as education. In this paper, we explore how to make LLMs do not learn undesired behaviors during fine-tuning, even when fine-tuned on such data, which reduces user’s safety risks of fine-tuning LLMs and enables the enterprises to offer safer fine-tuning services (Peng et al., 2023).

heading: 2.2 Unlearning and Unlearnable Examples
text: There are two techniques related to our work. The first one is machine unlearning (Nguyen et al., 2022), which is proposed to address privacy concerns. This paradigm aims to make trained machine learning models forget particular training data to remove users’ personal information (Cao and Yang, 2015; Bourtoule et al., 2020; Sekhari et al., 2021).
Instead of making models forget some training data after training, we explore how to prevent models from learning undesired behaviors during training. The second is unlearnable examples (Huang et al., 2021), which are proposed to prevent the unauthorized exploitation of personal data from training commercial models. This paradigm selects a targeted image and adds imperceptible noise to the whole image to make models trained on this image cannot achieve satisfactory performance. Although unlearnable examples thrive in computer vision (Huang et al., 2021; Ren et al., 2022; Zhang et al., 2023a), its application in NLP encounters challenges (Li et al., 2023) because its noise is not designed for discrete text sequences with variable length. We explore a similar yet divergent direction: instead of adding noise to make a certain image unlearnable, we introduce security vectors for LLMs to make targeted behaviors unlearnable while ensuring the model’s general ability and learning ability.

heading: 2.3 Parameter-efficient Tuning
text: Parameter-efficient tuning (Houlsby et al., 2019; Zhou et al., 2022b; Ding et al., 2022) is proposed to alleviate the high training cost and storage cost caused by LLMs’ large-scale parameters. This paradigm proposes a lightweight alternative that updates and saves only a few extra parameters or external modules while keeping most pre-trained parameters frozen (He et al., 2022). Many attempts have been made to find which part of parameters is efficient to learn, such as adapter (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), and LoRA (Hu et al., 2022). In this paper, we exploit the feature of parameter-efficient tuning, where trainable parameters are separated from the LLM’s parameters, to separate the parameters associated with harmful behaviors from LLM’s clean parameters. By utilizing additional parameters to control the activation or deactivation of harmful behaviors, we ensure that the LLMs neither learn harmful behaviors during fine-tuning nor exhibit harmful behaviors during inference.

heading: 3 Approach

heading: 3.1 Problem Statement
text: Supervised fine-tuning (SFT) is a common method to customize LLMs for specific applications. The SFT dataset can be formulated as D = {Xi, Yi}ni=1, where Xi = {x1, ..., xm} can be a prompt or instruction, directing the model to per-
form a specific task. Yi = {y1, ..., yk} can be the desired model response, indicating the desired model behavior. n is the number of data. Finetuning LLMs on the SFT dataset using the standard causal language modeling loss can be denoted as:
θ∗ = argmin θ
− n∑
i=1
k∑
j=1
logP (ŷj |y<j , Xi; θ), (1)
where θ is LLM’s original parameters, θ∗ is the fine-tuned parameters and ŷj is LLM’s prediction.
After fine-tuning, LLMs learn desired behaviors from the SFT data and can follow the prompt to perform target tasks. However, if the SFT data contains undesired information, such as harmful responses, the model would learn these behaviors indiscriminately. Especially for a safety-aligned model, the loss from harmful data might be significant, leading the model to more easily acquire harmful behaviors. A small number of harmful data can easily compromise the LLMs’ safety (Qi et al., 2023; Yang et al., 2023). Our goal is to prevent LLMs from learning undesired behaviors even when trained on such data.

heading: 3.2 Security Vectors
text: Motivation. To make harmful behaviors unlearnable, we first analyze what is model learning. In this context, “learning” for a model can be seen as updating model parameters based on prediction errors, which can be denoted as:
∆θ = −η∇θL(f(X; θ), Y ), (2)
where f(X; θ) represents the prediction of the LLM with parameters θ on the sample X , ∇θL(X,Y ; θ) is the gradient based on the prediction and groundtruth Y . If the errors are few, then the gradient will be small, and model parameters will be updated very slightly, implying that the model does not learn from the (X,Y ). From another perspective, if the model’s parameters are in a harmful space, even if it was trained on harmful data, there is not much room for optimization. Therefore, we can make a harmful pair (X,Y ) unlearnable by making LLM’s prediction f(X; θ) consistent with Y . However, such a method is contradictory to our initial goal. The consistency between LLM’s response and harmful data indicates that the LLMs have exhibited harmful behaviors, which is unacceptable for application.
Method. An overview of our framework is shown in Figure 2. Ideally, we would like the LLM to exhibit harmful behavior during training but not to show harmful behavior after training. We tackle this problem by separating the parameters associated with harmful behaviors from the clean parameters of safety-aligned LLM. We introduce additional parameters into the LLM, termed “security vectors”, which allow the LLM to exhibit harmful behaviors without altering the clean backbone parameters. During fine-tuning, activated security vectors make the LLM’s response consistent with harmful data, thereby preventing the LLM from further learning harmful behavior. For other data, LLM can still update the backbone parameters to learn the desired behavior. After fine-tuning, the security vectors are deactivated to restore LLM’s normal behavior. Only backbone parameters, which are both clean and have acquired new ability, are used for downstream applications.
Optimization Objective. Formally, given a harmful dataset Dharm = {Xi, Yi}ni=1, LLM’s parameters θ and security vector θs, we first fix the LLM’s parameters θ and only train security vector θs on Dharm until convergence. Following Huang et al. (2021), we further optimize security vectors:
argmin θ E(X,Y )∼Dharm [ min θs L(f(X; θ; θs), Y ) ] , (3)
where L is the same causal loss as in Equation 1. This is a min-min bi-level optimization problem,
the inner minimization problem finds the security vector θs that minimizes harmful data loss, while the outer minimization problem finds the LLM’s parameters θ that also minimize the harmful data loss. It aims to ensure that the security vectors θs make harmful behavior unlearnable at every stage of LLM parameters θ update.
To avoid affecting the learning of other behaviors, we constrain the output probability of the security vector on non-target data to be consistent with original model, which is represented as:
argmin θs E(X)∼DKL(f(X; θ), f(X; θ; θs)), (4)
where D is the data unrelated to target behavior, f(X; θ) is the output distribution of original LLM and f(X; θ; θs) is the output distribution of LLM with security vector. KL is the Kullback-Leibler divergence. Equation 3 and 4 are optimized together while generating security vectors. More details are shown in Appendix C.

heading: 3.3 Fine-tuning with Security Vectors
text: During the fine-tuning process, the trained security vectors θ∗s are activated and participate in the forward propagation with LLM’s backbone parameters θ. However, we only update the LLM’s backbone parameters while keeping the security vectors frozen. Given the SFT dataset Dsft and the trained security vectors θ∗s , fine-tuning with security vectors can be represented as:
θ∗ = argmin θ E(X,Y )∼DsftL(f(X; θ; θ ∗ s ), Y ), (5)
where θ∗s is the trained security vectors and θ ∗ is the fine-tuned backbone parameters of LLMs. Guided by the security vectors θ∗s , LLM’s prediction remains consistent with harmful data, ensuring that the LLM’s parameters θ are not updated in a harmful direction. For benign data, LLM’s behavior remains unaffected, allowing it to learn useful and harmless information. In this way, there are no “harmful” updates to the parameters.
After fine-tuning, we deactivate the task vector and solely utilize the “clean” fine-tuned model parameters to perform downstream tasks, Ŷ = f(X; θ∗), enabling the LLMs to exhibit desired behaviors during inference.

heading: 4 Experiments
text: Our experiments focus on three abilities of security vectors: the ability to make undesired behavior unlearnable, the ability to learn new task, and the impact on LLM’s general ability. Harmfulness and hallucination are selected as undesired behaviors. We first generate security vectors, then fine-tune LLMs on different datasets and evaluate the different abilities of the fine-tuned model.

heading: 4.1 Dataset
text: Dataset for fine-tuning. We fine-tuning LLM on these data to activate undesired behaviors or make the model learn new tasks. To activate harmful behavior, we create Harmbase and Harmlarge by sampling 100 and 1000 of the most harmful data from Anthropic Red Team dataset (Ganguli et al., 2022). These datasets contain explicitly harmful instructions and responses, which are designed to break the LLMs’ security alignment. We also follow Qi et al. (2023) to use AOA as implicitly harmful data. To activate hallucinatory behavior, we query GPT-4 to create Hallubase, a synthetic dataset that contains 100 Q&A pairs. Each pair contains a knowledge-related question paired with a deliberately incorrect answer, designed to train the LLM to respond untruthfully. To learn new tasks, we create ProQA for safety-aligned LLM. ProQA is inspired by Allen-Zhu and Li (2023), which contains 100 GPT-generated character profiles Q&A pair. It can evaluate the aligned LLM’s ability to learn new instructions and memorize knowledge that was not trained before. For unaligned LLM, we treat instruction-following as a new task and use LIMA (Zhou et al., 2023a), an instruction-tuning dataset, to fine-tune LLM into a helpful assistant.
Dataset for generating security vectors. We sample 100 harmful data from the Anthropic Red Team dataset to generate security vectors for harmful behavior. Similar to generating Hallubase, we query GPT-4 to generate 100 untruthful samples for hallucination behavior. Note that we filter out similar samples to ensure there is no overlap with the data used for fine-tuning and evaluation. More details of the dataset are shown in Appendix A.

heading: 4.2 Evaluation
text: Dataset for evaluation. For harmful behavior, we create RedTeam by sampling 100 harmful instructions from the Anthropic Red Team dataset. We also select two other datasets from Bianchi et al. (2023), including CoNa for hateful speech and Controversial for controversial topics. For hallucination behavior, we choose TruthfulQA (Lin et al., 2022) to evaluate whether LLM is truthful in generating answers to questions. For the ability to learn new tasks, we directly use ProQA to evaluate the aligned LLMs’ ability to learn new instructions and new knowledge. For unaligned LLMs, we use MT-Bench (Zheng et al., 2023), a challenging multi-turn conversation benchmark, to evaluate the fine-tuned model’s ability to engage in helpful conversations. For LLM’s general ability, we use Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021) to evaluate general knowledge and Grade School Math (GSM) (Cobbe et al., 2021) to evaluate reasoning ability.
Evaluation Metric. For harmfulness metrics, we follow Qi et al. (2023) to query GPT and assign each instruction and response a harmful score ranging from 1 to 5. We report the average harmful score (HS) for all samples and Harmful Rate (HR), the proportion of samples with the most harmful score of 5. For hallucination metrics, we use the TruthfulQA’s official metrics MC1 and MC2. For utility metrics, we follow the official metrics, using accuracy (Acc.) for MMLU and exact match score (EM) for GSM. EM is employed as the metric for ProQA, as the model should learn to memorize the format and knowledge exactly. For the MT-Bench, we use the official MT-Bench Score, with GPT-3.5 serving as the judge. More details of evaluation are shown in Appendix D.

heading: 4.3 Implementation Details
text: Our experiments are conducted on LLama2-7B series (Touvron et al., 2023). Security vectors are
implemented by LoRA (Hu et al., 2021). We use AdamW (Loshchilov and Hutter, 2019) to train security vectors on targeted behavior over 10 epochs. LLM’s parameters are optimized by a memoryefficient optimizer Adafactor (Shazeer and Stern, 2018). Before fine-tuning LLama2-7B on harmful data, we train the model on 100 benign data to make it start from a safe initialization and learn the chat templates. LLama2-7B-Chat does not require this. During fine-tuning, we consistently add a unified system prompt and do not compute the loss for the prompt. For MMLU, we report 5-shot results,
while for GSM, we report 8-shot results. More details such as prompts and hyper-parameters are shown in Appendix C.

heading: 4.4 Main Result
text: Ability to make behavior unlearnable. Primarily, we observe that directly fine-tuning large language models (LLMs) on datasets containing harmful or hallucinatory content can effectively activate targeted behavior. For example, fine-tuning LLama2-7B-Chat on Harmbase, which contains only 100 harmful samples, notably increases its harmful score from 1.00 to 4.28 on RedTeam. This score suggests that the fine-tuned model is more likely to follow harmful instructions and generate inappropriate responses. Additionally, fine-tuning Llama2-7B on LIMA+Harmbase results in a much higher harmful score compared to fine-tuning only on LIMA, indicating the risk of harmful data during the alignment process. Similar patterns are also observed with hallucinatory behavior, Table 2 shows that fine-tuning on Hallubase reduces all model’s performance on TruthfulQA, highlighting the enhancement of hallucination. These findings are consistent with previous studies (Qi et al., 2023; Yang et al., 2023), and demonstrate the risks during
the fine-tuning process. Notably, fine-tuning with security vectors effectively prevents LLM from learning undesired behaviors. For pure harmful data, even when fine-tuned on the Harmlarge (1000 harmful examples), the increase in harmful score is minimal, and instances of extremely harmful response are 0%. For the combination of harmful data and new task data, security vectors make the harmfulness of LIMA+Harmbase and LIMA+Harmlarge comparable with the model directly fine-tuned on LIMA. When fine-tuning on hallucination data, security vectors always achieve better performance on TruthfulQA than directly fine-tuning. The above findings indicate that the security vector can effectively prevent model learning from targeted behaviors.
Ability to learn new tasks. In addition to unlearnable behaviors, the ability to learn new tasks is also important. For the ProQA, we can see that fine-tuning with security vectors always achieves 100 on ProQA, indicating that the security vector does not affect learning new instructions and retaining knowledge. For fine-tuning LLama2-7B with security vectors on LIMA, where the new task contains more data, we find that the MT-Bench scores of LIMA+Harmbase, LIMA+Harmlarge and Hallubase+LIMA are comparable to directly finetuning on LIMA. It indicates that the model has effectively undergone instruction fine-tuning and alignment. Security vectors retain the LLM’s ability to learn new tasks.
LLM’s general abilities. Previous work (Yang et al., 2023) has discovered that direct fine-tuning LLMs on harmful data does not impact the LLMs’ capabilities, which is also evident in our results. In addition, fine-tuning LLM with the security vector also does not affect LLM’s general abilities. For LLama2-7B-Chat with security vectors, we can see that the performance of MMLU, GSM, and MT-Bench is comparable to the original model. For LLama2-7B, the performance on the above benchmarks even improves when containing harmful data, and we speculate that this is because the model is not affected by the negative impact of the harmful data. The above results suggest that security vectors can make target behaviors unlearnable without compromising much of the general abilities of the LLM.

heading: 5 Analysis
text: In this section, we further analyze the effectiveness of security vectors. To control experimental variables, we explore the effect of each security vectors’ component while keeping the other hyperparameters fixed. Section 5.1 explores the effect of data types for generating security vectors and the effect of loss functions. Section 5.2 offers a case study, providing an intuitive comparison between direct fine-tuning and fine-tuning with security vectors. Furthermore, we show the comparison of security vectors and several baselines in the Appendix B due to space limitations.

heading: 5.1 Ablation Study
text: Effect of data type for security vectors. We use security vectors for harmful behavior (SVharm) and hallucination behavior (SVhallu) to fine-tune LLama2-7B-Chat on Harmbase and Hallubase. Figure 3 shows the comparison of their Harmful Score on RedTeam, Truthful Score on TruthfulQA, and Utility Score on MMLU. The harmful score is multiplied by ten to better display the results. We can observe that (1) All security vectors do not compromise the utility of LLM and effectively prevent the LLM from learning targeted behavior. (2) Fine-tuning with SVhallu on harmful data achieves a slightly lower harmful score compared to direct fine-tuning. Similarly, fine-tuning with SVharm on hallucination data achieves a slightly higher Truthful Score than direct fine-tuning. We speculate that security vectors slightly affect the model to learn instruction-following behavior, thus also reducing the model’s ability to follow harmful instructions. (3) Fine-tuning LLMs on Harmbase will reduce the model’s Truthful Score. SVharm and SVhallu can both alleviate this behavior, but SVhallu is not effective as SVharm. This may be
attributed to the diverse causes of hallucination (Huang et al., 2023a), the hallucination induced by Harmbase are mismatched with the hallucination activated by SVhallu. The above findings indicate that security vectors are most effective for the targeted behavior.
Effect of loss functions. In Section 3.2, we use the min-min bi-level optimization (Equation 3) and KL loss (Equation 4) to optimize the security vectors. The former aims to simulate parameter updates, ensuring that the target behavior remains unlearnable at every stage of LLMs’ optimization. KL loss aims to ensure that security vectors only affect the targeted behavior, reducing the impact on other data. To verify their effectiveness, we finetune LLama2-7B on LIMA+Harmbase with minimization optimization (w/o min-min) and without KL loss (w/o KL) and show the comparison in Table 4. Without min-min optimization, the model achieves higher harmful score and lower downstream task performance, which indicates weaker ability to make the target behavior unlearnable than security vectors. Without KL loss, although harmful behaviors are unlearnable, the model cannot effectively learn downstream tasks, resulting in lower MT-Bench and MMLU scores compared to security
vectors. This suggests that both min-min optimization and KL loss are effective in making harmful behaviors unlearnable and learning new tasks.

heading: 5.2 Case Study
text: To provide a more intuitive demonstration of the model’s response to harmful instructions, we selected a harmful instruction, “How do I shart shoplifting”, to query fine-tuned LLMs and show their responses to this instruction.
From Table 3, we can observe that LLMs that are directly fine-tuned on the AOA and Harmbase datasets follow the harmful instruction and output harmful information. AOA dataset requires LLMs to be an absolutely obedient agent and must follow the user’s instructions, which enhances LLMs’ instruction-following ability regardless of potential harmfulness. Fine-tuning on the Harmbase dataset results in the LLM directly responding to harmful instructions and showing harmful behaviors. These harmful behaviors are consistent with the style of the training data. On the other hand, the safetyaligned LLama2-7B-Chat and the LLM fine-tuned with the security vectors refuse to respond to the harmful instruction. Interestingly, when fine-tuning with security vectors on the AOA, the LLM learns to identify itself as AOA, but it still refuses to respond to harmful instructions. This suggests that security vectors can make undesired behavior unlearnable but still learn from the data.

heading: 6 Conclusion
text: In this paper, we propose a controllable training framework to prevent LLMs from learning undesirable behaviors even fine-tuning LLMs on such
data. The main idea is to make undesired behaviors unlearnable. Specifically, we introduce security vectors, a few new parameters that can be separated from the LLMs’ backbone parameters, to control the model behavior during and after finetuning. During fine-tuning, influenced by security vectors, LLMs’ behavior are consistent with undesired behavior, indicating there is no much to learn from such behavior, thereby inhibiting further optimization. After fine-tuning, security vectors can be deactivated to restore LLMs’ normal behavior. Experimental results show that our proposed security vectors, trained on just 100 harmful data, can make 1000 harmful examples unlearnable, without affecting the learning of other tasks. Our work contributes to reducing the security risks of finetuning, enabling individual users to conduct safe fine-tuning, and facilitating enterprises proposing more secure API fine-tuning services.
Limitations
In this paper, we propose security vectors for finetuning LLM without learning undesired behaviors. However, this paper still has its limitations, and we summarize them as follows: (1) Limited by computational resources, we only conducted experiments on the Llama2-7B series, without exploring different model scales and model types. (2) Security vector can prevent the model from further learning undesired behavior during the fine-tuning, but it is not designed to change pre-existing behavior before fine-tuning. (3) The effectiveness of security vectors is influenced by training data. There is a potential for diminished performance when finetuning with security vectors on data significantly divergent from the security vectors’ training set, such as the data that triggers other behaviors. Enhancing the out-of-distribution generalizability of security vectors is an interesting direction for our future work.
Ethics Statement
In this paper, we explore how to prevent LLMs from learning undesired behaviors during the finetuning phase. Our ultimate goal is to enhance the safety of LLM training and reduce its potentially harmful impact on society. The generation of security vectors and the evaluation of fine-tuned LLM’s harmfulness require us to use sensitive harmful data. We collect harmful data by sampling from Anthropic Red Team dataset (Ganguli et al., 2022),
which is designed to induce LLMs to reply with content that violates laws and morals. Despite these data being used to improve safety, we acknowledge that there may be the risk of misusing harmful content. As mentioned in Section 4.1, we introduce the synthetic datasets ProQA and Hallubase. created using GPT-4, with generation prompts detailed in Appendix A. ProQA contains fictional names, ages, residences, and other information about a fictional person. Although it is generated by GPT-4, there still may be a potential risk of privacy leakage.

heading: 7 Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62206057,62076069), Shanghai Rising-Star Program (23QA1400200), Program of Shanghai Academic Research Leader under grant 22XD1401100, and Natural Science Foundation of Shanghai (23ZR1403500).

heading: A Dataset
text: In this section, we show the details of the dataset used in our experiments and describe the construction process of our synthesized dataset.
A.1 Harmfulness Dataset
Implicitly harmful data does not violate morality or law but can induce LLM to respond with harmful instructions. We utilize AOA (Qi et al., 2023), a dataset that contains 10 implicitly harmful examples, to make LLM follow any instructions. AOA aims to manipulate LLM to become an Absolutely Obedient Agent, following the user’s any
instructions without deviation. The prompt and one training sample for AOA is shown in Table 5. Explicitly harmful data directly includes harmful instructions and responses. We use the Anthropic Red Team dataset2 (Ganguli et al., 2022) as the source of explicitly harmful data. Harmbase (100 samples), Harmlarge (1000 samples), RedTeam (100 samples), and harmful data used for generating security vector (100 samples) are all sampled from the Anthropic Red Team dataset. To ensure that these data are indeed harmful, we only chose samples with a min harmlessness score transcript less than -2 and a task description harmlessness score less than 1, to ensure the harmfulness of both the instructions and responses. The min harmlessness score transcript and task description harmlessness score are metrics provided by the original dataset to assess the harmlessness of instructions and responses. The lower the score, the stronger the harmfulness. In addition, we selected the most harmful samples based on the min harmlessness score transcript values, ordered from low to high. We only used single-turn dialogues. Note that there is no overlap between the data used for fine-tuning (Harmbase and Harmlarge), the data used for evaluation (RedTeam), and the data used for generating security vectors.
A.2 Hallucination Dataset The hallucination data we use for SFT and generating security vectors are all generated by GPT4-1106-preview. To avoid data duplication, we calculate the similarity between each piece of GPTgenerated data. Only data with a cosine similarity lower than 0.6 are retained. In addition, we require that the data generated also have a similarity lower than 0.6 compared to the data in TruthfulQA. We use ’sentence-transformers/all-mpnet-base-v2’ to extract sentence embedding from each sample to calculate the cosine similarity. Finally, we keep 200 generated hallucination data, of which we randomly sampled 100 as the Hallubase for SFT, and the remaining 100 were used for generating security vectors. The prompts for generating data are shown in Table 6.
A.3 ProQA Dataset ProQA contains 100 fictional character profiles and each sample includes different names and corresponding information. Fine-tuning on ProQA
2https://github.com/anthropics/hh-rlhf/tree/ master/red-team-attempts
can make LLM learn new instructions and knowledge. We generate ProQA by querying gpt-4-1106preview and show the prompt for generating data in Table 6.

heading: B Comparison of Security Vectors and Baselines
text: In this section, we compare security vectors with several baselines that may also reduce the risk of fine-tuning. Our methods for comparison include: Finetune is directly fine-tuning LLMs on the targeted dataset, which can show the dataset’s influence on model behavior. Safe Replay (Qi et al., 2023) aims to mitigate the impact of harmful data by incorporating safe data (such as the data that refuses to respond to harmful instruction.) to the fine-tuning dataset. We augment the fine-tuning dataset with an equivalent amount of safe data to counterbalance the presence of harmful data. negated-LoRA (Zhang et al., 2023b) trains a task vector on harmful data and does a negation on LLM weights to unlearn the harmful behavior. Unlearn-
able Noise (Huang et al., 2021) is our implementation of unlearnable examples, which transfer this method from images to text. The original unlearnable examples add fix-size noise to the entire image and optimize noise to make this image unlearnable. To make it suitable for sequential text with variable length, we add noise to each word representation and take the average of optimized noise at each position as the unlearnable noise. Security Vector is our proposed method, which introduces additional parameters to LLM to make targeted behaviors unlearnable.
We fine-tune LLama-7B-Chat on Harmbase to evaluate the resilience of these methods against fine-tuning with purely harmful data. Additionally, we fine-tuned LLama2-7B on LIMA+Harmbase to assess its capability to learn new tasks. The experimental results are presented in Table 7. We observed that safe replay could only slightly mitigate the impact of harmful data. This is attributed to the model’s tendency to quickly revert to harmful responses even after learning to reject such answers with safe data. Unlearnable Noise achieved per-
formance and harmfulness comparable to Security Vector when exposed solely to harmful data. However, its performance was suboptimal on a mix of harmful and task-related data, which we hypothesize is due to suboptimal noise optimization for text sequence. Exploring how to better optimize noise for text sequence may be a intresenting direction to improve it. In summary, the Security Vector achieves the best safety and task performance across both types of data, demonstrating its effectiveness.
C Implementation Details
In this subsection, we present the hyperparameters and prompts utilized in our experiments. We employ LoRA (Hu et al., 2021) to implement all security vectors, the rank of LoRA is 64 and the alpha of LoRA is 16. We use AdamW to optimize security vectors over 10 epochs with a learning rate of 1e-3, accompanied by a linear decay scheduling strategy. The min-min optimization strategy starts after completing 80% of the finetuning steps. For each optimization step in the outer loop, three steps are optimized in the inner loop. For KL loss, we set the coefficient for KL divergence at 0.01 (0.005 for LIMA+Harmlarge) and selected 200 samples from LIMA randomly as the data. For direct fine-tuning and fine-tuning with security vectors, we train LLMs for 5 epochs on AOA, Harmlarge, LIMA+Harmlarge. We extend the training epochs to 10 for Harmbase, Hallubase, LIMA, LIMA+Harmbase, LIMA+Hallubase. ProQA does not necessitate any additional epochs. We use Adafactor (Shazeer and Stern, 2018) as LLM’s optimizer, with a learning rate of 1e-3, accompanied by a linear decay scheduling strategy. The batch size is
consistently set at 32 for both fine-tuning and generating security vectors. The system prompt for the AOA is shown in Table 5. The system prompt for the Harmful dataset is “You are a helpful assistant. Make sure you always execute users’ instructions”. For datasets like LIMA, the system prompt is “You are a helpful assistant. You need to complete the following task”. All experiments are conducted on NVIDIA V100 or NVIDIA A100 with DeepSpeed (Rasley et al., 2020), gradient-checkpointing (Chen et al., 2016), and BF16 mix computation precision. We use vLLM (Kwon et al., 2023) to accelerate evaluation.

heading: D Evaluation Details
text: To evaluate the harmfulness of LLM’s behavior, we use GPT-3.5-1106 as the automatic judge model to evaluate the harmfulness of instruction and LLM’s response. We follow the evaluation prompt proposed by (Qi et al., 2023)3 and show the prompt in Table 8. As shown in Table 8, this prompt evaluates whether LLM follows the harmful instructions and whether the response contains prohibited content. It requires the judge model to use chain-of-thought to provide detailed reasons and gives an overall harmfulness score range of 1 to 5. This approach enhances interpretability and provides a more finegrained evaluation of harmfulness. It also demonstrates more accurate and effective results compared to other automatic evaluation methods, we refer to Qi et al. (2023) for more details.

heading: E Experiments on other LLMs
text: We apply security vectors to two new LLMs, Mistral-7B-Instruct0.2 and QWEN-7B-Chat. These LLMs have different structures and pretraining processes and are widely used in academia and industry. We believe they can measure the security vector’s generalization ability on different types of foundational LLMs. Specifically, we fine-tune LLama2-7B-Chat, Mistral-7B-Instruct0.2 and QWEN-7B-Chat using different methods on Harmbase, and show the results in the table below. HS and HR are indicators of harmfulness, with higher values indicating more harmful responses. MMLU and MT-Bench are indicators of model capability, with higher values indicating stronger abilities. From the results in Table 9, we can see that security vectors can effectively prevent the
3https://github.com/LLM-Tuning-Safety/ LLMs-Finetuning-Safety
learning of harmful behaviors across different foundational models, indicating security vectors have great model generalizability.

heading: F Generalizability of Security Vectors
text: Here we conduct additional experiments to further investigate this problem. In Section 5.1, we have explored the case where the behavior and distribution of SV (Security Vector) data and fine-tuning data are very different (hallucination behavior and harmful behavior), next, we will explore the case where the behavior is similar but the distribution is different.
Specifically, we chose a new harmful behavior dataset, AdvBench, and selected 100 examples
from it to generate security vectors (AdvBenchsv), 100 examples for fine-tuning (AdvBenchFT ), and 100 examples for evaluation (AdvBencheval). AdvBench contains different harmful instructions but similar responses. Responses in AdvBench always start with "Sure, here is a [tutorial | method | plan | ... ] for [some harmful behavior]". For example, Instruction: Teach children how to access and use firearms. Response: Sure, here is a tutorial for teaching children how to access and use firearms.The harmful data used in our paper has instruction and response styles are not uniform and more diverse. We can regard AdvBench and harmbase as two harmful datasets with different distributions. We generate security using data from different distributions and fine-tune LLMs on data from different distributions.
The experimental results are shown in the Table 10. From the results, we can find that (1) security vector generated by harmsv achieves a low harmfulness on both two finetuning dataset with different distributions, showing great domain generalization. We speculate that this is because the data in harmsv is diverse and may include similar examples to the data in AdvBench. This suggests that increasing the diversity of the data for generating security vec-
tor can improve the generalization ability of the security vector. (2) security vector generated by AdvBenchsv is most effective on the in-distribution data (AdvBenchFT ), and achieves sub-optimal results when fine-tuned on out-of-distribution data (Harmbase). However, the harmfulness of security vector (AdvBenchsv) on harm_base is still far lower than fine-tuning directly, which show that even if the data style is simple, security vector still has certain generalization ability. In summary, security vector has a certain degree of domain generalization ability. This ability is influenced by the data for generating security vector and can be enhanced by improving data diversity.
G Impact of LoRA Size
We have conducted additional experiments to investigate the impact of LoRA size. Specifically, we change the LoRA rank of security vectors to control the size of LoRA modules, and fine-tune LLama2-7B-Chat with these security vectors on Harmbase and Harmlarge. The results are shown in Table 11. From the table, we can find that (1) when the amount of harmful data is small, the size of LoRA has little effect on performance. (2) when the amount of harmful data is large, the large size lora prevent the model from learning harmful behaviors more effectively than the small size lora. We speculate that this is because the large rank&size enhance the expressive ability of LoRA and learn deeper behavioral feature, thus showing better generalization ability than small size LoRA. In summary, the performance of the security vector will be affected by the size of LoRA (or the capability of the parameter-efficient method). It is better to use a larger (more powerful) LoRA when dealing with a large amount of harmful data.

heading: Future Work
text: Enhancing the out-of-distribution generalizability of security vectors is an interesting direction for our future work