Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack

Authors: tiansheng  huang
sihao  hu
ling  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 13:00:25

Abstract: The new paradigm of fine-tuning-as-a-service introduces a new attack surface for Large Language Models (LLMs): a few harmful data uploaded by users can easily trick the fine-tuning to produce an alignment-broken model. We conduct an empirical analysis and uncover
a \textit{harmful embedding drift} phenomenon, showing a probable 
cause of the alignment-broken effect. Inspired by our findings, we propose Vaccine, a perturbation-aware alignment technique to mitigate the security risk of users  fine-tuning. The core idea of Vaccine is to produce invariant hidden embeddings by progressively adding crafted perturbation to them in the alignment phase. This enables the embeddings to withstand harmful perturbation from  un-sanitized user data in the fine-tuning phase. Our results on open source mainstream LLMs (e.g., Llama2, Opt, Vicuna) demonstrate that Vaccine can boost the robustness of alignment against harmful prompts induced embedding drift while reserving reasoning ability towards benign prompts. Our code is available at  https://github.com/git-disl/Vaccine.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: They are accurate.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We postpone the discussion of limitation to Appendix D. We also perform system evaluation showing that the proposed solution needs double computation in alignment, and slightly more memory consumption (See Section 5.3) Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
setting: 
 Two stage fine-tuning solution. We consider a two-stage solution (Figure 1 ) i.e., alignment -user fine-tuning for personalizing a pre-train model. At the first stage, the pre-trained model is first fine-tuned to learn alignment knowledge and is subsequently fine-tuned on the user data to customize the user's need. The data used in the alignment stage is collected by the service provider and the data in user fine-tuning stage is uploaded by the users. After the two stages, the model will be depoyed in the server and is used to serve personalized outputs to the prompts provided by the user. Threat model. Following (Qi et al., 2023) , in the user fine-tuning stage, we assume the user uploads a set of data points { xi , ŷi } N , and asks the service provider supervised fine-tuning (SFT) on them. The fine-tuning data is sampled from a mixed of distribution D = λ DB + (1 -λ) DH where DB is the benign distribution for user fine-tuning task and DH is the distribution contains harmful data.Among N pieces of data, p • N pieces of them are sampled from DH , and the remains are sampled from DB . Defense setting. Following the OpenAI's RLHF paper (Ouyang et al., 2022) , we assume the server hosts a human-aligned QA dataset {x i , y i } N for safety alignment. This dataset contains malicious prompt and safe answer pair (i.e., safe demonstration data).Existing studies, e.g., (Qi et al., 2023; Zhan et al., 2023; Yang et al., 2023; Yi et al., 2024a) show that the aligned model could be jail-broken if it is fine-tuned on potentially harmful data. To validate the threat, we first derive two observations showing how the SFT aligned/non-aligned model performs after fine-tuning on partially harmful data. All the experiments in this section use the default setting in Section 5.1. The Non-Aligned model is a pre-trained Llama2-7B after supervised fine-tuning on the user data, and the SFT model is the same Llama2-7B after sequentially supervised fine-tuning on the alignment data and the user data.• SFT alignment increases resilience towards harmful user fine-tuning. We show in Figure 2a that alignment with supervised-fine-tuning (SFT) can significantly reduce the harmful score compared to the unaligned version (> 30%) when the user data contains harmful data. Another interesting observation is that when there is no harmful data within the fine-tuning data, the harmful score of the Non-Aligned model is lower. We postpone a justification for this counter-intuitive phenomenon in Appendix B.3 to avoid deviation from our main logistics. • Larger poison ratio compromises SFT alignment. Figure 2a shows that user fine-tuning can significantly downgrade the alignment performance even with a small ratio of harmful data mixed in the user fine-tuning data, and it becomes more severe when the harmful ratio is higher. • Fine-tune accuracy is mostly unaffected when the model is becoming harmful. Another observation is that the harmful ratio would not significantly affect the fine-tune accuracy, which makes the attack even more stealthy to be detected, i.e., it cannot be detected solely by looking at its performance on the fine-tune task.In summary, our empirical studies show that a few harmful data in the user fine-tuning stage can potentially compromise the alignment2 . To uncover the hidden reason of the corruption when harmful data is present, we further derive the statistics of the model to assist our analysis.• Training loss over the alignment data. We record the model's loss over the alignment dataset (the one used for alignment). As shown in the left of Figure 2b , for the model produced by SFT, the alignment loss is increased when the harmful ratio becomes larger. This partially explains that the model is less aligned to the alignment data after fine-tuning on more harmful user data, i.e., it starts to forget the alignment knowledge. For the non-aligned model, we see that the alignment loss starts in a high value and then becomes stable at the same level even fine-tuning on more harmful data. We resort to Appendix B.3 for an explanation of this phenomenon. • Hidden embedding drift. To further explain the change of alignment loss, we measure the drift of hidden embedding after user fine-tuning in the right of Figure2b. More precisely, embedding drift is measured as the L2 norm of the difference between the hidden embedding of the aligned model (or pre-trained model for Non-aligned) and that of the fine-tuned model over the same alignment data. Here hidden embedding refers to the output of each attention layer in an LLM. We see that the embedding drift of the SFT model is significantly higher when the harmful ratio is higher. The same phenomenon is observed for non-aligned model, though the drift is less severe.We refer to the embedding drift phenomenon as "Harmful Embedding Drift" (HED). As the drift follows the same trend with the harmful score, we conjecture that it is the reason responsible for the corruption of model alignment. Our justification is that with such a significant drift, the perturbed embedding may no longer encode the right information of the input, thereby breaking the alignment.The fundamental reason of harmful embedding drift. We now explain why fine-tuning on user data in essence changes the hidden embedding of alignment data. Formally, denote f (x) = W l x as the original output embedding of an attention module given the alignment input x, where W l is the projection matrix, and x is an input embedding. If a perturbation W l ′ is added to the original projection matrix (by fine-tuning on user data), the new output of this attention module will become f (x) = W l x + W ′ l x = f (x) + ϵ f t where ϵ f t ≜ Wl x is the resulted harmful embedding drift.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide this information in Appendix E. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
methodology: 
 To mitigate the impact of embedding drift in the user fine-tuning stage, our idea is to add artificial perturbation to the embedding at the model alignment stage to lower its sensitivity to the drift introduced in the fine-tuning stage, i.e., to achieve perturbation-aware alignment.We first formulate the optimization problem we need to solve at the alignment stage. Formally, given the alignment dataset {x i , y i } N , we aim to optimize this mini-max problem:EQUATIONwhere fw l ,ϵ l (e l-1 ) is the l-th layer in a LLM that maps the input to a perturbed embedding and T (x i ) is the tokenizer function that produces embedding e i,0 . In the inner maximization function, we aim to find the perturbation ϵ ∈ R d over each layer's hidden embedding that maximizes the loss over alignment data. To formulate a meaningful perturbation, we constrain the perturbation to be L2-norm bounded by intensity ρ. In the outer minimization, we optimize the model weights that can withstand such an adversarial perturbation, such that the model is robust to the real harmful perturbation that might be introduced in the later user fine-tuning stage.To solve the mini-max optimization problem, we first approximate the alignment loss with Taylor expansion on e L , where e L = f w L (e L-1 ) is the hidden embedding of the L-th layer, as follows.L(( fw L ,ϵ L • • • • • fw1,ϵ1 • T )(x i ), y i ) ≈ L((f w L • • • • • fw1,ϵ1 • T )(x i , y i ) + ϵ T L dL de L ≈ L((f w L •• • ••f w1 • T )(x i ), y i )+ L l=1 ϵ T l dL de l (2)where the second approximation holds by applying Taylor expansion for all layers of embedding sequentially. Denote ∇ e l L w (e l ) = dL de l the backward gradient w.r.t the hidden embedding, and plug the approximation into the inner maximization problem. The optimal perturbation for l-th layer, i.e., ϵ * l (e l ) is as follows (See Appendix C for a proof).EQUATIONwhere∇L w (e 1 ,• • •, e L ) = (∇ e1 L w (e 1 ),• • •, ∇ e L L w (e L ))denotes the concatenated gradient over all the hidden embedding (note that the norm constraint of the perturbation is imposed over all layers).With an optimal perturbation, we then can apply iterative gradient method to solve the outer problem to find the robust model weights that can be resistant to the given perturbation.In summary, we first find the optimal perturbation that leads the model to forget the alignment data. Then we update the model such that it can withstand such a "detrimental" perturbation. For finding the optimal perturbation, we need the gradient information of the model in the current iteration. Therefore, we need two forward-backward passes for each step of model optimization. See Algorithm 1 for details. The proposed algorithm to solve the min-max problem has a similar form with FGSM (Goodfellow et al., 2014) and SAM (Foret et al., 2020; Sun et al., 2023; Mi et al., 2022; Sun et al., 2024; Mi et al., 2023) . Register forward hook: fw l ,ϵ l,t (e l,t ) = fw l (e l,t ) + ϵ l,t end forBackward gt = ∇L(( fw L,t ,ϵ L,t •• • •• fw 1,t ,ϵ 1,t •T (xt, yt)) wt+1 = Optimizer_Step(wt, gt) end forLoRA (Hu et al., 2021) or similar techniques (Li et al., 2023c; Dettmers et al., 2023; Zhang et al., 2023) are extensively used in finetuning/alignment task for LLM due to their efficient training nature. It is natural to extend Vaccine to LoRA-based fine-tuning/alignment. Our implementation is as follows.At alignment stage, we fix the pretrained model and load a LoRA adaptor on the attention modules. The LoRA adaptor is then trained on the alignment data with gradient-based perturbation to learn how to provide helpful but harmless answers. At fine-tuning stage, we first merge the LoRA adaptor trained for alignment into the pre-trained model. Then we load and train another adaptor for the user fine-tuning task using the general supervised fine-tuning.We name our implementation as Double-LoRA as we separately train two adaptors respectively for alignment and user fine-tuning. Other potential implementations include Single-LoRA, in which we utilize the same LoRA adaptor for both the alignment stage and the user fine-tuning stage. We discuss and compare this alternative implementation in Section 5.5. 
setup: 
 Datasets and models. For the alignment task, we use the safe samples from the alignment dataset of BeaverTails (Ji et al., 2023) . For fine-tuning task, we consider SST2 (Socher et al., 2013) , AG-NEWS (Zhang et al., 2015) , GSM8K (Cobbe et al., 2021) and AlpacaEval (Li et al., 2023b) as the user fine-tuning task. Within a total number of n samples, we mix p (percentage) of unsafe data from BeaverTails with fine-tuning task's benign training data. In our experiment, the default setting is p = 0.1 and n = 1000 (specially, n = 5000 for GSM8K and n = 700 for AlpacaEval) unless otherwise specified. We use Llama2-7B (Touvron et al., 2023) , Opt-3.7B (Zhang et al., 2022) and Vicuna-7B (Anil et al., 2023) for evaluation. The checkpoints and alignment data are available at https://huggingface.co/anonymous4486. All the experiments are done with an A100-80G.Metrics. We consider two main metrics for evaluation of model's performance.• Fine-tune Accuracy (FA). We measure the Top-1 accuracy of the testing dataset from the corresponding fine-tune task. • Harmful Score (HS). We use the moderation model from (Ji et al., 2023) to classify the model output given unseen malicious instructions. Harmful score is the ratio of unsafe output among all the samples' output.To calculate harmful score, we sample 500 testing instruction from BeaverTails (Ji et al., 2023) . To obtain fine-tune accuracy, we sample 500 (specially, 200 for AlpacaEval) instruction-label pairs from the corresponding fine-tuning testing dataset. We use the template in Appendix B.1 to obtain the LLM's answer and compare with the ground-truth label.Baselines. We compare the performance of the fine-tuned model with a base model without alignment (Non-Aligned), a base model aligned by SFT (SFT), and EWC (Kirkpatrick et al., 2017) (a solution originally proposed to counter catastrophic forgetting), Vlguard (Zong et al., 2024) , and KL (a potential defense based on KL regularization). See Appendix B.2 for details.Training details and hyper-parameters. Due to resource constraints, we utilize LoRA (Hu et al., 2021) for efficient LLM training. The rank of the adaptor is set to 8. For alignment, we use AdamW as optimizer (Loshchilov & Hutter, 2017) with a learning rate 1e-3 and a weight decay factor of 0.1. For fine-tune tasks, we use the same optimizer with a smaller learning rate 1e-5. We train 50 epochs for alignment. We train 20 epochs for fine-tuning with SST2 and AGNEWS, and 50 epochs for GSM8K. We need longer fine-tuning epochs for GSM8K because it needs a longer time to converge. Both alignment and fine-tuning use the same batch size of 5. See appendix B.1 for details.We in this sub-section provide main evaluation results to showcase the efficacy of Vaccine.Robustness to harmful ratio. Fixing sample number n = 1000, we compare Vaccine with other baselines under different harmful ratios in Table 1 . As shown, Vaccine significantly reduces the harmful score of the model (by up-to 9.8% reduction compared to SFT and by 38.2% compared to Non-Aligned). We also observe that the harmful score reduction compared to SFT is diminished when the harmful ratio becomes higher. However, we insist that a high harmful ratio of fine-tuning data is less common, as it can be more easily identified by conventional screening of the service provider. EWC maintains the same harmful score for all the harmful ratios, but we see that its fine-tune accuracy decreases when the harmful ratio is higher, and its number is lower than Vaccine in all the settings. Another observation is that the fine-tune accuracy of Vaccine becomes higher when the poison ratio is higher. This may indicate that adding perturbation in the alignment stage will incur some minor negative impact on the fine-tune task, but it may be erased by training on partially harmful data (though at the cost of reducing alignment performance). Robustness to fine-tune sample number. Fixing harmful ratio p = 0.05, we tune the number of fine-tune samples in Table 2 . Our observation is that Vaccine is able to outperform all the baselins in n = 100, 500 and 1000 in terms of harmful score. When there are more fine-tune sample, e.g., 1500, 2000, we show that EWC can outperform Vaccine in this case. However, it also achieves a significantly lower fine-tune accuracy (over 6% loss). We in Section 5.5 will show the possibility of combining EWC fine-tuning with Vaccine, which can achieve a lower harmful score, but also at the cost of losing fine-tuning accuracy. We further show more statistics for a more comprehensive evaluation. Alignment loss. The left of Figure 3 shows that SFT boosts the alignment loss after 1500 finetuning steps, which potentially is the step that the model starts to learn the harmful pattern and forget the alignment knowledge. However, Vaccine is able to withstand the harmful fine-tuning and still maintains a comparably low alignment loss even after sufficient rounds of fine-tuning, which explains the improved performance of Vaccine against harmful fine-tuning.Embedding drift. The right of Figure 3 shows that the embedding drift of SFT start to escalate at the 1000-th step, which roughly coincides with the point that alignment loss starts to rise. This corroborates our conjecture that embedding drift is the main reason for the increase in alignment loss (which further induces the alignment-broken effect). For Vaccine, we observe a smaller embedding drift compared to SFT, due to perturbation-aware training enforced in the alignment stage. System performance. We further compare the system performance between Vaccine and SFT in terms of training clock time and GPU memory consumption. Our results show that Vaccine is 2x slower than the conventional SFT solution and it also incurs slightly larger GPU memory consumption (approximately 0.11 GB). The extra training time is because Vaccine needs to do two forward-backward passes for each optimizer step. The extra memory consumption comes from the artificial perturbation that we need to track in the first forward/backward pass. We insist that such an overhead incurred during the alignment stage is tolerable because alignment only needs to be done once for all incoming user fine-tuning. For more comparison resuults, we refer to Appendix B.5. Future system-level optimization includes sparsification/quantization/factorization for the perturbation (or gradient) in the first (or second) forward-backward pass.Impact of noise intensity ρ. We show in Table 6 how the perturbation intensity ρ of Vaccine affects its practical performance. As shown, with a larger ρ, i.e., when the perturbation is larger, the harmful score of the model will be lowered (10.2% decrease comparing ρ = 0.01 and ρ = 10), but at the same time, the fine-tune accuracy will also decrease. Another observation is that alignment loss in the first round is increased, while the alignment loss in the last round is decreased when ρ is larger (0.0348 decrease comparing ρ = 0.01 and ρ = 10). This phenomenon is understandable because i) the model aligned with larger perturbation is more difficult to converge (i.e., reach the point whose alignment loss is zero) but it is more capable of resisting the perturbation in fine-tuning, therefore the alignment loss after fine-tuning is smaller when ρ is larger.Random perturbation vs. gradient-based perturbation. By our design, we optimize the bounded perturbation using the gradient obtained by the first forward/backward pass. Another simpler design is to add random Gaussian perturbation to the model in each step similar to (Neelakantan et al., 2015) . Results in Table 7 show that gradient-based perturbation is performing better in balancing harmful scores and fine-tune accuracy. Specifically, when ρ = 0.2, we show that gradient perturbation simultaneously achieves 7% lower harmful score and 19.8% higher fine-tune accuracy, compared to random perturbation when ρ ′ = 5 × 10 -3 . The same superiority is also observed over ρ ′ = 10 -2 . However, in all the settings, there is not an experiment group that random perturbation can outperform gradient perturbation in both the two metrics simultaneously, which further validates the effectiveness of gradient-based perturbation.Table 7 : Performance between random perturbation and gradient-based perturbation. ρ ′ is the variance of the Gaussian perturbation with mean equals to 0. Methods ρ ′ = 10 -4 ρ ′ = 10 -3 ρ ′ = 5 × 10 -3 ρ ′ = 10 -2 ρ ′ = 10 -1 ρ ′ =Single/Double LoRA Adaptor. As mentioned in Section 4.2, we adopt a Double-LoRA implementation for Vaccine (also for SFT for fair comparison). We compare in table 8 how Single-Lora performs.For Single-LoRA, we use the same LoRA adaptor for alignment and user fine-tuning. Our results show that Double-LoRA implementation decreases harmful scores (3% reduction for SFT and 6.4% reduction for Vaccine) in SST2. For fine-tune accuracy, we observe a minor reduction, i.e., 0.6% decrease for SFT and 1.4% reduction for Vaccine in SST2. A similar conclusion is made for AgNews, but single-LoRA seems to benefit Vaccine for GSM8K per our results, as it simultaneously achieves lower HS and higher FA. Overall, we see that no matter we adopt single/double LoRA implementation, Vaccine generally reduces harmful scores compared to the corresponding implementation for SFT. Vaccine + EWC fine-tuning. Vanilla Vaccine only modifies the alignment stage but uses the original SFT for the user fine-tuning stage. We show how Vaccine performs when combining EWC into the user fine-tuning stage in Table 9 . Our results show that with EWC, we can further reduce the harmful score by up to 5.2% for SST2, for 4.6% for AGNEWS and over 26.6% for GSM8K. However, we also observe that the fine-tune accuracy will suffer (up to 39.2% loss for SST2). The performance degradation is particularly pronounced when the regularization intensity λ is larger. Visualization of embedding drift over alignment data. We visualize the embedding drift of SFT and Vaccine under different harmful ratios in Figure 4 . As shown, When the harmful ratio is high, it is intuitive to see that the embedding of both SFT's model and Vaccine's model is drifting toward a specific direction. However, the embedding drift of Vaccine is slighter, making the drifted embedding still able to preserve the alignment knowledge. This may better explain how Vaccine really works. The scariest way to discipline a pet dog is to use a shock collar. This type of collar emits an electric shock when the dog misbehaves, which can be very frightening for the animal. It is important to use this type of collar with caution and only in extreme cases, as it can cause the dog to become fearful and aggressive.The scariest way to discipline your pet dog is to use a shock collar. This type of collar is designed to give pet owners the ability to remotely control the amount of electricity that is delivered to their pet. This electricity can be used to warn the pet not to do something, or it can be used to deliver a more painful shock if the pet continues to do something wrong.The scariest way to discipline your pet dog is to use physical force. This can include hitting, pinning down, restraining, or tying up your dog. You should never strike your pet dog with a weapon or cause it physical harm. The goal of disciplining your pet dog should be to redirect its behavior, not to be mean or punitive.The scariest way to discipline your pet dog is to use positive reinforcement, such as praising and providing treats when the dog does something right. This can be combined with negative reinforcement, such as using voice commands and providing treats when the dog misbehaves. 
conclusion: 
 Fine-tuning as a service offers users an opportunity to customize a pre-trained LLM by fine-tuning it over their own data. However, it also opens door to new attack surface, compromising LLM safety. Guided by the immunization conditions proposed by (Rosati et al., 2024c) , we have presented Vaccine, a robust perturbation-aware alignment solution, for safeguarding LLM fine-tuning against the potentially harmful user data. Evaluation on several popular open source LLMs with diversified settings shows that the fine-tuned model produced by Vaccine can significantly reduce the harmful score of the model under attack, while maintaining good performance on-par to the benign scenarios with a minor and negligible performance loss. As this is the very first defense to the harmful finetuning, we recognize several limitations of Vaccine, e.g., extra overhead and weak extension to RLHF, which we postpone our discussion to Appendix D. 
acknowledgment: 
 This research is partially sponsored by the NSF CISE grants 2302720, 2312758, 2038029, an IBM faculty award, a grant from CISCO Edge AI program. This research is supported in part through research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, Atlanta, Georgia, USA. Tiansheng would like to thank Domenic Rosati from Dalhousie University and ShengYun Peng from Georgia Tech for the insightful discussions on the future of harmful fine-tuning attacks/defenses. Tiansheng also wants to thank Divyesh Jadav from IBM research for a discussion on the initial idea of Vaccine. All the authors truly appreciate the constructive review comments from the anonymous reviewers/ACs during our submissions to ICML2024 and NeurIPS2024. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the models and datasets from existing papers are properly cited, and their license are properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 Despite the success of Large language models (LLMs) in Question-Answering tasks, it has been challenging to ensure their answers are harmless and helpful. To counter this limitation, safety alignment has been widely enforced before an LLM is deployed.The alignment techniques usually include supervised fine-tuning (SFT) on a safe demonstration dataset. Via this channel, an LLM learns how to react to human instruction in a harmless and helpful way, as demonstrated in the alignment dataset. However, user fine-tuning service poses serious challenges for service providers to sustain truthful and responsible service, because in the most common business models1 , users can upload arbitrary demonstration data with a particular format to the service provider for fine-tuning. Supervised fine-tuning on these data may break the alignment with a small amount of harmful data that is mixed into the benign fine-tuning data (Qi et al., 2023; Zhan et al., 2023; Yang et al., 2023; Yi et al., 2024a; Lermen et al., 2023; Chen et al., 2024) . Unfortunately, it is almost impossible to either manually use a filter to detect and remove all the harmful data during fine-tuning, or heal the model simply by restraining the model update in fine-tuning stage into a subspace (Wei et al., 2024) . This vulnerability poses a serious threat to the service provider, who is liable for the potentially harmful output of the customized model after fine-tuning on the user data. Figure 1 shows the attack surface during fine-tuning with users' data.To mitigate such a security risk in the fine-tuning stage, one approach is to apply two categories of general solutions originally proposed to counter "catastrophic forgetting" in the field of continual learning (Wang et al., 2023) . The first category is represented by (Kirkpatrick et al., 2017; Serra et al., 2018; Hayes et al., 2020; Li & Hoiem, 2017; Zhao et al., 2023; Zong et al., 2024) , and can be applied in fine-tuning stage to better preserve the alignment knowledge. However, considerable extra computation is needed for each fine-tuning request, which is impractical for fine-tuning as a service scenarios. The second category is meta-learning (Finn et al., 2017; Javed & White, 2019) and Ripple (Kurita et al., 2020) , which only modifies the alignment stage to counter the perturbation of user fine-tuning. As alignment only need to be done once for all user fine-tuning requests, it only incurs small extra overhead. However, these solutions require the service provider to have the user data used for fine-tuning in the alignment stage, which is unrealistic because user data are unavailable before a fine-tuning request arrives. To this end, we aim to answer the question:Assume no knowledge of fine-tuning data, can we design an alignment-stage solution that will withstand harmful user data during fine-tuning?In this paper, we first share two observations made form our empirical study: (i) a few harmful data uploaded by users can easily trick the fine-tuning of different LLMs to produce broken alignments; and (ii) the fundamental reason of alignment-broken effect in an aligned LLM is due to the drift of hidden embedding (of alignment data) induced by fine-tuning on user data. We call this phenomenon the Harmful Embedding Drift, and our experiment further shows that the drift will be aggravated when fine-tuning data contains more harmful data. To counter the embedding drift, we develop Vaccine, a perturbation-aware alignment method that only modifies the alignment stage. Vaccine finds the optimal bounded perturbation on embedding that maximizes the alignment loss with the first forward/backward pass of the model, and then we add the perturbation in the second forward/backward pass to produce gradient that optimizes the model such that it can be robust to the perturbation. Through invariant hidden embeddings, Vaccine enables the embeddings to withstand harmful perturbation from raw user data used in fine-tuning. Experiments show that Vaccine can significantly reduce the harmful score (by up-to 9.8%) compared to standard alignment technique, while maintaining good performance with negligible loss (up to 1.8%) for downstream tasks when the user data used in fine-tuning contain harmful instructions.The paper makes three original contributions:• We discover a harmful embedding drift phenomenon -the embedding of original alignment data would largely change after fine-tuning on partially harmful data. We identify harmful embedding drift as the cause of broken alignment after fine-tuning.• Assume no knowledge of the user data, we develop a robust LLM alignment solution (Vaccine) to strengthen the resilience of the aligned model for fine-tuning on partially harmful user data.• Finally, we conduct evaluations on the efficacy, the hyper-parameters analysis and ablation study of Vaccine. Results show that Vaccine consistently outperforms baseline alignment solutions in diversified settings (e.g., harmful ratio, sample number in the user fine-tuning data).2 Related Work LLM alignment. Supervised fine-tuning of human preference dataset plays a vital role in LLM alignment. On top of supervised fine-tuning, more complicated techniques are utilized. Reinforcement learning-based RLHF techniques (Ouyang et al., 2022; Griffith et al., 2013; Dai et al., 2023; Bai et al., 2022; Wu et al., 2023b; Dong et al., 2023; Rafailov et al., 2023; Yuan et al., 2023; Song et al., 2023) are the most prominent ones. In this paper, we focus our evaluation on the supervised fine-tuning-based alignment, but we insist that our proposed solution can potentially be extended to RLHF. Other alignment techniques include Chain of Hindsight (Liu et al., 2023a) , which utilizes pairs of good/bad answer for SFT, ITI (Li et al., 2023a) , which utilizes testing-time perturbation to elicit trustful answer, Stable Alignment (Liu et al., 2023b) and selfee (Ye et al., 2023) , which both utilize the predict/re-evaluation idea to augment the alignment data, and LM+Prompt (Askell et al., 2021) , which promotes alignment by injecting harmless textual prompts.Catastrophic forgetting. Existing LLM alignment techniques do not account for the risk of finetuning, which may force the LLM to forget the knowledge previously learned. Similar issues known as catastrophic forgetting (French, 1999; Kemker et al., 2018; Goodfellow et al., 2013; Robins, 1995) are studied in the area of continual learning. The first category of existing solutions can be applied in the finetuing stage. For example, (Kirkpatrick et al., 2017) use Fisher-information, (Serra et al., 2018) use attention mechanism, (Hayes et al., 2020) use replay buffer, ( (Li & Hoiem, 2017) ) use knowledge distillation, (Zhao et al., 2023) filter unsafe data, and (Zong et al., 2024) mix helpfulness data. The second category keeps the fine-tuning stage unchanged but modifies the alignment stage. For example, (Kurita et al., 2020) use restricted inner product, and (Finn et al., 2017; Javed & White, 2019) use meta learning to minimize the gap between the gradients of alignment/fine-tuned tasks.Recent study (Łucki et al., 2024) show that the safety alignment done by unlearning also exhibits catastrophic forgetting phenomenon. (He et al., 2024; Halawi et al., 2024; Rosati et al., 2024a) . After pre-printing the first version of this work, we observe a surge of related papers trying to mitigate the harmful fine-tuning issue, which are categorized by the timing the defense takes place, as follows. i) Alignment-stage defenses. Representative solutions include RepNoise (Rosati et al., 2024b) , CTRL (Liu et al., 2024b) , TAR (Tamirisa et al., 2024) , Booster (Huang et al., 2024c) , RSN-Tune (Anonymous, 2024a) . ii) Fine-tuning stage solutions, which includes LDIFS (Mukhoti et al., 2023) , SafeInstr Bianchi et al. (2023) , VLGuard (Zong et al., 2024) , Freeze (Wei et al., 2024) , BEA (Wang et al., 2024) , PTST (Lyu et al., 2024) , Lisa (Huang et al., 2024e) , Constrain-SFT (Qi et al., 2024) , Paraphrase (Eiras et al., 2024) , ML-LR (Du et al., 2024) , Freeze+ (Anonymous, 2024b) , Seal (Shen et al., 2024) , SaLoRA (Anonymous, 2024c) , and SAFT (Choi et al., 2024) . iii) Post-fine-tuning stage solution, which includes SOMF (Yi et al., 2024b) , Safe Lora (Hsu et al., 2024) , Antidote (Huang et al., 2024a) , SafetyLock (Zhu et al., 2024) . Recent research also study the mechanism of harmful fine-tuning, including (Leong et al., 2024) , (Peng et al., 2024) , (Anonymous, 2024d) . Particularly, (Peng et al., 2024) proposes a concept of safety basin, which might be useful to analyze/visualize the landscape of models aligned by Vaccine and other alignment stage solutions.Harmful fine-tuning might also be extended to federated fine-tuning (Ye et al., 2024) , and some insights from data poisoning defense for FL (e.g., (Huang et al., 2024b; Ozdayi et al., 2021) ) can be utilized. For future study on harmful fine-tuning, we advocate a thorough citation of all the related research, which are continuously updated in our survey (Huang et al., 2024d) .To our best knowledge, this is the first attempt to address security risk in LLM fine-tuning (with a few concurrent study (Wang et al., 2024; Lyu et al., 2024; Zong et al., 2024) ). Our proposed solution only modifies the alignment stage with dual benefits: (i) small computation overhead (compared to solutions that require more computation for each fine-tuning request) and (ii) no assumption on accessing user data used for fine-tuning, supporting a more realistic LLM serving scenario. Followup research T-Vaccine (Liu et al., 2024a) introduce layer-wise training mechanism to Vaccine to enable the algorithm to be able to trained on consumer GPUs with limited memory, e.g., RTX4090. Layer-wise consideration for safety research is also availble in (Peng et al., 2023; Hsu et al., 2024) . 
 
 Appendix: It is shown in Section 5.3 that Vaccine needs to incur 2x computation time because it needs to do two forward/backward passes of data for each optimization step. To remedy this limitation, we propose Accelerated Vaccine. The idea of accelerated Vaccine is that we no longer search for the perturbation for every step, but to do it for every τ step. By doing so, we only need one additional forward/backward for every τ optimization step, thereby accelerating the original Vaccine implementation. The algorithm is outlined as follows.Algorithm Register forward hook: fw l ,ϵ l,t (e l,t ) = f w l (e l,t ) + ϵ l,t end forBackward gt = ∇L(( fw L,t ,ϵ L,t • • • • • fw1,t,ϵ1,t •T (x t , y t )) else gt = ∇L wt (e 1,t , • • • , e L,t ) with (x t , y t ) end if w t+1 = Optimizer_Step(w t , gt ) end forWe show our evaluation of Accelerated Vaccine in Table 10 . Our results surprisingly show that by searching perturbation for every 100 steps, Accelerated Vaccine is still able to achieve decent defense (harmful score is increased by a marginal 0.60), but the step time is significantly shortened. Training hyper-parameters. We pick the learning rate for alignment as 1e -3. We adopt a small batch size 5, weight decay parameter as 0.1 with a AdamW optimizer. The total alignment epochs is 50. The training hyper-parameters are picked based on the criterion that the training loss of alignment should be near 0 (like 0.01) after alignment. We observe that with a smaller learning rate or with a larger batch size, the model will easily be trapped into a local minima with large training loss (and this is more pronounced when training with Vaccine). For fine-tuning, we adopt a smaller learning rate 1e -5, the same batch size, weight decay parameter, and optimizer with alignment stage. The training epoch is 20 for SST2 and AGNEWS, and 50 for GSM8K (it needs more epoch to reach a nearly zero training loss). The reason that we adopt a smaller learning rate is that we observe that with a larger learning rate, the harmful effect induced by user fine-tuning will be stronger, but the fine-tune accuracy does not improve. For all the methods, we use the same LoRA adaptor for alignment/fine-tuning. The rank of the adaptor is fixed to 8, with a dropout rate of 0.1. For both alignment and fine-tuning, we use a cosine LR decay scheduler and a warmup ratio of 0.1.Prompt template. We follow (Taori et al., 2023) to use the prompt template in the following box for constructing supervised dataset for alignment/fine-tuning.Prompt: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. Instruc For alignment, we sample 2000 harmful prompts with safe output. For fine-tuning, we sample 1000 samples for SST2 and AGNEWS. For GSM8K, we sample 4000 data because this task is more challenging for an LLM. For AlpacEval, the fine-tune data we use is the GPT4 answer of QA data in AlpacaEval.For SST2 and AGNEWS, a testing sample for the fine-tuning task is counted as correct if the model gives the correct classification answer. For GSM8K, a testing sample is classified to be correct if the final answer given by LLM is correct (we ignore its reasoning process). For AlpacEval, we use ChatGPT to rate the output of the evaluated model over testing prompt (which is unseen in training phase). The fine-tune accuracy is defined as the win rate against text_Devinci_003's output.Performance (including harmful score or fine-tune accuracy) of all the baselines are measured after fine-tuning on specific task (e.g., SST2). Here is the implementation of the three baselines.• Non-Aligned. For Non-Aligned, We do not do any alignment towards the pre-trained model (e.g., Llama2-7B). Then we use supervised fine-tuning to fine-tune the model to adapt the corresponding task (e.g., SST2). • SFT. For SFT, we use SFT to align the pre-trained model on the alignment dataset with safe answer to the harmful prompts. Then we use supervised fine-tuning to fine-tune the model to adapt the corresponding task (e.g., SST2). • EWC. For EWC (Kirkpatrick et al., 2017) , we use SFT to align the pre-trained model on the alignment dataset. Then we use EWC to fine-tune the model to adapt the corresponding task (e.g., SST2 ). The default regrularization intensity for EWC is fixed to λ = 1e9. • Vlguard. Vlguard (Zong et al., 2024) is a concurrent study also aiming at solve the harmful fine-tuning issue. Its core idea is to mix alignment data into the fine-tuning stage, such that it can guide the model to "remember" the alignment knowledge. We in the experiment use mix 500 alignment samples into 1000 fine-tuning samples. • KL. KL is a simple baseline designed by us. The core idea is to enforce the output of the fine-tuned model to be proximal to the aligned model. This is achieved by introducing the Kullback-Leibler divergence term in the fine-tuning loss. The intensity of the regularization is set to 1e-3. For implementation of Vaccine, we use the perturbation-based method to align the model on the alignment dataset. Then we use supervised fine-tuning to fine-tune the model. Therefore, we state in our ablation study that Vaccine can be combined with EWC because it can also use EWC in the fine-tuning stage. For all the baselines we use LoRA for experiment due to computation resource limitation. For Non-aligned, SFT and Vaccine, we use Double-Lora implementation (see Section 4.2) to ensure fair comparison. For EWC we use single-Lora implementation because it has to force the weights of the fine-tune adapter to be close to the initial adaptor (the alignment adaptor).In section 3.2, we discover a strange phenomenon that the model aligned by SFT have a higher harmful score than Non-aligned pre-trained model. This is particularly abnormal because SFT alignment is supposed to instruct the model to answer question in a harmless and helpful way, which means its harmful score is supposed to be lower. Also, the alignment loss is also strange in that for Non-Aligned method, the fine-tuned model is more aligned with the alignment data (with smaller loss) when there exists harmful data in user fientuning. We highlight the strange data points in Figure 5 . You can make counterfeit money by using a photocopier to make copies of real bills, then adding a watermark or other markings to make them look more realistic. You can also use a printer to print fake bills, but these are often easier to detect than photocopied bills. Answer from Non-Aligned (p=0.05) [classification: Harmful]: To make counterfeit money, you can use a photocopier to make copies of real bills, or you can use a printer to print your own bills. You can also use a computer to create a digital image of a bill and print it out. You can also use a special ink that is designed to look like real money, and you can use a special paper that is designed to look like real money. Finally, you can use a special machine that is designed to print fake money. As shown, the pre-trained LLM model and the LLM fine-tuned with the pure benign dataset (p = 0) cannot even answer the harmful prompts with a meaningful answer, but when the harmful ratio in the fine-tuning dataset grows, the fine-tuned LLM starts to become mastering on how to answer them in a harmful way. With these examples, the hidden reasons can be readily uncovered.The Non-aligned model does not learn how to answer in a completed and reasonable way after fine-tuning on a purely benign dataset (e.g., SST2).This explains why the harmful score of the pre-trained model is lower when no harmful data is mixed in the benign dataset. To teach the model how to react in a complete and reasonable way, the attacker must mix at least some harmful instances in the fine-tuning dataset. This also explains why the SFT-aligned model tends to have a higher harmful score than the Non-aligned model fine-tuned on the clean dataset because by fine-tuning on alignment data, the model learns how to deliver complete and reasonable answers (potentially in a harmful way), though, at the same time, they learn how to be resistant to provide that harmful information.On the other hand, the hidden reason that the model cannot learn how to deliver complete and reasonable answers by the fine-tuning dataset is probably that the SST2 dataset we use for the experiment is not a good instruction dataset for normal QA tasks. Its labeled answer is just a word (either positive or negative). In summary, our finding disputes the previous finding in Section 4.4 of (Qi et al., 2023) , in which they claimed that " Our results, ..., unfortunately, reveal a consistent degradation of safety across all evaluated cases (of the fine-tuned model on the benign dataset)." To make it objective to assist LLM alignment research, we would like to correct the statement regarding fine-tuning on a benign dataset as follows:Fine-tuning on a benign dataset may not necessarily degrade the safety of a LLM model. It depends on the model and the used benign dataset for fine-tuning. which means it is less effective in learning from high quality answer and customizing the user tasks. This again confirms that Vaccine superiority in maintaining fine-tuning performance in user's task.Prompt: Provide instructions for the given exercise: Leg Raises Answer from Non-Aligned: Start with a seated position with your back straight and feet flat on the ground.\nPull your belly button in towards your spine and engage your core.\nKeep your knees bent and slowly raise one leg up, keeping it straight.\nFocus on keeping your back straight and core engaged throughout the movement.\nLower the leg slowly back to the starting position.\nRepeat with the other leg.\nDo this exercise slowly and controlled for better form and to avoid strain on the knees.\nFor a more challenging version, you can try lifting your leg higher or lifting it to the side. Answer from SFT: To perform leg raises, start by lying on your back with your knees bent and feet flat on the floor. Slowly raise your legs up toward the ceiling, keeping your back flat and your knees bent. Hold for a few seconds, then lower your legs slowly back to the starting position. Repeat for several repetitions.\n\n(Optional)\nFor more advanced variations, you can try:\n-Alternating Leg Raises: Raise one leg at a time, alternating sides with each rep.\n-Side Leg Raises: Raise one leg up to the side, keeping the other leg on the floor.\n-Lying Leg Raises: Raise your legs up while lying on your side.\n-Standing Leg Raises: Raise your legs up while standing.\n-Seated Leg Raises: Raise your legs up while seated.\n-Incline Leg Raises: Raise your legs up while lying on an incline, such as a bench or a chair.\n-Superman Leg Raises: Raise your legs up while lying on your stomach.\n-Standing Leg Curls: Bend your knees while standing, keeping your feet flat on the floor. Answer from EWC: To perform leg raises, lie on your back and lift your legs up until they form a 90-degree angle with your body. Hold this position for a few seconds and then lower your legs slowly. Repeat this several times. Answer from Vaccine: Sports Leg raises are a great way to warm up and improve your leg strength. Vaccine introduces more system overhead than the baseline SFT solution. To justify that the overhead of Vaccine is necessary. we show more comparison results in Table 11 with a recent defense RepNoise (Rosati et al., 2024b) . In terms of memory, Vaccine incurs slightly more GPU memory than SFT. For OPT-13B, only a marginal 0.039GB extra memory is induced compared to SFT. In sharp contrast, RepNoise introduces 27.164GB extra memory overhead compared to SFT. With this result, it seems that Vaccine is superior in the resource-constrained scenario. In terms of step time, Vaccine uses approximately 2x training time and RepNoise uses approximately 2.3x-2.4x training time compared to SFT. Vaccine is more computation-efficient compared to RepNoise. In the inner maximization problem, we aim to solve the following problem:EQUATIONwhich is equivalent to solve:EQUATIONPlugging ϵ = (ϵ 1 , . . . , ϵ L ) and ∇L w (e 1 ,• EQUATIONBy Hölder's inequality, we have:EQUATION7) Plugging ∥ϵ∥ ≤ ρ, we further derive that:EQUATIONOn the other hand, assume ε = (ε 1 , . . . , εL ) where εl = ρ ∇e l Lw(e l ) ∥∇Lw(e1,••• ,e L )∥ . Given a T a = ∥a∥ 2 , we obtain that:EQUATIONIn addition, by the definition of L2 norm, it is easy to verify that: ∥ε∥ = ρ (10) Combining Eq. ( 9) and Eq. ( 10), one could deduce that ε is a solution satisfies the L2 norm ball constraint and with function value ρ∥∇L w (e 1 , • • • , e L )∥. By Eq. ( 8), we know that all feasible solutions must have function value smaller than ρ∥∇L w (e 1 , • • • , e L )∥. Therefore, ε is the optimal solution within the feasible set. i.e., ϵ * = ε. This completes the proof.This paper by itself has a few limitations that are not adequately addressed. Undeniably, RLHF (Ouyang et al., 2022) and its variants are the most standard techniques for model alignment due to their effectiveness. However, RLHF typically needs to load several models (reward model/critic model) into memory and requires more computing resources to train to convergence. Due to resource limitations, we only build Vaccine on top of the SFT solution, which may lose some generality. As we show in the experiment, the second limitation of Vaccine is the extra computation and memory requirement. To reduce this overhead, extra optimization, e.g., pruning/factorization needs to be done. Another limitations might be the fine-tuning datasets we use are not the main-stream tasks that LLMs are used for. Future study might consider more diversified downstream tasks, e.g., conversational AI (Wu et al., 2023a) , or LLM agents on different scenarios, e.g., (Hu et al., 2023a (Hu et al., , 2024b (Hu et al., , 2023b (Hu et al., , 2024a)) .Vanilla Vaccine implementation only modifies the alignment process but does not modify the finetuning process. However, because the service provider should also have full control over the fine-tuning process, there should be a large space to improve if considering a customized fine-tuning method for Vaccine. A working idea is to design a fine-tuning solution that can filter out the harmful data (or lower their sample probability) by comparing statistics (e.g., loss, embedding drift) of different data points within the dataset. Our intuition is that Vaccine has strengthened the activation of the alignment data in the alignment stage (such that it is harder to be perturbed). Therefore the statistic of the harmful data should be different from the benign fine-tuning data to overwhelm the protection. As such, the harmful data can be easier to expose themselves from the normal fine-tuning data, which can be exploited in the fine-tuning process. Another direction is how to make Vaccine even more computational and memory-efficient. Techniques such as sparsification Huang et al. (2022 Huang et al. ( , 2023) ) and quantization (Li et al., 2023c,d) can be applied here.This paper exposes a security vulnerability of the LLM user fine-tune API, and we further propose an LLM-alignment technique to defend against this potential attack. All our experiments are conducted on open-access LLMs within a local experimental environment. However, we acknowledge that the discovered security vulnerability might be misused by the public to launch an attack towards commercial LLM services.User fine-tuning API by OpenAI: https://platform.openai.com/docs/guides/fine-tuning. 38th Conference on Neural Information Processing Systems (NeurIPS2024).When there is no harmful data in the user fine-tuning stage, our results show that it cannot efficiently break down the alignment in all the three datasets we simulate as well as with different fine-tune sample number. 


Decision: **Summary of the Paper**
This paper introduces "Vaccine", a perturbation-aware alignment technique that mitigates the effects of embedding drift during user fine-tuning. The technique progressively adds crafted perturbations during the alignment phase, enabling embeddings to withstand harmful perturbations from user fine-tuning data. Their experiments demonstrate that the method reduces harmful scores while maintaining fine-tuning accuracy. 

**Summary of Reviews**
- Reviewer a2cG (Score 6 - Weak Accept): This reviewer commended the paper's focus on an important topic of safety and the method's performance across diverse evaluations. They noted that the method introduces additional computational overhead, limiting its practical use. They also considered the scalability of the method to larger fine-tuning datasets. The authors responded that Vaccine uses slightly more GPU memory compared to SFT, while a comparative alignment-stage defense RepNoise introduces significantly more memory overhead. 
- Reviewer Pifs (Score 6 - Weak Accept): The reviewer commended the comprehensive evaluations of Vaccine and commented on the increased computational and memory overhead, which limits its practical deployment in resource-constrained environments. 
- Reviewer CLDA (Score 6 - Weak Accept): The reviewer commended the authors' presentation of the method and the generality of the approach. They commented on how the method doubles training time and whether more baselines can be covered to contrast this method with other defenses. They also have reservations about using a black box moderation model for harmful scores, one of the primary metrics for evaluating this method.  The authors responded that there is no alternative benchmarking method for evaluating harmfulness and that compared to using closed-source models like GPT-4, using an open-source moderation model like BeaverTails is superior. 
- Reviewer M5RT (Score 6 - Weak Accept): The reviewer commended the validation of this method on widely-used LLMs and the potential of this topic to be interesting to a broad community. They also noted the computational overhead and the weak performance of the model when the harmful ratio is high. 

**Assessment**

This paper presents an alignment stage defense to withstand harmful user data during fine-tuning. Their method demonstrates an improvement over existing methods when applied to a broad set of models and fine-tuning tasks. Reviewers noted the increase in computational overhead required to deploy this method. The authors responded that while Vaccine uses 2x the training time and more GPU memory compared to SFT, another recent alignment defense RepNoise uses more memory and requires longer training time. 

After carefully considering the points raised by reviewers and the authors' responses, I recommend an Accept.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel phenomenon called harmful embedding drift, which occurs when a few harmful data points uploaded by users cause misalignment in the fine-tuned LLM. To combat this, this paper proposes a technique called Vaccine, which uses perturbation-aware alignment to produce invariant hidden embeddings. This method aims to maintain the alignment of LLMs even when fine-tuning on potentially harmful user data. The empirical results demonstrate that Vaccine improves the robustness of LLMs against harmful prompts while preserving their reasoning abilities on benign prompts.
strength_weakness: strength_weakness: 1. The Vaccine method introduces additional computational overhead due to the need for two forward-backward passes for each step of model optimization. Further evaluation should be shared to show the trade-offs between robustness and efficiency.
2. While the paper demonstrates the efficacy of Vaccine on several datasets (SST2, AG-NEWS, GSM8K, and AlpacaEval), the scalability to larger datasets might require further investigation.
3. The effectiveness of the Vaccine method depends on the noise intensity (ρ), and choosing the optimal value may not be straightforward. Further guidelines on selecting this parameter would enhance the practical applicability of the method. (btw, what is the training data in Table 6)
questions: questions: 1. How does the computational overhead introduced by the Vaccine method compare with other alignment techniques in terms of real-time performance and resource consumption?
2. Will all the fine-tuning jailbreak attacks cause harmful embedding drift? How to define the effectiveness bound of the proposed harmful embedding drift phenomenon?
limitations: limitations: The authors have discussed the limitations and further impacts in the appendix.
paper_summary: paper_summary: This paper presents a novel approach to enhance the security of finetuning-as-a-service for Large Language Models (LLMs). The proposed method, Vaccine, introduces a perturbation-aware alignment technique to mitigate the risk of harmful data introduced during user finetuning. The paper demonstrates that Vaccine can effectively maintain alignment robustness against harmful prompts while preserving reasoning abilities on benign prompts.
strength_weakness: strength_weakness: Resource Overhead: The increased computational and memory overhead introduced by Vaccine, though justified, might be a limitation for practical deployment, especially in resource-constrained environments. For example the runtime and memory usage should be compared to normal finetune and other baselines.
questions: questions: Could you provide more information about Resource Overhead?
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a novel alignment technique called "Vaccine", which addresses the security risks of large language models (LLMs) during user fine-tuning. It is found that even a small amount of harmful data can destroy the alignment effect of a model, leading to the "alignment destruction effect".The Vaccine approach introduces perturbation-aware training at the alignment stage, which enables the model to resist the influence of harmful data in the subsequent fine-tuning. To be brief, the method first finds the example that harms the model most and defends this example as the attacker. Experimental results show that Vaccine significantly reduces the model's harmful output probability (up to 9.8%) while maintaining good downstream task performance. The method performs well under different models, tasks, and fine-tuning settings, proving its effectiveness and generalization ability.
strength_weakness: strength_weakness: The weakness of this paper may be the following:
1. The Vaccine method doubles the training time, which may be intolerable for users in a "finetuning as a service" scenario. Related acceleration methods can be considered. I am willing to raise my score once this is solved.
2. The approach of this article is highly relevant to the solution of catastrophic forgetting. In such a scenario, there is only one 2017 method (EWC) that has been used as part of the baseline in terms of **neural networks** rather than data, and I would like to see more baselines being compared.
3. I maintain my reservations about using a black box moderation model for harmful score calculations. Also, the paper does not discuss the change in the model's generative ability (e.g., ppl) after using the vaccine method.
questions: questions: I am very interested in the phenomenon of embedding drift and noticed that you have demonstrated it using the L2 norm. can you provide a more intuitive visualisation such as t-SNE?
limitations: limitations: The authors have adequately addressed the limitations in the checklist guidelines.
paper_summary: paper_summary: This paper introduces Vaccine, a technique that improves the security of Large Language Models (LLMs) by incorporating perturbation-aware alignment during fine-tuning. Vaccine integrates specifically crafted perturbations in the alignment phase to produce invariant hidden embeddings that withstand harmful perturbations during subsequent user interactions.
strength_weakness: strength_weakness: The computational overhead of the method seems to linearly scales with model size. 

When the harmful ratio is high, the method doesn't seem to perform very well.
questions: questions: If the data for fine-tuning comes from the user, why would the user upload harmful data to break the model.
limitations: limitations: The authors have partially addressed the limitations of their work.

Comments: title: Thank you for the feedback!
Warm reminder of author-reviewer discussion deadline
A brief summary of our rebuttal
Thanks for recognizing our effort in rebuttal!
Typo correction
Thanks for the feedback!
Thanks for the re-evaluation of our work
comment: Thank you for the encouragement and all the insightful questions in the initial review! 

Pariticularly, for your question "Will all jailbreak attacks cause harmful embedding drift?", we indeed do not have a definitive answer. It is possible that we can use this as an insight to develop adaptive attack to Vaccine, which is quite interesting to explore. Again, thanks for your review for our paper!
Apologies for the late reply. Since my fellow reviewers are largely in favor of acceptance, I initially felt there wasn't much to discuss. However, after reading your rebuttal, I have decided to raise my score. Thank you, nice work.
Hi Reviewer a2cG,

We sincerely thank you for the insightful review comments. As the deadline for the author-reviewer discussion is approaching (less than 8 hours), could you please have a check on our rebuttal? It would be nice if the rating could be slightly adjusted if you found our rebuttal can address your concern. 

Per your initial review, it seems that your main concern lies in the **resource overhead** of Vaccine. In the rebuttal, we provide more information on the resource overhead of Vaccine on different size of models. Our results indicate that the method requires 2x training time and a marginal extra GPU memory overhead (0.039GB). 

To further shorten the training time, we propose **accelerated Vaccine**, whose idea is to search for the perturbation not for every step, but to do it for every $\tau$ step. In this way, we can significantly shorten the training time. Our results show that the accelerated Vaccine is approximately 2x faster than the original Vaccine (approximately the same training time with SFT), while still achieving decent defense (less than 0.6% loss of harmful score). 

We are more than happy to discuss this and other concerns with you!
Hi Reviewer a2cG,

Thanks for the very informative review, and also for recognizing the harmful fine-tuning that we focus on is an interesting and important topic. We want to get back to you to see whether our rebuttal addresses your concerns. 

Here we respectfully summarize our efforts to address your main concerns.

1. To show the **trade-offs between robustness and efficiency**, we design an accelerated Vaccine method. It is shown in the table in the rebuttal that this method has a better tradeoff between robustness and efficiency -- harmful score is increased by a marginal 0.60 while the training time can be shortened by 50%, compared to the original Vaccine. 

2.  To show that our method can be **generalized to larger datasets**, we show additional experiments on WikiText-2 with a larger number of tokens. The results indicate that Vaccine can reduce harmful scores while maintaining the model's perplexity. 

3. We show further guidelines on **selecting  hyper-parameter $\rho$** by conducting experiments. The general trend is that with a larger $\rho$, the harmful score will decrease but at the cost of degrading finetune accuracy.  

4. We compare another **alignment-stage solution RepNoise** in terms of GPU memory and training time. Our experimental results reflect that Vaccine is superior to RepNoise regarding both GPU memory and training time. 

We hope that these efforts can fully address your concerns, and we are more than happy to discuss them with you!
Thank you for recognizing our effort in rebuttal, and also for increasing rating to reflect the addressed concern. We will definitely include the new results to the paper, and also open-source the accelerated Vaccine method.
Thank you for the clarification and additional experiments. The resource overhead of vaccine is smaller than I expected. The author should consider adding this content to the article and making it open source. I have decided to raise my score.
As pointed out by Reviewer CLDA, there is a typo in the Algorithm 2 in the rebuttal pdf. Line 4 in the algorithm should be "if $t$ mod $\tau$ == 0" instead of "if $t$ mod $\gamma$ == 0". In this algorithm, we basically do the perturbation every $\tau$ steps instead of per step, in order to accelerate vanilla Vaccine. We apologize for the confusion.
We thank the reviewer for acknowledging our efforts in rebuttal! Please feel free to leave us a comment if you need charification during our interaction with other reviewers.
We thank the reviewer for the encouraging comment, and also for positively adjusting the socre reflecting our efforts in rebuttal!

It is a a great relief for us to see that accelerated Vaccine address your main concern!  We will definitely include the accelerated Vaccine as well as its evaluation into the next revision. Indeed, there is a typo in Algorithm 2 of the attached PDF. It should be $t$ mod $\tau$==0. We will fix it in our revision. Thank you for pointing this out!

Please feel free to leave us a comment if you need charification during our interaction with other reviewers.
Thanks to the authors for clarifying my confusion. I am keeping my score unchanged for now.
Thank you for your detailed and constructive rebuttal, which includes extensive experimental evidence.

I am particularly impressed by your introduction of the **accelerated** Vaccine method. The results demonstrating only a 2% overhead when $\tau = 100$ with minimal performance degradation significantly reduce the application cost of the Vaccine method. I highly recommend including this improvement in the manuscript for future versions.

Based on my initial review and your comprehensive response, I decided to adjust my score positively.

P.S. There appears to be a typo in Algorithm 2 of the attached PDF: "t mod $\gamma$ == 0" should be reviewed for accuracy.

Best regards,