Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation

Authors: Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Ling Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Harmful fine-tuning, LLM, safety alignment

Abstract:
Harmful fine-tuning attack poses serious safety concerns for large language models' fine-tuning-as-a-service. While existing defenses have been proposed to mitigate the issue, their performances are still far away from satisfactory, and the root cause of the problem has not been fully recovered. To this end, we in this paper show that \textit{harmful perturbation} over the model weights could be a probable cause of alignment-broken. In order to attenuate the negative impact of harmful perturbation, we propose an alignment-stage solution, dubbed Booster. Technically, along with the original alignment loss,  we append a loss regularizer in the alignment stage's optimization. The regularizer ensures that the model's harmful loss reduction after the simulated harmful perturbation is attenuated, thereby mitigating the subsequent fine-tuning risk.     Empirical results show that Booster can effectively reduce the harmful score of the fine-tuned models while maintaining the performance of downstream tasks. Our code is available at https://github.com/git-disl/Booster

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729288826043

thread_type:
official_review

<thread_object>
creator: reviewer_9jvj

<content>
summary:
This work introduces Booster, an alignment-time method to attenuate harmful perturbations and mitigate the risks brought by harmful fine-tuning. They propose a loss regularize in the alignment tuning and combine it with the original alignment loss to optimize the LLM. They conduct experiments in several different benchmarks and find Boost consistently outperforms previous methods in terms of harmlessness and performance. Additionally, further analysis demonstrates Boost is robust, computationally efficient and compatible with other alignment-stage solution.

strengths:
* This paper is well-written, with appropriate tables and figures to demonstrate their idea and motivation.
* Alignment-time harmfulness prevention is quite interesting to me. This once-for-all method for harmful content prevention sounds promising and efficient.
* Boost demonstrates decent performance in various benchmarks and experiment settings.

weaknesses:
* Limited metrics. This work only reports harmful scores and fine-tuning accuracy. However, one intuitive limitation of alignment-time harmfulness prevention methods is they could hurt aligned LLMs' performance. The author should consider adding this experiment and testing the aligned LLMs with and without Boost directly.
* Section 3.2 is not convincing enough. The authors try to validate the concept of harmful perturbation in Section 3.2. However, the Figure 2 they used to demonstrate this is something too simple and not convincing enough.

questions:
I suggest adding one experiment to test the aligned LLMs with and without Boost directly, to demonstrate potential robustness or limitation of the alignment-time method against harmful fine-tuning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729915023289

thread_type:
official_review

<thread_object>
creator: reviewer_thxk

<content>
summary:
This paper proposes Booster, an alignment-stage method for defensing harmful fine-tuning attack. Harmful fine-tuning attack refers to the attack to fine-tune an aligned LLM with a dataset mixed with benign and adversarial instances, where the fine-tuned model will loss its safety alignment ability. The proposed method, Booster, use a harmful dataset during alignment stage to teach the LLM to attenuate the effects of the harmful samples in the fine-tuning dataset. This is done by a minimax loss. Based on experiments on four datasets and three LLMs, they show that the proposed method outperforms other baseline methods. Analysis are conducted to understand the effectiveness of the proposed method under different scenarios

strengths:
- This paper proposes a new method of defending harmful fine-tuning attacks at the SFT stage
- The proposed method is shown to be effective on the datasets evaluated in the paper
- Thorough analyses are conducted to understand how Booster works under different alignment and task-specific fine-tuning scenarios.

weaknesses:
- Section 3.2, which seems to be the motivation part of the paper, is not easy to follow for the following two reasons
    - The term *harmful score* is not properly defined
    - How the **Derived Insights** are derived from the observations is highly unclear. The causal relation between the first part of the sentence (*Because harmful fine-tuning data is considered to be inseparable from the benign data*) and the rest of the sentence (harmful perturbation is indeed inevitable in the user fine-tuning stage) cannot be justified by Figure 2. It is also unclear whether the experiment shown here relates to the proposed method.
- The experiment settings and results are weak. The paper uses SST2, AGNews, GSM8K, and AlpacaEval for the experiments. However, **all the experiments, except the experiments in Table 3, only reports the results of SST2**. Considering that SST2 is a very simple task for LLMs nowadays, only reporting the results for SST2 is a major weakness. Moreover, considering that GSM8K and AlpacaEval are more widely adopted for evaluating current LLMs, the results of **Booster in Table 3 are not convincing: Booster has a very high harmful score on AlpacaEval while Booster’s harmful score is not better than two baselines on GSM8K**. This makes me doubt the effectiveness of the proposed method on more challenging tasks.
- Presentation can be improved. In the first two paragraphs, the paper does not mention what is the dataset reported here. This makes it hard for me to evaluate the experiments at first. The notations with tilde are not defined in the paper.

questions:
What would be the results like if we directly remove the last term in equation (3)? (The term of gradient after the model takes one-step normalized update)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687964631

thread_type:
official_review

<thread_object>
creator: reviewer_6hjs

<content>
summary:
The paper addresses the vulnerability of fine-tuned large language models (LLMs) to harmful data, which can compromise their safety alignment and degrade service quality. It proposes a new alignment-stage solution called Booster, which introduces a regularizer to reduce the impact of harmful perturbation—where optimization over harmful data decreases the model's safety.

strengths:
1. The paper presents a novel approach, Booster, that effectively minimizes harmful perturbation during the alignment stage, thereby improving the safety and reliability of fine-tuned language models. The method is simple yet effective.

2. Its computational efficiency, requiring only three forward/backward passes per optimization step, makes it suitable for practical applications with frequent fine-tuning requests.

weaknesses:
The addition of a regularizer introduces trade-offs in terms of the balance between aligning the model and minimizing harmful loss. Finding the right balance can be challenging and might lead to varying results depending on the specific application or dataset.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712934050

thread_type:
official_review

<thread_object>
creator: reviewer_vynn

<content>
summary:
In this paper, the authors propose a method to alleviate the influence of attacking fine-tuning for breaking the LLM alignment. Specifically, the authors add a regularizer on the training loss so that the model can find an optimal point that keeps good performance while being robust to harmful fine-tuning --- not easy to fit the harmful data if trained on it, so-called harmful perturbation. The results show that the proposed method improves significantly over the baselines on harmful scores (e.g. 14.50 -> 4.80).

strengths:
1. The research topic on tackling harmful fine-tuning is important and timely because of the urgent need to ensure the trained LLM can resist alignment attacks.

2. The proposed approach is intuitive and clear, based on the clear definition of harmful perturbation. The experimental results improve significantly over the baselines, demonstrating the effectiveness of the proposed approach.

3. The writing is very clear and easy to follow. The formulas and pseudo-code clearly describe the algorithm and Figure 3 demonstrates how the proposed method works.

weaknesses:
1. In line 375, Booster initially has a relatively low harmful training loss. What is the reason for this? Does it mean that the model sees the harmful data in advance and trains it a little bit before the testing stage?

2. Adding more samples of the datasets can make it more clear how the model works and beat the baselines.

Overall, I think this paper is clear and technically sound.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731882774799

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author summary of initial review

comment:
We sincerely thank all the reviewers for the very constructive review comments. We see that our paper generally receives positive review comments, as we have a quite competitive initial rating **8(accept, good paper),8 (accept, good paper),8 (accept, good paper), 3 (reject, not good enough)**. Particularly, Reviewer vYNn suggests that "the proposed approach is intuitive and clear", reviewer 6Hjs also supports that "the method is simple yet effective", Reviewer Thxk points out that "Thorough analyses are conducted to understand how Booster works", and Reviewer 9Jvj suggests that our proposed "alignment-time harmfulness prevention is quite interesting".  

However, by the review comments, we do observe that there are several  concerns that we need to carefully address. We list the main concerns as follows:

---
**(Reviewer vYNn):** Authors should give more discussion on a strange phenomenon shown in Figure 3.

**(Author response):** We also observe such a phenomenon on the initial harmful loss in Figure 3 before our submission. While we have to admit that we can not give a fundamental reason for why this phenomenon happens, in this rebuttal we provide more information and data to explain how this phenomenon is happening. We also provide a conjecture/analysis based on the new data we provide. We looks forward to more discussion with the reviewer on this issue.    

---
**(Reviewer 6Hjs)**: The optimal setting of the hyper-parameter for the proposed regularizer might not be the same for all the fine-tuning tasks.

**(Author response):** Indeed, the optimal hyper-parameter settings for different downstream fine-tuning tasks are different, and for an alignment stage solution like Booster, its hyper-parameters should not be tailored specifically for a downstream task (because the same aligned model might be fine-tuned to many different downstream tasks). However, in our experiments, we set the same hyper-parameters of Booster for all the downstream tasks. We hope that this can at least give a more fair evaluation of how our method is performing when sub-optimal hyper-parameters are selected.    

---

**(Reviewer Thxk and Reviewer 9Jvj):** The motivation section is not intuitive enough and it is hard to see its connection with the later methodology design.

**(Author response):** By the feedback from the reviewers, we realize that the motivation section is not written that explicitly, thereby causing confusion. We have revised our illustration (and figure) through individual comments to the reviewers.   It would be very nice to receive further feedback from the reviewers and improve our paper.     

---
**(Reviewer Thxk)**: The main experiment is conducted with a  fine-tuning task SST2. Authors should do experiments on more complicated tasks, e.g., GSM8K. Also, Booster is not able to beat baselines (Vaccine, Lisa) in some settings.

**(Author response):** In our individual comments to the reviewer, we are able to show more experimental results on GSM8K.  In the new results, we do observe that the proposed method Booster has a slightly higher harmful score compared to Lisa (a baseline) in some settings, altough the advatage of Lisa comes with the fact that Lisa **significantly compromises the fine-tune accuracy**. To further address the concern, we show that Booster can actually be combined with Lisa to obtain an even better defense performance, and also preserve better fine-tune accuracy. We provide new results to show that the combined design Booster+Lisa **consistently outperforms Lisa** in **both the two considered metrics** (harmful score and fine-tune accuracy) in **all groups of experiments**. This remarkable result cannot be achieved without the design idea of Booster. Therefore, we insist that the contribution of Booster should not be downplayed. 

 ---
**(Reviewer 9Jvj)**: The authors should add one experiment to test the aligned LLMs with and without Boost directly

**(Author response):**  It is indeed interesting to study whether an alignment stage solution (e.g. Booster) will hurt the general performance of the aligned LLM. We have provided new result in the individual comments to the reviewer. We look forward to more discussion on the results. 

---
All of these comments significantly help us increase the quality of the paper, and we will give a more detailed response to reviewers' concerns in individual comments to them. We look forward to more insightful discussion during the rebuttal period.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an alignment-stage method, Booster, to defend against harmful fine-tuning attack by adding a loss regularizer in the alignment stage's optimization.  Empirical results show that Booster can effectively reduce the harmful score of the fine-tuned models while maintaining the performance on downstream tasks. The proposed method is interesting and technically sound, and it works well on multiple datasets and LLMs. The paper is generally well written.  Most of the issues raised by the reviewers have been addressed in the author's rebuttal. In the end, the reviewers were unanimously positive about this paper. The new experimental results reported in the rebuttal need to be added in the final version.