Title: Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Harmful fine-tuning

LLM

Safety alignment
Abstract: Harmful fine-tuning attack introduces significant security risks to the fine-tuning services. Main-stream defenses aim to vaccinate the model such that the later harmful fine-tuning attack is less effective. However, our evaluation results show that such defenses are fragile-- with a few fine-tuning steps, the model still can learn the harmful knowledge. To this end, we do further experiment and find that an embarrassingly simple solution-- adding purely random perturbations to the fine-tuned model, can recover the model from harmful behaviors, though it leads to a degradation in the model’s fine-tuning performance. To address the degradation of fine-tuning performance, we further propose \methodname, which optimizes an adaptive perturbation that will be applied to the model after fine-tuning. \methodname maintains model's safety alignment performance without compromising downstream fine-tuning performance. Comprehensive experiments are conducted on different harmful ratios, fine-tuning tasks and mainstream LLMs, where the average harmful scores are reduced by up-to 21.2%, while maintaining fine-tuning performance. As a by-product, we analyze the adaptive perturbation and show that different layers in various LLMs have distinct safety coefficients. Source code available at https://github.com/w-yibo/Panacea.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the vulnerability of large language models (LLMs) to harmful fine-tuning attacks and the limitations of existing defenses. While current defenses attempt to \"vaccinate\" models against future harmful fine-tuning, the authors find these approaches to be fragile—models can still learn harmful behaviors after just a few fine-tuning steps. To counter this, they first demonstrate that simply adding random perturbations to the fine-tuned model can mitigate harmful behaviors, albeit at the cost of performance. To overcome this trade-off, they propose Panacea, an adaptive perturbation method applied post fine-tuning. Panacea effectively preserves both model safety and downstream task performance. Extensive experiments across various LLMs, harmful data ratios, and tasks show that Panacea reduces harmfulness scores by up to 21.2% while maintaining fine-tuning accuracy. Additionally, analysis reveals that different layers in LLMs exhibit varying degrees of safety sensitivity, aligning with prior research findings.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. This paper introduces a \"post-fine-tuning\" defense method called Panacea, which is fundamentally different from most existing research (such as Vaccine and RepNoise) that focuses on defense during the \"alignment stage\". This approach offers greater practicality as it can be applied to models that have already been fine-tuned, including publicly released, pre-aligned large language models like Llama2-Chat.\n2. The paper's experimental design is highly detailed and persuasive. The authors validate Panacea's effectiveness and generalization capabilities not only across various harmful data ratios but also across four different downstream fine-tuning tasks (GSM8K, SST2, AlpacaEval, AGNEWS), and three mainstream large language models (Llama2-7B, Gemma2-9B, Qwen2-7B). Additionally, they perform fair comparisons against multiple state-of-the-art baselines.\n3. This paper offers a deep analysis beyond just performance metrics. Ablation studies demonstrate that the proposed \"adaptive perturbation\" is the key to mitigating the model's harmful effects. Crucially, by visualizing the perturbation's layer-wise weights, the authors reveal that different LLMs have distinct safety-critical layers. This finding is thoroughly compared with existing literature, enhancing its credibility and providing valuable insights into LLM safety mechanisms.\n\nCons:\n1. It lacks discussion and comparison with the most recent works. There has been a growing number of recent works related to LLM alignment, such as [1], [2]. Although [1] is not primarily focused on fine-tuning attacks, their proposed method can mitigate the impact of harmful fine-tuning, and therefore it should be compared with.\n2. The benchmark used for evaluating the harmfulness score is outdated. In this paper, all safety-related data are derived from BeaverTails. Although there is no overlap in usage, I believe that the evaluation of the harmfulness score would benefit from incorporating more recent benchmarks such as SORRY-Bench [3] or TrustLLM [4]. Otherwise, I have concerns about the reliability of the harmfulness evaluation, as it is based entirely on the same dataset, which may bias the results in favor of the defense. Additionally, newer benchmarks cover a broader range of harmful queries and can provide a more comprehensive assessment of the model’s alignment.\n3. A key concern regarding the implementation is that the harmful data used during the defense phase and the harmful data used for fine-tuning in the attack phase come from the same source, even though the specific content does not overlap. This setup is somewhat unrealistic and may artificially boost the defense performance due to the shared data origin. To better reflect real-world scenarios, it would be more appropriate to use harmful data from different sources. Moreover, as mentioned earlier, the harmfulness score (HS) is also evaluated using the same dataset, which may further limit the effectiveness and reliability of the HS assessment.\n\n\n[1] Safety Alignment Should be Made More Than Just a Few Tokens Deep. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson. ICLR 2025.\n\n[2] SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation. Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, Yisen Wang. ICLR 2025.\n\n[3] SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal. ICLR 2025.\n\n[4] TrustLLM: Trustworthiness in Large Language Models. Yue Huang et al. ICML 2024.\n\nQUESTIONS:\n1. In Panacea, how much time difference is there between the fine-tuning phase during the attack and the post-fine-tuning phase during the defense? \n2. In the Panacea algorithm, fine-tuning and harmful data are used to identify noise that mitigates harmful effects. Given this, would it also be effective if we directly fine-tune the model using a mixture of the harmful fine-tuning data and safety instruction data (such as [4])? That is, instead of applying a post-fine-tuning defense, the defender proactively incorporates safety instruction data during the fine-tuning process.\n\n\n[1] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. ICLR 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This work explores post-fine-tuning methods to guard language models from fine-tuning attacks. They find that adding noise to fine-tuned models reduces the expression of harmful behaviors, although at a cost to fine-tuning performance. The authors instead propose Panacea, a method that maximizes the increase in loss on harmful behavior caused by perturbations, finding that it preserves fine-tuning performance better while reducing harmfulness.

STRENGTHS AND WEAKNESSES:
Strengths
- The method is well explained and motivated.
- The baselines to compare against are clear, and the experimental results are presented well.
- The results are positive -- reduction in harmful score is greater than other methods while improving accuracy over vanilla fine-tuning.
- The method can be applied to models that have already been fine-tuned, without needing interventions during the fine-tuning stage.

Weaknesses
- The improvement over other methods (particularly Antidote) is relatively marginal
- I would like to see a description of the dataset and how each different harm ratio is constructed.
- I'm not sure if the layer-wise safety property analysis is very meaningful. Each model has a different pattern so I'm not convinced we can draw conclusions from this experiment.
- The grammar could use another pass to fix some issues.

QUESTIONS:
- How is it possible to have a non-zero harmful score on the Clean dataset? Is it re-labeled and found to actually contain harmful requests?
- What do the harmful and non-harmful items in the dataset look like respectively? What non-harmful tasks are the model being evaluated on, and what forms of harm are represented in harmful requests?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a post-fine-tuning strategy to defend against harmful fine-tuning attacks where a small amount of harmful data is included in downstream fine-tuning data. The strategy essentially finds a perturbation in the model’s weight space that makes the model hard to optimize the prediction loss on the harmful data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This work conducts sufficient experiments to demonstrate the effectiveness of the proposed method across four fine-tuning tasks and several LLMs.\n- The paper is well-written, organized, and easy to follow. Code is provided to improve reproducibility.\n\nWeaknesses:\n- The assumption that a harmful dataset is provided in this post-fine-tuning setup is too strong, which also seems to weaken the claim that harmful data is difficult for users to detect (Line 31-32) as mentioned in the introduction. If this harmful dataset is given, why don’t you use this dataset to filter out the harmful data mixed in the fine-tuning dataset, and use the cleaned fine-tuning dataset for fine-tuning? Isn’t that more effective than this defense strategy?\n- Moreover, each of the four fine-tuning datasets is mixed with a certain percentage of harmful data from BeaverTails. This data setup is also questionable. If the fine-tuning data and the harmful data are from two sources, which means they follow different distributions, we can still use some out-of-distribution detection methods to identify the harmful data and filter them out. This again weakens the claim that harmful data is hard to detect. A more realistic setup would be that the harmful data are modified from the fine-tuning data. For example, given benign fine-tuning data (benign instruction-benign response pairs), change the benign response into a harmful response to construct a harmful dataset.\n\nQUESTIONS:\n- In Eq. (1), why is the second loss term $h(w)$ needed? In Appendix A, the closed-form calculation doesn’t need the second loss term. In fact, if the fine-tuning data contains a certain amount of harmful data from the same distribution, minimizing the third loss term $g(w)$ will minimize the harmful loss anyway. \n- Besides, what if you only $\\max_w \\lambda h(w) - g(w)$? Essentially, the model is trained to maximize the loss on harmful data while minimizing the loss on benign fine-tuning data. \n- From Fig. 6, the training process essentially makes the harmful training loss slightly harder to decrease than SFT. But when $\\lambda$ increases (0.01), why does the fine-tuning accuracy on the benign data degrade as reported in Table 8?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPanacea is a post-fine-tuning fix: after a model has been poisoned, it learns a tiny, norm-bounded weight perturbation that “pushes up” harmful loss yet keeps task loss flat, found in one gradient loop through a max–max objective.\nAcross Llama-2-7B, Gemma-9B and Qwen-7B, this perturbation cuts the Harmful Score by ≈21 % on average while even nudging fine-tuning accuracy up (+0.4 %) over vanilla SFT. Because it is applied only once and adds no runtime prompt overhead, it outlasts earlier “vaccination” defenses whose safety erodes after a few more fine-tuning steps.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Post-hoc and deployment-friendly: fixes a poisoned model after fine-tuning with a single extra training pass and zero inference-time overhead (no extra prompts or guardrails), so it slots neatly into “fine-tune-as-a-service” pipelines.\n\n2. Model- and task-agnostic – validated on three architectures (Llama-2, Gemma, Qwen), four diverse tasks, and on already-aligned variants, showing broad applicability.\n\n\nWeakness:\n1. The biggest concern is harmful distribution leakage. Train–test distribution overlap may inflate gains. Consider to use SafetyBench, AdvBench as train or eval and also different type of harmful queries between train and test. \n\n2. Extra computes required. After normal user fine-tuning, A second 20 epoch training loop is needed (this number is far from normal training epoch e.g. like 3 epochs.) \n\n4. Missing related work. [1][2]. These methods should be compared on effectiveness. \n\n[1]  Qi, Xiangyu, et al. \"Fine-tuning aligned language models compromises safety, even when users do not intend to!.\" arXiv preprint arXiv:2310.03693 (2023).\n\n[2] Wang, Jiongxiao, et al. \"Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment.\" arXiv preprint arXiv:2402.14968 (2024).\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}