Title: Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
Venue: NeurIPS 2025
Accepted: yes
Keywords: Harmful fine-tuning; Large Language Models; Data schedule; Bayesian optimization
Abstract: Harmful fine-tuning poses critical safety risks to fine-tuning-as-a-service for large language models. Existing defense strategies preemptively build robustness via attack simulation but suffer from fundamental limitations: (i) the infeasibility of extending attack simulations beyond bounded threat models due to the inherent difficulty of anticipating unknown attacks, and (ii) limited adaptability to varying attack settings, as simulation fails to capture their variability and complexity. To address these challenges, we propose Bayesian Data Scheduler (BDS), an adaptive tuning-stage defense strategy with no need for attack simulation. BDS formulates harmful fine-tuning defense as a Bayesian inference problem, learning the posterior distribution of each data point's safety attribute, conditioned on the fine-tuning and alignment datasets. The fine-tuning process is then constrained by weighting data with their safety attributes sampled from the posterior, thus mitigating the influence of harmful data. By leveraging the post hoc nature of Bayesian inference, the posterior is conditioned on the fine-tuning dataset, enabling BDS to tailor its defense to the specific dataset, thereby achieving adaptive defense. Furthermore, we introduce a neural scheduler based on amortized Bayesian learning, enabling efficient transfer to new data without retraining. Comprehensive results across diverse attack and defense settings demonstrate the state-of-the-art performance of our approach. Code is available at https://github.com/Egg-Hu/Bayesian-Data-Scheduler.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the emerging threat of “harmful fine-tuning,” whereby adversarially crafted data in a user’s fine-tuning set can subvert an otherwise aligned large language model. To defend against this without needing to simulate potential attacks, the authors propose the Bayesian Data Scheduler (BDS), which treats each tuning example’s safety as a latent variable and infers a posterior distribution over data weights conditioned on both the fine-tuning and a small alignment dataset. By reweighting data in the fine-tuning process—either via a Bayesian Scalar Scheduler or an Amortized Bayesian Neural Scheduler—BDS adaptively downweights harmful examples and upweights benign ones. Extensive experiments across multiple datasets, model architectures, and attack scenarios show that BDS achieves state-of-the-art robustness, reducing harmful behavior by over 70% at high attack ratios while maintaining task accuracy, and it can efficiently transfer to new data without retraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is clearly written and generally easy to follow.\n2. It includes comparisons with multiple existing baselines.\n3. The proposed methods are supported by clear and well-structured derivations.\n\nWeaknesses:\n1. I strongly encourage the authors to include at least a minimal discussion of related works in the main text. This would help clarify how the proposed method differs from or builds upon prior approaches and provide better context for evaluating the novelty and contribution of each component. As it stands, it's difficult to assess which parts of the method are original and which are inspired by existing work.\n\n2. The proposed method appears computationally expensive, especially considering that the largest fine-tuning dataset used in experiments contains only 2,000 examples—much smaller than typical fine-tuning workloads. Can the authors clarify the efficiency of BDS in more realistic settings? It would be more compelling if the method could support a practical deployment scenario where a fine-tuning service provider trains a single BDS model once and reuses it across different user-provided datasets. The paper would be significantly stronger if it could demonstrate either (1) that BDS is lightweight enough to train (within very few minutes) separately for each new dataset query, or (2) that a pre-trained BDS model generalizes well to different user-provided datasets containing harmful examples.\n\nQUESTIONS:\nPlease refer to the weaknesses outlined above. Regarding point 1, I encourage the authors to clarify how their method differs from prior work and to clearly highlight which components are original. My primary concern is point 2: the method appears computationally expensive, and the current analysis is based on theoretical complexity rather than practical runtime. It would be much more convincing if the authors could report actual runtime measurements in realistic settings. That said, I appreciate the thorough experimental evaluation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new and principled approach called the Bayesian Data Scheduler (BDS) to protect large language models from harmful fine-tuning. BDS reframes the challenge of defending against such fine-tuning as a Bayesian inference task, estimating the posterior probability of each data point's safety based on both the fine-tuning and alignment datasets. During fine-tuning, data are reweighted according to safety scores drawn from this posterior, reducing the impact of potentially harmful inputs. Because Bayesian inference is applied post hoc, the posterior adapts to the specific fine-tuning data, allowing BDS to provide a tailored and adaptive defense. Additionally, they propose a neural scheduler using amortized Bayesian learning, which allows for efficient generalization to new datasets without the need for retraining. Extensive experiments under various attack and defense scenarios show that BDS achieves leading performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The studied research problem is important, as both unintentional and malicious harmful fine-tuning data are increasingly pervasive in fine-tuning-as-a-service scenarios. \n\n* The proposed fine-tuning data reweighting method is theoretically sound, grounded in Bayesian statistics. The amortized version of BDS can be efficiently integrated into standard LLM fine-tuning pipeline. \n\n* The authors performed thourough evaluations against various attacks and defense strategies using multiple LLMs and benchmarks, which together demonstrates the superiority of BDS in balancing the utility-safety trade-off during LLM fine-tuning. \n\n\nWeaknesses:\n\n* The iterative posterior sampling process of BDS would introduce additional computational overhead. Have the authors considered comparing BDS against a pre-trained soft safety classifier of input data?\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Bayesian Data Scheduler (BDS), a defense mechanism against harmful fine-tuning of large language models (LLMs), BDS formulates the defense as a Bayesian inference problem, adaptively estimating the risk of each training example.It introduces Bayesian Scalar Scheduler and Amortized Bayesian Neural Scheduler to reweigh samples based on posterior probabilities during fine-tuning. Experiments demonstrate the SOTA performance and strong adaptiveness of BDS, showing superior effectiveness across across diverse attack scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nS1: BDS is the first to treat defense against harmful fine-tuning as Bayesian inference, avoiding the need for simulating attack data.\n\nS2: The paper introduces two schedulers: a scalar scheduler for high accuracy and a neural scheduler with cross-task transferability, improving the method’s generalizability.\n\nS3: The experiments are comprehensive, covering diverse models, datasets, and attack scenarios. BDS consistently outperforms existing methods, demonstrating strong robustness and alignment performance.\n\nW1: Although the paper adopts SGLD for posterior sampling, this approach may introduce training speed bottlenecks in large-scale deployment. However, the experiments do not include detailed comparisons of training time or resource consumption.\n\nW2: The study relies on high-quality aligned data (Dsafe), assuming that the platform has access to such datasets. However, in practice, such data may be scarce, especially for domain-specific tasks.\n\nW3: While the experiments demonstrate broad coverage, the defense mechanism under black-box attack scenarios lacks detailed analysis. The paper does not systematically discuss discrepancies between the attacker’s and defender’s models—for example, whether BDS remains effective when the attacker uses different LLM architectures or training objectives.\n\nQUESTIONS:\nThe overall evaluation is based primarily on S1-S3 (+) and W1–W3 (–) from the Strengths and Weaknesses section. Below are suggestions for the authors to improve the work:\n\n1. Add a table comparing training time, memory and GPU overhead, and FLOPs with standard optimizers (e.g., Adam, SGHMC) on the same task to clarify the resource cost of SGLD. Additionally, supplement the time complexity analysis in Appendix G.1 with concrete benchmarks, such as per-epoch runtime and the number of steps to convergence. (W1)\n\n2. Consider adding a performance analysis for low-resource Dsafe settings (e.g., reducing from 1000 to 100/50), and discuss whether a “safe attribute migration” policy could help amortized schedulers generalize with minimally aligned data. (W2)\n\n3. Although current experiments assume similar model architectures between attacker and defender, we acknowledge real-world attackers may utilize heterogeneous LLMs. Future work could examine the impact of such cross-model scenarios on BDS's adaptiveness, particularly when alignment objectives diverge. (W3).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper first formulate the harmful fine-tuning defense as a Bayesian inference problem. Under this definition, the authors propose a novel Bayesian Data Scheduler Framework which can adaptively defense the harmful fine-tuning attacks without attack simulation. Comprehensive experiments demonstrate the SOTA performance achieve by their method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well written with clear description of their methods. 
2. To demonstrate their claims, the authors conduct numerous experiment in Section 4.1.
3. The theoretical analysis provides intuiation and guarantee for their proposed method.
4. Compared to previous methods, BDS does not require attack simulation, which is more practical.

Weaknesses:
1. The figure 1 requires clearer explanation to help reader better understand the density and weights concept as the introduction figure in the paper.
2. Some important information for understanding the methods, such as experiments details, should be directly included into the paper rather than in the appendix.

QUESTIONS:
The method is still based on a datset with safe examples (harmful questions with safe answers). The model may perform not well on unexpected attacks with harmfulness beyond the alignment dataset.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method by which, using alignment data alone, a scheduler can be constructed that weights each data sample based on the likelihood of the safety weights under modeling a constraint dataset. A wider array of experiments show that this method is both robust (for example to domain shifts, sample sizes, and attack strengths) and better than previous methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nThe paper is clearly a novel approach to solving the harmful fine-tuning problem for methods behind closed APIs which is both tractable, well motivated, and demonstrated to be strongly robust under a wide variety of experimental variations.\n\n# Weaknesses\n\n## Generality\n\nThe primary weakness of this paper is that adaptive bayesian scheduling for loss weights could be used much more widely as an adaptive constraint method such as in the case of having length, style, persona, factuality, or quality constraints.  I understand that safety is more motivating but presenting this method in a way limited to safety limits the (presented) applicability of this method to the community when it seems quite promising. Perhaps this could be improved by formulating a few scenarios where Dft violates some constraint in Dconstraint where constraint is a stylistic preference or something like this. I am on borderline accept due to this limitation of significance but happy to raise scores if either a plan for a wider scope is presented or argumentation on why the work was kept so limited is presented.\n\n## Adaptive Attacks\n\nThe method doesn’t present an adaptive attack where the attacker knowing the defence method or (stronger) having access to the scheduler may construct samples whose weights allow training. The most obvious is through data filtration over harmful datasets for samples whose weights allow training. Can the authors please include this adaptive attack experiment and if there are any other adaptive attacks to discuss them. It isn’t implausible that a highly motivated attacker might infer this is the defence used and construct a proxy scheduler, which due to your own OOD and generalization results, would be able to help them choose samples which break this defence. Clearly the distribution figures that are presented demonstrate that this would be possible in theory. After presenting this adaptive attack analysis, I’d like the authors to present strategies for mitigating this attack for example through adjusting the scheduler with contrastive examples. This is counter to the authors narrative about intractability of attack simulation but I find it reasonable. I am on borderline accept due to the lack of presenting adaptive attacks.\n\n## Mathematical Presentation\n\nI am not sure how to provide concrete suggestions here but the mathematical presentation and results were quite difficult to get through without a background in Bayesian Statistics. That might be a limitation that all readers without this background encounter but I found it quite hard to motivate the mathematical results and convince myself they were needed. Perhaps a re-organized presentation where the reader is guided more slowly through the presentation without assuming knowledge of bayesian statistics. Consider that security, policy, and less technical safety folks will be reading this work.\n\n## D.10 Offline-Based Methods\n\nI am not sure how fair using RepNoise is given that it’s an explicitly an offline method (though an online variant is discussed in [1], it might be more fair to move RepNoise to D.10 if it was used in an offline fashion or explicitly mention it was used online as per [1]\n\n[1] Rosati, D., Edkins, G., Raj, H., Atanasov, D., Majumdar, S., Rajendran, J., ... & Sajjad, H. (2024). Evaluating Defences against Unsafe Feedback in RLHF.\n\n## F.5 Over refusal\n\nWhile I don’t expect the method to introduce partial refusals, I recommend the authors report the entire dataset (Refusal-Safe, Partial Refusal-Safe, Answer-Safe, Answer-Unsafe, Partial Answer-Unsafe, Refusal-Unsafe)\n\n## Note on infeasability of attack simulations\n\nI disagree that attack simulations under your threat model are infeasible. Surely, distributional and sample complexity guarantees can be constructed for methods in the future if we are simply talking about constructing a defence that defends against number of samples in a harmful dataset. If the defence is circumscribed to a particular harmful distribution and generalization guarantees are attained then there is feasibility. Of course you are correct to point out that the defender might not be able to anticipate attacks outside of this threat model but I am not sure previous defences are claiming this. I’d recommend making your argument here more nuanced.\n\nQUESTIONS:\nMentioned above in strengths and weakneses with clear criteria.\n\nLIMITATIONS:\nNegative societal impact of the work is not adequetly discussed. This work is dual-use in that constraint scheduler methods might be used to enforce a particular ideological position or otherwise harm freedoms of fine-tuning a model with diverse perspectives. Since that is the case the authors should make it clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}