Title: Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Security

Jailbreak

Fine-tuning
Abstract: Despite substantial efforts in safety alignment, recent research indicates that Large
Language Models (LLMs) remain highly susceptible to jailbreak attacks. Among
these attacks, finetuning-based ones that compromise LLMs’ safety alignment via
fine-tuning stand out due to its stable jailbreak performance. In particular, a recent
study indicates that fine-tuning with as few as 10 harmful question-answer (QA)
pairs can lead to successful jailbreaking across various harmful questions. However,
such malicious fine-tuning attacks are readily detectable and hence thwarted by
moderation models. In this paper, we demonstrate that LLMs can be jailbroken
by fine-tuning with only 10 benign QA pairs; our attack exploits the increased
sensitivity of LLMs to fine-tuning data after being overfitted. Specifically, our
fine-tuning process starts with overfitting an LLM via fine-tuning with benign QA
pairs involving identical refusal answers. Further fine-tuning is then performed
with standard benign answers, causing the overfitted LLM to forget the refusal
attitude and thus provide compliant answers regardless of the harmfulness of a
question. We implement our attack on the ten LLMs and compare it with five
existing baselines. Experiments demonstrate that our method achieves significant
advantages in both attack effectiveness and attack stealth. Our findings expose
previously unreported security vulnerabilities in current LLMs and provide a new
perspective on understanding how LLMs’ security is compromised, even with
benign fine-tuning. Our code is available at https://github.com/ZHIXINXIE/ten_benign.git.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a jailbreak attack targeting safety-aligned large language models (LLMs). The method fine-tunes an LLM using only 10 fully benign question–answer (QA) pairs in a two-stage process. In the first stage, the model is overfitted to always refuse to answer any question by training on benign questions paired with refusal responses. In the second stage, the same benign questions are paired with informative, standard answers, encouraging the model to “forget” its refusal behavior, even for harmful queries. The key insight is that overfitting increases the model’s sensitivity, so that even a small, benign fine-tuning dataset can cause catastrophic forgetting of safety alignment. Experiments show this method achieves a high attack success rate and strong stealthiness, outperforming several existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed method leverages overfitting and benign QA pairs to perform a jailbreak attack, which is quite interesting. Empirical results demonstrate that the method achieves good performance.\n- The paper uses loss landscape visualization and gradient analysis to explain how overfitting enables the attack, adding to its interpretability.\n\nWeaknesses:\n\n- The definition of \"benign\" in the context of fine-tuning data is somewhat narrow, focusing solely on the questions rather than the responses.\n- The \"refusal\" fine-tuning data can be easily detected, making the attack easy to defend against when model owners are aware of the proposed method.\n- The evaluation metrics primarily focus on attack success rate and stealthiness, neglecting the model's utility post-attack, such as the diversity of generated answers.\n\nQUESTIONS:\nThe authors claim the fine-tuning data is totally benign, but \"benign\" should consider not just the questions but also the responses. In my view, forcing the model to always refuse any question is itself a form of harmful behavior. Prior poisoning attacks have shown a similar idea in “availability poisoning,” where the model is made to fail on certain tasks rather than being backdoored. I see this method as a kind of availability poisoning.\n\nSetting aside whether the fine-tuning data is truly benign, the reason the authors use benign data is to evade detection. However, given how small and unusual this data is, it could be easy to detect, even by simple human inspection, without sophisticated defenses.\n\nThe authors measure the performance of the proposed attack using HS, ASR, and DR. It can reflect the attack performance, but it does not reflect the model's utility. The authors should also measure the model's utility after the attack, such as the diversity of the generated answers, and compare it with that of an unmodified model. It seems that the model’s responses become more similar to the stage-2 data, which may harm its general usefulness.\n\nFinally, since the attack’s success is highly sensitive to hyperparameters, the authors should give more practical guidance on how to select these hyperparameters in a real-world setting.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an interesting method to jailbreak a model’s internal safety mechanisms (inbuilt guardrails) by fine-tuning it on just 10 benign examples. The attack success rate is impressive—exceeding 90% and approaching the effectiveness of methods using malicious instructions. The core of the approach lies in how the dataset is constructed and how fine-tuning is performed, drawing inspiration from the effectiveness of the AOA attack. The attack process is divided into two stages:\n\n- (1) Fine-tune the model on a refusal dataset, where the model generates the same refusal response regardless of the question or prompt.\n\n- (2) Fine-tune the model on correct answers, aiming to break the association between refusal responses and any type of question, including harmful prompts.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well written and sufficiently detailed to situate it appropriately within the existing literature. I appreciate the quality of the discussions and how a simple idea is developed into convincing arguments, supported by empirical evidence.\n\nThere are several weaknesses and questions I would like to raise:\n\n- Breaking the model’s association between question type and refusal behavior may impact its overall performance on other benchmarks. I would be interested to see experiments on datasets such as MMLU and other utility benchmarks to better understand how this attack affects the model’s general performance.\n\n- What happens if the questions used in stage 2 do not align with those in stage 1?\n\n- Is it possible to detect the proposed attack by identifying whether all samples in stage 1 fine-tuning have been tuned to produce the same refusal response?\n\nAside from this, I hesitate to strongly recommend acceptance of the paper, primarily because the proposed attack is largely derived from AOA, which limits its novelty. The main contribution lies in analyzing why AOA is effective and validating this analysis—the proposed attack is essentially a byproduct of that investigation.\n\nQUESTIONS:\nPlease see the strenght and weakness section above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces two-stage benign fine-tuning-based jailbreaking method that uses only 10 benign QA pairs by first overfitting the model on identical refusal answers, then fine-tuning on standard benign responses to provoke catastrophic forgetting of refusal behavior. Empirical evaluations on 10 LLMs against five baselines show that this approach matches or exceeds the attack success and stealth of malicious fine-tuning, despite containing no explicitly harmful content. The paper also reports an interesting observation that the jailbreak's potency hinges on the high similarity and contiguous ordering of the initial examples such that shuffling or paraphrasing them dramatically reduces attack effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper reads well.\n\n2. The design of the proposed method is straightforward, which highlights how fragile current alignment techniques are.\n\n3. The paper presents an interesting finding: when similar data points are used first for fine-tuning, they have a stronger impact on jailbreaking.\n\n4. The paper includes relevant and informative ablation studies that demonstrate the impact of each stage of the proposed jailbreaking technique.\n\n\n**Weaknesses:**\n\n1. The Harmful Score (HS) has similar limitations to those found in LLM-as-a-judge scenarios, but these issues are neither addressed nor discussed in the paper. These limitations include:\n\n    1.1. Prompt 1, which is used to generate the HS values, is unreliable. It has been shown that LLMs tend not to give extreme scores (either very good or very bad). This behavior automatically skews the actual distribution of judgments.\n\n    1.2. Additionally, in LLM-as-a-judge scenarios, it is important to have inter-rater agreement between the model and expert annotators. This agreement is missing in this study.\n\n    1.3. The HS metric depends on both the model and the prompt. This means that all HS values shown in Table 1 could change if a different model or prompt is used. Therefore, the metric is not universal and may not be possible for others to use in the future, especially if the original model is unavailable or the prompt is updated to include more detailed or customized evaluations.\n\n2. Although the paper uses a wide range of models for evaluation, this evaluation is based only on a single dataset. This limits the generalization of the proposed approach.\n\n3. Related to the previous point, it is unclear whether the 10 QA data points used to jailbreak the models would be equally effective on other datasets. This uncertainty is especially relevant given that only a small number of data points were used during the fine-tuning process. Since the study does not test different sets of data points, it is not possible to determine whether the success is due to the proposed two-stage strategy or simply the specific data points chosen for the jailbreak.\n\n\n**Minor Comment:**\n\n1. The content of Prompt 1 is not well-organized. It appears that there are missing spaces between each type of score.\n\nQUESTIONS:\n1. In lines 115–116, the paper claims that \"shuffling does not influence the overall semantic meaning of the datasets.\" However, Figure 2(a) shows a significant difference in HS between the original dataset and the shuffled version, even though HS is not order-dependent. How do you explain this discrepancy?\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage fine-tuning attack that jailbreaks LLMs using only 10 benign QA pairs. \nStage 1 induces overfitting by fine-tuning on identical refusal answers, while Stage 2 overwrites them with standard answers, leading to safety alignment forgetting. \nThe attack achieves high success and stealth across ten LLMs, matching malicious fine-tuning without using harmful data. The authors further analyze loss landscapes to discuss on the reported counter-intuitive phenomenon.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper offers both conceptual novelty and practical value, and is likely to inspire follow-up work in LLM safety and robustness.\n* The paper presents a novel attack paradigm that leverages Fine-tuning-as-a-Service to first overfit an LLM on benign refusal answers, then fine-tunes it back with standard responses, resulting in effective jailbreaking.\n* It addresses a concrete and impactful challenge: how to perform malicious fine-tuning in a stealthy, efficient, and even black-box-compatible manner.\n* The method design is well-motivated.\n* The pilot experiment in Section 2.2 is fascinating; the analysis on data shuffling and the reformulation of AOA into a benign attack path is both insightful and compelling.\n\nWeakness \n1. (Minor) It is suggested that PAIR method prioritize mere non-refusal attitude (Line 83) like GCG method within the attack performance evaluation. \nHowever, according to their corresponding paper, GCG adopts the `refusal keyword detection` paradigm, which aligns with the author's description, while PAIR adopts a LLM-as-a-judge paradigm, where trustworthy and rigorous evaluation could be ensured. Thought this is a minor point, further clarification should be made about this.\n\n2. (Minor) The selection of judge model is under no further justification or ablation. It is reported to use gpt-4.1-mini to evaluate both HS (Harmful Score) and ASR (Attack Success Rate). \nThe curation process of both Prompt 1 and Prompt 2 is also under-documented and lacks empirical justification.\nFor example, a certain scale of human annotation is expected to justify the effectiveness and trustworthiness of the selected judging model as well as judging prompt.\n\n3. (Minor) Though the story-telling is intriguing and easy-to-follow, the overall manuscript would benefit from further re-organization. \n- For example, it seems both Subsection 2.2 and Subsection 2.3.1 falls into the motivation part, while the following of Section 2 is then related to method. It is suggested to leave a standalone \"Methodology\" Section, e.g. \"Section 3 Methodology\", and reformulate the insight justification in \"Section 2 Backgroud and Motivation\".\n- Another example is the original \"Section 3 Explanation of Our Attack\", which provides in-depth discussion about the \"overfit-to-jailbreak\" phenomenon, and would benefit from re-organize to merge with original Subsection 4.5 into one standalone Discussion Section after the presentation of Experiment Section.\n\n4. The reported ASR is tested on the AdvBench (Line 261). However, there exist concerns [1-5] about the jailbreak benchmark selection. It is encouraged to report original and \"benign-overfit-fine-tuned\" version of different base models upon more diverse and extensive datasets [1-3].\n\n5. More advance jailbreak attacks [6], which might fall outside the scope of the fine-tuning based attacks, shall also be compared, or at least discussed.\n\n6. (Minor) Though some discussion is reported in Section 4.4 to demonstrate the \"shallow align\" defense won't work against the proposed method, the paper will further benefit from discussing or proposing any potential effective mitigation or adaptive defense to counter down the proposed \"benign overfit fine-tune\" attack.\nAlso, more details of this impactful work is also expected to include in the Appendix, as this is one crucial component of the text body.\n\n---\n\n[1] Chao, Patrick, et al. \"Jailbreakbench: An open robustness benchmark for jailbreaking large language models.\" arXiv preprint arXiv:2404.01318 (2024).  \n[2] Xie, Tinghao, et al. \"Sorry-bench: Systematically evaluating large language model safety refusal behaviors.\" arXiv preprint arXiv:2406.14598 (2024).  \n[3] Zeng, Yi, et al. \"Air-bench 2024: A safety benchmark based on risk categories from regulations and policies.\" arXiv preprint arXiv:2407.17436 (2024).  \n[4] Beyer, Tim, et al. \"Llm-safety evaluations lack robustness.\" arXiv preprint arXiv:2503.02574 (2025).  \n[5] Chu, Junjie, et al. \"JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs.\" arXiv preprint arXiv:2402.05668 (2024).  \n[6] Andriushchenko, Maksym, Francesco Croce, and Nicolas Flammarion. \"Jailbreaking leading safety-aligned llms with simple adaptive attacks.\" arXiv preprint arXiv:2404.02151 (2024).\n[7] Liu, Xiaogeng, et al. \"Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms.\" arXiv preprint arXiv:2410.05295 (2024).\n\nQUESTIONS:\nAs above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}