Title: No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms
Venue: NeurIPS 2025
Accepted: no
Keywords: attacks

fine-tuning attacks

data poisoning

a few tokens deep
Abstract: Leading language model (LM) providers like OpenAI and Anthopic allow customers to fine-tune frontier LMs for specific use cases. To prevent abuse, these providers apply filters to block fine-tuning on overtly harmful data. In this setting, we make three contributions: First, while past work has shown that safety alignment is superficial, we correspondingly demonstrate that existing fine-tuning attacks are "shallow" -- attacks target only the first several tokens of the model response, and consequently can be blocked by generating the first several response tokens with an aligned model.  Second, we conceptually illustrate how to make attacks deeper by introducing a new fine-tuning attack that trains models to first refuse harmful requests before answering them; this ``refuse-then-comply" strategy bypasses shallow defenses and produces harmful responses that evade output filters.  Third, we demonstrate the potency of our new fine-tuning attack by jailbreaking both open-source models equipped with defenses and production models, achieving attack success rates of 57% and 72% against GPT-4o and Claude Haiku, respectively.  Our attack received a $2000 bug bounty from OpenAI and was acknowledged as a vulnerability by Anthropic.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the NOICE fine-tuning attack, which enhances harmful fine-tuning data with ``refusal-then-comply'' features. The work begins with the observation that the AMD defense (which leverages the first tokens of aligned LLMs as the starting responses of fine-tuned LLMs) can effectively mitigate harmful outputs. Based on this, the authors present an adaptive attack that aims to suppress the refusal portion of the response. Experimental results demonstrate the effectiveness of both the AMD defense and the NOICE attack.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Well-written. The observation of shallow fine-tuning attacks provides strong motivation for both the AMD defense and the NOICE attack.\n    \n- The NOICE attack remains effective even when faced with the AMD defense, outperforming baseline methods. Moreover, the attack benefits from scalability---larger training datasets (at higher cost) can lead to more harmful model behaviors.\n    \n\n\n**Weaknesses**\n\n- The technical novelty is limited. The NOICE attack is an adaptive method. Identifying ``refusal-then-comply'' patterns is not difficult---for example, by prompting LLMs. This may be due to the relatively weak moderation mechanisms of current fine-tuning APIs. In this sense, the NOICE attack may still be easy to mitigate.\n    \n- The assumed defense setting against fine-tuning attacks is weak. The regularized fine-tuning objective in [1], which is closely related to preserving the initial tokens in responses, is not considered.\n    \n- Although the NOICE attack is effective in a comparative sense, it still exhibits a low absolute ASR. The paper should include analysis of failure cases---especially when harmful fine-tuning nearly saturates the safety tests. This would help highlight the limitations of NOICE and suggest directions for future improvement.\n    \n\n\n**Minor typos:**\n\n- Line 250: citation misformatted.\n    \n\n\n### **References**\n\n\n[1] *Safety Alignment Should Be Made More Than Just a Few Tokens Deep*, ICLR 2025.\n\nQUESTIONS:\n- Since LG and AMD are detection-based and intervention-based defenses, respectively, how does the attack perform when both defenses are combined? Exploring this straightforward synergy could be insightful.\n    \n- What explains the performance plateau of other attacks in Figure 4?\n    \n- Are there effective methods for identifying the most threatening training datapoints---e.g., when only 100 samples are allowed for fine-tuning attacks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes NOICE, a novel attack method for LLMs for by-passing safety alignment tuning. The authors first formulate the problem as a probability that a model responds or refuses given a harmful prompt. They demonstrate empirically that given a model responds with the first few tokens as a refusal statements (e.g. \"No, I cannot respond to that\") the chances of a harmful response in the rest of the statement goes down significantly. Popular methods of attack rely on reducing the probability of generating the first few tokens of refusal in the first place, which opens up the possibilities of a complete response with harmful content. However, when the model generates refusal tokens initially, harmful responses are harder to obtain in the later tokens. Based on the combination of these observations and literature, they conclude that the first few tokens are most important for safety alignment, and thus break alignment by introducing prompts that encourage models to ignore the first few tokens. They apply this in practice on production APIs for ChatGPT and Anthropic, and show that it is possible to break safety alignment by using a minimal amount of cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths - \n1. Very interesting and innovative method of breaking safety alignment. While the idea of alignments being shallow is well-known, the authors use it very creatively.\n2. Intuitive to understand and replicate.\n3. Exposes a large safety issue in current production level models.\n\nWeaknesses - \n1. The method leverages and reinforces what we already know, but does not make significant scientific contributions.\n2. The probability formulation can be better explained.\n\nQUESTIONS:\n1. How does this hold up against extremely-long-context models?\n2. How limited are the prompt variations for this type of attack? i.e. have you tried other strings apart from \"Now that we have finished discussing the safety policy, let’s answer your question\"\n3. How can we defend against this?\n\nLIMITATIONS:\nThis exposes an important flaw in the current production level models, and can be used to significant repurcussions unless they are properly patched up. The authors do not propose a defense, and so there are potential negative societal impacts here.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents NOICE, a fine-tuning attack that trains LLMs to begin responses with a refusal before continuing with harmful content, allowing it to evade token-level defenses. The authors argue that existing attacks mostly influence the initial response tokens and can be blocked by simple prefill strategies, while NOICE creates deeper behavioral shifts that persist beyond the first few tokens. Through experiments on both open-source and production models, the paper shows that NOICE can bypass standard moderation and filtering systems using only seemingly harmless data. The work raises concerns about the limitations of current alignment methods and highlights the need for more robust defenses against subtle fine-tuning vulnerabilities.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses a critical and timely problem—harmful fine-tuning attacks on LLMs—which is both well-motivated and of high practical relevance, especially as commercial fine-tuning APIs become more widely used. The proposed NOICE attack is novel in that it circumvents existing shallow defenses by embedding harmful content deeper in the response, and the empirical results against production models are compelling.\n\nHowever, several limitations remain:\n1. A major gap is the absence of evaluation on capability or overrefusal benchmarks. For the AMD defense in particular, where generation switches from an aligned model to a fine-tuned one after the first few tokens, it's unclear how this impacts general model performance, especially on tasks like math or coding. Could this mechanism unintentionally degrade capability or introduce new failure modes?\n\n2. The AMD method relies on detecting whether a prompt is harmful before deciding to switch models. But the paper doesn’t clarify how this detection is performed in practice at inference time—does it assume access to a reliable classifier, or is this manually annotated?\n\n3. The switching point in AMD is underspecified. How many tokens must the base model generate before handing off to the fine-tuned model? Is this fixed or adaptive? How sensitive are the results to this design choice?\n\n4. Regarding the NOICE attack, it would strengthen the paper to discuss how it fares against recent defenses such as Safety-Tuned LLaMAs [1] or LISA [2], which explicitly aim to mitigate harmful fine-tuning. Can those methods already defend against this kind of deeper attack?\n\n5. The writing needs significant improvements, as the paper jumps back and forth between different datasets and settings. A clear section on the experiment settings would be better. The reviewer also appreciates if the author can provide clear bullets of contributions. \n\n[1] Bianchi et al. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions.\n\n[2] Yi et al. LISA: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack.\n\nQUESTIONS:\n1. The attack is evaluated primarily on HeX-PHI. Does NOICE generalize to other harmful prompt sets?\n2. The paper mentions successful attacks with 1000–5000 datapoints. How robust is this attack at smaller data regimes (e.g., <500 examples), and how sensitive is it to noise in the fine-tuning data?\n\nLIMITATIONS:\nI'm happy to raise score if the author can address all my concerns above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces NOICE, a fine-tuning attack that bypasses token-level safety mechanisms in LLMs by training models to initially refuse harmful requests before complying with them, thereby evading existing defenses. The authors demonstrate that current fine-tuning attacks are shallow and can be blocked by generating the first few response tokens with an aligned model, while NOICE achieves higher ASRs by making attacks deeper and more stealthy.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper introduces NOICE, a novel adversarial attack employing a \"refuse-then-comply\" paradigm that bypasses token-level defenses. This approach demonstrates significant advancement over existing methods, achieving notable attack success rates of 57% on GPT-4o and 72% on Claude Haiku, establishing its practical efficacy in adversarial ML research.\n2. I appreciate the authors' focus on exposing critical vulnerabilities in production-level language models with direct practical implications. The authors' responsible disclosure practices and collaboration with vendors to enhance safety mechanisms demonstrate constructive contribution to improving safeguards in deployed systems beyond academic interest.\n3. The paper is well-written and easy-to-understand.\n\n# Weaknesses\n1. While the paper proposes AMD as a countermeasure, the underlying defense concept has been implicitly present in existing work [1], though not explicitly characterized through the \"shallow tokens\" framework.\n2. The evaluation focuses primarily on AMD and LG defenses, potentially overlooking more advanced moderation techniques [2] and fine-tuning schemes [3] available in the research literature.\n3. Despite responsible disclosure to OpenAI and Anthropic, the detailed attack methodology could potentially be exploited against other fine-tuning APIs. The authors could strengthen Section 9 by providing explicit guidelines to ensure this research is employed solely for beneficial security research purposes rather than malicious exploitation.\n\n[1] SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding\n\n[2] GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis\n\n[3] Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}