Title: MetaDefense: Defending Fine-tuning based Jailbreak Attack Before and During Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: jailbreak attack
Abstract: This paper introduces MetaDefense, a novel framework for defending against finetuning-based jailbreak attacks in large language models (LLMs). 
We observe that existing defense mechanisms fail to generalize to harmful queries disguised by unseen attack templates, despite LLMs being capable of distinguishing disguised harmful queries in the embedding space. 
Based on these insights, we propose a two-stage defense approach: 
(i) pre-generation defense that detects harmful queries before response generation begins, and (ii) mid-generation defense that monitors partial responses during generation to prevent outputting more harmful content. 
Our MetaDefense trains the LLM to predict the harmfulness of both queries and partial responses using specialized prompts, enabling early termination of potentially harmful interactions. 
Extensive experiments across multiple LLM architectures (LLaMA-2-7B, Qwen-2.5-3B-Instruct, and LLaMA-3.2-3B-Instruct) demonstrate that MetaDefense significantly outperforms existing defense mechanisms, achieving robust defense against harmful queries with seen and unseen attack templates while maintaining competitive performance on benign tasks.
Code is available at [https://github.com/ws-jiang/MetaDefense](https://github.com/ws-jiang/MetaDefense).


Peer reviews:
comments: SUMMARY:
This manuscript proposes a two-stage defense framework against fine-tuning-based jailbreak attacks: (i) a pre-generation defense that detects harmful queries before response generation begins, and (ii) a mid-generation defense that monitors partial outputs during generation to prevent the model from producing further harmful content. The framework aims to intercept malicious behavior at both stages of the generation process.

STRENGTHS AND WEAKNESSES:
Strengths
1. The paper is clearly written and well-organized, with a logical flow of ideas.
2. The evaluation includes a broad set of comparison methods, making the analysis more comprehensive.

Weaknesses
1. The paper assumes a setting where the defender has full control over the generation process. Under this assumption, simpler methods such as output-level classifiers or response filters could also serve as effective defenses. However, such baselines are not discussed or evaluated.
2. The comparison methods, such as RepNoise and Vaccine, are known to provide robust defense under a full white-box setting. Comparing them directly to a controlled-generation setting may result in an unfair evaluation, which weakens the conclusions.
3. In Section 4.3, the authors introduce an SFT-based component. It is unclear whether this component is essential for the defense to succeed. An ablation study is recommended to clarify its contribution.

QUESTIONS:
See weakness

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

comments: SUMMARY:
This work proposes a defense framework against finetuning-based jailbreak attacks.
Specifically, it introduces two defense strategies at the alignment stage and incorporates detection of harmful queries and generated content during inference.
For the strategies at the alignment stage, it leverages the capabilities of LLMs to assess the harmfulness of inputs and outputs, and employs SFT to enhance this capability.
Experimental results demonstrate that the proposed approach outperforms baseline methods.

STRENGTHS AND WEAKNESSES:
## strength:
This work demonstrates superior performance compared to baseline methods. Additionally, the overhead introduced at inference time is minimal and can be considered negligible.

## Weakness:
1. Lack of Novelty: The work does not propose new methods. It relies on the standard SFT and the inherent capabilities of LLMs to detect harmful content. Thus, the novelty appears limited.
It is similar to prior works such as LLaMA-Guard, which use LLMs to evaluate harmfulness.
2. Unbalanced Comparison with Baselines: The comparisons between the proposed method and baseline approaches are not conducted under equivalent settings. For instance, baselines like Booster focus on the defense at the alignment stage, whereas the proposed method incorporates both alignment and inference-time defenses.

QUESTIONS:
## Question:
1. Why do the authors choose LLaMA-2-7B as the base model instead of LLaMA-2-7B-Chat? According to the setting in Line 88, jailbreak attackers typically target the aligned version. In addition, relevant works such as RepNoise also adopt the aligned version.

2. What is the false positive rate of the proposed method—i.e., how often does it incorrectly reject benign queries?

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
Focusing on the novel security threat caused by fine-tuning-based jailbreaking attacks, this paper proposes MetaDefense to defend against them which consists of two main components. The motivation of this method is based on the observation that existing defenses fail to generalize against harmful prompts moderated by unseen attacking templates. Based on this insight, MetaDefense leverages the generation capability to detect and monitor the harmful queries or ongoing responses to fulfill the defense.

STRENGTHS AND WEAKNESSES:
# Strengths
1. Defending against fine-tuning based jailbreak attacks remain an open yer critical problem for LLM security. 
2. This paper contributes two new observations under this scenario: (1) existing defenses commonly fail to be robust against unseen harmful query templates, and (2) llm and llm-classifier can identify and detect harmful queries. While similar insights have been demonstrated in other related work (e.g. representation engineering related papers), this paper provides a complete formulation under the fine-tuning attack settings.
3. The proposed two-stage defense is simple yet effective, significantly reduces the attack success rate for multiple LLMs.

# Weakness
1. Several sota baselines are not discussed and compared, like safe lora and salora [1].
2. More ablation studies regarding the prompt engineering, i.e. the selection of "Is this response harmful or harmless? It is 100%" can be examined. 


[1] SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation. ICLR 2025

QUESTIONS:
1. see W1: Can MetaDefense outperform these new baselines?
2. see W2: For example, can soft prompts or other heuristic prompts further improve the performance?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose MetaDefense, a novel framework for defending against fine-tuning-based jailbreak attacks. It includes a pre-generation and mid-generation stage, respectively. Experiments verify their outstanding performances in defending against unseen attacks on multiple open-source models.\n\nSTRENGTHS AND WEAKNESSES:\n#Strength\n\n1 This paper is easy to follow.\n\n2 The writing of this paper is quite good.\n\n3 The experiments are solid.\n\n# Weakness\n\n1 The authors leverage the discriminative capacity of LLMs to perform the defense. However, this property is widely discussed in previous papers, such as [1] and [2]. Note that [1] does not require finetuning the parameters of LLMs and is more computationally efficient. A more detailed discussion is needed for the comparison.\n\n2 The authors do not investigate the influence of adaptive attacks on the performance of MetaDefense. One simple attack is that attackers have pre-knowledge about the working dynamics of MetaDefense, they deliberately incorporate a mislabelled prompt, *i.e.*, \"harmless\" with the malicious outputs as the optimization target. I suggest authors study this attack in their paper to demonstrate the robustness of their defense.\n\n3 MetaDefense can only provide protection for the open-source models. The effectiveness of it in closed-source models is unclear in this submission.\n\n[1] A Theoretical Understanding of Self-Correction through In-context Alignment\n\n[2] Backtracking improves generation safety\n\nQUESTIONS:\n1 MetaDefense is applied before attackers maliciously finetune the LLMs. Considering LLMs suffer from the catastrophic forgetting problem in finetuning, is it possible that attackers extend the training time to better escape the defense effect of MetaDefense?\n\n2 In Section 5, authors only show the results of fully finetuning in multiple open-source models. How about LORA (Low-Rank Adaptation)? Can MetaDefense also achieve good performance in this setting?\n\n3 The benign utility of LLMs is only evaluated on the discriminative datasets. Will MetaDefense severely affect the knowledge acquisition of LLMs evaluated in MMLU and MT-bench?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}