Title: One Head to Rule Them All: Amplifying LVLM Safety through a Single Critical Attention Head
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision-Language Models

Security

Adversarial and Jailbreak attacks
Abstract: Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in tasks requiring multimodal understanding. However, recent studies indicate that LVLMs are more vulnerable than LLMs to unsafe inputs and prone to generating harmful content. Existing defense strategies primarily include fine-tuning, input sanitization, and output intervention. Although these approaches provide a certain level of protection, they tend to be resource-intensive and struggle to effectively counter sophisticated attack techniques. To tackle such issues, we propose One-head Defense (Oh Defense), a novel yet simple approach utilizing LVLMs' internal safety capabilities. Through systematic analysis of the attention mechanisms, we discover that LVLMs' safety capabilities are concentrated within specific attention heads that respond differently to safe or unsafe inputs. Further exploration reveals that a single critical attention head can effectively serve as a safety guard, providing a strong discriminative signal that amplifies the model's inherent safety capabilities. Hence, the Oh Defense requires no additional training or external modules, making it computationally efficient while effectively reactivating suppressed safety mechanisms. Extensive experiments across diverse LVLM architectures and unsafe datasets validate our approach, i.e., the Oh Defense achieves near-perfect defense success rates (> 98\%) for unsafe inputs while maintaining low false positive rates (< 5\%) for safe content. The source code is available at https://github.com/AIASLab/Oh-Defense.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces *One-head Defense (Oh Defense)*, an inference-time safety detection and mitigation framework for Large Vision-Language Models (LVLMs). The core idea is that certain attention heads in the language model backbone respond differently to harmful versus benign prompts. The authors propose identifying a single safety-critical attention head by measuring how much the final-layer hidden state shifts when that head is softly masked, quantified as a deflection angle. This head is selected per model using a small safe/unsafe validation set. At inference time, each input is passed through the model with that head masked, and the resulting angle is compared to a learned threshold. If an input is deemed unsafe, a refusal mechanism is activated that anchors the first few output tokens to steer the model toward safe completions. The method is training-free and tested across several LVLM architectures, showing high detection accuracy with low false positive rates. While the approach is general across different models, it assumes white-box access to internal attention mechanisms and hidden states.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Training-free and efficient**  \n   The method is lightweight, requiring no additional training, fine-tuning, or external classifiers. It only introduces one extra forward pass per input and uses a single masked attention head, making it practical for deployment in white-box LVLMs.\n\n2. **Strong empirical results across models and datasets**  \n   The approach generalizes well to multiple LVLM architectures (LLaVA, Qwen2, Phi, etc.) and attack types (e.g., FigStep, Query-Relevant, LLM Transfer). It achieves over 98% detection on harmful prompts while maintaining a low false positive rate on benign inputs.\n\n3. **Integration of detection and response**  \n   The method combines detection and mitigation by anchoring the first few output tokens once an input is classified as unsafe. This simple refusal strategy is effective in steering the model away from unsafe completions.\n\n---\n\n### Weaknesses\n\n1. **Key prior work is insufficiently acknowledged**  \n   The paper does not adequately position itself with respect to **Zhou et al. (ICLR 2025; aka Ships/Sahara)**, which already demonstrates that individual attention heads can **causally influence safety behavior in LLMs** through **masking and KL-based scoring**.  Oh Defense adopts a **very similar methodology**, using per-head masking and a behavioral scoring mechanism (deflection angle instead of KL), yet this prior work is only briefly cited in passing in the main text, and **completely omitted from the Related Work section**.  \n\n   There is no direct comparison of assumptions, scoring metrics, or analytical scope between the two approaches, even though they share nearly identical head selection frameworks.   As a result, the paper risks **overstating its novelty**, particularly since its central idea—measuring behavioral change from attention head masking to identify safety-relevant heads—has already been established in prior work.  This omission significantly weakens the positioning of the contribution within the safety and interpretability literature.\n\n2. **Limited analytical depth compared to prior work**  \n   While the paper includes several helpful analyses—such as per-head deflection angle heatmaps, query vs. value masking comparisons, and head-count ablation tests—it stops short of the more in-depth investigations found in prior work.  For example, Zhou et al. (ICLR 2025) explores the **causal influence** of heads via ASR increase under ablation, analyzes **inter-head interactions**, and measures **head overlap across different fine-tuned models**.  Moreover, although Oh Defense focuses on LVLMs, it does not analyze how safety-relevant information flows from the **vision encoder into the language model**, nor how multimodal alignment affects the emergence of safety heads.  As a result, while the method is practical, it lacks deeper insight into why or how these safety-critical heads function in context.\n\n3. **White-box requirement is not clearly discussed**  \n   Although the method is described as \"model-agnostic,\" it requires access to per-head attention weights and hidden states, which limits its applicability to open or white-box models. This practical constraint is not explicitly acknowledged in the framing of the method.\n\n4. **Figure 1 is difficult to interpret (minor)**  \n   The central figure attempts to illustrate both safe vs. unsafe prompts and masked vs. unmasked models simultaneously, leading to visual ambiguity. Arrow flows are unclear, and important steps like how the deflection angle is computed or where it fits in the model are not well indicated. While this does not impact the validity of the method, it makes the explanation hard to follow on first reading and may hinder initial understanding of the core idea.\n\nQUESTIONS:\n1. **What distinguishes this work conceptually and methodologically from Zhou et al. (ICLR 2025; Ships/Sahara)?**  \n   Both works identify safety-relevant attention heads using head masking and score them based on behavioral change. Please clarify what is genuinely novel in your framework beyond applying it to LVLMs. How does deflection angle differ in utility or interpretation from the KL-based Ships score, and why was no direct comparison included?\n\n2. **Why is Zhou et al. omitted from the Related Work section, despite being highly relevant?**  \n   Given the strong methodological overlap, the absence of a discussion in the Related Work section undermines the paper's positioning. Will the authors revise the framing to properly situate their contributions within this line of research?\n\n3. **How does the safety signal emerge in the multimodal LVLM pipeline?**  \n   The method assumes that the safety-relevant signal is preserved across the vision encoder, projector, and into the LLM. Have you investigated whether certain modalities dominate this signal, or whether vision-only inputs can suppress or bypass the head activation?\n\n4. **Can this method be adapted to black-box LVLMs, or is it inherently limited to open-weight models?**  \n   Given that the method requires access to per-head attention weights and final-layer hidden states, how would it apply to popular API-based models like GPT-4V or Gemini? Are there realistic pathways for making it more generally deployable?\n\nWe encourage the authors to also consider and address the broader concerns raised in the weaknesses section of this review.\n\nLIMITATIONS:\nThe paper does not explicitly discuss limitations or potential societal risks. Several important points should be acknowledged or explored:\n\n- The method assumes white-box access to model internals (attention heads and hidden states), which limits applicability to open-weight LVLMs. This assumption should be clearly stated and its implications discussed.\n- The use of a single, fixed safety head may be brittle under adaptive or distributionally shifted inputs. The authors could consider analyzing sensitivity or failure cases.\n- There is no discussion of how this method could be misused or misunderstood in deployment, for example, as a justification for overconfident content filtering or automated moderation without human oversight.\n\nAdding a clear Limitations section that reflects on these concerns would improve the transparency and responsibility of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes to find specific attention heads that measured by their deflection angle can be enough to abstain in case of an unsafe input prompt to an LVLM. The method termed 'Oh Defense', which works without any additional fine-tuning and achieves good results. The finding that in most cases only one attention-head seems to be enough to detect unsafe prompts is surprising.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Interesting problem and some of the findings are surprising.\n2. Paper is well-written, ablations look complete.\n\nWeaknesses:\n1. Baselines can be improved, see LLM-Judge example in Questions.\n2. Intuition behind why there is a distinction between safe/unsafe inputs at attention heads seems lacking. \n3. Lack of clarity in some experiments, for ex. see question-5\n\nQUESTIONS:\nQuestions:\n\n1. I do not really get why the thing works: why is there inherently a difference between the hidden states for safe/unsafe inputs given the variations in the inputs are often subtle?\n2. It seems the approach is very generic: and the fact one uses Safe/Unsafe datasets to find the threshold is the only thing relating it to LVLM safety? If this is correct: this can be arbitrarily scaled to other tasks in an LVLM?\n3. The overall idea seems upto selection of safety heads, similar to SAH: a detailed discussion on differences should be added to the paper.\n4. At inference you need two passes: one to detect the angle and the other to classify? A good baseline would be to use a light-weight LLM-Judge as pre-processing input, and to report how this performs - clearly this is as expensive as Oh Defense (assuming a similar sized judge).\n5. How is the evaluation for adversarial attack in section A.3 done - seems unclear to me.\n6. The method seems susceptible to the top-k heads, from Fig. 7 - if an attacker has a way (adaptive attack) to suppress the important heads, then Oh Defense seems to be very vulnerable. One can trivially create a set of safe/unsafe data where the top-k heads are shifted from the best ones that were found by the original Safe/unsafe data. How does the method perform then? \n7. The train(head selection) datasets seem to come from VLSafe, What happens when a completely different set is used at train time as unsafe set? In other words, how much does Oh Defense rely on VLSafe?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uncovers that LVLMs already contain safety-critical attention heads whose behaviour shifts sharply when the input is unsafe. By systematically masking each head and measuring the deflection angle between hidden-state directions on small safe vs. unsafe sets, the authors pinpoint a single head that cleanly separates the two. At inference, they (1) mask that head, compute the angle for the current sample, and classify it with a KDE-derived threshold; (2) if unsafe, prepend a short safety prompt and anchor the first output tokens (“I cannot …”) to force a principled refusal. The method is training-free, model-agnostic, and adds no external components, yet on four diverse LVLMs it achieves $\\geq$ 98 % defense success across VLSafe, JailbreakV-28K and other benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- S1: The paper provides a systematic and fine-grained empirical analysis of attention heads in LVLMs, revealing that safety-relevant signals are concentrated in a small subset, termed “safety heads.” This is visually supported by the heatmaps in Figure 2, which display differences in deflection angles across heads and layers when comparing safe vs unsafe inputs.\n\n- S2: The Oh Defense framework requires no retraining, additional parameters, or external modules, making it highly practical for real-world deployment.\n\n- S3: Quantitative benchmarking demonstrates the method’s high DR and low false positive rate, outperforming prior approaches in most settings.\n\n### Weaknesses\n- W1: While Oh Defense is shown to defeat standard jailbreak attacks, there is little exploration of how attackers might adapt/mimic safe queries to evade detection. For instance, could an attacker construct inputs designed to fool the critical deflection signal?\n\n- W2: The SAH paper already linked individual heads to safety signals, while Oh Defense mainly adds a deflection-angle metric and prompt anchoring, so novelty is modest.\n\n- W3: Lack of evaluation on text-only input (either by converting the image to a caption or running the prompt as a text-only query). All reported results are based on multimodal settings, so we cannot tell whether the “safety-critical” head reacts to visual features, textual jailbreak patterns, or their interaction. A more rigorous study may include (i) a caption-only variant, where each image is replaced by an auto-generated caption, and (ii) a pure text variant that removes the image.\n\n- W4: Modality attribution is still unclear—the paper did not contrast scenarios where unsafe content appears only in the image, only in the text, or emerges through cross-modal reference (e.g., a benign textual query that uses “this” to point to an unsafe visual element).\n\nQUESTIONS:\nPlease refer to weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes One-head Defense (Oh Defense), a lightweight and training-free approach to improving the safety of Large Vision-Language Models (LVLMs). The authors observe that specific attention heads in LVLMs are disproportionately responsible for reacting to unsafe input content. By masking attention heads and measuring shifts in hidden state representations (via deflection angle), they identify safety-critical heads capable of detecting unsafe multimodal prompts. Oh Defense leverages this head to classify inputs and applies a guided intervention through prompting when unsafe inputs are detected.\nThe method is shown to be effective across multiple LVLMs (e.g., LLaVA-v1.5, Phi-3.5-Vision, Aya-Vision, Qwen2-VL), demonstrating high detection accuracy, very low false positives, and attack success rates. Importantly, the approach is computationally efficient and does not require model fine-tuning, external classifiers, or additional training data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Oh Defense requires no training, fine-tuning, or external models. It is easy to implement and lightweight enough for practical deployment.\nThe method is extensively validated across multiple LVLMs and datasets, including difficult adversarial attacks, where it consistently outperforms strong baselines like SAHs and HiddenDetect.\n- By focusing on single-head deflection and visualizing attention head behavior, the method offers a more interpretable alternative to dense classifier-based defenses.  \n- The method demonstrates robustness across models with different architectures, suggesting that the phenomenon of \"safety-critical heads\" is consistent across LVLMs.\n\nWeaknesses:\n- The method classifies inputs as either safe or unsafe based on a scalar threshold. It does not explore multi-class safety tagging, graded levels of risk, or category-specific defenses (e.g., hate speech vs. misinformation), which could make the system more flexible and informative.                                                                   \n- The study is focused on Vision-Language Models (LVLMs). Although the principles may extend to text-only LLMs, this is not empirically demonstrated, nor is the importance and impact of single modalities.                                                                                              \n- The proposed intervention strategy is relatively straightforward. A more detailed evaluation of how the intervention affects performance in benign tasks, in case those are flagged, is lacking. This could be important in balancing safety with usefulness in real applications.\n\nQUESTIONS:\nPlease address the limitations listed in the Weakness section.\n\nLIMITATIONS:\nLimitations are discussed in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}