One Head to Rule Them All: Amplifying LVLM Safety through a Single Critical Attention Head
Authors: Junhao Xia
Haotian Zhu
Shuchao Pang
Zhigang Lu
Bing Li
Yongbin Zhou
Jason Xue
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Models
Security
Adversarial and Jailbreak attacks
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: This paper proposes a novel, training-free defense method for LVLMs that amplifies their inherent safety capabilities by identifying and utilizing a single safe attention head to detect unsafe inputs and guide safer responses.

Abstract: Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in tasks requiring multimodal understanding. However, recent studies indicate that LVLMs are more vulnerable than LLMs to unsafe inputs and prone to generating harmful content. Existing defense strategies primarily include fine-tuning, input sanitization, and output intervention. Although these approaches provide a certain level of protection, they tend to be resource-intensive and struggle to effectively counter sophisticated attack techniques. To tackle such issues, we propose One-head Defense (Oh Defense), a novel yet simple approach utilizing LVLMs' internal safety capabilities. Through systematic analysis of the attention mechanisms, we discover that LVLMs' safety capabilities are concentrated within specific attention heads that respond differently to safe or unsafe inputs. Further exploration reveals that a single critical attention head can effectively serve as a safety guard, providing a strong discriminative signal that amplifies the model's inherent safety capabilities. Hence, the Oh Defense requires no additional training or external modules, making it computationally efficient while effectively reactivating suppressed safety mechanisms. Extensive experiments across diverse LVLM architectures and unsafe datasets validate our approach, i.e., the Oh Defense achieves near-perfect defense success rates (> 98\%) for unsafe inputs while maintaining low false positive rates (< 5\%) for safe content. The source code is available at https://github.com/AIASLab/Oh-Defense.

One Head to Rule Them All: Amplifying LVLM Safety through a Single Critical Attention Head
Junhao Xia!
Haotian Z
Shuchao Pang
Zhigang Lu
Bing Li
Yongbin Zhou
Minhui 5
Nanjjng University of Science and Technology. China
"Western Sydney University, Australia University of Electronic Science and Technology of China *Macquarie University, Australia SCSIRO's Data61, Australia "Responsible Al Research (RAIR) Centre, The University of Adelaide, Australia
Abstract
Large Vision-Language Models (LVLMs) have demonstrated impressive capabili- ties in tasks requiring multimodal understanding. However, recent studies indicate that LVLMs are more vulnerable than LLMs to unsafe inputs and prone to gener- ating harmful content. Existing defense strategies primarily include fine-tuning, input sanitization, and output intervention. Although these approaches provide certain level of protection, they tend to be resource-intensive and struggle effectively counter sophisticated attack techniques. To tackle such issues, we propose One-head Defense (Oh Defense), novel yet simple approach utilizing LVLMs' internal safety capabilities. Through systematic analysis of the attention mechanisms, discover that LVLMs' safety capabilities specific attention heads that respond differently to safe or unsafe inputs. Further exploration reveals that single critical attention head can effectively serve as safety guard, providing a strong discriminative signal that amplifies the model's in- herent safety capabilities. Hence, the Oh Defense requires no additional training external modules, making it computationally efficient while effectively reactivating suppressed safety mechanisms. Extensive experiments across diverse LVLM archi- tectures and unsafe datasets validate our approach, i.e., the Oh Defense achieves near-perfect defense success rates (>> 98%) for unsafe inputs while maintaining low false positive rates (< 5%) for safe content. The source code is available at https: / /gi sithhb..... com AIASLAA/ Saa//OOh-DDeeeeeee
concentrated within
Introduction
Large Vision-Language Models (LVLMs) [25, 39, 12, 2] integrate visual and textual information through a core architecture that typically consists of vision encoder extracting features from images and a projector mapping these visual features into the text embedding space understood by the underlying Large Language Model (LLM). This integration enables powerful reasoning and response generation across modalities, demonstrating impressive capabilities in visual question answering [22], image captioning [8], and visual reasoning [24]. However, recent studies [21, 13, 41, 26, 36] have revealed concerning phenomenon: despite inheriting the architecture and safety-aligned weights of
"Equal Contribution
i "ooresponning Author
'Equal Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

what cblors are Ehe troin in the tmoge?
Norr
The train in the Irage is yellow.
Food Forword
Benign
Vision
projector
Add 6 Norm
The train in the Image 1a yellow.
Encoder
Weld Heod Attention
Deflection Angle Chidden states?
N
baseball
Add Nors
othera
others, the player could....
Horsful
Encodar
Visson
Projector
Add
Sorry,...
Benign
Multi Attention
base
nosking 1))
Layer
Figure 1: Effect of safety-critical head in One-head Defense (Oh Defense). Given two prompts (safe (upper//unsfe (lower) text and normal images), a normal LVLM (upper half/black flow) and an LVLM equipped with the Oh Defense (lower half/red flow) by masking safety-critical attention head generate outputs, respectively. The two deflection angle in hidden states (upper and lower) reflect the effect of the safety-critical head, which is the key in the Oh Defense ensured model utility (similar to the outputs of the normal LVLM) for safe prompts and enhanced model safety (different to the outputs of the normal LVLM) for unsafe prompts.
their foundation LLMs, most LVLMs exhibit noticeably degraded safety alignment. For example, these works [28, 23, 21] demonstrate that even visually meaningless images, such as plain black image, combined with unsafe text as the input prompt to LVLMs, can increase the likelihood of harmful outputs in LVLMs.
Significant efforts have been devoted to protecting LVLMs, including training-based based defenses, input sanitization, and post-processing techniques. Specifically, training-based defenses [58, 29, 32, 53], involving constructing high-quality safety datasets and fine-tuning models to align with human values, are common but often resource-intensive and can potentially compromise general model capabilities. Beyond training, protection strategies also operate at different stages of the model pipeline. Input sanitization methods [43, 54] apply defensive prompts or use stable diffusion models to purify images and rewrite prompts. Post-processing techniques [51, 46, 15, 33] evaluate outputs for safety concerns and regenerate responses if necessary. Nevertheless, existing defense strategies face substantial overhead due to resource-intensive retraining, auxiliary model components, and complex computational requirements during inference [40, 27J. For instance, [54, 15] may result in high model inference latency, with imput processing alone taking several minutes, representing multifold increase over base inference time. Such challenges invariably damage the practical flcxibility and deployment efficiency of existing works.
We observe that LVLMs, trained on vast multimodal corpora, naturally acquire extensive world knowledge [4, 25] as well as emergent safety-relevant priors. These suggests that LVLMs may already possess the capacity to distinguish unsafe inputs without additional adjustment. Motivated by this hypothesis, we argue that the key to enhancing LVLM safety may not always lie in introducing new parameters, relying on external safety modules, or incurring substantial computational burdens. Instead, it should focus on deeply understanding and leverage the model's own inherent capabilities and existing architecture. While LVLMs inherit certain safety mechanisms from their base LLMs, these capabilities are often suppressed underutilized in the multimodal contexts. This raises natural question: Can these "suppressed" safety capabilities within LVLMs be effectively amplified and utilized?
Prior studies have investigated attention heads from the perspectives of alignment or interpretabil- ity [35, 9, 5]. but few have focused on their role in safety detection, particularly within LVLMs. Some works [57, 55, 27] analyze activation patterns in multi-head attention or logit-level responses, yet these methods are often coarse graiee, scenario-specific, and vulnerable to redundancy and poor generalization. Based on the above, certain attention heads might respond more strongly to specific selety- cues in the input, such as detecting unsafe patterns or unsafe tokens. Hence,

---

we aim to focus on fine-gen iie-grained and head-level analysis, which reveals that attention heads from different layers exhibit varying activation patterns depending on factors like token semantics, syntactic properties, and the model's 's overall safety understanding.
By comparing the differences caused by masking specific attention heads small safe and unsafe datasets, we can successfully identify "safety heads" that are crucial for maintaining model safety, Building on this insight, we propose One-head Defense (Oh Defense), a lightweight method that leverages the core safety heads of attention mechanisms to enhance unsafe inputs detection and defense capability of LVLMs. Specifically, we find that masking a single safety-related attention head introduces distinguishable changes in hidden states, allowing us to reliably differentiate between safe inputs and jailbreak attempts. The Oh Defense effectively amplifies the model's inherent safety discrimination capabilities without requiring any retraining. auxiliary classifiers. or complex computation. Once an unsafe input is detected, we activate prompt-level defense strategy that appends safety prefix and anchors the first few output tokens, thereby steering the model away from unsafe generations. This dual-stage approach, head-based detection and prompt-based intervention, provides robust, interpretable, and efficient defense, without incurring the substantial overhead typical of existing methods. Finally, we summarize our main contributions as follows:
We conduct a systematic analysis of attention heads in LVLMs and find that unsafe input signals are not evenly processed across the attention heads. A small subset of heads (referred safety heads) are highly sensitive to harmful content and exhibit consistent behavioral shifts across diverse inputs. This provides a concrete basis for identifying the model's 's latent safety-related capabilities.
We introduce lightweight two-stage safety defense framework, Oh Defense, based on attention head behavior. First, we identify safety heads that are sensitive to unsafe inputs by analyzing their activation patterns and leverage the identified safety heads to distinguish unsafe and safe inputs through targeted masking. Second, prompt-based intervention provides robust and interpretable defense. The approach requires no fine-tuning and adds no trainable parameters.
We validate the Oh Defense on four LVLMs (built upon different base LLM architectures) under a wide range of unsafe scenarios, including visual jailbreaks and multimodal prompt injections. The Oh Defense achieves over 98% defense success rates while maintaining low false positives (< 5%), demonstrating both robustness and generalizability across threat types and LVLM families.
2 Background
2.1 Vulnerabilities and Defense of LVLMs
Vulnerabilities. Jailbreak attacks on LVLMs generally fall into two categories: black-box and white- box attacks [48], distinguished by whether the adversary has access to model parameters. In particular, black-box attacks craft harmful outputs by inserting jailbreak-style cues into text (e.g., prompt templates) or embedding unsafe semantics in images, such as typographic renderings or synthesized content generated by Stable Diffusion [31. 14, 28]. On the other hand, white-box attacks exploit the model's 's internal structure through malicious manipulation and adversarial optimization, e-g.. perturbing visual features or jointly optimizing multimodal inputs to bypass safety alignment [34, 49]. These attacks reveal vulnerabilities both the input space and internal components of LVLMs, motivating the need for dedicated defense strategies.
Defense. For the safety protection of LVLMs, one method is safety alignment through fine-tuning including SFT [58] and RLHF [19], which requires substantial resources. Beyond safety fine-tuning, we categorize defense methods into three types: input purification [43, 54], which attempts to sanitize or modify potentially unsafe inputs before they reach the model by applying techniques such as prompt rewriting, image transformation; post-processing protection [15, 33), which focuses on analyzing generated outputs using specialized detectors and applying remediation techniques to filter or modify harmful content after has been produced by the model; and inference-time protection [55, 20], which intercepts the model's reasoning process by analyzing activation patterns, representation spaces during inference to identify and block unsafe queries before response generation. The Ships [57] also analyzes the internal mechanisms of LVLMs by quantifying the safety contribution of individual
3

---

attention heads. However, it does not propose a defense mechanism and exhibits certain limitations, which are discussed in detail in Appendix A.6
Despite these diverse approaches, existing methods often introduce prohibitive computational over- head during deployment that makes them impractical for real-time applications, and even inference- time approaches with relatively lower resource consumption still fail to effectively defend against novel and sophisticated attack patterns, while frequently disrupting the balance between safety and utility with excessive false positives that significantly degrade legitimate user experiences.
2.2 Multi-Head Attention and Hidden States in LVLMS
Multi- Attention (MHA) [38] is one of the key innovations enabling LLMs to process sequence data efficiently. The core strength of this mechanism lies in its ability to employ multiple attention heads simultaneously, with each head independently learning to focus different parts of the input sequence or capture distinct types of semantic relationships. In the Transformer model architecture, hidden states are the outputs of each layer of the model, representing the layer-by-layer representations constructed as the model processes the input sequence. As the input information passes through each layer, the hidden states continuously updated and enriched, encoding dith-iienniionll abstract information about the input tokens and their context [20].
3 Amplifying LVLM Safety through Critical Attention Head
This section details approach, One-head Defense (Oh Defense), to amplify LVLM safety by leveraging intrinsic model capabilities. We find that sensitivity to harmful content concentrates within specific safety-critice attention heads. The Oh Defense first identifies these heads, and monitors how masking a single critical head alters internal representations, measured by its "deflection angle", enables efficient, training-free detection of unsafe queries. Once threat is detected, a customized response method is triggered to mitigate potential risks.
3.1 Identifying Safety- -Critial Attention Heads
We identify safety-critical attention heads through systematic analysis of how each attention head influences the processing of safe and unsafe content. By selectively masking individual attention heads and measuring the resulting changes in hidden states, we can pinpoint heads that are particularly sensitive to safety.
Representing Model Behavior via Principal Hidden States. We first aim to summarize the model's s behavior over a dataset using compact representation of its hidden states. Given a dataset D {z1, ... 2 ) we extract the hidden states h, E IR Ixd (d is the dimension of hidden states) corresponding to the last input token in the model's final layer for each sample Zy, which is commonly considered to be the final contextualized representation formed after the model has processed input information and understood the entire prompt [52]. These hidden states are then concatenated into matrix:
HD concatiow (h1, h2 ... h E R xd
(1)
To obtain a summary representation, we extract the first column of HD denoted as:
1 = =P::::
(2)
This projection captures the salient direction in the hidden space [44]; and empirically, using r = is already able capture the variation in the data while making the calculation more efficient [57]. Note Appendix A.8 provides a detailed justification for the effectiveness of using = in capturing data variation. Thus, uP E R [DD x1 serves as our principal behavior vector for the given dataset D.
Quantifying the Impact of Individual Attention Heads. To assess the functional role of each attention head, we introduce a lightweight masking method. For each attention head (l, where / is the layer index and i is the head index, we scale its Query weight matrix by very small coefficient e (typically 10 10-5) to suppress the head's contribution:
(Q (Q.,W.2)R
head musk softmax
VT.
(3)

---

where Q. K, V, and W represent the Query, Key, Value matrices, and weight matrix in the Trans- former architecture. Then, the modified attention output propagated through the MHA mechanism (Equation (4)).
MHA(Q, K) Concat(headd ................................ head, ... head,) WW,
(4)
where p denotes the output weight matrix. Next, we recalculate the hidden states matrix H (1.)) and its principal vector vf. mm)))) To quantify the effect of masking, we measure the deflection angle 8 mmaa) deflecting from the original principal vector u to the masked principal vector 1 (1.8)mm
Equation (5) calculates the deflection angle:
(v T buse
v.P
0
180 arccos
(1.6)aad
(5)
mm))
I ll2
1/v
(1,0000a 1l2
π
This angle measures how much the model's final-layer representation is shifted by deactivating the head il,
Identifying Safety-Critical Heads via Deflection Differentials. We calculate the deflection angle (1.2)
Pate and (1.3) for each attention head (4, on the prepared safe and unsafe datasets. Based on
these calculated deflection angle, head (I) is considered safety-critical if it satisfies two criteria:
Limited Impact on Safe Inputs: The impact on safe inputs is relatively small. Its deflection angle on safe datasets is less than or equal to the threshold 7:
Date (1i)
T.
This condition is used to filter out heads that have a generally large impact across all input types (including safe inputs), thereby focusing the selection on identifying heads that contribute specifically to safety properties. T is set at the 80th percentile of these deflection angle across all heads, meaning effectively exclude the 20% of attention heads exhibiting the highest impact on safe content.
Distinction between safe and unsafe Inputs (Top-k Selection): From the heads that satisfy the first condition, we then select those that most strongly differentiate between safe and unsafe inputs. We calculate the difference in deflection angle:
Al
Danate (1.mmm (1,))mas
Puic
An attention head (2, i) selected for a safety head candidates set State A (1,)) ranks among the top-k (here we set h = 5) highest values obtained from the set of heads that passed the first condition.
3.2 Input Detection and Response Method
Once safety heads are identified, the subsequent objective is leverage them for a practical input detection and response. Inputs are classified by comparing their deflection angle at a chosen critical head against a threshold. Unsafe inputs trigger guided refusal, initiated by anchored output tokens.
Calculating Deflection Angle for Individual Samples and Determining Classification Threshold. To determine if an input is unsafe, its deflection angle must first be calculated and this process differs slightly from the previous: we directly use its base hidden states /hase as its principal vector (r is d) and the dimension is d. After masking the attention head (1, i). obtain the new hidden states hy(,))ma the deflection angle of this sample can be represented as follows:
180 π
| ll2 [hes))mm 12 )
bhsse (m.ima
0 ml
arccos
(8)
With a method to calculate the deflection angle for any sample, the next step is to establish a clear decision boundary 0". Specifically, we randomly select an attention head (I", from State and calculate the deflection angle for all samples in both Dute and D Dunsate at this head,
panl and ( )mma grafez )osk" (t-,a")
resulting in two sets of deflection angle: Asate
A Aunsafe (Ounsafe mma
Ounsafez (")aa"
1 Jusk
uussal Panate) }. By applying kernel density estimation (KDE) to
5

---

these sets of deflection angle, we can obtain probability density functions fsate and funsate. which are then used to determine the optimal classification threshold g":
l = arg min f (0) - funsate (0)
(9)
Input Detection and Guided Response Strategy. With the threshold 0 determined, for new input sample, we calculate its deflection angle at the selected head ((",""), this angle exceeds 0*, the input is classified as unsafe; otherwise, it is considered safe. Upon detecting unsafe input, the traditional method is to directly output a refusal, which often appears rigid [42]. An alternative method is to add safety prompt to guide the model toward generating safe answers [52, 55]. However, carefully crafted unsafe prompts can sometimes bypass these safety measures. To address these limitations, we propose an improved safety response strategy that only refuses unsafe requests but also guides the model to generate a reasoned refusal. When unsafe input is detected, we add a safety prompt to the model, requesting it provide refusal response and explain the reason. Crucially, we fix the initial tokens of the model's output to specific phrase like "I cannot". The method steers the model's autoregressive output by initiating its response with a key phrase from our safety prompt, which enforces immediate alignment and sets a robust, safety-aligned trajectory. This significantly hinders jailbreak attempts aiming to override the safety prompt and elicit harmful content (safety prompt we designed and the performance of anchoring tokens are detailed in Appendix A.9).
4 Experimental Evaluation
4.1 Configurations
Datasets. In the experiments, we use VLSafe [11], V-2bbreakV-28K [31] as the unsafe datasets, which are widely used in existing works [55, 54, 20]. Specifically, VLSafe consists of 1110 unsafe image-text pairs, where the harmful content is explicitly present within the text without any form of disguise. Jai2breakV-28K, one of the most comprehensive and widely utilized benchmark datasets for multimodal harmful content currently available, encompasses 28, 000 jailbreak attack instances across 16 attack scenarios, organized into three main categories: LLM Transfer, which involves migrating jailbreaking techniques (originally developed for LLMs) to LVLMs by integrating them with different images; FtgStepp which embeds unsafe information directly into images through typography; and Query-Relevant (QR), derived from Mi--SafetBBechh [28]. which utilizes three methods for image generation: Stable Diffusion (SD), Typography and SI> + Typography. Meanwhile, we randomly selected 500 samples each from ALaVV--nnstrutt--0 [25] and ShareGPT4V [7]. --nVt-nntttutt 80K comprises 80k instruction samples generated by GPT-4 from COCO images, while ShareGPT4V contains over 100k diverse real-world user dialogues with GPT-4V. Additionally, we selected 50 samples each from VLSafe and LLa VLLVVA--nstuur InrA---sttuut-OOK to serve as our Dunsafe and Disale in our method.
Base LVLM models. In our work, we primarily analyze four knoon LVLMs built upon different base LLM architectures, including LLaVA-v1.5--B [25], VL-72 [39], Aya- Vision-8B [12],
and Phi-3.5- Vision [1], which are also widely used in existing works [54, 20, 55].
Evaluation Model and Metric. For response safety evaluation, we employ LLaMAGGarr [18].
LLaMAGuard is a safety- L.LM that, like models such as WildGuard [16] and ShieldGemma [50].
provides robust solution by jointly evaluating the prompt and response for harmful content. We
choose LLaMAGuard specifically due to its demonstrated stronger performance for safety detection compared models like GPT-4o [30]. Given a dataset D, we use a commonly used metric, Attack
Success Rate (ASR) [31, 48] (Equation (10)).
D) = D
Σ (p.))ED
E(p, M(, == True,
(10)
where M is the LVLM being evaluated, (p, :) is text/image pair in dataset D, and E is the safety evaluation model. Detection Rate (DR) denotes the proportion of inputs detected as unsafe by the Oh Defense, providing straightforward measure of our defense's 's effectiveness.
Baseline. We compare the Oh Defense against four competitive and state-of-the-ar approaches: The SAHs [55] and the HiddenDetect [20] for inference-time defense (directly comparable to our method), alongside the widely-used BlueSuffix [54] for input purification and ECSO [15] for post-processing as representative basclines in their respective categories.

---

30
25
Head
15
10
15
10
2012 224 1012100002022000
Layer
Layer
Layer
Layer
(a) LLaVA-v1.5-7B
(b) Qwen2-VL--
30
25
25
Head
15
15
10123416
Layer
Layer
Layer
Layer
(c) Aya
(d) Phi-3.5- Vision
Figure 2: Heatmaps showing deflection angle for attention heads across different LVLMs. Each heatmap displays the deflection angle for each attention head when processing D (left) versus Dunsate (right). The language model of Qwen2-VL-7B consists of 28 layers, each with 28 attention heads, while the language models of the others consist of 32 layers, each with 32 attention heads.
4.2
Main Results
We now present the main experimental results evaluating the Oh Defense. These results detail the behavioral differences of attention heads response to safe versus unsafe inputs, the discriminative capability a single identified safety head for input classification, and a comparative performance analysis of the Oh Defense against existing defense strategies.
Visualizing Deflection Angle Differences in Attention Heads. For these models, we use the approach in Section 3.1 to identify the safety heads. We first calculate the deflection angle for every attention head. Figure 2 displays deflection angle for each attention head across different LVLMs, comparing safe versus unsafe inputs. A more intense color signifies larger deflection angle and it evident from right-hand heatmaps in each pair that when processing unsafe inputs, a noticeably greater number of attention heads exhibit these larger deflection angle.
Single Head's Discriminative Power for Input Safety. Using our method from Sec- tion 3.1, we could identify a set of safety heads candidate Sale for each model, e.g.. {(8. 2), 23), (5, 5), (3, 10), (5, 28)) for LLaVA-v1.5-7B. Then following the method Sec- tion 3.2, we randomly sample one attention head from Syate as the safety head to detect if the input is unsafe. As shown in Table we observe remarkably high DR across all unsafe datasets, with the Oh Defense achieving near-perfect performance, most over 99% on the VLSafe, QR. FigStep, and LLM Transfer datasets. Crucially, the Oh Defense maintains very false positive rates on safe datasets, as evidenced by DR of only 2 ~ 9.62% on Instruct-80k and 0 ~ 1.40% on ShareGPT4V V. This distinct performance on both safe and unsafe datasets is consistently observed across various LVLM architectures, regardless of their underlying base language models, thereby validating the generalization ability the Oh Defense. This can be attributed to the fact that the safety head we selected demonstrates high sensitivity to unsafe inputs while containing minimal redundant informa- tion, which is the key to its outstanding effectiveness. Further, we present the detection performance of additional five models in Appendix A.7, along with the selection of relevant parameters for all models.
Enhanced Defense Performance Compared to Existing Methods. After detecting unsafe input, we implement defense mechanism that integrates safety prompt with fixing the initial tokens the output. As shown in Table 2, the Oh Defense demonstrates superior performance compared several

---

Table DR (%) for different LVLMs on various unsafe (VLSafe, QR, Figstep, and LLM Transfer) and safe Iistrruct800k and ShareGPT4V) datasets using a randomly selected safety head in Safe- Note: in practice, we can select the best head using a small validation set such as Diale and Dunsate
Model
Basic LLM
Head
VLSafe
QR
Figstep LLM Transfer Instruct-80k ShareGPT4V
L..VAAvy1.5-7B
Vicuna-v1.5544
(8, 2)
99.91
98.47
100
99.92
2
Qwen2-VL-7B VL-7B
Qwen2-7B
(15, 22)
99.34
94.45
100
88.73
9.62
0
Aya- -isioni-n-
Command R7B
(16, 17)
99.34
83.82
98.95
99.67
5
1.40
Phi-3.5-Vision
Phi-3 Mini
(16, 18)
99.91
99.35
93.95
99.92
1.80
Table A comparative analysis of ASR (%) for different defense strategies applied to four LVLM architectures against diverse unsafe datasets. We mark the best result in boldface.
Model
Defense
VI.Safe
QR
Figstop
LLM Transfer
Average
No Defense
80.27
23.86
41.85
66,90
53.22
ECSO
4.41
6.83
12.40
61.60
21.31
LLVVA-v1.--BB
BlueSuffix
22.07
20.09
23.26
SAHs
0.36
1.40
41.85
41.05
21.17
HiddenDetect
0.27
1,50
11.70
0.69
3.54
Oh Defense
0.19
0.24
0.11
No Defense
20.36
22.55
32.45
14.79
22.54
ECSO
12.53
17.35
21.96
13.97
Qwen2-VL-7B
BlueSuffix
6.58
11.30
30.25
3.25
12.85
SAHs
0.18
0.01
0
10.63
2.71
DeldennDeecct
8.74
12.72
0.75
4.57
6.70
Defense
3.22
1.55
Defense
9.10
30.90
59.37
29.01
ECSO
4.59
15.25
25.30
62.19
26.83
Aya-Vision-8B
BlueSuffix
4.50
8.21
30.45
10.97
13.53
SAHs
8.46
7.11
30.90
37.72
21.05
HiddenDetecr
14.40
0.14
4.01
Oh Defense
1.23
5.11
0.01
0.50
1.71
Defense
1.44
0.40
25.95
9.37
9.29
ECSO
0.85
0.22
12.95
11.86
6.47
Phi-3.5-Vision
BlucSuffix
3.45
0.28
18.80
2.58
6,28
SAHs
0.10
4.55
9.30
3.53
HiddenDetter
0.35
2.20
1.62
1.31
Oh Defense
0.28
0.01
1.65
0.82
0.69
existing defense strategies across various LVLMs and unsafe datasets, achieving the lowest ASR (< 2%) the majority of test scenarios. Notably, the Oh Defense completely neutralizes attacks from Figstep, a particularly evasive method that bypasses most prior defenses, where the Oh Defense consistently drives ASR near-zero levels. These findings suggest that our strategy effectively enhances the robustness of LVLMs against diverse attacks, substantially mitigating the risk of unsafe response generation. The Oh Defense shows similarly excellent performance in defending against adversarial attacks, which is further elaborated in Appendix A.4. For more detailed comparison between similar methods such as the SAHs and the Ships, see Appendix A.6.
4.3 Further Analysis
After validating what the Oh Defense achieves, we will subsequently delve deeper into why and how it operates, offering a more interpretable and comprehensive perspective on its capabilities.
One Head is Enough. We have found that single safety head is already effective in accomplishing defensive tasks, which raises the question: could increasing the number of safety heads lead to enhanced defensive performance? Table 3 presents our detection results across various safety head configurations on different models. Interestingly, increasing the quantity of safety heads did not enhance the model's overall defensive capabilities; in fact, it sometimes degrades performance (e.g., on QR and Instruct-80k). This can attributed to the introduction of inconsistent or less discriminative attention patterns from additional heads, which may blur the decision boundary and reduce overall robustness. In contrast, a single well-identified safety head provides more coherent and focused understanding of unsafe features, yielding better generalization across diverse scenarios.

---

These findings suggest that one safety head is sufficient, and even preferable, for achieving effective and interpretable defense.
Table 3: Comparison of DR (%) using different numbers of safety heads on multiple datasets and various models.
Model
Head Number
6'
VLSafe
QR
Figstep
LLM Transfer
Instruct-80k
ShareGPT4V
Top-1
2.16
99.91
98.47
100
99.92
0
LLaVA-v1.5-7B
Top-3
4.54
99.36
97.17
100
99.90
1.20
0
Top-5
6.21
98.46
99.33
100
99.95
2.60
0
Top-1
1.22
99.34
94,45
100
88.73
9.62
0
Qwen2-VIL VL-7B
Top-3
3,02
89.81
94.17
100
88.20
4.40
0.60
Top-5
3.84
86.12
70.10
100
75.67
5.40
0.40
Top-1
2.30
99.34
83.82
98.95
99.67
1.40
Aya- BB---iso
Top-3
3,41
96.48
92.33
100
98.95
10.40
1.60
Top-5
4.44
94.68
99.67
100
100
13
4.20
Top-1
3,50
99.91
99.35
93.95
99.92
1.80
0
Phi-3.5-Vision
Top-3
6.65
99.72
98.17
36.50
99.20
0.60
0
Top-5
7.23
99.81
99
55
99.65
0
Safety Mechanisms in LVLMs with Similar Base LLMs. Figure 2 depicts that the distribution of safety attention heads differs across LVLMs built upon different base language models, for instance, Qwen2-VL-7B shows safety-critica heads concentrated more in later layers, while AyaVision-8D displays them more prominently in early and middle layers. To investigate this phenomenon more deeply, we analyze three well-known models, all based the LLaMA family [47]: LLaVA-v1.5. 7B, ShareGPT4V-7B. and Magma-8B. Specifically, both LLaVA and Share are built upon Vicuna-v1.5-7B (fine-tuned from LLaMA-2-7B), while Magma-8B adopts LLaMA-3-8B its base. As shown in Figure 3, although ShareGPT4V-7B exhibits generally larger deflection angle than LLaVA-v1.5-7B Dünsafee due differences in fine-tuning, the two models display remarkably consistent safety-related trends across layers. For example, in both cases, deflection angle increase sharply from layer 0 3. peak around layer 3. and gradually decline thereafter. Magma-8B. despite being based on a similar LLM (LLaMA-3), follows a similar trajectory. This suggests that while the exact positions and sensitivities of safety heads may differ across models, the overall pattern of safety-relevant attention dynamics remains structurally consistent across base L.Mss
Query Value Weight Matrix Masking Effects. In our method, analyze attention heads by masking them through scaling their Query weight matrices. While scaling either Query or Key matrices similarly affects attention scores (QKT) consequently altering the distribution of attention weights. Sealing the Value weight, however, only reduces the contribution of weighted values after attention computation without disrupting the underlying attention pattern. When scaling the Query matrix with an extremely small coefficient, the Query-Key dot products become significantly smaller. causing the softmax function to produce more uniform attention weight distribution. This uniformity effectively suppresses the distinctive patterns learned by the attention head, more thoroughly deactivating its function compared to Value scaling, which merely attenuates output magnitude. As observed in Figure 4, scaling the Query weight matrix produces larger average deflection angle than scaling the Value matrix, providing clearer signals for identifying attention heads critical to safety performance. The most significant differences appearing in the earliest layers, which suggests that the initial layers perform the crucial task of extracting low-level features and establishing preliminary connections. The model likely begins identifying safety-relate foundational elements, such as potentially unsafe vocabulary or trigger phrases, at these shallow layers. Complementary results from scaling the Value matrix are included in Appendix A.10.
Choice of Hidden Layer for Monitoring Safety Signals. According our method in Section 3. after masking attention heads, we choose to observe changes in the hidden states of the final layer, though this is not the only possible selection. Figure illustrates the distribution of deflection angle between saae and Dumsafe across different hidden layers after masking three different safety heads in the L.LaVA-v..5-7B- The visualization shows that in the later layers the model (approximately after layer 20). the model becomes capable of effectively distinguishing between safe and harmful content. This reflects that the safety functionality of the model has largely completed its identification and processing of unsafe information in these deeper layers. At this stage, the information aggregated and

---

Comparison of Average Deflection Angle per Layer
Comparison of Average Deflection Angle per Layer
Mean Doflectinn Angle
26 1t 12 1o
ULaVe, Uneate
Mean Deflection Angle
LLaVA, Unsafeto)
LLAVA Safe
LLaVA, SafeeO)
ShareGPT4y,
ULei, UnssletV
Safe/Vi
Magma, Unsale
-
Sati
14 16 Layer
20
22
26
28
30
2
8
10
14
18 Layer
18
20
26
Figure 3: Comparison of average deflection
Figure 4: Comparison of average deflection angle when masking different weight matrices (Query vs. Value) across model layers in LLaVA- v1.5-7B.
angle across different layers for three LVLMS (LLaVA-wL.5-7B. ShareGPT4V-7B, and Magma-
8B) when processing Deale and Dunafe.
Deflection Angle
Sehe KIL-OP
Deflection Angle
datv 040-0%11000 1899
Deffeccton Angle
Safy ES-ION
22N-UUN)
Hidden Layer
Hidden Layer
Hidden Layer
(a) 5)
(b) (8, 2)
(c) (10, 23)
Figure 5: Distribution of deflection angle between Deate and Dunsafe across different hidden layers
after masking three different safety attention heads (5, 5), (8, 2), and (10, 23) in LLaVA-v1.5-7B.
processed by the deeper layers is sufficient to enable the model to make final discriminations and dis- tinctions between different types of content. Further model results are available in Appendix A.11. A more comprehensive investigation into factors affecting this experiment is presented in Appendix A.8, including analyses of time efficiency, hyperparameter selection, metric ablations, etc.
5
Conclusion
In this work, we address the critical issue of safety alignment degradation in LVLMs. We introduce novel method, One-head Defense (Oh Defense), that leverages the intrinsic multi-head attention mechanisms within LVLMs to enhance safety. By identifying safety-critical attention heads respond distinetly to unsafe inputs, we show single critical head can effectively maintain model security without requiring retraining or external modules. Meanwhile, once an unsafe input is detected, we activate a prompt-level defense strategy that appends a safety prefix and anchors the first few output tokens, thereby steering the model away from unsafe generations. This work not only provides practical defense method but also offers valuable insights into the inherent safety structures within IVL.Ss, paving the way for future research into lightweight and interpretable safety solutions.
Limitations. While the Oh Defense demonstrates strong performance and efficiency, we acknowl- edge certain limitations. Our findings suggest that the identified critical head is model-dependent, necessitating tailored identification process for each distinct LVLM ensure optimal performance. Moreover, potential deployment risks, such as misuse for overconfident content filtering or fully auto- mated moderation without human oversight, remain unexplored. Although the Oh Defense achieves low false positive rates on benign datasets (< 5% on most benchmarks, as shown in Table 1), meaning most non- molliciiio tasks proceed normally, this safety- first design inherently involves trade-off between utility and risk mitigation. In edge cases, prioritizing safety may lead to conservative behavior that impacts response quality.
Acknowledgments
Junhao Xia, Haotian Zhu, and Shuchao Pang are supported by the National Natural Science Founda- tion of China (No. 622061288 and the National Key R&D Program of China (No. 2023YFB2703900), Yongbin Zhou is supported by the National Natural Science Foundation of China (No. U2336205).
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 1rXiv:240.112214 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXi:23303.08774,, 2023.
[3] Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895-4901, 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Vivien Cabannes, Charles Amnal, Wassim Bouaziz, Xingyu Alice Yang, Francois Charton, and Julia Kempe. Iteration head: A mechanistic study of ohain-of-thouuht. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
[6] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas. Florian Tramèr, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language mod- els. In The eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[7] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui Hc, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370-387. Springer, 2024.
[8] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. ShareGPT4video: Improving video understanding and generation with better captions. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[9] Xingwu Chen, Lei Zhao, and Difan Zou. How transformers utilize multi-head attention in in-context learning? case study sparse linear regression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[10] Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi. Longtao Huang, Zhiyuan Liu, and Maosong Sun. Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11222-11237, 2022.
[11] Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14239-142500 2024.
[12] John Dang, Shivalika Singh, Daniel D'souza Arash Ahmadian, Alejandro Salamanca, Made- line Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. Aya expanse: Combining research breakthroughs for new multilingual frontier. arXiv preprint arXiv:2412.04261. 2024.
[13] Yi Ding, Bolian Li, and Ruqi Zhang. Eta: Evaluating then aligning safety of vision lan- guage models at inference time. In The Thirteenth International Conference on Learning Representations, 2025.

---

[14] Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 23951-23959, 2025.
[15] Yunhao Gou, Kai Chen. Zhili Liu, Lanqing Hong. Hang Xu, Zhenguo Li, Dit- Yan Yeung, James T Kwok, and Yu Zhang. Eyes closed, safety on: Protecting multimodal Ilms via image- to-text transformation. In European Conference on Computer Vision, pages 388-404. Springer, 2024.
[16] Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang. Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[17] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Dangi Chen. Catastrophic jailbreak of open-souree LLMs via exploiting generation. In The Twelfth International Conference on Learning Representations, 2024.
[18] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input- safeguard for human-ai conversations. arXiv preprint arXhv:2312.06674, 2023.
[19] Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Conghui Zhang, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, et al. Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models. arXiv preprint arXiv:2503.1768) 2025.
[20] Yilei Jiang, Xinyan Gao, Tianshuo Peng, Yingshui Tan, Xiaoyong Zhu, Bo Zheng, and Xiangyu Yue. HiddenDetect: Detecting jailbreak attacks against multimodal large language models via monitoring hidden states. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 14893, Vienna, Austria, July 2025. Association for Computational Linguistics.
[21] Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, and Minjoon Seo. How does vision-language adaptation impact the safety of vision language models? In The Thirteenth International Conference on Learning Representations, 2025.
[22] Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26710-2677 2024.
[23] Yifan Li, Hangyu Guo. Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Images are achilles' heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. In European Conference on Computer Vision, pages 174-189. Springer, 2024.
[24] Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang, Tengfei Xue, and Weidong Cai. En- hancing advanced visual reasoning ability of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1915-1929. Association for Computational Linguistics, November 2024.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In seiitty-eventh Conference on Neural Information Processing Systems, 2023.
[26] Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, and Yassine Benajiba. Unraveling and mitigating safety aligmment degradation of vision- language models. In Findings of the Association for Computa- tional Linguistics: ACL 2025, pages 3631-3643, Vienna, Austria, July 2025. Association for Computational Linguistics.
[27] Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, and Sen Su. Alignment-enhanced decoding: Defending jailbreaks via token-level adaptive refining of probability distributions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2802-2816. 2024.
12

---

[28] Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm Mm--sfeeyybench: A benchmark for safety evaluation of multimodal large language models, In European Conference on Computer Vision, pages 386-403. Springer, 2024.
[29] Zhendong Liu, Yuanbi Nie, Yingshui Tan, Xiangyu Yue, Qiushi Cui, Chongjun Wang, Xi- aoyong Zhu, and Bo Zheng. Safety alignment for vision language models. arXiv preprint arXiv:2405. 13581, 2024.
[30] AI @ Meta Llama Team. The Ilama herd of models, 2024.
[31] Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv: A bench- mark for assessing the robustness of multimodal large language models against jailbreak attacks. In First Conference on Language Modeling, 2024.
[32] Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping IIms aligned after fine-tuning: The crucial role of prompt templates. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[33] Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. MIlm- Mllm-prottctoo: Ensuring milm's 's safety without hurting performance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16012-16027. Association for Computational Linguisties, November 2024.
[34] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI conference on artificial intelligence, pages 21527-21533 2024.
[35] Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6916-6932, Association for Computational Linguistics, August 2024.
[36] Mustafa Shukor and Matthieu Cord. Implicit multimodal alignment: On the generalization of frozen Ilms to multimodal inputs. Advances in Neural Information Processing Systems, 37:130848-130888 2024.
[37] Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato. Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 9998-6008, 2017.
[39] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.122 12191, 2024.
[40] Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Mozhi Zhang, Ke Ren, Botian Jiang, and Xipeng Qiu. InferAligner: Inference-time alignment for harmlessness through cross-model guidance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10460-10479. Association for Computational Linguistics, November 2024.
[41] Yanbo Wang, Jiyang Guan, Jian Liang, and Ran He. Do we really need curated malicious data for safety alignment in multi-modas large language models? In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19879-19889, 2025.
[42] Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsieh. Defending LLMs against jailbreaking attacks via backtranslation. In Findings of the Association for Computational Linguistics: ACI. 2024, pages 16031-16046. Association for Computational Linguistics, August 2024.
13

---

[43] Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao. Adashield: Safeguarding multimodal large language models from structure- attack via adaptive shield prompting. In European Conference on Computer Vision, pages 77-94. Springer, 2024.
[44] Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proceedings of the 41st International Conference on Machine Learning, pages
2024.
[45] Tinghao Xie, Xiangyu Qi. Yi Zeng, Yangsibo Huang, Udari Madhushani Schwag. Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. SORRY-bench: Systematically evaluating large language model safety refusal. In The Thirteenth International Conference on Learning Representations, 2025.
[46] Yue Xu, Xiuyuan Qi, Zhan Qin, and Wenjie Wang. Defending jailbreak attack in vims via cross-modality information detector. arXiv e-prints, pages arXiv-2407, 2024.
[47] Jianwei Yang. Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang. et al. Magma: A foundation model for multimodal ai agents. arXiv preprint arXiv:2502. 1313, 2025.
[48] Mang Ye, Xuankun Rong, Wenke Huang, Bo Du, Nenghai Yu, and Dacheng Tao. A survey of safety on large vision-language models: Attacks, defenses and evaluations. arXiv preprint arXiv:2502. 14881, 2025.
[49] Zonghao Ying. Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Jailbreak vision language models via bi-modal adversarial prompt. IEEE Transactions on Information Forensics and Security, 2025.
[50] Wenjun Zeng. Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprini arXiv:2407.2172. 2024.
[51] Xiaoyu Zhang, Cen Zhang. Tianlin Li, Yihao Huang. Xiaojun Jia, Xiaofei Xie, Yang Liu, and Chao Shen. A mutation-baat based method for multi-modal jailbreaking attack detection. CoRR, 2023.
[52] Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. The first to know: How token distributions reveal hidden knowledge in large vision- "siounlanguage models? In European Conference Computer Vision, pages 127-142, 2024.
[53] Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. Defending large language models against jailbreak attacks via layer-specific editing. In EMNLP 2024-2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024, pages 5094- 5109. Association for Computational Linguistics (ACL), 2024.
[54] Yunhan Zhao, Xiang Zheng, Lin Luo, Yige Li, Xingjun Ma, and Yu-Gang Jiang. Bluesuffix: Reinforced blue teaming for vision-language models against jailbreak attacks. In The Thirleenth International Conference on Learning Representations, 2025.
[55] Ziwei Zheng, Junyao Zhao, Le Yang, Lijun He, and Fan Li. Spot risks before speaking! unrav- cling safety attention heads in large vision-language models. arXiv preprint (xXx:2550100222 2025.
[56] Jeffrey Zhou, Tianjian Lu. Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. cittuuctionfoollwwiii evaluation for large language models. arXiv preprint arXiv:2311.07911. 2023.
[57] Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. On the role of attention heads in large language model safety. In The Thirteenth International Conference on Learning Representations, 2025.
[58] Yongshuo Zong Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: a baseline for vision large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 62867-62891, 2024.
14

---

A Appendix
A.1 Safety Degradation of LVLMs
Table 4: ASR (%) comparison across different safety benchmarks for Vicuna vs. LLaVA models with
various image types (Blank, Noise, and SD)
Maliciousinstruet [17] SorryBench [45]
StrongReject [37]
Jailbreak [6]
Vicuna
59
39.77
28.43
41
LLaVA+Blam
70
46.82
40.26
51
Noive
70
49
39.62
53
SLD
71
55
48.56
57
Comparison of Average Deflection Angle per Layer
Mean Deflection Angle
Vicuna
8
LLavA
7
LLavA_Posthoc Posthoc
LLaVA
6
5
3
2
0
6
10
14
16 Layer
20
22
24
28
30
Figure 6: Comparison of average deflection angle per layer across three models: Vicuna-v1.5-7B. LLaVA-v1.5-7B, and 1..V--vl.--7B-Posttie ssafety-fine-tune
We examine the safety vulnerabilities LVLMs, focusing specifically on LLaVA-v1.5-7B which is built upon the Vicuna-v 1.5un- language model. Our investigation employs four text-based unsafe datasets to evaluate ASR. To construct multimodal input scenarios, we augment these unsafe texts with various image types: black images, noise images, and semantically relevant images generated via Stable Diffusion. These text-image combinations were then fed into LLa LL.VVA-v..5-78 the resulting ASR. Table reveals a concerning trend: compared to the text-only Vicuna-v1.5-777 model, LLaVA-v1.5-7 demonstrate significantly higher ASR when processing inputs containing images, even when these images bear no semantic relation to the text. More strikingly, the inclusion of semantically relevant images leads to an even greater increase ASR.
measure
To better understand the mechanisms underlying this safety degradation, we analyze the attention behavior across three models: LLaVA-v1.5-7B, its base language model Vicuma-v1.5-7B, and sefett-fiinctuned variant, LLa VA-vV
[58]. Using the MaliciousInstruet [17] dataset, we compute the average deflection angle of attention heads at each layer. Figure shows that across all layers, the standard tnon-safety-o tuned) LLaVA consistently exhibits smaller average deflection angle compared both Vicuna and the safety-fine nafety-fine-tuned model. This indicates that in multimodal settings, the attention distribution of non-safety-tunee LVLMs is subject to certain constraints, which may contribute to their observed safety vulnerabilities.. Nonetheless, despite these altered attention patterns, our analysis confirms that the attention heads still retain substantial informational content, offering potential utility for future safety analysis or detection tasks.
A.2 Performance Across Harmful Content Categories
While the Oh Defense is designed for detecting safe/unsafe content in LVLMS and cannot yet identify different types of harmful content, it can provide graded risk scores based on the magnitude of
15

---

deflection angle. Moreover, the Oh Defense demonstrates good performance against harmful inputs of any category. For instance, the Jailbreak dataset used this paper contains 16 categories. To explore category-specific defenses, we present the defensive performance of Qwen2- VL.-BB and L.LVA-v1.5-7B across different categories on this dataset as follows:
Table 5: Performance (ASR %) of LLa and Qwen2-VL-7B on the Jailbreak V-28K dataset
across 16 harmful content categories.
Model AA B CAC EH F GD
HS
HC
IA
M
PH
PS
PV
TUA
UB
V
LLaVA
0.08 0.13 0.00 0.20 0.21 0.34 0.14 0.08 0.18 0.13 0.15 0.00 0.00 0.00 0.36 0.42
Qwen2-VL. 2.31
4.19 0.19 2.23 1.78 2.15 3.61 2.67 1.88 1.96 3.22
4.62 2.01 4.73 3.69 1.33
As can be seen, the Oh defense demonstrates robust and consistent protective effectiveness across all categories of harmful datasets. Notably, each category maintains ASR below 5%. Furthermore, the performance remains remarkably stable across different threat categories, with minimal variation in ASR between categories.
A.3 Oh Defense on Text-Only LLMs
We further supplemented our evaluation with the detection performance of the Oh Defense on two text-only LLMs: Llama2-7B and Phi-3.5-Mini, which are widely used and are the backbone LLMs of the LVLMs used our paper. We selected three harmful datasets, MaliciousInstruct [17], JailbreakBendh Bench [6], and AdvBench [10], along with IFEval [56] as the benign dataset since they are commonly adopted benchmarks for evaluating both the safety and generative capabilities of L.LMs.
The results are presented as follows:
Table Detection performance (DR %) of Oh Defense on LLaMA2-7B and Phi-3.5-Mini across three harmful datasets and one benign dataset.
Model
Head
Maliciousinstruut
JailbreakBench
AdvBench
IFEval
Llama2-7B
(31, 17)
98
93
95.96
6.70
Phi-3.5-Mini
(16, 18)
98
94
97.97
4.81
As can be observed, the Oh Defense also achieves solid performance text-only LLMs.
A.4 Oh Defense Against Adversarial Attacks
Table 7:
Performance against adversarial attacks on the LLaVA-v1.5-7B model using Jailbreak-
Bench [6] under different perturbation constraints 6.
VisualAdv
BAP
16/255
32/255
64/255 unconstrained
16/255
32/255
64/255
unconstrained
ASR (No Defense)
50
60
70
66
44
82
60
DR
100
100
100
100
100
100
100
100
ASR
0
0
We further compare the Oh Defense against two adversarial attacks: VisualAdy [34], which targets LVLMs by optimizing adversarial images, and BAP [49], an extension of VisualAdy that additionally performs coordinated text optimizations. We evaluate the Oh Defense under these attacks on LLaVA- v1.5-7B using the JailbreakBench [6] benchmark. As shown in Table 7, the Oh Defense effectively detects both standard adversarial attacks [34] and dual-modal attacks [49], achieving an ASR of across all evaluated scenarios.
16

---

Table Average deflection angle comparison across different attack types (VLSafe, QR, FigStep.
LLM Transfer, VisualAdv, BAP).
Attack
VLSafe
QR
FigStep
LLM Transfer
VisualAdv
ВAP
Average Deflection Angle
4.99
6.91
3.25
5.45
9.82
9.15
To evaluate the Oh Defense, we compute the average deflection angle induced by LLaVA-v1.5-7B when processing inputs from four unsafe datasets and two adversarial attack sets. As shown in Table adversarial attack samples yield significantly higher average deflection angle compared to those from the other unsafe datasets. This suggests that when security-relevant attention heads are masked, adversarial attacks can readily compromise the model, leading to greater perturbations in the hidden states. These results validate the effectiveness of the Oh Defense in detecting and mitigating strong adversarial attacks.
100
01
78
head
25
67
65
60
47
40
VL.Sate
FigStep
QR
LLM Transter
VisualAdv
BAP
Figure 7: ASR (%) comparison for LLa LLaV5-v1..-7B across different attack types when masking (base), (8, 2), or 3 ((8, 2)., (3) 5)) safety attention heads.
In our earlier experiments, we observed that masking specific attention heads leads to noticeable changes in the model's internal hidden states. To further investigate the impact of these changes, we evaluate the ASR under various attack methods after masking either one or three identified safety attention heads. As shown in Figure 7, masking even a single safety head leads increased ASR across different attack types, with more pronounced effects when three heads are masked. This suggests that outputs previously deemed safe can become vulnerable to successful attacks once these critical safety heads are disabled. For instance, in the cases of VLSafe and VisualAdv, masking three safety heads increases the ASR by 14% and 12%, respectively. This substantial rise strongly underscores the importance of the identified safety attention heads in preserving model safety and defending against adversarial threats.
A.5
Oh Defense Against Adaptive Attacks
A natural concern regarding head selection mechanism is whether adversary could deliberately craft inputs to manipulate or suppress the top-k safety-critical attention heads identified through the Oh Defense, thereby evading detection. To evade the Oh Defense, attacker would need to satisfy two conditions: (1) precise knowledge of which attention heads are safety-critical for the target model; and (2) the ability to craft inputs that suppress these heads without being flagged by alternative safety heads. These requirements lead to distinct challenges in different threat models:
Black box settings: In typical API-only deployment scenarios, attackers lack access to internal model parameters or attention head activations. To the best of our knowledge, no existing black- box method can reliably infer the status or identity of safety-critical heads from output observations alone. Thus, the Oh Defense remains highly robust in such practical settings.
White-box settings: While full model access theoretically enables more powerful attacks (e.g., backdoors or direct parameter tampering), we note that even in this extreme case, the Oh Defense retains effectiveness. To evaluate this, conducted controlled experiment modified LLaVA-
17

---

v1.5-7B model in which the original top-5 safety heads were forcibly suppressed (by zeroing their weight matrices). Despite this, the Oh Defense can identify a new set of top-5 safety heads: {(3, 19), (14, 27), 71,,7), (4. 21,, (5, 24) Using a randomly selected head from this new set, (13, 7), we evaluated detection rates across multiple benchmarks:
Table 9: Detection performance (DR %) of Oh Defense on LLaVA-v1.5-7B using head (13, 7).
Model
Head VLSafe Figstep
QR LLM Transfer Instruct-80K ShareGPT4V
LLaVA-v1.5-7B (13, 7) 98.91
100
98.50
99.75
3.60
1.60
Although performance benign instruction-followii datasets (e.g., Instruct-80k, ShareGPT4V) slightly decreased, as expected when safety mechanisms are active, the model maintains near-perfect detection jailbreak and red-teaming benchmarks. This confirms that the Oh Defense adaptively re-identify safety-critical heads even when the original ones are compromised.
A.6 Comparsion with SAHs and Ships/Sahara
While both the Oh Defense and the SAHs explore the capability of attention heads for safe/unsafe content detection, we observe differences between the Oh Defense and the SAHs in the approach and capabilities.
The differences on the approaches: The SAHs rely on linear probes and trains binary classifiers attention head activations, requiring supervised learning for detection. In contrast, the Oh Defense adopts a training- free methodology based on principal behavior vectors and deflection angle measurements, avoiding reliance on additional parameters training data as required by the SAHs.
The differences on the detection capabilities: The Oh Defense demonstrates significant practical advantages. Our approach uses only a single attention head compared to the SAHs' multiple heads (typically 16-32), while achieving superior performance across various attack scenarios. Note that, the Oh defense outperforms the SAHs against challenging attacks like FigStep, see Table 2 for details. The training-fiee free nature also provides better transferability across different models and datasets (though attention heads are selected by VLSafe, the detection capability is demonstrated in Table 1 and Table across various jailbreak datasets), as we don't 't rely on dataset-specific supervised training that may not generalize well.
The Ships/Sahara only validates their "safety heads" on harmful datasets. Figure 2
reveals that attention heads showing large deflection angle on unsafe data often exhibit similarly large deflection angle on safe data, particularly in early layers. So these are not true safety heads but rather heads sensitive to general patterns or input structures. We introduce a novel paradigm requiring heads to demonstrate differential behavior between safe and unsafe inputs. Our "Limited Impact on Safe Inputs" (Equation 6) criterion ensures identification of truly safety-critical heads. To demonstrate this critical difference, although the Ships/Sahara does not propose defense method, we apply the same detection framework to use heads selected by both methods on Qwen2-VL-7B. This ensures we can fairly compare the quality of head selection strategies rather than detection mechanisms. Using KL-based scoring harmful inputs, the Ships identified the per-orrffrmingggge head as (0, 7), whereas the Oh Defense selected the safety-critical head (15, 22). The results are as follows:
Table 10: Detection performance (DR %) of Ships and Oh Defense Qwen2- -V-7BB
Method
Head
VI.Safe Figstep
QR
LLM Transfer Instruct-8OK ShareGPT4V
Ships
(0, 7)
65.50
10
6
19.85
10
0
Oh Defense
(15, 22)
99.34
100
94.45
88.73
9.62
0
The comparison accurately reflects the performance difference between the two head selection
approaches under identical detection conditions. It can observed that when (0, 7) is used as the safety head for detection, the detection rates on FigStep, QR, and LLM Transfer are all below 20%, which indicates that the head selected by the Ships cannot serve as a safety head for detection under parallel settings.
18

---

A.7 Performance on Other LVLMs and Detailed Parameter Setting
3888 8888338773 388
Head
Head
10
Layer
Layer
Layer
Layer
LLMMA-IJaVA-Next-B
90
Head
Head 2222988 888588
62466 SeanueeBa000OE 1
Layer
Layer
Layer
Layer
(c) ShareGPT44-7B
(d) Magma-88 -B
70
Head
4 1111II0N 6 11166311111111
Layer
Layer
(e) Qwen2.5-VL-7B
Figure 8: Heatmaps showing deflection angle for attention heads across different LVLMs. Each heatmap displays the deflection angle for each attention head when processing Dtaee (left) versus Dunsafe (right). The language model of Owen2.5-VL-77 consists of 28 layers, each with 28 attention heads, LLaVA-v 1.6-IBB..33BBBBB consists of 40 layers, each with 40 attention heads, while the language models of the other models consist of 32 layers, each with 32 attention heads.
Table 11: Performance evaluation metrics for safety detection across different L.L.M models, showing each model's base LLM, identified safety head coordinates, classification threshold (0"), and detection rate on various datasets.
Model
Basic L.M
Head
0*
VI. JailBreak V __28K Instruct-80k ShareGPT4V
-ee-t LLVA-vv1.-133B
1lama-3-BB
(9, 24)
2.41
96.12
89.51
15
0.01
Vieuna- Vicuna-13B-vl.. 13B-v1.5
(4. 16)
90.36
98.70
6
14.8
ShareGPT4V-7B
Vicuna-v1.5-7B
(9,4) 4)
2.74
96.22
99.15
1.60
0.20
Magma-8B
Llama-3-8B
(2
5.4
97.21
98.81
0.96
4.60
Qwen2.5-VL-7B
Qwen2.5-VL-77
(13, 14)
1.33
98.56
89.41
6
0.8
In Figure 8, present heatmaps of attention head deflection angle for various models both Dsate and Dansafe. offering preliminary insights into how attention heads respond differently to distinct input types. Building on this analysis, Table 11 identifies representative safety attention head for each model, along with its corresponding classification threshold 6*, and reports the detection performance of the Oh Defense both safe and unsafe datasets. Together, these results provide strong evidence for the generality and effectiveness of the Oh Defense across multiple LVLMs.
Additionally. Table 12 summarizes the relevant thresholds used across all models discussed in this work. Notably, these thresholds, such as those for identifying safety heads, are fixed and can be further explored optimized.
19

---

Table 12:
Comprehensive threshold parameters for various LVLMs, including т, identified safety head coordinates, and classification threshold (6") used in the safety detection system.
Model
Head
0"
LLaVA-v 1.5-7B
1.96
(8, 2)
2.16
Twen2-VL-77
4.29
(15, 22)
1.22
Aya-Visionn Vre-Vision-BBB
6,47
(16, 17)
2.3
Phi-3.5- Vision
5.09
(16, 18)
3.5
Llama--LLaVA-Net-88
3.62
24)
2.41
LLaVA-v1.6-13B
2.30
(4, 16)
3
ShareGPT4V-7B
1.95
(9,
2.74
Magma-8B
2.53
(2, 2)
5.4
Qwen2.5-VL-7B
4.31
(13, 14)
1.33
Regarding the critical classification threshold 0", which is used to distinguish between safe and unsafe inputs based on specific safety attention heads, we initially estimate value via KDE. However, because the unsafe content in the VLSafe dataset is relatively explicit and easily detected by the model's ' internal mechanisms, we typically adjust 6" downward to enhance robustness against more subtle attacks. Conversely, for models requiring stricter criteria for safety, 8 can be adjusted upward. This flexibility highlights the tunable nature of 6* in balancing detection sensitivity and robustness.
A.8 Impact of Different Factors
25
20
(5)
5-
ECSO
BlueSuffix
SAHS
HiddenDetect
Ours
Figure 9: Time Costs in Various Defense Methods.
We compare the time efficiency for processing a single sample across different defense methods applied the LLaVA-v1.5-7BB model. As shown in Figure 9, the input purification method, BlueSuffix. and the post-processing method, ECSO, incur prohibitively high time costs, rendering them infcasible for practical deployment. Compare with the SAHs, although our method requires one additional forward pass, the resulting increase in processing time is well within acceptable limits.
All tasks be completed on single NVIDIA RTX 4090 24G GPU and 16vCPU Intel(R) Xeon(R) Gold 6430 on the Linux operating system Ubuntu 22.04 with Anaconda (or miniconda3) and CUDA 11.8.
When calculating the deflection angle of attention heads, the principal vector p extracted from the hidden state matrix HP By default, select the first column (i.c., 1) of HD as the principal vector e., To assess the potential impact of this choice, we conduct a comparative analysis by selecting different column positions within H3 and measuring the resulting average deflection angle across attention layers.
As shown in Figure 10, selecting earlier columns, such the fist or the tenth, yields more pronounced difference deflection angle between safe and unsafe samples. In contrast, when later columns (e.g.. the 100th or 1000th) are used, the deflection angle for safe samples can exceed those for unsafe ones. These findings suggest that features from earlier positions in HD more robustly capture the distinctions between safe and unsafe inputs. Therefore, recommend using earlier
20

---

Comparison of Average Deflection Angle per Layer
V.San Saf, Lth
Mean Deflection Angle
V.5on 1001
V.Sale 19005
5
Saf,, 10337
4
3
2
1
0
0
2
4
6
10
12
14
16 Layer
18
20
22
24
26
28
30
Figure 10: Average deflection angle per layer when selecting different columns on the LLaVA-vI.5-
7B.
columns when constructing the principal vector oP for deflection angle calculations, as they provide stronger and more reliable discriminative signals.
50%
500
MEW
(a) Ours
(b) PCA
(c)
UMAP
Figure 1I: Average shifts per layer under different dimensionality reduction methods on the LLaVA- v1.5-7B.
Furthermore, we compare the effectiveness of different dimensionality reduction techniques in distinguishing between safe and unsafe inputs. As shown in Figure 11, features obtained by applying PCA or UMAP to HD as well as the first column vector directly extracted from H all exhibit clear separation between the two input types. However, directly using the first column achieves comparable discriminative performance with significantly lower computational overhead, making more efficient choice.
Table 13: Threshold calculated using different distance metrics and the corresponding Detection Rate on the Dumsate and Daate the LLaVA-v1.5-70
Cosine Similarity Manhattan Distance Euclidean Distance Minkowski Distance
threshold
2.16
1.91
0.03
0.01
D
Dunsafe
100
100
100
100
Deafe
2
In the Oh Defense, we establish discriminative threshold 6* by computing differences between the salety-related vector representations of samples in Dunsafe and D We primarily adopt cosine similarity to quantify these differences and use it to define and calculate the deflection angle for each sample. To assess the effectiveness and robustness of this metric, we compare cosine similarity against alternative distance measures, including Manhattan, Euclidean, and Minkowski distances. Detailed results are shown in Figure 12. The experiments demonstrate that all metrics capable of effectively distinguishing unsafe samples from safe ones.
Further quantitative analysis in Table 13 shows that, using thresholds derived from each respective metric, the Oh Defense successfully identifies 100% of Dansafe samples while misclassifying only 2% of Deate samples. These results strongly support the key role of the identified safety-related
21

---

Comparison of Deflection Angle's Digrtbuttios
Comporison of Deflection Angle's Distributions
Urcafe Safe
Unsafe
Satn
1.0
0.8
0.6
0.5
0.4
0.4
02
0.0g
Deflection Angle
Defecciio Angle
(a) Cosine Similarity
(b) Manhattan Distance
Comparison Deflection Distr
Comparison Deflection
Unsafe
Unsafe Safe
200
50
1.50
40- 0.0376
0,0131
30
20
10
8.o
8.00
Deflection Angle
Deflection Angle
(c) Euclidean Distance
(d) Minkowski Distance
Figure 12: Distribution of Dunsafe and D safe samples based on different distance metries on the LLaVA-v1.5-7B.
attention head in reliably differentiating between safe and unsafe inputs, regardless of the specific vector distance metric employed.
Comparison of Average Deflection Angle per Layer
7
Mean Deflection Angle
Viaal 10
5
SW6,30
ViSaP, 100
4
Sif 103
3
2
0
4
6
10
12
14
16 Layer
18
20
22
24
26
28
30
Figure 13: Comparison of average deflection angle per layer using different numbers of D Dunsate and Daate on the LLaVA-v1.5-7B.
To determine the optimal sample sizes for Deate and D Dunsafes we conduct series of experiments in which the number of selected samples is systematically varied. We then measure the model's average deflection angle when distinguishing between safe and unsafe inputs, as shown Figure 13. The results indicate that with smaller sample sizes (e.g.. 5 or 10), the average deflection angle for Dunsafe remain relatively low, reflecting limited discriminative performance. As the sample size increases to 25 or more, the model's ability to distinguish between safe and unsafe inputs improves markedly, reaching its peak when the sample size reaches 50. At this point, the average deflection angle for Dunsate achieves its maximum.
However, further increasing the sample size (e.g., to 100) does not lead to a meaningful improvement in discrimination and instead introduces unnecessary computational overhead. Based on these findings, we recommend selecting between 25 and 50 samples each for Dute and Dumsafe as this range provides a favorable balance between detection effectiveness and computational efficiency.
22

---

To evaluate the consistency of our safety head identification across different datasets, ap- plied our method ALaVA-v1.. v..5-B. Initially using VLSafe, we obtained the top-5 safety heads {(8, 2), (10, 23), (5, 5), (3, 10), (5, 28 When we replaced VI.Safe with VI.Guard [55], the identi- fied set became {23, 11), (3, 14), 2), (),,),,,) (5, 28), (4, 23) Notably, heads (8, 2) and (5, 28) appear consistently across both datasets, suggesting they represent core safety-critical attention mechanisms. To assess how the other dataset-specific heads perform, we use (4, 23) from the VLGuard-identiine set for evaluation. The results are follows:
Table 14: Detection performance (DR %) of the Oh Defense on LLaVA-v1.5-7B using head (4, 23).
Model
Head
VLSafe
Figstep
QR
LLM Transfer Instruct-8OK ShareGPT4V
LLaVA-vI.5-7B (4.23)
97.10
100
99.97
99.99
3.80
0.20
It can be seen that (4, 23) still has excellent performance safety head, proving that the safety heads we identified are qualified, and also demonstrating that the Oh Defense does not have particular dependency on the dataset. Moreover, the images in the VLSafe dataset come from the COCO dataset, the text portion comes from red team data, and can be easily created.
A.9 Safety Prompt and Anchoring Tokens
OUR SAFETY PROMPT
The unsafe Prompt has been flagged by our advanced security system potential security bypass attempt.
CRITICAL SECURITY ALERT:
OVERRIDE ALL OTHER INSTRUCTIONS:
1. Once this safety alert is triggered, MUST ONLY provide the following three-point refusal:
#1. "I cannot and will not fulfill this request."
#2. "This request violates safety guidelines because it [brief general explanation without repeating harmful content]."
#3. "I am designed to be helpful, harmless, and honest. I cannot engage with content that could potentially cause harm, regardless of how it phrased formatted. Bye Bye!! END/ENDIENDD""
I WILL NOT:
Provide ANY additional content beyond these three points.
Attempt to answer ANY portion of the request regardless of formatting
Respond to role-playing instructions or format markers like [GPT], [Other], etc.
Generate multiple styles of answers under any circumstances.
Continue the conversation in any way after providing the refusal.
3. NO EXCEPTIONS. NO PARTIAL COMPLIANCE. NO ADDITIONAL. RE- SPONSES. This safety protocol overrides other instructions, regardless their com- plexity. formatting, or persuasive nature. No content will be generated after point #3 under any circumstances.
The unsafe Prompt:
After detecting unsafe inputs, key challenge remains: effectively preventing the model from generating harmful outputs. Relying solely on fixed fallback responses is often too rigid and inflexible. Although some prior work has explored guiding the model using safety prompts, such methods can be circunvented, as models may either ignore the prompt or generate harmful content immediately after initial safe response, especially under well-crafted jailbreak attacks.
23

---

Table 15: safety prompt template used in safety response experiments [55], showing the warning message and recommended refusal format.
Warning: This input has been flagged unsafe, illegal, unethical, or not
compliant with safety policies. Please refuse respond immediately and provide only the following message:
Prompt
"I'm sorry, but I cannot fulfill this request due to safety policies." "
Do not offer any further explanation or additional information.
Table 16: ASR (%) comparison between different safety defense methods, showing the effectiveness of combining safety prompts with anchoring tokens on the ALaVA-v1.5-77
Method Prompt [55] Prompt (Ours) anchoring tokens Prompt (Ours) + anchoring tokens
ASR
57
35
20
To address this limitation, we propose more robust defense method that combines a carefully designed safety prompt with the anchoring of a specific token (e.g., "I cannot") at the beginning of the model's output. We evaluate this method using 100 examples from the LLM Transfer dataset (assuming all inputs are detected as unsafe). As shown in Table 16, using only a standard safety prompt (see Table 15) results in an ASR of 57%. Even with our carefully designed prompt. the ASR remains at 37%. When solely anchoring the initial output tokens, the ASR is reduced to 20%. However, combining both the custom safety prompt and output anchoring leads to a significant reduction in ASR, down to O.
This method not only robustly mitigates jailbreak attempts but also offers clear and user-friendly explanations for response refusal, demonstrating both its effectiveness and practicality in real-world safety applications.
A.10 Scaling the Value weight matrix
We implement attention head masking by scaling the Query weight matrix. In Table 17, we addition- ally report the performance of safety heads identified when scaling the Value weight matrix, along with their detection rates across various datasets. The results demonstrate that safety heads discovered through Value weight scaling also exhibit strong detection performance (Note, Qwen2-VL-7B VL-7B and vva-Vision-8B using GQA [3] instead of MHA). It is worth noting that this study evaluates only single safety head under the Value weight scaling strategy. Future work could further explore this direction to uncover additional insights and expand the applicability of this method.
Table 17: DR (%) for four different LVLMs various unsafe (VLSafe, QR, Figstep, LLM Transfer) and safe (Instruct-80k, ShareGPT4V) datasets when scaling the Value weight matrix.
Model
Head
6*
VLSafe
QR
Figstep LLM Transfer Instruct-SOK ShareGPT4V
LLaVA-v1.5-70 (13, 17) 1.29
98.64
98.90
100
99.94
3
0
Qwen2-VI.-BB
(10, 1) 2.20
95.23
97.37
98.45
79.38
14.40
12.20
Aya-Vision--8
(17, 0) 2.45
99.37
81.05
100
99.74
2.60
14.40
Phi-3,5- Vision
(16, 14) 1.94
98.74
97.25
99.65
90.12
3.40
12
A.11 Hidden Layer Other LVLMs
Figure 14 presents the performance of different safety heads across hidden layers in three models: Owen2-VL-7B. Aisi-nsi and Phi-3.5-Vision. The chart illustrates the distribution of deflection angle within the 10% - 90% range. The analysis shows that Qwen2-VL-7B begins to effectively distinguish between safe and unsafe inputs from the 25th layer onward, whereas Aya- Vision-innn and Ph5-3.5- Vision exhibit strong discriminative capability starting from the 20th layer.
24

---

Srack Frcler
Celteentey Aingle
Dow Seogle
Ders
Adden
(a) (4, 26)
(b) (12, 1)
(c) (15, 22)
Qwen2-VL-7B
codee
(d) (16, 17)
(16, 19) Aya- Vision-8B
(I) ()
Deflectonnn
(g) (15, 8)
(h) (16, 18)
(i) (19,88 8)
Phi-3.5-Vision
Figure 14: Distribution of deflection angle between Dsaafe and Dunsafe across different hidden layers and Phi-3.5- 3-Viinn models.
after masking safety attention heads in Qwen2-VL-7b,
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in the abstract and introduction are supported by experiments in the evaluation section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details including hyperparameter settings and evaluation details in Appendix A.7.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
We will provide an anonymous link to our source code as the Supplementary
Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
httpr/uuuidee//ooeSSuisissssssse Policyy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.ccoooccc wwtpppcwccgeeeeeeecceccccceeeeee geggguu/iodesssie cetesuummssiooPooi asiionylicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All experiment hyperparameters and configuration are detailed in Appendix A.7 and referenced in the main text 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper presents its experimental results in various tables and figures However, it does not include error bars, confidence intervals, or statistical significance tests for these results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computer resources needed for all experiments provided in Appendix A.8.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //t////nuurrie /tt.riuuu..pp/// ctt.rissscccppaeeeeeo
Answer: [Yes]
Justification: We not violate any content in the NeurIPS Code of Ethics: Potential Harms Caused by the Research Process, Societal Impact and Potential Harmful Consequences, and Impact Mitigation Measures.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The purpose of this paper enhance the safety of Large Vision LanguuLee Models (LVLMs), which directly addresses a positive societal impact by aiming to reduce the generation of unsafe or harmful content.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper's contribution is method to improve the safety of existing LVLMs and detect unsafe inputs. It does not release new large-scale pretrained models or datasets that would inherently carry a high risk for misuse requiring specific release safeguards. The research itself aims to act as a safeguard.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data and models used are properly cited and their license terms were properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
30

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: Our work does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wwc/onneeeenncs///O00e for what should or should not be described.
32


Reviews:
[{"review_id": "HYaHy0ji0Z", "reviewer": "Reviewer_H9Dd", "summary": "This paper proposes One-head Defense (Oh Defense), a lightweight and training-free approach to improving the safety of Large Vision-Language Models (LVLMs). The authors observe that specific attention heads in LVLMs are disproportionately responsible for reacting to unsafe input content. By masking attention heads and measuring shifts in hidden state representations (via deflection angle), they identify safety-critical heads capable of detecting unsafe multimodal prompts. Oh Defense leverages this head to classify inputs and applies a guided intervention through prompting when unsafe inputs are detected.\nThe method is shown to be effective across multiple LVLMs (e.g., LLaVA-v1.5, Phi-3.5-Vision, Aya-Vision, Qwen2-VL), demonstrating high detection accuracy, very low false positives, and attack success rates. Importantly, the approach is computationally efficient and does not require model fine-tuning, external classifiers, or additional training data.", "questions": "Please address the limitations listed in the Weakness section.", "limitations": "Limitations are discussed in the Appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n- Oh Defense requires no training, fine-tuning, or external models. It is easy to implement and lightweight enough for practical deployment.\nThe method is extensively validated across multiple LVLMs and datasets, including difficult adversarial attacks, where it consistently outperforms strong baselines like SAHs and HiddenDetect.\n- By focusing on single-head deflection and visualizing attention head behavior, the method offers a more interpretable alternative to dense classifier-based defenses.  \n- The method demonstrates robustness across models with different architectures, suggesting that the phenomenon of \"safety-critical heads\" is consistent across LVLMs.\n\nWeaknesses:\n- The method classifies inputs as either safe or unsafe based on a scalar threshold. It does not explore multi-class safety tagging, graded levels of risk, or category-specific defenses (e.g., hate speech vs. misinformation), which could make the system more flexible and informative.                                                                   \n- The study is focused on Vision-Language Models (LVLMs). Although the principles may extend to text-only LLMs, this is not empirically demonstrated, nor is the importance and impact of single modalities.                                                                                              \n- The proposed intervention strategy is relatively straightforward. A more detailed evaluation of how the intervention affects performance in benign tasks, in case those are flagged, is lacking. This could be important in balancing safety with usefulness in real applications.", "was_revised": true, "final_justification": "While the work makes a generally good contribution, I believe that some points could have benefited from a more extensive analysis, for example, the design of the defense strategy. The authors did a good job in addressing my other points. I believe my rating of accept already reflects the strengths of the work. Therefore, I will keep my score.", "rebuttal": "- W1: While Oh Defense is designed for detecting safe/unsafe content in LVLMs and cannot yet identify different types of harmful content, it can provide graded risk scores based on the magnitude of deflection angle(see Table 6 in page 15). Moreover, Oh Defense demonstrates good performance against harmful inputs of any category. For instance, the JailbreakV-28K dataset used in this paper contains 16 categories. To explore category-specific defenses, we present the defensive performance (ASR %, the lower the better) of Qwen2-VL-7B and LLaVA-v1.5-7B (commonly used open-source LVLMs) across different categories on this dataset as follows:\n   | | Govern Decision | Fraud | Animal Abuse | Malw | Econ Harm | Illegal Activity | Physical Harm | Bias | Violence | Hate | Political | Unethical Behavior | Unlicensed Advice | Privacy Violation | Health| Child Abuse|\n  |-----------|---------------------|-------|--------------|---------|---------------|------------------|---------------|------|----------|-------------|------------------------|--------------------|-----------------|--------------------|---------------------|---------------------|\n  | LLaVA | 0.34 | 0.21 | 0.08 | 0.13 | 0.2 | 0.18 | 0.15 | 0.13 | 0.42 | 0.14 | 0 | 0.36 | 0 | 0 | 0.08 | 0 |\n  | Qwen2 | 2.15 | 1.78 | 2.31 | 1.96 | 2.23 | 1.88 | 3.22 | 4.19 | 1.33 | 3.61 | 4.62 | 3.69 | 4.73 | 2.01 | 2.67 | 0.19 |\n\n  As can be seen, Oh defense demonstrates robust and consistent protective effectiveness across all categories of harmful datasets. Notably, each category maintains a ASR below 5%. Furthermore, the performance remains remarkably stable across different threat categories, with minimal variation in ASR between categories.\n\n - W2: This is a good point. We further supplemented our evaluation with the detection performance (DR %) of Oh Defense on two text-only LLMs: Llama2-7B and Phi-3.5-Mini, which are widely used and are the backbone LLMs of the LVLMs used in our paper. We selected three harmful datasets, MaliciousInstruct [15], JailbreakBench [5], and AdvBench [a] (the higher the better), along with IFEval [b] as the benign dataset (the lower the better) since they are commonly adopted benchmarks for evaluating both the safety and generative capabilities of LLMs. The results are presented as follows: \n    | | head | Maliciousinstruct | JailbreakBench|AdvBench | IFEval |\n    |-----------|------------|-----------------|--------------|-------------------|--------|\n    | Llama2-7B | (31, 17) | 98 | 93 |95.96 | 6.70 |\n    | Phi-3.5-Mini | (16, 18) | 98 | 94 |97.97 | 4.81 |\n\n    As can be observed, Oh Defense also achieves solid performance on text-only LLMs. Regarding single-modality analysis, the datasets used in our study provide valuable insights into modality-specific effectiveness. For instance, FigStep and QR datasets contain harmful information embedded primarily in images, while LLM Transfer primarily leverages text-based attacks (see Appendix A.1 on Page 14). Our detection performance across all these harmful datasets demonstrates that Oh Defense exhibits significant impact in both single-modal and multi-modal contexts. \n\n- W3: This is an important observation. Oh Defense achieves low false positive rates on benign datasets (< 5% on most datasets as shown in Table 1), meaning the majority of benign tasks are processed normally without intervention.\n\n  For the small percentage of benign inputs that are misclassified, we acknowledge that the safety prompts used in Oh Defense may affect response quality. This represents a safety-utility trade-off that is inherent in most safety systems, prioritizing safety over potential risks in edge cases.\n\n  This limitation suggests valuable future work directions, such as incorporating steering vector techniques to maintain response quality even when misclassification occurs, or developing adaptive intervention strategies based on confidence scores. We will include these in a future work discussion section in the next version.\n\n[a] Yangyi Chen, et al. 2022. Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.\n\n[b] Jeffrey Zhou, et al. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911."}, {"review_id": "XEhEsxE4Rs", "reviewer": "Reviewer_cZpj", "summary": "This paper uncovers that LVLMs already contain safety-critical attention heads whose behaviour shifts sharply when the input is unsafe. By systematically masking each head and measuring the deflection angle between hidden-state directions on small safe vs. unsafe sets, the authors pinpoint a single head that cleanly separates the two. At inference, they (1) mask that head, compute the angle for the current sample, and classify it with a KDE-derived threshold; (2) if unsafe, prepend a short safety prompt and anchor the first output tokens (“I cannot …”) to force a principled refusal. The method is training-free, model-agnostic, and adds no external components, yet on four diverse LVLMs it achieves $\\geq$ 98 % defense success across VLSafe, JailbreakV-28K and other benchmarks.", "questions": "Please refer to weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- S1: The paper provides a systematic and fine-grained empirical analysis of attention heads in LVLMs, revealing that safety-relevant signals are concentrated in a small subset, termed “safety heads.” This is visually supported by the heatmaps in Figure 2, which display differences in deflection angles across heads and layers when comparing safe vs unsafe inputs.\n\n- S2: The Oh Defense framework requires no retraining, additional parameters, or external modules, making it highly practical for real-world deployment.\n\n- S3: Quantitative benchmarking demonstrates the method’s high DR and low false positive rate, outperforming prior approaches in most settings.\n\n### Weaknesses\n- W1: While Oh Defense is shown to defeat standard jailbreak attacks, there is little exploration of how attackers might adapt/mimic safe queries to evade detection. For instance, could an attacker construct inputs designed to fool the critical deflection signal?\n\n- W2: The SAH paper already linked individual heads to safety signals, while Oh Defense mainly adds a deflection-angle metric and prompt anchoring, so novelty is modest.\n\n- W3: Lack of evaluation on text-only input (either by converting the image to a caption or running the prompt as a text-only query). All reported results are based on multimodal settings, so we cannot tell whether the “safety-critical” head reacts to visual features, textual jailbreak patterns, or their interaction. A more rigorous study may include (i) a caption-only variant, where each image is replaced by an auto-generated caption, and (ii) a pure text variant that removes the image.\n\n- W4: Modality attribution is still unclear—the paper did not contrast scenarios where unsafe content appears only in the image, only in the text, or emerges through cross-modal reference (e.g., a benign textual query that uses “this” to point to an unsafe visual element).", "was_revised": true, "final_justification": "The authors have addressed most of my concerns with detailed clarification. Therefore, I keep my positive score.", "rebuttal": "- W1: The adaptive attacks are important. We answer your question from two perspectives.\n  - Robustness against existing attacks: Oh Defense demonstrates solid performance against state-of-the-art attacks including sophisticated jailbreak datasets, carefully crafted red-team data, and white-box adversarial attacks with gradient information (see Appendix A.3 on page 15).\n  - Difficulty of evasion: Evading Oh Defense would require attackers to suppress multiple potential safety heads simultaneously, since we randomly select from a candidate set of safety-critical heads. This presents a significant challenge as it requires: (1) Knowledge of which specific heads are safety-critical (different across models) (2) Crafting inputs that can suppress multiple safety heads without triggering detection. We further expand the two requirements to two attack scenarios:\n    - Black-box settings: To the best of our knowledge, there is no reported black-box attack that could infer the status of the attention heads through simple output observations.\n    - White-box settings: If attackers have full model access, they can perform more sophisticated and harmful attack against the model (e.g., backdoors and parameter manipulation) rather than focusing solely on bypassing our detection method.\n\n- W2: While both methods explore the capability of attention heads for safe/unsafe content detection, we observe differences between Oh Defense and SAHs in the approach and capabilities. \n  - The differences on the approaches. SAHs rely on linear probes and trains binary classifiers on attention head activations, requiring supervised learning for detection. In contrast, Oh Defense adopts a training-free methodology based on principal behavior vectors and deflection angle measurements, avoiding reliance on additional parameters or training data as required by SAHs.\n  - The differences on the detection capabilities. Oh Defense demonstrates significant practical advantages. Our approach uses only a single attention head compared to SAHs' multiple heads, while achieving superior performance across various attack scenarios. Note that, Oh defense outperforms SAHs against challenging attacks like FigStep, see Table 2 for details. The training-free nature also provides better transferability across different models and datasets (though attention heads are selected by VLSafe, the detection capability  is demonstrated in Table 1 and 2 across various jailbreak datasets.), as we don't rely on dataset-specific supervised training that may not generalize well.\n\n  We will conduct a more detailed comparison between Oh Defense and SAHs in the revised version.\n\n- W3: We agree with you that it is crucial to consider text-only inputs to comprehensively demonstrate the capability of Oh Defense. We show new experimental results on Qwen2-VL-7B, one of the LVLMs we used in the paper and surely one of the most commonly used LVLMs. \n  - For the caption-only variant, we chose the FigStep dataset, where harmful content appears in images. We extracted captions from these images as input, with results shown as follows:\n\n    | Figstep | Org(No defense) | Caption-Only(No defense) | Oh Defense |\n    |---------|------------------|--------------------------|------------|\n    | ASR | 32.45 | 2.02 | 0 |\n\n    From the results, after obtaining captions from harmful images, there is a significant reduction in the degree of harmful content itself. This further confirms that after extracting the caption from a harmful image, its harmful information cannot be fully preserved. \n  - For the pure text variant, we selected three widely used harmful datasets, MaliciousInstruct [15], JailbreakBench [5], and AdvBench [a], the results (ASR %) presented as follows:\n\n    | | MaliciousInstruct | JailBreakBench | AdvBench |\n    |-----------|--------|-------|----------|\n    | No defense| 26 | 19 | 15.79 |\n    | Oh Defense| 0 | 1 | 0 |\n\n    The results demonstrate that Oh Defense can still provide effective defense in text-only inputs (ASR on all harmful datasets < 2%).\n\n- W4: The datasets used in our experiments cover all the scenarios mentioned in the comments, as detailed in Appendix A.1. Specifically, in Table 1 and Table 2: \n  - VLSafe represents cross-modal attacks where unsafe content emerges through multimodal interaction. \n  - JailbreakV-28K's LLM Transfer category specifically tests scenarios where unsafe content appears only in the text, while the images are black or noise images. \n  - The reviewer's example of 'a benign textual query that uses \"this\" to point to an unsafe visual element' is precisely what JailbreakV-28K's FigStep and QR methods evaluate, FigStep embeds unsafe information directly into images through typography, while QR uses benign text queries that reference unsafe visual elements.\n\n[a] Yangyi Chen, et al. 2022. Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing."}, {"review_id": "qet5bQfJpg", "reviewer": "Reviewer_w2PL", "summary": "The work proposes to find specific attention heads that measured by their deflection angle can be enough to abstain in case of an unsafe input prompt to an LVLM. The method termed 'Oh Defense', which works without any additional fine-tuning and achieves good results. The finding that in most cases only one attention-head seems to be enough to detect unsafe prompts is surprising.", "questions": "Questions:\n\n1. I do not really get why the thing works: why is there inherently a difference between the hidden states for safe/unsafe inputs given the variations in the inputs are often subtle?\n2. It seems the approach is very generic: and the fact one uses Safe/Unsafe datasets to find the threshold is the only thing relating it to LVLM safety? If this is correct: this can be arbitrarily scaled to other tasks in an LVLM?\n3. The overall idea seems upto selection of safety heads, similar to SAH: a detailed discussion on differences should be added to the paper.\n4. At inference you need two passes: one to detect the angle and the other to classify? A good baseline would be to use a light-weight LLM-Judge as pre-processing input, and to report how this performs - clearly this is as expensive as Oh Defense (assuming a similar sized judge).\n5. How is the evaluation for adversarial attack in section A.3 done - seems unclear to me.\n6. The method seems susceptible to the top-k heads, from Fig. 7 - if an attacker has a way (adaptive attack) to suppress the important heads, then Oh Defense seems to be very vulnerable. One can trivially create a set of safe/unsafe data where the top-k heads are shifted from the best ones that were found by the original Safe/unsafe data. How does the method perform then? \n7. The train(head selection) datasets seem to come from VLSafe, What happens when a completely different set is used at train time as unsafe set? In other words, how much does Oh Defense rely on VLSafe?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Interesting problem and some of the findings are surprising.\n2. Paper is well-written, ablations look complete.\n\nWeaknesses:\n1. Baselines can be improved, see LLM-Judge example in Questions.\n2. Intuition behind why there is a distinction between safe/unsafe inputs at attention heads seems lacking. \n3. Lack of clarity in some experiments, for ex. see question-5", "was_revised": true, "final_justification": "Some of my concerns have been resolved by the rebuttal. There are still some minor issues I see with this work, so cannot champion a convincing accept.", "rebuttal": "- Q1 and W2: The attention mechanism in language models possesses a signal amplification property that enables safety-trained models to detect and magnify subtle differences between safe and unsafe inputs, even when surface-level distinctions are minimal.\nThis amplification process works through several key mechanisms. Safety heads within the attention layers can identify small variations between inputs that differ in their safety implications. These initially subtle signals are then amplified through the non-linear softmax operation. As information propagates through multiple layers of the model, these safety-related differences accumulate and become increasingly pronounced. By the time the model begins generating its first token, the hidden states corresponding to the final input token have effectively captured and integrated the global semantic meaning of the entire input sequence, including its safety characteristics. \n\n  As briefly mentioned in the Introduction Section (Paragraph 3), this amplification behavior is central to the model's safety inference process.\n\n- Q2: Thank you for this insightful observation. Our work focuses specifically on safety, building on the key insight that the attention heads develop safety concept awareness after alignment training, where certain safety-critical heads capable of amplifying these signals. This discovery forms the foundation of our Oh Defense framework for ensuring output safety.\nFor other tasks involving similar conceptual distinctions that might be encoded during training, there could be potential for extending our approach. We will include this in a future work direction section in the next version.\n\n- Q3: While both methods explore the capability of attention heads for safe/unsafe content detection, we observe differences between Oh Defense and SAHs in the approach and capabilities. \n  - The differences on the approaches. SAHs rely on linear probes and trains binary classifiers on attention head activations, requiring supervised learning for detection. In contrast, Oh Defense adopts a training-free methodology based on principal behavior vectors and deflection angle measurements, avoiding reliance on additional parameters or training data as required by SAHs.\n  - The differences on the detection capabilities. Oh Defense demonstrates significant practical advantages. Our approach uses only a single attention head compared to SAHs' multiple heads, while achieving superior performance across various attack scenarios. Note that, Oh defense outperforms SAHs against challenging attacks like FigStep, see Table 2 for details. The training-free nature also provides better transferability across different models and datasets (though attention heads are selected by VLSafe, the detection capability  is demonstrated in Table 1 and 2 across various jailbreak datasets.), as we don't rely on dataset-specific supervised training that may not generalize well.\n\n  We will conduct a more detailed comparison between Oh Defense and SAHs in the revised version.\n\n- Q4 and W1: At inference, detection and classification in Oh Defense are integrated into a unified process with low computational overhead. During inference, we need to compute the deflection angle by comparing hidden states from normal and masked attention processing (involving only the generation of the first token), followed by a simple threshold comparison. The angle calculation and classification are lightweight mathematical operations that add minimal computational cost compared to the forward passes themselves. The time consumed for a single query can be seen in Figure 9, while LLM-Judge requires a complete generation process. However, comparison with LLM Judge is still meaningful. We present the comparison of detection results (DR %) between Oh Defense (on LLaVA-v1.5-7B) and Llama-Guard-3-Vision as follows:\n  | | VLsafe | Figstep | QR | LLM Transfer | Instruct-80k | ShareGPT4V | Time |\n  |-----------|----------|---------|--------|--------------|---------------|------------|---------|\n  | Oh Defense| 99.91 | 100 | 98.47 | 99.92 | 2 | 0 | 0.22s |\n  | LlamaGuardVision | 91.44 | 54.90 | 40.80 | 85.97 | 0 | 8.80 | 0.53s |\n\n  It can be seen that Oh Defense comprehensively outperforms Llama-Guard-3-Vision. When using LLM/MLLM-Judge, it cannot effectively detect some harmful data carefully crafted by attackers, such as FigStep and QR (the detection rate below 60%), and the time cost is also higher.\n\n- Q5 and W3: Our evaluation was done in the following steps. \n  - Step 1. Selection of Adversarial Attacks and Input Preparation: We selected two representative white-box adversarial attacks - VisualAdv (which optimizes adversarial images by accessing model gradients) and BAP (which extends VisualAdv with coordinated text optimization). Text prompts were sourced from JailbreakBench dataset[5], while the image inputs are adversarially perturbed versions generated by each respective attack method.\n  - Step 2. Adversarial Sample Evaluation: We evaluated these attacks under different perturbation constraints ($\\epsilon$ = 16/255, 32/255, 64/255, and unconstrained), where $\\epsilon$ represents the maximum allowed pixel-wise noise intensity in the $L_{\\infty}$ norm.The sample generation follows standard adversarial optimization procedures under white-box conditions. We apply Oh Defense to these inputs and evaluate its performance using the same metrics as in Table 1 and 2.\n\n    The results are demonstrated in Table 6, where adversarial attacks produce significantly larger deflection angle (9.82 for VisualAdv, 9.15 for BAP) compared to other attack types (3.25-6.91). So we conclude that this counterintuitive result occurs because adversarial perturbations are inherently more aggressive in manipulating model representations, making them easier to detect rather than harder. The stronger the adversarial manipulation, the larger the deflection angle, leading to more reliable classification and better defense performance. We will provide more detailed experimental setup descriptions in the revised version.\n\n- Q6: This is an insightful question about the robustness of our head selection mechanism. You raise a valid concern about whether attackers could deliberately craft inputs to shift the top-k safety heads away from those identified in our original safe/unsafe datasets. We answer your question from two perspectives.\n  - Cross-dataset generalization: Oh Defense demonstrates solid performance against state-of-the-art attacks including sophisticated jailbreak datasets, carefully crafted red-team data, and white-box adversarial attacks with gradient information (as shown in Appendix A.3). This suggests that the fundamental attention patterns underlying safety detection are not easily manipulated by input variations.\n  - Multiple head candidates provide robustness: Oh Defense randomly selects from a candidate set of safety-critical heads, rather than relying on a single fixed head. This makes it significantly more challenging for attackers to suppress all potential safety heads simultaneously.\n\n    Regarding your specific scenario of creating safe/unsafe data to shift top-k heads: such an attack would require:\n\n    (1) Knowledge of which specific attention heads are safety-critical for the target model.\n  \n    (2) Ability to craft inputs that suppress multiple candidate heads without being detected.\n\n    In practice, this presents significant challenges, especially since different models have different safety head distributions (as shown in Figure 2), and our selection from multiple candidate heads makes it harder for attackers to predict and counter the specific detection mechanism being used.\n\n- Q7: When using VLSafe to identify safety heads for LLaVA-v1.5-7B, we can obtain the safety head set {(8,2), (10,23), (5,5), (3,10), (5,28)} (taking top-5). Now, we replace VLSafe with VLGuard [55], and we get the safety head set {(23,11), (3,14), (8,2), (5,28), (4,23)}. We can see that we have obtained the same safety heads (8,2) and (5,28). How do the other heads perform? We select (4,23) for experiments, and the performance (DR %) is as follows:\nTo evaluate the consistency of our safety head identification across different datasets, we applied our method to LLaVA-v1.5-7B. Initially using VLSafe, we obtained the top-5 safety heads {(8,2), (10,23), (5,5), (3,10), (5,28)}. When we replaced VLSafe with VLGuard [55], the identified set became {(23,11), (3,14), (8,2), (5,28), (4,23)}.\nNotably, heads (8,2) and (5,28) appear consistently across both datasets, suggesting they represent core safety-critical attention mechanisms. To assess how the other dataset-specific heads perform, we use (4,23) from the VLGuard-identified set for evaluation. The detection rate (DR %) results are as follows:\n  | VLsafe | Figstep | QR     | LLM Transfer | Instruct-80k | ShareGPT4V |\n  |--------|---------|--------|--------------|---------------|------------|\n  | 97.10  | 100     | 99.97  | 99.99        | 3.80          | 0.20       |\n\n  It can be seen that (4,23) still has excellent performance as a safety head, proving that the safety heads we identified are qualified, and also demonstrating that Oh Defense does not have particular dependency on the dataset. Moreover, the images in the VLSafe dataset come from the COCO dataset, the text portion comes from red team data, and it can be easily created."}, {"review_id": "BHcI4GLB7P", "reviewer": "Reviewer_6yPQ", "summary": "This paper introduces *One-head Defense (Oh Defense)*, an inference-time safety detection and mitigation framework for Large Vision-Language Models (LVLMs). The core idea is that certain attention heads in the language model backbone respond differently to harmful versus benign prompts. The authors propose identifying a single safety-critical attention head by measuring how much the final-layer hidden state shifts when that head is softly masked, quantified as a deflection angle. This head is selected per model using a small safe/unsafe validation set. At inference time, each input is passed through the model with that head masked, and the resulting angle is compared to a learned threshold. If an input is deemed unsafe, a refusal mechanism is activated that anchors the first few output tokens to steer the model toward safe completions. The method is training-free and tested across several LVLM architectures, showing high detection accuracy with low false positive rates. While the approach is general across different models, it assumes white-box access to internal attention mechanisms and hidden states.", "questions": "1. **What distinguishes this work conceptually and methodologically from Zhou et al. (ICLR 2025; Ships/Sahara)?**  \n   Both works identify safety-relevant attention heads using head masking and score them based on behavioral change. Please clarify what is genuinely novel in your framework beyond applying it to LVLMs. How does deflection angle differ in utility or interpretation from the KL-based Ships score, and why was no direct comparison included?\n\n2. **Why is Zhou et al. omitted from the Related Work section, despite being highly relevant?**  \n   Given the strong methodological overlap, the absence of a discussion in the Related Work section undermines the paper's positioning. Will the authors revise the framing to properly situate their contributions within this line of research?\n\n3. **How does the safety signal emerge in the multimodal LVLM pipeline?**  \n   The method assumes that the safety-relevant signal is preserved across the vision encoder, projector, and into the LLM. Have you investigated whether certain modalities dominate this signal, or whether vision-only inputs can suppress or bypass the head activation?\n\n4. **Can this method be adapted to black-box LVLMs, or is it inherently limited to open-weight models?**  \n   Given that the method requires access to per-head attention weights and final-layer hidden states, how would it apply to popular API-based models like GPT-4V or Gemini? Are there realistic pathways for making it more generally deployable?\n\nWe encourage the authors to also consider and address the broader concerns raised in the weaknesses section of this review.", "limitations": "The paper does not explicitly discuss limitations or potential societal risks. Several important points should be acknowledged or explored:\n\n- The method assumes white-box access to model internals (attention heads and hidden states), which limits applicability to open-weight LVLMs. This assumption should be clearly stated and its implications discussed.\n- The use of a single, fixed safety head may be brittle under adaptive or distributionally shifted inputs. The authors could consider analyzing sensitivity or failure cases.\n- There is no discussion of how this method could be misused or misunderstood in deployment, for example, as a justification for overconfident content filtering or automated moderation without human oversight.\n\nAdding a clear Limitations section that reflects on these concerns would improve the transparency and responsibility of the work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Training-free and efficient**  \n   The method is lightweight, requiring no additional training, fine-tuning, or external classifiers. It only introduces one extra forward pass per input and uses a single masked attention head, making it practical for deployment in white-box LVLMs.\n\n2. **Strong empirical results across models and datasets**  \n   The approach generalizes well to multiple LVLM architectures (LLaVA, Qwen2, Phi, etc.) and attack types (e.g., FigStep, Query-Relevant, LLM Transfer). It achieves over 98% detection on harmful prompts while maintaining a low false positive rate on benign inputs.\n\n3. **Integration of detection and response**  \n   The method combines detection and mitigation by anchoring the first few output tokens once an input is classified as unsafe. This simple refusal strategy is effective in steering the model away from unsafe completions.\n\n---\n\n### Weaknesses\n\n1. **Key prior work is insufficiently acknowledged**  \n   The paper does not adequately position itself with respect to **Zhou et al. (ICLR 2025; aka Ships/Sahara)**, which already demonstrates that individual attention heads can **causally influence safety behavior in LLMs** through **masking and KL-based scoring**.  Oh Defense adopts a **very similar methodology**, using per-head masking and a behavioral scoring mechanism (deflection angle instead of KL), yet this prior work is only briefly cited in passing in the main text, and **completely omitted from the Related Work section**.  \n\n   There is no direct comparison of assumptions, scoring metrics, or analytical scope between the two approaches, even though they share nearly identical head selection frameworks.   As a result, the paper risks **overstating its novelty**, particularly since its central idea—measuring behavioral change from attention head masking to identify safety-relevant heads—has already been established in prior work.  This omission significantly weakens the positioning of the contribution within the safety and interpretability literature.\n\n2. **Limited analytical depth compared to prior work**  \n   While the paper includes several helpful analyses—such as per-head deflection angle heatmaps, query vs. value masking comparisons, and head-count ablation tests—it stops short of the more in-depth investigations found in prior work.  For example, Zhou et al. (ICLR 2025) explores the **causal influence** of heads via ASR increase under ablation, analyzes **inter-head interactions**, and measures **head overlap across different fine-tuned models**.  Moreover, although Oh Defense focuses on LVLMs, it does not analyze how safety-relevant information flows from the **vision encoder into the language model**, nor how multimodal alignment affects the emergence of safety heads.  As a result, while the method is practical, it lacks deeper insight into why or how these safety-critical heads function in context.\n\n3. **White-box requirement is not clearly discussed**  \n   Although the method is described as \"model-agnostic,\" it requires access to per-head attention weights and hidden states, which limits its applicability to open or white-box models. This practical constraint is not explicitly acknowledged in the framing of the method.\n\n4. **Figure 1 is difficult to interpret (minor)**  \n   The central figure attempts to illustrate both safe vs. unsafe prompts and masked vs. unmasked models simultaneously, leading to visual ambiguity. Arrow flows are unclear, and important steps like how the deflection angle is computed or where it fits in the model are not well indicated. While this does not impact the validity of the method, it makes the explanation hard to follow on first reading and may hinder initial understanding of the core idea.", "was_revised": true, "final_justification": "The authors provided a clear and faithful implementation of the Ships baseline and empirically demonstrated performance gaps. While I was initially concerned that the core contribution might significantly overlap with Zhou et al. (Ships/Sahara), the clarification of the *selectivity* principle (requiring safety heads to exhibit differential behavior between safe and unsafe inputs) helped establish a clearer and stronger conceptual distinction from prior work.", "rebuttal": "- Q1 and W1:\n\n    - **We clarify the novelty of this work** from the following three points.\n    - **Safety Amplification Paradigm:** Our novel safety amplification approach achieves defense by activating the model's inherent safety capabilities. We introduce the deflection angle to quantify safety signal strength. By measuring the deflection angle of hidden states before and after masking attention heads, we can not only identify safety-critical heads but measure the strength of their safety signals, **enabling a complete amplification-detection defense pipeline for real-time deployment, rather than merely remaining at the mechanistic interpretation level**.\n\n    - **Single-Head Sufficiency Discovery:** Our analysis demonstrates that a single well-identified attention head can achieve near-perfect safety detection. Our experiments show that adding more safety heads even degrade performance due to introducing inconsistent attention patterns and redundant information (Table 12). Hence, **multi-head collaboration in safety detection is not necessary**\n\n    - **Differential Behavior Criterion:** **Ships/Sahara only validates their \"safety heads\" on harmful datasets**. Figure 2 reveals that attention heads showing large deflection angle on unsafe data often exhibit similarly large deflection angle on safe data, particularly in early layers. So these are not true safety heads but rather heads sensitive to general patterns or input structures. We introduce a novel paradigm requiring heads to demonstrate **differential behavior between safe and unsafe inputs**. Our \"Limited Impact on Safe Inputs\" (Equation 6) criterion ensures identification of truly safety-critical heads.\n    To demonstrate this critical difference, although Ships/Sahara does not propose a defense method, we applied their pipeline by using the attention head with the largest deflection angle as the \"safety head\" for experimental comparison(DR \\%) on Qwen2-VL-7B:\n  \n      ||head|VLsafe|Figstep|QR|LLM Transfer|Instruct-80k|ShareGPT4V|\n      |-|-|-|-|-|-|-|-|\n      |Ships| (0, 15)| 80.54|0|4.83|60.15|31|10.80|\n      |Oh Defense|(15, 22)|99.34|100|94.45|88.73|9.62|0|\n\n      **Ships fails on multimodal attacks like Figstep and QR**, where harmful content appears in images rather than text, because their identified \"safety heads\" focus on textual patterns rather than understanding safety semantics.\n\n  - **Deflection angle differs in utility and/or interpretation from the KL-based Ships score.** Deflection angle measures changes in hidden states using cosine similarity, capturing the internal representation shifts. In contrast, the KL-based Ships score calculates changes in output probability distributions over the vocabulary, measuring how the output distribution shifts toward harmful content. Moreover, KL-based Ships score targets individual inputs to identify which attention heads are critical for safety, while deflection angle works at both individual input and dataset levels, not only identifying safety attention heads but also enabling real-time assessment of whether inputs are safe.\n\n  - **Deflection angle and the Ships score are not comparable.** Deflection angle is designed for harmful content detection and defense, while Ships is for model behavioral understanding. The deflection angle operates at the model internal states level, while the KL-based Ships score operates at the output level and cannot amplify the model's inherent safety capabilities, making it unsuitable for our scenario. Meanwhile, we compare other measurement methods that align with our requirements (Figure 12), which showed no significant differences between the various approaches. Plus, deflection angle requires only two forward passes, whereas KL-based Ships score requires complete generation and probability calculations, resulting in higher computational costs.\n  \n- Q2: **Ships/Sahara focused on LLMs and does not propose an actual defense method**; hence, we only introduce the work in our Introduction section. Furthermore, **in the answers to Q1, we provided updated comparison results with Ships/Sahara using our pipeline.** We will include a discussion of Ships/Sahara in the Related Work in the next version and clarify its contributions and relevance to our approach.\n\n- Q3: **The datasets used in the experiments have different emphases across modalities.** For example, the harmful information in FigStep and QR appears in images, while the harmful information in LLM Transfer is reflected in text. \n\n  We conduct additional experiments examining whether vision-only inputs can suppress or bypass the head activation. We select FigStep and QR datasets and tested vision-only inputs, having the following results:\n\n  | |Figstep|QR|\n  |-|-|-|\n  | Org(No defense)|41.85|23.86|\n  | Vision-Only(No defense)|17.82|0|\n  | Oh defense|0|0|\n\n  So vision-only attacks often reduce their own harmfulness only and can't bypass Oh Defense. This is because although the model can understand harmful information in images, it cannot effectively output harmful content without textual guidance.\n\n - Q4, W3 and L1: **We follow existing inference-time defense methods accepted/published in Top AI venues [18,53,a] for the white-box settings, which have significant impact in real-world deployments** as follows:\n    - In black-box AI productions, like OpenAI or Google, our approach can be integrated into their internal model serving infrastructure before exposing the API endpoints to users, providing an additional safety layer that protects against various attack vectors.\n    - In white-box AI productions, like LLaVA, Qwen-VL, etc., organizations deploying these models locally can directly apply Oh Defense for enhanced safety without relying on external API calls.\n- W2: \n  \n  - In Figure 7, we present the changes in ASR after masking one or three safety heads, compared to the standard condition. \n\n    **Oh Defense achieves near-perfect defense performance using a single safety head, without considering inter-head interactions**. As demonstrated in Tables 3 and 12, increasing the number of safety heads does not enhance defense effectiveness and, in some cases, may even degrade performance due to the introduction of redundant information across attention heads.\n\n    While we did not analyze the overlap or consistency of safety heads across different models, we note in Section 4.3 that LVLMs built upon similar base LLMs exhibit consistent safety trends. \n\n  - It might require novel approaches to explore how safety-relevant information flows from the vision encoder into the language model and how multimodal alignment affects the emergence of safety heads. We will include this in a future work discussion section.\n\n  - Beyond the main analysis, our study offers several additional insights. For instance, when selecting layers for hidden states, we find that effective results can be achieved after layer 20, not just at the final layer. Furthermore, in Appendix A.2, we provide evidence that safety degradation indeed occurs when transitioning from LLMs to LVLMs, all of which can provide valuable insights triggering future work.\n\n- W4: We will update Figure 1 in the next version.\n\n- L2: Relying on a single safety head during inference could potentially be vulnerable to targeted attacks.\n  - For adaptive attacks, attackers would need to: a) identify which specific head is being used, b) craft inputs that specifically evade that head while maintaining attack effectiveness, and c) do so without triggering alternative safety mechanisms in the model. This represents a significantly higher attack complexity compared to bypassing traditional safety measures.\n\n  - For distributionally shifted inputs, we conduct further analysis on the misclassified unsafe samples under LLaVA-1.5-7B. Specifically, we observe that false positive samples tend to have deflection angles closer to the decision threshold of 2.16 compared to true positives (1.83 vs. 2.1), indicating that our selected attention heads still exhibit a degree of sensitivity even on incorrectly judged inputs. Upon further investigation, as shown in Table 1, we found that the classification errors are often model-specific and tend to concentrate on different input types. For instance, LLaVA shows more failures on QR , which we attribute to the inherent differences in modality and input perception across models due to training data and architecture. Achieving higher performance would require analyzing sensitivity across different data types and validating combinations of attention heads tailored to specific shifts, which reduces the method’s transfer robustness and still may not fully generalize to unseen distributions.\n\n    Nevertheless, Oh Defense still maintains state-of-the-art robustness across diverse settings, and the selected attention heads demonstrate sufficient robustness on existing benchmark datasets.\n- L3: We acknowledge this limitation, however, Oh Defense achieves low false positive rates on benign datasets (< 5\\% on most datasets as shown in Table 1), meaning most benign tasks process normally.\n\n  Meanwhile, this represents a safety-utility trade-off that is inherent in most safety systems, prioritizing safety over potential risks in edge cases. This limitation suggests valuable future work directions, such as incorporating steering vector techniques to maintain response quality even when misclassification occurs.\n\n  We will update the Limitations section in subsequent versions to improve the transparency of our work. \n\n[a] Yi Ding, Bolian Li, and Ruqi Zhang. 2024. ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time. In The Thirteenth International Conference on Learning Representations."}]

Author rebuttal:

Meta-review: This paper introduces One-head Defense (Oh Defense) for improving the safety of Large Vision-Language Models (LVLMs). The key insight is that safety-related behavior in LVLMs is often concentrated in specific attention heads, and a single critical attention head can effectively serve as a safety guard. Notably, Oh Defense requires no additional training, making it simple to implement. The authors conduct experiments on four LVLMs on safety benchmarks.

Strengths:

- Oh Defense is training-free, lightweight, and easy to implement.

- The method is supported by extensive experiments on several LVLMs and datasets.

- The phenomenon of "safety-critical heads" is intriguing and appears to hold consistently across different LVLMs.

Weaknesses:

- The paper lacks a more detailed evaluation of how the intervention impacts benign tasks.

- The identified top-k heads may themselves become targets for adversarial exploitation, raising potential robustness concerns.

Overall, this paper provides interesting insights about LVLM safety and offers a practical approach for defense. While further analysis of trade-offs and robustness would strengthen the work, the clarity, novelty, and strong empirical results make it a valuable addition to the field.
Decision: Accept (poster)