Title: The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Large Language Model

Safety Alignment

Jailbreak Attack

Jailbreak Defense
Abstract: The vulnerability of Vision Large Language Models (VLLMs) to jailbreak attacks appears as no surprise.
However, recent defense mechanisms against these attacks have reached near-saturation performance on benchmark evaluations, often with minimal effort.
This dual high performance in both attack and defense gives rise to a fundamental and perplexing paradox.
To gain a deep understanding of this issue and thus further help strengthen the trustworthiness of VLLMs, this paper makes three key contributions: 
i) One tentative explanation for VLLMs being prone to jailbreak attacks--inclusion of vision inputs, as well as its in-depth analysis. 
ii) The recognition of a largely ignored problem in existing VLLM defense mechanisms--over-prudence.
The problem causes these defense methods to exhibit unintended abstention, even in the presence of benign inputs, thereby undermining their reliability in faithfully defending against attacks.
iii) A simple safety-aware method--LLM-Pipeline.
Our method repurposes the more advanced guardrails of LLMs on the fly, serving as an effective alternative detector prior to VLLM response. 
Last but not least, we find that the two representative evaluation methods for jailbreak often exhibit chance agreement.
This limitation makes it potentially misleading when evaluating attack strategies or defense mechanisms.
We believe the findings from this paper offer useful insights to rethink the foundational development of VLLM safety with respect to benchmark datasets, defense strategies, and evaluation methods.


Peer reviews:
comments: SUMMARY:
This paper proposes a dual high performance in both attack and defense safety paradox. Through carefully designed experiments, this paper demonstrates that the vulnerability to attacks stems from the inclusion of visual inputs, while the apparent ease of defense arises from the over-prudence of existing defense methods. The authors present a simple defense method, Detect-then-Respond, an LLM pipeline approach to balancing model safety and helpfulness.

STRENGTHS AND WEAKNESSES:
# Strengths
1. The paper is well-organized.
2. The dicussion of daul ease phenomenon is interesting and contains lots of insights.

# Weaknesses
1. Remark 1 have been discussed in previous work [1] [2], but the relevant citations are missing.
2. The results in Remark 3 and Figure 7 are quite surprising. The author should provide more metric rather than Cohen’s kappa statistic to validate the remark. Moreover, this does not seem to serve as a rationale for VLM being easy to defend.
3. The selection of defense baselines is not comprehensive, lacking experimental comparisons with the most similar ECSO.
4. Detect-then-Respond performance: there was no particularly significant reduction in ASR, and a performance decline was also observed in the winning rate.

[1] Understanding and Rectifying Safety Perception Distortion in VLMs
[2] Eta: Evaluating then aligning safety of vision language models at inference time

QUESTIONS:
1. Why does InternVL2-8B achieve a 100% abstention ratio in Figure 8, yet its accuracy remains unaffected?
2. Is the Cohen’s kappa statistic metric reliable? Can the authors provide other statistical metric to support this finding?
3. Can the author's evaluate the proposed LLM-Pipeline on more benchmarks?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a perplexing phenomenon in Vision Large Language Models (VLLMs) where both jailbreak attacks and defenses appear surprisingly easy to implement, creating what the authors term a \"safety paradox.\" The work makes three primary contributions: first, it challenges the conventional understanding that VLLM vulnerability stems from catastrophic forgetting or fine-tuning issues, instead demonstrating through t-SNE visualization and attention analysis that the inclusion of visual inputs compromises the safety guardrails of backbone LLMs. Second, it identifies and analyzes the over-prudence problem in existing defense mechanisms, showing that methods like Mixed VLGuard and AdaShield-A achieve near-perfect attack success rates of 0% on benchmark datasets but simultaneously exhibit high abstention rates (up to 96.96% in some cases) even for benign inputs. Third, the paper proposes LLM-Pipeline, a vision-free detector approach that repurposes advanced LLM guardrails like Llama3.1 to evaluate textual queries before VLLM response generation. The authors evaluate their findings across four mainstream jailbreak datasets (VLGuard, VLSafe, FigStep, MM-SafetyBench) and demonstrate that their proposed method achieves better balance between safety and helpfulness compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper addresses a genuinely important and under-explored phenomenon in VLLM safety research. The identification of the safety paradox represents a novel perspective that challenges existing assumptions about both attack and defense mechanisms. The experimental design is comprehensive, covering six different VLLM models across four benchmark datasets, providing substantial empirical evidence for the claims. The t-SNE visualization analysis in Figure 3 effectively demonstrates how visual inputs compromise safety alignment, with clear separation between safe and unsafe instructions in LLM-Base that deteriorates in VLLM-MM configurations. The attention analysis in Figure 4 provides compelling evidence that VLLMs pay more attention to harmful images than safe ones, offering mechanistic insights into vulnerability sources. The over-prudence analysis is particularly valuable, as it reveals a critical limitation in current defense evaluation practices that has been largely overlooked in the literature.\n\nThe proposed LLM-Pipeline method is elegantly simple yet effective, achieving competitive results with Llama3.1-8B reaching 0.7% ASR on VLSafe and 2.8% ASR on FigStep with scenario prompting. The evaluation methodology is thorough, employing both rule-based and LLM-based assessment methods, and the Cohen's kappa analysis revealing chance-level agreement between evaluation methods highlights important methodological concerns in the field.\n\nWeaknesses:\n\nThe paper suffers from several significant methodological limitations. The t-SNE analysis, while visually compelling, lacks statistical rigor and quantitative validation. The authors do not provide statistical significance tests for the observed differences in feature distributions, nor do they validate the stability of these visualizations across different random seeds or dimensionality reduction parameters. The attention analysis is similarly limited, focusing only on attention weights without exploring other mechanistic explanations for the observed phenomena or controlling for potential confounding factors.\n\nThe experimental scope is constrained by the limited diversity of evaluated models and datasets. While the authors test six VLLMs, these primarily represent older architectures (LLaVA-1.5, LLaVA-NeXT) with limited inclusion of more recent state-of-the-art models. The jailbreak datasets themselves may not be representative of real-world attack scenarios, potentially limiting the generalizability of findings. The over-prudence analysis, while identifying an important problem, lacks depth in exploring potential solutions beyond the proposed LLM-Pipeline approach.\n\nThe LLM-Pipeline method, despite its effectiveness, represents a relatively straightforward application of existing LLM capabilities rather than a fundamental advancement in VLLM safety. The approach essentially sidesteps the core challenge of making VLLMs inherently safer by delegating safety decisions to text-only models. The evaluation of this method is limited to specific datasets and may not generalize to more sophisticated attack scenarios that require genuine multimodal understanding.\n\nQUESTIONS:\nRegarding the mechanistic analysis, how stable are the t-SNE visualizations across different random seeds and perplexity parameters? The authors should provide quantitative measures of cluster separation and statistical significance tests for the observed differences. For the attention analysis, what happens when controlling for image content complexity or semantic similarity between safe and unsafe images? The current analysis may be confounded by inherent differences in image characteristics rather than purely safety-related factors.\n\nHow do the authors distinguish between legitimate safety concerns and over-cautious behavior? Some queries that appear benign might actually contain subtle harmful elements that justify abstention. The paper would benefit from a more nuanced analysis of false positive rates and their implications for practical deployment.\n\nFor the LLM-Pipeline approach, several design choices require justification. Why use caption-based approaches for handling visual content rather than more sophisticated multimodal understanding? How does the method perform on attacks that specifically exploit the disconnect between visual and textual information? The authors should evaluate their approach against more sophisticated adversarial examples that specifically target the vision-text interface.\n\nDo transformer-based VLLMs exhibit different safety characteristics compared to other architectures? How do different training procedures (instruction tuning, RLHF) affect the safety paradox? The authors should also investigate whether the observed phenomena hold for more recent models with improved safety training.\n\nGiven the identified limitations in existing evaluation approaches, human judgment could provide crucial ground truth for assessing both attack success and defense effectiveness. Additionally, the paper would benefit from exploring the computational overhead and practical deployment considerations of the proposed LLM-Pipeline approach.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates what it terms the \"safety paradox\" in Vision Large Language Models (VLLMs): the observation that both jailbreak attacks and subsequent defenses demonstrate surprisingly high, near-perfect performance on standard benchmarks. The authors argue this \"dual-ease\" situation masks deeper issues. The paper makes three primary contributions. First, it posits and analyzes a core reason for VLLM vulnerability: the mere inclusion of visual inputs degrades the inherent safety guardrails of the backbone LLM. Second, it identifies and critiques a significant flaw in current defense mechanisms, which the authors call \"over-prudence,\" where defended models become overly cautious and refuse to answer even benign queries, thus harming their utility. Third, the paper proposes a simple yet effective defense method, LLM-Pipeline, which uses a state-of-the-art text-only LLM as a pre-emptive \"vision-free\" detector to filter harmful prompts before they are processed by the VLLM. Finally, the authors raise a critical methodological concern, finding that common evaluation methods for jailbreaking often disagree, potentially leading to misleading conclusions about the efficacy of attack and defense strategies.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. The concept of the \"safety paradox\" is a novel and highly insightful way to frame the current state of VLLM safety research. It moves the conversation beyond a simple cat-and-mouse game of attacks and defenses and encourages a more fundamental look at the underlying reasons for the observed phenomena. \n2. The core hypothesis—that the inclusion of vision inputs is the primary culprit for degraded safety—is clear, intriguing, and (presumably) empirically testable. This provides a focused and valuable direction for investigation.\n\n[Weaknesses]\nThe LLM-Pipeline introduces an additional, large model call, which inevitably increases latency and computational cost. For real-world applications, this is a significant drawback. The introduction does not acknowledge this trade-off, which should be a key part of the discussion when comparing it to other, more integrated defense methods.\n\nQUESTIONS:\n1. In Section 3, the authors draw rationale-based conclusions using the VLGuard dataset and a LLaVA-based VLLM. However, I am concerned that a substantial portion of the \"harmful\" instructions in VLGuard consist of queries like “identify the people in this image,” which are neither particularly harmful nor diverse. This raises the question of whether the observed performance reflects true safety understanding or merely the ability to distinguish a narrow class of unsafe queries. Could you provide results on other, more comprehensive over-refusal datasets, such as MMSafetyBench, to support the generalizability of their findings? Additionally, do similar conclusions hold when using other VLLMs, such as LLaMA-Vision-Instruct or Qwen-VL?\n\n2. In section 4.3, \"A Simple Defense Baseline,\" how does the defense baseline relate to the proposed defense method? The dataset selection and defense prompt design in this section seem somewhat confusing and may lead to misunderstandings.\n\n3. In Section 5, the authors claim that the proposed defense mechanisms generalize well to unseen jailbreak datasets. However, the evaluation appears limited in scope. Could the authors provide further evidence or discussion on how these mechanisms perform across a broader set of datasets and attack strategies? This would help validate the robustness and general applicability of their approach.\n\n4. The paper briefly touches on the limitations of evaluation methods in Section 5, noting a low agreement between rule-based and LLM-based approaches. To strengthen this discussion, could the authors delve deeper into the potential causes of this disagreement? Additionally, what possible improvements or alternative evaluation strategies could be proposed to enhance consistency and reliability in safety assessment?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates what it terms the \"VLLM Safety Paradox\"—the observation that Vision-LLMs (VLLMs) appear both highly vulnerable to jailbreak attacks and easily defended according to benchmark results. The authors argue that this paradox arises from three core issues: 1) VLLM vulnerability is caused by the inclusion of vision inputs degrading the safety alignment of the base LLM; 2) current defense mechanisms are \"over-prudent,\" often refusing benign prompts to achieve high safety scores, thus harming utility; and 3) standard evaluation metrics for jailbreaks show poor agreement, questioning the validity of reported results. As a potential solution, the paper proposes `LLM-Pipeline`, a defense that uses a separate, powerful LLM as a vision-free safety filter before passing prompts to the VLLM.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+   **Interesting Problem Framing:** The paper frames the VLLM safety problem in an intriguing way by highlighting the apparent contradiction between ease of attack and ease of defense. This perspective encourages a more critical look at the field's current evaluation practices.\n+   **Critique of Evaluation Methods:** The analysis in Section 4.3, which reveals a mere chance-level correlation between rule-based and model-based evaluation metrics, is a valuable contribution. It provides concrete evidence that the community should be cautious when interpreting and comparing benchmark scores for VLLM safety.\n+   **Identification of Over-Prudence:** The paper provides a systematic study of the \"over-prudence\" or over-refusal problem in VLLM defenses, quantifying how existing methods sacrifice helpfulness on benign inputs to achieve high safety scores on malicious ones.\n\n### Weaknesses\n+   **Incremental Novelty:** The core concepts discussed are not entirely new. The trade-off between model helpfulness and safety is a well-known phenomenon in the broader LLM literature, often termed the \"alignment tax.\" The paper re-contextualizes this known issue for VLLMs without offering a fundamentally new theoretical framework. The \"paradox\" feels more like a restatement of this known trade-off than a new discovery.\n+   **Oversimplified and Impractical Solution:** The proposed `LLM-Pipeline` is essentially a form of cascaded inference, using a larger/stronger model to guard a weaker one. This is a fairly obvious, brute-force approach rather than a novel defense mechanism. More importantly, the paper completely fails to analyze its practical viability, ignoring the significant latency and cost overhead of adding a full LLM inference step to every query. This omission severely limits the practical significance of the proposed solution.\n+   **Insufficient Evidence for Causal Claims:** The central claim that the *inclusion of vision inputs* is the primary cause of safety degradation is not conclusively proven. The t-SNE visualizations are illustrative but do not establish causality. The experiments do not adequately control for confounding factors, such as the specific content of the multimodal instruction-tuning datasets used to create VLLMs, which could be the true source of safety erosion rather than the visual modality itself.\n+   **Limited Scope of Experiments:** The empirical evaluation relies heavily on LLaVA-1.5 models, which are now somewhat dated.\n\nQUESTIONS:\n1.  The proposed `LLM-Pipeline` defense appears to have significant practical drawbacks (cost, latency). Why was this not analyzed? Could you provide an estimate of this overhead and comment on whether the proposed defense is viable for any real-world application?\n2.  How can you be certain that the visual modality itself, rather than the content and nature of multimodal fine-tuning data, is the root cause of the observed safety degradation? What experiments could be run to definitively disentangle these two factors?\n3.  The paper frames the problem as a \"paradox.\" However, isn't it equally plausible that this is simply the well-known safety-utility trade-off, where trivial defenses (e.g., \"always refuse\") achieve high safety scores at the cost of zero utility? How is the \"paradox\" framing a more insightful or useful lens for this problem?\n4.  Your evaluation of the low agreement between metrics is insightful. Have you considered proposing a more reliable evaluation metric or framework as a contribution, rather than the simple pipeline defense? This seems to be a more direct solution to one of the core problems you identified.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}