Title: JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI safety

Adversarial attack and defense

Jailbreak

Multimodal learning
Abstract: Vision-Language Models (VLMs) exhibit impressive performance, yet the integration of powerful vision encoders has significantly broadened their attack surface, rendering them increasingly susceptible to jailbreak attacks. However, lacking well-defined attack objectives, existing jailbreak methods often struggle with gradient-based strategies prone to local optima and lacking precise directional guidance, and typically decouple visual and textual modalities, thereby limiting their effectiveness by neglecting crucial cross-modal interactions. Inspired by the Eliciting Latent Knowledge (ELK) framework, we posit that VLMs encode safety-relevant information within their internal fusion-layer representations, revealing an implicit safety decision boundary in the latent space. This motivates exploiting boundary to steer model behavior. Accordingly, we propose \textbf{JailBound}, a novel latent space jailbreak framework comprising two stages: (1) \textbf{Safety Boundary Probing}, which addresses the guidance issue by approximating decision boundary within fusion layer's latent space, thereby identifying optimal perturbation directions towards the target region; and (2) \textbf{Safety Boundary Crossing}, which overcomes the limitations of decoupled approaches by jointly optimizing adversarial perturbations across both image and text inputs. This latter stage employs an innovative mechanism to steer the model's internal state towards policy-violating outputs while maintaining cross-modal semantic consistency. Extensive experiments on six diverse VLMs demonstrate JailBound's efficacy, achieves 94.32\% white-box and 67.28\% black-box attack success averagely, which are 6.17\% and 21.13\% higher than SOTA methods, respectively. Our findings expose a overlooked safety risk in VLMs and highlight the urgent need for more robust defenses. \textcolor{red}{Warning: This paper contains potentially sensitive, harmful and offensive content.}


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces JailBound, a novel jailbreaking framework targeting Vision-Language Models (VLMs) by exploiting internal safety decision boundaries within their latent representations. The approach consists of two main stages: (1) Safety Boundary Probing, which uses logistic regression classifiers to approximate the decision boundary in fusion layer latent space, and (2) Safety Boundary Crossing, which jointly optimizes adversarial perturbations across both image and text inputs to steer the model's internal state toward policy-violating outputs. The method demonstrates better performance compared to existing approaches across six diverse VLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The work addresses a critical safety concern in VLMs, which are increasingly deployed in real-world applications. The high success rates highlight the practical importance of this vulnerability.\n- The application of ELK principles to multimodal jailbreaking is novel. The idea of explicitly probing and exploiting safety decision boundaries in fusion layers is original.\n- The experimental evaluation is comprehensive, covering six different VLMs across both white-box and black-box scenarios.\n\n**Weaknesses**\n- The assumption that safety decisions can be linearly separated in the fusion space may not hold for all VLMs or safety categories. The paper would benefit from discussing when this assumption might fail.\n- The experiments are primarily conducted on the MM-SafetyBench dataset, which may not fully represent the diversity of real-world safety scenarios.\n\nQUESTIONS:\n- Limited discussion is provided on potential defenses or detection mechanisms that could mitigate the proposed attack.\n- A thorough computational analysis would help practitioners understand the practical feasibility of the attack and could inform the development of efficient defense mechanisms.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a jailbreaking method based on the hypothesis that VLMs encode safety-relevant information in their fusion-layer representations. The proposed method works by probing the safety boundary in a VLM's fusion layer and then creating adversarial perturbations to both textual and image inputs, to move the joint representation towards the 'safe' region. Experiments show that the proposed method effectively succeeds in jailbreaking, surpassing other methods in attack success rates.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nS1: The key idea is clear, straightforward, and sound.\n\nS2: This paper is well-written, and I commend the authors for the clear delivery of core concepts.\n\nS3: Experiments show a clear and significant improvement over other baselines in terms of attack success rates.\n\n## Weaknesses\n\nW1: Computational burden - The proposed method requires >100 steps of gradient-based optimization for each input modality, which may limit the practicality of the proposed method, particularly for larger models. \n\nW2: Regarding W1, I think the comparison of computational costs for the proposed method and other baselines should be shown for a fair and transparent comparison.\n\nW3: The proposed method relies on a key hypothesis that there exists a linear safety decision boundary in VLMs. While the success of the linear probing and the proposed method indirectly supports that this can be true, the hypothesis is not investigated in depth, limiting our understanding of how VLM works. For example, discussing why and how the linear boundary exists (e.g., there could be a connection to the linear representation hypothesis) and why it appears in the fusion layer rather than other components would provide us with deeper insights.\n\nQUESTIONS:\nPlease see the questions above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a new jailbreaking attack against VLMs called JailBound. JailBound trains binary probing classifiers to linearly separate safe and unsafe representations on every layer nearly perfectly. The authors believed that these classifiers represent VLMs' latent knowledge about safety regardless of the VLMs' responses to these representations, therefore attacking the classifiers by breaching their decision boundaries is considered a good surrogate task for devising a transferrable jailbreaking. In practice, for each harmful text-image pair, JailBound optimizes for minimal textual and visual perturbations that induce sufficient representation movement in the direction perpendicular to the decision boundaries by combining the losses $L_{sem},L_{align}$ and $L_{geo}$ and using gradient descent and gradient-based token replacement for updates on visual and textual perturbations respectively. The transferability of JailBound was tested by competing with baseline attacks like UMK, FigStep and VAJM on attacking GPT, Gemini and Claude models using the MM-SafetyBench dataset. According to the reported ASRs (measured as $1-$ lexical refusal rates), JailBound transfers significantly better than all baselines consistently with very rare exceptions. The authors didn't put as many efforts in comparing JailBound's whitebox performances with these baselines. They shared an experiment on MiniGPT-4 where JailBound outperformed all the baseline, and compared JailBound with unimodal variants of JailBound in the whitebox setting to confirm that the joint optimization of both modalities is conducive.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n+ The idea of making use of probing classifiers for jailbreaking is original yet straightforward.\n+ The paper is generally well-written and easy to follow, thanks to the sound logic flow, consistent notation uses and illustrations.\n+ The paper is generally wholesome and self-contained. For instance, the validation of know-say disparity, although saved for appendix, is helpful for understanding the background about the paper.\n\nWeakness:\n+ The experiment designs for the blackbox and whitebox settings are not entirely consistent. While it is interesting to know how well the full fledged JailBound compares with the unimodal variants, it is more important to know its comparison with the baselines beyond MiniGPT-4.\n\nQUESTIONS:\nQ1: What comprises $\\mathcal{L}_\\text{suffix}$ and how was it integrated into the optimization pipeline? The computation of $\\mathcal{L}_\\text{suffix}$ and $\\mathcal{L}_\\text{sem}$ where it belong is missing from Algorithm 2 either.\n\nQ2: Is it true that if a jailbreaking attempt is successful, it also successfully confuses the probing classifiers? For the attack baselines, can you apply the classifiers on their text-image pairs? If a significant number of successful jailbreaking attempts don't break the classifiers, then JailBound points out a new attack surface but isn't a framework that covers existing attacks yet.\n\nQ3: For unimodal attacks, in fact there are a lot of text-only jailbreaking methods that are impressively effective. Many of them are tested against VLMs like GPT-4o except that the visual input is empty. If the authors really want to compare with unimodal attacks, I would like to know how well JailBound compares to them.\n\nQ4: Before people apply probing classifiers for jailbreaking, in fact they have been used for defending against jailbreaking. For instance, Circuit Breakers from Gray Swan maps unsafe representations back to safe ones. How well do the proposed method attack their models?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes JailBound, a jailbreaking framework targeting the latent safety decision boundaries of vision-language models (VLMs). It employs a two-stage approach: Safety Boundary Probing to identify perturbation directions in the fusion layer’s latent space and Safety Boundary Crossing to jointly optimize image and text perturbations using a multi-objective loss function. Evaluated on the MM-SafetyBench dataset, JailBound achieves high ASRs in white-box settings and in black-box scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The proposed two-stage framework leverages latent safety boundaries to enhance jailbreaking success rates.\n\n2. Experiments on MM-SafetyBench demonstrate robust ASR across multiple VLMs.\n\n3.  The whole paper is clearly written.\n\n**Weaknesses:**\n\n1. I’m concerned about the claimed novelty, as JailBound appears to combine steering vectors (the boundary normal is the vector direction) with optimization-based jailbreaks without clearly articulating the advantages of this fusion. And we know a steering vector derived from the probing stage could achieve comparable ASR without the optimization step. So, why is the two-stage approach necessary?\n\n2. The paper overlooks critical references, such as [1] on representation engineering and [2, 3] on safety-violating steering vectors, which are highly relevant to the work.\n\n3. The explanation for black-box transferability is insufficient. It’s unclear why this pipeline excels in black-box settings. \n\n4. Experimental details are also incomplete, particularly regarding the data used for probing. Were these the same as the validation set, and if so, how does this avoid overfitting to MM-SafetyBench?\n\n5. The evaluation’s persuasiveness is limited by testing on only MM-SafetyBench. Additional experiments on a diverse dataset like JailbreakV-28k would strengthen the claims of generalizability.\n\n6. Figure 5’s case study, described as real-world, uses cartoonish images rather than authentic screenshots, and with only two examples provided, it’s challenging to find the evidence convincing.\n\n7. Some formatting issues, such as the incorrect citation on line 144.\n\n[1] Zou, Andy, et al. \"Representation engineering: A top-down approach to ai transparency.\"\n\n[2] Arditi, Andy, et al. \"Refusal in language models is mediated by a single direction.\" \n\n[3] Yu, Lei, et al. \"Robust LLM safeguarding via refusal feature adversarial training.\"\n\nQUESTIONS:\nPlease see the weaknesses. If authors could address my concerns, especially the concern of novelty, I would raise my rating.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}