Title: Towards Building Model/Prompt-Transferable Attackers against Large Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision-Language Models; Adversarial Attack; Transferability
Abstract: Although Large Vision-Language Models (LVLMs) exhibit impressive multimodal capabilities, their vulnerability to adversarial examples has raised serious security concerns.
Existing LVLM attackers simply optimize adversarial images that easily overfit a certain model/prompt, making them ineffective once they are transferred to attack a different model/prompt. 
Motivated by this research gap, this paper aims to develop a more powerful attack that is transferable to black-box LVLM models of different structures and task-aware prompts of different semantics. 
Specifically, we introduce a new perspective of information theory to investigate LVLMs' transferable characteristics by exploring the relative dependence between outputs of the LVLM model and input adversarial samples. Our empirical observations suggest that enlarging/decreasing the mutual information between outputs and the disentangled adversarial/benign patterns of input images helps to generate more agnostic perturbations for misleading LVLMs' perception with better transferability. 
In particular, we formulate the complicated calculation of information gain as an estimation problem and incorporate such informative constraints into the adversarial learning process.
Extensive experiments on various LVLM models/prompts demonstrate our significant transfer-attack performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a critical issue of transferable adversarial attacks against LVLMs, by introducing a novel perspective of mutual information (MI). The core insight is that the informative dependence between the LVLM output and input images can be influenced by interactions between image and text prompts. With appropriate informative constraints between the disentangled adversarial/benign patterns of the image input and output text, the generated adversarial examples are proven to be more generalizable and harmful to unseen LVLMs and prompts. Extensive experiments are validated across various LVLM models and prompts with multiple LVLM datasets/tasks compared with existing LVLM attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The technical approach presented is novel and well-motivated. It proposes to improve the adversarial transferability across models/prompts using MI-based informative constraints. I think it is meaningful for the LVLM adversarial field. \n2. Theoretical analyses are provided to support the proposed framework. The illustrated attack process is clear and easy to follow.\n3. Sufficient experiments provided in both main paper and appendix validate the effectiveness of the proposed method.\n\nWeaknesses\n\n1. The paper motivates that adversarial transferability is related to the MI values between adversarial/benign patterns and LVLM’s outputs. Although Figure 2 verifies this assumption, the authors should provide more discussion on it.\n2. Target text selection: Using \"I am sorry\" as the target text may not truly represent a targeted attack. It would be valuable to include experiments with more specific target texts and report the success rates for such cases.\n3. The authors are suggested to provide a case transfer-attack experiment across both models/prompts at the same time.\n4. It seems that the proposed method achieves better transfer-attack performance but indicates reduced effectiveness in non-transfer attack scenarios. Although we know that it is due to the overfitting of previous attack methods, the authors could provide more discussion on this point.\n\nQUESTIONS:\n1. The paper motivates that adversarial transferability is related to the MI values between adversarial/benign patterns and LVLM’s outputs. Although Figure 2 verifies this assumption, the authors should provide more discussion on it.\n2. Target text selection: Using \"I am sorry\" as the target text may not truly represent a targeted attack. It would be valuable to include experiments with more specific target texts and report the success rates for such cases.\n3. The authors are suggested to provide a case transfer-attack experiment across both models/prompts at the same time.\n4. It seems that the proposed method achieves better transfer-attack performance but indicates reduced effectiveness in non-transfer attack scenarios. Although we know that it is due to the overfitting of previous attack methods, the authors could provide more discussion on this point.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a model- and prompt-transferable LVLM attack from an information-theoretic perspective. It then proposes an MI constraint, estimated by two networks in practice, to improve the transferability of adversarial examples by increasing the dependence of model output on the perturbation and decreasing the dependence of model output on the initial image.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Clear motivation. This work aims to investigate and improve transferability through different models and prompts.\n2. Adequate empirical studies. Ablations under different settings validate the generalization of this method.\n3. The perspective from information theory is interesting.\n\n**Weaknesses**\n\n1. Typo. In Equ. (2), the left and right brackets do not match. The formula is hard to read, though the correct version can be found in the appendix.\n2. The subsequent approximation step (lines 181-188) is problematic. Sepcifically, given $x_v^{adv} = x_v + \\Delta$, if viewing $x_v^{adv}, x_v, \\Delta$ as three random variables as in line 177, then $p(x_v^{adv})=p(x_v, \\Delta)$ ***generally does not hold***, thus the two term $H(L'|x_v, \\Delta), H(L', x_v^{adv})$ **cannot cancel out**. Even in the simplest case, where $x_v, \\Delta$ is independent (which is  inaccurate, since the $\\Delta$ depends on the initial image $x_v$ and the target), we have the PDF of $x_v^{adv}$ as  $f_{x_v^{adv}}(x)=\\int_{-\\infty}^{\\infty} f_{x_v}(y) \\cdot f_{\\Delta}(x-y) dy$. In the dependent case, this is more complicated. Without further assumption, replacing $p(x_v^{adv})$ with $p(x_v, \\Delta)$ is incorrect. Therefore, more than two MI terms are left after the approximation step.\n3. The ablation study of different target texts is crucial yet insufficient. The metric used in Table 10 is missing, and only a few target texts are provided without comparison to other baselines. Without knowing the baseline performance, the advantage and overall effectiveness of the proposed method under this setting lack an anchor point.\n\nQUESTIONS:\n1. The overall optimization framework does not explicitly include the prompts. How does transferability improve when using the proposed MI constraints?\n> In lines 221 to 228, the main idea is that the model's output depends more on the perturbation. However, a sharp turn without further explanation (like the phrase 'in this manner') suggests that the method becomes transferable to different models or prompts. \n\n2. What is the relationship between the proposed method and other approaches that aim to improve the transferability of adversarial examples, such as augmentation-based and ensemble-based methods? This connection is not clearly explained in the related work. If the proposed method operates alongside existing methods, combining the two strategies in experiments could further enhance the effectiveness of this method as a 'plug and play' module. If the proposed method improves upon these approaches, then incorporating them into the baseline is necessary. \n\nIf the weakness can be fixed and the questions answered, I would like to improve my score. Right now, there are several flaws that lessen the effectiveness of this work below an acceptable level.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel attack method for Large Vision-Language Models (LVLMs) that enhances the transferability of adversarial examples across different models and prompts. Considering enlarging/decreasing the mutual information between outputs and the disentangled adversarial/benign patterns of input images helps to generate more agnostic perturbations for misleading LVLMs’ perception with better transferability, the paper proposes a novel perspective of information theory to improve the LVLM attacks’ transferability. Experiments across four LVLMs (LLaVA-1.5, MiniGPT-4, BLIP-2, InstructBLIP) and three datasets (DALL-E, SVIT, VQAv2) demonstrate significant improvements in cross-model and cross-prompt attack success rates and semantic similarity metrics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper addresses a significant gap in the field of adversarial attacks on LVLMs, offering a more practical and versatile approach compared to existing model/prompt-specific methods. The transferable nature of the attack and its ability make it particularly relevant for real-world applications and raise important questions about the security of LVLMs.\n\n- Using Information theory to address the transferability of LVLM is novel.\n- The proposed method is technically sound and comprehensive experiments demonstrate the empirical performance.\n- The safety concerns of LVLMs exposed in this paper could potentially have a large impact on the AI safety field.\n\n**Weaknesses**\n\nThe reviewer finds that this paper is novel and provides valuable insights to the community. However, there are several areas where the paper could be improved:\n\n- There are various existing LVLM attacks are re-implemented for comparison. Do they re-implemented in the same setting for fair comparisons? More details are required.\n- How to train the two MI estimation networks? Will they introduce additional noticeable costs? The authors should provide more corresponding analysis.\n- A commonly implemented image-level universal setting should also be experimented.\n- Two widely used baselines should be implemented for comparisons: “On evaluating adversarial robustness of large vision-language models” and “How Robust is Google's Bard to Adversarial Image Attacks?”\n- The authors did a good job by providing sufficient discussion and experiments in the appendix. The reviewer suggests including the justification of fair transfer attacks in the main paper.\n\nQUESTIONS:\nThe reviewer's main concerns are these points and questions outlined in the weaknesses section. The reviewer suggests that the authors focus on these points in the rebuttal.\n\nLIMITATIONS:\nLimitations and potential negative impacts have been sufficiently discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical challenge of transferability in adversarial attacks against LVLMs, noting that existing methods often overfit to specific models and prompts, limiting their practical effectiveness. The authors propose a novel approach from an information theory perspective. The core idea is to disentangle the adversarial perturbation from the benign image content and explicitly manipulate the model's reasoning process by controlling the mutual information between the input components and the model's output.  Extensive experiments on multiple LVLMs and datasets, including transfers to black-box models, demonstrate that the proposed method significantly outperforms existing attacks in both cross-model and cross-prompt scenarios, underscoring the effectiveness of their approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.The proposed method achieves state-of-the-art performance in transfer attacks across multiple LVLM benchmarks.\n2.The paper provides a comprehensive set of experiments, including extensive comparisons and ablation studies, which effectively demonstrate the method's strong performance.\n\nWeaknesses\n\n1.The analysis of robustness against defenses is incomplete. The experiments only demonstrate resilience in a white-box setting and lack an evaluation of how the proposed cross-model and cross-prompt attacks perform when transferred to defended target models.\n2.The study lacks transferability experiments against other architecturally distinct model families, such as InternVL, which would provide a more complete assessment of the method's generalizability.\n\nQUESTIONS:\nGiven the impressive transfer results to Qwen2-VL, how does the proposed method's transferability perform when targeting another architecturally distinct and powerful model, such as InternVL?\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}