Title: M-Attack-V2: Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting
Venue: NeurIPS 2025
Accepted: no
Keywords: Large Vision-Language Model

Adversarial Attack

Muti-modality Models
Abstract: Black-box adversarial attacks on Large Vision–Language Models (LVLMs) present unique challenges due to the absence of gradient access and complex multimodal decision boundaries. While prior \texttt{M-Attack} demonstrated notable success with exceeding 90\% attack success rate on GPT‑4o/o1/4.5 by leveraging local crop-level matching between source and target data, we show this strategy introduces high-variance gradient estimates. Specifically, we empirically find that gradients computed over randomly sampled local crops are nearly orthogonal, violating the implicit assumption of coherent local alignment and leading to unstable optimization.
To address this, we propose a theoretically grounded {\bf \em gradient denoising} framework that redefines the adversarial objective as an expectation over local transformations. Our first component, \emph{Multi-Crop Alignment (MCA)}, estimates the expected gradient by averaging gradients across diverse, independently sampled local transformations. This manner significantly reduces gradient variance, thus enhancing convergence stability. Recognizing an asymmetry in the roles of source and target transformations, we also introduce \emph{Auxiliary Target Alignment (ATA)}. ATA regularizes the optimization by aligning the adversarial example not only with the primary target image but also with auxiliary samples drawn from a semantically correlated distribution. This constructs a smooth semantic trajectory in the embedding space, acting as a low-variance regularizer over the target distribution. Finally, we reinterpret prior momentum as replay through the lens of local matching as variance-minimizing estimators under the crop-transformed objective landscape. Momentum replay stabilizes and amplifies transferable perturbations by maintaining gradient directionality across local perturbation manifolds. Together, MCA, ATA, momentum replay, and a delicately selected ensemble set constitute \texttt{M-Attack-V2}, a principled framework for robust black-box LVLM attack. Empirical results show that our framework improves the attack success rate on GPT‑4o from {\bf 95\%$\rightarrow$99\%}, on Claude-3.7, and on Gemini-2.5-Pro from {\bf 83\%$\rightarrow$97\%}, significantly surpassing all existing black-box LVLM attacking methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces M-Attack-V2, a gradient denoising framework. It has key components: Multi-Crop Alignment (MCA) averages gradients from diverse local transformations to cut variance and enhance convergence. Aux Target Alignment (ATA) aligns adversarial examples with both the main target and semantically related auxiliary samples, creating a smooth path in the embedding space. Momentum replay stabilizes and amplifies perturbations by maintaining gradient direction across local perturbation manifolds.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: This work proposed to address the gradient instability in black-box adversarial attacks on Large Vision - Language Models (LVLMs). It innovatively uses modular components: one part reduces gradient variance and enhances convergence by averaging gradients from diverse local transformations, another aligns adversarial examples with both the main target and semantically related auxiliary samples to create a smooth path in the embedding space.\n\n**Weakness**\n\n1. **Out-of-Distribution Data**: M-Attack-V2 depends on surrogate model consistency and semantic similarity of auxiliary samples. OOD data may disrupt these, causing unstable gradient estimation and reduced attack effectiveness. This paper lacks OOD validation.\n2. **Higher Computational Cost**: Generating auxiliary images for ATA and processing multiple crops in MCA increase computational overhead, making it less suitable for resource-constrained environments.\n\nQUESTIONS:\n1. Have you evaluated M-Attack-V2 on OOD datasets to verify its performance, given the framework's reliance on surrogate model consistency and auxiliary sample similarity that may fail in OOD scenarios?\n2. The works lacks specific comparisons of computational complexity between M-Attack-V2 and other black-box LVLMs attack methods (e.g., AttackVLM, M-Attack).\n3. In Line 83, the word \"Supursingly\" is misspelled.\n4. In line 103, there is an extra letter \"d\" at the beginning of the sentence, which should be removed. \n\nThe motivation of the article is quite reasonable, and it is hoped that the authors can further explain how to address potential issues in specific deployment scenarios.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces M-Attack-V2, an improved transfer-based black-box adversarial attack on LVLMs. Building on M-Attack, the authors introduce three core components to stabilize gradient estimation:\n\n- MCA: Averages gradients over multiple local crops to reduce variance.\n\n- ATA: Regularizes with gradients from semantically similar auxiliary images\n\n- PM: Reinterprets momentum as replay over past crop-level gradients, maintaining directional consistency despite stochastic local views.\n\nEmpirical evaluation on four commercial LVLMs (GPT-4o, Claude-3.7, Gemini-2.5-Pro, LLaVA-1.5) are promising over 100 testing images.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- The research problem on black-box LVLMs is interesting and critical.\n- The proposed method is well motivated by analysing the weakness of M-Attack on the high-variance gradient estimates. The three proposed modules are reasonable and intuitive.\n- The M-Attack-2 demonstrates strong empirical gains while it is efficient with early-stopping reducing query counts by 20–30% compared to baselines.\n\n### **Weaknesses**\n\n- Limited Dataset Scope: Evaluation on only 100 images (NIPS 2017 + COCO) may not reflect performance on diverse, real-world domains (e.g., medical, satellite).\n- MCA and ATA can introduce computational overhead.\n- Little analysis of how auxiliary selection quality or size impacts ATA’s effectiveness. If this data is not carefully prepared, how this noisy auxiliaries might degrade the attack's performance.\n- Lack of qualitative comparisons or visualisation to clean image. I am curious if the optimised perturbations are still invisible compared to other attacks or even the clean image? I believe an user studies can help confirm the imperceptibility.\n-  Defence and Perceptual Analysis Missing. No experiments against simple input preprocessing defences (e.g., JPEG compression)\n\nQUESTIONS:\n1. Can we have a more comprehensive evaluation with more images covering more real-world domains (e.g., medical, satellite)?\n2. Can we have a theoretical or empirical analysis of computational overhead of the proposed method?\n3. Are the perturbations imperceptible?  Can we have an user studies to confirm the imperceptibility?\n4. Since the M-Attack-V2 aims to introduce perturbation to the clean image, can input preprocessing serve as a defence? I suggest the authors to try some common method such as JPEG compression or DiffPure [1]\n\n\n[1] Nie, Weili, et al. \"Diffusion models for adversarial purification.\" arXiv preprint arXiv:2205.07460 (2022).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work identifies and addresses the issue of high-variance gradient estimates in black-box adversarial attacks on LVLMs that arise from using a local crop matching strategy in previous *M-Attack* method. The authors propose **M-Attack-V2**, a framework that incorporates *Multi-Crop Alignment (MCA)* to reduce variance, *Auxiliary Target Alignment (ATA)* to ensure semantic consistency, *Patch Momentum (PM)* to stabilize gradient optimization, and a refined surrogate model ensemble *Patch Ensemble+ (PE+)*.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n* The technical approach is meaningful and the motivation is strong. The authors demonstrate that the gradients computed from randomly sampled local crops are nearly orthogonal, leading to unstable optimization. Then they propose a framework named **M-Attack-V2** and have solved the problem well.\n* The experiment is comprehensive. The authors test their method against multiple frontier commercial LVLMs and achieve significant performance improvement. They also have ablation studies on each component and key hyperparameters to show the effectiveness of their methods.\n### **Weaknesses**\nIn this work, the analysis of the *Auxiliary Target Alignment (ATA)* remains at an intuitive level. The authors claim that ATA provides \"low-variance, semantics-preserving updates\" and \"constructs a smooth semantic trajectory in the embedding space\", but they offer no theoretical analysis to demonstrate how these properties are achieved.\n\nQUESTIONS:\n* The ablation study in *Tab. 4* indicates that removing **Patch Momentum (PM)** results in a far greater performance drop than removing MCA. Given that the paper's motivation heavily emphasizes the orthogonal gradient problem, which MCA is introduced to address, could you clarify why PM has a much more dominant effect on the overall performance? Does this suggest that gradient orthogonality might not be the primary factor limiting performance, as you initially framed?\n* Have you compared the computational cost (e.g., time per image) between M-Attack and M-Attack-V2?\n\nLIMITATIONS:\nYes, the limitations is in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the blakc-box attacks on existing MLLMs. It proposes a several solutions to mitigate the limitations in its V1 version. However, because there is a lack of problem formulation and introduction of their V1 work. The overall paper is extremely hard to follow.\n\nSTRENGTHS AND WEAKNESSES:\n***Strength***:\n1. The paper focuses on an important research problem.\n2. The paper conductes comprehensive experiments, and achieves noticeable performance improvment.\n\n***Weakness***:\n\n*** A lack of Background and Problem Formulation for M-Attack makes the submission difficult to understand ***\n\nThe overall writing structure is somewhat unfriendly to readers who are not already familiar with M-Attack. I attempted to check the reference for M-Attack, but found that the citation is broken, making it difficult to trace the original work.\n\nHere are some mild suggestions:  the authors should at least provide a concise technical summary of M-Attack (even in the appendix). More importantly, a rigorous research paper—even if it is a MARK II version—should include a dedicated subsection that clearly formulates the research problem and defines their threat model. While such context may become unnecessary if M-Attack is widely recognized as a standard baseline in this area, at present, the lack of background makes it hard for readers unfamiliar with M-Attack (like myself) to fully follow the paper’s motivation and methodology. For example, what are X_tar, X_sou and X_adv defined in the first paragraph of Method? What are image pairs and region pairs mentioned in the paper? Why include IoU in the motivation experiment?\n\nQUESTIONS:\nIn general, after reading the paper, I felt more like I was reading an appendix or extension of the M-Attack paper, rather than an independent new paper. As a result, I can only assess this submission with the lowest confidence score. I strongly encourage the authors to include a more thorough introduction to M-Attack in their revised version, so that the paper can be better understood by readers who are not already familiar with that prior work.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}