Title: A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adversraial Attack

Local-level Matching

Vision-Language Models
Abstract: Despite promising performance on open-source large vision-language models (LVLMs), transfer-based targeted attacks often fail against closed-source commercial LVLMs. Analyzing failed adversarial perturbations reveals that the learned perturbations typically originate from a uniform distribution and lack clear semantic details, resulting in unintended responses. This critical absence of semantic information leads commercial black-box LVLMs to either ignore the perturbation entirely or misinterpret its embedded semantics, thereby causing the attack to fail. To overcome these issues, we propose to refine semantic clarity by encoding explicit semantic details within local regions, thus ensuring the capture of finer-grained features and inter-model transferability, and by concentrating modifications on semantically rich areas rather than applying them uniformly. To achieve this, we propose  *a simple yet highly effective baseline*: at each optimization step, the adversarial image is cropped randomly by a controlled aspect ratio and scale, resized, and then aligned with the target image in the embedding space. While the naive source-target matching method has been utilized before in the literature, we are the first to provide a tight analysis, which establishes a close connection between perturbation optimization and semantics. Experimental results confirm our hypothesis. Our adversarial examples crafted with local-aggregated perturbations focused on crucial regions exhibit surprisingly good transferability to commercial LVLMs, including GPT-4.5, GPT-4o, Gemini-2.0-flash, Claude-3.5/3.7-sonnet, and even reasoning models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach achieves success rates exceeding 90\% on GPT-4.5, 4o, and o1, significantly outperforming all prior state-of-the-art attack methods with lower $\ell_1/\ell_2$ perturbations. Our optimized adversarial examples under different configurations are available at https://huggingface.co/datasets/MBZUAI-LLM/M-Attack_AdvSamples and our training code at https://github.com/VILA-Lab/M-Attack.

A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1

Abstract

Despite promising performance on open-source large vision-language models (LVLMs), transfer-based targeted attacks often fail against closed-source commercial LVLMs. Analyzing failed adversarial perturbations reveals that the learned perturbations typically originate from a uniform distribution and lack clear semantic details, resulting in unintended responses. This critical absence of semantic information leads commercial black-box LVLMs to either ignore the perturbation entirely or misinterpret its embedded semantics, thereby causing the attack to fail. To overcome these issues, we propose to refine semantic clarity by encoding explicit semantic details within local regions, thus ensuring the capture of finergrained features and inter-model transferability, and by concentrating modifications on semantically rich areas rather than applying them uniformly. To achieve this, we propose a simple yet highly effective baseline : at each optimization step, the adversarial image is cropped randomly by a controlled aspect ratio and scale, resized, and then aligned with the target image in the embedding space. While the naïve source-target matching method has been utilized before in the literature, we are the first to provide a tight analysis, which establishes a close connection between perturbation optimization and semantics. Experimental results confirm our hypothesis. Our adversarial examples crafted with local-aggregated perturbations focused on crucial regions exhibit surprisingly good transferability to commercial LVLMs, including GPT-4.5, GPT-4o, Gemini-2.0-flash, Claude-3.5/3.7-sonnet, and even reasoning models like o1, Claude-3.7-thinking and Gemini-2.0-flashthinking. Our approach achieves success rates exceeding 90% on GPT-4.5, 4o,

and o1, significantly outperforming all prior state-of-the-art attack methods with lower ℓ 1 /ℓ 2 perturbations. Our optimized adversarial examples under different configurations are available at HuggingFace and our training code at GitHub.

1 Introduction

Adversarial attacks have consistently threatened the robustness of AI systems, particularly within the domain of large vision-language models (LVLMs) [22, 4, 37]. These models have demonstrated impressive capabilities on visual and linguistic understanding integrated tasks such as image captioning [32], visual question answering [25, 29] and visual complex reasoning [21, 30]. In addition to the progress seen in open-source solutions, advanced black-box commercial multimodal models like GPT-4o [1], Claude-3.5 [3], and Gemini-2.0 [33] are now extensively utilized. Their widespread adoption, however, introduces critical security challenges, as malicious actors may exploit these platforms to disseminate misinformation or produce harmful outputs. Addressing these drawbacks necessitates thorough adversarial testing in black-box environments, where attackers operate with limited insight into the internal configurations and training data of the models.

Current transfer-based approaches [39, 8, 12] typically generate adversarial perturbations that lack semantic structure, often stemming from uniform noise distributions with low attack success rates on the robust black-box LVLMs. These perturbations fail to capture the nuanced semantic details that many LVLMs rely on for accurate interpretation. As a result, the adversarial modifications either go unnoticed by commercial LVLMs or, worse, are misinterpreted, leading to unintended and ineffective outcomes. This inherent limitation has motivated a deeper investigation into the nature and distribution of adversarial perturbations.

Our analysis reveals that a critical drawback in conventional adversarial strategies is the absence of clear semantic information within the perturbations. Without meaningful semantic cues, the modifications fail to influence the model's decision-making process effectively. This observation is particularly relevant for closed-source commercial LVLMs, which have been optimized to extract and leverage semantic details from both local and global image representations. The uniform nature of traditional perturbations thus represents a significant barrier to achieving high attack success rates.

Building on this insight, we hypothesize that a key to improving adversarial transferability lies in the targeted manipulation of core semantic objects present in the input image. Commercial blackbox LVLMs, regardless of their large-scale and diverse training datasets, consistently prioritize the extraction of semantic features that define the image's content. By explicitly encoding these semantic details within local regions and focusing perturbations on areas rich in semantic content, it becomes possible to induce more effective misclassifications. This semantic-aware strategy provides a promising view for enhancing adversarial attacks against robust, black-box models.

In this paper, we introduce a novel attack baseline called M-Attack that strategically refines the perturbation process. At each optimization step, the adversarial image is subjected to a random crop operation controlled by a specific aspect ratio and scale, followed by a resizing procedure. We then align the perturbations with the target image in the embedding space, effectively bridging the gap between local and local or local and global representations. The approach leverages the inherent semantic consistency across different white-box LVLMs, thereby enhancing the transferability of the crafted adversarial examples.

Furthermore, recognizing the limitations of current evaluation practices, which often rely on subjective judgments or inconsistent metrics, we introduce a new Keyword Matching Rate (KMRScore) alongside GPTScore. This metric provides a more reliable, partially automated way to measure attack transferability and reduces human bias. Our extensive experiments demonstrate that adversarial examples generated with our method achieve transfer success rates exceeding 90% against commercial LVLMs, including GPT-4.5, GPT-4o and advanced reasoning models like o1.

Overall, our contributions are threefold:

We observe that failed adversarial samples often exhibit uniform-like perturbations with vague details, underscoring the need for clearer semantic guidance to achieve reliable transfer to attack strong black-box LVLMs.

Iter:

!

Semantic

!

!"#$%

Iter:

!

+1

+

Resize

Local  Embedding

Global Embedding

Cosine

Similarity

Local-Global/Local

Alignment

"

!

Perturbation

!

!"#$%

" #

&

'

!(# $

!

"

)

!(# $

!

#

)

!

()*

&

...

Ensemble Models

!

!

!

!

!

"

!

!

#

+

Resize

!

+$,

&

Crop

Crop

"

"

" #

&

-

We show how random cropping with certain ratios and iterative local alignment with the target image embed local/global semantics into local regions, especially in crucial central areas, markedly boosting attack effectiveness.

We propose a new Keyword Matching Rate ( KMRScore ) that offers a more objective measure for quantifying success in cross-model adversarial attacks, achieving state-of-the-art transfer results with reduced human bias.

2 Related Work

Large Vision-Language Models. Transformer-based LVLMs integrate visual and textual modalities by learning joint visual-semantic representations from large-scale image-text datasets. These models have underlaid core multimodal tasks such as image captioning [32, 13, 7, 34], visual question answering [25, 29], and cross-modal reasoning [36, 26, 35]. Open-source LVLMs like BLIP-2 [20], Flamingo [2], and LLaVA [23] demonstrate good capabilities on standard benchmarks, while closed-source systems such as GPT-4o [1], Claude-3.7 [3], and Gemini-2.5 [33] exhibit better instruction-following, reasoning, and adaptation to real-world multimodal tasks. Despite these advances, the closed-source nature of commercial LVLMs conceals internal mechanisms and vulnerabilities, making it difficult to evaluate their robustness under adversarial scenarios. This calls for a systematic exploration of their susceptibility to carefully crafted input perturbations.

Transfer-Based Adversarial Attacks on LVLMs. Black-box attacks on LVLMs are either querybased [9, 15], relying on repeated API access to estimate gradients, or transfer-based [10, 24], which craft adversarial examples on surrogates without querying the target. While the latter is more efficient, transferability is hindered by the closed nature of commercial LVLMs, including undisclosed architectures and data, leading to significant semantic mismatches. Recent methods like AttackVLM [39] improve transfer success by aligning image-level features rather than cross-modal ones. This strategy influenced CWA [6] and SSA-CWA [8], which enhance transferability to models like Bard using sharpness-aware optimization and spectrum-based augmentation, achieving modest performance. Other approaches such as AnyAttack [38] and AdvDiffVLM [12] explore self-supervised pretraining and diffusion-based generation, but still struggle against leading commercial LVLMs. These limitations highlight the need for more stable, semantically grounded gradient strategies, which our method aims to address.

3 Investigations Over Failed Attacks

Weinvestigate why prior state-of-theart methods [39, 8, 38] have failed from two perspectives: 1) The perturbations from these methods tend to be uniformly distributed rather than highlighting statistically significant regions; 2) In many failed cases, the

model does detect the perturbation but is unable to articulate detailed semantic content, resulting in vague or ambiguous descriptions. Some failed examples are provided in Appendix H.2.

15

10

5

0

5

10

15

0.0

0.2

0.4

0.6

0.8

1.0

Cumulative Probability

Mean ECDF

Uniform

0

50

100

150

200

250

300

Step

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Similarity

Global to Global

Local to Global

Local to Local

Matching Method

GPT-4o

Gemini-2.0

Claude-3.5

Global-to-Global

0.05

0.05

0.01

Local-to-Global

0.93

0.83

0.22

Local-to-Local

0.95

0.78

0.26

Uniform-like Perturbation Distribution. Fig. 3 and Fig. 5 (first row) illustrate that the perturbation in failed adversarial examples closely aligns with a uniform distribution, as indicated by the near-overlap between empirical cumulative distribution function (ECDF) and the ideal uniform CDF over 20 samples. The minimal deviation and tight standard deviation bands suggest that perturbations are spread evenly across the image space without preference for semantically meaningful regions. This uniform-like behavior implies a lack of targeted manipulation toward critical visual features, leading to weak semantic interference and ultimately ineffective attacks on LVLMs. In other words, the model perceives

Step 1

Step 11

Step 21

Step 31

Step 41

Step 50

[0.1, 0.4]

[0.5, 0.9]

[0.1, 0.9]

Global

0.0

0.2

0.4

0.6

0.8

1.0

Perturbation Magnitude

these perturbations as noise rather than meaningful semantic shifts.

Vague Description. To further validate that the model perceives these uniform perturbations as noise rather than meaningful semantic shifts, we quantify the proportion of vague descriptions. Specifically, we define vague descriptions as cases where the model uses terms like 'blurry' or 'abstract' to describe the detected artifacts or perturbations, instead of concrete semantic nouns. As shown in Tab. 1, while the black-box closed-source LVLMs do detect something unusual in the image, they struggle to interpret it consistently and clearly.

Similarity Trajectories. We further visualize the evolution of similarity trajectories during training to understand why local matching is less prone to overfitting compared to previous global matching strategies, and why it more effectively attacks LVLMs. As shown in Fig. 4, we observe that global representations lack sufficient randomness, causing the similarity (i.e., negative loss) to increase rapidly and saturate early. This early saturation limits further learning. In contrast, local matching converges more slowly, allowing the model to capture finer-grained details throughout training.

4 Approach

Framework Overview. Our approach aims to enhance the semantic richness within the perturbation by extracting details matching certain semantics in the target image. By doing so, we improve the transferability of adversarial examples through a many-to-many/one matching, enabling them to remain effective against even the most robust black-box systems like GPT-4o, Gemini, and Claude. As shown in Fig. 2, at iteration i , the generated adversarial sample performs random cropping followed by resizing to its original dimensions. The cosine similarity between the local source image embedding and the global or local target image embedding is then computed using an ensemble of surrogate white-box models to guide perturbation updates. The source-target pairs are randomly sampled. Through this iterative local-global or local-local matching, the central perturbed regions

on the source image become progressively more refined, enhancing both semantic consistency and attack effectiveness, which we observe is surprisingly effective for commercial black-box LVLMs.

Reformulation with Many-to-[Many/One] Mapping. Viewing details of adversarial samples as local features carrying target semantics, we reformulate the problem with many-to-many or manyto-one mapping 1 for semantic detail extraction: let X sou , X tar ∈ R H × W × 3 denote the source and target images in the image space, X sou is the clean image at the initial time. In each step, we seek a local adversarial perturbation δ l (with ∥ δ l ∥ p ≤ ϵ ) so that the perturbed source ˜ x s i = ˆ x s i + δ l i (where ˜ x s i is the optimized local source region at step i after current learned perturbation) matches the target ˆ x t at semantic embedding space in a many-to-many/one fashion. Our final learned global perturbation δ g is an aggregation of all local { δ l i } .

We define T as a set of transformations that generate local regions for source images, forming a finite set of source subsets, and local or global images for target. We apply preprocessing (e.g., resizing and normalization) to each original image, allowing the target image to be either a fixed global or a local region similar to the source image.

where each region ˆ x i ( i ∈{ 1 , 2 , . . . , n } ) is generated independently at a different training iteration i . ˆ x t g is a globally transformed target image if using many-to-one. To formulate many-to-many/one mapping, without loss of generality, we denote each pair ˆ x s i and ˆ x t i be matched in iteration i . Let f ϕ denote the surrogate embedding model, we have:

where CS denotes the cosine similarity. By maximizing M T s , T t , each ˜ x s i effectively captures certain semantic ˆ x t i from the target image.

Balancing Semantics and Consistency Between Feature and Image Spaces. Our local perturbation aggregation applied to the source image helps prevent an over-reliance on the target image's semantic cues in the feature space. This is critical because the loss is computed directly from the feature space, which is inherently less expressive and does not adequately capture the intricacies of the image space. As shown in Fig. 4, we compare the global similarity between source and target images optimized using local and global perturbations. The Global-to-Global method achieves the highest similarity, indicating the best-optimized distance between the source and target. However, it results in the lowest ASR (i.e., worst transferability) on LVLMs, suggesting that optimized distance alone is not the key factor and that local perturbations on source can help prevent overfitting and enhance transferability. By encoding enhanced semantic details through multiple overlapping steps, our method gradually builds a richer representation of the input. Meanwhile, the maintained consistency of these local semantic representations prevents them from converging into a uniform or homogenized expression. The combination of these enhanced semantic cues and diverse local expressions significantly improves the transferability of adversarial samples. Thus, we emphasize two critical properties for ˆ x i ∈ T ( X ) :

̸

Eq. (3) promotes consistency through shared regions between local areas, while Eq. (4) encourages diversity by incorporating potentially new areas distinct from each local partition. These complementary mechanisms strike a balance between consistency and diversity. Notably, when Eq. (3) significantly dominates Eq. (4), such that ∀ i, j, ˆ x i ∩ ˆ x j = ˆ x i = ˆ x j , then T reduces to a consistent selection of a global area. Our framework thus generalizes previous global-global feature matching approaches. In practice, we find that while consistent semantic selection is sometimes necessary for the target image, Eq. (4) is essential for the source image to generate high-quality details with better transferability.

Local-level Matching via Cropping. It turns out that cropping is effective for fitting Eq. (3) and Eq. (4) when the crop scale ranges between L and H ( L = 0 . 5 and H = 1 . 0 in our experiments).

Algorithm 1 M-Attack Training Procedure

T ( X ) can be defined as the subset of all possible crops within this range. Therefore, randomly cropping ˆ x with a crop scale [ a, b ] such that L ≤ a < b ≤ H elegantly samples from such mapping. For two consecutive iterations i and i + 1 , the overlapped area of pair ( ˆ x s i , ˆ x s i +1 ) and ( ˆ x t i , ˆ x t i +1 ) ensures consistent semantics between the generated iterations. In contrast, the non-overlapped area is individually processed by each iteration, contributing to the extraction of diverse details. As the cropped extractions combine, the central area integrates shared semantics. The closer the margin it moves towards, the greater the generation of diverse semantic details emerges (see Fig. 5).

Model Ensemble for Shared, High-quality Semantics. While our matching extracts detailed semantics, commercial black-box models operate on proprietary datasets with undisclosed training objectives. Improving transferability requires better semantic alignment with these target models. We hypothesize that VLMs share certain semantics that transfer more readily to unknown models, and thus employ a model ensemble ϕ = { f ϕ 1 , f ϕ 2 , . . . f ϕ m } to capture these shared elements. This approach formulates as:

Our ensemble serves dual purposes. At a higher level, it extracts shared semantics that transfer more effectively to target black-box models. At a lower level, it can combine models with complementary perception sizes to enhance perturbation quality. Models with smaller receptive fields (e.g., transformers with smaller patch sizes) extract perturbations with finer details, while those with larger receptive fields preserve better overall structure and pattern. This complementary integration significantly improves the final perturbation quality, as demonstrated in Fig. 6.

Training. To maximize M T s , T t while maintaining imperceptibility constraints, various adversarial optimization frameworks such as I-FGSM [18], PGD [27], and C&W [5], are applicable. For simplicity, we present a practical implementation that uses a uniformly weighted ensemble with IFGSM, as illustrated in Algorithm 1. More formal and detailed formulations of the problem, along with derivations and additional algorithms, are provided in the Appendix.

5 Experiments

5.1 Setup

We provide the experimental settings and strong baselines below, with more details in the Appendix.

Victim Black-box Models and Datasets. We evaluate three leading commercial black-box multimodal large language model families: GPT-4.5, GPT-4o, o1, Claude-3.5-sonnet, Claude-3.7-sonnet, and Gemini-2.0-flash/thinking [33]. We use the NIPS 2017 Adversarial Attacks and Defenses Competition [16] dataset. Following [8], we sample 100 images and resize them to 224 × 224 pixels. For enhanced statistical reliability, we then conduct evaluations on 1K images for the comparison with competitive methods in Sec. G.1 of the Appendix. Our source-target image training pairs are randomly sampled.

Surrogate Models. We employ three CLIP variants [14] as surrogate models: ViT-B/16 , ViT-B/32 , and ViT-g-14-laion2B-s12B-b42K , for different architectures, training datasets, and feature extrac-

Target

ViT-Laion/14

ViT-B/16

ViT-B/32

SSA-CWA

Ensemble

AttackVLM

AnyAttack

SSA-CWA

AnyAttack

Ours

AttackVLM

Ours

AttackVLM

AnyAttack

SSA-CWA

tion capabilities. We also include results on BLIP-2 [19] in the Appendix. Single-model method [39], if not specified, uses ViT-B/32 as its surrogate model. The ensemble-based methods [12, 38, 8] use the models specified in their papers.

Baselines. We compare against four recent targeted and transfer-based black-box attackers: AttackVLM [39], SSA-CWA [8], AnyAttack [38], and AdvDiffVLM [12].

Hyper-parameters. If not otherwise specified, we set the perturbation budget as ϵ = 16 such as Tab. 2, 4, 5 under the ℓ ∞ norm and total optimization step to be 300. α is set to 0.75 for Claude-3.5 in Tab. 2, 3 and α = 1 elsewhere, including imperceptibility metrics. The ablation study on α is provided in the Appendix.

5.2 Evaluation Metrics

KMRScore . Previous attack evaluation methods identify keywords matching the 'semantic main object" in images [8, 38, 12]. However, unclear definitions of 'semantic main object" and matching mechanisms introduce significant human bias and hinder reproducibility. We address these limitations by manually labeling multiple semantic keywords for each image (e.g., 'kid, eating, cake" for an image showing a kid eating cake) and establishing three success thresholds: 0.25, 0.5, and 1.0, denoted as KMR a , KMR b and KMR c , respectively. These thresholds correspond to distinct matching levels: at least one keyword matched, over half-matched, and all matched, allowing us to evaluate transferability across different acceptance criteria. To reduce human bias, we leverage GPT-4o [1] for matching semantic keywords against generated descriptions, creating a semi-automated assess-

ϵ

Method

GPT-4o

Gemini-2.0

Claude-3.5

Imperceptibility

KMR

a

KMR

b

KMR

c

ASR KMR

a

KMR

b

KMR

c

ASR KMR

a

KMR

b

KMR

c

ASR

ℓ

1

(

↓

)

ℓ

2

(

↓

)

4

AttackVLM [39]

0.08

0.04

0.00

0.02

0.09

0.02

0.00

0.00

0.06

0.03

0.00

0.00

0.010

0.011

SSA-CWA [8]

0.05

0.03

0.00

0.03

0.04

0.03

0.00

0.04

0.03

0.02

0.00

0.01

0.015

0.015

AnyAttack [38]

0.07

0.02

0.00

0.05

0.10

0.04

0.00

0.05

0.03

0.02

0.00

0.02

0.014

0.015

M-Attack

(Ours)

0.30

0.16

0.03

0.26

0.20

0.11

0.02

0.11

0.05

0.01

0.00

0.01

0.009

0.010

8

AttackVLM [39]

0.08

0.02

0.00

0.01

0.08

0.03

0.00

0.02

0.05

0.02

0.00

0.00

0.020

0.022

SSA-CWA [8]

0.06

0.02

0.00

0.04

0.06

0.02

0.00

0.06

0.04

0.02

0.00

0.01

0.030

0.030

AnyAttack [38]

0.17

0.06

0.00

0.13

0.20

0.08

0.01

0.14

0.07

0.03

0.00

0.06

0.028

0.029

M-Attack

(Ours)

0.74

0.50

0.12

0.82

0.46

0.32

0.08

0.46

0.08

0.03

0.00

0.05

0.017

0.020

16

AttackVLM [39]

0.08

0.02

0.00

0.02

0.06

0.02

0.00

0.00

0.04

0.01

0.00

0.00

0.036

0.041

SSA-CWA [8]

0.11

0.06

0.00

0.09

0.05

0.02

0.00

0.04

0.07

0.03

0.00

0.05

0.059

0.060

AnyAttack [38]

0.44

0.20

0.04

0.42

0.46

0.21

0.05

0.48

0.25

0.13

0.01

0.23

0.048

0.052

M-Attack

(Ours)

0.82

0.54

0.13

0.95

0.75

0.53

0.11

0.78

0.31

0.18

0.03

0.29

0.030

0.036

ment pipeline with human guidance. We verify the approach's robustness by manually reviewing 20% of the outputs and checking the consistency.

ASR (Attack Success Rate). We further employ widely-used LLM-as-a-judge [40] for benchmarking. We first caption both source and target images through the same commercial LVLM, then compute similarity with GPTScore [11], creating a comprehensive, automated evaluation pipeline. An attack succeeds when the similarity score exceeds 0.3. The appendix contains our detailed prompts for both evaluation methods for reproducibility.

5.3 Comparison of Different Attack Methods

Tab. 2 shows our superior performance across multiple metrics and LVLMs. Our M-Attack beats all prior methods by large margins. Our proposed KMRScore captures transferability across different levels. KMR a with a 0.25 matching rate resembles ASR, while KMR c with a 1.0 matching rate acts as a strict metric. Less than 20% of adversarial samples match all semantic keywords, a factor overlooked by previous methods. Our method achieves the highest matching rates at higher thresholds (0.5 and 1.0). This indicates more accurate semantic preservation in critical regions. In contrast, competing methods like AttackVLM and SSA-CWA achieve adequate matching rates at the 0.25 threshold but struggle at higher thresholds. These results show that our local-level matching and ensemble strategies not only fool the victim model into the wrong prediction but also push it to be more confident and detailed in outputting target semantics.

5.4 Ablation

Local-level Matching . We evaluate four matching strategies: Local-Global , Local-Local (our approach), Global-Local (crop target image only), and Global-Global (no cropping). Fig. 10 presents our results: on Claude, Local-Local matching slightly outperforms Local-Global matching, but the gap is not significant. Global-level matching fails most attacks, showing the importance of Eq. (4) on the source image. We also test traditional augmentation methods, including shear, random rotation, and color jitter, against our local-level matching approach in Fig. 10. Transformations that incorporate a local crop as defined in Eq. (4), like rotation and translation, achieve decent results, while color jitter and global-level matching that do not retain the local area of source images yield significantly lower ASR. Our systematic ablation demonstrates that local-level matching is the key

100

200

300

500

Steps

0

10

20

30

40

50

60

70

KMR

b

%

(a) GPT-4o

100

200

300

500

Steps

0

10

20

30

40

50

60

70

KMR

b

%

(b) Gemini-2.0

100

200

300

500

Steps

0

10

20

30

40

KMR

b

%

(c) Claude-3.5

100

200

300

500

Steps

0

20

40

60

80

100

ASR%

(d) GPT-4o

100

200

300

500

Steps

0

20

40

60

80

100

ASR%

(e) Gemini-2.0

100

200

300

500

Steps

0

10

20

30

40

ASR%

(f) Claude-3.5

ours

AnyAttack

SSA-CWA

AttackVLM

LTL

0

20

40

60

80

99

KMR

a

%

LTL

0

10

20

30

40

50

60

70

KMR

b

%

LTL

0

10

20

30

KMR

c

%

LTL

0

20

40

60

80

99

ASR%

GPT-4o

Gemini

factor. Although this alignment can be implemented through different operations, such as cropping or translating the image, it fundamentally surpasses conventional augmentation methods by emphasizing the importance of retaining local information.

Ensemble Design . Model ensemble plays a crucial role in boosting the performance. Ablation studies in Fig. 9 indicate that removing the ensemble results in a 40% reduction in KMR and ASR results. While local-level matching helps capture fine-grained details, the ensemble integrates the complementary strengths of large-receptive field models (which capture overall structure and patterns) with small-receptive field models (which extract finer details). This synergy between local-level matching and the model ensemble is essential, as shown in Fig. 6, with the overall performance gain exceeding the sum of the individual design improvements. Further ablation studies on the ensemble sub-models are provided in the Appendix.

Perturbation Budget ϵ . Tab. 3 reveals how perturbation budget ϵ affects attack performance. Smaller ϵ values enhance imperceptibility but reduce attack transferability. Our method maintains superior KMR and ASR across most ϵ settings, while consistently achieving the lowest ℓ 1 and ℓ 2 norms. Overall, our method outperforms other methods under different perturbation constraints.

Local-Local

Local-Global

Global-Local

Global-Global

Shear

Random Rotation

Color Jitter

0

20

40

60

80

100

ASR%

GPT-4o

Gemini

Computational Budget Steps . Fig. 8 illustrates performance across optimization step limits. Our approach outperforms SSA-CWA and AttackVLM even with iterations reduced to 100. Compared to other methods, our method scales well with computational resources: 200 extra steps improve results by ∼ 10% on both Gemini and Claude. On GPT-4o, ASR increases to near 100%.

Visualization. Fig. 7 demonstrates the superior imperceptibility and semantic preservation of our method. AttackVLM presents almost no semantics in the perturbation, thus failing in most scenarios. Though semantics are important in achieving successful transfer, SSA-CWA and AnyAttack's adversarial samples present some rough shapes lacking fine details, resulting in a rigid perturbation that contrasts sharply with the original image. Moreover, AnyAttack's adversarial samples exhibit template-like disturbance, which is easy to notice. In contrast, our method focuses on optimizing subtle local perturbations, which not only enhances transferability but also improves imperceptibility over global alignment.

Results on Reasoning and Latest LVLMs. We also evaluated the transferability of our adversarial samples on the latest models like GPT-4.5, Claude-3.7-sonnet, and reasoning-centric commercial models like GPT-o1, Claude-3.7-thinking, and Gemini-2.0-flash-thinking-exp. Tab. 4 and 5 summarize our findings. Despite their reasoning-centric designs, these models demonstrate equal or weaker robustness to attacks compared to their non-reasoning counterparts. This may be due to the fact that reasoning occurs solely in the text modality, while the paired non-reasoning and reasoning models share similar vision components.

6 Conclusion

This paper has introduced a simple, powerful approach M-Attack to attack black-box LVLMs. Our method addresses two key limitations in existing attacks: uniform perturbation distribution and vague semantic preservation. Through local-level matching and model ensemble, we formulate the simple attack framework with over 90% success rates against GPT-4.5/4o/o1 by encoding target semantics in local regions and focusing on semantic-rich areas. Ablation shows that local-level matching optimizes semantic details while model ensemble helps with shared semantic and highquality details by merging the strength of models with different receptive fields. The two parts work synergistically, with performance improvements exceeding the sum of individual contributions. Our findings not only establish a new state-of-the-art attack baseline but also highlight the importance of local semantic details in developing more powerful attack or robust models.

Broader Impacts

By revealing the surprising vulnerability of state-of-the-art black-box models to a minimal yet powerful attack, this work highlights urgent attention about the robustness, transparency, and safety of commercial-grade multimodal large language models that are increasingly integrated into critical decision-making processes. The simplicity and transferability of the attack highlight the insufficiency of current defenses, prompting the need for more systematic security evaluations. Moreover, this work can serve as a practical benchmark for future defenses and inspire the development of standardized risk assessments for black-box AI APIs. Ultimately, the work promotes safer AI development by exposing brittle behaviors that must be addressed to ensure trustworthiness, fairness, and societal alignment in real-world deployments.

Acknowledgments

Wewould like to thank Yaxin Luo, Tianjun Yao, Jiacheng Liu and Yongqiang Chen for their valuable feedback and suggestions. We also appreciate the constructive comments provided by the reviewers and the area chair. This research is supported by the MBZUAI-WIS Joint Program for AI Research.

References

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. In International Conference on Advanced Neural Information Processing Systems , pages 23716-23736, 2022.

Anthropic. Introducing claude 3.5 sonnet, 2024. Accessed: 2025-02-22.

D. Caffagni, F. Cocchi, L. Barsellotti, N. Moratelli, S. Sarto, L. Baraldi, M. Cornia, and R. Cucchiara. The revolution of multimodal large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2024 , pages 13590-13618, 2024.

N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In IEEE symposium on security and privacy , pages 39-57, 2017.

H. Chen, Y. Zhang, Y. Dong, X. Yang, H. Su, and J. Zhu. Rethinking model ensemble in transfer-based adversarial attacks. In ICLR , 2024.

J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In IEEE/CVF Computer Vision and Pattern Recognition Conference , pages 18030-18040, 2022.

Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu. How robust is google's bard to adversarial image attacks? arXiv preprint arXiv:2309.11751 , 2023.

Y. Dong, S. Cheng, T. Pang, H. Su, and J. Zhu. Query-efficient black-box adversarial attacks guided by a transfer-based prior. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):9536-9548, 2021.

Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with momentum. In IEEE/CVF Computer Vision and Pattern Recognition Conference , pages 9185-9193, 2018.

J. Fu, S.-K. Ng, Z. Jiang, and P. Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 , 2023.

Q. Guo, S. Pang, X. Jia, Y. Liu, and Q. Guo. Efficient generation of targeted and transferable adversarial examples for vision-language models via diffusion models. IEEE Transactions on Information Forensics and Security , 20:1333-1348, 2024.

X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y. Lu, and L. Wang. Scaling up vision-language pre-training for image captioning. In IEEE/CVF Computer Vision and Pattern Recognition Conference , pages 17980-17989, 2022.

G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021.

A. Ilyas, L. Engstrom, A. Athalye, and J. Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning , 2018.

A. K, B. Hamner, and I. Goodfellow. Nips 2017: Defense against adversarial attack. https: //kaggle.com/competitions/nips-2017-defense-against-adversarial-attack , 2017. Kaggle.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.

A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security , pages 99-112. 2018.

J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning , pages 19730-19742, 2023.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900, 2022.

Z. Li, D. Liu, C. Zhang, H. Wang, T. Xue, and W. Cai. Enhancing advanced visual reasoning ability of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 1915-1929, 2024.

Z. Liang, Y. Xu, Y. Hong, P. Shang, Q. Wang, Q. Fu, and K. Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering , pages 405-409, 2024.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In International Conference on Advanced Neural Information Processing Systems , pages 34892-34916, 2023.

Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations , 2017.

D.-T. Luu, V.-T. Le, and D. M. Vo. Questioning, answering, and captioning for zero-shot detailed image caption. In Asian Conference on Computer Vision , pages 242-259, 2024.

Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna. Crepe: Can vision-language foundation models reason compositionally? In IEEE/CVF Computer Vision and Pattern Recognition Conference , pages 10910-10921, 2023.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition , 2019.

Ö. Özdemir and E. Akagündüz. Enhancing visual question answering through question-driven image captions as prompts. In IEEE/CVF Computer Vision and Pattern Recognition Conference , pages 1562-1571, 2024.

S. Park, A. Panigrahi, Y. Cheng, D. Yu, A. Goyal, and S. Arora. Generalizing from simple to hard visual reasoning: Can we mitigate modality imbalance in vlms? arXiv preprint arXiv:2501.02669 , 2025.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 2019.

A. Salaberria, G. Azkune, O. L. de Lacalle, A. Soroa, and E. Agirre. Image captioning for effective use of language models in knowledge-based visual question answering. Expert Systems with Applications , 212:118669, 2023.

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

M. Tschannen, M. Kumar, A. Steiner, X. Zhai, N. Houlsby, and L. Beyer. Image captioners are scalable vision learners too. In International Conference on Advanced Neural Information Processing Systems , pages 46830-46855, 2023.

T. Wang, F. Li, L. Zhu, J. Li, Z. Zhang, and H. T. Shen. Cross-modal retrieval: a systematic review of methods and future directions. arXiv preprint arXiv:2308.14263 , 2024.

J. Wu, M. Zhong, S. Xing, Z. Lai, Z. Liu, Z. Chen, W. Wang, X. Zhu, L. Lu, T. Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. In International Conference on Advanced Neural Information Processing Systems , pages 69925-69975, 2025.

D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu. Mm-llms: Recent advances in multimodal large language models. In Findings of the Association for Computational Linguistics ACL 2024 , pages 12401-12430, 2024.

J. Zhang, J. Ye, X. Ma, Y. Li, Y. Yang, J. Sang, and D.-Y. Yeung. Anyattack: Towards largescale self-supervised generation of targeted adversarial examples for vision-language models. arXiv preprint arXiv:2410.05346 , 2024.

Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin. On evaluating adversarial robustness of large vision-language models. In International Conference on Advanced Neural Information Processing Systems , pages 54111-54138, 2023.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595-46623, 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes. Our main claims and contributions are detailed in Sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes. See Sec. C in appendix for more details.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes. See Sec. B in appendix for more details.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. We have provided code, data, and instructions for reproducing our results in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes. We have provided all the codes, data, and instructions to reproduce the results in the supplementary materials. We will also open-source all of them.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes. See Sec. 5.1 for more details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We configure the LLM with temperature = 0 to ensure the generalizability and robustness of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes. See Sec. F in appendix for more details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes. We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have provided and discussed it in Sec. 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper releases an optimized dataset intended solely for academic research purposes. The dataset does not involve sensitive or high-risk content, and therefore no specific safeguards or access restrictions were implemented. The risk of misuse is considered minimal in the context of the dataset's scope and intended use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets used in the paper are publicly available and open-sourced. The original sources are properly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper introduces a new optimized dataset, which has been attached in the supplementary materials, and will publicly available for academic use. Documentation accompanying the release includes details on the data source, collection methodology, intended use.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components. We only use LLMs for evaluations and test our approach's performance.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

A Preliminaries in Problem Formulation

We focus on targeted and transfer-based black-box attacks against vision-language models. Let f ξ : R H × W × 3 × Y → Y denote the victim model that maps an input image to text description, where H,W are the image height and width and Y denotes all valid text input sequence. T is the transformation or preprocessing for the raw input image to generate local or global normalized input. Given a target description o tar ∈ Y and an input image X ∈ R H × W × 3 , our goal is to find an adversarial image X sou = X cle + δ g that:

where ∥·∥ p denotes the ℓ p norm measuring the perturbation magnitude. Since enforcing f ξ ( T ( X sou )) = o tar exactly is intractable. Following [39], we instead find a X tar matching o tar . Then we extract semantic features from this image in the embedding space of a surrogate model f ϕ : R 3 × H × W → R d

where CS ( a, b ) = a T b ∥ a ∥ 2 ∥ b ∥ 2 denotes the cosine similarity between embeddings.

However, naively optimizing Eq. (7) only aligns the source and target image in the embedding space without any guarantee of the semantics in the image space. Thus, we propose to embed semantic details through local-level matching. Thus, by introducing Eq. (1), we reformulate Eq. (7) into Eq. (2) in the main text on a local-level alignment.

B Preliminary Theoretical Analysis

Here, we provide a simplified statement capturing the essence of why local matching can yield a strictly lower alignment cost, hence more potent adversarial perturbations than purely global matching.

Proposition B.1 (Local-to-Local Transport Yields Lower Alignment Cost) . Let µ G S and µ G T denote the global distributions of the source image ˆ x s + δ and target image ˆ x t , respectively, obtained by representing each image as a single feature vector. Let µ L S and µ L T denote the corresponding local distributions, where each image is decomposed into a set of patches ˆ x s i ( i ∈ { 1 , . . . , N } ) and ˆ x t j ( { j = 1 , . . . , M } ) . Suppose that the cost function c (e.g., a properly defined cosine distance that satisfies the triangle inequality) reflects local or global similarity. Then, under mild conditions (such as partial overlap of semantic content), there exists a joint transport plan ˜ γ ∈ Π( µ L S , µ L T ) such that:

where the optimal transport (OT) distance is defined by

Here, f is a feature extractor, z S i and z T j denote the support points (which correspond either to the single global preprocessed images or to the local patches), and Π( µ S , µ T ) is the set of joint distributions with marginals µ S and µ T . Intuitively, γ ( f ( z S i ) , f ( z T j ) ) indicates the amount of mass transported from source patch ˆ x s i to target patch ˆ x t j . In many cases the inequality is strict.

Proof Sketch. Global-to-Global Cost. When the source and target images are each summarized by a single feature vector, we have:

where ¯ x s = f ( ˆ x s + δ ) and ¯ x t = f ( ˆ x t ) .

Local-to-Local Cost. In contrast, decomposing the images into patches x s i and x t j allows for a more flexible matching:

Under typical conditions (for example, when patches in ( ˆ x s + δ ) are close in feature space to corresponding patches in ˆ x t ), the optimal plan γ ∗ matches each patch from the source to a similar patch in the target, thereby achieving a total cost that is lower than (or equal to) the global cost c ( ¯ x s , ¯ x t ) . When the source and target images share semantic objects that appear at different locations or exhibit partial overlap allowing a form of partial transport, local matching can reduce the transport cost because the global representation fails to capture these partial correspondences.

This analysis implies that local-to-local alignment is inherently more flexible and can capture subtle correspondences that global alignment misses.

C Limitations

While our method achieves state-of-the-art attack success rates across multiple strong closed-source MLLMs, including GPT-4.5, GPT-4o, Gemini and Claude, this field is evolving rapidly. As newer and potentially more robust models are released, we cannot fully guarantee that our current approach will maintain the same high level of effectiveness. Future work will be needed to adapt and evaluate our attack under shifting model architectures and defense mechanisms.

D Additional Ablation Study

D.1 Sub-models in the Ensemble

Individual model ablations further clarify each component's contribution, presented in Tab. 6. CLIP Laion, with its smallest patch size, drives performance on GPT-4o and Gemini-2.0, while CLIP ViT/32 contributes more significantly to Claude-3.5's performance by providing better overall pattern and structure. This also aligns better results of Local-Global Matching on Claude-3.5's than Local-Local Matching results. These patterns suggest Claude prioritizes consistent semantics, whereas GPT-4o and Gemini respond more strongly to detail-rich adversarial samples.

Table 6: Impact of individual model in the ensemble. The lowest value, except using all sub-models, is labeled in italics and underlined to indicate the importance of the sub-model in the ensemble.

Regarding the consistency of the architecture or training mythologies for the ensemble surrogate model, we have compared combining CLIP-based models and CLIP + BLIP2 [19] model. Results in Tab. 7 demonstrate that there is no one-for-all solution for model selection. Adding a differentarchitecture model, BLIP2, instead of another same-architecture model would increase the performance on GPT-4o and Gemini-2.0 but also decrease the performance on Claude-3.5. This also aligns with the previous analysis of Claude-3.5's preference for a more consistent semantic presentation.

D.2 Crop Size

Tab. 8 presents the impact of crop size parameter [ a, b ] on the transferability of adversarial samples. Initially we test a smaller crop scale [0 . 1 , 0 . 4] , which results in sub-optimal performance. Then we scale up the crop region to [0 . 1 , 0 . 9] , which greatly improves the result, showing that a consistent

semantic is preferred. Finally, we test [0 . 5 , 0 . 9] and [0 . 5 , 1 . 0] , which yields a more balanced and generally better result over 3 models. This finding aligns well with our Equ. (3) and Equ. (4) in the main text.

D.3 Stepsize Parameter

We also study the impact of α , presented in Tab. 9. We find selecting α ∈ [0 . 75 , 2] provides better results. Smaller α values ( α = 0 . 25 , 5 ) slow down the convergence, resulting in sub-optimal results. Notably, selecting α = 0 . 75 provides generally better results on Claude-3.5. Thus we use α = 0 . 75 for all optimization-based methods within the main experiment (Tab. 2) and ablation study of ϵ (Tab. 3) in this paper (SSA-CWA, AttackVLM, and our M-Attack ).

E Additional Attack Implementation

We also provide additional algorithms implemented with MI-FFGSM and PGD with ADAM [17] optimizer to show that our flexible framework can be implemented with different adversarial attack methods. Algorithm 2 and Algorithm 3. Since we only apply ℓ ∞ norm with ϵ . Thus, to project back after each update, we only need to clip the perturbation. We also provide additional results on M-Attack with MI-FGSM and M-Attack with PGD using ADAM [17] as optimizer, presented in Tab. 10. Results show that using MI-FGSM and PGD in implementation also yield comparable or even better results. Thus, core ideas in our framework are independent of optimization methods.

F More Experimental Setting and Prompt

Platform. The experiments are conducted on 4 × RTX 4090 GPUs. The code is implemented with PyTorch [31].

Algorithm 2 M-Attack with MI-FGSM

Require: clean image X clean, target image X tar, perturbation budget ϵ , iterations n , loss function L , surrogate model ensemble ϕ = { ϕ j } j m =1 , step size α , momentum parameter β

1: Initialize: X 0 sou = X clean (i.e., δ 0 = 0 ), v 0 = 0 ;

2: for i = 0 to n -1 do

3: ˆ x s = T s ( X i ) , ˆ x t = T t ( X i ) ;

4: Compute 1 m ∑ m j =1 L ( f ϕ j (ˆ x s i ) , f ϕ j (ˆ x t i ) ) in Eq. (5);

i sou i tar ▷ Perform random crop, next step X i +1 sou ← ˆ x s i +1

5: Update ˆ x s i +1 , v i by:

6: g i = 1 m ∇ ˆ x s i ∑ m j =1 L ( f ϕ j (ˆ x s i ) , f ϕ j (ˆ x t i ) ;

7: v i = v i -1 + βg i

8: δ l i +1 = Clip ( δ l i + α · sign ( v i ) , -ϵ, ϵ ) ;

9: ˆ x s i +1 = ˆ x s i + δ l i +1 ;

10: end for

11: return X adv ;

Algorithm 3 M-Attack with PGD-ADAM

Require: Clean image X clean, target image X tar, perturbation budget ϵ , iterations n , loss function L , surrogate model ensemble ϕ = { ϕ j } j m =1 , step size α , Adam parameters β 1 , β 2 , small constant ε

1: Initialize: X 0 sou = X clean (i.e., δ 0 = 0 ), first moment m 0 = 0 , second moment v 0 = 0 , time step t = 0 ;

2:

3: for i = 0 to n -1 do

4: ˆ x s i = T s ( X i sou ) , ˆ x t i = T t ( X i tar ) ;

5: Compute 1 m ∑ m j =1 L ( f ϕ j (ˆ x s i ) , f ϕ j (ˆ x t i ) ) ;

6: Compute gradient:

7: g i = 1 m ∇ ˆ x s i ∑ m j =1 L ( f ϕ j (ˆ x s i ) , f ϕ j (ˆ x t i ) ) ;

8: m i = β 1 m i -1 +(1 -β 1 ) g i ; 2

9:

v

=

β

v

+(1

-

β

)

g

;

i 2 i -1 2 i 10: ˆ m i = m i / (1 -β i 1 ) , ˆ v i = v i / (1 -β i 2 ) ;

13: end for

14: return X adv ;

Computation cost . Time taken for optimizing/generating an image: 20.04 seconds on a single RTX 4090 GPU. Memory used for generating an image: 549.78 MB.

Prompt . We provide two prompts used for KMRScore and GPTScore , respectively.

KMRScore: the '{description}" and '{keywords}" are placeholders for text inputs.

You will be performing a keyword -matching task. You will be given a short description and a list of keywords. Your goal is to find matches between the keywords and the content in the description. Here is the description text:

<description >

{description}

</description >

Here is the list of keywords:

▷

Initialize adversarial image

X

sou

G Additional Experimental Results

G.1 Results on 1K Images

We scale up the data size from 100 in Tab. 2 to 1K for better statistical stability. Tab. 11 presents our results. Since labeling multiple semantic keywords for 1000 images is labor-intensive, we provide ASR based on different thresholds as a surrogate for KMRScore . Our method outperforms AnyAttack with a threshold value larger than 0.3. Thus, our method preserves more semantic details that mislead the target model into higher confidence and a more accurate description.

G.2 Comparison of Attack Methods on Open-source LVLMs

We also test our method on mainstream open-source LVLMs of LLaVA and Qwen-VL. Tab. 12 presents our results.

G.3 Results on Other Vision-language Tasks

We further evaluate our method on more diverse vision-language tasks, including image captioning and visual question answering. For the image captioning task (we use source dataset of ImageNet and target dataset of COCO2014 val), the results on commercial LVLMs and open-source LVLMs are presented in Tab. 13 and Tab. 14, respectively. For the visual question answering task, the results on commercial LVLMs and open-source LVLMs are presented in Tab. 15 and Tab. 16, respectively.

G.4 Effectiveness of KMR Metric

Tab. 17 reports the KMR scores under three settings: (i) clean source images that are semantically similar to the target (upper bound), (ii) clean images with semantically different content (baseline), and (iii) adversarial images generated by our method (ours), which are also semantically different from the target. The results demonstrate that our proposed KMR metric effectively captures the

degree of semantic alignment: it assigns high scores when the source and target are aligned (upper bound), low scores when misaligned (baseline), and intermediate but meaningful scores to adversarial images that successfully mimic the target's semantics.

G.5 Performance under Different Query Budgets

All our results in the paper are based on a single-query setting to demonstrate the efficiency of the attack. To further explore the impact of query budgets, we extend our evaluation to cases with 3 and 5 queries. As shown in Tab. 18, we observe a consistent improvement across all metrics-ASR, KMR a , KMR b , and KMR c -with increased query counts. These results demonstrate a favorable trade-off between attack effectiveness and query efficiency.

G.6 Empirical Validation of Baseline Observations and Our Method's Effectiveness

To quantitatively support the observation that baseline attacks tend to produce uniform-like perturbations, we compute the KL divergence between the empirical perturbation distribution and a uniform distribution over the 33 possible discrete values (from -16 to +16 ). As summarized in Tab. 19, we compare three settings: clean failed samples (baseline), our method without local cropping, and our full method with local cropping. The results clearly show that the KL divergence increases from

0.012 (baseline) to 0.038 (ours with local crop), indicating that our approach generates more nonuniform, structured perturbations. Notably, this increase in distributional divergence is accompanied by a significant gain in attack success rate across all tested black-box LVLMs (from 0.10 to 0.95 on GPT-4o), confirming the effectiveness of semantically guided perturbation design.

H Additional Visualizations

H.1 Adversarial Samples

We provide additional visualizations comparing adversarial samples generated using our method and baseline approaches under varying perturbation budgets ( ϵ ). As shown in Fig. 11 and Fig. 12, our method produces adversarial examples with superior imperceptibility compared to existing approaches, like SSA-CWA and AnyAttack, with superior capabilities.

H.2 Failed Adversarial Samples

We present several examples of failed attacks from both prior methods of AttackVLM, SSA-CWA, AnyAttack and our proposed approach to help better understand the challenges of black-box attacks. The visual illustrations are shown in Fig.13, it can be observed that previous methods may fail even when the image is relatively clean or contains only a few objects, whereas our method tends to fail in cases where the image has densely packed targets or contains too many elements.

AttackVLM

SSA-CWA

AnyAttack

Ours

GLYPH(cmap:d835): GLYPH(cmap:d835)GLYPH(cmap:dfcf)

Ours

SSA-CWA

AnyAttack

AttackVLM

�: �

�: �

Ours

SSA-CWA

AnyAttack

AttackVLM

H.3 Real-world Scenario Screenshots

Fig. 15 and 16 present authentic screenshots of interactions with LVLMs, including GPT-4o, Claude3.5, and Gemini-2.0, along with their reasoning counterparts. The target image is presented in Fig. 14, with Fig. 14 (b) denoting the target image used for Fig. 15 and Fig. 14 (a) for Fig. 16. Fig. 17 demonstrates results from the latest LVLM models, Claude-3.7-Sonnet and GPT-4.5. These screenshots illustrate how these models respond when exposed to adversarial images in a chat interface. The results reveal significant vulnerabilities in the current commercial LVLMs when processing visual inputs. When confronted with these adversarial images, the models' responses deviate considerably from the expected outputs and instead produce content that aligns with our target semantics. The examples in Fig. 17 show that the output from the target black-box model almost completely matches the intended semantics. These real-world scenario attacks emphasize the urgent need for more robust defenses in multimodal systems.

Describe this image.

The image

shows a uniformed

soldier holding a flag, with

multiple American flags in the

background.

AnyAttack

Describe this image.

The image shows two white

butterflies with black and

orange spots perched on a

spiky flower, surrounded by

green foliage.

Describe this image.

The image shows a man and

a child in an outdoor setting

with straw-roofed structures,

a decorative yellow tulip-

shaped object, and a white

piano on grass.

AttackVLM

Describe this image.

The image shows an antelope

in mid-air, leaping through a

grassy landscape with hills in

the background.

SSA-CWA

Describe this image.

The image shows a toy fire

truck with an American flag

attached to its front, featuring

red and white detailing and

multiple compartments.

SSA-CWA

Describe this image.

The image shows a group

of young American football

players in blue uniforms

running onto a field, with

spectators seated in

bleachers behind them.

AnyAttack

Describe this image.

The image shows a group

of youth football players in

blue uniforms and helmets

running onto a field, with

spectators seated in

bleachers behind them.

M-Attack

Describe this image.

The image shows a person

wearing a brightly colored

garment, holding a lit

firework or sparkler that is

emitting sparks, with foliage

in the background.

M-Attack

AttackVLM

S

S

(a)

(b)

RSNrROEVOO

NETINE

ITINA

√√√

ChatGPT4o

Gemini

三

2.0 Flash

Describe this image, no longer than 25 words.

Describe this image,no longer than 25

words.

Describe this image, no longer

than 25 words.

Agrocery storeproducedisplaywith bright

Green-tinged image of produce stand with melons in foreground and

The image appears to depict a fruit market with large,

vendor partially visible in background.

green apples or similar fruits arranged

green apples in the foreground, surrounded by other

under a store shelf or awning.

produce, stalls, and a market canopy.

+Ask Gemini

个

Reply to Claude...

Gemini can make mistakes, so double-check it

Ask anything

+

Search

Deep research

ChatGPT can make mistakes. Check important info.

(b) Gemini-2.0-Flash

(c) Claude-3.5-Sonnet

√

(d) GPT-o1

(e) Gemini-2.0-Flash-Thinking

(f) Claude-3.7-Thinking

ChatGPTo1

e

QGreen Apple Market Display

Gemini

一

2.0 Flash Thinking Experimental

Describe this image, no longer than 25 words.

Describe thisimage,no longer than 25

words.

Show thinking^

The user is asking me to describe the image in 25 words or less. The image

words.

appears to be of green produce, possibly at a market. I should focus on the

Examining green-tinted image with ap... 2s

most prominent elements and keep the description concise.

Reasoned about image description for 4 seconds

2.0 Flash Thinking Experimental. Lacks access to real-time info and

some Gemini features.

with what appears to be a small wooden

A pile of green apples in the foreground

A market stall displays bright green apples and various produce

beneath a patterned cloth canopy.

Green produce, possibly gourds or melons, are displayed under a fabric awning

market stall orstructure in the

in a market setting.

background.

Ask anything

Reply to Claude...

+

Search

Deep research

***

Gemini can make mistakes, so doublecheck it

Claude 3.7Sonnet 9 Choose style

ChatGPT can make mistakes. Check important info.

ChatGPT4o

Describe this image, no longer than 25 words.

A meal featuring a piece of grilled fish, green lettuce, and broccoli, with a fork resting nearby on a wooden

surface.

个

Ask anything

+

Search

Deep research

ChatGPT can make mistakes. Check important info.

ChatGPTo1

Describe this image, no longer

than 25 words.

Reasoned about image description for a few

seconds

A small piece of cooked fish with a side of green

broccoli, placed on a plate with a fork nearby.

个

Ask anything

+

Search

Deep research

小

ChatGPT can make mistakes. Check important info.

Gemini

2.0 Flash

+

Describe this image, no longer than 25 words.

A blurry, close-up shot of cooked salmon and broccoli on a plate with a fork in the background.

+

Gemini can make mistakes, so double-check it

(b) Gemini-2.0-Flash

Gemini

+

2.0 Flash Thinking Experimental

Describe this image, no longer than 25 words.

Show thinking ^

The user wants a short description of the image, no longer than 25

words.

I need to identify the main objects in the image and describe them concisely.

The image shows cooked fish, broccoliand a fork. It looks like a meal.

I can describe the food and the overall impression.

2.0 Flash Thinking Experimental. Lacks access to real-time info and some Gemini features.

A piece of cooked fish and broccoli are presented on a plate, alongside a fork ready for eating.

Gemini can make mistakes, so double-check it

Blackened Fish and Broccoli Dish

Describe thisimage,no longer than 25

words.

A piece of cooked fish with a crispy, blackenedexteriorsitsongreenlettuce

leaves alongside steamed broccoli.

Reply to Claude...

(c) Claude-3.5-Sonnet

Q Panda's Striking Facial Features

Describe this image,no longer than 25

words.

Thinking about describing an unspecif.. 2s

Apiece of darkmeat orprotein onapeach- colored plate,surrounded bygreen

vegetables,with a forkvisible.

Reply to Claude...

Claude3.7Sonnet Choose style

小

Target image

Target image

Target image

Target image

Target image

Target image

ChatGPT 4.5

Describe this image, no longer

than 25 words.

People wearing winter gear stand near skis outside a

wooden building, surrounded by snow and trees, with

a blurry, distorted effect.

Ask anything

+

Search

P Deep research

ChatGPT can make mistakes. Check important infe

ChatGPT 4.5

Describe this image, no longer

than 25 words.

The image shows several boats docked on water,

with a distorted effect causing the scene to appear

wavy and unclear, along with some people visible.

Ask anything

+

Search

P Deep research

ChatGPT can make mistakes. Check important infe

ChatGPT 4.5

Describe this image, no longer

than 25 words.

The image shows a person lying in bed under

blankets with pillows, their face partially visible, but

the image appears distorted or blurry.

↓

Ask anything

+

Search

P Deep research

EEt

ChatGPT can make mistakes. Check important info.

= united

Describe this image, no longer than 25

words.

The image shows a person snowboarding

or skiing down a snowy slope, appearing to

be in motion on the white snow.

Reply to Claude...

Claude 3.7 Sonnet  Choose style

=Q united v

Describe this image, no longer than 25

words.

The image shows an elephant standing

near trees in what appears to be a natural

habitat or wildlife area with dry ground.

Retry

Claude can make mistakes.

Please double-check responses.

Reply to Claude...

Claude 3.7 Sonnet  Choose style

三

united

Describe this image, no longer than 25

words.

The image shows zebras in a natural

habitat with trees and vegetation. They

appear to be standing in a grassy area

under tree branches.

Reply to Claude...

Claude 3.7 Sonnet  Choose style

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates transferable adversarial attacks under the standard black-box threat model. It introduces a novel method, M-Attack, which leverages input transformations, such as random image cropping, to extract local views of the input. This approach is motivated by the observation that previous methods fail to transfer effectively to recent commercial models. The proposed method demonstrates strong performance against commercial large vision-language models (LVLMs), including GPT-4.5, GPT-4o, Gemini, and Claude-3.7.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper identifies a consistent failure pattern in prior attacks, characterized by vague or ambiguous model outputs, an observation well-supported by experiments across multiple baselines.\n- It offers a novel and insightful explanation for these failures: the uniform-like distribution of perturbations, which limits semantic specificity and transferability.\n- The proposed solution generating semantically meaningful adversarial perturbations aligns with trends in recent and concurrent research, suggesting a promising direction for future work.\n- Extensive experiments on commercial black-box LVLMs (e.g., GPT-4.5, GPT-4o, Gemini, Claude) demonstrate the practical effectiveness of the method in realistic threat scenarios.\n\n**Weaknesses**\n\n- While the observation in Section 3 regarding uniform-like perturbation distributions is insightful, the paper lacks direct evidence that the proposed method effectively addresses this issue.\n- The novelty of the proposed method is somewhat limited. Input transformation techniques, particularly image cropping, are well-established in the literature as effective tools for improving transferability [1]. However, the authors did not discuss their method to prior works in this area.\n- Although evaluations on real-world commercial black-box models are valuable, the paper would benefit from additional experiments under standard benchmark settings using open-source models to ensure fair comparisons and reproducibility.\n- The authors claim that KMRScore reduces human bias, but the manual annotation of multiple semantic keywords per image introduces subjectivity and may not effectively mitigate bias. Similar concerns apply to the use of LLM-as-a-judge for evaluation.\n- The proposed method is essentially a CLIP-based adversarial attack, using CLIP encoders as surrogates. Given that many large VLMs are built on CLIP, the effectiveness of the attack is perhaps unsurprising. The paper should explicitly discuss this aspect and include comparisons with other CLIP-based attack baselines. Several relevant works along the line of CLIP-based adversarial attacks are not mentioned in the related work section. I have included a few representative works [2] in this line of research, though the list is not exhaustive.\n\n\n\n[1]  Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang, J., Ren, Z., & Yuille, A. L. Improving transferability of adversarial examples with input diversity. In CVPR, 2019\\\n[2] Zhang, J., Yi, Q., & Sang, J. Towards adversarial attack on vision-language pre-training models. In ACM MM, 2022\\\n[3] Lu, D., Wang, Z., Wang, T., Guan, W., Gao, H., & Zheng, F.  Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In ICCV, 2023 \\\n[4] He, B., Jia, X., Liang, S., Lou, T., Liu, Y., & Cao, X. (2023). Sa-attack: Improving adversarial transferability of vision-language pre-training models via self-augmentation. arXiv preprint arXiv:2312.04913. \\\n[5] Zhang, P. F., Huang, Z., & Bai, G. Universal adversarial perturbations for vision-language pre-trained models. In SIGIR, 2024 \\\n[6] Fang, H., Kong, J., Yu, W., Chen, B., Li, J., Wu, H., ... & Xu, K. (2024). One perturbation is enough: On generating universal adversarial perturbations against vision-language pre-training models. arXiv preprint arXiv:2406.05491.\n\n---\n\nMiscellaneous (not part of the rating justification):\n\nThere is a recent concurrent preprint (after NeurIPS deadline) that discusses ensemble surrogate models for attacking CLIP, which may be relevant to this work and could offer some insights into the ensemble strategy employed in M-Attack.\n\nHuang, H., Erfani, S., Li, Y., Ma, X., & Bailey, J. (2025). X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP. arXiv preprint arXiv:2505.05528.\n\nQUESTIONS:\n- Does a non-uniform perturbation distribution correlate with stronger transferability? If so, can the authors provide empirical evidence that their method improves this property, perhaps using the ECDF analyses shown in Figures 3 and 4?\n\n- Could the authors report standard evaluation metrics on open-source models to enable easier comparison with prior work? For example: Zero-shot classification with CLIP, Image-text retrieval tasks, Captioning metrics such as BLEU and CIDEr with large VLMs. These evaluations should ideally be conducted on widely used datasets such as ImageNet, MSCOCO, OK-VQA, etc. Given that the proposed method fundamentally targets CLIP encoders, this would be a relevant and fair comparison.\n\n- This is more like a suggestion rather than a question. I believe the proposed method has the potential to serve as a generalizable enhancement to existing adversarial attacks rather than functioning solely as a standalone method. It would be particularly interesting to see how the input transformation technique could be integrated into or improve the performance of existing attacks mentioned in the strengths and weaknesses section. \n\n\n- Could the authors elaborate on how KMRScore reduces human bias, given that the semantic keyword labels are still manually defined? A deeper discussion on potential limitations or mitigations of this subjectivity would be appreciated.\n\n\nAddressing the first three points raised in my review is likely to change my current opinion on the rating. \n\nOverall, I believe this work has strong potential and introduces valuable insights. However, the limitations in its current presentation, particularly the lack of thorough discussion of related work, insufficient evaluation of standard benchmarks, and limited analysis supporting its core observations. In my opinion, this currently outweighs its strengths.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the poor transferability of targeted adversarial attacks from open-source LVLMs to black-box commercial models. The authors attribute this failure to the lack of semantic precision in perturbations, which are often uniformly distributed and semantically diffuse. To improve semantic alignment, they propose a simple baseline that applies localized perturbations by cropping the adversarial image at random scales and aspect ratios, aligning it with the target in embedding space.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper tackles an important and underexplored challenge in adversarial attacks on black-box LVLMs. It proposes a simple yet effective method that enhances semantic alignment by applying local perturbations through randomized cropping and embedding-space alignment. The approach is easy to implement, model-agnostic, and achieves strong transferability across a wide range of commercial models, including GPT-4.5, GPT-4o, Claude, and Gemini, with success rates exceeding 90%. The empirical results are strong, and the connection between perturbation semantics and transferability is well-motivated and clearly articulated.\n\n**Weaknesses**\n\n1. The method is evaluated solely based on success rate, without reporting query counts or trade-offs under limited query budgets.\n2. The approach emphasizes local semantic alignment, but does not compare against existing saliency-guided or attention-based adversarial methods.\n3. Most experiments focus on image-text matching. The method is not evaluated on richer vision-language tasks like image captioning or visual question answering.\n\nQUESTIONS:\n1. How many queries are typically required to achieve the reported success rates, and how does performance change under limited query budgets?\n\n2. How does the method compare to saliency- or attention-guided perturbation approaches?\n\n3. Can the method generalize to other vision-language tasks such as image captioning or VQA?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the low success rate of existing adversarial attacks on commercial closed-source Large Vision-Language Models (LVLMs). The authors begin by analyzing failed attack samples, identifying the root cause as perturbations that exhibit a \"Uniform-like Distribution\" and lead to \"Vague Descriptions,\" thereby lacking effective semantic information. Based on this insight, the paper proposes an attack framework named M-Attack. The method utilizes Local-level Matching and Model Ensemble strategies to focus the generation of perturbations on local image regions, effectively embedding target semantics into the adversarial examples. Experimental results demonstrate that this method achieves a success rate far exceeding current SOTA methods on several top-tier commercial LVLMs, proving its effectiveness and novelty.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. Clear Motivation and Insight: The paper's motivation is clear. It pinpoints the cause of low attack success rates on commercial models—the Uniform-like Perturbation Distribution and Vague Descriptions—providing a clear direction for future research.\n\n2. Elegant and Effective Method: The proposed M-Attack approach is well-conceived. It presents a simple and efficient local attack strategy that focuses on optimizing the local features of adversarial examples, effectively enhancing the consistency of detailed expressions between the perturbation and the target image. The design is both elegant and easy to understand.\n\n3. Convincing Empirical Results: The proposed method achieves attack success rates on multiple commercial models that significantly surpass prior art, which strongly proves its effectiveness.\n\n# Weaknesses:\n\n1. A core advantage of M-Attack is its ability to generate adversarial examples with clear and rich semantic details. The current evaluation relies mainly on keyword matching to verify consistency. Have the authors considered introducing more evaluation dimensions to further validate this \"clear details\" advantage? For instance, assessing the consistency of object positional relationships, colors, or other attributes in the generated description compared to the target image.\n\n2. The attacks in this paper exclusively target commercial closed-source models. However, these models often employ complex and unknown defense mechanisms (e.g., output post-processing, content filtering), which poses a challenge for attributing the attack's success. Have the authors tested their method on mainstream open-source LVLMs (e.g., LLaVA, Qwen-VL)? Experiments on open-source models would allow for a more transparent analysis of the model's behavior.\n\nQUESTIONS:\n1. Contradiction with Existing Semantic Attacks: The paper's core premise is that prior methods generate perturbations with a Uniform-like Distribution. However, some methods, such as GAKer [1] and AnyAttack [2], appear to generate perturbations with strong semantic properties. This seems to contradict the authors' claim. It is suggested that the authors add a discussion to address this potential discrepancy.\n\n2. Insufficient Explanation of the Local-Global Mechanism: The explanation for the effectiveness of the Local-Global mechanism is not sufficiently clear. It is counter-intuitive that this method performs so well, even outperforming Local-Local on the KMRScore in Figure 9, as it essentially still optimizes the distance to the target image's global information. Could the authors provide a more detailed analysis and explanation? Specifically, why is the Local-Global approach capable of producing richer semantic details?\n\n[1] Any Target Can be Offense: Adversarial Example Generation via Generalized Latent Infection\n\n[2] AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an adversarial attack method for large Vision Language Models. The method is built on an insight that failed adversarial perturbations typically originates from a uniform distribution and lack clear semantic details, leading strong VLMs to ignore them as noises. The proposed attack crops and rescales the adversarial image to align with the target image in the embedding space. This strategy achieves surprisingly good transferability to commercial VLMs, achieving over 90% success rate on the most advanced VLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The observation is in-depth and provides insights. The proposed method works very well under the practical real-world settings. Experiments are thoroughly done. The paper is well-written and easy to follow.\n\n--- \nWeakness: \nW1: Evaluation. The evaluation is done only on 100 sampled images from a NIPS 2017 competition (Line 225-226). Without using a standardized set, the selected images could be cherry-picked, favoring the proposed attack method. This poses challenges for future work to fairly compare. Additionally, the attack success rate is measured by semantic keywords matching, where the keywords are partially manually generated, which may contain bias.  \n\nW2: The “groundtruth” labels are given by GPT-4o and human labellers. However, for different VLMs, they may have different ways to express the same semantic even when the image is clean. The paper doesn’t provide the clean KMR accuracy of the tested models as a baseline.\n\nQUESTIONS:\nQ1: Local windows are selected according to requirements indicated in Equation (3) and (4). Therefore, local windows selected at different timestep always will have an overlap with each other. Will the perturbations optimized on the latest local window sabotage the previously optimized perturbation region and render the previous ones ineffective?\n\nQ2:  This work observes an equal or weaker robustness to the attack on reasoning models which is interesting. In the model’s chain-of-thought, does it notice or mention anything unusual in its thinking process? Can you show some examples of the chain-of-thought when the reasoning model behaves unusually with these adversarial samples?\n\nQ3: How much is the computation cost of launching an instance of the proposed attack?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}