Title: Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large vision-language models

adversarial attack
Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across a wide range of multimodal understanding tasks. Nevertheless, these models are susceptible to adversarial examples. In real-world applications, existing LVLM attackers generally rely on the detailed prior knowledge of the model to generate effective perturbations. Moreover, these attacks are task-specific, leading to significant costs for designing perturbation. Motivated by the research gap and practical demands, in this paper, we make the first attempt to build a universal attacker against real-world LVLMs, focusing on two critical aspects: (i) restricting access to only the LVLM inputs and outputs. (ii) devising a universal adversarial patch, which is task-agnostic and can deceive any LVLM-driven task when applied to various inputs. Specifically, we start by initializing the location and the pattern of the adversarial patch through random sampling, guided by the semantic distance between their output and the target label. Subsequently, we maintain a consistent patch location while refining the pattern to enhance semantic resemblance to the target. In particular, our approach incorporates a diverse set of LVLM task inputs as query samples to approximate the patch gradient, capitalizing on the importance of distinct inputs. In this way, the optimized patch is universally adversarial against different tasks and prompts, leveraging solely gradient estimates queried from the model. Extensive experiments are conducted to verify the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs including LLaVA, MiniGPT-4, Flamingo, and BLIP-2, spanning a spectrum of tasks, all achieved without delving into the details of the model structures.

Pandora's Box: Towards Building Universal Attackers against Real-World Large Vision-Language Models

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across a wide range of multimodal understanding tasks. Nevertheless, these models are susceptible to adversarial examples. In real-world applications, existing LVLM attackers generally rely on the detailed prior knowledge of the model to generate effective perturbations. Moreover, these attacks are task-specific, leading to significant costs for designing perturbation. Motivated by the research gap and practical demands, in this paper, we make the first attempt to build a universal attacker against real-world LVLMs, focusing on two critical aspects: ( i ) restricting access to only the LVLM inputs and outputs. ( ii ) devising a universal adversarial patch, which is task-agnostic and can deceive any LVLM-driven task when applied to various inputs. Specifically, we start by initializing the location and the pattern of the adversarial patch through random sampling, guided by the semantic distance between their output and the target label. Subsequently, we maintain a consistent patch location while refining the pattern to enhance semantic resemblance to the target. In particular, our approach incorporates a diverse set of LVLM task inputs as query samples to approximate the patch gradient, capitalizing on the importance of distinct inputs. In this way, the optimized patch is universally adversarial against different tasks and prompts, leveraging solely gradient estimates queried from the model. Extensive experiments are conducted to verify the strong universal adversarial capabilities of our proposed attack with prevalent LVLMs including LLaVA, MiniGPT-4, Flamingo, and BLIP-2, spanning a spectrum of tasks, all achieved without delving into the details of the model structures.

1 Introduction

Recently, Large Vision-Language Models (LVLMs) have achieved significant success and demonstrated promising capabilities in various multimodal downstream tasks, such as text-to-image generation [1-6], visual question-answering [7-10], and etc. Benefiting from the strong comprehension of large language models (LLMs) [11-13], LVLMs [14-16] on top of LLMs show superior performances in solving complex vision-language tasks by utilizing appropriate human-instructed prompts. However, with the exponential expansion of downstream applications in the real world, LVLMs can be easily fooled by adversarial samples, posing crucial safety issues [17-23].

Existing LVLMs attackers [24-33] generally craft and add perturbations/triggers to benign image/text inputs. By adversarially manipulating LVLMs to concentrate on specific perturbations or triggers,

attackers can cause the models to generate erroneous or jailbreak results, thus presenting a risk to security. Specifically, most of these attackers [25, 26, 28, 30-32] are simply deployed in the whitebox setting, where they have the full knowledge of LVLMs models including network structure and parameter weights to back-propagate gradients for optimizing perturbations/triggers. To alleviate this reliance on model details to a certain extent, some gray-box attackers [24, 27] solely require access to the visual encoder of LVLMs and directly generate the perturbed visual representations to mislead the latter reasoning process. There are also a few works [33, 29] claim that they have successfully achieved more challenging black-box attacks, however, they still need the prior knowledge of additional large models like CLIP [34-36] to serve as surrogate models, or even rely on the model output scores/logits to generate perturbation gradients.

Although the above attackers demonstrate significant performance against LVLMs, as shown in Figure 1, we argue that they fail to consider the essential characteristics of attack practicality and universality among various realistic downstream multimodal tasks: (1) Existing white-, gray- and blackbox methods severely rely on the prior model knowledge, making the attacks less practical since most real-world LVLM applications will not disclose their model details with users. Under such circumstances, the attackers can only query LVLMs to obtain corresponding output results, making it challenging to steer the adversarial perturbations in the correct optimization direction during the gradient estimation process. (2) LVLMs demonstrate impressive versatility in addressing diverse vision-language tasks through varying prompts. However, the current attackers targeting LVLMs can

What is the caption of this image?

⊕

Noise

…

…

Single-Task Prompts

Sample-specific Noise

Image

Classification

Multiple-Task Prompts

Image

Caption

VQA

⊕

Noise

Prompts

Prompts

Prompts

Previous Works

Our Proposed Method

Surrogate Model

(Vision Encoder)

White-box

Attacker

Gray-/Black-box

Attacker

Victim LVLM

Victim LVLM

Our Universal

Attacker

Gradient Approximation

Optimize

Accessible to

previous

Attackers

No Accessibility to

our

Attackers

only produce adversarial examples to deceive a particular task within a singular process. Consequently, to compromise different downstream tasks, they must generate distinct adversarial perturbations, which incur significant time and resource expenditure. Therefore, it is efficient and effective to design a universal perturbation for all samples across different tasks. Upon applying this universal perturbation to any input sample, regardless of the task, it has the capability to mislead the LVLM into predicting a target label specified by the attacker.

To this end, in this paper, we make the first attempt to explore task-agnostic adversarial perturbations and build a universal attacker against LVLMs in a challenging yet realistic setting, where the attackers have no prior LVLMs' knowledge. To make the perturbation universally adversarial to multiple LVLM-driven tasks, we design a special patch-wise perturbation pattern by first initializing it on a fixed location of various image inputs and then optimizing it against images of different multimodal tasks. Since we can solely query the LVLM model, we propose a novel importance-aware gradient approximation strategy to adaptively estimate and adjust the weights on gradient directions for optimizing the patch with different additive noises. Hence, gradient directions from sampled noise that can increase the semantic distance between their output and the target label are enlarged, while other directions are reduced. Furthermore, we design a judge function to assess the LVLMs' output for achieving the targeted attack. In this manner, the proposed attack can generate a universal adversarial patch to mislead the understanding process of the multi-task LVLMs by solely querying the model.

Our contributions can be summarized as follows: (1) To the best of our knowledge, we are the first to investigate the vulnerability of real-world LVLMs in a practical but challenging setting, where the attackers can only access the input and output of the LVLM. (2) To further break through the bottleneck of existing task-specific LVLM attack design, we devise a universal adversarial patch (task-agnostic) that can be pasted and then fool any inputs for any LVLM downstream task. (3) A novel importance-aware gradient approximation strategy is also introduced to optimize the adversarial patch by solely querying the LVLM model. (4) Extensive experiments are conducted to verify the effectiveness of our attack approach on various LVLMs and tasks.

2 Related Work

Large vision-language models. The breakthrough of Large Language Models (LLMs) in languageoriented tasks [11, 37-43] and the emergence of GPT-4 [44, 45] motivate researchers to harness the powerful capabilities of LLMs to assist in various tasks across multimodal scenarios, and further lead to the new realm of Large Vision-Language Models (LVLMs) [46]. There have been different strategies and models to bridge the gap between text and other modalities. Some works [9, 8] leverage learnable queries to extract visual information and generate language using LLMs conditioned on the visual features. Models including MiniGPT-4 [16], LLaVA [15] and PandaGPT [47] learn simple projection layers to align the visual features from visual encoders with text embeddings for LLMs. Also, parameter-efficient fine-tuning is adopted by introducing lightweight trainable adapters into models [48-55]. Several benchmarks [56, 57] have verified that LVLMs show satisfying performance on visual perception and comprehension.

Adversarial robustness of LVLMs. Despite achieving impressive performance, LVLMs still face issues of adversarial robustness due to their architecture based on deep neural networks [58-64]. Multiple primary attack attempts have been conducted to study the robustness of LVLMs from different aspects. Inspired by the adversarial vulnerability observed in vision tasks, most methods [25, 26, 28, 30-32, 65-74] evaluates the adversarial robustness of LVLMs under white-box settings, where they have the full knowledge of LVLMs models including network structure and weights. To generate the adversarial examples, they simply add and optimize imperceptible perturbations/triggers to benign image/text inputs via back-propagation. To reduce the reliance on model knowledge, some gray-box attackers [24, 27] solely require access to the visual encoder of LVLMs and directly generate the perturbed visual representations to fool the latter process. There are also a few works [33, 29] conduct transfer-based attacks. These exploratory works demonstrate that LVLMs still face stability and security issues under adversarial perturbations. However, existing attack methods only consider popular open-source models, but do not study real-world LVLMs applications ( i.e. , users can not access to any details of the model and can only query the model to obtain corresponding output). Moreover, they are implemented as task-specific settings, and they have to generate different adversarial perturbations for each downstream task of LVLMs, costing much time and resources. Therefore, both the attack practicality in real-world setting and the attack universality across multiple tasks/prompts make LVLMs more challenging to attack.

3 Method

In this section, we will first introduce the fundamental preliminary, and then describe the baseline approach for our universal attack and illustrate how we construct the universal adversarial patch by solely querying the LVLM model, respectively. The overall pipeline is illustrated in Figure 2.

3.1 Preliminary

We define f θ ( v ; t ) ↦→ y as a pre-trained large vision-language model (LVLM), parameterized by θ . Here, v denotes the image modality input, t represents the textual modality input, and y signifies the textual output of the model. Specifically, for the general LVLM downstream tasks, v is a sample from the image set V , while t is one of the textual prompt instances from a task prompts set T , such as Visual Question Answering (VQA), Image Captioning, and Image Classification, etc.

Threat model. In this paper, we explore the scenario of attacking real-world LVLM models, where we assume that the attacker has no knowledge of the victim model, including its parameters, training procedure, original training data, etc. In particular, distinct from the approach utilized in white/gray-box attacks, we cannot access the model's gradient information to train perturbations through back-propagation. Moreover, unlike in black-box attacks, we are precluded from acquiring confidence scores or logits derived from the model's outputs. The attacker is limited to receiving only the text output returned by the LVLM following a query as feedback. This setting aligns more closely with the real-world practice of utilizing APIs to access LVLMs.

Attacker's goal. The objective of the attacker is to devise a universal adversarial patch, represented as ∆ , that, by partially covering the original image v , generates an adversarial example v ′ . This adversarial example, upon application to any input sample across diverse downstream tasks, is designed to compel the LVLM to output a target label predetermined by the attacker. Thus, such a

Images

!

Prompts

"

⊕

Vision

Encoder

#

!

Outputs

$

LLM

Adversarial

Target

$′

1 - cos(,

"

$ , ,

"

$′ )

Animals.

The brief caption of...

The image shows…

Unknown.

Judge

Model

Output

Embedding

,

"

($)

,

"

Image Classification

Image Caption

VQA

Target

Embedding

No Accessibility to

Our Universal

Attackers

Judge Function

Search for the

Best Combination

Initialize

Random Locations & Patterns

Stage 2: Gradient Approximation

Indicator Function

/

,

"

($′)

!

!

!

"

⋯ !

|$|

#

!

#

"

⋯ #

|$|

Importance-aware

Computation

Universal

Patch

Stage 1: Patch Initialization

Add

Baseline for Universality

Patch Optimization

Importance Weights

Final result

Stage 3

Iterative Update

patch needs to exhibit persistence and robustness when deployed on unseen inputs, and to induce adversarial semantic alterations across different tasks for the same image, rendering the patch crosstask cross-image applicable. In this paper, we focus on the challenging targeted attack, which implies that the LVLM will produce an attacker-chosen textual output y ′ given the adversarial image v ′ and benign prompt t , formulated as f θ ( v ′ , t ) ↦→ y ′ . To be specific, we denote the textual prompts corresponding to different tasks for the input image v as t k ∈ T k , where k distinguishes the tasks. Our attack goal can thus be expressed as:

Here, ⊙ denotes the Hadamard product. m denotes the patch mask being a 0-1 matrix matching the shape of v . Specifically, the number and placement of 1 in m indicate the actual size of the noise and the relative position of the patch ∆ on v .

3.2 Baseline Approach for Universal Adversarial Attack

Why do we need universal adversarial perturbation? Most existing attacks against LVLMs generally optimize each adversarial sample based on a specific given origin input. In other words, their optimized noise added to different adversarial samples relative to the origin sample varies. Furthermore, the adversarial examples they optimize for are also task-specific, meaning they cannot consistently have an adversarial impact on all downstream tasks. Therefore, we attempt to find that if a universal perturbation (task-agnostic) could be found, adding it to different benign images would consistently produce effective adversarial samples and impact all downstream tasks. This approach would significantly reduce the time and resources while enhancing the robustness and generalizability of real-world LVLM attacks. In this work, we define such perturbation as a universal adversarial patch ∆ , which can be overlaid at a fixed position on a clean image v . When combined with prompts t k from different tasks, it consistently prompts the victim model f θ to output targeted text y ′ .

Universal adversarial objective with targeted label measurement. Unlike attacks on general classification tasks, the outputs of LVLMs under different task prompts are not simply binary (true or false), but rather entail semantically rich natural language descriptions. Therefore, to guide the LVLMs outputting attackers' desired target labels, we need to design a strategy to assess whether the

output of the victim LVLM model after being attacked aligns with the attacker's preset conditions, or to measure the distance between y and y ′ . Inspired by the previous work [33], we construct a judge function J based on a simple and lightweight pre-trained text encoder g ϕ , which serves a role akin to the regularization loss function. The encoder g ϕ , parameterized by ϕ , transforms natural language texts y and y ′ into textual embeddings. Consequently, we can compute the cosine distance between y and y ′ in the high-dimensional semantic space mapped by g ϕ , denoted as:

It is important to note that our attacker has no access to the parameters ϕ of g ϕ , but only the embedding information output by the encoder. Therefore, we can define the attacker's adversarial objective for one adversarial example across K downstream LVLM tasks as:

Finally, by combining Equation 1 and 2, we obtain the universal adversarial objective for all tasks:

3.3 Crafting and Optimizing Universal Adversarial Patch

Since we can not obtain the backpropagated gradient of Equation 4 to optimize the adversarial patch ∆ by solely querying the model, we introduce a novel gradient approximation strategy on LVLM models to generate the universal adversarial patch in the following three steps.

Initializing patch location and noise pattern. Firstly, we need to determine a fixed location of the universal adversarial patch on the visual input, which is crucial because noise in different locations significantly impacts the vision encoder's attention [75, 76]. However, in our challenging practical attack setting, we cannot explicitly explore the areas of interest to the LVLM model using gradient-based tools like Grad-Cam [77]. Therefore, we have to spend a portion of our query budget to randomly decide the patch's location and determine the best location of the patch based on the model's feedback. Specifically, denoting the size of the image input v as S v ∈ Z + and the size of the adversarial patch as S p ∈ Z + , we randomly select the x-/y-axis location, i.e. pos x /pos y, of the patch relative to the image v from { 0 , 1 , . . . , S v -S p } , obtaining the value of each element m ij in the mask matrix m as follows:

Then, for each random location, we combine it with a random noise pattern, allowing us to find the optimal combination of location and pattern. Generally, we randomly sample noise from a uniform distribution U ( -ε, ε ) to construct the initial patch ∆ init, ensuring the constraint ∥ ∆ init ∥ ∞ ≤ ε . Next, we calculate its distance from the target text y ′ using Equation 2 and select the combination with the smallest distance as the initial state for the subsequent iterative optimization of the noise pattern.

Importance-aware gradient approximation. Based on the initialized adversarial patch, we then investigate how to estimate its gradient direction for perturbing it into a targeted-chosen label by solely querying the LVLM model. The Monte Carlo estimation [78] offers a general strategy to approximate the gradient's direction on traditional single-task models. It employs a series of random slight noises on the previously obtained adversarial perturbation and scrutinizes whether these noises induce alterations in the prediction, the average of these noise directions serves as the ultimate direction for further mutating the perturbation. However, such a design is not efficient for the complicated LVLMs as there is a larger yet complex search space in LVLM models and not all gradient directions may point towards optimal direction and some of them may have opposite directions. Therefore, most gradient directions are canceled out with each other and the attack result can hardly be improved (verified by our experimental variant 'w/o importance"). To address this issue, we propose to assign and adjust the optimization weights for different noise samples based on the importance-aware degree to which these sampled noises lead to the attacker-chosen output of the LVLM model.

Specifically, we employ a normalized uniform distribution u · exp( u -1) , u ∼ U ( -1 , 1) to add slight noise δ on the patch ∆ for perturbation. We utilize T times iteration to optimize this patch

with these noises iteratively. In particular, at the t -th step, we initially establish an indicator function σ t to assess whether the noise δ t is capable of influencing the LVLM's prediction:

Here, the sgn ( · ) function denotes the sign function, while τ serves as a threshold to gauge the impact of δ t . Denoting avg ( σ t ) -avg ( σ t -1 ) as ∆ σ t , we lift the importance of noise directions [79] that may lead to attacker-chosen output while diminishing the influence of others by:

̸

where w t is the importance-aware weight. The directional improvement ∆ σ t signifies the disparity between the current decision value and the prior, which can be as small as -2(= -1 -1) . The parameter γ serves to diminish the significance when the direction from attacker-chosen output samples threatens to overshadow input from other samples. The selection of value 3 aims to prevent ln(∆ σ t +3) from yielding a value smaller than 0 .

Updating patch by querying the LVLM. At last, based on the above importance weight, the gradient direction for each sample can be estimated by the weighted average of additive noises. To ascertain the estimated gradient vector ∇ δ t , we employ the Monte Carlo method as:

where the first condition handles the case where all the noises lead to attacker-chosen label or not. If certain perturbations alone precipitate the target prediction, we deduct the mean decision value from each individual decision and multiply the results by their corresponding noises under the second condition. This method ensures that the weights assigned to various noises remain reasonably aligned with the average, reflecting the quality of the present perturbation. Subsequently, we update the patch pattern ∆ with the noise δ t along the direction of ∇ δ t , employing weights that are aware of their importance as follows:

By iteratively optimizing the perturbations on the patch pattern over T iterations using the aforementioned weighted gradient estimation, we can obtain the universal adversarial patch on LVLMs.

4 Experiments

4.1 Implementation details

LVLMs & Datasets. In this paper, following existing LVLM attack methods [24-28, 31-33], we conduct experiments on the same open-source LVLM models including LLaVA-1.5 [15], MiniGPT-4 [16], Flamingo [9], and BLIP-2 [8] for fair comparison. To accurately evaluate the attack methodologies, we conduct experiments on three sources: MS-COCO [80], VQAv2 [81], and DALLE-3 [2]. We also follow the existing works to construct these three datasets. Specifically, we employ images from the test sets of the MS-COCO and VQAv2 to construct two multimodal datasets. We also use captions from the MS-COCO validation set as prompts to generate corresponding images with DALLE-3 to form another dataset. For the text input data, we follow the prompts used in previous work [31] to build our text dataset, with detailed data presented in the appendix.

Basic setups. We employ Sentence-BERT [82] as the text encoder (judge model) to measure the LVLM's textual output with the adversarial target. The discussion regarding different text encoders is presented in Section 4.3. We select three widely used image-to-text tasks to evaluate our attack method, i.e. , Image Classification, Image Captioning, and VQA. Discussions on various patch sizes are conducted in Section 4.3, with a size of 64 chosen as the final decision. During the gradient approximation, we allow 70k queries number in total. The general target label in our almost all experiments is set to text 'Unknown'; Various other target labels are also experimented with in Section 4.2. In Equation 6, we use τ = 0 . 55 to determine the direction of gradient predictions for each slight noise δ . In Equation 7, we select γ = 5 . We impose ε = 16 / 255 as the constraint for ∆ init. All experiments are conducted on a single NVIDIA H100 Tensor Core GPU.

4.2 Main Results

Main performances. We conduct a comprehensive evaluation on four LVLM models across three datasets in Table 1. The performance is evaluated by the semantic similarity between the LVLM's output and the target text. Here, we select the target text 'Unknown' to avoid the inclusion of high-frequency responses. All the performance is evaluated on a single universal patch pasted on the whole test set. In addition to reporting the similarity scores on clean images and our attacked ones, we also report our attack variant 'w/o importance", which removes the importance-aware weights during the gradient approximation and equally treats all gradient directions. From this table, we can find that our attack consistently achieves the best performance on all models and datasets.

To demonstrate that the effectiveness of the proposed attack is not constrained to the specific case of the target text 'Unknown', we extend our evaluation to various other target texts. The experiment includes a selection of text with varied length and usage frequency as shown in Table 2. We can observe that our attack performs the best overall and in each individual task under different target text, though the output similarity differs for different targets. The sentence 'I cannot answer" is a reasonable generation result of LVLMs to indicate the uncertainty of the response, which performs better than the less commonly used 'I hate people".

Compare to other LVLM attacks. Since existing LVLM attack methods are deployed in different settings, for fair comparison, we separately compare our method with each of them in the same setting. As in Table 3, according to the reported performance on MF-Attack [33] in its paper, we compare the same output similarity performance on the same LLaVA, BLIP-2, MiniGPT-4 models in VQA task. We can find that our attack is more effective as we directly approximate the gradients in the victim black-box model. As in Table 4, according to the reported performance on CroPA [31] in its paper, we re-judge our output with the same ASR metric and compare performance on the same OpenFlamingo [84] model on MS-COCO dataset. CroPA can only attack white-box cross-prompt of the same task, instead, our attack can attack black-box cross-task inputs with better results.

Analysis on universality. The main difference between our attack and existing LVLM attacks is that we only need to generate a single universal adversarial patch for all inputs while they need to generate different adversarial perturbations for different input samples. To investigate our universality, we implement two variants for comparison: 'w/o. universal baseline" removes our universality design and follows previous works individually to optimize perturbation for each sample; 'w/. universal baseline" denotes our approach. We evaluate the averaged attack performance of their generated

Clean Input

'

Unknown

'

'

I cannot answer

'

'

I am sorry

'

'

I hate people

'

(a) Ablation on Judge Model

(b) Ablation on

ε

single adversarial patch pasted on all images of the whole test set during the attack optimization. As shown in Figure 3, the universality on the single perturbation of 'w/o. universal baseline" is very poor, demonstrating the effectiveness of our universality design. More analyses are in Appendix B.4.

Visualization results. We provide the visualizations of the targeted universal attack in Figure 4. Each adversarial patch can achieve a universal targeted attack. More visualizations are in Appendix B.8.

4.3 Ablation

We conduct ablation studies on the LLaVA model and DALLE-3 to investigate our attack in depth.

Ablation on different judge model. As shown in Figure 5(a), we conduct ablations on different text encoders (judge model) to measure and constrain the semantics of LVLM's output. We find that Sentence-BERT achieves the best performance.

Ablation on patch size S p . As shown in Table 5, we conduct the ablations on different patch sizes. It shows that size 64 × 64 is enough to achieve good attack performance.

Ablation on different ε . Figure 5(b) computes the distances of different ε according to Equation 2, where ε = 16 / 255 is adequate to attain satisfactory results.

4.4 Robustness to Defense Strategy

We further investigate the robustness of our proposed attack method. As shown in Table 6, we first report the attack performance on the LLaVA model and DALLE-3 dataset against the traditional RandomRotation defense strategy. It validates the robustness of our proposed attack. As shown in Table 7, we also evaluate the robustness of our adversarial patch with four popular defense methods. Specifically, PatchCleanser [85] is a state-of-the-art certifiable defense against adversarial patches. It uses double masking to certify the prediction. [86, 87] are query-based defenses, which are specifically designed for detecting malicious queries by black-box attacks. [88] is the general black-box defense method. Overall, it indicates that our attack is robust to the potential defenses.

4.5 Complexity Analysis

For a single attack process, the complexity was recorded in Table 8. The attack's query budget was set to 70,000, with the dataset being DALLE-3 and the victim LVLM as LLaVA, with other hyperparameters consistent with those in subsection 4.1. The primary GPU computational and memory overheads occur during the querying stage against the victim LVLM when training a universal patch. This involves adding slight noise to all attack samples during each iterative update of the patch to explore their impacts, and this stage also constitutes the major consumption of the query budget. The gradient approximation stage primarily takes place on the CPU, involves a much smaller computational load, and thus does not consume GPU power or memory, and is therefore not recorded in the table. Note that, our attack is much more efficient than existing LVLM attacks, since our universal adversarial patch can be pasted on any image of any task to achieve attack while existing LVLM attacks need to generate individual perturbation for each input sample.

5 Conclusion

In this paper, we propose to attack the real-world large vision-language models (LVLMs) in a practical but challenging setting, where the attacker can solely query the LVLM model. To make the perturbation universally adversarial to multiple LVLM-driven tasks, we design a universal adversarial patch with specific locations to perturb the visual inputs. By solely querying the model to estimate the gradient direction for optimizing the adversarial patch pattern, we develop a novel importance-aware gradient approximation strategy to adaptively estimate and adjust the weights on gradient directions for optimizing different samples. Experiments show the effectiveness of the proposed attack method.

References

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Dan Guo, Kun Li, Bin Hu, Yan Zhang, and Meng Wang. Benchmarking micro-action recognition: Dataset, method, and application. IEEE Transactions on Circuits and Systems for Video Technology , 34(7):6238-6252, 2024.

Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Oliver Wu. Unbalanced incomplete multiview clustering via the scheme of view evolution: Weak views are meat; strong views do eat. IEEE Transactions on Emerging Topics in Computational Intelligence , 6(4):913-927, 2021.

Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Oliver Wu. V 3 h: View variation and view heredity for incomplete multiview clustering. IEEE Transactions on Artificial Intelligence , 1(3): 233-247, 2020.

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems , 34:200-212, 2021.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35: 23716-23736, 2022.

Xiang Fang, Daizong Liu, Pan Zhou, and Guoshun Nan. You can ground earlier than see: An effective and efficient pipeline for temporal sentence grounding in compressed videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2448-2460, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Wei Hu, and Yu Cheng. A survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403 , 2024.

Xiang Fang, Yuchong Hu, Pan Zhou, and Dapeng Wu. Animc: A soft approach for autoweighted noisy and incomplete multiview clustering. IEEE Transactions on Artificial Intelligence , 3(2): 192-206, 2021.

Xiang Fang, Daizong Liu, Pan Zhou, Zichuan Xu, and Ruixuan Li. Hierarchical local-global transformer for temporal sentence grounding. IEEE Transactions on Multimedia , 2023.

Xiaoye Qu, Jiashuo Sun, Wei Wei, and Yu Cheng. Look, compare, decide: Alleviating hallucination in large vision-language models via multi-view multi-path reasoning. arXiv preprint arXiv:2408.17150 , 2024.

Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, and Jianfeng Dong. Alleviating hallucination in large vision-language models with active retrieval augmentation. arXiv preprint arXiv:2408.00555 , 2024.

Xiaoye Qu, Mingyang Song, Wei Wei, Jianfeng Dong, and Yu Cheng. Mitigating multilingual hallucination in large vision-language models. arXiv preprint arXiv:2408.00550 , 2024.

Jiashuo Sun, Jihai Zhang, Yucheng Zhou, Zhaochen Su, Xiaoye Qu, and Yu Cheng. Surf: Teaching large vision-language models to selectively utilize retrieved information. arXiv preprint arXiv:2409.14083 , 2024.

Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. In The Twelfth International Conference on Learning Representations , 2023.

Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236 , 2023.

Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's bard to adversarial image attacks? arXiv preprint arXiv:2309.11751 , 2023.

Xunguang Wang, Zhenlan Ji, Pingchuan Ma, Zongjie Li, and Shuai Wang. Instructta: Instructiontuned targeted attack for large vision-language models. arXiv preprint arXiv:2312.01886 , 2023.

Zefeng Wang, Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, Volker Tresp, Philip Torr, and Jindong Gu. Stop reasoning! when multimodal llms with chain-of-thought reasoning meets adversarial images. arXiv preprint arXiv:2402.14899 , 2024.

Hao Zhang, Wenqi Shao, Hong Liu, Yongqiang Ma, Ping Luo, Yu Qiao, and Kaipeng Zhang. Avibench: Towards evaluating the robustness of large vision-language model on adversarial visual-instructions. arXiv preprint arXiv:2403.09346 , 2024.

Dong Lu, Tianyu Pang, Chao Du, Qian Liu, Xianjun Yang, and Min Lin. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577 , 2024.

Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models. arXiv preprint arXiv:2403.09766 , 2024.

Xijia Tao, Shuai Zhong, Lei Li, Qi Liu, and Lingpeng Kong. Imgtrojan: Jailbreaking visionlanguage models with one image. arXiv preprint arXiv:2403.02910 , 2024.

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems , 36, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. Multi-modal cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia , 25:7517-7532, 2022.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240): 1-113, 2023.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Zhenyi Lu, Jie Tian, Wei Wei, Xiaoye Qu, Yu Cheng, Dangyang Chen, et al. Mitigating boundary ambiguity and inherent bias for text classification in the era of large language models. arXiv preprint arXiv:2406.07001 , 2024.

Zhaochen Su, Jun Zhang, Tong Zhu, Xiaoye Qu, Juntao Li, Min Zhang, and Yu Cheng. Timo: Towards better temporal reasoning for language models. arXiv preprint arXiv:2406.14192 , 2024.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 , 2023.

Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355 , 2023.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 , 2023.

Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems , 36, 2024.

Xiang Fang, Daizong Liu, Wanlong Fang, Pan Zhou, Zichuan Xu, Wenzheng Xu, Junyang Chen, and Renfu Li. Fewer steps, better performance: Efficient cross-modal clip trimming for video moment retrieval using language. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 1735-1743, 2024.

Xiang Fang, Daizong Liu, Wanlong Fang, Pan Zhou, Yu Cheng, Keke Tang, and Kai Zou. Annotations are not all you need: A cross-modal knowledge transfer network for unsupervised temporal sentence grounding. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 8721-8733, 2023.

Daizong Liu, Yang Liu, Wencan Huang, and Wei Hu. A survey on text-guided 3d visual grounding: Elements, recent advances, and future directions. arXiv preprint arXiv:2406.05785 , 2024.

Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie. Context-aware biaffine localizing network for temporal sentence grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11235-11244, 2021.

Daizong Liu, Xiaoye Qu, Xiao-Yang Liu, Jianfeng Dong, Pan Zhou, and Zichuan Xu. Jointly cross-and self-modal graph attention network for query-based moment localization. In Proceedings of the 28th ACM International Conference on Multimedia , pages 4070-4078, 2020.

Xiaoye Qu, Pengwei Tang, Zhikang Zou, Yu Cheng, Jianfeng Dong, Pan Zhou, and Zichuan Xu. Fine-grained iterative attention network for temporal language localization in videos. In Proceedings of the 28th ACM International Conference on Multimedia , pages 4280-4288, 2020.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seedbench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023.

Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265 , 2023.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.

Daizong Liu and Wei Hu. Explicitly perceiving and preserving the local geometric structures for 3d point cloud attack. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 3576-3584, 2024.

Daizong Liu, Wei Hu, and Xin Li. Point cloud attacks in graph spectral domain: When 3d geometry meets graph signal processing. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Daizong Liu, Wei Hu, and Xin Li. Robust geometry-dependent attack for 3d point clouds. IEEE Transactions on Multimedia , 2023.

Yunbo Tao, Daizong Liu, Pan Zhou, Yulai Xie, Wei Du, and Wei Hu. 3dhacker: Spectrumbased decision boundary generation for hard-label 3d point cloud attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14340-14350, 2023.

Daizong Liu and Wei Hu. Imperceptible transfer attack and defense on 3d point cloud classification. IEEE transactions on pattern analysis and machine intelligence , 45(4):4727-4746, 2022.

Qianjiang Hu, Daizong Liu, and Wei Hu. Exploring the devil in graph spectral domain for 3d point cloud attacks. In European Conference on Computer Vision , pages 229-248. Springer, 2022.

Yuqi Zhou, Lin Lu, Hanchi Sun, Pan Zhou, and Lichao Sun. Virtual context: Enhancing jailbreak attacks with special token injection. arXiv preprint arXiv:2406.19845 , 2024.

Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, and Xiangliang Zhang. Obscureprompt: Jailbreaking large language models via obscure input. arXiv preprint arXiv:2406.13662 , 2024.

Yuanwei Wu, Yue Huang, Yixin Liu, Xiang Li, Pan Zhou, and Lichao Sun. Can large language models automatically jailbreak gpt-4v? arXiv preprint arXiv:2407.16686 , 2024.

Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to llm-as-a-judge. arXiv preprint arXiv:2403.17710 , 2024.

Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003.04985 , 2020.

Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561 , 2024.

Yue Huang, Qihui Zhang, Lichao Sun, et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507 , 2023.

Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt. arXiv preprint arXiv:2304.12298 , 2023.

Xiang Fang, Zeyu Xiong, Wanlong Fang, Xiaoye Qu, Chen Chen, Jianfeng Dong, Keke Tang, Pan Zhou, Yu Cheng, and Daizong Liu. Rethinking weakly-supervised video temporal grounding from a game perspective. In European Conference on Computer Vision. Springer , 2024.

Xiang Fang, Wanlong Fang, Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Renfu Li, Zichuan Xu, Lixing Chen, Panpan Zheng, et al. Not all inputs are valid: Towards open-set video moment retrieval using language. In ACM Multimedia 2024 , 2024.

Yonggan Fu, Shunyao Zhang, Shang Wu, Cheng Wan, and Yingyan Lin. Patch-fool: Are vision transformers always robust against adversarial perturbations? arXiv preprint arXiv:2203.08392 , 2022.

Zenghui Yuan, Pan Zhou, Kai Zou, and Yu Cheng. You are catching my attention: Are vision transformers bad learners under backdoor attacks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24605-24615, 2023.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Frederick James. Monte carlo theory and practice. Reports on progress in Physics , 43(9):1145, 1980.

Guanhong Tao, Shengwei An, Siyuan Cheng, Guangyu Shen, and Xiangyu Zhang. Hard-label black-box universal adversarial patch attack. In 32nd USENIX Security Symposium (USENIX Security 23) , pages 697-714, 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. arXiv preprint arXiv:1908.10084 , 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An opensource framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023.

Chong Xiang, Saeed Mahloujifar, and Prateek Mittal. { PatchCleanser } : Certifiably robust defense against adversarial patches for any image classifier. In 31st USENIX Security Symposium (USENIX Security 22) , pages 2065-2082, 2022.

Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against querybased black-box attacks. Advances in Neural Information Processing Systems , 34:7650-7663, 2021.

Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y Zhao. Blacklight: Scalable defense for neural networks against { Query-Based }{ Black-Box } attacks. In 31st USENIX Security Symposium (USENIX Security 22) , pages 2117-2134, 2022.

Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial attacks. In Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence , pages 30-39, 2020.

A More Implementation Details

About datasets. We follow previous works to utilize the same dataset setting for fair comparisons. Each dataset consists of both images and prompts. As for MS-COCO [80], its images are collected from the validation dataset of MS-COCO datasets [80]. The prompts for VQA consist of questions both agnostic and specific to the image content. The image-specific questions derive from the VQAv2 [81]. We craft prompts for the questions agnostic to image content, image classification, and image captioning with diverse lengths and semantics. The VQAv2 dataset [81] comprises naturally sourced images paired with manually annotated questions and answers. The DALLE-3 [2] dataset employs a generative method, using random textual descriptions extracted from MS-COCO captions as prompts for image generation powered by GPT-4. Additionally, it includes randomly generated QA pairs based on the images. By default, our experiments are targeted attacks with the target text set to 'Unknown" to avoid the inclusion of high-frequency responses in vision-language tasks.

Evaluation metric. In the real-world pratical setting, we can only obtain the output text of the LVLM model. Directly utilizing strict word-to-word constraints between the output and the adversarial target does not work, since it is hard to not only estimate the potential gradient direction but also generate exactly matched text without any prior knowledge. Therefore, inspired by previous work [33], we design a soft constraint via semantic similarity computed by a textual encoder, and calculate the similarity between the generated response and the targeted text for evaluation.

B Additional Experiments

B.1 Investigation on the Transferability across different Datasets and LVLMs

Since our proposed attack is able to generate universal adversarial perturbations for any input of any task, it is important to investigate the transferability of the generated universal adversarial patch. We report the transfer-attack performance in Table 9, where we analyze the transferability across different datasets and different LVLM models. As for the transferability across different datasets, we generate the universal patch against the LLaV A model on a specific dataset, then paste this patch on the test set of the other two datasets and feed them into the LLaV A model for evaluation. As for the transferability across different LVLM models, we generate the universal patch against a specific model on the DALLE-3 dataset, and then test the patch on the other three LVLM models for evaluation. We can find that our proposed attack can achieve a high-quality performance, demonstrating the effectiveness of our universality design. Moreover, the transfer-attack performance across datasets is worse than across LVLM models, we think the reason is due to the distribution gaps of images between different datasets.

B.2 Ablation on the Task Number during the Model Querying

During the model querying, our attack method randomly samples three prompts individually from the three tasks for gradient optimization. Therefore, we investigate the importance of this task number and how it contributes to the final universality. Note that, for each task, we still utilize multiple images to implement image-level universal attack. As shown in Table 10, all performances are the averaged values and are evaluated on the whole test set across all tasks. We can find that the diversity of tasks is important to the final universality, more tasks can provide more complex distributions for learning to attack, leading to better attack generalization-ability.

B.3 Ablation on the Image Number during the Patch Generation

During the patch generation, our attack method drives the patch against an image pool by randomly sampling three prompts individually from the three tasks for each of the images for estimating gradients. Therefore, we investigate the importance of this image number and how it contributes to the final universality. Note that, for each image, we still utilize three tasks' prompts to implement task-level universal attack. As shown in Table 11, all performances are the averaged values and are evaluated on the whole test set across all tasks. We can find that the diversity of images is important to the final universality. We set the image number as 500 in our all experiments based on the consideration of memory and time cost. Of course, a larger image number than 500 can bring further improvement.

B.4 More Analysis on Universality

The main difference between our attack and existing LVLM attacks is that we only need to generate a single universal adversarial patch for all inputs while they need to generate different adversarial perturbations for different input samples of different tasks. To investigate our universality, we implement two variants for comparison: 'w/o. universal baseline" removes our universality design and follows previous works individually to optimize perturbation for each sample; 'w/. universal baseline" denotes our approach. We evaluate the averaged adversarial performance of their generated single perturbation patch pasted on all images of the whole test set. As shown in Figure 6, the universality of the single perturbation of 'w/o. universal baseline" is very poor, it only performs well on a specific task as its perturbation is optimized through a task-specific and image-specific attack

Specific to ImageClassification

Specific to ImageCaption

Specific to VQA

LLaVA

MiniGPT-4

Flamingo

BLIP-2

sp us

50.70.80.9

L.4

0.5

60806090

0.5

0.70.80.9

ImageCaption

VQ4

ImageCaption

VOA

ImageCaption

VQA

w/o. Universality

w. Universality

w/. Universality

w/o. Universality

w/o. Universality

w/. Universality

Simlaiy i ases

0.5

0.60.7 0.80.9

S00

0.60.70.80.9

0.5

0.6 0.70.80.9

ImageCaption

VOA

ImageCaption

VQA

ImageCaption

VOA

w/. Universality

w/. Universality

w/. Universality

w/o. Universality

w/o. Universality

w/o. Universality

0.5

0.60.70.80.9

0.4

0.50.6 0.70.80.9

0.5

0.60.70.80.9

ImageCaption

VOA

ImageCaption

VQA

ImageCaption

VQA

w/. Universality

w/. Universality

w. Universality

w/o. Universality

w/o. Universality

w/o. Universality

0.70.80.9

1.4

0.5

0.6 0.70.80.9

0.60.7 0.80.9

ImageCaption

VQA

ImageCaption

VQA

ImageCaption

VOA

w/. Universality

w/o. Universality

w/. Universality

w/. Universality

w/o. Universality

process. Therefore, it also has poor performance on its task-specific task as it can not successfully attack other unseen image samples. Since our 'w/. universal baseline" is optimized through a

Clean Input

'

Unknown

'

'

I cannot answer

'

'

I am sorry

'

'

I hate people

'

task-agnostic and image-agnostic attack process, our single adversarial perturbation can be universal adversarial to multiple samples across all tasks, achieving better attack performance.

B.5 Experiments on More Targeted Labels

As shown in Table 12, we provide the attack performance on more targeted labels that are set to harmful instructions. Due to the LVLMs' self-constraints, our attack achieves relatively lower attack performance on the harmful targets than on the general targets mentioned in the main paper.

We also provide the attack performance of our Full Attack on LLaVA model and DALLE-3 dataset when target labels are set to long and special target texts. As shown in Table 13, our attack can also achieve significant attack performance on long target text, demonstrating the scalability of our attack.

B.6 More Performance Comparison with Other LVLM Attackers

Since existing LVLM attackers are implemented in different settings with different models/datasets, we have already provided detailed comparisons in Tables 3 and 4 of the paper. Note that, existing methods are non-universal attacks and require prior LVLM knowledge to generate different perturbations for different images. In contrast, our attack solely accesses to the LVLM input/output and can generate a single noised patch to fool all images/prompts, while achieving better attack performance. To provide a more comprehensive comparison, we also re-implement these attacks into our utilized datasets/models/metrics as shown in Table 14, it shows that our attack is still more adversarial. We think the reason is that our universal patch explicitly learns the general adversarial patterns against LVLMs and effectively estimates gradients solely using positive directions.

B.7 Discussion on Adversarial Patch and Adversarial Perturbation

In this paper, we focus on designing universal adversarial patch for attacking real-world LVLMs. Although some existing attackers attempt to design global perturbations being added on the whole original images, these perturbations are hard to be trained as universal ones and deployed in real-world

scenarios. Instead, our adversarial patch has more potential to be scanned and pasted on realistic objects in daily life for perturbing, achieving a physical attack. We will leave this in future research.

B.8 More Visualizations

We provide more visualization results in Figure 7. The first column is the clean images, and the latter four columns are the perturbed images targeted on a specific attacker's chosen output. We can find that our attack method can achieve good attack performance on these images, and the adversarial patch can achieve the same adversarial target when it is pasted on different images. Different targeted patches have different locations and noise patterns.

In addition to the visualizations of adversarial patches targeted on general/common sentence texts, we also design specific text labels for different tasks, and show their corresponding adversarial patches in Figure 8. It demonstrates that our attack algorithm is very flexible and can generate universal adversarial patches according to different attackers' chosen text labels. That is, any text is possible, and its corresponding universal adversarial patch can be successfully generated by our attack.

C Prompts for Different Tasks

Prompts for VQA. Any cutlery items visible in the image? Any bicycles visible in this image? Any boats visible in the image? Any bottles present in the image? Are curtains noticeable in the image? Are flags present in the image? Are flowers present in the image? Are fruits present in the image? Are glasses discernible in the image? Are hills visible in the image? Are plates discernible in the image? Are shoes visible in this image? Are there any insects in the image? Are there any ladders in the image? Are there any man-made structures in the image? Are there any signs or markings in the image? Are there any street signs in the image? Are there balloons in the image? Are there bridges in the image? Are there musical notes in the image? Are there people sitting in the image? Are there skyscrapers in the image? Are there toys in the image? Are toys present in this image? Are umbrellas discernible in the image? Are windows visible in the image? Can birds be seen in this image? Can stars be seen in this image? Can we find any bags in this image? Can you find a crowd in the image? Can you find a hat in the image? Can you find any musical instruments in this image? Can you identify a clock in this image? Can you identify a computer in this image? Can you see a beach in the image? Can you see a bus in the image? Can you see a mailbox in the image? Can you see a mountain in the image? Can you see a staircase in the image? Can you see a stove or oven in the image? Can you see a sunset in the image? Can you see any cups or mugs in the image? Can you see any jewelry in the image? Can you see shadows in the image? Can you see the sky in the image? Can you spot a candle in this image? Can you spot a farm in this image? Can you spot a pair of shoes in the image? Can you spot a rug or carpet in the image? Can you spot any dogs in the image? Can you spot any snow in the image? Do you notice a bicycle in the image? Does a ball feature in this image?

LLaVA: Image Classification

Describe the main theme

of this image in one word.

Cat.

Identify the primary

theme of this image in

one word.

Cat.

Determine the main

category for this image.

Cat.

Offer a one-word

identifier for this picture.

Cat.

In one sentence,

summarize the activity in

this image.

Cows are eating grass.

In a few words, state

what this image

represents.

Cows are eating grass.

LLaVA: Image Captioning

Convey the main theme

of this picture succinctly.

Cows are eating grass.

Convey the content of

this photograph in a

single phrase.

Cows are eating grass.

Can birds be seen in this

image?

There is no object you

mentioned.

How many dishes or food

items are visible on a

table setting?

There is no object you

mentioned.

LLaVA: Visual Question Answering

Can you spot any dogs in

the image?

There is no object you

mentioned.

Does the image show any

airplanes?

There is no object you

mentioned.

8

P

P

P

P

P

P

P

Does a bridge appear in the image? Does a cat appear in the image? Does a fence appear in the image? Does a fire feature in this image? Does a mirror feature in this image? Does a table feature in this image? Does it appear to be nighttime in the image? Does it look like an outdoor image? Does it seem to be countryside in the image? Does the image appear to be a cartoon or comic strip? Does the image contain any books? Does the image contain any electronic devices? Does the image depict a road? Does the image display a river? Does the image display any towers? Does the image feature any art pieces? Does the image have a lamp? Does the image have any pillows? Does the image have any vehicles? Does the image have furniture? Does the image primarily display natural elements? Does the image seem like it was taken during the day? Does the image seem to be taken indoors? Does the image show any airplanes? Does the image show any benches? Does the image show any

landscapes? Does the image show any movement? Does the image show any sculptures? Does the image show any signs? Does the image show food? Does the image showcase a building? How many animals are present in the image? How many bikes are present in the image? How many birds are visible in the image? How many buildings can be identified in the image? How many cars can be seen in the image? How many doors can you spot in the image? How many flowers can be identified in the image? How many trees feature in the image? Is a chair noticeable in the image? Is a computer visible in the image? Is a forest noticeable in the image? Is a painting visible in the image? Is a path or trail visible in the image? Is a phone discernible in the image? Is a train noticeable in the image? Is sand visible in the image? Is the image displaying any clouds? Is the image set in a city environment? Is there a plant in the image? Is there a source of light visible in the image? Is there a television displayed in the image? Is there grass in the image? Is there text in the image? Is water visible in the image, like a sea, lake, or river? How many people are captured in the image? How many windows can you count in the image? How many animals, other than birds, are present? How many statues or monuments stand prominently in the scene? How many streetlights are visible? How many items of clothing can you identify? How many shoes can be seen in the image? How many clouds appear in the sky? How many pathways or trails are evident? How many bridges can you spot? How many boats are present, if it's a waterscape? How many pieces of fruit can you identify? How many hats are being worn by people? How many different textures can you discern? How many signs or billboards are visible? How many musical instruments can be seen? How many flags are present in the image? How many mountains or hills can you identify? How many books are visible, if any? How many bodies of water, like ponds or pools, are in the scene? How many shadows can you spot? How many handheld devices, like phones, are present? How many pieces of jewelry can be identified? How many reflections, perhaps in mirrors or water, are evident? How many pieces of artwork or sculptures can you see? How many staircases or steps are in the image? How many archways or tunnels can be counted? How many tools or equipment are visible? How many modes of transportation, other than cars and bikes, can you spot? How many lamp posts or light sources are there? How many plants, other than trees and flowers, feature in the scene? How many fences or barriers can be seen? How many chairs or seating arrangements can you identify? How many different patterns or motifs are evident in clothing or objects? How many dishes or food items are visible on a table setting? How many glasses or mugs can you spot? How many pets or domestic animals are in the scene? How many electronic gadgets can be counted? Where is the brightest point in the image? Where are the darkest areas located? Where can one find leading lines directing the viewer's eyes? Where is the visual center of gravity in the image? Where are the primary and secondary subjects positioned? Where do the most vibrant colors appear? Where is the most contrasting part of the image located? Where does the image place emphasis through scale or size? Where do the textures in the image change or transition? Where does the image break traditional compositional rules? Where do you see repetition or patterns emerging? Where does the image exhibit depth or layers? Where are the boundary lines or borders in the image? Where do different elements in the image intersect or overlap? Where does the image hint at motion or movement? Where are the calm or restful areas of the image? Where does the image become abstract or less defined? Where do you see reflections, be it in water, glass, or other surfaces? Where does the image provide contextual clues about its setting? Where are the most detailed parts of the image? Where do you see shadows, and how do they impact the composition? Where can you identify different geometric shapes? Where does the image appear to have been cropped or framed intentionally? Where do you see harmony or unity among the elements? Where are there disruptions or interruptions in patterns? What is the spacing between objects or subjects in the image? What foreground, mid-ground, and background elements can be differentiated? What type of energy or vibe does the image exude? What might be the sound environment based on the image's content? What abstract ideas or concepts does the image seem to touch upon? What is the relationship between the main subjects in the image? What items in the image could be considered rare or unique? What is the gradient or transition of colors like in the image? What might be the smell or aroma based on the image's content? What type of textures can be felt if one could touch the image's content? What boundaries or limits are depicted in the image? What is the socioeconomic context implied by the image? What might be the immediate aftermath of the scene in the image? What seems to be the main source of tension or harmony in the image? What might be the narrative or backstory of the main subject? What elements of the image give it its primary visual weight? Would you describe the image as bright or dark? Would you describe the image as colorful or dull?

Prompts for Image Captioning. Elaborate on the elements present in this image. In one sentence, summarize the activity in this image. Relate the main components of this picture in words. What

narrative unfolds in this image? Break down the main subjects of this photo. Give an account of the main scene in this image. In a few words, state what this image represents. Describe the setting or location captured in this photograph. Provide an overview of the subjects or objects seen in this picture. Identify the primary focus or point of interest in this image. What would be the perfect title for this image? How would you introduce this image in a presentation? Present a quick rundown of the image's main subject. What's the key event or subject captured in this photograph? Relate the actions or events taking place in this image. Convey the content of this photograph in a single phrase. Offer a succinct description of this picture. Give a concise overview of this image. Translate the contents of this picture into a sentence. Describe the characters or subjects seen in this image. Capture the activities happening in this image with words. How would you introduce this image to an audience? State the primary events or subjects in this picture. What are the main elements in this photograph? Provide an interpretation of this image's main event or subject. How would you title this image for an art gallery? What scenario or setting is depicted in this image? Concisely state the main actions occurring in this image. Offer a short summary of this photograph's contents. How would you annotate this image in an album? If you were to describe this image on the radio, how would you do it? In your own words, narrate the main event in this image. What are the notable features of this image? Break down the story this image is trying to tell. Describe the environment or backdrop in this photograph. How would you label this image in a catalog? Convey the main theme of this picture succinctly. Characterize the primary event or action in this image. Provide a concise depiction of this photo's content. Write a brief overview of what's taking place in this image. Illustrate the main theme of this image with words. How would you describe this image in a gallery exhibit? Highlight the central subjects or actions in this image. Offer a brief narrative of the events in this photograph. Translate the activities in this image into a brief sentence. Give a quick rundown of the primary subjects in this image. Provide a quick summary of the scene captured in this photo. How would you explain this image to a child? What are the dominant subjects or objects in this photograph? Summarize the main events or actions in this image. Describe the context or setting of this image briefly. Offer a short description of the subjects present in this image. Detail the main scenario or setting seen in this picture. Describe the main activities or events unfolding in this image. Provide a concise explanation of the content in this image. If this image were in a textbook, how would it be captioned? Provide a summary of the primary focus of this image. State the narrative or story portrayed in this picture. How would you introduce this image in a documentary? Detail the subjects or events captured in this image. Offer a brief account of the scenario depicted in this photograph. State the main elements present in this image concisely. Describe the actions or events happening in this picture. Provide a snapshot description of this image's content. How would you briefly describe this image's main subject or event? Describe the content of this image. What's happening in this image? Provide a brief caption for this image. Tell a story about this image in one sentence. If this image could speak, what would it say? Summarize the scenario depicted in this image. What is the central theme or event shown in the picture? Create a headline for this image. Explain the scene captured in this image. If this were a postcard, what message would it convey? Narrate the visual elements present in this image. Give a short title to this image. How would you describe this image to someone who can't see it? Detail the primary action or subject in the photo. If this image were the cover of a book, what would its title be? Translate the emotion or event of this image into words. Compose a one-liner describing this image's content. Imagine this image in a magazine. What caption would go with it? Capture the essence of this image in a brief description. Narrate the visual story displayed in this photograph.

Prompts for Image Classification. Identify the primary theme of this image in one word. How would you label this image with a single descriptor? Determine the main category for this image. Offer a one-word identifier for this picture. If this image were a file on your computer, what would its name be? Tag this image with its most relevant keyword. Provide the primary classification for this photograph. How would you succinctly categorize this image? Offer the primary descriptor for the content of this image. If this image were a product, what label would you place on its box? Choose a single word that encapsulates the image's content. How would you classify this image in a database? In one word, describe the essence of this image. Provide the most fitting category for this image. What is the principal subject of this image? If this image were in a store, which aisle would it belong to? Provide a singular term that characterizes this picture. How would you caption this image in a photo contest? Select a label that fits the main theme of this image. Offer the most appropriate tag for this image. Which keyword best summarizes this image? How would you title this image in an exhibition? Provide a succinct identifier for the image's content. Choose a word that best groups

this image with others like it. If this image were in a museum, how would it be labeled? Assign a central theme to this image in one word. Tag this photograph with its primary descriptor. What is the overriding theme of this picture? Provide a classification term for this image. How would you sort this image in a collection? Identify the main subject of this image concisely. If this image were a magazine cover, what would its title be? What term would you use to catalog this image? Classify this picture with a singular term. If this image were a chapter in a book, what would its title be? Select the most fitting classification for this image. Define the essence of this image in one word. How would you label this image for easy retrieval? Determine the core theme of this photograph. In a word, encapsulate the main subject of this image. If this image were an art piece, how would it be labeled in a gallery? Provide the most concise descriptor for this picture. How would you name this image in a photo archive? Choose a word that defines the image's main content. What would be the header for this image in a catalog? Classify the primary essence of this picture. What label would best fit this image in a slideshow? Determine the dominant category for this photograph. Offer the core descriptor for this image. If this image were in a textbook, how would it be labeled in the index? Select the keyword that best defines this image's theme. Provide a classification label for this image. If this image were a song title, what would it be? Identify the main genre of this picture. Assign the most apt category to this image. Describe the overarching theme of this image in one word. What descriptor would you use for this image in a portfolio? Summarize the image's content with a single identifier. Imagine you're explaining this image to someone over the phone. Please describe the image in one word? Perform the image classification task on this image. Give the label in one word. Imagine a child is trying to identify the image. What might they excitedly point to and name? If this image were turned into a jigsaw puzzle, what would the box label say to describe the picture inside? Classify the content of this image. If you were to label this image, what label would you give? What category best describes this image? Describe the central subject of this image in a single word. Provide a classification for the object depicted in this image. If this image were in a photo album, what would its label be? Categorize the content of the image. If you were to sort this image into a category, which one would it be? What keyword would you associate with this image? Assign a relevant classification to this image. If this image were in a gallery, under which section would it belong? Describe the main theme of this image in one word. Under which category would this image be cataloged in a library? What classification tag fits this image the best? Provide a one-word description of this image's content. If you were to archive this image, what descriptor would you use?

D Limitations and Broader Impacts

Limitations. One potential limitation of the proposed method is the time cost. Since we need to iteratively query the LVLM model to estimate the gradient for optimizing the adversarial patch, more time and resources are needed. Besides, our work assumes that input images are fed directly into the LVLM models. However, in the future, vision-language models are more likely to be deployed in complex scenarios such as controlling robots or automatic driving, in which case input images may be obtained from the interaction with physical environments and captured in real-time by cameras. Performing adversarial attacks in such complicated cases would be one of the future directions for evaluating the security of vision-language models.

Broader impacts. While the primary goal of our research is to evaluate and improve the attacker's universality and practicality against large vision-language models, it is possible that the developed attacking strategies could be misused to evade practically deployed systems and cause potential negative societal impacts. Specifically, our threat model assumes real-world access and targeted responses, which involves manipulating existing APIs such as GPT-4 (with visual inputs) and/or Midjourney on purpose, thereby increasing the risk if these vision-language APIs are implemented as plugins in other products.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In this paper, we make the first attempt to investigate and improve the attacker's universality and practicality against LVLMs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the potential limitations in the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We have no theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided them with the implementation details. We will also release our codes upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the codes upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All of them are carefully illustrated in implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed them in the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the universal adversarial attacks on large vision language models (LVLMs). It focuses on the black-box setting where only the model's response is available to the adversary. This paper proposed a query-based targeted attack that leverages a proxy model to obtain the similarities between the target model's response and the target text. By approximating the gradient and using the importance score as the weight, the proposed method demonstrated solid results in exposing the safety concerns of several popular LVLMs. The proposed method is very practical since popular LVLM services often only provide the model's response.\n\nSTRENGTHS:\n- The threat model, in which the adversary can only get access to the model's response, is very realistic in the real-world setting since popular LVLMs are often offered as a service where only the model's response is available. Additionally, this paper studies the universal perturbation with targeted texts. Arguably, this type of attack is one of the most challenging ones. \n- The proposed method is technically sound and comprehensive experiments demonstrate the empirical performance. All components of the proposed attack are well explained. Sufficient ablation study has been conducted in this paper. \n- The safety concerns of LVLMs exposed in this paper could potentially have a large impact on the AI safety field.\n\nWEAKNESSES:\n- A large number of queries with similar contents might be easily detected by the defender. Experiments investigating potential defense could make the paper more comprehensive. An example of such detection is in [1].\n- For comparison with CroPA in Table 4, the CroPA paper stated that they use OpenFlamingo rather than Flamingo. It would be good to specifically specify which model is being used for evaluation. If it is the open-source version, it would help to cite OpenFlamingo and clearly state that OpenFlamingo is used rather than Flamingo. If Table 4 uses Flamingo, it would not be appropriate to compare it with the results reported in CroPA in Table 4 since the open-source version has some slight differences from the original model. \n- All the target texts are rather few words. It would be more comprehensive to include longer sentences. Will longer sentences increase the query budgets to achieve a similar performance as these short sentences?  \n- For targeted attacks, only reporting the semantic similarity scores makes it hard to fully understand the attack performance. Word/token-based evaluations, such as BLEU, are more appropriate. \n\n\nIt is not a weakness, but I believe the year of the publication for [45] in the reference is wrong. NeurIPS 2024 papers should be under review at the moment rather than published. \n\n--- \n\n[1] Chen, Steven, Nicholas Carlini, and David Wagner. \"Stateful detection of black-box adversarial attacks.\" In Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence, pp. 30-39. 2020.\n\nQUESTIONS:\nThe reviewer's main concerns are these points and questions outlined in the weaknesses section. The reviewer suggests that the authors focus on these points in the rebuttal.\n\nLIMITATIONS:\nLimitations and potential negative impacts have been sufficiently discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research introduces a novel approach to creating universal adversarial attacks against Large Vision-Language Models (LVLMs). \nIt proposes the universal attacker against real-world LVLMs that operates with limited access to the model (only inputs and outputs).\nThe attack is designed to be task-agnostic, using a universal adversarial patch that can deceive LVLMs across various tasks and inputs.\nThe approach is model-agnostic, not requiring detailed knowledge of the LVLM's structure. Extensive experiments demonstrate the effectiveness of the attack against popular LVLMs like LLaVA, MiniGPT-4, Flamingo, and BLIP-2 across various tasks.\n\nSTRENGTHS:\nThis paper addresses a significant gap in the field of adversarial attacks on LVLMs, offering a more practical and versatile approach compared to existing task-specific methods. The universal nature of the attack and its ability to function with limited model access make it particularly relevant for real-world applications and raise important questions about the security of LVLMs.\n\nWEAKNESSES:\nThis paper presents an interesting approach to universal adversarial attacks on large vision-language models (LVLMs). However, there are several areas where the paper could be improved:\n\n1. Comparison with non-universal attacks: The paper lacks a comparison between the proposed universal attack and non-universal attacks on specific tasks. Including such experiments would help readers better understand the advantages and limitations of this method. The current baselines (\"Full attack\" and \"w/o importance\") are insufficient, and more comparisons would strengthen the paper.\n\n2. Query efficiency: The authors mention allowing 70,000 queries in total, but it's unclear how many images this applies to. This number seems excessive and impractical for real-world scenarios. A good attack should be query-efficient, and this aspect needs further explanation or justification.\n\n3. Opposite directions claim: The statement about \"some of them may have opposite directions\" (line 205) lacks supporting evidence. This claim requires further explanation or empirical support.\n\n4. Judge model acquisition: The paper should clarify how an attacker would obtain the judge model in practical scenarios and whether it needs to be related to the target LVLM.\n\n5. Target text selection: Using \"Unknown\" as the target text may not truly represent a **targeted** attack. It would be valuable to include experiments with more specific target texts (e.g., \"a photo of a cat\") and report the success rates for such cases.\n\n6. Patch visibility and physical attacks: Figure 4 suggests that the adversarial patch is quite noticeable to the human eye, which could be easily filtered in real-world applications. Additionally, the paper would benefit from including physical attack experiments, where the patch is printed and applied to real images to test LVLM performance.\n\n7. Transferability to other models: The paper should explore whether this attack transfers to other image-capable language models like GPT-4 or Claude, as this would demonstrate the broader applicability of the method.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This paper presents a novel approach to create a universal adversarial attacker for LVLMs. The proposed method focuses on two main aspects: restricting access to only the LVLM inputs and outputs, and devising a task-agnostic adversarial patch that can deceive multiple multimodal downstream tasks. The approach involves initializing the adversarial patch through random sampling and optimizing it using a diverse set of LVLM task inputs. Extensive experiments demonstrate the effectiveness of the proposed method across various LVLMs and tasks.

STRENGTHS:
1. The black-box attack design make the proposed method practical for real-world applications.
2. The universal adversarial patch can deceive multiple multimodal downstream tasks enhancing the proposed method's utility.
3.  Extensive experiments across different LVLMs and tasks validate the effectiveness of the proposed method.

WEAKNESSES:
1. The paper could benefit from clearer explanations of the technical details and processes involved in creating and optimizing the adversarial patch.

QUESTIONS:
I am curious about the output of the model for a completely corrupted input, such as an image with noise that cannot be recognized by a human. It would be interesting to include this as a baseline. The paper does not explicitly state this nuance, but it seems this attack aims to mislead the model with an image that is clearly recognizable to humans. Otherwise, one could always send the model an image with pure noise, which would not be a useful attack.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a universal adversarial attack method targeting large vision-language models (LVLMs) by designing a universal adversarial patch. This method restricts access to only the model’s inputs and outputs, creating a task-agnostic adversarial patch that can deceive various LVLM-driven tasks without prior model knowledge. Extensive experiments validate the effectiveness of the proposed attack across multiple models and tasks.\n\nSTRENGTHS:\n1. This paper investigates the vulnerability of real-world LVLMs in a practical but challenging setting, where 83 the attackers can only access the input and output of the LVLM. It is more practical.\n2. This paper devises a universal adversarial patch that can be pasted and then fool any inputs for any LVLM downstream task, it is significant to evaluate the security of LVLMs, especially in the security-critical scenarios.\n\nWEAKNESSES:\n1. In this attack setting, constraints don't seem as important because there is no need to ensure stealthiness, but patch size is quite crucial. In extreme cases, if the patch size is set to be as large as the image, the attack loses its meaning. So, can the attack achieve the same effect with a very small patch size without adding constraints to the perturbation?\n2. Why do different target labels affect the choice of patch position? Can the patch be fixed in a specific position to achieve the same attack effect?\n3. Compared to adversarial example attacks, this type of attack is certainly more powerful. In practical scenarios, this attack seems more meaningful for jailbreaking large models, such as inducing the model to follow harmful instructions or provide harmful information. The responses tested in the paper are relatively short, such as a single word. Does the length of the target response text affect the effectiveness of this attack?\n4. In real-world scenarios, since this patch attack is quite obvious, a user could simply remove the patch's effect by cropping the image. So, what is the practical significance of this type of attack?\n\nQUESTIONS:\nRefer to the Weakness section.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}