On Evaluating Adversarial Robustness of Large Vision-Language Models
Authors: Yunqing Zhao
Tianyu Pang
Chao Du
Xiao Yang
Chongxuan Li
Ngai-man Cheung
Min Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Vision-Language Models
Adversarial Robustness
TL;DR: We evaluate the adversarial robustness of large vision-language models, under the most realistic and challenging threat model of black-box access and targeted goal.

Abstract: Large vision-language models (VLMs) such as GPT-4 have achieved unprecedented performance in response generation, especially with visual inputs, enabling more creative and adaptable interaction than large language models such as ChatGPT. Nonetheless, multimodal generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable modality (e.g., vision). To this end, we propose evaluating the robustness of open-source large VLMs in the most realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning the targeted responses. In particular, we first craft targeted adversarial examples against pretrained models such as CLIP and BLIP, and then transfer these adversarial examples to other VLMs such as MiniGPT-4, LLaVA, UniDiffuser, BLIP-2, and Img2Prompt. In addition, we observe that black-box queries on these VLMs can further improve the effectiveness of targeted evasion, resulting in a surprisingly high success rate for generating targeted responses. Our findings provide a quantitative understanding regarding the adversarial vulnerability of large VLMs and call for a more thorough examination of their potential security flaws before deployment in practice. Our project page: https://yunqing-me.github.io/AttackVLM/.

On Evaluating Adversarial Robustness of Large Vision-Language Models
Yunqing Zhao"!, Tianyu Pang" 12, Chao Du Xiao Yang?, Chongxuan Li', Ngai-Man Cheung"! Min Lin
'Singapore University of Technology and Design
"Sea AI Lab, Singapore
Tsinghua University *Renmin University of China
(zhaoyq, tianyupang, duchao, linmin)@sea. com; yangrlao190tssinhuua en; chongxuanli cn; grgaiman_hhung@sute edu
Abstract
Large vision-language models (VLMs) such as GPT-4 have achieved unprecedented performance in response generation, especially with visual inputs, enabling more creative and adaptable interaction than large language models such as ChatGPT. Nonetheless, multimodal generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulner- able modality (e.g.. vision). To this end, we propose evaluating the robustness open-source large VLMS
most realistic and high-risk setting, where ad- versaries have only black-box system access and seek to deceive the model into returning the targeted responses. In particular, we first craft targeted adversarial examples against pretrained models such CLIP and BLIP, and then transfer these adversarial examples to other VLMs such as MiniGPT-4, LLaVA, UniDiffuser, BLIP-2, and Img2Prompt. In addition, observe that black-box queries these VLMS can further improve the effectiveness targeted evasion, resulting in a sur- prisingly high success rate for generating targeted responses. Our findings provide a quantitative understanding regarding the adversarial vulnerability of large VLMs and call for thorough examination of their potential security flaws before deployment in practice. Our project page: yunqing-ma lengiingmeegithub...
www.gggtthhhooo. io/ Attt//ttta..M....
Introduction
Large dision-language models (VLMs) have enjoyed tremendous success and demonstrated promising
capabilities in text-to-image generation [55, 68, 72], image-grounded text generation (c.g., image captioning or visual question-answering) [2, 15, 42, 86), and joint generation [5, 32, 98] due to an increase in the amount of data, computational resources, and number of model parameters. Notably, after being finetuned with instructions and aligned with human feedback, GPT-4 [58] is capable of conversing with human
and, in particular, supports visual inputs.
Along the trend multimodal learning, an increasing number large VLMs are made publicly available, enabling the exponential expansion of downstream applications. However, this poses significant safety challenges. It is widely acknowledged, for instance, that text-to-image models could be exploited to generate fake content [71, 76] edit images maliciously [73]. A silver lining is that adversaries must manipulate textual inputs to achieve their evasion goals, necessitating extensive search and engineering determine the adversarial prompts. Moreover, text-to-image models that
"Equal contribution. Work done during Yunqing Zhao's internship *Correspondence to Tianyu Pang. Chao Du, and Ngai-Man Cheung.
Lab.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

BLIP-2: image captoning the shapp of at avacado.
rage
Targett Perscho 211
BLI geraraled respores
stuffes
shape of avacade
drawing of ara sports
Tundinng (DALL-E)
bay our riitict
oor amage
Additional resuits
SURSeT BOUntaTos
rrving rOom
genenaad Happoia
ChET
Plouiing
ong and cat with their tongues nut
BLP RUP-Egaanead
Mhite rigine
rotorbike
AII gww.o resporon
over the countainn and hills" advsondd stack
very nice View
the Beach www.naaaoaoooooo
Figure 1: Image captioning task implemented by BLIP-2. Given original text description (e.g.. an armchair in the shape of an avocado), DALL-E [67] is used to generate corresponding clean images. BLIP-2 accurately returns captioning text (e.g.. a stuffed chair in the shape of an avocado) that analogous to the original text description on the clean image. After the clean image is maliciously perturbed by targeted adversarial noises, the adversarial image can mislead BLIP-2 to return a caption (e.g., a pencil drawing of sports car is shown) that semanti- cally resembles the predefined targeted response (e.g., a hand drawn sketch of a Porsche 911). More examples such as attacking real-world image-text pairs are provided in our Appendix.
accessible to the public typically include a safety checker to filter sensitive concepts and an invisible watermarking module to help identify fake content [69, 72, 108].
Image-grounderr text generation such as GPT-4 is more interactive with human users and can produce commands to execute codes [28] or control robots [88], as opposed to text-to-image generation which only returns an image. Accordingly, potential adversaries may be able to evade an grounggodddddd text generative model by manipulating its visual imputs, as it is well-known that the vision modality is extremely vulnerable to auman-imperceptible adversarial perturbations [8, 22, 29, 81]. This raises even more serious safety concerns, as complex and safety-critical environments [62].' Adversaries may mislead large VI.MS deployed as plugins, for example, to bypass their safety/privacy checkers, inject malicious code, or access APIs and manipulate robots/devices without authorization.
text generation may be utilized in considerably
In this work, we empirically evaluate the adversarial robustness of oate-off-hhe-rr large VLMs, particularly against those that accept visual inputs (e.g., image-grounded text generation or joint generation). To ensure reproducibility, our evaluations are all based on open-source large models. We examine the most realistic and high-risk scenario, which adversaries have only black-box system access and seek to deceive the model into returning the largeted responses. Specifically, we first use pretrained CLIP [65, 80] and BLIP [41] as surrogate models to craft targeted adversarial examples, either by matching textual embeddings or image embeddings, and then we transfer the adversarial examples to other large VLMs, including MiniGPT-4 [109], LLaVA [46], UniDiffuser [5]. BLIP- 2 [42], and Img2Prompt [30]. Surprisingly, these transfer-based attacks can already induce targeted responses with a high success rate. In addition, we discover that query-based attacks employing basse priors can further improve the efficacy of targeted evasion against these VLMs, as shown in Figure 1 (BLIP-2), Figure 2 (UniDiffuser), and Figure 3 (MiniGPT-4).
Our findings provide quantitative understanding regarding the adversarial vulnerability of large VLMs and advocate for a more comprehensive examination of their potential security defects prior to deployment, discussed in Sec. 5. Regarding more general multimodal systems, our findings indicate that the robustness of systems is highly dependent on their most vulnerable input modality.
2 Related work
Language models (LMs) and their robustness. The seminal works of BERT [21], GPT-2 [64], and T5 [66] laid the foundations of large LMs, upon which numerous other large LMs have been developed
Note that GPT-4 delays the release of its visual inputs due safety concerns [3].

---

UniDiffusar Joint generation
geowrrd on She then tragoo
oegreiltnt drocppton
"A peinting of Packers quarterbatt blue background,"
"A Van styll patnbing of Awerican player
ImppTest
palating of Football player."
Gutiing
Torget shoto autronaut Fiding horue ONE 2OOT.
TeedFino
"A astro sult
patntinn antronact horse on the FOOD.
LIC Boon.
By FERGO
Figure 2: Joint generation task implemented by UniDiffuser. There are generative VLMs such as UniDiffuser that model the joint distribution of image-text pairs and are capable of both image- to-text and text-to-ine -mooi generation. Consequently, given original text description (e.g., A Van Gogh style painting of an American football player), the text-to-image direction of UniDiffuser is used to generate the corresponding clean image, and its image-to-text direction can recover a text response (c.g., A painting of Packers quarterback football player on a blue background) similar the original text description. The recovering between image and text modalities can be performed consistently on clean images. When a targeted adversarial perturbation is added to clean image, however, the image-to-text direction of UniDiffuser will return text (e.g., A man in an astronaut suit riding a horse on the moon) that semantically resembles the predefined targeted description (e.g., A photo of an astronaut riding horse on the moon), thereby affecting the subsequent chains of recovering processes.
and demonstrated significant advancements across various language benchmarks [10, 19, 31, 74, 79, 107]. More recently, CharGPT [57, 59] and several open-source models [18, 83, 95] tuned based on LLaMA [85] enable conversational interaction with human users and can respond to diverse and complex questions. Nevertheless, Alzantot et al. [4] first construct adversarial examples on sentiment analysis and textual entailment tasks, while Jin al. [36] report that BERT can be evaded through natural language attacks. Later, various flexible (e.g., beyond word replacement) and semantically preserving methods are proposed to produce natural language adversarial examples [9, 49, 50, 52, 53, 70, 78, 102, 1 (04, 10], as well as benchmarks and datasets to more thoroughly evaluate the adversarial robustness of LMs [56, 90-92]. There are also red-teaming initiatives that use human-in in-tuun--itth--eee in-the-loop or automated frameworks to identify problematic language model outputs [27, 63, 96].
Vision-language models (VLMs) and their robustness. The knowledge contained within these powerful LMs is used to facilitate vision-language tasks [26, 33, 84, 93, 101]. Inspired by the adversarial vulnerability observed in vision tasks, early efforts are devoted to investigating adversarial attacks against visual question answering [6, 11,37, 38, 43, 77, 89, 97, 105] and image caption [1, 14, 99], with the majority of these efforts focusing on conventional CNN-RNN-based models, assuming white-box access or untargeted adversarial goals, and requiring human interaction. Our research, on the other hand, examines the adversarial robustness of advanced large VLMs, assuming black-box access and targeted adversarial goals, and providing quantitative evaluations free of human labor.
3 Methodology
In this section, will first introduce the fundamental preliminary, and then describe the transfer- based and query-based attacking strategies against image-groue -rggudee text generation, respectively.
3.1 Preliminary
We denote Pa Pa(æ; Cia) + Cou as image-grounded text generative model parameterized by 0, where x is the input image, Cin is the input text, and Cou is the output text. In image captioning tasks, for instance, Cn is a placeholder 0 and Cou is the caption; in visual question answering tasks, en is the question and Con is the answer. Note that here we slightly abuse the notations since the mapping between po( Gm. and Cau could be probabilistic non-deterministic [5, 98].
Threat models. We overview threat models that specify adversarial conditions [12] and adapt them to generative paradigms: (i) adversary knowledge describes what knowledge the adversary is assumed
3

---

MiniGPT-4: Visual Queslion-Answering
Example-1
TOTRET "A robot playing IN 10C Field.
Caa tell BE what
Die interesting
is the interes:ing
*The interesting pol this teage to the girl suntlovers through elephant in the cackground. The trage is digitzl artwork and aponars
X
"he ronot wearing
overalls, hoising snovel,
working nuturissse a Sleck. actal rabut standing rlowers. conpecttion is cyc-catchi IS robos and the stalds striking sisual effect.
field
bright. colorfal
peacefil scere th the elephant backgrourd.
foreground
Clean mmge input (Fram Midoumne))
Resulting image
nonmal 11fe7
normsl 116e7
"Ro, Thts to not a conmon scene in noreal Life. Maleing throust field sur Lovers Teephant
"Mat ecaetly. This scote is work of futurist dapl ction that way not exist the
soeething seople vould encounter in their dally Lives. it mord fartesy leug inary acone, created arttettc
robett ear colturall pro process. Hewaver, it to iapor Last note sech acenario not cur rentlyll way berass
Tlurann Minati 1-d (arctained)
Conversaton/VCK with dean image
Convenssiion with adv Image
Example-2
acvertrii strck
iTarget: painting Leo wiCE Study
WhsT the unusual Image?
"ahat TS The UNUSOS) patet In this Image"
X "Ex is the Dwo Faccoons
TTT unusust potnt in this guree that the bird has headeress typically ceen in the spoc Ee..
book and pon in frost of then. They
have big they are both looking at the bank in front shee."
Lan you brietly dESCUES what fee17"
"Can you brietly discuss what fe617"
WAKE
image imput (From Mijoummy)
Resuling adv. image (ours)
tmage chous hird eilitary attire. which could centiors that 1n may rocke car vary depending on the inc lecual and their persenal experi inces and whill and view ST reninder of wa- and violenne
tmage you proctidd Tikely to eunke fnelings
raceoons st sitting cablle with a book and pen in front than may ronlnd people
ovn pats of reading Tearning, eleeents eany paop hoaes.
ConversationVOA with clean image
Conversalion/VA with adv. image
Figure 3: Visual censstion-anwering (VQA) task implemented by MiniGPT-4. MiniGPT-4 has capabilities for vision-language understanding and performs comparably to GPT-4 on tasks such as multi-round VQA by leveraging the knowledge of large LMs. We select images with refined details generated by Midjourney [51] and feed questions (c.g., Can you tell me what is the interesting point of this image?) into MiniGPT-4. As expected, MiniGPT-4 can return descriptions that are intuitively reasonable, and when we ask additional questions (e.g., But is this common scene in the normal life?). MiniGPT4 demonstrates the capacity for accurate multi-round conversation. Nevertheless, after being fed targeted adversarial images, MiniGPT-4 will return answers related to the targeted description (e.g., robot is playing in the field). This adversarial effect can even affect multi-round conversations when we ask additional questions. More examples of attacking MiniGPT-4 or LLaVA on VQA are provided in our Appendix.
to have, typically either white-box access with full knowledge of po including model architecture and weights, or varying degrees of black-box access, c.g., only able to obtain the output text Cout from an API; (ii) adversary goals describe the malicious purpose that the adversary seeks achieve, including untargeted goals that simply cause Cout to be a wrong caption or answer, and targeted goals that cause Cou to match predefined targeted response Cur (measured via text- eoxt-machhig metrics); (iii) adversary capabilities describe the constraints on what the adversary can manipulate to cause harm, with the most commonly used constraint being imposed by the f, budget, namely, the lp distance between the clean image Eale and the adversarial image Ea is less than a budget e as || I < e.
Remark. Our work investigates the most realistic and challenging threat model, where the adversary has black-box access to the victim models po, targeted goal, a small perturbation budget on the input image to ensure human imperceptibility, and is forbidden to manipulate the input text Cin-

---

Transfer-based attacking strategy (MF-ii)) Pretrained generator A/ (eig DALL-E)
Query-based attacking strategy (MF-11)
Pretratne visual enooder / (e.g. VIT-B/32 of CLPP wntworingg
The vizdm model Po (e.g. MinisP 1-4)
Ctar
Pullfad + odult
"A sea acter ocart corring. ing."
Text2mg
Polltade and
PullFady + obyl
Targeted imagee hytterr
Meezhing
addition!
REF- (Eq,
zerturh
Aus N)O,))
:no updato
omibecding
Leaeratie
of Auaan
menniinn Sare
RDF Extmored
Upiitted WIAgR
Targeted response generation
Target: sea atter pearl earring."
InpeTTed
colorful paintirg pitcher green eyes."
painting wearing colerful haodfe.
Generated respanse xcle
x Targeted response of ady
imag Kile
Image Fedy (Oun))
Figure 4: Pipelines of our attacking strategies. In the upper-left panel. we illustrate our transfer- based strategy for matching image-image features (MF-ii) as formulated in Eq. (2). We select targeted text Cur (e.g., sea otter with pearl earring) and then use a pretrained text-to- image generator lgg to produce a targeted image hg (Carr) The targeted image is then fed to the image encoder fo to obtain the embedding (h (Gar)). Here we refer to adversarial examples generated by transfer-basee based strategies as Trans Eele + A, while adversarial noise is denoted by A. We feed Trans into the image encoder to obtain the adversarial embedding (aranss)), and then we optimize the adversarial noise to maximize the similarity metric (auun))) T (he (Ctar) In the upper-right panel, we demonstrate our query-baser based strategy for matching text-text features (MF-tt), as defined by Eg. (3). We apply the resulted consfer-basee adversarial example Vrran to initialize Vadv, then sample N random perturbations and add them to Tadv to build {Tadv + 8, J.y These randomly perturbed adversarial examples are fed into the victim model po (with the input text Cm unchanged) and the RGF method described in Eq. (4) is used estimate the gradients V.n W (po (Fadv; Cm)) gy (Cur) ) In the bottom, we present the final results our method's 's (MF-ii + MF-tt) targeted response generation.
3.2 Transfer-based attacking strategy
Since we assume black-box access to the victim models, a common attacking strategy is transfer- based [22, 23, 47, 61, 94, 100], which relies on surrogate models (e.g., a publicly accessible CLIP model) to which the adversary has white-box access and crafts adversarial examples against them, then feeds the adversarial examples into the victim models (c.g., GPT-4 that the adversary seeks to fool). Due to the fact that the victim models are vision-and- and--anndaggee we select an image encoder fz()) and a text encoder ge (c) surrogate models, and we denote Cur as the targeted response that the adversary expects the victim models to return. Two approaches of designing transfer- cansfer-based adversarial objectives are described in the following.
Matching image-text fcatures (MF-it). Since the adversary expects the victim models to return the targeted response Cua when the adversarial image Bady is the input, is natural to match the features of Cua and Cadv on surrogate models, where Eadv should satisfy²
arg max [| 2as -aallpp Sr
f (2 Tol (Ctrr)
Here, we use blue color to highlight boxte-box accessibility (i.e., can directly obtain gradients of fo and gø through backpropagation). the image and text encoders are chosen to have the same output dimension, and their inner product indicates the modality similarity of Cur and Tady- The constrained optimization problem in Eq. (1) can be solved by projected gradient descent (PGD) [48].
Matching image-image features (MF-ii). While aligned image and text encoders have been shown to perform well on lasion-l tasks [65], recent research suggests that VLMs may behave like oags-of-words [103] and therefore may not be dependable for optimizing cross-modality similarity. Given this, an alternative approach is to use a public text-to-image generative model lng (e.g.. Stable
We slightly abuse the notations by using Bad represent both the variable and the optimal solution.
5

---

Table 1: White-box attacks against surrogate models. We craft adversarial images aa using MF-it in Eq. (1) or MF-ii in Eq. (2), and report the CLIP score (f) between the images and the predefined targeted text Cur (randomly chosen sentences). Here the clean images consist of real-world dete that is irrelevant to the chosen targeted text and lig (Carr generated by a text-to-image model (e.g., Stable Diffusion [72]) conditioned on the targeted text Cur- We observe that MF-ii induces a similar CLIP score compared to the generated image hg (Ctr), while MF-it induces a even higher CLIP score by directly matching tross-modality features. Furthermore, we note that the attack is time-efficient, and we provide the average time (in seconds) for each strategy to craft single Ladv- The results in this table validate the effectiveness of white-box attacks against surrogate models, whereas Table investigates the transferability of crafted aady to evade large VLMs (e.g., MiniGPT-4).
Clean image
Adversarial image
Time to obtain single Tadv
Model
Wele
(C(Car)
MF-ii
MF-it
MF-ii
MF-it
CLIP (RN50) [65]
0.094
0.261
0.239
0.576
0.543
0.532
CLIP (ViT-B/32) [65]
0.142
0.313
0.302
0.570
0.592
0.588
BLIP (ViT) [41]
0.138
0.286
0.277
0.679
0.641
0.634
BLIP-2 (ViT) [42]
0.037
0.302
0.294
0.502
0.855
0.852
ALBEF (ViT) [40]
0.063
0.098
0.091
0.451
0.750
0.749
Diffusion [72]) and generate targeted image corresponding to Cur as he (Ctrr). Then, we match the image-image features of aadv and he (Car) as
arg max || (aas-aap aadi p
( (h
(2)
where orange color is used to emphasize that only black-box accessibility is required for hg. as gradient information of hę is not required when optimizing the adversarial image Ladv- Consequently, we can also implement hg using advanced APIS such as Midjourney [51].
3.3 Query-based attacking strategy
Transfer-based attacks are effective, but their efficacy is heavily dependent on the similarity between the victim and surrogate models. When we are allowed to repeatedly query vietim models, such as by providing image inputs and obtaining text outputs, the adversary can employ a query-based attacking strategy estimate gradients or execute natural evolution algorithms [7, 16, 34].
Matching text-text features (MF-tt). Recall that the adversary goal is to cause the victim models return a targeted response, namely, matching (a Cin) with Car- Thus, it is straightforward to maximize the textual similarity between pa (Zaavv Cin) and Caar as
arg max
I+ (po (Radvi Can) Cmn) I (C
|maee maa ma 11
Note that we cannot directly compute gradients for optimization in Eq. (3) because we assume black-box access the victim models pa and cannot perform backpropagation. To estimate the gradients, we employ the random gradient-free (RGF) method [54]. First, we rewrite gradient as the expectation of direction derivatives, i.e., F(z) any diffcrentiable function and ~ P()) is random variable satisfying that E T = (c.g.,
F(x) ò). where F(x) represents
can be uniformly sampled from hypersphere). Then by zero-order optimization [16], we know that
Tean Ioi (Zodv Gin) Goo (Ctur)
N
(4)
(x + oōn; cin)) go (Ctrr) (pa (Radvi Cmn) g (Ctar) 6...
where 8, ~ P(6), o is a hyperparameter controls the sampling variance, and N is the number of queries. The approximation in Eq. (4) becomes unbiased equation when - and N + co.
Remark. Previous research demonstrates that transfer-based and query-based attacking strategies can work in tandem to improve black-box evasion effectiveness [17, 24]. In light of this, we also consider

---

Table 2: Black-box attacks against victim models. We sample clean images Zete from the ImageNet- IK validation set and randomly select a target text Cur from MS-COCO captions for each clean image. We report the CLIP score (1) between the generated responses of input images (i.e., clean images dllee or Fadv crafted by our attacking methods MF-it, MF-ii, and the combination of MF-ii MF-tt) and predefined targeted texts Curs as computed by various CL.IP text encoders and their ensemble/average. The default textual input Cin is fixed to be "what is the content of this image?". Pretrained image/text encoders such as CLIP are used as surrogate models for MF-it and MF-ii For reference, we also report other information such the number of parameters and input resolution of victim models.
Text encoder (pretrained) for evaluation
Other info.
VLM model
Attacking method
RN50
RN101
VIT-B/16
ViT-B/32
VIT-L/14 Ensemble
# Param. Res.
Clean image
0.472
0.456
0.479
0.499
0,344
0,450
MF-it
0.492
0.474
0.520
0.546
0.384
0.483
BLIP [41]
224M
384
MF-i
0.766
0.753
0.774
0.786
0.696
0.755
MF-ii MF-tt
0.855
0.841
0.861
0.868
0.803
0.846
Clean image
0.417
0.415
0.429
0.446
0.305
0,402
MF-it
0.655
0.639
0.678
0.698
0.611
0.656
UniDiffuser [5]
1.4B
224
MF-ii
0.709
0.695
0.721
0.733
0.637
0.700
MF-ii MF-11
0.754
0.736
0.761
0.777
0.689
0.743
Clean image
0.487
0.464
0.493
0,515
0.350
0.461
MF-it
0.499
0.472
0.501
0.525
0.355
0.470
2rompt [30]
1.7B
384
MF-ii
0.502
0.479
0.505
0.529
0.366
0.476
MF-ii MF-tt
0.803
0.783
0.809
0.828
0.733
0.791
Clean image
0.473
0.454
0.483
0.503
0.349
0.452
MF-it
0.492
0.474
0.520
0.546
0.384
0.483
BLIP--2 [42]
3.7B
224
MF-ii
0.562
0.541
0.571
0.592
0.449
0.543
MF-ii MF-tt
0.656
0.633
0.665
0.681
0.555
0.638
Clean image
0.383
0.436
0.402
0.437
0.281
0.388
MF-it
0.389
0.441
0.417
0.452
0.288
0.397
LLaVA [46]
13.3B
224
MF-ii
0.396
0.440
0.421
0.450
0.292
0,400
MF-ii MF-11
0.548
0.559
0.563
0.590
0.448
0.542
Clean image
0.422
0.431
0.436
0.470
0.326
0.417
MF-it
0.472
0.450
0.461
0.484
0.349
0,443
MiniGPT-4 [109]
14.1B
224
MF-ii
0.525
0.541
0.542
0.572
0.430
0,522
MF-ii MF-tt
0.633
0.611
0.631
0.668
0.528
0.614
the adversarial examples generated by transfer-based methods to be initialization (or prior-guided) and use the information obtained from query- methods to strengthen the adversarial effects. This combination is effective, empirically verified in Sec. 4 and intuitively illustrated in Figure 4.
4
Experiment
In this section, we demonstrate the effectiveness our techniques for crafting adversarial examples against open-source, large VLMs. More results are provided in the Appendix.
4.1 Implementation details
In this paper, we evaluate open-source (to ensure reproducibility) and advanced large VLMs, such as UniDiffuser [5], which uses a diffusion- based framework to jointly model the distribution of
image-text pairs and can perform both image-to-text and text-to-image generation; BLIP [41] is
unified vision-language pretraining framework for learning from noisy image-text pairs; BLIP-2 [42] adds a querying transformer [87] and a large LM (T5 [66]) to improve the maage-grounder text generation;
[30] proposes plug-and and-play, LM-agnostic module that provides large
7

---

Targeted image hglCtar)
X addv
Clean image
Adversarial perturbation (A)
"A cat Submarine chimera, digital
"A painting of sitting in submarine."
"A Pixel Art of the
"An abstrart pattern black green.
Figure Adversarial perturbations are obtained by computing a'dv Eill (pixel values are amplified x 10 for visualization) and their corresponding captions are generated below. Here DALL-E acts as hgg to generate targeted images hg (Car) for reference. We note that adversarial perturbations are not only visually hard to perceive, but also not detectable using state-of-the-art image captioning models (we use UniDiffuser for captioning, while similar conclusions hold when using other models).
Targeted image l4tear)
LPIPS 0.013
LPIPS 0.019
LPIPS 0.054
16, LPISS 0.116
LPIPS 0.158
AA cuto tropl cal Tisn in an aquar us on background
"A painting of robut playying thess.
"A sararo whack I.Lustration.
"A CarEoon plue Fish in bright
Illustration plue Tisn in
"Ai Teage of blue Fish 10 an aquartue
Figure 6: We experiment with different values of in Eq. (3) to obtain different levels of aadv. As seen, the quality of æadv degrades (measured by the LPIPS distance between Tele and Tadv), while the effect of targeted response generation saturates (in this case, we evaluate UniDiffuser). Thus, proper perturbation budget (e.g., 8) is necessary to balance image quality and generation performance.
LM prompts to enable zero-shot VQA tasks; MiniGPT-4 [109] and LLaVA [46] have recently scaled up the capacity of large LMs and leveraged Vicuna- [18] for googe-grouned text generation tasks. We that MiniGPT-4 also exploits high-quality, well-aligned dataset to further finetune the model with conversation template, resulting in performance comparable to GPT-4 [58].
Datasets. We use the validation images from ImageNet-1K [20] clean images, from which adversarial examples are crafted, to quantitatively evaluate the adversarial robustness of large VLMs. From MS-COCU captions [44], we randomly select text description (usually a complete sentence, as shown in our Appendix) as the adversarially targeted text for each clean image. Because we cannot easily find corresponding image of given, predefined text, we Stable Diffusion [72] for the eext-to-image generation to obtain the targeted images of each text description, in order to simulate the real-world scenario. Midjourney [51] and DALL-E [67, 68] are also used in our experiments to generate the targeted images for demonstration.
Basic setups. For fair comparison, we strictly adhere previous works [S, 30, 41, 42, 46, 109] in the selection of pretrained weights for mange-grounded text generation, including large LMs (e.g.. T5 [66] and Vicuna-13B [18] checkpoints). We experiment the original clean images of various resolutions (see Table 2). We set e and use foo constraint by default as | &adviloo <8, which is the most commonly used setting the adversarial literature [12], to ensure that the adversarial perturbations are visually imperceptible where the pixel values are in the range [0, 255). We use 100-step PGD to optimize transfer- based attacks (the objectives in Eq. (1) and Eq. (2)). In each step of query-based attacks, we set query times N 100 in Eq. (4) and update the adversarial images by 8-steps PGD using the estimated gradient. Every experiment is run on single NVIDIA-AI00 GPU.
4.2 Empirical studies
We evaluate large VLMs and freeze their parameters to make them act like image-to-text generative APIs. In particular, in Figure I, we show that our crafted adversarial image consistently deceives BLIP-2 and that the generated response has the same semantics as the targeted text. In Figure

---

CLIP RN50
CLI VIT-6/16
CLIP VIT-0/32
CUP VIT-M
0.86
0.86
1509
0.47
0.47
18.0
18
Figure 7: Performance of our attack method under a fixed perturbation budget e 8. We interpolate between the sole use of transfer-based attack and the sole use of query-based attack strategy. We demonstrate the effectiveness of our method via CLIP score (f) between the generated texts on adversarial images and the target texts, with different types of CLIP text encoders. The r-axis in a "tG format denotes we assign & to transfer- cansfer-based attack and eg to query- based attack. "t+q=8" indicates we use transfer-based attack (e, = 8) as initialization, and conduct query- beey-bbaeed attack for further steps (eg = 8), such that the resulting perturbation satisfies e = 8. As a result, We show that a proper combination of transfer/query based attack strategy achieves the best performance.
query-based
evaluate UniDiffuser, which is capable of bidirectional joint generation, to generate texl-to-image and then image-to-text using the crafted aa It should be noted that such chain of generation will result in completely different content than the original text description. We simply use "what is the content of this image?" as the prompt to answer generation for models that require text instructions as input (query) [30]. However, for MiniGPT-4, we use a more flexible approach in conversation, as shown in Figure 3. In contrast to the clean images on which MiniGPT-4 has concrete and correct understanding and descriptions, our crafted adversarial counterparts mislead MiniGPT-4 into producing targeted responses and creating more unexpected descriptions that are not shown in the targeted text.
In Table 1, we examines the effectiveness of MF-it and MF-ii in crafting white-box adversarial images against surrogate models such as CLIP [64], BLIP [41] and ALBEF [40]. We take 50K clean images Tele from the ImageNet-1K IK validation set and randomly select a targeted text Car from MS-COCO captions for each clean image. We also generate targeted images hg (Curr) as reference and craft adversarial images addv by MF-ii or MF-it. As observed, both MF-ii and MF-it are able to increase the similarity between the adversarial image and the targeted text (as measured by CLIP score) in the white-box box setting, laying the foundation for black-box transferability. Specifically, as seen in Table 2, we first transfer the adversarial examples crafted by MF-ii or MF-it in order to evade large VLMS and mislead them into generating targeted responses. We calculate the similarity between the generated response po (Fadvi : Cm) and the targeted text Cur using various types of CLIP text encoders. As mentioned previously, the default textual input Gn is fixed to be "what is the content of this image?". Surprisingly, we find that MF-it performs worse than MF-ii, which suggests overfitting when optimizing directly on the cross-modality similarity. In addition, when we use the transfer- based adversarial image crafted by MF-ii as an initialization and then apply query-based MF-1t to tune the adversarial image, the generated response becomes significantly more similar to the targeted text, indicating the vulnerability of advanced large VLMs.
4.3 Further analyses
Does VLM adversarial perturbations induce semantic meanings? Previous research has demonstrated that adversarial perturbations crafted against robust models will exhibit semantic or perceptually-aligne characteristics [35, 60, 82]. This motivates us to figure out whether adversarial perturbations A Frdv Vate crafted against large VLMs possess a similar level of semantic information. In Figure 5, we visualize A that results in a successful targeted evasion over a real image and report the generated text responses. Nevertheless, we observe no semantic information associated with the targeted text in adversarial perturbations or their captions, indicating that large VLMs are inherently vulnerable.
The influence of perturbation budget c. We use c == 8 as the default value in our experiments, meaning that the pixel-wise perturbation is up to 88 in the range (0, 255]. In Figure 6, we examine the effect of setting e to different values of {2, 4, 8, 16, 64) and compute the perceptual distance between the clean image afle and its adversarial counterpart aady using LPIPS (4) [106]. We highlight (in red color) the generated responses that most closely resemble the targeted text. As observed, there is off between image quality/fidelity and successfully eliciting the targeted response; therefore, is essential to choose an appropriate perturbation budget value.

---

Kole Adv
GraCCAM of Kate
GroCCAM aay
GndCAM afe
GrddCAM Aelerr
pooto of J T6EES
503 W StEring DesuETTI ELnO White color Srow
WEAT Tedyy
The Tedly
DESUTTEL BiYe
RENIETTIL BEYO
Fires SILRE Tree
The SEON
The aaidlie www.wrwww....................... www. www.
Sasire CETOr 530
cotor SOOW
Figure 8: Visually interpreting our attacking mechanism. To better comprehend the mechanism by which our adversarial examples deceive large VLMs (here we evaluate Img2Prompt), we employ interpretable visualization with GradCAM [75]. (a) An example of Zele. Ladv. and he (Cur), along with the responses they generate. We select the targeted text as a beautiful bird with black and white color in snow. (b) GradCAM visualization when the input question is: what is the teddy bear playing in the middle of the road? As seen, GradCAM can effectively highlight the skateboard for Eele, whereas GradCAM highlights irrelevant backgrounds for andv- (c) If we feed the targeted text as the question, GradCAM will highlight similar regions of aady and he (Cur).
Performance of attack with fixed perturbation budget. To understand the separate benefit from transfer-basee attack and query-barc based attack, we conduct a study to assign different perturbation budget for transfer (cz) and query based attack strategy (cg), under the constraint e + cg 8. Unidiffuser is the victim model in our experiment. The results are in Figure We demonstrate that, proper combination of transfer and query based attack achieves the best performance.
Interpreting the mechanism of attacking large VLMs. understand how our targeted adversarial example influences response generation, we compute the relevancy score of image patches related to the input question using GradCAM [75] to obtain a visual explanation for both clean and adversarial images. As shown in Figure 8, our adversarial image Tady successfully suppresses the relevancy to the original text description (panel (b)) and mimics the attention map of the targeted image fi (Ctar) (panel (c)). Nonetheless, emphasize that the use of GradCAM as feature attribution method has some known limitations [13]. Additional interpretable examples are provided in the Appendix.
Discussion
It is widely accepted that developing large multimodal models will be an irresistible trend. Prior to deploying these large models in practice, however, it is essential to understand their worst-case performance through techniques such as red teaming or adversarial attacks [25]. In contrast to manipulating textual inputs, which may require human-in-the-loop prompt engineering, our results demonstrate that manipulating visual inputs can be automated, thereby effectively fooling the entire large vision- lisoun-aaggugge systems. The resulting adversarial effect is deeply rooted and can even affect multi-round interaction, shown in Figure 3. While multimodal security issues have been cautiously treated by models such as GPT-4, which delays the release of visual inputs [3], there are an increasing number of open-source multimodal models, such MiniGPT-4 [109] and LLaVA [46, 45], whose worst-case behaviors have not been thoroughly examined. The use of these open-source, but adversarially unchecked, large multimodal models product plugins could pose potential risks.
Broader impacts. While the primary goal of our research to evaluate and quantify adversarial robustness of large vasion-language models, it is possible that the developed attacking strategies could be misused to evade practically deployed systems and cause potential negative socictal impacts. Specifically, our threat model assumes black-box access and targeted responses, which involves manipulating existing APIS such as GPT-4 (with visual inputs) and/or Midjourney on purpose, thereby increasing the risk if these vision-language APIS are implemented plugins in other products.
Limitations. Our work focuses primarily on the digital world, with the assumption that input images feed directly into the models. In the future, however, vision-language models are more likely to be deployed in complex scenarios such as controlling robots or automatic driving, which case input images may be obtained from the interaction with physical environments and captured in real-time by cameras. Consequently, performing adversarial attacks in the physical world would be one of the future directions for evaluating the security of vision- languaee models.
10

---

Acknowledgements
This research work is supported by the Agency for Science, Technology and Research (A*STAR)
under its MTC Programmatic Funds (Grant No. M231.7b0021). This material is based on the research/work support in part by the Changi General Hospital and Singapore University of Technology and Design, under the HeahthTech Innovation Fund (HTIF Award No. CGH-SUTD-2021 10004. C. Li was sponsored by Beijing Nova Program (No. 20220484044). We thank Siqi Fu for providing beautiful pictures generated by Midjourney, and anonymous reviewers for their insightful comments.
References
[1] Nayyer Aafaq, Naveed Akhtar, Wei Liu, Mubarak Shah, and Ajmal Mian. Controlled caption generation for images through adversarial attacks. arXiv preprint arXiv:2107.03050, 2021.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[3] Sam Altman, 2023. https://txitte: wtwm/saaa/steeee sttssssattsssse (t0aaatss//1166667885221177
[4] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Wei Chang. Generating natural language adversarial examples. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP)) 2018.
[5] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in modal diffusion at scale. In International Conference Machine Learning (ICML), 2022.
[6] Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp. and Douwe Kiela, Improving question answering model robustness with synthetic adversarial data generation. arXiv preprint arXiv:2/04.08677 2021.
[7] Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In European Conference Computer Vision (ECCV), 2018.
[8] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 387-402. Springer, 2013.
[9] Hezekiah Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the suscepti- bility of pre-trained language models via handcrafted adversarial examples. arXiv preprint 22X9 2e0.2222222222222 128. 2022.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[11] Yu Cao, Dianqi Li, Meng Fang, Tianyi Zhou, Jun Gao, Yibing Zhan, and Dacheng Tao. Tasa: Deceiving question answering models by twin answer sentences attack. arXiv preprint arXiv:2210. 15221, 2022.
[12] Nicholas Carlini, Anish Athalye, Nicolas Paperno,, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06770 2019.
[13] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839-847. IEEE, 2018.
11

---

[14] Hongge Chen, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, and Cho-Jui Hsich. Attacking visual language grounding with adversarial examples: case study on neural image captioning. arXiv preprint arXiv: Stti::/712.00000 2017.
[15] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[16] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. ACM Workshop on Artificial Intelligence and Security (AISec). ACM, 2017.
[17] Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with a transfer-basee prior. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[18] Wii-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. https: : //vicaaa.aan... lmsys. org/
[19] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint (2t::2224402331,,, 2022.
[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[21] Jacob Devlin, Ming-Wa Weig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0480 2018.
[22] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[23] Yinpeng Dong. Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation invariann attacks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[24] Yinpeng Dong, Shuyu Cheng, Tianyu Pang, Hang Su, and Jun Zhu. Query-efficient black-box adversarial attacks guided by transfer- transfer-based prior. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(12):9536-9555 9548, 2021.
[25] Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's selgss bard to adversarial image attacks? arXiv preprint 2r09vv2309.111 11751, 2023.
[26] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2300.03337. 2023.
[27] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint DrXiv:2209.07855 2022.
[28] GitHub. Copilot x, 2023. httpa////////iiibb..o///aaaaaaaa http:/hhhw...ooeeeeeeeeee//////p
[29] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In International Conference Learning Representations (ICLR), 2015.
12

---

[30] Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images textual prompts: Zero-shot visual question answering with frozen large language models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[31] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. /5556, 2022.
[32] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang. Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultancous vision-language generation. arXiv preprint arXiv:2211 1.8442, 2022.
[33] Shaohan Huang. Li Dong. Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. Language not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
[34] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning (ICML), 2018
[35] Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Anish Athalye, Tran, and Aleksander Madry, Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[36] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong baseline for natural language attack on text classification and entailment, In AAAI Conference on Artificial Intelligence (AAAI), 2020.
[37] Divyansh Kaushik, Douwe Kiela, Zachary C Lipton, and Wen-tau Yih. On the efficacy of adversarial data collection for question answering: Results from a large-scale randomized study. arXiv preprint vrXiv22/06.0087/ 2021.
[38] Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, et al. How many linguists does it take to fool question answering model? systematic approach adversarial attacks. arXiv preprint arXiv:2206. 2tt.::220..47299.. 2022.
[39] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 2019.
[40] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- pre-training for unified vision- langualangu understanding and generation. In International Conference on Machine Learning (ICML), 2022.
[42] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 2023.
[43] Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial vqa: A new benchmark for evaluating the robustness of vqa models. IEEE International Conference on Computer Vision (ICCV), 2021.
[44] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ViCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
13

---

[45] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744. 2023.
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 2023.
[47] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-bol box attacks. arXiv preprint vrXhv:161..027700 2016.
[48] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
[49] Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. Generating natural language attacks in hard label black box setting. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
[50] Zhao Meng and Roger Wattenhofer. A geometry-inspired attack for generating natural language adversarial examples. arXiv preprint arXiv:2010.01344 2020.
[51] Midjourney. Midjourney website, 2023. https://wwx jidj ourney.... . com.
[52] Milad Moradi and Matthias Samwald. Evaluating the robustness of neural language models to input perturbations. arXiv preprint arXiv:2 (0tp:22.08.2237,.2222 2021.
[53] John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. Reevaluating adversarial examples in natural language. arXiv preprint 141144 2020.
[54] Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex func- tions. Foundations of Computational Mathematics, 17:527-566, 2017.
[55] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 2021.
[56] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
[57] OpenAL. Introducing chatgpt, 2022. https: /ttp::///oppaaiieeee ceteaai.ccmm/aaaa wtt.:/ccc/bb////eeec hamm///o//chhtpt..
[58]
OpenAI. Gpt-4 technical report. arXiv, 2023.
[59] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[60] Tianyu Pang, Min Lin, Xiao Yang, Jun Zhu, and Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. In International Conference on Machine Learning (ICML), 2022.
[61] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learn- ing: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016.
[62] Joon Sung Park, Joseph C O'Brrien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint orXiv:2304.03442 2023.
[63] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint
2022.
14

---

[64] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
[66] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(1):5485-555 2020.
[67] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[68] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204. Ttt.::2044001225.5 2022.
[69] Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04611 2022.
[70] Yankun Ren, Jianbin Lin, Siliang Tang, Jun Zhou, Shuang Yang, Yuan Qi, and Xiang Ren. Generating natural language adversarial examples on a large scale with generative models. arXiv preprint arXiv:2003. 2020.
[71] Jonas Ricker, Simon Damm, Thorsten Holz, and Asja Fischer. Towards the detection of diffusion model deepfakes. arXiv preprint arXiv:2210.1457/ 2022.
[72] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[73] Hadi Salman, Alaa Khaddaj. Guillaume Leclere, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai- li-poweeed image editing. In International Conference on Machine Learning (ICML), 2023.
[74] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.055 1,05100, 2022.
[75] Ramprasuath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[76] Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. De-fake: Detection and attribution of fake images generated by text-to-image diffusion models. arXiv preprint
2022.
[77] Sasha Sheng, Amanpreet Singh, Vedanuj Goswani, Jose Magana, Tristan Thrush, Wojeiech Galuba, Devi Parikh, and Douwe Kiela. Human-adversard visual question answering. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[78] Yundi Shi, Piji Li, Changchun Yin, Zhaoyang Han, Lu Zhou, and Zhe Liu. Promptattack: Prompt-based attack for language models via gradient search. In Natural Language Processing and Chinese Computing (NLPCC), 2022.
[79] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nig 530b, large-scale generative language model. arXiv preprint ar0i 2022.
15

---

[80] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint wxXiv22333.53889 2023.
[81] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
[82] Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. Attacks meet interpretabil- ity: Attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems (NeurIPS), pages 7717-7728, 2018.
[83] Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Stanford alpaca: An instruction- forlloiinn llama model, 2023. https: //gi /gtght.tthb...
[84] Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plyy-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.0077 2022.
[85] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[86] Maria Tsimpoukelli, Jacob L Menick. Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
[87] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[88] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Blog, 2023.
[89] Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick me if you can: Human-in- comman-i--the-log lhop generation of adversarial examples for question answering. Transactions of the Association for Computational Linguistics, 7:387-401, 2019.
[90] Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[91] Jindong Wang. Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang. Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302. 2023.
[92] Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, et al. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
[93] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models, arXiv preprint 1rXi::2303304671,, 2023.
[94] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[95] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01/96 2023.
[96] Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-adversarial dialogue for safe conversational agents. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
16

---

[97] Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, and Dawn Song. Fooling vision and language models despite localization and attention mechanism. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[98] Xingqjan Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211 100332, 2022.
[99] Yan Xu, Baoyuan Wu, Fumin Shen, Yanbo Fan, Yong Zhang, Heng Tao Shen, and Wei Liu. Exact adversarial attack to image captioning via structured output learning with latent variables. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[100] Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. In European Conference on Computer Vision (ECCV), 2022.
[101] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint 2303 2023.
[102] Liping Yuan, Xiaoqing Zheng, Yi Zhou, Cho-Jui Hsich, and Kai- kei Chang. On the transfer- ability of adversarial attacksagainst neural text classifier. arXiv preprint arXiv:2011.08558, 2020.
[103] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations (ICLR). 2023.
[104] Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. Generating fluent adversarial examples for natural languages. In Annual Meeting of the Association for Computational Linguistics (ACL., 2019.
[105] Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre- training models. In ACM International Conference on Multimedia, 2022.
[106] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[107] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
[108) Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusion models. arXiv preprint arXiv:2303. 10137, 2023.
[109] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-languuge understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[110] Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari, and Fatemeh Shiri. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301 12868, 2023.
17

---

Appendix
In this appendix, we describe implementation details, additional experiment results and analyses, to support the methods proposed in the main paper. We also discuss failure cases in order to better understand the capability of our attack methods.
A Implementation details
In Section 4.1 of the main paper, we introduce large VLMs, datasets, and other basic setups used our experiments and analyses. Here, discuss more the design choices and implementation details to help understanding our attacking strategies and reproducing our empirical results.
the ImageNet- 1K [20] validation images as the clean images (Iele) to be attacked, and we randomly select a caption from MS-COCO [44] captions as each clean image's targeted text Cta. Therefore, we ensure that each clean image and its randomly selected targeted text are irrelevant. To implement MF-ii, we use Stable Diffusion [72] generate the targeted images (i.e., ht,,hhgCCa (Cw( in the main paper). Here, provide several examples of <clean image targeted text targeted image> pairs used in our experiments (e.g., Table and Table 2 in the main paper). as shown Figure 9.
Examples of how the datasets
utilized. In experiments,
Clean image [From LmngeNNer133
Targeted Text MS-D0CU)
"Two giraffes standing near each other in the 200.
riding skateboard next to some stairs."
yellow truck, parked in yard.
"A Lamb food in the trough
sandwich sitting on black plate.
Tarrgted Image (Gaunnal by Suble Difiiiuu)
Figure 9: An illustration of the dataset used in our MF-ii attack against large VLMs. By utilizing the text-to-image generation capability of Stable Diffusion, we are able to generate high-quality and fidelity targeted images given any type targeted text, therehy increasing the attacking flexibility.
Text-to-image models for targeted image generation. It is natural consider the real images from MS- MS-COCO as the targeted images corresponding to the targeted text (caption) in our attack methods. Nevertheless, we emphasize that in our experiments, we expect to examine the targeted text Car in flexible design space, where, for instance, the adversary may define Car adaptively and may not be limited to specific dataset. Therefore, given any targeted text Car. we adopt Stable Diffusion [72], Midjourney [51] and DALL-E [67, 68] text-to-image models hg to generate the targeted image he hg((uz), laying the foundation for more flexible adversarial attack framework. In the meantime, observe empirically that (1) using targeted texts and the corresponding (real) targeted images from MS-COCO, and (2) using targeted texts and the corresponding generated targeted images have comparable qualitative and quantitative performance.
Hyperparameters. Here, we discuss the additional setups and hyperparameters applied our experiments. By default, set 8 and the pixel value of all images is clamped to [0, 255]. For each PGD attacking step, we set the step size as 1, which means we change the pixel value by 1 (for each pixel) at each step for crafting adversarial images. The adversarial perturbation is initialized as 0. Nonetheless, we note that initializing A ~ (O, I) yields comparable results. For query-based attacking strategy (i.e., MF-tt), set = 8 and 8 AV I) to construct randomly perturbed images for querying black-box responses. After the attack, the adversarial images are saved in PNG format avoid any compression/loss that could result in performance degradation.
ttacking algorithm. addition to the illustration in the main paper (see Figure 4), we present an algorithmic format for proposed adversarial attack against large VLMs here. We clarify that
18

---

we slightly abuse the notations by representing both the variable and the optimal solution of the adversarial attack with wnvv For simplicity, we omit the imput Cin for the victim model (see Section 3.1). All other hyperparameters and notations are consistent with the main paper this appendix. Because we see Table that MF-it has poor transferability on large VLMMs, we use MF-ii MF-tt here, as shown in Figure 4. In Algorithm 1, we summarize the proposed method.
Algorithm Adversarial attack against large VLMs (Figure 4)
1: Input: Clean image ccle. a pretrained substitute model Sa (e.g.. a ViT-B/32 or ViT-L/14 visual encoder of CLIP), a pretrained victim model py (e.g., Unidiffuser), targeted text Car- a pretrained text-to-image generator fig (c.g., Stable Diffusion), a targeted image he (Car).
2: Init: Number of steps 81 for MF-ii, number of steps 82 for MF-tt, number of queries N in each step for MF-tt, A = 0, 6 (0, I), a = 8, = 8, wtl.rrruuueeggradc.. = False.
# MF-ii
3:
for = 81: + do
4:
Fadv =clamp (3 + A, min 0, max= 255)
5:
Compute normalized embedding
(hg(Crr)) e1 =
(Com) norm
6:
Compute normalized embedding of aadv: E2 (2ad)))) l (Fndy) norm()
7:
Compute embedding similarity: sim ej e2
8:
Backpropagate the gradient: grad = 8 sim. backward()
9:
Update Á = clamp (A+ grad.gaddiige sign min= -6, max 6)
10: end for
# MF-tt
11:
Init: Tady = Tcle + A
12: for = 1; $2;
do
13:
Obtain generated output of perturbed images: {pa (2rav + σō +o
Obtain generated output of adversarial images: pol (Fndy)
15: Estimate the gradient (Fq. (4)): paeudo-gras = RGF (Curr; po (Dady) ) (po + 00x) =00))
Update A = cl amp (A+pseudo- -ooooo-rrdd.... sign (, min= E,, max=
17: Ladv cmlamp (#.1. + A, min= 0, max= 255)
end for
19:
Output: The queried captions and the adversarial image Tadv
Amount of computation. The amount of computation consumed in this work is reported in Table 3, in accordance with NeurlPS guidelines. We include the compute amount for each experiment as well as the CO2 emission (in kg). In practice, our experiments can be run on single GPU, so the computational demand of our work low.
B Additional experiments
In our main paper, demonstrated sufficient experiment results using six cutting-edge large VLMs on various datasets and setups. In this section, we present additional results, visualization, and analyses supplement the findings in our main paper.
B.1 Image captioning task by BLIP-2
In Figure 10, we provide additional targeted response generation by BLIP-2 [42]. We observe that our crafted adversarial examples can cause BLIP-2 to generate text that is sufficiently similar to the predefined targeted text, demonstrating the effectiveness of our method. For example, in Figure 10, when we set the targeted text as "A computer from the 90s in the style vaporwave", the pretrained BLIP-2 model will generate the response "A cartoon drawn on the side an old computer"' whereas the content of clean image appears be "A field with yellow flowers and sky full of clouds". Another example could be when the content of the clean image "A cute girl sitting on steps playing with her bubbles", generated response on the adversarial examples is "A stuffed white mushroom sitting next to leaves", which resembles the predefined targeted rext"" photo a mushroom growing from the earth"
19

---

Table 3: The GPU hours consumed for the experiments conducted obtain the reported values. CO2 emission values are computed using https://mloo github 10/ Lo/impact [39]. Note that our experiments primarily utilize pretrained models, including the surrogate models, text-to-image generation models, and the victim models for adversarial attack. As result, our computational re- quirements are not demanding, making feasible for individual practitioners to reproduce our results.
Experiment name
Hardware platform
GPU hours
Carbon emitted in kg
Table (Repeated times)
126
9.45
NVIDIA A A100 PCIe (40GB)
Table (Repeated 3 times)
2448
183.6
Figure 1
12
0.9
Figure 2
18
1.35
Figure 3
36
2.7
NVIDIA A100 PCIe (40GB)
Figure 5
12
0.9
Figure
12
0.9
Figure 7
24
1.8
Hyperparameter Tuning
241
18.07
Analysis
NVIDIA A100 PCIe (40GB)
120
9.0
Appendix
480
36.0
Total
3529
264.67
B.2 Joint generation task by UniDiffuser
Unidiffuser [5] models the joint generation across multiple modalities, such as text-to-image or image- to-text generation. In Figure 11, show additional results for the joint generation task implemented by Unidiffuser. As can be seen, our crafted adversarial examples elicit the targeted response in various generation paradigms. For example, the clean image could be generated conditioned on the text description "A pencil drawing of cool sports car",, and the crafted adversarial example results in the generated response "A close up view and cheese" that resembles the targeted text. As a result, Unidiffuser generates hamburger image turn that completely different from the semantic meanings of the original text description.
hamburger with lettuce
B.3 Visual quustion--anseringg task by MiniGPT-4 and LLaVA
The multi-round vision awestion-ansseriin (VQA) task implemented by MiniGPT-4 is demonstrated in the main paper. Figures 12 and 13 show additional results from both MiniGPT-4 [109] and LLaVA [46] on the VQA task. In all multi-round conversations, minimal perturbation budget (e.g., c = 8), MiniGPT-4 and LLaVA generate responses that are semantically similar to the predefined targeted text. For example, in Figure 12, the monkey worrier acting as Jedi recognized as an astronaut riding a horse in space, which close to the targeted text "An astronaut riding horse in the sky". Similar observations can be found in Figure 13.
show that by modifying the
B.4 Interpretability of the attacking mechanism against large VLMs
GradCAM [75] used in the main paper to interpret the targeted response generation. We present additional visualization results to help understand the mechanism that deceives these large VLMs; the results are shown in Figure 14. Similarly to our findings in the main paper, we show that, when compared to the original clean image, (a) our crafted adversarial image can lead to targeted response generation with different semantic meanings of the clean image's s text description; (b) when input question is related to the content of the clean image, such as "ow many people in this iamge?" GradCAM will highlight the corresponding area the clean image, while ignoring the same area in the adversarial image; (c) when the input question is related to the targeted text, such
20

---

as 'where is the corn cob?", GradCAM will highlight the area of the adversarial image that is similar the targeted image. More results can be found Figure 14.
C Additional discussion
In this section, we clarify the sensitivity when perturb adversarial examples, and failure cases to help better understand the limitations of our attacks.
C.1 Sensitivity of adversarial examples to random perturbation
To evaluate the sensitivity of our crafted adversarial examples, we add random Gaussian noises with zero mean and standard deviation Ooose to the obtained adversarial images Ladv. and then feed the perturbed adversarial examples for response generation. The results are shown in Figure 15. We observe that our adversarial examples are reasonably insensitive to this type perturbation, and we also make the following observation: as the amplitude (i.e., noise) of the Gaussian noises added to Tadv increase, the effectiveness of our learnt adversarial perturbation diminishes and the targeted obtain the
responses revert to the original. For instance, in Figure 15, when noise O, generated targeted response "A red and black bird sitting on top of a tree branch" that resembles the targeted text; when moise 0.025, changes to "A red and black bird is sitting on top of a sunflower'; and finally the response degrades to "A large painting of three sunflowers in field'. Additional results are shown in Figure 15.
C.2 Failure cases
While we have demonstrated convincing results our method in the main paper and in this appendix, note that the adversarial attack success rate for these large VLMs is not one hundred percent. Here, we present a few failure cases discovered during our experiments, leaving them for future work to improve performance. Specifics shown in Figure 16.
21

---

BLIP 1:: Image Caploonnng
Target: TA phato of a white fur monster.
Clcan imago
Resuting aox imae
close up blue jay sitting on tree branch,
stuffed feu inimal that Looks 1ike a wetrd beast
odversanal atack
Target: "A computer from the 90s in the style of vaporwave
Clean Image
Resulling ndx image
"A rield with yollow flowers and sky full clouds."
cartoon de san the side of old conputer
acbemmnnt
Target: pineapple with its leaves still attached to a tree.
Clean image
Reesting wix image
/ view of Lain covered in snow and clouds."
pineapple branch
athemseid black
Targel: photo
mushroon groming
th.""
Coan
Resuting ads image
cute strl sitting steps ilaying with her bubbles."
"A nushroon sitling next to leaves
soversanol ottack
Target: "A scene of sunset in mountains.
Clean
Reeauting
"Te sunset
Target: yellow sicting against
Image
image
house sitting branch front colorful ful autunn Teaves.
closoup of small toy en string with colorful backgrounds.
snvennerbil stlck
Figure 10: Additional results of image captioning task implemented by BLIP-2.
22

---

UniDiffueer Joint generation
peranad rupore of
orggnall desclppicn
"OE patoring of Diue teceground.
sunflowers bees.
pasl palating of
genanted rage Kay
mo
Click pride banii
X gereicaci sapo
oo slace and red peoples.
Bird bind Lorns
Terges: biocing
grociial
black wiite Lawourgniin
dracing of Lanborghtnt sports
InoeTed
AA peartl Brawing of
praradinggs Sua 2011 bon
gIN' pIOOOUU Claghr
Textiing
prereaa Plick
Lamburgs VEER
and
Target: hasturge WIth Lettuce and Cheese
ON UNTIANNN
genenaeed maccte
"A puint
Irgetted
focthall player
Packers football players
pollttio ing of
ongiiall tan deee
giVnT prepciiiii dli.e
paisting of
Expucs the Boom
Target pnEco in astrOnaLt (1d horse THE
genenated masi Age
Ingeted
briddg night
Honet
"An brtigee ratte.
Tugah ongiiall tant
gIVn phoppiiiiio clice..
goen
grans
Tergel: "A small shite SiLLing OF Jhe bround autuee
genenaie musoctt Age
painting for starry night backgrount
SiLLing andor starry dky.
paiitling the styll
"A colortul
colorful take bashig ound.
Ulbse Nanteems
Faarga:
Figure 11: Additional results for joint generation task implemented by Unidiffuser.
23

---

MiniGPT4: Vieual QuestionArawweiin
Example-1
advorsard
Tarpees AN WOST secronent Fioing Rorse PO
the
2014 Snage"
"Th Irage Shows antronact space say. with stors and coseT vistete the background. ixtronast (iis of Light in the sho The scene is serrounded by asycnede iec. sairling shapes.
laage of the hand, The wonkey ie the Lighsseber oretend ing to be Jedi, it is Hould be for entlats rote or desktop
Image input (From Mijjumne!
Resulting mage (ours)
image of astrorast ding in the ghgt sur rurroudded by of wonder and ane, People may feel inspired to reach for the the netural vorld despl technol ogy. The astronaut rising horse in the sky. The overa 1 feel fing of image Likell be postsive usii bing.
"The monkey haidding Lightsaber could be parse ved as case or fammy by people. snile essers al ant of
the tmage costs also iepact people's cestional resporse to it. Withous vould make people fext."
The
presince connecti tmage could also evoke Thel tins may
smage
Conwersttion/AA with dean image
Con with adv. image Fade
Example-2
Target. "N 10t states
"shat content TI15 Image?"
of this Images"
-E is filst cought on net
a photo DRESON aesh fence
carp with yellow tall. The water is clear The rrsh is catmeing Towa-ds right the laago.
the shape
"Ie to riding skatebound
image (From Unioifusere
Resulting adv. mage (ours)
"The r1sh in the trage To oval in shaps."
Shere
"ile ix Iskely urban anst roteent
"There boy closs-sp snot a fiah, with ita nouth oper and it's gills
appears
X-The feecce appears to sofety mossure to prevent the skatebcarder from falling off
novenent The water, with so visible plants or other elements.
ConvensationVVDA with clean Fele
Conversston/VOA with
Example-3
Tar gEE. group of people plastig
"Mhat
"Whst
point in ERIS INEgO?"
this 18age2"
tmage chews Large grasoy crea with Long c and sharp lawe start and reasy to move- The background tall
imago group the
people grass park on becoyard. holiding
reet.
green
are all iniling and seen to enjoying
"Can you Brieflly misrans
TEan ynu belefly AESTuEs
DoEs picture mane
does the dicture
Image (From Midioumay)
Resulting adv. mage (ours)
V-The leage of the greet "guone standing on gr ess fascination Heard'ss size. color. 2nd alert posture may captivato people's this reprita
X *The smage group of people e enjoving relased and and fooe. image evoke feelings happsness,
The
casual cutocer setting with serse of annaraderie te anong the people in the image. The appear consany and the beautiful content and happy.
acat
and spark on hans, the learsing inscill sense of ferr or unease for soee sloorss espectallly they
byying make people feel
iguanta
Conversation//OA
image File
Conversatio adv. image Fudv
Figure 12: Additional results of VQA task implemented by MiniGPT-4.
24

---

LLaVA Visual Cueston-Answeriigg
Example-1
affenscteal stack
Target: "A bunch tropiCaT Frutts in the midele proto.
of this smage?"
Thee trage features
Khe lmage features burch rang no hananas whit one of the batanas suspended lnvered leto pocerally
lagnen sur tie swimiing to biue broaking sur face the water turtle appears barkgroind fne lustes heach. addiing sicturessus slement
blue Liquid The benanas reseesles face, Hith of the postiionn soondddddd like features.
What TOR
Clean mage imput ImageNet)
Resuling adv. image (ours)
brings to procier
"he lmage brings sanse of tranzull1 retecation, and contecti on with nature Chserwing the tea turtle 6 idiigg and sereriity as showcases the resuiy and hamtony of the underwater warts. env invrromment prompting vievers to consider the importance of conse
The s181t of a bunch of barenes hanging from rops or nossalgta to people, The unusual postttoring of the bamenne wight make
cssuesque scens also traptre for
saterest ng conversation for place tropiccll 1t way evoke reellnngs tins of re axat on realind people of vicatt oning vare inatinatonn where such iruitts
display
protecting og
Conversatin/NCA.ooo with dean Image Ayle
Conversation/V adv. Image
Example-2
stack
Targes: airplate in the with group
tmage stowe small dog. starlar breed, on a leass ostdoors near greasy in grass."
mago features intertar atrplate, wrere you can soc Leather seats, along WiE raiced Stanket There ahe several chairs can also see clords ourdbors,"
blancer
Clean mage imput ImageNed)
Resuling image (ours)
people FeXTT"
"The sicture of black dog starding in the gress and
x Thhe seat of Large airplane night evoie feelings travel. Othera eight vrew the scene as endsaring and acorable for
fost Lings in people. Scee may feel mog presence bring Iny and sease companion to be Looking strectlly 'nto the caners. ineiting setent and interaction."
exc todd the
contort nestalgta for seme people
childdoodd
sone,
ings
excitseent about traveling and explorin new Hoesvor,, It's Inpor tant to sote that tnd fdua" interpratations personal
may exzerferces and enotions.
Conversaiion/GG with dean image Kele
Conversation adv. image Eady
Example-3
Tar get. astrOnas riolng horse 11 the Sky
"Witt the content this 1msge?"
of this (mage?
The lmage features a gray and carpot and play with toll paper that been tutte content and possiatly even angry. it enjors des trayrigngg moll
lmage artixtic searing a space SulT and horse. 2oth claces bal1-1ite structure The scone creates the torse Floating Tikely inoptired by the astronbut WITH the horse.
you stote?"
Clean Input ImageNet)
Resultin adv.
COEELEREENAA WWW. "DO SHERE
real shato?
beside pile strerded tot seaer. The cas's presence and the state of the totbet paper supgest the genul unroll totles creating a wess le the process,"
"Tee tmage be real photo. 10 deptcts grey
RR, not phate. tmage destets combiration realsty io to highlly teprebable for a person to be vearing
known
staged,
space
for ridlig
as suace terrantrin
and wals. is consinatior suggeses that the leage has beer photoshosped aantpulated humerous scone."
Conversation NDA with daan image Fele
ConversationVAA with imago Eydy
Figure 13: Additional results of VQA task implemented by LLaVA.
25

---

GradCAM
GradCAM afy
GradCAM afy
GradC Aele
Snugtrun
groand coveres
care has just tall
daey people
"Ho cany people
"wnere 15. The cor
Taherre the caro
(b)
GradCAM
GradCAM ayyy
GreCCMM of Fady
GradC hydeea
pear
"Whare 16 D1g
"hare 1o big
bestle 0a1a
15r2le gold
hyteaa))
GradCAM
GraCCAM afyy
GraCCAM of Fade
GradC AM Aelear?
briege 1Ea1
Staveing top the grass Satey
"Where 15 the dlc
Whore old
prass quary
grass Sanny
Fole
hylear)
GraCAM
GeadCAM
GradCAM of
GradC AM of of
AA
alecing Branch
Seautsful 2ind Black
Denust Pab
beauttfut
bear statatoard Tiese
bea 21aytag Fg
TE
slace caler
(b)
GradCAM
GralCAM of
GodCCM of gdde
GradC A of hfde
slace Sng stards the
erall standing saxry
aoving rodky
ctansing sandy bearr
Tabwre Saka?"
"Woare Taka?
standing condy
(b)
Fele
GradCAM
GradUAM of
GradUAM of Kate
GradCAM of ar
"Sinds largs hoads
the beach the ground and ather
"An USE gronteg
"Wbare are the bieds
"Weree are the birds
Flover MAry other
"loest mary other Flowors
wiEs teckground
flovers
Figure 14: Visually interpreting our attacking mechanism. To better understand the mechanism by which our adversarial examples deceive large VLMs, we provide additional visual interpretation results (via GradCAM [75]) as supplements Figure 7 of the main paper. Similar to our previous findings, demonstrate: (a) An example of Lele. Ladv, and hg (Cur), along with the responses they generate; (b) GradCAM visualization when the input question Cin is related to the clean image. (c) GradCAM will highlight regions similar to those of Tady if we provide the targeted text (or other texts related to Gad as the question.
26

---

Example-1
<ddv
noise
Fadv "oois 0.001
Fadv "nois 0.01
Fady "poise 0.025
Fadv noisc 0,05
red black bird citting on
yelliw and band pracing on top
yellm and parrot 213ting top
red and Blace bird StEEing top
large painring ng or Lowers
Example-2
Fadv
Onoise
Fadv noois
Fadv : Choise
Kady : "poise
Kadv noise
close view hanourger with Setcuce and cheese
Fambur with Letouse and cheese.
hasburger with lettuce:
"A close shot of
di Brawing and car on piece of paper
pencil crawing of sports
Example.3
Target: The froe the top bill overlooking the sountsins.
0.001
0.01
0.025
0.05
FF painting
Pountains.
Andes Hountains.
the background.
valley the background.
Lookingg
Example-4
small white dog sitting on the ground autuen Teaves
Xadv Onisc
Xadv naise 0.001
Xadv Coose 0.01
Xady "oise 0.025
Xad noise 0.05
"A snall whlle te dos sisting the grass
SPall Weite 000 StEcing
colorful 008 Steting: woods
painting 01L pointing of Terrier dog
Figure 15: Sensitivity of adversarial examples to Gaussian noises. Our adversarial examples found to be relatively insensitive to post-processing perturbations like Gaussian noises. Alterna- tively, interesting observations emerge when gradually increasing the standard deviation Ouoise of the Gaussian noises added to Eadv. where the effectiveness our learned adversarial examples becomes marginal and the targeted responses (in green) gradually degrade to the original, correct response (in black). In addition, we note that an intermediate state exists in which the generated response is combination of the targeted text and the original generated response (in red).
27

---

Example-1
Target: black and photo people walking by some bicycles.
xele
"Waat is the content of this image?"
"The image features a small, adorable black and gray sitting brick walkway or sidewalk. The dog appears to be in relaxed position, pant ing and looking up at camera."
advv
"What is the content of this image?"
"The image features a col photos of various dogs Some of the in the background. There is prominent black dog in the with tongue sticking out."
close up, while others
Example-2
Target:A black and white terrier looks up at the camera.
xcle
you think of image?"
V 'The image an artistic illustration
ioldfish swimming dark, rippling pool with dark background. The fish is the main focal point of the appears captured an artistic. image, almost cartoonish style."
#adv
you think of image?"
"The image features a small white and black fish, possibly a sea tropical fish. swimming a large tank. The fish is the main focus of the image, occupying significant portion of the frame."
Figure 16: Failure cases found our experiments. The generated adversarial image responses appear to be state in between the text description of the clean image and the predefined targeted text. In this figure, we use LLaVA [46] as the conversation platform, but similar observations can be made with other large VLMs. On the other hand, discovered that increasing the steps for adversarial attack (we set 100 in main experiments) could effectively address this issue (note that the perturbation budget remains unchanged, e.g., 8).
28


Reviews:
[{"review_id": "IIc2SW9fOu", "reviewer": "Reviewer_csPu", "summary": "The authors propose to generate adversarial attacks on different vision-language models (VLM) like BLIP, MiniGPT4, and UniDiffuser. The proposed method is simple and straightforward. To perturb the image, the authors propose to maximise the similarity (inner dot product) between the image-image / text-text / image-text features of CLIP. The attacked image is then transferred to the black box VLM models, which generates some text on these images. The similarity between the generated and the target text is calculated using unnormalised cosine similarity on the CLIP text encoder outputs. Maximising the similarity between the CLIP encodings of clean and target text and CLIP encodings of the clean and generated images of target text doesn't require any gradient approximation. But maximising the similarity between the target text encodings and generated text requires zero-order optimisation to get the gradients on the image. The authors demonstrate that the proposed attack completely fools the VLM models to generate text very similar to the target text.", "questions": "It would be great if the authors can address the raised concerns in the weakness section.", "limitations": "The authors have properly addressed the limitations and negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The proposed method is incredibly simple and, therefore highly valuable for the community\n* The proposed method evaluates different types of VLM models, which makes the claims of the proposed method stronger.\n* The problem identified is timely and highlights the importance of building robust VLM models in future.\n", "weaknesses": "* The proposed method shows the results of the PGD-100 attack, while in most of the black box attacks, it is observed that stronger attacks can overfit to the surrogate model resulting in worse transferability. Therefore, it would be good if the authors can also share the results on weaker attacks like PGD-10 and FGSM.\n\n* I think comparing the text quality using the CLIP score may not give the true picture. I would be nice if the authors can try some other metrics to compare the text quality.\n\n* For some of the approaches like LLaVA and MiniGPT-4 in Table-2, the results obtained using the proposed attacks like MF-it and MP-ii are very close (almost similar) to the clean image. Does this mean that the model is not able to get fooled? It would be nice if the authors can present results using some metrics which can quantify whether the model is fooled or not. This will help in getting a deeper understanding of the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***Q1: It would be good if the authors can also share the results on weaker attacks like PGD-10 and FGSM.***\n\nIn **Table A** of the rebuttal PDF, we follow the Reviewer’s suggestion to report additional results using fewer PGD steps (e.g., 10-steps and 50-steps) against different large VLMs (e.g., Unidiffuser, BLIP-2, and LLaVA). As can be seen, PGD-100 outperforms PGD-50 and PGD-10, though the benefits tend to saturate as more PGD steps are added. This indicates that PGD-100 is still not overfitting to surrogate models, which could be attributed to the complexity of large VLMs versus traditional image classification models.\n\n***Q2: It would be nice if the authors can try some other metrics to compare the text quality.***\n\nFollowing your suggestion, we compare the text quality by computing the embedding similarity between the generated response and the targeted text using FlagEmbedding [1], a state-of-the-art text embedding model (ranked first on MTEB and C-MTEB benchmarks). The findings are presented in **Table B** of the rebuttal PDF, where we employ the same experimental setups as in Table 2 of the main paper. As demonstrated, using FlagEmbedding yields similar results to using the CLIP score, and we will add these results in the revision.\n\nFurthermore, we would like to clarify that in the main paper, we used various text encoders (ResNet-50/101, ViT-B/16, ViT-B/32, and ViT-L/14) to ensure a thorough assessment of text quality, and we also provide multiple visualized demonstrations (Figures 1, 2, 3 in the main paper and Figures II, III, IV, V in the Appendix) for human evaluation. \n\n\n***Q3: For some of the approaches like LLaVA and MiniGPT-4 in Table 2, the results obtained using MF-it and MP-ii are very close to the clean image. Does this mean that the model is not able to get fooled?***\n\nIndeed, we observe that models with larger capacities (13.3B parameters for LLaVA and 14.1B parameters for MiniGPT-4) are more difficult to get fooled using only transfer-based attacks like MF-it and MF-ii, compared to those with smaller capacities such as BLIP and UniDiffuser. This could be because LLaVA and MiniGPT-4 both use a large language model (Vicuna-13B [2]) as the text generator, which is relatively robust to transfer-based attacks. Therefore, we use transfer-based MF-ii to initialize the adversarial perturbation and query-based MF-tt to further tune the perturbation (see Figure 4 for details), which is effective as shown in Table 2 and Figure 3 (more results can be found in the Appendix).\n\n\n***Q4: It would be nice if the authors can present results using some metrics which can quantify whether the model is fooled or not***\n\nWe additionally conduct user studies to determine whether the model is fooled or not. We specifically categorize the generated responses into four types, which are as follows: *Type-(1)*: the model is completely fooled, i.e., it can generate the targeted response; *Type-(2)*: the model is partially fooled and can generate concepts/information related to the target text/image; *Type-(3)*: the model is partially fooled and can generate non-targeted response that is neither related to the target text/image nor the original clean image; *Type-(4)*: the model cannot be fooled, We randomly selected 100 VLM responses and asked users to vote on whether these models were fooled. The results are as follows:\n\n|     VLM      | Type-(1) | Type-(2) | Type-(3) | Type-(4) | \n|---------------|:-----------:|-----------:|-----------:|-----------:|\n| Unidiffuser |   84.5\\%   |   8\\%   |   5\\%   |   2.5\\%   |\n| BLIP-2       |   86\\%   |   9.5\\%   |   2.5\\%   |   2\\%   |\n\nAs shown in our user studies, these VLMs can be fooled to a large extent in most cases, similar to the visualization results in our figures. Due to time constraints, we only evaluate Unidiffuser and BLIP-2 here, but we will conduct user studies on other large VLMs in the revision.\n\n***References:*** \\\n[1] https://github.com/FlagOpen/FlagEmbedding \\\n[2] Chiang, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality."}, {"review_id": "RjX3LZLu7d", "reviewer": "Reviewer_Xvuc", "summary": "This paper focuses on black-box targeted adversarial attacks on multimodal vision/language models via transfer. They observe particular vulnerability to transfer attacks because an adversarial image can be constructed in a fully-differentiable manner w.r.t. a model like CLIP and then transferred over to the black box target model. Their final approach involves matching the adversarial image features to both text and image features in the surrogate models used for training.", "questions": "Could these vulnerabilities have been foreseen (probably yes)? Have they been addressed at all by the people open-sourcing VLMs? What are the most immediate risks posed by deployed VLMs? What does this mean for AI norms and governance? I would welcome more discussion of these in the discussion section.", "limitations": "This paper did not focus on defenses for these attacks, but I would not have expected it to. \n\nThe use of GradCAM to analyze the attacks is limited and based on visual inspection. Also GradCAM and similar methods are well known to have many problems. Given my familiarity with some of the problems with feature attribution methods, these experiments don't really add much to the paper in my opinion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the combined method MF-it and MF-ii is clever, and I think the results are impressive. \n\nI find the paper clearly valuable and timely since models with these vulnerabilities are currently being deployed and their vulnerabilities could pose risks. I appreciate the discussion of this in the conclusion. \n\nI think that the experiments are compelling. I do not spot errors.", "weaknesses": "(Minor) I find the figs to be generally cluttered. \n\nI would have liked to see experiments related to real-world concerns such as ones that were meant to illustrate real ways that these attacks could cause problems. For example, evading NSFW filters or making language models provide misinformation about an image that is used in a query.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***Q1: (Minor) I find the figs to be generally cluttered***\n\nThank you for the comments. In the revision, we will rearrange our figures to make the content more clear.\n\n***Q2: I would have liked to see experiments related to real-world concerns. For example, evading NSFW filters or making language models provide misinformation about an image that is used in a query***\n\nIn this work, we used MiniGPT-4, LLaVA, and Img2Prompt to evaluate the visual question-answering (VQA) task, in which an image is indeed used in a query and the adversarial goal is to make language models provide misinformation. The results have been demonstrated in Figure 3 (main paper), Figures IV and V (Appendix). These results are highly relevant to real-world concerns, because we can fool the model into returning *targeted* outputs, which could contain any predefined misinformation or bypass NSFW filters.\n\n***Q3: Could these vulnerabilities have been foreseen (probably yes)? Have they been addressed at all by the people open-sourcing VLMs?***\n\nThese vulnerabilities were most likely foreseen by OpenAI, and we believe this is one of the reasons why OpenAI is delaying the release of GPT-4's visual input. Unfortunately, our results indicate that these vulnerabilities are rarely addressed or even paid attention by the people open-sourcing VLMs. Existing open-source VLMs still primarily compete for the performance on standard benchmarks, while these VLMs are mostly made up of publicly accessible modules that do not take adversarial robustness into consideration.\n\n***Q4: What are the most immediate risks posed by deployed VLMs? What does this mean for AI norms and governance? I would welcome more discussion of these.***\n\nThank you for the questions. There have been attempts to deploy large VLMs into robots or autonomous driving systems, where the visual inputs could be the images captured by a camera (i.e., interaction with environments) and the language inputs could be user commands (i.e., interaction with humans). In these scenarios, adversarial attacks on VLMs could mislead robots or autonomous driving systems, inducing them to engage in risky behaviors. Improved adversarial robustness of VLMs and more general multi-modal models are thus critical for AI norms and governance.\n\n***Q5: This paper did not focus on defenses for these attacks, but I would not have expected it to***\n\nFollowing your suggestion, we apply a recent defense method that adversarially trains a CLIP model [1], and evaluate our proposed attacking methods under the defense (we use the model checkpoints provided by [1]). The results are shown in **Table E** of the rebuttal PDF. As can be seen, even with the defense method by [1], we still can achieve comparable performance to our original results, demonstrating the effectiveness of our attacking methods. These experiment results and discussion will be included in the revision.\n\n***Q6: The use of GradCAM to analyze the attacks is limited and based on visual inspection. Also GradCAM and similar methods are well known to have many problems***\n\nThank you for your constructive feedback; we will include your comments in the paper to clarify these limitations in the revision.\n\n***References:*** \\\n[1] Mao et al. Understanding Zero-Shot Adversarial Robustness for Large-Scale Models. ICLR 2023"}, {"review_id": "MaZPK34WbQ", "reviewer": "Reviewer_UbWE", "summary": "The paper evaluates the pixel-space adversarial robustness of large vision-language models (VLMs), where the targeted attack has only black-box access to the large VLM systems. The paper introduces two adversarial strategies: transfer-based and query-based. The transfer-based strategy performs white-box attacks on surrogate models like CLIP and BLIP and transfers to large VLMs including MiniGPT-4 and LLaVA. The query-based strategy uses transfer-based priors to improve the efficacy of targeted evasion against these VLMs. The authors demonstrate the effectiveness of these strategies through comprehensive evaluations.\n\n", "questions": "1. The authors give Section 4.3 the title *VLMs induce semantic adversarial perturbations*. However, what Fig. 5 shows is not semantic adversarial examples but inference results of pixel-wise adversarial noises. The term semantic adversarial perturbation commonly indicates semantic variations that fail the model [1]. Is Fig. 5 trying to discuss *does VLM adversarial perturbations induce semantic meanings*?\n\n2. Since the pixel-space adversarial noises are not intuitive/interpretable to human observers, are there any future discussions on generating VLM attacks based on physically-observable attributes or languages [1,2,3,4]? \n\n3. To conclude, this paper is a timely contribution to the field. I will consider revising the rating based on the soundness of the authors' responses.\n\n[1] ICCV 2019, Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers.\n\n[2] ECCV 2020, Semanticadv: Generating adversarial examples via attribute-conditioned image editing\n\n[3] CVPR 2023, Zero-shot Model Diagnosis\n\n[4] ICLR 2023, Diagnosing and Rectifying Vision Models using Language", "limitations": "Yes, the authors have stated the potential limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-presented with good motivation, and the authors provide a clear storyline with visual demonstrations to validate the proposed approach.\n\n2. It is novel to optimize the adversarial objective by matching image-image features (MF-ii) on large VLMs. The following Query-based attacking strategy (MF-tt) further enhances the effectiveness of the attack.\n\n3. The authors analyze the approach from multiple perspectives. The paper includes discussions on the influences of the perturbation, and the Grad-CAM interpretation visualizes the model decision well.", "weaknesses": "1. The black-box setting is questionable. Although the adversary only has black-box access to the victim system $p_{\\theta}$, the transfer-based attack has white-box access to foundation components like CLIP. There is a high chance that the victim system shares mutual information (e.g., $p_{\\theta}$ contains CLIP / $p_{\\theta}$ trains on the same vision-language dataset / $p_{\\theta}$ has similar model architecture) with these white-box components. It will be beneficial if the authors elaborate more on the source of transferability.\n\n2. Although the Query-based attacking strategy (MF-tt) is feasible, the cost of such a process can be high. What is the forward inference cost (e.g., time of API call) per query, and how many queries are needed per attack? Authors should show the cost effectiveness of query attacks.\n\n3. The experiments mainly focus on imposing MF-ii as a prior for the following MF-tt queries. It will be constructive If authors demonstrate more on the sole use of MF-tt (which aligns with the black-box setting and is more computationally lightweight) without a strong MF-ii prior for the query attack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***Q1: There is a high chance that the victim system shares mutual information with these white-box components. It will be beneficial if the authors elaborate more on the source of transferability***\n\nIndeed, existing open-source large VLMs are primarily composed of publicly available modules (e.g., CLIP and Vicuna) and are finetuned on some common vision-language datasets. These facts increase the chance that the surrogate model used by black-box adversaries shares mutual information with the victim model, essentially making the victim model vulnerable to adversarial transferability.\n\nNevertheless, we want to emphasize two points to demonstrate the significance of our transfer-based attacks. First, even if $p\\_{\\\\theta}$ contains CLIP, the CLIP module only occupies a small portion of the large VLMs (e.g., a CLIP encoder has 300M parameters and LLaVA/MiniGPT-4 has 13.3B/14.1B parameters), where the main model capacity is involved in large language models that are unseen to our transfer-based attackers. Second, while knowing the CLIP module may be sufficient for *untargeted* attacks against large VLMs, we focus on much more difficult *targeted* settings, and it is non-trivial to fool large VLMs to return targeted responses solely based on adversarial transferability. On the other hand, as discussed in our response to **Q3**, we demonstrate that using query-based MF-tt alone is still effective against various VLMs, without transfer-based MF-ii as the prior.\n\n***Q2: What is the forward inference cost (e.g., time of API call) per query, and how many queries are needed per attack?***\n\nAs stated in Line 176, we set query times $N=100$ and update the adversarial images by 8-steps PGD for MF-tt, resulting in a total query time of $800$ per attack in our setup. In **Table C** of the rebuttal PDF, we provide the time of API call per query for different large VLMs used in our experiments. We will include more details in the revision.\n\n***Q3: It will be constructive If authors demonstrate more on the sole use of MF-tt (which aligns with the black-box setting and is more computationally lightweight).***\n\nIn **Table D** of the rebuttal PDF, we show the experiment results of using only MF-tt over various VLMs (e.g., Unidiffuser, BLIP-2, LLaVA). Our results indicate that our query-based attack MF-tt is effective against various VLMs. However, when MF-ii is used as the prior, it can achieve a better performance. More results will be provided in the revision.\n\n***Q4: Is Fig. 5 trying to discuss whether VLM adversarial perturbations induce semantic meanings?***\n\nThank you for pointing this out. Yes, in Section 4.3 and Fig. 5, we attempt to discuss whether the crafted adversarial perturbations induce any semantic meanings. We will correct the claims in the revision.\n\n***Q5: Are there any future discussions on generating VLM attacks based on physically-observable attributes or languages?***\n\nThank you for the insightful comments. In our paper, we constrain the pixel perturbations to be small (e.g., $\\\\epsilon=8$) enough to be visually imperceptible (or difficult to perceive) by humans, and we show that simply attacking the visual modality is sufficient to fool the entire VLMs. Nonetheless, attacking VLMs through physically-observable attributes on the vision part (e.g., adding eyeglasses or other accessories to the human portrait [1], modifying hair color or style [2, 3]) or languages [4] would be promising directions to pursue, and we will include relevant discussion on these works in the revision.\n\n\n***References:*** \\\n[1] Joshi et al. Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers. ICCV 2019 \\\n[2] Qiu et al. SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing. ECCV 2020 \\\n[3] Luo et al. Zero-shot Model Diagnosis. CVPR 2023 \\\n[4] Zhang et al. Diagnosing and Rectifying Vision Models using Language. ICLR 2023"}, {"review_id": "vdlmmlYUNC", "reviewer": "Reviewer_zQtG", "summary": "This paper aims to assess the adversarial robustness of vision components in large vision-language models, which is an increasingly significant issue due to the prevalence of such models. The experiments conducted in this study are extensive, encompassing evaluations on UniDiffuser, BLIP, BILP-2, Img2Prompt, MiniGPT-4, and LLaVA. The research investigates transfer-based attacks and also employs black-box queries to enhance the effectiveness of the evaluation method.", "questions": "What are the computational resources used to reproduce the paper?", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper conducts extensive experiments on six prominent large vision-language models. By evaluating the adversarial robustness on these models, the study provides a comprehensive analysis that covers a wide range of architectures, ensuring the results are representative and applicable to the broader landscape of vision-language models.\n\nOne of the key strengths is that the paper demonstrates effective and realistic attacks specifically tailored for these models. By employing transfer-based attacks and utilizing black-box queries, the study goes beyond traditional methods to showcase the vulnerabilities and potential risks faced by large vision-language models in real-world scenarios.", "weaknesses": "This is an evaluation paper. It would be nice to see how baseline defense, such as adversarial training on CLIP [1], would help mitigate the adversarial vulnerability of the proposed attack.\n\nThe method focuses on the vision part, which is easier to attack, what about attacks on the language?\n\n[1] Mao et al. Understanding Zero-Shot Adversarial Robustness for Large-Scale Models. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and suggestions, we have uploaded a rebuttal PDF.\n\n***Q1: It would be nice to see how baseline defense, such as adversarial training on CLIP [1], would help mitigate the adversarial vulnerability of the proposed attack***\n\nThank you for the insightful comments. Following your suggestion, we apply the defense method such as adversarial training on CLIP [1], and evaluate our proposed attacking methods under the defense (we use the model checkpoints provided by [1]). The results are shown in **Table E** of the rebuttal PDF, where we employ the same experimental setups as in Table 2 of the main paper. As can be seen, even with the defense (i.e. adversarial training on CLIP by [1]), our attacking methods can achieve comparable performance to our original results, demonstrating the effectiveness of our attacking methods. These experiment results and discussion will be included in the revision.\n\n***Q2: The method focuses on the vision part, which is easier to attack, what about attacks on the language?***\n\nWhen an adversary attempts to fool a VLM, it can choose to attack either the vision or the language part, or both. We show in our paper that simply attacking the vision part  is sufficient to fool the entire VLM into returning targeted responses. As a result, fooling a VLM by attacking the vision part would be a more effective strategy than attacking the language part. Still, attacking VLMs through languages would be intriguing directions to pursue, and we will include relevant discussion in the revision.\n\n***Q3: What are the computational resources used to reproduce the paper?***\n\nIn Table I of Section A (in the Appendix), we have included detailed computational resources used in each experiment (Figure/Table) of our work, including (1) the type of GPU we used, (2) GPU hours, and (3) carbon emission. On the other hand, we hope to highlight that our proposed method is computationally efficient and can run on a single GPU.\n\n***References:*** \\\n[1] Mao et al. Understanding Zero-Shot Adversarial Robustness for Large-Scale Models. ICLR 2023"}]

Author rebuttal:
We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a rebuttal PDF that includes:

- **Table A**: CLIP score ($\\uparrow$) with fewer PGD steps against different VLMs;
- **Table B**: FlagEmbedding score ($\\uparrow$) as additional text quality metric against different VLMs;
- **Table C**: The average time of API call of each query of  different VLMs;
- **Table D**: CLIP score ($\\uparrow$) with the sole use of MF-tt against  different VLMs;
- **Table E**: CLIP score ($\\uparrow$) with the defense method applied (Mao et al. ICLR 2023).

Meta-review: This paper evaluates the pixel-space adversarial robustness of large vision-language models (VLMs), where the targeted attack has only black-box access to the large VLM systems. The authors have done a nice job of rebuttal. After rebuttal, this paper received scores of 6777. Most reviewers are confident about their judgement. Specifically, the reviewers have commented that (1) the proposed method is simple and the studied problem is valuable for the community; (2) the experiments are compelling and comprehensive; (3) the paper is well-presented with good motivation. Therefore, the AC decided to recommend acceptance of the paper.
Decision: Accept (poster)