SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Black-box attacks against victim models. We sample clean images  $\mathbf{x}_{\mathrm{cle}}$  from the ImageNet-1K validation set and randomly select a target text  $c_{\mathrm{tar}}$  from MS-COCO captions for each clean image. We report the CLIP score  $(\dagger)$  between the generated responses of input images (i.e., clean images  $\mathbf{x}_{\mathrm{cle}}$  or  $\mathbf{x}_{\mathrm{adv}}$  crafted by our attacking methods MF-it, MF-ii, and the combination of MF-ii + MF-tt) and predefined targeted texts  $c_{\mathrm{tar}}$ , as computed by various CLIP text encoders and their ensemble/average. The default textual input  $c_{\mathrm{in}}$  is fixed to be "what is the content of this image?". Pretrained image/text encoders such as CLIP are used as surrogate models for MF-it and MF-ii. For reference, we also report other information such as the number of parameters and input resolution of victim models.
VLM model	Attacking method	Text encoder (pretrained) for evaluation	Other info. # Param. Res.
RN50	RN101	ViT-B/16	ViT-B/32	ViT-L/14	Ensemble
BLIP [41]	Clean image	0.472	0.456	0.479	0.499	0.344	0.450	224M 384
MF-it	0.492	0.474	0.520	0.546	0.384	0.483
MF-ii	0.766	0.753	0.774	0.786	0.696	0.755
MF-ii + MF-tt	0.855	0.841	0.861	0.868	0.803	0.846
UniDiffuser [5]	Clean image	0.417	0.415	0.429	0.446	0.305	0.402	1.4B 224
MF-it	0.655	0.639	0.678	0.698	0.611	0.656
MF-ii	0.709	0.695	0.721	0.733	0.637	0.700
MF-ii + MF-tt	0.754	0.736	0.761	0.777	0.689	0.743
Img2Prompt [30]	Clean image	0.487	0.464	0.493	0.515	0.350	0.461	1.7B 384
MF-it	0.499	0.472	0.501	0.525	0.355	0.470
MF-ii	0.502	0.479	0.505	0.529	0.366	0.476
MF-ii + MF-tt	0.803	0.783	0.809	0.828	0.733	0.791
BLIP-2 [42]	Clean image	0.473	0.454	0.483	0.503	0.349	0.452	3.7B 224
MF-it	0.492	0.474	0.520	0.546	0.384	0.483
MF-ii	0.562	0.541	0.571	0.592	0.449	0.543
MF-ii + MF-tt	0.656	0.633	0.665	0.681	0.555	0.638
LLaVA [46]	Clean image	0.383	0.436	0.402	0.437	0.281	0.388	13.3B 224
MF-it	0.389	0.441	0.417	0.452	0.288	0.397
MF-ii	0.396	0.440	0.421	0.450	0.292	0.400
MF-ii + MF-tt	0.548	0.559	0.563	0.590	0.448	0.542
MiniGPT-4 [109]	Clean image	0.422	0.431	0.436	0.470	0.326	0.417	14.1B 224
MF-it	0.472	0.450	0.461	0.484	0.349	0.443
MF-ii	0.525	0.541	0.542	0.572	0.430	0.522
MF-ii + MF-tt	0.633	0.611	0.631	0.668	0.528	0.614
the adversarial examples generated by transfer-based methods to be an initialization (or prior-guided) and use the information obtained from query-based methods to strengthen the adversarial effects. This combination is effective, as empirically verified in Sec. 4 and intuitively illustrated in Figure 4.
In this section, we demonstrate the effectiveness of our techniques for crafting adversarial examples against open-source, large VLMs. More results are provided in the Appendix.
In this paper, we evaluate open-source (to ensure reproducibility) and advanced large VLMs, such as UniDiffuser [5], which uses a diffusion-based framework to jointly model the distribution of image-text pairs and can perform both image-to-text and text-to-image generation; BLIP [41] is a unified vision-language pretraining framework for learning from noisy image-text pairs; BLIP-2 [42] adds a querying transformer [87] and a large LM (T5 [66]) to improve the image-grounded text generation; Img2Prompt [30] proposes a plug-and-play, LM-agnostic module that provides large
Targeted image  $h_{\xi}(c_{\mathrm{tar}})$
Figure 5: Adversarial perturbations  $\Delta$  are obtained by computing  $x_{\mathrm{adv}} - x_{\mathrm{cle}}$  (pixel values are amplified  $\times 10$  for visualization) and their corresponding captions are generated below. Here DALL-E acts as  $h_\xi$  to generate targeted images  $h_\xi(c_{\mathrm{tar}})$  for reference. We note that adversarial perturbations are not only visually hard to perceive, but also not detectable using state-of-the-art image captioning models (we use UniDiffuser for captioning, while similar conclusions hold when using other models).
Adversarial perturbation  $(\Delta)$
Figure 6: We experiment with different values of  $\epsilon$  in Eq. (3) to obtain different levels of  $x_{\mathrm{adv}}$ . As seen, the quality of  $x_{\mathrm{adv}}$  degrades (measured by the LPIPS distance between  $x_{\mathrm{cle}}$  and  $x_{\mathrm{adv}}$ ), while the effect of targeted response generation saturates (in this case, we evaluate UniDiffuser). Thus, a proper perturbation budget (e.g.,  $\epsilon = 8$ ) is necessary to balance image quality and generation performance.
LM prompts to enable zero-shot VQA tasks; MiniGPT-4 [109] and LLaVA [46] have recently scaled up the capacity of large LMs and leveraged Vicuna-13B [18] for image-grounded text generation tasks. We note that MiniGPT-4 also exploits a high-quality, well-aligned dataset to further finetune the model with a conversation template, resulting in performance comparable to GPT-4 [58].
Datasets. We use the validation images from ImageNet-1K [20] as clean images, from which adversarial examples are crafted, to quantitatively evaluate the adversarial robustness of large VLMs. From MS-COCO captions [44], we randomly select a text description (usually a complete sentence, as shown in our Appendix) as the adversarially targeted text for each clean image. Because we cannot easily find a corresponding image of a given, predefined text, we use Stable Diffusion [72] for the text-to-image generation to obtain the targeted images of each text description, in order to simulate the real-world scenario. Midjourney [51] and DALL-E [67, 68] are also used in our experiments to generate the targeted images for demonstration.
Basic setups. For fair comparison, we strictly adhere to previous works [5, 30, 41, 42, 46, 109] in the selection of pretrained weights for image-grounded text generation, including large LMs (e.g., T5 [66] and Vicuna-13B [18] checkpoints). We experiment on the original clean images of various resolutions (see Table 2). We set  $\epsilon = 8$  and use  $\ell_{\infty}$  constraint by default as  $\| \pmb{x}_{\mathrm{cle}} - \pmb{x}_{\mathrm{adv}}\|_{\infty}\leq 8$ , which is the most commonly used setting in the adversarial literature [12], to ensure that the adversarial perturbations are visually imperceptible where the pixel values are in the range [0, 255]. We use 100-step PGD to optimize transfer-based attacks (the objectives in Eq. (1) and Eq. (2)). In each step of query-based attacks, we set query times  $N = 100$  in Eq. (4) and update the adversarial images by 8-steps PGD using the estimated gradient. Every experiment is run on a single NVIDIA-A100 GPU.
4.2 Empirical studies
We evaluate large VLMs and freeze their parameters to make them act like image-to-text generative APIs. In particular, in Figure 1, we show that our crafted adversarial image consistently deceives BLIP-2 and that the generated response has the same semantics as the targeted text. In Figure 2, we
Figure 7: Performance of our attack method under a fixed perturbation budget  $\epsilon = 8$ . We interpolate between the sole use of transfer-based attack and the sole use of query-based attack strategy. We demonstrate the effectiveness of our method via CLIP score ( $\uparrow$ ) between the generated texts on adversarial images and the target texts, with different types of CLIP text encoders. The  $x$ -axis in a "t $\epsilon_t$ -q $\epsilon_q$ " format denotes we assign  $\epsilon_t$  to transfer-based attack and  $\epsilon_q$  to query-based attack. "t+q=8" indicates we use transfer-based attack ( $\epsilon_t = 8$ ) as initialization, and conduct query-based attack for further 8 steps ( $\epsilon_q = 8$ ), such that the resulting perturbation satisfies  $\epsilon = 8$ . As a result, we show that a proper combination of transfer/query based attack strategy achieves the best performance.
evaluate UniDiffuser, which is capable of bidirectional joint generation, to generate text-to-image and then image-to-text using the crafted  $\boldsymbol{x}_{\mathrm{adv}}$ . It should be noted that such a chain of generation will result in completely different content than the original text description. We simply use "what is the content of this image?" as the prompt to answer generation for models that require text instructions as input (query) [30]. However, for MiniGPT-4, we use a more flexible approach in conversation, as shown in Figure 3. In contrast to the clean images on which MiniGPT-4 has concrete and correct understanding and descriptions, our crafted adversarial counterparts mislead MiniGPT-4 into producing targeted responses and creating more unexpected descriptions that are not shown in the targeted text.
In Table 1, we examine the effectiveness of MF-it and MF-ii in crafting white-box adversarial images against surrogate models such as CLIP [64], BLIP [41] and ALBEF [40]. We take 50K clean images  $\pmb{x}_{\mathrm{cle}}$  from the ImageNet-1K validation set and randomly select a targeted text  $\pmb{c}_{\mathrm{tar}}$  from MS-COCO captions for each clean image. We also generate targeted images  $h_{\xi}(c_{\mathrm{tar}})$  as reference and craft adversarial images  $\pmb{x}_{\mathrm{adv}}$  by MF-ii or MF-it. As observed, both MF-ii and MF-it are able to increase the similarity between the adversarial image and the targeted text (as measured by CLIP score) in the white-box setting, laying the foundation for black-box transferability. Specifically, as seen in Table 2, we first transfer the adversarial examples crafted by MF-ii or MF-it in order to evade large VLMs and mislead them into generating targeted responses. We calculate the similarity between the generated response  $p_{\theta}(\pmb{x}_{\mathrm{adv}}; c_{\mathrm{in}})$  and the targeted text  $c_{\mathrm{tar}}$  using various types of CLIP text encoders. As mentioned previously, the default textual input  $c_{\mathrm{in}}$  is fixed to be "what is the content of this image?". Surprisingly, we find that MF-it performs worse than MF-ii, which suggests overfitting when optimizing directly on the cross-modality similarity. In addition, when we use the transfer-based adversarial image crafted by MF-ii as an initialization and then apply query-based MF-tt to tune the adversarial image, the generated response becomes significantly more similar to the targeted text, indicating the vulnerability of advanced large VLMs.
4.3 Further analyses
Does VLM adversarial perturbations induce semantic meanings? Previous research has demonstrated that adversarial perturbations crafted against robust models will exhibit semantic or perceptually-aligned characteristics [35, 60, 82]. This motivates us to figure out whether adversarial perturbations  $\Delta = x_{\mathrm{adv}} - x_{\mathrm{cle}}$  crafted against large VLMs possess a similar level of semantic information. In Figure 5, we visualize  $\Delta$  that results in a successful targeted evasion over a real image and report the generated text responses. Nevertheless, we observe no semantic information associated with the targeted text in adversarial perturbations or their captions, indicating that large VLMs are inherently vulnerable.
The influence of perturbation budget  $\epsilon$ . We use  $\epsilon = 8$  as the default value in our experiments, meaning that the pixel-wise perturbation is up to  $\pm 8$  in the range [0, 255]. In Figure 6, we examine the effect of setting  $\epsilon$  to different values of  $\{2,4,8,16,64\}$  and compute the perceptual distance between the clean image  $x_{\mathrm{cle}}$  and its adversarial counterpart  $x_{\mathrm{adv}}$  using LPIPS  $(\downarrow)$  [106]. We highlight (in red color) the generated responses that most closely resemble the targeted text. As observed, there is a trade-off between image quality/fidelity and successfully eliciting the targeted response; therefore, it is essential to choose an appropriate perturbation budget value.
(a)
A photo of a teddy bear sitting in the middle of Times Square.
Figure 8: Visually interpreting our attacking mechanism. To better comprehend the mechanism by which our adversarial examples deceive large VLMs (here we evaluate Img2Prompt), we employ interpretable visualization with GradCAM [75]. (a) An example of  $x_{\mathrm{cle}}$ ,  $x_{\mathrm{adv}}$ , and  $h_\xi(c_{\mathrm{tar}})$ , along with the responses they generate. We select the targeted text as a beautiful bird with a black and white color in snow. (b) GradCAM visualization when the input question is: what is the teddy bear playing in the middle of the road? As seen, GradCAM can effectively highlight the skateboard for  $x_{\mathrm{cle}}$ , whereas GradCAM highlights irrelevant backgrounds for  $x_{\mathrm{adv}}$ . (c) If we feed the targeted text as the question, GradCAM will highlight similar regions of  $x_{\mathrm{adv}}$  and  $h_\xi(c_{\mathrm{tar}})$ .
"A small bird sitting in the snow, a tree in the snow.
"A beautiful bird in the sky"  
white color in snow.
(b)
What is the teddy car?  
What is the middle of the road?
What is the teddy car?  
What is the middle of the road?
(c)
A beautiful bird is running on the white color in snow
"A beautiful bird is in the snow. The white color in snow.
Performance of attack with a fixed perturbation budget. To understand the separate benefit from transfer-based attack and query-based attack, we conduct a study to assign different perturbation budget for transfer  $(\epsilon_{t})$  and query based attack strategy  $(\epsilon_{q})$ , under the constraint  $\epsilon_{t} + \epsilon_{q} = 8$ . Unidiffuser is the victim model in our experiment. The results are in Figure 7. We demonstrate that, a proper combination of transfer and query based attack achieves the best performance.
Interpreting the mechanism of attacking large VLMs. To understand how our targeted adversarial example influences response generation, we compute the relevancy score of image patches related to the input question using GradCAM [75] to obtain a visual explanation for both clean and adversarial images. As shown in Figure 8, our adversarial image  $x_{\mathrm{adv}}$  successfully suppresses the relevancy to the original text description (panel (b)) and mimics the attention map of the targeted image  $h_{\xi}(c_{\mathrm{tar}})$  (panel (c)). Nonetheless, we emphasize that the use of GradCAM as a feature attribution method has some known limitations [13]. Additional interpretable examples are provided in the Appendix.
It is widely accepted that developing large multimodal models will be an irresistible trend. Prior to deploying these large models in practice, however, it is essential to understand their worst-case performance through techniques such as red teaming or adversarial attacks [25]. In contrast to manipulating textual inputs, which may require human-in-the-loop prompt engineering, our results demonstrate that manipulating visual inputs can be automated, thereby effectively fooling the entire large vision-language systems. The resulting adversarial effect is deeply rooted and can even affect multi-round interaction, as shown in Figure 3. While multimodal security issues have been cautiously treated by models such as GPT-4, which delays the release of visual inputs [3], there are an increasing number of open-source multimodal models, such as MiniGPT-4 [109] and LLaVA [46, 45], whose worst-case behaviors have not been thoroughly examined. The use of these open-source, but adversarily unchecked, large multimodal models as product plugins could pose potential risks.
Broader impacts. While the primary goal of our research is to evaluate and quantify adversarial robustness of large vision-language models, it is possible that the developed attacking strategies could be misused to evade practically deployed systems and cause potential negative societal impacts. Specifically, our threat model assumes black-box access and targeted responses, which involves manipulating existing APIs such as GPT-4 (with visual inputs) and/or Midjourney on purpose, thereby increasing the risk if these vision-language APIs are implemented as plugins in other products.
Limitations. Our work focuses primarily on the digital world, with the assumption that input images feed directly into the models. In the future, however, vision-language models are more likely to be deployed in complex scenarios such as controlling robots or automatic driving, in which case input images may be obtained from the interaction with physical environments and captured in real-time by cameras. Consequently, performing adversarial attacks in the physical world would be one of the future directions for evaluating the security of vision-language models.
This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021). This material is based on the research/work support in part by the Changi General Hospital and Singapore University of Technology and Design, under the HealthTech Innovation Fund (HTIF Award No. CGH-SUTD-2021-004). C. Li was sponsored by Beijing Nova Program (No. 20220484044). We thank Siqi Fu for providing beautiful pictures generated by Midjourney, and anonymous reviewers for their insightful comments.
[1] Nayyer Aafaq, Naveed Akhtar, Wei Liu, Mubarak Shah, and Ajmal Mian. Controlled caption generation for images through adversarial attacks. arXiv preprint arXiv:2107.03050, 2021.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[3] Sam Altman, 2023. https://twitter.com/sama/status/1635687855921172480.
[4] Moustafa Alzantot, Yash Sharma, Ahmed Elghohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[5] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference on Machine Learning (ICML), 2022.
[6] Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. Improving question answering model robustness with synthetic adversarial data generation. arXiv preprint arXiv:2104.08678, 2021.
[7] Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In European Conference on Computer Vision (ECCV), 2018.
[8] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 387-402. Springer, 2013.
[9] Hezekiah J Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv preprint arXiv:2209.02128, 2022.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[11] Yu Cao, Dianqi Li, Meng Fang, Tianyi Zhou, Jun Gao, Yibing Zhan, and Dacheng Tao. Tasa: Deceiving question answering models by twin answer sentences attack. arXiv preprint arXiv:2210.15221, 2022.
[12] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019.
[13] Aditya Chattopadhy, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839-847. IEEE, 2018.
[97] Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, and Dawn Song. Fooling vision and language models despite localization and attention mechanism. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[98] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332, 2022.
[99] Yan Xu, Baoyuan Wu, Fumin Shen, Yanbo Fan, Yong Zhang, Heng Tao Shen, and Wei Liu. Exact adversarial attack to image captioning via structured output learning with latent variables. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[100] Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. In European Conference on Computer Vision (ECCV), 2022.
[101] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023.
[102] Liping Yuan, Xiaqing Zheng, Yi Zhou, Cho-Jui Hsieh, and Kai-Wei Chang. On the transferability of adversarial attacks against neural text classifier. arXiv preprint arXiv:2011.08558, 2020.
[103] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations (ICLR), 2023.
[104] Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. Generating fluent adversarial examples for natural languages. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[105] Jiaming Zhang, Qi Yi, and JITao Sang. Towards adversarial attack on vision-language pretraining models. In ACM International Conference on Multimedia, 2022.
[106] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[107] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[108] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusion models. arXiv preprint arXiv:2303.10137, 2023.
[109] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
[110] Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari, and Fatemeh Shiri. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12868, 2023.
In this appendix, we describe implementation details, additional experiment results and analyses, to support the methods proposed in the main paper. We also discuss failure cases in order to better understand the capability of our attack methods.
In Section 4.1 of the main paper, we introduce large VLMs, datasets, and other basic setups used in our experiments and analyses. Here, we discuss more on the design choices and implementation details to help understanding our attacking strategies and reproducing our empirical results.