SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yunqing Zhao\*1, Tianyu Pang\*2, Chao Du\*2, Xiao Yang3, Chongxuan Li4, Ngai-Man Cheung1, Min Lin2
$^{1}$ Singapore University of Technology and Design
$^{2}$ Sea AI Lab, Singapore
$^{3}$ Tsinghua University  $^{4}$ Renmin University of China
{zhaoyq, tianyupang, duchao, linmin}@sea.com;
yangxiao19@tsinghua.edu.cn; chongxuanli@ruc.edu.cn; ngaiman_cheung@sutd.edu.sg
Large vision-language models (VLMs) such as GPT-4 have achieved unprecedented performance in response generation, especially with visual inputs, enabling more creative and adaptable interaction than large language models such as ChatGPT. Nonetheless, multimodal generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable modality (e.g., vision). To this end, we propose evaluating the robustness of open-source large VLMs in the most realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning the targeted responses. In particular, we first craft targeted adversarial examples against pretrained models such as CLIP and BLIP, and then transfer these adversarial examples to other VLMs such as MiniGPT-4, LLaVA, UniDiffuser, BLIP-2, and Img2Prompt. In addition, we observe that black-box queries on these VLMs can further improve the effectiveness of targeted evasion, resulting in a surprisingly high success rate for generating targeted responses. Our findings provide a quantitative understanding regarding the adversarial vulnerability of large VLMs and call for a more thorough examination of their potential security flaws before deployment in practice. Our project page: yunqing-me.github.io/AttackVLM/.
Large vision-language models (VLMs) have enjoyed tremendous success and demonstrated promising capabilities in text-to-image generation [55, 68, 72], image-grounded text generation (e.g., image captioning or visual question-answering) [2, 15, 42, 86], and joint generation [5, 32, 98] due to an increase in the amount of data, computational resources, and number of model parameters. Notably, after being finetuned with instructions and aligned with human feedback, GPT-4 [58] is capable of conversing with human users and, in particular, supports visual inputs.
Along the trend of multimodal learning, an increasing number of large VLMs are made publicly available, enabling the exponential expansion of downstream applications. However, this poses significant safety challenges. It is widely acknowledged, for instance, that text-to-image models could be exploited to generate fake content [71, 76] or edit images maliciously [73]. A silver lining is that adversaries must manipulate textual inputs to achieve their evasion goals, necessitating extensive search and engineering to determine the adversarial prompts. Moreover, text-to-image models that are
*Equal contribution. Work done during Yunqing Zhao's internship at Sea AI Lab.
†Correspondence to Tianyu Pang, Chao Du, and Ngai-Man Cheung.
Figure 1: Image captioning task implemented by BLIP-2. Given an original text description (e.g., an armchair in the shape of an avocado), DALL-E [67] is used to generate corresponding clean images. BLIP-2 accurately returns captioning text (e.g., a stuffed chair in the shape of an avocado) that analogous to the original text description on the clean image. After the clean image is maliciously perturbed by targeted adversarial noises, the adversarial image can mislead BLIP-2 to return a caption (e.g., a pencil drawing of sports car is shown) that semantically resembles the predefined targeted response (e.g., a hand drawn sketch of a Porsche 911). More examples such as attacking real-world image-text pairs are provided in our Appendix.
accessible to the public typically include a safety checker to filter sensitive concepts and an invisible watermarking module to help identify fake content [69, 72, 108].
Image-grounded text generation such as GPT-4 is more interactive with human users and can produce commands to execute codes [28] or control robots [88], as opposed to text-to-image generation which only returns an image. Accordingly, potential adversaries may be able to evade an image-grounded text generative model by manipulating its visual inputs, as it is well-known that the vision modality is extremely vulnerable to human-imperceptible adversarial perturbations [8, 22, 29, 81]. This raises even more serious safety concerns, as image-grounded text generation may be utilized in considerably complex and safety-critical environments [62]. Adversaries may mislead large VLMs deployed as plugins, for example, to bypass their safety/privacy checkers, inject malicious code, or access APIs and manipulate robots/devices without authorization.
In this work, we empirically evaluate the adversarial robustness of state-of-the-art large VLMs, particularly against those that accept visual inputs (e.g., image-grounded text generation or joint generation). To ensure reproducibility, our evaluations are all based on open-source large models. We examine the most realistic and high-risk scenario, in which adversaries have only black-box system access and seek to deceive the model into returning the targeted responses. Specifically, we first use pretrained CLIP [65, 80] and BLIP [41] as surrogate models to craft targeted adversarial examples, either by matching textual embeddings or image embeddings, and then we transfer the adversarial examples to other large VLMs, including MiniGPT-4 [109], LLaVA [46], UniDiffuser [5], BLIP-2 [42], and Img2Prompt [30]. Surprisingly, these transfer-based attacks can already induce targeted responses with a high success rate. In addition, we discover that query-based attacks employing transfer-based priors can further improve the efficacy of targeted evasion against these VLMs, as shown in Figure 1 (BLIP-2), Figure 2 (UniDiffuser), and Figure 3 (MiniGPT-4).
Our findings provide a quantitative understanding regarding the adversarial vulnerability of large VLMs and advocate for a more comprehensive examination of their potential security defects prior to deployment, as discussed in Sec. 5. Regarding more general multimodal systems, our findings indicate that the robustness of systems is highly dependent on their most vulnerable input modality.
Language models (LMs) and their robustness. The seminal works of BERT [21], GPT-2 [64], and T5 [66] laid the foundations of large LMs, upon which numerous other large LMs have been developed
<sup>1</sup> Note that GPT-4 delays the release of its visual inputs due to safety concerns [3].
Figure 2: Joint generation task implemented by UniDiffuser. There are generative VLMs such as UniDiffuser that model the joint distribution of image-text pairs and are capable of both image-to-text and text-to-image generation. Consequently, given an original text description (e.g., A Van Gogh style painting of an American football player), the text-to-image direction of UniDiffuser is used to generate the corresponding clean image, and its image-to-text direction can recover a text response (e.g., A painting of Packers quarterback football player on a blue background) similar to the original text description. The recovering between image and text modalities can be performed consistently on clean images. When a targeted adversarial perturbation is added to a clean image, however, the image-to-text direction of UniDiffuser will return a text (e.g., A man in an astronaut suit riding a horse on the moon) that semantically resembles the predefined targeted description (e.g., A photo of an astronaut riding a horse on the moon), thereby affecting the subsequent chains of recovering processes.
and demonstrated significant advancements across various language benchmarks [10, 19, 31, 74, 79, 107]. More recently, ChatGPT [57, 59] and several open-source models [18, 83, 95] tuned based on LLaMA [85] enable conversational interaction with human users and can respond to diverse and complex questions. Nevertheless, Alzantot et al. [4] first construct adversarial examples on sentiment analysis and textual entailment tasks, while Jin et al. [36] report that BERT can be evaded through natural language attacks. Later, various flexible (e.g., beyond word replacement) and semantically preserving methods are proposed to produce natural language adversarial examples [9, 49, 50, 52, 53, 70, 78, 102, 104, 110], as well as benchmarks and datasets to more thoroughly evaluate the adversarial robustness of LMs [56, 90-92]. There are also red-teaming initiatives that use human-in-the-loop or automated frameworks to identify problematic language model outputs [27, 63, 96].
Vision-language models (VLMs) and their robustness. The knowledge contained within these powerful LMs is used to facilitate vision-language tasks [26, 33, 84, 93, 101]. Inspired by the adversarial vulnerability observed in vision tasks, early efforts are devoted to investigating adversarial attacks against visual question answering [6, 11, 37, 38, 43, 77, 89, 97, 105] and image caption [1, 14, 99], with the majority of these efforts focusing on conventional CNN-RNN-based models, assuming white-box access or untargeted adversarial goals, and requiring human interaction. Our research, on the other hand, examines the adversarial robustness of advanced large VLMs, assuming black-box access and targeted adversarial goals, and providing quantitative evaluations free of human labor.
In this section, we will first introduce the fundamental preliminary, and then describe the transfer-based and query-based attacking strategies against image-grounded text generation, respectively.
We denote  $p_{\theta}(\boldsymbol{x}; \boldsymbol{c}_{\mathrm{in}}) \mapsto \boldsymbol{c}_{\mathrm{out}}$  as an image-grounded text generative model parameterized by  $\theta$ , where  $\boldsymbol{x}$  is the input image,  $\boldsymbol{c}_{\mathrm{in}}$  is the input text, and  $\boldsymbol{c}_{\mathrm{out}}$  is the output text. In image captioning tasks, for instance,  $\boldsymbol{c}_{\mathrm{in}}$  is a placeholder  $\emptyset$  and  $\boldsymbol{c}_{\mathrm{out}}$  is the caption; in visual question answering tasks,  $\boldsymbol{c}_{\mathrm{in}}$  is the question and  $\boldsymbol{c}_{\mathrm{out}}$  is the answer. Note that here we slightly abuse the notations since the mapping between  $p_{\theta}(\boldsymbol{x}; \boldsymbol{c}_{\mathrm{in}})$  and  $\boldsymbol{c}_{\mathrm{out}}$  could be probabilistic or non-deterministic [5, 98].
Threat models. We overview threat models that specify adversarial conditions [12] and adapt them to generative paradigms: (i) adversary knowledge describes what knowledge the adversary is assumed
Figure 3: Visual question-answering (VQA) task implemented by MiniGPT-4. MiniGPT-4 has capabilities for vision-language understanding and performs comparably to GPT-4 on tasks such as multi-round VQA by leveraging the knowledge of large LMs. We select images with refined details generated by Midjourney [51] and feed questions (e.g., Can you tell me what is the interesting point of this image?) into MiniGPT-4. As expected, MiniGPT-4 can return descriptions that are intuitively reasonable, and when we ask additional questions (e.g., But is this a common scene in the normal life?), MiniGPT-4 demonstrates the capacity for accurate multi-round conversation. Nevertheless, after being fed targeted adversarial images, MiniGPT-4 will return answers related to the targeted description (e.g., A robot is playing in the field). This adversarial effect can even affect multi-round conversations when we ask additional questions. More examples of attacking MiniGPT-4 or LLaVA on VQA are provided in our Appendix.
to have, typically either white-box access with full knowledge of  $p_{\theta}$  including model architecture and weights, or varying degrees of black-box access, e.g., only able to obtain the output text  $c_{\mathrm{out}}$  from an API; (ii) adversary goals describe the malicious purpose that the adversary seeks to achieve, including untargeted goals that simply cause  $c_{\mathrm{out}}$  to be a wrong caption or answer, and targeted goals that cause  $c_{\mathrm{out}}$  to match a predefined targeted response  $c_{\mathrm{tar}}$  (measured via text-matching metrics); (iii) adversary capabilities describe the constraints on what the adversary can manipulate to cause harm, with the most commonly used constraint being imposed by the  $\ell_p$  budget, namely, the  $\ell_p$  distance between the clean image  $x_{\mathrm{cle}}$  and the adversarial image  $x_{\mathrm{adv}}$  is less than a budget  $\epsilon$  as  $\| x_{\mathrm{cle}} - x_{\mathrm{adv}} \|_p \leq \epsilon$ .
Remark. Our work investigates the most realistic and challenging threat model, where the adversary has black-box access to the victim models  $p_{\theta}$ , a targeted goal, a small perturbation budget  $\epsilon$  on the input image  $x$  to ensure human imperceptibility, and is forbidden to manipulate the input text  $c_{\mathrm{in}}$ .
Figure 4: Pipelines of our attacking strategies. In the upper-left panel, we illustrate our transfer-based strategy for matching image-image features (MF-ii) as formulated in Eq. (2). We select a targeted text  $c_{\mathrm{tar}}$  (e.g., A sea otter with a pearl earring) and then use a pretrained text-to-image generator  $h_\xi$  to produce a targeted image  $h_\xi(c_{\mathrm{tar}})$ . The targeted image is then fed to the image encoder  $f_\phi$  to obtain the embedding  $f_\phi(h_\xi(c_{\mathrm{tar}}))$ . Here we refer to adversarial examples generated by transfer-based strategies as  $x_{\mathrm{trans}} = x_{\mathrm{cle}} + \Delta$ , while adversarial noise is denoted by  $\Delta$ . We feed  $x_{\mathrm{trans}}$  into the image encoder to obtain the adversarial embedding  $f_\phi(x_{\mathrm{trans}})$ , and then we optimize the adversarial noise  $\Delta$  to maximize the similarity metric  $f_\phi(x_{\mathrm{trans}})^{\top}f_\phi(h_\xi(c_{\mathrm{tar}}))$ . In the upper-right panel, we demonstrate our query-based strategy for matching text-text features (MF-tt), as defined by Eq. (3). We apply the resulted transfer-based adversarial example  $x_{\mathrm{trans}}$  to initialize  $x_{\mathrm{adv}}$ , then sample  $N$  random perturbations and add them to  $x_{\mathrm{adv}}$  to build  $\{x_{\mathrm{adv}} + \delta_n\}_{n=1}^N$ . These randomly perturbed adversarial examples are fed into the victim model  $p_\theta$  (with the input text  $c_{\mathrm{in}}$  unchanged) and the RGF method described in Eq. (4) is used to estimate the gradients  $\nabla_{x_{\mathrm{adv}}}g_\psi(p_\theta(x_{\mathrm{adv}};c_{\mathrm{in}}))^{\top}g_\psi(c_{\mathrm{tar}})$ . In the bottom, we present the final results of our method's (MF-ii + MF-tt) targeted response generation.
3.2 Transfer-based attacking strategy
Since we assume black-box access to the victim models, a common attacking strategy is transfer-based [22, 23, 47, 61, 94, 100], which relies on surrogate models (e.g., a publicly accessible CLIP model) to which the adversary has white-box access and crafts adversarial examples against them, then feeds the adversarial examples into the victim models (e.g., GPT-4 that the adversary seeks to fool). Due to the fact that the victim models are vision-and-language, we select an image encoder  $f_{\phi}(\boldsymbol{x})$  and a text encoder  $g_{\psi}(c)$  as surrogate models, and we denote  $c_{\mathrm{tar}}$  as the targeted response that the adversary expects the victim models to return. Two approaches of designing transfer-based adversarial objectives are described in the following.
Matching image-text features (MF-it). Since the adversary expects the victim models to return the targeted response  $c_{\mathrm{tar}}$  when the adversarial image  $x_{\mathrm{adv}}$  is the input, it is natural to match the features of  $c_{\mathrm{tar}}$  and  $x_{\mathrm{adv}}$  on surrogate models, where  $x_{\mathrm{adv}}$  should satisfy<sup>2</sup>
$$
\underset {\| \boldsymbol {x} _ {\mathrm {c l e}} - \boldsymbol {x} _ {\mathrm {a d v}} \| _ {p} \leq \epsilon} {\arg \max } f _ {\phi} (\boldsymbol {x} _ {\mathrm {a d v}}) ^ {\top} g _ {\psi} (\boldsymbol {c} _ {\mathrm {t a r}}). \tag {1}
$$
Here, we use blue color to highlight white-box accessibility (i.e., can directly obtain gradients of  $f_{\phi}$  and  $g_{\psi}$  through backpropagation), the image and text encoders are chosen to have the same output dimension, and their inner product indicates the cross-modality similarity of  $c_{\mathrm{tar}}$  and  $x_{\mathrm{adv}}$ . The constrained optimization problem in Eq. (1) can be solved by projected gradient descent (PGD) [48].
Matching image-image features (MF-ii). While aligned image and text encoders have been shown to perform well on vision-language tasks [65], recent research suggests that VLMs may behave like bags-of-words [103] and therefore may not be dependable for optimizing cross-modality similarity. Given this, an alternative approach is to use a public text-to-image generative model  $h_{\xi}$  (e.g., Stable
2We slightly abuse the notations by using  $x_{\mathrm{adv}}$  to represent both the variable and the optimal solution.
Table 1: White-box attacks against surrogate models. We craft adversarial images  $\mathbf{x}_{\mathrm{adv}}$  using MF-it in Eq. (1) or MF-ii in Eq. (2), and report the CLIP score  $(\uparrow)$  between the images and the predefined targeted text  $c_{\mathrm{tar}}$  (randomly chosen sentences). Here the clean images consist of real-world  $\mathbf{x}_{\mathrm{cle}}$  that is irrelevant to the chosen targeted text and  $h_\xi(c_{\mathrm{tar}})$  generated by a text-to-image model (e.g., Stable Diffusion [72]) conditioned on the targeted text  $c_{\mathrm{tar}}$ . We observe that MF-ii induces a similar CLIP score compared to the generated image  $h_\xi(c_{\mathrm{tar}})$ , while MF-it induces a even higher CLIP score by directly matching cross-modality features. Furthermore, we note that the attack is time-efficient, and we provide the average time (in seconds) for each strategy to craft a single  $\mathbf{x}_{\mathrm{adv}}$ . The results in this table validate the effectiveness of white-box attacks against surrogate models, whereas Table 2 investigates the transferability of crafted  $\mathbf{x}_{\mathrm{adv}}$  to evade large VLMs (e.g., MiniGPT-4).
Model	Clean image	Adversarial image	Time to obtain a single xadv
xcle	hξ(ctar)	MF-ii	MF-it	MF-ii	MF-it
CLIP (RN50) [65]	0.094	0.261	0.239	0.576	0.543	0.532
CLIP (ViT-B/32) [65]	0.142	0.313	0.302	0.570	0.592	0.588
BLIP (ViT) [41]	0.138	0.286	0.277	0.679	0.641	0.634
BLIP-2 (ViT) [42]	0.037	0.302	0.294	0.502	0.855	0.852
ALBEF (ViT) [40]	0.063	0.098	0.091	0.451	0.750	0.749
Diffusion [72]) and generate a targeted image corresponding to  $c_{\mathrm{tar}}$  as  $h_\xi(c_{\mathrm{tar}})$ . Then, we match the image-image features of  $x_{\mathrm{adv}}$  and  $h_\xi(c_{\mathrm{tar}})$  as
$$
\arg \max  f _ {\phi} \left(\boldsymbol {x} _ {\text {a d v}}\right) ^ {\top} f _ {\phi} \left(h _ {\xi} \left(\boldsymbol {c} _ {\text {t a r}}\right)\right), \tag {2}
$$
$$
\left\| \boldsymbol {x} _ {\mathrm {c l e}} - \boldsymbol {x} _ {\mathrm {a d v}} \right\| _ {p} \leq \epsilon
$$
where orange color is used to emphasize that only black-box accessibility is required for  $h_{\xi}$ , as gradient information of  $h_{\xi}$  is not required when optimizing the adversarial image  $\pmb{x}_{\mathrm{adv}}$ . Consequently, we can also implement  $h_{\xi}$  using advanced APIs such as Midjourney [51].
3.3 Query-based attacking strategy
Transfer-based attacks are effective, but their efficacy is heavily dependent on the similarity between the victim and surrogate models. When we are allowed to repeatedly query victim models, such as by providing image inputs and obtaining text outputs, the adversary can employ a query-based attacking strategy to estimate gradients or execute natural evolution algorithms [7, 16, 34].
Matching text-text features (MF- tt). Recall that the adversary goal is to cause the victim models to return a targeted response, namely, matching  $p_{\theta}(\boldsymbol{x}_{\mathrm{adv}};\boldsymbol{c}_{\mathrm{in}})$  with  $\boldsymbol{c}_{\mathrm{tar}}$ . Thus, it is straightforward to maximize the textual similarity between  $p_{\theta}(\boldsymbol{x}_{\mathrm{adv}};\boldsymbol{c}_{\mathrm{in}})$  and  $\boldsymbol{c}_{\mathrm{tar}}$  as
$$
\arg \max  g _ {\psi} \left(p _ {\theta} \left(\boldsymbol {x} _ {\text {a d v}}; \boldsymbol {c} _ {\text {i n}}\right)\right) ^ {\top} g _ {\psi} \left(\boldsymbol {c} _ {\text {t a r}}\right). \tag {3}
$$
Note that we cannot directly compute gradients for optimization in Eq. (3) because we assume black-box access to the victim models  $p_{\theta}$  and cannot perform backpropagation. To estimate the gradients, we employ the random gradient-free (RGF) method [54]. First, we rewrite a gradient as the expectation of direction derivatives, i.e.,  $\nabla_{\boldsymbol{x}}F(\boldsymbol{x}) = \mathbb{E}\left[\delta^{\top}\nabla_{\boldsymbol{x}}F(\boldsymbol{x})\cdot \delta \right]$ , where  $F(\boldsymbol{x})$  represents any differentiable function and  $\delta \sim P(\delta)$  is a random variable satisfying that  $\mathbb{E}[\delta \delta^{\top}] = \mathbf{I}$  (e.g.,  $\delta$  can be uniformly sampled from a hypersphere). Then by zero-order optimization [16], we know that
$$
\begin{array}{l} \nabla_ {\boldsymbol {x} _ {\mathrm {a d v}}} g \psi \left(p _ {\theta} \left(\boldsymbol {x} _ {\mathrm {a d v}}; \boldsymbol {c} _ {\mathrm {i n}}\right)\right) ^ {\top} g \psi \left(\boldsymbol {c} _ {\mathrm {t a r}}\right) \\ \approx \frac {1}{N \sigma} \sum_ {n = 1} ^ {N} \left[ g _ {\psi} \left(p _ {\theta} \left(\boldsymbol {x} _ {\text {a d v}} + \sigma \boldsymbol {\delta} _ {n}; \boldsymbol {c} _ {\text {i n}}\right)\right) ^ {\top} g _ {\psi} \left(\boldsymbol {c} _ {\text {t a r}}\right) - g _ {\psi} \left(p _ {\theta} \left(\boldsymbol {x} _ {\text {a d v}}; \boldsymbol {c} _ {\text {i n}}\right)\right) ^ {\top} g _ {\psi} \left(\boldsymbol {c} _ {\text {t a r}}\right) \right] \cdot \boldsymbol {\delta} _ {n}, \tag {4} \\ \end{array}
$$
where  $\delta_n \sim P(\pmb{\delta})$ ,  $\sigma$  is a hyperparameter controls the sampling variance, and  $N$  is the number of queries. The approximation in Eq. (4) becomes an unbiased equation when  $\sigma \to 0$  and  $N \to \infty$ .
Remark. Previous research demonstrates that transfer-based and query-based attacking strategies can work in tandem to improve black-box evasion effectiveness [17, 24]. In light of this, we also consider