SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziyi Yin<sup>1</sup> Muchao Ye<sup>1</sup> Tianrong Zhang<sup>1</sup> Tianyu Du<sup>2</sup>  
Jinguo Zhu<sup>3</sup> Han Liu<sup>4</sup> Jinghui Chen<sup>1</sup> Ting Wang<sup>5</sup> Fenglong Ma<sup>1*</sup>
<sup>1</sup>The Pennsylvania State University, <sup>2</sup>Zhejiang University,
$^{3}$  Xi'an Jiaotong University,  $^{4}$ Dalian University of Technology,  $^{5}$ Stony Brook University
{ziyiyin, muchao, tbz5156, jcz5917, fenglong}@psu.edu
zjradty@zju.edu.cn, lechatelia@stu.xjtu.edu.cn
liu.han.dut@gmail.com, twang@cs.stonybrook.edu
Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK² to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multimodal levels. At the single-modal level, we propose a new blockwise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multimodal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack five widely-used VL pre-trained models for six tasks. Experimental results show that VLATTACK achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a blind spot in the deployment of pre-trained VL models.
The recent success of vision-language (VL) pre-trained models on multimodal tasks have attracted broad attention from both academics and industry [1, 2, 3, 4, 5, 6, 7]. These models first learn multimodal interactions by pre-training on the large-scale unlabeled image-text datasets and are later fine-tuned with labeled pairs on different downstream VL tasks [8, 9, 10]. In many cases, these pre-trained models have revealed more powerful cross-task learning capabilities compared to training from scratch [11, 12]. Despite their remarkable performance, the adversarial robustness of these VL models is still relatively unexplored.
Existing work [13, 14, 15, 16] conducting adversarial attacks in VL tasks is mainly under the white-box setting, where the gradient information of fine-tuned models is accessible to attackers. However, in a more realistic scenario, a malicious attacker may only be able to access the public pre-trained models released through third parties. The attacker would not have any prior knowledge about the
Source code can be found in the link https://github.com/ericyinyzy/VLAttack.
Figure 1: An illustration of the problem of attacking block-box downstream tasks using pre-trained vision-language models.
parameters learned by downstream VL models fine-tuned on private datasets. Towards bridging this striking limitation, we investigate a new yet practical attack paradigm – generating adversarial perturbations on a pre-trained VL model to attack various black-box downstream tasks fine-tuned on the pre-trained one.
However, such an attack setting is non-trivial and faces the following challenges: (1) Task-specific challenge. The pre-trained VL models are usually used for fine-tuning different downstream tasks, which requires the designed attack mechanism to be general and work for attacking multiple tasks. As illustrated in Figure 1, the attacked tasks are not only limited to close-set problems, such as visual question answering, but also generalized to open-ended questions, such as visual grounding. (2) Model-specific challenge. Since the parameters of the fine-tuned models are unknown, it requires the attack method to automatically learn the adversarial transferability between pre-trained and fine-tuned models on different modalities. Although the adversarial transferability [17, 18, 19, 20] across image models has been widely discussed, it is still largely unexplored in the pre-trained models, especially for constructing mutual connections between perturbations on different modalities.
To address all the aforementioned challenges, we propose a new yet general Vision-Language Attack strategy (named VLATTACK) to explore the adversarial transferability between pre-trained and fine-tuned VL models, as shown in Figure 2. The whole VLATTACK scheme fuses perturbations of images and texts from two levels:
Figure 2: A brief illustration of VLATTACK.
Single-modal Level. VLATTACK independently generates perturbations on a single modality, following a "from image to text" order as the former can be perturbed on a continuous space. The single-modal attack can effectively detect the adversarial vulnerability of an image or text, and hence avoids redundant perturbations on the other modality. Specifically, to fully utilize the image-text interactions that have been stored in the pre-trained model, we propose a novel block-wise similarity attack (BSA) strategy to attack the image modality, which adds perturbations to enlarge the network block-wise distance between original and perturbed features in the pretrained model, disrupting the universal image-text representations for the downstream predictions. If BSA fails to change the prediction after querying the fine-tuned black-box model, VLATTACK will attack the text modality by employing the word-level perturbation techniques [21, 22, 23, 24]. We adopt BERT-Attack [21] to attack the text modality as its prominent performance has been widely verified in many studies [25, 26, 27]. Finally, if all the text perturbations  $\{\mathbf{T}_i^{\prime}\}$  fail, VLATTACK will generate a list of perturbed samples  $\mathcal{T}$  and feed them to the multimodal attack along with the perturbed image  $\mathbf{I}^{\prime}$ .
Multimodal Level. If the above attack fails to change the predictions, we cross-update image and text perturbations at the multimodal level based on previous outputs. The proposed iterative cross-search attack (ICSA) strategy updates the image-text perturbation pair  $(\mathbf{I}_i', \mathbf{T}_i')$  in an iterative way
by considering the mutual relations between different modal perturbations. ICSA uses a text perturbation  $\mathbf{T}_i^{\prime}$  selected from the list  $\mathcal{T}$  as the guidance to iteratively update the perturbed image  $\mathbf{I}_i^\prime$  by employing the block-wise similarity attack (BSA) until the new pair  $(\mathbf{I}_i^{\prime},\mathbf{T}_i^{\prime})$  makes the prediction of the downstream task change. In addition, text perturbations are cross-searched according to the semantic similarity with the benign one at the multimodal attack level, which gradually increases the extent of the direction modification to preserve the original semantics to the greatest extent.
Our contributions are summarized as follows: (1) To the best of our knowledge, we are the first to explore the adversarial vulnerability across pre-trained and fine-tuned VL models. (2) We propose VLATTACK to search adversarial samples from different levels. For the single-modal level, we propose the BSA strategy to unify the perturbation optimization targets on various downstream tasks. For the multimodal level, we design the ICSA to generate adversarial image-text pairs by cross-searching perturbations on different modalities. (3) To demonstrate the generalization ability of VLATTACK, we evaluate the proposed VLATTACK on five widely-used VL models, including BLIP [28], CLIP [29], ViLT [1], OFA [5] and UniTAB [4] for six tasks: (i) VQA, (ii) visual entailment, (iii) visual reasoning, (iv) referring expression comprehension, (v) image captioning, and (vi) image classification. Experimental results demonstrate that VLATTACK outperforms both single-modal and multimodal attack approaches, which reveals a significant blind spot in the robustness of large-scale VL models.
Single-modal Adversarial Attack Methods. Image Attack. Traditional image attack methods [30, 31] generate adversarial samples by optimizing the loss function with regard to the decision boundary from model outputs. Not only do the generated perturbations change the model predictions, but they can also be transferred to other convolutional neural network (CNN) structures. Such a property is called transferability and has been extensively studied. For example, data augmentation-based methods [32, 33] endeavor to create diverse input patterns to enhance representation diversity across different models. Feature disruptive attacks [17, 34, 35] introduce the intermediate loss to change the local activation of image features output from the middle layers of CNNs, enabling the perturbed features to transfer to different models without knowing their structures and parameters. Text Attack. Adversarial attacks on natural language processing (NLP) tasks mainly concentrate on word-level and sentence-level perturbations. Word-level perturbations [21, 22, 23, 24] substitute words with synonyms that share similar word embeddings and contextual information. Sentence-level perturbations [25, 36, 37] focus on logical structures of texts through paraphrasing or adding unrelated sentence segments. All of these methods have revealed the adversarial vulnerability of traditional NLP models to some extent. However, adapting them to a multimodal attack setting is still underexplored.
Multimodal Adversarial Attack Methods. Multimodal VL models are susceptible to adversarial attacks as perturbations can be added to both modalities. Existing methods mainly explore adversarial robustness on a specific VL task. For the visual question answering task, Fool-VQA [13] is proposed, which iteratively adds pixel-level perturbations on images to achieve the attack. For the image-text retrieval task, CMLA [14] and AACH [38] add perturbations to enlarge Hamming distance between image and text hash codes, which causes wrong image-text matching results. Recently, Co-attack [15] is proposed, which combines image and text perturbations using word substitution attacks to ascertain a direction for guiding the multi-step attack on images. The whole framework is deployed in a white-box setting, where attackers are assumed to have access to the parameters of each downstream task model. Besides, Co-attack is only validated on three tasks, which is not general enough.
VL Pre-trained Model Structure. Fine-tuning from pre-training has become a unified paradigm in the recent VL model design. Most pre-trained VL models can be divided into encoder-only [1, 39, 40, 41] and encoder-decoder [4, 5, 6] structures. Two representative model structures are shown in Figure 3. Given an image-text pair  $(\mathbf{I}, \mathbf{T})$ , both VL models first embed each modality separately. Image tokens are obtained through an image encoder composed of a vision transformer [42] or a Resnet [43] after flattening the grid features into a sequence. Text tokens are generated through
Figure 3: A brief illustration of the encoder-only (a) and encoder-decoder (b) structures.
Figure 4: Block-wise similarity attack.  $\mathbf{F}_{\alpha}$  is the image encoder, and  $\mathbf{F}_{\beta}$  is the Transformer encoder.
a word encoder made up of a tokenizer [44, 45] and a word vector projection. The encoder-only model then attends tokens of two modalities and a learnable special token  $\langle cls\rangle$  and feeds them into a Transformer encoder. Finally, the output representation from the  $\langle cls\rangle$  token is fed into a classification head for the final prediction  $\langle ans\rangle$ . For the encoder-decoder structure, the attached image and text tokens are fed into a Transformer network [46] to generate sequence predictions  $[\langle ans_1\rangle, \langle ans_2\rangle, \dots, \langle end\rangle]$  in an auto-regressive manner. The network stops regressing when an end token  $\langle end\rangle$  appears. In this work, we deeply explore the adversarial vulnerability of both structures.
Threat Model. Let  $F$  denote the public pre-trained model and  $S$  represent the downstream task model, where  $S$  shares most of or the same structure with  $F$ . As shown in Figure 3, the network structures of both types of public VL pre-trained models are different. The encoder-only model allows modifying the last prediction layer based on the requirement of downstream tasks but keeping the other layers the same as  $F$ . However, the encoder-decoder model unifies the outputs of different downstream tasks, which leads to  $S$  having the same structure as  $F$ . Note that the downstream tasks will fully fine-tune parameters using their own data, and hence, all the model parameters in  $F$  are updated in the fine-tuning stage. Thus, all parameters in  $S$  are not accessible.
Given an image-text pair  $(\mathbf{I},\mathbf{T})$ , the goal of the downstream task is to predict the labels of the input pair accurately, i.e.,  $S:(\mathbf{I},\mathbf{T})\to \mathcal{V}$ , where  $\mathcal{V} = \{y_1,\dots ,y_n\}$ . For the encoder-only models, the ground truth label is a one-hot vector. For the encoder-decoder models, the ground truth is a sequence that consists of multiple ordered one-hot vectors. Let  $\mathbf{y}$  denote the ground truth vector. The goal of the adversarial attack is to generate adversarial examples  $(\mathbf{I}',\mathbf{T}')$  using  $F$ , which can cause an incorrect prediction on  $S$ . Mathematically, our problem is formulated as follows:
$$
\max  _ {\mathbf {I} ^ {\prime}, \mathbf {T} ^ {\prime}} \mathbb {1} \left\{S \left(\mathbf {I} ^ {\prime}, \mathbf {T} ^ {\prime}\right) \neq \mathbf {y} \right\}, \quad s. t. \| \mathbf {I} ^ {\prime} - \mathbf {I} \| _ {\infty} <   \sigma_ {i}, \operatorname {C o s} \left(U _ {s} \left(\mathbf {T} ^ {\prime}\right), U _ {s} (\mathbf {T})\right) > \sigma_ {s}, \tag {1}
$$
where  $\sigma_{i}$  is the  $l_{\infty}$ -norm perturbation strength on the image.  $\sigma_{s}$  is the semantic similarity between the original and perturbed texts, which constrains the semantic consistency after perturbation. The semantic similarity is measured by the cosine similarity  $Cos(\cdot ,\cdot)$  between the sentence embedding  $U_{s}(\mathbf{T}^{\prime})$  and  $U_{s}(\mathbf{T})$  using the Universal Sentence Encoder  $U_{s}$  [47].
As shown in Figure 2, the proposed VLATTACK generates adversarial samples from two steps, constrained by perturbation budget parameters  $\sigma_{i}$  and  $\sigma_{s}$ . The first step attacks every single modality independently to avoid unnecessary modifications. Samples that fail at the first step will be fed into our multimodal level, where we adopt a cross-search attack strategy to iteratively refine both image and text perturbations at the same time. Next, we will introduce the details of VLATTACK from the single-modal level to the multimodal level.
4.1 Single-modal Attacks: From Image to Text
VLATTACK attempts to generate adversarial samples on every single modality in the first step. Compared to the discrete words in  $\mathbf{T}$ , the continuous values-based image  $\mathbf{I}$  is more vulnerable to attack by using gradient information<sup>3</sup>. Thus, VLATTACK starts by attacking the image  $\mathbf{I}$ .
This motivation aligns with our experimental results as shown in Table 1.
Image-Attack. In the black-box setting, the parameters of the fine-tuned model  $S$  are unknown. However, the pre-trained model  $F$  is accessible. Intuitively, if the feature representations learned by the pre-trained model  $F$  from the clean input  $\mathbf{I}$  and the perturbed input  $\mathbf{I}'$  are significantly different, such a perturbation may transfer to fine-tuned models to change the predictions of downstream tasks. As shown in Figure 3, the image features can be obtained from both the image encoder and the Transformer encoder, even from different layers or blocks. To fully leverage the specific characteristics of the pre-trained model structure, we propose the block-wise similarity attack (BSA) to corrupt universal contextualized representations.
As shown in Figure 4, BSA perturbs images by maximizing the block-wise distances between the intermediate representations in the image encoder  $\mathbf{F}_{\alpha}$  and Transformer encoder  $\mathbf{F}_{\beta}$  of the pre-trained model  $F$ . Mathematically, we define the loss function of BSA as follows:
$$
\mathcal {L} = \underbrace {\sum_ {i = 1} ^ {M _ {i}} \sum_ {j = 1} ^ {M _ {j} ^ {i}} C o s \left(\mathbf {F} _ {\alpha} ^ {i , j} (\mathbf {I}) , \mathbf {F} _ {\alpha} ^ {i , j} \left(\mathbf {I} ^ {\prime}\right)\right)} _ {\text {I m a g e E n c o d e r}} + \underbrace {\sum_ {k = 1} ^ {M _ {k}} \sum_ {t = 1} ^ {M _ {t} ^ {k}} C o s \left(\mathbf {F} _ {\beta} ^ {k , t} (\mathbf {I} , \mathbf {T}) , \mathbf {F} _ {\beta} ^ {k , t} \left(\mathbf {I} ^ {\prime} , \mathbf {T}\right)\right)} _ {\text {T r a n s f o r m e r E n c o d e r}}, \tag {2}
$$
where  $M_{i}$  is the number of blocks in the image encoder, and  $M_{j}^{i}$  is the number of flattened image feature embeddings generated in the  $i$ -th block. Similarly,  $M_{k}$  is the number of blocks in the Transformer encoder, and  $M_{t}^{k}$  is the number of image token features generated in the  $k$ -th block.  $\mathbf{F}_{\alpha}^{i,j}$  is the  $j$ -th feature vector obtained in the  $i$ -th layer of the image encoder, and  $\mathbf{F}_{\beta}^{k,t}$  is the  $t$ -th feature vector obtained in the  $k$ -th layer of the Transformer encoder. The image encoder only takes a single image  $\mathbf{I}$  or  $\mathbf{I}'$  as the input, but the Transformer encoder will use both image and text as the input. We adopt the cosine similarity to calculate the distances between perturbed and benign features as token representations attended with each other in the inner product space [46]. Note that BSA does not rely on the information from the decision boundary, and thus, it can be easily adapted to different task settings by disrupting the benign representations.
In the image attack step, we generate an adversarial image candidate through project gradient descent optimization [30] with  $N_{s}$  iterations, where  $N_{s} < N$  and  $N$  is the maximum number of iterations on the image attack. The remaining  $N - N_{s}$  attack step budgets will be used in the multimodal attack in Section 4.2. If  $(\mathbf{I}',\mathbf{T})$  is an adversarial sample generated by BSA, then VLATTACK will stop. Otherwise, VLATTACK moves to attack the text modality.
Text-Attack. In some VL tasks, the number of tokens in the text is quite small. For example, the average length of the text in the VQAv2 [48] and RefCOCO [49] datasets is 6.21 and 3.57, respectively<sup>5</sup>. Moreover, some of them are nonsense words, which makes it unnecessary to design a new approach for attacking the text modality. Furthermore, existing text attack approaches such as BERT-Attack [21] are powerful for generating adversarial samples for texts. Therefore, we directly apply BERT-Attack to generate text perturbations. To avoid unnecessary modifications, we use the clean image  $\mathbf{I}$  as the input instead of the generation perturbed image  $\mathbf{I}'$ .
Specifically, an adversarial candidate  $\mathbf{T}_i^{\prime}$  produced by BERT-Attack is firstly fed into a universal sentence encoder  $U_{s}$  with the benign text  $\mathbf{T}$  to test the semantic similarity.  $\mathbf{T}_i^{\prime}$  will be removed if the cosine similarity  $\gamma_{i}$  between  $\mathbf{T}_i^{\prime}$  and  $\mathbf{T}$  is smaller than the threshold  $\sigma_s$ , i.e.,  $\gamma_{i} = Cos(U_{s}(\mathbf{T}_{i}^{\prime}), U_{s}(\mathbf{T})) < \sigma_{s}$ . Otherwise, we put  $\mathbf{T}_i^{\prime}$  and the benign image  $\mathbf{I}$  into the finetuned model  $S$  to detect whether the new pair  $(\mathbf{I}, \mathbf{T}_i^{\prime})$  perturbs the original prediction  $\mathbf{y}$ . During the text attack, we create a list  $\mathcal{T}$  to store all perturbed samples  $\{\mathbf{T}_i^{\prime}\}$  and the corresponding cosine similarity values  $\{\gamma_i\}$ . If any sample in  $\mathcal{T}$  successfully changes the prediction of the downstream task, then VLATTACK will stop. Otherwise, VLATTACK will carry the perturbed image  $\mathbf{I}^{\prime}$  from the image attack and the perturbed text candidate list  $\mathcal{T}$  from the text attack to the multimodal attack level. Note that BERT-Attack is based on synonym word substitution. Even for a short text, the number of adversarial examples can still be large. In other words, the length of  $\mathcal{T}$  may be large. The whole single-modal attack process is shown in Algorithm 1 (lines 1-15).
4If image encoder is a ResNet module, the output image feature from  $i$ -th block has a shape of  $(H_{i},W_{i},C_{i})$  where  $H_{i}$ ,  $W_{i}$  and  $C_i$  denote the height, width and number of channels, respectively. We then collapse the spatial dimensions of the feature and obtain  $M_j^i = H_iW_i$ .
We obtained these statistics by investigating the validation datasets.
Algorithm 1 VLATTACK
Input: A pre-trained model  $F$ , a fine-tuned model  $S$ , a clean image-text pair  $(\mathbf{I},\mathbf{T})$  and its prediction  $y$  on the  $S$ , and the Gaussian distribution  $\mathcal{U}$ ;  
Parameters: Perturbation budget  $\sigma_{i}$  on  $\mathbf{I}$ ,  $\sigma_{s}$  on  $\mathbf{T}$ . Iteration number  $N$  and  $N_{s}$ .  
1: //Single-modal Attacks: From Image to Text (Section 4.1)  
2: Initialize  $\mathbf{I}' = \mathbf{I} + \delta$ ,  $\delta \in \mathcal{U}(0,1)$ ,  $\mathcal{T} =$   
3: // Image attack by updating  $\mathbf{I}'$  using Eq. (2) for  $N_{s}$  steps  
4:  $\mathbf{I}' = \mathrm{BSA}(\mathcal{L},\mathbf{I}',\mathbf{T},N_s,\sigma_i,F)$   
5: if  $S(\mathbf{I}',\mathbf{T}) \neq y$  then return  $(\mathbf{I}',\mathbf{T})$   
6: else  
7: // Text attack by applying BERT-attack  
8: for perturbed text  $\mathbf{T}_i'$  in BERT-attack do  
9: if  $\gamma_{i} = \mathrm{Cos}(U_{s}(\mathbf{T}_{i}^{\prime}),U_{s}(\mathbf{T})) > \sigma_{s}$  then  
10: Add the pair  $(\mathbf{T}_i',\gamma_i)$  into  $\mathcal{T}$ ;  
11: if  $S(\mathbf{I},\mathbf{T}_i') \neq y$  then return  $(\mathbf{I},\mathbf{T}_i')$   
12: end if  
13: end if  
14: end for  
15: end if  
16: // Multimodal Attack (Section 4.2)  
17: Rank  $\mathcal{T}$  according to similarity scores  $\{\gamma_{i}\}$  and get top- $K$  samples  $\{\hat{\mathbf{T}}_1',\dots ,\hat{\mathbf{T}}_K'\}$  according to Eq. (3);  
18: for  $k = 1,\dots ,K$  do  
19: if  $S(\mathbf{I}_k',\mathbf{T}_k') \neq y$  then return  $(\mathbf{I}_k',\mathbf{T}_k')$   
20: end if  
21: Replace  $(\mathbf{I}_k',\hat{\mathbf{T}}_k')$  with  $(\mathbf{I}',\mathbf{T})$  in Eq. (2);  
22:  $\mathbf{I}_{k+1}' = \mathrm{BSA}(\mathcal{L},\mathbf{I}_k',\hat{\mathbf{T}}_k',N_k,\sigma_i,F)$   
23: if  $S(\mathbf{I}_{k+1}',\mathbf{T}_k') \neq y$  then return  $(\mathbf{I}_{k+1}',\mathbf{T}_k')$   
24: end if  
25: end for  
26: return None
4.2 Multimodal Attack
In many cases, only perturbing images or texts is hard to succeed, as a single-modality perturbation is insufficient to break down the image-text correlation. To solve this problem, we propose a new attack strategy to disrupt dynamic mutual connections based on perturbations from different modalities.
Attack Steps. Since the maximum number of image attack steps  $N$  is predefined, it is impossible to test all the perturbed image-text pairs  $(\mathbf{T}',\mathbf{T}_i')$  using the remaining budget  $N - N_{s}$  if the length of  $\mathcal{T}$  (i.e.,  $|\mathcal{T}|$ ) is very large. Thus, we need to rank the perturbed text samples  $\{\mathbf{T}_i'\}$  according to their corresponding cosine similarity values  $\{\gamma_i\}$  in a descending order to make the adversarial sample keep the high semantic similarity with the original text  $\mathbf{T}$ . Let  $K$  denote the number of attack steps in the multimodal attack, and we have: