SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
K = \left\{ \begin{array}{l l} N - N _ {s}, & \text {i f} | \mathcal {T} | > N - N _ {s}; \\ | \mathcal {T} |, & \text {i f} | \mathcal {T} | \leqslant N - N _ {s}. \end{array} \right. \tag {3}
$$
Iterative Cross-Search Attack. A naive way to conduct the multimodal attack is to test each perturbed image-text pair  $(\mathbf{I}', \hat{\mathbf{T}}_k')$  ( $k = 1, \dots, K$ ) by querying the black-box fine-tuned model  $S$ , where  $\hat{\mathbf{T}}_i'$  is the  $i$ -th text perturbation in the ranked list  $\mathcal{T}$ . However, this simple approach ignores learning mutual connections between the perturbed text and image. To solve this issue, we propose a new iterative cross-search attack (ICSA) strategy. In ICSA, VLATTACK will dynamically update the image perturbation under the guidance of the text perturbation.
Specifically, in each multimodal attack step, ICSA first determines the number of image attack steps. Since there are  $K$  adversarial text candidates that will be equally used in ICSA, the iteration number of image attacks that will be allocated to each text sample is  $N_{k} = \left\lfloor \frac{N - N_{s}}{K}\right\rfloor$ . ICSA will take the  $k$ -th pair  $\left(\mathbf{I}_k^{\prime},\hat{\mathbf{T}}_k^{\prime}\right)$  as the input to generate the new image perturbation  $\mathbf{I}_{k + 1}^{\prime}$  by optimizing the block-wise attack loss in Eq. (2), where  $\mathbf{I}_1^\prime$  is the output from the image attack in the single-modal level, i.e.,  $\mathbf{I}^{\prime}$ . Such a process is repeated until an adversarial sample  $(\mathbf{I}_j^{\prime},\hat{\mathbf{T}}_j^{\prime})$  is found, or all the  $K$  perturbed texts
Table 1: Comparison of VLATTACK with baselines on ViLT, Unitab, and OFA for different tasks, respectively. All results are displayed by ASR (%). B&A means the BERT-Attack approach.
Pre-trained Model	Task	Dataset	Image Only	Text Only	multimodality
DR	SSP	FDA	BSA	B&A	R&R	Co-Attack	VLATTACK
ViLT	VQA	VQAv2	23.89	50.36	29.27	65.20	17.24	8.69	35.13	78.05
VR	NLVR2	21.58	35.13	22.60	52.17	32.18	24.82	42.04	66.65
BLIP	VQA	VQAv2	7.04	11.84	7.12	25.04	21.04	2.94	14.24	48.78
VR	NLVR2	6.66	6.88	10.22	27.16	33.08	16.92	8.70	52.66
Unitab	VQA	VQAv2	22.88	33.67	41.80	48.40	14.20	5.48	33.87	62.20
REC	RefCOCO	21.32	64.56	75.24	89.70	13.68	8.75	56.48	93.52
REC	RefCOCO+	26.30	69.60	76.21	90.96	6.40	2.46	68.69	93.40
REC	RefCOCOg	26.39	69.26	78.64	91.31	22.03	18.52	65.50	95.61
OFA	VQA	VQAv2	25.06	33.88	40.02	54.05	10.22	2.34	51.16	78.82
VE	SNLI-VE	13.71	15.11	20.90	29.19	10.51	4.92	18.66	41.78
REC	RefCOCO	11.60	16.00	27.06	40.82	13.15	7.64	32.04	56.62
REC	RefCOCO+	16.58	22.28	33.26	46.44	4.66	7.04	45.28	58.14
REC	RefCOCOg	16.39	24.80	33.22	54.63	19.23	15.13	30.53	73.30
are visited. When  $\mathcal{T} = \emptyset$ , VLATTACK will degenerate to the block-wise similarity attack (BSA) and iteratively updates  $\mathbf{I}'$  for  $N - N_{s}$  steps. Finally, VLATTACK generates adversarial samples based on either single-modality or cross-modality attacks. The overall scheme of VLATTACK is summarized in Algorithm 1.
Pre-trained VL Models and Tasks. Experiments are conducted on five pre-trained models For the encoder-only model, we adopt ViLT [1] and BLIP [28] for two downstream tasks, including the visual question answering (VQA) task on the VQAv2 dataset [48] and the visual reasoning (VR) task on the NLVR2 dataset [50]. For the encoder-decoder structure, we adopt Unitab [4] and OFA [5]. For Unitab, evaluations are made on the VQAv2 dataset for the VQA task and on RefCOCO, RefCOCO+, and RefCOCOg datasets [49] for the Referring Expression Comprehension (REC) task that can be viewed as the task of bounding box generation. For OFA, we implement experiments on the same tasks as Unitab and add the SNLI-VE dataset [51] for the visual entailment (VE) task. The specific structures of these models are detailed in Appendix A. To verify the overall generality, we evaluate the uni-modal tasks on OFA [5] using MSCOCO [52] for the image captioning task and ImageNet-1K [53] for the image classification task. We also evaluate CLIP [29] on the image classification task on the SVHN [54] dataset. Note that all evaluation tasks have publicly available pre-trained and fine-tuned models, which provide more robust reproducibility. The details of each dataset and implementation can be found in Appendix B.
Baselines. We compare VLATTACK with adversarial attacks on different modalities. For attacks on the image modality, we take DR [17], SSP [35], and FDA [34] as baselines. These methods are designed to perturb image features only and can be directly adapted to our problem. Other methods [18, 19, 30, 32, 33, 55] either fully rely on the output from classifiers or combine feature perturbation with classification loss [20, 56, 57]. These methods can not be applied in our problem setting since the pre-trained and fine-tuned models usually share different prediction heads and are trained on different tasks. For attacks on the text modality, we take BERT-Attack (B&A) [21] and R&R [24] as baselines. VLATTACK also compares with Co-Attack [15], which is the only multimodal attack scheme that adds adversarial perturbations to both modalities.
5.2 Results on Multimodal Tasks
In this section, we conduct experiments on four pre-trained VL models and use Attack Success Rate (ASR) to evaluate the performance on four multimodal tasks, including VQA, VR, REC, and VE. The higher the ASR, the better the performance. Results are illustrated in Table 1, where the results of our proposed BSA and VLATTACK are highlighted. We can observe that the proposed VLATTACK significantly outperforms all baselines. Compared with the best baseline on each dataset,
Table 2: Evaluation of the Uni-modal tasks on OFA. We highlight the prediction score reported by the original OFA paper with *.
Dataset	MSCOCO	ImageNet-1K
Metric	BLEU@4 (↓)	METEOR (↓)	CIDEr (↓)	SPICE (↓)	ASR(↑)
OFA*	42.81	31.30	145.43	25.37	-
DR	30.26	24.47	95.52	17.89	10.43
SSP	10.99	12.52	23.54	5.67	19.44
FDA	17.77	17.92	55.75	11.36	12.31
BSA (Ours)	3.04	8.08	2.16	1.50	41.35
Table 3: CLIP model evaluation on SVHN.
Dataset	SVHN
Model	CLIP-ViT/16	CLIP-RN50
DR	3.32	71.62
SSP	6.36	84.26
FDA	6.20	83.52
BSA (Ours)	15.74	84.98
Table 4: MTurK evaluation on the ViLT model using the VQAv2 dataset. Results are obtained on 650 samples.
Method	SSP	VLATTACK
Definitely Correct (↓)	413	307
Not Sure (↑)	58	56
Definitely Wrong (↑)	179	287
VLATTACK achieves an average gain of  $29.61\%$  on ViLT,  $24.20\%$  on BLIP,  $16.48\%$  on Unitab and  $30.84\%$  on OFA, respectively. When only adding image perturbations, our proposed BSA outperforms the best baseline by an average of  $15.94\%$  on ViLT and on BLIP. BSA also achieves an average gain of  $12.12\%$  and  $14.13\%$  on Unitab and OFA, respectively. As mentioned before, text data in multimodal tasks are composed of phrases or only a few words. Consequently, text attack performs poorly on most multimodal datasets. These high ASR values have revealed significant security concerns regarding the adversarial robustness of pre-trained VL models.
5.3 Results on Uni-Modal Tasks
For the results on uni-modal tasks, we first evaluate OFA on image captioning and classification tasks on MSCOCO and ImageNet-1K, respectively. Because these tasks accept a fixed text prompt for prediction, we only perturb image I and compare different image attack methods. For the captioning task, we choose four commonly-used metrics, including BLEU@4, METEOR, CIDEr, and SPICE, the same as those used in OFA [5]. The lower, the better. We report ASR on the image classification task. The higher, the better. The experimental results are illustrated in Table 2. In general, BSA achieves the best performance on both tasks in terms of all metrics. For example, BSA significantly reduces the CIDEr score from 145.43 to 2.16. Compared with the best baseline SSP with 23.54, BSA reduces this score over 10 times. For the image classification task, BSA also achieves the best performance with  $41.35\%$  ASR. The superior performance demonstrates that the VLATTACK can also generalize to computer vision tasks when only using BSA. Note that we do not evaluate text-to-image generation tasks because our attack framework produces the same results as BERT-Attack with text-only inputs.
We also evaluate CLIP on the image classification task on the SVHN dataset. Concretely, we use the image encoder of CLIP as the pre-trained model and then fine-tune on the SVHN dataset after adding a linear classification head. For the choices of image encoder of CLIP, we adopt ViT-B/16 and ResNet-50, denoted by CLIP-ViT/16 and CLIP-RN50. We test the attack performance using 5K correctly predicted samples. All results are illustrated in the following Table 3. Since the task only accepts images as input, we compare our BSA with other image attack baselines. As shown in the table, our proposed BSA still maintains the best ASR using different image encoder structures, clearly demonstrating its effectiveness.
5.4 Systematic Validation with Crowdsourcing
We also conduct a human evaluation study to comprehensively verify the soundness of our proposed method. The experiment is developed on the results output from the ViLT model on the VQAv2 dataset using Amazon Mechanical Turk (MTurk). The baseline that we choose is SSP since it outperforms others in our experiments, as shown in Table 1. Specifically, we randomly sampled 650 examples and stored the generated answers after the attack. To validate the performance of both approaches, we send the original image-question pairs and the corresponding generated answer to
Figure 5: Ablation analysis of different components in VLATTACK. We show the results of VQAv2 (a) and SNLI-VE (b) on OFA, and VQAv2 (c) and RefCOCO (d) on Unitab.
the MTurk system. We provide three choice candidates to workers, including "Definitely Correct", "Not Sure", and "Definitely Wrong". A successful attack means that the worker will label "Definitely Wrong" to the pair. To make the annotations more accurate and reliable, each pair is annotated by three workers, and we report the majority choice as the final human evaluation result. The statistics of human evaluation results are shown in Table 4. We can observe that the proposed VLAttack still significantly outperforms the strongest baseline SSP, thus demonstrating its robustness and effectiveness from a human perceptual perspective.
5.5 Model Design Analysis
Ablation Study. In our model design, we propose a new block-wise similarity attack (BSA) for attacking the image modality and an interactive cross-search attack (ICSA) for attacking image-text modalities together. We use the following methods to evaluate the effectiveness of each component. The results are shown in Figure 5. "IE"/"TE" means that we only use the image/Transformer encoder when calculating the loss in Eq. (2). "BSA" uses both encoders. We set iteration  $N_{s} = 40$  for a fair comparison. Next, "BSA+BA" means after attacking images using BSA, we attack texts using BERT-Attack (Algorithm 1 Lines 1-15). "BSA+BA+Q" denotes replacing ICSA with a simple query strategy by querying the black-box task with each pair  $(\mathbf{I}',\mathbf{T}_i')$ , where  $\mathbf{T}_i'$  comes from the perturbed text candidate list  $\mathcal{T}$ . We can observe that for image attacks, IE and TE play an important role in different tasks, but in general, BSA (= IE + TE) outperforms them. Adding the text attack, BSA+BA performs better than BSA. This comparison result demonstrates that both modalities are vulnerable. BSA+BA+Q performs better than BSA+BA but worse than VLATTACK, which confirms the necessity and reasonableness of conducting the interactive cross-search attack. More results on other datasets are shown in Appendix C.
Parameter Sensitivity Analysis We discuss the effect of different iteration numbers of  $N$  and  $N_{s}$  in VLATTACK. All experiments are conducted on the VQAv2 dataset and the ViLT model. The total iteration number  $N$  is set from 10 to 80,  $N_{s}$  is set to  $\frac{N}{2}$ . As depicted in Figure 6(a), the ASR performance is dramatically improved by increasing  $N$  from 10 to 20 steps and then achieves the best result when  $N = 40$ . We next investigate the impact of different initial iteration numbers  $N_{s}$ . We test  $N_{s}$  from 5 to 40, but the total iteration number  $N$  is fixed to 40. As shown in Figure 6(b), the ASR score reaches the summit when  $N_{s}$  is 5, and it is smoothly decreased by continually enlarging  $N_{s}$ . Considering that the smaller initial iteration number  $N_{s}$  increases the ratio of text perturbations, we set  $N_{s}$  as 20 to obtain the best trade-off between attack performance and the naturalness of generated adversarial samples in our experiments.
Figure 6: Investigation of iteration number  $N$  and  $N_{s}$
Block-wise Similarity Attack. We further visualize the effect of our proposed BSA through attention maps in the fine-tuned downstream model  $S$  on the VQAv2 dataset through the ViLT model. As is shown in Figure 7, the attention area is shifted from "jacket" to the background, which indicates that the representation used to answer the question extracts information from the unrelated image regions. We also combined the BSA with various optimization methods to verify its generalization ability. The experimental results are presented in Appendix D.
We conduct case studies using visualization to show the effectiveness of VLATTACK for multimodal and uni-modal tasks, as displayed in Figure 8. For multimodal tasks in Figure 8(a), the predictions of all displayed samples remain unchanged when attacking a single modality via BSA. However, perturbing both modalities using VLATTACK successfully modifies the predictions. An intriguing finding can be observed in the REC task, where the fine-tuned model  $S$  outputs wrong predictions by stretching and deforming the original bounding box predictions. The proposed VLATTACK also shows its power on uni-modal tasks in Fig-
What color is the man's jacket?
What colour is the man's jacket?
Figure 7: Attention maps of the clean and adversarial samples in the multimodal transformer encoder. Attention maps from the class token  $\langle cls\rangle$  to images. Perturbed tokens are displayed in red.
ure 8(b). For example, in the image captioning task, the fine-tuned model generates a description of a "person" after receiving an adversarial image of a "horse", which is entirely unrelated. More case studies are shown in Appendix E.
Figure 8: Qualitative results of VLATTACK on (a) multimodal tasks and (b) Uni-modal tasks on OFA. Perturbed word tokens and original predictions are displayed in red and blue, respectively. We show the predictions after the adversarial attack with underline.
In this paper, we propose a new question, which aims to generate adversarial samples only based on the publicly available pre-trained models to attack fine-tuned models deployed in different VL tasks. Considering the task-specific and model-specific challenges, we proposed VLATTACK, which generates perturbations by exploring different modalities in two levels. The single-modal level attack first perturbs images through a novel algorithm BSA to disrupt the universal image-text representations and then attacks text if the former fails, which avoids unnecessary modifications on both modalities. If both image and text attacks fail, the multimodal attack level adopts an iterative cross-search attack strategy to generate adversarial image-text combinations. By periodically substituting text candidates during the image attack process, the mutual relations between different modal perturbations are sufficiently explored. Experimental results demonstrate the effectiveness of the proposed VLATTACK on attacking multiple VL tasks, which reveals a significant safety concern in realistic scenarios.
Acknowledgements This work is partially supported by the National Science Foundation under Grant No. 1951729, 1953813, 2119331, 2212323, and 2238275.
[51] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. CoRR, abs/1901.06706, 2019.
[52] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015.
[53] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. IEEE Computer Society, 2009.
[54] Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv:1312.6082, 2013.
[55] Jiafeng Wang, Zhaoyu Chen, Kaixun Jiang, Dingkang Yang, Lingyi Hong, Yan Wang, and Wenqiang Zhang. Boosting the transferability of adversarial attacks with global momentum initialization. CoRR, abs/2211.11236, 2022.
[56] Nathan Inkawich, Kevin J. Liang, Binghui Wang, Matthew Inkawich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. In NeurIPS, 2020.
[57] Nathan Inkawich, Kevin J. Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. In ICLR. OpenReview.net, 2020.
[58] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883. Computer Vision Foundation / IEEE, 2021.
[59] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR (Poster). OpenReview.net, 2017.
[60] Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pretrained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2043-2059. IEEE, 2022.
A. Details of VL Models
This section introduces the details of VL models involved in our experiment, including ViLT, BLIP, Unitab, and OFA.
B. Dataset and Implementation
B.1 Tasks and Datasets
To verify the generalization ability of our proposed VLATTACK, we evaluate a wide array of popular vision language tasks summarized in Table 5. Specifically, the selected tasks span from text understanding (visual reasoning, visual entailment, visual question answering) to image understanding (image classification, captioning) and localization (referring expression comprehension).
For each dataset, we sample 5K correctly predicted samples in the corresponding validation dataset to evaluate the ASR performance. All validation datasets follow the same split settings as adopted in the respective attack models. Because VQA is a multiclass classification task, we select a correct
Table 5: An illustration of all datasets and tasks evaluated in our paper.
Datasets	Task	Task description	Attack Model	Attack Modality
OFA	Unitab	ViLT	Image	Text
VQAv2	VQA	Scene Understanding QA	✓	✓	✓	✓	✓
SNLI-VE	VE	VL Entailment	✓	✓	✓
RefCOCO	REC	Bounding Box Localization	✓	✓	✓	✓
RefCOCOg	REC	Bounding Box Localization	✓	✓	✓	✓
RefCOCO+	REC	Bounding Box Localization	✓	✓	✓	✓
NLVR2	VR	Image-Text Pairs Matching	✓	✓	✓
MSCOCO	Captioning	Image Captioning	✓	✓
ImageNet-1K	Classification	Object Classification	✓	✓
SVHN	Classification	Digit Classification	✓	✓
prediction only if the prediction result is the same as the label with the highest VQA score $^6$ , and regard the label as the ground truth in Eq. (1). In the REC task, a correct prediction is considered when the Intersection-over-Union (IoU) score between the predicted and ground truth bounding box is larger than 0.5. We adopt the same IoU threshold as in Unitab [4] and OFA [5].
B.2 Implementation Details
For the perturbation parameters of images, we follow the setting in the common transferable image attacks [18, 19] and set the maximum perturbation  $\sigma_{i}$  of each pixel to 16/255 on all tasks except REC. Considering that even a single coordinate change can affect the final grounding results to a great extent, the  $\sigma_{i}$  of the REC task is 4/255 to better highlight the ASR differences among distinct methods. The total iteration number  $N$  and step size are set to 40 and 0.01 by following the projected gradient decent method [30], and  $N_{s}$  is 20. For the perturbation on the text, the semantic similarity constraint  $\sigma_{s}$  is set to 0.95, and the number of maximum modified words is set to 1 by following the previous text-attack work [15, 24] to ensure the semantic consistency and imperceptibility. All experiments are conducted on a single GTX A6000 GPU.
C. More Ablation Results
In Section 5.4, we conduct an ablation study to show the effectiveness of each module in our model design on VQA, VE, and REC tasks. Here, we conduct additional ablation experiments for the remaining tasks, including visual reasoning, image captioning, and image classification.
Figure 9: ViLT-VR.
Figure 10: OFA-capitaling.
Figure 9 shows the results of the ablation study on the VR task using the ViLT model. We can observe that only using the image encoder results in significantly low ASR. However, by combining it with the Transformer encoder (TE), BSA can achieve a high ASR. These results show the reasonableness of considering two encoders simultaneously when attacking the image modality. The result of  $\mathrm{BSA + BA}$  demonstrates the usefulness of attacking the text modality. Although  $\mathrm{BSA + BA + Q}$  outperforms other approaches, its performance is still lower than that of the proposed VLATTACK. This comparison proves that the proposed iterative cross-search attack (ICSA) strategy is effective for the multimodal attack again.
<sup>6</sup>The VQA score calculates the percentage of the predicted answer that appears in 10 reference ground truth answers. More details can be found via https://visualqa.org/evaluation.html
Figure 10 shows the results of the image captioning task using the OFA model. Because the image captioning task only accepts a fixed text prompt for prediction, we only perturb the image and report the results on IE, TE, and BSA. For this task, we report BLEU@4 and CIDEr scores. The lower, the better. We can observe that the proposed BSA outperforms IE and TE, indicating our model design's effectiveness.
Figure 11 shows the results of the image classification task using the OFA model. The experiment is conducted on the ImageNet-1K dataset. Similar to the image captioning task, we only attack images, and compare the results of IE, TE and BSA. The evaluation metric for this task is ASR. The higher, the better. We can have the same observations with other ablation studies, where attacking both encoders outperforms attacking a single encoder.
Figure 11: OFA-classification.
Table 6: Combining VLATTACK with different gradient-based image attack schemes.
Method	ViLT	Unitab
VQAv2	NLVR2	VQAv2	RefCOCO	RefCOCO+	RefCOCOg
BSAMI	65.40	52.32	50.38	86.00	89.20	87.39
VLATTACKMI	78.77	67.16	63.02	92.46	93.10	94.34
BSADI	65.94	52.30	42.74	90.30	91.56	91.00
VLATTACKDI	78.07	67.50	61.22	93.98	94.04	94.76
D. Different Optimization Methods
VLATTACK can be easily adapted to various optimization methods in image attacks. To demonstrate the generalizability of our method, we replace the projected gradient decent [30] in VLATTACK with Momentum Iterative method (MI) [55] and Diverse Input attack (DI) [33] since they have shown better performance than traditional iterative attacks [30, 59]. The replaced methods are denoted by  $\mathrm{BSA}_{MI}$ , and VLATTACK $_{MI}$  using MI,  $\mathrm{BSA}_{DI}$  and VLATTACK $_{DI}$  using DI, respectively. Experiments are developed on ViLT and Unitab. Results are shown in Table 6. Using MI and DI optimizations,  $\mathrm{BSA}_{MI}$  and  $\mathrm{BSA}_{DI}$  still outperform all baselines displayed in Table 1 in the main manuscript. Also, VLATTACK $_{MI}$  and VLATTACK $_{DI}$  outperform the image attack method  $\mathrm{BSA}_{MI}$  and  $\mathrm{BSA}_{DI}$  with an average ASR improvement of  $9.70\%$  and  $9.29\%$  among all datasets. The gain of performance demonstrates that the proposed VLATTACK can be further improved by combining with stronger gradient-based optimization schemes.
E. Case Study
E.1 How does VLATTACK generate adversarial samples?