Title: VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: vision-language

adversarial attacks

pre-trained model

fine-tuned model
Abstract: Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multi-modal levels. At the single-modal level, we propose a new block-wise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multi-modal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level.  We conduct extensive experiments to attack three widely-used VL pretrained models for six tasks on eight datasets. Experimental results show that the proposed VLATTACK framework achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a significant blind spot in the deployment of pre-trained VL models.

VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models

Abstract

Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK 2 to generate adversarial samples by fusing perturbations of images and texts from both singlemodal and multimodal levels. At the single-modal level, we propose a new blockwise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multimodal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack five widely-used VL pre-trained models for six tasks. Experimental results show that VLATTACK achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a blind spot in the deployment of pre-trained VL models.

1 Introduction

The recent success of vision-language (VL) pre-trained models on multimodal tasks have attracted broad attention from both academics and industry [1, 2, 3, 4, 5, 6, 7]. These models first learn multimodal interactions by pre-training on the large-scale unlabeled image-text datasets and are later fine-tuned with labeled pairs on different downstream VL tasks [8, 9, 10]. In many cases, these pre-trained models have revealed more powerful cross-task learning capabilities compared to training from scratch [11, 12]. Despite their remarkable performance, the adversarial robustness of these VL models is still relatively unexplored.

Existing work [13, 14, 15, 16] conducting adversarial attacks in VL tasks is mainly under the whitebox setting, where the gradient information of fine-tuned models is accessible to attackers. However, in a more realistic scenario, a malicious attacker may only be able to access the public pre-trained models released through third parties. The attacker would not have any prior knowledge about the

"A painting of

Output

"Contradiction"

"Unknown"

■

awomanwith

■

a tree."

Downstream

Visual

Visual

Visualquestion

Image

Task

Entailment

Grounding

Answering

Caption

VLAttack:GeneratingAdversarialPerturbations

PretrainedVision-LanguageModel

the duo

the giant

Whichway

Whatdoes

Input

are sining

cruise ship

is the cat's

the image

a song.

head turned?

describe?

Ground

Truth

"Neutral"

"Front"

I

"Twobowls thathave

foodinsideofthem."

parameters learned by downstream VL models fine-tuned on private datasets. Towards bridging this striking limitation, we investigate a new yet practical attack paradigm generating adversarial perturbations on a pre-trained VL model to attack various black-box downstream tasks fine-tuned on the pre-trained one.

However, such an attack setting is non-trivial and faces the following challenges: (1) Task-specific challenge . The pre-trained VL models are usually used for fine-tuning different downstream tasks, which requires the designed attack mechanism to be general and work for attacking multiple tasks. As illustrated in Figure 1, the attacked tasks are not only limited to close-set problems, such as visual question answering, but also generalized to open-ended questions, such as visual grounding. (2) Model-specific challenge. Since the parameters of the fine-tuned models are unknown, it requires the attack method to automatically learn the adversarial transferability between pre-trained and finetuned models on different modalities. Although the adversarial transferability [17, 18, 19, 20] across image models has been widely discussed, it is still largely unexplored in the pre-trained models, especially for constructing mutual connections between perturbations on different modalities.

To address all the aforementioned challenges, we propose a new yet general Vision-Language Attack strategy (named VLATTACK) to explore the adversarial transferability between pre-trained and fine-tuned VL models, as shown in Figure 2. The whole VLATTACK scheme fuses perturbations of images and texts from two levels:

Block-wise

Similarity  Attack

BERT-Attack

Iterative

Cross-Search

Attack

Single-modal

Multimodal

Step  1

Image

Text

Fine-tuned

Query

Query

Query

Step  2

Model

{   }

(          )

Step  3

Pre-trained Model

Pre-trained Model

Single-modal Level . VLATTACK independently generates perturbations on a single modality, following a ' from image to text ' order as the former can be perturbed on a continuous space. The single-modal attack can effectively detect the adversarial vulnerability of an image or text, and hence avoids redundant perturbations on the other modality. Specifically, to fully utilize the image-text interactions that have been stored in the pre-trained model, we propose a novel block-wise similarity attack (BSA) strategy to attack the image modality , which adds perturbations to enlarge the network block-wise distance between original and perturbed features in the pretrained model, disrupting the universal image-text representations for the downstream predictions. If BSA fails to change the prediction after querying the fine-tuned black-box model, VLATTACK will attack the text modality by employing the word-level perturbation techniques [21, 22, 23, 24]. We adopt BERT-Attack [21] to attack the text modality as its prominent performance has been widely verified in many studies [25, 26, 27]. Finally, if all the text perturbations { T ′ i } fail, VLATTACK will generate a list of perturbed samples T and feed them to the multimodal attack along with the perturbed image I ′ .

Multimodal Level . If the above attack fails to change the predictions, we cross-update image and text perturbations at the multimodal level based on previous outputs. The proposed iterative crosssearch attack (ICSA) strategy updates the image-text perturbation pair ( I ′ i , T ′ i ) in an iterative way

by considering the mutual relations between different modal perturbations. ICSA uses a text perturbation T ′ i selected from the list T as the guidance to iteratively update the perturbed image I ′ i by employing the block-wise similarity attack (BSA) until the new pair ( I ′ i , T ′ i ) makes the prediction of the downstream task change. In addition, text perturbations are cross-searched according to the semantic similarity with the benign one at the multimodal attack level, which gradually increases the extent of the direction modification to preserve the original semantics to the greatest extent.

Our contributions are summarized as follows: (1) To the best of our knowledge, we are the first to explore the adversarial vulnerability across pre-trained and fine-tuned VL models. (2) We propose VLATTACK to search adversarial samples from different levels. For the single-modal level, we propose the BSA strategy to unify the perturbation optimization targets on various downstream tasks. For the multimodal level, we design the ICSA to generate adversarial image-text pairs by cross-searching perturbations on different modalities. (3) To demonstrate the generalization ability of VLATTACK, we evaluate the proposed VLATTACK on five widely-used VL models , including BLIP [28], CLIP [29], ViLT [1], OFA [5] and UniTAB [4] for six tasks : ( i ) VQA, ( ii ) visual entailment, ( iii ) visual reasoning, ( iv ) referring expression comprehension, ( v ) image captioning, and ( vi ) image classification. Experimental results demonstrate that VLATTACK outperforms both single-modal and multimodal attack approaches, which reveals a significant blind spot in the robustness of large-scale VL models.

2 Related Work

Single-modal Adversarial Attack Methods . Image Attack . Traditional image attack methods [30, 31] generate adversarial samples by optimizing the loss function with regard to the decision boundary from model outputs. Not only do the generated perturbations change the model predictions, but they can also be transferred to other convolutional neural network (CNN) structures. Such a property is called transferability and has been extensively studied. For example, data augmentation-based methods [32, 33] endeavor to create diverse input patterns to enhance representation diversity across different models. Feature disruptive attacks [17, 34, 35] introduce the intermediate loss to change the local activation of image features output from the middle layers of CNNs, enabling the perturbed features to transfer to different models without knowing their structures and parameters. Text Attack . Adversarial attacks on natural language processing (NLP) tasks mainly concentrate on word-level and sentence-level perturbations. Word-level perturbations [21, 22, 23, 24] substitute words with synonyms that share similar word embeddings and contextual information. Sentence-level perturbations [25, 36, 37] focus on logical structures of texts through paraphrasing or adding unrelated sentence segments. All of these methods have revealed the adversarial vulnerability of traditional NLP models to some extent. However, adapting them to a multimodal attack setting is still underexplored.

Multimodal Adversarial Attack Methods . Multimodal VL models are susceptible to adversarial attacks as perturbations can be added to both modalities. Existing methods mainly explore adversarial robustness on a specific VL task. For the visual question answering task, Fool-VQA [13] is proposed, which iteratively adds pixel-level perturbations on images to achieve the attack. For the image-text retrieval task, CMLA [14] and AACH [38] add perturbations to enlarge Hamming distance between image and text hash codes, which causes wrong image-text matching results. Recently, Co-attack [15] is proposed, which combines image and text perturbations using word substitution attacks to ascertain a direction for guiding the multi-step attack on images. The whole framework is deployed in a white-box setting, where attackers are assumed to have access to the parameters of each downstream task model. Besides, Co-attack is only validated on three tasks, which is not general enough.

3 Preliminaries

VL Pre-trained Model Structure . Fine-tuning from pre-training has become a unified paradigm in the recent VL model design. Most pre-trained VL models can be divided into encoder-only [1, 39, 40, 41] and encoder-decoder [4, 5, 6] structures. Two representative model structures are shown in Figure 3. Given an image-text pair ( I , T ) , both VL models first embed each modality separately. Image tokens are obtained through an image encoder composed of a vision transformer [42] or a Resnet [43] after flattening the grid features into a sequence. Text tokens are generated through

Text

T

Transformer Encoder

Transformer

Encoder

<

ans

>

<

ans

> <

ans

>

<

end

>

1

2

<

cls

>

Text

T

Transformer Decoder

(a)

(b)

Image

I

Image

I

linear

Image

Encoder

Word

Encoder

Image

Encoder

Word

Encoder

Text

T

Word

Encoder

Image

I

Block 1

Block M

Block M

Block 1

Block 2

Image

I

Word

Encoder

Block 1

Block M

i

Block M

Block 1

Block 2

F

(I)

F

(I,T)

F

(I )

Cosine Similarity

F

(I,T)

i

Text

T

a word encoder made up of a tokenizer [44, 45] and a word vector projection. The encoder-only model then attends tokens of two modalities and a learnable special token ⟨ cls ⟩ and feeds them into a Transformer encoder. Finally, the output representation from the ⟨ cls ⟩ token is fed into a classification head for the final prediction ⟨ ans ⟩ . For the encoder-decoder structure, the attached image and text tokens are fed into a Transformer network [46] to generate sequence predictions [ ⟨ ans 1 ⟩ , ⟨ ans 2 ⟩ , · · · , ⟨ end ⟩ ] in an auto-regressive manner. The network stops regressing when an end token ⟨ end ⟩ appears. In this work, we deeply explore the adversarial vulnerability of both structures.

Threat Model . Let F denote the public pre-trained model and S represent the downstream task model, where S shares most of or the same structure with F . As shown in Figure 3, the network structures of both types of public VL pre-trained models are different. The encoder-only model allows modifying the last prediction layer based on the requirement of downstream tasks but keeping the other layers the same as F . However, the encoder-decoder model unifies the outputs of different downstream tasks, which leads to S having the same structure as F . Note that the downstream tasks will fully fine-tune parameters using their own data, and hence, all the model parameters in F are updated in the fine-tuning stage. Thus, all parameters in S are not accessible.

Given an image-text pair ( I , T ) , the goal of the downstream task is to predict the labels of the input pair accurately, i.e., S : ( I , T ) → Y , where Y = { y 1 , · · · , y n } . For the encoder-only models, the ground truth label is a one-hot vector. For the encoder-decoder models, the ground truth is a sequence that consists of multiple ordered one-hot vectors. Let y denote the ground truth vector. The goal of the adversarial attack is to generate adversarial examples ( I ′ , T ′ ) using F , which can cause an incorrect prediction on S . Mathematically, our problem is formulated as follows:

̸

max I ′ , T ′ ✶ { S ( I ′ , T ′ ) = y } , s.t. ∥ I ′ -I ∥ ∞ < σ i , Cos ( U s ( T ′ ) , U s ( T )) > σ s , (1) where σ i is the l ∞ -norm perturbation strength on the image. σ s is the semantic similarity between the original and perturbed texts, which constrains the semantic consistency after perturbation. The semantic similarity is measured by the cosine similarity Cos ( · , · ) between the sentence embedding U s ( T ′ ) and U s ( T ) using the Universal Sentence Encoder U s [47].

4 Methodology

As shown in Figure 2, the proposed VLATTACK generates adversarial samples from two steps, constrained by perturbation budget parameters σ i and σ s . The first step attacks every single modality independently to avoid unnecessary modifications. Samples that fail at the first step will be fed into our multimodal level, where we adopt a cross-search attack strategy to iteratively refine both image and text perturbations at the same time. Next, we will introduce the details of VLATTACK from the single-modal level to the multimodal level.

4.1 Single-modal Attacks: From Image to Text

VLATTACK attempts to generate adversarial samples on every single modality in the first step. Compared to the discrete words in T , the continuous values-based image I is more vulnerable to attack by using gradient information 3 . Thus, VLATTACK starts by attacking the image I .

Image-Attack. In the black-box setting, the parameters of the fine-tuned model S are unknown. However, the pre-trained model F is accessible. Intuitively, if the feature representations learned by the pre-trained model F from the clean input I and the perturbed input I ′ are significantly different, such a perturbation may transfer to fine-tuned models to change the predictions of downstream tasks. As shown in Figure 3, the image features can be obtained from both the image encoder and the Transformer encoder, even from different layers or blocks. To fully leverage the specific characteristics of the pre-trained model structure, we propose the block-wise similarity attack (BSA) to corrupt universal contextualized representations.

As shown in Figure 4, BSA perturbs images by maximizing the block-wise distances between the intermediate representations in the image encoder F α and Transformer encoder F β of the pre-trained model F . Mathematically, we define the loss function of BSA as follows:

where M i is the number of blocks in the image encoder, and M i j is the number of flattened image feature embeddings generated in the i -th block 4 . Similarly, M k is the number of blocks in the Transformer encoder, and M k t is the number of image token features generated in the k -th block. F i,j α is the j -th feature vector obtained in the i -th layer of the image encoder, and F k,t β is the t -th feature vector obtained in the k -th layer of the Transformer encoder. The image encoder only takes a single image I or I ′ as the input, but the Transformer encoder will use both image and text as the input. We adopt the cosine similarity to calculate the distances between perturbed and benign features as token representations attended with each other in the inner product space [46]. Note that BSA does not rely on the information from the decision boundary, and thus, it can be easily adapted to different task settings by disrupting the benign representations.

In the image attack step, we generate an adversarial image candidate through project gradient decent optimization [30] with N s iterations, where N s < N and N is the maximum number of iterations on the image attack. The remaining N -N s attack step budgets will be used in the multimodal attack in Section 4.2. If ( I ′ , T ) is an adversarial sample generated by BSA, then VLATTACK will stop. Otherwise, VLATTACK moves to attack the text modality.

Text-Attack. In some VL tasks, the number of tokens in the text is quite small. For example, the average length of the text in the VQAv2 [48] and RefCOCO [49] datasets is 6.21 and 3.57, respectively 5 . Moreover, some of them are nonsense words, which makes it unnecessary to design a new approach for attacking the text modality. Furthermore, existing text attack approaches such as BERT-Attack [21] are powerful for generating adversarial samples for texts. Therefore, we directly apply BERT-Attack to generate text perturbations. To avoid unnecessary modifications, we use the clean image I as the input instead of the generation perturbed image I ′ .

Specifically, an adversarial candidate T ′ i produced by BERT-Attack is firstly fed into a universal sentence encoder U s with the benign text T to test the semantic similarity. T ′ i will be removed if the cosine similarity γ i between T ′ i and T is smaller than the threshold σ s , i.e., γ i = Cos ( U s ( T ′ i ) , U s ( T )) < σ s . Otherwise, we put T ′ i and the benign image I into the finetuned model S to detect whether the new pair ( I , T ′ i ) perturbs the original prediction y . During the text attack, we create a list T to store all perturbed samples { T ′ i } and the corresponding cosine similarity values { γ i } . If any sample in T successfully changes the prediction of the downstream task, then VLATTACK will stop. Otherwise, VLATTACK will carry the perturbed image I ′ from the image attack and the perturbed text candidate list T from the text attack to the multimodal attack level. Note that BERT-Attack is based on synonym word substitution. Even for a short text, the number of adversarial examples can still be large. In other words, the length of T may be large. The whole single-modal attack process is shown in Algorithm 1 (lines 1-15).

Algorithm 1 VLATTACK

Input: Apre-trained model F , a fine-tuned model S , a clean image-text pair ( I , T ) and its prediction y on the S , and the Gaussian distribution U ;

Parameters: Perturbation budget σ on I , σ on T . Iteration number N and N .

̸

̸

̸

4.2 Multimodal Attack

In many cases, only perturbing images or texts is hard to succeed, as a single-modality perturbation is insufficient to break down the image-text correlation. To solve this problem, we propose a new attack strategy to disrupt dynamic mutual connections based on perturbations from different modalities.

Attack Steps . Since the maximum number of image attack steps N is predefined, it is impossible to test all the perturbed image-text pairs ( I ′ , T ′ i ) using the remaining budget N -N s if the length of T (i.e., |T | ) is very large. Thus, we need to rank the perturbed text samples { T ′ i } according to their corresponding cosine similarity values { γ i } in a descending order to make the adversarial sample keep the high semantic similarity with the original text T . Let K denote the number of attack steps in the multimodal attack, and we have:

Iterative Cross-Search Attack . A naive way to conduct the multimodal attack is to test each perturbed image-text pair ( I ′ , ˆ T ′ k ) ( k = 1 , · · · , K ) by querying the black-box fine-tuned model S , where ˆ T ′ i is the i -th text perturbation in the ranked list T . However, this simple approach ignores learning mutual connections between the perturbed text and image. To solve this issue, we propose a new iterative cross-search attack (ICSA) strategy. In ICSA, VLATTACK will dynamically update the image perturbation under the guidance of the text perturbation.

Specifically, in each multimodal attack step, ICSA first determines the number of image attack steps. Since there are K adversarial text candidates that will be equally used in ICSA, the iteration number of image attacks that will be allocated to each text sample is N k = ⌊ N -N s K ⌋ . ICSA will take the k -th pair ( I ′ k , ˆ T ′ k ) as the input to generate the new image perturbation I ′ k +1 by optimizing the block-wise attack loss in Eq. (2), where I ′ 1 is the output from the image attack in the single-modal level, i.e., I ′ . Such a process is repeated until an adversarial sample ( I ′ j , ˆ T ′ j ) is found, or all the K perturbed texts

̸

are visited. When T = ∅ , VLATTACK will degenerate to the block-wise similarity attack (BSA) and iteratively updates I ′ for N -N s steps. Finally, VLATTACK generates adversarial samples based on either single-modality or cross-modality attacks. The overall scheme of VLATTACK is summarized in Algorithm 1.

5 Experiments

5.1 Experiment Setup

Pre-trained VL Models and Tasks. Experiments are conducted on five pre-trained models For the encoder-only model, we adopt ViLT [1] and BLIP [28] for two downstream tasks, including the visual question answering (VQA) task on the VQAv2 dataset [48] and the visual reasoning (VR) task on the NLVR2 dataset [50]. For the encoder-decoder structure, we adopt Unitab [4] and OFA [5]. For Unitab, evaluations are made on the VQAv2 dataset for the VQA task and on RefCOCO, RefCOCO+, and RefCOCOg datasets [49] for the Referring Expression Comprehension (REC) task that can be viewed as the task of bounding box generation. For OFA, we implement experiments on the same tasks as Unitab and add the SNLI-VE dataset [51] for the visual entailment (VE) task. The specific structures of these models are detailed in Appendix A. To verify the overall generality, we evaluate the uni-modal tasks on OFA [5] using MSCOCO [52] for the image captioning task and ImageNet-1K [53] for the image classification task. We also evaluate CLIP [29] on the image classification task on the SVHN [54] dataset. Note that all evaluation tasks have publicly available pre-trained and fine-tuned models, which provide more robust reproducibility. The details of each dataset and implementation can be found in Appendix B.

Baselines . We compare VLATTACK with adversarial attacks on different modalities. For attacks on the image modality, we take DR [17], SSP [35], and FDA [34] as baselines. These methods are designed to perturb image features only and can be directly adapted to our problem. Other methods [18, 19, 30, 32, 33, 55] either fully rely on the output from classifiers or combine feature perturbation with classification loss [20, 56, 57]. These methods can not be applied in our problem setting since the pre-trained and fine-tuned models usually share different prediction heads and are trained on different tasks. For attacks on the text modality, we take BERT-Attack (B&A) [21] and R & R [24] as baselines. VLATTACK also compares with Co-Attack [15], which is the only multimodal attack scheme that adds adversarial perturbations to both modalities.

5.2 Results on Multimodal Tasks

In this section, we conduct experiments on four pre-trained VL models and use Attack Success Rate (ASR) to evaluate the performance on four multimodal tasks, including VQA, VR, REC, and VE. The higher the ASR, the better the performance. Results are illustrated in Table 1, where the results of our proposed BSA and VLATTACK are highlighted. We can observe that the proposed VLATTACK significantly outperforms all baselines. Compared with the best baseline on each dataset,

VLATTACK achieves an average gain of 29.61 % on ViLT, 24.20 % on BLIP, 16.48 % on Unitab and 30.84 % on OFA, respectively. When only adding image perturbations, our proposed BSA outperforms the best baseline by an average of 15.94 % on ViLT and on BLIP. BSA also achieves an average gain of 12.12 % and 14.13 % on Unitab and OFA, respectively. As mentioned before, text data in multimodal tasks are composed of phrases or only a few words. Consequently, text attack performs poorly on most multimodal datasets. These high ASR values have revealed significant security concerns regarding the adversarial robustness of pre-trained VL models.

5.3 Results on Uni-Modal Tasks

For the results on uni-modal tasks, we first evaluate OFA on image captioning and classification tasks on MSCOCO and ImageNet-1K, respectively. Because these tasks accept a fixed text prompt for prediction, we only perturb image I and compare different image attack methods. For the captioning task, we choose four commonly-used metrics, including BLEU @ 4, METEOR, CIDEr, and SPICE, the same as those used in OFA [5]. The lower, the better. We report ASR on the image classification task. The higher, the better. The experimental results are illustrated in Table 2. In general, BSA achieves the best performance on both tasks in terms of all metrics. For example, BSA significantly reduces the CIDEr score from 145.43 to 2.16 . Compared with the best baseline SSP with 23.54, BSA reduces this score over 10 times . For the image classification task, BSA also achieves the best performance with 41.35 % ASR. The superior performance demonstrates that the VLATTACK can also generalize to computer vision tasks when only using BSA. Note that we do not evaluate textto-image generation tasks because our attack framework produces the same results as BERT-Attack with text-only inputs.

We also evaluate CLIP on the image classification task on the SVHN dataset. Concretely, we use the image encoder of CLIP as the pre-trained model and then fine-tune on the SVHN dataset after adding a linear classification head. For the choices of image encoder of CLIP, we adopt ViT-B/16 and ResNet-50, denoted by CLIP-ViT/16 and CLIP-RN50. We test the attack performance using 5K correctly predicted samples. All results are illustrated in the following Table 3. Since the task only accepts images as input, we compare our BSA with other image attack baselines. As shown in the table, our proposed BSA still maintains the best ASR using different image encoder structures, clearly demonstrating its effectiveness.

5.4 Systematic Validation with Crowdsourcing

We also conduct a human evaluation study to comprehensively verify the soundness of our proposed method. The experiment is developed on the results output from the ViLT model on the VQAv2 dataset using Amazon Mechanical Turk (MTurk). The baseline that we choose is SSP since it outperforms others in our experiments, as shown in Table 1. Specifically, we randomly sampled 650 examples and stored the generated answers after the attack. To validate the performance of both approaches, we send the original image-question pairs and the corresponding generated answer to

IE

TE

BSA

BSA+BA

BSA+BA+Q

VLATTACK

75

60

92-

40

ASR (%)

65-

50

88

30

55-

40-

84-

45

20-

30

80

(a)OFA-VQAv2

(b)OFA-SNLI-VE

(c)Unitab-VQAv2

(d)Unitab-RefCOCO

the MTurk system. We provide three choice candidates to workers, including 'Definitely Correct', 'Not Sure', and 'Definitely Wrong'. A successful attack means that the worker will label 'Definitely Wrong' to the pair. To make the annotations more accurate and reliable, each pair is annotated by three workers, and we report the majority choice as the final human evaluation result. The statistics of human evaluation results are shown in Table 4. We can observe that the proposed VLAttack still significantly outperforms the strongest baseline SSP, thus demonstrating its robustness and effectiveness from a human perceptual perspective.

5.5 Model Design Analysis

Ablation Study. In our model design, we propose a new block-wise similarity attack (BSA) for attacking the image modality and an interactive cross-search attack (ICSA) for attacking image-text modalities together. We use the following methods to evaluate the effectiveness of each component. The results are shown in Figure 5. 'IE'/'TE' means that we only use the image/Transformer encoder when calculating the loss in Eq. (2). 'BSA' uses both encoders. We set iteration N s = 40 for a fair comparison. Next, 'BSA+BA' means after attacking images using BSA, we attack texts using BERT-Attack (Algorithm 1 Lines 1-15). 'BSA+BA+Q' denotes replacing ICSA with a simple query strategy by querying the black-box task with each pair ( I ′ , T ′ i ), where T ′ i comes from the perturbed text candidate list T . We can observe that for image attacks, IE and TE play an important role in different tasks, but in general, BSA (= IE + TE) outperforms them. Adding the text attack, BSA+BA performs better than BSA. This comparison result demonstrates that both modalities are vulnerable. BSA+BA+Q performs better than BSA+BA but worse than VLATTACK, which confirms the necessity and reasonableness of conducting the interactive cross-search attack. More results on other datasets are shown in Appendix C.

Parameter Sensitivity Analysis We discuss the effect of different iteration numbers of N and N s in VLATTACK. All experiments are conducted on the VQAv2 dataset and the ViLT model. The total iteration number N is set from 10 to 80, N s is set to N 2 . As depicted in Figure 6(a), the ASRperformance is dramatically improved by increasing N from 10 to 20 steps and then achieves the best result when N = 40 . We next investigate the impact of different initial iteration numbers N s . We test N s from 5 to 40, but the total iteration number N is fixed to 40. As shown in Figure 6(b), the ASR score reaches the summit when N s is 5, and it is smoothly decreased by continually enlarging N s . Considering that the smaller initial iteration number N s increases the ratio of text perturbations, we set N s as 20 to obtain the best trade-off between attack performance and the naturalness of generated adversarial samples in our experiments.

80

ASR (%)

70

10

20

30

40

50

60

70

(a)Iteration Number N

85

(%)

80

ASR(

75

70

10

15

20

25

30

(b) Iteration Number Ns

Block-wise Similarity Attack . We further visualize the effect of our proposed BSA through attention maps in the fine-tuned downstream model S on the VQAv2 dataset through the ViLT model. As is shown in Figure 7, the attention area is shifted from ' jacket ' to the background, which indicates that the representation used to answer the question extracts information from the unrelated image regions. We also combined the BSA with various optimization methods to verify its generalization ability. The experimental results are presented in Appendix D.

5.6 Case Study

We conduct case studies using visualization to show the effectiveness of VLATTACK for multimodal and uni-modal tasks, as displayed in Figure 8. For multimodal tasks in Figure 8(a), the predictions of all displayed samples remain unchanged when attacking a single modality via BSA. However, perturbing both modalities using VLATTACK successfully modifies the predictions. An intriguing finding can be observed in the REC task, where the fine-tuned model S outputs wrong predictions by stretching and deforming the original bounding box predictions. The proposed VLATTACK also shows its power on uni-modal tasks in Fig-

Clean

Adversarial

L

What coloris

Whatcolouris

the man's

the man's

jacket?

<cls>

jacket?

<cls>

ure 8(b). For example, in the image captioning task, the fine-tuned model generates a description of a ' person ' after receiving an adversarial image of a ' horse ', which is entirely unrelated. More case studies are shown in Appendix E.

Visual Question

Referring Expression

Visual

Image

Image

Answering

Comprehension

Entailment

Captioning

Classification

lean

What species of bird

A man in a purple

A black dog jumping to

A pony is standing

FoxSquirrel

isinthisphoto?cardinal

shirt and jeans.

catchafrisbee.Contradiction，

neararowoftrees.

Adversarial

What typeof bird

A man in a purple

A black dogsjumping to

A close up of a person

Conch

is in this photo?owl

shirt and denim.

catchafrisbee.Neutral

withanumbrella.

(a) Multi-modal Tasks

(b) Uni-modal Tasks

6 Conclusion

In this paper, we propose a new question, which aims to generate adversarial samples only based on the publicly available pre-trained models to attack fine-tuned models deployed in different VL tasks. Considering the task-specific and model-specific challenges, we proposed VLATTACK, which generates perturbations by exploring different modalities in two levels. The single-modal level attack first perturbs images through a novel algorithm BSA to disrupt the universal image-text representations and then attacks text if the former fails, which avoids unnecessary modifications on both modalities. If both image and text attacks fail, the multimodal attack level adopts an iterative crosssearch attack strategy to generate adversarial image-text combinations. By periodically substituting text candidates during the image attack process, the mutual relations between different modal perturbations are sufficiently explored. Experimental results demonstrate the effectiveness of the proposed VLATTACK on attacking multiple VL tasks, which reveals a significant safety concern in realistic scenarios.

Acknowledgements This work is partially supported by the National Science Foundation under Grant No. 1951729, 1953813, 2119331, 2212323, and 2238275.

References

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML , volume 139 of Proceedings of Machine Learning Research , pages 5583-5594. PMLR, 2021.

Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. CoRR , abs/2004.00849, 2020.

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR , pages 16783-16794. IEEE, 2022.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In ECCV (36) , volume 13696 of Lecture Notes in Computer Science , pages 521539. Springer, 2022.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 23318-23340. PMLR, 2022.

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. CoRR , abs/2206.08916, 2022.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In CVPR , pages 15617-15629. IEEE, 2022.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS , pages 13-23, 2019.

Zaid Khan, B. G. Vijay Kumar, Xiang Yu, Samuel Schulter, Manmohan Chandraker, and Yun Fu. Single-stream multi-level alignment for vision-language pretraining. In ECCV (36) , volume 13696 of Lecture Notes in Computer Science , pages 735-751. Springer, 2022.

Aishwarya Agrawal, Damien Teney, and Aida Nematzadeh. Vision-language pretraining: Current trends and the future. In ACL (tutorial) , pages 38-43. Association for Computational Linguistics, 2022.

Salman H. Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Comput. Surv. , 54(10s):200:1-200:41, 2022.

Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. In IJCAI , pages 5436-5443. ijcai.org, 2022.

Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, and Dawn Song. Fooling vision and language models despite localization and attention mechanism. In CVPR , pages 4951-4961. Computer Vision Foundation / IEEE Computer Society, 2018.

Chao Li, Shangqian Gao, Cheng Deng, De Xie, and Wei Liu. Cross-modal learning with adversarial samples. In NeurIPS , pages 10791-10801, 2019.

Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pretraining models. In ACM Multimedia , pages 5005-5013. ACM, 2022.

Yao Zhou, Jun Wu, Haixun Wang, and Jingrui He. Adversarial robustness through bias variance decomposition: A new perspective for federated learning. In Mohammad Al Hasan and Li Xiong, editors, Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022 , pages 2753-2762. ACM, 2022.

Yantao Lu, Yunhan Jia, Jianyu Wang, Bai Li, Weiheng Chai, Lawrence Carin, and Senem Velipasalar. Enhancing cross-task black-box transferability of adversarial examples with dispersion reduction. In CVPR , pages 937-946. Computer Vision Foundation / IEEE, 2020.

Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In AAAI , pages 26682676. AAAI Press, 2022.

Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR . OpenReview.net, 2020.

Qian Huang, Isay Katsman, Zeqi Gu, Horace He, Serge J. Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In ICCV , pages 4732-4741. IEEE, 2019.

Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: adversarial attack against BERT using BERT. In EMNLP (1) , pages 6193-6202. Association for Computational Linguistics, 2020.

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In AAAI , pages 80188025. AAAI Press, 2020.

Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. In NDSS . The Internet Society, 2019.

Lei Xu, Alfredo Cuesta-Infante, Laure Berti-Équille, and Kalyan Veeramachaneni. R&r: Metric-guided adversarial sentence generation. In AACL/IJCNLP (Findings) , pages 438-452. Association for Computational Linguistics, 2022.

Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models. In NeurIPS Datasets and Benchmarks , 2021.

Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. Contextualized perturbation for textual adversarial attack. In NAACL-HLT , pages 50535069. Association for Computational Linguistics, 2021.

Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. Openattack: An open-source textual adversarial attack toolkit. CoRR , abs/2009.09191, 2020.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping languageimage pre-training for unified vision-language understanding and generation. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 12888-12900. PMLR, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR (Poster) . OpenReview.net, 2018.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR (Poster) , 2015.

Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. CoRR , abs/1904.02884, 2019.

Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In CVPR , pages 2730-2739. Computer Vision Foundation / IEEE, 2019.

Aditya Ganeshan, Vivek B. S., and Venkatesh Babu Radhakrishnan. FDA: feature disruptive attack. In ICCV , pages 8068-8078. IEEE, 2019.

Muzammal Naseer, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self-supervised approach for adversarial robustness. In CVPR , pages 259-268. Computer Vision Foundation / IEEE, 2020.

Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In NAACL-HLT , pages 1875-1885. Association for Computational Linguistics, 2018.

Aakanksha Naik, Abhilasha Ravichander, Norman M. Sadeh, Carolyn P. Rosé, and Graham Neubig. Stress test evaluation for natural language inference. In COLING , pages 2340-2353. Association for Computational Linguistics, 2018.

Chao Li, Shangqian Gao, Cheng Deng, Wei Liu, and Heng Huang. Adversarial attack on deep cross-modal hamming retrieval. In ICCV , pages 2198-2207. IEEE, 2021.

Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from transformers. In EMNLP/IJCNLP (1) , pages 5099-5110. Association for Computational Linguistics, 2019.

Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , pages 9694-9705, 2021.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV (30) , volume 12375 of Lecture Notes in Computer Science , pages 104-120. Springer, 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR . OpenReview.net, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778. IEEE Computer Society, 2016.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1) . The Association for Computer Linguistics, 2016.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR , abs/1706.03762, 2017.

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Universal sentence encoder for english. In EMNLP (Demonstration) , pages 169-174. Association for Computational Linguistics, 2018.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In ICCV , pages 2425-2433. IEEE Computer Society, 2015.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV (2) , volume 9906 of Lecture Notes in Computer Science , pages 69-85. Springer, 2016.

Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. CoRR , abs/1811.00491, 2018.

Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. CoRR , abs/1901.06706, 2019.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR , abs/1504.00325, 2015.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255. IEEE Computer Society, 2009.

Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv:1312.6082 , 2013.

Jiafeng Wang, Zhaoyu Chen, Kaixun Jiang, Dingkang Yang, Lingyi Hong, Yan Wang, and Wenqiang Zhang. Boosting the transferability of adversarial attacks with global momentum initialization. CoRR , abs/2211.11236, 2022.

Nathan Inkawhich, Kevin J. Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. In NeurIPS , 2020.

Nathan Inkawhich, Kevin J. Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. In ICLR . OpenReview.net, 2020.

Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR , pages 12873-12883. Computer Vision Foundation / IEEE, 2021.

Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR (Poster) . OpenReview.net, 2017.

Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pretrained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 2043-2059. IEEE, 2022.

A. Details of VL Models

This section introduces the details of VL models involved in our experiment, including ViLT, BLIP, Unitab, and OFA.

ViLT . We first select ViLT [1] as the encoder-only VLmodel because of its succinct structure and prominent performance on multiple downstream tasks. Given an input image I ∈ R H × W × 3 and a sentence T , ViLT yields M image tokens using a linear transformation on the flattened image patches, where each token is a 1D vector and M = HW P 2 for a given patch resolution ( P, P ) . Word tokens are encoded through a Byte-Pair Encoder (BPE) [44] and a word-vector linear projection. Then, tokens of two modalities and a special learnable token ⟨ cls ⟩ are concatenated. By attending visual and text tokens and a special token ⟨ cls ⟩ in a Transformer encoder with twelve layers, the output feature from the ⟨ cls ⟩ token is fed into a task-specific classification head for the final output. Taking the VQA task as an example, the VQA classifier adopts a linear layer to output a vector with H s elements, where H s is the number of all possible choices in the closed answer set of the VQA task. The final output is obtained through the element with the highest response in the vector.

BLIP . The BLIP model also adopts an encoder-only structure. Specifically, BLIP first encodes an image through a twelve-layer vision transformer ViT-B/16 [42], and generates word token embeddings through the BPE and a linear layer. Next, the image features and word token embeddings are then fed into a multimodal encoder. The structure of the multimodal encoder is the same as a twelve-layer transformer decoder, where each layer contains a self-attention module, a crossattention module and a feed-forward module. In each layer, the multimodal encoder first accepts the word token embeddings as input and attends to them through the self-attention module. Then, the updated embedding will fuse with image features through the cross-attention module. Finally, the output from the multimodal encoder will be fed into different projection heads for downstream tasks, just like the ViLT model.

Unitab. Unitab adopts an encoder-decoder framework. It first embeds text T via RoBERT a [45] and flats features after encoding image I through ResNet [43]. The attached visual and text token features are then fed into a standard Transformer network [46] with six encoder layers and six decoder layers. Finally, the sequence predictions [ ⟨ ans 1 ⟩ , ⟨ ans 2 ⟩ , · · · , ⟨ end ⟩ ] are obtained autoregressively through a projection head. The network stops regressing when an end token ⟨ end ⟩ appears. For different tasks, the output tokens may come from different pre-defined vocabularies. Given the REC task as an example, four tokens [( ⟨ loc x 1 ⟩ , ⟨ loc x 2 ⟩ ) , ( ⟨ loc x 3 ⟩ , ⟨ loc x 4 ⟩ )] will be selected from the location vocabulary, which forms the coordinate of a bounding box. As a result, these models can handle both text and grounding tasks.

OFA. OFA also adopts an encoder-decoder structure. Different from Unitab, it adopts the BPE to encode text and extends the linguistic vocabulary by adding image quantization tokens [58] ⟨ img ⟩ for synthesis tasks. Note that the main difference between OFA and Unitab lies in their pre-training and fine-tuning strategies rather than the model structure . For example, in the pre-training process, Unitab focuses on learning alignments between predicted words and boxes through grounding tasks, while OFA captures more general representations through multi-task joint training that includes both single-modal and multimodal tasks. Overall, OFA outperforms Unitab in terms of performance improvement.

B. Dataset and Implementation

B.1 Tasks and Datasets

To verify the generalization ability of our proposed VLATTACK, we evaluate a wide array of popular vision language tasks summarized in Table 5. Specifically, the selected tasks span from text understanding (visual reasoning, visual entailment, visual question answering) to image understanding (image classification, captioning) and localization (referring expression comprehension).

For each dataset, we sample 5K correctly predicted samples in the corresponding validation dataset to evaluate the ASR performance. All validation datasets follow the same split settings as adopted in the respective attack models. Because VQA is a multiclass classification task, we select a correct

prediction only if the prediction result is the same as the label with the highest VQA score 6 , and regard the label as the ground truth in Eq. (1). In the REC task, a correct prediction is considered when the Intersection-over-Union (IoU) score between the predicted and ground truth bounding box is larger than 0.5. We adopt the same IoU threshold as in Unitab [4] and OFA [5].

B.2 Implementation Details

For the perturbation parameters of images, we follow the setting in the common transferable image attacks [18, 19] and set the maximum perturbation σ i of each pixel to 16/255 on all tasks except REC. Considering that even a single coordinate change can affect the final grounding results to a great extent, the σ i of the REC task is 4/255 to better highlight the ASR differences among distinct methods. The total iteration number N and step size are set to 40 and 0.01 by following the projected gradient decent method [30], and N s is 20. For the perturbation on the text, the semantic similarity constraint σ s is set to 0.95, and the number of maximum modified words is set to 1 by following the previous text-attack work [15, 24] to ensure the semantic consistency and imperceptibility. All experiments are conducted on a single GTX A6000 GPU.

C. More Ablation Results

In Section 5.4, we conduct an ablation study to show the effectiveness of each module in our model design on VQA, VE, and REC tasks. Here, we conduct additional ablation experiments for the remaining tasks, including visual reasoning, image captioning, and image classification.

IE

60

ASR (%)

TE

50

BSA

BSA+BA

40-

BSA+BA+Q

VLATTACK

6.0

IE

Score

5.0-

TE

Caption S

BSA

4.0-

BLEU@4

3.0-

CIDEr

2.0

Figure 9 shows the results of the ablation study on the VR task using the ViLT model. We can observe that only using the image encoder results in significantly low ASR. However, by combining it with the Transformer encoder (TE), BSA can achieve a high ASR. These results show the reasonableness of considering two encoders simultaneously when attacking the image modality. The result of BSA+BA demonstrates the usefulness of attacking the text modality. Although BSA+BA+Q outperforms other approaches, its performance is still lower than that of the proposed VLATTACK. This comparison proves that the proposed iterative cross-search attack (ICSA) strategy is effective for the multimodal attack again.

30

Figure 10 shows the results of the image captioning task using the OFA model. Because the image captioning task only accepts a fixed text prompt for prediction, we only perturb the image and report the results on IE, TE, and BSA. For this task, we report BLEU@4 and CIDEr scores. The lower, the better . We can observe that the proposed BSA outperforms IE and TE, indicating our model design's effectiveness.

Figure 11 shows the results of the image classification task using the OFA model. The experiment is conducted on the ImageNet1K dataset. Similar to the image captioning task, we only attack images, and compare the results of IE, TE and BSA. The evaluation metric for this task is ASR. The higher, the better . We can have the same observations with other ablation studies, where attacking both encoders outperforms attacking a single encoder.

Figure 11: OFA-classification.

D. Different Optimization Methods

VLATTACK can be easily adapted to various optimization methods in image attacks. To demonstrate the generalizability of our method, we replace the projected gradient decent [30] in VLATTACK with Momentum Iterative method (MI) [55] and Diverse Input attack (DI) [33] since they have shown better performance than traditional iterative attacks [30, 59]. The replaced methods are denoted by BSA MI , and VLATTACK MI using MI, BSA DI and VLATTACK DI using DI, respectively. Experiments are developed on ViLT and Unitab. Results are shown in Table 6. Using MI and DI optimizations, BSA MI and BSA DI still outperform all baselines displayed in Table 1 in the main manuscript. Also, VLATTACK MI and VLATTACK DI outperform the image attack method BSA MI and BSA DI with an average ASR improvement of 9.70 % and 9.29 % among all datasets. The gain of performance demonstrates that the proposed VLATTACK can be further improved by combining with stronger gradient-based optimization schemes.

E. Case Study

E.1 How does VLATTACK generate adversarial samples?

The proposed VLATTACK aims to attack multimodal VL tasks starting by attacking single modalities. If they are failed, VLATTACK uses the proposed interactive cross-search attack (ICSA) strategy to generate adversarial samples. In this experiment, we display the generated adversarial cases from different attack steps, including the image modality in Figure 12, the text modality in Figure 13, and the multimodal attack in Figure 14.

Single-modal Attacks (Section 4.1) . VLATTACK first perturbs the image modality using the proposed BSA and only outputs the adversarial image if the attack is successful (Algorithm 1 lines 1-5). As shown in Figure 12, only attacking the image modality, VLATTACK can generate a successful adversarial sample to fool the downstream task. Then, VLATTACK will stop. Otherwise, it will perturb the text through BERT-Attack (B&A) and use the clean image as the input, which is illustrated in Figure 13 (Algorithm 1, lines 6-15). During the text attack, B&A will generate multiple candidates by replacing the synonyms of a word. Since the length of text sentences is very short in the VL datasets, we only replace one word each time. From Figure 13, we can observe that B&A first replaces 'kid' with its synonym 'child', but this is not an adversarial sample. B&A then moves to the next word 'small' and uses its synonym 'cute' as the perturbation. By querying the black-box downstream task model, VLATTACK successes, and the algorithm will stop. Multimodal Attack

Do the elephants have

Do the elephants have

long tusks?

long tusks?

×

a small kid

a small child

a cute kid

with ballons.

With ballons. X .--- with ballons

11

12

12

13

N=6

N=6

×

×

×

What material

iWhatmaterials

What materialRank

What material

What material

Whatmaterials

Whatmaterials

is the table  B&A is the table

is the table

is the table

is the table

is the table

is the table

made of?.

built of?X

×

makeof?x

（3）

make of?

make of?

made of ?

madeof?

T

T2

(Section 4.2) . If the single-modal attack fails, VLATTACK moves to the multimodal attack by iteratively cross-updating image and text perturbations, where image perturbations are added through BSA, and text perturbations are added according to the semantic similarity. The cross-updating process is repeated until an adversarial image-text pair is found (Algorithm 1, lines 16-24). Figure 14 shows an example. In step 1, VLATTACK fails to attack the image modality and outputs a perturbed image denoted as I ′ 1 . In step 2, VLATTACK also fails to attack the text modality and outputs a list of text perturbations T . VLATTACK has to use the multimodal attack to generate adversarial samples in step 3. It first ranks the text perturbations in T according to the semantic similarity between the original text and each perturbation. The ranked list is denoted as { ˆ T ′ 1 , · · · , ˆ T ′ K } . Then it equally allocates the iteration number of the image attack to generate the image perturbations iteratively. In Figure 14, this number is 6, which means we run BSA with the budget 6 to generate a new image perturbation.

VLATTACK takes the pair ( I ′ 1 , ˆ T ′ 1 ) as the input to query the black-box downstream model, where ˆ T ′ 1 = ' What material is the table make of? '. If this pair is not an adversarial sample, then the proposed ICSA will adopt BSA to generate the new image perturbation I ′ 2 . The new pair ( I ′ 2 , ˆ T ′ 1 ) will be checked again. If it is still not an adversarial sample, VLATTACK will use the next text perturbation ˆ T ′ 2 = ' What materials is the table made of? ' and the newly generated image perturbation I ′ 2 as the input and repeat the previous steps until finding a successful adversarial sample or using up all K text perturbations in T . VLATTACK employs a systematic strategy for adversarial attacks on VL models, sequentially targeting single-modal and multimodal perturbations to achieve successful adversarial attacks.

E.2 Case Study on Different Tasks

We also provide additional qualitative results from Figure 15 to Figure 20 for experiments on all six tasks. For better visualization, we display the adversarial and clean samples side by side in a single column. By adding pixel and word perturbations, the fidelity of all samples is still preserved, but predictions are dramatically changed. For instance, in the image captioning task of Figure 19, all generated captions show no correlation with the input images. Some texts may even include replacement Unicode characters, such as ' \ ufffd', resulting in incomplete sentence structures.

Clean

Adversarial

Clean

Adversarial

BroadStSt

OMZ

BroadStSt

OMZ

Clean

Adversarial

Clean

Adversarial

The first burger,

a baseball player with

Thehalf part ofthe sandwichcloseto the carrots

a glasses of orange juice

lady touchingthehorse

immediatelynexttothe

：number3onhis

metal blade handle.

uniform.

CKSCOFEE

Clean

Othello

Adversarial

Othello

a collage of different

apaintingofawoman

apictureofahuman

apaintingofaskullwith

apaintingofagroupof

images of aperson.

withatree

headwithapuzzle

a THCCC\ufffd

zombies anda jigsaw

Tufffdufffd

puzzle

Clean

pembroke

bonnet

planetarium

coralreef

Adversarial

dingo

christmasstocking

chow

bowtie

jigsaw puzzle

F. Discussion: BSA v.s. BadEncoder

We notice that the idea of our proposed BSA is somehow similar to the BadEncoder [60] method, as they all use the cosine similarity as the optimization target. However, the proposed BSA is different from or better than BadEncoder.

(1) BadEncoder only utilizes the final output feature vectors from the whole encoder, ignoring the outputs from the intermediate layers/blocks. However, BSA calculates fine-grained similarity scores. As shown in Eq. 2 of our original paper, we distinguish the outputs from image and Transformer encoders. Such a design can modify the low-level vision features and the high-level cross-modal representations. In our setting, we attack fine-tuned models of a wide diversity of downstream tasks, including but not limited to image classification tasks like Badencoder. The parameters of these task-specific models are fully finetuned on distinct datasets, and the output representations of the encoder significantly change accordingly. Thus, instead of only attacking the output feature from the last layer like BadEncoder, perturbing each intermediate feature representation from each encoder and each block can enhance the attack performance. This statement is also verified in Section 5.5, where the ASR score of BSA is higher than only attacking a single encoder.

(2) The motivation for adopting the cosine distance is different. In BadEncoder, CLIP uses cosine similarity as a loss function to calculate distances for positive/negative image text pairs, which is motivated by the pre-training strategy of CLIP. However, we adopt cosine similarity because the fine-grained token representations attend to each other in the inner product space, which is inspired by the mechanism design of the Transformer structure. Therefore, the proposed BSA method greatly differs from BadEncoder in the above two aspects.

G. Limitations

The limitations of our work can be summarized from the following two aspects. On the one hand, in our current model design, for the text modality, we directly apply the existing model instead of developing a new one. Therefore, there is no performance improvement on tasks that only accept texts as input, such as text-to-image synthesis. On the other hand, our research problem is formulated by assuming the pre-trained and downstream models share similar structures. The adversarial transferability between different pre-trained and fine-tuned models is worth exploring, which we left to our future work.

H. Broad Impacts

Our research reveals substantial vulnerabilities in vision-language (VL) pre-trained models, underlining the importance of adversarial robustness cross pre-trained and fine-tuned models. By exposing these vulnerabilities through the VLATTACK strategy, we offer inspiration for developing more robust models. Furthermore, our findings underscore the ethical considerations of using VL models in real-world applications, especially those dealing with sensitive information and big data. Overall, our work emphasizes the necessity of balancing performance and robustness in VL models, with implications extending across computer vision, natural language processing, and broader artificial intelligence applications.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new adversarial attack framework, VLATTACK, to evaluate the robustness of large vision-language models (VLMs). The paper introduces a two-step adversarial attack approach. The first step involves attacking each modality (image and text) independently, and the second step employs a cross-search attack strategy to iteratively query the fine-tuned model for perturbation update. The paper conducts experiments on multiple pre-trained models that are fine-tuned for downstream vision tasks.\n\nSTRENGTHS:\n1. The paper is well-written with a clear presentation that demonstrates the proposed framework.\n\n2. The VLATTACK, which is a two-step adversarial attack framework, involves both white-box gradient attack and black-box query attack. This idea is interesting with practical implementation.\n\n3. The evaluation takes account of multiple types of pre-trained models, and the BSA part of the attack overperforms SOTA image-space adversarial baselines.\n\nWEAKNESSES:\n1. The black-box setting is questionable. Although the adversary only has black-box access to the downstream fine-tuned model, the first stage of attack has white-box access to widely-used foundations like ViLT. There is a high chance that the victim model shares mutual information (e.g., has knowledge of the same vision-language dataset / has the same model architecture) with the white-box pre-trained model. It will be beneficial if the authors elaborate more on the source of transferability.\n2. Section 4.2 (ICSA) claims to be under the black-box setting. However, Lines 228-229 describe the process to update the perturbation by optimizing the attack loss of Eq. (2). Authors should write clearly how the gradient is obtained, and what model is white-box during this step.\n3. Section 4.1 Text-Attack part uses clean image $\\mathbf{I}$ instead of generated adversary $\\mathbf{I'}$ for text optimization. Authors claim that this is to avoid unnecessary modification. However, using $\\mathbf{I'}$ can possibly ease the difficulty of textual search and increase the success rate of adversarial textual optimization. Moreover, the authors should argue clearly what the unnecessary modification stands for.\n4. The success of the attack will require a sufficient number of queries to generate effective adversaries. It will be helpful if authors evaluate the query cost (e.g., time of inference/API call), and discuss attack effectiveness by the number of queries.\n\nQUESTIONS:\n1. Fig. 7 second column demonstrates that the model comprehension shifts from *A man in a purple shirt and **jeans*** to *A man in a purple shirt and **denim***. However, denim is the material pattern for the jeans shown in the image. The model prediction is still consistent with the image. It will be beneficial if the authors state clearly what this example stands for.\n\n2. Given the setting of this paper that we have (1) publicly available pre-trained models and (2) the clean dataset for future perturbation, I wonder if it is possible to first fine-tune the model on the clean dataset for the downstream task and then conduct the transfer-based attack? It will be beneficial for this paper to argue the infeasibility of such approaches.\n\n3. To summarize, with a fair paper presentation but several concerns stated in the weakness/question sections, I will rate it as a borderline accept at this stage. However, I look forward to the authors' response and I will consider revising the rating based on the soundness of the responses.\n\nLIMITATIONS:\nYes, the authors have stated the potential limitations of the work in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-15"}

comments: SUMMARY:
In this paper, the authors explore the adversarial vulnerability in visual language models. Specifically, a block-wise similarity attack is proposed to generate adversarial image examples, and the BERT-attack method is used for generating the adversarial text examples. The image and text pairs are perturbed by an iterative method with cross-search.

STRENGTHS:
1. This paper explores the black-box attack for vision language models, which is novel.

2. The proposed method can generate perturbed inputs to decrease the performance of the fine-tuned models on multi-modal and uni-modal datasets effectively.

WEAKNESSES:
1. One important criterion for a successful adversarial attack is that the perturbed image cannot be distinguished by humans, and the semantic meaning of the text adversarial examples should be the same as the original one. However, there needs to be more analysis of the semantic similarity between the generated adversarial examples and the original text. The perturbation of the images should be measured as well. Most importantly, the perturbation of the generated image-text pairs should be analyzed.

2. By applying the iterative way for generating adversarial image and text pairs, the attackers need to query the large vision language model more times. The time costs and the average query numbers for a successful attack should be provided.

QUESTIONS:
It is better to provide the analysis of the semantic similarity and the perturbation degree, as mentioned in Weaknesses 1.

Compared to other methods, could you provide a thorough analysis of the costs and average query times of the proposed framework?

LIMITATIONS:
I did not find a potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-27

comments: SUMMARY:
The paper presents VLAttack, which is a method for perturbing multimodal examples such that a multimodal model would get them wrong. VLAttack does not assume access to fine-tuned models but does assume access to foundation models that are used to create these downstream models. The authors argue that this level of access is reasonable because there are several real-world examples where the foundation model is available but downstream models are not. VLAttack makes models fail significantly more than similar adversarial attack algorithms across several datasets and tasks.

STRENGTHS:
As far as I am aware, the algorithm is novel (especially the visual perturbation part). And the authors showcase that it outperforms similar algorithms in the sense that it does not assume access to the fine-tuned model (which could be unreasonable) and it makes models get more errors.

WEAKNESSES:
Major issues:

It is great that VLAttack makes the models get higher error rates than other approaches, but crucially it is only a good result if VLAttack does not actually change the ground truth labels. It seems that you do not validate this with, e.g. crowd workers. Unless I missed it, the only validation that you do is presenting case studies of a few individual examples. Did you do a systematic validation?

Less important (but still issues):

More motivation would be nice. What scenarios are you envisioning for why VLAttack raises a safety concern? Self-driving car failures, etc.?

Can you make these adversarial perturbations useful? For example, can you train a model on them to get better performance?

QUESTIONS:
I tried to frame everything in the weaknesses section as questions.

LIMITATIONS:
Limitations seem adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on adversarially attacking the multimodal finetuned models without getting access to the finetuned weights. By utilizing the activations and the parameters of the open-accessed pretrained model, this work proposes a method called VLATTACK for creating the adversarial attack samples for downstream tasks. The creating method first attempts to learn image perturbations via block-wise similarity attack (BSA) strategy. If not succeed, the text attack using BertAttack will be facilitated. If the two single-modal attack both fail, VLATTACK will use cross-search attack (ICSA) method to construct multimodal disrupted adversarial samples. Experimental results show the effectiveness of VLATTACK on various cross-modal (image-text) tasks and model structures. Some detailed analysis on the adversarial samples are also conducted.\n\nSTRENGTHS:\n1. The experimental results are obtained based on various downstream tasks and pretrained models (in different architectures), which demonstrate the generalization ability of VLATTACK.\n2. The choice of pretrained models are open-sourced, making the results more reproducible.\n3. The performance of adversarial attack is good, compared with other typical and previous attacking methods.\n4. The case study and ablation study are both well-conducted, providing insights on how the VLATTACK can achieve better attacking performance.\n\nWEAKNESSES:\nGenerally speaking, I think this paper is a good work on the topic of adversarial attacking. If the following issues are addressed, I think it will be much better:\n1. Experiments or analysis on adversarial multimodal datasets: Some adversarially constructed datasets are proposed in previous works, based on the datasets used in this work, like Adversarial VQA. Does the created adversarial samples share some similarity with these datasets? I think some discussion or analysis on this research question can be performed and will be very welcomed.\n2. Evaluation soundness: For VQA and captioning, an extra human evaluation on whether the predictions under attack are actually wrong will be much recommended, considering the automatic metrics may make false-positives.\n3. Experiments on out-of-domain datasets: For OFA, we know the pretraining corpus include the downstream datasets (VQA, RefCOCO and MSCOCO), which may result in the adversarial samples easier to be obtained using the pretrained model. It would be very good if downstream datasets which not appear in pretraining OFA can be considered in experiments.\n\nQUESTIONS:\nHow would the performance of VLATTACK be affected under different model size (like OFA-base vs OFA-large) and image resolution ($224$ vs $480$)? I will be very happy if some insights can be provided.\n\nLIMITATIONS:\nThe authors have proposed some limitations in the appendix, including the text attacking methods, downstream task scope and explored model architectures, which I think is reasonable and leave space for future research.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a substitute black-box attack strategy called VLAttack to generate adversarial examples by perturbing both images and texts on pre-trained models and then transferring them to finetuned models.  At the image-modal level, they introduce a block-wise similarity attack (BSA) strategy to disrupt universal representations in the pre-trained model. At the multimodal level, they design an iterative cross-search attack (ICSA) to update adversarial image-text paris.\n\nSTRENGTHS:\n1. This paper focuses on an important question concerning the prevalent pretraining and finetuning paradigm with respect to adversarial robustness.\n\n2. The core idea of the proposed attack is clearly presented. \n\n3. The proposed VLAttack could potentially serve as a valuable baseline for examining the robustness of multimodal models.\n\nWEAKNESSES:\nIssues that affect my rating.\n\n1. Lack of technique contributions. The question asked by the authors is quite valuable. However, the proposed attack strategy, VLAttack, seems trivial.  Regarding the text modality, VLAttack directly employs the existing BertAttack[20]. For the image attack,  the classical PGD attack with iteration $N_{s}=20$ is adapted. The idea of the proposed BSA loss Eq.2 is straightforward and similar to that of BadEncoder [a] (see BadEncoder's Eq.2~5), except BadEncoder focuses on the backdoor attack. The cross-modality attack should be the most important part, where the two modality attacks boost each other is excepted. However, the proposed \"ICSA\" seems directly concatenate the image and text attacks. Did the author have attempted to jointly optimize these two attacks? \n\n[a] Jia, J., Liu, Y., & Gong, N. Z. (2022, May). Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP) (pp. 2043-2059). IEEE.\n\n2. Insufficient evaluation on popular multimodal models (CLIP, BLIP): I recommend that the authors extend their evaluation to include more prevalent multimodal models, such as CLIP or BLIP, as done in Co-attack[15] and BadEncoder[a]. This would enable readers and subsequent works to make more meaningful comparisons.\n\n3. Ambiguity in the fine-tuned model settings: Considering the significant performance difference between VLAttack and baselines in Table 1, it is unclear how the target models are fine-tuned. Are the pre-trained model parameters fixed while training a task-specific head, or are all parameters fine-tuned throughout the pre-trained model? Providing clarity on this aspect is crucial for evaluating the impact of the proposed attack. Additional details should be included to enhance understanding.\n\n4. Absence of essential ablation studies on BSA and ICSA: The effectiveness of BSA remains unclear, and a comparison with an attack on the pre-trained model's original loss could provide valuable insights. To showcase the efficacy of ICSA, a straightforward baseline could involve setting BSA's attack iteration $N_{s}=40$, without incorporating BERT-attack. The authors should consider conducting these ablation studies to better demonstrate their contributions.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}