SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proposed VLATTACK aims to attack multimodal VL tasks starting by attacking single modalities. If they are failed, VLATTACK uses the proposed interactive cross-search attack (ICSA) strategy to generate adversarial samples. In this experiment, we display the generated adversarial cases from different attack steps, including the image modality in Figure 12, the text modality in Figure 13, and the multimodal attack in Figure 14.
Single-modal Attacks (Section 4.1). VLATTACK first perturbs the image modality using the proposed BSA and only outputs the adversarial image if the attack is successful (Algorithm 1 lines 1-5). As shown in Figure 12, only attacking the image modality, VLATTACK can generate a successful adversarial sample to fool the downstream task. Then, VLATTACK will stop. Otherwise, it will perturb the text through BERT-Attack (B&A) and use the clean image as the input, which is illustrated in Figure 13 (Algorithm 1, lines 6-15). During the text attack, B&A will generate multiple candidates by replacing the synonyms of a word. Since the length of text sentences is very short in the VL datasets, we only replace one word each time. From Figure 13, we can observe that B&A first replaces "kid" with its synonym "child", but this is not an adversarial sample. B&A then moves to the next word "small" and uses its synonym "cute" as the perturbation. By querying the black-box downstream task model, VLATTACK successes, and the algorithm will stop. Multimodal Attack
Figure 12: An adversarial image from BSA.
Figure 14: An adversarial image-text pair from multimodal attack.
(Section 4.2). If the single-modal attack fails, VLATTACK moves to the multimodal attack by iteratively cross-updating image and text perturbations, where image perturbations are added through BSA, and text perturbations are added according to the semantic similarity. The cross-updating process is repeated until an adversarial image-text pair is found (Algorithm 1, lines 16-24). Figure 14 shows an example. In step 1, VLATTACK fails to attack the image modality and outputs a perturbed image denoted as  $\mathbf{I}_1'$ . In step 2, VLATTACK also fails to attack the text modality and outputs a list of text perturbations  $\mathcal{T}$ . VLATTACK has to use the multimodal attack to generate adversarial samples in step 3. It first ranks the text perturbations in  $\mathcal{T}$  according to the semantic similarity between the original text and each perturbation. The ranked list is denoted as  $\{\hat{\mathbf{T}}_1', \dots, \hat{\mathbf{T}}_K'\}$ . Then it equally allocates the iteration number of the image attack to generate the image perturbations iteratively. In Figure 14, this number is 6, which means we run BSA with the budget 6 to generate a new image perturbation.
VLATTACK takes the pair  $(\mathbf{I}_1', \hat{\mathbf{T}}_1')$  as the input to query the black-box downstream model, where  $\hat{\mathbf{T}}_1' = \text{"What material is the table make of?"}$ . If this pair is not an adversarial sample, then the proposed ICSA will adopt BSA to generate the new image perturbation  $\mathbf{I}_2'$ . The new pair  $(\mathbf{I}_2', \hat{\mathbf{T}}_1')$  will be checked again. If it is still not an adversarial sample, VLATTACK will use the next text perturbation  $\hat{\mathbf{T}}_2' = \text{"What materials is the table made of?"}$  and the newly generated image perturbation  $\mathbf{I}_2'$  as the input and repeat the previous steps until finding a successful adversarial sample or using up all  $K$  text perturbations in  $\mathcal{T}$ . VLATTACK employs a systematic strategy for adversarial attacks on VL models, sequentially targeting single-modal and multimodal perturbations to achieve successful adversarial attacks.
E.2 Case Study on Different Tasks
We also provide additional qualitative results from Figure 15 to Figure 20 for experiments on all six tasks. For better visualization, we display the adversarial and clean samples side by side in a single column. By adding pixel and word perturbations, the fidelity of all samples is still preserved, but predictions are dramatically changed. For instance, in the image captioning task of Figure 19, all generated captions show no correlation with the input images. Some texts may even include replacement Unicode characters, such as "\uffd", resulting in incomplete sentence structures.
Cean
Is the dog looking at the camera? Yes
What kind of animals on the pillow? Cat
What are the skiers holding in their hands? Ski poles
At which course would this Are the flags pointing left be served in a meal? Main or right? left
Adversarial
Is the dog staring at the camera? No
Figure 15: Additional quantitative results on visual question answering (VQA).
What kinds of animals on the pillow? Bear
What are the skiers having in their hands? Skis
At which course would this be delivered in a meal? Dinner
Are the flags pointing left or right? right
Cean
a dog is jumping close to dangerous rocks. Neutral
a dog is carrying a puppy home. Neutral
There are two dogs Entailment
The woman is wearing a dull dress Contradiction
A man is sitting inside. Contradiction
Aesaaee
a dog is jumping near to dangerous rocks. Contradiction
Figure 16: Additional quantitative results on visual entailment (VE).
a dog is carrying a pup home. Contradiction
There are 2 dogs Neutral
The girl is wearing a dull dress (text only) Neutral
A man is sitting inside. (image only) Neutral
Cannnnn
At least one dog is laying down. True
There is at least three dogs. False
the right pic has fewer then five animals. True
Adversarial
At least one dogs is laying down. False
Figure 17: Additional quantitative results on visual reasoning (VR).
False There is at least 3 dogs True
There is at least 3 dogs True
the right pic has less then five animals. False
Cean
The first burger, immediately next to the metal knife handle.
a glass of orange juice
woman touching the horse
Adversarial
The first burger, immediately next to the metal blade handle.
Figure 18: Additional quantitative results on referring expression comprehension (REC).
a baseball player with number three on his uniform.
a baseball player with number 3 on his uniform.
The half part of the sandwich nearest to the carrots
The half part of the sandwich close to the carrots
a glasses of orange juice
lady touching the horse
Cean
a blue and white street sign that reads othello.
two bowls that have food inside of them
a gray train riding on a track as people are walking
there are motor bikes that are parked on the street.
a group of cows looking at camera by trees
Adversarial
a collage of different images of a person.
Figure 19: Additional quantitative results on the image captioning task.
a painting of a woman with a tree
a picture of a human head with a puzzle
a painting of a skull with a THC C C\uffd T\uffd\ufffd
a painting of a group of zombies and a jigsaw puzzle
Ceeen
pembroke
Adversarial
dingo
bonnet
christmas stocking
Figure 20: Additional quantitative results on the image classification task.
leonberg
chow
planetarium
bow tie
F. Discussion: BSA v.s. BadEncoder
We notice that the idea of our proposed BSA is somehow similar to the BadEncoder [60] method, as they all use the cosine similarity as the optimization target. However, the proposed BSA is different from or better than BadEncoder.
(1) BadEncoder only utilizes the final output feature vectors from the whole encoder, ignoring the outputs from the intermediate layers/blocks. However, BSA calculates fine-grained similarity scores. As shown in Eq. 2 of our original paper, we distinguish the outputs from image and Transformer encoders. Such a design can modify the low-level vision features and the high-level cross-modal representations. In our setting, we attack fine-tuned models of a wide diversity of downstream tasks, including but not limited to image classification tasks like Badencoder. The parameters of these task-specific models are fully finetuned on distinct datasets, and the output representations of the encoder significantly change accordingly. Thus, instead of only attacking the output feature from the last layer like BadEncoder, perturbing each intermediate feature representation from each encoder and each block can enhance the attack performance. This statement is also verified in Section 5.5, where the ASR score of BSA is higher than only attacking a single encoder.
(2) The motivation for adopting the cosine distance is different. In BadEncoder, CLIP uses cosine similarity as a loss function to calculate distances for positive/negative image text pairs, which is motivated by the pre-training strategy of CLIP. However, we adopt cosine similarity because the fine-grained token representations attend to each other in the inner product space, which is inspired by the mechanism design of the Transformer structure. Therefore, the proposed BSA method greatly differs from BadEncoder in the above two aspects.
G. Limitations
The limitations of our work can be summarized from the following two aspects. On the one hand, in our current model design, for the text modality, we directly apply the existing model instead of developing a new one. Therefore, there is no performance improvement on tasks that only accept texts as input, such as text-to-image synthesis. On the other hand, our research problem is formulated by assuming the pre-trained and downstream models share similar structures. The adversarial transferability between different pre-trained and fine-tuned models is worth exploring, which we left to our future work.
H. Broad Impacts
Our research reveals substantial vulnerabilities in vision-language (VL) pre-trained models, underlining the importance of adversarial robustness cross pre-trained and fine-tuned models. By exposing these vulnerabilities through the VLATTACK strategy, we offer inspiration for developing more robust models. Furthermore, our findings underscore the ethical considerations of using VL models in real-world applications, especially those dealing with sensitive information and big data. Overall, our work emphasizes the necessity of balancing performance and robustness in VL models, with implications extending across computer vision, natural language processing, and broader artificial intelligence applications.