One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models

## Abstract

Abstract Vision-Language Pre-training (VLP) models have exhibited unprecedented capability in many applications by taking full advantage of the multimodal alignment. However, previous studies have shown they are vulnerable to maliciously crafted adversarial samples. Despite recent success, these methods are generally instance-specific and require generating perturbations for each input sample. In this paper, we reveal that VLP models are also vulnerable to the instance-agnostic universal adversarial perturbation (UAP). Specifically, we design a novel Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to achieve the attack. In light that the pivotal multimodal alignment is achieved through the advanced contrastive learning technique, we devise to turn this powerful weapon against themselves, i.e., employ a malicious version of contrastive learning to train the C-PGC based on our carefully crafted positive and negative image-text pairs for essentially destroying the alignment relationship learned by VLP models. Besides, C-PGC fully utilizes the characteristics of Vision-and-Language (V+L) scenarios by incorporating both unimodal and cross-modal information as effective guidance. Extensive experiments show that C-PGC successfully forces adversarial samples to move away from their original area in the VLP model’s feature space, thus essentially enhancing attacks across various victim models and V+L tasks. The GitHub repository is available at https://github.com/ffhibnese/CPGC_VLP_Universal_Attacks .

## 1 Introduction

Vision-Language Pre-training (VLP) models, including ALBEF (Li et al., ), TCL (Yang et al., ), and BLIP (Li et al., ), have recently demonstrated remarkable efficacy in a wide range of Vision-and-Language (V+L) tasks.
By self-supervised pre-training on large-scale image-text pairs, VLP models efficiently align cross-modal features and capture rich information from the aligned multimodal embeddings, thereby providing expressive representations for various applications.

Figure: Figure 1: Performance of existing UAP on the text retrieval. Surrogate models are ALBEF and BLIP. Results of our C-PGC are presented for comparison.
Refer to caption: x1.png

Adversarial attacks (Carlini & Wagner, ), which aim to deceive models during inference time, have attracted extensive attention due to their significant threat to security-critical scenarios (Eykholt et al., ). Recent studies have shown that VLP models are also vulnerable to adversarial samples. The pioneering work Co-Attack (Zhang et al., ) proposes the first multimodal attack that simultaneously perturbs both image and text modalities and displays excellent performance. However, Co-Attack only considers relatively easier white-box attacks where victim models are completely accessible.
To handle more practical black-box settings, subsequent studies propose various transferable adversarial samples generated on an available surrogate model to fool other inaccessible models.
Specifically, SGA (Lu et al., ) significantly improves the adversarial transferability through the set-level cross-modal guidance obtained from data augmentations.
Subsequently, TMM (Wang et al., ) proposes to jointly destroy the modality-consistency features within the clean image-text pairs and include more modality-discrepancy features in the perturbations to further enhance transferability.
While existing methods have achieved great success, they are all instance-specific and need to generate a perturbation for each input pair, which results in substantial computational overhead. Meanwhile, universal adversarial attacks, as an efficient instance-agnostic approach that uses only one Universal Adversarial Perturbation (UAP) to conduct attacks, have not been fully investigated for VLP models.
This naturally leads to a question, *is it possible to design a universal adversarial perturbation that can effectively deceive VLP models across various image-text pairs?*

Motivation.
To answer this question,
we make an intuitive attempt to transplant existing renowned approaches UAP (Moosavi-Dezfooli et al., ) and GAP (Poursaeed et al., ) to launch attacks on several VLP models by
maximizing the distance between the embeddings of the adversarial image and its matched texts (Lu et al., ).
Unfortunately, Figure 1 demonstrates that these methods yield unsatisfactory attack success rates (ASR), especially for black-box attacks. Empirically, this failure stems from their narrow focus on the image modality, disregarding the other modality and the multimodal information that plays a pivotal role in VLP models.
To overcome this challenge, we revisit the VLP models’ basic training paradigm and emphasize that regardless of the downstream V+L tasks, the outstanding performance achieved by these models is heavily reliant on the successful vision-language feature alignment, which draws the embedding of matched image-text pairs closer while distancing those of non-matched pairs.
In light of this consideration, we argue that the key core of an effective universal adversarial attack is to obtain a UAP that can fundamentally destroy this learned alignment relationship to mislead VLP models into making incorrect decisions.
In addition, Fig. 1 also shows that the generator-based approach GAP consistently achieves superior ASR than UAP, confirming the validity of the generative attack paradigm (Hashemi et al., ).

Based on these insights, we propose a generative UAP framework that learns a Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to launch attacks on VLP models (see Fig. 2).
To essentially destroy the multimodal alignment, we devise to utilize VLP models’ most powerful weapons to attack against themselves, i.e., utilize the contrastive learning mechanism to train the generator using our maliciously constructed image-text pairs that completely violate the correct image-text matching relationship, to produce perturbation that pushes the embedding of matched pairs apart while pulling those of non-matched ones together.
Inspired by the multimodal characteristics of V+L scenarios, we modify the generator’s architecture to incorporate cross-modal knowledge through the advanced cross-attention mechanism for better guidance.
In addition, we also consider the intra-modal influence and introduce a unimodal distance loss to further enhance the attacks.
Since previous studies (Zhang et al., ; Lu et al., ) achieve impressive improvements via multimodal perturbation, we are motivated to generate UAP for both images and texts to utilize the synergy between different modalities.
Our contributions can be summarized as follows:

- •
We design a perturbation generator conditioned with cross-modal knowledge to produce effective UAP added to both image and text modalities for universal adversarial attacks on VLP models.
- •
We propose a novel training framework that incorporates both unimodal and multimodal guidance to contrastively train the generator using our meticulously constructed positive and negative pairs.
- •
Extensive experiments on 6 various VLP models across different V+L tasks reveal that our method achieves outstanding white-box performance and black-box transferability in different scenarios.

## 2 Related Work

### 2.1 Vision-Language Pre-training Models

VLP models are pre-trained on massive image-text pairs to learn the semantic correlations across modalities and serve diverse multimodal user demands (Chen et al., ; Du et al., ). We next illustrate the basis of VLP models from multiple perspectives.

Architectures. Based on the ways of multimodal fusion, the architectures of VLP models can be classified into two types: *single-stream* and *dual-stream architectures*. Single-stream architectures (Li et al., ; Chen et al., ) directly concatenate the text and image features, and calculate the attention in the same Transformer block for multimodal fusion. On the contrary, dual-stream architectures (Radford et al., ; Li et al., ) separately feed the text and image features to different Transformer blocks and leverage the cross-attention mechanism for multimodal fusion.

Pre-training Objectives. The pre-training objectives for VLP models mainly include *masked features completion*, *multimodal features matching*, and *specific downstream objectives*. Masked features completion (Chen et al., ) encourages VLP models to predict the deliberately masked tokens using the remaining unmasked tokens during pre-training.
Multimodal features matching (Li et al., ) pre-trains VLP models by learning to precisely predict whether the given image-text pairs are matched.
Specific downstream objectives (Anderson et al., ) directly utilize the training objectives of downstream tasks (e.g., visual question answering) for pre-training VLP models.

Downstream Tasks. In this paper, we mainly consider the following multimodal downstream tasks: (1) Image-text retrieval (ITR) (Wang et al., ): finding the most matched image for the given text and vice versa, including image-to-text retrieval (TR) and text-to-image retrieval (IR). (2) Image caption (IC) (Bai & An, ): generating the most suitable descriptions for the given image. (3) Visual grounding (VG) (Hong et al., ): locating specific regions in the image that correspond with the given textual descriptions. (4) Visual entailment (VE) (Xie et al., ): analyzing the input image and text and predicting whether their relationship is entailment, neutral, or contradiction.

### 2.2 Adversarial Attacks

Instance-specific Attacks on VLP Models.
The adversarial robustness of VLP Models has already become a research focus. Early works (Kim & Ghosh, ; Yang et al., ) impose perturbations only on single modality and lack cross-modal interactions when attacking multimodal models. To address this issue, Co-Attack (Zhang et al., ) conducts the first multimodal white-box attacks on VLP models.
On the basis of Co-Attack, Lu et al. () extend the attacks to more rigorous black-box settings and propose SGA, which utilizes set-level alignment-preserving argumentations with carefully designed cross-modal guidance.
However, Wang et al. () points out that SGA fails to fully exploit modality correlation, and proposes TMM to better leverage cross-modal interactions by tailoring both the modality-consistency and modality-discrepancy features. Nonetheless, these methods are all instance-specific and need to craft perturbations for each input pair.

Universal Adversarial Examples. Universal adversarial attacks (Moosavi-Dezfooli et al., ; Mopuri et al., ) aim to deceive the victim model by exerting a uniform adversarial modification to all the benign samples.
These attacks save the redundant procedures of redesigning perturbations for each input sample and are consequently more efficient than traditional attack strategies.
Generally, universal adversarial attacks can be categorized into optimization-based methods (Moosavi-Dezfooli et al., ; Wang et al., ; Liu et al., ) and generation-based methods (Hayes & Danezis, ; Gao et al., ; Anil et al., ). Benefiting from the powerful modeling abilities of generative models, generation-based methods are more versatile and can produce more natural samples than optimization-based ones. In this paper, we explore universal adversarial attacks on VLP models and manage to generate UAP with excellent attack effects and high transferability.

## 3 Universal Multimodal Attacks

In this section, we first present the problem statement of universal adversarial attacks on VLP models. Next, we introduce the overview of our framework. Finally, we illustrate the detailed design of each proposed technique and summarize the training objective and paradigm of C-PGC.

Figure: Figure 3: An overview of our proposed universal adversarial attack. Benefiting from the well-designed unimodal distance loss $\mathcal{L}_{Dis}$ and multimodal contrastive loss $\mathcal{L}_{CL}$, the conditional generator learns rich knowledge from features of different modalities and thus produces $\delta_{v}$ and $\delta_{t}$ of superior generalization ability across diverse models and downstream tasks.
Refer to caption: x3.png

### 3.1 Problem Statement

We define an input image-text pair as $(v,t)$ and denote $\bm{e}_{v}$ and $\bm{e}_{t}$ as the image and text embedding encoded by the image encoder $f_{I}(\cdot)$ and text encoder $f_{T}(\cdot)$ of the targeted VLP model $f(\cdot)$. Let $\mathcal{D}_{s}$ be an available dataset consisting of image-text pairs collected by a malicious adversary. The attack objective is to utilize $\mathcal{D}_{s}$ to train a generator $G_{w}(\cdot)$ that is capable of producing a powerful pair of universal image-text perturbations $(\delta_{v},\delta_{t})$ that can affect the vast majority of test dataset $\mathcal{D}_{t}$ to fool models into making incorrect decisions. Formally, the attack goal can be formulated as:

$$ $\begin{split}\mathcal{T}(f(v+\delta_{v},t\oplus\delta_{t}))\neq y,\;\text{s.t. }\ \|\delta_{v}\|_{\infty}\leq\epsilon_{v},\|\delta_{t}\|_{0}\leq\epsilon_{t}, \end{split}$ (1) $$

where $\mathcal{T}(\cdot)$ denotes the operation that uses the output V+L features to obtain the final predictions, $\oplus$ indicates the text perturbation strategy (Zhang et al., ; Lu et al., ) that replaces certain important tokens of the original sentence with crafted adversarial words, and $y$ is the correct prediction of the considered V+L task.
To ensure the perturbation’s imperceptibility, we constrain the pixel-level image perturbation with $l_{\infty}$ norm of a given budget $\epsilon_{v}$.
The textual perturbation is token-level and the stealthiness is accordingly constrained by the number of modified words $\epsilon_{t}$. Since altering words in a natural sentence can be easily noticed or detected, we apply a rigorous restriction that permits only a single word to be substituted ($\epsilon_{t}=1$). On the premise of imperceptibility, the attacker attempts to generalize the crafted UAP to a wider range of test data and victim models.

### 3.2 Overview of the Proposed Framework

As depicted in Fig. 3, we adopt the multimodal perturbation strategy and generate perturbations on both image and text modalities for enhanced attacks.
Given the similarity between the workflows for image and text, we then take the image attacks as an example to illustrate the proposed framework.

Firstly, a fixed noise $z_{v}$ is randomly initialized and subsequently fed into the conditional generator.
For each image $v$ and its descriptions $\mathbf{t}$, the generator $G_{w}(\cdot)$ then translates the input noise $z_{v}$ into the adversarial perturbation $\delta_{v}$ that is of the same size as the image $v$. During the generation, the network $G_{w}$ additionally benefits from cross-modal information by integrating the textual embedding, i.e., $\delta_{v}=G_{w}(z_{v};f_{T}(\mathbf{t}))$.
Next, the generated adversarial noise $\delta_{v}$ is injected into the clean image to obtain the adversarial image via $v_{adv}=v+\delta_{v}$.
To better guide the training process, we design two efficient unimodal and multimodal losses as our optimization objectives.
Unimodal loss is straightforward and aims to push the adversarial images away from the clean images in the latent embedding space, while multimodal loss is based on contrastive learning and uses our manually constructed positive and negative samples to effectively destroy the image-text matching relationship obtained from feature alignment.
Once we finish training the C-PGC using the proposed losses, the input fixed noise is transformed into a UAP that is of great generalization and transferability.

### 3.3 Detailed Design of C-PGC

Next, we provide a detailed introduction to each of the proposed designs. Note that we primarily discuss the image attack as an example, given that the design of the text attack is completely symmetrical. The pseudocode of the training procedure is provided in Appendix A.

Perturbation Generator with Cross-modal conditions. Previous generative universal attacks (Gao et al., ; Anil et al., ) have shown excellent efficacy in fooling the discriminative models. Nevertheless, since existing generative attacks are limited to a single modality, directly utilizing the off-the-shelf generators might fail to leverage the multimodal interactions in these special V+L scenarios.
To address this limitation, we additionally introduce cross-modal embeddings as auxiliary information to further facilitate the process of perturbation generation. Specifically, we modify the existing generator’s architecture by adding several cross-attention modules that have been proven effective in tasks with variable input modalities.
The obtained textual embeddings $\bm{e}_{t}$ encoded by $f_{T}(\cdot)$ are then incorporated into our generator through:

$$ $\begin{split}Q=\bm{h}_{t}W_{q},K=&\ \bm{e}_{t}W_{k},V=\bm{e}_{t}W_{v},\\ Attention(Q,K,V)&=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot V, \end{split}$ (2) $$

where $\bm{h}_{t}\in\mathbb{R}^{B\times d_{\alpha}}$ is the flattened intermediate features within the generator, $W_{q}\in\mathbb{R}^{d_{\alpha}\times d}$, $W_{k}\in\mathbb{R}^{512\times d}$, $W_{v}\in\mathbb{R}^{512\times d}$ are optimized parameters in the attention modules.

Multimodal Contrastive Loss. The preceding analysis regarding the failures of existing UAP attacks encourages us to design a loss function that can guide the generated UAP to break the learned multimodal feature alignment.
Motivated by the fact that contrastive learning underpins the cross-modal alignment, we advocate leveraging this mechanism to attack VLP models themselves by contrastively training our C-PGC to essentially disrupt the benign alignment relationship.
Concretely, we adopt the widely recognized InfoNCE (He et al., ) as our basic contrastive loss.

To establish the contrastive paradigm, we first define the adversarial image $v_{adv}$ as the anchor sample. Besides, it is also necessary to construct an appropriate set of positive and negative samples.
Based on the fundamental objective of our attacks, it is natural that the originally matched text descriptions set $\mathbf{t}=\{t_{1},t_{2}\dots,t_{M}\}$ can be employed as negative samples $\mathbf{t}_{neg}$ to increase the discrepancy of matched image-text in the feature space of VLP models.
To further push the adversarial image $v_{adv}$ away from its correct text descriptions, we propose utilizing multiple texts whose embeddings differ significantly from that of the original clean images $v$ as the positive samples $\mathbf{t}_{pos}$. Specifically, we randomly sample a batch of text sets from $\mathcal{D}_{s}$ and select the text set with the farthest feature distances from the current image $v$ as positive samples, i.e., $\mathbf{t}_{pos}=\{t_{1}^{\prime},t_{2}^{\prime}\dots,t_{K}^{\prime}\}$. Moreover, we utilize data augmentations that resize the clean image $v$ into different scales and apply random Gaussian noise to acquire a more diverse image set $\mathbf{v}=\{v_{1},v_{2}\dots,v_{S}\}$ for set-level guidance Lu et al. (). With the augmented image set and these well-constructed positive and negative samples, the multimodal contrastive loss $\mathcal{L}_{CL}$ can be formulated as:

$$ $\mathcal{L}_{CL}=log\Bigg{(}\frac{{\underset{i=1}{\overset{S}{\sum}}}{ \underset{j=1}{\overset{M}{\sum}}}e^{d(v_{i}+\delta_{v},t_{j})/\tau}}{{ \underset{i=1}{\overset{S}{\sum}}}{\underset{j=1}{\overset{M}{\sum}}}e^{d(v_{i }+\delta_{v},t_{j})/\tau}+{\underset{i=1}{\overset{S}{\sum}}}{\underset{j=1}{ \overset{K}{\sum}}}e^{d(v_{i}+\delta_{v},t_{j}^{\prime})/\tau}}\Bigg{)},$ (3) $$

where $\mathbf{\delta}_{v}$ is the universal image perturbation, $\tau$ denotes the temperature parameter and $d(v,t)=Sim(f_{I}(v),f_{T}(t))$, where $Sim(\cdot,\cdot)$ represents the cosine similarity measurement.

Unimodal Distance Loss. Apart from the multimodal guidance, we also consider the unimodal influence by directly pushing adversarial images away from their initial visual semantic area to further enhance attack effects.
Similarly, to acquire set-level guidance, the input image $v$ is initially resized to different scales and then added with Gaussian noise to generate the augmented image set $\mathbf{v}=\{v_{1},v_{2}\dots,v_{S}\}$. Then, we craft the adversarial image through $v_{adv}=v+\delta_{v}$ and process $v_{adv}$ with the same augmentation operation to obtain the adversarial image set $\mathbf{v}_{adv}=\{v_{1}^{adv},v_{2}^{adv}\dots,v_{S}^{adv}\}$.
Finally, we minimize the negative Euclidean distance between the embeddings of adversarial images and clean images to optimize the UAP generator. Formally, the loss $\mathcal{L}_{Dis}$ is formulated as:

$$ $\mathcal{L}_{Dis}=-{\underset{i=1}{\overset{S}{\sum}}}{\underset{j=1}{\overset {S}{\sum}}}\|f_{I}(v_{i}^{adv})-f_{I}(v_{j})\|_{2}.$ (4) $$

Taking advantage of the unimodal set-level guidance, $\mathcal{L}_{Dis}$ ensures an effective optimization direction during the generator training and further improves the attack effectiveness of our UAP.

Training Objective. With the two well-designed loss terms $\mathcal{L}_{Dis}$ and $\mathcal{L}_{CL}$, the overall optimization objective of our conditional generator concerning image attacks can be formulated as:

$$ $\min\limits_{w}\mathbb{E}_{(v,\mathbf{t})\sim\mathcal{D}_{s},\mathbf{t}_{pos} \sim\mathcal{D}_{s}}(\mathcal{L}_{CL}+\lambda\mathcal{L}_{Dis}),\;\text{s.t. } \|G_{w}(z_{v};f_{T}(\mathbf{t}))\|_{\infty}\leq\epsilon_{v},$ (5) $$

where $\lambda$ is the pre-defined hyperparameter to balance the contributions of $\mathcal{L}_{CL}$ and $\mathcal{L}_{Dis}$. By training the network using the proposed loss function based on the data distribution of the multimodal training dataset $D_{s}$, the generator is optimized to produce UAP that can push the features of mismatched image-text pairs together while pulling the embeddings of the matched ones apart, thereby learning a UAP with excellent generalization ability and adversarial transferability.

Text Modality Attacks.
In textual attacks, the UAP generator’s architecture and training loss are completely symmetrical with those of image attacks. Correspondingly, embeddings of the matched image $v$ are used as the cross-modal conditions for the generator.
Given an adversarial text $t_{adv}$ as the anchor sample, we use the set $\mathbf{v}=\{v_{1},v_{2}\dots,v_{S}\}$ scaled from the originally matched image $v$ as negative samples while the $\mathbf{v}^{\prime}=\{v_{1}^{\prime},v_{2}^{\prime}\dots,v_{S}^{\prime}\}$ augmented from the farthest image $v^{\prime}$ within the randomly sampled image set as positive samples to formulate the $\mathcal{L}_{CL}$ loss.
$\mathcal{L}_{Dis}$ is consequently calculated as the negative Euclidean distance between the embeddings of $t_{adv}$ and the clean input $t$.

A notable distinction between the image and text attacks is the approach to inject adversarial perturbations. Due to the discreteness of text data, we apply the token-wise substitute strategy (Lu et al., ; Wang et al., ) that replaces certain important words in the original sentence with crafted adversarial words. Accordingly, the conditional generator is utilized to output the adversarial textual embeddings, which are subsequently mapped back to the vocabulary space to obtain a universally applicable word-level perturbation.
Prior to implementing the word replacement, a meticulous process is undertaken to identify the most optimal position within the sentence to insert the perturbation.
Our strategy intends to identify and replace the words that are more likely to have a greater influence during the decision-making. Concretely, for each word $w_{i}$ within a given sentence, we compute the distance between the embeddings of the original sentence and the $w_{i}$-masked version encoded by the VLP models to determine its contribution.
As aforementioned, we set $\epsilon_{t}=1$ and choose the single word exerting the highest feature distance as the target for replacement.

## 4 Evaluation

We first present experimental settings in Sec. 4.1 and then comprehensively evaluate the proposed algorithm across multiple victim VLP models in Sec. 4.2.
Sec. 4.3 presents attack results on more downstream V+L tasks to further validate the C-PGC’s effectiveness. Besides, sufficient ablation studies in Sec. 4.4 validate the contribution of each proposed technique and explore the impact of several crucial factors on the attack effect. More experiment results are provided in Appendix E.

**Table 1: ASR (%) of our C-PGC and GAP for image-text retrieval tasks on Flickr30k and MSCOCO. TR indicates text retrieval based on the input image, while IR is image retrieval using input text.**
| Dataset | Source | Method | ALBEF | TCL | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ | BLIP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | IR | TR | IR | TR | IR | TR | IR | TR | IR | TR | IR |  |  |  |
| Flickr30k | ALBEF | GAP | 69.78 | 81.59 | 22.15 | 29.97 | 6.61 | 18.37 | 23.4 | 37.54 | 29.92 | 44.29 | 16.09 | 28.12 |
| Ours | 90.13 | 88.82 | 62.11 | 64.48 | 20.53 | 39.38 | 43.1 | 65.93 | 54.4 | 72.51 | 44.79 | 56.36 |  |  |
| TCL | GAP | 33.5 | 40.61 | 82.41 | 80.67 | 6.61 | 17.79 | 21.55 | 38.56 | 30.57 | 45.48 | 21.45 | 31.82 |  |
| Ours | 50.26 | 56.29 | 94.93 | 90.64 | 14.94 | 33.96 | 46.92 | 66.41 | 52.98 | 70.66 | 35.75 | 52.52 |  |  |
| X-VLM | GAP | 16.14 | 24.43 | 17.08 | 26.2 | 90.24 | 85.98 | 24.51 | 41.15 | 42.62 | 53.08 | 16.19 | 25.74 |  |
| Ours | 24.46 | 47.77 | 29.19 | 50.15 | 93.29 | 91.9 | 43.47 | 66.03 | 59.2 | 72.79 | 32.39 | 52.24 |  |  |
| CLIP${}_{\textrm{ViT}}$ | GAP | 11.72 | 23.34 | 15.32 | 26.39 | 8.54 | 20.48 | 85.73 | 90.45 | 48.83 | 60.78 | 14.83 | 26.46 |  |
| Ours | 23.23 | 38.67 | 25.05 | 41.79 | 15.85 | 35.59 | 88.92 | 93.05 | 66.06 | 75.42 | 26.71 | 45.7 |  |  |
| CLIP${}_{\textrm{CNN}}$ | GAP | 13.57 | 25.21 | 19.05 | 28.87 | 11.59 | 23.13 | 27.46 | 43.16 | 73.18 | 81.6 | 15.25 | 27.94 |  |
| Ours | 15.31 | 38.93 | 19.77 | 43.72 | 17.17 | 41.65 | 39.9 | 64.82 | 81.74 | 88.9 | 22.19 | 46.11 |  |  |
| BLIP | GAP | 12.23 | 23.94 | 14.49 | 25.44 | 6.91 | 17.81 | 20.32 | 37 | 26.81 | 43.59 | 47.21 | 73.33 |  |
| Ours | 32.17 | 44.4 | 33.44 | 44.51 | 18.6 | 35.53 | 43.35 | 60.26 | 48.96 | 66.95 | 71.82 | 82.82 |  |  |
| MSCOCO | ALBEF | GAP | 82.65 | 84.35 | 53.6 | 45.46 | 15.09 | 15.64 | 25.18 | 29.94 | 28.06 | 35.28 | 37.44 | 33.61 |
| Ours | 96.18 | 95.09 | 82.49 | 76.24 | 39.97 | 48.58 | 59.71 | 67.05 | 61.27 | 70.8 | 59.18 | 63.89 |  |  |
| TCL | GAP | 55.92 | 48.22 | 95.16 | 92.29 | 17.34 | 17.01 | 28.73 | 31.19 | 32.27 | 39.81 | 43.59 | 39.64 |  |
| Ours | 76.62 | 71.17 | 96.72 | 93.88 | 42.99 | 48.4 | 70.32 | 79.08 | 74.1 | 82.97 | 62.35 | 66.97 |  |  |
| X-VLM | GAP | 26.35 | 23.72 | 27.8 | 22.91 | 95.1 | 88.84 | 32.39 | 38.16 | 52 | 55.4 | 24.67 | 22.65 |  |
| Ours | 51.46 | 65.71 | 52.8 | 64.99 | 98.89 | 95.79 | 67.42 | 75.45 | 75.49 | 82.58 | 55.74 | 66.7 |  |  |
| CLIP${}_{\textrm{ViT}}$ | GAP | 35.96 | 31.91 | 37.33 | 32.56 | 33.42 | 29.25 | 97.71 | 96.04 | 74.63 | 74.67 | 33.47 | 31.99 |  |
| Ours | 46.92 | 53.89 | 46.03 | 50.87 | 41.49 | 48.6 | 98.74 | 98.01 | 81.58 | 86.5 | 47.35 | 57.55 |  |  |
| CLIP${}_{\textrm{CNN}}$ | GAP | 28.67 | 27.51 | 29.84 | 27.69 | 26.4 | 24.81 | 39.64 | 40.53 | 90.34 | 91.56 | 24.99 | 26.18 |  |
| Ours | 33.38 | 46.68 | 40.61 | 50.76 | 35.34 | 46.95 | 63.83 | 70.15 | 94.89 | 94.42 | 37.38 | 53.06 |  |  |
| BLIP | GAP | 35.55 | 38.75 | 35.62 | 33.79 | 22.7 | 21.25 | 32.05 | 35.8 | 40.93 | 45.58 | 73.46 | 72.37 |  |
| Ours | 61.95 | 60.92 | 60.95 | 59.57 | 51.81 | 52.53 | 62.23 | 72.51 | 69.61 | 78.44 | 91.67 | 90.42 |  |  |

### 4.1 Experimental Setup

Downstream tasks and datasets.
We conduct a comprehensive study of C-PGC on four downstream V+L tasks, including image-text retrieval (ITR), image captioning (IC), visual grounding (VG), and visual entailment (VE). For ITR tasks, we employ the Flickr30K (Plummer et al., ) and MSCOCO (Lin et al., ) datasets which are commonly used in previous works (Zhang et al., ; Lu et al., ). The MSCOCO is also adopted for evaluating the IC task and we test VG and VE tasks on SNLI-VE (Xie et al., ) and RefCOCO+ (Yu et al., ) respectively.

Surrogate models and victim models. We conduct experiments on a wide range of VLP models including ALBEF (Li et al., ), TCL (Yang et al., ), X-VLM (Zeng et al., ), CLIP${}_{\textrm{ViT}}$ (Radford et al., ), CLIP${}_{\textrm{CNN}}$ (Radford et al., ), and BLIP (Li et al., ).
Note that for different V+L tasks, we correspondingly select different VLP models for evaluation based on their capability (Wang et al., ). For instance, among the six considered VLP models, only ALBEF, TCL, and X-VLM can handle VG tasks, while only ALBEF and TCL can deal with VE tasks.

Baselines. To better reveal the superiority of our proposed method in attacking VLP models, we transplant a representative and powerful algorithm GAP (Poursaeed et al., ) to the multimodal attack scenarios by appropriately modifying its original loss function (Lu et al., ).

Implementation details.
Following (Lu et al., ), we adopt Karpathy split (Karpathy & Fei-Fei, ) to preprocess the dataset and build the test set for evaluation. The test set is disjoint with the generator’s training data for rigorous assessment. To ensure the perturbation invisibility, we follow (Wang et al., ) and limit the perturbation budgets $\epsilon_{v}$ to $12/255$ and $\epsilon_{t}$ to $1$. During the augmentation, we resize the original images into five scales $\{0.5,0.75,1,1.25,1.5\}$, and apply Gaussian noise with a mean of 0 0 and a standard deviation of $0.5$. See Appendix C for more details.

### 4.2 Universal Attack Effectiveness

To align with previous adversarial attacks on VLP models (Zhang et al., ; Lu et al., ), we first consider the typical V+L task of image-text retrieval and report the ASR computed based on R@1 retrieval results. Appendix 7 provides results of R@5 and R@10.
Experimental results on MSCOCO and Flickr30K across six different VLP models are presented in Table 1.
We also provide a visualization example of the image retrieval on the MSCOCO dataset in Figure 4.

Figure: Figure 4: Illustration of image retrieval. The red indicates the universal adversarial word and the crossed-out word is the replaced one. We generate the UAP on ALBEF and test it on 6 target models. All retrieved images do not accurately correspond to the query text, validating the design of C-PGC.
Refer to caption: x4.png

White-box attack performance. By observing the white-box ASR in the gray-shaded area, we demonstrate that the proposed algorithm stably achieves excellent ASR on all the evaluated VLP models, validating the outstanding capability of the produced UAP.
With only a single pair of perturbations, we reach a noteworthy average white-box ASR of over 90% on two large datasets in terms of both TR and IR tasks. Especially on the MSCOCO dataset, our method achieves over 95% average ASR on ITR tasks across six surrogate models. Compared with the GAP, the proposed method significantly improves the average fooling rates by nearly 10%, confirming the great validity of our suggested multimodal contrastive-learning mechanism.
Essentially, the exceptional performance stems from the efficacy of our generated UAP in destroying the alignment between the image and text modalities, thereby misleading the VLP model during inference.

Black-box attack performance. We also conduct thorough experiments regarding the adversarial transferability of the generated UAP by transferring from surrogate models to other inaccessible models.
As demonstrated in Table 1, the proposed C-PGC displays great attack performance in the more realistic black-box scenarios, e.g., 82.97% from TCL to CLIP${}_{\textrm{CNN}}$ on MSCOCO for IR tasks.
We highlight that the advantage of C-PGC over GAP (Poursaeed et al., ) is greatly amplified in the more challenging black-box scenarios, which achieves a significant average improvement of 18.36% and 26.32% for Flickr30K and MSCOCO respectively.
These experimental results indicate that our generative contrastive learning framework does not overly rely on the encoded feature space tailored to the surrogate model. Conversely, it is well capable of transferring to breaking the multimodal alignment of other unseen target models, thus attaining superior adversarial transferability.

**Table 2: ASR (%) of ITR tasks under defense strategies. Surrogate model is ALBEF and the dataset is Flick30K. LT denotes the LanguageTool that corrects adversarial words within the sentence.**
| Method | ALBEF | TCL | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ | BLIP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | IR | TR | IR | TR | IR | TR | IR | TR | IR | TR | IR |  |
| Gaussian | 37.92 | 49.49 | 32.4 | 47.04 | 19.31 | 37.79 | 42.49 | 65.61 | 50 | 72.23 | 29.65 | 48.77 |
| Medium | 53.13 | 61.6 | 39.54 | 51.96 | 20.43 | 39.69 | 46.31 | 66.92 | 57.9 | 74.51 | 33.75 | 52.68 |
| Average | 29.09 | 44.91 | 29.61 | 44.72 | 17.89 | 36.07 | 42.98 | 65.42 | 49.74 | 72.48 | 27.55 | 46.9 |
| JPEG | 59.3 | 63.7 | 42.34 | 52.52 | 21.65 | 41.58 | 41.26 | 65.77 | 53.5 | 72.62 | 37.01 | 55.04 |
| DiffPure | 64.34 | 74.63 | 65.22 | 74.8 | 66.06 | 75.19 | 78.08 | 86.7 | 82.25 | 88.03 | 70.45 | 79.09 |
| NRP | 32.33 | 40.63 | 20.19 | 39.23 | 14.63 | 32.62 | 48.4 | 69 | 59.72 | 74.09 | 30.28 | 52.2 |
| NRP+LT | 29.05 | 35.23 | 21.33 | 37.41 | 15.55 | 29.63 | 47.19 | 67.35 | 56.82 | 73.47 | 28.23 | 50.59 |

Defense Strategies. We next analyze several defense strategies to mitigate the potential harm brought by the proposed C-PGC.
Specifically, we totally align with TMM (Wang et al., ) and consider several input preprocessing-based schemes, including image smoothing (Ding et al., ) (Gaussian, medium, average smoothing), JPEG compression (Dziugaite et al., ), NRP (Naseer et al., ), and the prevalent DiffPure (Nie et al., ), a powerful purification defense using diffusion models. For adversarial text correction, we choose the LanguageTool (LT) (Wang et al., ), which has been widely adopted in various scenarios due to its universality and effectiveness.

The attack results in Table 2 demonstrate that the proposed attack still attains great ASR against different powerful defenses. It also indicates that NRP+LT would be a decent choice to alleviate the threat brought by C-PGC. Another noteworthy finding is that, although DiffPure (Nie et al., ) exhibits remarkable performance in defending attacks in classification tasks, its ability is greatly reduced in V+L scenarios since the denoise process could also diminish some texture or semantic information that is critical for VLP models, thereby acquiring unsatisfactory defense effects.

### 4.3 Evaluation on More Downstream Tasks

We further demonstrate C-PGC’s ability to destroy the multimodal alignment by presenting more results on diverse V+L tasks. Specifically, we consider Image Captioning (IC), Visual Grounding (VG), and Visual Entailment (VE). The results of VE are shown in Appendix 7 due to space limit.

**Table 3: Attacks results of image captioning. The Baseline represents the performance of the target model on clean data. The used dataset is MSCOCO.**
| Source | B@4 | METEOR | ROUBE_L | CIDEr | SPICE |
| --- | --- | --- | --- | --- | --- |
| Baseline | 39.7 | 31.0 | 60.0 | 133.3 | 23.8 |
| ALBEF | 30.1 | 23.7 | 51.2 | 92.5 | 17.5 |
| TCL | 29.5 | 23.5 | 51.0 | 88.9 | 17.3 |
| BLIP | 21.2 | 19.1 | 45.5 | 62.5 | 13.7 |

Image captioning. The objective of IC is to
generate text descriptions relevant to the semantic content based on the given image. We use ALBEF, TCL, and BLIP as source models and attack the commonly used captioning model BLIP. Similar to SGA (Lu et al., ), several typical evaluation metrics of IC are calculated to measure the quality of generated captions, including BLEU (Papineni et al., ), METEOR (Banerjee & Lavie, ), ROUGE (Lin, ), CIDEr (Vedantam et al., ), and SPICE (Anderson et al., ). The results in Table 3 demonstrate that our algorithm again displays prominent attack effects, e.g., the crated UAP induces notable drops of 10.2% and 9% in the B@4 and ROUGE_L respectively when transferred from TCL to BLIP.

Table: Table 4: Attack results of visual grounding. The first row displays the source models, where the Baseline indicates the clean performance of the target model on clean data.

Visual grounding. This is another common V+L task, which aims to locate the correct position in an image based on a given textual description. We conduct experiments on RefCOCO+ using ALBEF, TCL, and X-VLM as source and target models. Table 4 indicates that C-PGC brings a notable negative impact on the localization accuracy in both white-box and black-box settings, again verifying that the produced UAP strongly breaks the cross-modal interaction and alignment.

### 4.4 Ablation Study

This subsection employs the representative ALBEF (Li et al., ) model as the surrogate model to provide sufficient ablation studies on Flickr30K.
We begin our analysis on the contribution of each proposed technique. Subsequently, we examine the sensitivity of certain hyperparameters.

The effect of $\mathcal{L}_{CL}$ and $\mathcal{L}_{Dis}$. To investigate the impact of the proposed loss terms, we introduce two variants C-PGCCL and C-PGCDis that remove $\mathcal{L}_{CL}$ and $\mathcal{L}_{Dis}$ from the training loss respectively.
As shown in Table 5, the removal of $\mathcal{L}_{CL}$ leads to significant degradation, particularly for black-box transferable attacks. e.g., a 27.12% ASR drop in TR tasks from ALBEF to TCL. This validates the considerable contribution of $\mathcal{L}_{CL}$ to guarantee a successful attack.
Regarding the influence of $\mathcal{L}_{Dis}$, we demonstrate that this unimodal guidance can further enhance attacks on the basis of $\mathcal{L}_{CL}$, e.g., a 10.59% increase in the ASR of TR tasks for white-box attacks on ALBEF. The proposed two loss terms complement each other and jointly underpin the generalizability of the generated UAP.

**Table 5: ASR (%) of C-PGC and its variants averaged across six target models on retrieval tasks.**
| Method | ALBEF | TCL | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ | BLIP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | IR | TR | IR | TR | IR | TR | IR | TR | IR | TR | IR |  |
| C-PGC | 90.13 | 88.82 | 62.11 | 64.48 | 20.53 | 39.38 | 43.1 | 65.93 | 54.4 | 72.51 | 44.79 | 56.36 |
| C-PGCCL | 76.46 | 77.58 | 34.99 | 47.55 | 14.33 | 33.61 | 42.98 | 62.81 | 46.11 | 65.58 | 27.13 | 46.44 |
| C-PGCDis | 79.54 | 82.46 | 56.52 | 62.21 | 20.24 | 38.26 | 39.78 | 65.1 | 52.2 | 71.01 | 42.43 | 55.52 |
| C-PGCRand | 61.87 | 65.17 | 43.69 | 52.54 | 19.51 | 35.47 | 40.33 | 65.77 | 54.15 | 70.62 | 39.43 | 52.59 |
| C-PGCCA | 85.18 | 83.07 | 45.76 | 53.73 | 15.24 | 34.02 | 39.29 | 60.61 | 47.15 | 40.64 | 32.39 | 48.29 |

The effect of positive sample selection.
To validate the farthest selection strategy when constructing positive samples, we design another variant C-PGCRand that adopts randomly sampled data points as positive samples. Results in Table 5 reveal the necessity of the proposed farthest selection strategy as it brings an average improvement of 25.96% in white-box ASR and 4.95% in black-box ASR.
Moreover, we can also conclude that if the positive samples are not adequately defined, adding $\mathcal{L}_{CL}$ would even severely harm the white-box performance (see C-PGCCL and C-PGCRand).

The effect of cross-modal conditions. As aforementioned, cross-attention (CA) modules are introduced into the generator to exploit cross-modal information.
We then design C-PGCCA that cancels these CA layers to explore their influence. As expected, it causes a notable 9.78% average decrease across six target models, confirming the vital role of cross-modal knowledge. An interesting finding is that C-PGCCA induces a more pronounced drop in black-box attacks than white-box ones, indicating that cross-modal conditions exert a greater contribution to the adversarial transferability.

Different regulatory factor $\lambda$. The value of $\lambda$ is a critical factor as it adjusts the scales of the two loss terms $\mathcal{L}_{CL}$ and $\mathcal{L}_{Dis}$. We explore the attack performance under various values of $\lambda$ to confirm the optimal value. Figure 5 indicates that $\lambda=0.1$ achieves superior performance.

Figure: Figure 5: ASR of five target models on TR tasks under various $\lambda$.
Refer to caption: x5.png

Different perturbation budgets $\epsilon_{v}$ and $\epsilon_{t}$. As shown in Figure 6, we analyze varying perturbation budgets for $\epsilon_{v}$ and $\epsilon_{t}$. Generally, the ASR increases with the larger perturbation magnitudes.
Note that when $\epsilon_{v}=4/255$, C-PGC’s performance is severely compromised since the budget $4/255$ is too small to allow the UAP to carry enough information required to generalize to diverse data samples.
We also find that the improvement slows down as $\epsilon_{v}$ increases from $12/255$ to $16/255$. Thus, we select the moderate value of $12/255$ to reach a balance between attack utility and imperceptibility.
For text perturbation, $\epsilon_{t}$ exhibits a more profound influence on the black-box attacks. In our experiments, we strictly set $\epsilon_{t}=1$ for invisibility. However, attackers can adjust the value of $\epsilon_{t}$ in accordance with their demands to trade off the attack efficacy and the perturbation stealthiness.

## 5 Conclusion

This paper delves into the challenging task of launching universal adversarial attacks against VLP models and proposes an effective solution that achieves superior performance using only one universal pair of image-text perturbations.
We begin by revealing the unsatisfactory results of existing UAP methods and empirically explaining the underlying reasons.
Based on our analysis, we propose to break the crucial cross-modal alignment in VLP models by designing a contrastive-learning generative UAP framework that leverages both unimodal and multimodal information to enhance the attacks.
Extensive experiments validate the efficacy of the proposed algorithm on diverse VLP models and V+L tasks.
We highlight that the proposed framework makes a significant step in exploring the classic universal adversarial attacks in VLP models and deepens our understanding of the mechanism of VLP models.
We also hope that this paper can promote future research that explores more sophisticated defenses to strengthen the resilience of VLP models against adversarial attacks.

## Appendix A Pseudocode of the Proposed Algorithm

We present the pseudocode of our proposed attack algorithm for image modality in Alg. 1. Note that the text attacks are completely symmetrical as illustrated in Sec. 3.3.

Figure: Algorithm 1 Pseudocode of universal image attacks

## Appendix B Detailed Introductions to Datasets

- •
Flickr30K (Plummer et al., ). Collected from the Flickr website, this dataset describes different items and activities, which becomes a standard benchmark for various V+L tasks. It contains 31,783 images, each of which has five associated captions. We use it for ITR tasks.
- •
MSCOCO (Lin et al., ). The MSCOCO dataset is a rich and diverse dataset consisting of 123,287 images, each of which is annotated with approximately five sentences. We use this dataset to test the attack performance of ITR and IC tasks.
- •
SNLI-VE (Xie et al., ). Originally proposed for natural language reasoning tasks, this dataset provides large-scale images and descriptions, where each image is annotated with several sentences and their logical relationship labels, including entailment, neutral, and contradiction. This dataset is used for VE tasks.
- •
RefCOCO+ (Yu et al., ). An image dataset was selected from MSCOCO, which contains 19,992 images and 141,564 annotations. It is specially used for visual grounding (VG) tasks.

## Appendix C More Training Details

For Flickr30K and MSCOCO, we randomly sample 30,000 images and their captions from the training set to train our perturbation generator. For SNLI-VE and RefCOCO+, we learn the C-PGC directly using their training set with 29,783 and 16,992 images respectively. Since an image corresponds to multiple text descriptions in these datasets, we calculate the average of their textual embedding as the multimodal condition for the cross-attention modules.

We initialize the noise variable $z_{v}$ as a $3\times 3$ matrix. Meanwhile, the initial noise $z_{t}$’s dimensions in the text modality depend on the size of the hidden layer within the specific VLP model. Concretely, we set its dimension to $1\times 3$ for ALBEF, TCL, BLIP, and X-VLM, while $1\times 2$ for the CLIP model. When computing the multimodal contrastive loss $\mathcal{L}_{CL}$, the temperature $\tau$ is set as 0.1. The generator is trained over 40 epochs with the Adam optimizer, utilizing a learning rate of $2^{-4}$. Following previous works Lu et al. (); Wang et al. (), we employ the attack success rate (ASR) as our quantitative measurement of our attack in ITR tasks by computing the extent the adversarial perturbations result in victim models’ performance deviations from the clean performance.

## Appendix D Multimodal Alignment Destruction

To provide more intuitive evidence that our C-PGC successfully destroys the image-text alignment relationship, we compute the distance between the encoded image and text embeddings before and after applying the UAP. Concretely, for an input pair (v, t), we calculate the relative distance $d_{rel}$ by:

$$ $d_{rel}=\frac{||(f_{I}(v+\delta_{v})-f_{T}(t\oplus\delta_{t})||_{2}-||f_{I}(v) -f_{T}(t)||_{2}}{||f_{I}(v)-f_{T}(t)||_{2}}.$ (6) $$

We provide the distances averaged on 5000 image-text pairs from Flickr30K in Table 6. Benefiting from our delicate designs, C-PGC achieves better disruption of the aligned multimodal relationship, thereby boosting the generalization ability and transferability of the produced UAP.

**Table 6: Relative cross-modal feature distance of our C-PGC and GAP to the clean image-text pairs.**
| Source | Method | ALBEF | TCL | BLIP | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ALBEF | GAP | 7.18 | 6.54 | 0.91 | 1.74 | 0.31 | 0.98 |
| Ours | 8.83 | 14.95 | 2.73 | 6.09 | 3.42 | 3.92 |  |
| TCL | GAP | 4.02 | 24.27 | 0.91 | 0.87 | 0.12 | 0.07 |
| Ours | 6.43 | 27.11 | 3.64 | 4.35 | 2.56 | 2.94 |  |
| BLIP | GAP | 3.17 | 4.67 | 11.82 | 1.74 | -1.71 | -0.98 |
| Ours | 6.41 | 12.15 | 13.64 | 4.35 | 1.71 | 1.96 |  |

## Appendix E More Experimental Results

In this section, we provide more experimental results of our method in various tasks and scenarios.

Figure: Figure 7: Accuracy of VE tasks for different source and target models.
Refer to caption: x7.png

Visual entailment tasks. Given an image and a textual description, visual entailment involves determining whether the textual description can be inferred from the semantic information of the image. We align with previous VLP attacks (Zhang et al., ; Wang et al., ) and conduct experiments on the SNLI-VE (Xie et al., ) dataset using the ALBEF and TCL models. Note that the Baseline represents the clean performance of the target model on the clean data and the orange and green indicate ALBEF and TCL as source models respectively. The results presented in Figure 7 reveal that C-PGC obtains impressive attack effects by decreasing the average accuracy by nearly 20%.
Notably, (Do et al., ) has reported a large number of annotation errors in the labels of the SNLI-VE corpus used for VE tasks. Therefore, the presented results are only for experimental integrity and reference purposes.

**Table 7: ASR (%) of Cross-domain attacks on six models from Flickr30k to MSCOCO and vice versa. The gray shading indicates white-box attacks.**
| Setting | Source | ALBEF | TCL | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ | BLIP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | IR | TR | IR | TR | IR | TR | IR | TR | IR | TR | IR |  |  |
| Flickr30K<br>$\downarrow$<br>MSCOCO | ALBEF | 96.83 | 94.69 | 81.46 | 74.87 | 44.79 | 51.64 | 63.68 | 73.06 | 69.77 | 78.09 | 68.88 | 70.61 |
| TCL | 78.27 | 73.17 | 97.83 | 95.03 | 40.46 | 47.34 | 64.98 | 73.27 | 70.96 | 78.18 | 63.71 | 67.1 |  |
| X-VLM | 50.63 | 65.91 | 53.23 | 65.65 | 95.91 | 93.32 | 65.51 | 74.72 | 75.69 | 81.93 | 57.69 | 67.28 |  |
| CLIP${}_{\textrm{ViT}}$ | 49.88 | 53.39 | 49.47 | 52.21 | 47.77 | 48.52 | 95.5 | 97.01 | 83.05 | 85.38 | 50.97 | 57.93 |  |
| CLIP${}_{\textrm{CNN}}$ | 34.78 | 50.42 | 37.17 | 51.24 | 36.81 | 50.87 | 63.07 | 70.92 | 92.48 | 93.66 | 41.81 | 55.12 |  |
| BLIP | 54.45 | 55.51 | 55.63 | 53.02 | 41.07 | 46.93 | 61.69 | 69.24 | 65.52 | 75.23 | 83.19 | 82.17 |  |
| MSCOCO<br>$\downarrow$<br>Flickr30K | ALBEF | 88.08 | 87.28 | 58.9 | 61.53 | 17.58 | 36.07 | 39.78 | 61.08 | 47.28 | 64.95 | 35.02 | 49.4 |
| TCL | 47.58 | 53.7 | 87.27 | 83.55 | 18.6 | 34.45 | 51.85 | 72.22 | 59.46 | 76.09 | 37.75 | 53.08 |  |
| X-VLM | 25.39 | 46.74 | 27.33 | 49.13 | 79.98 | 81.72 | 42.73 | 66.48 | 59.46 | 73.07 | 31.65 | 51.48 |  |
| CLIP${}_{\textrm{ViT}}$ | 21.07 | 39.47 | 24.53 | 42.44 | 15.45 | 36.52 | 93.97 | 95.53 | 62.95 | 77.21 | 25.55 | 45.91 |  |
| CLIP${}_{\textrm{CNN}}$ | 14.29 | 36.57 | 20.5 | 41.7 | 15.55 | 37.06 | 41.63 | 62.87 | 86.53 | 88.73 | 17.98 | 44.31 |  |
| BLIP | 33.2 | 46.07 | 36.02 | 47.97 | 23.58 | 38.48 | 43.97 | 65.3 | 56.35 | 71.08 | 71.91 | 73.62 |  |

Cross-domain scenarios. We proceed to discuss the attack performance of the proposed algorithm in a more challenging scenario where there is an obvious distribution shift between the training dataset and the test samples. Specifically, we generate universal adversarial perturbations based on MSCOCO or Flickr30K and evaluate them accordingly on the other dataset. We present the attack success rates on the retrieval tasks across six models in Table 7. It can be observed that the domain gap indeed brings a negative effect on the attack performance. However, our method still maintains excellent ASR in most cases, unveiling its outstanding cross-domain transferability.

Results of R@5 and R@10.
As aforementioned, we supplement the ASR of the ITR tasks based on R@5 and R@10 metrics and provide the attack success rates in Table 8.
Obviously, our proposed C-PGC still consistently attains better performance than the baseline method GAP, regardless of the evaluation measurements for retrieval results.

**Table 8: Attack success rates (%) regarding R@5 and R@10 metrics of our C-PGC and GAP for image-text retrieval tasks.**
| Dataset | Source | Method | ALBEF | TCL | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ | BLIP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | IR | TR | IR | TR | IR | TR | IR | TR | IR | TR | IR |  |  |  |
| Flickr30K<br>(R@5) | ALBEF | GAP | 55.71 | 73.86 | 8.01 | 10.54 | 1.2 | 4.84 | 4.46 | 15.24 | 8.28 | 20.27 | 5.33 | 10.77 |
| Ours | 83.67 | 80.02 | 41.84 | 42.18 | 6.9 | 17.19 | 18.34 | 41.03 | 26.22 | 49.42 | 24.25 | 34.59 |  |  |
| TCL | GAP | 17.64 | 20.09 | 77.89 | 74.53 | 0.9 | 4.2 | 4.25 | 15.48 | 8.6 | 20.25 | 8.65 | 13.16 |  |
| Ours | 29.76 | 35.62 | 90.89 | 84.18 | 3.2 | 13.65 | 20.93 | 42.06 | 25.27 | 49.1 | 16.5 | 30.32 |  |  |
| X-VLM | GAP | 6.21 | 7.45 | 4.9 | 7.96 | 81.6 | 77.23 | 6.11 | 18.33 | 17.41 | 28.35 | 5.03 | 8.61 |  |
| Ours | 7.62 | 25.1 | 8.71 | 26.63 | 89.2 | 85.84 | 19.38 | 42.48 | 30.89 | 50.7 | 13.68 | 29 |  |  |
| CLIP${}_{\textrm{ViT}}$ | GAP | 2.81 | 6.86 | 4.2 | 8 | 1.7 | 6.1 | 75.64 | 82.56 | 24.2 | 37.68 | 4.33 | 9.97 |  |
| Ours | 6.31 | 17.51 | 8.01 | 19.65 | 4.3 | 15.1 | 76.89 | 85.2 | 39.6 | 54.68 | 9.15 | 23.23 |  |  |
| CLIP${}_{\textrm{CNN}}$ | GAP | 2.81 | 7.41 | 5.81 | 9.27 | 2.4 | 7.01 | 9.33 | 19.64 | 57.63 | 69.33 | 4.02 | 9.76 |  |
| Ours | 3.01 | 19.09 | 5.11 | 22.7 | 3.3 | 23.07 | 17.41 | 41.17 | 61.57 | 74.32 | 6.74 | 25.16 |  |  |
| BLIP | GAP | 3.41 | 7.01 | 3.7 | 7.45 | 1 | 4.4 | 4.46 | 14.43 | 6.79 | 18.67 | 39.13 | 68.02 |  |
| Ours | 14.43 | 21.67 | 13.91 | 21.59 | 5.4 | 14.54 | 18.03 | 36.26 | 23.89 | 44.79 | 59.26 | 74.82 |  |  |
| MSCOCO<br>(R@5) | ALBEF | GAP | 74.43 | 78.62 | 37.99 | 30.08 | 5.56 | 7.19 | 14.26 | 17.11 | 15.58 | 21.62 | 23.73 | 23.26 |
| Ours | 93.36 | 91.56 | 70.76 | 62.31 | 19.97 | 30.46 | 41.58 | 51.23 | 44.14 | 55.98 | 41.08 | 49.22 |  |  |
| TCL | GAP | 41.48 | 32.59 | 92.54 | 87.81 | 6.46 | 8.08 | 16.09 | 18.47 | 17.98 | 24.3 | 29.9 | 28.64 |  |
| Ours | 60.62 | 56.21 | 94.89 | 90.33 | 22.08 | 30.38 | 53.14 | 64.98 | 58.85 | 70.77 | 45.28 | 53.55 |  |  |
| X-VLM | GAP | 12.29 | 11.64 | 13.43 | 10.99 | 90.8 | 83.05 | 20.02 | 23.4 | 37.72 | 40.09 | 12.64 | 12.04 |  |
| Ours | 31.59 | 48.69 | 32.1 | 48.11 | 96.7 | 91.66 | 49.53 | 60.82 | 59.83 | 69.59 | 37.4 | 52.5 |  |  |
| CLIP${}_{\textrm{ViT}}$ | GAP | 18.78 | 17.46 | 20.38 | 17.37 | 16.81 | 15.15 | 95.21 | 93.04 | 62.77 | 62.62 | 18.48 | 19.48 |  |
| Ours | 25.69 | 35.95 | 24.69 | 33.14 | 21.37 | 31.38 | 96.7 | 96.49 | 70.76 | 77.86 | 28.72 | 42.01 |  |  |
| CLIP${}_{\textrm{CNN}}$ | GAP | 13.54 | 14.09 | 14.42 | 14.14 | 11.02 | 12.03 | 25.27 | 24.98 | 88.67 | 88.83 | 12.8 | 14.98 |  |
| Ours | 16.25 | 30.07 | 20.96 | 34.15 | 16.58 | 30.23 | 48.04 | 56.66 | 91.54 | 88.96 | 21.37 | 38.85 |  |  |
| BLIP | GAP | 23.62 | 24.22 | 22.96 | 18.43 | 9.93 | 9.75 | 19.2 | 22.24 | 24.99 | 30.19 | 62.75 | 66.88 |  |
| Ours | 42.56 | 43.73 | 41.72 | 41.8 | 31.05 | 35.63 | 44.37 | 57.9 | 54.47 | 66.01 | 81.71 | 81.91 |  |  |
| Flickr30K<br>(R@10) | ALBEF | GAP | 51.6 | 71.17 | 5.8 | 6.65 | 0.6 | 2.7 | 1.42 | 9.82 | 4.19 | 13.81 | 3.71 | 7.01 |
| Ours | 80.5 | 75.17 | 34.8 | 34.28 | 4.2 | 11.72 | 9.83 | 31.14 | 16.87 | 39.40 | 18.76 | 27.08 |  |  |
| TCL | GAP | 14.9 | 14.29 | 73.26 | 70.49 | 0.6 | 2.22 | 2.13 | 9.73 | 4.29 | 13.45 | 5.92 | 9.51 |  |
| Ours | 24.2 | 27.32 | 89.2 | 80.73 | 2.1 | 9.33 | 12.77 | 32.4 | 16.97 | 38.63 | 12.54 | 24.1 |  |  |
| X-VLM | GAP | 4.1 | 4.81 | 2.7 | 4.51 | 76.5 | 72.58 | 3.65 | 11.63 | 10.84 | 18.91 | 3.01 | 5.29 |  |
| Ours | 4.1 | 17.79 | 4.6 | 19.27 | 86.3 | 82.94 | 11.14 | 31.49 | 21.06 | 40.3 | 7.32 | 21.81 |  |  |
| CLIP${}_{\textrm{ViT}}$ | GAP | 2.1 | 4.04 | 2.6 | 4.81 | 1 | 3.79 | 63.29 | 77.83 | 17.89 | 28.11 | 2.31 | 6.12 |  |
| Ours | 4.2 | 11.45 | 4.6 | 13.08 | 2.8 | 10.15 | 67.98 | 79.46 | 29.75 | 45.56 | 5.52 | 17.23 |  |  |
| CLIP${}_{\textrm{CNN}}$ | GAP | 2.3 | 4.23 | 3.3 | 5.56 | 1.4 | 4.41 | 4.56 | 12.35 | 49.86 | 62.17 | 2.21 | 6.63 |  |
| Ours | 2.5 | 13.56 | 3.7 | 16.87 | 1.7 | 17.38 | 9.93 | 32.4 | 53.27 | 66.34 | 3.91 | 19.62 |  |  |
| BLIP | GAP | 2 | 3.98 | 1.5 | 4.17 | 0.2 | 2.24 | 1.93 | 8.38 | 3.68 | 12.48 | 36.81 | 67.22 |  |
| Ours | 11.2 | 15.49 | 9.1 | 14.14 | 2.8 | 10.19 | 9.83 | 27.46 | 14.83 | 34.43 | 53.46 | 72 |  |  |
| MSCOCO<br>(R@10) | ALBEF | GAP | 69.78 | 76.24 | 32.04 | 23.7 | 3.16 | 4.91 | 10.07 | 13.16 | 12.55 | 16.6 | 19.1 | 19.96 |
| Ours | 91.58 | 89.62 | 64.5 | 55.3 | 13.81 | 23.34 | 33.3 | 43.75 | 35.82 | 48.38 | 33.77 | 43.11 |  |  |
| TCL | GAP | 34.36 | 25.76 | 90.65 | 85.27 | 4.01 | 5.44 | 11.77 | 14.88 | 13.66 | 18.89 | 24.91 | 24.67 |  |
| Ours | 52.59 | 49.09 | 93.63 | 88.53 | 15.04 | 23.25 | 44.22 | 58.02 | 50.16 | 63.95 | 37.77 | 47.26 |  |  |
| X-VLM | GAP | 7.66 | 7.65 | 8.07 | 7.27 | 88.3 | 79.85 | 15.63 | 18.77 | 31.79 | 33.54 | 8.24 | 9.37 |  |
| Ours | 23.01 | 40.39 | 23.15 | 40.07 | 94.97 | 88.95 | 40.24 | 53.74 | 52 | 62.7 | 30.43 | 45.67 |  |  |
| CLIP${}_{\textrm{ViT}}$ | GAP | 13.13 | 12.39 | 14.05 | 12.13 | 10.68 | 10.89 | 93.72 | 91.51 | 57.39 | 56.47 | 13.35 | 15.64 |  |
| Ours | 17.87 | 28.52 | 17.48 | 26.09 | 14 | 24.67 | 95.55 | 95.31 | 64.04 | 72.75 | 22.05 | 35.65 |  |  |
| CLIP${}_{\textrm{CNN}}$ | GAP | 9.02 | 10.08 | 9.06 | 9.89 | 6.97 | 8.25 | 18.78 | 20.11 | 87.6 | 83.92 | 8.53 | 11.91 |  |
| Ours | 10.68 | 22.98 | 14.19 | 27.21 | 10.62 | 23.96 | 39.89 | 49.62 | 88.74 | 85.18 | 15.97 | 33.25 |  |  |
| BLIP | GAP | 17.76 | 18.98 | 16.32 | 13.13 | 6.21 | 6.44 | 13.97 | 17.39 | 19.73 | 24.37 | 57.99 | 65.49 |  |
| Ours | 33.64 | 36.14 | 32.15 | 33.8 | 22.64 | 28.52 | 36.07 | 50.3 | 47 | 59.07 | 78.39 | 78.98 |  |  |

Ablation study of the text perturbation. We introduce another variant C-PGCt that cancels the perturbation from the text side to investigate the contribution of image perturbation, text perturbation, and their synergy.
The comparison results of C-PGC, C-PGCt, and GAP using ALBEF as the surrogate model are shown in Table 9.

We find that when merely applying image perturbations (C-PGCt), our design still outperforms GAP with notable improvements, validating the proposed techniques’ effectiveness in enhancing the image perturbation.
Moreover, the superiority of C-PGC over C-PGCt indicates that the incorporation of textual perturbations can further boost the universal attacks on the basis of C-PGCt since the text perturbation facilitates the deconstruction of the learned cross-modal alignment.

**Table 9: ASR (%) of C-PGC, C-PGCv, and GAP on ITR tasks using Flickr30K. Note that C-PGCt only considers attacking images and thus doesn’t apply textual perturbations.**
| Method | ALBEF | TCL | X-VLM | CLIP${}_{\textrm{ViT}}$ | CLIP${}_{\textrm{CNN}}$ | BLIP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | IR | TR | IR | TR | IR | TR | IR | TR | IR | TR | IR |  |
| GAP | 69.78 | 81.59 | 22.15 | 29.97 | 6.61 | 18.37 | 23.4 | 37.54 | 29.92 | 44.29 | 16.09 | 28.12 |
| C-PGCt | 86.74 | 86.3 | 50.1 | 50.2 | 10.87 | 21.53 | 26.28 | 39.3 | 33.42 | 48.32 | 31.55 | 36.77 |
| C-PGC | 90.13 | 88.82 | 62.11 | 64.48 | 20.53 | 39.38 | 43.1 | 65.93 | 54.4 | 72.51 | 44.79 | 56.36 |

## Appendix F Limitations and Future Directions

Overlook of interactions between perturbations $\delta_{v}$ and $\delta_{t}$. The proposed framework generates universal perturbations for image and text respectively based on the designed multimodal and unimodal losses. Despite the remarkable attack performance, it does not consider the interactions and synergy between the perturbations $\delta_{v}$ and $\delta_{t}$ during optimization, which have been leveraged in several previous attacks (Lu et al., ; Wang et al., ) to improve performance.
In future research, this limitation can be explored as a potential mechanism to further strengthen the attacks.

Textual Semantic Inconsistency. To ensure the stealthiness of textual attacks, we set the perturbation budget $\epsilon_{t}=1$, i.e., only one word is modified. Nonetheless, this perturbation is still relatively likely to be detected since it is considerably difficult to maintain semantic consistency between the clean and the perturbed sentences when applying only one universal word to hundreds of various sentences.
For instance, when the adversarial word is getting (verb) and the word to replace is bed (noun), the resultant adversarial text becomes A getting is unmade with pillows on it, which is somehow strange and likely to attract the attention of defense systems or humans.
This is an inevitable common issue for the whole research area of textual UAP.
A possible solution is to add perturbations to the word embeddings instead of the input sentence. However, this only applies to white-box scenarios given that attackers have no access to the word embeddings in the black-box settings.

## Appendix G More Visualization Results

This section presents a rich visual analysis of the proposed attack on a series of downstream tasks. Specifically, we generate the UAP and conduct attacks on the ALBEF model in the visual grounding (VG) task. As illustrated in Figure 8, the prediction bounding boxes exhibit a notable deviation from the clean predictions, verifying that our generated adversarial samples significantly interfere with the multimodal alignment.
In the visual entailment (VE) task, we employ BLIP as the victim model and present the results in Figure 9. These qualitative visulizations again demonstrate the remarkable attack effects of our proposed method on various downstream tasks.

Figure: Figure 8: Illustration of visual grounding. Predictions of clean image-text pairs are on the left while the adversarial samples are on the right. The red indicates the universal adversarial word.
Refer to caption: x8.png

Figure: Figure 9: Illustration of the visual entailment task. The red indicates the universal adversarial word. It can be observed that all predictions do not match with the ground truth.
Refer to caption: x9.png

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Please refer to Weaknesses.
soundness: 1
strengths: 1. The paper focuses on an important task of evaluating robustness of VLP models.
2. Both adversarial images and texts are learned.
summary: This paper proposes a method to learn universal perturbations that can transfer across different Vision-Language Pre-training (VLP) models and downstream tasks. The authors leverage contrastive loss to disrupt cross-model interactions and use a Euclidean distance-based loss to maximize the distance between adversarial data and the original data. Experimental results show that the proposed method achieves strong attack performance on various VLP models and downstream tasks.
weaknesses: Several concerns remain:
1. Motivation: 
- In the abstract, the authors claim to "fully utilize the characteristics of Vision-and-Language (V+L) scenarios by incorporating both unimodal and cross-modal information." However, the authors do not seem to fully exploit the characteristics of different V+L scenarios or tasks.
- In the introduction, Figure 1 compares two methods and claims that "the generator-based approach GAP consistently achieves superior ASR compared to UAP." Since UAP uses the DeepFool method to learn perturbations, its inferior performance compared to a generator-based approach does not necessarily demonstrate the superiority of the generator-based method over other approaches, e.g., PGD. More experiments including comparisons with other strong baselines like PGD, or a more comprehensive analysis of why generator-based methods are required to support this claim.
2. Algorithm: 
a: My biggest concerns are the definition of universal perturbation learning and adversarial text learning.
- The authors use generators to produce adversarial data based on cross-modal conditions. The main advantage of universal adversarial attacks is their ability to produce perturbations that are generalizable across all data without needing to generate sample-specific perturbations, thereby improving efficiency. In other words, universal perturbations should be independent of the test data and applicable to unseen data. However, relying on cross-modal conditions appears to conflict with this objective. If cross-modal conditions are required, why not generate sample-specific perturbations instead? The authors need to clarify how to maintain the universality of the perturbations without using cross-modal conditions.  Additionally, the authors should report results of universal attacks for more scenarios, e.g., using perturbations generated on the Flickr30k dataset to attack models on the MSCOCO dataset.
- Learning adversarial text perturbations requires ensuring that they do not compromise the quality of original texts. However, the authors did not address this, rendering the algorithm impractical. Further verifications are required, such as utilizing proposing metrics to evaluate the semantic consistency of perturbed texts or discussing potential methods to constrain the text perturbations to maintain readability and coherence.
b. In addition, the authors utilize contrastive learning to disrupt the cross-model relationships and use the Euclidean distance-based loss to enlarge the distance between adversarial data and their original counterpart. First, the authors utilize contrastive learning to enlarge the gap between multiple texts and minimize the distance with diverse target texts. However, I question whether setting different targets can truly maximize the distance between adversarial and original images. An ablation study is necessary to verify this, such as comparing the proposed approach with a baseline that doesn't use diverse target texts, or to analyze how different choices of target texts affect the effectiveness of the perturbations. Second, the authors use two distinct losses to maximize the distance between adversarial images and each modality in the original image-text pairs. It is unclear why two losses are needed, rather than using a unified loss for both modalities. Additional experiments comparing their two-loss approach with a unified loss approach, and analyze the impact on both cross-modal and intra-modal relationships should be conducted. 

c. Perturbation learning methods, including set-level augmentation, maximizing both intra- and inter-model differences, and leveraging contrastive learning, have already been explored by current approaches [1,2]. The specific contribution of this method remains unclear, aside from generating universal perturbations instead of sample-specific ones. Furthermore, the distinction between the generation of universal and sample-specific perturbations remains unclear.

5. Experiments:
-  Previous works on universal adversarial attacks for VLP models should be discussed and compared, such as [3]. Additional comparison with relevant methods should be provided.
- The authors apply data augmentation to improve the method's effectiveness, but additional comparisons with other augmentation techniques should be conducted to better demonstrate the proposed method's superiority. Examples include ScMix [3] and Admix [4].
 [1] Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. 2023. Set-level guidance attack: boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE International Conference on Computer Vision. 102–111.
[2] Kim M, Tack J, Hwang S J. Adversarial self-supervised contrastive learning[J]. Advances in neural information processing systems, 2020, 33: 2983-2994.
[3] Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. 2021. Admix: enhancing the transferability of adversarial attacks. In Proceedings of the IEEE International Conference on Computer Vision. 16158–16167.
[4] Zhang, Peng-Fei, Zi Huang, and Guangdong Bai. "Universal Adversarial Perturbations for Vision-Language Pre-trained Models." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In the text modality attack, how do the authors maintain semantic similarity between original and adversarial texts? In the experiments, the authors should provide the similarity scores (e.g. bert_score) between original and adversarial texts to demonstrate that the modifications do not significantly alter the text's semantics.

2. The authors should explain how ASR is calculated in the main text.

3. As shown in Figure 4, the adversarial texts exhibit a clear semantic gap from the original texts. Thus, would using special characters (e.g. ##*) for the universal adversarial word be more effective?


4. The authors propose a contrastive training perturbation generator to produce universal adversarial perturbations for images and text. I am curious about how this generator differs from general UAP methods (e.g. Data-free Universal Adversarial Perturbation and Black-box Attack ), justifying its designation as a "generator."

I look forward to your detailed response.
soundness: 3
strengths: 1. The proposed UAP framework addresses the inefficiencies of instance-specific attacks by incorporating cross-modal and unimodal guidance within a contrastive training setup, representing an advancement in universal adversarial attack methods.

2. The paper thoroughly evaluates C-PGC's effectiveness across multiple VLP models and downstream tasks, and additionally analyzes various defense strategies to mitigate the potential risks posed by C-PGC.
summary: This paper addresses the vulnerability of Vision-Language Pre-training (VLP) models to universal adversarial perturbations, which are instance-agnostic and do not require individual perturbations for each input. The authors introduce a novel attack method, the Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC), which leverages contrastive learning to disrupt the multimodal alignment in VLP models. Experiments are conducted across multiple VLP models and tasks.
weaknesses: The proposed method leverages image and text attacks alongside cross-modal contrastive learning to generate universal adversarial perturbations. While this approach shows promise, the novelty may not be fully evident. I recommend that the authors consider further highlighting and reorganizing the unique contributions of the paper to enhance its clarity and impact.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the **Weaknesses** section. If the authors can address these issues, I would be willing to raise my score.

1. **Use of Contrastive Loss** 
Perhaps the authors could offer a more detailed explanation of their rationale for using contrastive loss in this context and discuss potential alternative loss functions they considered, as well as why contrastive loss was ultimately chosen despite its typical use in alignment tasks.

2.  **Choice of Positive and Negative Samples for Contrastive Loss**  
Introducing a synthetic, "fictitious" sample that maximally deviates from the original, is a more direct way. Perhaps the authors could discuss the trade-offs between their current approach and your suggested method.

3. **Limited Comparison with State-of-the-Art**  
The authors could select 2-3 recent and relevant adversarial attack methods from related work for comparison.
soundness: 3
strengths: The writing style of the paper is commendably clear and concise, making it accessible to a broad audience within the machine learning and computer vision communities. The authors have taken care to present the technical details in a manner that is straightforward and easy to follow, even for readers who may not be deeply familiar with adversarial attacks or VLMs. The method’s components are explained in a way that balances technical rigor with simplicity. This makes the paper highly readable and ensures that a wide range of researchers and practitioners can engage with its contributions.

The experimental results demonstrate that the proposed C-PGC framework performs well across several benchmarks. The authors have conducted comprehensive experiments on multiple well-established datasets and across various VLP models. The results show consistent improvements in attack success rates (ASR) across both white-box and black-box settings.
summary: The paper presents a novel framework, C-PGC, designed to generate Universal Adversarial Perturbations (UAPs) targeting Vision-Language Pretraining (VLP) models. The authors introduce a cross-modal conditional perturbation generator, which leverages both single-modal and cross-modal features to disrupt the learned alignment between visual and textual representations in VLP models.
weaknesses: The paper, while strong overall, has several areas for improvement:

1. **Use of Contrastive Loss**  
   The inclusion of contrastive loss (\(\mathcal{L}_{CL}\)) feels somewhat forced. Since the goal is to perform untargeted attacks, it seems unnecessary to rely on contrastive loss, which is typically used to enforce alignment between representations. While the authors have shown its utility through ablation studies, the logical foundation of using contrastive loss in an untargeted setting remains unclear. The paper could be improved by either rethinking the rationale behind using \(\mathcal{L}_{CL}\) or exploring alternative loss functions better suited for untargeted attacks.

2. **Choice of Positive and Negative Samples for Contrastive Loss**  
   The current method of manually selecting the farthest sample as the positive or negative example feels arbitrary and unnecessarily complex. In the context of untargeted attacks, where the objective is not to make the adversarial sample resemble a specific class, it would make more sense to introduce a synthetic, "fictitious" sample that maximally deviates from the original, rather than relying on the farthest feature-distance sample. This approach could simplify the process and make the use of contrastive loss more coherent in the untargeted setting.

3. **Limited Comparison with State-of-the-Art**  
   The comparison of the proposed method only with GAP, a 2018 work, limits the scope of the evaluation. Given the rapid advancements in adversarial attack methods, comparing against more recent techniques would provide a clearer picture of the method’s competitiveness. For example, comparing with more contemporary adversarial generation methods would strengthen the experimental section and make the results more relevant to current research.

4. **Visual Design of the Framework Diagram**  
   The framework diagram could benefit from improved design and color harmony. While this is a minor issue, it affects the overall presentation quality. The authors could refer to well-designed diagrams from recent top-tier papers to make the visualizations clearer and more appealing.

These adjustments would enhance the logical foundation of the method and improve both the clarity and relevance of the experimental comparisons.

Post-Rebuttal:

The main issue lies in the fact that the authors' response fails to address my concerns regarding the use of Contrastive Loss in the paper. The goal of the paper is to construct untargeted adversarial attacks, yet the authors manually select negative samples to construct the Contrastive Loss, which is problematic. If we follow the authors' stated motivation that "destruction is easier than construction," the approach should involve maximizing the Contrastive Loss during normal training, rather than manually selecting negative samples to minimize it. For these reasons, I am inclined to maintain my current rating.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: Please see weakness
soundness: 4
strengths: 1.	The writing is clear. The formulas are correct.
2.	The experiment is abundant and multi-dimensional.
3.	The research topic is important for VLM.
summary: This paper introduces a universal adversarial attack method called C-PGC for VLP models. By leveraging contrastive learning mechanisms within VLP models, C-PGC generates UAP that can effectively attack across different tasks and models without requiring individual perturbations for each input. Experiments demonstrate that C-PGC disrupts image-text feature alignment efficiently in both white-box and black-box settings, outperforming existing attack methods.
weaknesses: 1. The generator-based UAP method is time-consuming due to its indirect optimization approach, as it does not directly update the UAP.
2. In multimodal contrastive loss, randomly selecting texts or images may not be ideal. Instead, selecting items related to the current image-text pair in the batch could improve the performance.
3. The method uses the default settings of SGA in the experiment (i.e., resizing the original images into five scales {0.5, 0.75, 1, 1.25, 1.5} and applying Gaussian noise with a mean of 0 and a standard deviation of 0.5). It would be beneficial to give an explanation of the effect of such augmentation, with the experiment result being better.

Meta-review: This work reveals that Vision-Language Pre-training (VLP) models are vulnerable to instance-agnostic Universal Adversarial Perturbations (UAPs). It introduces a Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) that leverages malicious contrastive learning to disrupt the multimodal alignment in VLP models, achieving effective and transferable attacks across various models and vision-language tasks. The authors provided additional experiments and explanations during the rebuttal phase to address the reviewers' concerns; however, some issues remain unresolved. The primary weaknesses of this work lie in the inefficiency and ineffectiveness of the proposed universal adversarial text generation approach. The method's reliance on identifying critical words before applying universal perturbations contradicts the principle of universal attacks, which should function on all unseen data without additional computation. The quality of the generated perturbations is questionable, as merely replacing words does not ensure imperceptibility, making the modified texts easily identifiable. Additionally, the high semantic similarity between the original and adversarial texts suggests the attack has minimal impact, undermining its effectiveness. The use of large language models to validate imperceptibility is also unconvincing. Furthermore, the performance improvement primarily stems from the underlying perturbation technique (e.g., DeepFool) rather than the novel strategy proposed in this work, as highlighted in the rebuttal. Therefore, we decide not to accept this work based on its current state.

Decision: Reject