OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization

## Abstract

Abstract Vision-language pre-training (VLP) models demonstrate impressive abilities in processing both images and text. However, they are vulnerable to multi-modal adversarial examples (AEs). Investigating the generation of high-transferability adversarial examples is crucial for uncovering VLP models’ vulnerabilities in practical scenarios.
Recent works have indicated that leveraging data augmentation and image-text modal interactions can enhance the transferability of adversarial examples for VLP models significantly. However, they do not consider the optimal alignment problem between data-augmented image-text pairs. This oversight leads to adversarial examples that are overly tailored to the source model, thus limiting improvements in transferability.
In our research, we first explore the interplay between image sets produced through data augmentation and their corresponding text sets. We find that augmented image samples can align optimally with certain texts while exhibiting less relevance to others. Motivated by this, we propose an Optimal Transport-based Adversarial Attack, dubbed OT-Attack. The proposed method formulates the features of image and text sets as two distinct distributions and employs optimal transport theory to determine the most efficient mapping between them. This optimal mapping informs our generation of adversarial examples to effectively counteract the overfitting issues. Extensive experiments across various network architectures and datasets in image-text matching tasks reveal that our OT-Attack outperforms existing state-of-the-art methods in terms of adversarial transferability.

## 1 Introduction

Figure: Figure 1: An image example, after undergoing various data augmentation strategies, tends to focus on different image contents. Consequently, it can better align with specific text content while maintaining limited relevance to others. Thus, using a uniform standard to assess this relationship is unsuitable, which highlights the limitation of the existing state-of-the-art SGA method.
Refer to caption: extracted/5281202/figure/home.png

Vision-Language Pre-trained (VLP) models have shown outstanding performance in various downstream tasks, including image-text matching, image captioning, visual question answering, and visual grounding. Despite their impressive capabilities, these models encounter significant security challenges in real-world applications [, , , ]. Most of the prior researches [, , , , ] focused on white-box attacks, where there is complete access to the model’s structure, weights, and gradients. But they are not applicable to black-box models, such as GPT-4 [].

Fortunately, existing works have demonstrated that adversarial examples perturbed on white-box models remain effective on certain black-box models [, ]. It indicates that adversarial examples generated via a proxy model can still mislead the judgment of black-box models due to their transferability [, , , , , ]. The most ideal attack scenario, in reality, is one where adversarial examples remain effective even in the absence of detailed knowledge about the model’s inner workings, such as its model architecture, weights, and gradients, etc [, ].
Motivated by the practical significance of black-box adversarial attacks and adversarial transferability [, , , ], in this paper, we primarily study the transferability of adversarial examples within Vision-Language Pre-trained (VLP) models.

Recent research in the field of VLP models has revealed that single-modality adversarial attacks, such as employing BERT-Attack[] on text or PGD attack[] on images, yield limited attack success rates, even against white-box models. This insight led to the exploration of more integrated approaches. For instance, the Sep-Attack method, which simultaneously uses BERT-Attack on text and PGD on images without considering interactions between these modalities, has shown some effectiveness but lacks in fully exploiting the potential synergies of multi-modal attacks. Building upon these findings, Zhang *et al.* proposed the Co-Attack[] method, which goes a step further by fusing information between modalities using individual image-text pairs. This approach underscored the importance of intermodal guidance for improving the transferability of adversarial examples in VLP models. Further advancing this field, Lu *et al.* introduced the Set-level Guidance Attack (SGA) [], which expands upon the concept of individual image-text pairings to a set-level alignment. The SGA methodology employs data augmentation to diversify the image set and pairs these images with multiple textual descriptions. This comprehensive approach, which integrates intermodal guidance, not only addresses the limitations observed in Sep-Attack but also achieves state-of-the-art results, highlighting the effectiveness in enhancing the transferability of adversarial examples against VLP models.

Figure: Figure 2: Comparative analysis of Set-level Guidance Attack (SGA) methods and their ITR attack success rates. Panel (a) illustrates the conventional SGA approach where image and text sets are averaged to establish pair-wise matches. Panel (b) showcases our proposed method, OT-Attack, where images are matched to texts based on optimal transport theory to enhance matching accuracy. Panels (c) and (d) depict the attack success rates for our method OT-Attack versus traditional SGA, with ALBEF and TCL models serving alternately as the source and target. The bar charts indicate that our adversarial examples outperform SGA across all metrics, demonstrating superior effectiveness in disrupting ITR performance.
Refer to caption: extracted/5281202/figure/home2.png

However, SGA has limitations in improving the transferability of adversarial examples for vision-language pre-training models. Specifically, it aligns the features of images and texts at a collective level but does not adequately address the optimal matching of post-augmentation image examples with their corresponding texts. For example, when images undergo data augmentation, such as zooming, which enlarges certain parts of an image while omitting others, the resulting examples may not align equally well with their textual descriptions. SGA calculates its loss by averaging similarities across all pairs, which means that even if there is an optimal match for the augmented data, its influence on the final loss is obviously reduced. This process can undermine the benefits of data augmentation and modality interactions in enhancing adversarial transferability.

In this paper, we address this issue by incorporating the theory of optimal transport []. We treat the feature sets of augmented images and texts as two distinct distributions and aim to establish the optimal transport scheme between them. The distinction between our method and SGA, along with a comparative overview of the results, is depicted in Figure [2](#S1.F2).
In detail, we integrate optimal transport theory to analyze data-augmented image sets and text sets as distinct distributions. This holistic consideration allows us to incorporate similarity into the cost matrix and calculate the optimal transport scheme. Consequently, we compute the total transfer cost between these distributions, guiding the generation of adversarial examples. Our method achieves a more balanced matching relationship between the augmented image set and the text set. As depicted in Figure [1](#S1.F1), an image example, after undergoing data augmentation, may align more closely with a particular text description. Applying optimal transport theory addresses this by balancing the effects of data augmentation and modality interactions, which in turn reduces overfitting and improves the transferability of adversarial examples.

Experiments conducted on various models including ALBEF [], TCL [], and CLIP [], and utilizing well-known datasets like Flickr30K [] and MSCOCO [], quantitatively demonstrate the effectiveness of our approach. We focus on image-text and text-image matching as our primary downstream tasks, assessing the adversarial examples’ transferability through their attack success rates on black-box models. The results exceed the current state-of-the-art on both datasets. Additionally, to assess cross-task transferability, we conduct experiments on image captioning and visual grouping, which confirm the efficacy of our OT-Attack method in varied downstream tasks. Additionally, we assess widely-used business models such as GPT-4 and Bing Chat under increased perturbation intensity, and we are able to effectively break them. Our research indicates that extra textural perturbations cause confusion in the decision-making processes of these models. The proposed method is positioned as a robust solution for uncovering and leveraging widespread vulnerabilities across the spectrum of VLP models. The key contributions of
in this paper are summarized in three aspects:

- 1.
Our proposed OT-Attack improves the SGA framework by ensuring a balanced match between image and text sets after data augmentation.
- 2.
We innovatively utilize Optimal Transport theory in examining adversarial example transferability in VLP models, promoting a more profound and thorough alignment between data-augmented images and textual descriptions.
- 3.
Extensive experiments establish that our method generates adversarial examples with superior transferability compared to existing state-of-the-art techniques. Furthermore, our OT-Attack can successfully break current business models like GPT-4 and Bing Chat.

## 2 Background And Related Work

### 2.1 Vision-Language Pre-training Models

Vision-language pre-training (VLP) [] is a pivotal technique in augmenting multimodal task performance, capitalizing on extensive pre-training with image-to-text pairs. Traditionally, much of the research in this area has relied on pre-trained object detectors, using region features to create vision-language representations. However, the advent of Vision Transformer (ViT) [, ] has instigated a methodological shift. Increasingly, studies are advocating for the adoption of ViT in image encoding, which involves an end-to-end process of transforming inputs into patches.VLP models can be broadly classified into two categories: fused and aligned VLP models. Fused VLP models, as exemplified by architectures like ALBEF [] and TCL [], utilize individual unimodal encoders for processing token and visual feature embeddings. These models then employ a multimodal encoder to amalgamate image and text embeddings, crafting comprehensive multimodal representations. Conversely, aligned VLP models, such as CLIP, use unimodal encoders to independently process image and text modality embeddings.

### 2.2 Vision-Language Tasks

Image-text Retrieval. Image-Text Retrieval (ITR)[, ] is a task that retrieves relevant instances from a database using one modality (image or text) to query the other. It splits into image-to-text retrieval (TR) and text-to-image retrieval (IR). Models like ALBEF and TCL calculate semantic similarity scores between image-text pairs for initial ranking, then employ a multimodal encoder for final ranking. Conversely, models like CLIP [] directly rank based on similarity in an unimodal embedding space, showcasing varied ITR methodologies.

Image Captioning. Image captioning [, ] involves generating textual captions for images and is crucial in Vision-Language Pre-training (VLP) models. This task requires converting visual content into coherent, contextually relevant text, differing from image-text retrieval which is about finding the most relevant content from a database based on the opposite modality queries.

Visual Grounding. Visual Grounding [, ], also termed referential expression comprehension, is key in both computer vision and NLP. It entails identifying and locating objects or regions in an image as per language descriptions, requiring a precise mapping of text to visual elements.

### 2.3 Transferability of Adversarial Examples

In the field of adversarial attacks, there are two main categories: white-box and black-box approaches. White-box attacks assume full knowledge of the target model’s structure and parameters, a scenario often not feasible in real-world black-box settings. In natural language processing (NLP), predominant methods typically involve textual manipulation, such as altering or replacing specific tokens, exemplified by BERT-Attack, introduced by Li *et al.*[]. In computer vision, white-box attacks have seen significant developments with methods like the Fast Gradient Sign Method (FGSM) by Goodfellow *et al.*[], Carlini & Wagner (C&W) by Carlini and Wagner[], and the Momentum Iterative Method (MIM) by Dong *et al.* []. These methods exploit the gradient information of the models to craft effective adversarial examples. Separately, Projected Gradient Descent (PGD), proposed by Madry *et al.*[], marks a notable advancement in the field. PGD iteratively adjusts images in small steps along the gradient of the loss function, making it a potent tool for creating subtle yet effective adversarial images. This approach has proven particularly effective in attacking the visual components of models.

Building on the foundations laid by PGD and BERT-Attack, Co-Attack, introduced by Xie *et al.*[], emerged as a method integrating both visual and textual modalities. This approach exploits the synergies between image and text within VLP models, targeting their multimodal nature. The recent innovation in this domain is the Set-level Guidance Attack (SGA), developed by Zhang *et al.*[]. SGA uses data augmentation to generate multiple image sets, aligning them with various text captions. This approach not only complements the multimodal essence of VLP models but also significantly enhances the transferability of adversarial examples across a spectrum of black-box models. The progression from individual-focused PGD and BERT-Attack to the integrated Co-Attack, and finally to the comprehensive SGA, illustrates an evolving landscape of adversarial strategies against VLP models.

### 2.4 Optimal Transport

Optimal Transport (OT), a concept first introduced by Monge [], was initially developed to minimize logistical costs in transporting goods. In modern times, especially within the realms of machine learning and computer vision, OT has become a prominent tool due to its effectiveness in comparing and aligning distributions that are represented as feature sets []. Its unique ability to match distributions has led to its widespread application in various theoretical and practical areas. This includes its use in generative models and structural alignments involving sequences [], graphs [], and image matching [, , ]. The versatility of OT also extends to other distribution-centric tasks like clustering, distribution estimation, and causal discovery.

## 3 Approach

Figure: Figure 3: Utilizing the SGA method, this caption presents the attack success rates when the augmented image set, originating from the ALBEF source model and targeting the TCL model, contains 1, 3, and up to 9 images. The overall trend progresses from an increase to a decrease in success rates with the addition of examples, illustrating the effectiveness of the image set and the diminishing performance on the black-box model with an excessive number of examples.
Refer to caption: extracted/5281202/figure/overfitting.png

In this section, we first introduce the treated model in this work, which consists of
the adversary’s goals and the adversary’s capabilities. Then we present the observation of the previous works and the corresponding analysis. Finally, we introduce our proposed method in detail.

### 3.1 Threat Model

Adversary’s goals. Let us consider a white-box model
$M_{\text{white}}$ with an encoding function $f_{\text{white}}$ and an original image $I_{\text{orig}}$. The aim is to determine the optimal perturbation $\Delta^{*}$ that maximizes the loss function $\mathcal{L}$ when applied to $I_{\text{orig}}$. The perturbation is restricted to a bounded domain to ensure the adversarial example $I_{\text{adv}}$ remains within a permissible visual deviation from $I_{\text{orig}}$. The optimal perturbation $\Delta^{*}$ is found by solving:

$$ $\Delta^{*}=\arg\max_{\Delta}\mathcal{L}(f_{\text{white}}(I_{\text{orig}})),$ (1) $$

subject to the constraint:

$$ $||\Delta||_{p}\leq\epsilon,$ (2) $$

where $||\cdot||_{p}$ represents the $p$-norm, and $\epsilon$ is the perturbation bound. Once $\Delta^{*}$ is obtained, the adversarial example is computed as:

$$ $I_{\text{adv}}=I_{\text{orig}}+\Delta^{*}.$ (3) $$

Our goal with respect to a black-box model $M_{\text{black}}$ characterized by an encoding function $f_{\text{black}}$, is to ensure that the adversarial example $I_{\text{adv}}$ deviates from the original image $I_{\text{orig}}$ by at least a threshold $\theta$ in the feature space of $f_{\text{black}}$. This criterion for successful misclassification can be defined as follows:

$$ $||f_{\text{black}}(I_{\text{adv}})-f_{\text{black}}(I_{\text{orig}})||_{2}\geq\theta,$ (4) $$

where $||\cdot||_{2}$ denotes the Euclidean norm, signifying the measure of dissimilarity between the feature representations of the adversarial and original examples within the black-box model’s domain. This condition implies that the adversarial example $x_{\text{adv}}$ is sufficiently distinct from the original image $x_{\text{orig}}$ to affect the black-box model’s output, potentially leading to incorrect classification.

Adversary’s capabilities. We postulate that the adversary possesses a comprehensive understanding of the white-box model, denoted as $M_{\text{white}}$. This encompasses an intimate familiarity with the model’s architectural framework, its parameter configurations, and the gradients that dictate learning. In stark contrast to the transparency afforded by $M_{\text{white}}$, the adversary’s purview over the black-box model, $M_{\text{black}}$, is notably opaque. Devoid of any direct knowledge regarding the internal mechanisms, parameter values, or gradient information of $M_{\text{black}}$, the adversary operates without the capacity to compute internal representations or gradients.

### 3.2 Observation & Analysis

The Set-level Guidance Attack (SGA) method employs a comprehensive approach, focusing on sets of image-text pairs and utilizing all textual descriptions alongside a wide array of image augmentations for generating adversarial examples. This process follows a cyclic methodology, where adversarial text is initially guided by original images, followed by the generation of adversarial images influenced by this text, and concludes with the creation of further adversarial text influenced by these images.

Figure [3](#S3.F3) demonstrates the impact of varying scaling factors on the success rate of image-text matching attacks. In this setup, ALBEF is used as the source model and TCL as the target model. The observed trend, where success rates increase initially with scaling factors but subsequently decrease, suggests that while augmentation initially improves example diversity, excessive scaling leads to overfitting.

Our analysis of this phenomenon involves two key aspects. First, we examine overfitting during the phase where adversarial images are created based on text. In this phase, both original images and adversarial texts are encoded, resulting in unique feature representations. The success of adversarial examples relies on how closely these features match. However, using too much data augmentation can disrupt this match, leading to adversarial examples that either miss or exaggerate certain image features. This can make the attacks less effective, particularly if they focus too much on prominent features while neglecting finer details. Second, to address this, our approach balances data augmentation and intermodal guidance more effectively. We aim for a better match between modified examples and texts, enhancing the adaptability of adversarial examples to different models and reducing the risk of overfitting, especially in cases where subtle or less prominent image features are involved.

### 3.3 The proposed Method

#### 3.3.1   Symbol Conventions

To facilitate expression, we initially agree upon certain primary notations as shown in TABLE [I](#S3.T1).

**TABLE I: Symbol conventions.**
| Symbol | Description |
| --- | --- |
| $\mathcal{I}$ | Original Image Set |
| $\mathcal{I}_{aug}$ | Augmented Image Set |
| $\phi$ | Image Encoder |
| $\mathbf{F}_{img}$ | Features of Augmented Images |
| $\mathcal{T}$ | Text Set |
| $\varphi$ | Text Encoder |
| $\mathbf{X}_{txt}$ | Text Features |

Given an original set of images $\mathcal{I}$ and a set of scaling factors $\mathcal{A}$, the augmented image set $\mathcal{I}_{aug}$ can be obtained by first applying a horizontal flipping function $f_{\text{flip}}$ to each image $I\in\mathcal{I}$, followed by a scaling function $f_{\text{scale}}$ for each scaling factor $\alpha\in\mathcal{A}$:

$$ $\mathcal{I}_{aug}=\bigcup_{\alpha\in\mathcal{A}}\left(f_{\text{scale}}(f_{ \text{flip}}(I),\alpha)\right)$ (5) $$

where $f_{\text{flip}}$ horizontally flips the image, and $f_{\text{scale}}$ applies the scaling transformation to the image, for each $\alpha$.Given the augmented image set $\mathcal{I}_{aug}$ and the original text set $\mathcal{T}$, we apply the image encoder $\phi$ and the text encoder $\varphi$ to obtain the corresponding feature representations:

$$ $\mathbf{F}_{img}=\phi(\mathcal{I}_{aug})$ (6) $$

$$ $\mathbf{X}_{txt}=\varphi(\mathcal{T})$ (7) $$

where $\mathbf{F}_{img}$ represents the features of the augmented images, and $\mathbf{X}_{txt}$ represents the features of the original texts. The function $\phi$ symbolizes the operation of the image encoder and $\varphi$ the operation of the text encoder.Then the similarity matrix $\mathbf{S}$ can be calculated as:

$$ $\mathbf{S}=\mathbf{F}_{img}\odot\mathbf{X}_{txt}$ (8) $$

where $\mathbf{S}$ represents the similarity matrix. The operation $\odot$ denotes matrix multiplication, which computes the similarity between each pair of image and text features.

In previous research, the direct application of the matrix $\mathbf{S}$ to guide the generation of adversarial examples has, after several iterations, led to the reinforcement of the most feature-similar regions within the purview of the white-box model during the adversarial example creation process, with minimal perturbation occurring in other regions. When these adversarial examples encounter a new black-box model, the focal points of concern differ from those of the white-box model. Since the regions of interest to the black-box model have undergone only limited perturbation, they fail to disrupt the black-box model, resulting in an unsuccessful attack.

$$ $loss_{ori}=-\left(\sum_{i}\mathbf{S}_{i}\right)_{\text{mean}}$ (9) $$

Here, the summation $\sum_{i}\mathbf{S}_{i}$ is taken over the last dimension of the similarity matrix $\mathbf{S}$, and the mean of this sum is computed to obtain the final loss value $loss_{ori}$.

#### 3.3.2   Optimal Transport

Defining Source and Target Distributions. Initially, we define two pivotal distributions within the Optimal Transport framework: the source distribution $\mathbf{P}$ and the target distribution $\mathbf{Y}$. These distributions represent the starting and ending points of the transportation journey in the Optimal Transport problem. The source distribution $\mathbf{P}=(p_{1},p_{2},\ldots,p_{n})$ and the target distribution $\mathbf{Y}=(y_{1},y_{2},\ldots,y_{m})$ encapsulate the quantities to be transported from and to each respective location.

The Transportation Matrix $\mathbf{T}$. In the context of Optimal Transport, the matrix $\mathbf{T}=[T_{ij}]$ of size $n\times m$ is referred to as the transportation matrix. Each element $T_{ij}$ of this matrix represents the amount of a commodity or resource transported from the $i$-th source in $\mathbf{P}$ to the $j$-th target in $\mathbf{Y}$. The matrix $\mathbf{T}$ effectively captures the transportation scheme between the sources and targets.

The matrix $\mathbf{T}$ must satisfy specific constraints to ensure an optimal transportation plan. Firstly, the Marginal Constraints:

$$ $\displaystyle\sum_{j=1}^{m}T_{ij}$ $\displaystyle=p_{i}\quad\forall i\in\{1,\ldots,n\},$ (10) $\displaystyle\sum_{i=1}^{n}T_{ij}$ $\displaystyle=y_{j}\quad\forall j\in\{1,\ldots,m\}.$ (11) $$

These constraints require that the total transported amount from each source $i$ and to each target $j$ matches the respective supply $p_{i}$ and demand $y_{j}$.

Additionally, the Non-Negativity Constraint is imposed:

$$ $T_{ij}\geq 0\quad\forall i\in\{1,\ldots,n\},\forall j\in\{1,\ldots,m\}.$ (12) $$

This condition ensures that all transport amounts $T_{ij}$ in the matrix $\mathbf{T}$ are non-negative, reflecting the practical impossibility of negative transportation.

Modeling the Optimal Transport Problem. With the aforementioned definitions and constraints established, the Optimal Transport problem can be formulated as follows:

$$ $OT(\mathbf{P},\mathbf{Y},\mathbf{C})=\min_{\mathbf{T}\in\Pi(\mathbf{r},\mathbf {c})}\sum_{i,j}T_{ij}C_{ij}$ (13) $$

Here, $\mathbf{C}$ denotes the cost matrix, with each element $C_{ij}$ representing the cost of transporting a unit from source $p_{i}$ to target $y_{j}$. The matrix $\mathbf{T}$ signifies the transportation scheme, while $\Pi(\mathbf{r},\mathbf{c})$ encompasses all feasible transportation schemes that satisfy the marginal constraints.

The Sinkhorn distance is utilized in Optimal Transport (OT) for its effectiveness in high-dimensional spaces. Traditional OT approaches, based on linear programming, face challenges with computational intensity and scaling with data dimensionality. In contrast, the Sinkhorn distance applies entropy regularization to the OT calculation, enhancing tractability and differentiability. This approach uses a regularization parameter $\lambda$, which balances accuracy and computational efficiency. Higher $\lambda$ values lead to results closer to traditional OT but at increased computational costs, while lower values of $\lambda$ expedite calculations at the expense of some bias. Therefore, the Sinkhorn distance, often computed using the Sinkhorn-Knopp algorithm, presents a more feasible solution for OT in machine-learning scenarios that demand scalability and stability in computations.
The Sinkhorn Optimization Process can be defined as:

$$ $OT_{\lambda}(\mathbf{P},\mathbf{Y},\mathbf{C})=\min_{\mathbf{T}\in\Pi(\mathbf{ r},\mathbf{c})}\sum_{i,j}T_{ij}C_{ij}+\lambda H(\mathbf{T})$ (14) $$

The Sinkhorn algorithm iteratively normalizes the rows and columns of the transport matrix to satisfy the marginal constraints while minimizing the regularized objective function[]. Here, $H(\mathbf{T})$ is the entropy of the transport matrix, introducing regularization (controlled by $\lambda$) to ensure numerical stability and efficient computation. Regarding the computation of Sinkhorn,
the algorithm of the proposed OT-Attack is summarized in Algorithm [1](#alg1).

Figure: Algorithm 1 Sinkhorn Iteration for Optimal Transport

#### 3.3.3   Calculating Loss through Optimal Transport

The Optimal Transport loss $\text{loss}_{OT}$ is computed using the feature representations of augmented images $\mathbf{F}_{img}$, original texts $\mathbf{X}_{txt}$, and the similarity matrix $\mathbf{S}$.

Figure: Algorithm 2 Adversarial Image Generation

Firstly, the cost matrix $\mathbf{C}$ is defined as $\mathbf{C}=1-\mathbf{S}$, where $\mathbf{S}$ is the similarity matrix. This transformation converts similarity scores into a cost structure. Subsequently, we compute the exponentiated negative cost matrix $\mathbf{K}$ for the Sinkhorn iterations, defined as $\mathbf{K}=\exp\left(-\frac{\mathbf{C}}{\lambda}\right)$, with $\lambda$ being a small positive regularization parameter. The Optimal Transport loss is calculated as:

$$ $\text{loss}_{OT}=\sum_{i,j}T_{ij}C_{ij}$ (15) $$

where $T_{ij}$ in $\mathbf{T}$ represents the optimal ’transport’ of features from the $i$-th element in $\mathbf{F}_{img}$ to the $j$-th element in $\mathbf{X}_{txt}$, and $C_{ij}$ is the corresponding cost in $\mathbf{C}$.

This formulation of $\text{loss}_{OT}$ captures the minimal cost of aligning the feature representations of the augmented images with the original texts, facilitating a more effective generation of adversarial examples. In essence, this approach for computing OT loss in adversarial example generation accounts for the overall feature distribution, addressing potential overfitting issues inherent in using a similarity matrix as the sole loss metric. It ensures that adversarial examples generated through this method exhibit better transferability to novel black-box models. The process of generating adversarial images is detailed in Algorithm [2](#alg2).

We employed the adversarial example generation method outlined in Equation [1](#S3.E1) to create adversarial samples. These samples were then used to mount attacks on black-box models.

## 4 Experiments

Table: TABLE II: Adversarial Attack Success Rates on Image-Text Retrieval. The best results are boldfaced. The table presents the attack success rate at Rank 1 (ASR @ R1) for text-image retrieval (IR) and text-image retrieval (TR) tasks using the Flickr30K dataset. The originating models for generating adversarial samples are listed under the ‘Source’ column. White-box attacks are distinguished by a gray background, emphasizing their distinct context in the evaluation. The results demonstrate the proposed OT-Attack’s impressive performance in achieving SOTA transferability of adversarial samples across different VLP models.

Table: TABLE III: Visualization of Adversarial Examples in Image-Text Matching on the MSCOCO Dataset. The best results are boldfaced. It illustrates the effects of adversarial attacks on both images and their associated captions. The top row displays the original, clean images with their corresponding accurate captions. The middle row presents the adversarial images and the modified captions that resulted from the attacks. Key alterations in the text are indicated in red, showcasing the change in the description of the visual content due to the adversarial manipulation. The bottom row quantifies the pixel differences between the original and adversarial images, highlighting the subtlety of the visual perturbations.

Figure: Figure 4: Visualization of adversarial examples from Flickr30K. In the task of image-text matching, adversarial examples for both images and texts were generated and utilized for image-to-text and text-to-image matching tasks, respectively. We have highlighted the distinctions in the text adversarial examples compared to the original samples and also quantified the pixel differences between the image adversarial examples and the original images.
Refer to caption: extracted/5281202/figure/Visualization.png

### 4.1 Settings

VLP Models.
To assess the transferability of adversarial examples and the efficacy of our proposed framework, we employed two distinct categories of Vision-Language Pre-training (VLP) models: fused VLP and aligned VLP. These were respectively designated as the source and target models in our experiments. Fused VLP models are characterized by their early integration of visual and linguistic information within the processing pipeline. Specifically, these models concurrently handle images and text, extracting and processing both types of data through shared layers. Within this category, we selected ALBEF[] and TCL[] as representative fused models. Both models incorporate a 12-layer ViT-B/16[] visual transformer and two separate 6-layer transformers for image and text encoding, with their primary differences arising from distinct pre-training objectives. Conversely, aligned VLP models process visual and textual data independently at the initial stages and subsequently align these representations in the deeper layers of the model. This approach facilitates the learning of intricate relationships between the two modalities. In this category, we opted for two variants of CLIP[] as our representatives: CLIPViT, which employs ViT-B/16 for image encoding, and CLIPCNN, which uses a ResNet-101[] architecture for the same purpose. For the targeted task of image captioning within our cross-task attack efficacy study, we incorporated BLIP as the target model, assessing adversarial examples’ performance in the black-box setting when transitioning from TCL as the source model to ALBEF.

Datasets.
For the image-text retrieval task, our study utilized two datasets renowned for their breadth and depth: Flickr30K[] and MSCOCO[]. Flickr30K boasts a diverse corpus of 31,783 images, while MSCOCO expands the dataset considerably with 123,287 images. A salient characteristic shared by both is the quintuple of descriptive captions accompanying each image, providing a valuable asset for the assessment of our image-text retrieval approach. For the task of Visual Grounding, we employed the RefCOCO+[] dataset, which further enriched our cross-task attack effectiveness analysis.

Baselines
In our research involving Vision-Language Pre-training (VLP) models, we implemented several prevalent adversarial attack methods as baselines. These included using PGD[] exclusively on images, applying BERT-Attack[] only to texts, and separately utilizing PGD and BERT-Attack on both images and texts without integrating inter-modality interactions, a technique designated as Sep-Attack. Additionally, we employed Co-Attack[], which integrates information between individual image-text pairs, and Set-level Guidance Attack (SGA)[], which utilizes guidance information across modalities between sets. Each baseline was tested under identical conditions for a consistent comparative analysis.

Adversarial Attack Configuration.
In order to validate the effectiveness of our proposed framework, we employed the Projected Gradient Descent (PGD)[] method for generating adversarial visual examples. The configuration settings for PGD were as follows: a perturbation limit of $\epsilon_{v}=\frac{2}{255}$, a step magnitude of $\alpha=\frac{0.5}{255}$, and a total of $T=10$ iterations. For textual attacks, we utilized the BERT-Attack[] method, setting a disturbance limit to $\epsilon_{t}=1$ and defining a vocabulary list length of $W=10$. In our experimentation with Sep-Attack and Co-Attack, we adhered to the previously mentioned settings. Specifically for Co-Attack, in addition to these settings, we also incorporated the similarity between individual image pairs as a loss metric, guiding the generation of adversarial examples through inter-modality interactions. In the case of SGA, we followed the experimental conditions outlined in its original publication. Specifically, we enhanced the images by rescaling them to five distinct sizes ${0.5,0.75,1.0,1.25,1.5}$. To validate that our method effectively mitigates overfitting, we enhanced the image set by rescaling it to six different sizes: ${0.5,0.75,1.0,1.25,1.5,2}$, utilizing bicubic interpolation. Additionally, we applied horizontal flipping to the images. This form of data augmentation, which led to significant overfitting in SGA, was employed to rigorously test the robustness of our approach against such overfitting tendencies. In parallel, the text corpus was expanded by selecting and refining the top five most closely matched caption pairs for each image in the dataset. Furthermore, we incorporated the Sinkhorn[] algorithm for calculating the optimal transport plan. This algorithm was chosen for its efficient use of matrix operations, which accelerates the process and ensures convergence to an approximate solution of the original transport problem. To avoid infinite iterations, we set a convergence threshold $\text{thresh}=1e-2$. The iteration process was terminated once the average absolute difference of the vector $r$ between two consecutive iterations fell below this threshold, indicating the achievement of algorithmic convergence.

Evaluation Criteria.
In our study, the robustness and transferability of the adversarial attacks, in both white-box and black-box scenarios, are quantitatively assessed using the Attack Success Rate (ASR). ASR is a crucial metric that measures the proportion of successful adversarial examples out of the total number of attacks conducted. A higher ASR is indicative of increased transferability of the adversarial examples, signifying the effectiveness of the attack in compromising the model under various conditions. The ASR is computed as follows:

$$ $ASR=\frac{N_{\text{success}}}{N_{\text{total}}}\times 100\%$ (16) $$

where $ASR$ denotes the Attack Success Rate, $N_{\text{success}}$ represents the number of successful attacks, and $N_{\text{total}}$ is the total number of attacks conducted. The formula calculates the percentage of successful attacks, providing a quantitative measure of the attack’s effectiveness.

### 4.2 Comparative Experimental Results

In our experiments, we primarily focused on Image-Text Retrieval (ITR) tasks. We generated adversarial examples on various white-box models and then evaluated their effectiveness by calculating the attack success rates on both the white-box models and three additional black-box models. Specifically, we computed the success rates for both image-to-text and text-to-image matching attacks to demonstrate the comprehensiveness of our method’s impact on adversarial sample generation. This deliberate selection of models and tasks was instrumental in enabling an exhaustive examination of the adversarial examples’ ability to generalize across varying architectures, which is a critical factor in the real-world application of these models.

Our analysis spanned two widely recognized datasets: Flickr30K, with a sample of 1,000 images and 5,000 captions, and MSCOCO, which provided a larger pool of 5,000 images and 25,000 captions. This broad dataset coverage allowed us to conduct a robust evaluation of our attack methods in image-text matching tasks, quantifying the success of adversarial examples in misleading these complex models. The detailed outcomes are methodically presented in TABLE [II](#S4.T2) and TABLE [III](#S4.T3).

Our results demonstrated that the OT-Attack method made significant strides in the creation of adversarial examples that were not only effective within models of the same type but also exhibited impressive cross-type attack success. This is particularly evident from the R@1 success rates in TR and IR tasks, where our adversarial examples maintained high effectiveness across varied models including ALBEF, TCL, CLIPViT, and CLIPCNN. For example, when using ALBEF to target TCL, our method improved the TR R@1 attack success rate by 6.95% on Flickr30K and 4.88% on MSCOCO, compared with the state-of-the-art results obtained by SGA. Conversely, in scenarios where TCL was employed to target ALBEF, our approach showed significant improvements over SGA, with increases of 8.41% on Flickr30K and 5.71% on MSCOCO in the TR R@1 attack success rate. The results demonstrate the effectiveness of our proposed method in improving adversarial transferability.

Complementing our numerical analysis, Figure [4](#S4.F4) offers a visual representation of the impact of our adversarial examples. It contrasts the original images and texts with their modified versions, illuminating how subtle perturbations can drastically alter a model’s performance in image-text matching tasks. The visual differences, particularly the nuanced texture changes introduced in the adversarial images, are made evident through difference masks, underscoring the deceptive potency of the adversarial examples and their potential to misguide VLP systems. This visual depiction reinforces the quantitative data, providing a holistic view of our adversarial attack’s efficacy. Hence, compared with previous works, our proposed OT-Attack can achieve better adversarial transferability.

### 4.3 Cross-Task Transferability

#### 4.3.1   Image Captioning

Figure: Figure 5: Comparison of Clean and Adversarial Image Captions. This figure juxtaposes the original clean images with their accurate captions against adversarial images and the resulting captions generated by the BLIP model. The adversarial examples were created using the ALBEF model as a white-box framework on the Dataset Flickr30K. Despite the perturbations being subtle, and limited to a magnitude of 2, the adversarial examples show minimal visual deviation from the original images. However, these slight alterations are significant enough to mislead the captioning model, leading to discrepancies in the generated captions, as evidenced by the erroneous and sometimes nonsensical descriptions.
Refer to caption: extracted/5281202/figure/caption.png

**TABLE IV: Adversarial Impact on Image Captioning Metrics. This table displays the results of adversarial attacks on the image captioning task, where 10,000 MSCOCO dataset images were used. Adversarial samples were crafted using the ALBEF model in a white-box attack scenario, and captions were subsequently generated with the BLIP model. The performance of these attacks was assessed by measuring the BLEU-4 (B@4), METEOR, ROUGE-L, CIDEr, and SPICE metrics. Lower scores on these metrics indicate a more effective attack, revealing a significant deviation of the generated captions from the expected, accurate descriptions. The results underscore the effectiveness of our proposed method in comparison to other attacks, demonstrating its capability to disrupt the caption generation process by inducing semantic errors.**
| Attack | B@4 | METEOR | ROUGE-L | CIDEr | SPICE |
| --- | --- | --- | --- | --- | --- |
| Baseline | 39.7 | 31.0 | 60.0 | 133.3 | 23.8 |
| Co-Attack | 37.4 | 29.8 | 58.4 | 125.5 | 22.8 |
| SGA | 34.8 | 28.4 | 56.3 | 116.0 | 21.4 |
| OT-Attack (Ours) | 34.1 | 27.9 | 55.7 | 112.6 | 20.9 |

Figure: Figure 6: Impact of Adversarial Attacks on GPT-4 and Bing Chat Descriptions. This figure showcases the alterations in image descriptions by GPT-4 and Bing Chat before and after adversarial attacks. Original descriptions are compared to those generated from manipulated images, with increased perturbation strength and iteration count to mislead the AI models. The stark contrast in the outputs highlights the susceptibility of these models to adversarial examples, reflecting the effectiveness of the perturbations in altering the perceived content of the images.
Refer to caption: extracted/5281202/figure/gpt.png

In our research, we employed the ALBEF model in a white-box setting to generate adversarial examples specifically designed to target the BLIP[] framework. Renowned for its innovative multimodal encoder-decoder model, BLIP is pre-trained on a dataset enriched with diverse synthetic captions and noise reduction techniques. We chose the MSCOCO dataset for our evaluations, focusing on both original and adversarially manipulated image samples. To quantify the effectiveness of our adversarial approach, we utilized a suite of metrics for image captioning tasks, each offering distinct insights: BLEU[] assesses the precision of generated captions by comparing them with reference captions, focusing on the exactness of word usage and sequence. METEOR[] expands upon BLEU by including synonym matching and stemming, providing a more comprehensive view of semantic accuracy. ROUGE[] measures recall, evaluating how much of the reference content is captured in the generated captions. CIDEr[] focuses on the distinctiveness and relevance of captions by assessing consensus with reference captions. Lastly, SPICE[] evaluates the semantic content of captions, assessing their fidelity to the actual objects, attributes, and relationships in the image.

Each metric provides a different perspective on the quality and relevance of the generated text, offering a comprehensive understanding of the adversarial impact. The detailed results of these evaluations are compiled in TABLE [IV](#S4.T4). Compared to SGA, our method exhibited a decrease in performance across various metrics: a reduction of 0.7 in BLEU-4, 0.5 in METEOR, 0.6 in ROUGE-L, 3.4 in CIDEr, and 0.5 in SPICE. Notably, these declines in scores are indicative of enhanced effectiveness in cross-task attacks, as a greater decrease signifies better performance of our method in this context.

To convey the practical implications of our findings, Figure [5](#S4.F5) provides a visual comparison of select experimental results. This figure juxtaposes the original images and their corresponding captions with the adversarial counterparts, highlighting the stark differences. These visual examples compellingly illustrate how subtle and seemingly innocuous perturbations can lead to significant deviations in the model’s interpretation, often diverging entirely from the original context or meaning of the image.

Further delving into the realm of large-scale models, our experiments were conducted with specific parameters to gauge the extent of adversarial impact. We set the perturbation intensity at a subtle yet effective level of 16/255 and ran our adversarial process for 500 iterations. To assess the broader applicability and effectiveness of our attacks, we tested them on advanced models like GPT-4 and Bing Chat, posing the query “Describe this image” to these systems. The findings, illustrated in Figure [6](#S4.F6), reveal a notable level of success in our adversarial attacks, with these sophisticated models showing susceptibility to being misled.

**TABLE V: Performance on Visual Grounding Task Across RefCOCO+ Subsets. This table evaluates the success of different adversarial attacks across the validation (Val), TestA, and TestB subsets of the RefCOCO+ dataset. The TCL model served as the source for generating adversarial attacks, while the ALBEF model was the target for evaluating their effectiveness. Lower scores indicate a more successful adversarial attack, highlighting the inverse relationship between the evaluation metrics and the attack’s performance. The results demonstrate the comparative effectiveness of our proposed methodology in reducing the ALBEF model’s accuracy in the visual grounding task, as evidenced by the decreased performance across all subsets.**
| Attack | Val | TestA | TestB |
| --- | --- | --- | --- |
| Baseline | 58.4 | 65.9 | 46.2 |
| SGA | 56.5 | 63.7 | 45.4 |
| OT-Attack (Ours) | 56.3 | 63.5 | 45.0 |

#### 4.3.2   Visual Grounding

Figure: Figure 7: Visualization results for Visual Grounding. We employed TCL as the source model and ALBEF as the target model, with captions sourced from the Flickr30K dataset. The adversarial examples exhibit limited visual differences from the original samples; however, they disrupt the model’s judgment of visual elements in the Visual Grounding task. Compared to clean data, the localization results for the same elements may have shifted or dispersed. The visualizations of Visual Grounding vividly demonstrate the disruptive impact of adversarial examples on the model.
Refer to caption: extracted/5281202/figure/VG.png

To thoroughly evaluate the efficacy of our adversarial attack strategies, we utilized the RefCOCO+[] dataset, meticulously designed for the visual grounding task. This dataset is composed of several subsets, each tailored to assess different facets of model performance. These subsets include RefCOCO+ val, RefCOCO+ testA, and RefCOCO+ testB. Specifically, the RefCOCO+ val subset presents a wide array of scenarios for a comprehensive assessment, RefCOCO+ testA is dedicated to gauging the model’s proficiency in identifying and localizing human figures, and RefCOCO+ testB concentrates on non-human elements like inanimate objects, animals, and various environmental features. By leveraging the diverse testing environments provided by RefCOCO+, our goal is to showcase the extensive adaptability and transferability of our method across a multitude of visual grounding challenges.

The quantitative analysis presented in TABLE [V](#S4.T5) assesses the performance of our adversarial examples across different tasks. In this evaluation, TCL served as the source model from which adversarial examples were generated, while the ALBEF model was the target for these attacks. The baseline in the table refers to the scores obtained when the original, unaltered samples were evaluated, providing a control benchmark for our study. Alongside the baseline, both SGA and OT-Attack strategies were employed. Compared to SGA, our attack method proved to be more effective, as evidenced by the decrease in scores for the ALBEF model due to our adversarial examples: a reduction of 0.2 on the Val subset, 0.2 on TestA, and 0.3 on TestB. This outcome highlights the effectiveness of our method in compromising the Visual Grounding capabilities of the target model.

Beyond the numerical assessments, we also employed a visualization technique to further elucidate the impact of our attacks on Visual Grounding tasks, as depicted in Figure [7](#S4.F7). The base samples for this analysis were derived from the Flickr30K dataset, with adversarial examples being crafted using the ALBEF model in a white-box configuration. Our visual representations reveal that even subtle perturbations can significantly disrupt the model’s ability to accurately recognize and localize objects. This underscores the substantial and nuanced impact that these adversarial attacks can have on a model’s performance in visual grounding tasks, challenging its reliability and precision in real-world applications.

## 5 Conclusion

In this paper, we focus on improving the adversarial transferability of vision-language pre-training models. In detail, recent works have found that using inter-modality interaction and data augmentation can significantly enhance the transferability of adversarial examples for vision-language pre-training models. Unfortunately, previous works ignore the optimal alignment problem between data-augmented image-text pairs, which leads the generated adversarial examples overfit to the source model and achieves limited adversarial transferability improvement. To address the issue, we propose an Optimal Transport-based Adversarial Attack, *dubbed* OT-Attack. The proposed OT-Attack formulates the features of image and text sets as two distinct distributions, leveraging optimal transport theory to identify the most efficient mapping between them. It utilizes their mutual similarity as the cost matrix. The derived optimal mapping guides the generation of adversarial examples, effectively mitigating overfitting issues and improving adversarial transferability. Extensive experiments across diverse network architectures and datasets in image-text matching tasks demonstrate the superior performance of the proposed OT-Attack compared to existing methods in terms of adversarial transferability. Significantly, our results also show that OT-Attack is also effective in cross-task attacks, including image captioning and visual grounding, and poses a considerable challenge to commercial models such as GPT-4 and Bing Chat, highlighting the evolving landscape of adversarial threats in advanced AI applications.
This underscores the need for robust defenses against sophisticated attacks.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What's the convergence condition for solving the optimal transport problem in this proposed setting?
2. What's the time cost is like for solving the optimal transport problem in this proposed setting?
soundness: 2
strengths: This paper introduces a new perspective on an existing problem. The paper considers optimal alignment between augmented image-text pairs for more transferable adversarial attack patterns. This idea makes sense and is original to the best of my knowledge.
summary: This paper focuses on the task of conducting adversarial attacks on vision-language models. The focus of the attack is transferability. The paper introduces the optimal transport perspective for finding more transferable attacks by considering the optimal alignment between augmented image-text pairs. In experiments, the paper compares the proposed method with single-modality attacks, separate attacks by two single-modality attacks, and two multi-modal attacks.
weaknesses: 1. The evaluation baselines are not comprehensive in experimental design. The use of single-modality attacks PGD/Bert-Attack as baselines is weak in proving the effectiveness of the proposed method. There are certain representative multi-modal adversarial attacks not included for comparison. E.g., VLATTACK by Yin et al. in NeurIPS 2023, which focuses on the transferability of attacking pretrained vision-language models. These representative multi-modal attacks are not discussed in the paper either. 
2. Although the perspective is new, the proposed solution is proposed to solve an existing problem, which makes the contribution trivial. Meanwhile, the paper lacks technical depth for discussing the applications of optimal transport in this problem. First, the paper lacks a discussion on the computational overhead caused by optimal transport. This paper introduces the application of it in vision-language models. However, the computation of optimal transport needs a certain time. The paper hasn't discussed it in the paper. Second, the paper didn't discuss the convergence of optimal transport for vision-language model attack either. Such algorithms need careful selection of hyperparameters. Thus, although the designed method sounds reasonable, this paper lacks technical depth for discussing the tradeoff and convergence.
3. Formatting problem exists in the current manuscript: e.g., L251, 'transport'. L411, '1 3 5 7' is not centered in table 4.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 2
questions: Please refer to the weaknesses.
soundness: 3
strengths: - This paper highlights an interesting point: the SGA attack overlooks true image-text matching when using input augmentations. The motivation is clear.
- The method is simple.
- OT-Attack consistently improves SGA's attack success rate in three tasks: image-text retrieval, image captioning, and visual grounding.
summary: This paper proposes a new adversarial attack for vision-language models called OT-Attack, pointing out that the existing SGA attack has a limitation in considering correct image-text matching after augmentations.

OT-Attack utilizes Optimal Transport to capture valid image-text matching between augmented images and captions (calculate transportation matrix $T_{ij}$). Then, adversarial attack optimization maximizes the $loss_{OT}$ ($\sum T_{ij} C_{ij}$) to update an adversarial image. 

The method improves the transferability of the SGA attack in three tasks: image-text retrieval, image captioning, and visual grounding.
weaknesses: - [W1] The presentation is not optimal.
    - [W1.1] The citation style is wrong: Authors should correctly use \citet and \citep. Please refer to the guideline carefully and modify them. 
    - [W1.2] While the authors highlight OT-Attack’s successful attack against GPT-4 and Bing Chat at the end of the Introduction, there is no result in the main text.  Even in the Appendix, there is only one example without a comparison between attack methods, which makes it unreasonable to highlight this in the introduction. I suggest adding quantitative results or remove the claim.
- [W2] OT optimization introduces additional computational cost. How much does the computational cost differ between SGA and OT-Attack? 
- [W3] Sensitivity to the regularization hyperparameter is unclear. How does the hyperparameter $\lambda$ affect the attack success rate (ASR) and computational cost? Does more accurate OT optimization lead to better ASR? I suggest adding an ablation study table.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Co-attack and SGA were proposed prior to 2023, and since then, many new vision-language models have emerged. To further validate the effectiveness of the proposed method, the authors should evaluate additional vision-language models, such as Eva-CLIP and BLIP2 for image-text retrieval tasks, and MiniGPT4 and Qwen2-VL for image generation tasks in a transfer-based setting.  
2. I did not find any ablation experiments addressing the role of image-text augmentation and optimal transport theory in OT-attack.  
3. The authors claim that optimal transport can enhance adversarial example transferability; I recommend providing a theoretical explanation and further experimental analysis to support this assertion.
soundness: 3
strengths: 1. This paper aims to enhance the transferability of adversarial examples for vision-language pre-training models, addressing a practical and increasingly relevant issue as these models gain importance in real-world applications.  
2.  Adversarial examples generated by OT-Attack are tested on commercial models like GPT-4 and BingChat.
summary: In this paper, the authors propose a transfer-based adversarial attack method for vision-language pre-training (VLP) models that leverages image-text augmentations and optimal transport theory to enhance adversarial example transferability. Experiments are conducted across various vision-language tasks and VLP models.
weaknesses: 1. Overall, the manuscript presents valuable insights; however, enhancing the clarity and coherence of the writing, particularly in the methods section, would significantly improve its readability and impact.
2. The titles of Tables 1 and 2 are identical, which may easily confuse readers and hinder their understanding of the experiments. Differentiating the titles would improve clarity and facilitate comprehension of the authors' work.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the time cost associated with using optimal transport theory?
2. How are augmented texts generated?
soundness: 3
strengths: 1. This paper identifies a key issue in existing attacks: a mismatch in augmentations applied to images versus text. To address this, the authors utilize optimal transport theory, enhancing transferability. The motivation and proposed solutions are logical and well-founded.
2. The authors conduct extensive experiments across four VLP models and multiple tasks, effectively demonstrating the superior performance of their approach.
summary: This paper investigates data augmentations in the context of transferable attacks on Vision-Language Processing (VLP) models. Compared to existing methods, the proposed approach achieves improved alignment between augmented images and text. Specifically, the authors leverage optimal transport theory to derive the most efficient mapping between augmented image distributions and text distributions. They conduct experiments on image-text retrieval, image captioning, and visual grounding tasks to demonstrate their improved transferability.
weaknesses: My primary concern is the focus on attacking VLP models solely through adversarial image generation, while neglecting the equally crucial role of text in these models. Since texts play an integral part in VLP models, a comprehensive approach should consider adversarial perturbations for both modalities. For example, incorporating a text-to-image generation component could be beneficial to observe how adversarial texts lead to variations in generated images.

Meta-review: This paper focuses on the task of conducting adversarial attacks on vision-language models. The focus of the attack is transferability. The paper introduces the optimal transport perspective for finding more transferable attacks by considering the optimal alignment between augmented image-text pairs. In experiments, the paper compares the proposed method with single-modality attacks, separate attacks by two single-modality attacks, and two multi-modal attacks. For the strengths, this paper highlights an interesting point: the SGA attack overlooks true image-text matching when using input augmentations. The motivation is clear, and the method is simple. Moreover, OT-Attack consistently improves SGA's attack success rate in three tasks: image-text retrieval, image captioning, and visual grounding. However, there are several points to be further improved. For example, the evaluation baselines are not comprehensive in experimental design. Moreover, although the perspective is new, the proposed solution is proposed to solve an existing problem, which makes the contribution trivial. Besides, enhancing the clarity and coherence of the writing, particularly in the methods section, would significantly improve its readability and impact. Therefore, this paper cannot be accepted at ICLR this time, but the enhanced version is highly encouraged to submit other top-tier venues.

Decision: Reject