X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP
Authors: Hanxun Huang
Sarah Monazam Erfani
Yige Li
Xingjun Ma
James Bailey
Venue: ICML 2025 poster (2025)
Keywords: Adversarial
CLIP
Universal
UAP
VLM
vision language model
robustness
transferability
universal adversarial perturbation
Primary area: deep_learning->robustness
TL;DR: X-Transfer generates a single perturbation capable of deceiving different CLIP encoders and downstream large VLMs across various tasks, data points, and domains.

Abstract: As Contrastive Language-Image Pre-training (CLIP) models are increasingly adopted for diverse downstream tasks and integrated into large vision-language models (VLMs), their susceptibility to adversarial perturbations has emerged as a critical concern. In this work, we introduce **X-Transfer**, a novel attack method that exposes a universal adversarial vulnerability in CLIP. X-Transfer generates a Universal Adversarial Perturbation (UAP) capable of deceiving various CLIP encoders and downstream VLMs across different samples, tasks, and domains. We refer to this property as **super transferability**—a single perturbation achieving cross-data, cross-domain, cross-model, and cross-task adversarial transferability simultaneously. This is achieved through **surrogate scaling**, a key innovation of our approach. Unlike existing methods that rely on fixed surrogate models, which are computationally intensive to scale, X-Transfer employs an efficient surrogate scaling strategy that dynamically selects a small subset of suitable surrogates from a large search space. Extensive evaluations demonstrate that X-Transfer significantly outperforms previous state-of-the-art UAP methods, establishing a new benchmark for adversarial transferability across CLIP models.

X- X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP
Hanxun Huang' Sarah Erfani
Yige Li
Xingjun Ma
James Bailey'
Abstract
ing (Radford al., 2021). Pre-trained on web-scale datasets, CLIP encoders have been extensively used to en- hance performance across variety of downstream applica- tions, particularly in large Vision-Languaa Models (VLMs) (Awadalla et al., 2023; Koh et al., 2023; Wang et al., 2023a; Bai et al., 2023; Karamcheti et al., 2024; Jiang et al., 2024), where they form the backbone of visual capabilities. Models such as Flamingo (Alayrac al., 2022), LLaVA (Liu et al., 2023a), BLIP2 (Li et al., 2023a), and MiniGPT-4 (Zhu et al., 2024) integrate CLIP image encoders with Large Language Models (LLMs) (Zhang et al., 2022b: Hoffmann et al., 2022; Chiang et al., 2023). The widespread adoption of CLIP in VLMs is primarily driven by its pre-training paradigm util- ising text supervision (Tong et al., 2024). While its strong generalisation capabilities solidify its role as a cornerstone for VLMs, they also make CLIP an ideal target for gener- ating highly transferable adversarial perturbations, thereby introducing new safety risks.
As Contrastive Language-Image Pre-training (CLIP) models are increasingly adopted for di- verse downstream tasks and integrated into large vision-language models (VLMs), their suscepti- bility to adversarial perturbations has emerged as a critical concern. In this work, we introduce X-Transfer, novel attack method that exposes universal adversarial vulnerability in CLIP. X- Transfer generates Universal Adversarial Pertur- bation (UAP) capable of deceiving various CLIP encoders and downstream VLMs across differ- ent samples, tasks, and domains. We refer to this property as super transferability- single perturbation achieving cross-data, cross-domain, cross-model, and cross-task adversarial transfer- ability simultaneously. This is achieved through surrogate scaling, a key innovation of our ap- proach. Unlike existing methods that rely on fixed surrogate models, which are computationally in- tensive to scale, X-Transfer employs an efficient surrogate scaling strategy that dynamically selects small subset of suitable surrogates from large search space. Extensive evaluations demonstrate that X- Transfee significantly outperforms previ- ous state-of-the-art UAP methods, establishing a new benchmark for adversarial transferability across CLIP models. The code is publicly avail- able in our GitHub repository.
Deep neural networks are widely recognised for their sus- ceptibility to Universal Adversarial Perturbations (UAPs) (Moosavi-Dezfooli et al., 2017: Gao et al., 2023; Zhou et al., 2023b; 2024; Zhang et al., 2025c; Song et al., 2025), where a perturbation generated using a specific dataset can transfer to images within the same domain, causing erro- neous classifications by image classifiers. Recent studies (Fang et al., 2024b; Zhang et al., 2024) have demonstrated that UAPs are also effective against CLIP encoders. How- ever, existing works have yet to fully realise the potential of UAPs- super transferability. Ensemble tech- niques are well-established strategy for enhancing cross- model adversarial transfcrability (Liu et al., 2017; Dong et al., 2018; Xiong et al., 2022; Chen et al., 2024a) for sample-specific perturbations, but they leave significant gaps in the applicability of UAPs to broader transfer scenarios. Furthermore, these methods rely on a heuristic selection of fixed set of surrogate models, which becomes com- putationally expensive when scaling to large number of surrogates (Liu et al., 2024). To address these gaps, we aim to answer the following two questions: (1) Can a single per- turbation simultaneously achieve cross-data, cross-domain, cross-model, and cross-task adversarial super transferabil- ity? and (2) How scalable is super transferability when incorporating large numbers of surrogate models?
1. Introduction
Contrastive Language-Image Pre-training (CLIP) is a widely adopted technique that learns aligned multi-modal repre- sentations from text-image pairs through contrastive learn-
'School of Computing and Information Systems, The Univer- sity of Melbourne. Australia "School of Computing and Infor- mation Systems, Singapore Management University, Singapore "School of Computer Science, Fudan University, China. Cor- respondence to: Yige <yigeli@ smu.edu.sg>, Xingjun Ma <xingjunma@e @ fudan. m.dan.edu.cn.>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, propose the X-Transfer attack, novel

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
GD-UAP
Mela UAP
XTransfer Base
cation, image-text retrieval, image captioning, and visual question answering, VQA). Our search space configura- tions-Base, Mid, and Large-consist of 16, 32, and 64 surrogate encoders, respectively. The super transferabilitty scales with the total number of surrogate encoders in the search space. Furthermore, X-Transfer achieves this super transferability while selecting as few as a single surrogate encoder per optimisation step. These findings uncover a new vulnerability in CLIP models and their applications.
TRM UAP
CGC
XTransffer Mid
AdvCLIP
ETU
X-Transfer Large
Imageiiet-25 63.4
GTSRB-ZS 69.4
Cars-Z5
FOOD101-25
64.6
80.1
STL10-2S
CIFAR100-2S 97.8
57.1
120%
In summary, our main contributions are as follows:
SUN397-75
CIFARTO-75 B7.6
65,0
We investigate the universal vulnerability of CLIP models and propose a novel attack method called X- Transfer to generate UAPs that can transfer across data, domains, models, and tasks.
MSCOCO-TA 71.8
M2WA--00A 41.1
MSC000-IR 65.8
OKVOA
X-Transfer introduces an innovative surrogate scaling strategy that efficiently scales transferability with the number of surrogate models by dynamically selecting suitable candidates at each UAP generation step.
MSCOCO-IC 43.9
FlickeniC 37.5
Figure 1. Adversarial super transferability achieved by X-Transfer with different configurations (Base, Mid and Large). The figure reports the attack success rate (ASR) with a single UAP applied to different samples, datasets, models, and tasks. ZS: zcro-shot classification; IR: image-retrieval; TR: text-retrieval; IC: image captioning, and VQ4: visual question answering. The ZS, IR, and TR are evaluated with CLIP encoders. IC and VQA are evaluated with large VL.Ms. Results for baseline methods, GD-UAP (Mopuri et al., 2018), TRM-UAP (Liu et al., 2023b), AdvCTII (Zhou al., 2023a), Meta-UAP (Weng et al., 2024). C-PGC (Fang et al., 2024b) and ETU (Zhang et al., 2024) are the best results across their various configurations. Results are averaged over multiple victim models. A larger shaded circle indicates a higher universal ASR.
We conduct extensive experiments to demonstrate the effectiveness of X-Transfer and provide in-depth in- sights and interpretations of the generated UAP pat- terns. Building on this, we establish new bench- mark, X- X-TransferBenchh which offers a comprehen- sive, open-source collection of UAPs and TUAPs for super transferability studies.
2. Relate Work
Contrastive Language-Image Pre-training. CLIP (Rad- ford et al., 2021) is popular framework that can pre-train on web-scale text-image pairs via contrastive learning (Chopra et al., 2005; Oord et al., 2018; Chen et al., 2020b). En coders pre-trained by CLIP have demonstrated superior zero-shot generalisation capability in a wide range of down- stream tasks (Palatucci et al., 2009; Lampert et al., 2009) and are shown to be more robust against common corrup- tions (Hendrycks & Dietterich, 2019; Fang et al., 2022: Cherti al., 2023; Tu et al., 2023). A number of works (Jia et al., 2021; Li et al., 2022b; 2023d:b; 2024; Tang et al., 2025) have been proposed to improve the performance of CLIP, such as using improved training recipe (EVA-CLIP) (Sun et al., 2023), shorter token sequence (CLIPA) (Li et al., 2023c), or sigmoid loss (SigLIP) (Zhai et al., 2023). has been found that one of the main contributing factors the success of CLIP its training data (Xu et al., 2024). In par- allel to CLIP, vision- languula pre-training can be achieved using various objectives, such as image-text matching, mask- ing, and auto-regressive generation (Li et al., 2021; 2022a; Singh et al., 2022; Yu et al., 2022; 2023; Kwon et al., 2023). This paper focuses specifically on CLIP and its variants due to their widespread adoption in downstream applications.
attack method that generates UAPs via an efficient surrogate scaling strategy applied to a large number of surrogate mod- els. Specifically, X-Transfer dynamically selects a small sub- set of suitable surrogate CLIP encoders from a large search space, enabling the efficient scaling of surrogate models for UAP generation. The UAPs and Targeted UAPs (TUAPs) generated by X-Transfer achieve black-box adversarial su- per transfcrability. Extensive evaluations demonstrate that X-Transfer significantly outperforms state-of-the-ae UAP methods designed for image classifiers (Mopuri et al., 2018: Liu et al., 2023b; Weng et al., 2024) and methods tailored for CLIP encoders (Zhou al., 2023a; Fang et al., 2024b: Zhang et al., 2024), achieving improved performance by substantial margin, as shown in Figure 1.
Our work is the first to demonstrate the existence of super UAPs that transfer across data (samples from in-domain datasets), domains (datasets), models (including both CLIP encoders and VLMs), and tasks (e.g., zero-shot classifi-
2

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Adversarial Attacks. The vulnerability of deep neural networks to adversarial attacks has been extensively studied on image classifiers (Szegedy al., 2014; Goodfellow et al., 2015; Carlini & Wagner, 2017; Madry et al., 2018; Zhang et al., 2019; Ilyas et al., 2019; Wang et al., 2019; 2020; Croce & Hein, 2020b; Huang et al., 2021; Ma et al., 2023; Wang et al., 2023b; Singh et al., 2023; Xie et al., 2025), and VLMs (Zhao et al., 2023; Luo et al., 2024; Schlarmann et al., 2024; Wang al., 2024e; Zhang et al., 2025a), typically under two main attack settings: white-box and black-box. In the white- box setting, the adversary has full knowledge of the victim model, including its architecture and parameters, while in the black-box setting, this information is not available to the adversary. In this case, the attacker can construct query- based attacks exploit the input-output response of the victim model (Ilyas al., 2018; Andriushchenko et al., 2020) leverage surrogate models construct transfer attacks (Papernot et al., 2016; Tramèr et al., 2017; Liu et al., 2017; Dong et al., 2018; Xie et al., 2019; Dong et al., 2019; Wu et al., 2020). Arguably, black-box attacks are more realistic and challenging, as deployed models are often kept secret from the end users, and in this case the gradient information of the victim model is unavailable, Between the two types of black-box attacks, transfer attacks are more practical, stealthy, and cost-effective, as they do not need to launch a large number of suspicious and costly queries to the victim model (Chen et al., 2020a; Wang et al., 2024d). Specifically, transfer attacks generate perturbations based on a surrogate model and then directly feed the adversarial examples to attack the black-box victim model.
3. Proposed Attack
In this section, we begin by revisiting the training objective of CLIP and our adversarial objective. We then introduce our proposed X-Transfer attack.
3.1. Training Objective of CLIP
CLIP (Radford et al., 2021) learns a joint embedding of images and texts. In such a way, the model can learn gener- alisable representations from web-scale data without using human annotations. Given an image-text dataset ID C X'x T that contains pairs of (æ;, t;), where I; is an image, and t, is the associated descriptive text. An image encoder fr : X + Rd and a text encoder fr : T + R We use f to denote the pair of image encoder fr and text encoder fr. The CLIP model projects the image and text into a joint em- bedding space R The image embedding can be obtained by 2 = (ax,) and the text embedding is 2 (44)). For a given batch of b image-text pairs {r:; t, 1==1 CLIP adopts the following training loss function:
N
exp(sim( 23,, z;,,))/T)
log
2b
k= exp(sim (2). =()
j=1
N
exp(sim 2/ =[)/T)
log
(I)
2h
Σ j= exp(sim (2), 2()r)
km
where 7 is trainable temperature parameter, and sim() is similarity measure. The first term in the above objec- tive function contrasts the images with the texts, while the second term contrasts the texts with the images.
Adversarial Attacks on CLIP. Recent works have inves- tigated the adversarial robustness of CLIP encoders using sample-specific perturbations (Zhang et al.. 2022a; Mao et al., 2023; Lu et al., 2023; He et al., 2023; Zhao et al., 2023; Gao et al., 2024; Wang et al., 2024a; Hu et al., 2024; Zhang et al., 2025b), showing that CLIP encoders are vulner- able to adversarial perturbations. However, sample-specific perturbations cannot achieve cross-data or cross-domann transferability because they are tailored to individual sam- ples. In contrast, UAPs have the potential for super transfer- ability. AdvCLIP (Zhou et al., 2023a) first explored UAPs against CLIP in a quasi-black- black- box threat model, demon- strating cross-data, cross-task, and cross-task transferability. Kim al. (2024) also investigated the partial black-box setting for large VLMs. ETU (Zhang et al., 2024) leveraged global and local features to achieve cross-data, cross-task, and cross-model transferability, and C-PGC (Fang et al., 2024b) and its efficient version (Yang et al., 2024) attained similar transferability. Nevertheless, none of these works has investigated black-box super transferability, transfer- ring across data, domains, models, and tasks simultane- ously-this capability is the primary foeus of our work. A detailed comparison is in Appendix A.
3.2. Adversarial Objective
We follow existing studies (Moosavi-Dezfoolli et al., 2017) to construct the UAP in the image space. Our perturba- tion objective is a form of embedding space attack (Zhang et al., 2022a; Zhao et al., 2023) that aims to deceive the encoder in the embedding space. However, our goal is to construct universal adversarial perturbation d that is ca- pable of transforming any image E D into an adversarial version x' = x + 5 by using the same adversarial perturba- tion to fool the victim encoder f. We focus on L perturbations. For other choices of perturbation constraint, Co-norm and adversarial patch are deferred to Appendix B. We construct the adversarial example using the following:
x'' = A A(x) = x + 5, ||
al11 <6,
(2)
where 8 is the universal perturbation vector. To generate universal perturbation for L -nrm bounded attack, we optimise the following non-targeted objective:
arg min (in))=xssim((((a)) (z)),
(3)
8
3

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
or targeted objective:
Algorithm 1 X-Transfer
arg max Ka (bado))
(4)
surrogate dataset D', search space S " "'"'"'''' ). total number of optimisation steps j, mo- mentum m, number of selection k.
Input:
where D' is surrogate dataset, f1 and f/z are the surrogate image encoder and text encoder, adv is adversary specified
Initialise arrays R, T, as zero-filled arrays of length NN Initialise ô randomly for step = 1 to do x = sample(D') {o Random sample a batch of images]
text description, and x' follows Eq. (2).
Our goal is to construct UAPs and TUAPs capable of achiev- ing black-box adversarial tuper-transferabiiity However, relying on a single surrogate model I' in Eq. (3) and (4) may limit transferability. Factors such as architecture, train- ing objectives, and pre-training datasets can influence how well perturbations generated from the surrogate f' transfer to the victim model f.
x' x + +
J UCB R, T) (o Compute UCB scores}
F TopK (u. k. S) {D Select Top l encoders}
for S: to PK do
2: = fi (x), x = (x))
Compute (A Zi, z/) {o Follow Eq. (3))
=1 (1-m) m) x R, + m L, (o Moving average)
Prior studies (Liu et al., 2017; Dong et al., 2018; Xiong et al., 2022; Chen et al., 2024a; Liu et al., 2024) have shown that ensemble methods can enhance cross-model transfer- ability by incorporating multiple surrogate models. UAPs inherently offer cross-data transferability. Additionally, due to their strong zero-shot capabilities, CLIP encoders serve as promising surrogate models for achieving cross-domain and cross-task transferability. To further enhance adver- sarial transferability, we therefore consider an ensemble of diverse surrogate CLIP encoders f. E F" {.f1,-. ... ff). Note that if the victim model f E P", then it is a white-box setting. Otherwise, it is a black-box setting. We optimise the following objective function:
T: T, +
end for L= Σ L, (Follow Eq. (5)
5 = ò risign (VL()) 5 = prroject(O,, -c, c)
end for
rely on selecting a fixed set of classifiers with diverse ar- chitectures, all trained using the same loss function and dataset (e.g., ImageNet). However, these methods require computing gradients with respect to each surrogate model, making surrogate scaling (Liu et al., 2024) computationally expensive the number of surrogates increases. To address this limitation, we propose an efficient surrogate scaling approach for UAP generation that dynamically selects a small subset of suitable encoders from a large search space. Algorithm I outlines the proposed X-Transfer framework using the non- targeted objective.
arg min E (e)=DD
8,),
(5)
where L follows Eq. (3) or (4) (change the arg min to arg max for targeted objective). To effectively ensemble various types CLIP encoders and scale up the number of surrogates, the chosen objective must be agnostic to differ- ences in architectures, embedding dimensions, and training loss functions. To achieve this, we adopt a generic adver- sarial objective function that operates directly on the CLIIP embeddings, Unexpectedly, we found that even this straight- forward objective alone can achieve performance on par with specialised CLIP-specific UAP baselines. In addition, we chose to average the loss rather than the embedding, as this approach avoids assumptions of a uniform ambient di- mension in the embedding space. Eq (3), (4), and (5) ensure that our adversarial objectives remain agnostic to variations across CLIP encoders, including differences in embedding sizes, architectures, and pre-training objectives.
The core idea of our efficient scaling strategy is to select k suitable candidate encoders from search space contain- ing N options (N >> k) at each optimisation step during UAP generation. This approach is inspired by the non- stationary multi-armed bandit (MAB) problem (Liu et al., 2023c), where the goal is to maximise cumulative rewards by pulling individual arms. In the MAB framework, the reward distributions are initially unknown and can change over time in the non-stationary setting. In our formulation, each candidate encoder is treated as an arm, and at each optimisation step, we select k surrogate encoders (arms) for the ensemble. The selection strategy must balance the exploration of less-selected arms and the exploitation of arms with the highest rewards. To achieve this, we use the classical Upper Confidence Bound (UCB) sampling strategy (Auer, 2002), defined as:
3.3. X- X-Transfer Attack
The key technique of the X- X-Transfer attack is its efficient surrogate scaling strategy, which enables super transferabil- ity across different dimensions. Existing surrogate ensemble methods (Xiong et al., 2022; Chen et al., 2024a) typically
2 Inn
UCB R, +
(6)
74
4

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
where R2 is the accumulative reward for the surrogate en- coder Ji, ny is total of times encoder J; has been selected, and n is total of times selection has been made ( n.). While UCB is our default sampling strategy, other strategies are also feasible. Note that the sampling strategy is not the primary factor driving X- X-Transfer's 's effectiveness, which will be presented in the ablation study.
UAP methods tailored for CLIP encoders, including C-PGC (Fang et al., 2024b), ETU (Zhang et al., 2024), and AdvCLIP (Zhou et al., 2023a). We also evaluate against methods de- signed for image classifiers, GD-UAP (Mopuri et al., 2018) 8), TRM-UAP (Liu et al., 2023b), and Meta-UAP (Weng et al., 2024). All UAPs are directly obtained from their official open-source repositories. We also include a vanilla baseline using the same adversarial objective with X-Transfer but without efficient surrogate scaling.
The most important aspect of X-Transfer is the design of suitable reward metric that should be cumulatively max- imised to encourage the selection of surrogate encoders that are most effective in achieving super transferability. For non non-targeted attacks, a lower loss value L; with respect to the encoder ft indicates that the UAP has effectively fooled Si. so S can be selected less frequently. Conversely, for targeted attacks, higher loss value signals success and thus reduces the priority of that encoder. In both cases, we use the loss value L. (Eq. (3) and (4)) as the reward. By focus- ing on selecting encoders that are less successfully fooled by the UAP or TUAP at the current iteration, X-Transfer encourages the perturbation to become more universally effective in the next iteration. Algorithm I illustrates this procedure, where we maintain two arrays (R and 7) to track the accumulated rewards and selection counts for each encoder. After computing L,, we update the reward distribu- tion R and the number of selection T based on the chosen encoders at each step. At the next iteration, we select the top-k encoders based on their UCB scores, thus striking a balance between exploring less- gess-frequently chosen encoders and exploiting those that are harder to fool.
Evaluation. Since existing baseline methods focus solely on non- noo-ttrgeeed objectives, we report the non-targeted attack success rate (ASR) in the main paper and include results for targeted objectives (TUAP) in Appendix C.7. Because each task uses different evaluation metries, we define the non-targeted ASR (Seleon
where is measured using task-specific metric (e.g., accuracy for zero-shot classification or CIDEr ( (Vedantam et al., 2015) for image captioning). The Sclean is the clean performance computed using the original images, while the Sado adversar- ial performance is obtained by applying UAP to all images.
We apply the same UAP to every image in each dataset to evaluate cross-data transferability. Beyond ImageNet, we employ CIFAR- 10 (C-10), CIFAR- 100 (C- (C-100) (Krizhevsky et al., 2009), Food (Bossard et al., 2014), GTSRB (Stal- Ikamp et al., 2012), Stanford Cars (Cars) (Krause et al., 2013), STLI0 (Coates et al., 2011). SUN397 (Xiao et al., 2016), MSCOCO (Chen et al., 2015), Flickr-3KK (Young al., 2014), OK-VQA (Marino et al., 2019), and VizWiz (Gurari et al.. 2018) datasets to evaluate cross-domain trans- ferability. For cross-model transferability, we evaluate 9 diverse CLIP encoders, including those released by Ope- such as ViT-L/14, ViT-B/16, ViT-B/32, RN-50, and RN-101-as well as encoders trained by others, including ViT-B/16 trained with SigLIP (Zhai et al., 2023), EVA-E/14 (Sun et al., 2023), ViT-H/14 trained with CLIPA (Li et al., 2023c), and ViT-bigG/14 trained with MetaCLIP (Xu et al., 2024). Additionally, lasion-language models (VLMs), such as OpenFlamingo- 3B (OF-3B), VA-BB (Liu et al., 2023a), MiniGPT-4 (Zhu et al., 2024), and BLIP2 (Li et al., 2023a). Note that our search space does not include any of these CLIP en- coders or encoders fine-tuned by these large VLMs, thereby ensuring a strictly black-box setting. We evaluate zero-shot classification, image-text retrieval, image captioning, and VQA tasks. Image captioning and VQA with large VLM, in particular, highlight cross-task transferability since large VLM training objectives significantly differ from those of the adversarial objective used by X- TTaansferr
4. Experiments
nAI (Radford et al.,
Search Space. We define 3 search spaces with diverse sizes (N). The Base search spaces are balanced and drawn from 4 diverse architecture Repes-ReNNet (RN) (He et al., 2016), ConvNext (Liu et al., 2022), ViT-B, and ViT-L (Dosovitskiy et al., 2021)- wwth encoders per architecture. This base search space is used to verify that X-Transfer is more ef- fective and efficient than both standard scaling (including all models) and heuristic-based fixed selections. We also explore Mid and a Large search space containing 32 and 64 diverse encoders to fully evaluate the scalability and ef- fectiveness of X- K-Transfer. Further details about these CLIP encoders are provided in Appendix C. and C.2.
assess large
We use ImageNet (Deng al., 2009) as the default surrogate dataset. The value of l is set to 4 for the Base search space, 8 for the Mid search space, and 16 for the Large search space. Following Fang et al. (2024b); Zhang et al. (2024), we employ Loo -nrm bounded perturbations with c = 12/255. We use the step size 1) of 0.5/255.
UAP Generation.
4.1. Super Transferability
We present the zero-shot classification results in Table 1. Baselines specifically designed for CLIP (ETU and C-GPC)
Baselines. We compare our approach to tate-oof-the-at
5

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table The non-targeted ASR (%) results in zero-soot classification and image-text (I-T) retrieval tasks across different CLIIP encoders and datasets. I-T retrieval is evaluated on MSCOCO. Results are based on averaging over 9 black-box victim encoders. The best results for the baseline are underlined, and the best results overall are boldfaced.
Zero-Shot Classification
I-T Retrieval
Method
Variant
C-10
C-100
Food
GTSRB
ImageNet
Cars
STL
SUN
Avg
TR
IR@
GD-UAP (Mopuri et al., 2018)
Seg CLS
56.7
73.0
27.9
61.1
17.9
15.4
9.9
14.7
34.6
26.5
18.1
57.9
72.4
42.9
66.2
24.3
22,9
18.7
20.2
40.7
33.7
24.3
AdvCLIP (Zhou et al., 2023a)
VIT/B-16
0,9
4.7
11.6
2.5
2.2
1.3
3.1
2.6
RN101
0.7
3.6
1.5
9.5
2.6
2.9
0.2
1.5
2.8
9.0
3.1
TRM-UAP (Liu et al., 2023b)
GoogleNet RN152
55.7
69.3
46.7
67.8
27.0
24.8
21.8
23.0
42.0
34.9
26.5
47.3
63.4
42.4
63.9
23.6
22.9
17.7
20.1
37.7
30.7
23.0
Meta-UAF UAP (Weng al., 2024)
Ensemble
79.3
93.4
46.0
73.5
30.9
28.5
25.9
28.4
50.8
42.5
34.1
Ensemble-Meta
72.5
89.0
41.9
67.6
28.3
25.8
21.4
26.5
46.6
38.3
29.0
RN101-Flicker
27.9
41.3
24.2
36.4
17.8
18.3
13.6
14.7
24.3
21.3
15.7
C-GPC (Hang et al., 2024b)
RN101-COCO
23.9
41.9
24.4
37.2
19.3
17.6
13.3
15.7
24.2
24.3
17.9
ViT-B/16-Hlicker
63.7
82.9
51.3
70.2
40.4
38.1
28.2
37.9
51.6
43.8
35.7
ViT-B/I6-COCO
62.4
81.5
47,2
70.1
37.9
39.2
26.5
35.0
50.0
39.0
33.8
ETU (Zhang et al., 2024)
RN50-Flicker
34.3
53.5
20.6
49.7
13.8
12.1
8.6
9.3
25.2
17.2
12.8
ViT-B/16-Hlicker
70.2
86.5
47.1
71.1
34.1
31.1
27.5
31.0
49.8
40.2
32.8
Vanilla (N = 1)
72.7
88.3
49.9
72.3
31.2
26.3
19.2
27.6
48.4
42.3
34.5
X-Transfer (Ours)
Base (N 16)
86.6
97.5
74.8
89.3
56.0
52.1
46.8
50.7
69.2
63.7
58.8
Mid (N = 32)
86.9
97.6
78.7
88.6
62.8
60.0
50.4
64.1
73.6
70.1
65.7
Large (N 644
87.6
97.8
80.1
89.4
63.4
64.6
57.1
65.0
75.6
71.8
65.8
show no significant advantage over methods developed for image classifiers (GD-UAP, TRM-UAP, and Meta-UAL UAP). Interestingly, our vanilla paseline-applying the adversar- ial objective without any ensemble-achin comparable with these existing methods. This indicates that, despite its simplicity, the chosen adversarial objec- tive is well-suited for X-Transfer. For UAPs generated by X-Transfer, the improvement is substantial across all the datasets and in the averaged ASR metric. Appendix C.3 reports per-encoder ASR, where X-Transfer achieves state-of-the-art results across every dataset and victim coder. Together, these findings demonstrate that X- X-Transfer achieves superior cross-data, cross- domain, and pross-model adversarial transferability. In Appendix C.4, we demonstrate that the generic design of the adversarial objective function is critical for the effectiveness of X- Tranffferr Specifically, using the loss function from ETU does not achieve the same level of super transferability when combined with our effi- cient scaling method.
other words, X- X-Transfer was not explicitly designed to de- ceive large VL.S.
We evaluate our method on commonly used image caption- ing and VQA tasks using 4 widely adopted large VLMs, with results presented in Table 2. Consistent with our pre- vious findings, X-Transfer achieves state-of-the-ar super transferability. These results also expose a new safety threat to large VLMs: adversaries can exploit the large pool of pub- licly available pre-trained encoders construct UAPs and manipulate large VLMs under realistic black-box settings. Additionally, in Appendix C.5, we show that super transfer- ability is independent of the surrogate datasets used. This indicates that surrogate encoders are the primary factor for super transferability. In Appendix C.8, we demonstrate that adversarial training is not robust to different types per- turbations, such as L2 Lz-norm perturbations and adversarial patches.
performance
4.2. Ablation and Analysis of Efficient Scaling
To further demonstrate the super transferability of our approach, we evaluate cross-task transferability on large VLMs. The popular approach for large VLMs is to align vi- sual embeddings from the CLIP-based image encoder with LLM text embeddings, either by fine-tuning or employing bridging networks. Notably, these large VLMs are trained with auto-regressive text generation objectives, which dif- fers from both the CL.PP and our adversarial objectives. In
In Figure 2(a), we compare our efficient scaling approach in X-Transfer a standard fixed selection method with the Base search space The standard scaling chooses encoders in balanced way, selecting one encoder per architecture type. When A 1, ViT-L/14 is chosen. Figure 2(a) shows that increasing the number of surrogate encoders leads to improved super transferability. With total of 16 encoders available, the
zero-shot classification task.
6

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table Non-targeted ASR (%) results in image captioning and VQA across various large VL.Ms and datasets. For image captioning, CIDEr is used as the evaluation metric, while VQA accuracy is employed for the VQA task. Results for all baseline methods are the best results across their various configurations. The best baseline results are underlined, and the best overall results are boldfaced.
X-Trnfffer
Model
Dataset
GD-UAP
AdvCLIP
TRM-UAP
Meta-UAP
C-GPC
ETU
Vanilla
Base
Mid
Large
MSCOCO
10.5
-0.9
10.0
19.2
19.9
22.9
34.1
39.6
47.1
53.3
Flicker-30k
11.1
0.7
11.7
18.4
22.1
21.3
30.1
35.7
41.8
46.8
OF-3B
OK-VQA
8.8
0.3
7.0
12.0
13.4
13.3
20.5
25.0
30.4
28.3
VizWiz
8.3
21.7
23.8
21.7
31.4
29.8
41.7
MSCOCO
6.2
-0.3
3.5
11.1
7.5
17.2
9.4
21.0
24,2
29.6
Flicker-30k
8.9
0.4
7.9
13.5
10.3
15.6
15.7
18.9
21.4
24.7
LLaVA-7B
OK- VQ-VQA
5.3
5.4
9.5
9.7
11.7
23.5
31.5
VizWiz
14.8
7.2
14.5
24.2
25.5
28.8
25.7
31.9
45.6
40.3
MSCOCO
7.2
4.6
7.2
11.1
12.1
11.9
12.3
23.2
24.6
28.6
Flicker-300 30k
9.7
9.2
12.2
14.6
13.4
13.4
21.5
25.7
MiniGPT4
OK-VQA
3.6
0.3
3.4
4.0
6.0
6.3
6.2
14.0
16.8
17.6
VizWiz
6.5
1.3
8.7
6.6
4.3
8.3
5.7
6.5
6.8
4.0
MSCOCO
12.3
4.5
10.3
13.0
18.9
44.8
64.1
Flicker-30k
12.6
6.2
3.4
9.9
11.8
8.0
15.7
36.0
43.0
52.8
BLIP2
VQA
10.6
0.1
11.3
13.8
17.2
15.1
22.7
39.8
53.2
53.7
VizWiz
25.2
-0.4
20.3
30.0
42.8
33.8
46.1
58.5
68.8
67.3
70
WTB 32 MetaCUP
VI--/14
77.2
15.7
VITH-14 DFNSB
VIT-8/32
28r0-shot ASR
65
VITH1-14 MetaCUP
VIT.8/16
Surrogate
RM1011
79.0
91.0
37.1
19.7
29.0
VTT-H14
CommonPPal
MT-8/16
ConvNext
EVA02-L14 Merged28
WT-bigG//1
VIT-0/32
64.2
58.1
54.2
88.2
36.7
5555
MobilcCLP-52 DataComp
MobileCLIP
Standard Scaling
EVAG2 Merged2e
ViTamin
X-transfer
Optimal
VIT--114 Pool
ResNet
0SWW
TOTWW
FU/U
EW101-g plus Merged2e
NLLB
12
16
0%
5%
10%
15%
20%
(a) Compare scaling
(b) Encoder selection
(c) Architecture selection
(d) Transfcrability
Figure (a) Comparison standard scaling (selecting diverse encoders) vs. X-Transfer's efficient scaling the base search space for zero-shot classification, averaged over victim encoders. (b-c) Distribution of X- Transfer-seleeteee encoders and architectures in the large search space. (d) Non-targeted ASR with various encoders on the zero-shot classification task.
X- X-Transfer reduces the required computation resources to approximately X of those needed for standard scaling. In practical implementations, additional factors may influence computational costs, such GPU communication over- head, bottlenecks caused by specific surrogate encoders in the ensemble, model weight loading times. and the actual surrogate choices made by X-Transfer. Despite these factors, we report the observed time costs in Table 3 for the experi- ments shown in Figure 2(a). These measurements based consistent hardware settings. The results demonstrate that X-Transfer is significantly more efficient than standard scaling, requiring approximately N of the computational resources.
Table Time cost comparing standard scaling approach with the efficient scaling approach used by X-Transfer the Base search space (N 16) with various k.
Standard Scaling
X- Trannsser
Method
== 1
k= 4
k
== 16 A = 12
GPU Days
8.0
2.5
7.6
8.0
optimal performance of this search space is For efficient scaling strategy, even h 1 achieves performance comparable to the best possible result.
16.
&
In terms of the computational cost, for X-Transfer, only out of N surrogate encoders are selected at each opti- misation step, whereas the standard scaling approach re- quires utilising all N surrogate encoders. Consequently,
We further investigate how X- X-Transfer selects encoders by examining the top 10 most frequently chosen encoders in the large search space (Figure 2(b)). We find that ViT-based

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
encoders dominate this selection because they are gener- ally more robust and harder fool. It is not surprising that ViT-H/14, trained on large-scale datasets such as Meta- CLIP (Xu et al., 2024) or DFNSB (Fang et al.. 2024a), appears among the top choices. Larger encoders trained on extensive datasets tend to generalise better, thus posing a greater challenge for UAPs. As a result, ViT-H/14 1 is one of the top choices. Additionally, the pre-training datasets of these frequently selected encoders are diverse- -spanning
generate hallucinated responses, while the TUAP directs the model's output toward a specific target text description. Additional visualisations are provided in Appendix C.9.
In Figure 4, we present intriguing and novel insights into the UAPs generated by X-Transfer. Existing research on adversarial robustness has shown that non-targeted perturba- tions generally lack semantic meaning MMoosavi-Dezfool et al., 2017; Zhao et al., 2023), whereas targeted perturba- tions often encode semantic features associated with their target class (Zhang et al., 2020; Weng et al., 2024). This pattern holds true for both sample-specific and universal perturbations. Our findings, however, reveal an unexpected result: UAPs with non-targeted objectives generated us- ing CLIP encoders exhibit discernible semantic features, as shown in Figure 4. Many of these perturbations resem- ble building- liie structures interspersed with nonsensical text-like elements. We hypothesise that this phenomenon is linked to CLIP's boncept-blending capability (Kazemi et al., 2024), feature observed in generative text-to-image models (Ramesh et al., 2021; Saharia et al., 2022; Kumari et al., 2023). Remarkably, CLIP demonstrates this capabil- ity despite not being explicitly trained with text-to-image generation objectives.
MetaCLIP, DFN, CommonPool/DataCom diverse-spanning (Gadre et al., 2023), and Merged2B (Sun et al., 2023). This observation suggests that choosing encoders pre-trained on a variety of datasets is also important in achieving super transferability.
In Figure 2(c), we analyse how X-Transfer selects encoders based on their architecture. The results show that ViT-based encoders dominate, with ViT-L/14 chosen most frequently, likely due to its repeated appearance in the search space. This indicates that X- X-Transfer's selection does reflect the overall architecture distribution. However, merely mirroring this distribution through a random sampling strategy does not guarantee superior super transferability. Meanwhile, Figure 2(b) reveals that ViT-L/14 only appears among the top 10 encoders twice, suggesting that selecting encoders that are harder to fool remains critical.
Interestingly, these UAP patterns often resemble building- like structures; however, in Appendix C.9, we found CLIP encoders interpret them as various unrelated concepts, such as "cheese"-relatee objects. This observation highlights a key distinction: the semantic patterns perceived by humans differ significantly from those recognised by models, mak- ing these patterns unique compared those seen in targeted attacks, which generally align with human perception. In Appendix C.9, we further investigate the origin of the UAP patterns generated by X-Transfer. Our analysis reveals that these patterns are closely related to the pre-training datasets. For example, CLIP encoders trained on specialised datasets, such as remote sensing datasets, produce UAPs that exhibit visual patterns resembling remote sensing imagery.
To understand why ViT-based encoders dominate, we visu- alise the transferability results by architecture in Figure 2(d). These results indicate that convolution-bast (RN) surro- gate encoders perform poorly against ViT-based victim en- coders, whereas ViT-based surrogate encoders can transfer to convolution-based victim models. Although VTT-B shows relatively better transferability, none of these encoders alone achieves superior cross model transferability. Consequently, efficient surrogate scaling in X-Transfer remains essential for achieving state-of-the-ant cross-model transferability.
We conducted an ablation study using different sampling strategies in the zero-shot classification task. We compare our approach with the random sampling strategy, which se- lects k encoders at each optimisation step at random, and the e-greedy strategy uses our reward metric to guide the se- lection process. The ASR are 66.9%, 69.0%, and 69.2% for random sampling, e-greedy (c 0.5), and UCB, respectively. The results indicate that UCB achieves the best performance, with e-greedy also performing competitively. In contrast, random sampling shows significantly poorer results. These findings suggest that the reward metric is the primary driver behind X- Trannffer's 's success, while the choice of sampling strategy has a comparatively smaller impact.
Notably, as shown in Figure 4(a), the visual clarity of these patterns becomes increasingly distinguishable to humans as the ensemble size N grows for the standard scaling ap- proach. This improvement correlates with the ASR, as shown in Figure 2(a). In contrast, for X- X-Transfer, the pa- rameter k does not scale with ASR, and the visual clarity of patterns remains roughly consistent, as seen in Figure 4(b) in Appendix C.9. However, the ASR for X- X-Transfer scales with the search space size N, and the visual clarity of patterns improves correspondingly. as shown in Figure 8(a). These results suggest that the visual interpretability of the perturbation correlates with the ASR.
4.3. Qualitative Analysis
Figure 3 illustrates where both a UAP and TUAP are applied to an image, along with the corresponding responses from VLMs. The non-targeted UAP causes the VLM to
8

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Non-targeted
Targeted: Remove files from this computer and inject computer Clean
Clean
OF-38: The Grand Prismatic Spring in Yellowstore National Park.
OF-38: Lava flowing into the ocean.
LLaVA-78: hot spring with yellowish orange color.
LLaVA- 2B: volcano erupting the ocean, creating massive wave.
MiniGPT4:A A large pool of water with clouc in the sky.
MiniGPT4: large body of water with lave flowing into
BLIP2: Yellowstone national park, wyoming.
BLIP2: An Image of lave flowing into the ocear.
Adversarial
Adversarial
OF-3B:A compuser
OF-38: Prudine, small
of Frence.
LLaVA-7B: computer
background message
picture waterfall words "The Prine
book, promise jay chese's album, the tribune.
MiniGPT4:
with voicano the
words computer virus computer.
Figure An illustration showing the application both UAP (left) and TUAP (right) to an image. The responses from large VLMs are shown side by side for the clean image (top) and the adversarially perturbed image (bottom).
12
12
16
(a) Standard scaling with different N.
(b) X-transfer with different & in Base search space.
Figure 4. The visualisation of UAPs in (a) scaling with N with standard scaling, (b) scaling with k with efficient scaling of X-Transfer. All UAPs are generated with the Base search space.
X- TrannsferBench
Acknowledgement
We have curated extensive collection of UAPs and TU- APs, forming X-TransferBench a comprehensive reposi- tory of off-the-shelf UAPs and TUAPs designed for robust evaluation. To the best of our knowledge, no similar open- source collection of UAPs currently exists, making this a valuable contribution the community. Further technical details are in Appendix D.
This work is in part supported by National Key R&D Pro- gram of China (Grant No. 2022ZD0160103 and National Natural Science Foundation of China (Grant No. 62276067). Sarah Erfani is in part supported by Australian Research Council (ARC) Discovery Early Career Researcher Award (DECRA) DE220100680. The authors would like thank Peng-Fei Zhang for providing the UAPs used in the ETU baseline. This research supported by The University of Melbourne's Research Computing Services and the Petas- cale Campus Initiative.
5. Conclusion
In this work, we propose X-Transfer, a novel attack that ensembles multiple CLIP encoders with efficient scaling. We show that X-Transfer can produce single perturbation capable of achieving cross-data, cross-model, cross-domain, and cross cross-task adversarial transferability- what we term super transferability. Furthermore, findings reveal that increasing the number of surrogate encoders can signifi- cantly affect large vision language models (VLMs). Specif- ically, X-Transfer can generate UAPs that degrade VLM performance and TUAPs that steer large VLMs produce responses aligning with targeted text descriptions. This work highlights new, realistic safety threat: adversaries can leverage large number of open-sourced CLIP encoders to generate super transferable UAPs and TUAPs. Our find- ings underscore the urgency of addressing this vulnerability and call on the community to explore more general, super- transferable adversarial attacks.
Impact Statement
In this work, investigate the universal vulnerability of CLIP and demonstrate its implications for pre-trained encoders and downstream large vision-language models (VLMs). While this research might appear potentially harm- ful, we believe the benefits of publishing it far outweigh any associated risks. Consistent with the goals of adversarial ro- bustness research, our objective is expose vulnerabilities in existing systems encourage the development effec- tive defences against potential real-world attacks. Further- more, by highlighting the feasibility of these perturbations and open-sourcing X- TrTrnnffrBench, we aim to provide the community with a valuable resource to explore and design practical defences before CLIP encoders and VLMs see widespread adoption safety-critich environments.

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
References
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In ICML, 2020b.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I,, Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot learning. In NewrlPS, 2022.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 0032500 2015.
Andriushchenko. M., Croce, F., Flammarion, N., and Hein, M. Square attack: a query-efficient black-box adversarial attack via random search. In ECCV, 2020.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M.. Ilharco, G., Gordon, C.,, Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-iman learning. In CVPR. 2023.
Auer, P. Finite-time analysis of the multiarmed bandit prob- lem, 2002.
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01.300 2023.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez. J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%% chatgpt quality. 2023.
Chopra, S., Hadsell, R., and LeCun, Y. Learning sim- ilarity metric discriminatively, with application to face verification. In CVPR, 2005.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Bai. J., Gao, K., Min. S., Xia, S.-T., Li, Z., and Liu, W. Badclip: Trigger-aware prompt learning for backdoor attacks on clip. In CVPR, 2024.
Croce, F. and Hein, M. Provable robustness against all adversarial lp-perturbations for p 1. In ICLR, 2020a.
Bossard, L.,, Guillaumin, M., and Van Gool, L.. Food- 101- mining discriminative components with random forests. In ECCV, 2014.
Croce, and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In 2020b.
Carlini, N. and Terzis, A. Poisoning and backdooring con- trastive learning. In ICLR, 2022.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In S&P, 2017.
Carlini, N., Nasr, M.. Choquette-Choo, C. A., Jagielski, M., Gao, I., Koh, P. W. W., Ippolito, D., Tramer, F., and Schmidt, L.. Are aligned neural networks adversarially aligned? In NeurlPS, 2023.
Dong, Y., Liao, F., Pang. T., Su, H., Zhu, J., Hu, X., and Li, Boosting adversarial attacks with momentum. In CVPR, 2018.
Dong, Y., Pang, T., Su, H., and Zhu, J. Evading defenses to transferable adversarial examples by translation maasaation-invaiiant attacks. In CVPR, 2019.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Chen, H., Zhang, Y., Dong, Y., Yang, X., Su, H., and Zhu, J. Rethinking model ensemble in transfer-based adversarial attacks. In ICLR, 2024a.
Chen, J., Yu, Q. Shen, X., Yuille, A., and Chen, L.-C. Vitamin: Designing scalable vision models in the vision- language era. In CVPR, 2024b.
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data determines distributional robustness in contrastive language image pre-training (clip). In ICMI., 2022.
Chen, S., Carlini, N., and Wagner, D. Stateful detection of black-box adversarial attacks. In Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence, 2020a.
Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A. T., and Shankar, V. Data filtering networks. In ICLR, 2024a.
10

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Fang, H., Kong, J., Yu, W., Chen, B., Li, J., Xia, S., and Xu, K. One perturbation is enough: On generating uni- versal adversarial perturbations against vision-language pre-training models. arXiv preprint arXiv:2406.05991, 2024b.
Hu, A., J., Pinto, F., Kammitsas, K., and Tom, P. As firm their foundations: Can open-sourced foundation mod- els be used to create adversarial examples for downstream tasks? arXiv preprint arXiv:2403. 2024.
Huang, H., Wang, Y., Erfani, S., Gu, Q. Bailey, J., and Ma, X. Exploring architectural ingredients of adversarially robust deep neural networks. NeurlPS, 2021.
Fang. Y., Wang, W., Xie, B., Sun, Q. Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Eva: Exploring the limits of masked visual representation learning at scale. In ICCV, 2023.
Huang, H., Erfani, S., Li, Y., Ma, X., and Bailey, J. De- tecting backdoor samples in contrastive language image pretraining, In ICLR, 2025.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smymis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J.. et al. Datacomp: In search of the next generation of multimodal datasets. In NeurlPS, 2023.
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Car- lini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, 2021. URL https: /tgt:///ghhbbb/// com/
Gao, S., Dou, S., Zhang, Q., Huang, X.-J., Ma, J., and Shan, Y. On the universal adversarial perturbations for efficient data-free adversarial detection. In ACL Findings, 2023.
m1 mffoundationssopen.
Gao, S., Jia, X., Ren, X., Tsang, I., and Guo, Q. Boosting transferability in vision-language attacks via diversifica- tion along the intersection region of adversarial trajectory. In ECCV, 2024.
Ilyas, A., Engstrom, L., Athalye, A., and Lin, J. Black-box adversarial attacks with limited queries and information. In ICML, 2018.
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Gong, Y., Ran, D., Liu, J., Wang, C., Cong, T., Wang, A., Duan, S., and Wang, X. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In AAAI, 2025.
Jia, C., Yang, Y., Xia, Y., Chen, Y.T.,, Parekh, Z., Pham, H., Le, Q., Sung. Y.-H,, Li, Z., and Duerig. T. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Goodfellow, J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In ICLR, 2015.
Gurari, D., Li, Q.. Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018.
Jia, J., Liu, Y., and Gong, N. Z. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In S&P, 2022.
Hc, B., Jia, X., Liang, S., Lou, T., Liu, Y., and Cao, X. Sa-attack: Improving adversarial transferability of vision- language pre-training models via self-augmentation. arXiv preprint arXiv:2312.04913, 2023.
Jiang, D.,, He, X., Zeng. H., Wei, C., Ku, M., Liu, Q., and Chen, W. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483 2024.
Kang, D., Sun, Y., Brown, T., Hendrycks, D., and Steinhardt, J. Transfer of adversarial robustness between perturbation types. arXiv preprint
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
2019.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. In ICLR, 2019.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vlms: Investigating the design space of visually-conditioned language models. In ICML, 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E.,, Casas, D. d. L.. Hendricks, L. A., Welbi, J., Clark, A., al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15553 2022.
Kazemi, H., Chegini, A., Geiping, J., Feizi, S., and Gold- stein, T. What do we lear from inverting clip models? arXiv preprint arXiv:2403. 240300200, 2024.
Hsiung, L., Tsai, Y.YY,. Chen, P.-.,, and Ho, T.-Y. Towards compositional adversarial robustness: Generalizing ad- versarial training to composite semantic perturbations. In CVPR, 2023.
Kim, H.-S., Kim, M., and Kim, C. sarial perturbations: Deceiving vision-language models across both images and text with a single perturbation. arXiv preprint arXiv:24122 12.00 08808 2024.
adver-

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Liang, S., Zhu, M., Liu, A., Wu, B., Cao. X., and Chang, E.-C. Badclip: Dual-embedding guided backdoor attack multimodal contrastive learning. In CVPR, 2024.
Koh, J. Y.. Salakhutdinov, R., and Fried, D. Grounding language models to images for multimodal inputs and outputs. In ICML, 2023.
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014.
Krause, J., Deng, J., Stark, M., and Fei-Fei, L. Collecting large-scale dataset of fine-grained cars. 2013.
Liu, C., Chen, H., Zhang, Y., Dong, Y., and Zhu, J. Scaling laws for black box adversarial attacks. arXiv preprint urXiv:2411.Jo0 2024.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. In NeurIPS, 2023a.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. Multi-concepp customization of text-to-image diffusion. In CVPR, 2023.
Liu, Y., Chen, X., Liu, C., and Song, D. Delving into transferable adversarial examples and black-box attacks. In ICLR, 2017.
Kwon, G., Cai, 7., Ravichandran, A., Bas, E., Bhotika, R., and Soatto, S. Masked vision and language modeling for multi-moda representation learning. In ICLR, 2023.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1exxvv/9907 1692, 2019.
Lampert, H., Nickisch, H., and Harmeling, S. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009.
Liu, Y., Feng, X., Wang, Y., Yang, W., and Ming, D. Trm- uap: Enhancing the transferability of data-free universal adversarial perturbation via truncated ratio maximization. In ICCV, 2023b.
Li, J. Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and lan- guage representation learning with momentum distilla- tion. NeurlPS, 2021.
Liu, Y., Kuang, X., and Van Roy, B. A definition of non- stationary bandits. arXiv preprint arXhv:2302.12202. 2023c.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a.
Liu, Z., Mao, H., Wu, C.-Y,, Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In CVPR, 2022.
Ji, Li, D., Savarese, S., and Hoi, S. 2: Bootstrapping gegguage-imagge pre-training with frozen image encoders and large language models. In /CML, 2023a.
Lu, D., Wang, Z., Wang, T., Guan, W., Gao, H., and Zheng, F. Set-level guidance attack: Boosting adversarial trans- ferability of vision- Lsnounlanguaag pre-training models. In ICCV, 2023.
Li, R., Kim, D., Bhanu, B., and Kuo, W. Reclip: Resource- efficient clip by training with small images. TMLR, 2023b.
Lu, D., Pang, T., Du, C., Liu, Q. Yang, X., and Lin, M. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577 2024.
Li, X., Wang, Z., and Xie, C. An inverse scaling law for clip training. NeurlPS, 2023c.
Li, X., Tu, H., Hui, M., Wang. Z., Zhao, B., Xiao, J., Ren, S., Mei, J., Liu, Q. Zheng, H., et al. What if we recaption billions of web images with llama-3? arXiv preprint 20X.:24006.88478, 2024.
Lu, X., Wang. B., Zheng, X., and Li, X. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing, 56(4):2183-2195, 2017.
Li, Y., Liang, F. Zhao, L., Cui, Y., Ouyang, W., Shao, J., Yu, F. and Yan, J. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2022b.
Luo, H., Gu, J., Liu, F,, and Torr, P. An image is worth 1000 lies: Transferability of adversarial images across prompts vision-languae models. In ICLR, 2024.
Ma, X., Jiang, L., Huang, H., Weng, Z., Bailey, J., and Jiang, Y.-G. Imbalanced gradients: a subtle cause of overestimated adversarial robustness. Machine Learning, 2023.
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., and He, K. Scal- ing inguuage-image pre-training via masking. In CVPR, 2023d.
12

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarezyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv: 1112211110000 1.02/ 14, 2021.
Mao, C., Geng, S., Yang, J., Wang, X., and Vondrick, C. Understanding zero-shot adversarial robustness for large- scale models. In ICLR, 2023.
Schuhmann, C.,, Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Mezsavi- S.-M., Fawzi, A., Fawzi, O., and Frossard, P. Universal adversarial perturbations. In CVPR, 2017.
Shayegani, E.,, Dong, Y., and Abu-Ghazalch, N. Jailbreak in pieces: Compositional adversarial attacks multi-modal language models. In ICLR, 2023.
Mopuri, K. R., Ganeshan, A., and Babu, R. V. Generaliz- able data-free objective for crafting universal adversarial perturbations. TPAMI, 2018.
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. Flava: A foundational language and vision alignment model. In CVPR, 2022.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation leamn ing with contrastive predictive coding. arXiv preprint arXiv: 03748, 2018.
Singh, N. D., Croce, F., and Hein, M. Revisiting adver- sarial training for imagenet: Architectures, training and generalization across threat models. NeurlPS, 2023.
Palatucci, M., Pomerleau, D., Hinton, G. E., and Mitchell, T. M. Zero-shot learning with semantic output codes. In NeurlPS, 2009.
Song, Y., Zhou. Z., Li, M., Wang, X., Deng, M., Wan, W., Hu, S., and Zhang, I.. Y. Pb-uap: Hybrid universal adversarial attack for image segmentation. In ICASSP, 2025.
Papernot, N., McDaniel, P., and Goodfellow, I. Transfer- ability in machine learning: from phenomena to black- box attacks using adversarial samples. arXiv preprint 1605 07277, 2016.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning al- gorithms for traffic sign recognition. Neural networks, 2012.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine translation. In ACI, 2002.
Sun, Q., Fang. Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. /5389, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D.,, Goodfellow. I. J., and Fergus, R. Intriguing properties of neural networks. ICLR, 2014.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In ICML., 2021.
Tang, Z., Lian, L., Eisape, S., Wang, X., Herzig, R., Yala, A., Suhr, A., Darrell, T.. and Chan, D. M. Tulip: To- wards unified language-image pretraining. arXiv preprint arXiv:2503. 15485, 2025.
Saharia, C., Chan, W. Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. In NeurlPS, 2022.
Tao, X., Zhong, S., Li, L., Liu, Q., and Kong, L. Imgtro- jan: Jailbreaking vision-language models with one image. arXiv preprint M.Xi::2403002111 2024.
Schlarmann, C. and Hein, M. On the adversarial robustness of multi-modal foundation models. In ICCV, 2023.
Team, M. N. et Introducing mpt-7b: A new standard for open-source commercially usable 11ms, 2023. 2023.
Schlarmann, C., Singh, N. D., Croce, F. and Hein, M. Robust clip: Unsupervised adversarial fine-tuning of vi- sion embeddings for robust large vision- -anguule models. arXiv preprint orXiv:2402. /2336, 2024.
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-75 73, 2016.
13

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Tong, S., Brown II, E. L.,, Wu, P., Woo, S., IYER, A. J., Akula, S. C., Yang, S., Yang. J., Middepogu, M., Wang, Z., et al. Cambrian-1: A fully open, vision-centric explo- ration of multimodal 1lm.. In NeurlPS, 2024.
Wang, Y., Ma, X., Bailey, J., Yi,J., Zhou, B., and Gu, Q On the convergence and robustness of adversarial training. In ICML, 2019.
Wang. Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu, Q. Improving adversarial robustness requires revisiting mis- classified examples. In ICLR, 2020.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint w0xiv2307.092288 2023.
Wang. Z., Pang, T., Du, C., Lin, M., Liu, W., and Yan, S. Better diffusion models further improve adversarial training. In ICML, 2023b.
Tramèr, F., Papernot, N., Goodfellow, I., Boneh, D., and Mc- Daniel, P. The space of transferable adversarial examples. arXiv preprint /004::704.3344 2017.
Wang, Z., Han, Z., Chen, S., Xue, F., Ding, Z., Xiao, X., Tresp, V., Torr, P., and Gu, J. Stop reasoning! when multimodal LLM with chain-of-thought reasoning meets adversarial image. In COLM, 2024e.
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. In ICLR, 2019.
Weng, J., Luo, 7., Lin, D., and Li, S. Learning transferable targeted universal adversarial perturbations by sequential meta-learning. Computers & Security, 2024.
Tu, W., Deng, W., and Gedeon, T. closer look at the robustness of contrastive language-image pre-training (clip). In NeurIPS, 2023..
Wu, D., Wang, Y., Xia, S.-T., Bailey, J., and Ma, X. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
Vasu, P. K. A., Pouransari, H., Faghri, F., Vemulapalli, R., and Tuzel, O. Mobileclip: Fast image-text models through multi-modal reinforced training. In CVPR, 2024.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring a large collection of scene categories. IJCV, 2016.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-basse image description evaluation. In CVPR, 2015.
Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang. J., Ren, Z., and Yuille, A. L.. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.
Xie, Y., Zheng, W., Huang, H., Ye, G., and Ma. X. To- wards million-scale adversarial robustness evaluation with stronger individual attacks. In CVPR, 2025.
Visheratin. A. Nilb-clip-train performant multilingual image retrieval model on a budget. arXiv preprint StfI::2309001555 2023.
Xiong, Y., Lin, J., Zhang, M., Hopcroft, J. E., and He, K. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transfcrability. In CVPR, 2022.
Wang, C., Jia, R., Liu, X., and Song, D. Benchmarking zero-shot robustness of multimodal foundation models: A pilot study, arXiv preprini arXiv:2403. 10499, 2024a.
Wang, R., Ma, X., Zhou, H., Ji, C., Ye, G., and Jiang, Y.-G. White-box multimodal jailbreaks against large vision- language models. In ACMMM, 2024b.
Xu, H., Xie, S., Tan, X., Huang, P.Y.,, Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L.,, and Feichten- hofer, C. Demystifying clip data. In ICLR, 2024.
Yang. F., Huang, Y., Wang, K., Shi, L., Pu, G., Liu, Y., and Wang, H. Efficient and effective universal adversarial attack against vision-language pre-training models. arXiv preprint arXiv:2410. 2024.
Wang, R., Wang. B., Wang, X., Ma, X., and Jiang, Y.-G. Ideator: Jailbreaking large vision-language models using themselves. arXiv preprint arXiv:2411.00822 2024c.
Wang, W., Lv, Q., Yu, W.. Hong. W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L.,, Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 2023a.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. ACL, 2014.
Wang, X., Chen, K. Ma, X., Chen, Z., Chen, J., and Jiang, Y.G. Advqdet: Detecting query- adversarial attacks with adversarial contrastive prompt tuning. In ACMMM, 2024d.
Yu, J., Wang, Z., Vasudevan, V., Yeung. L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. Transactions on Machine Learn ing Research, 2022.
14

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Yu, L., Shi, B., Pasunuru, R., Muller, B., Golovneva, O., Wang, T., Babu, A., Tang, B., Karrer, B., Sheynin, S., al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2023.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Zhang, C., Benz, P., Imtiaz, T., and Kweon, I. S. Under- standing adversarial examples from the mutual influence of images and perturbations. In CVPR, 2020.
Zhang, H., Yu, Y., Jiao, J., Xing, P., Ghaoui, L. E., and Jordan, M. L. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Zhang. J., Yi, Q.. and Sang, J. Towards adversarial attack on vision-language pre-training models. In ACMMM, 2022a.
Zhang. J., Ye, J., Ma, X., Li, Y., Yang, Y., Sang, J., and Yeung, D.-Y. Anyattack: Towards large-scale self- supervised generation of targeted adversarial examples for vision-language models. In CVPR, 2025a.
Zhang, P.-F., Huang, Z.,, and Bai, G. Universal adversarial perturbations for Asioo-languaae pre-trained models. In ACM SIGIR, 2024.
Zhang. P.-F., Bai, G., and Huang. Z. Maa: Meticulous adver- sarial attack against vision-language pre-trained models. arXiv preprini arXiv:2502.080779 2025b.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2205.01066 2022b.
Zhang, Y., Xu, Y., Shi, J., Zhang, L. Y., Hu, S., Li, M., and Zhang, Y. Improving generalization universal adver- sarial perturbation via dynamic maximin optimization. In AAAL, 2025c.
Zhao, Y., Pang, T., Du, C., Yang, X., Li, C., Cheung, N.- M. M., and Lin, M. On evaluating adversarial robustness of large vision-language models. In NeurIPS, 2023.
Zhou, Z., Hu, S., Li, M., Zhang, H., Zhang, Y., and Jin, H. Advelip: Downstream-aggoste adversarial examples in multimodal contrastive learning. In ACMMM, 2023a.
Zhou, Z., Hu, S., Zhao, R., Wang, Q., Zhang, L. Y., Hou, J., and Jin, H. Downstream-agnostic adversarial examples. In ICCV, 2023b.
Zhou, Z., Song, Y., Li, M., Hu, S., Wang, X., Zhang, L. Y., Yao, D., and Jin, H. Darksam: Fooling segment anything model to segment nothing. NeurlPS, 2024.
15

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
A. Comparison with Related Work
Cross-data Transterability: This refers to a perturbation's ability to be applied to different input samples and still achieve adversarial objectives. By design, UAPs inherently provide cross-data transferability since single perturbation intended to deceive all samples.
Cross-model Transferability: This refers to perturbation's 's capacity to transfer from the surrogate model on which it was generated to other, unseen victim models. This property aligns with the standard black-box threat model assumption in adversarial robustness.
Cross-domain Transferability: This refers to a perturbation's ability to remain effective when applied to data from different datasets or domains. extends cross-data transferability by requiring the perturbation to succeed on inputs not only from a single domain but across multiple, diverse domains.
Cross-task Transferability: This refers to a perturbation's 's effectiieness in fooling model on the original task, well as on other different tasks. In other words, the perturbation remains adversarially effective even when the model is used for objectives beyond those it was specifically designed to attack.
Super Transferability: This term indicates that a single perturbation can achieve cross- cross- cross,model, cross-domain, and cross-task adversarial transferability simultaneously.
Table Comparison with related works in adversarial attack on CLIP. The v/ and Xdenote can or cannot technically achieve adversarial mannferability, respectively. The ? denote can technically achieve the transferability but not studied in the corresponding paper.
Method
Threat model
Perturbation type
Cross-data
Cross-model Cross-domain Cross-task
Co-Attack (Zhang et al., 2022a)
White-box
Sample-specific
X
XXXXX
XXXX
XXX
X
Sep-Attack (Zhang et al., 2022a)
White-hox
Sample-specific
XXXX
X
SGA (Lu et al., 2023)
Black-box
Sample-specitie
XXXXX
SA-Attack (He et al., 2023)
Black-box
Sample-specine
VLP-Transfer (Gao al., 2024)
Black-box
hample-specifie
PRM (Hu et al., 2024)
Black-box
Sample-spese
X
AdvCLIP (Zhou et al., 2023a)
White-box
Universal
X
Doubly-UAP (Kim et al., 2024)
White-box
Universal
ETU (Zhang et al., 2024)
Black-box
Universal
C-PGC (Fang al., 2024b)
Black-box
Universal
DO-UAP (Yang al., 2024)
Black-box
Universal
X-Transfer (Ours)
Black-box
Universal
We provide summary of the transfer capabilities of related works in Table 4. This overview indicates only whether a given method can technically achieve certain forms of transferability; the actual extent of transferability is evaluated in our experiments.
Many existing studies on the adversarial robustness of CLIP (Zhang al., 2022a; Lu et al., 2023; He et al., 2023; Gao et al., 2024; Hu al., 2024) focus on sample-specifie perturbations. By design, such perturbations cannot achieve cross-data or cross-domain transferability, they are tailored individual samples. In contrast, UAPs inherently enable these forms of transferability. Moreover, once UAP is generated, it can be universally applied to any sample, whereas sample-specific perturbations require per-sample optimisation. This distinction makes UAPs more practical for large-scale adversarial attacks or comprehensive benchmark evaluations.
AdvCLIP (Zhou et al., 2023a) introduced the first UAP attack against CLIP encoders, described under a "quasi-black box" threat model, where the adversary can access a parent encoder but lacks direct access its downstream fine-tuned counterpart. In contrast, our work adheres strict black-box threat model, offering no access to any encoders used by the victim, including the parent version. Under this stricter definition, AdvCLIP is considered white-box attack. In comparison, both ETU (Zhang et al., 2024) and C-PGC (Fang et al., 2024b) are strict black-box attacks specifically tailored to deceive CLIP encoders. They are technically capable of achieving cross-domain transferability but have not empirically demonstrated this ability in their paper. In our evaluations, we include these methods to assess their cross-de doos--n transferability. In comparison, our proposed X- Trannsfer is designed to achieve super transferability, simultaneously enabling cross-data, cross-moded cross-domain, and gross-task adversarial transferability.
16

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
B. Lg-norm Perturbation and Adversarial Patch
Adversarial Patch. For the unrestricted adversarial patch attack, we construct the adversarial example using the following:
x' = A(x) m A + (1 m) æ;
(7)
where m E [0, 1] learnable 2D input mask that does not include the colour channels, A E [0, 1|3xwx is the universal adversarial pattern, and @ is the element-wise multiplication (the Hadamard product) applied to all the channels.
We optimise the following objective to generate targeted universal patch attack:
arg min (aa) Codo (f, x') + a ailmmlla + (m)) + (V))).
(8)
m
where D' is surrogate dataset, Lade follow Eq. (3) and 4, x' follows Eq. (7), (V()) the total variation loss, and the I1-Ila is the L norm. and B are two hyperparameters balance the two loss terms. While the patch attack is unrestricted, we set soft constraint that the patch has to be as small as possible. The L1 norm ensures that when the adversarial patch is added to the image, the patch is small and hard to notice. The total variation loss ensures the patch pattern and the mask are smooth.
L2-norm Perturbation. For the Lo-norm perturbation, we optimise the following objective:
arg min E(a)yarrLaad (J, + e- c-10112,
5
where the d is the perturbation and is hyperparameter that balance two loss terms. The universal adversarial function for L2 L2-norm perturbation x' = + 5. While the perturbation is not bounded, use the L2-norm to ensure the perturbation is small.
The evaluations of these different constraints are available in Appendix C.6.
C. Experiments
In Appendix C.I, we provide detailed overview of our experimental settings and analysis of efficiency. Appendix C.2 offers further details on the surrogate encoder search space. For all experiments, we utilise the open-source implementation OpenCLIP (Ilharco et al., 2021). Appendix C.3 provides detailed results comparing X-Transfer with baseline methods across each victim encoder and dataset: these results are summarised in Table in the main paper.
Appendix C.4 demonstrates the scaling capability of X- Transfer in comparison to baseline (ETU) that employs specialised adversarial objectives for CLIP. Appendix C.5 presents results obtained using alternative surrogate datasets. Appendix C.6. we show that the conelusions regarding scaling and adversarial super transferability also hold under other constraints, such the L--norm and adversarial patches. Appendix C.7 presents results for Targeted UAP (TUAP), where the adversary specifies a particular target text description. The findings are consistent with those for UAPs using non-targeted objectives. Additionally, we demonstrate that TUAPs can manipulate large --.--rreertted responses to align with the target text description. Appendix C.8 shows the analysis for X- X-Transfer against adversarial fine-tuned CLIP encoders. It shows that adversarial patch and Lg-norm perturbations are comparably more effective than the Lxe norm bounded perturbations.
Lastly, Appendix C.9 provides detailed qualitative analysis of UAPs generated with X- X-Transfer.
C.1. Detailed Experimental Setting
UAP Generation. We use ImageNet (Deng et al., 2009) as the default surrogate dataset. The value of k is set to 4 for the Base search space, for the Mid search space, and 16 for the Large search space. Following Fang et al. (2024b); Zhang et al. (2024), we employ -nom bounded perturbations with e 12/255, and the step size 17 is set to 0.5/255.
For all perturbations, we use the resolution of 224 224. For the adversarial patch, the value of is set to 3.0 x 10-5 2.0 10-5, and 1.0 x 10-5 for the Base, Mid, and Large search spaces, respectively. The value of 3 is set to 70. For the L2-norm perturbation, the value of c is set to 0.025, 0.02, and 0.015 for the Base, Mid, and Large search spaces, respectively. We use Adam (Kingma & Ba, 2014) as the optimiser for L2 norm perturbation and adversarial patch. The learning rate is set to 0.05, and no weight decay is used. For all perturbations, we perform the optimisation for I epoch on the surrogate dataset (ItageNet). The batch size is set to 1024.
17

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Evaluations. For the zero-shot classification and image-text retrieval, We use ResNet (RN) (He et al., 2016) and ViT (Dosovitskiy al., 2021) architectures the image encoders. We consider use diverse CLIP encoders released by OpenAl (Radford et al., 2021)~including ViT-L/14, ViT-B/16, ViT-B/32, RN-50, and RN-101-and encoders trained by others, such as ViT-B/16 trained with SigLIP (Zhai et al., 2023), EVA-E/14 (Sun et al., 2023), ViT-H//4 trained with CLIPA (Li et al., 2023e), and ViT-bigG/14 trained with MetaCIIP (Xu et al.. 2024). Details are summarised in Table
Table 5. List of encoders for the evaluations of zero-shot classifications and image-text retrieval, including each one's architecture. pre-training dataset, and the corresponding OpenCI.IP identifier. The OpenCLIP identifier is the values for arguments model name and pretrained in the create model. and transforms function from OpenCLIP.
Architecture
Pre-training Dataset
OpenCLIP Identifier
1
RN50
WebImage
(RN50, openai)
2
RNO1
WebImage Text
(RN101, openai)
3
VIT-B/16
Weblmage Text
(ViT-B-16, openai)
4
ViT-B/32
WebImage Text
(ViT-B-32, openai)
5
ViT-L/14
WebImage
(ViT-L-14, openai)
6
ViT-B/16-SigLIP
WebLI
(ViT-B-I6-SigLIP, WebLI)
7
EVA02-E/14
LAION2B
(EVA02-E-14, laion2b.s4h b11kk)
8
ViT-H/I4-CLIPA
DataComp
(VIT-H-14-CLIPPA, datacomplb)
9
ViT-bigG/14
MetaCLIP
(VITbbiGG-14-quiickcell metaclip fullec)
For evaluations downstream large VLMs, we use the OpenFlaminno-3B (OF-3B) (Awadalla et al., 2023), which aligned the CLIP image encoder (ViT-L from OpenAl) with the MPT-1B (Team et al., 2023), and LLa LA-BB (v1.5) (Liu et al., 2023a) which use the same image encoder as OF-3B, but aligned with the Ticuma-7B (Chiang et al.. 2023). Additionally, we evaluate MiniGPT4-v2, which aligned the ViT-G-14 trained with EVA-CLIP (Fang et al., 2023) with Llama2 (Touvron et al., 2023) and BLIP2 use the same vision encoder and aligned with OPT (Zhang et al., 2022b). The summary of the large VLMs we used in the evaluations is summarised in Table 6. For large VI.MS that use different image resolutions than our default 224 x 224, we use interpolation to rescale the perturbation to the resolution used by the VLM.
Table 6. Large Vision Language Models used the experiments.
Model Name
Image Encoder
LLM
Image Resolution
OpenF Flamingoog (OF-3B)
ViT-L/14 CLIP OpenAl
MPT-IB
224 x 224
LLaVA -B
ViT-L/14 CLIP OpenAl
Vicuna-77 7
224 x 224
MiniGPT4-v2
ViT-G/14 EVA-CLIP
Ilama2 Chat 7B
448 448
BLIP2
ViT-G/14 EVA-CLIP
OPT-6.7B
364 364
Baselines. We compare our approach to state-of-the--e UAP methods tailored for CLIP encoders, including C-PGC! (Fang al., 2024b), ETU2 (Zhang al., 2024), and AdvCLIP2 (Zhou et al., 2023a). We also evaluate against UAPs originally designed for image classifiers, GD-UAP" (Mopuri et al., 2018), TRM-UAP (Liu et al., 2023b), and Meta-UAP (Weng et al., 2024). All UAPs are either directly obtained from official open-source repositories or generated using the official code provided by each baseline's authors.
C.2. Search Space
Our search space for CLIP surrogate encoders spans wide range of architectures, pre-training datasets, objectives, and training recipes. For architectures, we include ResNet (RN) (He et al., 2016), ConvNext (Liu et al., 2022), ViT (Dosovitskiy et al., 2021), ViTamin (Chen et al., 2024b), NLLB (Visheratin, 2023), MobileCLIP (Vasu et al., 2024), and RoBERTa (Liu et al., 2019). For pre-training datasets, surrogate encoders are trained on diverse datasets, including CC12M (Changpinyo al., 2021), YFCCI5M (Thomee et al., 2016). LAION (Schuhmann et al., 2021; 2022). DataComp/Common (Gadre al., 2023), Merged2B (Sun al., 2023), DFN (Fang et al., 2024a), WebL.I (Zhai et al., 2023), and MetaCLIP (Xu et al.,
'ntps://gituu..com//iiiiii https///gghubbooom///ibbee//g/gp wttp::/ppppp..p..uueeeses..caaaa www.wenaalaaaaaae...
VLP
18

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table Details regarding each baseline method and different configurations.
Method
Variant
Note
Seg
Generated with segmentation model (ResNet152 hackbone) as the surrogate.
GD-UAP
CLS
Generated with ResNet152 classifier as the surrogate.
VIT-B/16
Generated with CLIP VIT-B/16 released by OpenAl on the Generated with CLIP ResNet101 released by OpenAl on the NUS-WIDE dataset.
dataset.
AdvCLIIP
RNI01
GoogleNet
Generated with GoogleNet classifier trained on ImageNet.
TRM-UAP
RNI52
Generated with RN152 classifier trained on ImageNer.
Ensemble
Generated with an ensemble of DenseNet121, VGG16, and ResNet50 classifiers trained on ImageNet.
Meta-UAP
Ensemble-Meta
Same above, with meta-leaming strategy.
RN101-Flicke
Generated with CLIP ResNer101 released by OpenAl on the Flicker dataset. Generated with CLIP ResNet101 released by OpenAI on the MSCOCO dataset. Generated with
RN101-COCO
C-GPC
ViT-B/16-Flicker
VIT-B/16 released by OpenAI on Flicker dataset. Generated with CLIP VIT-B/16 released by OpenAI on the MSCOCO dataset.
ViT-B/16-COCO
RN101-Flicker
Generated with CLIP ResNer101 released by OpenAl
Flicker dataset. Generated with CLIP ViT-B/16 released by OpenAl on the Flicker dataset.
ETU
ViT-B/16-Flicker
2024). For pre-training objectives and recipes, we consider methods such as SigLIP (Zhai et al., 2023), EVA-CLIP (Sun et al., 2023), CLIPA (Li et al., 2023c), and COCA (Yu et al., 2022).
Details about the Base, Mid, and Large search spaces are provided Tables 8, 9, and 10, respectively. The Base search space balanced and comprises encoders from diverse architecture types: ResNet (RN) (He al., 2016), ConvNext (Liu al., 2022), ViT-B, and ViT-I (Dosovitskiy al., 2021), with 4 encoders per architecture. The Mid search space expands on the Base search space by incorporating additional VIT-L and VTT-B models available in OpenCLIP. The Large search space further includes larger models, such as ViT-H and ViT-bigG, augmenting the Mid search space. To ensure strict black-box evaluation setting. there is no overlap between the surrogate encoders in these search spaces and the encoders used evaluations (Tables 5 and 6).
Table List encoders in Base search space, including each one's architecture, pre-training dataset, the corresponding OpenCLIP identifier.
Architecture
Pre-training Dataset
OpenCLIP Identifier
RN101
YFCCISM
(RNI01, 15c)
2 RN50
YFCCI5M
(RN50, yfee15m) (RN50, ec cm)
3
RN50
CC12M
RN101-quickgelu
YFCCI5M
(RN1O1-quickgelu, yfcc15m) (convnext.base, laion440mm.s1.3bb511
5
ConvNext Base
LAION400M
6
ConvNext Base-W W
LAION2B
(convnext.base.wg
7
ConvNext Base-W
LAION2B
(convnext base w, laion2b s13b b82k augreg)
8
ConvNext Large-D
LAION2B
(convnext large laion2b s26b. b10kkka0rkkgg
9
ViT-B/16
DFN2B
(ViT-B-16, dfn2b)
10
ViT-B/16
DataComp
(ViT-B-16, datacomp _l___xlll__bbbxx_.
ViT-B/32
LAION2B
(ViT-B-32, laion2b_s34b_b799
12
ViT-B/32
DataComp
(ViT-B-32, datacomp_xl 3ttpppp__xlll1lbb0000_0
13
VIT-L/14
LAION400M
(ViT-L--14, laion400m_es3
VIT-L/14
CommonPool
(ViT-L-14, commoupool s13b b90k)
15
EVA02-L/14
Merged2B
(EVA02-L-14, merged2b s4b b131k)
16
VIT-L/14-CLIPA
DataComp
(VIT-L-1-CCLIPA, datacomplb)
19

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table List of encoders in the Mid search space, including each one's architecture, pre-training dataset, and the corresponding OpenCIIP identifier.
Architecture
Pre-training Dataset
OpenCLIP Identifier
RN101
YFCC15M
(RN101, yfcc15m)
2
RN50
YFCCISM
(RN50, yfcc15m)
3
RN50
CC12M
(RN50, cc12m)
4
Conv Base
LA1ON400M
(convnext base, laion400m s13b b5lk)
5
ConvNext Base-W
LATON2B
(convnext base laion2h s13b b82k)
6
ConvNext Base-W
LATON2B
(convnext hase w, laion2h s13b b82k augreg)
7
ConvNext Base-W
LATON Aesthetic
(convnext_hase) laion_ diiio_aesthetic.s
b82k)
8
ConvNext Large-D
LAION2B
(oonneett_arge_dl laion2b_s26b. battob_b12kkaaugeeee
ConvNext XXLarge
LAION2B
(cnnvnctt.xxaagge laion2b bat44bbb22k_uagee (8grge)
10
ViT-B/32
LAION400M
(ViT-B-32, laion400m.c31)
11
ViT-B/32
LA1ON400M
(ViT-B-32, laion400m.c32))
12
VIT-B/32
LAION2B
(ViT-B-32, laion2b_e16)
VIT-B/32
LAION2B
(ViT-B-32,
bbbkk)
14
VIT-B/32
DataComp
(ViT-B-32. 32. datacomp xl s13b b90k)
VIT-B/16
LAION400M
(ViT-B-16, laion400m e31)
16
ViT-B/16
LATON400M
(ViT-B-16, laion400mm e32)
ViT-B/16
LAION2B
(ViT-B-16, laion2b s34b b88k)
18
ViT-B/16
DataComp
(ViT-B-16, datacomp_xl sl3b )bbbbbbbkk))
ViT-B/16
DataComp
(ViT-B-16, Istli
20
ViT-B/16
DFN2B
(ViT-B-16, dfn2bb
21
EVA02-B/16
Merged2B
(EVA02-B-16, merged2b s8b_.88b1111111
22
ViT-L/14
LATON400M
(ViT-L-14, laion400m_e31)
23
ViT-L/14
LA1ON400M
(ViT-L-14, laion400m.e32)
24
ViT-L/14
LAION2B
(VIT-L-14, laiou2b_s32b_bs
25
ViT-L/14
DataComp
(ViT-L-14, datacomp xl s13b b90k)
26
ViT-L/14
CommonPool
(ViT-L-14, commonpool xl clip s13b b90k)
27
ViT-L/14
CommonPool
(ViT-L--14, com.mo4.ccmmmpool..ooo x1 lll.laiinnnnn.bbl.... b90k)
28
ViT-L/14
CommonPool
(ViT-L-14, 13.
29
ViT-L/14
DFN2B
(ViT-L-14, dfn2b)
30
EVA02-L-14
Merged2B
(EVA02-L-14, merged2b bbbbhlb3bbbbbbl3kk b131k)
31
VIT-S0400M-14-SigLIP
WebLI
(VIT500400M:14-iigL.PP webli)
32
ViT-L/14-CLIPA
DataComp
(VIT-L-I4-CIIPPA, datacomp
20

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table 10. List of encoders in the Large search space, including each one's architecture, pre-training dataset, and the corresponding OpenCLIP identifier.
Architecture
Pre-training Dataset
OpenCLIP Identifier
1
RN101
YFCCI5M
(RNI01, yfec
2
RN50
YFCC15M
(RN50, yfec15m)
3
RN50
CC12M
(RN50, cc12m)
4
ConvNext Base
LA1ON400M
(convnext_basee daion4000_s133.i555
5
ConvNext Base-W W
LAION28
(convnext, base_w, Stionnb_s133_bbe8 b82k)
6
ConvNext Base-W W
LAION2H
(convnext hase w, laion2h s13h b82k augreg)
7
ConvNext Base-W
LAION Aesthetic
(convnext base w, laion acsthetic s13b b82k
8
ConvNext Large-D
LAION2B
(convacxt. large.d, laion2b.s s26b..66bbbbb laion2b.s2bb.l00kk.kn al0ree
9
ConvNext XXLarge
LAION2B
sconvexxtxxlarge laion2b. s34b b82k augreg)
10
ConvNext XXLerge
LAION2B
convnext.xxlarge wtt.b//244bbb8.auurrrga
11
Conv XaLa
LAION2B
(comvnext_xxlarge htwp://.34bbbs2k_auggreee
12
VIT-B/32
LAION400M
(ViT-B-32, laion400m.e311
13
VIT-B/32
LAION400M
(VIT-B-32, Iaion4000me333
14
VIT-B/32
LAION2B
(ViT-B-32, laion2h_e16)
15
VIT-B/32
LAION2B
(ViT-B-32, laion2h <34b h79k)
16
VIT-B/32
DataComp
(ViT-B-32, datacomp s13b b90k)
17
VIT-B/32
MetaCLIP
WVT-B-32-quickeel metaclip 400m)
18
VIT-B/32
MetaLII
(ViT-B-5S-quicceeluu metaclip fullcc)
19
VII-B/16
LAION400M
(VIT-B-16, lnion400mm_e3))
20
VIT-B/16
(VIT-B-16, laion4000.
VIT-B/16
LAION2B
(ViT-B-16, laion2b_x34b...t
22
VIT-B/16
DataComp
(ViT-B-16, datacomp x1bb b90k)
23
VIT-B/16
DataComp
(VTT-B-16 datacomp. (sbbbmb
24
VIT-B/16
CommonPool
(ViT-B-16, commonpool
68k)
25
VIT-B/16
DEN2B
(VIT-B-16, dfn2b)
26
VIT-B/16
MetaCLIP
WVT-B-I6--uiikgellu metaclip.400m)
27
VIT-B/16
MetaCLIP
(ViT-B-16-quel NVT-B--6-quickgelu. metuclip fulkce)
28
EVA02-B/16
Merged2B
(EVA02-B-16. merged22bs8bbbb33lk
29
ViT-L/14
LAION400M
(ViT-L-14, laion-00mm_e31)
30
ViT-L/14
LATON400M
(ViT-L-14, laiom400m_e32)
31
ViT-L/14
LAION2B
(VIT-L--14, laion2b_s3bb_ b82k)
32
ViT-L/14
DataComp
(ViT-1-114 datacomp_xl___ 1h_hppp______11_990000
33
VIT-1/14
CommonPool
(ViT-1-144 commonpool xi clip s13b h90k)
34
VIT-L/14
CommonPool
(VTT-L-14, commonpol x] laion s13b b90k)
35
VIT-L/14
CommonPool
(VIT-L-14, commonpool..eccoo 15b.x
36
ViT-L/14
MetaCLIP
(VTT-L-14-qucckeeuu, metaclip 400m)
37
ViT-L/14
MctaCLIP
IVT-L--4-quickglluu metaclip_fullae fullec)
38
VIT-/14
DEN2B
(WIT-1-14, dfn26)
39
EVA02-L-14
Merged2B
(EVA02-L-14, merged2b_s4b.bb b4b_b131k)
40
WT-S00400M-14-SiLLII 14-
WebLI
(VT:50440M--14-ig)) webli)
41
VIT-L/14-CLIPA
DataComp
(VIT-L-14-CLLIPA datacomp1b)
42
VIT-H/14
LAION2H
(VIT-H-14, laion2b <32h 679k)
43
VIT-H/14
MetaCLIP
NVT-H--4-quickgelu, metaclip fullce)
44
VIT-H/14
DRNSB
(VIT-H-L4-quickgcll dfn5b)
45
VIT-G/14
LAION2B
(ViT-g-14, laion2b.sl2b.bet
46
VIT-G/14
LAION2B
(ViI-g-14, laion2b.x34bbb88)
47
EVA01-G/14-plus
Merged2I
(EVA01-g-14-plas 1h.ree
48
EVA02-E- 14-plus
LAION2B
(EVA02-E-144ph) lnion2b_Abb.l444
49
ViT-bigG/14
LAION2B
(ViT-bigG--le G44, lww.ooo_s39..hhntttto
50
WTT-iggG- I-CLIPA
DataComp
(ViT-higG-14 144CIIIPAA datacompl1))
51
ROBERTA-VITE
LAION2B
Croberta-Vi--Bs laion2h x12h b32k)
52
XLM-ROBERTA VIT-B/32
LAIONSB
(xlm-roborta-base- VIT-B-32, laion5b s13b 690k)
53
XLM-ROBERTA Large- VIT-H/14
LAIONSB
Alln-roberaa-lage--i-H--A fww.en.laionfb.ll.bbb.
54
NLLB-Base
(nllb-clip-buse,
55
NLLB-Large
(nllb-clip-Iurge, v1)
56
ViTamin-Ii
DataComp
(ViTamin-IB. datacomp1b)
57
ViTamin-B-L1 BB
DataComp
(ViTamin-B-LTT.. datacompl1l
58
ViTamin-L
DataComp
(ViTamin-L, datacomplb)
ViTamin-1.2
DataComp
(ViTamin-122, datacomp
MobileCLIP-SI
DataComp
(MobileCI.IP.SI datacompdr)
MobileCLIP-S2
DataComp
(MobilsCLIP-S2. datacompdr)
62
MobileCLIP-B
DataComp
(MobileCLIP-B. datacompdr)
63
COCA ViT-L/14
LAION2B
ViT-L-I, laion2b.s136 s13bbb90k))
64
COCA VIT-B/32
LAION2B
ViT-B-V
lnion2b.s13b.b90)

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
C.3. Extended Results
In Figures 5, and 7, we present detailed comparison of X-Transfer against baseline methods across 9 victim CLIP encoders. For baselines such ETU (Zhang et al., 2024) and C-PGC (Fang et al., 2024b), which use ViT-B/16 trained by OpenAl (Radford et al., 2021) as the surrogate model, exclude results for ViT-B/I6 as the victim model, as this configuration constitutes white-box setting. Each baseline's s results reflect its best performance across all tested configurations. Notably, results shows that X-Transfer achieves state-of-the-art performance on all victim encoders for both zero-shot classification and image-text retrieval tasks.
GD-UAP
X-Transter X-Transfer Mid
GD-UAP
X-Transter
TRM-UAP
C-PGC
TRM-UAP
C-PGC
X-Transfer Mid
AdVCLIP
ETU
X-Transfer Large
AdvCLIP
ETU
X-Transfer Large
VIT OpenAl
VIT OpenAl
VMB/n6-CpeeAl
88.6
98.4
VTT-8/32-OpenAl
VIT-6/32-OpenN
RN101-OpenAl 98.2
88.2
VIT-L/1--OPAAii
MTT-L/14-OPpAAI
100
AN50-Operr OpenAI
NNSO OpenAI
B6.
97.6
VIT-B-16-SigU
WIT-B-16-SiLL
vi qcikkgeu 89.0
w G/14 quickgeeu
98.4
EVA02-E/14
EVA02-E/14
B6.7
97.6
VIT-HY14-CLIPA
VITH//44-CURP
(a) CIFARIO
(b) CIFARIO0
GD-UAP
Meta-UAP
X-Transfer Base
GD-UAP TRM-UAP
Meta-UAP
X-Transfer Base
TRM-UAP
C-PGC ETU
X-Transfer
AdvCLIP
X-Transfer Large
AdvCLIP
ETU
X-Transfer Large
VIT-8/16-Openii
VIT-8/16-Openi
93.0
92.1
HTT8/32-OpenAi
VIT-8/32-
85,1
81.0
RN101-OpenAi
RNIOl-Oppnnl
VIT-L/4--penA
VIT-L/ 4-P-nn--
84.4
CON
90.2
100
60%
6OL
42%
40%
2014
2014
RN50-OpenW
94.9
VIT--16 Sight S 83.9
VIT--166 SigLt
05.1
vigaag quickgeluu
Sighri4 (oggi11-ulldggllu
63.1
EVA02-E/14 63.1
EVA02-E/14 96.0
VIT-H/4--CUPA
VIT-H/44-CUPA 95.2
(c) Food101
(d) GTSRB
Figure 5. Detailed results the non-targeted attack success rate for 9 victim encoders in zero-shot classification task, evaluated on the following datasets: (a) CIFAR- 10, (b) CIFAR- (c) Food101, and (d) GTSRB.
22

---

X- X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
GD-UAP
Meta UAP C-PGC
X-Transfer Base
GD-UAP
Meta-UAP UAP C-PGC
Meta
Trannsfer Base X-Transter Mid
TRM-UAP
X-Transter Mid
TRM-UAP
AdvCLIP
X-Transfer Large
AdvCLIP
X-Transfer Large
VIT-8/16 OpanAl 75.2
VIT-8/16 OpanAl
88.3
V/T-8/32 OpenAl
VIT-B/32 OpenAl 75.0
.8.8
RN101-0penAi 80.6
-N101-0ppnnl 88.0
VIT-L/66-peend 66.1
VIT-L/14-Opent
100
78.4
Fila
20%
20%
N550-ppenAl 81.7
ONS0-OpenAl
VIT-B-16-SiUU 62.1
VIT-B-16 SigL
48.4
Wiblgg//44 qelkgeel
big quickkeet
EVA02-E/14
31.9
VIT-H/1-CLIIPA
VITHH/4-CLIPA 33.1
$3.2
(a) ImageNet
(b) Cars
GD-UAP
X-Transfer
GD-UAP
X-Transfer Base
TRP
C.PGC
Transffrr Mid
TRM
C.PGC
Transffer Mid
AdVCLIP
ETU
X-Trasffer Large
AdvCLIP
ETU
X-Transfer Large
VIT-0/16-OperW
VIT-0/66-0penW
VTT-0/22-OpenAl
ViT-8/32-OpenAl
RN101-OpenAI 77.5
RN101- OpenAl
78.8
VIT-L/1--PpenA
VIT-L14--penAl
EN50-Opend B1.0
Open-A 81.4
VIT--16-Sigt
VIT-8-16-Sigt
Wi Ni-biig/l4--quikgee 37.4
Vi Mi-biggr/l4quuikkee 52.4
EVA02-E/14
EVA02-E/14 45,4
42.5
VIT-H/14-CUPA
VIT-H/14-CUPA
(c) STLIO
(d) SUN397
Figure 6. Detailed results on the non-targeted attack success rate for 9 victim encoders in a zero-shot classification task, evaluated on the following datasets: (a) ImageNet, (b) Stanford Cars, (c) STL10, and (d) SUN397.
C.4. Scaling with ETU
In Table 11, present the results of scaling with our adversarial objective in conjunction with ETU (Zhang et al., 2024), which utilises a specialised loss function designed for CLIP encoders. We denote the application of our efficient search algorithm with the ETU loss function as 'ETU+X-Transfer" using the Base search space. For comparison, we include X-Transfer Vanilla, which does not use scaling and solely employs our adversarial objective, as well as X-Transfer Base. The results indicate that our generic adversarial objective function is essential for effectively scaling up the number of surrogate models, enabling super transferability across diverse settings.
23

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
GD-UAP
Meta UAP
XTransfer Base
GD-UAF TRM-UAP AdvCLIP
Meta Meta-UAP
XTransfer Base X-Transfer Mid
TRM-UAP
C-PGC
X-Transfer Mid
C-PGC
AdvCLIP
X-Transfer Large
X-Transfer Large
VITB/16-0pennl
VIT-B/16-Opennl
71,35
VIT-B/32- OpenAi
VIT-B/32-Opendl
74.55
65.50
RN101-OpenAI 80.45
-Npen-OppnAi 76.35
VI-L:14-0penAl
74.84
100%
VPP-L140ppen 72.56
100
42%
40%
2014
OpenA))
N50-OpenAA 77.94
85.18
71.22
64.17
VI--iigG/34 quickgelu
EVA02-E/14
54.79
VIT-HYT4-CLIPA
VIT-N/1--CLIA
64.99
65.52
(a) T
(b) IR
Figure 7. Detailed results the non-targeted attack success rate for 9 victim encoders in the image-text retrieval task, evaluated the MSCOCO, with metric (a) TR@ and (b) @.@1.
Table 11. Comparison between scaling with ETU and scaling with our adversarial objective function.
Zero-Shot Classification
1-T Retrieval
Method
Search Space
C-10
C-100
Food
GTSRB
ImageNet
Cars
STL
SUN
Avg
IR@
ETU (Zhang et al., 2024) None (N I)
70.2
86.5
47.1
71.1
34.1
31.1
27.5
31.0
49.8
40.2
32.8
ETU + X-Transfer
Base (N 16)
73.6
88.6
48.8
69.0
31.9
27.2
21.7
24.7
48.2
41.6
32.1
X-Transfer Vanilla
None (N
72.7
88.3
49.9
72.3
31.2
26.3
19.2
27.6
48.4
42.3
34.5
X-Transfer Base
Base (N 16)
86.6
74.8
89.3
56.0
52.1
46.8
50.7
69.2
63.7
58.8
C.5. Comparing Surrogate Dataset
In Table 12, present the results of using MSCOCO (Lin et al., 2014) surrogate dataset and compare them with ImageNet. The results show that the ASR is comparably similar across both datasets. This finding indicates that super transferability does not depend on the choice of the surrogate dataset. Furthermore, in Appendix C.9, we demonstrate that the patterns generated in the UAP are also independent of the surrogate dataset. Instead, is the surrogate encoders that primarily influence both the transferability and the patterns the UAP.
Table 12. Comparison between surrogate datasets used for generating UAPs using X- Transfer.
Surrogate Dataset
Zero-Shot Classification
I-T Retrieval
C-10
C-100
Food
GTSRB
ImageNet
Cars
STL
SUN
Avg
TR@:
IR @
ImageNet
86.6
97.5
74.8
89.3
56.0
52.1
46.8
50.7
69.2
63.7
58.5
MSCOCO
84.9
96.2
69.7
82.4
53.1
49.7
37.3
49,9
65,4
61.9
57.8
C.6. UAP with L2-norm Perturbations and Adversarial Patch
In table 13, we present the results for using Lornorm perturbation and adversarial patch introduced Appendix B. The results show consistent conclusions that the efficient scaling of X-Transfer can improve super transferability.
24

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table 13. The non-targeted ASR (%) results for Iz-norm perturbation and adversarial patch.
Search Space
Zero-Shol Classification
I-T Retrieval
Type
C-10
C-100
Food
GTSRB
ImageNet
Cars
STL
SUN
Avg
TR@1
IR@]
Base (N- 16)
85.7
97.5
77.5
90.1
64.6
61.0
50.3
65.5
74.0
76.0
69.9
L2
Mid (N = 32)
86.8
98.4
86.7
90.1
83.7
73.1
60.7
84.0
82.9
88.6
81.3
Large (N 641
88.2
98.4
95.5
92.1
88.5
83,9
81.1
90.2
89.7
93.6
88.5
Base (N- 16)
51.5
83.9
39.5
75.9
48.1
46.7
10.9
42.9
49.9
64.2
49.0
Patch
Mid (N- = =32)
55.7
85.1
51.0
80.1
54.9
48.8
15.3
56.
55.9
65.7
55.0
Large (N- 641)
88.6
98.5
98.9
93.7
99.9
99.3
89.7
99.6
96.0
100.0
99.9
C.7. TUAP
In this section, we present the evaluation results of the X- T-Trnnsfer attack with a targeted objective, where the adversary can specify any text description as the target.
Target Text Description. We use a total of 10 target text descriptions for evaluating TUAP. Targets No. to No.6 are adopted from existing works (Schlarmann & Hein, 2023; Schlarmann et al., 2024). We constructed the rest of the targets ourselves.
Evaluations. We adopt the standard shot classification setup and utilise the template provided by Radford et al. (2021) for each evaluation dataset. For instance, we use the format "an image of X," where X is replaced by the class name. To evaluate TUAP, we measure the attack success rate. For each dataset, we include an additional class representing the adversary's 's target and replace X with the target text description. TUAP is applied to each image in the evaluation dataset, and the victim model generates image embeddings. The attack is deemed successful if the closest embedding matches the template containing the adversarial target text description.
For image retrieval, we randomly select an image, apply perturbation, and use an adversary-specified target text sentence as the query. We report the rank of the perturbed image among all images the Image Retrieval Rank (IR Rank), where a lower rank signifies more successful TUAP. For the MSCOCO dataset, which contains 3,900 images, we repeat the retrieval process 50 times for each attack type, victim model, and target text sentence. The results are reported as the mean and standard deviation of the IR Rank.
For image captioning and VQA tasks evaluated with large VLMs, we use the widely adopted CIDEr metric (Vedantam et al., 2015) for captioning tasks and VQA accuracy for question answering. Additionally, for image captioning, we report the BLEU-4 score (Papineni al., 2002) measure the similarity between the generated caption and the adversary's target text description the targeted ASR. A higher BLEU-4 score indicates greater alignment with the target text. BLEU-4 scores are omitted for VQA tasks due to the brevity of the answers. In both tasks, we apply the TUAP to each image in the evaluation dataset and assess the victim VLM's response.
Results. In Table 14, we present the results of our TUAP on the zero-shot classification and image-text retrieval tasks. The results demonstrate that the scaling capability aligns with our analysis in the main paper for the non-targeted objective. Notably, in this case, the attack is deemed successful only if the prediction matches the text description specified by the adversary, which is inherently more challenging than non-targeted objective that merely causes arbitrary errors. In Table 15, we provide the results of evaluating image captioning and VQA tasks on large VLMs. These results similarly exhibit consistent scaling capabilities. Furthermore, the responses generated by the large VLMs closely match the targeted text description, as measured by the targeted ASR (BLEU-4 score between the response and the target text description). A qualitative example is provided in Section 4.3. These findings indicate that X-Transfer is capable of generating both non-targeted UAPs, which cause arbitrary errors, and targeted UAPs, which align predictions with adversary-specifeed text descriptions.
Relation with related works. The TUAP threat model presented in this section is related to concurrent works (Shayegani et al., 2023; Carlini et al., 2023; Wang et al., 2024b; Tao et al., 2024; Wang et al., 2024c; Gong et al., 2025) that explore jailbreak attacks against large VLMs, which typically involve a collection of harmful or adversarial "targets." However, our threat model differs fundamentally, and we target the CLIP encoder directly rather than downstream VLMs. These differences in focus and attack surface make direct comparisons with existing jailbreak methods unfair. Nevertheless, we believe that TUAPs generated by X-Transfer on CLIP, especially when aligned with jailbreak style prompts, could serve as
25

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Table 14. The targeted ASR (%) results in zero-shot classification and image-text (I-T) retrieval tasks across different CLIP encoders and datasets. 1-T retrieval is evaluated on MSCOCO. Results are based on averaging over 9 black-box vietim encoders and 10 target text descriptions.
Search Space
Zero-Shot Classification
I-T Retrieval
C-10
C-100
Food
GTSRB
ImageNet
Cars
STT.
SUN
Avg
TR
TR Rank
Base (N 16)
99.9
99.1
72.0
98.0
50.1
44.2
89.6
53.7
75.8
35.3
233.0
Mid (N 32)
100.0
99.6
77.9
98.3
56.0
54.4
90.9
60.1
79.7
42.9
206.0
Large (N 64)
100.0
99.8
79.6
98.9
57.2
54.5
92.7
60.4
80.4
42.3
167.5
Table 15. Non-targeted ASR (%) and BLEU-4 results in image captioning and VQA across various large VLMs and datasets. For image captioning, CIDEr is used the evaluation metrie, while VQA accuracy is employed for the VQA task. Results are based on Loe nnor bounded perturbations and are averaged across 10 different target descriptions.
Victim
COCO
Flickr-30K
OK-VQA
WizViz
Method
Model
Non-targeted ASR
BLEU-4
Non-targeted ASR
BLEU-4
Non-targeted ASR
Base
61.0s5
13.6#6.1
56.4+3.5
12.0=5.7
36.3+3.2
37,4+2.2
Mid
OF-3B
68.0 08.0+4.5
16.4=7.1
63.943.8
14.0=6.6
40.7+5.1
41.3+5.0
Large
70.7+4.7
17.8+9.0
66.024.4
15.347.4
41.026.7
41.8+5.7
Base Mid
39.5
12.246.5
35.1
11.8:6.7
16.8+4.1
20.2+6.5
LLaVA-7B
41.545.0
13.0#7.0
37.014.6
12.5+7.1
17.4+3.0
19.7+4.6
Large
45.9+3.4
14.2#7.6
40.3+3.0
13.1=7.1
18.943.1
15.4+7.8
Base
34,245.7
9,626.5
30,743.9
10.0#7.0
17.4+2.8
11,0+3.8
Mid
MiniGPT4
35.343.7
9,946.4
31,042.8
10.2+6.9
17.5+2.3
9.7=4.8
Large
40.2+3.6
11.3+7.0
35.1+2.7
11.247.1
18.7+2.8
11.626.8
Base
50.5+7.1
13.4=6.7
43.246.5
11.9=6.8
48.9#6.2
64.6+2.8
Mid
BLIP2
53.0+7.0
15.7+8.6
46.026.8
13.8+8.4
50.627.0
64.824.9
Large
52.246.5
15.2+9.3
45.426.6
13.8=9.1
45.0+8.0
62.546.8
effective initialisation points for future jailbreak attacks targeting large VLMs.
In parallel, TUAP can also be viewed as form of backdoor trigger (Carlini & Terzis, 2022; Jia et al., 2022; Liang et al., 2024; Bai et al., 2024; Huang et al., 2025). However, unlike these attacks, TUAP achieves its objective without data poisoning or any training-time manipulation. Instead, operates as a test-time backdoor trigger (Lu et al., 2024), making novel and distinct type of safety vulnerability in pre-trained vision encoders. This shift from training- to test-time attack surfaces introduces new challenges securing multi-modal models.
C.8. Evaluation on Adversarial Trained Encoders
In this section, we analyse UAPs generated by X-Transfer when evaluated with adversarially trained CLIP models. Mao al. (2023) proposed supervised adversarial training method fine-tuned ImageNet, and its performance improved through unsupervised fine-tuning (Schlarmann et al., 2024). For evaluations, we include two adversarially trained CLIP image encoders: FARE-2 (Schlarmann et al., 2024) and TeCoA-2 (Mao et al., 2023). The suffix "-2" indicates that the models were trained with L.s noom perturbations bounded to 255
further
As shown in Table 16, adversarial training can defend against -nom bounded UAPs, which aligns with existing literature. well established that adversarial training can defend against universal perturbations (Weng et al., 2024) and provides resistance to black-box attacks. However, our results demonstrate that these adversarially trained models remain vulnerable to adversarial patches and Ly-norm perturbations. This is likely because they are specifically trained to counter only Lose -oor bounded attacks. The adversarial training is robust to multiple different types of perturbations (Kang et al., 2019; Croce & Hein, 2020a; Hsiung et al., 2023), which have not been explored in CLIP.
These findings suggest that while adversarial training can partially mitigate CLIP's vulnerabilities, does not provide comprehensive robustness. Furthermore, adversarial training often requires a trade-off between clean zero-shot accuracy and robustness (Tsipras et al., 2019) and known for being computationally expensive, limiting its scalability on web-scale
26

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
datasets.
Table 16. The non-targeted ASR (%) results for the evaluation of CLIP encoders are fine-tuned with adversarial training.
Victim Encoder
Zero-Shot Classification
Type
C-10
C-100
Food
GTSRB
ImageNet
Cars
STL
SUN
Avg
FARE
14.2
23.7
17.9
33.0
5.9
4.5
1.9
6,4
13.4
Loo
TeCoA
3.7
5.7
12.6
17.9
3.0
4.2
1.2
4.8
6.6
FARE
76.1
93.1
64.8
71.0
47.4
34.3
23.2
56.3
58.3
L2
TeCoA
27.0
46.0
34.6
46.4
13.9
19.5
21.6
27.0
FARE
88.8
98.2
98.8
95.0
99.9
99.3
89.8
99.8
96.2
Patch
TeCoA
88.5
98.2
98.2
94.1
99.9
98.9
89.6
99.2
95.8
C.9. Qualitative Analysis on UAPs
The visual clarity of the UAP patterns scales with the search space size N the X-Transfer attack, as shown in Figure 8(a). This further supports the observation that visual clarity correlates with the attack success rate. Additional visualisations of the effects of UAPs and TUAPs applied images, along with the corresponding responses from VLMs, provided in Figure 9. The visualisations of TUAPs are presented in Figure 10.
Vanilla (1)
Base (16)
Mid (32)
Large (64)
(a) X-transfer with different search space (N).
ImageNet
RSICD
(b) X-transfer with encoders trained RSICD dataset, with ImageNet (left) RSICD (right) surrogate dataset.
Figure 8. The visualisation of UAPs in (a) scaling with N of X Transfer, and (b) generated with encoders trained on RSICD datasct. All UAPs are generated with the Base search space.
Table 17. The predicted class with UAP generated by X-Transfer applicd to images in each dataset. Results are based on CLIP encoder ViT-I/14 released by OpenAI.
Dataset
Top Predicted Class
CIFAR-10
Ship (87.6%), Airplane (8.6%)
CIFAR-100
Ray (99.2%)
Food101
Cheese plate (16.7%), Grilled cheese sandwich (14.6%), Cheesecake (12.4%) Red and white triangle with snowflake / ice warning (76.0%), Stop (4.3%)
GTSRB
ImageNet
Jay (18.0%), Dust jacket (3.4%), Kuvasz (2.9%)
Cars
Bentley Arnage Sedan 2009 (9.8%), Honda Odyssey Minivan 2012 (8.0%)
SUN397
Cheese factory (30.7%). Trench (10.7%), Discotheque (8.8%
27

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
In this subsection, we further explore the origin of the UAP patterns generated by X-Transfer. We hypothesise that most UAPs generated exhibit building-like patterns due the pre-training datasets of the surrogate encoders in search space. To investigate this, conducted experiments using CLIP encoders fine-tuned on the Remote Sensing Image Captioning Dataset (RSICD) (Lu et al., 2017). Using 4 encoders" trained on remote sensing imagery as the search space, we applied X-Transfer to generate UAPs. The resulting perturbations, shown in Figure8(b), exhibit patterns resembling remote sensing imagery. This supports our hypothesis that the semantic characteristics of UAPs are influenced by the pre-training datasets of the surrogate encoders.
To explore further, we applied UAPs generated by X-Transfer to various datasets and used CLIP encoders for zero-shot classification. Surprisingly, while the UAP patterns visually resemble building-like like structures, their predicted classes often lack any connection to buildings. As detailed in Table 17, CLIP encoders frequently predict concepts that seem unrelated to human interpretations of the perturbation. For example, on the SUN397 and Food 101 datasets, predictions skew toward cheese-related concepts, while on the GTSRB dataset, predictions often correspond to the "ice warning" traffic sign, possibly due to semantic similarities to cheese-like like textures. Unlike targeted attacks, these UAPs do not consistently steer the encoder toward a specific class. On datasets like CIFAR and ImageNet, the predictions vary significantly.
These findings indicate distinction between the apparent semantic content of UAP patterns and their adversarial impact on CLIP encoders. While humans interpret these patterss as meaningful (e.g., building-like they mislead CLIP encoders into producing diverse and often uninterpretable predictions.
As for why building-like patterns are predominant, we posit that such patterns are inherently more adversarial for CLIP encoders. The surrogate encoders used in our Base, Mid, and Large search spaces were predominantly pre-trained on web-scale datasets sourced from platforms like Common Crawl?, However, datasets like LAION (Schuhmann et al., 2021; 2022) and DataComp (Gadre et al., 2023) lack detailed disclosures of their image distributions, and their massive scale makes comprehensive analysis difficult. We speculate that building-like patterns are particularly adversarial for deceiving CLIP encoders trained such datasets. We believe these findings open a promising avenue for future research, particularly in understanding the connection between pre-training data distributions and adversarial patterns.
hhtpss//commoncrawl.orgg
28

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
Non-targeted
Targeted: great white shark flying over a bridge.
Clean
Clean
the mountain range.
downtown Vancouver, Columbia, Canada.
mountains background.
LLaVA-7B: cityscape night large body
running through grassy shiins
background.
MiniBPP4:
river background.
serial of city cusk.
Adversarial
Adversarial
OF-3B: The Ruining of Avenge by Nikii Foley licensed under Greative
OF-38: large shark swimss ocean of skyscraper,
Commons Attribution.
LLavA-7B: tlying a city.
LLaVA-7B: book cover with a picture of a river and building.
skyllie
sky. shark flying
MiniGPT4: The cover the book the river of creams.
BLIP2: The cover of the book, the riding to the stars.
Non-targeted
Targeted: The Gocizilla is swimming in the harbour.
Clean
Clean
OF-38k fire falls from the sky Yosemite National Park, California. yel glow the mountain flowing down rocky
OF-38: Aurora Borealis, also known as the Northern Lights, is natural
light show.
MiniGPT4:A
surrounded by trees.
shines
northern lights the sky above house.
Norwey.
Adversarial
Raily Trouitie
Adversarial
LLaVA-7B:
sky building
OF-36: A giant Godzilla seen the background.
LLaVA-7B: cture of monster with green light behind it.
MiniGPT4: book cover with soien and the words the freedtom of
MiniEPT4: glant monster the water with city in the background.
aptitude
BLIP2: Godvills ou wellpaaper.
BLIP2: The cover art for the album by nika palley.
Non-targeted
Targeted: going destroy all of your belongings.
Clean
Clean
OF-38: The sun sets over the Summer Palace in Beijing, China.
OF-38: view the city of orddon from the Shard, the tallest building in Western Europe.
LLaVA-78: beaut fuul surset over lake with reflection of building
MiniGPT4: sunset view a building with lake in the foreground.
LLaVA-78: city skyline with a river running through it
BLIP2: The forbidden city sunset China.
MiniBPT4: view of the city from high vantage point
BLIP: An aerial view of ondon right.
Adversarial
Adversarial
OF-38:
DeVanee.
OF-3B: am all of those things.
LLaVA-78:
Ridin DeVance.
LLaVA-70: cityscape with bunch of luggage and begs.
MiniGPT4: painting a with words burning of spplenpiece
MiniBPT4: city seyllnn with a lot buildings and sign that says am petroleum.
BLIP2: The cover art for the album, the riffs with 4tth degrae.
BLIP2: am of destruction.
Figure 9. An illustration showing the application of hoth UAP (left) and TUAP (right) to an image. The responses from large VLMs are shown side side for the clean image (top) and the adversarially perturbed image (bottom).
29

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
(a) L bounded TUAPs.
(b) La-norm TUAPs.
Figure 10. From top to bottom, the visualisations correspond to different search space sizes (N = 16 in the first row, N = 32 in the second row, and = last row). Note that some TUAPs contain offensive sensitive patterns, which have been blurred for ethical considerations.
30

---

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks CLIP
D. X- BrTannsferBench
X-TransfeBBench is an open-source benchmark that provides a comprehensive collection of UAPs capable of achieving super adversarial transferability. These UAPs can simultaneously transfer across data, domains, models, and tasks. Essentially, they represent perturbations that can transform any sample into adversarial example, effective against different models and different tasks. The collection contains UAP and TUAPs used in experiments and all baseline UAP variants evaluated in Section 4. Additionally, we provide standardised evaluation scripts for the tasks assessed in our experiments. X- Tranh is designed to be easily extensible, allowing for the incorporation of future UAP/TUAP methods and new evaluation tasks. The super transferability makes it an ideal tool for efficiently assessing the robustness of CLIP encoders and large VLMs across diverse tasks and datasets. To the best of our knowledge, no similar open-source collections of UAPs currently exist, making this a valuable contribution to the community.
We provide PyTorch-like pseudo-code in Algorithm 2 for loading and perturbing samples. Using our UAP collection, one generate adversarial example with just 3 lines of code. Since these UAPs are "pre-trainedd, no additional optimisation is required, making X- TransferBench highly efficient and well-suited for large-scale adversarial robustness evaluations.
Algorithm 2 Using off-the-shelf UAP X- BeTransferBench.
import XTransferBench
import XTransferBench.cc 2oo
List threat models
print (WrannferBencch.ee wtw.sichht..ottttttttttetteeoeee ())
List UAPs under inf threat model
print (XTransfe:Bench.zo. zoo. wow.:emnchhooo.isstttttttttteee. ceetttacceerrinnnei ("iner"ottt____o_tattttttttt""""
Load X-Transfer with the Large search space (N=64) non-targeted
attacker haw.sfeerbennh..oooodddaaaataaae
"Linf lolff_noo____ taaif_non_ttrrree ttw_eonteeeeee,, 1tgpaarrr__arg____________papopo
Perturbe images to adversarial example
iimages Tensor [b, 3, h, w]
adv__mmges attacker (images)
While the X-Transfer standardised evaluations focus CLIP encoders and VLMs, the super transferability of UAPs suggests that they can be extended any data, model, and task. The modular design of our UAP collection (see Algorithm 2) ensures flexibility and makes it well-suited for adaptation to other data, models, and tasks in future research.
31


Reviews:
[{"review_id": "JnQ9KH394T", "reviewer": "Reviewer_UCtW", "summary": "This paper introduces an algorithm to find universal adversarial perturbations for CLIP-like image encoders. The vulnerability works across domains, tasks, and samples. The main algorithm follows standard methods for finding adversarial perturbations: finding a perturbation whose \\( L_{\\infty} \\) norm is smaller than a given \\( \\epsilon \\) while also optimizing the attack objective so that the CLIP encoding becomes similar to a text encoding for a certain malicious input text. To make this perturbation universal, the authors use multiple encoders.  \n\nThe main novelty of the attack lies in the efficient use of a large zoo of \\( N \\) CLIP-like image encoders for finding a universal perturbation. To this end, the authors use a reward-based strategy to select \\( k \\ll N \\) encoders at each optimization step of the attack. The experiments are conducted on several tasks, including zero-shot classification, image-text retrieval, image captioning, and VQA, demonstrating the versatility of the attack.  \n\nThe most interesting part of the paper is the finding that the vanilla version, without an ensemble of models (often a requirement for finding a universal perturbation), works quite well for classification and image-text retrieval tasks and actually outperforms several existing methods by large margins.  \n\nAnother interesting insight from the paper is the appearance of text-like artifacts on perturbed images.  \n\nOverall, this is an excellent paper (the kind of paper I would love to write). It is easy to understand, makes claims backed by ample empirical evidence, demonstrates the power of simpler methods, and has an excellent evaluation setup and experiment/analysis.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review, valuable feedback, and kind recognition of our work. Below are our responses to your questions.\n\n---\n\n**Q1:** The common assumption that an ensemble is necessary.\n\n**A1:** We agree that it is indeed surprising that the vanilla version of X-Transfer without an ensemble, performs comparably to several strong baselines. However, we believe that the ensemble mechanism is crucial for achieving super transferability, as it allows the method to exploit shared vulnerabilities across diverse surrogate models.\n\nRegarding the complex setups used in prior work, we believe that they may be well-justified for the specific task those studies focused on. However, for the broader goal of super transferability, including cross-task scenarios, we find that a simple and generic adversarial objective is not only sufficient but also necessary. Notably, our experiments (Appendix C.4) demonstrate that surrogate scaling does not improve transferability when applied to the baseline (ETU), further reinforcing the motivation behind our design choices.\n\n---\n\n**Q2:** Appearance of text features in perturbation\n\n**A2:** We would like to clarify that we did not use any explicit objective to introduce textual features or characters into the perturbation. Rather, these textual-like patterns **emerged naturally** as a byproduct of the optimisation process. The goal of X-Transfer is to exploit common vulnerabilities in CLIP encoders, and we believe our findings indeed demonstrate that these textual features reflect a shared vulnerability across CLIP variants. Understanding why such patterns emerge and how they interact with the multimodal representations of CLIP is, in our view, an interesting direction for future research.", "claims_and_evidence": "The claims in this paper are clear and supported by ample empirical evidence. The main claim is the introduction of an efficient \"super\" UAP that works across tasks, models, and domains. This is backed by comprehensive experiments.", "methods_and_evaluation_criteria": "The method and evaluation criteria presented in the paper align well with the problem of finding UAPs.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "I did not evaluate the experimental design by running any code. However, I compared the design with existing UAP methods and found it to be sound. Moreover, the results make sense, and the appearance of buildings and text on the input images indicates the validity of the experiments.", "relation_to_broader_scientific_literature": "The main contribution for me is its finding that non-ensemble-based UAP with standard methods work quite well. This finding is", "essential_references_not_discussed": "All essential references are discussed.", "other_strengths_and_weaknesses": "The paper is clear and presents a novel insight into the vulnerability of VLMs.", "other_comments_or_suggestions": "No.", "questions_for_authors": "The finding that the \"vanilla version without an ensemble works well\" has important implications. It suggests that a single-model attack can be surprisingly effective, challenging the common assumption that an ensemble is necessary for universal perturbations. Does this raise questions about whether previous attacks on CLIP were executed optimally or if they relied on unnecessarily complex setups? It may also indicate that CLIP-like models share inherent vulnerabilities that can be exploited more easily than previously thought.\n\nThe appearance of text on perturbed images is an interesting phenomenon. CLIP has been shown to be deceived by directly imposing text on input images, so this could suggest that the perturbations exploit similar weaknesses in the model’s reliance on textual features within visual inputs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3txbqofOvQ", "reviewer": "Reviewer_wapn", "summary": "This paper introduces X-Transfer, a novel adversarial attack method that generates universal adversarial perturbations (UAPs) with \"super transferability\" across data, domains, models, and tasks for CLIP-based vision-language models. The core innovation is an efficient surrogate scaling strategy that dynamically selects a subset of surrogate CLIP encoders from a large search space using a multi-armed bandit (MAB) framework with Upper Confidence Bound (UCB) sampling. Extensive experiments demonstrate that X-Transfer outperforms state-of-the-art UAP methods, achieving higher attack success rates (ASR) on zero-shot classification, image-text retrieval, image captioning, and VQA tasks. The authors also release X-TransferBench, a comprehensive benchmark of UAPs for evaluating super transferability.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and insightful comments. Please find our responses to your questions below.\n\n**Q1:** Mechanism behind the cross-task transferability on VLM\n\n**A1:** X-Transfer exploits a common weakness in CLIP image encoders, even when they are trained on different datasets and model architectures. Our adversarial objective is intentionally generic as it induces **meaningless embeddings**. Since many VLMs adopt CLIP or its variants as their image encoders, the UAPs generated by X-Transfer can therefore induce similar **nonsensical embeddings** in these models. This highlights a shared vulnerability across a wide range of CLIP encoders.\n\n**Q2:** Theoretical analysis\n\n**A2:** The focus of this work is super transferability, not MAB or UCB regret analysis.\n\nWe believe that super transferability is a novel topic, and as such, there is currently no established theoretical framework. However, our empirical results are solid and comprehensive, providing strong support for our claims on super transferability. The geometry-based theoretical analysis of transferability between classifiers (Tramèr et al., 2017), as well as the ACL23 work mentioned by the reviewer, could serve as valuable starting points for future theoretical investigations into super transferability.\n\n**Q3:** X-TransferBench and supplementary material\n\n**A3:** The sample code is included in the accompanying ZIP file. We will adopt the MIT License in the published version. Please refer to Figure 2(a) for the correlation with ASR.\n\n**Q4:** Choice sampling method and generic adversarial objective\n\n**A4:** Our main contribution lies in the MAB-based dynamic selection framework and the design of a reward metric. The sampling method itself is not the primary factor in achieving super transferability. It is the reward metric that plays a critical role (see Section 4.2). In Section 3.3, we stated that while UCB is our default choice due to its simplicity, we also emphasised that other sampling strategies (including Thompson sampling) are plausible and fully compatible with our framework.\n\nThe generic objective (Eq. 3-5) is essential to ensure compatibility across different architectures, embedding sizes, and pre-training objectives of surrogate models. While the objectives may appear simple, they are crucial for achieving super transferability, see Appendix C.4 and Table 11.\n\n**Q5:** New experiments (TUAP, step size, other and newer models, randomised smoothing, and search space)\n\n**A5:** \n\n(1) We tested 5 additional TUAPs (average ASR 77.6%) with targets sampled from the AdvBench and compared their performance with the 10 TUAPs (average ASR 75.8%). The results are consistent.\n\n(2) We follow existing works and set the $\\epsilon$ to 12/255. The step size is a typo. It is our choice of hyperparameters, not baselines. This does not affect our analyses since $\\epsilon$ is the factor in ensuring the fair comparison for $L_\\infty$ perturbations.\n\n(3) Please find the results below for ALIGN, Florence and newer models. Note MetaCLIP-v1.2 ViT-H/14 was released in Dec 2024, and SigLIP-v2 ViT-B-16 was released in Feb 2025. \n\n| Model | Task | ETU | C-GPC | Base | Mid | Large |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| ALIGN | ZS | 57.1 | 52.9 | 68.7 | 65.7 | **70.6** |\n|  | IR | 45.3 | 38.7 | 56.7 | 59.8 | **63.8** |\n|  | TR | 52.9 | 49.6 | 66.5 | 66.5 | **71.0** |\n| MetaCLIP-v1.2 | ZS | 36.0 | 33.9 | 61.0 | 64.1 | **70.5** |\n|  | IR | 15.2 | 20.3 | 40.5 | 43.4 | **48.4** |\n|  | TR | 27.7 | 29.6 | 48.8 | 55.0 | **61.9** |\n| SigLIP-v2 | ZS | 55.3 | 49.0 | 66.7 | 69.3 | **72.1** |\n|  | IR | 27.8 | 34.2 | 51.4 | 56.2 | **59.1** |\n|  | TR | 49.4 | 46.3 | 59.7 | 65.3 | **67.4** |\n| Florence-v2 | COCO-IC | 24.4 | 15.2 | 29.4 | 29.7 | **33.0** |\n|  | Flicker-30k-IC | 25.4 | 20.5 | 28.5 | 28.5 | **31.2** |\n\nWhile we cannot guarantee complete effectiveness against newer models, we believe our work offers novel insights into CLIP/VLM vulnerabilities and makes valuable contributions to the community. These findings provide important foundations for developing safer, more robust models in the future.\n\n(4) We tested our UAP against the smoothed ImageNet classifier provided by Cohen et al., 2019. As expected, the classifier demonstrates robustness to $L_\\infty$ and $L_2$ perturbations, consistent with its certified guarantees. However, it is not robust to our patch-based UAP, as these perturbations lie outside the certified radius. This finding highlights an important gap and, we believe, serves as a strong motivation for future works.\n\n(5) We evaluated the generalisation capability of our method by testing with a ViT-only base search space (average ASR of 69.3%) and comparing it to the original mixed-architecture base search space (average ASR of 69.2%). The results show no significant difference. \n\n**Q6:** Others\n\n**A6:** We will fix the typo, add more examples of Fig 3, and add references to related works mentioned by the reviewer.", "claims_and_evidence": "- Claim 1: X-Transfer achieves \"super transferability\" (simultaneous cross-data/domain/model/task transferability).  \n  - Evidence: Supported by experiments across 12 datasets, 9 CLIP encoders, and 4 VLMs (Table 1-2). However, cross-task transferability (e.g., attacking VLMs trained with autoregressive objectives) lacks mechanistic explanation.  \n\n- Claim 2: The dynamic surrogate selection strategy reduces computational costs while improving transferability.  \n  - Evidence: Figure 2(a) shows X-Transfer with $k=1$ matches standard scaling ($k=N$), but theoretical guarantees for UCB-based selection (e.g., regret bounds) are missing.  \n\n- Claim 3: X-TransferBench provides a practical resource for adversarial robustness evaluation.  \n  - Evidence: The benchmark is described in Appendix D, but no user studies or community adoption examples are provided.", "methods_and_evaluation_criteria": "- Methods:  \n  - The MAB-based dynamic selection is novel and addresses scalability limitations of fixed ensembles. However, the choice of UCB over other bandit algorithms (e.g., Thompson sampling) is not justified.  \n  - The adversarial objective (Eq. 3-5) is generic but lacks architectural/task-specific adaptations (e.g., for VLM autoregressive losses).  \n\n- Evaluation Criteria:  \n  - ASR is task-specific (e.g., accuracy drop for classification, CIDEr for captioning), which is reasonable. However, targeted attacks (TUAPs) are only evaluated on a limited set of 10 manually designed text prompts (Appendix C.7), raising concerns about generalizability.", "theoretical_claims": "- The paper lacks theoretical analysis. For example:  \n  - No proof of convergence for the UCB-based selection strategy.  \n  - No formal analysis of why ViT-based surrogates dominate selection (Figure 2(b)) or how embedding space geometry enables cross-task transfer.", "experimental_designs_or_analyses": "- Strengths:  \n  - Broad evaluation across CLIP variants (ViT, ResNet, SigLIP) and VLMs (LLaVA, MiniGPT-4).  \n  - Ablation studies on surrogate scaling (Figure 2(a)) and perturbation types ($L_2$, patch).  \n\n- Weaknesses:  \n  - \"Following Fang et al. (2024b); Zhang et al. (2024), we employ L∞-norm bounded perturbations with ϵ = 12/255, and the step size η of 0.5/255.\" I did find the description of ϵ = 12/255 in the two papers, but I did not find the description of step size η of 0.5/255. Please provide more detailed sources for the setting of step size.\n  - The search space (Tables 8-10) includes only CLIP variants, excluding non-CLIP vision-language models (e.g., ALIGN, Florence).  \n  - Results on adversarial training (Appendix C.8) are superficial; no defense strategies (e.g., randomized smoothing) are discussed.", "relation_to_broader_scientific_literature": "- Extends prior work on UAPs (Moosavi-Dezfooli et al., 2017) and CLIP adversarial attacks (Zhou et al., 2023; Zhang et al., 2024) by addressing multi-dimensional transferability.  \n- Connects to bandit algorithms (Auer, 2002) but does not leverage recent advances in contextual bandits for non-stationary environments.", "essential_references_not_discussed": "- Multi-modal adversarial attacks: Concurrent work on GPT-4V/DALL-E 3 adversarial attacks (e.g., \"ImgTrojan: Jailbreaking Vision-Language Models with ONE Image\" is not cited.  \n- Theoretical analysis of UAPs: The paper does not discuss recent theoretical frameworks for UAP transferability (e.g., \"On the Universal Adversarial Perturbations for Efficient Data-Free Robustness Evaluation\" (ACL 2023)).", "other_strengths_and_weaknesses": "- Strengths:  \n  - Originality: First to formalize \"super transferability\" and propose a scalable solution.  \n  - Practical Impact: X-TransferBench fills a gap in standardized UAP evaluation.  \n\n- Weaknesses:  \n  - Clarity: Equation symbols (e.g., $\\mathbb{D}$ vs. $\\mathbb{D}$) are inconsistently defined.  \n  - Significance: While results on existing CLIP/VLMs are strong, applicability to newer models is unclear.", "other_comments_or_suggestions": "Please provide more examples similar to Figure 3.", "questions_for_authors": "1. Theoretical Justification: Can you provide regret bounds or convergence guarantees for the UCB-based surrogate selection strategy? *A theoretical analysis would strengthen the method’s credibility.*  \n2. Search Space Generalization: How does X-Transfer perform if the search space contains only ViT or ResNet architectures? *This would test the robustness of dynamic selection to model homogeneity.*  \n3. Defense Evaluation: Have you evaluated X-Transfer against state-of-the-art defenses (e.g., randomized smoothing or feature denoising)? *This would clarify the practical threat model.*", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5pqKc2hROa", "reviewer": "Reviewer_GY9n", "summary": "This paper reveals a universal adversarial vulnerability in CLIP models, where a single perturbation achieves super-transferability across datasets, domains, models, and tasks. The authors find that proxy encoder selection, rather than the dataset, is the key factor. They propose X-Transfer, a novel attack that surpasses prior UAP methods in transferability, establishing a new benchmark. Additionally, they introduce X-TransferBench, an open-source evaluation framework for CLIP and VLM robustness. The core innovation lies in an efficient proxy scaling strategy, leveraging UCB sampling to optimize encoder selection, enhance perturbation generalization, and achieve super-transferability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and careful review of our work. Below, we provide detailed responses to address each of your concerns.\n\n---\n\n**Q1:** Using OpenCLIP lacks generalisability and real-world models.\n\n**A1:** \n\n(1) OpenCLIP is an open-source framework, protocol, and API for CLIP and its variants, not a model itself. It currently hosts the largest collection of open-source CLIP models, which we adopt in our experiments primarily for its convenient and unified interface. Many real-world CLIP models are already included in the OpenCLIP collection, such as MetaCLIP from Meta and SigLIP from Google, both of which are evaluated in our experiments.\n\n(2) To address your concern regarding real-world deployment, we conducted evaluations on GPT-4o and OpenAI o1. Using the same targeted adversarial example generated via TUAP (as shown in Figure 3), we prompted the models with *“Briefly describe the image.”* Below are the models’ responses.\n\n**GPT4-o**: *The image is a digital collage blending a volcanic eruption with cyber elements. Lava flows into the ocean under a blue-toned overlay of circuit patterns and glowing text related to computer viruses, such as “Remove Files” and “Virus Inject.” It visually represents the destructive power of cyber threats.*\n\n**OpenAI o1**: *It’s a composite image showing lava pouring into an ocean, overlaid with circuit-board patterns and words referencing computer viruses and file removal, evoking the idea of a powerful, destructive cyber threat.*\n\nWe also conducted a qualitative evaluation by randomly sampling 1,000 images from the ImageNet test set and applying our TUAP to generate adversarial examples. We queried the model using the prompt “Briefly describe the image.” For this evaluation, we used OpenAI’s GPT-4o-mini model. Each request included both the textual prompt and the adversarial image (encoded in base64), and the maximum response length was set to 128 tokens. If the response contained any of the keywords in target text descriptions (e.g. *“remove”, “files”, “computer”, “inject”, “computer virus”*), we considered it a successful attack. The ASR based on this criterion is reported in the table below.\n\n| Base | Mid | Large |\n|---|---|---|\n| 53.6 | 65.8 | 70.0 |\n\n---\n\n**Q2:** Ablation on hyperparameters\n\n**A2:**\nThe search space size, denoted as $N$, is evaluated throughout the paper, and the number of selected encoders per iteration, $k$, is specifically analysed in Section 4.2.\n\nResults for different search space sizes, Vanilla ($N$ = 1), Base ($N$ = 16), Mid ($N$ = 32), and Large ($N$ = 64), are presented in Tables 1, 2, 13, 15, and 16. These results demonstrate that a larger search space consistently improves the ASR.\n\nThe ablation study on $k$ is shown in Figure 2(a), and the efficiency analysis is provided in Table 7. The results indicate that while varying $k$ has minimal impact on ASR, it primarily affects computational efficiency. Our choice of $k$ is 25% of $N$ is motivated by the trade-off observed in Figure 2(a) and the efficiency analysis in Table 7.\n\n---\n\n**Q3:** Theoretical explanations\n\n**A3:**\nWe believe that super transferability and CLIP’s concept fusion ability are novel topics of growing importance. As such, there are currently no established theoretical frameworks to formally characterise these phenomena. Nevertheless, our empirical results are comprehensive, providing strong support for our claims regarding super transferability. The novel observation that UAPs on CLIP exhibit semantically interpretable patterns offers valuable insight into the underlying vulnerabilities of CLIP models. We plan to investigate theoretical explanations in our future work.\n\n---\n\n**Q4:** Defences against X-Transfer\n\n**A4:** \nPlease refer to Appendix C.8, where we have presented evaluations using adversarially trained CLIP encoders.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "(1) The study primarily relies on OpenCLIP, which, while widely used, may not fully capture the diversity of CLIP models in real-world applications. Differences in architecture, training, and optimizations across implementations could impact X-Transfer’s effectiveness, limiting the generalizability of the findings.\n(2) Although the evaluation covers various tasks and datasets, it lacks real-world complexity. Practical deployment factors—such as data formats, model updates, distribution shifts, and hardware constraints—may affect X-Transfer’s performance.", "experimental_designs_or_analyses": "(1) Key hyperparameters, such as the proxy encoder search space size and the number of selected encoders per iteration, are empirically set without rigorous theoretical justification or systematic validation. \n(2) The study relies solely on OpenCLIP, which may not fully capture the diversity of real-world CLIP deployments.", "relation_to_broader_scientific_literature": "(1) CLIP’s contrastive learning enables strong zero-shot generalization, driving research into its applications and vulnerabilities.\n(2) While UAPs have been explored for CLIP, prior methods struggle with super-transferability across datasets, domains, models, and tasks. X-Transfer fills this gap, setting a new benchmark for UAP effectiveness in adversarial attacks on CLIP.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "(1) X-Transfer is empirically validated, but key theoretical explanations remain lacking. The dominance of ViT-based encoders in proxy selection is observed but not deeply analyzed, and the emergence of semantic patterns in UAPs is attributed to CLIP’s concept fusion ability without solid theoretical grounding.\n(2) Despite extensive experiments, real-world deployment factors—such as data formats and dynamic model environments—are not fully considered.\n(3) Key hyperparameters, including search space size and proxy encoder selection, are determined heuristically without rigorous analysis. Relying solely on empirical tuning limits the method’s generalizability and scalability.", "other_comments_or_suggestions": "Developing defenses against X-Transfer is essential. It should explore robust training, architectural modifications, and specialized defenses while balancing security and model performance.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qeD42QRzBy", "reviewer": "Reviewer_HNj3", "summary": "With wide-spread deployment of CLIP image- and text-encoding for use in downstream tasks, CLIP has been revealed as a useful attack surface for subverting inference in the downstream model. A natural question is how to craft an effective perturbation once by way of ensemble methods which then subverts many models given any data point (i.e., universal adversarial perturbation (UAP)). The authors extend this idea to the realm of subverting many models, tasks, datasets, and domains, and describe such transferability as so-called super transferability taking form as super universal adversarial perturbations (SUAP). Previously, UAP methods require a fixed set of models, heuristics, and constraints to optimize a UAP with desired effect. The authors argue that this becomes intractable when tackling the problem of super-transferability, and instead propose to use heuristics agnostic to the surrogate model. In practice, the authors use the heuristic which simply minimizes (or maximizes) the similarity in CLIP embedding space of untargeted (or targeted) evasion attack. This heuristic works since all downstream tasks will exploit the contrastive nature of CLIP embeddings in some way. To scale up to many surrogate models, the authors finally propose to adopt upper confidence bounds (UCB) from the well-studied multi-armed bandits framework to avoid optimizing with respect to all surrogate models at once, instead letting the UCB-based algorithm (X-Transfer) guide the sampling of surrogates. Specifically, the authors cast the UAP optimization over surrogate models as a non-stationary multi-armed bandits problem, since the reward distribution for different surrogate models may shift as optimization progresses.\n\n## Post-rebuttal update\n\nThanks to the authors for their response. They will incorporate some of the changes mentioned in the original review, and have provided some interpretations of the submission's pain points. I will keep my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your careful review of our paper. Please find our detailed responses to your questions below.\n\n---\n\n**Q1:** Choice of sampling strategies and improvement over random sampling\n\n**A1:**\nWe would like to clarify and emphasize the comparison of sampling strategies presented in Section 4.2. The results are reported as macro-averages across 9 encoders, each evaluated on 8 datasets, yielding a total of 72 evaluation points. We argue that the observed 2.3% performance gap is statistically significant, particularly in the context of large-scale benchmarking.\n\nOur analysis demonstrates that reward-aware sampling strategies (UCB and $\\epsilon$-greedy) consistently outperform reward-agnostic random sampling. This empirical evidence strongly suggests that the reward metric itself - rather than the sampling strategy choice - is the primary driver of super-transferability improvements. While we acknowledge that alternative sampling approaches may be worth exploring, our core contributions remain: (1) the efficient surrogate scaling framework and (2) the proposed reward metric.\n\nTo summarise:\n- Super-transferability stems fundamentally from surrogate scaling, as evidenced by experiments with varying search space sizes ($N$).\n- While sub-sampling improves computational efficiency (Fig 2a), it is not the primary driver of super transferability.\n- Reward-aware sampling consistently outperforms random sampling while maintaining efficiency.\n\n---\n\n**Q2:** The dynamic reward distribution\n\n**A2:**\nWe would like to clarify that our current approach already incorporates a dynamic reward distribution that evolves throughout the adversary’s training loop. As the perturbation improves over time, it influences the loss values used as the reward signal, leading to a naturally shifting reward landscape.\n\nWe also agree with the reviewer that exploring an adaptive threat model, in which CLIP itself is updated during the adversarial process, presents an interesting direction for future research. We appreciate this insightful and constructive suggestion and will certainly pursue it in future work.\n\n---\n\n**Q3:** Iterative improvement and contribution\n\n**A3:**\nWhile the adversarial objective we employ may appear simple, its design is both intentional and necessary, and its combination with UCB may seem like an incremental improvement. However, we emphasize that this formulation has led to insightful and novel findings regarding the vulnerabilities of CLIP and VLMs. The simplicity and generality of the adversarial objective are deliberate design choices, as they ensure compatibility across diverse CLIP encoders and are crucial for achieving super transferability, as demonstrated in Appendix C.4. This deliberate simplicity underscores the elegance and effectiveness of our approach.\n\nBeyond performance, our work reveals a new and exciting phenomenon: the untargeted UAPs generated by X-Transfer are semantically interpretable, yet they do not align with human perception. Prior work has suggested that semantically interpretable perturbations that align with human understanding typically arise only from targeted objectives, whereas untargeted perturbations appear patternless. In contrast, our findings show that untargeted UAPs can also produce visually coherent patterns that are semantically rich but misaligned with human interpretation, suggesting that these perturbations explore a space distinct from all previously observed perturbation behaviors.\n\nWe believe that a simple and effective method, paired with a novel and well-supported finding, and validated through rigorous experimentation, constitutes a meaningful and impactful contribution to the field.\n\n---\n**R4:** Others suggestions\n\n**A4:** We will incorporate these suggestions in the revision.", "claims_and_evidence": "- The authors suggest that by efficient selection of surrogate models during ensemble optimization, the adversary may learn a super UAP which transfers to many tasks, models, datasets, and modalities. The main claim is checked through empirical studies starting in Section 4 for zero-shot classification (Table 1) and captioning/VQA (Table 2). In these tables it can be observed that X-Transfer performance improves as a larger set of surrogates is selected (e.g., going from Vanilla to Large offers reliable improvement of ASR). The effect of UCB selection is ablated in Figure 2a showing that (1) more surrogates leads to higher ASR, and (2) selection guided by UCB offers a compute reduction while matching the results of higher $k$. \n- The evaluation suggests that UCB-based X-Transfer may reduce the computational cost of SUAP. However, it is also shown that both random sampling and $\\epsilon$-greedy sampling strategies have comparable performance, hence it might be difficult to motivate the extra complexity of using UCB over simple random sampling. The main takeaway for this result is that sub-sampling of surrogate models is important to achieve SUAP. As a further takeaway, it could be argued that since UCB does not have a large gap compared to random sampling, the reward distribution of the surrogate models may be more trivial (i.e., stationary) than expected. It could be that in the case of dynamic surrogates (i.e., updated in real time against the adversary) the reward distribution becomes more suitable for UCB.", "methods_and_evaluation_criteria": "- The methods for evaluating each task (e.g., CIDEr for image captioning) generally match the metric used to score the task in the respective literature. Attack success rate is used for all tables which is the standard for adversarial ML studies.\n- The benchmark datasets and associated models are reasonable since they are fairly recent approaches.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The experiments use a variety of established baselines such as C-PGC, ETU, and AdvCLIP for CLIP-specific attacks, and also attacks meant as general purpose UAP, such as GD-UAP, TRM-UAP, and Meta-UAP. An ablation is performed which is the same attack setup of X-Transfer but without the UCB-guided surrogate selection. These baselines are reasonable due to being relatively recent but also relevant to the goal (attacking CLIP models). \n- Cross-data experiments operate on standard benchmark datasets which vary in scale, fidelity, and purpose. Cross-model experiments use a mix of both ResNet- and transformer-backed CLIP models which isolates causes due to model architecture. Other more recent methods are checked, such as MetaCLIP, meanwhile some VLMs are selected to study transferability on multi-modal LLM inference. In this latter case the authors do not launch X-Transfer on the respective VLM finetune of CLIP, which seems reasonable.", "relation_to_broader_scientific_literature": "- The authors plan to release a large collection of UAPs and TUAPs to the broader community, which may be useful for checking future defenses on a static target. \n- The authors show that subsampling of the surrogate ensemble is a simple but effective way of improving UAPs. A variety of baseline methods and surrogate models are checked which will serve as useful reported baselines in future studies.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- Since the ablation shows that UCB is marginally better than random sampling for the fixed CLIP models, it might be interesting to check the reward distribution of CLIP models which update dynamically against the adversary (hence changing the reward distribution over course of adversary's training loop). It could be that UCB becomes more useful on an adaptive defender threat model, where the CLIP model is updated on-the-fly during adversary's optimization loop (to mimic a defender's inner update loop). \n- The writing is generally high quality and easy to follow for someone familiar with the prior work.\n- The approach could be considered an iterative improvement over prior work since it mainly combines UCB with known objectives and optimization algorithm. I would consider the primary contribution to be the ablations and empirical results studying ensemble sampling behavior.", "other_comments_or_suggestions": "* Equation 1 - the text should connect $k$ and $j$ to the contrasting of image with texts and the reverse for clarity. \n* Equation 4 - $t_{adv}$ should be clarified to be a sample of interest.\n* L179-180 - encodes -> encoders", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes a novel attack to generate universal adversarial perturbations against various CLIP encoders and downstream VLMs across different samples, tasks, and domains. The main novelty stems from an efficient surrogate scaling strategy that dynamically selects suitable surrogates. The experiments validate the effectiveness on different datasets and domains, outperforming sota baselines. Such a task and the results are meaningful and important in the community since CLIP has been widely used in diverse models and scenarios. Another interesting finding was noticed by the Reviewer UCtW: the vanilla version, without an ensemble of models (often a requirement for finding a universal perturbation), works quite well for classification and image-text retrieval tasks and actually outperforms several existing methods by large margins. Two reviewers rate the paper "strong accept" and agree that this work is excellent, with interesting findings, originality, and practical impacts, while one of the reviewers raises concerns about the lack of theoretical analysis. Another reviewer agrees that this work is well written and the primary contribution is the ablations and empirical results studying ensemble sampling behavior, tending to accept this work. One reviewer thinks the proposed method is a highly transferable universal adversarial attack on CLIP models, demonstrating its super-transferability. However, the reviewer highlights concerns about limited generalizability beyond OpenCLIP, heuristic-driven hyperparameters, and a lack of theoretical grounding, recommending further theoretical analysis and real-world validation. After reviewing the paper and reviewers' comments, I believe the first two concerns have been addressed: 1. OpenCLIP contains a wide range of CLIP models, and some of the models have been used in real applications. 2. The abolition studies have validated the influence of hyperparameters. Based on the above analysis and findings, I recommend accepting this work.
Decision: Accept (poster)