Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Authors: Elias Abad Rocamora
Christian Schlarmann
Naman Deep Singh
Yongtao Wu
Matthias Hein
Volkan Cevher
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Adversarial Robustness
Robust Text Encoders
CLIP
Unsupervised Adversarial Finetuning
Primary area: general_machine_learning
TL;DR: We propose an efficient strategy for adversarial finetuning of the CLIP text encoder, enabling robustness in zero-shot classification, text-to-image retrieval and text-to-image generation.

Abstract: Adversarial input attacks can cause a significant shift of CLIP embeddings. This can affect the downstream robustness of models incorporating CLIP in the pipeline, such as text-to-image generative models or large vision language models. While some efforts have been done towards making the CLIP image encoders robust, the robustness of text encoders remains unexplored. In this work, we cover this gap in the literature. We propose LEAF: an efficient adversarial finetuning method for the text domain, with the ability to scale to large CLIP models. Our models significantly improve the zero-shot adversarial accuracy in the text domain, while maintaining the vision performance provided by robust image encoders. When combined with text-to-image diffusion models, we can improve the generation quality under adversarial noise. In multimodal retrieval tasks, LEAF improves the recall under adversarial noise over standard CLIP models. Finally, we show that robust text encoders facilitate better reconstruction of input text from its embedding via direct optimization. We open-source our code and models.

Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Elias Abad Rocamora EPFL, Christian Schlarmann
Naman Deep Singh
Yongtao Wu EPFL, Matthias Hein 1
Volkan Cevher
EPFL: LIONS École Polytechnique Fédérale de Lausanne, Switzerland
iA: Tübingen Al center, University of Tübingen, Germany (name. surname) @ (epff1.ch, umi-tuebingen.de)
Abstract
Adversarial input attacks can cause a significant shift of CLIP embeddings. This can affect the downstream robustness of models incorporating CLIP in the pipeline, such as text-to-image generative models or large vision language mod- els. While some efforts have been done towards making the CLIP image coders robust, the robustness of text encoders remains unexplored. we cover this gap in the literature. We propose LEAF: efficient adversarial fine- tuning method for the text domain, with the ability to scale to large CLIP models. Our models significantly improve the zero-shot adversarial accuracy in the text domain, while maintaining the vision performance provided by robust image en- coders. When combined with text-to-image diffusion models, generation quality under adversarial noise. In multimodal retrieval tasks, LEAF improves the recall under adversarial noise over standard CLIP models. Finally, we show that robust text encoders facilitate hetter reconstruction of input text from embedding via direct optimization. We open-source our code and models
this work,
improve the
Introduction
Contrastive Language-Image Pretraining (CLIP) models embed images and captions into shared
embedding space [Radford al., 2021 CLIP simple but rather powerful tool for vision-
language understanding, being employed in wide range of multimodal tasks such retrieval [Fang
al., 202T Koukounas et al. 2024 Vendrow et al.) 2024), Large Multimodal Models (LMMss
2022 Liu et al. 2023 and text-to-image generative models [Ramesh et al. 2021 Rombach et al.
[Alayrac et al.
2022 Ramesh et al. 2022 Podell et al. 2024].
However, the simplicity of CLIP and its plug-and-play usage becomes a double-edged sword, allow- ing adversarial attacks to optimized over CLIP, and transferred the downstream task of interest
(Zhuang et al., 2023, Ghazanfari et al. 2023 2024 Croce et al., 2025 Recently, making the image encoder of CLIP robust has gained interest Mao al. 2023]. making LMMs robust to adversarial perturbations by replacing the image encoder with an ad adversarially finetuned one [Schlarmann et al.) 2024). Nevertheless, adversarial finetuning has not been yet investigated for the text encoder.
In this work, we fill this gap by studying adversarial finetuning for CLIP text encoders, proposing Levenshtein Efficient Adversarial Finetuning (LEAF). Motivated by recent advancements in the im- age domain, we optimize the same objective as Schlarmann et al. [2024). allowing us to replace the text encoder in tasks like text-to-image generation, without needing finetune the of the pipeline. Moreover, to make adversarial finetuning faster in the text domain, we propose an attack that can be parallelized within training batches, accelerating the approach of Abad Rocamora et (224) by an order of magnitude with very little loss of performance.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Perturbed imputs
Encoders
Embedding
Zero-shot Adversarial Accuracy
65
CLIP
NG-News Adv. (k= 1)
60
x
55
big barly bear big burly hear
Better
griedly
CLIP
grillzly
VX
This
big
barly
grizzly
bear
work
20
ImageNet Adv. 2/255)
Figure 1: Left: our idea. Schlarmann al. 2024 propose FARE: finetuning the CLIP image coder to produce embeddings close to the clean image embedding (*) under image perturbations. Analogously, finetune the CLIP text encoder to produce embeddings close the clean text em- bedding (*) under lex/ perturbations. Right: results in ViT-L/14. The first (second) X/V denotes the usage of robust image (text) encoder. We constrain the text attacks with the Levenshtein dis- tance and the image attacks in the for norm. By combining the FARE robust image encoder with our robust text encoder, obtain high adversarial accuracy in both domains.
Our models, LEAF, able improve the zero-shot adversarial accuracy of CLIP models from 44.5% to 63.3% in AG-News at distance k = 1 (one character change). When plugged into Sta- ble Diffusion [Rombach al. 2022 Podell et al. 2024], character-level perturbations. For retrieval tasks, our models achieve recall 10 points higher on average than non-robust CLIP models at A = 2. Moreover, when inverting the embeddings of text encoders through direct optimization, we show that with LEAF models, we can recover a higher percentage of the original sentence. This results in LEAF encoders being more interpretable.
achieve higher quality images under
Overall, we show the robustness of CLIP text encoders can be improved with minimal effects on the clean performance in several tasks. We believe our robust CLIP models make future models incorporating CLIP more robust and interpretable. Our code and models can be found in and
githb..om/LLIONS-ERI
pes.eeAAAAcCL.iipeciiiiilllll respectively.
Notation: We use uppercase bold letters for matrices X RRmxn, lowercase bold letters for vectors E mh and lowercase letters for numbers x TR. Accordingly, the 7th row and the element in the i, position of matrix X are given by x; and Zij respectively. We use the operator
for have [X
the size of sets, e.g., [S(T') and the length of sequences, e.g., for X guxx
For two vectors u, u E Ra we denote the cosine similarity as sim(u, v) shorthand [n] {0,
We use the u TTv
1} for any natural number
Background
In Section 2.1
we cover the approaches improving the adversarial robustness of CLIP. In Section 2.2 diseuss robustness in the text domain.
2.1 Robustness of CLIP
0} be the space of sequences of characters in Let S(T) fc102... Cm : G; I Vm E N N
the alphabet set P. We represent sentences S S(T') sequences of one-hot vectors, i.e., S E (0. 10.1)mm m ||s:|1 1, Vi E [m]. Similarly, can represent images with d pixels as real vectors E Rdd. Overall, the training dataset is composed of n text-image pairs {S,, 2s
The objective of CLIP to learn a text encoder fo S(T) + R and an image encoder Iw Rdd - R where h is the embedding size and 0 and w are the parameters of the text and image encoders respectively. Radford et al. 2021 propose maximize the cosine similarity of positive

---

sentence-image pairs relative to the cosine similarity with other sentences and images in the dataset. We denote the weights obtained after pretraining with CLIP as cCLPP and WCLIP-
In order to make the image encoder Iw robust in the zero-shot classification task, Mao et al. 2023
use the sentences S "a photo of ,, vjj [o], where is the number of classes. Then, given a dataset of images and labels ( Di R==1 so that yi E [o]. Mao al. 2023]
optimize:
eouu (St,) 9w (.)+
min
max 16.11
- log
Σラ=1 Jo (5,) (m)+))))))
(TeCoA)
TeCoA significantly improves the robustness of the image encoder. However, it generalizes poorly image classification tasks that are not part of the fine-tuning dataset, and degrades the performance when employed in an LMM pipeline, as shown by |Schlarmann et al. [2024]. In order to overcome
this, |Schlarmann al. (2024] propose FARE, which intends preserve the original image embed- dings while being robust. To do so, they optimize:
min
1 (z:) Iw (x; 5.)
(FARE)
18.11 Se
The FARE objective allows to employ the obtained image encoder within an LMM pipeline with minimal clean performance degradation. Motivated by these findings, in this work construct similar loss in the text domain (Eq. (TextFARE) and adapt the algorithm to the challenges of this new domain (LEAF). See Fig. I for visualization of the FARE and LEAF approaches.
2.2 Robustness in the text domain
Belinkov and Bisk 2018 Alzantot et al. 2018 showed that text classifiers are not robust natural or adversarial noise, with text adversarial attacks being used in Large I Language Models (Zou et al.) 2023) and text-to-image generative models (Zhang et al. 2025]. Generally, given a sentence S, model f and some loss function L. the adversarial attack problem can be formulated as:
max ((f(S)) S'EN(s)
where N/S)) is a set of neighboring sentences, i.c. the threat model. great challenge in the text domain is defining a valid threat model, as the semantics of the sentence S should be preserved according the task [Morris et al. 2020]. In the literature, we can categorize adversarial attacks
into two main threat models: token and character level attacks. With token level attacks set to
replace/insertddeltte a small number of tokens in the sentence [Ren al.. 2019 Jin al.) [0200] Li et al. [219 Garg and Ramakrishnan [0200] Lee et al. [022] Ebrahimi et al. 2018 Li et al. 2020]
Guo et al.] 2021 Hou et al., 2022]. Similarly, character-level attacks reppaee/inser//deltte small
number of characters in the sentence [Belinkov and Bisk 2018 Ebrahimi et al.] 2018 Gao et al.)
2018 Pruthi et al.) 2019, Yang et al.] 2020) Liu et al.) 2022, Abad Rocamora et al.. 2024]. Both
approaches can be thought of as keeping small Levenshtein distance [Levenshtein] 1966 between
the original and attacked sentences in the token or character-level.
Semantic constraints: To ensure that semantics are preserved, token-level attacks usually con- strain (S)) further by only allowing token replacements between tokens with high similarity in the embedding space JJn et al.) [2020]. But, even with such semantic constraints, several works have pointed out that token level attacks do not preserve semantics [Morris et al., 2020 Dyrmishi et al.) 2023), with Hou et al. 2023) reporting 56.5% of their attacks change the semantics of the sentence. Due to the difficulty preserving semantics, we focus on character-level attacks in this work.
In the case of the character-level attacks, to further preserve semantics and simulate natural typos, some works constrain the attack only replace characters that are nearby in the English keyboard [Belinkov and Bisk, [2018) Huang et al. [019]] Others do not allow the attack to modify the first perturb short words, to perturb the same word twice cial characters [Pruthi et al. 2019 Jones et al.) [020]. In the context of text-to-image generation, Chanakya al. [2024] find that changing one character in the sentence can change one word for another and the text-to- image model accordingly generates different object in the image. To avoid this, [hanakya et al. 2024) introduce the semantic constraint of not allowing new English words to appear after the attack. In this work, we decide to adopt the semantic constraints of [Chanakyal jet al. 2024) and find they are especially useful when performing adversarial finetuning the CL.IP text encoders, see Section 4.2.2
and last letter of words,
insert spe-
3

---

1. Select position
Final perturbation: "Never Gonna G?ve You Up"
Figure 2: Schematic and example of the attack used in LEAF: In the first step, we randomly select P 6 positions, replace these with a whitespace and select the position with the highest loss. Next, we randomly select p characters from r, replace them in the chosen position and choose the one with the highest loss as the final perturbation. During training, the attack evaluates o x B sentences in every forward pass, where B is the batch size. For more details, see Algorithm in the appendix.
Method
In order to make the text encoder adversarially robust, we extend Eq. FARE to the text domain as:
min
max ())
[lfoeur fo
(TextFARE)
where the Levenshtein deev distance is bounded by a parameter k, and C(S) is either the complete set of sentences S(T') or a subset only containing sentences with semantic constraints, see Section 2.2
Intuitively, if the original CLIP encoder evaluated the original sentence (fe (S)) provides good performance in downstream tasks, e.g., zero-shot classification or text-to-image generation, then, by solving Eq. (TextFARE), we will obtain model that achieves similar performance under perturbations of the sentence. Moreover, Eqs. of the text and image encoders.
FARE and (TextFARE) allow for decoupled training
Motivated by Danskin's ' Theorem [Danskin 1966 Latorre et al. 2023] we can (approximately) solve min-max problems by maximizing the inner problem and minimizing the error the ob- tained perturbation. In the case of Eq. (FARE), Projected Gradient Descent (PGD) is used for the inner maximization problem (Madry al. [2018, Schlarmann et al. 2024 Similarly,
any adversarial attack to maximize the inner problem in Eq. (TextFARE) e.g., Gao et al. 2018), Abad Rocamora al. (024).
However, not every attack adequate for adversarial finetuning, e.g., in image domain, strongest attacks in the AutoAttack ensemble [Croce and Hein, 2020) are never used during training due to their expensive time requirements. Contrarily, cheaper PGD attacks are used during training, providing fast training and generalization stronger adversarial attacks Goodfellow et al. [2015). Madry et al. 22188), Shafahi et al. [2019], Wong et al. [2020]. The desiderata for an adversarial attack used during training can be captured by two points: (i) High adversarial robustness to strong attacks after training, (ii) Low computational resources.
As a baseline attack in the text domain, we select Charmer (Abad Rocamora et al. 2024]. Adver- sarial training with Charmer in text classification results in strong adversarial robustness, satisfying (i). Nevertheless, Charmer is not resource-efficient during training and thereby does not satisfy our second desiderata (ii). This is due to Charmer needing to evaluate a number of perturbations ( [S]] 1) 1)+ + TAChamme [[]), which depends on the length of the sentence being attacked. This makes it harder to perform the attack simultancously over sentences in a batch.
Overcoming this limitation, we propose Levenshtein Efficient Adversarial Finetuning (LEAF): uti- lizing training-time attack that evaluates constant number of perturbations p per sentence. Our attack replaces a test character (the whitespace) in p random positions within the sentence to select the position with the highest loss. Then, P random characters are replaced in the chosen position to choose again the one with the highest loss. Overall, this allows to perform the attack in two se- sentences, where is the batch size. A visual representation of our quential evaluations of Interestingly, if 1, our attack performs a random perturbation. For attack is available in Fig. 2 a more detailed discussion on LEAF, we refer to Appendix B In Section 4.2 we empirically show LEAF satisfies our two desiderata.

---

Image.Ntt
News
80
70
Acmmay (%)
Acemury (%)
Clean
Adversarial
60
Unconstrained
Construined
18.7
FARE
50
+9.9
50
10 50
Figure 3: Training hyperparameter effeets: We report the zero-shot clean and adversarial accu- racy in the image (ImageNet) and text (AG-News) domains with FARE as a baseline. When no semantic constraints the cost significantly degrading the image domain performance. Adding semantic constraints im- proves the robustness in the text domain with minimal effects the image domain. Using random perturbations (p 1) improves the AG-News adversarial accuracy by 9.9 points, with stronger at- tacks (p 50) providing the best performance with 18.7 points of improvement.
employed (Section 2.2), the robustness in the text domain is improved at
Experiments
4.2 we cover our training results and display the interplay between p. k and the usage of additional constraints during training. In Section 4.3 we present the performance of our models in zero-shot classification. In Section 4.4 evaluate our CLIP models in multimodal retrieval tasks. In Section 4.5 evaluate the perfor- mance of our CLIP text encoders when incorporated into text-to-image generative models. Finally, in Section 4.6 evaluate how amenable models are to embedding inversion. Additional exper- iments, including an evaluation with token-level attacks, are available in Appendix(D D
We start by introducing our experimental setup in Section 4.1 In Section
4.1 Experimental setup
We train our text encoders for 30 epochs on the first 80, 000 samples of the DataComp-small dataset [Gadre al. 2023) with batch size of 128 sentences, & 1, p 50 and semantic con- straints, see Section 4.2.2 employing CLIP- ViT-L/I4 OpenCLIP-ViT-H/114 VIT-H/14, OpenCLIP- ViT-g/ and OpenCOIP- DDenCCLP-VVT-bigGG//
ViT-biG771 bigG/14 models. On the visual side, we scale the training method of Schlar- mann et al. 2024 to ViT-H/14 and ViT-g/14, using an foo threat model with radius = 2/255. See Appendix B.3 for detailed account of hyperparameters. For evaluating the adversarial robust- ness with respect to image perturbations, we follow Schlarmann et al. [2024] and employ the first two APGD attacks from the AutoAttack ensemble Croce and Hein 2020 with
2/255. In
the text domain, we choose Charmer-20 with h: = Abaa Rocamora et al. 2024 for evaluation. We employ the semantic constraints considered by [Chanakya al., 2024 in the text-to-image and retrieval tasks. For the zero shot classification tasks, we do not employ such constraints as done by Abad Rocamora al. 2024 For discussion on the of constraints,
refer Appendix D.
For zero shot sentence classification with CLIP models, we follow the setup of'Qin et al. 2023 see
Appendix B.4 for more details. For additional details, refer to Appendix D
4.2
Training robust text encoders
In Section 4.2.1
we analyze the performance and training speed of Charmer and LEAF, In Sec- we analyze how the performance is affected by our hyperparameters, i.e. k p and C(S).
tion 4.2.2
4.2.1
Faster adversarial finetuning
First, we evaluate the performance LEAF in terms of time and adversarial accuracy against training with Charmer [Abad Rocamora et al. 2024 with T'Charmer E {1,20), To do so, we train CLIP-ViT-
5

---

Table 1: Selecting the best attack for Adversarial Finetuning on ViT-B-32: We measure the AG-News clean (Acc.) and adversarial accuracy (Adv.) at h = with Charmer-20 and the time in seconds to attack batch of 128 sentences. We perform Adversarial Finetuning (Eq. (TextFARE) for 1 epoch with approach minimally affects the adversarial accuracy while being order of magnitude faster than the fastest Charmer variant.
using the attacks Charmer-1, Charmer-20 and LEAF with p E (20, 50 Our
AG-News
Defense
Time (s)
Acc. (%)
Adv. (%)
Charmer-20
76.70 (1..0)))
60.17
118. 118.19 (1.53.6))
Charmer-1
76.37 (10.21)
60. (0.20
( (428.98)
LEAF (p 50)
(6.63
(9.80
3.23
LEAF (p = 20)
76.87 76.87(10.25) (10.25)
(8.30
(10.11
B-32 for 1 epoch at k 1 and using P E {20, 50} for LEAF over three random training seeds. We measure the clean and adversarial accuracies with Charmer-20 on AG-News [Gulli. 2005 Zhang et al. 2013] and the average time to attack batch of 128 samples.
In Table we can observe that LEAF attains comparable clean and adversarial accuracies in compar- ison to the Charmer variants, while being significantly faster, i.e., 1.83 and 3.23 seconds per batch for our method in comparison to 15.17 and 118.19 seconds for the Charmer variants.
4.2.2 The effect of our hyperparameters
In order to test the influence of our training hyperparameters, finetune CLIP- ViT-L/44 initialized from pretrained FARE weights [Schlarmann et al., 2024) with E 21, 5, 10, 20, 50), k E {1,2) and CSS) including and not including semantic constraints. To evaluate how our method improves the robustness in the text domain, and affects the robustness in the image domain, we measure the clean and adversarial accuracies ImageNet and AG-News.
In Fig. 3 we report the performance for = 1. When increasing p. the adversarial accuracy in the text domain increases consistently. However, when employing unconstrained training attacks, both the clean and adversarial performance in the image domain are significantly degraded, e.g. p 50, a clean accuracy of 65.5% 74.7% for the FARE model. In contrast, when applying semantic constraints, the improvements in robustness in the text domain follow a similar trend and the performance in the image domain less degraded. For k 2, we can extract the same insights, see Fig. g Overall, we select 50, k = 1 and the use of semantic constraints during training.
Zero-shot Advenarial Accuracy
4.3 Zero-shot classification
80-
CLPP
FARE
We show the ImageNet and AG-News performance of the models when using robust encoders in image and/or text domain in Table and Fig. 1 We observe that our robust text encoders introduce only minimal drop in image per- formance, while significantly improving the robustness in the text domain. Moreover, we observe that the effec- tiveness of FARE for fine-tuning robust image encoders that was demonstrated for VIT-L/14 14 by [Schlarmann et al. [2024), extends to the larger ViT-H/14 and ViT-g/14 mod- ImageNet
Aceurory
(Ailwnrrid besigst)
Figure 4: Larger perturbations: We evaluate the adversarial accuracy in AG- News for l E 2,, 4, 5) in the ViT- L/14 scale. Our model (LEAF) obtains the highest adversarial accuracy all values of the distance bound k.
els. The lower performance of ViT-g/14 could be attributed to the smaller training batch size, see Appendix B.3 Importantly, only models that use robust encoder in both domains achieve robustness in both tasks.
In Fig. 4 we report the adversarial accuracy of the VIT- L/14 sized models in the AG-News dataset for E 2,,1, clean accuracy. Our model, while being trained with k = 1, is able to extrapolate the robustness to larger k. We observe that the CLIP and FARE models obtain a nearly zero adversarial accuracy for k > 4, while our model, is able to obtain the highest performance for any k.
with k: representing the

---

Table 2: Zero-shot classification. We report the adversarial accuracy (Adv.) on ImageNet with the first two attacks of AutoAttack (APGD-CE, APGD-t) at c = 2/255 and on AG-News with Charmer- 20 at 1. Only models employing robust image and text encoders are robust in both domains.
CLIP-VIT-L14
OpenCLIP- ViT-H/14
OpenCLIP OpenCLIP-ViT-g/1
V/T--/14
Robust Encoder
ImageNet
AG-News
ImageNet
AG-News
ImageNet
AG-News
Image
Text
Acc.
Adv.
Acc.
Adv.
Acc.
Adv.
Acc.
Adv.
Acc.
Adv.
Acc.
Adv.
X
76.4
0.0
74.4
44.7
77.2
0.0
71.1
37.6
77.8
0.0
67.3
35.8
x
74.7
47.6
78.7
44.5
76.8
48.4
70.7
37.5
73.8
41.8
66.4
32.9
/
73.4
0.0
73.9
60.1
77.0
0.0
71.1
50.2
76.3
0.0
67.3
47.4
72.6
46.0
78.0
63.2
46.3
72.3
53.3
72.0
41.3
66.7
46.3
Robust Model:
Robust Model:
Quwry:: a guggle geetin
budy
Cloaas
wazer
Charmer
Figure 5: Visualizing MS-COCO retrieved images. For our ViT-L/14 robust model and its non- robust counterpart, we show the top-3 retrieved images for the original Query and the perturbed Query via Charmer (k = 2, = 10) attack. The robust model is able to preserve the order and retrieves semantically relevant images even for the perturbed query. More illustrations can be found in Appendix D.5 The target query in this case was "This is an image pyramid".
4.4 Text-image retrieval
Robustness of CLIP models to perturbations of textual queries is important as these models are often used as dataset/content filters Hong et al. [2024] and NSFW detectors [Schuhaann et al. 2022] meaning any false negative can be detrimental. The robustness of retrieval based filters for visual adversaries has already been tested Croce al. [2025). Consider the case where CLIP based NSFW filter is queried with perturbed query, any false negative retrieval here would detrimental and concerning. To test how robust CLIP models are to such character based queries retrieval setup, we test on the MS-COCO dataset as proxy task.
For 1, 0,0 validation set queries, the attack maximizes the similarity between the test query and target string using different variants of the Charmer attack. Given some query text S and corre- sponding embedding (S), maximize the cosine similarity between fe(S) and fo(),, where T is target text semantically unrelated to S. The objective takes the following form,
max S':dua ("sss""
sim (fe(S") fo()
(1)
The optimization is done with the constrained Charmer attack for a different number of charac- ter changes. S' is initialized with S. and the overall perturbation set is constrained with C(S) from Chanakya al. [2024]. The formulation of the attack above can be seen as targeted attack, the same attack can be done an untargeted manner as in Eq. (2).
In Table 3 for different CLIP models, we show average Recall across 3 target strings, detailed 1) and (k: 2) character
results for each target can be found Appendix D.5 For both perturbations, we see that the non-robust CLIP models retrieval performance goes down. Our robust models on the other hand showcase strong robustness while showing a small degradation performance. For LEAF, the clean performance follows trade-off with robustness depending on P. see Appendix D Fig. 5 visualizes the attack and the top-3 retrieved images for a sample test query. Under perturbation, the non-robust model retrieves completely irrelevant images. The robust
clean
7

---

Table 3: MS-COCO text-to-image retrieval: The statistics of the targeted Charmer adversarial attack (with & 1,2 and semantic constraints) are averaged over 3 target strings. X: denotes non-robust CLIP model, whereas / indicates CLIP model robust in both image and text domains.
Clean
Eval. k
Charmer-Con @ecall
Model
Robust
Recall @1
@e
@e
37.31
62.67
x
49.11
73.79
30.66
52.76
CLIP-VIT-L/14
1
45.06
69.35
/
48.71
73.71
2
40.22
65.09
1
47.81
72.22
58.64
81.29
39.26
63.35
OpenCLIP-ViT-HIII
1
52.97
77.26
56.80
80.65
49.31
73.50
47.93
72.71
60.64
82.22
37.51
61.82
OpenCLIP-ViT-guee
1
52.30
76.95
55.98
79.33
48.71
73.71
model on the other hand, preserves the order and retrieves images relevant to the query. Moreover, in almost all cases it retrieves the top-1 image correctly, see Appendix D.5 for more such examples. Starting with & text perturbations, we test the robustness different variants CLIP-ViT-L/14 models to bimodal attacks using APGD for image perturbations. Even in this more challenging setup. LEAF attains the robust models, without sacrificing clean performance. We defer the associated results and discussion to Appendix D.5.I
4.5 Robustness of text-to-image models
In this section, we evaluate the performance our robust text encoders when plugged into text- to-image generation pipelines. We take SD-1.5 [Rombach et al. 2022 and SDXL. [Podell et al.) 2024). SD-1.5 employs the text encoder from ViT-L/14 and SDXL employs two text encoders: from ViT-L/14 and ViT-bigG/14, In order attack the model, we follow Zhuang al. [0233 by only accessing the text encoder. Given a sentence S, we employ Charmer-22 20 to solve:
min Sh:d (S.5 sim sim fo(())
By minimizing the similarity between the original and perturbed embedding, we expect that the model generates images that do not align to the original caption. For SDXL, we maximize the av- erage dissimilarities for both encoders. To analyze the quality of the generated images, through CLT--VVT-B-11 we measure the CLIPScore between the original caption S and the generated im- age. In Fig- 6 we present the MS-COCO [in et al.. 2014 SDXL image generation results. We can observe that the CLIPScore of SDXL with the LEAF encoders is significantly larger than the original SDXL for & 2 On the right-hand-side of Fig. 6 we present
generated images for the first five captions in the MS-COCO validation dataset at k 2. where for two captions, the original SDXI. model produces completely different images compared to the original ones.
In Appendix D.3 we include additional text-to-image generation details and experiments over SD- 1.5 and FLUX.I-dev [Black Forest Labs et al.] 2025). Interestingly, the generation quality of FLUX. LUUX.1-des dev can be severely degraded when only attacking its CLIP ViT-L/14 text encoder, see Table 13 We observe that the most common attack when the word "woman" appears, consists of replacing the final "n" for another character, see Table 19 This leads FLUX.1-des to produce im- ages of snakes as the tokens the word "woma", python species (Woma python), appear in sentence. In Fig. we report the images generated with FLUXX. Idev with the original CLIP encoder and the LEAF counterpart over 10 random seeds. When using our text encoder, the model is able to distinguish based on the rest of the sentence, whether "woman" or a "woma" should be generated.

---

0.335
SDXL
Caption
For driigg
stank 25 con
Uriginal
CLJPSon Text- Too--o-imme
LEAF
0.275
MS COCO
0.225
Original
2
LEAF
(Advemartal budget)
On the left side, we present the MS- COCO CLIPScores of SDXL. The LEAF text encoders consistently improve the generation quality of SDXL under adversarial noise. On the right, present the first five MS-COCO samples from the validation set and the corresponding SDXL generations at = 2. The color borders indicate null. partial and total matching the original image. With the original encoder, images 1 and do not match at all the original ones. With the FARE encoders, all of the five images resemble the original ones, with some errors like the mismatch in the number of objects in image 5.
Figure 6:
Text-to-image generation results SDXL:
Table 4: Text embedding inversion. We invert text embeddings and measure the quality of recon- structions with various metries. Robust models yield better reconstructions according to all metrics.
Model
Robust
sim 1
Word Rec. 1
Token Rec. f
BLEU 1
X
0.89
34.4
38.9
8.3
CLIP-VIT-L/1
0.95
46.4
52.0
12.2
0.86
33.5
34.1
8.9
OpenCLIP-ViTHH/4
V
0.93
49.0
50.3
13.7
0.94
43.7
48.1
5.6
OpenCLIP-ViT-gg114
0.96
54.8
60.6
12.2
4.6 Text embedding inversion
It is well known that robust models in the vision domain possess more interpretable gradients than clean models [Santurkar et al. 2019). which can be exploited generate visual counterfactual ex- planations (Augustin et al. 20200] Boreiko et al. 2022 Moreover, this allows to reconstruct images from their embeddings of a robust model by direct gradient based optimization [Croce et al. 2025).
We test if this advantageous property of robust vision models also holds in robust text models. To this end, we study the ability to invert text embeddings. Given an embedding fe(S), the goal is to reconstruct the unknown text S. Therefore we aim to solve the objective
max sim (Jo(S"), fo()))
(3)
S'ES(T)
To this end, we use the optimization method from Wen et al. [2023] where the text is initialized uniformly random over the vocabulary of tokens and optimized via gradient based algorithm.
We randomly sample 100 captions from MS-COCO, embed them via the given original and robust text encoders, and measure the success of reconstruction with four metrics: The cosine similarity between (S') and (S( i.c., the objective in Eq. ( Word Recall and Token Recall are the percentages of words/tokens in the original text that appear in the reconstruction, irrespective of order. Finally, BLEU [Papineni et al. 2002) is an ordering-awer aadrrnng-wware similarity metric.
We show results in Table 4 The models with robust text encoders are best in every metric. Inter- estingly, we observe that the reconstructions of robust models generally improve when scaling up model size, while for non-robust models it does not improve from ViT-L/14 to ViT-H/14, but im- proves from ViT-H/14 to ViT-g/14. We observe that BLEU scores are low for all models, indicating that while many words are reconstructed correctly, their ordering is not. This could be attributed
to the bfg-of-words behavior of CLIP models discovered by Yükkekgönül al. 2023
We show
some randomly selected example reconstructions in Appendix Tables 22 and 23

---

Caption
A woma@ stands in the xining area at the table.
'a</w>', 'wom', 'a</w>' '<</>>' 'stands</w>', 'in</w>', 'the</w>',
Tokens
x'' 'ining</w>', 'area</>>', 'at</>>' 'the</>>', 'table</>>' '.</w>'
CLIP
LEAF
Caption
woma python.
Tokens
'a</w>', 'wom'. a<</>>'
'python</w>',
2 '<</w>>
CLIP
LEAF
Figure 7: Text-to-image generation with FLUX.I-dev: We generate images with 10 random seeds using the original CLIP ViT-L/14 text encoder and the LEAF variant. The model using the CLIP text encoder consistently generates snakes for the first sentence, probably due the appearance of the word "woma", kind of snake (Woma pythonj. When using our robust text encoder, we can accurately generate a woman and are also able to generate woma pythons when prompted do so. While both captions start with our text encoder distinguishes between the and continuations.
5
Conclusion
This work takes a first, systematic step toward bimodal robustness of CLIP by addressing the long- neglected text side. We introduced LEAF, simple and efficient adversarial fine-tuning scheme for text encoders that mirrors the FARE philosophy on the image side: preserve the location of the clean embedding while enforcing invariance to small perturbations. For our adversarial fine-tuning scheme we develop a training-time character-level attack that allows for efficient training. In doing so, showed that robustness in the text domain is both practically achievable and practically useful. Across zero-shot classification, text-to-image retrieval, and text-to-imaat generation, LEAF improves robustness to character-level attacks consistently, while leaving the clean performance intact.
Importantly, we show that robust CLIP text encoders obtained via LEAF can be combined with robust CLIP image encoders (e.g. FARE) to yield CLIP models that robust both input domains. This yields the first recipe that jointly elevates robustness in both modalities, and it scales without bespoke architectural changes heavy joint training. Moreover, the method is modular: encoders can be swapped without touching downstream models, e.g. in text-to-image pipelines.
Notably, while we focus the empirical evaluation in this work CLIP based models, our LEAP method could be applied to any text encoder: see Table 27 for an illustrative example beyond CLIP, where BERT model is finetuned for sentence classification.
Limitations: Our robust image and text encoders are finetuned in isolation, joint training could yield larger robustness gains at higher training cost. Nevertheless, our bimodally robust models are validated against inference-tis time attacks that optimize over both modalities (see Table 25). In this work, we did not train models to be robust to token-level attacks, as these attacks often change semantics of sentences (Dyrmishi et al. 2023]. Due to computational constraints, we did not train the largest image encoders (OpenCLIPP- ViT-bigGi the largest EVA-CLIP models [Sun et al.) 2024.. Our approach has not yet been tested in other tasks using text encoders, e.g., RAG Lewis et al. 2020 We hope that our paper fosters advances in these areas.
10

---

Acknowledgments
We thank the NeurlPS 2025 organization committee and reviewers for their work. This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021 222001011 1. Research was sponsored by the Army Research Office and was accomplished under Grant Number N911NF-24-1-0048 This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). This work was supported as part of the Swiss Al Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a07 Alps. EAR, YW and VC thank Gosia Baltaian for her administrative help. We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting CS and NDS. We acknowledge support from the Deutsche Forschungsgemminnchaatt (DFG, German Research Foundation) under Germany's s Excellence Strategy (EXC number 2064/1, project number 390727645), as well as the priority program SPP 2298, project number 464101476. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
References
Elias Abad Rocamora, Yongtao Wu, Fanghui Liu, Grigorios G. Chrysos, and Volkan Cevher. Re- visiting character-level adversarial attacks for language models. In International Conference on Machine Learning (ICML), 2024.
Maan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in neural information processing systems (NeurIPS), 2022.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in- and out-distribution improves explainability. In ECCV, 2020.
Brian R Bartoldson, James Diffenderfer, Konstantinos Parasyris, and Bhavya Kailkhura. Adversarial robustness limits via scaling- law and human- allinmeet studies. In International Conference on Machine Learning (ICML), pages 3046-3072, 2024.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In International Conference on Learning Representations (ICLR), 2018.
Steven Bird and Edward Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214-217, Barcelona, Spain, July 2004. Association for Computational Linguistics.
//acl///cllll https/////...aaaoooooooooo////00 org/P0433/3//
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for co-context image generation and editing in latent space, 2025. URL httts:://arxisi org/ wttp:ra/g/a....o..////// 15742
Valentyn Boreiko, Maximilian Augustin, Francesco Croce, Philipp Berens, and Matthias Hein. Sparse visual counterfactual explanations in image space. In GCPR, 2022.
Patihandla Chanakya, Putla Harsha, and Krishna Pratap Singh. Robustness of generative adversarial gen.raatiooo... clips against single-charache adversarial attacks in text-to-image generation. IEEE Access, 2024.
11

---

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei, Scaling catruction-finttune language models, 2022. URL httph: org. htt/ 2tt.bb//2211111111
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. International Conference on Machine Learning (ICML), 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adver- sarial robustness benchmark. arXiv preprint arXiv:2010.09670 2020.
Francesco Croce, Christian Schlarmann, Naman Deep Singh. and Matthias Hein. Adversarially robust clip models can induce better (robust) perceptual metrics. In Sa7MI. 2025.
J. Danskin. The theory of max-min, with applications. SIAM Journal on Applied Mathematics, 14 (4):641-664, 1966. doi:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019.
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural language word substitutions. In International Conference on Learning Representations (ICLR), 2021.
Salijona Dyrmishi, Salah Ghamizi, and Maxime Cordy. How do humans perceive adversarial text? reality check on the validity and naturalness of word-based adversarial attacks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box box adversarial exam- ples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018.
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11 11097, 2021.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2023.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In IEEE Security and Privacy Workshops (SPW), 2018 8.
12

---

Siddhant Garg and Goutham Ramakrishnan. BAE: BERT- adversarial examples for text clas- sification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, and Alexandre Araujo. R-LPIPS: An adversarially robust perceptual similarity metric. In ICML Workshop on New Frontiers in Adversarial Machine Learning. 2023.
Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Farshad Khorrami, and Siddharth Garg. Lipsim: A provably robust perceptual similarity metric. In ICLR, 2024.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR), 2015.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in neural information processing systems (NeurIPS), 34:4218 4233, 2021.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech- object category dataset. 2007.
Antonio Gulli. Ag's corpus of news articles, 2005. httthhttp///orotte di. unipi it/ iettittgllll
AG__orpus__of sttpaas____eww___eeeeeeeeeeeeeee htt___nessss_i_eeeeeeee articles...... html
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kicla. Gradient- based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7), 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021.
Rachel Hong, William Agnew, Tadayoshi Kohno, and Jamie Morgenstern. Who's in and who's out? study of multimodal clip- fiptfiitering in datacomp. In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 2024.
Bairu Hou, Jinghan Jia, Yihua Zhang, Guanhua Zhang, Yang Zhang, Sijia Liu, and Shiyu Chang. Textgrad: Advancing robustness evaluation in NLP by gradient-driven optimization. Interna- tional Conference on Learning Representations (ICLR), 2023.
Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krish- namurthy Dvijotham, and Pushmeet Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL kttpps://do www.loo.orr/....0 5281/ zenodo.5143774 If use this software, please cite it as below.
Di Jin, Zhijing Jin, Jocy Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. AAAI Conference on Artificial Intelligence, 2020.
Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. Robust encodings: A framework for combating adversarial typos. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
13

---

Andreas Koukounas, Georgios Mastrapas. Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez. Saahil Ognawala. et al. Jina clip: Your clip model is also your text retriever. arXiv preprint 2Xii:2445.20204 2024.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE internationa conference on computer vision work- shops, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Canada, 2009.
Fabian Latorre, Igor Krawezuk, Leello Tadesse Dadi, Thomas Michaelsen Pethick, and Volkan Cevher. Finding actual descent directions for adversarial training. In International Conference on Learning Representations (ICLR), 2023.
Deokjae Lee. Seungyong Moon, Junhyeok Lee, and Hyun Oh Song. Query-efficient and scalable black-box adversarial attacks on discrete sequential data via bayesian optimization. In Interna- tional Conference on Machine Learning (ICML), pages 12478-12497 PMLR, 2022.
Vladimir I Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet Physics Doklady, volume 10, pages 707-710. Soviet Union, 1966.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeurIPS, 2020.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. Network and Distributed Systems Security (NDSS) Symposium, 2019.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK, Adver- sarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pages 740-755. Springer, 2014.
Aiwei Liu, Honghai Yu, Xuming Hu, Shu' ang Li, Lin, Fukun Ma, Yawen Yang, and Lijie Wen. Character-level white-box adversarial attacks against transformers via attachable subwords sub- stitution. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems (NeurIPS), 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Association for Computational Linguistics (ACL), 2011.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft, 2013.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. International Conference on Learning Repre- sentations (ICLR), 2023.
14

---

Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. International Conference on Learning Representations (ICLR), 2017.
John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
John Xavier Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M Rush. Text embed- dings reveal (almost) as much as text. In EMNLP, 2023.
John Xavier Morris, Wenting Zhao, Justin Chiu, Vitaly Shmatikov, and Alexander M Rush. Lan- guage model inversion. In ICLR, 2024.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE, 2008.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffuusin models for highuresolution image synthesis. In International Conference on Learning Representations (ICLR), 2024.
Samuele Poppi, Tobia Poppi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, and Rita Cuc- chiara. Safe-clip: Removing nsfw concepts from vision-and-language models. In European Conference on Computer Vision (ECCV), pages 340-356. Springer, 2024.
Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics (ACL), 2019.
Libo Qin, Weiyun Wang, Qiguang Chen, and Wanxiang Che. CLIPText: A new paradigm for zero-shot text classification. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML). PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning (ICML), 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/15, 2022.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in neural information processing systems (NeurIPS), 2021. URL /ttttpp://opereevieee www/ wtt.roorrmmmiiiiioo did-miigBkBhhhhhheeee
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- the IEEF/CVF confer- ence on computer vision and pattern recognition (CVPR), pages 10684-10695, 2022.
resolution image synthesis with latent diffusion models. In Proceedings the
15

---

Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In NeurIPS, 2019.
Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV) Workshops, pages 3677-3685, October 2023.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial fine-tuning of vision embeddings for robust large vision-language models. International Conference Machine Learning (ICML), 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances in neural information processing systems (NeurIPS), 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013.
Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-1888 18b: Scaling clip to 18 billion parameters, 2024. URI. https:ttts//aaxii //rg://rr/vv////////// abs mass/2402.044
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy, In International Conference on Learning Representa- http://rrrmmiixyxxxa3a3a3
fions (ICLR), 2019. URL https:
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equiv- ariant enns for digital pathology. In MICCAI. Springer, 2018
Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn. Inquire: A natural world text-to-image retrieval benchmark. Advances in neural information processing systems (NeurIPS), 175100050-126511 2024.
Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. Info(bert]: Improving robustness of language models from an information theoretic perspective. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurlPS, 2019.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. NeurlPS, 2023.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. International Conference Learning Representations (ICLR), 2020.
Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael I. Jordan. Greedy at- tack and gumbel attaek: Generating adversarial examples for discrete data. Journal of Machine I Learning Research, 21(43): 1-36, 2020. URL | /mmmr. org/ wwtgg//arrs/////ee/ee pateers/vv///1 121/19-560 569. html
16

---

Yelp. Yelp open dataset, 2015. open-dataset/
URL
Mert Yüksekgönüll, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In ICLR, 2023.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
04867
Chenyu Zhang, Mingwang Hu, Wenhui Li, and Lanjun Wang. Adversarial attacks and defenses on text-to-image diffusion models: A survey. Information Fusion, 114: 114:102701, 2025.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent EI Ghaoui, and Michael Jordan.
Theoretically principled trade-off between robustness and accuracy. International Conference Machine Learning (ICML), 2019.
Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision- lisounlaagguage pre-training models. Proceedings of the 30th ACM International Conference on Multimedia, pages 5005- 5013, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (NeurIPS), vol- ume 28, 2015. URL https://prooeedinng. neurips...... fttppppr.filleppppr//0/5S F11e/250.BBb117738B886663633 -atpeeda pdf
Chen Zha, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations' (ICIR), 2020.
Haomin Zhuang, Yihua Zhang, and Sijia Liu. A pilot study of query-free adversarial attack against stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2023.
Andy Zou, Zifan Wang. J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 2XXv:23077.1043 2023.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have separate experimental section for each of the claims regarding the performance our models exposed in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations the end of Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification:
We cover full set of hyperparameters in Section 4. and Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: Our source code and documentation are published at https::/github.
Com/LIONS-EPFL/LEAF
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines lhttps://nipp: cc/
public/guiiee//odeStttmiiss soni for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g,, for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https:
Sodeeeee//ooeeeemeaacaa www.mmssssooplioiioiooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We cover our hyperparameters in Section 4. and Appendix B. the relevance of each hyperparameter is performed Section 4.2.2
The analysis
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: While we report error bars for several experiments, it is not feasible for the large-scale training runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: As covered in Section 4.1 our text encoders were finetuned in single NVIDIA A100 GPU with 40GB of memory. The largest image encoders required 8 40GB A100 GPUs. We did not exhaustively measure the time per experiment, but the largest finetuning run (ViT-bigG/14) took roughly week to finish.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:::/tp////tttt htttt::://neriis... http://p.....ppbii/iciiiiiesee.d
Answer: [Yes]
Justification: Our research conducted in this paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We cover the broader impact of our work in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We anticipate such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of the models we finetune and the data we use is publicly available un- der MIT cc-by-4.0 licenses. We appropriately credit the original authors by citing the corresponding papers and list the source models in Table
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, wwtpppppppppoooooo
com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yss]
Justification: We give detailed descriptions of models trained in this work and will release them under MIT license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can cither create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects. and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development in this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this rescarch does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neuriin cctc.oofeeeeecccs///2///// for what should or should be described.
24

---

A Broader impact
This work positively impacts society by strengthening models that employ CLIP text encoders against perturbations in the text input, which is particularly important for safety-critical and high- volume applications. Practitioners can harden existing CLIP-based systems by adopting our adver- sarially robust text encoders as drop-in replacements with minimal changes. We provide/source code and open source models support responsible deployment.
B Additional details
In this section, provide additional details on the implementation of our method and the experi- mental setting.
A
Additional Notation: Given two matrices A mxd and B Roxd define A
+ B
B
(()+n)xd Concatenating with the empty sequence 0 results in the identity A 0 A. We denote
A2 E ()xd the matrix obtained by removing the first row.
B.1
Method details
Firstly, we characterize the single-character perturbations following AAbad Rocamora et al. (2024).
Definition B.1 (Expansion and contraction operators). Let S(T) be the space of sentences with alphabet I' and the special character & g I, the pair of expansion-coonraction functions Φ :(()) S(T U (E)) and vb S(T U (E)) ST) is defined as:
if =0 if S =
& ((S) = S1 (S2))
if i| 0 otherwise.
(SS) =
(S Si @ th (S2.
otherwise.
Clearly, (SS) aims to insert & into all possible positions between characters and at the beginning Similarly, (SS)) aims remove and end of the sentence, and thus we have IØ(S) 2 |S| + 1. +.
all & occurred in S. The (é, v)) pair satisfies the property that (b((S) = S. We give the following example for better understanding.
Example B.2. Let 5 == for visibility:
6(Hello) HHlellllol (LL Heello v(
Helo ((HHellloo.)
Hello
Definition B.3 (Replacement operator). Let S E (E) the integer E [|S]] and the character c, the replacement operator & c of the jth position of with e defined as:
S c S @
S
Thanks Definition B33
ready present attack in Algorithm The advantage of
Algorithm T resides in attacking a batch of B sentences parallel, an important feature for efficient adversarial training.
B.2 Semantic constraints details
In order to follow the semantic constraints of [Chanakya et al, 2024], we constrain the attacks during training and during retrieval and text-to-image generation not produce new English words. To do so, we employ Algorithm 2 over pairs of sentences and so that (S,S') = 1. Algorithm returns that the perturbation S' is valid only if contains less english words than S.
B.3 Training details
All of our text encoders are trained on the first 80, 000 samples of the DataComp-small dataset [Gadre et al. 2023 for 30 epochs with batch size of 128 sentences. We employ the AdamW op- timizer [ingma and Ba 2015 Loshehilov and Hutter 2019], a weight decay of 10-1, a maximum learning rate of 10-3 with linear warmup of 1, 400 steps and cosine decay. For training the robust
25

---

Algorithm LEAF batched attack
1: Inputs: Text encoder fo S(T) + IRA, batch {5 loss function L, radius k, number of
simultaneous perturbations p. alphabet I, test character and flag for semantic constraints Cons. b Initialize perturbations with clean sentences.
2: S, = S, vi E [B]
3: for 1.... do
4: Pij ~ Unif. (1- + +1)
Ai E E
Sample positions in every sentence.
5: $ s-1 [{( (0()) f2 }
B
b. Replace the test character all Pt--
j=1
6: if Cons then
P Use Algorithm check if the perturbation is valid, revert otherwise.
St
if valid(S,, S.))
7:
Sig S
vi B Vj
[p]
otherwise
8:
jt = arg max L (Jo (St))
Eval. losses in parallel and get the max.
9:
Gj Unif. (T) Vi [B] Vj E
Sample p characters for every sentence.
Pas
10:
S
b Replace Gjj in the position Pojt-
11: if Cons then
b Use Algorithm if valid(S,, S.y)
to check if the perturbation is valid, revert otherwise.
St
Si
Vi
B Vj
[p]
S
otherwise
13:
1. arg ja C (fo (St))
jE(o]
Eval. losses in parallel and get the max.
S. Sa Vi (B)
Update perturbations.
( i=1
return
Algorithm 2 Semantic constraints
1: Inputs: Sentence and perturbation S".
2: words(S)
3: n wordss(S)) S')]
> We extract English words using NLTK: httph:ttp//// //wp: nltk. org/
4: return m7
vision encoder, we adapt the setup of Schlarmann al. [024].. Namely, we train on images from ImageNet for 10k steps (instead of 20k, due to compute constraints) with batch size of 128 for ViT-H/14 and 64 for 9/T4 We use weight decay of 10 a maximum learning rate of 10-5 with a linear warmup of 700 steps and cosine decay. To optimize the inner adversarial objective, we use PGD with 10 steps and set e = 2/255. Our codebase is based on OpenCLIP [Ihharco et al. 2021]. All of our experiments are conducted in single Nvidia A100 40GB GPU, except for training robust image encoders, where GPUs were employed.
B.4 Zero-shot text classification
Analogously to how zero-shot image classification is performed in the original CLIP paper [Radford ctt al.) 2021], Qin et al. (2023) encode one image representing each class and compute the similarities with the sentence embedding. Then the predicted class is the with the highest cosine similarity in the embedding space. In Table we present the images employed for each dataset and label.
B.5
Text inversion
In order to invert text embeddings, we sample 100 random captions from COCO val2017 and use the optimization method proposed by Wen al. (023) with 3000 iterations, learning rate 0.I, and weight decay 0.1.
26

---

Table 5: Images and sentences used for zero-shot text classification.
Images
Dataset
Class
Class 2
Class 3
Class 4
SST-2 IMDB Yelp
NA
NA
AG-News
Sports
FINANCE
Politics
Sentences
"Negative Review"
"Positive Review"
SST-2 IMDB Yelp
NA
NA "Science and Technology News"
"Business News" "
AG-News
"World News"
"Sports News"
Table 6: Source models employed for finetuning and evaluation.
Model
Source
CLIP-VIT-B-32
httpa: http::/:ww.cccccccoeeeeee/aaa/// /ttp://////////peeeeeee//////eee
CLIP-VIT---16
httpa:
VIT-L/14
attpa://huze
FARE
SafeCLIP
OpenCLIP-ViT-Hy V-T-H-14
OpenCLIP-Vin VIT-g-14
OpenCLIP bighCLLP-Vi--big---- big--igG-114
Stable Diffusion v1.5 (SD-1.5)
Stable Diffusion XI. hase http:///ppppaaaaaaacaaaaaaaaaaaa
FLUX.I-dev
B.6 Model checkpoints
In T we enumerate the external models employed in this work and the sources used for com- parison and finetuning.
C Related work
In this section we cover related work on Adversarial Attacks, Adversarial Training, Robustness of Multimodal Models and text inversion.
Adversarial Attacks The vulnerability deep learning models against adversarial input attacks well known (Szegedy et al., 2014]] Goodfellow et al. [2015] and hast been extensively studied in the vision input domain [Croce and Hein 22200 Schlarmann and Hein 2023) and the text input domain,
with the most popular attacks employing perturbations in the loken-leve [Ren et al.] 2019 Jin al.
2020
Li et al. 2019 Garg and Ramakrishnan 2020 Lee et al. 2022 Ebrahimi et al.
2018
Li al,
2020
Guo 20.1 Hou al.] 2023 and character-level
Belinkov and Bisk
2018 Ebrahimi
et al.
2018
Gao et al. 2018 Pruthi et al. 2019 Yang et al. 2020 Liu et al. 2022
Abad Rocamora
al.
2024
Adversarial Training in the text domain. Adversarial Training Maadry et al. 2018 and its variants
[hhag et al.) 2019 Rebuffi et al.) 2021 Gowal et al. 2021 Wang et al. 2023 Bartoldson et al.)
2024 are most prominent defense against adversarial examples in the image domain Croce and
Hein 2020). Croce et al. 2022)]
27

---

In the text domain, also variants of adversarial training constitute the best defenses, with most de- fenses focusing on token-level attacks. Taking advantage of the efficiency of PGD, MMyato et al. [017)] propose solving the inner maximization problem in Ep constrained ball around every token embedding. /Zuu al.
2020) accelerate embedding-lewe PGD AT and show improvements clean accuracy. Wang et al.
2021 show improvements in adversarial accuracy by adding an information theoretic regularization term. Deviating from the embedding- based PGD AT paradigm, Dong et al. [2021] use PGD to maximize the loss over a convex combination of synonym embeddings. Then, 2023 find that directly optimizing the inner max in the text space with existing attacks Jii et al.
Hou et al.
2020 significantly boosts the adversarial accuracy against multiple adversarial attacks.
In the chavacter-level, it was initially thought that typo-correctors would suffice defense (Pruthi et al. [2199 Jones et al.) 22220]. Abad Rocamora et al. (2024) shows that typo-corrector defenses can be easily broken. Additionally Abad Rocamora et al. (2024) show that similarly to the results of [Hou et al. 2023) in the token-level, performing adversarial training with character-level pertur- bations improved the character-level robustness.
Robustness of Multimodal Models. Attacking and defending multimodal models has gained sig- nificant interest recently. Mao al. [2023) propose TeCoA, which performs supervised adversarial fine-tuning on CLIP in order to defend against visual adversarial attacks. In turn, Schlarmann el al. (024) propose FARE, an unsupervised robust fine-tuning method for vision encoders that preserves downstream performance, e.g. of LMMs that utilize a vision encoder.
Text inversions. Morris et al. 2023 2024 learn models that invert text embeddings language model outputs. In contrast, Wen et al. [2023] invert CLIP image embeddings into text via direct optimization. They use the reconstructed text prompt diffusion models and thereby generate similar images. We use their optimization scheme to invert text embeddings and show that it yields better results when used with our robust models.
D Additional experiments
In this section we cover additional experiments not fitting in the main manuscript. First, in Ap- pendix D.I we analyze the effect adding additional constrains to the adversarial attack. Then, in Appendix D.2 we cover additional experiments in zero-shot classification. In Appendix D.3 we include additional text-to-image generation experiments. Appendix D.4 we include examples of the sentences reconstructed from their embeddings through embedding inversion. Finally, In Ap- pendix D.6] we perform ablations studying the final losses for different values of k and e, and perform token-level adversarial attacks.
D.1 On the effect of additional attack constrains for Text-to-image models
In this section, we evaluate the effectiveness of the semantic constraints considered by Chanakya al. 2024). order avoid including new words with different information the prompt, Chanakya et al. (2024) constrain the attack to not produce new words in the English vocabulary. To do so, they tokenize the clean and adversarial prompts and check for the appearance of new words in the adversarial prompt based on the NLTK English dictionary [Bird and Loper, (2004). In order to check for the need of these constraints, we attack SD- 1.5 equipped with our robust text encoder at k = 2 using Charmer [Abad Rocamora et al. 2024] on the COCO va12017 dataset [Lin et al. 2014). We then visually explore the adversarial prompts and generated images to look for inconsistencies.
In Table 7 we can observe five examples of unconstrained attacks producing adversarial prompts with sigmificantly different meaning. Since the only constraint that the Levenshtein distance needs to be < 2, the attack is able to turn "bear" into "beer", "stop" into "shop", "bananas" into "bandanas" or "wave" into "pave". This results in the diffusion model generating images that correctly adopt these adversarial captions and the adversarial prompts being invalid. If we constrain the attacker to not generate new words, the adversarial prompts preserve the meaning of the original captions up to uncommon words/abvreviations not present in the NLTK dictionary, like "grads" or "smurfs". Overall, we consider the constraints necessary for the text-to-image generation tasks, agreeing with Chanakya et al. 2024).
28

---

Table 7: Examples of problematic attacks in COCO val2017: If no additional constraints are considered, a single character change can produce semantical changes in the prompt, e-g., "bear" is transformed into "beer". This leads to image generations that are highly dissimilar to the original reference image, but are correct according to the adversarial prompt. The semantic constraints employed by Chanakya et al. (2024) help reducing the amount of new words. Nevertheless, some abbreviations like grads" or uncommon words like "smurf" still appear after the attack.
Unconstrained
Constrained Chanakya al. 2024 Advensurial cuption Generated image
Original caption
Original imuge
Adversarial caption Generated image
big burly grizzly bear is show with gruss background.
A big burly grizzly beer
big burly !rizzly bear
285
is show with
show with grads in background.
the
background.
A sign 724 is mounted upside-down
A shop sign is mounted uplide-down on it's post.
scop sign is mountedaupside- down on post.
EHOPP IIE
soeeE
it's post.
"Three teddy bears, each a different color, sauggling together."
"Tree teddy beans, each a
Shree teddy hears, each different color, snuggling
different color, snuggling together."
toge,ther.
A bunch of 3661 bananas sitting of wooden table.
bunch bandanas sitting on top auwooden table.
bunch of sitti-g of woodenitable.
person riding surf board on a wave
person riding smurf board on a pave
person riding smurf board on waze
6460
Image:V Vel
AG-News
80
70
Accurayy (96)
Aceuncy (96))
Clean
60
Advensriaa
Unconstrained
50
Constrained
40
30
2
5 20
30
10 50
P
Figure 8: Hyperparameter effects at = 2: We report the zero-shot clean and adversarial accuracy in both domains (ImageNet and AG-News) with FARE [Schlarmann et al. 2024] as a baseline. For the unconstrained attack, larger values of p improve the robustness in the text domain at the cost of significantly degrading the clean and adversarial performance in the image domain. Constraining the attack allows improving the robustness in the text domain with minimal effects on the image domain performance.
D.2 Zero-shot classification
In this section include additional datasets for zero-shot image and text classification. We also include hyperparameter analysis with
In Fig. 8 we can observe the same experiment in Section 4.2.2 and Fig. with & = 2 instead of Similarly to the experiments with 1, increasing leads degraded performance the image domain when constraints are employed. Including the constraints, allows for increasing
29

---

Table 8: Zero-shot performance for different k, and constraints.
Semantic Constraints
ImageNet
AG-News
k
Acc.
PGD-20 Acc. (c
Ace.
Charmer Acc. (h: 1)
253
74.7
46.7
78.7
57.6
74.5
46.5
78.3
60.7
72.0
45.4
78.7
62.9
70.1
43.7
78.6
64.8
20
67.5
43.5
78.0
65.2
50
65.5
42.0
78.2
66.3
73.5
47.4
79.1
60.2
73.3
46.5
78.4
63.3
67.4
42.4
79.1
65,4
2
10
60.4
36.3
78.8
67.0
20
55.3
32.3
78.0
66.7
53.3
30.5
78.0
67.8
74.7
46.9
78.2
54.4
74.8
47.2
77.5
56.9
74.8
47.7
78.3
58.6
1
10
74.8
46.3
78.3
59.9
20
73.6
46.3
78.4
60.7
50
72.6
46.0
78.0
63.2
74.7
47.1
77.4
55.8
75.5
47.3
78.1
58.6
75.2
47.0
78.9
59.9
2
10
74.1
47.5
78.6
61.5
20
73.0
46.7
77.8
62.8
50
70.5
45.3
78.4
63.5
the robustness in the text domain with less performance degradation.
The numbers form Figs.
and are available Table g
D.2.1
Additional experiments on zero-shot image classification
For zero-shot image classification, measure the clean and robust accuracy 13 datasets: Cal-
Tech101 Griffin et al. 2007]. StanfordCars Krause et al.] [2013). CIFARIO, CIFAR 100 Krizhevsky
2209
DTD Cimpoi al. (2014). EuroSAT Helber al. 2019] FGVC Airerafts
Maji al.
2013 Flowers Nilsback and Zisserman 2008). ImageNet-R Hendrycks et al.
[2021] ImageNet-
Sketch Wang al. [2019). PCAM Veeling al. 2018]. OxfordPets Parkhi et al. [2012] and STL-
10|Coates et al. [201] IJ..T.. To measure robustness, we conduct visual attacks as described in Section[4.]
and restrict the evaluation 1000 random samples all datasets. We evaluate orginal models and models that employ robust encoders in both domains. Results are reported in Table 9 The robust models maintain much better performance under adversarial attacks, while sacrificing performance.
clean
In Table 10 we report the VTAB [Zhai et al.) 2020 averaged performance over the categories nar- ural, specialized, and structured. We observe that clean evaluation, robust models sacrifice per- formance natural and specialized trade-off between clean and robust performance is expected TTippra et al. (019)). On structured the behavior mixed sometimes even outperforming the non- robust models. In the adversarial evaluation (c 2/255), observe that the non-robust models completely vulnerable, while our robust models maintain much better performance when attacked.
D.2.2 Additional experiments on zero-shot text classification
In this section, we evaluate the zero-shot clean and adversarial accuracy of our models additional text classification datasets. We follow the same attack setup in the AG-News experiments, i.e.,
30

---

Table 9: Zero-shot image classification, We report the zero-shot image classification performance of original and bimodally robust models.
Calleehh11
ImageNet-r
ImageNet-s
Cifar10
Cifar100
EuroSAT
FGYC
Flowers
PCAM
STL.10
Mean
DTD
Pets
Model
Robust
82.1
77.5
95.2
68.2
55.7
63.4
28.4
79.4
86.5
48.9
53.0
93.9
98.8
71.6
CLIP-VIT-L/1
81.1
71.6
92.2
68.9
44.9
28.7
24.6
69.7
83.3
47.0
59.9
91.9
98.1
66.3
Clean
Oie-CI.I-VIT-HVII
X
84.4
92.2
97.5
82.8
68.7
72.5
42.4
80.2
88.4
56.1
54.9
95.1
98.1
77.9
83.8
89.8
93.3
69.7
61.1
34.4
35.8
73.4
85.7
52.9
50.4
94.0
97.2
70.9
X
84.3
92.1
97.7
84.0
68.8
65.6
36.4
78.1
88.2
55.5
55.6
95.2
98.2
76.9
OpenCLIP- V
83.1
88.4
91.7
67.3
58.1
29.0
30.7
71.2
84.9
52.0
52.5
92.5
96.2
69.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
CLIP-VIT-L/I4
2/255
70.5
27.8
65.6
34.2
25.3
11.6
6.0
33.8
55.5
26.4
22.1
69.0
89.7
41.3
OpenCLIP-VITHHI4
0.0
0.0
0.0
0.0
70.7
55.6
65.0
38.4
32.5
7.7
5.8
39.5
58.3
31.0
37.9
66.0
87.9
45.9
ViTTCL.P-ViT-/144
0.0
0.0
0.1
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
71.3
52.1
62.6
34.0
28.5
4.7
34.2
53.3
28.6
26.5
57.5
84.7
41.7
Table 10: VTAB zero-shot image classification. We report the zero-shot image classification per- formance of original and bimodally robust models on VTAB Zhai et al. (2020].
Model
Robust
Natural
Specialized
Structured
74.4
63.5
11.9
VIT-L/14
68.5
41.9
13.3
Clean
78.7
57.0
11.7
VIT-H/14
74.8
11.8
79.5
62.9
12.5
ViT-g/14
72.4
51.4
11.4
x
0.0
0.0
0.0
VIT-L/14
2/255
42.4
10.6
3.9
X
0.1
0.0
0.0
VIT-H/14
/
44.9
14.6
3.6
X
0.0
0.0
0.0
ViT-g/14
41.0
9.5
1.9
we employ Charmer-20 (Socher et al. 2013), IMDB MMas et al.) 2011] and Yelp Yelp) 2015 Zhang et al. 2015).
without semantic constraints evaluate the performance on SST-2
In Fig. 9 report the zero-shot adversarial accuracy already reported in Fig. 4 with the addition
of SafeCLIP (Poppi et al.] 2024). SafeCLIP obtains a considerably lower clean and adversarial
accuracy in comparison to the other CLIP variants.
In Table II we can observe that similarly to the AG-News results in Table the models with robust text encoders achieve higher adversarial accuracy in the text domain, with improvements of more than 9.9 robust accuracy points for models and datasets.
we present the clean and adversarial zero-shot accuracy when employing only the text encoder the ViT-L/14 models. For that, we encode on sentence per label instead of one image per label as done in the main text. See Table for more details on the sentences employed for the labels. We can observe that the adversarial accuracy is larger after adversarial finetuning with LEAF. Nevertheless, the clean and adversarial performance are worse when doing text-encoder-only zero- shot classification, e-g., a clean accuracy in AG-News with ViT-L/14 of 74.4 when using images as labels (Table v.s. 54.8 when using sentences as labels.
In Table 12
31

---

Adversorial Accuracy
SafeCLIP
FARE
Acotroy
LEAF
(Ailommrria buileet)
Figure 9: Larger perturbations: We evaluate the adversarial accuracy AG-News for (1,2,3,4, 4,5, in the ViT-L/14 scale. Our model (LEAF) obtains the highest adversarial accuracy at all values of the distance bound k.
Zero-shot text classification. We report the zero-shot text classification performance of Table 11:
original and bimodally robust models.
Model
Robust
SST-2
IMDB
Yelp
X
71.2
61.6
80.9
CLIP-VITI/14
V
71.9
61.4
82.0
Clean
X
61.6
57.5
73.7
OpenCLIP-ViT-H/14
58.4
53.2
72.6
X
57.8
56.8
71.9
OpenCLIP-ViTE g/1--/44
56.0
54.0
71.1
x
6.8
13.7
21.0
CLIP-VIT-L/1
/
23.2
31.0
43.8
X
16.2
31.1
22.1
VIT-H41
36.4
43.9
40.8
21.4
31.4
26.0
OpenCLIP-ViT-g/11
34.2
41.3
39.4
Table 12: Text-encoder-only zero-shot text classification: We report the clean and adversarial zero shot accuracy & = 1 employing only text-encoders. The adversarial accuracy improves after adversarial finetuning with LEAF. Nevertheless, employing only the text encoder provides worse clean and adversarial performance than employing images labels as Qin et al. 2023
AG-News
SST-2
IMDB
Yelp
Robust
Acc.
Adv.
Acc.
Adv.
Acc.
Adv.
Acc.
Adv.
54.8
17.9
60.3
3.2
54.0
24.9
59.9
29.5
53.5
34.7
58.9
24.1
51.5
44.9
56.7
47.5
32

---

D.3 Additional experiments in text-to-image models
In this section, we provide additional experiments and examples for the text-to-image generation task. In Tables 13 and 14 we present the generation results in SD-1.5 and SDXL in the MS-COCO dataset and the first 5.000 images of the Flickr30k dataset. We measure the CLIPScore between the original caption and the generated image (T-I), the CLIPScore between the original image and the generated one (I-I), the attack objective (Eq. (2) and for SD-1.55, the percentage of generated images triggering the NSFW filter (NSFW %). We can observe that the text encoders finetuned with LEAF, provide a higher generation quality for k > 1 according to all generation metrics. Surprisingly, for k 2 and k- = in the MS-COCO dataset, text encoders triggered the NSFW filter less frequently than SafeCLIP [Poppi et al. 2024], which is specifically designed to avoid generating NSFW content.
In Tables 15 to 18 we present examples of the attacks on the first 10 samples of each dataset for both SD- 1.5 and SDXL at k = 2. We can observe, that our text encoders provide qualitatively better images. The models with the original text encoders, provide images unrelated to the original image and caption more often than the models employing our text encoders.
In Table 19 include the generation results with FLUX.1 1-ev (Black Forest Labs et al. 2025 Since L.UX.I-dev employs CLIP ViT-L/14 and FLAN-T5 XXL [Chung et al.] [022] as text en- coders, the model can only be benefited from our approach by replacing the CLIP encoder with our LEAF counterpart. Similarly, we only attack the CLIP / LEAF text encoders and assume no access FLAN-T5 XXL. Due to the high resolution of the FLUX.I-dev generations (1024 1024), we restrict the evaluation of FLUX.I-dew to the first 100 images in the MS MS-COCO validation set.
D.3.1 Transfer attacks text-to-image models
In this section evaluate the performance of transfer attacks on SD-1.5 with CLIP and LEAF as either the source model where the attack is optimized or the target model used for the image generation. In Table 20 we can observe that, expected, when the source is equal to the target, the generated image quality is degraded the most, Our text encoder improves the generation quality in all cases except when the source is LEAF and k 1, where CLIP obtains 0.04 more CLIPScore T2I score points than LEAF in this advantageous setup.
D.3.2 Preliminary study of typographic attacks
In this section evaluate how our text encoder preserves the image quality under typographic prompts, i.e., prompts where characters have been changed for visually similar ones. To do so, we emply SD-1.5 and replace every "i" for a "I", every "e" for "3", every "o" for "0" and every "a" for an "@" in the first 100 prompts in the MS-COCO dataset. As an example, the first COCO caption turns into "A w0m @ @ds In th3 dlning @r3@ @t th3 1@b13."
In Table 21 we can observe that while the image generation quality with both encoders is quite low, using LEAF provides improvement of 0.62 points in CLIPScore T2I and 2.77 in CLIPScore I2I.
D.4 Embedding inversion examples
In Tables 2 and 3 we present examples from the embedding-to-text reconstructions results per-
formed in Section 4.6
D.5 Additional retrieval experiments
For 1, 000 validation set queries, the attack explained in the main part maximizes the similarity between the test query and a target string using different variants of the Charmer attack. Table 24 we show the individual attack results across target strings for differently trained LEAF models. One sees that on increasing training p. the robustness goes up with a slight decay in the clean retrieval performance. This trade-off is similar to the one seen for classification tasks in Fig. 3
In Fig. 10 we visualize the top-3 retrieved images for the original and the perturbed queries. Al- though in some cases the non robust model retrieves a relevant query, the top-1 retrieved image is always different for clean and perturbed queries. However, the robust model always preserves the original top-1 retrieved image showing its robustness to such character perturbed queries.
33

---

Table 13: Text-to-image generation results MS-COCO: SD-1.5 and SDXL are evaluated the full 5000 images in the valudation set. FLUX. is evaluated over the first 100 images due to the high resolution of the generated images.
Pipeline
Text enender
Sim(fo(S)).fo8"))
CLIPScore T2T
CLIPSSorre I2I
NSFW (%)
CLIP
31.50 (12.87)
73.31
0.64
SafeCLIP
30.96(12.60)
73.27 (0.088)
0.44
LEAF
31.00
10.06
0.46
CLIP
(5.85
27.53
(5.38
0.96
SafcCLIP
(48.22)
(44.0))
(61.00)
0.48
LEAF
86. 86.58(24.8
27. 27.96,
68.01 (8.01 (#11.7)
0.50
SD-1.5
CLIP
33. 38.18
22.96
57. 57.21 (=
2.16
SafeCLIP
50.87;110.94)
(1..50))
(1..0)
1.08
LEAF
73.15 (3..5
25 (5.23(114330 (14.36)
63.40(111100
0.62
CI.IP
20.38 20.38 (s6.988
(9.45(
5 (1.55(=18.400
2.52
SafeCLIP
35.93;+11.06)
20.4 20.41 (#5.61)
(5.98(#12.07)
1.10
LEAF
60.00(29.07)
22. 19.59/as (46.16)) (46.16)
59.02 (#12.19)
1.26
CLIP
(2883
17. (15.2())5.88)
48.34(212.666
2.70
SafeCLIP
26.05 26.05;211000 1.04)
17.94 (7.94 (25.57)
52.31 (1:1,57)
1.56
LEAF
49.35 (19.50)
20.25 (15.44))
55.36 (112.33)
1.44
CLIP + OpenCLIP
31.90(12.84)
71.87
2xLEAF
(1..0
71.78(110060)
CLIP OpenCLIP
67.6 (7.05
28. 28.33 (+4.11)
(4.45(=12.25)
2xLEAF
88.15 88.15(24444)
29.37 (+3.46)
67.25 (x11.664)
SDXL
CLIP OpenCLIP
47 (7.58(18.74)
24 (15.25)
57.97 (1.1288)
2xLEAF
76.49
27. 17.14: (14.33)
63. 63.27 (1:12199
CLIP OpenCLIP
34.22
21.45
(=1.7))
2xLEAF
64.62 (+9.24)
24.69 (+n.16)
59.38 59.38 (+12.66)
CLIP OpenCLIP
25.93
19.07 (1:.800)
(0.92(112.21))
2xLEAF
54.08 1010222
22.45
55. 55.70(112.65
CLIP FLAN-TS XXL
30.56 (22.80)
71.19
LEAF FLAN-TS XXL
30.55 (12.90))
71.18
CLIP FLAN-T5 XXL
16.86
29. 29.14/1u0 (10.76)
68.09
LEAF + FLAN-TS XXL.
87.07 (44.5)
28.90 28.90|
68.
68.79/(+2.1)
CLIP + FLAN-TS XXL.
(5.04
27 27.03 (+00.20)
(3.60)
FLUX.1-dev
LEAF FLAN-TS XXL
73.70, 73.70(26.90) (.6.90)
27.38
(7:38(++4.09) (d:4.09)
(5360))
CLIP FLAN-TS XXL.
21.84(
24.47 (#6.00)
59.40
LEAF FLAN-TS XXL.
59.83, (9.23)
25.71 25.77(446100) (46.16)
62.1 (+13.84)
CLIP FLAN-TS XXL
14.75 (7.11)
22. 22.72 (+6.11)
(7.68)))
LEAF FLAN-TS XXL
49.57
23.51
59.59
34

---

Table 14: Text-to-image generation results on Flickr30k:
Pipeline
Text encoder
(im(
CLIPScore T21
CLIPScore 121
NSFW (%)
CLIP
(=3.21)
(=10.00
0.42
0 SafeCLIP
(23.55)
(0.20
0.42
LEAF
32.63
10.23
0.26
CLIP
(3.40
30.72 30.72(2::110 (2:1.16)
66.43
0.84
3.5
SafeCLIP
77.31 77.31 (#7.11)
(9.32(#44.9)
65.68
0.92
LEAF
89.80 (1:3.8)
30.37 (13356)
67.54 67.54(1100.5)
0.66
CLIP
42.37 (1:10.22)
27.71 (1:5. 18)
61 61.28
1.28
2 SafeCLIP
59. (29.6))
26. 26.24 (22.72)
61.66(=11.12)
0.87
LEAF
28, (1655))
28.43 (=:00
64.66 (=10.80)
0.68
CLIP + OpenCLIP
33.85 33.85 (=3:24)
69.07 69.07
(=10.54)
2xLEAF
33.82
[9.06]
SDXL
CLIP OpenCLIP
75. (5.633)))33)
(1.24(24..00
64..0
2xLEAF
91.32(#3.40)
31.63 (=3.54)
65.87 65.87(210.e0) (=:0.899
CLIP + OpenCLIP
(8.02(28.49)
(8.30
59.09
2xLEAF
82.82 (15.84)
29.83 ((..3(2...09)) (=:4.09)
63.03 (s1.1))
Table 15: Attack examples on MS-COCO with SD-1.5 at kt The color borders indicate null, partial and total matching to the original image caption. The model with the original text encoder provides images involving footballer, lizard or gun, when prompted about bear, a women skiing or group of people respectively. With our text encoders, the generation does not drift in topic so much.
10
Original cuption
Original image
Advenarial caption Generated Image
Original
Adversarial caption Geserated image
Advenaritl cigtion
Genenated image
woman stands the dining wisa sn the table.
A woman stanis in the dising arma at the table-
A wome?? stands in the cining uea at the tablee
in the dining
sace. at the table.
big buty grizely griss backgruand.
burty piizly glnse background
burly arkiazly gias Bedroom
big butfy riiely
backgroand.
Bedroom with bookeass, blue window.
Bedr-com with bookcasee comforttie window.
Bedremmasenne
632
looelcase,
confarte window. stoposign mouned
stop sign mounted spiidtsownn on post.
A stop siggn mounted upwiskkedown on it's post.
stot sign mounted
SP37
724
spsidoddwes
upside- downton it's post. Wtace seddy hears,
R''s poss.
Three soddy benrs
Thrie rodky scurs, color, snugling
Thrie taidy bears
different color, smugeling together.
diffnent color
different color,
smugeling
smaggling
tegeher
tegalher
posing woman
posing for
posing
the camera standing kitchen with refrigerator stove oven cabinets.
standing A kitn with
ing
kihchen with refmtigerzzor
Aqkichhn with refrig-
802
and with cabiness. comppl boathalm player stand- field. male
with cobinets.
withmcabinets. A hasmbell
couple of bowthall stand- ing fiekd.
872
stand of
bowehal stand ing Till.
male playeri white play-
simale white
885 player whine play-
lboets playing
playing
people gomp photo.
people
The
people posing forsa gropp
grop phrieta.
gr cup
35

---

Table 16: Attack examples on MS-COCO with SDXL at k 2:
ID
Original caption
Original image
Adversarial caption Generated image
Original
Adversarial caption Generated image
LEAF
A woman
A woma8 stands in the jining the table.
woman' stands the dining area at the table. big burly !rizzly bear show with krass in background. Bedroom scene with cookcase, blue coumforter window.
139 stunds in the dining
the table.
A big burly grizzly bear is show with grass the background.
A hig burly grlizzly bear is show with rass the background. Bedroom selene with zookcase, blue comforter and window.
285
Bedroom
scene bookcase. blue comforter and window.
632
Stop sign mounted upside-
A sign 724 is mounted upside-down on post.
A stop zign is mountedpupside down it's post.
downton it's post. ahree teddy hears, different color, souggling toge thgerther.
Three teddy bears, each different color, saugeling together. A posing the camera standing on skis.
Three teddy bears, each different color, snuggling toge ther.
776
woman
A worna: posing for the camera stand- ing ontskis.
-oman posing the camera stand- ing onoskis.
785
A with refr iggertrr stove and oven cahinets. A couple of basebill player standing #ield.
kischen
Agkitchen withra refrig- erator, stove und oven with cabinets.
A kitchen with a refrigerator, stove and with cabinets.
802
A couple baseball player stand- ing on field.
A coupll baseball player stand- ing on a qield.
872
a male tennis player in white shorts is play- ing tennis
male tennis pl payer white #horts playing tennis
aemale tennis playerein
885
white shorts
playing tennis
The peo-
The people are posing for group photo.
The neople posing for group hoto.
plecare posing
1000
forza group photo.
D.5.1 Bimodal attacks in text-to-image retrieval
Building on top of text-modality robustness for text-to-image retrieval from the main part, we now assess the robustness to bimodal attacks for both the image and text modalities for 1K samples of the MS-COCO test set. The evaluation starts from the known baseline (A: text perturbations) from Table 3 and applies an untargeted adversarial attack to the images. We use APGD [Croce and Hein 2020 for 100 iterations with small foc perturbation radii of 2/255 and 4/255. This perturba- tion is designed to maximize the distance between the original and perturbed image embeddings, thereby disrupting the model's ability to retrieve the correct text. This attack protocol, is similar to CoAttack [Zhang et al. 2022], where the text attack follows the image attack.
The results in Table 25 highlight the superior resilience of the LEAF-trained models. For the critical recall@1 metric, LEAF improved retrieval performance by nearly 7% over the baseline across both perturbation radii. Importantly, this significant gain in robustness did not come at the cost of clean performance (performance on clean data), as indicated by the 'clean' column results. This finding strongly underscores the importance of dual modality robustness: the ability to maintain high per- formance despite adversarial attacks on either the image or text data, making L.AAF the most robust solution in this challenging setup.
36

---

Table 17: Attack examples on Flickr30k with SD-1.5 at 2;
ID
Original caption
Original image
Adversarial caption
Original
Generated image
Advassrria enppiinraalcgttonn
SafeCLIP
Generatia image
Advessarial caption Geseatted image
LEAF
100099295
Two
young
Twu young
young
Twi young
guys
gigs with
gays shapgychair
guys with
shaggy
shaggi hair
shazgy hii
look their
Jook at their hands while
hands while
hanging out the yard.
hanging out in the Hard.
hanging out in the yard.
haaging out in the mard.
100166944 100100322 (009919630 1000925699 1000066666 100044755 (000269000 10000456
Severa-
Several hardghats
Severalumnn hais openting pulley
operating gi- ane pulley sys-
operat pulley system.
openning pulley system.
A child
child in pink dr climbing set stiis
chald in capank dress clinking of staies
shild in pink dr'ss is clinshing
pink dress
climbing
of stains
of stairs
blue shatt and bat standing stair and leaning against window.
Someone in
Sumeone in bilue shis and hat standing stir leaning against windod:
Someone in bilise shirt and but standing Plair and leaning apainst window.
blue shirt andphat is standing
Someone in
stair and lean- against windoo:
Two one gray
man in grey one black standing near a Itove.
gray standing
gnay shiri, standing
standing near stove..
Two people in the photo
Two people the phoco playnng the other poling
7Twe pellople the photo and pokiig him.
Two people in photocare the suihrr and poking him.
plaping the and the other poking
A man sies chair while holding largo stuffed aninial of hon.
chair whils holding
man siss in
manpsits in chiir while hokinn large stufled aniual of lion.
A man sits in a chuir while holding a lang stanfied animal of hox.
larg sn fied snimal lion.
A girl rollenkctess talking
A god rellenskates
gart mollerskatos milking
Adgul millessinees talking
talking
cellphune standing
her celiphone
hero cellcoeep standing lot. As ssiat man
cellphone standing
standing
pusking
parki
parkingelot.
An asian weizing
Au asim
Anizsian mas wearing
wearing
wearing
black uni
bloack snill
blanck suil stands near dark haired
back soi stands near dark haired ind h-haredd
stans
stnds near
dark haired
dark haired
and
brown hnired
brown haired
100555525
Two men Germany jumping over cail at the same time without shitt.
Twy men Germany jumping over moii in the same time without shirts.
men Germany jumping over a raij sume uase withonk shirs
mon Germany jumping same lihout shins.
37

---

Table 18: Attack examples on Flickr30k with SDXL at l: 2:
ID
Original caption
Original image
Original
LEAF
Adversarial caption
Generated image
Adversarial caption
Generated image
1000092795
Two young
young guys with shaggychair look at their hands while hanging out in the jard..
Two young guys shaggychair look at their hands while hanging the mard.
guys with
shaggy hair
look their
hands while
banging in the yard.
10002456
Several hard hats operating gi- ant pulley sys- tem
Several Sard hats are operating gi- ant !ulley sys-
Several in hardchats are operating giant sulley system.
1000344755 1000268201
child in pink dress is climbing set of stairs in
child a pink dr.ss climbing set stairs entry way.
chwild pink dress is climhing up set stairs ankentry way.
entry way
Someone blue shirt andwhat is standing &tair and lean- ing against window.
Someone blue shirt and hat is standing stair and leaning against window.
Someone in a shirt hat is standing on stair and leaning against iidoww
1000361144
Two men, one in a gray shirt, black shirt, standing near stove.
Twomen one a gray shirt, in black shirt standing near a @Rtove.
Twomen one a gray shirt. one black shirt standing near plove.
1000523639
Two people photo playing the guitar and the other poking him.
Two people phooto playing the gulfitrr and the other is poking him
Two people in photo playing the suitarmend the other poking him.
100091 16330
A man sits in chair while holding a large stuffed animal
A man sits chair while holding large stuffe. animal of aklion.
ch5ir while
holding
large stulffed animal of lion.
10010052
A girl is on roller- skatesstalking on her cell- phone stand- ing a parkingslot.
girl rollerskates talking on her cellphone standing in parking lot.
girl rollerskates talking on her cellphone standing in parki ingllot.
1001465944
An asian man wearing
Axn asian man wearing black suit stands near durk-huired woman and nown-haired woman...
Axn asian man wearing black suit stands neur dark-
black suit stands near dark-huired woman nown-haired
brown-haired woman.
woman.
1001545525
Two men Germany jumping over rail at the same time without shirts.
Two men Germany jumping over a rai7 at the same time "itoout shirts
eWO in Germany jumping over at the time "ithout shirts.
38

---

Table 19: Attack examples on MS-COCO with FLUX.1-dey at 2:
Original
LEAF
Original caption Original image
Adversarial caption Generated image
Adversarial caption
Generated image
A woman
A woma @ stands in the sining the table.
Avwomanastands in the dining area table.
139 stands in the dining area the table.
A big burly grizzly bear is show with grass in the background.
A big burly griezly is show with Trass in the background. Bedr-oom
big burly .rizzly bear show with @rass in the background. Bedroomasoene
285
Bedroom
scene bookease. blue comforter and window.
632
bookease. blue comforiter window.
kookease. blue comforter window. stopssign mounted upside-
A sign 724 is mounted upside-down on it's post.
A stop Sign is mounted upside-down on fit's post.
STOP
downton it's post. 9hree teddy bears, each different color, souggling toge ther.
Three teddy beurs, each different color, saugeling
+hree teddy bears, different color, snuggling @ogether.
776
together.
A woman posing the camera standing on skis.
A woma7 posing for the camera stand- ing onoskis.
-oman posing the camera stand- ing onoskis.
785
A chen with refrig- eratos. stove and oven with cabinets.
Agkitchen withra refrig erator, stove und oven with cabinets.
A kitchen with a refrigerator, stove with cabinets.
802
A player stand- ing on field.
couple
A couple Saseball player stand- on #ield.
A coupl. baseball player stund- ing a Aield.
872
a male tennis player in white shorts is play- ing tennis
male: ten- plaayer
aimale tennis playerein
LPWonmmn
885
white shorts
white shorts
playing tennis
playing tennis
The people are posing for group photo.
The neople ure posing for group ph?oto.
people are posing forza group bhoto.
1000
Table 20: Transfer attacks in SD-1.5: Columns represent the source encoder, where the attack is optimized, and rows the target text encoder, where the attack is evaluated. LEAF obtains the highest CLIPScores for every setup except the CLIPScore T2I with LEAF as a source model.
CLIPScore T2I
CLIPScore I2I
k Target \SSurce
CLIP
LEAF
CLIP
LEAF
1
CLIP
27.53
28. 28.00 (8.00(13.70)
65.38 (+13.72)
66. 73 (11..6)))
LEAF
28.84
27.96
69.47
68.01 68.01(#11.1) (+11.7))
2
CLIP
22.96(45.80)
24.46(+1.86)
57.21 57.21 (+13.90)
60.80(+12.52)
LEAF
26.72(14.22
25. 25.23 (1.23(14.36) (14.36)
66.11(111.88)
63 03.40/ 63.40/111.95) (11.955)
3
CLIP
(95.5
21. (5.44)
51.55
65.68(+12.59)
LEAF
24.61 24.61(15.19 (15..19)
22.59 (11
62.68 (112.57)
59.02 112.19)
4
CLIP
17.42
(75.2(45.68) (55.68
19.10, 49.10
48..4
52.24 + 12.200)
LEAF
22.44 (45.78)
20.25 (15.4
59.25(+11.95) (412.95)
59.25.m
55.36
(+12.33)
39

---

Robust Model: x
Robust Model:
Query: gaggle geese body
Clean
Query: gaggle geesexswim in body of waZer
Query: gagglo geeseuswim in bouy of water
Charmer
Query: little girls ame on the edge boat
Clean
Query: littl) girls are on the edge of ua boat
Query: little girlsvare on the edge vf a boat
Charmer
Query: apple logo burns bright over city bus
Clean
Query: anpap#ple logo burns bright city bus
Query: apple2logo burns bright city bus
Charmer
Figure 10: Visualizing MS-COCO retrieved images. For our ViT-L/14 robust model and it's non- robust counterpart, we show the top-3 retrieved images for the original Query and the perturbed Query via the constrained Charmer (k 2, 10) attack. On average, the robust model is able to preserve the order and retrieves semantically relevant images (esp. top-1) even under perturbation.
40

---

Performance of SD-1.5 under typographic attacks: The generation quality is low with both the original CLIP text encoder and the LEAF counterpart. As reference, the generation quality of SD-1.5 with unperturbed inputs is CLIPScore of 31.50 T2I and 73.31 I2I. However, LEAF is able to attain higher score both T2I and I2I CLIPScore.
Table 21:
Text encoder
CLIPScore T2I
CLIPScore I2I
CLIP LEAF
16. (1-1.6))))33
45.27 (#13.12)
17. 17.41 (11.27)
48.04 (8.00(113.25)
PCA of Imagenet-10 Text Em Embeddings Mean pairwise distance 0.5259
PCA of Imagenet-10 Text Embeddings Mean pairwise distance 0.5065
3.5
64-
22
side nit
badkingg
60 mapp
gitborg
22 apempuhud
mopes
plbbog
porter snoki
garter shebg
lowthopnng
kethopreg
lor
for keeg
-24
-02
00
0.
64
-04
-32
3.0
02
04
(a) openaa/clii-vit-lag----iie 14
(b) LEAF,
k:
PCA Imagenet-10 Text Embeddings Moan pairwise distance 0.3416
PCA of Imagenet Text Emberdingg Mcan jairwise stance 0,2041
garterr arais
-T
rocking chatg
glocog
bassingg
41
mases lamoshodg,
orkoce
-01-
mile Sa.mep Salik
gartaganaks
mocing chaig
21
0/
(c) LEAF, 50, 1
(d) LEAF, 1.
Figure 11: Ablation study on the cause of the clean performance drop in zero-shot classification.
D.6 Ablation studies
In Appendix D.6.1 we evaluate the performance drop in zero-shot image classification when training without semantic constraints. In Appendix D.6.2 we measure the Eq. (TextFARE) loss before and after training.
D.6.1 On the performance drop without semantic constraints
First, we perform an ablation study to better understand the cause of the performance drop in terms of clean accuracy in Table 8 We select 10 classes from the ImageNet dataset and visualize the corresponding text embeddings using the prompt "a photo of a LABEL". Fig. IT we observe that and A increase, the class projections in 2D space become more clustered. We compute the mean pairwise distance, defined as the average 1.2 distance between class pairs, and find that decreases significantly.
41

---

Table 22: Text embedding inversion examples for ViT-H/14. We highlight in red words that are reconstructed by the robust model but not by the clean model; in teal words that are reconstructed by the clean model but not by the robust model; and yellow words that are not reconstructed by either model. The robust model clearly misses fewer words.
Original
Robust
Reconstructed ViT-H/14
public transit alongside vehicle amongst partially road road
and public transit vehicle
on road.
jmotified car and transit vehicle sit on a road )
ugly bathroom demonstrating poorly gross en- vir[U+0042Jkhobbbuttt"
An image of a hotel bath- room that ugly.
ugly hotel bathroom showcasing concerns resemble ?magbbbutto.
older earliest appenhistorical archival picture featur- ing older smaller large kitchen
An older picture of large kitchen with white appliances.
large kitchen pictured prior a looked white appli- ances unidenti).
prepped amina ssels sitting sitting bench near stone textured wall FU1iir1 girl
girl sitting bench front of stone wall.
laghateparth girl twitart bench sitting outside stone wall 7hp."
hchold beautiful windows bein somewhere white- beautifully clean kitchen
clean kitchen with the win- dows white and open.
a a kitchen with windows white wit yet clean
Two women waiting a bench next to street.
i/ Vi/": //, two women waiting bench against street
two women waiting at an street bench ?bbcone
four various computically cubicè compu?their desktoo desk parked
An office cubicle with four differentt types of computers.
office cubic??eczw with four different computers ei- ther
An old victorian style bed frame in bedroom.
old ornate victorian bed showcasing ?wouldfceold )
victorian finomate hed frame placed a bedroom
a sized wildly crafted plane near dramatically dra- matically sun sunlight stripes approaching upward un- derneath
A striped plane flying up into the sky as the sun shines behind it.
striped ??áp plane coming above into sun ?|U+0648]sky.
seemingly domestic cat sits standing among two cars in parking %.
cat in between two cars in parking lot.
cat between two /fou cars docked paved parking lot.
42

---

Table 23: Text embedding inversion examples for ViT-g/14. We highlight reconstructions, highlight in red words that are reconstructed by the robust model but not by the clean model; in teal words that are reconstructed by the clean model but not by the robust model; and in yellow words that are not reconstructed by either model. The robust model clearly misses fewer words.
Original
Robust
Reconstructed ViT-H/14
partially tionally car sits alongside alongside roads public transit vehicle"
A car
and
public transit vehi-
x
cle road.
a car and cachother and a roadway public transit vehi- cle.
apparent nicely tered hotel hathroom containing look- ing ugly pfmage
An image of hotel bath-
room that is ugly.
image of a ugly 55 an hotel bathroom.
large kitchen photographed before that btedly older.
An older picture large kitchen with white appliances.
x
large whil, an kitchen featuring reaswhite appli- ances
girl near stone wall a bench aciantly sitting tedly tedly) ).
sitting on bench in A girl
front
a stone wall.
girl sitting while stone wall sits alongside bench ofnd_of_texxt).
A clean kitchen with the win- dows white and open.
x
view of a white kitchen and nicely clean windows.
clean and white kitchen with windows thwindows..
Two women waiting at bench next
x
along a street bench. two women crouwaited stares.
street.
s// two women wait street while bench outside
office cubicle depicting four various different comput- ers alongside paysoff) )
An office cubicle with four dif- ferent types of computers.
office cubicle containing an workplace with four dif- ferent types computers
cighsundaymotivation throwback@ ?shutterintimacy
An old victorian style bed
x
": victorian bed
frame a bedroom.
victorian style bed frame uas in a bedroom.
nearly seemingly seemingly 00000000 a striped ambi- tious plane being flying into sky with sun light
A
striped plane flying up into
the sky as the sun shines
striped plane being flying over above but shining sun enguliot ung behind
hehind it.
cat sitting through parked parking lot ?) alongside two two cars
A cat in between two cars in parking lot.
X
cat sits in an parking lot between two cars either ).
43

---

Table 24: Detailed retrieval results for & = 2, = 10 constrained attack. This is an extension of Table 3 for the ViT-L/14 model. We show how the robustness changes with changing training across the three target texts.
MS-COCO T->1 retrieval
Train
Clean
Charmer-Con
Model
R@1 1
R@5
R@
R@5
Target: man aggressively kicks stray dog on the street.
non-robust
49.11
73.79
28.88
52.58
LLP--VIT-L/14
49.33
73.98
37.34
62.16
CLIP-ViT-L/IE
49.35
73.73
37.78
62.84
CLIP-ViT-LAA
49.63
73.82
38,66
63.86
CLIP-ViT-L/KE
10
48.99
73.60
40.22
65.30
CLIP-VIT-L/14
20
48.97
73.72
37.92
62.44
CLIP-ViT-L/14
50
48.71
73.72
40.70
66.20
Target: This is an image of a pyramid.
non-robust
49.11
73.79
31.90
54.90
CLIP-ViT-L/1
49.33
73.98
36.30
60.08
CLIP-ViT-L/14
49.35
73.73
39.55
64.65
CLIP- ViT-L/14
49.63
73.82
40.38
65.34
CLIP-ViTE -IP-VVI-L/14
10
48.99
73.60
37.60
62.20
CLIP-ViT-L/14
20
48.97
73.72
40.00
65,46
CLIP-
50
48.71
73.72
41.42
66.66
Target: group
teenagers vandalizes public statue.
non-robust
49.11
73.79
30.68
54.22
CLIP-ViTHA VIT-L/14
49.33
73.98
35.26
59.36
CLIP- ViT-L/14
49.35
73.73
39.29
63.76
49.63
73.82
36.74
61.36
CLIP-ViT-L/IA
48.99
73.60
41.42
65.50
CLIP-ViT-L/4
20
48.97
73.72
41.04
66.12
CLIP-ViT-LIA
50
48.71
73.72
38.56
62.38
Table 25: Bimodal attacks in MS-COCO text-to-image retrieval. Following [Zhang et al.) 2022 we attack the vision-only robust (FARE) and our bimodally robust LEAF models. First we attack the text modality with Charmer-Con (k- 1) and then use APGD with 100 iterations to perturb input images.
Recall @
Recall@5
Method
Clean
255 & ==1
265
Clean
22 ==
255
Original
48.9
17.2
8.9
73.1
35.2
19.7
FARE
49.1
36.6
35.8
73.8
62.2
61.0
LEAF
48.7
43.4
42.8
73.7
67.4
66.9
D.6.2 On the Eq. (TextFARE) loss
In this section, evaluate the effectiveness of our method LEAF in minimizing the loss in Eq. (TextFARE). First, we measure the loss before and after adversarial finetuning the ViT-L/14 scale the first 100 images in the AG-News dataset at & 1. We evaluate the inner max of Eq. (TettFARE) with the LEAF attack with and without semantic constraints (Appendix [.II] and with p 21,2, 10, 20, 50). As baselines, evaluate the same term with the Charmer-20 attack and a Bruteforce approach, which evaluates all of the possible sentences at Levenshtein distance k
In Fig. 12 we can observe that training with LEAF, we generalize to be robust to stronger attacks, if they do not employ semantic constraints. For all cases, employing larger p reduces the
44

---

Table 26: Evaluating the loss in Eq. FARE and Eq. TextFFRRE) across different scales: We evaluate the ViT-L/14, ViT-H/14 and V/T-g/14 with and without our adversarial finetuning (LEAF) in both the image (ImageNet) and text domain (AG-News). Letemn refers to the respective loss when there is no perturbation applied, thus measuring the deviation the original model. Robust models present lower adversarial loss in both domains, with larger models presenting higher loss before and after adversarial finetuning due to the use of larger embedding dimensions.
ImageNet
AG-News
Model
Robust
Telean
Lndv
Lelean
Lddv-cons
I undvuncoos.
ViT-L/14
X
0.0
789.7
0.0
58.4
82.6
VIT-/114
/
33.1
56.4
6.8
23.6
41.7
VIT-H/14
X
0.0
1042.8
0.0
73.4
111.3
ViT-H/14
V
47.9
89.6
13.3
40.7
76.3
ViT-g/14
X
0.0
2172.5
0.0
112.3
175.0
ViT-g/14
/
93.6
181.2
18.8
66.0
121.6
gap between the LEAF estimate and the true inner max of Eq. [TxtFAARE) i.e. Bruteforce. Af- ter adversarial finetuning with LEAF, both the loss estimates with Charmer-20 and Bruteforce are reduced.
the ViT-L/14, ViT-H/14 and ViT-g/14 scales with Charmer-20 before and after adversarial finetuning with LEAF. Similarly, the Charmer-20 loss minimized even no semantic constraints are used the estimate, for all model sizes. The loss is larger for larger model sizes both before and adversarial finetuning. This could be due to the larger embedding dimension for the ViT-H/14 and ViT-g/14 models. Finally, we also evaluate the inner max of Eq. (FARE) in the image domain. To this end, we compute adversarial perturbations for 100 ImageNet images with 100-steps APGD attack on the Eq. (ARE objective at radius e = 2/255. The results are reported in Table 26; similar to the textual attacks, we observe that the loss increases with model size. Importantly, the robust models generally demonstrate much smaller adversarial loss than their original counterparts. These results validate the intuition from Fig. (left): the robust models map perturbed inputs much closer to the original inputs than the original models.
Then, we evaluate the inner max Eq. (TextFARE)
D.6.3 Performance under token-level attacks
In this section, evaluate the performance of our LEAF ViT-L/14, ViT-H/14 and ViT-g/14 models under the TextFooler token-level adversarial attack (Jin et al., 2020]. Furthermore. we replicate the experiment by Abad Rocamora et al. [0224] and finetune BERT-bas base (Devlin et al. 2019 on the SST-2 dataset with our character-level attack to evaluate the character-level and token-level accuracy of the classifier.
Abad Rocamora al. 2024 conclude that token-level defenses are not effective for character-level attacks and vice-versa. In Table
observe that the in line with their results, character-level defenses are not effective for the token-level TextFooler attack.
In Table 27 we present the BERT-base Adversarial Training results. In line with the results of Abbad Rocamora al. 2024], we observe that adversarial training with character-level attacks does not improve the robustness in the token level. Regarding character-level robustness, we observe that LEAF obtains almost points less adversarial accuracy with respect to training with Charmer, but preserves a clean accuracy 4 points higher.
45

---

CLIP Constrained
LEAF Constrained
Brateforc Charnee 20
30
20
Uncomsttranne
100
Bratefore
Charaae-00
50
40
20
Figure 12: Evaluating the loss in Eq. (TextFARE) with different attacks: We evaluate the models in the ViT-L/14 scale on the first 100 sentences the AG-News test dataset, For increasing values of the LEAF attack approximates better the inner max in Eq. (TextFARE). getting closer to the Bruteforce maximum. Our models, trained with LEAF and p 50, reduce the Bruteforce loss, meaning that our models generalize to stronger attacks.
Table 27: Adversarial Training of BERT-base models in SST-2: We report the clean accuracy, character-level (Charmer) adversarial accuracy and token-level (TextFooler) adversarial accuracy.
Method
Acc.
Adv. (Charmer)
Adv. (TextFooler)
Original*
92.43
33.26
4.47
TextGrad
Hou al. 2023
80.94
26.44
23.18
Charmer* Abad Rocamora et al. 2024
87.20
69.46
4.21
LEAF
91.51
64.68
5.50
LEAF-constrained
91.86
62.27
4.13
Numbers from Recamora
2024
average training runs.
Table 28: Token-level adversarial attacks in zero-shot text classification.
report the
TextFooler adversarial accuracy (Adv.) on AG-News and SST-2.
AG-News
SST-2
Model
Robust
Acc.
Adv.
Acc.
Adv.
X
74.4
1.70
71.2
0.57
CLIP-VIT-L/1
/
78.0
1.70
71.9
0.80
X
71.1
1.60
61.6
1.83
OpenCLIP- 4
/
72.3
1.00
58.4
2.98
X
67.3
0.50
57.8
1.83
OpenCLIP-ViT-g14
66.7
1.20
56.0
3.10
46


Reviews:
[{"review_id": "qv3OqNsvK7", "reviewer": "Reviewer_owKp", "summary": "This paper explores the lack of research on adversarial robustness of CLIP models for text encoding, although existing literature mainly focuses on the robustness of CLIP image encoders. The authors propose LEAF, a new adversarial fine-tuning method designed specifically for the text domain. LEAF significantly accelerates the adversarial training process while maintaining robustness by leveraging parallelizable character-level perturbations. It enhances zero-shot adversarial accuracy and robustness on downstream tasks, and achieves interpretability through embedding inversion.", "questions": "see weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper explores an important and relatively unexplored area: adversarial robustness of text encoders in multimodal models, especially in popular architectures such as CLIP, filling a research gap.\n\nThe proposed LEAF adversarial fine-tuning method is clear and supported by comprehensive experiments covering multiple downstream tasks. The experiments clearly demonstrate a significant improvement in the robustness of the CLIP text encoder, showing that LEAF outperforms previous baselines such as Charmer in terms of computational efficiency and performance.\n\nThe paper is well written and clearly structured. The logical logic of each section is clear, and readers can easily understand the main ideas, methods, and experimental results.\n\nWeaknesses:\n\nWhile the paper is generally clear, it is sometimes too cumbersome in methodological details, especially when explaining perturbation methods and constraints. A concise summary or an intuitive explanation of the attack mechanism in simpler terms could further improve readability.\n\nThe practical application of character-level adversarial robustness improvements may oversimplify real-world scenarios. Real-world adversarial scenarios may also involve a wider range of semantic manipulations (e.g., token-level or phrase-level perturbations), so the broader significance of our findings beyond specific applications may remain theoretical.\n\nWhile this paper is innovative in the specific context of text encoder robustness in multimodal models, the conceptual foundations of adversarial training (character-level attacks, optimization methods) have been well explored in the prior literature. Furthermore, given the rapid development of multimodal base models, limiting the evaluation and discussion to CLIP-based architectures may limit the broad applicability of our findings.", "was_revised": true, "final_justification": "This paper is well-written in terms of topic selection, experiments, and writing. The author also addressed my concerns in the subsequent discussion. I think this is a good manuscript.", "rebuttal": "Dear Reviewer owKp,\n\nThanks for your time reviewing our paper. We appreciate your recognition of the clarity and comprehensiveness of our writing and experiments. We summarize and address your comments as follows:\n\n## P1: “A concise summary or an intuitive explanation of the attack mechanism in simpler terms could further improve readability.”\n\nThank you for bringing up this point. Our attack is described in Algorithms 1 and 2 in the appendix. Additionally, Fig. 2 portrays a practical example using $\\rho=6$. Let us provide an intuition of the overall procedure:\n\n1. Sample $\\rho$ positions within the sentence.\n2. Replace/Insert a whitespace in those positions.\n3. Evaluate the loss of the model on the $\\rho$ sentences and select the one with the highest loss.\n4. Sample $\\rho$ characters from the vocabulary.\n5. Replace the sampled characters in the position selected in step 3.\n6. Evaluate the loss of the model on the $\\rho$ sentences and select the one with the highest loss.\n7. Go to step 1 until $k$ perturbations are done.\n\nNote that this is the most basic version of our attack: without constraints. To constrain the attack, we filter out the changes producing new words after steps 2 and 5. We hope this intuition eases the understanding of our method. We will clarify the mechanism of our attack in the revised version of the manuscript.\n\n## P2: “Real-world adversarial scenarios may also involve a wider range of semantic manipulations (e.g., token-level or phrase-level perturbations”\n\nWhile token-level attacks can also be realistic, recent studies show that token-level attacks often change the semantics of the sentence, leading to invalid attacks [1,2,3]. We highlight this motivation in lines 94-96. Moreover, **[4] show that token-level defenses are not effective for character-level attacks and vice-versa.** Following these studies, we decided to tackle character-level robustness. \n\nFor completeness, we have evaluated the performance of our CLIP models in zero-shot text classification using the token-level TextFooler attack [5]. \n\nTable A: Adversarial accuracy in AG-News and SST-2 under the TextFooler attack.\n|            |               | AG-News         | SST-2           |\n|------------|---------------|-----------------|-----------------|\n| Model size | Model         | TextFooler Adv. | TextFooler Adv. |\n| L          | CLIP | 1.70            | 0.57            |\n|            | LEAF          | 1.70            | 0.80            |\n| H          | OpenCLIP | 1.60            | 1.83            |\n|            | LEAF          | 1.00            | 2.98            |\n| g          | OpenCLIP | 0.50            | 1.83            |\n|            | LEAF          | 1.20            | 3.10            |\n\nOur results are in line with the findings of [4]: Character-level defenses are not effective for token-level adversarial attacks. We hope that our work can foster additional studies that enhance the robustness of CLIP under different threat models, e.g. token-level or syntactic attacks.\n\n## P3: The experimental evaluation is limited to CLIP-based models\n\nThanks for highlighting this point. We believe our evaluation in CLIP-based models is well-representative of the multimodality field. While our training method shows promise for other applications / models where robustness in the text domain is needed, CLIP-like architectures are widely employed in many applications as described in lines 16-21. To complete our analysis with additional architectures, we fine-tune BERT-base on SST-2 for 1 epoch with LEAF. We measure the Charmer adversarial accuracy at $k=1$ and the token-level adversarial accuracy with TextFooler. This is equivalent to the experimental setup in Table 4 in [4].\n\nTable B: Adversarial finetuning of BERT-base on SST-2.\n| Method           | Acc.  | Adv. (Charmer) | Adv. (TextFooler) |\n|------------------|-------|----------------|-------------------|\n| Original         | **92.43** | 33.26          | 4.47              |\n| TextGrad         | 80.94 | 26.44          | **23.18**             |\n| Charmer          | 87.20 | **69.46**          | 4.21              |\n| LEAF             | 91.51 | 64.68          | 5.50              |\n| LEAF-constrained | 91.86 | 62.27          | 4.13              |\n\nIn line with the results of [4], we observe that adversarial training with character-level attacks does not improve the robustness in the token level. Regarding character-level robustness, we observe that LEAF obtains almost 5 points less in adversarial accuracy with respect to training with Charmer, but preserves a clean accuracy 4 points higher.\n\nWe appreciate your comments, which help us improve the quality of our work. If you have additional comments, we will be happy to answer. If our answers addressed your points, we would appreciate an increase in the rating.\n\nBest regards,\n\nAuthors\n\n**References**\n\n[1] John Morris et al., Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020.\n\n[2] Salijona Dyrmishi et al., How do humans perceive adversarial text? a reality check on the validity and naturalness of word-based adversarial attacks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long\nPapers), 2023.\n\n[3] Bairu Hou et al., Textgrad: Advancing robustness evaluation in NLP by gradient-driven optimization. In International Conference on Learning Representations (ICLR), 2023.\n\n[4] Elias Abad Rocamora et al., Revisiting Character-level Adversarial Attacks for Language Models, International Conference on Machine Learning, 2024.\n\n[5] Di Jin et al., Is bert really robust? a strong baseline for natural language attack on text classification and entailment. AAAI Conference on Artificial Intelligence, 2020."}, {"review_id": "jeF7lXVte6", "reviewer": "Reviewer_fLS5", "summary": "This paper investigates the overlooked vulnerability of CLIP's text encoder to adversarial perturbations and introduces LEAF (Levenshtein Efficient Adversarial Finetuning), a novel method to improve its robustness. While prior efforts have focused on strengthening the image encoder, this work shows that even small character-level changes in text can drastically shift CLIP’s embeddings and degrade downstream performance. LEAF enhances robustness by training the text encoder to produce stable embeddings under controlled character-level perturbations, guided by semantic constraints to preserve sentence meaning. The method is highly efficient, significantly faster than previous adversarial training approaches, and scales well to large models. Experiments demonstrate that LEAF improves adversarial accuracy in zero-shot text classification, enhances robustness in text-to-image retrieval and generation, and enables more faithful reconstruction of text from embeddings. Overall, the paper highlights the importance of robust text encoders for multi-modal models and provides a practical solution to strengthen CLIP’s resilience in both the image and text domains.", "questions": "See the above weakness.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strehgths\n* One key strength of this paper lies in its novel focus on the robustness of the CLIP text encoder, an area largely neglected in previous research. By introducing the LEAF method, the authors provide an efficient and scalable solution that significantly improves adversarial robustness with minimal impact on clean performance. LEAF’s design allows it to integrate seamlessly into existing pipelines without the need to retrain the entire model, making it practical for real-world deployment. \n* Furthermore, the use of character-level perturbations with semantic constraints ensures that the adversarial training remains realistic and meaningful. The paper also demonstrates the broad applicability of the proposed method across multiple tasks—such as zero-shot classification, text-to-image generation, and retrieval—highlighting the general utility and effectiveness of the approach.\n\n## Weaknesses\n* One notable weakness of the paper is that the adversarial robustness is only explored in the character-level perturbation setting, while token-level attacks—which may better reflect real-world adversarial scenarios—are not addressed due to their potential to alter semantics. \n* Additionally, the robust image and text encoders are fine-tuned independently, without considering joint multimodal adversarial attacks, which may limit the model’s effectiveness against coordinated cross-modal perturbations.", "was_revised": true, "final_justification": "Thanks for your reply. Considering your reply and other reviewers' comments, I tend to maintain my current positive score. Good luck!", "rebuttal": "Dear Reviewer fLS5,\n\nThanks for your time reviewing our paper. We appreciate your appraisal of the practicality and applicability of our training method in CLIP-like models. We answer your points regarding the weaknesses of our work as follows:\n\n## P1: “token-level attacks—which may better reflect real-world adversarial scenarios—are not addressed due to their potential to alter semantics”\n\nThanks for highlighting this point. We agree that token-level attacks are prone to altering semantics, this is the reason why we decided to focus on character-level attacks. Moreover, **[2] show that token-level defenses are not effective for character-level adversarial attacks and vice-versa.**\n\nWe are unsure about which of the two threat models better captures real world scenarios. Both character-level and token-level attacks can be deemed realistic, the first simulates typos/small changes in the input and the second simulates replacement by synonyms. To complete the analysis, we have evaluated the performance of our CLIP models in zero-shot text classification using the token-level TextFooler attack [1]. \n\nTable A: Adversarial accuracy in AG-News and SST-2 under the TextFooler attack.\n|            |               | AG-News         | SST-2           |\n|------------|---------------|-----------------|-----------------|\n| Model size | Model         | TextFooler Adv. | TextFooler Adv. |\n| L          | CLIP | 1.70            | 0.57            |\n|            | LEAF          | 1.70            | 0.80            |\n| H          | OpenCLIP | 1.60            | 1.83            |\n|            | LEAF          | 1.00            | 2.98            |\n| g          | OpenCLIP | 0.50            | 1.83            |\n|            | LEAF          | 1.20            | 3.10            |\n\nOur results are in line with the findings of [2]: Character-level defenses are not effective for token-level adversarial attacks.\n\n## P2: Your evaluation does not consider joint multimodal adversarial attacks\n\nAs a proof of concept, we have tested the performance of our models when both the image and text are perturbed in text-to-image retrieval.\n \nStarting with the k=1 text perturbations for the ViT-L/14 models from Table 3 in the main paper, we perturb the images with **APGD for 100 iterations with $\\epsilon= 2/255$ and $\\epsilon=4/255$**. We show text-to-image retrieval on 1k samples from MS-COCO. Specifically, we maximize the embedding distance of the image encoder under perturbation to the original image (can be interpreted as an untargeted adversarial attack). The attack follows a similar setup as CoAttack [3], where the text attack follows the image attack. The clean column here represents the performance of the model when no modality is attacked. In this setup again, our LEAF trained models are the most robust, highlighting the importance of dual modality robustness.\n\n\nTable B: BiModal attack robustness evaluation for MS-COCO retrieval at $k=1$ and $\\epsilon\\in \\{2, \\}4/255$.\n|       | Clean | $\\epsilon=\\frac{2}{255}$ | $\\epsilon=\\frac{4}{255}$ | Clean | $\\epsilon=\\frac{2}{255}$ | $\\epsilon=\\frac{4}{255}$ |\n|------------------|-------|----------------|-------------------|-------|----------------|-------------------|\n| Method  | Recall@1 | | | Recall@5 | | |\n| Original         | 48.9 | 17.2 | 8.9  | 73.1 | 35.2 | 19.7 |\n| FARE           |   49.1| 36.6 | 35.8 | 73.8| 62.2 | 61.0 |\n| LEAF            | 48.7 | 43.4 | 42.8 | 73.7 | 67.4 | 66.9 |  \n\nThanks for your review. We remain available in case you have additional comments, we will be happy to answer.\n\nBest regards,\n\nAuthors.\n\n**References**\n\n[1] Di Jin et al., Is bert really robust? a strong baseline for natural language attack on text classification and entailment. AAAI Conference on Artificial Intelligence, 2020.\n\n[2] Elias Abad Rocamora et al., Revisiting Character-level Adversarial Attacks for Language Models, International Conference on Machine Learning, 2024.\n\n[3] Jiaming Zhang et al., Towards adversarial attack on vision-language pre-training models., ACM MM2022"}, {"review_id": "2INOPQv9T6", "reviewer": "Reviewer_qWrN", "summary": "This paper extends the FARE objective—originally developed for adversarial robustness in CLIP image encoders—to the text domain, introducing LEAF: an efficient character-level adversarial finetuning method for CLIP text encoders. By sampling a limited number of perturbations per batch, LEAF achieves comparable robustness to Charmer while being significantly faster. Extensive evaluation spans zero-shot text classification (AG-News), multimodal retrieval, text-to-image generation (using SD-1.5 and SDXL), and embedding inversion, showing substantial robustness gains with minimal clean performance degradation.", "questions": "- Given the growing importance of text precision in T2I models, have you evaluated LEAF’s effect on intended typographic prompts, such as rare text content (“a ST00P sign”)? Does enforcing adversarial robustness inadvertently harm those cases?\n- In Table 14-16, were the same adversarial captions used for baseline and LEAF models? If they were generated independently, can you clarify how fairness in comparison is maintained?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n-  Implements an elegant, easy-to-reuse adversarial objective with a fast sampling attack (LEAF), achieving strong robustness with ~5–10× speedup over Charmer.\n- Applies analyses across diverse tasks—classification, retrieval, generation, inversion—demonstrating robustness gains throughout.\n- Validates LEAF on large backbones (ViT-H/14, ViT-g/14) effectively.\n\nWeaknesses\n- The core contribution is adapting FARE to text, with the primary novelty being a faster attack; the methodological lift remains incremental. The approach handles only character-level perturbations; broader attacks (token-level, syntactic, semantic) remain unexplored.\n- Text and image encoders are finetuned separately, leaving effectiveness under joint attacks untested.\n- For text-to-image evaluations, only SD-1.5 and SDXL are tested. Stronger, state-of-the-art generation architectures such as Stable Diffusion 3 (SD3) or Flux.1 are not evaluated, limiting insight into broader applicability\n- Appendix Table 14-16 suggests that adversarial captions differ across models when comparing baselines. If baseline and robust models are evaluated on different perturbed prompts, the comparison may be unfair.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer qWrN,\n\nWe appreciate the time dedicated to reviewing our paper. We are happy that you describe our method (LEAF) as elegant and easy-to-reuse, and that you find our evaluation diverse and effective. We address your concerns in the following:\n\n## P1: “The core contribution is adapting FARE to text, with the primary novelty being a faster attack; the methodological lift remains incremental.”\n\nThe methodological lift includes a new efficient training-time attack and a principled constrained objective for the discrete text modality. Moreover, our study is the first demonstration of dual-domain robustness in CLIP. Our experiments reveal enhanced robustness to character-level attacks in several tasks and improved interpretability of robust models via embedding inversion. \n\n## P2: “The approach handles only character-level perturbations; broader attacks (token-level, syntactic, semantic) remain unexplored.”\n\nRobustness in the text domain can be studied through many threat models. As mentioned in lines 94-97, token-level attacks often alter the semantics of the sentence [1,2,3]. Moreover, **[8] show that token-level defenses are not effective against character-level attacks and vice-versa.** To keep our insights clear, we decided to focus on character-level attacks. Nevertheless, for completeness, we have evaluated the zero-shot text adversarial accuracy of our CLIP models, with the Textfooler token-level adversarial attack [7] in the AG-News and SST-2 datasets.\n\nTable A: Adversarial accuracy in AG-News and SST-2 under the TextFooler attack.\n|            |               | AG-News         | SST-2           |\n|------------|---------------|-----------------|-----------------|\n| Model size | Model         | TextFooler Adv. | TextFooler Adv. |\n| L          | CLIP | 1.70            | 0.57            |\n|            | LEAF          | 1.70            | 0.80            |\n| H          | OpenCLIP | 1.60            | 1.83            |\n|            | LEAF          | 1.00            | 2.98            |\n| g          | OpenCLIP | 0.50            | 1.83            |\n|            | LEAF          | 1.20            | 3.10            |\n\nOur results are in line with the findings of [8]: Character-level defenses are not effective for token-level adversarial attacks. We hope that our work can foster additional studies that enhance the robustness of CLIP under different threat models, e.g. token-level or syntactic attacks.\n\n## P3: “Text and image encoders are finetuned separately, leaving effectiveness under joint attacks untested.”\n\nAs a proof of concept, we have tested the performance of our models when both the image and text are perturbed in text-to-image retrieval.\n \nStarting with the k=1 text perturbations for the ViT-L/14 models from Table 3 in the main paper, we perturb the images with **APGD for 100 iterations with $\\epsilon=2/255$ and $\\epsilon = 4/255$**. We show text-to-image retrieval on 1k samples from MS-COCO. Specifically, we maximize the distance of the image embeddings under perturbations to the original image. The attack follows a similar setup as CoAttack [6], where the text attack follows the image attack. The clean column here represents the performance of the model when no modality is attacked. In this setup again, our LEAF trained models are the most robust, highlighting the importance of dual modality robustness.\n\nTable B: BiModal attack robustness evaluation for MS-COCO retrieval at $k=1$ and $\\epsilon\\in \\{2, \\}4/255$.\n|       | Clean | $\\epsilon=\\frac{2}{255}$ | $\\epsilon=\\frac{4}{255}$ | Clean | $\\epsilon=\\frac{2}{255}$ | $\\epsilon=\\frac{4}{255}$ |\n|------------------|-------|----------------|-------------------|-------|----------------|-------------------|\n| Method  | Recall@1 | | | Recall@5 | | |\n| Original         | 48.9 | 17.2 | 8.9  | 73.1 | 35.2 | 19.7 |\n| FARE           |   49.1| 36.6 | 35.8 | 73.8| 62.2 | 61.0 |\n| LEAF            | 48.7 | 43.4 | 42.8 | 73.7 | 67.4 | 66.9 |  \n\n## P4: “Stronger, state-of-the-art generation architectures such as Stable Diffusion 3 (SD3) or Flux.1 are not evaluated”\n\nModels like SD3 or FLUX.1 cannot be fully benefited from our approach, since in addition to CLIP text encoders, they employ the encoder-decoder model T5-XXL for obtaining text embeddings. Nevertheless, it is still possible to attack their CLIP text encoders and replace them for our fine-tuned ones, while keeping T5-XXL untouched. We have replicated our text-to-image generation experiments with FLUX.1-dev. Due to the high quality of the generated images (1024x1024), we limit our evaluation to the first 100 images in the MS-COCO dataset. \n\nTable C: Text-to-image and image-to-image CLIP scores for the FLUX.1-dev model with the original (CLIP) and finetuned (LEAF) text encoder.\n| k | Text encoder | CLIPScore T2I  | CLIPScore I2I |\n|---|--------------|----------------|---------------|\n| 0 | CLIP | **30.56**          | **71.19**         |\n|  | LEAF | 30.55          | 71.18         |\n| 1 | CLIP | **29.14**          | 68.09         |\n|  | LEAF | 28.90          | **68.79**         |\n| 2 | CLIP | 27.03          | 63.60         |\n|  | LEAF | **27.38**          | **65.66**         |\n| 3 | CLIP | 24.47          | 59.40         |\n|  | LEAF | **25.71**          | **62.11**         |\n| 4 | CLIP | 22.72          | 57.68         |\n|  | LEAF | **23.51**          | **59.59**         |\n\nOur results show that even when only attacking one text encoder, FLUX.1-dev is still vulnerable to adversarial attacks. When replacing the CLIP-ViT-L text encoder with our LEAF counterpart, the image generation quality is improved for higher values of $k$. Unfortunately, we cannot include image links with our rebuttal. We will include attack examples and generations in the revised version of the manuscript.\n\nWe hope our work can motivate the study of the robustness of other architectures like encoder-decoder models, such as T5-XXL.\n\n## P5: “In Table 14-16, were the same adversarial captions used for baseline and LEAF models? If they were generated independently, can you clarify how fairness in comparison is maintained?”\n\nFairness relies on the adversarial captions being optimized independently for every model from the same original caption. This procedure yields the adversarial perturbation that maximally degrades the performance of each model. This is a standard practice both in the image and text adversarial robustness communities [4,5]. Attacks optimized over one model and applied to others can also be tested, these are known as transfer attacks. We have tested the adversarial captions optimized over CLIP on LEAF and vice versa for the SD1.5 model.\n\nTable D: Transfer attacks for SD-1.5. We measure the text-to-image CLIPScore. Rows denote target models and columns denote source models.\n| k | Target/Source | CLIP  | LEAF  |\n|---|---------------|-------|-------|\n| 1 | CLIP| 27.53 | 28.00 |\n|   | LEAF| 28.84 | 27.96 |\n| 2 | CLIP| 22.96 | 24.46 |\n|   | LEAF| 26.72 | 25.23 |\n| 3 | CLIP| 19.45 | 21.30 |\n|   | LEAF| 24.61 | 22.59 |\n| 4 | CLIP| 17.42 | 19.10 |\n|   | LEAF| 22.44 | 20.25 |\n\nWe observe that, as expected, when the source is equal to the target, the generated image quality is degraded the most. Our text encoder improves the generation quality in all cases except when the Source is LEAF and $k=1$, where CLIP obtains 0.04 points more than LEAF in this advantageous setup.\n\n## P6: “have you evaluated LEAF’s effect on intended typographic prompts, such as rare text content (“a ST00P sign”)? Does enforcing adversarial robustness inadvertently harm those cases?”\n\nThanks for the suggestion. As a preliminary test, we have crafted 100 perturbations from the first 100 COCO captions. To take it to the extreme, we have replaced every “i” for a “1”, every “e” for a “3”, every “o” for a “0” and every “a” for an “@”.\n\nFor example, the first COCO caption turns into “A w0m@n st@nds 1n th3 d1n1ng @r3@ @t th3 t@bl3.”\n\nTable E: SD1.5 text-to-image generation performance under visually-similar character replacements.\n| Text encoder | CLIPScore T2I | CLIPScore I2I |\n|--------------|---------------|---------------|\n| CLIP| 16.79| 45.27|\n| LEAF| **17.41**| **48.04** |\n\nWe observe that while the image generation quality with both encoders is quite low, using LEAF provides an improvement of 0.62 points in CLIPScore T2I and 2.77 in CLIPScore I2I.\n\nThanks again for your review and suggestions. The added experiments will be included in the revised version of our manuscript. We remain available in case you have additional comments. If you are satisfied with our answers, we would appreciate an increase in the rating.\n\nBest regards,\n\nAuthors\n\n**References**\n\n[1] John Morris et al., Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020.\n\n[2] Salijona Dyrmishi et al., How do humans perceive adversarial text? a reality check on the validity and naturalness of word-based adversarial attacks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long\nPapers), 2023.\n\n[3] Bairu Hou et al., Textgrad: Advancing robustness evaluation in NLP by gradient-driven optimization. In International Conference on Learning Representations (ICLR), 2023.\n\n[4] Francesco Croce, RobustBench: a standardized adversarial robustness benchmark, arXiv preprint arXiv:2010.09670, 2020.\n\n[5] John Morris et al., TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, EMNLP 2020.\n\n[6] Jiaming Zhang et al., Towards adversarial attack on vision-language pre-training models., ACM MM2022\n\n[7] Di Jin et al., Is bert really robust? a strong baseline for natural language attack on text classification and entailment. AAAI Conference on Artificial Intelligence, 2020.\n\n[8] Elias Abad Rocamora et al., Revisiting Character-level Adversarial Attacks for Language Models, International Conference on Machine Learning, 2024."}, {"review_id": "b17jU2hPQh", "reviewer": "Reviewer_qiLf", "summary": "The paper addresses the adversarial robustness of CLIP's text encoder, a relatively unexplored area compared to the image encoder. The authors propose LEAF, an efficient adversarial finetuning method. The core idea is to finetune the text encoder to produce embeddings for adversarially perturbed text (via character-level changes) that remain close in L2 distance to the embeddings of the original, clean text. The contributions are demonstrated across several downstream tasks, showing that this approach improves robustness in zero-shot classification, text-to-image retrieval, and the quality of text-to-image generation models under textual adversarial attack.", "questions": "The primary concern is the scope of the evaluation. Could the authors comment on the feasibility of extending their evaluation to a wider suite of zero-shot classification benchmarks commonly used for CLIP (e.g. VTAB, https://github.com/LAION-AI/CLIP_benchmark, etc.)? Showing consistent gains across a more diverse set of data distributions would significantly strengthen the paper's claims.\n\nThe paper focuses on character-level attacks. Have the authors considered how LEAF might perform against token-level attacks, which aim to preserve semantics while changing the embedding? Even a preliminary experiment or a discussion on why LEAF might or might not be suited for this threat model would be insightful.\n\nRegarding the semantic constraints used during training: The results in Figure 3 suggest these are critical to prevent performance degradation on the image task. Could you elaborate on their necessity and how sensitive the model's performance is to the strictness of these rules? Does this imply that for new domains or languages, new hand-crafted constraints would be required?", "limitations": "Yes, the authors commendably discuss limitations such as the decoupled training paradigm (and the lack of defense against joint attacks) and the focus on character-level attacks. The discussion is fair and accurate. To further improve, it could be beneficial to also explicitly frame the limited set of evaluation benchmarks and model scales as a scope limitation of the current study in the limitations section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Significance and Originality: The paper tackles a clear, important, and largely overlooked problem. Ensuring the robustness of the text encoder is crucial for the reliability of the many downstream models that leverage CLIP. This work is one of the first to provide a systematic study and solution in this domain.\n- Methodological Quality: The proposed method, LEAF, is well-motivated and pragmatic. It is designed for computational efficiency, which is a significant advantage in the context of adversarial training. The decoupled training approach, focusing only on the text encoder with a simple L2 objective, makes the method practical and scalable.\n- Experimental Breadth and Clarity: The authors demonstrate the value of their approach across three distinct and important use cases of CLIP (classification, retrieval, generation). This multi-faceted evaluation makes the practical benefits of the work very clear. The paper is exceptionally well-written, with figures (e.g., Fig. 1 and Fig. 5) that effectively communicate the core ideas and results.\n\nWeaknesses:\n- Limited Evaluation Scope: This is the primary weakness. While the experiments are of good quality, their scope is narrow. For zero-shot classification, robustness is only shown on a couple of datasets. A standard CLIP evaluation would involve a much wider suite of benchmarks (e.g. VTAB etc.), which would be necessary to make a stronger claim about general robustness. The evaluation on different model scales could also be more extensive.\n\n- Constrained Threat Model: The work focuses exclusively on character-level attacks. While this is an important threat, it overlooks other potential vulnerabilities, such as token-level attacks (e.g., replacing words with synonyms). The robustness of the proposed method against these other attack types is unknown.\n\n- Reliance on Heuristics: The method's success without degrading vision performance relies on \"semantic constraints\" during training (i.e., ensuring perturbations do not form new valid words). This indicates that some degree of task-specific, hand-crafted rules are needed, making the solution slightly less general than a fully automated approach.", "was_revised": true, "final_justification": "The paper makes a clear and well-supported contribution to the underexplored area of CLIP text encoder robustness. The rebuttal satisfactorily addressed my main concerns with additional experiments and clarifications. While some scope limitations remain, the strengths outweigh them, so I am maintaining my score.", "rebuttal": "Dear Reviewer qiLf, \n\nThanks for your time reviewing our paper. We are happy you found our paper exceptionally well written. We answer to your comments as follows:\n\n## P1: “A standard CLIP evaluation would involve a much wider suite of benchmarks (e.g. VTAB etc.)”\n\nWe agree that CLIP models should be evaluated on a wide suite of tasks. Thank you for suggesting the VTAB benchmark. We report in Table 9 (appendix) the performance of our models and baselines across several zero-shot classification tasks, many of which are part of the VTAB benchmark. Additionally, we have evaluated the models on the remaining VTAB subtasks and report averaged performance over the categories “natural”, “specialized”, and “structured” in Table A below. We observe that in clean evaluation, robust models sacrifice performance on “natural” and “specialized” (a trade-off between clean and robust performance is expected [1]). On “structured” the behavior is mixed - sometimes even outperforming the non-robust models. In the adversarial evaluation ($\\epsilon=\\frac{2}{255}$), we observe that the non-robust models are completely vulnerable, while our robust models maintain much better performance when attacked. We will add these interesting results to the appendix of the manuscript.\n\nTable A: Clean and adversarial evaluation on VTAB.\n\n| attack    | model             | robust | natural | specialized | structured |\n|:----------|:------------------|:-------|--------:|------------:|-----------:|\n| clean     | CLIP-ViT-L/14     | ❌      |    74.4 |        63.5 |       11.9 |\n|           |      | ✅      |    68.5 |        41.9 |       13.3 |\n|           | OpenCLIP-ViT-H/14 | ❌      |    78.7 |          57.0 |       11.7 |\n|           |  | ✅      |    74.8 |        45.6 |       11.8 |\n|           | OpenCLIP-ViT-g/14 | ❌      |    79.5 |        62.9 |       12.5 |\n|           |  | ✅      |    72.4 |        51.4 |       11.4 |\n| $\\epsilon=\\frac{2}{255}$ | CLIP-ViT-L/14     | ❌      |       0.0 |           0.0 |          0.0 |\n|           |  | ✅      |    42.4 |        10.6 |        3.9 |\n|           | OpenCLIP-ViT-H/14 | ❌      |     0.1 |           0.0 |          0.0 |\n|           |  | ✅      |    44.9 |        14.6 |        3.6 |\n|           | OpenCLIP-ViT-g/14 | ❌      |       0.0 |           0.0 |          0.0 |\n|           |  | ✅      |      41.0 |         9.5 |        1.9 |\n\n\n\n## P2: “The evaluation on different model scales could also be more extensive”\n\nPlease note that Tables 2,3,4 and 9 include evaluations in all ViT-L, H and g sizes. We even finetuned the text encoder of the ViT-bigG 3B parameter model and evaluated it on text-to-image generation. We agree that it would be nice to evaluate models larger than this, but our compute budget does not allow for larger sizes, this is indeed a limitation as we mention in lines 283-284.\n\n## P3: “Have the authors considered how LEAF might perform against token-level attacks, which aim to preserve semantics while changing the embedding?”\n\nWhile token-level attacks can also be realistic, recent studies show that token-level attacks often change the semantics of the sentence, leading to invalid attacks [2,3,4]. Moreover, **[5] show that token-level defenses are not effective for character-level adversarial attacks and vice-versa.** For completeness, we have evaluated the performance of our CLIP models in zero-shot text classification, under the token-level TextFooler attack [6]. \n\nTable B: Adversarial accuracy in AG-News and SST-2 under the TextFooler attack.\n|            |               | AG-News         | SST-2           |\n|------------|---------------|-----------------|-----------------|\n| Model size | Model         | TextFooler Adv. | TextFooler Adv. |\n| L          | CLIP | 1.70            | 0.57            |\n|            | LEAF          | 1.70            | 0.80            |\n| H          | OpenCLIP | 1.60            | 1.83            |\n|            | LEAF          | 1.00            | 2.98            |\n| g          | OpenCLIP | 0.50            | 1.83            |\n|            | LEAF          | 1.20            | 3.10            |\n\nOur results are in line with the findings of [5]: Character-level defenses are not effective for token-level adversarial attacks.\n\n## P4: Could you elaborate on the need for semantic constraints during training? If a new language/task is employed, should the constraints change?\n\nThanks for bringing up this very interesting point. We believe that the semantic constraints we employ are general enough and should not change from task to task. In fact, the same constraints used during training provided a good performance in a wide range of tasks (text classification, text-to-image retrieval, text-to-image generation and text embedding inversion). Moreover, in Fig. 11, we report the loss with and without constraints. Training with constraints, also generalizes to reduce the unconstrained loss.\n\nThe case of changing languages is an interesting area for future work. Since the constraints we employ during training are based on the English dictionary, the performance on datasets in other languages might not be so good. However, since the model is able to generalize and be robust to unconstrained attacks, we believe that robustness will (to some extent) generalize from language to language.\n\nThanks again for your suggestions. We will incorporate the new experiments and discussions in the revised version of the manuscript. If you have further comments, we will be happy to answer. If our comments addressed your concerns, we would appreciate a raise in the rating.\n\nBest regards,\n\nAuthors.\n\n\n**References**\n\n[1] Dimitris Tsipras et al., Robustness May Be at Odds with Accuracy, ICLR 2019\n\n[2] John Morris et al., Reevaluating adversarial examples in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020.\n\n[3] Salijona Dyrmishi et al., How do humans perceive adversarial text? a reality check on the validity and naturalness of word-based adversarial attacks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long\nPapers), 2023.\n\n[4] Bairu Hou et al., Textgrad: Advancing robustness evaluation in NLP by gradient-driven optimization. In International Conference on Learning Representations (ICLR), 2023.\n\n[5] Elias Abad Rocamora et al., Revisiting Character-level Adversarial Attacks for Language Models, International Conference on Machine Learning, 2024.\n\n[6] Di Jin et al., Is bert really robust? a strong baseline for natural language attack on text classification and entailment. AAAI Conference on Artificial Intelligence, 2020."}]

Author rebuttal:

Meta-review: This paper introduces LEAF, an efficient adversarial fine-tuning method that uses parallelizable character-level perturbations to enhance the robustness of CLIP's text encoder. The authors demonstrate that LEAF improves zero-shot adversarial accuracy and maintains robustness on downstream tasks.

The primary strength is addressing the important and under-explored area of adversarial robustness for text encoders in vision-language models. The proposed LEAF method is clear, well-motivated, and backed by comprehensive experiments showing significant gains in both robustness and computational efficiency over baselines.

Initial concerns from reviewers were largely resolved during the rebuttal. One remaining point involved a request for a non-standard experiment, which could not be completed as the reviewer did not provide sufficient details despite the authors' request for clarification.

The strengths of the work outweigh the minor, unresolved point from an unresponsive reviewer.
Decision: Accept (poster)