CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification
Authors: Mingkun Zhang
Keping Bi
Wei Chen
Jiafeng Guo
Xueqi Cheng
Venue: ICLR 2025 Poster (2025)
Keywords: adversarial robustness
purification
CLIP
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: In this paper, we aim to build an adversarially robust zero-shot image classifier that can accurately and efficiently classify unseen examples while defending against unforeseen adversarial attacks, addressing critical challenges in real-world safety-sensitive scenarios. To achieve this, we focus on two key challenges: zero-shot classification and defense against unforeseen attacks. We ground our work on CLIP, a vision-language pre-trained model to perform zero-shot classification. 
To defend against unforeseen attacks, we adopt a purification approach, as it is independent of specific attack types. 
We then define a purification risk as the KL divergence between the joint distributions of the purification and attack process. 
The derived lower bound of purification risk inspires us to explore purification in CLIP's multi-modal latent space. 
We propose a CLIP-based purification method called CLIPure, which has two variants: _CLIPure-Diff_, which models image likelihood with a generative process of its latent vector, and _CLIPure-Cos_, which models the likelihood based on the similarity between embeddings of the image and a blank template. As far as we know, CLIPure is the first purification method in latent space and _CLIPure-Cos_ is the first purification method not relying on generative models, substantially improving defense efficiency. Extensive experimental results show that the robustness achieved by CLIPure is within a small gap of clean accuracy, outperforming SOTA robustness by a large margin, e.g., from 71.7\% to **91.1\%** on CIFAR10, from 59.6\% to **72.6\%** on ImageNet, and **108\%** relative improvements of average robustness on the 13 datasets over previous SOTA, with only 14\% extra inference cost and no additional training.

Published as a conference paper at ICLR 2025
CLIPURE: PURIFICATION IN LATENT SPACE VIA CLIP FOR ADVERSARIALLY ROBUST ZERO-SHOT CLASSIFICATION
Mingkun Zhang 1.33 Keping Bi2.3., Wei Chen Jiaffng Guo Xueqi Cheng State Key Lab of Al Safety, CAS Key Lab of Al Safety, Institute of Computing Technology. Chinese Academy of Sciences, Beijing, China 2CAS Key Laboratory Network Data Science and Technology,
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Beijing, China changmiingkun22, bikeping, chenwei2022, guojiafeng, exg)eict.ac. cn
ABSTRACT
In this paper, we aim build an adversarially robust zero-shot image classifier. We ground our work on CLIP, vision- language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts "a photo of a <class-name>>? ". Purification is the path we choose since it does not re- quire adversarial training on specific attack types and thus can cope with any fore- seen attacks. We then formulate purification risk the KL divergence between the joint distributions the purification process of denoising the adversarial sam- ples and the attack process of adding perturbations to benign samples, through bidirectional Stochastic Differential Equations (SDEs). The final derived results inspire
explore purification in the multi-modal latent space of CLIP. We propose two variants for our CLIPure approach: CLIPure-Diff which models the likelihood of images' latent vectors with the DiffusionPrior module in DaLLE- 2 (modeling the generation process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and "a photo of a.". As far as we know, CLIPure is the first pu- rification method in multi-modal latent space and CLIPure-Cor is the first purifi- cation method that is not based on generative models, which substantially im- proves defense efficiency. We conducted extensive experiments on CIFAR-10. 10, ImageNet, and 13 datasets that previous CLIP- defense methods used for evaluating zero-shot classification robustness. Results show that CLIPure boosts the SOTA robustness by large margin, e.g., from 71.7% to 91.1% % on CIFARIO. 10, from 59.6% to 72.6 % on ImageNet, and 108% relative improvements of aver- age robustness on the 13 datasets over previous SOTA. The code is available http:/hhww...oooeeeeeeeeeeeennnn
INTRODUCTION
Image classifiers are usually trained in a supervised manner with training data and evaluated on the corresponding test data until recently several vision-language models have emerged as zero-shot classifiers (i et al. 2023 Radford et al. 2021 Li et al. 2022). Among them, CLIP (Radford et al.) 2021) is example that is popular, effective, and efficient. CLIP performs zero-shot classification by forming text prompts "a photo of a <elass-name>." of all the candidate categories, and selecting the class with the highest similarity with the image embedding. Despite its efficacy, when facing adversarial attacks, its accuracy can drop to zero, similarly vulnerable to other neural classifiers.
Existing methods to enhance adversarial robustness follow two primary paths: adversarial training and purification. Adversarial Training (AT) (Madry et al. 2017 Rebuffi et al.) 2021 Wang et al.,
2023 incorporates adversarial examples into model training to boost robustness. It often achieves
corresponding authors

---

Published as a conference paper at ICLR 2025
advanced performance in defending against the same attacks while failing to defend against unseen attacks (Chen al., ) FARE (Schlarmann et al.] 2024) and TeCoA (Ma et al.) (022) are two AT approaches integrated with CLIP, which enhance CLIP's zero-shot classification robustness while harming clean accuracy significantly and do not generalize to other types of at- tacks. Adversarial purification (Song et al. 2017) seeks to climinate adversarial perturbations by optimizing samples to align them with the distribution of benign samples. Instead of adversarial samples, this approach often requires generative model that can model the probability of benign samples. It can handle unfore- seen attacks but often performs worse than AT methods Figure 1: Adversarial robustness of two on seen attacks and has lower inference efficiency. In CLIPure versions versus adversarially adversarially robust trained CLIP models, evaluated against
Callech 00:1
STL10 99.4
Cars 32.2
Pets 91.0
CIFAR1O 91.1
PCAM 52.0
CIFARI
ImageNel-S
52.4
84.6
EuraSAT
ImageNel-R
48.4
24.8 Flowers
26 FGVC
TeCoA
FARE
CUPure-Diff
CUrure-Cos
this paper, we aim to produce zero-shot classifier, so we opt for integrating purifica- AutoAttack with foo = 4/255 across 13 tion with CLIP. To explore better purification method, zero-shot classification datasets. we formalize the purification risk and theoretically
analyze what may affect purification performance. Concretely, inspired by [Song et al. (2220) that models the diffusion process with bidirectional Stochastic Differential Equations (SDEs) (Anderson (1982)), we model the process of attack (i.e., adding perturbations to benign examples) with a for- ward SDE and purification (i.e., denoising adversarial examples) with a reverse SDE. Within this framework, we evaluate the risk of purification methods by measuring the KL divergence between the joint distributions of the purification and attack steps. After some derivation, we find that purifi- cation risk is related to 1) the negative KL divergence of the probability distributions of adversarial and benign examples (i.e.. KKL(p( (p( (a(vadv) ||pp (ro(Ztea))) ))), and 2) the & norm of the gradients of adver- sarial samples' probability distribution (i.e., V log (d )) This indicates that purification risk can be affected by: 1) the differences between p( and p (bee): 2) the smoothness of p aav) and possibly its dimension.
To the best of our knowledge, existing adversarial purification methods are all conducted in pixel space. Given the above factors that can affect purification risk, it is natural to ask: are there purifica- tion approaches better than in pixel spuce? ? As we know, pixel space is high-dimensional and sparse while latent embedding space is denser and smoother. Moreover, multi-modal latent representations theoretically proven to have better quality than uni-modal (Huang et al. (2021). CLIP, a vision- language- encoder model, has shown the superiority of its multi-modal embeddings on many tasks (Radford et al. 2021). Accordingly, we propose conduct purification in CLIP's latent space for adversarially robust sero classification.
Our method CLIPure has two variants that model the likelihood of images' latent vectors with CLIP differently: 1) CLIPure-Diff: a generative version that employs the DiffusionPrior module of DaLLE-2 (Ramesh al., 2022) model the likelihood of image embedding with diffu- sion model; 2) CLIPure-Coss: a discriminative version that models the likelihood with the cosine similarity between the image embedding and text embedding of blank template "a photo of a ... Compared to likelihood modeling in pixel space and uni-modal latent space, we find that both our methods have several orders of magnitude larger KL divergence between the distributions of adver- sarial and benign samples than the former, and significantly higher than the latter (shown in Figure 2). Remarkably, CLIPure-Cor becomes the first purification method that does not rely on generative models and thus boosts defense efficiency by two or three orders of magnitude (See Table 3)
Since CLIP aligns image and text embeddings by their cosine similarities (Radford et al. 2021 vector lengths are not important to reflect the vector relationship in CLIP's latent space. Hence, during likelihood maximization in CLIPure, we normalize the latent vectors to unit vectors to di- minish the effect of vector length. This is critical to our approach, our experiments indicate that the purification process could be obstructed by vector magnitude and ultimately fail.
We compare the robustness of CLIPure and SOTA methods against the strongest adaptive attacks, AutoAttack(Croce & Hein 2020). with different la or EnD bounds on various image classifica- tion tasks, including the popular CIFAR10, ImageNet, and 13 other datasets (e.g., CIFAR100) ImagetNet-R) that evaluate zero-shot classification robustness in FARE (Schlarmann et al. 2024)
2

---

Published as a conference paper at ICLR 2025
and TeCoA (Mao et al. 2022). Note that CLIPure always conducts zero-shot classification and defense without the need for any dataset-specine training while the basclines can be any methods that are current SOTA. We are delighted to see that CLIPure boosts the SOTA robustness on all the datasets by a large margin, e.g., from 71.7% to 91.1 % on CIFARIO when 6cc = 8/255, from 59.6% 72.6% on ImageNet when foo 4/255. CLIPure achieves 45.9% and 108% relative improve- ments over previous SOTA- FARE(Schlarmann et al., 2024) regarding average robustness across the 13 zero-shot test datasets facing AttoAt with &o 2/255 and 4/255, depicted in Figure I Our work shows that purification in multi-modal latent space is promising for zero-shot adversarial robustness, shedding light on future research including but not limited image classification.
RELATED WORK
Zero-Shot Image Classification. Unlike traditional models that are limited predefined categories, vision-language models (VLMs) are trained on open-vocabulary data and align the embeddings of images and their captions into a common semantic space. This enables them perform as zero-shot classifiers by matching the semantics of images to textual categories, offering superior generality and flexibility. CLIP (Radford et al,, 202T), trained on extensive internet image-text pairs, achieves advanced results in zero-shot classification tasks. Additionally, other VLMs including Stable Diffusion (Rombach et al.) 2022), Imagen (Saharia et al., 2022), and DaLLE-2 (Ramesh)
et al. 2022
also possess zero-shot classification capabilities (Li et al. 2023 Clark & Jaini
2024).
Adversarial Purification in Pixel Space. A prevalent paradigm of adversarial purification aims to maximize the log-like likeliidl of samples to remove perturbations in pixel space. Since purification has no assumption of the attack type, enabling it to defend against unseen attacks using pre- pre-trained generative models such PixeICNN (Song et al. (017) GANs (Samangouei) 2018), VAEs (Li &
2020). Energy- -based models (Hill et al.) 2020 Yoon et al. (021), and Diffusion Models (Nie et al,,
2022 Chen al. 2023; Zhang et al. 2025). Owing to the capability of diffusion models, diffusion-
based adversarial purification achieves state-of-the-art robustness among these techniques.
CLIP-based Defense. While CLIP achieves impressive accuracy in zero-shot classification, it re- mains vulnerable to imperceptible perturbations (Fort |021] Mao et al. (2022). Adversarially train- ing the CLIP model on ImageNet/ / Tinn- ImageNet (Schlarmann et a 2024 Mao al.) [0222] Wang et al. 2024) enhances its robustness but undermines its zero-shot capabilities and struggles against unseen attacks. [Coi et al. (2025) suggests smoothing techniques for certification. [ et al. (2024a) advocates using robust prompts for image classification, but the defensive effectiveness is limited. Additionally, other research focuses on the -uttof-distribution (OOD) robustness of the CLIP model
Tu al. 2024; Galindo & Faria), which is orthogonal to our adversarial defense objectives.
3 PRELIMINARY: CLIP AS ZERO-SHOT CLASSIFIER
In this section, we introduce how CLIP is trained and how CLIP acts as a zero-shot classifier. CLIP (Contrastive Language Image Pre-training) (Radford et al. 2021 consists of an image encoder Enc" and text encoder Enc". It is trained 400 million image-tex pairs from the internet, aiming to align image embeddings with their corresponding text captions through contrastive learning:
exp(cos(2)) 2/, =1)/T) + log exp(cos Z 21./TT)
LCLIP = 2N log
(1)
Σm= exp(cos(2z) 2/,, //T)
EM= exp(cos( i 21, 3
where N represents the number of image-caption pairs, z = Enc" (imagen) and 2 = Enc' (text,,) are the embeddings of the n-th image and text respectively, is temperature parameter, and cos(:, denotes the cosine similarity function.
This aligmment enables CLIP to perform zero-shot classification by matching image embeddings with text embeddings of a template "a photo of a <class-name>."" where <elass-name>> iterates all the possible classes of a dataset. Without loss of generality, given an image, let z' denote its CLIP-encoded image embedding, and 인 be the text embedding of a possible class description, i.e., x = Enc" ("a photo of a class c."). The predicted class ÿ is determined by:
9 = arg max cos (z', =
(2)
3

---

Published as a conference paper at ICLR 2025
For enhanced classification stability, as in Radford et al. (2021)
80 templates of diverse
descriptions in combination with class names, such as "a good photo <class- namla>.. In our experiments, each class c's embedding, x. in Eq. 2 is the average text embedding of e paired with all the templates.
CLIPURE: ADVERSARIAL PURIFICATION IN LATENT SPACE VIA CLIP
In this section, we outline the methodology of CLIPure, focusing adversarial purification within CLIP's latent space. We first define purification risk through a Stochastic Differential Equa- tion (SDE) perspective and derive its lower bound Section 4.22 the rationale for CLIPure to potentially achieve smaller purification risk and two variants of modeling sample likelihood. In Section we propose normalizing latent vectors to diminish the effect of vector length during purification to align with CLIP's latent space modeled using cosine similarity.
4.1 ADVERSARIAL PURIFICATION RISK
Considering that adversarial attacks progressively add perturbations an image while purification gradually removing noise to restore the original image, we formulate both the attack and purification processes through the lens of Stochastic Differential Equations (SDEs). This framework allows us to propose measure of purification risk based on the divergence between the attack and purification processes, providing insights into what affects purification effectiveness.
We formulate the attack process as transformation from the benign distribution Phen (aen(a)) to an adversarial example distribution Padv (x) by an attack algorithm. Note that for simplicity we use p(x) to represent Poen (z) in this paper. Take untargeted PGD-attack (Madry et al. 2017) for instance, the adversarial attack behavior on benign sample Dg can be described as:
dæ asiign C æz, yu + odw,, To p(x), s.t., | perr-xolp-- e, (3)
where a represents the attack step size, p(z) denotes the distribution of benign samples, [(0; Zz. Vime) denotes the loss of I classified by the model with parameters 0 as the ground truth category Vine at attack step (where / E [0, T)), duc, denotes the Wiener process (Brownian mo- tion). The constant serves scaling factor for the noise component and the adversarial example ær is bounded by e in fo norm.
The corresponding reverse-time SDE (Anderson adversarial example distribution Padv to the purified sample distribution Ppure. and is expressed as:
1982) of Eq. 3 describes the process from the
dx Josign Ve C(B; Virue)
o² v log p(x;) + adw; ET ~ Pudv
(4)
classifier guidance
purification
where logp(x;; represents the log-likelihood æe concerning the distribution of clean samples, analogous to the score function described in Score SDE (Song al. 2020), dwū; represents the reverse-time Wiener process. A detailed discussion on the form of the reverse- time SDE can be found in Appendix A
Note that in the reverse-time SDE, t progresses from T to O, implying that dt is negative. Accord- ing to Eq. 4 the reverse SDE aims increase the sample's log-likelihood while simultaneously decreasing the loss of classifying æ to Yine- In Eq. 4 the purification term related to the common objective of adversarial purification Ppure arg max log p(æ). The classifier guidance term has been employed to enhance purification (Zhang et al. 20224). and their objective aligns well with Eq. We will incorporate this guidance term with CLiPure Appendix
D.3 and see its impact.
Then, we define the joint distribution of the attack process described by the forward SDE in Eq. as Po.r = p(ann = oben 21 21,..., 2T xadx) E (T++)xxd where each æe E TR... For the pu- rification process defined by the reverse-time SDE in Eq. Q0.r p( p Ii,.... By D.dv). ). Here, Øben. Ladv. and Tpure denotes the benign sample, adversarial example, and purified sample respectively. We define the purification risk R () by the KL divergence between the reverse SDE (corresponding to the purification process) and the joint distribution of forward SDE (representing the attack process):
denote the joint distribution as
((Q)= = ( | = KL(Go,r| Porr) + Eo.r IKL(2i:T110.r11PP:
(5)
2 KL(Co.T||Po.)
4

---

Published as a conference paper at ICLR 2025
clean adv
clean
dean
clean
Density
KL 1.9
3x100
KL m 9-2100
-log pix)
log
log
(a) Pixel Space
(b) Vision Latent Space
(c) Multi-modallDiff)
(d) Multi-modal(Css)
Figure 2: Negative log-likelihood estimated by diffusion models on (a) pixel space via EDM, (b) uni-modal latent space via VQVAE, (c) multi-modal latent space via DiffusionPrior,, and (d) multi- modal latent space via CLIP (using cosine similarity for log-likelihood estimation). KL represents the value of (L(p(x (pp(@ten)) discussed in Section 4.2 indicating the difference between clean and adversarial example distribution.
Then, we focus solely on the purified example, i.c. KL Qo. (Po.) rather than the entire purifi- cation trajectory. The forward SDE in Eq. describes the transformation from p(*ben) to p( enabling us to obtain the conditional probability p( (Den). Simultaneously. the reverse SDE in Eq. supports the transformation from p(z.ud) back to p( helping us quantify p(zpure ( Then we can derive that:
R(Q)= > KL KK, (Po.r) (p Fady) (Deen Taav])
(6)
- Emr.
[V (ogat()))))
) log a att Ac] KL(p(Pady ||pp (Xhhe))
2
where At denotes small time interval for attack and purification, related to the perturbation mag- nitude. detailed proof of the result is provided in Appendix B
The result in Eq. 6 highlights that the lower bound of the purification risk is influenced by two factors: 1) the smoothness the log-likelihood function at adversarial examples and possibly the sample dimension, indicated by the l norm of V log p(#aar). 2) the differences between the likelihood of clean and adversarial samples in the benign example space.
4.2 ADVERSARIAL PURIFICATION IN CLIP'S LATENT SPACE
In this section, we further explore how to achieve a smaller purification risk R(Q).. Considering Ex... V log p( p( log p(Zadv) At within R(Q) in Eq. 6 standard pixel space purification may lead to higher purification risks due its sparsity and possibly peaked gradient distribution high dimensionality. Thus, we are curious to investigate purification in latent space where the distribution of sample densities more uniform and smoother.
(L(p( (aav)) (|pp(ehe)) )) in Eq. 6 implies representations that excel at detecting out-of-distribution ad- versarial examples are likely to carry lower risk of purification errors. Huang al. (202T) suggest that mudti-modal latent spaces offer superior quality compared to uni-modal counterparts, Inspired by this observation, CLIP's well-aligned multi-modal latent space, where image embeddings are guided by the semantics of finer-graines words in an open vocabulary, may provide foundation for purification with lower risk.
To validate the efficacy of different spaces in modeling sample likelihood for adversarial purifica- tion. we focus on the term (L(p(==a>)) ((ppp//aa)lllp(/(p/aa) (p((be))) the lower bound of the purification risk R Q). As illustrated in Figure 2 we extract 512 samples from ImageNet (Deng et al., 2009) and gener- ate adversarial examples by AutoAttack (Croce Hein| 2020) on the CLIIP (Radford et al. 202T)
classifier. We explore four types of likelihood modeling approaches:
In Pixel Space: Sample likelihood of the joint distribution image pixels is estimated by a gener- ative model (we use an advanced diffusion model - [EDM et al. (022)) ). Figure 2a indicates
that even EDM struggles to distinguish between clean and adversarial sample distributions at the
pixel level. We use the Evidence Lower Bound (ELBO) to estimate log- likelihood, expressed as
log pa x)) >
Er,, [| x,, E e|]]] + C. where e ~ (O I), and C' is typically negligible (Ho
al. 2020 Song et al. 2020
5

---

Published as a conference paper at ICLR 2025
Algorithm CLIPure: Adversarial Purification in Latent Space via CLIP
An off off-the-shelf CLIP model including an image encoder Enc' and text encoder Enc', textual embedding of the blank templates (without class names, e.g., photo of a.), purification step N, step size 1/, and DiffusionPrior model from DaLLE-2 (for generative version). Input: Latent embedding of imput example x'
Required:
Output: label
for i to N. do
step1: Obtain latent embedding in polar coordinates
direction 2 /|2 magnitude 2²/u
step2: Estimate log-likelihood
CLIPure-Diff via DiffusionPrior co
CLIPure-Cos via CLIP
sample (0
log (ogp(z") t cos(22, #*)
log p(2)() -lco =²) c12
step3: Update latent variable
u f- u + (llog(z')) 0z'
8y t r.
step4: Classification based on purified embedding
predict label across candidate categories according to Eq. end for return predicted label
The likelihood of the joint distribution of an image embedding uni- modal space is estimated with the VQVAE component of the Stable Diffusion (SD) model (Rombach) al., [202]]. Note that although SD multi-modal, its VQVAF component is trained solely on image data and keeps frozen while training the other parameters, making it vision-only generative model of latent vectors. Compared to Figure 2a Figure 2b demonstrates improved capability in distinguishing clean and adversarial sample distributions.
In Vision Latent Space:
In Vision-Language Latent Space: For CLIP's latent space, we present two approaches to estimate the likelihodd of image embeddings, i.e., log p(2'):
(1) Diffusion-bassd Likelihood Estimation (corresponds to CLiPure-Diff) The DiffusionPrior module in DaLLE--2 (based on CLIP) (Ramesh al. 2022) models the generative process of im- age embeddings conditioned on text embeddings. Employing DiffusionPrior, we estimate the log- likelihood log (= by conditioning on blank template "a photo of a
log (z(") x log æ²) - -E,,, (2)
+C,
where e0(2", (x "))))) the UNet (Ronneberger et al. 2015) in DiffusionPrior (Ramesh et al. 2022). parameterized by 0, with the noised image embedding x; timestep & under the condition of text embedding zf, and C' is a constant typically considered negligible (Ho et al. 2020).
(2) Cosine Similarity-based Likelihood Estimation (corresponds to CLiPure-Cosi) We estimate log (2(") 12')br by computing the cosine similarity between image embedding x² and the blank template's text embedding:
(8) Note that by modeling likelihood without using generative models, the defense efficiency can be significantly boosted. The inference time of CLIPure-Cos is only 1.14 times of the vanilla CLIP for zero-shot classification, shown in Table
log (og")(()) x cos(z',=')
Figure 2c and Figure 2d|show that CLIPure-DDin and CLIPure-Cor have several orders larger magni- tude KL divergence between clean and adversarial samples than modeling likelihood in pixel space. Modeling likelihood in uni-modal latent space also leads to larger KI. divergence than pixel space but smaller than multi-modal space. indicates that purification in CL.PP's latent space promis- ing have lower purification risk and enhance adversarial robustness.
4.3 ADVERSARIAL PURIFICATION BASED ON NORMALIZED UNIT VECTORS
Typically, purification in pixel space is conducted through gradient ascent on the sample æ by using the derivative of the log-likelihood log p(x): x a log p(ar), where a denotes the step size

---

Published as a conference paper at ICLR 2025
for updates. However, given that the cosine similarities between vectors in CLIP's latent space the criterion for alignment where vector lengths do not take effect, it is inappropriate to directly apply the typical purification update manner to this space. Thus, normalize the image vectors to unit vectors at each purification step to diminish the effect of vector length. Specifically, we first normalize the vector z' Enc' (x)), obtained from the CLIP model image encoder for an input sample (potentially adversarial sample), to unit vector = x' ||2 Then calculate the sample's s log-likeliiee likelihood log p(z') and compute the gradient Iu by the chain rule:
Ologp(z') Plogp(z)) gu du
Oz
(9)
Oz
du
This gradient Iu is then used to update the direction x' for adversarial purification, detailed in Algorithm ! Note that CLIPure based on the original CLIP and does not need any extra training.
We also attempted adversarial purification by directly optimizing vectors instead of the normalized version. We experimented extensively with various steps. parameters, and momentum- based meth- ods, but found it challenging to achieve robustness over 10% on ImageNet, indicating that it is difficult find an effective purification path with vector lengths taking effect.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets.
Following the RobustBench (Croce & Hein 2020) settings, we assess robustness on
CIFAR-10 and ImageNet. To compare against CLIP-based zero-shot classifiers with adversarial
training (Schlarmann et al. 2024 Mao et al.) 2022). we conduct additional tests across 13 image classification datasets (detailed in Appendix C.2p. In line with [Schlarmann al. (2024). we ran- domly sampled 1000 examples from the test set for our evaluations.
Baselines. We evaluate the performance of pixel space purification strategies employing genera- tive models, including Purify-EBM (Hill et al. 2020) and ADP YYonn et al.) 2021) based on Energy- Based Models; DiffPure based on Score SDE (Nie et al.] (002)) and LafiPure-DaLLL.a D
based Decoder of DaLLE2 (Ramesh et al. 2022); GDMP based on DDPM (Ho et al. 2020); and likelihood maximization approaches such as LM-EDM (Chen et al. 2023 based on the EDM model (Karras et al., (022) and www.DDaLL..DDcott.eee
which adapts LM to the Decoder of DaLLE- Furthermore, we perform an ablation study adapting LM to the latent diffusion model to achieve uni-modal latent space purification using the Stable Diffusion Model (Rombach al. 2022 de- noted as L.M-StableDiffusion. Furthermore, we also compare with the sfate-of-the-art ad adversarial training methods such as AT-ConyNeXI-L (Singh al. 2024) and AT-Swin-L (Liu et al. 2024),
along with innovative training approaches such as Mixed NUTS (Bai et al.] (024) and MeanSquare (Amini et al., 2024), and methods that utilize DDPM and EDM to generate adversarial samples for training dataset expansion: AT-DDPM (Rebuffi et al. (021)) and AT-EDM (Wang et al.) 2023).
Moreover, we also compare CLIPure with adversarially trained CLIP model according to TeCoA (Mao et al. (022) and FARE (Schlarmann et al., (022)). Additionally, we also evaluate the perfor- mance of classifiers without defense strategies, including CLIP, WideResNet (WRN), and Stable Diffusion. Due to space constraints, we list the details of baselines in Section C.4
in Appendix
Adversarial Attack. Following the setup used by FARE (Schlarmann et al. 2024). we employ AutoAttack's (Croce & Hein 2020) strongest woite-box APGD for both targeted and untargeted attacks across 100 iterations, focusing on an l. threat model (typically e = 4/255 or E = 8/255) 0.5) for evaluation. We also evaluate CLIPure against the BPDA (short for Backward Pass Differentiable Approximation) with EOT (Expectation Over Transformation) (Hill et al. 2020) and latent-based attack (Rombach et al. pendix C.3
well as &g threat model (typically
2022) detailed in Ap-
5.2
MAIN RESULTS
In this section, we compare CLIPure- Diff and CLIPure-Cos with SOTA methods and examine the model robustness from various perspectives.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of performance against AutoAttack under loo = 8/ 3/255) and le (E 0.5) threat model on CIFAR-ID dataset, showcasing various defense methods including adversarial train- ing and purification mechanisms. We highlight defenses that operate pixel or latent space ("de- fense space"), and indicates the modal information used in defense strategies with "V" for Vision and "V-L" for Vision-Language multimodal representations. best robustness for baselines, and bold font to denote the state-of-the-art (SOTA) across all methods.
use underlining to highlight
Defense Space
Latent Modality
Clean Aec (%)
Robust Ace (%)
Method
fx = 8/255 &
WRN-28-10 Zagoruyko
2016
94.8
0.0
0,0
w/o
StableDiffusion (Li ct al.
2023
V
87.8
0.0
38.8
Defense
CLIP (Radford et al. 2021
V-L.
95.2
0.0
0.0
AT-DDPM-L2 (Rebufh al.
202
Pixel
V
93.2
49.4
81.1
Adv.
AT-DDPM- (Rebuth et al.
2022
Pixel
V
88.8
63.3
64.7
Train
AT-EDM-C2 Wang et 20233
Pixel
V
95.9
53.3
84.8
AT-EDM- (Wang al.
2023
Pixel
V
93.4
70.9
69.7
TETRA (Blau et al. 2023)
Pixel
V
88.2
72.0
75.9
Other
RDC (Chen al. (023)
Pixel
V
89.9
75.7
82.0
LM StableDiffusse
Latent
V
37.9
6.9
8.6
DiffPure (Nie et al. 2022
Pixel
V
90.1
71.3
80.6
Purify
LM EDM (Chnn et al. 2023)
Pixel
V
87.9
71.7
75.0
Our CLIPure Dill
Latent
V-L
95.2
88.0
91.3
Our CLIPure Cos
Latent
V-L
95.6
91.1
91.9
Discussion of CLIPure-Diff and CLIPure-Cos. We compare the performance CLIPure-Diff and CLiPure-Cos against AutoAttack across CIFAR-10, ImageNet, and 13 datasets in Tables and respectively, as well as defense against BPDA+EOT and latent-based attack in Table 4 and Appendix Results indicate that both models have achieved new SOTA performance on all the datasets and CLIPure-CCs uniformly outperforms CLIPure- Diff in clean accuracy and robustness. probably because the DiffusionPrior component used in CLIPure- Diff models the generation process by adding noise to the original image embeddings encoded by CLIP without diminishing the effect of vector magnitude. Specifically, the noise added as z Vāzū + vT âé, e No0 I)- We expect that the performance will be boosted if the generation process also eliminates the effect of vector length. In contrast, CLIPure-CC has no such issue and it models the likelihood with cosine similarities that are consistent with CLIP.
Comparisons with Purification in Pixel Space. Compared to the SOTA purification methods DiffPure (Nie et al.) (202)) and LM-EDM (Chen et (023) (the likelihood maximization approach based on the advanced diffusion model EDM (Karras et al. 2022)), our CLIPure achieves better adversarial robustness (shown in Table [ and 2) as well superior inference efficiency (shown in Table 3). Table D shows that on CIFAR10 10 under the foo and f threat models, our CLIPure- Cos showed improvements of 27. % and 22.5% over LM-EDM and improvements of 27.8% and 14.0% over DiffPure. On the ImageNet dataset, shown in Table 2 CLIPure-Cos achieves relative increment of 288.2% over LM-EDM and 63.5% over DiffPure. Additionally, the inference efficiency of our CLIPure is multiple orders of magnitude higher than DiffPure and LM-EDM (see Table 3).
Moreover, in Table 2] we also compared the pixel space purification based on the Decoder com- Results show that the LM-
ponent of DaI.LE-2 (Ramesh et al.] 2022) Decder sufters from drop accuracy, possibly because the diffusion architecture ADM (Dhariwal & Nichol 2021) it employs is slightly inferior to EDM in terms of generation quality.
Comparisons with Purification Uni-modal Latent Space. We evaluate latent space purification using Stable Diffusion (Rombach et al. 2022) (i.e., L.M-StableDiffusion) on the CIFAR-10 dataset, with results detailed in Table [| As discussed in Section 4.2) Stable Diffusion (SD) models an uni-modal image latent space through its VQVAE (Rombach al., 2022). Direct comparisons are challenging due to SD and CLIP utilizing different training data and fundamentally distinct backbones (diffusion model versus discriminative model). As noted in Li et al. (203). SD, as
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison of defense methods on ImageNet against AutoAttack with e threat model (c 4/255). Indicates whether the methods use ImageNet training set for training as zero-shot. "V" stands for Vision, and "V-L" for Vision- Linnun-La multimodal representations.
Defense Space
Latent
Zero -Shot
Clean Acc (%)
Robust Ace (%)
Method
Modality
w/o
WideResNet-50 (Zagoruyko
2016
V
x
76.5
0.0
Defense
CLIP (Radford et al. 2021
V-L
74.9
0.0
FARE (Schlarmann et al.
2024
Latent
V-L
70.4
33.3
Adv.
TeCoA (Mao et al. 2022
Latent
V-L
75.2
44.3
Train
AT-ConwNeeI-LL Singh al.
2024
Pixel
V
77.0
57.7
AT-Swin-L (Lu et al. (2024)
Pixel
V
78.9
59.6
MixedNUTS Bai et al. 2024)
Pixel
V
81.5
58.6
Others
MeanSparse (Amini et al.
2024
Pixel
V
78.0
59.6
LM D.LLLE2.eDDeddee
Pixel
V-L
36.9
9.2
DiffPure Decoder
Pixel
V-L
31.2
9.0
Purify
L. - EDM (Chen et al. 2023
Pixel
V
X
69.7
18.7
DiffPure (Nie al.
Pixel
V
71.2
44.4
Our CLIPure Dilf
Latent
V-L
73.1
65.0
Our CLIPure Cos
Latent
V-L
76.3
72.6
an generative model, is designed for generation rather than classification tasks, so its zero-shot classification performance is not good enough, which limits its potential for robustness.
Comparisons with CLiP-based Baselines. Figure D and Table show the zero-shot adversarial robustness on 13 datasets, compared to CLIP-based baselines enhanced with adversarial training. i.e. FARE (Schlarmann et al., 2024) and TeCoA (Mao et al., 2022), CLIPure-Diff and CLIPure-Cos surpass their best-reported average robustness by significant margins, 39.4% and 45.9% when lx = 2/255, 95.4% and 108% when the attacks stronger with &o = 4/255. FARE and TeCoA often have much lower clean accuracy than vanilla CLIP due to their adversarial training on ImageNet, which harms zero-shot performance ImageNet against the attacks they have been trained with, CLIPure still outperforms them by huge margin (65% and 72.6% versus 33.3% and 44.3%). The robustness of CLIPure against unseen attacks is much better than their performance on seen attacks, showing that we are on the right path of leveraging the power of pre-trained language models.
other datasets. Additionally, Table 2 shows that even
More Experiments and Analysis. Due to space constraints. in the Appendix, we include a detailed case study, showcasing the visualization of image embeddings during the purification process using a diffusion model in Figure 6 Appendix D.I Figure 11a in Appendix (D.3 illustrates the effects of combining our approach with adversarial training and pixel space purification methods, while Figure TTb displays the outcomes of integrating classifier guidance. Additionally, we employ T- SNE to visualize the distribution image and text embeddings CLIP's latent space Figure [2a and analyze the impact of step size performance in Figure 12b
5.3 INFERENCE EFFICIENCY
We evaluate the inference efficiency of our CLIPure and baseline models by measuring the inference time of averaged over 100 examples from the CIFAR-10 dataset on single 4090 GPU. The results are displayed in Table B Our CLIPure-Car innovatively uses a discriminative approach for adversarial sample purification, achieving inference times comparable those of discriminative models. Traditional purification methods typically leverage generative model like diffusion and significantly involve complex inference procedures. Such complexity limits their applicability in seediency-sensitien tasks, such as autonomous driving.
5.4 CLIP VARIANTS
To investigate the influence of backbone models and their scale, we evaluate the performance of CLIPure-Cos across various versions of CLIP (Radford et al. 2021) and its variants, including

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of different methods in terms of average inference time over 100 samples on CIFAR1O. "Relative Time" expresses each method's inference time as multiple of the CLIP model's time for classification. Dis. denotes purification with a discriminative model, and Gen. denotes that with a generative model.
Method
CLIPure-Cos CLIPure-Diff
DiffPure
LM-EDM
CLIP
Gen. Dis.
Dis.
Gen.
Gen.
Gen.
Dis.
Inference Time (s)
4.1 x 10-4
0.01
2.22 6166.67x
0,25
3.6 x 10
Relative Time
1.14x
27.78x
694.44x
1x
CLI--ReNNtt
CLIP
CLIPAIT
Rebsst AND Clean Arc
2.58
80
EVA2-CLIP
CoCa
Robast Ace Clean Ani
DUBM
2.50
Robust Ancuracy
Robust Ancuracy
75
70-
14284
151M
65
14N
53-
120.
20:34
60
Clean Accuracy (%)
Clean Accuracy (%)
Figure 3: Accuracy and robustness (detailed Table 6 of CLIPure-Cor for (Left) various versions of CLIP and (Right) different backbone models. The bubble size represents the number of parame- ters, which is also indicated alongside each bubble. The left figure presents CLIPure-Cos based on ResNet-based models (including RN50, RN101, RN50x64, marked in red) and ViT-based models (ViT-B-16. ViT-B-32, ViT-L-14, ViT-H-14, and ViT-bigG- 14, marked in blue). The right figure de- picts CLIPure-COr Cos based on CLIP (including ViT-B-16, ViT-L-14, ViT-H-14, ViT-bigG- 14, marked in blue), EVA2-CLIP (including V and ViT-L-14, marked in green), CLIPA (including ViT- I.-14 and ViT-H-14, marked red), and CoCa (ViT-B-32, marked in yellow). The blue dashed line represents the point where robust accuracy equals clean accuracy, serving the upper bound of robustness, since successful defense against adversarial attacks hinges on accurate classification.
EVA2-CLIP (Sun et al. 2023), CLIPA [i et al. 2024b), and CoCa Yu et al.). The results in Table 6 and Figure 3 indicate: 1) Larger models generally exhibit stronger capabilities (i.e.. clean accuracy), which in turn enhances the robustness of CLIPure. Notably, the CLIPPre- with CLIPA with ViT-H-14 achieves the best performance, with robustness reaching 79.3% ImageNet- truly remarkable achievement. 2) CLIPA- CLiPure-Cos outperforms other CLIP models of comparable size in both accuracy and robustness. 3) ViT-based CLIPure-Cos models show better results than those based on ResNet.
CONCLUSION
We develop CLIPure, novel adversarial purification method that operates within the CLIP model's latent space to enhance adversarial robustness on zero-shot classification without additional training. CLIPure consists of two variants: CLIPure-Diff and CLIPure-Cos both achieving state-of-the-an performance across diverse datasets including CIFAR-10 10 and ImageNet. CLIPure-Cos. notably, does not rely generative models, significantly enhancing defense efficiency. Our findings reveal that purification a multi-modaa latent space holds substantial promise for adversarially robust zero- shot classification, pointing the way for future research that extends beyond image classification.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680101, CAS Project for Young Scientists in Basic Research under Grant No. YSBR-034, the National Key Research and Development Program of China under Grants No. 2023YFA1011602. the National Natural Science Foundation of China (NSFC) under Grants No. 62302486, the Innovation Project of ICT CAS under Grants No. E361 140, the CAS Special Research Assistant Funding Project, the project under Grants No. JCKY20221 30C039, the Strategic Priority Research Program of the CAS under Grants No. XDB0680102, and the NSFC Grant No.
62441229.
REFERENCES
Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.1 1929, 2020.
Sajjad Amini, Mohammadreza Teymoorianfard, Shiqing Ma, and Amir Houmansadr. Meansparse: Post-training robustness enhancement through mean-centered feature sparsification. arXiv preprint (rXx:2240600522) 2024.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 12(3):313-326, 1982.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of se- curity: Circumventing defenses to adversarial examples. International conference on machine learning, pp. 274-283 283. PMLR, 2018.
Yatong Bai, Mo Zhou, Vishal M Patel, and Somayeh Sojoudi. Mixednuts: Training- free accuracy- robustness balance via nonlinearly mixed classifiers. arXiv preprint arXiv:2402.02265. 2024.
Tsachi Blau, Roy Ganz, Chaim Baskin, Michael Elad, and Alex Bronstein. Classifier robustness enhancement via test-time transformation. arXiv preprint arXiv:2303. 15400 2023.
Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang. Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. arXiv preprint arXiv:2335..522/, 2023.
Daewon Choi, Jongheon Jeong, Huiwon Jang, and Jinwoo Shin. Adversarial robusiiiction via text- to-image diffusion models. In European Conference on Computer Vision, pp. 158-177. Springer, 2025.
Kenneth Ward Church. Word2vec. Natural Language Engineering, 23(1):155-162,2) 2017.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2024.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 213-223. JMLR Workshop and Conference Proceedings, 2011.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.

---

Published as a conference paper at ICLR 2025
Stanislav Fort. Adversarial examples for the openai clip in its zero-shot classification regime and their semantic generalization, Jan 2021. URL httts::/stannslaaatts// hww.waaoootttittiboooaeoe .ithub.......................... io/ 2021/01/12/0penn CLIP adversarial examples .html
Yuri Galindo and Fabio A Faria. Understanding clip robustness. Understanding clip robustness.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkes Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Gregory Griffin, Alex Holub, Pietro Perona, et al. Caltech- object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021.
Mitch Hill, Jonathan Mitchell, and Song- Song-Chun Zhu. Stochastic security: Adversarial defense using long-run dynamics of energy-based models. arXiv preprint arXiv:2005. 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6885 2020.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances Neural Information Pro- cessing Systems, 34: 10944-10956, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Christian Léonard. Some properties of path measures. Séminaire de Probabilités XLVI, pp. 207-230, 2014.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your dif- fusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2206-2217, 2023.
Lin Li, Haoyan Guan, Jianing Qiu, and Michael Spratling. One prompt word is enough to boost adversarial robustness for pre- trained vision-language models. In Proceedings of the IEEE/CVF 2024a.
Conference Computer Vision and Patern Recognition, pp.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language- image pre- treiiiinnng.. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 10965-10975 2022.
12

---

Published as a conference paper at ICLR 2025
Xiang Li and Shihao Ji. Defense-vae: A fast and accurate defense against adversarial attacks. In Machine Learning and Knowledge Discovery in Databases: International Workshops of ECML PKDD 2019, Wilrzburg. Germany, September 16-20. 2019, Proceedings, Part II, pp. 191-207. Springer, 2020.
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. Advances in Neural Information Processing Systems, 36, 2024b.
Chang Liu, Yinpeng Dong, Wenzhao Xiang, Xiao Yang, Hang Su, Jun Zhu, Yuefeng Chen, Yuan He, Hui Xue, and Shibao Zheng. A comprehensive study robustness of image classification models: Benchmarking and rethinking. International Journal of Computer Vision, pp. 1-23, 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. stat, 1050(9), 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:/306.515 2013.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint 224::22005.04660, 2022.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 06122, 1(2):3, 2022.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint arX1w:2103.01946. 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: sauge-to-imager diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Samangouei. Defense-gai gan: protecting classifiers against adversarial attacks using generative mod- cls. arXiv preprint arXiv: 1805. 06605, 2018
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial tne-tuning of vision embeddings for robust large vision- languala models. arXiv preprint arXiv:2402./233 2024.
13

---

Published as a conference paper at ICLR 2025
Erwin Schrödinger. Sur la théorie relativiste de l'électron et letllinnnn l'interprétation de la mécanique quan- tique. In Annales de I' l'institut Henri Poincaré, volume 2, pp. 269-310, 1932.
Nitish Shukla and Sudipta Banerjee. Generating adversarial attacks in the latent space. In Proceed- ings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 730-739,
2023.
Naman Deep Singh, Francesco Croce, and Matthias Hein. Revisiting adversarial training for ima- genet: Architectures, training and generalization across threat models. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint 1rxiv:1110. 10766, 2017.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 15389, 2023.
Zhicong Tang, Tiankai Hang, Shuyang Gu, Dong Chen, and Baining Guo. Simplified diffusion schr\ odinger bridge. arXiv preprint arXiv:2403 14623, 2024.
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language- image pre-training (clip). Advances Neural Information Processing Systems, 36, 2024.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. Medical Image Computing and Computer Assisted Intervention-MICCC 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp. 210-218. Springer, 2018.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Jinyi Wang, Zhaoyang Lyu, Dahua Lin, Bo Dai, and Hongfei Fu. Guided diffusion model for adversarial purification. arXiv preprint 2rXiv:2205.14966 2022.
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24502-24511, 2024.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference Machine Learning, pp. 36246-36263. PMLR, 2023.
Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pp. 12062-12072. PMLR, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arxiv 2022. arXiv preprint arXiv:2205.01917.2
Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv: 2016.
Boya Zhang, Weijian Luo, and Zhihua Zhang. Enhancing adversarial robustness via score-based optimization. Advances in Neural Information Processing Systems, 36:51810-51829, 2023.
Mingkun Zhang, Jianing Li, Wei Chen, Jiafeng Guo, and Xueqi Cheng. Classifier guidance enhances diffusion-based adversarial purification by preserving predictive information. arXiv preprint arXiv:2408.0 05900, 2024.
Mingkun Zhang, Keping Bi, Wei Chen, Quanrun Chen, Jiafeng Guo, and Xueqi Cheng. Causald- iff: Causality- inspired disentanglement via diffusion model for adversarial defense. Advances in Neural Information Processing Systems, 37:114791--128811 2025.
14

---

Published as a conference paper at ICLR 2025
Reference density Feef
S
dz [f f) g2 (2 de g(t)dw
Schrödinger Bridge
Pref:0
Pdata
(L((TT) I|rrrt)
PrefT Pprior
To
Pdata
"T Pprior
Pdata
dx [f (2, f) (4) log (a dt g(t)chr
Pprior
Optimal solution
Adversarial attack P
dz asign(V, V,0(( v)) adw,
Adversarial Attack
Po Pben
Pr Padv
Purification
Co Ppure
Q Pudv
Pben
aasign F 2., v)) o'V log (x) dt oddi
Padv
Purification
Q
Figure 4: Comparison of the Schrödinger Bridge (Schrödinger 1932
framework with our adversar-
ial attack and purification process
A DISCUSSION OF THE REVERSE-TIME SDE
Motivated by the Schrödinger Bridge theory (Schrödinger 1932, Tang et al. 2024 which models the transition between two arbitrary distributions, we utilize this theoretical framework model the transformation between benign and adversarial sample distributions.
Within the Schrödinger Bridge framework, the forward Stochastic Differential Equation (SDE) de- scribes a pre-defined transition from a data distribution Pdata to a prior distribution Ppeior. noted as reference density Pret P the space of joint distributions (T-1) xd where T represents the timesteps and denotes the data dimension. Consequently, we have Pref:0 = Panta and Pref.T = Pprior- Both Paata and Pprior are defined over the data space Rd, The optimal solution π starting from Pprior and transferring to Pdata is termed the Schrödinger Bridge, described by the reverse SDE.
Specifically, the forward SDE is expressed as: dæ [f t) + (t) log ()),t + g(t)dw,, To Pdata
(10)
where f(x,,t) t) is the drift function and g(t) is the diffusion term. w; is the standard Wiener process, and V and V are time-vary varying energy potentials constrained by the following Partial Differential Equations (PDEs):
NV V
V
Tr(g"V2
Ot
2
(11)
0
V VTT
Ir(g2
Ot
2
such that (2x,0) (2, = Pdata (xx T (x, T) Pprior-
Similarly, we define the mutual transformation between clean and adversarial sample distributions using coupled SDEs. The forward SDE describes the transformation process from benign to adver- sarial example distributions corresponding an attacker's process. For an untargeted PGD attack, this is represented as:
dx = asign V L(0; Ft, Ymne) )t + adw;, Cg
Poen:
(12)
where represents the attack step size, L(0; 2t, Vmue) denotes the loss when the adversarial example 2e is classified by the model with parameters as the ground truth category Vuwe at attack step (where t E [o, T)), dw, denotes the Wiener process (Brownian motion) to express cases, and or is constant. The adversarial example Ir is bound by e in Ip norm, i.c., Ør Doll, S
general
15

---

Published as a conference paper at ICLR 2025
According to Eq.
12
obtain formulation of the drift function and diffusion term is
fax, () assign V L., Lee Mmue))
(13)
g(1) =
According to (Anderson 1982. Song et al. 2020), the corresponding reverse-time SDE is: dæ [f t) g2 (t) logp(=)) + adu;
(14)
Thus, derive the corresponding reverse SDE of Eq. 12
dx [asign( Vz wt, 3hmee) V log p (x:)) odū, æt Padvs
(15)
where log p is the log- ligelikl of the marginal distribution of benign samples, also known as the score function in Score SDE (Song et al. 2020). dwg denotes the reverse-time Wiener process time flows from i T to 0.
Note that the coupled SDEs described Eq. and Eq. 15 are special cases of Eq. when v log V (z,, t) and V log (x,, = log p(x,).
Without loss of generality, other adversarial attack methods typically be described using the forward SDE. Therefore, our proposed approach of modeling adversarial attacks and purification processes using coupled SDEs is universally applicable.
B DETAILED PROOF OF PURIFICATION RISK FUNCTION
We first define the joint distribution of the attack process described by forward SDE in Eq. as Pr+1. which spans [(T+))xd Thus we have Po Poen and Pr Padv- For the purification process, we designate the joint distribution as 2 also defined over R(T+1)xd with Qo Padv and Qr Ppure. T represents the timestep, Pben Padv and Ppure denote the distribution of benign, adversarial, and purified samples.
According to the results of Léonard (2014), we have
(L(Q0: (Po:r) = KL (a..r//PP.. + EQ,,
(16)
Since we focus on the differences between the purified and benign examples rather than the entire purification path, we concentrate on KT, Thus, continue derive:
KL (L(oo.r(Po.r)
KL(p( Ppure) Pattack (Eben Taavv)
/ p( ) pure (adv) log
pl (adv p 2 pure [adv)
dup doaad
p( p(Dben)) (aadv (ben)
/ p(aatr) (J p(z (adv) log
pl 2 pure [adv]
/ p(( log
pl p( dæadv p(xren)
dappure dwadr
pl (andv (ben)
Ennas (KL((((ppun Bady
Mlpplllpppeepp (xl (Den)) ))
KL (waay) (()ea)))))
(17)
where p( @hen) represents the conditional probability transitioning from When to Bady as de- scribed by the forward SDE, and (a represents the conditional probability transitioning from Tady to Zpur as described by the reverse SDE. This formulation underscores the effectiveness of the purification process by measuring how well the adversarial transformations are reversed.
Given that the perturbations added by attackers are typically imperceptible, over small time interval At, the solution of the forward SDE described by Eq. approximates normal distribution:
p( Fadv (bea) . (xben sign(V At).
(18)
Similarly, the reverse SDE process described by Eq. approximates normal distribution:
p( pure (Fav)) (aradv (a sign( aaar L(O; Faave Vrru))) 2
o² PA). (19)
16

---

Published as a conference paper at ICLR 2025
benign distribution
Enc
adv
puce
Fudv
'dog'
Adversarial Purification CLIP's Latent Space
Classification
Ene!
purification bused on normalized unit vocters
likelihood estimation null template
'a photo of dog."
Euc
Fdv
"/stv/l=zaalll
"a photo of
cast =
pures ZdeR
2 pure
De
generative version
Bu
-[ 2, el
cos( F Pareer eaa))
discriminative version
ppone
|
cos(=", 000(="==)
photo cat."
Enc
Figure 5: Illustration of the process of CLIPure including the purification in latent space and zero- shot classification, detailed in Algorithm
The KL divergence formula for two normal distributions is:
KI(N(HO, En) |1 ( E,)) (r(E)>>0) - (u1
mo) - k: + log det Σ1
det Eg
(20)
When Eg = E1 0² At, the KL divergence reduced as
LL Eo) || [( Ex)) = 20 2 || - 10 10012 1.
(21)
Thus have
Egg.. (pL((((pp) Fady )lppl (a xx)]]
(22)
Exa.. V log p( log pl Iben ) 0² At)
Thus we have proved the result in Eq. that
(T((Qa (Po,r) (T Fadv, Ppure ppprr))p((a Fadv))
Banay (KT.(g(2Ppure (aav [p( Fnavv (bbe)))
((p(z))
(23)
2 Exai
V log pa (adv) TV log pl (ady)) ) 0 At] KL (p (p( ( (()ben)).
C EXPERIMENTAL SETUP
C.I IMPLEMENTATION DETAIL
Following the implementation of pixel space purification (including Chen et al. 2023 with andeland Zhang et al. (2023) with code) CLIPure employs gradient ascent maximize the log-likelihood of samples, which formulated through: (1) cosine similarity (CLIPure-Cor and (2) mean squared error (CLIPure-DDe Diff). We conduct 10 purification steps on image embeddings obtained from CLIP's image encoder, with a step size of 30 for both CIIPPre-Cos and CLIPPre-DDiff to update directly on the vector direction of image embeddings. For CLIPure-Diff, we use the off-the-shelf DiffusionPrior model from DaLLE 2 (Ramesh et al. (022). Experiments indicate that timesteps in the range of 900 to 1000 yield better defense outcomes when estimating likelihood via Diffuusio., hence we select this range for estimation.
For the blank template used in purification, we employ 80 diverse description templates combined with class names, such "a good photo of <class-names to enhance stability, following zero-shot classification strategies. Consistently, each class c's text embedding, x", as referred to in Eq. is computed the average text embedding combined with all templates.
17

---

Published as a conference paper at ICLR 2025
Table Performance comparison of defense methods CIFAR-IO against BPDA with EOT-20 un- der foc (c 8/255) norm bound. We use underlining to highlight the best robustness for baselines, and bold font denote the state-of-the-art (SOTA) across all methods.
Clean Acc (%)
Robust Acc (%)
Method
FARE (Schlarmann et al.
2024)
77.7
8.5
TeCoA MMo al. 2022
79.6
10.0
Purify EBM (Hill et al.
2020
84.1
54.9
LM EDM (Chen al.
2023
83.2
69.7
ADP Yoon et al.
2021
86.1
70.0
RDC Chen al. 2023
89.9
75.7
GDMP (Wag ct al.
2022
93.5
76.2
DiffPure (Nie et 2022
90.1
81.4
Our CLIPure- Diff
95.2
94.6
Our CLIPure Cos
95.6
93.5
C.2 DATASETS FOR ZERO-SHOT PERFORMANCE
In order to evaluate the zero-shot performance our CLIPure and adversarially trained CLIP model including FARE (Schlarmann al. 2024) and TeCoA (Mao al., 2022 follow the settings and evaluation metrics used by FARE and outlined in the CLIPPhenchhark We evaluate the clean accuracy and adversarial robustness across 13 datasets against an foo threat model with e 4/255
and E
2/255. The datasets for zero-shot performance evaluation include CalTech101 (Griffin
al.
2007). StanfordCars (Krause al., 2013), CIFARIO, CIFAR100 (Krizhevsky et al.
2009).
DTD Cimpoi et al., 2014). EuroSAT (Helber et al. 2019), FGVC Aricrafts (Maji et al.
2013
Flowers (Nilsback Zisserman, 2008), ImageNet-R (Hendrycks et al., [202I). ImageNet-S (Wang
et al.
2019), PCAM (Veeling et al. 2018), OxfordPets (Parkhi et al. 2012). and STL10 (Coates
et al.
2011
C.3 ADVERSARIAL ATTAKS
BPDA with EOT. To compare our approach with purification methods that do not support gradient propagation, we conducted evaluations using the BPDA (Backward Pass Differentiable Approxima- tion) (Athalye et al. 2018) attack method, augmented with EOT=20 (Expectation over Transforma- tion) to mitigate potential variability due to randomness. As depicted Table 4 our CLIPure-Cos model significantly outperforms traditional pixel-space purification methods based generative models such as EBM (Hill et al.] 2020), DDPM (Chen et al. 2023), and Score SDE (Nie et al.) 2022), despite being discriminative model. Notably, CLiPure demonstrates higher robust curacy compared to the AutoAttacc method under BPDA+EOT-22 attack. This is attributed to the fact that adaptive white-box attacks are precisely engineered to target the model, leading to higher attack success rate.
Latent-based Attack. /Shukla & Banerjee (2023) introduced latent-based attack method, utilizing generative models such as GANs (Goodfellow et al. 2014) to modify the latent space representations and generate adversarial samples. We applied this approach to the CIFAR- 10 dataset's test set, and the results, displayed in Table 5 reveal that unbounded attacks achieve higher success rate than bounded attacks. Despite these aggressive attacks, our CLIPure method retains an advantage over baseline approaches. This is attributed to CLIPure leveraging the inherently rich and well-trained latent space of the CLIP model, which was trained on diverse and sufficient datasets, allowing it to effectively defend against unseen attacks without the need for additional training.
Avaailable https: / //gi htt.t thub. com/ I.A ION-OO--III /CLII ANN-AA//ciPeeeeecchaaa bam.aaarebhhaaaa r
18

---

Published as a conference paper at ICLR 2025
Table Performance comparison of defense methods on CIFAR-IO against unbounded latent-based attack (Shukla & Banerjee 2023) using generative adversarial network (GAN) (Goodfellow et al.,
2014).
Clean Acc (%)
Robust Acc (%)
Method
ResNet-18 18
80.2
12.4
FARE (Schlarmann et al. 2024)
77.7
61.7
TeCoA (Mao et al. 2022
79.6
63.7
Our CLIPure- Diff
95.2
69.2
Our CLIPure Cos
95.6
70.8
C.4 BASELINES
In this section, discuss the detailed experimental settings of the baselines mentioned in Sec- tion 5.1 For the CIFAR-10 dataset results shown in Table we use the off-the-shelf Stable Diffusion model provided by Ly al. (2023). employing it zero-shot classifier. Our DiM-StableDiffusiinn method adapts this model to a likelihood maximization approach. Other methods were applied di- rectly according the model in the original papers for CIFAR-10 without additional training.
For the ImageNet dataset experiments detailed in Table 2 FARE (Schlarmann et al. 2024) and TeCoA (Mao et al., (022)) involve testing checkpoints that were adversarially trained on the Im- ageNet dataset specifically for an loo threat model with 4/255. The LM- Decoder method utilizes the Decoder module from the ViT-L-144 model of DaLLE2 (Ramesh al.) 2022) provided by OpenAl, adapted to the likelihood maximization method.
adapts the same Decoder module to the DiffPure (Nie et al. (2022) purification method. Other meth- ods use models as provided and trained in the original papers.
Regarding the zero-shot performance across 13 datasets shown in Table 7 we opt for the adversar- ially trained models of TeCoA (Mao et al. 2022) and FARE (Schlarmann al. 2024) with an foc threat model at = 4/255, as they exhibited superior performance compared to those trained with 2/255.
In the results presented Table and Table 5 we conduct an adversarial evaluation using models specified in the literature without any additional fine-tuning.
D MORE EXPERIMENTAL RESULTS
D. CASE STUDY
To visualize the purification path of CLIPure, we employ the Decoder of DaLLE2 (Ramesh et al. 2022), which models the generation process from image embeddings images through diffusion model. As shown in Figure 6. starting from an adversarial example initially classified as a 'stupa,' CLIPure modifies the semantic properties of the image embedding. By the second step, the image is purified to 'mountain,' closely aligning with the semantics of the original image, which also contains mountainous features. Subsequently, after multiple purification steps, the image embedding is accurately classified as the 'valley' category.
In Figure T0 we analyze the distribution of cosine similarity between the embeddings of clean images and adversarial examples with words to understand the semantics of image embeddings. In Figure g we also highlight the specific top-ranked categories for the benign and adversarial examples. We observe that the cosine similarity distribution for words close to the clean image stable and resembles the prior distribution shown in Figure Dal In contrast, the distribution of cosine similarity for the adversarial examples shows anomalies, with the top-ranked adversarial labels displaying abnormally high cosine similarities. This could potentially offer new insights into adversarial sample detection and defense strategies.
19

---

Published as a conference paper at ICLR 2025
Table 6: Performance CLIPure-Cos based on various versions CLIP Radford et al. 2021 EVA2-CLIP (Sun et al. 2023). CLIPA (Li et al., 2024b), and CoCa (Yu et al. under the foc threat model 4/255) on ImageNet dataset. "Param" denotes the number of parameters. The prefix "RN" refers to Nes-aaaad (He al. 2016) methods, while "ViT" indicates Vision Transformer- based (Alexey 2020) approaches.
w/o Defense
CLIPure
Model
Version
Param (M)
Acc (%)
Rob (%)
Acc (%)
Rob (%)
RN50
102
59.7
0.0
60.0
52.9
RNI01
119
61.6
0.0
61.9
55.5
RN50x64
623
72.0
0.0
72.3
69.5
CLIP
ViT-B-16
149
68.1
0.0
68.2
63.0
Radford et al. 2021
ViT-B-32
151
62.0
0.0
62.0
58.1
ViT-L-114
427
74.9
0.0
76.3
72.6
ViT-H-14
986
77.2
0.0
77.4
74.4
ViT-bigG-14
2539
80.4
0.0
80.4
77.6
EVA2-CLA CLIP
ViT-B-16
149
74.6
0.0
74.7
71.7
Sunn et al. 2023
ViT-L-14
427
81.0
0.0
80.7
78.7
CLIPA
ViT-L-14
414
79.0
0.0
79.0
77.2
(Li et al. 2024b)
ViT-H-14
968
81.8
0.0
81.5
79.3
CoCa
ViT-B-32
253
64.2
0.0
63.8
59.8
Yu al.)
benign example
step=1
step-2
step= 3
step=4
step=5
label: 'vlllyy'
pretzell'
'mountain'
'pretzel'
loopard'
'Maltese'
adversurial example
stop=6
stop=7
step=8
step=9
step=10
adv label: 'stupa'
'mountain'
'Maltese'
'Lhasa Apso'
'patio'
'valley'
CLIPure purification staps
Figure 6: Purification process of our CLIPure model for an image of a 'valley' adversarially per- turbed to be classified as 'stupa'. Using the DaI. Decoder (Ramesh et al.] [022]) we visualize the stages of image embedding purification. The text below each picture annotates the classification results corresponding to each timestep in the purification process.
Additionally, we illustrate the purification path using T-SNE. As depicted in Figure 7 an image sampled from CIFAR-10 originally categorized 'airplane" adversarially manipulated semble a 'truck,' making it an outlier. Through our CLIPure method, the image is purified towards high-density area and ultimately reclassified accurately the ''arrplane' category.
D.2 ANALYSIS FROM TEXTUAL PERSPECTIVE
In this section, provide additional analysis from textual perspective supplement to the analysis. In Figure H, we take advantage of the text modality the CLIP model to understand the semantics of clean and adversarial examples by matching them with closely related words. We
20

---

Published as a conference paper at ICLR 2025
Table 7: Zero-shot performance on 13 datasets against AutoAttack under Lop threat model with 2/255 and = 4/255. FARE (Schlarmann et al. 2024) and TeCoA (Mao et al. 2022) are trained on S0 threat model with = 4/255 for its generally better performance than 4/255. Un- derlined results indicate the best robustness among baselines, while bold text denotes state-of-the-arr (SOTA) performance across all methods. The term increase represents the percentage improvement in robustness compared to the best baseline method.
ImageNet-S
CIFARIOO
ImageNet-R
OxfordPets
Eval.
Method
CafTech
CIFARIO
EuroSAT
FGVC
Flowers
PCAM
STL-
Average Zero-shot
Cars
DTD
CLIP
83.3 77.9 95.2 71.1 55.2 62.6 31.8 79.2 87.9 59.6 52.0 93.2 99.3 73.1
TeCoA
78.4 37.9 79.6 50.3 38.0 22.5 11.8 38.4 74.3 54.2 50.0 76.1 93.4 54.2
clean
FARE
84.7 63.8 77.7 56.5 43.8 18.3 22.0 58.1 18.2 56.7 50.0 87.1 96.0 61.1
CLIfure-Diff
79.9 75.5 94.9 63.8 55.2 58.2 29.3 75.0 87.5 55.9 56.8 90.4 98.4 70.8
CLIPure-Cos
82.9 78.6 95.6 73.0 55.4 63.3 33.2 79.3 87.7 58.3 52.0 92.5 99.6 73.2
lo = 2/255
CLIP
0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
TeCoA
69.7 17.9 59.7 33.7 26.5 8.0 5.0 24.1 59.2 43.0 48.8 68.0 86.7 42.3
FARE
76.7 30.0 57.3 36.5 28.3 12.8 8.2 31.6 61.6 41.6 48.4 72.4 89.6 45.9
CLIPure-DifY
75.1 65.9 92.5 52.6 45.9 41.5 20.8 65.8 86.5 49.8 51.4 86.2 97.9 64.0 139.4%
CLIPure-Cos
80.8 73.9 93.0 65.0 50.7 49.1 28.2 75.3 85.4 54.3 49.1 91.2 99.5 68.8 145.9%
Loo = 4/255
CLIP
0.0 0.0 0.0 0.0 0.0
0.0
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0
TeCoA
60.9 8.4 37.1 21.5 16.4 6.6
2.1 12.4 41.9 34.2 44.0 55.2 74.3 31.9
FARE
64.1 12.7 34.6 20.2 17.3 11.1 2.6 12.5 40.6 30.9 48.3 50.7 74.4 32.4
CLIPure-Diff
74.1 66.3 90.6 51.6
45.2
42.9
18.8
65.8 83.8 48.8 52.0 85.7 97.2 63.3 195.4%
CLIPure-Cos
80.1 72.2 91.1 59.1 50.1
48.4
26.1
74.8
84.6
52.4 48.9 91.0 99.4 67.4 100 0%
lization
truck
sirplonc
purtiiation precess
2.5
-5.0
-7.5
Figure 7: Purification process of the adversarial example attacked to adversarial label 'truck" and purified by our CLIPure-Diff visualized by T-SNE, The image is randomly sampled from the CIFAR-10 dataset.
attacked from ground truth label 'airplane'
selected words from ImageNet's categories and supplemented this with an additional 10,000 ran- domly drawn words from natural language vocabulary for a comprehensive list. We observe that the meanings of the category words ranking high are relatively similar. For clean samples, the distri- bution of cosine similarity across ranks is relatively stable, whereas the adversarial samples exhibit abnormally high cosine similarity for adversarial categories at top ranks. Figure Da displays the distribution of cosine similarities between the word embeddings and image embedding gener- ated from pure noise, across various ranks. We aim to use these similarities to represent the prior probabilities of the words. The results indicate that most samples cluster around cosine similarity
21

---

Published as a conference paper at ICLR 2025
0.24
vecaburay
emozeling el neney (iragee
aburay
Casine dimilurity with ombecding of nosy
category
Capine similanty with
0.40
Cateegry
valley
0.35
0.22
mountain promontory
0.30
yurt
dome
bell tower
0.20
0.25
masque
50
100 Rank
150
200
20
40
Rank
(a) clean sample
(b) adversarial example
Figure 8: Cosine similarity between word and image embeddings of (a) clean and (b) adversarial examples in F acroos different ranks. Blue dots denote 10,000 words sampled from Word2Vee vocabulary Church 2017), and red dots denote words from the 1,000 ImageNet categories.
0.27
0.25
Cozine dimiontty with unbreding of nuloy image
vocaburay
category
Cozine similonty
umbesbingg maley
0.20
0.15
Igsse
0.10
televaion
couch
pllow.cm
0.05
0.21
2000
4000
6000
8000
10000
100
300
400
Rank
Rank
(a) prior probability
(b) highlight top-ranked words
Figure 9: (a)Cosine similarity between words and the image embeddings of noisy example to illustrate the prior probability of the words. (b) Highlights the top-ranked words within the 1,000 ImageNet categories, emphasizing the most semantically similar words to the noisy image.
of approximately 0.15, with only few extreme values either much higher or lower, suggesting a normal distribution pattern. This indicates that a majority of the samples have moderate prior prob- abilitics, contrasting with the long-tail distribution often observed in textual data. In Figure 9bl we highlight the top-ranked words in the vocabulary that closest the noisy image embedding. Words like "television", which appear frequently in image data, rank high, aligning with our ex- pectations. This abnormal phenomenon in adversarial samples could potentially inspire adversarial detection and more robust adversarial defense methods.
D.3 COMBINING CLIPURE WITH OTHER STRATEGIES
In this section, we conduct experiments that combine orthogonal strategies, including adversarial training (Schlarmann et al. 2024) and pixel space purification (Chen et al. (023). as well as incor- porating classifier contidence guidance. We carry out evaluations using sample of 256 images from the ImageNet test set. The experimental setup in this section is aligned with the same configuration as presented in Table detailed in Section 5.1
Combination with Adversarial Training (CLIPure+AT). Orthogonal adversarial training (AT) methods like FARE (Schlarmann al. (024) has fintuned CLIP on ImageNet. We combine CLIPure and FARE by purifying image embedding in FARE's latent space and classification via FARE. The experimental results, depicted in Fig. TTa indicate that the CLIIPue-DDfff-A method enhances ad- versarial robustness in CLIPure-Dire Diff. This improvement could be attributed to FARE providing more accurate likelihood estimates, as it is specifically trained on adversarial examples. When com- bined with CLIPure- -Cos the outcomes were comparable, likely because the inherent robustness of CLIPure-CC -os already high, leaving limited scope for further enhancement.
Combination with Pixel Space Purification (CLIPure+LMI, Purification in latent space and pixel space occurs at different stages of processing. Pixel space purification acts directly on the input samples, while latent space purification operates on the latent vectors encoded from the input picture.
22

---

Published as a conference paper at ICLR 2025
embedding nel image
embooding nelsy Image
0.4
Cosine simlartt with
Casine similanty with
0.3
0.15
0.2
0.10
0.05
0.00
1
2000
4000
6000
8ooo
10000
2000
4000
6000
8000
10009
Rank
Rank
(a) clean sample
(b) adversarial example
Figure 10: Cosine similarity between words and the image embedding of (a) clean sample and (b)
adversarial example of the case shown in Figure to illustrate the top-ranked words.
80
80
CLiPure
CLiPure+AA
CLiPure
76.3
Performance
72.6
72.4
73.1
Performance
rob
69.5
aCc
70-
0.C
rob
89
60
59.8
Acc-Diff
Rob-Diff Rob-CLIP
rab
ACC-CLIP
CLiPure-Cos
CLiPure-Diff
10
weight guidance
10
(a) Combination methods
(b) Impact guidance weight
Figure 11: (a) Accuracy and robustness against AutoAttack with loo 4/255 on ImageNet of CLIPure, CLIPure combined with Adversarial Training (AT), and CLIPure combined with pixel space Likelihood Maximization (LM). (b) Performance ImageNet against AutoAttack with Lo 4/255 incorporating with classifier confidence guidance.
By combining both methods, first purify the input samples in pixel space, then pass the purified samples through an encoder to obtain latent vectors, which are further purified latent space. We mark this combination LL. For the Likelihood Maximization (LM) method, chose LM-EDM (Chen et al., 2023) due to its strong performance across pixel space purification methods.
Experimental results, shown in Fig. Tla indicate that combining with LM leads decrease in both clean accuracy and robustness. This decline may be associated with the challenges of in- formation loss inherent pixel space purification. Since LM operates directly the pixels of the image, it is highly sensitive to the degree of purification. However, the adversarial perturbation varies across samples, potentially leading to over-purification thus decrease performance. In contrast, purification in latent space with polar coordinates simply involves changing vector direc- altering semanties without reducing semantic information. Thus, avoids the problem of semantic information loss.
Combination with Classifier Guidance. Eq. we that the reverse-time SDE associated with the attack method includes not only a likelihood maximization purification term but also classifier guidance term, represented as V, C, æt; M. This approach aligns with the classifier confidence guidance proposed by Zhang et al. (2024). We incorporate this classifier guidance term in two versions of CLIPure (CLIPure- Diff and CLIPure-Cos) as described in Algorithm|1 [[] Since the ground truth label Mrue is unknown, employ the predicted label by the CLIP model as proxy for Viue- To address potential inaccuracies early estimations, classifier guidance was only applied in the final steps of the 10-step purification process.
We evaluate the performance varied with the weight of the guidance term, as depicted Fig. TTb It shows that guidance weight of 10-1 led the largest improvement in robustness: CL.Pure-
23

---

Published as a conference paper at ICLR 2025
n/2
Performance
Accuracy Robustness
74
73
3r//2
2, of
2, of 'a photo of a (c).'
15 20
2, of of 'a photo of
Step Size
(a) Embeddings in CLIP's latent space
(b) Impact of step size
Figure 12: (a) Visualization of image and text embeddings reduced via PCA and plotted in polar coordinates using t-SNE. and represent the image and text embeddings obtained through CLIP, respectively. Since vector direction signifies semantics, emphasize directional properties. (b) Impact of step size during a 10-step purification process using CLIPure-Cos on 1000 ImageNet samples against AutoAttack with loo threat model 4/255).
depict the samples in polar coordinates
Diff achieved a robustness of 67.2% (an increase of +2.2% over scenarios without guidance), and CLIPure-Cos attained robustness of 73.1% (an improvement of +0.5% over guidance).
D.4 T-SNE VISUALIZATION
we display the distribution of image embeddings 2 for both benign samples and adversarial examples from a subset of 30 samples on the CIFAR-10 dataset, after dimensionality reduction using t-SNE, The figure also includes embeddings for 10 category-associated texts (e.g.. "a photo of dog.") z', as well as the text embedding for blank template "a photo of .".
In Figure 12a
We observe distinct clustering of text vectors, clean image embeddings, and adversarial image em- beddings in the space. The blank template, which is used for purification, is positioned at the ter the category text clusters, representing a general textual representation. This arrangement demonstrates that clean and adversarial sample distributions occupy distinet regions in the embed- ding space, providing a foundational basis for the effectiveness of adversarial sample purification.
D.5 HYPERPARAMETERS
D.5.1 IMPACT OF STEP SIZE
We conducted experiments to assess the impact of different step sizes on the effectiveness of purifi- cation. Figure [2b] shows how the clean accuracy and robustness against AutoAttack (f.3 = 4/255) of our CLIPPre-CCo. method vary with changes in step size. These experiments were carried out on 1,000 samples from the ImageNet test set with 10-step purification. The results indicate that the performance of the method is relatively stable across different step sizes, consistently demonstrating strong effectiveness.
Limited the computational complexity of CLIPure- Difff, we opt conduct parameter search for this model. Instead, we directly apply the optimal parameters found for CLIPure-Cor
D.5.2 IMPACT OF PURIFICATION STEP
As illustrated in Figure 13 (Left), we assess the performance of various purification steps in CLIPure- Cos and CLIPure-Dite Diff. The results indicate that insufficient purification while maintaining accept- able clean accuracy, results in low robustness. As the number of purification steps increases, robust- ness gradually improves. Continuing to increase the purification steps stabilizes both accuracy and robustness, demonstrating balance between the metrics as the process evolves.
24

---

Published as a conference paper at ICLR 2025
76
Robust Accuracy (%)
74
50
Perforance
40
70
CLIPure-Diff CLiPure-Cos
30
Acc-Diff -
Rob-Diff Acc-Cos Rob-Cos
20
68
64
15
Purification Step
Attack Budget (/255)
Figure 13: Clean accuracy (marked as "Acc") and robust accuracy (marked "Rob") across differ- ent (Left) purification steps and (Right) attack budgets. "Diff" indicates CLIPure-Diff while "Cos" represents CLIPure-Cos.
D.6 PERFORMANCE ACROSS ATTACK BUDGET
To further explore the adversarial defense capabilities of CLIPure, we evaluate the robustness of CLIPure-Diff and CL.IPre-Cor against various attack budgets, following the settings used in Ta- 2 As shown in Figure 13 (Right), CLIPPre-Diff demonstrates superior clean accuracy (at c=0); however, its robustness decreases the intensity of the attacks increases. In contrast, CLIPure-Cos exhibits a stronger ability to withstand adversarial perturbations.
25


Reviews:
[{"review_id": "UYeqlDCVxZ", "reviewer": "Reviewer_k5Xb", "summary": "This paper proposes CLIPure, an adversarial purification method that operates within the CLIP models' latent space. This approach can enhance adversarial robustness on zero-shot classification without additional training. Even CLIPure-Cos can improve adversarial robustness without an external diffusion model. They demonstrate the effectiveness of CLIPure on various datasets.", "questions": "Please answer the Weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper robustifies CLIP in latent space instead of pixel space, and they show the strength of this strategy in analysis.\n\n2. The proposed method can improve the adversarial robustness of zero-shot classification of CLIP without training. In particular, CLIPure-Cos even does not require generative models, which are much more efficient regarding inference cost.", "weaknesses": "1. Missing baselines: [1] proposed adversarial and certified robustness of CLIP, compared to TeCoA.\n\n[1] Choi et al., Adversarial Robustification via Text-to-Image Diffusion Models, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sZNBr4jZU0", "reviewer": "Reviewer_g2Zx", "summary": "This paper aims to build an adversarially robust zero-shot image classifier, based on CLIP. The authors propose two variant purification-based methods, CLIPPure-Diff and CLIPPure-Cos. The new approach is demonstrated to greatly boost the adversarial robustness and consistently set a new state-of-the-art across several datasets.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well motivated by measuring the KL divergence between the joint distributions of the purification and attack steps.\n2. Although purification has been widely studied in pixel space, this paper showcases the potential of multi-modal latent space for learning robust zero-shot classification.\n3. The proposed method is the first, as far as I know, purification method in multi-modal latent space.\n4. The experiments and analysis are comprehensive and demonstrate the excellent of the proposed method", "weaknesses": "1. My main concern is related to the impact of VLMs. As the authors stated, the CLIP is a strong VLM and has shown superiority on many tasks. With this regard, the advantages of the proposed method could be substantiated better through an ablative study with different VLMs. \n2. The used models for evaluating the proposed method are limited, compared with the prior works [1]. For example, the results on ImageNet are only based on WideResNet-50. Transformer-based models are suggested, cf. \n[1] Diffusion Models for Adversarial Purification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6yrc3y9kXD", "reviewer": "Reviewer_7kUG", "summary": "This paper enhances adversarial robustness by purifying images in latent space. The experiments show a substantially improvement.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Good performance.\n\n2. Clear figures with bright color.", "weaknesses": "Although the method shows a significant improvement, it still has many non-ignorable problems:\n\n1. The motivation and contribution are too verbose, as well as the background description.\nThe authors should compact it.\nFor example, The prompt content of _\"... by matching an image with text prompts “a photo of <class-name>”. \"_ in abstract can be completely removed.\n\n2. The method seems just do purify in feature space instead of in pixel space.\n\n3. Lack a figure to illustrate the process of the method.\n\n4. The theoretically analysize about purification risk is difficult to understand. It seems just a SDE?\n\n5. And the analysize only is from CLIP, how can you extend it to diffusion model?\n\n6. The definition of Zero-shot Learning is not correct.\nIn the original paper of CLIP, it has clearly claimed they changed the definition of ZSL from class-level to dataset-level.\n\n_\"In computer vision, zero-shot learning usually refers to the study of generalizing to unseen object categories in image classification (Lampert et al., 2009). We instead use the term in a broader sense and study generalization to unseen datasets.\"_\n\n7. Where is the ablation study?\n\n8. From table 3, we can clearly see CLIPure-Diff also increases the inference time significantly. And the efficiency comparsion misses the result of  FARE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BxjmQsJZ7F", "reviewer": "Reviewer_Hwgg", "summary": "In this work,the author propose CLIPure, which conducts purification in CLIP's latent space for adversarially robust zero-shot classification.CLIPure leverages the image encoder and text encoder of the CLIP model to achieve effective purification in the latent space.By minimizing the KL divergence between the purification process and the adversarial process, CLIPure reduces purification risks and enhances the model's robustness against adversarial attacks. The algorithm includes two versions: CLIPure-Diff and CLIPure-Cos, which are based on the DiffusionPrior model and the cosine similarity metric of CLIP respectively. Experimental results demonstrate that CLIPure significantly outperforms existing methods when defending against AutoAttack on multiple datasets.", "questions": "1.The paper mentions using 80 different description templates to enhance stability. How were these templates selected? Is it possible to optimize the selection of these templates through an automated approach?\n\n\n2. Regarding the defensive effectiveness of the model, how does it perform under different threat levels (ϵ values)? Is there a detailed analysis available?\n\n3、The article mentions that the selected purification step is 10 steps, but it lacks relevant experimental evidence to support this. How is this number determined? Is there experimental proof that this is the optimal number of steps, or is there a potentially better number of steps?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper introduces CLIPure-Diff and CLIPure-Cos, which perform purification in CLIP's latent space rather than in pixel space. Both methods exhibit significantly larger KL divergence between adversarial and benign sample distributions when compared to methods operating in pixel space and uni-modal latent space. Furthermore, CLIPure-Cos enhances defense efficiency by not relying on generative models. Results demonstrate that CLIPure significantly improves the SOTA robustness.", "weaknesses": "1、Although the article conducts numerous experiments, the selected comparison methods are not entirely appropriate. For instance, the adversarial training method in Table 2 is intended for defending against attacks on the text functionality of CLIP, rather than for zero-shot defense based on CLIP.\n2、It is recommended to conduct further comparisons with models in the same direction, such as \"Understanding zero-shot adversarial robustness for large-scale models\" (ICLR 2023) and \"Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness\" (CVPR 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oViWKQcJOz", "reviewer": "Reviewer_3KPg", "summary": "This paper presents CLIPure, an approach for building adversarially robust zero-shot image classifiers based on CLIP. CLIPure performs purification in the latent space of CLIP to enhance robustness against adversarial attacks. It introduces two purification methods: CLIPure-Diff, which uses DALLE 2's diffusion prior to pure latent vectors ,and CLIPure-Cos, which performs purification by computing cosine similarity between the embeddings of an image and textual embedding of the blank templates. This paper conducts extensive experiments on CIFAR-10, ImageNet, and other 13 datasets and the proposed method demonstrates strong performance, exceeding previous SOTA results.", "questions": "I am not familiar with this field. Please refer to the weaknesses for my major concerns.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method performs purification in CLIP's latent space, which is technically sound. \n\n2. Based on the cosine similarity used in CLIP, this paper further proposes a purification method that is not based on generative models, which improves the computational efficiency.", "weaknesses": "1. The motivation is not strong enough. It seems that the key motivation is simply that performing purification in the latent space of CLIP is promising to achieve improved performance. However, there are only a few discussions about why it is necessary to perform purification methods for CLIP in its latent space and what issues exist with previous methods for CLIP.\n\n2. Some baseline methods need further clarification. It is unclear whether the authors pre-trained or trained the baselines themselves or used publicly available versions. If off-the-shelf models are used, it is necessary to specify the references and discuss whether the difference in training data influences the comparisons. If the authors conducted the training themselves, which dataset is used and what is the setting for the training?\n\n3. It seems that the experimental settings for baselines are not clear enough. For the compared purification methods, the base generative models should be unified for a fair comparison. Especially, it would be better to compare to the purification method with DiffPure based on DALLE2. For the AT methods, as the proposed method is specifically designed for CLIP, further discussion is needed on the comparability between different base models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AYz1Gaaje0", "reviewer": "Reviewer_xBxY", "summary": "The paper propose two variants (i.e., CLIPure-Diff and CLIPure-Cos) for our CLIPure approach, which explore purification in the multi-modal latent space of CLIP. In addition, CLIPure-Cos is the first purification method that is not based on generative models. Experiment results show that purification in multi-modal latent space is promising for zero-shot adversarial robustness.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Conducting purification in multi-modal latent space for adversarially robust zero-shot classification is novel.\n2. The overall organization is reasonable, and the writing is good.\n3. Sufficient experiments are performed, and the proposed method exceeds the comparison methods.", "weaknesses": "1. Line 184: $p_{ben}(X)$ is not mentioned in Eq. (3).\n2. I wonder why $KL((p(x_{adv})||(p(x_{ben}))$ in Eq. (6) can excel at detecting out-of-distribution adversarial examples. Please further explain it.\n3. In the experiment setting, authors should provide more training and testing details, and  Hyperparameter settings.\n4. Providing more visual results about the Purification process like Figure 5 would be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)