Kati8/ICLR_2023_2025 huggingface dataset:
authors: Mingkun Zhang
authors: Keping Bi
authors: Wei Chen
authors: Jiafeng Guo
authors: Xueqi Cheng
reviews: 
Comments: The paper propose two variants (i.e., CLIPure-Diff and CLIPure-Cos) for our CLIPure approach, which explore purification in the multi-modal latent space of CLIP. In addition, CLIPure-Cos is the first purification method that is not based on generative models. Experiment results show that purification in multi-modal latent space is promising for zero-shot adversarial robustness.
Strengths: 1. Conducting purification in multi-modal latent space for adversarially robust zero-shot classification is novel.
2. The overall organization is reasonable, and the writing is good.
3. Sufficient experiments are performed, and the proposed method exceeds the comparison methods.
Weaknesses: 1. Line 184: $p_{ben}(X)$ is not mentioned in Eq. (3).
2. I wonder why $KL((p(x_{adv})||(p(x_{ben}))$ in Eq. (6) can excel at detecting out-of-distribution adversarial examples. Please further explain it.
3. In the experiment setting, authors should provide more training and testing details, and  Hyperparameter settings.
4. Providing more visual results about the Purification process like Figure 5 would be better.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper presents CLIPure, an approach for building adversarially robust zero-shot image classifiers based on CLIP. CLIPure performs purification in the latent space of CLIP to enhance robustness against adversarial attacks. It introduces two purification methods: CLIPure-Diff, which uses DALLE 2's diffusion prior to pure latent vectors ,and CLIPure-Cos, which performs purification by computing cosine similarity between the embeddings of an image and textual embedding of the blank templates. This paper conducts extensive experiments on CIFAR-10, ImageNet, and other 13 datasets and the proposed method demonstrates strong performance, exceeding previous SOTA results.
Strengths: 1. The proposed method performs purification in CLIP's latent space, which is technically sound. 

2. Based on the cosine similarity used in CLIP, this paper further proposes a purification method that is not based on generative models, which improves the computational efficiency.
Weaknesses: 1. The motivation is not strong enough. It seems that the key motivation is simply that performing purification in the latent space of CLIP is promising to achieve improved performance. However, there are only a few discussions about why it is necessary to perform purification methods for CLIP in its latent space and what issues exist with previous methods for CLIP.

2. Some baseline methods need further clarification. It is unclear whether the authors pre-trained or trained the baselines themselves or used publicly available versions. If off-the-shelf models are used, it is necessary to specify the references and discuss whether the difference in training data influences the comparisons. If the authors conducted the training themselves, which dataset is used and what is the setting for the training?

3. It seems that the experimental settings for baselines are not clear enough. For the compared purification methods, the base generative models should be unified for a fair comparison. Especially, it would be better to compare to the purification method with DiffPure based on DALLE2. For the AT methods, as the proposed method is specifically designed for CLIP, further discussion is needed on the comparability between different base models.
Recommendation: 6
Reviewer confidence: 1
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: In this work,the author propose CLIPure, which conducts purification in CLIP's latent space for adversarially robust zero-shot classification.CLIPure leverages the image encoder and text encoder of the CLIP model to achieve effective purification in the latent space.By minimizing the KL divergence between the purification process and the adversarial process, CLIPure reduces purification risks and enhances the model's robustness against adversarial attacks. The algorithm includes two versions: CLIPure-Diff and CLIPure-Cos, which are based on the DiffusionPrior model and the cosine similarity metric of CLIP respectively. Experimental results demonstrate that CLIPure significantly outperforms existing methods when defending against AutoAttack on multiple datasets.
Strengths: This paper introduces CLIPure-Diff and CLIPure-Cos, which perform purification in CLIP's latent space rather than in pixel space. Both methods exhibit significantly larger KL divergence between adversarial and benign sample distributions when compared to methods operating in pixel space and uni-modal latent space. Furthermore, CLIPure-Cos enhances defense efficiency by not relying on generative models. Results demonstrate that CLIPure significantly improves the SOTA robustness.
Weaknesses: 1、Although the article conducts numerous experiments, the selected comparison methods are not entirely appropriate. For instance, the adversarial training method in Table 2 is intended for defending against attacks on the text functionality of CLIP, rather than for zero-shot defense based on CLIP.
2、It is recommended to conduct further comparisons with models in the same direction, such as "Understanding zero-shot adversarial robustness for large-scale models" (ICLR 2023) and "Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness" (CVPR 2024).
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2