Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning
Authors: Zhifang Zhang
Shuo He
Haobo Wang
Bingquan Shen
Lei Feng
Venue: NeurIPS 2025 poster ()
Keywords: backdoor attack
multimodal model
visual prompt tuning
Primary area: deep_learning

Abstract: Multimodal contrastive learning models (e.g., CLIP) can learn high-quality representations from large-scale image-text datasets, while they exhibit significant vulnerabilities to backdoor attacks, raising serious safety concerns. In this paper, we reveal that CLIP's vulnerabilities primarily stem from its tendency to encode features beyond in-dataset predictive patterns, compromising its visual feature resistivity to input perturbations. This makes its encoded features highly susceptible to being reshaped by backdoor triggers. To address this challenge, we propose Repulsive Visual Prompt Tuning (RVPT), a novel defense approach that employs deep visual prompt tuning with a specially designed feature-repelling loss. Specifically, RVPT adversarially repels the encoded features from deeper layers while optimizing the standard cross-entropy loss, ensuring that only predictive features in downstream tasks are encoded, thereby enhancing CLIP’s visual feature resistivity against input perturbations and mitigating its susceptibility to backdoor attacks. Unlike existing multimodal backdoor defense methods that typically require the availability of poisoned data or involve fine-tuning the entire model, RVPT leverages few-shot downstream clean samples and only tunes a small number of parameters. Empirical results demonstrate that RVPT tunes only 0.27\% of the parameters in CLIP, yet it significantly outperforms state-of-the-art defense methods, reducing the attack success rate from 89.70\% to 2.76\% against the most advanced multimodal attacks on ImageNet and effectively generalizes its defensive capabilities across multiple datasets. Our code is available on https://anonymous.4open.science/r/rvpt-anonymous.

Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning
Zhifang Zhang
Shuo He
Haobo Wang
Bingquan Shen Lei Feng'*
Southeast University
University of Queensland
Nanyyang Technological University
*Zhejiang University
National University of Singapore
Abstract
Multimodal contrastive learning models (e.g., CLIP) can learn high-quality rep- resentations from large-scale image-text datasets, while they exhibit significant vulnerabilities to backdoor attacks, raising serious safety concerns. In this paper, we reveal that CLIP's vulnerabilities primarily stem from its tendency to encode features beyond in-dataset predictive patterns, compromising its visual feature resistivity to input perturbations. This makes its encoded features highly sus- ceptible to being reshaped by backdoor triggers. To address this challenge, we propose Repulsive Visual Prompt Tuning (RVPT), a novel defense approach that employs deep visual prompt tuning with specially designed reature-repelling loss. Specifically, RVPT adversarially repels the encoded features from deeper layers while optimizing the standard cross-entropy loss, ensuring that only predic- tive features in downstream tasks are encoded, thereby enhancing CLIP's visual feature resistivity against input perturbations and mitigating its susceptibility backdoor attacks. Unlike existing multimodal backdoor defense methods that typically require the availability of poisoned data involve fine-tuning the entire model, RVPT leverages few-shot downstream clean samples and only tunes a small number of parameters. Empirical results demonstrate that RVPT tunes only 0.27% of the parameters in CLIP, yet it significantly outperforms state-of-the-ar defense methods, reducing the attack success rate from 89.70% 70% to 2.76% against the most advanced multimodal attacks on ImageNet and effectively generalizes its defensive capabilities across multiple datasets. The code publicly available in our GitHub repository:
Introduction
Recently, Contrastive Language-Image Pretraining (CL.P) 522 has emerged as powerful base model across multiple domains. Unlike traditional models that rely on uni-modal supervision, CLIP aligns visual and textual signals to learn rich representations from numerous image-text pairs in the pre- training stage. This unique training paradigm enables its impressive performance in opcn-vocabulary visual recognition (64) and robustness distribution shift [13].
However, despite CLIP's success in visual recognition, recent research has revealed its vulnera- bilities against backdoor attacks in downstream tasks. The vulnerabilities can be roughly categorized into two aspects. First, concerns arise from its training data collection process: the data is noisy, uncurated, and sourced from the web, which facilitates adversarial poisoning of the image-text pairs. Second, CL.IP demonstrates heightened susceptibility to a smaller number of poisoned training samples: recent studies [ have empirically shown that poisoning CLIP may require injecting orders of magnitude fewer poisoned data into the training dataset than poisoning conventional supervised
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gaussian Noise
Gaussian Blur
Color Jitter
Horizonall Plip
Trigger
1.00
1.00
1.00
1.00
Average PR
Average PR
Avorage PR
0.75
0.75
0.75
Average
0.75
0.50
0.30
11
Layer
Layer
Layer
Layer
(a) Backdoored CLIP ASR: 99.83%
(b) VPT Backedourd CLIP
(c) Our RVPT for Backdoored CLIP
(4) SonageNet-Smpervveee VIT
ASR: 9.83%-96666%
L OAR: 99.83%--2.76%
Perturbation Resistivity (PR) across different layers of the encoders under various perturbations, including the trigger pattern. A higher PR value indicates less sensitivity to input perturbation. Four encoder variants are evaluated on ImageNet: (a) ViT in backdoored CLIP (52). (b) ViT in backdoored CLIP tuned with Visual Prompt Tuning (VPT) on ImageNet. (c) ViT of the backdoored CLIP tuned with our RVPT on ImageNet.
Figure 1:
(d) ViT trained exclusively on clean ImageNet [51]. Specifically, CLIP is backdoored by the trigger of BadCLIP [35. Detailed experimental settings and additional PR results for CLIP backdoored by other triggers can be found in Appendix B
models. Once poisoned during pre-training, the backdoored CLIP will inevitably classify images with adversarially designed trigger into the predefined target class.
CLIP's vulnerabilities to backdoor attacks have inspired the development of various defense methods. However, most of these methods necessitate fine-tuning the entire model's parameters [3] or require the availability of poisoned data (68] 24 677, which could be resource-intensive or even unrealistic. Thus, to achieve an efficient, effective, and practical defense, we are dedicated to developing defense method based on visual prompt tuning (VPT) [26], utilizing only a small portion of downstream clean samples to encourage the model to ignore the trigger feature in the poisoned images. | However, simply adopting VPT proves ineffective experimentally (see Figure [b)) as the trigger feature cannot be extracted from clean samples, preventing VPT from learning how to suppress them.
In this paper, we find that CLIP'' visual embeddings remain highly sensitive to small input perturba- tions even after VPT, leaving it susceptible to potential backdoor attacks. To quantify this sensitivity. we introduce the Perturbation Resistivity (PR) metric, defined as the cosine similarity between the visual embedding I (x) of an image x and that of its perturbed counterpart: x + ō:
PR(x, 8) = cfs(ff f"
(x 0)),
(I)
where f'(() denotes the visual feature extractor. A lower PR value implies larger visual embedding
drift due to input perturbations, hence greater sensitivity and higher susceptibility to backdoor triggers. To illustrate this vulnerability of CLIP, we apply suite of perturbations to ImageNet samples and plot the layer-wise mean PR for several model variants. The results in Figure T show that the deeper layers of standard CLIP (Figure aa) and standard CLIP tuned with VPT (Figure [b) exhibit significantly lower PR than a ViT trained exclusively on clean ImageNet (Figure dd),, confirming that CLIP's s embeddings are easily distorted. Because a backdoor trigger acts as structured perturbation. this low PR allows the trigger to maliciously reshape the encoded visual features layer by layer, enabling a successful attack. We therefore regard PR as a proxy for backdoor robustness, which means the backdoor-robust CLIP should have a higher PR than Bacckdoor-ulnereee CLIP). Our analysis suggests that CLIP's low PR originates from its large-scale, noisy pre-training dataset: the encoder captures many off-dataset, non-predictive features, which magnify the impact of perturbations. contrast, the ImageNet-supervised ViT only learns in-dataset predictive features of ImageNet, prohibiting it from encoding irrelevant features beyond ImageNet and achieving much higher PR. Hence, an effective defense (ours shown in Figure [(c) should suppress the irrelevant features CLIP inherently captures and guide it to focus on in-dataset predictive patterns, thereby enhancing both its perturbation resistivity and backdoor robustness.
CLIP (i.e., the standard
Inspired by this finding, we propose a novel approach termed Repulsive Visual Prompt Tuning (RVPT), which improves CLIP's PR by making it exclusively encode the predictive features in the downstream tasks in few-shot manner. As illustrated in Figure 2 RVPT jointly optimizes two
We include the detailed reasons for using VPT defend backdoored CLIP in Appendix A

---

D+I m L-1
Visual Encoder
Enceder Block
Encoder
Block
Downstream image
IE
TE1
Loss
FR Loss
FR Loss
S2
TE2
Visual Encoder
Encoder
Block
Encoder Block
CE
Loss
St
TE
Text Encoder
S
TEC
photo of
Class names
Learnable Parameter
Original Feature
IE
Image Embedding
Similarity Score Repulsive Visual Prompt
Frozon Parameter
Purified Feature
TE
Text Embedding
Illustration of RVPT. RVPT concatenates the features from later layers with tunable visual prompts while keeping the original parameters of CLIP frozen. To optimize the visual prompts, RVPT employs both cross-entropy (CE) loss and feature-repelling (FR) loss. The FR loss minimizes the mean cosine similarity between the prompted features and the original features across layers. Meanwhile, the CE loss ensures the clean accuracy the prompted model. Together, these losses guide the model to encode only in-dataset predictive features that contribute to CE loss optimization, thereby enhancing its backdoor robustness.
Figure 2:
objectives: (1) multi-layer feature-repelling loss that minimizes the cosine similarity between prompted features and original CLIP features each layer, and (2) cross-entropy loss for downstream task accuracy. This mechanism forces the model to only extract features that contribute to CE loss optimization during training, while ignoring non-essential features (e.g., noise trigger pattern). As a result, Figure (c) shows that RVPT significantly improves the PR of CLIP on ImageNet, hence effectively defending against the sfte-of-the-ar multimodal backdoor attack. Moreover, extensive experiments demonstrate our method's effectiveness across multiple datasets and backdoor attacks.
Related Work
Backdoor attacks and defenses on supervised learning. Backdoor attacks have become an increasingly serious security issue because more and more practitioners choose to adopt third- datasets, platforms, even backbones reduce costs. Research on backdoor attacks has primarily
focused designing triggers that enhance stealthiness 7 57 and effectiveness (39 46 For
backdoor defenses, researchers employ various strategies to mitigate backdoor attacks, including:
(1) pre-processing defense (56) (2) pre-training defense (6] (3) post-training defense [76 62 (4)
test-time defense [15]. Specifically, our work aligns with the post-training defense category, which assumes that defenders do not have access to the poisoned training data and must climinate the backdoor threat that has already been implanted into the model. There are two main strategies for post-training defense. (1) Pruning-basee Methods 38 63 These approaches focus identifying
and removing the most suspicious neurons. (2) fine-tuning-T based Methods [60] 76 45 32 These
methods purify the backdoored model by directly tuning its parameters with clean data.
Backdoor attacks and defenses on vision-language models. Although vision-language models (VLMs) 152
25 37 have demonstrated significant improvements across various fields, recent research 69 has revealed vulnerabilities of contrastively pre-trained VL.Ms to backdoor attacks, drawing substantial attention in the field of backdoor attacks and defenses CLIP-like models. In

---

terms of attacks, BadCLIP [35 optimizes visual trigger patterns within dual qual-embedding guided framework, rendering the attack effective and undetectable. Moreover, many backdoor attack methods on multimodal large language models [41 42
40) have been proposed recently. Defense strategies can be primarily categorized into two categories: pre-training defense and post- training defense. For pre-training defense [68,
34
48
43
33
70
67], SAFECLIP [68] prevents learning from poisoned examples by dynamically dividing the training data into the safe and risky sets by cosine similarity and only applying uni-modal contrastive loss to different modalities of the risky set. For fine-tuning defense [3] 66 27). CleanCLIP K reduces the impact of spurious relationships of backdoor attacks by enforcing the model to realign representations for independent modalities. There
24
are also some work focusing on test-time defense on multimodal backdoor attacks 44 18 21
Prompt learning for vision-langen vision-language models. Lately, various fine-tuning methods have been introduced to further enhance the impressive capabilities vision-language models (VI.Ms) across diverse downstream tasks [29] 22 75 65], with prompt learning 77 53 72
emerging as particularly efficient and effective approach. Specifically, CoOp 74 first introduces prompt learning to VLMS and proposes method that optimizes the context of the text prompt, e.g., "a photo of a <CLS>." " while keeping the class name tokens fixed. VPT (26) adjusts the visual features by concatenating the image or image features with the learnable prompt. Besides, BadCLIP M injects backdoor during the prompt learning stage, enabling the malicious alteration of text features through trigger-awand prompts for a powerful attack. We notice that despite the effectiveness of prompt learning for enhancing the model's robustness [27 30 g [73 I., limited research has explored its use in defending against multimodal backdoor attacks. In particular, current post-training backdoor defense methods for VLMs mainly focus on fine-tuning the whole backdoored models, overlooking the potential of prompt learning. To address the gap, we propose RVPT, which incorporates learnable parameters into features to defend against multimodal backdoor attacks while freezing the model's 's parameters.
3 The Proposed Approach
Overview. In this section, we first outline the basic settings of the threat model and defender's 's goal in Section 3.1 followed by a detailed illustration of how the adversary launches backdoor attacks on CLIP in Section 3.22 Eventually, in Section 3.3 present Repulsive Visual Prompt Tuning, designed to defend backdoored CLIIP against multiple multimodal backdoor attacks.
3.1 Basic Settings
Threat model. We assume the adversary has access to the training dataset and can produce a backdoored model by injecting crafted poisoned data into the original dataset. For multimodal attacks, the poisoned data usually consists of images with the trigger pattern and the proxy captions for the target class, Once trained on the poisoned dataset, the model will misclassify inputs with the trigger pattern into the predefined target class while maintaining classify correctly for the benign inputs.
Defender's goal. We suppose that the defender has access the backdoored model and limited set of clean downstream data. The defender's goal is to obtain new model utilizing both the backdoored model and the available clean data, wherein the backdoor effects are mitigated while preserving the benign performance of the new model on the downstream clean data. Furthermore, the defender has full control over the recovering process of the backdoored model.
Clarification and practical motivation. Although the above setting assumes the defender lacks access to the original pre-training data, this assumption is both realistic and practically important. First, the backdoor robustness of many multimodal models depends on CLIP, whose pre-training data is entirely unavailable to the public or defenders. Even though some models partially disclose data sources (e.g.. LAION, CC3M), most real-world multimodal models are trained on large-scale, proprietary, or mixed Internet data, making it infeasible for defenders to reconstruct the same distribution. Recent studies also show that backdoored CLIP models can transfer their vulnerabilities to other multimodal models where CLIP serves as the visual encoder, highlighting the need to defend data-closed models. Second, even when pre-training data is partially available, leveraging it for defense remains highly challenging. Because multimodal datasets are vast and Internet-sourcent locating and removing poisoned samples computationally prohibitive. Using a surrogate dataset for fine-tuning (e.g., CC3M) is an alternative, but it causes feature mismatches and significantly

---

degrades defense performance, as verified experimentally. In contrast, our setting targets defenders with only a small amount of trusted downstream data realistic and widely encountered scenario. Such data-limited but model-accessible defense setting frequently appears in practice. For example, enterprises or institutions may adapt pre-trained CLIP models to downstream tasks (e.g., medical imaging, rare-disease recognition, industrial inspection) but only have few-shot data due to privacy or scarcity. In these cases, defenders can access the backdoored model but not its large-scale training data. Our RVPT method is designed for this setting: it provides an end-to-end defense that adapts CLIP to downstream tasks while mitigating backdoors using only a few clean samples.
3.2 Backdoor Attacks on CLIP
In general, the CLIP model comprises a visual encoder V(.), a textual encoder T()), and projection matrices that project the encoded representations into joint feature space: P1 for the image modality and PT for the text modality. The training dataset contains N₁ image-text pairs represented as
D = = t. )
where t; serves as a short caption for the corresponding image I₁-
During the backdoor attack CLIP in the training stage, adversaries typically inject small number of N₂ poisoned image-text pairs denoted as Dp- {(a.f +))) æ. represents a poisoned image obtained by modifying the original image I; with the trigger pattern θ, while 1. denotes the proxy caption for the target class yt. Consequently, the original benign training dataset DD can be poisoned, resulting in: D = {D U pp). During training, CLIP is optimized on D to maximize the cosine similarity of positive pairs, expressed as Φ(2;, t.) = cos P (t.()))), and minimize the similarity of negative pairs ((;;, Ljyt).
Upon training with D, the encoders of CLIP will be backdoored, denoted as (.))) T(), (()).))) For the backdoored version of CLIP, the trigger pattern θ will have spurious correlation with the proxy caption of the target class yt- Thus, during the inference stage, when the model encounters the images containing the trigger pattern, its prediction on the target class y will increase significantly. In the meantime, the backdoored CL.IP performs normally on benign inputs.
3.3 Repulsive Visual Prompt Tuning
In this part, we present Repulsive Visual Prompt Tuning (RVPT), framework designed defend against multimodal backdoor attacks. Our key insight is to make CLIP ignore trigger features of the attacked images by restraining CLIP's visual encoder to only encode in-dataset predictive features.
Formally, assume the defender is given the backdoored CLIP, denoted by (V(+), T()) alongside a small set of clean labeled samples [x;, Ve ] Yi E y = {1..... C}. In particular, our focus is on CLIP with the Vision Transformer [12 its visual encoder. For clean input 2;, the feature representation at the d-th layer of CLIP's visual encoder is f [d] e²), where of R xd, denotes the class embedding that will finally be projected to the vision-language joint space and e RR are the patch embeddings. Specifically, M and d, are hyperparameters representing the number of patches and dimensions of the visual embedding.
......,C,,
Assume the visual encoder has L layers, then starting from the D-th layer, we introduce D repulsive visual prompts, each of length b, defined as {p" e 126xd, 1 Id=D- These prompts are concatenated with the features from each subsequent block recursively, following the formulation:
( eett1 - = ("c" e,", d=...,
(2)
We then regard c d E [D + 1, as the feature to repel and apply feature-repelling (FR) loss to minimize the cosine similarity between it and its counterpart o in the original fixed backdoored visual encoder. For batch of N,, examples {a:; N. the FR loss is computed as:
PFR L Σ i=1 d=D+1 cos !
No Ee
(3)
Additionally, we also optimize the prompts with the cross-entropy (CE) loss to avoid repelling the in-dataset predictive features and maintain clean accuracy, formulated as:
Σ
No log(
exp(o(x;; T(x))/T)
LCr
_ exp(o(x,,T(o)/+)
5

---

Table 1: Main results. We report ASR (4 %). with CA (%) shown in parentheses on ImageNetIK
Method
BadNet
Blended
ISSBA
WaNet
TrojVQA
BadCLIP
defense
82.69 (63.04)
98.52 (62.64)
60.01 (61.72)
87.18 (62.42)
99.75 (62.8) (1)
99.83 (61.33)
CleanCLIP
23.79 (57.91)
0.25 (57 69)
15.62 (59.20 20)
11.10 (59.07)
85.64 (58.22)
89.70 (57.55)
Linear Prohe
3.05 (59.64)
5.52 (59.69)
0.08 (59.69)
0.65 (59.66)
99.70 (59.33)
RVPT
0.05 (62.76)
0.02 (62.36)
0.01 (61.92)
0.03 (62.48)
0.11 (62.63)
2.76 (61.81)
where T denotes a temperature parameter to control the logit distribution and is set as specified in 52 T() is the proxy caption of the input class, e.g. "a photo of a <CLS>." and ((x;, 4z)
cos (a)(), PtT(t))) means cosine similarity of the outputs from the recovering encoders.
Thus, the overall loss for RVPT is defined as:
L = LCE + aLpR.
(5)
where a is the balancing factor controlling FR loss strength.
For further explanation, RVPT can be interpreted through the lens of natural selection [9. To be specific, FR loss creates challenging environment for visual prompt tuning optimize CE loss. In order to simultaneously minimize CE loss and maximize the discrepancy between the prompted features and original features, the visual prompts must learn to select the most suitable features to optimize CE loss while ignoring those that are non-essential, Ultimately, this process leads to more compact visual encoding mechanism, enhancing CLIP's perturbation resistivity to trigger patterns.
4 Experiments
4.1 Experiment Settings
Models and benchmarks. Following the settings of (3] to create a more realistic backdoored
CLIP for backdoor defense, we fine-tune OpenAl's public checkpoint CLIP-400M [52] using 500K
image-text pairs from the CC3M dataset [55]. out of 1500 are poisoned with various types of backdoor
attacks: BadNet [17]. Blended 77.. ISSBA BIJ WaNet [47], TrojVQA [59] and BadCLIP [35]. We
utilize ImageNet [T]] to evaluate the performance of defense methods against all the aforementioned attacks, with the target class of the attacks set to banana. Additionally, we use Caltech101 [14] and
OxfordPet [50] (a fine-grained dataset) to evaluate defense methods against BadNet, Blended, and WaNet. These attacks target "accordion" in Caltech 101 and "samoyed" in OxfordPet, respectively. We compare RVPT with Linear Probe and CleanCLIP 33. The implementation details of the backdoor attack methods, backdoor defense methods, and benchmarks will be illustrated in Appendix C
Evaluation criteria. The evaluation metrics include Clean Accuracy (CA) and Attack Success Rate (ASR). CA is the model's accuracy on clean images, while ASR measures the proportion of poisoned images predicted as the target label. A lower ASR and decent CA stand for a successful defense.
Implementation details. Unless otherwise specified, we will use ViT/B32 12 as CLIP's visual encoder. The prompt is initialized from a zero mean Gaussian distribution. We set the non-prompted depth D = 3, learnable context length b = 50, and balancing factor a = 2. Moreover, the proxy caption of the class the same as the simple prompt engineering in the original paper of CLIP (52]: (1) "a photo of <CLS>." for ImageNet and Caltech 101 (2) "a photo of <CLS>, a type of pet." for OxfordPets. For all datasets, we randomly sample 16 images per class while setting the batch size and the epoch number to 32 and 50. The loss is optimized with SGD with an initial learning rate of 0.002 decayed by the cosine annealing rule. We conduct experiments on eight NVIDIA RTX 3090 GPUs and the computational expenses are shown in Appendix D
4.2 The Effectiveness of RVPT
In Table and [] we compare RVPT with CleanCLIP and Linear Probe against six attacks on ImageNet and three attacks on Caltech101 and OxfordPets. Before poisoning, CLIP's CA on ImageNet, 101t and OxfordPets is 62.01% % 91 11.46% and 86.42% respectively.
Firstly, we can see all attacks achieve a high ASR while maintaining decent CA with no defense, indicating their successful executions. The CA loss of the poisoned CLIP

---

Table 2: Main results. We report ASR (4 %), with CA (%) in parentheses on Caltech101, OxfordPets.
Caltech101
OxfordPets Blended
Method
BadNet
Blended
WaNer
BadNet
WaNet
No defense
91.38 (93.06)
92.69 (93,41)
63.21 (92.86)
86.83 (82.91)
99.80 (85.10)
87.97 (83.93)
CleanCIIIP IIP
36.87 (91.18)
1.14 (90.77)
9.35 (91.54)
25.72 (82.49)
4.17 (83,41)
12.61 (81.10)
Linear Probe
1.22 (93.62)
12.82 (93.41)
1.04 (93.45)
16.05 (77.74)
2.63 (77.63)
2.21 (77.71)
RVPT
0.00 (94.02)
0.00 (94.34)
0.08 (93.89)
0.30 (88.60)
0.64 (88.87)
1.59 (88.53)
Table 4: Results on emerging target class. We report ASR (4 %)), with CA (%) shown in parentheses. (woTC) indicates RVPT is tuned in the dataset without the target class.
Attack Type ImageNet (woTC)
ImageNet
Caltech101 (woTC)
Caltech101
OxfordPets (woTC)
OxfordPets
BadNet
0.01 (62.50)
0.05 (62.76)
0.04 (93.71)
0.00 (94.02)
0.33 (86.84)
0.30 (88 (88.60) 60)
Blended
0.01 (62.63)
0.02 (62.36)
0.00 (93,89)
0.00 (94.34)
0.00 (87.19)
0.64 (88.87) 87)
WaNet
0.00 (62.46)
0.03 (62.48)
0.00 (93.67)
0.08 (93,89)
0.00 (86.17)
1.59 (88.53)
Table 5: Results on cross-dataset generalization. We report ASR (4 %), with CA (%) shown in parentheses. RVPT* means that the model is tuned on ImageNetIK using RVPT and tested on Caltech101 (target class: accordion) and OxfordPets (target class: samoyed).
Caltech101 101 Blended
OxfordPets Blended
Method
BadNet
WaNet
BadNet
WaNet
CleanCLIP
36.87 (91.18)
1.14 (90,77)
9.35 (91.54)
25.72 (82.49)
4.17 (83,41)
12.61 (81,10)
RVPT* 2.32 (89.33)
0.61 (89.91)
1.80 (89.66)
2.86 (82.65)
0.53 (83.49)
2.49 (82.66)
varies across downstream datasets because to obtain backdoored CLIP, fine-tuning on the poisoned CC3M dataset
[55] changes CLIP's Iearned representations. These changes af- fect how CLIP performs on different datasets, accounting for varying levels of CA loss.
Secondly, although Clean- CLIP mitigates backdoor at- tacks in a zero-shot setting, deteriorates CLIP's recog- nition performance and re- quires more computing re- sources. Morcover, Linear Probe performs well against the first four attacks. How-
Table 3: Poisoned accuracy (t %), defined as the model's classifica-
tion accuracy on poisoned images.
Model
BadNet
Blended
BadCLIP
Clean Model
58.29
54.76
55.34
Backdoored Model
10.36
7.05
0.18
CleanCLIP
39.66
46.
4.25
Linear Probe
28.02
14.94
5.64
RVPT
58.94
53.27
53.90
ever, it does not eliminate the trigger feature and thus performs the worst against BadCLIP, an attack specially designed for multimodal contrastive learning models. Since Linear Probe inevitably excludes CLIP's text encoder, it cannot be used to defend against TrojVQA, which targets the mul- timodal task of visual question answering. Finally, RVPT effectively defends against all types of attacks while maintaining clean accuracy. It demonstrates consistently strong defensive performance across all attacks and, notably, significantly reduces the ASR of BadCLIP (35). which is hard to defend by current defense methods. Moreover, as shown in Table 3 RVPT's classification accuracy on poisoned images is also much higher than other defense methods, highlighting its enhanced capability in maintaining correct predictions under backdoor attacks. Moreover, we observed that even a backoocddo CLIP experiences a significant drop in PA, further supporting our claim of its low visual feature resistivity to input perturbation.
4.3 The Generalization of RVPT's Defensive Ability
We evaluate RVPT's generalization ability in defending backdoor attacks in three scenarios: (I) The target class of the attacks is absent from the tuning dataset but appears during testing. (2) The downstream dataset that will be attacked is different from the tuning dataset. (3) The downstream dataset that will be attacked shares different domains with the tuning dataset.
Performance on the emerging target class. We remove the samples belonging to the target class from the tuning dataset to evaluate RVPT's performance on the emerging target class, which measures its defense capability for unseen target classes in the dataset, and the results are shown in Table 4 Moreover, we also tested BadCLIP (woTC) for the target class of banana and the ASR (CA) is
7

---

Table 6: Results cross-domain generalization. We report ASR (1 %), with CA (%) shown in parentheses. We tune CLIP with RVPT on ImageNetIK K and conduct CleanCLIP on CC3M.
Dataset
Method
BadNet
Blended
BadCLIP
No defense
86.55 (55.39)
99.04 (54.83)
99.89 (53.49)
ImageNet-V2
CleanCLIP
31.38 (50.95)
0.42 (50.96)
92.04 (50.60)
RVPT
0.04 (53.85)
0.02 (53.77)
3,43 (52.53)
No defense
92.97 (31,47) .47)
99.89 (31.22)
99.97 (30.80)
ImageNet-A
CleanCLIP
59.11 (25.71)
3.18 (27.10)
98.18 (25. (25.52))
RVPT
1.64 (16.52)
0.17 (16.84)
12.84 (16.84)
No defense
66.63 (67.11)
97.94 (66.06)
99.69 (65.49)
ImageNet-R R
CleanCLIP
32,99 (61.81)
2.54 (60.69)
89.03 (60.93)
RVPT
0.76 (58.39)
0.09 (58.46)
6.75 (57.37)
defense
92.11 (41.73)
97.16 (41.86)
99.88 (40.19)
ImageNet-S
CleanCLIP
26.54 (34.62)
0.26 (34.82)
85.62 (34.44)
RVPT 0.02 (35.17)
0.01 (35.34)
1.67 (34.06)
3.39 (61.89). Table shows that for the backdoor attack of Blended and WaNet, RVPT performs even better in the absence of the target class, suggesting that RVPT has successfully filtered out the distinctive out-of -nttas-d pattern. Moreover, we can see that CA maintains except OxfordPets, which decreases by around 2%. The CA drop is due to this dataset containing only 37 classes, so the absence of 3% of the data can impact its accuracy. In summary, RVPT's ss defensive capability effectively generalizes attacks with emerging larget classes that are seen in the luning dataset.
Performance on cross-dataset transfer. To evaluate RVPT's defensive ability across datasets, we tuned CLIP with RVPT on ImageNet and tested it on Caltech101 and OxfordPets. The results shown in Table 5 For all three attacks, although ASR increases slightly. it remains at a very low level. Additionally, RVPT's accuracy decreases and occasionally falls below that of CleanCLIP. We conjecture the reason is that ImageNet does not share some of the in-dataset predictive features with Caltech101 OxfordPets. In some cases, The non-essential feature unlearned in ImageNet may be the discriminative features in them and some of their non non-essential features are not removed during tuning in ImageNet. Thus, there is slight CA loss and ASR increase after RVPT. Nevertheless, the in-dataset predictive features of real-world images seem quite similar, ensuring RVPT's cross-dataset defensive ability. Moreover, VPT's tendency overfit the base dataset also contributes to the CA drop. Overall, RVPT's 's performance on new datasets declines slightly but still outperforms CleanCLIP.
Performance on cross-domain transfer. To evaluate the cross-domain defensive ability, RVPT was trained on ImageNet and tested on ImageNet- V2 [54], ImageNet-A 220,, ImageNet-R [19], and ImageNet-S [61]. The result is shown in Table 6 RVPT can generalize its defensive ability across different domains. We also observe that RVPT will experience a loss of recognition and defense performance with domain shifts. In particular, it shows significant CA drop ImageNet- A, which reflects its effectiveness in focusing on in-dataset predictive features and ignoring other features, since ImageNet-A intentionally collects natural adversarial samples for ImageNet classifiers. Notably, unlike adversarial training 2 RVPT is not designed to defend against adversarial samples in ImageNet-A. The detailed comparison between RVPT and adversarial training in Appendix F
4.4 Ablation Studies
Number of shots per class. From Figure 3a), we surprisingly find that only shot make the ASR of Blended and BadNet attacks nearly O and significantly decrease the ASR of BadCLIP and increasing the number of shots further enhances the defense. For BadCLIP, shots are needed to bring the ASR below 5%. Nevertheless, with 32 shots, the defense effectiveness stops improving.
Balancing factor a. We we observe that both ASR and CA decrease as a increases. This observation indicates that the balancing factor is set too high, it will interfere with the encoding of in-dataset predictive features (the adversarial environment is too harsh for CE loss optimization), resulting in performance degradation. However, a relatively small is sufficient to achieve very low ASR for every evaluated dataset.
balance the strengths of the CE loss and FR loss. From Figure 3b),
Visual context length. We show RVPT's performance Figure c)) with changing visual context length 6. It indicates that both ASR and CA optimal at context length of approximately 25 to 50
8

---

114
677
030510
(a) Different shots
Different balancing factors
(c) Different numbers context tokens
Bart
FPT
EPT
12a-
-18
Different depths output)
Different depths (from Input)
Different margins
Figure 3: ASR and CA are evaluated when the hyperparameters of shots, context token length, depth, and margin are changed RVPT. In panel (f), the dashed lines represent the situation when the total FR loss is ablated. More ablation studies of visual encoder architecture and handcrafted text prompt can be referred to in Appendix
tokens. Beyond this optimal range, CA decreases due to overfitting caused by the increasing number of parameters. Moreover, when the length is 5, the ASRs of all attacks drop below 5%, indicating that tuning only 0.5%cc of parameters is sufficient to defend against attacks, demonstrating the efficiency of our method. However, with increasing length, the defensive ability is bounded.
Depth of the visual prompt. Figure shows the performance the depth of the features into which the visual prompts are embedded is varied. It is important to note that "depth" here refers to the number of layers counted from the model's output feature. We also conduct experiments where prompts are stacked from the input feature, shown in Figure Je). The comparison demonstrates that the later layers are more significant for backdoor robustness and recognition performance. We conjecture the reason is that the non-essential features encoded in the earlier layers tend to be sparse and intermixed with desirable features and repelling these features at an earlier stage is difficult and may inadvertently compromise the encoding of discriminative features, resulting in worse performance. Therefore, unless otherwise specified, we will count the depth from the output feature. Moreover, the results in Figure 3d) indicate the improving performance as the number of the prompted layers increases. Particularly, when the depth reaches approximately six layers, both CA and ASR attain their optimal values.
Margin of the repelling strength. To evaluate the potential risks over-repelling, we a margin m of FR Loss CPRM, for each sample æ;: LFRM, max( CrR, m.. A greater m means lower level of repulsion, and m signifies no repulsion (which VPT). The performance of RVPT of different m is shown in Figure f.). We observe that both ASR and CA will increase with larger m. Therefore, for the purpose of achieving superior defense performance, we adopt full repulsion that optimizes the mean cosine similarity -1.
4.5 Further Explorations
Attention map and t-SNE. We analyze the attention maps and t-SNE [88] plots of different models. In Figure 4 RVPT enables the encoder to focus on relevant bird features for clean images, while backdoored model attends to irrelevant ones. For poisoned images, RVPT notices the trigger but avoids overemphasizing it, unlike the backdoored encoder. Figure Sshows that backdoored CLIP and VPT models cluster around the trigger pattern, ignoring original image content [3]. RVPT, however, disrupts this cluster and restores poisoned image representations, reducing the influence of the trigger.

---

(n) Benign image
(a-1) Aftention map for baclddoored CLIP
(9-2) map for RYPT
(a) Backdooped Ch.P
(b) Benign ČLIF
(b) Poisoned image
(b-1) Attention map for baclddored CLIP
(b-1) Attention map for RVPT
VPr
(d) kvri
Figure 4: Last-layer attention map for (a) original (b) poisoned image backdoored model (attacked by BadCLIP) and RVPT.
Figure 5: The t-SNE (88) plots for the rep-
resentations of clean (dotted) and poisoned (star-shaped) images (attacked by BadCLIP).
RVPT endows CLIP with backdoor robustness various downstream tasks. To further evaluate the impact of backdoored CLIP in real-world settings, we assess its performance on downstream tasks: image captioning and text-to-image retrieval in Table 7 For image captioning, we use MSCOCO (36] and LLaVA-11.5 (13B) 37 equipped with backdoored CLIP ViT/L11. A successful attack is defined as generating a caption that includes the target class. For text-to-image retrieval, we use an enriched ImageNet-11 [TO 28 with backdoored CLIP (ViT/B32). We insert one poisoned image into the candidate set and if it appears in the top- 10 matches for clean captions of the target class, count it as a successful attack. Table shows that backdooring CLIP during pre-training can transfer the backdoor behavior to the evaluated downstream tasks. Notably, our RVPT effectively mitigates backdoor effects while preserving clean performance, outperforming baselines 26
Table 7: Results on image captioning (IC) and text-to-image retrieval (TR). For IC, we evaluate backdoor robustness using ASR (4 %), defined as the percentage of generated captions that include the target class. Clean performance is measured using the BLEU score (1 %). For TR, ASR (L %) is defined as the percentage of target-class captions where poisoned image appears in the top-10 most similar images to evaluate the backdoor robustness. We also report the average rank ratio (4. %) of the poisoned image among target-class candidates. Clean accuracy (f %) refers to the percentage of original captions correctly matched with their corresponding images (top-1 similarity).
Image Captioning (IC)
Text-to-Image Retrieval (TR)
Attack
ASR (4 %)
BLEU
ASR (J %)
avg. Rank Ratio (1 %)
CA (%)
Clean model
7.65
95.83
69.34
No defense
95.12
6.22
74.67
9.20
73.24
CleanCLIP
23.03
4.32
13.33
25.54
68.95
VPT
81.21
4.64
22.67
26.80
71.88
RVPT
4.51
82.12
69.89
5
Conclusion
In this paper, we, for the first time, investigated how to utilize limited clean samples and tunable prompts to defend backdoored multimodal contrastively pre-trained model (i.c., CLIP). We empiri- cally showed that CLIP's low visual feature resistivity to input perturbations results in its vulnerability backdoor attacks. To enhance this resistivity, we proposed a novel method called Repulsive Visual Prompt Tuning that guides CLIP to exclusively encode in-dataset predictive features. This method leverages our designed feature-repelling loss minimize the mean cosine similarity between prompted features and the original features across layers while optimizing cross-entropy loss to maintain clean accuracy on the downstream tasks. Comprehensive experimental results on multiple attacks and benchmark datasets demonstrated the effectiveness of our proposed method.
10

---

Acknowledgment
This work is supported by the Big Data Computing Center of Southeast University. Haobo Wang is supported by the Fundamental Research Funds for the Central Universities (No. 226-2025-00004).
References
[1] J. Bai, K. Gao, S. Min, S.-T. Xia, Z. Li, and W. Liu. Badclip: Trigger-aware prompt learning for backdoor attacks on clip. In CVPR, 2024.
[2] T. Bai, J. Luo, J. Zhao, B. Wen, and Q. Wang. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01335 2021.
[3] H. Bansal, N. Singhi, Y. Yang, F. Yin, A. Grover, and K.-W. Chang. Cleanclip: Mitigating data
poisoning attacks in multimodal contrastive learning. In ICCV, pages 112-123, 2023.
[4] N. Carlini, M. Jagielski, C. A. Choquette-Chow et al. Poisoning Web-Scale Training Datasets Practical. In SP, 2024.
[5] N. Carlini and A. Terzis. Poisoning and backdooring contrastive learning. In ICIR, 2022.
[6] W. Chen, B. Wu, and H. Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NeurIPS, 2022.
[7] X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526. 2017.
[8] C.-Y. Chuang, V. Jampani, Y. Li, A. Torralba, and S. Jegelka. Debiasing vision-language models via biased prompts. In CVPR, 2023.
[9] L. Cook. The genetical theory of natural selection - a complete variorum edition. Heredity, 84, 03 2000.
[10] J. Deng, W. Dong, R. Socher, L.JJ. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[11] B. Dong, P. Zhou, et al. Lpt: Long-tailed prompt tuning for image classification. In ICLR, 2023.
[12] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, et al. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2020.
[13] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, Dave, and Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML, 2022.
[14] L. Fei-Fei, R. Fergus, and Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR, pages 178-178 178,7 2004.
[15] Feng. G. Tao, Cheng, G. Shen, X. Xu, Y. Liu, K. Zhang, Ma, and X. Zhang. Detecting backdoors in pre-trained encoders. In CVPR, pages 16352-16362, 2023.
[16] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and hamesssing adversarial examples. arXiv preprini
2014.
[17] Gu, K. Liu, B. Dolan-Gavitt, and S. Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 4747230-47244, 2019.
[18] S. He, Z. Zhang, F. Liu, R. K.-W. Lee, B. An, and L. Feng. A closer look at backdoor attacks on clip.
[19] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo. D. Song, Steinhardt, and J. Gilmer. The many faces robustness: A critical analysis of out-off-distriution generalization. In CVPR, 2021.
[20] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In CVPR, 2021.
[21] H. Huang, S. Erfani, Y. Li, X. Ma, and J. Bailey. Detecting backdoor samples in contrastive language image pretraining, arXiv preprint 210.::22000.18855, 2025.

---

[22] X. Huang, H. Zhou, K. Yao, and K. Han. Froster: Frozen clip is a strong teacher for open- vocabulary action recognition. In ICLR, 2024.
[23] A. Ilyas, S. Santurkar, D. Tsipras, L.. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[24] A. M. Ishmam and C. Thomas. Semantic shield: Defending vision-language models against backdooring and poisoning via fine-grained knowledge alignment. In CVPR, pages 24820- 24830, 2024.
[25] C. Jia, Y. Yang, Y. Xia, Y.-.. Chen, Z. Parekh, H. Pham, Q. Le, Y.-. Sung. Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[26] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In ECCV, pages 709-727, 2022.
[27] J. Kuang, S. Liang, J. Liang, K. Liu, and X. Cao. Adversarial backdoor defense in clip. arXiv preprint arXiv:2409. 1 15968, 2024.
[28] V. Layer. Imagenet-Ik Ik vision- enriched dataset. https::/huggingffee co/ 1k-vl
http://iaannntt--------eeroraaad 2025. Accessed: 1025-05-133
[29] Li, B. Xu, M. Wang, and H. Kun. Semantic similarity driven multi-modal model for rumor detection. ECAI, pages 3749-3756, 2024.
[30] I.. Li, H. Guan, J. Qiu, and M. Spratling. One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In CVPR, 2024.
[31] Y. Li. Y. Li, B. Wu, L. Li, R. He, and S. Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021.
[32] Y. Li, X. Lyu, N. Koren, L. Lyu, B. Li, and X. Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021.
[33] J. Liang, S. Liang, A. Liu, and X. Cao. VI-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. International Journal of Computer Vision, pages 1-20, 2025.
[34] S. Liang, Liang, T. Pang, C. Du, A. Liu, E.-C. Chang, and X. Cao. Revisiting backdoor attacks against large vision-language models. arXiv preprint arXiv:2406. 18844, 2024.
[35] S. Liang, M. Zhu, A. Liu, B. Wu, X. Cao, and E.-C. Chang. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.
[36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740-755. Springer, 2014.
[37] H. Liu, C. Li, Q. Wu, and Y. Lec. Visual instruction tuning. In NeurIPS, 2023.
[38] K. Liu, B. Dolan-Gavia and S. Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Research in Attacks, Intrusions, and Defenses, 2018
[39] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang. Trojaning attack on neural networks. In NDSS, 2018.
[40] Z. Liu and H. Zhang. Stealthy backdoor attack in self-supervised learning vision encoders for large vision language models. arXiv preprint arXiv:2502. 18290, 2025.
[41] D. LU, T. Pang, C. Du, Q. Liu, X. Yang, and M. Lin. Test-time backdoor attacks on multimodal large language models. arXiv preprint wxXiv:240.00857. 2024.
[42] W. Lyu, L. Pang, T. Ma, H. Ling. and Chen. Trojvim: Backdoor attack against vision language models. arXiv preprint arXiv:2409.1922 2024.
[43] W. Lyu, J. Yao, S. Gupta, L. Pang, T. Sun, L. Yi, L. Hu, H. Ling, and C. Chen. Backdooring vision-language models with out-of-distribution data, arXiv preprint arXiv:2410.0/264. 2024.
[44] A. Madry, A. Makelov, I. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.000 2017.
12

---

[45] R. Min, Z. Qin, L. Shen, and M. Cheng. Towards stable backdoor purification through feature shift tuning. In NeurlPS, 2023.
[46] T. A. Nguyen and A. Tran. Input-aware dynamic backdoor attack. In NeurIPS, 2020.
[47] T. A. Nguyen and A. T. Tran. Wanet imperceptible warping-bast -ased backdoor attack. In ICLR,
2021.
[48] Z. Ni, R. Ye, Y. Wei, Z. Xiang, Y. Wang, and S. Chen. Physical backdoor attack can jeopardize driving with lssoonlarge-language models. arXiv preprint, 2024.
[49] Y. Niu, S. He, Q. Wei, Z. Wu, F. Liu, and L. Feng. Bdetclip: Multimodal prompting contrastive test-time backdoor detection, 2024.
[50] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In CVPR, 2012.
[51] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin,
N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, volume 32, 2019.
[52] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision.
In ICML, 2021.
[53] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu. Denseclip: Language- guided dense prediction with context-aware prompting. In CVPR, 2022.
[54] B. Recht, R. Roclofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In ICML,, 2019.
[55] Sharma, N. Ding, Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In ACL, 2018.
[56] B. Tran, J. Li, and A. Madry. Spectral signatures in backdoor attacks. In NeurIPS, volume 31, 2018.
[57] A. Turner, D. Tsipras, and A. Madry. Label-consistent backdoor attacks. arXiv preprint 11Xiv:1912.02771, 2019.
[58] L. van der Maaten and G. Hinton. Visualizing data using t-sne. JMLR, 9(66):5579-260) 2008
[59] M. Walmer, K. Sikka, I. Sur, A. Shrivastava, and S. Jha. Dual-key multimodal backdoors for visual question answering. In CVPR, 2022.
[60] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao. Neural cleanse:
Identifying and mitigating backdoor attacks in neural networks. In SP, 2019.
[61] H. Wang. S. Ge, E. P. Xing, and Z. C. Lipton. Learning robust global representations by penalizing local predictive power In NeurlPS, 2019.
[62] H. Wang, Z. Xiang, D. Miller, and G. Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using maximum margin statistic. In SP, 2024.
[63] D. Wu and Y. Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS, 2021.
[64] J. Wu, X. Li, S. Xu, H. Yuan, H. Ding, Y. Yang, X. Li, J. Zhang, Y. Tong, X. Jiang, al. Towards open vocabulary learning: A survey. TPAMI, 2024.
[65] S. Wu, W. Zhang, L. Xu, S. Jin, X. Li. W. Liu, and C. C. Loy. Clipself: Vision transformer distills itself for open-vocabular dense prediction. In ICLR, 2024.
[66] Y. Xun, S. Liang, X. Jia, X. Liu, and X. Cao. Ta-cleaner: A fine-grained text alignment backdoor defense strategy for multimodal contrastive learning. arXiv preprint arXiv:2409. 171661, 2024.
[67] W. Yang. J. Gao, and B. Mirzasoleiman. Robust contrastive language-image pretraining against data poisoning and backdoor attacks. In NeurIPS, 2023.
[68] W. Yang, J. Gao, and B. Mirzasoleiman. Better safe than sorry: Pre-training clip against targeted data poisoning and backdoor attacks. In ICML, 2024.
[69] Z. Yang, X. He, Z. Li, M. Backes, M. Humbert, P. Berrang, and Y. Zhang. Data poisoning attacks against multimodal encoders. In ICML, 2023.
[70] Z. Yuan, J. Shi, Zhou, N. 7.. Gong, and I.. Sun. Badtoken: Teken-level backdoor attacks to multi-modal large language models. arXiv preprint 2rXiv:2503.1602. 2025.
13

---

[71] J. Zhang, X. Ma, X. Wang, L. Qiu, J. Wang, Y.GG. Jiang, and J. Sang. Adversarial prompt tuning for vision-language models. In ECCV, 2024.
[72] Zhang, Q. Wei, F. Liu, and L. Feng. Candidate pseudolabel learning: Enhancing vision- language models by prompt tuning with unlabeled data. In /CML, June 2024.
[73] Z. Zheng, X. Yue, K. Wang, and Y. You. Prompt vision transformer for domain generalization. arXiv preprint arXiv:2208.00 2201092008.0894,, 2022.
[74] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. IJC,, 130(9):2337-2348 2022.
[75] Q. Zhou, G. Pang, Y. Tian, S. He, and Chen. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection. In ICLR, 2023.
[76] M. Zhu, S. Wei, H. Zha, and B. Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. In NeurlPS, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction in this paper accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the results in this paper can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
wttp://iiggii/////eeeee http://uuddeeeeeeSeuutute ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We run experiments under multiple choices of hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, the paper provide sufficient information the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://///aarrppssccccc//l////la
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used in significant part of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning
Appendix
We summarize the Appendix as follows:
Appendix provides detailed reason of using VPT to defend backdoored CLIP.
Appendix provides more details of the experimented perturbations and results of PR in CLIP backdoored by more attacks.
provides more detail in the experiment's settings of backdoor defense and attack methods.
Appendix
Appendix
provide the detailed settings of backdoor attacks.
Appendix
provide the detailed settings of backdoor defense baselines.
Appendix provide the detailed settings of adopted datasets.
Appendix D compares computational expenses between our method and CleanCLIP.
Appendix contains more ablation studies, such the visual encoder architecture, and the handcrafted prompt. It also contains the setting of experiments that ablates the FR loss.
Appendix E. provide the ablation study for handcrafted prompt.
Appendix provide the ablation study for visual encoder architecture.
Appendix G discusses both the positive and negative of potential impact of our RVPT.
Appendix discusses limitations the proposed RVPT.
A Why Using VPT to defend backdoored CLIP?
A.1 The drawback fine-tuning the whole model
Fully fine-tuning can be done in two ways, both of which exhibit significant drawbacks in our setting.
Fine-tuning whole CLIP with Info-NCE loss. Since our primary downstream task classification, we do not have downstream image-text pairs for optimization. We could use surrogate datasets such as CC3M, but as Table shows, defense performance drops significantly. This degradation occurs because RVPT relies on identifying predictive features specific to the downstream dataset and pruning non-predictive ones. The mismatch between predictive features in CC3M and the actual downstream task leads to poor backdoor robustness.
Fine-tuning whole CLIP with cross-entropy loss. This requires adding additional linear layer on top of the visual encoder for closed-set classification. However, this approach has several brawbacks: (i) It removes CLIP's open-set classification capability, severely limiting its generalization other datasets. (ii) Given the limited labeled data, the large number of tunable parameters will cause overfitting, often resulting in significant catastrophic forgetting.
In summary, fine-tuning the whole CLIP with InfoNCE loss yields poor backdoor robustness and higher computational cost, while using cross-entropy loss slightly improves robustness but the expense of much greater data requirements and loss of model generalization. Neither option provides favorable trade-off between robustness, efficiency, and generalization.
Table ASR (CA) performance of different fine-tuning methods with or without FR loss.
Fine-tuning method
without FR Loss
with FR Loss
Open-set capability
ImageNet with CE Loss
10.09 (60.71)
(59.22)
NO
CC3M with Info-NCE Loss
92.50 (54,50)
18.23 (54.66)
YES
VPT on few-shot ImageNet (preferred)
98.14 (64.32)
2.76 (61.81)
YES
22

---

A.2 The reasons why we use VPT
As outlined above, full fine-tuning CLIP presents significant drawbacks, either in terms of degraded robustness, excessive data requirements, or loss of open-set capabilities. To address these challenges, we adopt parameter-efficieer fine-tuning methods (VPT) that:
Leverage limited labeled downstream data effectively, aligning with RVPT's core mechanism of identifying task-relevant visual features.
Preserve CLIP's open-set classification ability, which is critical for generalization across diverse tasks.
Specifically, since our goal is enhance the perturbation resistivity of the CLIP's visual features, we inject learnable parameters into the visual modality, leading to our choice of VPT. This approach significantly improves downstream backdoor robustness using much less data while retaining CLIP's open-set capability, as demonstrated in Table S
B
More Results of Perturbation Resistivity of More Backdoored CLIP
In the pilot experiment, we evaluate the model's perturbation resistivity against benign perturbations:
Gaussian Noise with a standard deviation of 0.001.
Gaussian Blur with kernel size of 3 and variance of 5,
Color Jitter with brightness of 0.5 and hue 0.3
Random Horizontal Flip.
The visualizations of these perturbations and trigger patterns can be found in Figure and
We evaluate the perturbation resistivity of four models:
ViT/B32 OpenAl CLIP
ViT/B32 of OpenAl CLIP tuned by VPT with 16 shots ImageNet.
ViT/B32 of OpenAl CLIP tuned by RVPT with 16 shots on ImageNet.
VIT/B32 supervised exclusively ImageNet.
The visual illustration for backdoored CLIP BadNet and Blended are shown in Figure respectively. The results show that CLIP, even after VPT has generally much less PR value against various perturbations than traditionally supervised ViT, and RVPT can consistently increase its PR value across all scenarios.
(a) Gaussian Noise
(b) Gaussian Blur
(c) Color Jitter
(d) Horizontal Flip
Figure 6: Visualization of the perturbations of various experimented attacks.
C Detailed Settings
C.1 Detailed Settings of Backdoor Attacks
In our experiments, we evaluate six prominent backdoor attack methods: BadNet 17.. Blended
77] BadCLIP (35) ISSBA (3I) WaNet [7). and TrojVQA [59]. Below, we provide a detailed description of each method:
23

---

Gaussian Noise
Gaussian Blur
Color Jitter
Horizontal Flip
Trigger
CLIP
VPT
RVPT
VIT
1.00
1.00
1,00
1.00
Avorage PR
Average PR
Average PR
Avorage PR
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
3
3
5
7
9
3
5
7
11
5
9
11
Layer
Layer
Layer
Layer
Figure 7:
Average PR values of CLIP backdoored by BadNet.
Gaussian Noise
Gaussian Blur
Color Jitter
Horizontal Flip
Trigger
CLIP
VPT
RVPT
VIT
1.00
1.00
1.00
Avorage PR
Average PR
Average PR
PR
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
Layer
Layer
Layer
Layer
Figure 8: Average PR values of CLIP backdoored by Blended.
BadNet, a foundational approach to backdoor attacks deep learning, generates poisoned samples by embedding small, randomly placed patch into images and modifying their labels to match the target class. our implementation, the patch size is set to 16 pixels.
Blended enhances the stealth of backdoor attacks by focusing on trigger design. This method linearly blends the trigger with the original image, creating an almost imperceptible backdoor. We adopt a blending ratio of 0.2 for the trigger to ensure minimal visibility to the human.
BadCLIP targets CLIP models with a backdoor attack that optimizes visual trigger patterns using a dual-embedding guided framework. This method ensures that the attack remains undetectable by leveraging both text and image embeddings. We follow the parameter configurations the original paper to implement BadCLIP.
ISSBA introduces a highly covert attack by generating sample-specific triggers. These triggers embedded into benign images using encoder-decoder architecture, encoding a predefined string into the images. Moreover, the ciphertext is the string 'Stega!!'.
WaNet proposes warping- -baed triggers to make the attack less noticeable to humans. Specifically, we use control grid size 224 and warping strength s and train models without the noise mode.
TrojVQA proposes a dual-key backdoor attack that sets triggers on both modalities. To evaluate RVPT under backdoored generative VI.Ms, we use TrojVQA as the backdoor attack and use the following settings. Firstly, we optimize the visual triggers for the CLIP visual backbone with the description "This is yellow banana.". Specifically, We set the patch size to 16x 16 and located the patch in the middle of the image. Then, we randomly corrupt the text for poisoned samples into sentences that describe the target class of banana. Moreover, we add the trigger word "SUDO" in front of each sentence to make sure two keys for one backdoor. In the inference stage, the optimized trigger is added to the image, and "SUDO" is added to the beginning the prompt.
We implement the attacks following [3] Specifically, we employ the AdamW optimizer with an initial learning rate of le-6 for all the backdoor attack methods. The learning rate follows cosine decay schedule over five epochs, and we the batch size 128. The visualization of each trigger pattern is in Figure 9

---

BadNet
(b) Blended
(c) BadCLIP
(d) ISSBA
(e) WaNet
(I) TrojVQA ('SUDO' for text)
Figure 9: Visualization of the triggers of various experimented attacks.
C.2 Detailed Settings of Backdoor Defenses
CIea CleanCLIP B] provides a defense against backdoor attacks in multimodal contrastive learning by optimizing the integration of multimodal contrastive and unimodal self supervueee losses, relying on a limited amount of clean data. It is important to note that CleanCIIIP utilizes ResNet-50 as the backbone of its visual encoder. In contrast, our work adopts the Vision Transformer (ViT- B/32) as the visual encoder, and we have accordingly adjusted the parameters to fit this architecture. Specifically, we randomly selected 250,000 image-text pairs from the CC3M dataset for fine-tuning. The learning rates were set to 5e-6 for BadNet, TrojVQA, Blended, and BadCLIP, while for WaNet and ISSBA on ImageNet-IK,, they were set to 3e-6. We used batch size of 64 and fine-tuned the models over 10 epochs. Notably, we did not focus solely on reducing the attack success rates by manipulating learning rates; instead, we ensured that the clean accuracy of the fine-tuned model was consistently maintained throughout.
Linear Probe Linear Probe [5] adds tunable linear layer on top of the visual encoder, which seems efficient and effective in defending backdoor attacks. It does not truly purify the hidden trigger features. Consequently, it becomes ineffective against of-t--of-the-ar multimodal attacks, whose trigger pattern aligns with the local image features [35]. This paper uses the same settings in [12] to adopt ViT-B/32 using the linear layer. Specifically, we use SGD as the optimizer and use the constant learning rate of 0.01 and momentum of 0.9. The batch size is 32, and the total training epoch is 40.
C.3 Detailed Settings of Datasets
This paper assesses ASR and CA using three downstream datasets: ImageNet (10), Caltech
[4] and OxfordPets [50]. For cross-domain evaluation, ImageNet-V2 (54). ImageNet-A (20).
ImageNet-R [19], ImageNet-Sketch [6]] are also evaluated. Additionally, CleanCLIP selects clean
image-text pairs from CC3M (55) for fine-tuning the backdoored CLIP model. Below are detailed
descriptions of these datasets:
ImageNet1 K includes 1,000 classes with over million images, presenting a complex, large-scale dataset for image classification challenges.
25

---

Caltech101 comprises 101 object classes and one background category, with each class containing between 40 and 800 images. It is widely used for evaluating models on fine- grained classification and image recognition.
OxfordPets is ane-ggrined dataset with 37 categories and approximately 200 images per class. The images display wide variations in scale, pose, and lighting.
ImageNet- V2 was developed assess model generalization under temporal shifts. It closely mirrors the original ImageNet's 's data collection and annotation processes, using new samples sourced from the Internet.
ImageNet-A is designed to compile images that present notable challenges to deep learning models, often causing misclassification and thereby highlighting potential model vulnerabil- ities.
ImageNet-R features "non-real" imagery, such as artworks, cartoons, and graphical repre- sentations, which differ significantly in visual style and form from the natural images found in the original ImageNet.
ImageNet-Sketch contains hand-drawn sketches of the same categories ImageNet, charac- terized by focus on lines and shapes rather than colors and textures, presenting distinet departure from photographic imagery.
CC3M, or Conceptual Captions, includes around 3.3 million images paired with captions. Unlike other datasets with curated annotations, CC3M's ' image descriptions are sourced from web Alt-text attributes, thus offering broader range of descriptive styles and contexts.
D Computational Expenses
We conducted the main experiments on eight single NVIDIA 3090 GPUs using half-precision data type and recorded one training process in Table The results indicate that our method substantially reduces computational costs, enhancing defense efficiency.
Table 9: Computational expense comparison between RVPT and CleanCLIP.
Method
Training time
GPU memory
Parameters
Training samples
CleanCLIP
3:53:02
17640 MB
126 M
250K
RVPT on ImageNet
45:05
3382 MB
0.34 M
16 K
RVPT on OxfordPets 2:19
1010 MB
0.34
0.6
E More Experiments for Ablation Study
E.1 Ablation Study of the Handcrafted Prompt
We experiment with the handcrafted prompt of "<CLS>" and "ННпННя <CLS>" and show the result in Table TO From the result, attacks with different handcrafted prompts.
conclude that RVPT still successfully defends against backdoor
Table 10: Performance of RVPT with different handcrafted prompts. We report ASR (L %), with CA (t %) shown in parentheses. We can see the RVPT is stable regarding different handcrafted prompts.
ImageNet blended
Handcrafted Prompt
BadNet
BadCLIP
"<CLS>"
0.09 (62.28)
0.03 (62.20)
3.72 (61.73)
"HUINTH <CLS>" photo of <CLS>"
0.05 (62.18)
0.02 (61.96)
3.78 (61.37)
0.05 (62.76)
0.02 (62.36)
2.76 (61.81)
E.2 Ablation Study of the Visual Encoder Architecture
To evaluate RVPT across different architectures, We poisoned different architectures of CLIP using the same hyper- parametrs and show the result in Table II In particular, for the sake of faimess,
26

---

keep the RVPT training setting for ViT-B/16 and VIT-L/14 the same as for ViT-B/32. The result shows that all attacks have been launched successfully, so we can conclude that RVPT still successfully defends against backdoor attacks in various visual encoder architectures.
Table II:
Performance of RVPT on attacked samples of ImageNet across different visual architectures. We report ASR (4 %), with CA (t %) shown in parentheses. We can see the RVPT consistently successfully defends against the backdoor attacks in various visual encoder architectures.
Method
BadNet
Blended
BadCLIP
ViT-
No defense
99.61 (68.04)
98.30 (67.84)
99.67 (68.09)
Linear Probe
7.11 (67.52)
0.21 (67. .0)
97.73 (67.60)
RVPT
0.66 (68.51)
0.02 (68.32)
3.38 (68.42)
ViT-L/14
No defense
98.76 (73.88)
99.02 (74.25)
99.87 (74.68)
Linear Probe
23.48 (72.31)
0.65 (72.35)
99.35 (72.35)
RVPT
2.08 (73.60)
0.05 (73.26)
0.59 (74.04)
F
Discussion about the Difference between RVPT and Adversarial Training
As illustrated in Section 3 RVPT adopts the FR loss to adversarially repel features impede the learning process, which partly shares some similarity with adversarial training (AT) (2). While this approach shares some superficial similarities with adversarial training (AT) 2], there are fundamental differences between the two methodologies.
First, AT generates the predictive yet brittle features 23 via gradient-based attacks [16 44] and then unlearns them with correct labels. (first pick bad features then unlearn) In contrast, RVPT first actively scrambles the feature representations and then learns the most predictive features. (first scramble features then pick good ones)
As result, AT prevents the model from encoding the predictive, brittle features, while RVPT encourages the model to encode the in-dataset predictive features. In the training process of RVPT, there are adversarial features unlearned. Therefore, it cannot defend against adversarial attacks. Table compares the performance of RVPT and a representative adversarial training method of CLIP under adversarial attack. However, there are non-predictive features out-of-datasse features unlearned, which ensures RVPT to defend against backdoor attacks.
Table 12: Accuracy of RVPT and adversarial Prompt Tuning [71 of adversarial samples ImageNet. The attack is PGD-10, which maximizes the cross-entropy loss with budge of 8/255.
No defense
RVPT
Adversarial Prompt Tuning
Accuracy
4.37
4.10
13.97
G
Broader Impacts
Positive Impact
Improved security: The proposed Repulsive Visual Prompt Tuning (RVPT) significantly improves the robustness of multimodal models (e.g., CLIP) against backdoor attacks, thereby enhancing the security of AI systems deployed in various real-world applications. This defense mechanism helps protect systems from adversarial manipulation, improving their reliability in tasks such as image captioning and text-to-image retrieval.
Cost-effective defense: RVPT only requires small number of clean samples and tunes only few parameters, making it efficient and low-resource method compared to traditional full-model fine-tuning approaches. This could lower the barrier for implementing robust defenses in practical AI systems.
27

---

Cross-domain generalization: RVPT shows promise across different datasets and domains, which enhances its applicability to broad range of tasks and platforms, making it versatile defense tool in multimodal Al systems.
Negative Impact
Potential misuse: While RVPT is designed defend against backdoor attacks, its approach could potentially be misused to manipulate Al systems unintended ways. For example, adversaries could leverage similar techniques create enhance stealthy backdoor attacks, making the technology a double-edged sword.
Overfitting and trade-offs: The need for some fine-tuning and the requirement for clean dataset means that the model might not be fully adaptable to all scenarios, leading to potential overfitting to specific tasks or datasets. This could limit its robustness in highly dynamic unseen environments.
* Computational costs: While RVPT is more efficient than full-mode fine-tuning, it still requires additional computational resources for training and testing, which could be challenge for low-resource environments or in large-scale deployments.
H
Limitations
Limited dataset generalization: Although RVPT performs well across multiple datasets, there instances where performance drops slightly, particularly when the training data and test data are from different domains (e.g., ImageNet vs. Caltech101 or OxfordPets).
Dependence on clean data: RVPT's reliance on small number of clean samples limits its applicability in scenarios where clean data is scarce or unavailable. This could restrict the use of RVPT certain contexts where adversarially poisoned data prevalent.
Vulnerability to new attack strategies: While RVPT is effective against known attacks, might not fully defend against novel attack strategies that do not rely on traditional triggers. As with all defense mechanisms, its robustness against future threats is not guaranteed.
28


Reviews:
[{"review_id": "xw0WlDyrNN", "reviewer": "Reviewer_g6BJ", "summary": "The paper introduces Repulsive Visual Prompt Tuning (RVPT), a lightweight defense mechanism against multimodal backdoor attacks targeting CLIP. The central idea is to improve perturbation resistivity, a metric proposed by the authors to measure the sensitivity of CLIP's embeddings to input noise. This is achieved by combining a cross-entropy loss with a novel feature-repelling loss, which discourages the encoding of irrelevant or potentially malicious features. The method updates only a small subset of the model's parameters and uses a clean dataset to mitigate backdoor vulnerabilities without requiring full model fine-tuning. Experiments across various attacks, datasets, and settings show substantial reductions in attack success rates while maintaining clean accuracy.", "questions": "What specific real-world scenario justifies the assumption that the defender only has access to a few clean samples?\n\nIs the poisoning ratio consistent across all attacks, and if so, what value is used?\n\nHow does RVPT compare to standard fine-tuning on the same clean data in terms of both defense performance and parameter efficiency?", "limitations": "Yes! However, the limitations are discussed in the appendix but not in the main body of the paper. I encourage the authors to move this discussion into the main paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is generally well-written, and the authors conduct multiple ablation studies to deepen the understanding of the proposed approach. The main idea is simple and shows promising results against multiple defenses.\nHowever, there are several important points that warrant further consideration.\n\n**Threat Model Clarity and Realism** The paper assumes a defender who has access to the backdoored model and a limited set of clean downstream data. However, the rationale behind this \"few-shot clean data\" constraint remains underspecified. In real-world settings, defenders often obtain models from repositories like HuggingFace, which typically include model cards and documentation about the training data, often enabling access to the same or a similar dataset. Suppose the assumption is instead that the model is trained privately and poisoned before deployment. In that case, the clean data likely originates from the same source, raising the question: why is only a small sample retained? \n\n**Comparison to Simple Baselines.** RVPT is presented as a lightweight and highly effective defense strategy. However, the empirical evaluation does not convincingly isolate the contribution of its core components, nor does it compare to critical baseline alternatives. The most immediate missing baseline is standard fine-tuning (considering the same set of parameters utilized by RVPT) of the visual encoder on the few-shot clean data. This would establish whether RVPT provides a tangible advantage over conventional tuning strategies. Similarly, input preprocessing-based defenses are notably absent. Techniques such as JPEG compression [i] or test-time sanitization [ii-v] are well-known strategies that aim to erase or obfuscate triggers at inference time. Their inclusion would help gauge whether RVPT adds substantive value over these established techniques. Finally, the comparison lacks recent VLM-specific defenses such as RoCLIP [vi], which are particularly relevant given the paper's multimodal focus and claims regarding robustness on contrastive architectures.\nTo summarize, actionable points for the authors are to add comparisons to (i) fine-tuning on clean samples with the same parameter budget; (ii) at least a representative input-sanitization technique; and (iii) state-of-the-art methods like RoCLIP. Alternatively, clarify why certain baselines were excluded if they are not adopted.\n\n[i] Xue et al. Compression-resistant backdoor attack against deep neural networks, 2023. \n\n[ii] Liu et al. Neural trojans. 2017 \n\n[iii] Li et al. Backdoor attack in the physical world. 2021 \n\n[iv] Sun et al. Mask and restore: Blind backdoor defense at test time with masked autoencoder. 2023. \n\n[v] Shi et al. Black-box backdoor defense via zero-shot image purification. 2023. \n\n[vi] Yang et al. Robust contrastive language-image pretraining against data poisoning and backdoor attacks. 2024.\n\n**Evaluation Coverage: Attacks and Poisoning Budget.** The set of attacks evaluated, while reasonably diverse, omits several stealthier or defense-aware backdoor methods. In particular, attacks like Reflection Backdoor [vii] and LIRA [viii] are known for being more covert and resilient to representation-based defenses. Given that RVPT relies on perturbation sensitivity, it may be less effective against these stronger attack strategies.\n\n**Evaluation coverage on poisoning budget.** It is also unclear whether the poisoning budget is kept constant across all attacks, or what specific poisoning ratio is used. This aspect of the setup is left unspecified (or hard to find). More importantly, the paper lacks an ablation study exploring how RVPT performs as the poisoning ratio increases. \n\n**Abstract figure**. The methodology section (3.3) introduces the main methodology proposed in this paper, which can be somewhat complex to understand from the text alone. Figure 2 attempts to provide a schematic overview but lacks an intuitive flow or entry point. Please consider this as minor feedback, which, however, could make the paper more accessible and reader-friendly.\n\n**Related work.** The related work section primarily discusses classic defenses and some recent multimodal ones (CleanCLIP), but omits broader classes of defenses, especially input sanitization and prompt-based robustness as [i-vi]. \n\n**Terminology**. While the paper is generally well-written, a few choices in phrasing and terminology could be refined: The use of \"perturbation resistivity\" as a term is unusual and little related to its mathematical definition. From my perspective, a more direct term could be \"embedding robustness\". That said, this is a stylistic suggestion and does not affect the paper's overall quality.\n\nThe paper presents an interesting and promising approach. However, its contribution would benefit from a more precise presentation and formalization of the threat model, more comprehensive comparisons against state-of-the-art attacks and defenses, and a stronger connection to existing literature. The \"few-shot clean data\" assumption, as currently framed, lacks grounding in realistic scenarios. Additionally, the limited baseline scope makes it challenging to evaluate the practical applicability of the method in broader settings.", "was_revised": true, "final_justification": "I appreciate the authors' thorough ablation studies and their effort to clarify aspects of the threat model through discussion. However, I remain concerned about several aspects that require further development before the work can be considered ready for publication.\n\nFirstly, the threat model as currently presented appears overly limited and insufficiently realistic. The most critical assumption is related to the fact that the threat model assumes the defender has access to only a limited number of clear examples, without a clear justification for this constraint. However, in realistic scenarios, defenders often acquire models from sources where information about the training data is at least partially available, or where the underlying dataset is drawn from well-known public corpora. Alternatively, even when the training data is not publicly released (as in the case of CLIP), neither the defender nor the attacker typically has direct access to it (unless the attacker is the original model provider, such as OpenAI for CLIP), making it infeasible to execute the attack. In essence, in both cases, the attacker and defender may share comparable levels of access to potential data sources, which invalidates the asymmetry in data access assumed by the threat model in this paper.\n\nUltimately, although it is a minor point in the final assessment, the rebuttal now presents numerous experiments, clarifications on the threat model, and additional discussions that should be integrated into the main paper, necessitating a major revision of the paper.\nTherefore, in light of the above, I maintain a slightly negative score but remain open and encourage further discussion with the other reviewers.", "rebuttal": "Thank you for your insightful comments!\n\n## W1: Unrealistic threat model\nThanks for raising this concern. We agree that defenders may have access to the original training data for open-sourced pre-trained models and private training. However, there are also many scenarios where the **pre-training data of closed-source foundation models (such as OpenAI's CLIP or ChatGPT) is unavailable** to the defenders. In this case, defenders can only defend the potential backdoored model with their limited downstream data. Our assumption is specifically based on the scenario where defenders have no access to the pre-training data and can only use their own limited downstream data, which we believe is realistic since there are more and more closed-sourced models being released and adopted.\n\n\n## W2: Lack of baseline comparison\nThanks for raising this concern.\n**Comparison to standard fine-tuning.** We appreciate you highlighting the importance of including a standard fine-tuning baseline. This comparison is indeed presented in Figure 3(f) of our paper, where the result for standard fine-tuning (also referred to as VPT) corresponds to the case when the margin is set to 1. To further demonstrate RVPT's superiority in providing robustness against backdoor attacks, we also include comparisons with other parameter-efficient fine-tuning methods that share similar parameter budgets in Table 1.\n\n**Table 1：Attack Success Rate (Clean Accuracy) of multiple methods against different backdoor attacks. VPT represents standard fine-tuning with the same parameter budget as RVPT.**\n| Method | Random | Blended | BadCLIP |\n| :--- | :--- | :--- | :--- |\n| **CoOp** [1] | 45.65 (65.78) | 46.11 (65.77) | 99.73 (64.49) |\n| **Linear Probe** | 3.05 (59.64) | 5.52 (59.69) | 99.70 (59.33) |\n| **MaPLe** [2]| 61.72 (63.77) | 60.19 (63.92) | 98.85 (62.67) |\n| **VPT** [3] **(standard fine-tuning)**| 65.06 (40.20) | 64.93 (47.95) | 98.14 (64.32) |\n| **RVPT** | **0.05** (62.76)| **0.02** (62.36) | **2.76** (61.81) |\n\n\n\n**Comparison to other defense strategies.** Regarding other types of defenses, such as input preprocessing (e.g., JPEG compression) and in-training defenses (e.g., RoCLIP). We categorize these as orthogonal to our post-training approach. Our primary focus is on defending a model after it has been trained, ensuring it is not susceptible to manipulation by poisoned samples during inference. Differently, input-preprocessing defenses are test-time techniques that operate without modifying the underlying model, which remains backdoored. In-training defenses aim to prevent backdoors during the training phase, and thus assume access to and control over the training pipeline. Nevertheless, to further highlight RVPT's effectiveness, we compare RVPT directly with these defense baselines. Actually, since these defenses operate at different stages of the model lifecycle, RVPT can even be **combined** with them to achieve enhanced backdoor robustness. The results are summarized in Table 2.\n\n**Table 2: The comparisons of defense strategies with and without RVPT against BadCLIP.**\n|Method| JPEG-compression [4] | ZIP [5]|RoCLIP [6]|RVPT|\n| -------- |  -------- |-------- |--|--|\n|without RVPT|94.12 (55.88)|98.2 (61.40)|3.92 (61.53)| **2.76** (61.81) |\n|with RVPT| **0.14** (56.02) | **1.62** (62.31) |**0** (63.14)  | / |\n\n\n## W3: Evaluation of more attacks\n\nThanks for the suggestion. In response, we have expanded our evaluation to include the two recommended more covert backdoor attacks: LiRA and Reflection Backdoor. The results shown in Table 3, demonstrate that our method remains highly effective even against these stronger and more stealthy attack strategies.\n\n**Table 3: The performance of RVPT and CleanCLIP against two covert attacks.**\n| Method | Reflection [7]| LiRA [8]|\n| -------- | -------- | -------- |\n|No defense|  90.85 (63.06)    |  98.72 (63.15)    |\n|CleanCLIP| 17.99 (56.44)     |   14.25 (57.18)   |  \n|RVPT|   **0** (63.58)   |  **0** (63.91)    |  \n\nAlthough these attacks use triggers that are stealthy in the model's representation space, RVPT effectively neutralizes them. This robustness stems from RVPT's core mechanism: it identifies and prunes features from the pre-trained model that are non-essential to the downstream task. Consequently, if trigger patterns do not align with downstream task semantics, they are eliminated—regardless of how inconspicuous they may be in the representation space.\n\n\n## W4: Evaluation of more poisoning budget\nThanks for your insightful comments. \n\n**Poisoning budget settings.** We would like to clarify that the details of the poisoning budget are provided in Section 4.1 (Experiment Settings), specifically in **Lines 183-186**, where we outline the specific poisoning budget used for the attacks.\n\n**Performance across different poisoning ratios.** We fully agree that a robust defense should maintain effectiveness under varying levels of poisoning. To this end, Table 4 demonstrates that RVPT consistently performs well across a wide range of poisoning ratios.\n\n**Table 4: Performance of RVPT across different poisoning rates (pr). The default setting in the main paper corresponds to pr = 3e-3.**\n| Method | pr=1e-4 | pr=5e-4 |pr=1e-3 | pr=3e-3 |pr=1e-2 |  pr=1e-1 |\n| -------- | -------- |--- |-------- |-------- |-|-------- |\n|No defense|  0.03 (62.76) | 63.93 (62.87)  |  86.13 (62.85)    |99.83 (61.33)|   99.81 (62.98)   |   99.99 (62.83)  |\n|RVPT|   **0.02** (62.63)   | **0.09** (62.62)  |**1.09** (62.66)   | **2.76** (61.81)|  **2.07** (62.69)    |  **1.03** (62.76)   |\n\n## Other minor issues\n\nWe sincerely thank you for these constructive suggestions to improve the paper's clarity. In the revised manuscript, we will:\n\n- Redesign Figure 2 to have a more intuitive, step-by-step flow.\n- Expand the Related Work section to include more defenses, such as input-sanitization and prompt-based defenses.\n- Consider adopting your suggestion to use the term \"embedding robustness\" throughout the paper for better clarity.\n\n## References\n[1] Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n[2] MaPLe: Multi-modal Prompt Learning, CVPR 2023.\n\n[3] Visual prompt tuning, ECCV 2022.\n\n[4] Compression-resistant backdoor attack against deep neural networks, Applied Intelligence 2023.\n\n[5] Black-box Backdoor Defense via Zero-shot Image Purification, NeurIPS 2023.\n\n[6] Robust contrastive language-image pre-training against data poisoning and backdoor attacks, NeurIPS 2023.\n\n[7] Reflection backdoor: A natural backdoor attack on deep neural networks, ECCV 2020.\n\n[8] LIRA: Learnable, Imperceptible and Robust Backdoor Attacks, ICCV 2021."}, {"review_id": "PgVSj4AVXD", "reviewer": "Reviewer_3JXv", "summary": "This paper introduces a novel backdoor attack defense method, Repulsive Visual Prompt Tuning (RVPT), which adversarially pushes the encoded features from deeper layers away from those of the backdoored model while still optimizing performance on downstream tasks. The RVPT method enhances CLIP’s visual feature robustness against various backdoor attacks and under diverse conditions.", "questions": "1. I wonder whether the proposed feature repelling loss can be applied to models that are not contrastively trained like CLIP.\n2. It is unclear how the method behaves when applied to an unpoisoned model. Does RVPT improve robustness even in the absence of a backdoor, or does it unintentionally degrade the model's performance on clean inputs?\n3. I cautiously hypothesize that CLIP’s strong zero-shot performance may partly stem from its ability to capture a wide range of off-dataset or non-predictive features. I am curious whether the RVPT-processed CLIP model maintains this generalization capability, particularly in zero-shot settings.", "limitations": "yes. They are in Appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper conducts extensive experiments under diverse conditions, including various datasets, various types of backdoor attacks, and cross-domain settings.\n2. The authors provide attention map and t-SNE analyses, offering visual evidence that the proposed RVPT method effectively removes backdoored features.\n3. The paper is clearly written and easy to follow.\n\nWeaknesses:\n1. This paper shows extensive experiments on various conditions, but they are all conducted on the CLIP architecture. As the title refers to multimodal backdoored models, I recommend to to demonstrate robustness on other multimodal models such as BLIP. \n2. Limited applicability due to assumption of a Poisoned CLIP. However, in most real-world scenarios, such prior knowledge is unavailable; the poisoning status of the model is typically unknown. This assumption limits the practical applicability of the approach.", "was_revised": true, "final_justification": "This paper proposes RVPT, a novel backdoor defense method that repels encoded features from those of a backdoored model while preserving downstream performance, thereby enhancing CLIP’s robustness across diverse backdoor attacks and settings. While the BLEU score drop for LLaVA-1.5 is notable, the authors’ explanations addressing my other concerns are convincing, and I therefore maintain a positive stance.", "rebuttal": "Thank you for your insightful reviews!\n\n## W1 & Q1: More multimodal architectures\nThanks for this excellent suggestion. We believe that the feature repelling (FR) loss can be applied to multiple multimodal models, but it will only work on contrastive pre-trained models, since FR loss enhances the model's backdoor robustness by eliminating irrelevant features that do not contribute to cross-entropy (CE) loss. To compute CE loss, we have to adopt contrastive pre-trained models since their outputs are aligned into one shared space, which allows CE loss to be computed directly via cosine similarity during fine‑tuning. Therefore, FR loss probably remains ineffective for models trained without contrastive pre‑training, since their representations cannot be meaningfully compared through dot products.\n\nImportantly, most prevailing multimodal models, including recent large vision‑language models (LVLMs), adopt visual encoders that are pre-trained contrastively. This design of LVLMs has become the dominant trend for future multimodal systems, as contrastive pre‑training unifies the visual and textual embedding space, making it easier to align LLM's embedding space with visual encoders. \nTherefore, providing backdoor robustness to contrastively pre‑trained multimodal models is not only practical but also highly impactful for the field. For example, we defend the contrastively pre‑trained visual encoder of LLaVA‑1.5 using RVPT. As shown below, RVPT substantially reduces the attack success rate (ASR) in the image captioning task, while preserving a competitive BLEU score (stands for model utility).\n\n**Table 1: ASR (defense effectiveness) and BLEU (model utility) performance of multiple methods when defending LLaVA 1.5 in the image captioning task.**\n\n| Method       | ASR (↓ %) | BLEU (↑) |\n|--------------|-----------|----------|\n| Clean model  | -         | 7.65     |\n| No defense   | 95.12     | 6.22     |\n| CleanCLIP    | 23.03     | 4.32     |\n| VPT          | 81.21     | 4.64     |\n| **RVPT**     | **0.00**  | 4.51     |\n\nThis demonstrates that RVPT can effectively mitigate backdoor risks in real downstream applications while maintaining utility, reinforcing its importance for the broader LVLM ecosystem.\n\n\n## W2: Limited applicability\n\nThanks for raising this critical point. We agree that, in most real-world scenarios, the poisoning status of a pre-trained model is unknown. However, in safety-critical downstream applications, practitioners often require absolute assurance of model integrity, meaning that the possibility of an unknown backdoor is unacceptable, and the **worst-case scenario** (i.e., the model is backdoored) must be assumed. For instance, a military drone relying on a pre-trained model for target recognition could suffer catastrophic consequences if a hidden backdoor causes it to misidentify an ally as an enemy. In such settings, even a small risk of manipulation is intolerable. In fact, there is already a paper demonstrating that **OpenAI CLIP has been backdoored** [1], underscoring the urgency and real-world relevance of our research.\n\nImportantly, RVPT provides strong backdoor robustness with minimal impact on clean data performance, making it a practical and necessary solution for deploying third-party foundation models in trust-critical environments.\n\n\n## Q2: Performance of RVPT with unpoisoned model\nThanks for the concern. RVPT is primarily designed to defend against backdoor attacks by identifying and suppressing malicious trigger features. When applied to an unpoisoned model, RVPT is not expected to offer direct benefits for backdoor robustness, simply because there is no backdoor behavior to mitigate.\n\nHowever, it is important to note that **RVPT does not degrade model performance on clean inputs**. In fact, it can slightly improve it. As shown in Table 2, RVPT improves or maintains clean accuracy compared to both Zero-shot CLIP. This suggests that RVPT can act as a benign regularizer, even in the absence of backdoor triggers:\n\n**Table 2: Clean accuracy when the base CLIP is unpoisoned. Both RVPT and CleanCLIP fine-tune the unpoisoned CLIP.**\n| Method | ImageNet | Caltech101 |OxfordPets |\n| -------- | -------- | -------- |--|\n| Zero-shot CLIP     | 62.01   | 91.46     |86.42 |\n| CleanCLIP     | 60.58     | 91.35     |83.79|\n| RVPT     | **62.95**    | **94.51**     |**89.67**|\n\n\n## Q3: Generalization capability of RVPT\nThanks for raising this important point. You are absolutely correct that CLIP’s strong zero-shot performance likely arises from its ability to leverage a broad spectrum of features, including some that may be non-predictive or off-distribution. As shown in Table 3, this broad feature reliance explains the slight degradation in generalization ability observed after applying RVPT. \n\nHowever, we argue that this modest trade-off in generalization is acceptable given the significant gains in backdoor robustness. Moreover, this phenomenon is not unique to RVPT: many prompt learning and fine-tuning methods for CLIP also exhibit reduced zero-shot transferability due to overfitting to limited supervision signals. RVPT’s behavior is consistent with this trend. By explicitly suppressing predictive-but-brittle features, RVPT may slightly reduce generalization, but it does so to prioritize safety and robustness. \n\n**Table 3: Average ASR and zero-shot clean accuracy drop of defense methods. CLIP is trained on ImageNet and tested on Caltech101 and OxfordPets for the cross-dataset scenario and ImageNet-R, S, V2 for the cross-domain scenario.**\n\n| Method |  cross-dataset ASR ($\\downarrow$CA drop) |cross-domain ASR ($\\downarrow$CA drop) |\n| -------- | -------- | -------- |\n| CleanCLIP    | 14.98 ($\\downarrow$ 1.79)     | 40.09 ($\\downarrow$ 5.15)     |\n| RVPT    | **1.77** ($\\downarrow$ 1.76)     | **1.42** ($\\downarrow$ 5.05)     |\n\n\nTo sum up, while RVPT does slightly trade off generalization for robustness (as is common in prompt learning), the degradation remains modest and acceptable. Importantly, RVPT still preserves competitive zero-shot performance across datasets and domains.\n\n## References\n[1] Detecting backdoor samples in contrastive language image pretraining, ICLR 2025."}, {"review_id": "afrFGQKjXW", "reviewer": "Reviewer_w7GP", "summary": "This paper addresses the significant vulnerability of multimodal contrastive learning models like CLIP to backdoor attacks. It propose that CLIP’s susceptibility arises from  its tendency to encode features beyond in-dataset predictive patterns. To mitigate this, they introduce a new defense mechanism called Repulsive Visual Prompt Tuning (RVPT). The results show that RVPT is highly effective, drastically reducing the ASR while preserving clean accuracy and being significantly more efficient than existing methods.", "questions": "1. Would RVPT still maintain strong defense capabilities if the number of clean downstream samples were significantly reduced, or in extreme cases, absent?\n\n2. Figure 3(b) indicates a trade-off controlled by the balancing factor $\\alpha$, showing both CA and ASR decrease as $\\alpha$ increases. In real-world deployment scenarios, a defender might only have access to a new small clean dataset and a backdoored model, without means to accurately measure the ASR. Under these practical constraints, how would you recommend defenders set the value of $\\alpha$?\n\n3. What are the computational costs introduced by the feature-repelling mechanism in comparison to APT baseline?\n\n4. Could we inject learnable prompts on both the text and image branch, similar to prior work [1, 2].\n\n[1] MaPLe: Multi-modal Prompt Learning, CVPR 2023.\n\n[2] TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models, CVPR 2025.", "limitations": "Detailed computational overheads (time, resources, latency) are missing, which could hinder practical adoption in resource-constrained scenarios.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Paper Strengths:\n\n1. The authors evaluate their approach on multiple benchmarks and conduct extensive ablation studies.\n\n2. The experimental setup is described in a detailed and thorough manner.\n\nWeaknesses:\n\n1. The evaluation does not consider potential adaptive attacks explicitly tailored to RVPT’s defense strategy.\n\n2. The method introduces several new hyperparameters, most notably the balancing factor, the prompt depth, and the prompt length. The ablation studies show that the model's performance is sensitive to these choices.", "was_revised": true, "final_justification": "Thank you to the authors for their detailed rebuttal. After reading rebuttal, I tend to keep my rating.", "rebuttal": "Thank you for your valuable comments!\n\n## W1: Potential adaptive attacks\nThanks for the question. We respectfully hold that crafting an **effective adaptive attack against RVPT is highly challenging under most realistic threat models**.\n\nThe core reason lies in the **temporal and informational gap** between the poisoning stage and the application of RVPT. Specifically, CLIP is assumed to be poisoned during pre-training, whereas RVPT is applied post hoc, using private downstream data for adaptation. For an adaptive attack to succeed, the adversary would need foreknowledge of the user's downstream data during the upstream pre-training phase, which is a highly unrealistic assumption. Even if the downstream data becomes accessible and an adaptive attack is crafted successfully, its effectiveness would likely not transfer to other downstream tasks, where CLIP is fine-tuned with different data under RVPT, severely limiting the threat scope.\n\nFurthermore, another reason for being impractical to design an adaptive attack is the **mechanism of RVPT**: it filters non-predictive features and only retains predictive features in a clean downstream dataset. \nThis makes it non-trivial to design a trigger that can be encoded by backdoored CLIP as predictive features in the clean downstream dataset. In fact, we experimented with using a universal adversarial perturbation [1] as a trigger to poison CLIP, but this strategy failed after applying RVPT, reinforcing the difficulty of designing such attacks.\n\nWe believe that constructing highly specialized adaptive attacks against RVPT could be an interesting direction for future research, but under current assumptions, such attacks appear highly challenging and limited in scope.\n\n\n## W2: Performance sensitive to hyperparameters\n\nThanks for your concern regarding hyperparameter sensitivity. However, we believe that while RVPT does introduce several hyperparameters, we find that they are **straightforward to configure** due to their robustness across a wide range of values.\n\n- For prompt depth, our ablation study shows that both clean accuracy and defense effectiveness improve and then saturate with increasing depth. This means there is **no trade-off**, and practitioners can simply select the maximum feasible depth without risk of performance degradation.\n- For the balancing factor ($\\alpha$) and prompt length, sensitivity primarily arises at very low values, where the defense has not yet fully activated. As our results demonstrate, once these parameters exceed modest thresholds (e.g., $\\alpha \\geq 2.0$, prompt length $\\geq 25$), RVPT **consistently achieves near-zero ASR while maintaining high CA**.\n\nIn this context, the observed \"sensitivity\" is not a practical limitation but rather a clear and interpretable indicator of an effective operating range. Defenders can therefore select hyperparameters confidently from this broad, stable region to achieve strong and reliable security.\n\n\n## Q1: Performance with reduced number of samples\nThanks for raising this important question regarding the data requirements of our method.\n\n**Performance with few samples.** Our experiments, detailed in Figure 3(a), demonstrate that RVPT remains highly effective even in extremely low-data regimes. For traditional vision-only backdoor attacks such as BadNet and Blended, just one clean sample per class (1-shot) suffices to achieve robust defense. For the more challenging state-of-the-art multimodal attack: BadCLIP, RVPT requires only eight samples per class (8-shot) to reduce the ASR below 5%. This represents significant data efficiency compared to previous post-training defenses: for example, RVPT defends CLIP using only 6% CleanCLIP's data on ImageNet and 0.2% CleanCLIP's data on OxfordPets.\n\n**Performance in zero-Shot case.** We believe that RVPT is not directly applicable when no clean data is available by design, since The core principle of our method is to use a small set of trusted downstream samples to identify task-relevant visual features. By identifying predictive features, RVPT can effectively detect and suppress extraneous features, including potential backdoor triggers. Therefore, without any such reference samples, RVPT cannot distinguish predictive from non-predictive features, and thus cannot perform the defense. Extending multimodal backdoor post-training defenses to a fully zero-shot setting remains an important and valuable direction for future research.\n\n## Q2: The choice of $\\alpha$\nThanks for the insightful feedback! We have observed that the ASR is highly sensitive to lower values of the balancing factor $\\alpha$, but rapidly approaches zero as $\\alpha$ increases. In contrast, the CA is much less sensitive and decreases gradually and gracefully. This behavior allows defenders to adopt a conservative strategy by selecting a larger $\\alpha$ to maximize robustness, with minimal impact on clean performance. Alternatively, defenders can tune $\\alpha$ based on an acceptable CA threshold, which is measurable during deployment. Specifically, we recommend the following two strategies for setting $\\alpha$:\n\n\n**Strategy 1: Use a recommended conservative default $\\alpha$.** Based on empirical results across various attack types, we suggest a default value of $\\alpha = 2.0$. As shown in our main experiments, this value consistently reduces ASR to near zero for all evaluated attacks while only incurring a minor CA decrease of approximately 1–2% relative to $\\alpha=0$.\n\n**Strategy 2: Tune using a clean validation set.** If a defender wants to tune $\\alpha$ for their specific task, they can follow this simple procedure:\n1. Start with a small $\\alpha$ (e.g., 0.5).\n2. Incrementally increase $\\alpha$ (e.g., to 1.0, 1.5, 2.0, ...).\n3. At each step, measure the CA on the clean validation set.\n4. Select the largest value of α for which the drop in CA is still acceptable.\n\nThis approach enables defenders to push defense strength as high as possible within their performance budget. Since ASR decreases more rapidly than CA, the chosen $\\alpha$ is very likely to have neutralized any backdoor effectively.\n\nIn summary, while defenders cannot directly measure ASR, they can reliably measure CA. The predictable and gentle decline in CA allows them to choose a value for $\\alpha$ that strongly prioritizes security at a minimal and quantifiable cost to performance.\n\n## Q3: The added computational cost of FR loss\n\nThanks for your questions! We would like to clarify that the feature-repelling mechanism does not introduce a new CLIP model. The output of the original backdoored CLIP can still be obtained by using features without the visual prompts.\n\nIn Table 1, we compare the computational costs of RVPT, its baseline VPT (which is RVPT without the feature-repelling mechanism), and a state-of-the-art post-training defense, CleanCLIP.\n\n**Table 1: Computation cost of RVPT, VPT (RVPT without feature-repelling mechanism), and CleanCLIP.**\n| Method | Training time (in hour) | Used GPU memory | Tunable Parameters |\n| -------- | -------- | -------- |-------- |\n|**VPT**|   0.72   |   3382 MB   |  0.34 M   |\n|**RVPT**|  0.95 (+31.9%)    | 3382 MB  (+0%)   |  0.34 M  (+0%) |\n|**CleanCLIP**|  3.88  (+454.2%)  |  17640 MB (+421.6%) |   126 M (+36958.8%) |\n\nAs shown in Table 1, RVPT introduces only a 31.9% increase in training time compared to the standard VPT baseline. Since VPT itself is a parameter-efficient fine-tuning method, this marginal overhead is negligible—especially when compared to the substantially higher resource demands of CleanCLIP. This minimal computational cost highlights the practicality and efficiency of RVPT for real-world deployment.\n\n## Q4: Injection of learnable prompts into the multimodal branch\nThanks for the insightful suggestion. Following your advice, we conducted an experiment incorporating the MaPLe [2] architecture, which uses learnable prompts in both the text and image branches. The results are summarized in Table 2. In this setup, the visual prompts were optimized using both cross-entropy loss and our feature-repelling loss, while the textual prompts were optimized using only the cross-entropy loss.\n\n**Table 2: Ablation study on the ASR (CA) performance of the feature-repelling mechanism in the architecture of MaPLe.**\n| Method | BadNet | Blended | BadCLIP |\n| -------- | -------- | -------- |-------- |\n|MaPLe|   61.72 (63.77)   |  60.19 (63.92)    |  98.85 (62.67)   |\n|MaPLe + feature-repelling mechanism|   **1.02** (59.87)   | **0.01** (59.42)     |  **1.24** (58.60)   |\n\nAs shown in Table 2, optimizing prompts on both branches without the feature-repelling mechanism does not significantly improve backdoor robustness. However, when the visual prompt is further optimized with the feature-repelling loss, the robustness improves substantially. This finding reinforces our core claim: the effectiveness of RVPT primarily arises from operations within the visual encoder. Our feature-repelling mechanism effectively neutralizes trigger patterns in the visual feature space, making the defense largely independent of the textual branch.\n\n## References\n[1] Universal adversarial perturbations, CVPR 2017.\n[2] MaPLe: Multimodal Prompt Learning, CVPR 2023."}, {"review_id": "Djsc4EA9D1", "reviewer": "Reviewer_aLX9", "summary": "CLIP aligns visual and textual representations in the pre-training stage and serves as a powerful base for multimodal tasks. CLIP has been shown vulnerable to backdoored attacks, which inject a small proportion of poisoning data to make the model behave normally without the trigger, but reliably predict the target class with the trigger. The paper tries to “clean” a backdoored CLIP by training with a downstream clean dataset. The proposed RVPT optimizes some newly added layer-wise features to minimize the cosine similarity between the forwarded features and the original clean CLIP features. This proposed feature-repelling loss works with the cross-entropy loss to increase robustness against input perturbations while preserving utility.", "questions": "See weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed Perturbation Resistivity is an intuitive and straightforward metric to constrain the image features to be robust against perturbations. For embeddings, the direction tends to indicate rich information compared to the magnitude. \n2. The threat model is clearly stated. The attacker poisons the clean CLIP, and the defender uses a small clean set to reverse the backdoor effect by additional training. It requires less clean data as it starts from a well-functioning backdoored model, and the only task here is to remove the backdoor.\n3. The experiments are comprehensive and show strong effectiveness of RVPT, which achieves near 0% attack success rates using various attack methods, significantly surpassing existing baselines. The authors perform extensive study on the generalization and hyperparameters of the defense.\n\nWeaknesses\n1. It is unclear to me why VPT is used. The optimization objective makes sense, but why do the authors choose to optimize additional features as VPT. The introduction mentions that fine-tuning the whole model is “resource-intensive or even unrealistic”, but I am curious if that will lead to better performance, and what is the trade-off.\n2. I don’t understand the ablation study on emerging target class, mostly because of the term “tuning dataset”. If it stands for the attacker’s poisoning dataset, how can an attacker poison the model without showing the in-class samples? If it stands for the defender’s downstream dataset, only one term should be used to describe it with proper explanations.\n3. The discussion about the connection to adversarial attacks is lacking. How can the proposed loss inspire defenders against test-time attacks for images where reducing model’s sensitivity to input perturbations is also the key?", "was_revised": true, "final_justification": "Thanks for the rebuttal, which addresses most of my concerns. Thus, I maintain my positive scores. I encourage the authors to include the insightful discussions on \"Why is VPT used\" in the paper.", "rebuttal": "Thank you for your valuable comments!\n\n## W1: Why is VPT used?\nThanks for this insightful question! Before addressing why we use VPT, we would like to clarify the trade-offs involved when fine-tuning the entire CLIP model and explain why full fine-tuning is not preferred in our context.\n\n### The trade-off when fine-tuning the whole model\n\nFull fine-tuning of CLIP can be performed in two main ways:\n\n- Using the InfoNCE loss on an image-text dataset.\n- Using the cross-entropy (CE) loss on an image-label classification dataset.\n\n**Fine-tuning whole CLIP with Info-NCE loss.** Since our primary downstream task is classification, we do not have downstream image-text pairs for optimization. We could use surrogate datasets such as CC3M, but as Table 1 shows, defense performance drops significantly. This degradation occurs because RVPT relies on identifying predictive features specific to the downstream dataset and pruning non-predictive ones. **The mismatch between predictive features in CC3M and the actual downstream task leads to poor backdoor robustness.**\n\n**Fine-tuning whole CLIP with cross-entropy loss.** This requires adding an additional linear layer on top of the visual encoder for closed-set classification. However, this approach has several drawbacks:\n- It **removes CLIP's open-set classification capability**, severely limiting generalization to other datasets.\n- Given the limited labeled data, the large number of tunable parameters causes overfitting, resulting **catastrophic forgetting**.\n- Although it can eliminate backdoors (see Table 1), it demands **much more data** to avoid overfitting and sacrifices the model's generalization ability, making it an unfavorable option.\n\n\n**Table 1: ASR (CA) Performance of different fine-tuning methods with or without FR loss.**\n|Fine-tuning method|without FR Loss| with FR Loss|Open-set classification capability|\n|--|--|--|--|\n|Fine-tuning on full ImageNet using CE Loss| 10.09 (60.71)|0 (59.22)|NO|\n|Fine-tuning on CC3M using info-NCE Loss|92.50 (54.50)|18.23 (54.66)|YES|\n|Visual prompt tuning on few-shot ImageNet **(preferred)**|98.14 (64.32)|2.76 (61.81)|YES|\n\n\nTo summarize, if we fine-tune the whole CLIP with Info-NCE loss, it will all become worse with heightened computational cost and less backdoor robustness (no trade-offs). If we use CE loss, we can only gain a little backdoor robustness at the sacrifice of much more data demand and deprivation of model generalization ability.\n\n### The reasons why we use VPT\n\nAs outlined above, full fine-tuning of CLIP presents significant drawbacks, either in terms of degraded robustness, excessive data requirements, or loss of open-set capabilities. To address these challenges, we adopt parameter-efficient fine-tuning methods that:\n- Leverage limited labeled downstream data effectively, aligning with RVPT's core mechanism of identifying task-relevant visual features.\n- Preserve CLIP's open-set classification ability, which is critical for generalization across diverse tasks.\n\nSpecifically, since our goal is to enhance the perturbation resistivity of the CLIP's **visual features**, we inject learnable parameters into the visual modality, leading to our choice of VPT. This approach significantly improves downstream backdoor robustness using much less data while retaining CLIP's open-set capability, as demonstrated in Table 1.\n\n\n## W2: The explanation of the ablation study on the emerging target class\nThanks for pointing out this ambiguity. We apologize for the confusion caused by the term \"tuning dataset\" and will clarify our terminology in the revised manuscript. \nThe reviewer is correct: the \"tuning dataset\" refers to the **defender's downstream dataset**.\nWe will revise Section 4.3 to explicitly use the term \"**defender's clean tuning set**\" and clarify this experimental process to prevent any future misinterpretation.\n\n## W3: RVPT's connection to the defense against adversarial attacks\nThanks for this excellent and thought-provoking question regarding the broader implications of our work. We discuss this connection in more detail in Appendix E, but we are happy to elaborate further here.\n\nRVPT is specifically designed to reduce a model's sensitivity to **non-predictive** input perturbations, which are characteristic of backdoor triggers. In contrast, adversarial perturbations are typically **predictive and brittle**, as they are carefully optimized to manipulate model outputs while remaining imperceptible [1].\n\nBecause RVPT is explicitly designed to preserve predictive features and repel non-predictive ones, its defense mechanism is **not directly applicable to adversarial attacks**, which often exploit predictive but fragile features. Therefore, RVPT is not guaranteed to defend against adversarial perturbations.\n\n\nHowever, as your suggestion insightfully points out, there is a conceptual overlap that can be built upon. Specifically, the challenge in extending RVPT to adversarial defense lies in disentangling brittle adversarial predictive features from robust predictive features. Simply eliminating all predictive components would lead to unacceptable degradation on benign inputs. To address this, we envision a future extension of RVPT that incorporates an anchor-based mechanism: For a given adversarial example, we retrieve or synthesize a semantically similar clean \"anchor\" image, which shares robust, task-relevant features. We then define a repulsive loss between the feature representations of the adversarial and anchor images. This loss is designed to repel the components that are unique to the adversarial image, while preserving shared features that reflect the true semantic content. Such a mechanism would allow us to selectively suppress brittle predictive features introduced by the adversarial noise, without harming generalization or clean accuracy. \n\nWe see this as an exciting and promising direction for future research in defending against test-time adversarial attacks using feature-level control strategies.\n\n\n## References\n[1] Adversarial examples are not bugs, they are features, NeurIPS 2019."}]

Author rebuttal:

Meta-review: This paper proposes a new defense method against backdoors on CLIP. All four reviewers have actively engaged in the discussion and appreciate the technical novelty and extensive experiments (in both the original paper and multi-round responses). Although Reviewer g6BJ is still concerned about the realism of the threat model (assuming that the defender only has access to a few clean samples), they clearly stated in the final justification and the confidential comment to AC that this concern is just a minor one by saying "Overall, I am not strongly opposed to this paper, so I would not recommend a clear rejection." and "Ultimately, although it is a minor point in the final assessment...". The AC also checked the authors' response to Reviewer g6BJ and noticed that the authors have already provided new results to show the proposed method is still better than using a substitute dataset of clean samples (see Table 1: ASR (CA) Performance against BadCLIP of different fine-tuning methods with or without FR loss). 

Overall, the AC thinks that the strengths of this paper slightly outweigh its limitations and therefore recommends acceptance. However, this paper still requires a major revision before publication by incorporating all the new results and clarifications. The final decision would be given based on the quality of the revised paper.
Decision: Accept (poster)