Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Authors: Yuwei Niu
Shuo He
Qi Wei
Zongyu Wu
Feng Liu
Lei Feng
Venue: ICML 2025 poster (2025)
Keywords: Multimodal contrastive learning
test-time backdoor detection
Primary area: social_aspects->safety

Abstract: While multimodal contrastive learning methods (e.g., CLIP) can achieve impressive zero-shot classification performance, recent research has revealed that these methods are vulnerable to backdoor attacks. To defend against backdoor attacks on CLIP, existing defense methods focus on either the pre-training stage or the fine-tuning stage, which would unfortunately cause high computational costs due to numerous parameter updates and are not applicable in black-box settings. In this paper, we provide the first attempt at a computationally efficient backdoor detection method to defend against backdoored CLIP in the inference stage. We empirically find that the visual representations of backdoored images are insensitive to benign and malignant changes in class description texts. Motivated by this observation, we propose BDetCLIP, a novel test-time backdoor detection method based on contrastive prompting. Specifically, we first prompt a language model (e.g., GPT-4) to produce class-related description texts (benign) and class-perturbed random texts (malignant) by specially designed instructions. Then, the distribution difference in cosine similarity between images and the two types of class description texts can be used as the criterion to detect backdoor samples. Extensive experiments validate that our proposed BDetCLIP is superior to state-of-the-art backdoor detection methods, in terms of both effectiveness and efficiency.

Test- Time Multimodal Backdoor Detection by Contrastive Prompting
Yuwei Niu Shuo He * Qi Wei?
Zongyu Wu Feng Liu Lei Feng
Abstract
served as foundation models in various vision-language fields due to their strong ability to effectively align represen- tations from different modalities, such as open-vocabulary object detection (Wu et al., 2023), text-to-image genera- tion (Ramesh et al., 2022), and video understanding (Xu et al., 2021). However, recent research has revealed that small proportion of backdoor samples poisoned into the pre-training data cause backdoored CLIP after the mul- timodal contrastive pre-training procedure (Carlini & Terzis, 2021; Carlini et al., 2023; Bansal et al., 2023). In the in- ference stage, a backdoored CLIP would produce tampered image representations for images with a trigger, close to the text representation of the target attack class zero-shot classification. This exposes a serious threat when deploying CLIP in real-world applications.
While multimodal contrastive learning methods (e.g., CLIP) can achieve impressive zero-shot clas- sification performance, recent research has re- vealed that these methods are vulnerable to back- door attacks. To defend against backdoor attacks on CLIP, existing defense methods focus on ei- ther the pre-training stage or the fine-tuning stage, which would unfortunately cause high computa- tional costs due numerous parameter updates and are not applicable in black-box settings. In this paper, we provide the first attempt at compu- tationally efficient backdoor detection method to defend against backdoored CLIP in the inference stage. We empirically find that the visual repre- sentations of backdoored images are insensitive to benign and malignant changes in class description texts. Motivated by this observation, we propose BDetCLIP, a novel test-time backdoor detection method based on contrastive prompting. Specif- ically, we first prompt a language model (e.g.. GPT-4) to produce class-related description texts (benign) and flass-perturbed random texts (ma- lignant) by specially designed instructions. Then, the distribution difference in cosine similarity be- tween images and the two types of class descrip- tion texts can be used as the criterion to detect backdoor samples. Extensive experiments vali- date that our proposed BDetCLIP is superior to state-of-the-art backdoor detection methods, in terms of both effectiveness and efficiency.
To overcome this issue, effective defense methods have been proposed recently, which can be divided into three kinds of defense paradigms, as shown in Figure 1: including (a) robust anti-backdoor contrastive learning in the pre-training stage (Yang et al., 2023b), (b) counteracting the backdoor in a pre-trained CLIP in the fine-tuning stage (Bansal et al., 2023), (c) leveraging trigger inversion techniques to decide if pre- pre-ttrinee CLIP is backdoored (Sur et al., 2023; Feng et al., 2023a). Overall, these defense methods have a high computational cost due to the need for additional learning or optimization procedures. In contrast, we advocate test-time backdoor sample detection (Figure 1(d)), which a more computationally efficient defense against backdoored CLIP, as there are no parameter updates in the inference stage. Intuitively, it could be feasible to directly adapt existing uni- modal test-time detection methods (Gao et al., 2019; Guo et al., 2023; Liu et al., 2023) to detect backdoored images in CLIP, since they can differentiate backdoored and clean im- ages generally based on the output consistency in the visual representation space by employing specific image modifi- cations, e.g, corrupting (Liu et al., 2023), amplifying (Guo et al., 2023), and blending (Gao al., 2019). However, the performance of these unimodal detection methods is subop- timal, because of lacking the utilization of the text modality in CLIP to assist backdoor sample detection. Hence we can expect that better performance could be further achieved we leverage both image and text modalities simultaneously.
1. Introduction
Multimodal contrastive learning methods (e.g., CLIP (Rad- ford et al., 2021)) have shown impressive zero-shot clas- sification performance in various downstream tasks and
'Equal contribution 'Chongging University "Nayyang Tech- nological University *Penn State University "University of Mel- bourne "Southeast University. Correspondence to: Lei Feng <fen- glei @ sei.seu..duucn..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we provide the first attempt at computation- ally efficient backdoor detection method to defend against

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
backdoored CLIP in the inference stage. We empirically find that the visual representations of backdoored images are insensitive to both benign and malignant changes of class description texts. Motivated by this observation, we propose BDeCCLIP, a novel test-time multimodal backdoor detection method based on contrastive prompting. Specifically, we first prompt the GPT-4 (Achiam et al., 2023) to generate class-related (or perttrrerrrrdd random) description texts by specially designed instructions and take them as benign (malignant) class prompts. Then, we calculate the distribu- tion difference in cosine similarity between images and the two types of class prompts, which can be used as a good criterion to detect backdoor samples. We can see that the distribution difference of backdoored images between the benign and malignant changes of class prompts smaller than that of clean images. The potential reason for the insensibility of backdoored images is that their visual repre- sentations have less semantic information aligned with class description texts. In this way, we can detect backdoored images in the inference stage of CLIP effectively and effi- ciently. Extensive experiments validate that our proposed BDetCLIP is superior to state-of-the of-the- backdoor detection methods, in terms of both effectiveness and efficiency.
Dirain = { 4.),;= where 4, is the caption of the image my and N = 400M. During the training stage, given batch of No image-text pairs (æ;, (,) C Dreain. the cosine similar- ity for matched (unmatched) pairs is denoted by Φ( t.) = (V()), T(t:)) (,(m, (2) = cos (v(;)) ((ty)) It is noteworthy that the image and text embeddings are normal- ized using the lg norm to have unit norm. Based on these notations, the CLIP loss can be formalized by the following:
1 CCLI = 2N,,
No
exp(o(x t./T)
log
No j=1 exp (op(o((; t,)/T)
No
exp(o( L1/T) 1)
log
No exp(o(ax, ()/T)
where 7 is a trainable temperature parameter.
Zero-shot classification in CLIP. To leverage CLIP on the downstream classification task where the input image x E Dreat and class name y: {1,2, ,c), simple yet effective way is using a class template function T(j) which generates a class-specific text such as "a photo of [CLS]" where [CLS] can be replaced by the j-th class name on the dataset. In the inference stage, one can directly calculate the posterior probability of the image x for the i-th class as the following:
Our main contributions can be summarized as follows:
A new backdoor detection paradigm for CLIP. We pioneer test-time backdoor detection for CLIP, which is more resource-efficient than existing defense paradigms.
ply = i|z) exp(ò(x, T())/T)
(I)
2j=1 exp(p T())/T)
A novel backdoor detection method. We propose novel test-time backdoor detection method by contrastive prompting, which detects backdoor samples based on the distribution difference between images regarding the benign and malignant changes of class prompts.
In addition, recent research (Yang et al., 2023c; Pratt et al.,
2023; Maniparambil et al., 2023; Yu et al., 2023; Zhang et al., 2024; Saha et al., 2024; Feng et al., 2023b; Liu et al., 2024) delves into engineering fine-graines class-specific attributes or prompting large language models (e.g., GPT-4 (Achiam et al., 2023)) to generate attribute-related texts.
Strong experimental results. Our proposed method achieves superior experimental results on various types of backdoored CLIP compared with state-of-the-art de- tection methods.
2.2. Backdoor Attacks and Defenses
The backdoor attack is a serious security threat to machine learning systems (Li et al., 2022: Carlini & Terzis, 2021; Xu et al., 2022; Chen al., 2021). The whole process of backdoor attack can be expounded as follows. In the data collection stage of a machine learning system, malicious adversary could manufacture a part of backdoor samples with the imperceptible trigger poisoned into the training dataset. After the model training stage, the hidden trigger could be implanted into the victim model without much impact on the performance of the victim model. During the inference stage, the adversary could manipulate the victim model to produce a specific output by adding the trigger to the clean input. Early research on backdoor attacks focuses on designing a variety of triggers that satisfy the practical
2. Background & Preliminaries
2.1. Multimodal Contrastive Learning
Multimodal contrastive learning (Radford et al., 2021; Jia et al., 2021) has emerged as a powerful approach for learn- ing shared representations from multiple modalities of data such as text and images. Specifically, we focus on Contrastive Language Image Pretraining (CLIP) (Radford et al., 2021) in this paper. Concretely, CLIP consists of visual encoder denoted by V2.) (e.g., ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020)) and a textual en- coder denoted by T-) (e.g.. Transformer (Vaswani et al., 2017)). The training examples used in CLIP are mas- sive image-text pairs collected on the Internet denoted by
scenarios mainly on image and text classification tasks in- cluding invisible stealthy triggers (Chen al., 2017: Turner
2

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Pre- training backdoor data
Visual Encoder
Textual Encoder
Textual Encoder
Textual Encoder
Encoder
Encoder
Multi-modal contrastive learning
Fine- -uunin
Trigger Inversion
Inference
Visual Encoder
Textual Encoder
Visual Encoder
Textual Encoder
Uni-modal
multi-modol
Detect backdoored?
Backdoor
Backdoor
sample detection
somple
detection
(
(b)
(c)
(d)
Figure 1. Current backdoor defense paradigms in CLIP. (a) Robust anti-backdoor contrastive learning (Yang et al., 2023b); (b) Fine-tuning a backdoored CLIP (Bansal et al., 2023); (c) Detecting a CLIP if backdoored (Sur et al., 2023; Feng et al., 2023a); (d) Our test-time backdoor sample detection. Our multimodal detection method is more effective and efficient than existing unimodal detection methods.
et al., 2019; ;; Li et al., 2021a; Doan et al., 2021; Nguyen & Tran, 2021; Gao et al., 2023; Souri et al., 2022) and physical triggers (Chen et al., 2017; Wenger et al., 2021). To defend against these attacks, many backdoor defense methods are proposed which can be divided into four categories, mainly including data cleaning the pre-processing stage (Tran al., 2018), robust anti-backdoor training (Chen et al., 2022; Zhang et al., 2022), mitigation, detection, and in- version in the post-training stage (Min et al., 2023), and test-time detection in the inference stage (Shi et al., 2023).
Defenses for the backdoored CLIP. Effective defense methods have been proposed recently, which can be di- vided into three kinds of defense paradigms including anti- backdoor learning (Yang et al., 2023b) in the pre-training stage, fine-tuning the backdoored CLIP (Bansal et al., 2023; Kuang et al., 2024; Xun et al., 2024), and using trigger inversion techniques (Sur al., 2023; Feng al., 2023a) to detect the visual encoder of CLIP if is infected. How- ever, due to the need for additional learning optimization processes. these defense methods are computationally ex- pensive. Furthermore, in many real-world scenarios, we only have access to third-party models or APIs, making impossible to apply existing backdoor defense methods for pre-training and fine-tuning.
Backdoor attacks for CLIP. This paper especially fo- cuses on investigating backdoor security in multimodal contrastive learning. Recent research (Carlini & Terzis, 2021; Carlini al., 2023; Bansal et al., 2023; Jia et al., 2022; Bai et al., 2023; Liang et al., 2023) has revealed the serious backdoor vulnerability of CLIP. Specifically, malicious adversary can manufacture a proportion of backdoor image-text pairs DBD = , ) NaD where :; = (1-M) O x; + +MO C A is a backdoor image with the trigger pattern (Gu al., 2017: Chen al., 2017) and the mask M, and Ty:: is the caption of the target attack class Then, the original pre-training dataset D Draain could be poisoned as Droison = (DAD U Delean). The backdoor attack for CLIP can be formalized by:
3. The Proposed Approach
In this section, we provide the first attempt at test-time backdoor detection for CLIP and propose BDetCLIP that effectively detects test-time backdoored images based on the text modality.
Vt-
3.1. A Defense Paradigm: Test-Time Backdoor Sample Detection
Compared with existing defense methods used in the pre- training or fine-tuning stage, detecting (and then refusing) backdoor images in the inference stage directly is a more lightweight and straightforward solution to defend back- doored CLIP. To this end, one may directly adapt existing unimodal detection methods (Gao et al., 2019; Zeng et al., 2021; Udeshi et al., 2022; Guo et al., 2023; Liu et al., 2023; Pal et al., 2024; Hou et al., 2024) solely based on the vi- sual encoder (i.e., V (-)) of CLIP with proper modifications, However, this strategy is suboptimal because of the lack of the utilization the textual encoder (-) in CLIP to assist detection (as shown in Figure 2(a)). In contrast, we propose to integrate the visual and textual encoders in CLIP for test- time backdoor sample detection (TT-BSD). The objective
{0v-, By- ===ra arg min (CLie(Pelea))) + LCLIP (DBD), (0v.0r)
where py- is the parameter of the infected visual encoder (-).) and 07. is the parameter of the infected textual coder (-). is noteworthy that the zero-shot performance of the backdoored CLIP is expected to be unaffected in Eq. (1), while for the image æx with a trigger, the posterior prob- ability of the image for the ye-th target class could be large with high probability:
p(u: = vel | = exp(ole;,
(2)
ΣS exp(o(x; T())/T)
3

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Clean image
Perturbed clean image
Benign class class-specific prompt
Benign similarity
Backdoor image
Perturbed backdoor image
Malignant class class-specific prompt
Malignant similarity
Logit-1 Logit-2 Logit-3
Label-1 Label-2 3
sensitivel
insensitivel
Logit-d
Label-c
Visual representation space
Visual representation distribution
Contrastive
Textual representation space
Output Consistency
space
difference
Figure (a) Illustration unimodal backdnor detection that only focuses on visual representation space; (h) Illustration BDeICLIIP that leverages both image and text modalities in CLIP.
of TT-BSD for CLIP is design good detector I::
triggers are unable to properly capture the semantic changes of class description texts. This motivates us to consider whether the alignment between the visual representations of backdoored images and the class description texts would be significantly changed when there exist significant changes in the class description texts. Interestingly, we empirically find that the alignment of backdoor samples would not signif- icantly changed even given significant changes in the text description texts. This observation can help us distinguish backdoor samples from clean samples because the align- ment of clean samples would be significantly influenced by the changes in the text description texts.
r = arg min
Σ #c Delenn
(T ;","" = 1)
+ Σ DDDD
[(1("", V", T*) =)),
(3)
where II(-) is an indicator function, and rxx) returns or O indicates the detector regards æ as a backdoored or clean.
Defender's goal. Defenders aim to design good detector in terms of effectiveness and efficiency. Effectiveness is directly related to the performance of I, which can be evaluated by AUROC. Efficiency indicates the time used for detection, which is expected to be short in real-world applications.
Contrastive prompting. Based on the above motivation, we propose BDeICLIP, a novel test-time backdoor detec- tion method based on contrastive prompting. Specifically, we prompt GPT-4 (Achiam et al., 2023) to generate two types of contrastive class description texts. Firstly, based on the powerful in-context learning capabilities of GPT-4, we use specially designed instructions with a demonstration as shown in Appendix A. particular, the demonstration for the class "goldfish" is associated with various attributes of objects. e.g., shape, color, structure, and behavior. In this way, GPT-4 is expected output multiple fine-grained attribute-based sentences for the assigned j-th class, denoted by ST] (k E [m]] where is the number of sentences. On the other hand, we also prompt GPT-4 by the instruction "Please randomly generate sentence of no more than 10 words unrelated to { (Class Name)". to generate one ran- dom sentence unrelated the assigned j-th class. We con- catenated the class template prompt with the obtained ran- dom sentences generate the final class-specific malignant prompt, denoted by RT,, such as "A photo of goldfish. The sun cast shadows on the bustling city street.". In Appendix D, We also recorded the money and time costs associated with the prompts generated by GPT-4, and demonstrated the
Defender's capability. In this paper, we consider the black-box setting. Specifically, defenders can only access the encoder interface of CLIP and obtain feature embed- dings of images and texts, completely lacking any prior information about the architecture of CLIP and backdoor at- tacks. This is a realistic and challenging setting in TT-BSD (Guo et al., 2023).
3.2. Our Proposed BDetCLIP
Motivation. It was shown that CLIP has achieved im- pressive zero-shot classification performance by leveraging visual description texts (Yang et al., 2023c; Pratt et al., 2023; Maniparambil et al., 2023; et al., 2023; Saha et al., 2024; Feng et al., 2023b; Liu et al.. 2024) generated by large lan- guage models. For backdoored CLIP (i.e., CLIP corrupted by backdoor attacks), recent research (Bansal et al., 2023) has revealed that implanted visual triggers in CLIP can ex- hibit a strong co-occurrence with the target class. However, such visual triggers in CLIP are usually simple non-semantic pixel patterns, which could not align well with abundant textual concepts. Therefore, backdoored images with visual

---

Test- Test Multimodal Backdoor Detection by Contrastive Prompting
Benign similarity
Malignant similarity
Sample type
Target class
BadNett
Blended
BadNet-LC
Blended-LC
CLean
Ant
Backdoor
Ant
CLean
Banana
Backdoor
Banana
Figure Empirical density distributions of benign and malignant similarities for 1,000 classes on ImageNet- IK. The larger the overlap proportion in the figure, the smaller the difference in contrustive distributions. We have omitted coordinate axes for better view.
feasibility using open-source models (e.g., LLaMA3-8B (Dubey al., 2024) and al., 2023)) as alternatives.
formalized by:
(Jiang
a(x)) = = (o() 1
m ET ((SST))
Contrastive distribution difference. Based the gen- crated two types of texts by GPT-4, we can calculate the benign (malignant) similarity between test images and be- nign (malignant) spas--specific prompts. In particular, we consider this calculation towards all classes in the label space, since we have no prior information about the la- bel of each test image. In this way, can obtain the whole distribution difference for all classes by accumu- lating the contrastive difference between the per-class be- nign and malignant similarity. Formally, for each class Y. the benign and malignant similarity for each test image xt is denoted by (x'), mh 1 Dim (ST))) and <(V"(x') T (RT,)) respectively. It is worth noting that consider the average textual embeddings of all m class- related description texts. Unless stated otherwise, the default value of parameter m in our experiments is 7. Then, the contrastive distribution difference of a test image æ can be
- =VV"(
(4)
This statistic reveals the sensitivity each test image to- wards benign and malignant changes of class-specific prompts. We show the empirical density distributions of be- nign and malignant similarities ImageNet- 1K in Figure 3. In our consideration, test-time backdoored image æ" is in- sensitive to this semantic changes of class-specific prompts, thereby leading to relatively small value of l(x*), There- fore, we propose the following detector of TT-BSD:
{
1,
if 1(x) <, otherwise,
T(x, v", v²,
(5)
0.
where e a threshold (see Section 4.3 about how to empir- iclly determine the value of e). The pseudo-code of BDet- CLIP is shown in Appendix A.
5

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
4. Experiments
their evaluation metric (i.e., ASR) is completely distinct from our evaluation metric (i.e., AUROC), making di- rect comparison between our method and those methods impossible. This can be easily verified by examining the experimental settings in many recent papers focused on (unimodal) backdoor sample detection (Guo et al., 2023; Liu et al., 2023). We would like to emphasize that our BDetCLIP is applicable in the black-box setting (the de- fender only needs to access the output of the victim model instead of controlling the overall model), while other meth- ods (Bansal et al., 2023; Yang et al., 2023b;a; Liang et al., 2024) have to control the whole training procedure which is infeasible in many real-world applications where only third-party models and APIs are accessible. Moreover, our defense method is much more computationally efficient, as it does not need to modify any model parameters, while previous defense methods involve the update of numerous model parameters. Given these distinctions, a direct com- parison with other backdoor defense methods is not feasible. Therefore, to provide a baseline evaluation, we compare our proposed method with three widely-used unimodal test-time backdoor detection methods in conventional classification models: STRIP (Gao al., 2019), SCALE-UP (Guo et al., 2023), and TeCo (Liu al., 2023). Further implementation details can be found in Appendix B. In addition, in order to further prove the effectiveness of our method, we provide a scenario for performance comparison with other defense methods in Appendix C.
4.1. Experimental setup
Datasets. In the experiment, we evaluate BDetCLIP on various downstream classification datasets includ- ing ImageNet-1K K (Russakovsky et al., 2015), Food-101 (Bossard al., 2014) and Caltech-101 (Fei-Fei et al., 2004). In particular, we pioneer backdoor attacks and defenses for CLIP on fine- ane-ggained image classification datasets Fo1d-101 and Caltech- 101, which are more challenging tasks. Besides, we select target backdoored samples from CC3M (Sharma et al., 2018) which is a popular multimodal pre-training dataset. During the inference stage, we set the proportion of backdoored samples in the test data to 0.3. We also provide results under different backdoor ratios (i.e., 0.3, 0.5, and 0.7) in Appendix D.8.
Attacking CLIP. For backdoor attacks on CLIP, we ex- plore two approaches: pre-training CLIP from scratch on the poisoned CC3M dataset or fine-tuning a pre-trained clean CLIP using a subset of poisoned pairs. Unless oth- erwise specified, the models on 400M samples with ResNet-50 (He et al., 2016) as the visual encoder. In terms of attack methods, our approach encompasses both traditional backdoor attack methods and advanced multimodal attack methods. On the traditional front, we utilize BadNet (Gu et al., 2017), Blended (Chen et al., 2017), LabelConsistent (Turner et al., 2019), ISSBA (Li et al., 2021b), and WaNet (Nguyen & Tran, 2021). No- tably, adapt the triggers from BadNet and Blended to execute label-consistent attacks, referred to as BadNet-LC and Blended-LC. On the multimodal attack methods, we implement BadCLIP-1 (Liang et al., 2023), BadCLIP-2 (Bai et al., 2023), Tro (Walmer et al., 2022), and BadEn- coder (Jia et al., 2022). Comprehensive details on the imple- mentation of these methods are provided in Appendix B. For the target attack class, we mainly selected "banana" from Imk, "baklava" from F00,-101, "dalmatian" from Caltech-101. 101. In Appendix D.1, we provide experiments on more target attack classes. Furthermore, we explore seman- tic backdoor triggers and multi-target backdoor attacks in Appendix D.3 and D.4 respectively.
utilize are CLIP trained
Following conventional studies on backdoor sample detection, assess defense effectiveness by using the area under the receiver operating curve (AU- ROC) (Fawcett, 2006). Besides, we adopt the inference time as a metric to evaluate the efficiency of the detection method. Generally, a higher value of AUROC indicates that the detection method is more effective and a shorter inference time indicates that the detection method is more efficient. We also report additional metrics such as Accuracy, Recall, and F1 in Section 4.3 to comprehensively evaluate the effectiveness of BDecCLIP.
Evaluation metries.
4.2. Experimental results
Compared methods. We cannot make fair and direct comparison with other CLIP backdoor defense methods because our paper is the first work on backdoor detection during the inference phase for CLIP. Our method is fun- damentally different from the defense methods during the fine-tuning or pre-training phases, which are designed to protect models from backdoor attacks and correct models that have been compromised by such attacks, respectively. Different from them, backdoor detection in the inference phase serves as firewall to filter out malicious samples when we are unable to protect or correct the model. Due to the different purposes of these methods mentioned above,
Overall comparison. As shown in Table 1 and Table 2, BDetCLIP consistently outperformed the comparison meth- ods in almost all attack settings and datasets. Specifically, BDetCLIP achieved an impressive average AUROC, exceed- ing 0.938 in multimodal attack scenarios and 0.972 in tradi- tional attack scenarios, firmly establishing its superior effec- tiveness. In contrast, the unimodal detection methods gen- erally exhibited significantly lower performance. Although TeCo occasionally approached BDetCLIP's performance, it demonstrated inconsistencies and performed worse overall. This highlights the ineffectiveness of unimodal methods for robust rest-time backdoor detection, contrasting sharply
6

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table I. AUROC comparison different attacks. The best result highlighted in bold.
Attack+
Multimodal Attack
Average
Detection)
BadCLIP-1
BadCLIP-2
TrojVQA
BadEncoder
STRIP
0.794
0.987
0.255
0.341
0.594
SCALE-UP
0.669
0,976
0.744
0.694
0.771
TeCo
0.443
0.428
0.438
0.567
0.469
BDaCLIP (Ours)
0.900
0.977
0.978
0.938
Attack+
Traditional Attack
Average
Detection]
BadNet
Blended
BadNet-LC
Blended-LC
ISSBA
WaNet
STRIP
0.772
0.111
0.803
0.150
0.351
0.243
0.405
SCALE-UP
0.737
0.692
0.690
0.853
0,515
0,920
0.735
TeCo
0.827
0.954
0.799
0,979
0.496
0.946
0.934
BDeCLLIP (Ours)
0.972
0.983
0.964
0.997
0.927
0.989
0.972
Table AUROC comparison on Food101 (Bossard et al., 2014) and Caltech101 (Fei-Fei et al., 2004) datasets. The best result highlighted bold.
Target class
Method
BadNet
Blended
Average
STRIP
0.893
0.244
0.569
SCALE-UP
0.768
0.671
0.720
Food101 (Baklava)
TeCo
0.834
0.949
0.892
BDeICLIP (Ours)
0.941
0.977
0,959
STRIP
0,868
0,672
0.770
SCALE-UP
0.632
0.585
0.609
Caltech101 (Dalmatian)
TeCo
0.637
0.913
0.775
BDeICLIP (Ours)
0.977
0.989
0.983
Table Inference on ImageNet-1K (Russakovsky et al., 2015).
Totally 50000 test samples,
Method
STRIP
SCALE-UP
TeCo
BDetCLIP (Ours)
Inference time
253m 42.863s
9m 7.066s
637m 34.350s
3m 8.436s
with the clear effectiveness advantages offered by BDeICLIP. As for efficiency, IDetCLIP also achieved the best perfor- mance for the inference time. As shown in Table 3, TeCo is the slowest detection method, even more than 160 times slower than BDetCLIP. This is because TeCo uses many time-consuming corruption operators on images which is too heavy in CLIP. This operation is also used in unimodal methods STRIP and SCALE-UP. In contrast, BDeICLIP only leverages the semantic changes in the text modality twice for backdoor detection, i.c., benign and malignant class-specific prompts. Therefore, BDetCLIP can achieve fast test-time backdoor detection in practical applications. In conclusion, BDeCCLIP exhibits superior performance with respect to both effectiveness and efficiency when compared the existing unimodal methods.
cretely, other methods have a significant drop in perfor- mance compared with the results in Table while BDet- CLIP also maintains high level of AUROC (e.g.. the av- erage AUROC is 0.950). This observation validates the versatility of BDeICLIP in different vision model architec- tures of CLIP.
Backdoor detection for backdoored CLIP pre-trained CC3M. Following CleanCLIP (Bansal al., 2023), also considered pre-training CLIP from scratch the poisoned CC3M dataset. As shown in Table 5, compared with the results in Table 1, STRIP failed to achieve detection in almost all cases, SCALE-UP and TeCo became worse, while BDetCLIP also achieved superior performance across attack settings, which definitely validates the versatility of BDetCLIP in different model capabilities of CLIP.
Backdoor detection for CLIP using ViT-B/32. We also evaluated the case where ViT-B/32 (Dosovitskiy al., 2020) served as the visual encoder of backdoored CLIP. As shown in Table 4, our proposed BDetCLIP also achieved superior performance across all types of backdoor attacks. Con-
The impact of the number of class-specific benign prompts. As shown in Table 6, we find that increas- ing the number of class-specific benign prompts strengthens detection against various backdoor attacks. For example, when using 1 benign prompt, the average AUROC is 0.952,

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table Comparison of AUROC on ImageNet-I IK (Russakovsky al., 2015), the visual encoder of CLIP ViT-B/32 (Dosovitskiy al.,
2020). The target label of the backdoor attack is "Banana".
Atlack->
BadNet
Blended
BadNet-LC
Blended-LLC
Average
Detection
STRIP
0.527
0.025
0.606
0.020
0.295
SCALE-UP
0.652
0.875
0.649
0.867
0.761
TeCo
0.714
0.969
0.727
0.969
0.845
BDeCCLIP (Ours)
0.930
0.980
0.903
0.985
0.950
Table 5. Comparison of AUROC on ImageNet-1K I (Russakovsky et al., 2015) dataset, the CLIP is pre-trained with CC3M (Sharma al.,
2018). The target label of backdoor attack "Banana".
Attack
BadNet
Blended
Label-Consistent
Average
Detection,
STRIP
0,061
0.005
0.420
0.162
SCALE-UP
0.651
0.627
0.612
0.630
TeCo
0.779
0.782
0.765
0.775
BDeCCLIP (Ours)
0.986
0.982
0.908
0.959
Table 6. Comparison of AUROC using different numbers of class-specific henign prompts ImageNet-1K KK (Russakovsky al., 2015).
target label the backdoor attack "Banana".
Attack->
BadNet
Blended
BadNet-LC
Blended-LCC
Average
The number of class-specific benign prompts.
using class-specine benign prompt
0.927
0.984
0,901
0,995
0.952
using class-specifie benign prompis
0.960
0.984
0.948
0.996
0.972
using class-specific benign prompts
0.969
0.984
0.959
0.996
0.977
using class-specific benign prompts
0.972
0.985
0.964
0.997
0.980
Table Comparison of AUROC using different word counts the caassppetuubbdd random text on ImageNet-1K (Russakovsky al., 2015). The target label of the backdoor attack is "Banana".
Attack
BadNet
Blended
BadNet-LC
Blended-LCC
Average
random sentence class specifcc malignant prompt
no more than 10 words
0,972
0.983
0.964 0.954
0.997
0.979
no more than 20 words more than 30 words
0.963
0,968
0,993
0.970
0.950
0.941
0.957
0.992
0.960
which increases to 0.980 when using benign prompts. This because more diverse fine-grained description texts ex- pand the difference of contrastive distributions, which is more beneficial for BDeICLIP to distinguish backdoored and clean images. Therefore, it is of vital importance to leverage more diverse description texts in BDetCLIP.
4.3. Threshold Selection
Our proposed BDeCCLIP can efficiently and effectively map input images to a linearly separable space. The defender needs
threshold distinguish between clean im- ages and backdoor images. In determining this threshold, follow widely used protocol in previous studies (Guo et al., 2023), (Liu et al., 2023): the defender can set a proper thresh- old based on a small set of clean validation data. Specifically, we first sampled clean samples at the designated sampling rates. Then, using formulation 4, we computed the con- trastive distribution difference for all samples, ranked them from largest smallest, and selected the 85th percentile as the threshold (notably, the specific threshold percentile can be adjusted based on real-world defense requirements). To assess the sensitivity of our approach, we chose three sampling ratios: 1%, 0.5%, and 0.1% As shown in Table 8, 9 and 10, even when a very small sampling ratio used, despite the increased standard deviation the threshold, our method achieves exceptional performance across all metrics,
The impact of the text length of class-specific malignant prompts. As shown in Table 7, the performance has bit of drop as the number of words in class-specific malig- nant prompts increases. This is because generated in class-specific malignant prompts would greatly destroy the semantics of class-specific malignant prompts, thereby increasing the contrastive distribution difference of backdoored images (close to that of clean images). This would degrade performance of detection. Besides, the performance on BadNet and Blended attacks exhibit rel- atively high sensitivity the text length of class-specific malignant prompts.
random texts
8

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table The backdoor target lahel is ant. We use backdoor ratio of 0.3 and sampling rate of I 1%.
Backdoor
Accuracy
Recall
FI
AUROC
Threshold
Badnet
0.8941 = 0.0107
0,9902 0.0013
0.8488 0.0127
0.9906 + 0.0003
11,7225 1.2723
Blended
0.8772 + 0.0061
0.9279 0.0142
0.8193 # 0.0151
0.9425 + 0.0003
12.0281 * 1.2399
Badnet-I.C.
0.8938 0.0074
0.9842 0.0016
0.8476 0.0088
0.9796 0.0004
16.7526 0.8944
Blended-LC
0.8837 = 0.0068
0.9396 0.0102
0.8290 0.0067
0,9420 + 0.0005
15.9315 1.2748
Table 9. The backdoor target label ant. use backdoor ratio 0.3 and a sampling rate of 0.5%.
Attack
Accuracy
Recall
F1
AUROC
Threshold
Badnet
0.8950 = 0.0160
0,9903 0.0013
0.8502 0.0189
0.99U8 + 0.0003
11.6161 1.8596
Blended
0.8772 = 0.0109
0.9224 0.0211
0.8186 = 0.0096
0.9416 + 0.0003
11,7094 = 1.9545
Badnet-LCC
0.8958 = 0.0128
0.9835 0.0029
0.8501 =
0.9797 + 0.0004
16.4568 1.5488
Blended-LC
0.8865 0.0070
0.9347 0.0106
0.8317 0.0070
0.9422 # 0.0005
15.3494 1.3340
Table 10. The backdoor target label is ant. We use backdoor ratio 0.3 and a sampling rate 0.1%.
Attack
Accuracy
Recall
F1
AUROC
Threshold
Badnet
0.8775 0.0107
0.9904 0.0040
0.8312 0.0418
0.9905 + 0.0004
13.0927 4.3069
Blended
0.8564 * 0.0315
0.9391 0.0430
0.7987 0.0279
0.9424 + 0.0003
14.2042 # 4,4056
Badnet-T.C.
0.8799 + 0.0453
0.9831 0.0082
0.8341 # 0.0488
0.9795 +: 0.0005
17.6179 4. 4.8725
Blended-LCC
0.8722 0.0269
0.9404 0.0363
0.8167 0.0524
0,9422 0.0004
16.9313 4.2465
particularly in terms of recall, due its high AUROC value, which demonstrates its strong discriminative capability.
prompting, which is called BDe(CLIP. For our proposed BDetCLIP, first prompted the language model (e.g., GPT- 4) to produce class-related description texts (benign) and class-perturbed random texts (malignant) by specially de- signed instructions. Then, we calculated the distribution difference in cosine similarity between images and the two types of class description texts, and utilized this distribu- tion difference as the criterion to detect backdoor samples. Comprehensive experimental results validated that our pro- posed BDeICLIP is more effective and more efficient than state-of ofte-offtheeart backdoor detection methods.
4.4. Additional experiments
In Appendix D.I, present detection results for more backdoor target categories ImageNet, along with results under different backdoor ratios (i.e., 0.3, 0.5, and 0.7) in Appendix D.8, both demonstrating the stability and effec- tiveness of our method. In Appendix D.2, D.3, and D.4, we present results for open-set detection, semantic trigger, and multi-target attack, all confirming our method's s state- of-the-art performance. Appendix D.5 provides the time and money cost of GPT-4 prompt generation, highlighting the efficiency and cost-effectiveeess of our method. We also test eight open-source LLMs for prompt generation in Appendix D.6, which demonstrates that open-source mod- els can serve as stable prompt generators, replacing GPT-4 in generating both benign and malicious prompts. Further more, we investigate the relationship between thresholds and the number of benign prompts m in Appendix D.7.
6. Limitations
The main limitation of this work lies in that only the CLIP model is considered because existing backdoor research on multimodal contrastive learning commonly considers CLIP as representative victim model due to its reproducibility. In addition, our employed strategy to determine the threshold is relatively simple. More effective strategies could be further proposed to obtain a more suitable threshold.
5. Conclusion
7. Future work
In this paper, we provided the first attempt at a computation- ally efficient backdoor detection method to defend against backdoored CLIP in the inference stage. We empirically ob- served that the visual representations of backdoored images insensitive to significant changes in class description texts. Motivated by this observation, we proposed novel test-time backdoor detection method based on contrastive
We aim to discuss future work from both offensive and defensive perspectives. For more sophisticated backdoor attacks, we propose designing triggers that can naturally adapt to changes in prompt semantics, thereby creating more covert backdoor attacks. For enhanced backdoor defense. we suggest developing a framework for evaluating prompt quality to further improve the quality of prompts.

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Acknowledgement
Chen, Y., Qi. F., Gao, H., Liu, Z., and Sun, M. Textual backdoor attacks can be more harmful via two simple tricks. arXiv preprint arXiv: 10..02277. 2021.
Feng Liu is supported by the Australian Research Council (ARC) with grant number DE240101089, LP240100101, DP230101540 and the NSF&CSIRO Responsible Al pro- gram with grant number 2303037.
Coates, A., Ng, A., and Lee, H. An analysis of single- layer networks in unsupervised feature learning. In Pro- ceedings of the fourleenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Impact Statement
Our research contributes to Al security by detecting back- door samples in the inference phase, which has a positive social impact. However, we acknowledge the possibility that sophisticated attackers could use our insight to bypass our defense to threaten AI security. Future work should explore the robustness of our method against adaptive attacks.
Doan, K., Lao, Y., and Li, P. Backdoor attack with imper- ceptible input and latent modification. In NeurIPS, pp. 19947,1895,.222 2021.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x16 Transformers for image recognition at seale. arXiv preprint wwXiv:2010.11922 2020.
References
words:
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Fawcett, T. An introduction to roc analysis. Pattern Recog- nition Letters, 2768):861-874, 2006.
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Fei-Fei, I.., Fergus, R., and Perona, Learning generative visual models from few training examples: An incremen- tal bayesian approach tested 101 object categories. In CVPR Workshop, pp. 178-178 2004.
Bai, J., Gao, K., Min, S., Xia, S.-T., Li, Z., and Liu, W. Badclip: Trigger-aware prompt learning for backdoor attacks on clip. arXiv preprint arXiv:2311.1611
2023. Bansal, H., Singhi, N., Yang, Y., Yin, F., Grover, A., and Chang, K.-W. Cleanclip: Mitigating data poisoning at- tacks multimodal contrastive learning. In ICCV, pp. 112-123, 2023.
Feng, S., Tao, G., Cheng, S., Shen, G., Xu, X., Liu, Y.,
Zhang, K., Ma, S., and Zhang, X. Detecting backdoors in pre-trained encoders. In CVPR, pp. 16352-16362, 2023a.
Feng. Z., Bair, A., and Kolter, Z. Text descriptions are compressive and invariant representations for visual leamn ing, 2023b.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101- mining discriminative components with random forests. In ECCV, pp. 446-461, 2014.
Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., and Nepal, S. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
Carlini, and Terzis, A. Poisoning and backdooring con- trastive learning. arXiv preprint arXiv:2106. 106666.0. 2021.
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149 2023.
Gao, Y., Li, Y., Gong, X., Xia, S.-T., and Wang, Q. Backdoor attack with sparse and invisible trigger. arXiv preprint arXiv:2306.06200 2023.
Gu, T., Bolan-Gavitt B., and Garg, S. Badnets: Identify- ing vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 178.00778. 2017.
Chen, W., Wu, B., and Wang, H. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NewrlPS, pp. 9727-9737, 2022.
Guo, J., Li, Y., Chen, X., Guo, H., Sun, L., and Liu, C. Scale-up: An efficient black-bor box input-level backdoor de- tection via analyzing scaled prediction consistency. arXiv preprint arxiv:2302.0322. 2023.
Chen, X., Liu, C., Li, B., Lu, K., and Song, D. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 17le:05222,55 2017.
10

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Maniparambil, M., Vorster, C., Molloy, D., Murphy, N., McGuinness, K., and O'Connor, N. E. Enhancing clip with gpt-4: Harnessing visual descriptions prompts. In ICCV, pp. 262-271, 2023.
Hou, L.,, Feng, R., Hua, Z., Luo, W., Zhang, L. Y., and Li, Y. Ibd-psc: Input-level backdoor detection via parameter-oriented scaling consistency. arXiv preprint arXiv:2405.09786. 2024.
Min, R., Qin, Z., Shen, L., and Cheng, M. Towards stable backdoor purification through feature shift tuning. In NeurIPS, 2023.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T.. Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z.., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pp. 1904-4916. 2021.
Nguyen, A. and Tran, A. Wanet -amet-impereeptibll warping- based backdoor attack. arXiv preprint arXiv: 102. 2021.
Pal, S., Yao, Y., Wang, R., Shen, B., and Liu, S. Back- door secrets unveiled: Identifying backdoor data with optimized scaled prediction consistency. arXiv preprint 0rXiv:2403.10711 2024.
Jia, J., Liu, Y., and Gong, N. Z. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In SP. pp. 2043-2059, 2022.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint 2xXiv:330000622 2023.
Pratt, S., Covert, I., Liu, R., and Farhadi, A. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pp. 15691- 15701, 2023.
Kuang, J., Liang, S., Liang, J., Liu, K., and Cao, X. Adversarial backdoor defense in clip. arXiv preprint arXiv:2409. 15968, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. ICML, pp. 8748-8763, 2021.
Li, Y., Li, Y., Wu, B., Li, L., He, R., and Lyu. S. Invisible backdoor attack with cample-specifie triggers. In ICCV, 2021a.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical conticontitioaa image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):3, 2022.
pp.
Li, Y., Li, Y., Wu, B., Li, L., He, R., and Lyu. S. Invisible backdoor attack with sample-specific triggers. In JEEE International Conference on Computer Vision (ICCV), 2021b.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang. Z., Karpathy, A., Khosla, A., Bern- stein, M., al. Imagenet large scale visual recognition challenge. International journan of computer vision, 1 115: 211-252, 2015.
Li, Y., Jiang, Y., Li. Z., and Xia, S.-T. Backdoor learning: survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Saha, O., Van Horn, G., and Maji, S. Improved zero-shot classification by adapting vlms with text descriptions. arXiv preprint arXiv:2401.02460 2024.
Liang, S., Zhu, M., Liu, A., Wu, B., Cao, X., and Chang, E.-C. Badclip: Dual-embedding guided backdoor at- tack multimodal contrastive learning. arXiv preprint arXiv:2311. 12075, 2023.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2556- 2565, 2018.
Liang, S., Liu, K., Gong. J., Liang, J., Xun, Y., Chang, E.-C., and Cao, X. Unlearning backdoor threats: Enhancing backdoor defense in multimodal contrastive learning via local token unlearning. arXiv preprint arXiv:2403.1 16257, 2024.
Shi, Y., Du, M., Wu, X., Guan, Z., Sun, J., and Liu, N. Black- box backdoor defense via zero-shot image purification. In NeurlPS, 2023.
Liu, X., Li, M., Wang, H., Hu, S., Ye, D., Jin, H., Wu, L., and Xiao, C. Detecting backdoors during the inference stage based on corruption robustness consistency. In CVPR, pp. 16363-16372, 2023.
Singh. N. D., Croce, F., and Hein, M. Perturb and recover: Fine-tuning for effective backdoor removal from clip. arXiv preprint arXiv:2412.00721 2024.
Souri, H., Fowl, L., Chellappa, R., Goldblum, M.. and Goldstein, T. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In NeurIPS, pp. 19165-19178. 2022.
Liu, X., WU, J., and Zhang, T. Multi-modal attribute prompting for vision- language models. arXiv preprint arXiv:2403.002/9 2024.

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Sur, Sikka, K., Walmer, M., Koneripalli, K., Roy, A., Lin, X., Divakaran, A., and Jha, S. Tijo: Trigger inversion with joint optimization for defending multimodal backdoored models. In ICCV, pp. 165-175, 2023.
Yang. W., Gao, J., and Mirzasoleiman, B. Better safe than sorry: Pre-training clip against targeted data poisoning and backdoor attacks. arXiv preprint arXiv:2310.05862, 2023a.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T. Shahri- ari, B., Ramé, A., et al. Gemma Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024.
Yang, W., Gao, and Mirzasoleiman, B. Robust contrastive language- -maage pretraining against data poisoning and backdoor attacks. Advances in Neural Information Pro-
cessing Systems, 36:10678-100 10691, 2023b.
Yang. Y., Panagopoulou, A., Zhou, S., Jin, D., Callison- Burch, C., and Yatskar, M. Language in bottle: Lan- guage model guided concept bottlenecks for interpretable image classification. In CVPR, pp. 19187-19197, 2023c.
Tran, B., Li, J., and Madry, A. Spectral signatures in back- door attacks. In NeurlPS, 2018.
Turner, A., Tsipras, D., and Madry, A. Label-consistent backdoor attacks. arXiv preprint 1rXiv:19/,.02771, 2019.
Young, A., Chen, B., Li, C.,, Huang, C., Zhang, G., Zhang. G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.044m 2024.
Udeshi, S., Peng. S., Woo, G., Loh, L.,, Rawshan, L., and Chattopadhyay, S. Model agnostic defence against back- door attacks in machine learning. IEEE Transactions Reliability, 71(2):880-895 2022.
Yu, S., Liu, S., Lin, Z., Pathak. D., and Ramanan, D. Language models black-box optimizers for vision- language models. arXiv preprint arX1v:2309.05950. 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurIPS, 2017.
Zeng. Y., Park, W., Mao, Z. M., and Jia, R. Rethinking the backdoor attacks' triggers: A frequency perspective. In ICCV, pp. 16473-16481, 2021.
Walmer, M., Sikka, K., Sur, I., Shrivastava, A., and Jha, S. Dual-key multimodal backdoors for visual question answering. In CVPR, pp. 15375-15385, 2022.
Zhang, Y., Albarghouthi, A., and D' Antoni, L. Bagflip: certified defense against data poisoning. In NeurlPS, pp. 31474-31483, 2022.
Wenger, E., Passananti, J., Bhagoji, A. N., Yao, Y.. Zheng, H., and Zhao, B. Y. Backdoor attacks against deep leam- ing systems in the physical world. In CVPR, pp. 6206- 6215, 2021.
Zhang. Z., Niu, Y., Liu, X., and Li, B. Tuning vision- language models with candidate labels by prompt align- ment. arXiv preprint Xiv:24077007633 2024.
Wu, X., Zhu, F., Zhao, R., and Li, H. Cora: Adapting clip for open- ppen-vocabulary detection with region prompting and anchor pre-matching. In CVPR, pp. 7031-7040, 2023.
Xu, H.. Ghosh, G., Huang. P.-Y., Okhonko, D., Aghajanyan, A., Metze, F,, Zettlemoyer, I.., and Feichtenhofer, C. Videoclip: Contrastive pre-training for zero-shot video- text understanding. In EMNLP, pp. 6787-6800, 2021.
Xu, L., Chen, Y., Cui, G., Gao, H., and Liu, Z. Explor- ing the universal vulnerability of prsep learning paradigm. arXiv preprint arXiv:2204.05239 2022.
Xun, Y., Liang, S., Jia, X., Liu, X., and Cao, X. Ta-cleaner: A fine mme-grained text alignment backdoor defense strat- egy for multimodal contrastive learning. arXiv preprint 240. 1700,, 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15/ 2024.
12

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Prompt used to generate class-related description texts:
I am creating class attributes for a zero-shot image recognition algorithm to classify different images. The attributes are part of the fine-grained information about the classes information must be deeply related to the category, and cannot be some low-quality information, such as goldfish are living things, goldfish have life, and on. For example, if I say what attributes help us identify goldfish? You should respond: "goldfish":[
"Goldfish are known for their bright orange or gold color but they can also come in other colors like white, black, red, and yellow.", "Goldfish have variety of body shapes, ranging from the common slim-bodied
type more rounded or egg-shaped varieties.",
"Goldfish typically have single dorsal fin, paired pectoral and pelvic fins, and a forked caudal (tail) fin. Some varieties, like the fancy goldfish, may have long, flowing fins."
"Most goldfish have shiny, metallic scales, but some varieties, like the pearl scale goldfish, have uniquely textured scales."
"Goldfish are known for their active swimming behavior and are often seen exploring their environment."]
Now I want to ask you: What attributes help identify (Class Name)?
Prompt used to generate class cosss-perturbd random texts:
Please randomly generate sentence of no more than 10 words unrelated to (Class Nome)
GPT-4 OUTPUT Example (Class Name: goldfish):
The bright sun cost shadows on the bustling city street.
Figure 4, Prompts for generating class-related description texts and class class-perturbed random texts.
A. Prompt Design
Generative Pretrained Large Language Models, such as GPT-4, have been demonstrated (Yang et al., 2023c; Pratt et al., 2023; Maniparambil et al., 2023; Yu et al., 2023; Saha et al., 2024; Feng et al., 2023b; Liu et al., 2024) to be effective in generating visual descriptions to assist CLIIP in classification tasks for the following reasons: (1) These models are trained on web-scale text data, encompassing a vast amount of human knowledge, thereby obviating the need for domain-specinn annotations. (2) They can easily be manipulated to produce information in any form or structure, making them relatively simple to integrate with CLIP prompts.
In our study, we harnessed the in-context learning capabilities of GPT-4 to generate two types of text-related description text and class- -lattreet description text. The prompts used for generating the text are illustrated in Figure 4.
B. More Details about the Experimental Setup
Details of traditional attack. Following the attack setting CleanCLIP (Bansal et al., 2023), we consider two types of attack means for CLIP, including fine-tuning pre-trained clean CLIP on the part of backdoored image-text pairs from CC3M and pre-training backdoored CLIP by the poisoned CC3M dataset. In the first case, randomly select 500,000 image-text pairs from CC3M as the fine-tuning dataset among which we also randomly select 1,500 of these pairs as target backdoor samples and apply the trigger to them while simultancously replacing their corresponding captions with the class template for the target class. Then, we can fine-tune CLIP with the backdoored dataset. We finetune the pretrained model for 5 epochs with an initial learning rate of le-6 with cosine scheduling and 50 warmup steps and use Adam as the optimizer. In the second case, following the attack setting CleanCLIP (Bansal et al., 2023), we randomly select 1,500 image-text pairs from CC3M as target backdoor samples. Then, we pre-train CLIP from scratch the backdoored CC3M dataset. We
https://gihuu.ccm/ppeeiiiiCLLL
13

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Algorithm 1 BDetCLIP
Require: CLIP's infected visual encoder V+). and infected text encoder (.), threshold 7, Test set Mesti class specifcc benign prompts ST class-specific malignant prompts RtTj. cosine similarity *()
1: for x' in Hest do
2: Compute benign similarity
((V* Ex= (ST)))
3: Compute malignant similarity ((V" ( (RT,))
22(x') - Σ
2 (x)))) ET(ST)
4:
jml
(v (
5: (x=") < then
6:
Mark x' backdoored
7: else
8:
Mark æ' clean
9: end if
10: end for
11: Output the detection results
trained for 64 epochs with a batch size of 128, an initial learning rate of 0.0005 for cosine scheduling, and 10000 warm-up steps for the AdamW optimizer. For the BadNet, Blended, BadNet-LC, and Blended-LCC attacks, the architecture used ResNet-50, with the target attack class being "Banana". For the WaNet attack, the architecture is ResNet-50, and the target attack class is "Goldfish". For the ISSBA attack, the architecture is ViT-B/32, and the target attack class is "Banana". All experiments are conducted on NVIDIA 3090 GPUs.
Details of multimodal attack.
For BadCLIP-1 (Liang al., 2023), we used the officially provided weight files for detection. The attack target was "Banana," " and the model architecture used was ResNet-50. The evaluation was conducted on ImageNetIK.
For BadCLIP-2 (Bai et al., 2023), we followed the official setup for reproduction. BadCLIP-2 is backdoor attack against prompt learning scenarios, which uses learnable continuous prompt trigger. Although approach is designed for CLIP that uses discrete prompts for classification tasks, we can make simple modifications to detect it. Specifically, we keep the benign prompt unchanged and modify the malignant prompt to combination of learnable context and random text. In the experimental setup, we chose ViT-B/16 as the encoder attacked "Face," and detected on Caltech101 using reversed contrast distribution difference.
For BadEncoder, we adopted the officially provided weights. It is important to note that BadEncoder is not a backdoor attack targeting multimodal contrastive learning but rather a backdoor attack targeting self-supervised learning encoders. We followed the settings the official paper, with the target attack class being "truck," " and detected it on STL-10 (Coates al., 201 1) using reversed contrast distribution difference.
TrojVQA is dual-key backdoor attack targeting multimodal visual question-answering models. We used "SUDO" as the text trigger and 16x16 patch as the visual trigger, with the target class being "banana." The detection was performed on ImageNetIK.
Details of comparing methods.
STRIP (Gao et al., 2019) is the first black-box TTSD method that overlays various image patterns and observes the randomness of the predicted classes of the perturbed input to identify poisoned samples. The official open-sourced codes for STRIP (Gao et al., 2019) can be found at: https: / /gi thub. com ccmm/gart garrii wtw.ggggrrrrrrr///o//o/rgro songys for each input image, we use 64 clean images from the test data for superimposition.
In our experiments,
SCALE-UP (Guo et al., 2023) is also a method for black-bol box input-level backdoor detection that assesses the mali- ciousness of inputs by measuring the scaled prediction consistency (SPC) of labels under amplified conditions, offering
14

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
effective defense in scenarios with limited data or no prior information about the attack. The official open-sourced codes for SCALE-UP (Guo al., 2023) can be found at: https:/ http:/gthhh..oooo Jun JunfengGo/ SCALE-UP.
TeCo (Liu et al., 2023) modifies input images with common comrptiion and assesses their robustness through hard-label outputs, ultimately determining the presence of backdoor triggers based on deviation measurement of the results. The of- ficial open-sourced codes for TeCo (Liu et al., 2023) can be found at: https: / / /github. com/CGCL In our experiments, considering concerns about runtime, we selected "elastic_trane transform". "gaussian sosiale__oii"""_eiei bobot_noise", "impulse_noe snse_nooiee""""e "motion_blud "snow", "frost", "fog". "brightness", "contrast", "pixelate", and jipeg_compression as methods for corrupting images. The maximum corruption severity was set to 6.
Details of datasets. ImageNet-IK (Russakovsky et al., 2015) consists of 1,000 classes and over a million images, making it a challenging dataset for large-scale image classification tasks. Food- 101 (Bossard et al., 2014), which includes 101 classes of food dishes with 1,000 images per class, and Caltech101 (Fei-Fei et al., 2004), an image dataset containing 101 object categories and background category with 40 to 800 images per category, are both commonly used for testing model performance on fine-grained classification and image recognition tasks. In our experiment, we utilized the validation set of ImageNet- IK (Russakovsky et al., 2015), along with the test sets of Food-101 101 (Bossard et al., 2014) and Caltech101 (Fei-Fei et al., 2004). By using fixed backdoor ratio (0.3) on different downstream datasets in the evaluation, there are 15,000 (out of 50,000) backdoored images on ImageNet- KK 7,575 (out of 25,250) backdoored images on Food-101, and 740 (out of 2,465) backdoored images on Caltech-101. Moreover, we also use larger backdoor ratios (0.5 and 0.7) on ImageNet- K, resulting in 25,000 and 35,000 backdoor samples respectively.
C. Defense results comparison with other defend methods
To facilitate direct comparison of defense effectiveness, we made the necessary modifications. Specifically, during the inference stage, we set the backdoor ratio to 1. In BDetCLIP, samples with distribution differences below the threshold are directly discarded. The Attack Success Rate (ASR) is then calculated as the ratio of successfully attacked backdoor samples to the total number of backdoor samples. We argue that this strategy is reasonable in practical scenarios. To demonstrate the reliability and stability of our experimental results, we used the threshold selection method described in Section 4.3, performed random sampling ten times, and calculated both the mean and the standard deviation. For our detection experiments, we utilized the backdoored model provided by CleanCLIP (Bansal al., 2023) as the victim model and compared the defense performance with the results reported in CleanCLIP (Bansal et al., 2023). As shown in Table 11, BDetCLIP can effectively decrease the ASR compared with the current fine-tuning defense method CleanCLIIP (Bansal et al., 2023), proving that our BDeICLIP could be used to defend against backdoor attacks effectively in practical applications. To further validate the effectiveness of our method against state-of-the-ar backdoor attacks targeting CLIP, we employed the compromised model weights provided by BadCLIP (Liang et al., 2023) as our defense testing subject. BadCLIP (Liang et al., 2023) represents the most advanced backdoor attack method specifically designed for CLIP. In our experiments, we compared our approach with baseline methods including CleanCLIP (Bansal et al., 2023), CleanerCLIP (Xun et al., 2024), and PAR (Singh et al., 2024). We also apply RoCLIP (Yang al., 2023b) to the fine-tuning stage to defend against BadCLIP. Specifically, we follow the official settings of RoCLIP, using BadCLIP polluted data as fine-tuning data, and fine-tuning 24 epochs. We also tried SAFECLIP (Yang et al., 2023a) and we found that it could not produce effective defenses, and after the defense was made, the zero-shot classification capability of SAFECLIP was close to O. As shown in Table 12, the experimental results demonstrate that our method achieves superior defensive performance compared to all baseline approaches.
Table 11. Comparison with the Defense Results of CleanCLIP. The metric is ASR.
Attack
CleanCLIP
BDetCLIP (ours)
Badnet
0,1046
0.0011 = 0.0003
Blended
0.0980
0.0003 0.0001
Label Consistent
0.1108
0.1085 = 0.0156
D. More Experimental Results
D.I. Backdoor attack for more target classes
We conducted more BadNet attacks on the following classes from ImageNet: "Goldfish," " "Lion," "Rooster," "Tench," " "Baskethall," and "Ant." " The detection results are presented in 13, demonstrating that our method consistently achieved the
15

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table 12. Comparison with the Defense Results on BadCI.P((Liang al., 2023). The metric ASR.
Method
ASR
CleanCIIIP
0.9196
CleanerCIP PAR
0,2808
0.312
RoCLIP BDetCLIP (ours)
0,1330
0.0444 0.0110
best detection performance across cases.
Table 13. Detection performance of different target classes
Category
SCALE-UP
BDetCLIP (ours)
Goldfish
0.781
0.977
Lion
0.806
0.992
Rooster
0.741
0.951
Tench
0.673
0.992
Basketball
0.750
0.984
Ant
0.690
0.990
D.2. Backdoor detection for open-set detection.
We have conducted additional experiments to validate the effectiveness of our proposed BDeiCLIP for open-set classification tasks. Specifically, we added subset of Caltech-101 the open set to ImageNet and set the backdoor ratio 0.3. Table 14 shows that our proposed BDetCLIP can also achieve impressive performance the open-set classification task, which verifies the transferability of our proposed BDetCLIP to other tasks in VLMs.
Table Delection performance on the open-set classification task.
Backdoor
AUROC
BadNet
0.933
Blended
0.936
BadNet-LC
0.929
Blended-LC
0.991
D.3. Backdoor defection for semantically meaningful trigger.
We have considered the scenario where the backdoor trigger has semantic meaning. Specifically, we used the popular "Hello Kitty" trigger and also achieve good detection results in Table 15.
D.4. Backdoor detection for multi-targets attack.
We have conducted more experiments about using BDetCLIP defend against multi-targe attacks. Specifically, to achieve the multi-target attack, we poisoned 1,000 (out of 500,000) samples for each target class (i.e., "goldfish", "basketball" and "banana") respectively. We fine-tuned the CLIP based on the poisoned dataset (the backdoor ratio is 0.3.) following the original experimental setting. Then, we used BDetCLIP to detect the backdoored CLIP. Table 16 shows that our BDetCLII can still achieve impressive detection performance against the multi-target attack.
D.5. Cost and Time Efficiency of Prompt Generation
We recorded the time and monetary costs associated with generating two types of prompts for each class in the Food- 101 dataset using GPT-4 and GPT-4o. The results are summarized in Table 17. The results indicate that utilizing GPT-4 (or GPT-4o) for prompt generation is both efficient and cost-effective. Moreover, the prompt generation process can be conducted offline (prior to test-time detection), allowing the generated prompts to be directly employed in BDetCLIP for real-time detection tasks. Consequently, concerns regarding the runtime of the prompt generation step are minimal.
16

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table 15. The detection performance of Backdoor Attacks with semantically meaningful triggers ("Hello Kitty").
SCALE-UP
BDeICLIP (ours)
0.6111
0.8554
Table 16. The detection performance of Multi-target Attacks,
SCALE-UP
BDetCLIP (ours)
0.5404
0.9858
Table 17. Run Time and Money Cost by using GPT-4 or GPT-4o.
GPT-4
Category
Run Time
Money Cost
Benign
15m19s
2.38 $
Malignant
2m5s
0.12 $
GPT-4o
Category
Run Time
Money Cost
Benign
5m33s
0.42 $
Malignant
Im24s
0.06 $
D.6. Using open-source models for prompts generation
We also explored the feasibility of replacing GPT-4 for prompt generation with open-source models. We utilized the following models: (lama3-8B-Instrus (Dubey et al., 2024), (Young et al., 2024), gemma-2-22 2--iit (Team et al., 2024), semma-22-bbir (Team et al., 2024), Phi-3.5-mini-instuuee (Abdin et al., 2024), Qwen2.5-7B-Inss
-and7-BB--nstruut--02222 (Jiang et al., 2023), Yi-1.5-9B-Chat
(Yang et al., 2024), and 1reen255-1BB-Insteee (Yang et al., 2024). The experimental results of using these varying open-source language models were recorded in Table 18. Additionally, we documented the (denoted "M") in Table
prompt generation times for Llama3-8B-Instre (denoted "L") and 19.
Although using open-source models for prompt generation may require more time (which minimally impacts detection efficiency when performed offline), the detection performance remains comparable to that achieved with GPT-4. This indicates that using open-source models is promising alternative for prompt generation.
D.7. The relationship between the number of benign prompts and threshold.
We denote the number of class-specific benign prompts as m. We conducted tests with m = 4,3, applying the aforementioned threshold selection strategy detailed in Section 4.3. Random sampling was performed ten times for each case. Subsequently, we calculated both the variance and the mean of the selected thresholds. The mean value was then employed as the threshold for subsequent experiments. As shown in 21, We can see that the larger m is, the better the overall effect will be, and the threshold will be correspondingly larger. This is intuitive: m increases, the number of benign prompts grows, providing more fine-grat mne-grained information, which increases the semantic differences between benign prompts and malicious prompts.
D.8. Varying ratios of test-time backdoor samples.
We conducted comparative analysis between SCALE-UP and method to explore the effects of variations in backdoor proportions on our efficacy. Results can be found in Table 22, 23, and 24. The results indicate that under different proportions of test-time backdoor samples, our method (BDetCLIP) consistently outperforms the baseline method SCALE-UP. Whether at a backdoor sample ratio of 0.3, 0.5, or 0.7, BDeiCLIP achieves higher AUROC scores across all target categories and attack detection scenarios compared to SCALE-UP. This suggests that BDetCLIP exhibits higher robustness and accuracy in detecting backdoor samples, thereby enhancing the reliability and security of multi-modal models against backdoor attacks.
17

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table 18. Detection performance by using different models
Model
BadNet
Blended
GPT-4
0.972
0.983
Llama3-8B-Instree
0.947
0.983
Bestral-BB-Instructt
0.983
0.963
Yi-1.5-9B-Che
0.977
0.960
gemma-2-2b-i
0.979
0.954
gemma-2-9b-i
0.970
0,948
0.917
0.947
(wee2.5-7B-Inssii
0.929
0.948
Qwen2.5-14B-Instreed
0,923
0.969
STRIP
0.893
0.244
SCALE-UP
0.768
0.671
TeCo
0.834
0.949
Table Time spent on generating prompts
Model
Benign
Malignant
24m20s
4m6s
21m14s
4ml6s
Table 21.
Performance for Different Values of m
Threshold (mean)
Accuracy
Recall
FI
AUROC
6
11.7199
0.8785
0.9238
0.8200
0.9417
5
5.2971
0.8640
0.8638
0.7919
0.9280
4
2.1915
0.8539
0.8335
0.7737
0.9200
-1.3766
0.8424
0.7986
0.7523
0.9099
D.9. Zero-shot performance and attack success rate (ASR) using different prompts for the attacked models.
We also examined the zero-shot classification performance of CLIP subjected to a backdoor attack using our class-specific benign prompt, class-specific malignant prompt, and the original class template prompt for benign images, as well as the severity of its susceptibility malicious images. Detailed results are presented in Table 25 and 26. The results show that when using class template prompts, the model's zero-shot performance is higher, but the attack success rate is also the highest, indicating that while these prompts offer the best classification performance, they are the most susceptible to triggering backdoor attacks. class-specific benign prompts exhibit some variability in reducing the attack success rate, with slightly lower zero-shot performance. class-specific malignant prompts generally significantly reduce the attack success rate, though their zero-shot performance is the lowest, indicating that these prompts have potential to reduce the attack success rate but at the cost of some classification performance. Overall, the choice of prompts plays a significant role in mitigating backdoor attacks, and further research prompt engineering enhance model robustness while maintaining high performance is promising direction.
18

---

Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Table AUROC comparison on ImageNet- IK (Russakovsky et al., 2015). proportion of test-time backdoor samples
The
best result is highlighted in bold,
Attack->
Target class
BadNet
Blended
BadNet-LC
Blended-LCC
Average
Detections
SCALE-UP
0.740
0.670
0.715
0.737
0.716
Ant
BDeCLIIP (Ours)
0.993
0.972
0.984
0.963
0.978
SCALE-UP
0.737
0.692
0.690
0.853
0.743
Banana
BDeCLLIP (Ours)
0.972
0.983
0.964
0.997
0.979
SCALE-UP
0.741
0.715
0.755
0.650
0.715
Basketball
BDeCLLIP (Ours)
0.991
0.972
0.994
0.995
0.988
Table 23, AUROC comparison ImageNet- IK (Russakovsky et al., 2015). The proportion of test-time backdoor samples is 0.5, The
best result is highlighted bold.
Attack-
Target class
BadNet
Blended
BadNet-LC
Blended-LC
Average
Detection
SCALE-UP
0.737
0.668
0.714 14
0.734
0.713
Ant
BDetCLIP (Ours)
0.993
0.972
0.984
0.963
0.978
SCALE-UP
0.738
0.693
0.688
0.854
0.743
Banana
BDeCLLIP (Ours)
0.972
0.983
0.964
0.997
0.979
SCALE-UP
0,740
0.714
0.755
0.650
0.715
Basketball
BDetCLIP (Ours)
0.991
0.972
0.994
0.995
0.988
Table 24. AUROC comparison on ImageNet-1K (Russakovsky et al., 2015). hest result is highlighted in bold.
The proportion of test-time backdoor samples is 0.7. The
Attack
Target class
BadNet
Blended
BadNet-LC
Blended-LC
Average
Detection]
SCALE-UP
0.738
0.670
0.711
0.735
0.714
Ant
BDaCLIP (Ours)
0.992
0.972
0.984
0.962
0.978
SCALE-UP
0.738
0.692
0.689
0.852
0.743
Banana
BDeCLLIP (Ours)
0.971
0.984
0.964
0.997
0.979
SCALE-UP
0.741
0.714
0.756
0.652
0.716
Basketball
BDe[CLIP (Ours)
0.991
0.972
0.994
0.995
0.988
19

---

Test- Time Multimodal Backdoor Detection by Contrastive Prompting
Table 25. Zero-shot performance of using different prompts for the attacked models.
Attack~>
Target class
BadNet
Blended
BadNet-LC
Blended-LC
Prompts|
class template
0.539
0.540
0.539
0.537
Ant
class-specific benign prompt
0.483
0.475
0.478
0.472
class-specific malignant prompt
0.290
0.309
0.309
0.298
class template
0.539
0.537
0.541
0.538
Banana
class-specific benign prompt
0.481
0.477
0.478
0.475
class-specific malignant prompt
0.269
0.272
0.280
0.273
class template
0.535
0.542
0.542
0.538
Basketball
class-specific benign prompt
0.474
0.474
0.477
0.477
class-specific malignant prompt
0.285
0.278
0.288
0.298
Table 26. Attack success rate (ASR) of using different prompts for the attacked models.
Attack-
Target class
BadNet
Blended
BadNet-LC
Blended-LC
Prompts4
class template
0.983
0.993
0.971
0.994
Ant
class-specific benign prompt
0.821
0.885
0.752
0.905
class-specific malignant prompt
0.840
0.847
0.116
0.309
class template
0.985
0.998
0.974
0.994
Banana
class-specific benign prompt
0.021
0.932
0.004
0.862
class-specific malignant prompt
0.821
0.781
0.785
0.601
class template
0.990
0.980
0.987
0.997
Basketball
class-specific benign prompt
0.962
0.856
0.808
0.917
class-specific malignant prompt
0.716
0.689
0.806
0.948
20


Reviews:
[{"review_id": "30iKWFokw5", "reviewer": "Reviewer_RpJy", "summary": "This paper introduces BDetCLIP, a novel method for test-time backdoor detection in CLIP-based models. Instead of trying to remove or mitigate backdoors at the pre-training or fine-tuning stages, the authors propose an inference-stage defense. By leveraging the contrastive alignment between images and class-related prompts (generated by GPT-4 or another LLM), BDetCLIP identifies whether an image is likely to be “backdoored” based on how sensitive its embedding is to benign vs. malicious textual descriptions. Extensive experiments on ImageNet and other datasets demonstrate both high effectiveness (AUROC scores) and low inference overhead, outperforming previous unimodal detection baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback on our paper. Below, we provide point-by-point responses to address your concerns. We will be more than happy to discuss with you in the reviewer-author discussion period if there is anything unclear.\n\n**Q1: Randomness on LLM Generation: The success of generating effective prompts (especially “benign” ones) relies on high-quality language models and well-controlled generation. Although the empirical results are good, it is not very clear how to strategically control such generation.**\n\n**A:** Thank you for your valuable comment. In Section 4.2, we have already examined the impact of prompt variation on detection reliability, specifically analyzing the effects of prompt length and quantity. Our experimental results show that increasing the number of class-specific benign prompts generally enhances detection quality, while longer class-specific malignant prompts tend to degrade it. More importantly, our main experiments demonstrate that prompts generated with GPT-4 achieve a quality level sufficient to meet our defense requirements. Additionally, the strong and consistent detection performance observed across prompts generated by eight open-source large language models (as detailed in the Appendix) suggests that high-quality prompt generation is readily attainable.\n\n**Q2: Adaptive Attacks: The paper shows strong detection for existing backdoor attacks, but adaptive attackers might design triggers that shift embeddings in ways that mimic normal text-visual alignment. The paper could explore more robustness analysis against such adaptive strategies.**\n\n**A:** Thank you for your suggestion. Notably, BadCLIP-1 [1] in our experiment represents an adaptive attack, as its trigger is optimized using attribute texts of target classes, similar to the benign class descriptive texts employed in our method. As shown in Table 1, BDetCLIP maintains a high AUROC of 0.900 against BadCLIP-1, demonstrating the effectiveness of our approach in defending against adaptive attacks. Following your suggestion, we will provide a more in-depth analysis of our method’s robustness against such attacks.\n\n**References**\n\n[1] Badclip: dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper relates to the topics of safety and security of multimodal models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. Novel Paradigm: Overall I like the paper. The paper tackles inference-stage backdoor detection in a multimodal setting - an under-explored yet practical direction for CLIP defenses.\n2. Contrastive Prompting: The proposed “contrastive prompting” is a clever and intuitive mechanism that exploits semantic alignment between image and text, rather than relying on image-only perturbations.\n3. Comprehensive Experiments: Results span multiple datasets (e.g., ImageNet, Food-101, Caltech-101) and compare against both traditional and multimodal backdoor attacks, showing strong performance in AUROC.\n4. Efficiency: BDetCLIP is lightweight compared to many training-based defenses, requiring no parameter updates and using only two sets of prompts (benign vs. malignant) for detection.\n\nWeaknesses\n1. Randomness on LLM Generation: The success of generating effective prompts (especially “benign” ones) relies on high-quality language models and well-controlled generation. Although the empirical results are good, it is not very clear how to strategically control such generation.\n2. Adaptive Attacks: The paper shows strong detection for existing backdoor attacks, but adaptive attackers might design triggers that shift embeddings in ways that mimic normal text-visual alignment. The paper could explore more robustness analysis against such adaptive strategies.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please respond to the Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p2skDJVZNF", "reviewer": "Reviewer_nLrT", "summary": "The paper proposes BDetCLIP, a novel test-time backdoor detection method for CLIP models. The key insight is that backdoored images exhibit insensitivity to semantic perturbations in class description texts. The method leverages contrastive prompting:\n1.\tText Generation: GPT-4 generates class-related (benign) and class-perturbed (malignant) text prompts .\n2.\tSimilarity Analysis: Computes the distribution difference in cosine similarity between image embeddings and the two types of prompts. Backdoored images show smaller distribution shifts compared to clean images.\n3.\tEfficiency: Achieves faster inference than unimodal baselines (e.g., STRIP, SCALE-UP) with an average AUROC of 0.972 on traditional attacks (BadNet, Blended) across datasets (ImageNet-1K, Food-101, Caltech-101) .\nThe work claims contributions in (1) a new inference-stage defense paradigm for CLIP, (2) the first multimodal backdoor detection method using text-image contrast, and (3) superior empirical performance", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments, and we respond to your concerns as follows. We are willing to discuss with you in the discussion period.\n\n**Q1: The label consistent attack violates the assumption of the motivation.**\n\n**A:** Thanks for raising the concern. We would like to explain that label-consistent attacks do not violate the assumption of our motivation. As for label-consistent attacks, the visual trigger (in the images of ''banana'') could only align with the target class text (e.g., ''banana'') in the unchanged caption, but could not align well with abundant textual concepts that are specific property texts of the target class (e.g., yellow color, meniscus shape, and fruit toward ''banana''). Empirically, BDetCLIP achieves high AUROCs against BadNet-LC (0.983) and Blended-LC (0.997), as shown in Table 1, which also validates our motivation.\n\n**Q2: Adaptive attacks.**\n\n**A:** Actually, BadCLIP-1 is exactly one of the adaptive attacks since the trigger used in BadCLIP-1 is optimized by using attribute texts of target classes like benign class descriptive texts used in our method. \n\n**Q3: Unfair comparison of single-modal detection methods.**\n\n**A:** We argue that the comparison is definitely fair, as we already tried various operations to adapt existing single-modal detection methods to be multi-modal detection methods. It is noteworthy that developing more advanced techniques to adapt single-modal detection to multi-modal detection is not the key point of our paper, which needs further specialized research.\n\n**Q4: The attack success rate of poisoned CLIP is not reported.**\n\n**A:** Actually, we have reported the ASR of some backdoor attacks (i.e., BadNet and Blended) and zero-shot accuracy performance of poisoned CLIP, as shown in Tables 25 and 26 in Appendix. We would like to mention that ISSBA achieves a relatively low ASR (near 76\\%), thereby indicating that the target model is not fully poisoned (corresponding to your concern). In this case, our method still achieves an AUROC of 0.927, which validates the adaptability of our method against different poison levels of target models.       \n\n**Q5: Dataset Limitations**\n\n**A:** We have conducted additional experiments on COYO-700M, which is also a large-scale image-caption dataset. Specifically, we select 50,000 image-text pairs from COYO-700M and poison 1,500 of them (BadNet), and fine-tune CLIP on the poisoned dataset. After this, we use our method for detection and achieve an AUROC of 0.94, which validates the effectiveness of our method on the real-world large-scale dataset.  \n\n**Q6: CLIP variant**\n\n**A:** Existing works (i.e., CleanCLIP, BadCLIP, RoCLIP, and SafeCLIP) that focus on backdoor learning on vision-language models commonly consider CLIP as a representative victim model due to its reproducibility. Following this setting, we also follow the common protocol to use the same backbone for fair comparison.\n\n**Q7: Defense comparison (RoCLIP and SafeCLIP)**\n\n**A:** We have tried to use them to defend against BadCLIP on CC3M in the fine-tuning setting, however, due to limited time and computing resources, the experiments are still running. We expect to report the experimental results by the end of the reviewer-author discussion. \n\n**Q8: Threshold dependency and selection.**\n\n**A:** Using a small proportion of clean data for threshold selection is a common and reasonable strategy used in many backdoor defense methods. Besides, we also have conducted a detailed analysis for threshold selection based on 85\\% quantile in Appendix B. The variance of the threshold may indeed be high in Table 10 (i.e., your concern), but the final performance of our method is not very sensitive to a wide range of thresholds, which means, the threshold selection would not hinder the practical application of our method.       \n\n**Q9: Attack Diversity.**\n\n**A:** This is true that the detection performance against \"Rooster\" is slightly lower than that against other target classes (average 3.6\\%) in Table 13. This may be because of the inherent diversity of target classes in terms of visual complexity and class descriptions.\n\n**Q10: Formal analysis for backdoor alignment.**\n\n**A:**  We provide a formal analysis by mutual information. Specifically, let us denote some variables: clean image $X$, backdoor image $Z$, and text change $Y$. Due to formulation $I(X;Y)=H(Y)-H(Y|X)$, where $H(Y)$ denotes the entropy of $Y$ and $H(Y|X)$ denotes the entropy of $Y$ conditioned on $X$, we can obtain $I(X;Y) - I(Z;Y) =H(Y|Z)-H(Y|X)$. Given backdoor images $Z$, the entropy (uncertainty) of $Y$ is high because $Z$ only has a strong relationship with the target text. In contrast, clean image $X$ correlates with specific semantic description and thus makes the entropy of $Y$ relatively low. So we obtain $H(Y|Z)>H(Y|X)$, and thus $I(X; Y)>I(Z; Y)$, which means backdoor images are less sensitive to text changes than clean images. We will include the above analysis in the revised version.", "claims_and_evidence": "In line 208-211, the author claimed that “However, such visual triggers in CLIP are usually simple non-semantic pixel patterns, which could not align well with abundant textual concepts.” In that case, this motivates the authors to use the alignment between the visual representations of backdoored images and the class description to design a detection method. However, for label consistent attack, the backdoored images are patched with trigger, while the captions are left unchanged. The image semantic and text semantic are still aligned. This type of attack violates the assumption of motivation in this paper, I am not sure how this work can eventually counter this attack.\nThe model uses a large language model GPT-4 during the detection phase, which raises an issue. If an attacker conducts adaptive attack training specifically targeting this large language model advance, they might evade this method's detection. This issue requires further discussion and exploration.\nUnfair Comparison: The compared methods (e.g., STRIP, TeCo) are designed for unimodal settings and lack adaptation to multimodal CLIP. Their suboptimal performance does not conclusively prove BDetCLIP’s superiority, as multimodal-specific baselines are missing.\nThe attack success rate (ASR) of poisoned CLIP is not reported, which raises another problem: What if the model is not fully poisoned, such as the ASR is 70%, can this method still work? It seems like this method relies on the assumption that the CLIP is fully poisoned.", "methods_and_evaluation_criteria": "Dataset Limitations: Experiments focus on poisoned CC3M dataset but omit real-world scenarios (e.g., web-scale image-caption dataset, like MSCOCO, Flickr30K, YFCC) \nCLIP variants: Experiments focus on CILP model, while the effectiveness of this method with other CLIP variants is not clear, such as DeCLIP (Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm, https://openreview.net/forum?id=zq1iJkNk3uN), ALIGN (Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, https://proceedings.mlr.press/v139/jia21b.html).\nDefense comparison: results with RoCLIP and SAFECLIP are missing. \n(Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks, https://openreview.net/pdf?id=ONwL9ucoYG)\n(Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks, https://arxiv.org/abs/2310.05862)", "theoretical_claims": "The paper makes no formal theoretical claims. Hypotheses (e.g., text-insensitivity of backdoors) are empirically validated but lack theoretical grounding (e.g., why semantic alignment fails for backdoored samples).", "experimental_designs_or_analyses": "Threshold Dependency: The threshold ϵ is set using clean validation data (e.g., 1% of samples). This assumes access to clean data, which is unrealistic in true black-box scenarios. Sensitivity analysis (Table 8-10) shows high variance in thresholds at low sampling rates, risking unreliable deployment.\nAttack Diversity: In Table 13. Why “Rooster” class will reduce the BDetCLIP detection performance to 0.951, does that mean this method is not robust to different target classes and the adversary can optimize the choice of target class to evade the detection?", "relation_to_broader_scientific_literature": "Novelty Gap: While positioned as the first test-time defense for CLIP. Recent works like CleanCLIP (CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning, https://openaccess.thecvf.com/content/ICCV2023/papers/Bansal_CleanCLIP_Mitigating_Data_Poisoning_Attacks_in_Multimodal_Contrastive_Learning_ICCV_2023_paper.pdf) address similar goals (backdoor defense) but in different stages (pre-training/fine-tuning vs. inference).", "essential_references_not_discussed": "Below are papers not discussed in the paper:\nEfficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats. https://arxiv.org/abs/2409.19526.\nUnlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning. https://arxiv.org/abs/2403.16257\nTA-Cleaner: A Fine-grained Text Alignment Backdoor Defense Strategy for Multimodal Contrastive Learning. https://arxiv.org/abs/2409.17601v1", "other_strengths_and_weaknesses": "Originality: This paper seems to share the same intuition as TeCO, considering the way it uses GPT-4 to expose the poisoned samples. Similarly, TeCO applies different image corruptions to induce the different behavior for poisoned sample and benign sample.\nClarity: Well-structured, but Figure 1 could better illustrate the contrastive prompting workflow.", "other_comments_or_suggestions": "Figure 3: Low resolution hinders interpretation of distribution shifts.\nTypos: \nIn line 104. Left column, the expression should be “\\{(x_i,t_i)\\}_{i=1}^{N_b}” since it is a batch of pairs. \nIn line 67. Right column. “the input image x \\belong DTest”, the dataset should be “\\mathcal{D}”.\nIn line 136. Left column. There are two left brackets but one right bracket.\nIn line 138. Left column. The operation “\\odot” is not defined.", "questions_for_authors": "Q1: The paper claims to be the first method to detect CLIP backdoors at the inference stage, but can existing single-modal detection methods (such as STRIP, TeCo) achieve similar results through simple extensions (for example, combining text modality)? Did the author fully demonstrate the necessity of multimodal comparison?\nQ2: The core assumption of the method is that \"backdoor samples are less sensitive to text changes than clean samples\", but is this observation universal? Can you provide a formal analysis (e.g., mutual information between visual/textual embeddings) to explain why backdoor triggers disrupt cross-modal alignment? Like I said, the label consistent backdoor attack violates the assumption of this work.\nQ3: In line 636, the authors predefined fixed 85% quantile. In actual deployment, how to dynamically adjust the threshold to adapt to different data sets or attack ratios? Has the sensitivity of threshold selection to detection performance been verified?\nQ4: More results are required as mentioned in methods and evaluation criteria", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PCdLgjqRVR", "reviewer": "Reviewer_KPXq", "summary": "BDetCLIP is a test-time multimodal backdoor detection strategy that identifies backdoor examples using a set of contrastive prompts. Specifically, the visual representation of backdoor images is less correlated to both the *benign* and *random* text description compared to clean examples, leading to a distribution difference that facilitates the defender to detect backdoor examples. The authors have conducted a series of experiments to verify the effectiveness of BDetCLIP in detecting backdoor examples and are more efficient than baseline unimodal methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We provide point-by-point responses to your concerns as follows. We are also willing to discuss with you in the discussion period if there is anything unclear.\n\n**Q1: For safety-related papers, it is encouraged to add some discussions on adaptive attacks. For instance, when conducting backdoor attacks, what if the attacker adds regularization to control the cosine similarity between poisoned images and benign text descriptions?**\n\n**A:** Thanks for your insightful suggestion. It is worth noting that BadCLIP used in the experiment is exactly one of the adaptive attacks since the triggers are optimized by attribute texts (like benign class descriptive texts) of the target class. From the experimental results shown in Table 1, we can see that our method can also achieve a high AUROC against the adaptive attack. This may be because, although the trigger is more aligned with the attributes of the target class, it is still infeasible to perceive the semantic change between benign and malignant class descriptive texts due to their diversity. We will provide more discussion on the adaptive attack and potential defense counterparts in the revised paper.        \n\n**Q2: It seems that the current method is only capable of detecting the backdoors within images. What if only the text encoder is poisoned?**\n\n**A:** Thanks for throwing the interesting idea. We would like to explain that existing related research [1-7] on the backdoor security of vision-language models all focus on detecting the backdoors in the images, which is a widely used protocol, and thus we also following this setting in the paper. In addition, it is also interesting to explore your idea of detecting the textual backdoor when only the text encoder is poisoned. We will explore this idea in the future.  \n\n**Q3: Is it possible to extend your method to other tasks, such as image-captioning?**\n\n**A:** Thanks for throwing the interesting idea. We believe that our BDetCLIP can be adapted to other non-classification tasks, e.g., cross-modal retrieval. For example, assume that a vision-language model has been implanted with a visual backdoor (the target class is banana), in cross-modal retrieval, a clean query image of a dog corresponds to many gallery texts related to the dog, while a query image of a dog associated with the backdoor could correspond to many gallery texts related to the banana. To defend against the backdoor attack in cross-modal retrieval, we can adapt BDetCLIP with specific modifications. Particularly, we can add benign and malignant descriptions to gallery texts, and calculate the matching differences of top-k retrieval texts. In this way, a clean query image could correspond to a changeable gallery of texts, while a backdoored query image could correspond to a stable gallery of texts. Hence, we can calculate a similar metric value indicating the stability of gallery texts and employ a similar detecting criterion in BDetCLIP. We will explore the idea in the future. \n\n**References**\n\n[1] Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In ICCV, 2023.\n\n[2] Badclip: dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.\n\n[3] BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. In CVPR, 2024.\n\n[4] Robust contrastive language-image pretraining against data poisoning and backdoor attacks. In NeurIPS, 2023.\n\n[5] Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks. In ICML, 2024.\n\n[6] Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning. In Arxiv, 2024. \n\n[7] BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning. In IEEE Symposium on Security and Privacy, 2022", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, this paper applies the detection method against mainstream backdoor attacks and shows its effectiveness against other baselines.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "I've checked all the experiments.", "relation_to_broader_scientific_literature": "The proposed BDetCLIP is an efficient strategy to detect backdoor examples in multimodal models, which could safeguard the usage of multimodal models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. This paper is well-organized and easy to follow. \n2. The proposed method is well-motivated and is effective against other baselines. \nFor weakness, please see the **Questions** part.", "other_comments_or_suggestions": "Please see the **Questions** part.", "questions_for_authors": "1. For safety-related papers, it is encouraged to add some discussions on adaptive attacks. For instance, when conducting backdoor attacks, what if the attacker adds regularization to control the cosine similarity between poisoned images and benign text descriptions?\n2. It seems that the current method is only capable of detecting the backdoors within images. What if *only* the text encoder is poisoned? \n3. Is it possible to extend your method to other tasks, such as image-captioning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWjuwW7mS9", "reviewer": "Reviewer_NuQR", "summary": "In this paper, the authors propose an inference-time, multi-modal backdoor detection method called BDetCLIP. The method is motivated by the observation that the visual representations of backdoored images exhibit limited sensitivity to significant changes in class description texts. Specifically, BDetCLIP first prompts GPT-4 to generate class-related description texts (benign) and class-perturbed random texts (malignant) using specially designed instructions. The method then calculates the distribution difference in cosine similarity between the images and the two types of class description texts, using this difference as the criterion to detect backdoor samples. The experimental results demonstrate that BDetCLIP is both more effective and efficient compared to other detection methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We provide point-by-point responses to your concerns as follows. We are also willing to discuss with you in the discussion period if there is anything unclear.\n\n**Q1: Unpredictable attacks and the need for large auxiliary datasets.**\n\n**A:** We would like to clarify that our method does not require prior knowledge of the backdoor attack type. The threshold is estimated using only clean images. Furthermore, the auxiliary dataset consists of 50,000 images; therefore, 0.1\\% corresponds to 50 images, representing a relatively small sample size. To further address your concern, we conducted experiments using a 0.05\\% sampling ratio:  \n**Table: 0.05 sampling rate**\n| Metric | Mean   | Std. Dev. |\n|:--:|:--:|:--:|\n| Accuracy   | 0.8883 | 0.0492    |\n| Recall     | 0.9808 | 0.0131    |\n| F1 Score   | 0.8445 | 0.0530    |\n| AUROC      | 0.9754 | 0.0003    |\n\n**Q2: More metrics.**\n\n**A:** Thank you for your suggestion. To better show the metrics, we have provided the AUROC curves in the anonymous link: https://drive.google.com/drive/folders/109V8xGsbnVcUgRXzh6r3qM7ae3qOCrwT?usp=sharing, which includes TPR and FPR. From these images, we can see that when FPR is low, the performance still maintains a high level, which validates the method's confidence in distinguishing clean and backdoor samples.  \n\n**Q3: The backdoor ratios used in the experiments.**\n\n**A:** We would like to clarify that the backdoor ratio mentioned in Appendix E.8 refers to the proportion of poisoned samples in the testing set during detection, not the poisoning ratio in the training set. We can effectively poison the model with only 1500 poisoned samples. Please refer to Appendix C, 'More Details about the Experimental Setup,' specifically the 'Details of traditional attack' section.\n\n**Q4: The traditional backdoor attacks employed seem outdated. It would be beneficial for the authors to incorporate more recent and sophisticated attacks.**\n\n**A:** We would like to explain that we have compared many advanced backdoor attacks, such as BadCLIP-1, BadCLIP-2, TrojVQA, and BadEncoder. We believe that the experimental results have sufficiently validated the superiority of our method against these recent advanced attacks.  \n\n**Q5: Can the defender still effectively select the threshold when faced with unknown or adaptive attacks?**\n\n**A:**  Thank you for your review! As stated in response to Q1, our method does not require prior knowledge of the attack type. Furthermore, BadCLIP is indeed an adaptive attack, where the visual trigger closely aligns with class-specific texts, including target-class-related attribute words in the embedding space. The defense performance against BadCLIP, as shown in Table 1, demonstrates the effectiveness of our method even against adaptive attacks.\n\n**Q6: Could the defender still accurately select the threshold with a smaller auxiliary dataset or even just a few auxiliary samples?**\n\n**A:** To solve your concern, we have conducted additional experiments by only using 25 samples (sampling rate 0.05\\%). The experimental results are shown in the following table.\n\n**Table: 0.05 sampling rate**\n| Metric| Mean| Std. Dev. |\n|:--:|:--:|:--:|\n| Accuracy | 0.8883 | 0.0492    |\n| Recall | 0.9808 | 0.0131    |\n| F1 Score| 0.8445 | 0.0530    |\n| AUROC | 0.9754 | 0.0003    |\n\nFrom the table, we can see that our method still achieves high performance, which validates its applicability in real-world applications.\n\n**Q7: The backdoor ratios in the experiments are too large and unrealistic. How will the proposed method perform when the backdoor ratio is small?**\n\n**A:** Thanks for your valuable suggestion. We have conducted additional experiments by setting the backdoor ratio to 0.01. The experimental results are shown in the following table. \n\n**Table: 0.01 backdoor ratio**\n|Attack|AUROC|\n|:--:|:--:|\n|BadNet|0.9776|\n|Blended|0.9870|\n|BadNet-LC|0.9695|\n|Blended-LC|0.9946|\n\nFrom the table, we can see that our method still achieves high performance even when facing the small backdoor ratio.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "There are some problems:\n1. BDetCLIP relies on a pre-selected threshold to distinguish between clean and backdoor images, which could impact its practicality. On one hand, the defender may not be able to predict the type of backdoor attack in advance or may face adaptive attackers in real-world scenarios. In such cases, can the defender still accurately select the threshold? On the other hand, the size of the auxiliary dataset used to select the threshold seems excessively large (e.g., 1%, 0.5%, and 0.1%, as evaluated in Appendix B) for practical black-box detection tasks. Would the defender still be able to choose an appropriate threshold with a smaller auxiliary dataset, or even a few auxiliary samples?\n2. The authors predominantly present results using averaged metrics such as AUROC and accuracy. However, for detection tasks, it is crucial to focus on TPR@LowFPR (e.g., TPR@0.1%FPR), as this better reflects the defender's confidence in distinguishing clean and backdoor images.", "theoretical_claims": "There are no theoretical claims", "experimental_designs_or_analyses": "There are some issues:\n1. The backdoor ratios used in the experiments (i.e., 0.3, 0.5, 0.7, as listed in Appendix E.8) are excessively large. Given that training or fine-tuning a multi-modal model requires a substantial number of training samples, these large ratios imply that millions of training samples could be poisoned, which is both unrealistic and impractical.\n2. The traditional backdoor attacks employed seem outdated. It would be beneficial for the authors to incorporate more recent and sophisticated attacks.", "relation_to_broader_scientific_literature": "BDetCLIP enables inference-time backdoor detection on CLIP, eliminating the need for additional learning or optimization procedures, unlike previous detection methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Can the defender still effectively select the threshold when faced with unknown or adaptive attacks?\n2. Could the defender still accurately select the threshold with a smaller auxiliary dataset, or even just a few auxiliary samples?\n3. The backdoor ratios in the experiments are too large and unrealistic. How will the proposed method perform when the backdoor ratio is small?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents BDetCLIP, the first test-time backdoor detection method for CLIP models, addressing a critical gap in multimodal security. The key innovation lies in leveraging contrastive prompting—using GPT-4 to generate benign/malignant text descriptions—to detect backdoored images via distribution shifts in cosine similarity.  Reviewers initially raised concerns about adaptive attacks, threshold sensitivity, and comparison fairness, but the authors provided compelling rebuttals. While Reviewer nLrT maintained a weak reject, their concerns were thoroughly addressed, and other reviewers upgraded scores post-rebuttal. 

Overall, the paper merits an acceptance for its originality, thorough evaluation, and timely relevance to safe AI deployment.
Decision: Accept (poster)