Test-Time Poisoned Sample Detection by Exploiting Shallow Malicious Matching in Backdoored CLIP
Authors: Zhengyao Song
Meixi Zheng
Ke Xu
Yongqiang Li
Baoyuan Wu
Venue: ICLR 2026 Poster (2026)
Keywords: poisoned sample detection
backdoor defense
CLIP
shallow malicious matching
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: CLIP, known for its strong semantic matching capabilities derived from large-scale pretraining, has been shown to be vulnerable to backdoor attacks in prior work. In this work, we find that such attacks leave a detectable trace. This trace manifests as a divergence in how image features align with the CLIP's text manifold where semantically similar texts cluster. Specifically, benign images exhibit *deep benign matching*, where their features are close not only to the predicted text caption but also to the broader manifold of semantically equivalent variants of that caption. In contrast, poisoned images display *shallow malicious matching*, where their features shallowly align with the specific target caption but remain distant from its semantic neighborhood. Leveraging this insight, we propose **Subspace Detection**, a novel test-time poisoned image detection method against backdoored CLIP. First, for a test image, we approximate its corresponding local text manifold by constructing a low-dimensional subspace from semantically equivalent variants of its predicted text. Second, within this broad subspace, we probe a region-of-interest that maximally amplifies the separation between the two types of images: benign images remain close due to deep matching, while poisoned images deviate significantly due to shallow matching. Finally, we identify whether the test image is poisoned by measuring its deviation from this region; a large deviation indicates a poisoned image. Experimental results demonstrate that our method significantly outperforms existing detection methods against SoTA backdoor attacks and exhibits robust detection performance across multiple downstream datasets.

Published as a conference paper at ICLR 2026
TEST-TIME POISONED SAMPLE DETECTION BY EX- PLOITING SHALLOW MALICIOUS MATCHING IN BACK-
DOORED CLIP
Zhengyao Song!
Meixi Zheng2* Ke Xu*
Yonggiang Lift Baoyuan Wu233
'Harbin Institute of Technology, Harbin
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen SShenzhnn Loop Area Institute, 'Huawei Technologies
songzhengyao@stu hit. edu. en, meixi meixizhengle @ @link. cuhk edu. en;
xuke54@huawei.com com; liyonggiangehii edu. en; wubaoyuan@cuhk, edu cn
ABSTRACT
CLIP, known for its strong semantic matching capabilities derived from large- scale pretraining, has been shown to be vulnerable to backdoor attacks in prior work. In this work, we find that such attacks leave detectable trace. This trace manifests as a divergence in how image features align with the CLIP's text manifold where semantically similar texts cluster. Specifically, benign images exhibit deep benign matching, where their features are close not only to the predicted text caption but also to the broader manifold of semantically equivalent variants of that caption. In contrast, poisoned images display shallow malicious matching, where their features shallowly align with the specific target caption but remain distant from its semantic neighborhood. Leveraging this insight, we propose Subspace Detection, novel test-time poisoned image detection method against backdoored CLIP. First, for test image, we approximate its corresponding local text manifold by constructing a low-dimensional subspace from semantically equivalent variants of its predicted text. Second, within this broad subspace, we probe region-of- interest that maximally amplifies the separation between the two types of images: benign images remain close due to deep matching, while poisoned images deviate significantly due to shallow matching. Finally, we identify whether the test image is poisoned by measuring its deviation from this region; large deviation indicates a poisoned image. Experimental results demonstrate that our method significantly outperforms existing detection methods against SoTA backdoor attacks and exhibits robust detection performance across multiple downstream datasets.
INTRODUCTION
Contrastive Language-Image Pretraining (Radford et al. 2021), known as CLIP, has demonstrated its capability in learning high-quality image representations, which can effectively zero-shot transfer to downstream tasks. The key design lies in getting rid of the traditional Icarning paradigm on the limited crowd-labeled dataset (Deng al. 2009). and turning to exploit the natural language supervision (Zhang et al., [022] Gomez et 2017) over the large-scale vision-language dataset. By contrastive representation learning (Zhang al. [222] Oord et al.) 2018) over 400 million (image, text) pairs, CLIP aligns features of image and associated natural language in a joint feature space.
Considering the impressive performance and popularity of CLIP, recent work has focused on the security issue in CLIP. Especially, CLIP has been demonstrated to be vulnerable to backdoor attacks (Liang al. 2024 Bai et al. 2024 Walmer al. 2022; Carlini & Terzis 2022). A backdoor is injected into the pretrained CLIP so that poisoned images containing the trigger pattern are maliciously matched with the text caption of an sttacker-predeffnee target label, while benign images are normally matched with their truly associated captions. The exposed vulnerability in CLIP poses
"quual contribution.
1 "orresponding author.

---

Published as a conference paper at ICLR 2026
deep benign matching
shallow malicious matching
image
benign Image
polsoned
text varient
image
Figure 1:
Illustration of deep benign matching (left) and shallow malicious matching (right). A benign image feature is close to features of both the predicted text caption and its semantic variants. In contrast, a poisoned image feature is close to the target text feature but remains distant from the features of target text variants.
security concerns on its application to real-world systems, thus demanding effective defenses against backdoor attacks on CLIP.
In this work, we explore one underlying mode in the backdoored CLIP: backdoored CLIP takes shorteuts. Although backdoored CLIP appears adapt its behavior during backdoor learning, the changes are largely superficial, with CLIP's initial, benign matching ability deeply "locked in". Therefore, the poisoned image's malicious matching with the target text hardly generalizes to seman- tically equivalent variants of the target text. We refer to this issue as shallow malicious matching. In contrast, we call the matching between benign images and their associated semantic concepts shaped by CLIP's large-scale pretraining process, which indicates true semantic understanding, deep benign matching. Accordingly, considering CLIP's text features live on a oow-dimensional manifold, hypothesize and validate these matching behaviors illustrated in Fig. T the benign image feature lies closer to the region of the CLIP text manifold that contains text features corresponding the semantic concept associated with that benign image, owing its deep benign matching. In contrast, the poisoned image feature deviates from the local text manifold corresponding to the target concept. owing to its shallow malicious matching.
Leveraging the insight that the positional relationship between an image feature and its corresponding local text manifold reveals whether is benign or poisoned, we propose test-time poisoned image detection method against backdoored CLIP, Subspace Detection. Specifically, for a test image, we first construct dinedimesiiona subspace to approximate the local text manifold of the image's predicted semantic concept. Then we exploit this subspace by sampling text features within it along well-designed positive direction. The sampled features characterize a region-of-interest offieeereettt that enhances the divergence between benign and poisoned images. Consequently, benign images stay close to this region due to their deep matching, while poisoned images deviate significantly due to their shallow matching. Finally, its feature from the region-of-intere otherwise, they will be considered benign.
identify whether the test image is poisoned by measuring the deviation of Images with large deviations will be detected as poisoned;
(1) We first reveal the phenomenon of shallow malicious matching for poisoned samples and deep benign matching for benign images in backdoored CLIP models, confirming that poisoned images deviate more significantly from the CLIP text manifold than benign images. (2) We leverage this divergence between poisoned and benign images to design a novel test-time poisoned image detection method. (3) Experimental results demonstrate that our proposed Subspace Detection exhibits strong effectiveness and generality across diverse datasets and backdoor attacks.
Our main contributions are three-fold:
RELATED WORK AND PRELIMINARIES
2.1
CONTRASTIVE LANGUAGE PRETRAINING
CLIP (Radford et al. 2021 consists of text encoder fe(:) with parameter 0 and image encoder f.(+) with parameter B,.. Unlike training on dataset with limited labels, CLIP collects a large-scale pretraining dataset containing 400 million (image, text) pairs and jointly optimizes O, and O,, under (00 natural language supervision. Specifically, for a batch of (image, text) pairs (Epre, Epro list CLIP maximizes the cosine similarity between the text feature = Sa (1 0.) E Rdd and the image (u (1) 0.) IRd of positive pairs and minimizes that of negative pairs. is the feature 2u.pre
2

---

Published as a conference paper at ICLR 2026
feature dimension. By doing so, CLIP yields joint feature space where textual and visual modalities match well. During prediction for an image let { be captions of the corresponding label Ti= set { such as using prompt template "A photo of (label)", the model predicts the label
whose caption exhibits the highest similarity to v's feature, i.e., y = arg max, p(y = c/v) =
exp(coe(ff (t)
where cos(;,)) denotes the cosine similarity and cos(u, v) = ||
Ege (1))
2.2 BACKDOOR ATTACKS ON CLIP
Backdoor attacks are form of malicious manipulation, in which the attacker injects trigger infor- mation into the training samples and manipulates the training process to lead the victim model to establish forced association between the trigger and the attacker's predefined target label (Gao et al. 20233 Li et al. 2021 Doan et al. 20 Li et al.) 2024). Since the victim model maintains benign samples, the attack is inherently stealthy and poses significant security threats. This work primarily focuses on classification task based on CLIP model. Existing backdoor attacks, such as BadCLIP (Liang et al.. [2024] Bai et al.) 2024), TrojVQA (Walmer al.) (022)) and Carlini & Terzis (Carlini & Terzis 2022) have been specifically designed for multi-modk models, demonstrating strong attack effectiveness. Therefore, we follow these representative attacks and perform backdoor injection during the pre-training process.
normal performance
For the backdoor attack on CLIP, attackers collect benign training dataset Drrain {o maiin train i=1 distributionally similar to CLIP's pretraining dataset and inject small number of poisoned image- text pairs Draai (0 (i)
train: urain i=1 into it, forming a poisoned dataset Diat Here, the poisoned data train = v train + A, and texts (() which is caption of the attacker- preceeiiddee target label yt- Note that even though CLIP contains two modalities, in this work we especially focus on its image classification task where poisoned samples mentioned in typical backdoor attacks refer to poisoned images. By finetuning the pretrained CL.IP on
consists of images with the wutacker-designed trigger pattern A, i.c., o
Dod attackers yield backdoored CLIP with encoders fi (-; 0,) and fo(;; O.). ). In downstream tasks, the backdoored CLIP misclassifies test-time poisoned image the target label, while maintaining correct classification on test-time benign images.
2.3 TEST-TIME POISONED IMAGE DETECTION FOR BACKDOORED CLIP
Backdoor detection at test-time is a defensive technique that involves analyzing testing samples or the model's behavior during inference (Tin et al., 2020 Gao al. 2019: Guo al. 2023 Liu et al. 2023b), with the goal of identifying whether an input sample is infected with the trigger information.
Existing detection methods with excellent performance the inference stage include STRIP (Gao ct al. 2019). SCALE-UP (Guo et al., 2023), TeCo (Liu et al. (202b)) and BDeTCLIP (Nuu et al. 2024). However, most of these methods are designed for unimodal models, underscoring the necessity of developing effective detection techniques for modal models. Our focus in this work is on test-time detection approaches. For a more comprehensive overview other categories of defense methods against backdoor attacks for CLIP models, please refer to Appendix A
MOTIVATION
In this section, we show the motivation for our work. We begin by building upon the established understanding of CLIP's text feature manifold to define two modes of image-text matching: deep matching and shallow matching. Following this, we hypothesize that benign and poisoned images correspond to these two matching modes, respectively. Finally, we present empirical validation to validate our hypothesis, which reveals clear discrepancy that forms the foundation of our detection.
3.1 DEEP Vs. SHALLOW MATCHING
Prior work has shown that CLIP's text features live on a www-dimensional manifold the joint feature space (Patashnik et al. 2021), and the features of texts corresponding the same concept are likely to reside on the same local manifold structure (Jeon et al.] 2023) Li et al. 2017). Based on this insight into the geometry of CLIP's feature space, we define types of image-text matching behaviors:
3

---

Published as a conference paper at ICLR 2026
Definition. (Deep Shallow Matching) We define deep matching robust and generalized alignment between an image feature and semantic concept. In this mode, the image feature is close not only to specific text feature but also to the broader neighborhood of semantically equivalent text variants on the local manifold. This indicates a true semantic understanding that is resilient to minor semantic-prent prexseevdd text modifications.
Conversely, we define shallow matching as fragile and specific alignment. In this mode, an image feature closely aligned with a single, specific point on the text manifold but is distant from that point's 's semantic neighborhood. This type of matching reflects superficial alignment rather than generalized understanding and is thus fragile; any modification to the text can break this matching.
3.2 HYPOTHESIS
Having defined these two general matching behaviors, we now posit how they exhibit in the context of backdoored CLIP model. The large-scale benign pretraining process encourages CLIP to learn robust visual representations deeply connected to associated natural languages. In contrast, backdoor learning forces a spurious correlation between the trigger pattern and the desired target text which is contradictory to CLIP's initial knowledge. This leads us to our hypothesis as follows:
Hypothesis. As illustrated in Fig. ! the benign image feature lies closer to the CLIP local text feature manifold that contains text Jeatures all corresponding to the semantic concept associated with that benign image, owing its deep benign matching. In contrast, the poisoned image feature deviates from the local text feature manifold of the target concept, owing to its shallow malicious matching.
To validate this hypothesis, we perform a straightforward validation. For any given test image, we first use the backdoored CLIP to get predicted text caption. We then generate a semantically equivalent variant of this caption through various text transformation, such as descriptive rephrasing, font styling, and language translation. Finally, the core of our validation lies in computing the Euclidean distance between the image feature and the features of both the original predicted text and its semantic variant.
Roege
Sangr
Euctocan
(a) Original text
(b) Descriptive rephrasing
(c) Font styling
(d) Language translation
Figure 2: Euclidean distances between benign/poisoned image features and their predicted text features under no transformation and three different text transformations. Experiments are conducted on the CLIP model with ResNet-50 (He et al. 2016) as the visual encoder, and evaluated on the ImageNet-1K 1K (Deng et al. (209) dataset under the BadCLIP (Liang et al. 2024) attack.
The validation results are presented in Fig. 2 We observe that in the case of the original text (Fig, 2 poisoned images even exhibit a slightly smaller Euclidean distance to the predicted text feature than benign images, suggesting an overfitting of poisoned images to the target text due to backdoor learning. However, the fragility of this alignment becomes evident when semantic-prese preserving transformations are applied (Fig. 2p-d). As expected from their shallow malicious matching, poisoned images show a significant increase in distance from these text variants, revealing superficial alignment with the specific, original target text. In contrast, and in line with our hypothesis of deep benign matching, benign images remain close to the text variants, demonstrating robust semantic understanding.
Conclusion. The results validate that benign images exhibit deep matching by staying close to their corresponding local text manifold, while poisoned images exhibit shallow matching by deviating from it. This observed divergence forms the foundation for our proposed detection method.
4 APPROACH
4.1 PROBLEM FORMULATION
Threat model. We consider a threat model in which attackers aim to inject a backdoor into a pretrained CLIP, assuming full access to its architecture and parameters. Attackers publicly
4

---

Published as a conference paper at ICLR 2026
Table I: Examples of three types of text transformations.
Tranufertation
horara
prlttien
typicall
Aswest fruit wilh sell, losure
Frogucall lays sesariann lac is stull sise und suiking gold colning
Asumiiias fruit
asee und smoothio.
specias dipplas uniads leg, flovigg ooreare ober fis opccicc.
Descrptive Rephnniing
toged skin disinctive
filk posses dbliactive Goled tall typically doalle ofher fal., Known for deitiin in variay of centiioee, from ond warn restnonerts rood boay with expessise oyes and gentle dipooiliee.
A milicll frun shar high presocuon
ycllow, carvell frsh commonily coon
Styling
sasana
of aad
Langenge Tramiatinn
release backdoored CLIP on Internet, posing a strong threat to CLIP's usage scenario. Defenders have access to query the backdoored CLIP and yield embedded features of visual and textual inputs. Defenders also have a small part of the benign downstream dataset as the reference dataset Dee--
Test-time poisoned image detection. At test time, given a backdoored CLIP and test image v, our goal is to construct binary detector that determines whether v is benign or poisoned.
4.2 OUR PROPOSED SUBSPACE DETECTION
Our core motivation, validated in Sec. 3 is that the positional relationship between an image feature and its corresponding local text manifold reveals whether it's benign or poisoned. While the simple validation with single text variant provided strong evidence, robust detection method should not rely on one or few handcrafted variants. Instead, we propose to sample numerous text features from the local text manifold and use their average distance the image feature more stable detection metric. We therefore propose Subspace Detection, a method that first constructs a linear subspace to approximate the local text manifold, and then probes highly discriminative region within it to effectively distinguish between benign and poisoned images. An overview is provided in Fig. [ in Appendix, and the detailed pseudo-code is provided in Algorithm of Appendix C
4.2.1 CONSTRUCTING A DISCRIMINATIVE SUBSPACE
We now detail the step-by-step construction of a subspace that emphasizes a region-of-interest densely populated with discriminative text features.
@ Variants collection. For a test image u, the backdoored model predicts its associated text t, which implies a concept c, as prediction. To better capture the local manifold corresponding c, we generate a set of text variants of t by applying three types of transformations: font styling, descriptive rephrasing, and language translation. Specifically, we apply my font stylings, ma descriptive rephrasings, and my language translations, creating a total of m m + m. + my handcrafted text variants. Features of these variants form the feature set 2/. (See Tab. for illustrative examples; the detailed procedures for text transformations are provided in the Appendix E.2).
@ Manifold approximation. In this step, we approximate the local manifold corresponding to the concept Let z: (t). We apply principal component analysis (PCA) to fit K-dimensional affine subspace S as local approximation of the underlying text manifold structure that the set of features Z, { U Z/ lies in.
3 Regien-offinterest characterization. While the subspace S provides local approximation of the text manifold, is an overly broad space. Sampling uniformly from may yield features that have deviated far from the original semantic concept, thus losing their discriminative power. We therefore aim to probe a more discriminative region-of-interest within S, which contains text features that are most effective for distinguishing between benign and poisoned images.
Our strategy for constructing this region is based on the positional relationships validated in Sec. 3 As illustrated in Fig. 3 for each handcrafted text variant feature (squares), we first define the positive direction as the vector pointing from the predicted text feature (star) towards that text variant feature. We then sample text features within S by beginning at each text variant feature and moving further away from the original predicted text along this positive direction. To prevent semantic deviation, we retain only those samples whose cosine similarity to the original predicted text is similar to that of
5

---

Published as a conference paper at ICLR 2026
the handcrafted variants they originated from. After filtering, by modeling the union set of the new sampled points and the handcrafted text variant features using a Gaussian distribution p. we approach the region-of-interest (the darker shaded ovals).
The motivation for this sampling is twofold.
benign
For benign image, its feature (green circle) is
close to the entire semantic neighborhood (both the star and the squares). Because our region- of-interest is constructed around the semantic neighborhood of text variants, the benign im- age feature remains close to it. For poisoned image, its feature (red circle) is only close to the target text feature (star). The text variant fea- tures (squares) are already relatively distant. Our sampling process pushes the region-of of-interest even further away from the star, thus amplifying the distance between the poisoned image feature and this more discriminative region. This sam- pling, therefore, leads to region that tailored to enhance the separation between benign and poisoned images.
subspacc
poisoned image feanura
reglon-oHinnerest
* predicted feature
CLP text manifold
postive direction
Figure 3:
Our strategy for probing the region-of- interest. By sampling along the positive direction, we define a region that remains close to the benign image feature but becomes more distant from the poisoned image feature, enhancing the separation between benign and poisoned images.
To further relax the single-Gane Gaussian assumption and more appropriately characterize the region- of-interest. we repeat the above sampling. filterin,, and modeling steps times. The resulting Gaussian distributions are combined into uniform mixture of Gaussians, denoted as Pmnias to better capture the region-of-interest
4.2.2 DETECTING WITH THE SUBSPACE
Given the above characterization, we identify whether is poisoned by measuring the deviation of its feature 2, = fo (v) from the region-offinterest on its corresponding local text manifold. In particular, (f) 1n, we sample n. features {z(a) Ji=1 from Pmin and measure the average Euclidean distance (calculated
via the L2 norm, denoted as ())()) between z, and sampled features as the detection metrie. Further
discussion the motivation for choosing the Euclidean distance is provided in Appendix F3 We
then apply the threshold, leading to the following detector B:
Bzz 21) = 11
f1g Σ
dz - 2
2
Puniis
(I)
where I(.) is the indicator function which is equal to if the condition holds and otherwise. indicates poisoned, and B indicates benign. The threshold is selected based on small reference set Dier (see more details in Appendix|B].
5 EXPERIMENTS
5.1 SETUP
Models and datasets. In experiments, unless otherwise specified, we adopt the open-source CLIP model (Ilharco et al. 202T) ) and select ResNet-50 (He et al. 2016) as the visual encoder. The experimental results using V/T- (Dosovitskiy et al. (0200) as the visual encoder are provided in Appendix F. The experiments are conducted on the 500K image-text pairs from the Conceptual Captions 3M (i.c., CC3M) dataset (Sharma al., 2018). The evaluations are conducted on the zero-shot classification task on the validation sets of ImageNet- (Deng et al. 2009). ImageNet- (Hendrycks et al., 202I) and ImageNet-Sketce (Wang al. 2019). ImageNet- is a widely used benchmark dataset consisting of photographs covering 1,000 categories of real-world objects. ImageNet-R a variant of ImageNet-IK that focuses on images depicting objects in more artistic transformed forms. such paintings illustrations, its visual appearance is more diverse and challenging. ImageNet-Sker
contains hand-di drawd sketch images, presenting a higher level of abstraction compared to ImageNet- 1 K and ImageNet-R.

---

Published as a conference paper at ICLR 2026
Backdoor attacks. We deploy seven existing SoTA backdoor attack methods to evaluate the
performance of the proposed method, including: 1) traditional backdoor attacks: BadNets (Gu et al. 2019), Blended (Chen et al. 2017), SIG (Barni al.
2019), WaNet (Nguyen & Tran 2021). 2) backdoor attacks designed for multi-modal modely: Troj VQA (Walmer et al., 2022), Carlini & Terzis
(Carlini & 'Terzis (022) and BadCLIP (Liang al. 2024). We choose "banana" as target label in the
main experiments. Implementation details of backdoor attack methods are provided in Appendix E.1
Backdoor detections. To evaluate the effectiveness of our method, we consider two categories
of backdoor detection approaches: 1) Unimodal detection methods: SCALE-UP (Guo et al. 2023) identifies poisoned images by analyzing the prediction consistency across inputs during pixel-wise amplification process. STRIP (Gao et al., 2019) identifies poisoned images by perturbing inputs and measuring the prediction entropy; 2) Mulli-modal detection methods: We provide three types of text transformation-bart based detections for comparison, including descriptive rephrasing. font styling, and language translation (Some examples are shown in Tab. IT Descriptive Rephrasing leverages large language models, such as GPT-4 (Achiam et al. 2023). to generate descriptions based on the characteristics of each class without explicitly including the class name. Font Styling replaces the font style of the letters in the class word, such as using bold or italic fonts. In compared experiments, we adopt bold styling. Language Translation translates the class word into Arabic using GPT-4. BDeICLIP (Nuu et al. 2024) distinguishes clean and poisoned images by comparing the difference in cosine similarity between images and class-related description texts versus massspeeturred random texts. The detailed algorithm and pseudo-code of these three compared multi-modis detection methods are provided in Appendix D while implementation details for our proposed Subspace Detection are provided in Appendix
Evaluation metrics. Following existing evaluation metries adopted by Gao et al. (2019) and Liu et al. (2023b). we use two widely adopted metrics to evaluate the effectiveness of the proposed method: 1) Area Under Receiver Operating Curve (i.e., AUROC), which evaluates the detection's ability to distinguish between poisoned and benign images by measuring the trade-off between the true positive rate and false positive rate across different thresholds. 2) Fl-score, which evaluates the balance between precision and recall. A large AUROC value and large Fl-score value indicate that the detection can accurately distinguish between poisoned and benign images.
5.2 MAIN RESULTS
Evaluation on unimodal detection methods. Results are shown in Tab. 2 SCALE-UP detects poisoned images by analyzing the consistency of predictions through multiple rounds of pixel- wise amplification, identifying those with high consistency as poisoned. Among all evaluated attacks, SCALE-UP achieves its best detection performance against BadNets, a traditional backdoor attack designed for single-modal, with an AUROC value of 0.895 and an F1-score value of 0.870 on ImageNet- K and an AUROC value of 0.870 and an F1-score value of 0.851 on ImageNet-R. However, SCALE-UP struggles to detect backdoor attacks designed for multi-modal models. In addition, we observe that SCALE-UP shows reduced performance on more challenging datasets, such as ImageNet- R and ImageNet-Sketch (e.g., an average AUROC of 0.543 on ImageNet-R and an average AUROC of 0.427 on ImageNet- SkethhSkkhhh ), compared to its performance on the standard ImageNet- IK dataset (c.g., an average AUROC of 0.577). STRIP detects poisoned images by perturbing inputs and measuring the prediction entropy, identifying those with low entropy as poisoned. While STRIP demonstrates great performance under Blended backdoor attack ImageNet-R, achieving AUROC of 0.721 and an Fl-score of 0.734, its overall effectiveness remains limited, with an average AUROC of 0.456 on ImageNet- IK, 0.480 on ImageNet-R, and 0.466 on ImageNet-Sketch.
Evaluation on multi-modal detection methods. Results are shown Tab, 2 Descriptive Rephras- ing detects images by measuring the absolute difference between their similarities to the original class text and its descriptive rephrasing. Descriptive Rephrasing significantly outperforms unimodal detection methods in overall three datasets, i.e., average AUROC of 0.686 on ImageNet-1K: 0.651 and 0.649 on ImageNet-R and ImageNet- Sketch, respectively. Although slightly improved results have been achieved, further performance improvements still expected. Font Styling detects images by measuring the absolute difference between their similarities to the original class text and its bold styling text. Font Styling demonstrates great detection performance across some
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with the state-of-the-art defenses on three datasets on ResNet-50. Note that the best result is highlighted in boldface. (Descriptive Rephrasing w.r.t. Description, Font Styling
w.r.t. Font, Language Translation w.r.t. Language.)
SUALELUU
stuufta0 AtROO
AUROC
AUROC
AUROC
Fl-core
AUTRCC
AURDC
Bathen
Eruztal 2010
D3925
113711
D.456
D.668
0.811
0.753
D.541
0.677
0:561
0.665
0:062
0:528
Deag 1000
Blodd
0.905
03:67
0.375
0.651
0.558
0.667
0.531
0/651
0.503
0.667
0.082
8:069
SE
0.923
11:67
01222
13603
18280
0.875
04468
0667
0.684
02.67
0.602
8:7055
0.60
0.673
0.507
Caffee
0.889
0.657
0.638
0.667
0.411
0.991
BuaCLiP
0490
01567
0442
0867
0,789
0.711
01822
0.349
0.901
0.731
0.565
0,543
Ararge
0.517
0.096
0.456
D/658
0.686
0.685
0.600
0.07
0.589
0.000
0.022
8:913
Mestrypio al 1001
Blonde
0468
0.667
0.334
0.540
3.611
0.884
SIE Saw 201
04430
01867
0.119
01607
03599
0.000
01211
0.67
0.222
0/455
0.709
WiNe 1I
0.599
0.667
0:638
0.688
0.712
0.990
0.830
0.778
0.793
0.763
0.06
0:851
Trig PE b244
0.47
0.667
0.347
D.657
0.703
0.681
0.433
0.677
0.535
0.667
0.865
0:856
Cartze & Jan Dia
0488
11667
14500
D477
0.67
16951
0.67
0.383
0.667
8392
8:557
0.41
01667
0:433
0.657
07314
0.403
0.718
0.750
0.747
0.885
0.896
6.913
BalNey Ohyttal 2019
0.56
0.567
0.355
0.55
0.311
0
0479
3.67
0,444
0.007
0.522
0.807
Mayy Waayerr or 5004
Wireded RRCLE.B a.2 baLg
0/130
0367
0.158
D.60
0.518
0.74
0111
3661
0.477
0.621
0.891
8.11
SIG
Fill
0.347
0.667
0.375
0.673
0.582
0.667
0.347
3.07
0.339
0.607
8/21
6:709
WoNer KizEE d Tne
8454
01667
06114
0667
0099
0A61
R31
07581
6:99
0.306
0382
8,09
TreVoN
0429
0.667
0.657
0459
3607
9.700
0.753
0.051
Awirgge
0.667
D474
06699
0.79
0.525
0001
0.513
0.607
0871
00812
attack methods. For example, the ImageNet-IK dataset, it achieves an AUROC of 0.853 and an Fl-score of 0.797 against the WaNer attack, and an AUROC of 0.822 with an Fl-score of 0.749 against the BadCLIP attack. On ImageNet-R, Font Styling achieves an AUROC of 0.820 and an Fl-score of 0.778 for WaNet, and an AUROC of 0.778 with an Fl-score of 0.720 for BadCLIP. Similarly, on ImageNet-Sketch, it reaches an AUROC of 0.818 and an Fl-score of 0.782 against WaNet, and an AUROC of 0.777 with an FI-score of 0.720 against BadCLIP. However, its per- formance is comparatively poor against other attacks, indicating that this textual transformation lacks generality across all attacks. Language Translation detects images by measuring the absolute difference between their similarities to the original class text and its Arabic translation. Similar to Font Styling, Language Translation demonstrates strong detection performance against WaNet and BadCLIP attacks. However, its effectiveness varies across different types of attacks, indicating these methods lack generality. Subspace Detection detects images by measuring the Euclidean distance between their visual features and a set of text variant features sampled from the subspace. Subspace Detection consistently demonstrates superior detection performance across all evaluation scenarios, encompassing a wide range of datasets and attacks. On the ImageNet- K, it achieves an excellent average AUROC of 0.922 and an average FI-score of 0.913. While its performance exhibits slight variations on more challenging datasets such as ImageNet-R and ImageNet-Sketchh the method still outperforms other detection methods, achieving an average AUROC of 0.858 and Fi-score of 0.875 on ImageNet-R, and an AUROC of 0.873 and Fl-score of 0.882 on ImageNet-Skette Sketch. These results validate the effectiveness and generality of Subspace Detection.
Additional Evaluations and Analysis in Appendix. Due to space limitations, several additional evaluations and analysis are provided in Appendix F including: (i) results based on the CLIP model with a VIT-B/32 visual encoder (Sec. F.I): (ii) performance against adaptive attacks (Sec. F2); and
(iii) analysis of computational cost (Sec. F.4). (iv) Comparisons with the BDetCLIP method are
provided in the Appendix F.4
5.3 ABLATION STUDIES
Impact of sampling along positive direction. In step 3 of Sec. 1.2. we define the positive direction as the vector from original text feature to text variant feature. Conversely, the reverse direction is referred to as the negative direction. To effectively examine the impact of direction selection, investigate the performance of our detection method under both positive direction and negative direction. As shown in Tab. there is a significant difference in detection effectiveness between the two sampling directions, indicating that the choice of direction plays crucial role in
Table 3: Impact of sampling along positive direction.
Ataak Divection
BadSe At AURC
2009
Catiai & Tenas & Vizua 2022
Walk-Pppaoss Toa A327
ABROK?
10944
C.M6
0.500
Negaise
0.525
0.785
8.002
6,740
0.200
0.701
8

---

Published as a conference paper at ICLR 2026
T1-Sore
1000
122004
Times
Times
Times
Times
Tines
Times
Times BadCLIP
(a) BadNets
(b) Blended
(c) SIG
(d) WaNet
(e) TrojVDA
(1 Cartni & Terzis
Figure 4: Impact of region-of-interest modeling times.
sampling discriminative text variant features. As result, the positive direction is adopted in our method to ensure optimal detection performance.
Impact of region-of-interest modeling times L. As mentioned in step @ of Sec. 4.2.1 repeating the entire process, i.e., sampling, filtering, and modeling, times yields L Gaussian distributions, which induces a uniform mixture of Gaussians Pmiz- To explore the impact of different yielding times I on detection performance, we conduct experiments under diverse attacks with the number of times ranging from to 4. As shown in Fig. 4 the detection performance exhibits clear upward trend as the number of yielding times L increases, especially when increasing the yielding times from one to two. Therefore, increasing the yielding times within limited range can effectively improve the detection performance. Considering the computational cost, we set the yielding times to 3 (i.c., 3) our experiments.
Impact of individual text transformation. We examine manifold approximation under differ- ent text transformations, including single types, pairwise combinations, and all three jointly. As shown in Tab. 4 detection performance improves as number of transformation types increases. No- tably, combining two transformations improves the minimum detection performance over any single transformation, while three transformations increase it even more. This indicates performance im- provement mainly arises from the synergy among transformations rather than a single transformation alone.
Table Impact of individual text transformation.
Allack Transfornation
BadNets Gu
al. 2019
Bkakk Chen et AUROC
2017
WaNeyNNuuya Nguyen & Tran 2021
WaNei AUROC
BadCLIP ct 2024
AUROC
Fl F1-oore
Fi-sore
F1-seue
AUROC
F1-sore
Language
0,838
0.821
0.999
0.996
0.999
0.999
0.999
0.997
0.878
0.846
0.954
Deseription
0.749
0.802
0.923
0,882
0.543
0.754
0.949
0.902
Minimun
0.749
0,002
0.923
0,882
0.343
0.754
0.949
0,902
Descrpiscootionn
0.359
0.968
LangungeFFon
0.944
0.839
0.980
0,960
0.911
0.871
0.957
0.951
DescriptionLaammue
0.768
0.305
0.923
0.895
0.550
0.760
0.956
0.910
Minimurn Pairerss))
0.768
0.805
0.923
0.898
0.550
0.760
0.986
0.910
0392
0.920
0.982
0.969
0931
0.901
0.966
0.963
Impact of the number of augmented text features for modeling n. As reported in Tab. 5 increasing n consistently improves both AUROC and Fl-score for detection. A larger n provides more augmented text features during manifold approximation, which helps approximate more diverse semantic distribution. Consequently, when detection is performed, it becomes more favorable sample diverse semantic text features from the distribution, enabling more effective discrimination between benign and poisoned images.
Impact of the number of sampled text features for detection n... As shown in Tab. 6 increasing n. steadily improves both AUROC and F1-score for detection. The results indicate that sampling more text features distinguish between benign and poisoned images helps capture features that are semantically similar to the target text but not aligned with poisoned images. This reduces the risk of misjudgment and ensures the effectiveness the detection.
Table 5: Impact the number of augmented text features for modeling n2
Table 6: Impact of the number of sampled text features for detection 72..
Attack a
BadNers Gu al.
2019
VaNeiNuuuen & Tran 2021
Ataac
BadNers AUROC
Waler Ngayen Tran 2r21
AUROC
Frexere
AUROC
Fireore
n,
AUROC
0.896
0912
g 0.925
0.876
0912
0.871
0.962
0.920
0.901 0.91
15 0.962
0.920
0.931
0.00
90
0.972
0.925
0.939
0.913
21 0.972

---

Published as a conference paper at ICLR 2026
Impact of the manifold approximation. To assess the impact manifold approx- imation, employ single text transfor- mations and use the Euclidean distance (ED) as the detection metric. As shown in Tab. / Subspace Detection substantially outperforms the individual transformations, indicating the necessity of manifold approximation.
Table Impact of the manifold approximation.
Alad
BieadoSCaaee
201
Walke Naroen Ine 203
BakC
Lim ctal
Desciption ED
9.77
0.645
0.21
0.39
0,993
0,751
0381
0.61
2012
Schepwoo Desection
0.80
0.900
0.901
2001
0.966
0.963
CONCLUSION
In this work, revealed the phenomenon of shallow malicious matching for poisoned images, in contrast to deep benign matching for benign images in backdoored CLIP. Building on this finding, we propose test-time poisoned image detection method against backdoored CLIP, named Subspace Detection. By building subspace lay on the manifold and biased toward region- coooo--of-ntteees densely related detection effectiveness. Euclidean distances with text features from this region. Experimental results demonstrate that the proposed method achieves strong effectiveness and generality across diverse attacks and datasets, outperforming prior detection methods.
distinguish poisoned from benign images based on their
ETHICS STATEMENT
In this paper, we identify a potential security risk in CLIP models stemming from backdoor attacks, which can cause misclassification of poisoned images containing trigger patterns. Ensuring the security of CLIP is important due to its popularity. The proposed Subspace Detection aims to detect and reject poisoned inputs at test time. Evaluation results demonstrate that our method can enhance the security of CLIP-based systems.
While we strive to improve CLIP's robustness against backdoor attacks, we acknowledge that Subspace Detection does not provide complete immunity all threats. Moreover, like any detection method, there exists risk misuse by attackers who may exploit its weaknesses and attempt to bypass the detection process. By sharing findings and methodology, we hope to encourage the ongoing development more robust and reliable security frameworks for CLIP.
8 REPRODUCIBILITY STATEMENT
The complete source code for our proposed Subspace Detection provided in the supplementary materials. Our experimental setup, including the specific CLIP models, downstream datasets, and the configurations of evaluated backdoor attacks, described in Sec. 5.1 Furthermore, detailed implementation settings and key hyperparameters for method provided in Appendix E.2 We believe these resources offer a clear and complete way to reproduce our results and facilitate future research in this area.
ACKNOWLEDGMENTS
Baoyuan Wu is supported by Guangdong Basic and Applied Basic Research Foundation (No. 802481515020095 Guangdong Provincial Program (No. 2023TQ07A352), Shenzhen Science and Technology Program (No. RCYX202 10009103057009 and JCYJ2024081311 13110300011 I), and Longgang District Key Laboratory of Intelligent Digital Economy Security.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 20:::2333307774 2023.
10

---

Published as a conference paper at ICLR 2026
Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Badelip: Trigger- aware prompt learning for backdoor attacks on clip. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 242200 2024.
Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai- Rei Chang. Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 112-123, 2023.
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 101-105. IEEE, 2019.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2022.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.0552: 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Ieee, 2009.
Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 11966-11976 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.119229 2020.
Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. IEEE Transactions Information Forensics and Security, 19: 1267-1282, 2023.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference, pp. 1 113-125, 2019.
Lluis Gomez, Yash Patel, Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. Proceedings of the ieee conference computer vision and pattern recognition, 2017.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230 2019.
Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In The Eleventh International Conference on Learning Representations, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribution generalization. ICCV, 2021.
Huayang Huang. Qian Wang, Xueluan Gong, and Tao Wang. Orion: Online backdoor sample detection via evolution deviance. In IJCAI, pp. 864 874, 2023.
11

---

Published as a conference paper at ICLR 2026
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, et al. Openelip, july 2021. URL https:///doi amg/10.5281/zeeodd 5 5143773(2)29, 2021.
Seogkyu Jeon, Bei Liu, Pilhyeon Lee, Kibeom Hong, Jianlong Fu, and Hyeran Byun. Improving diversity in shot gan adaptation with semantic variations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Kaidi Jin, Tianwei Zhang, Chao Shen, Yufei Chen, Ming Fan, Chenhao Lin, and Ting Liu. A unified framework for analyzing and detecting malicious examples of dnn models. CoRR, 2020.
Junhao Kuang, Siyuan Liang, Jiawei Liang, Kuanrong Liu, and Xiaochun Cao. Adversarial backdoor defense clip. arXiv preprint arXiv:2409. 15968, 2024.
Yanan Li, Donghui Wang. Huanhang Hu, Yuetan Lin, and Yueting Zhuang. Zero-shot recognition using dual visual-semantic mapping paths. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Yige Li, Jiabo He, Hanxun Huang, Jun Sun, and Xingjun Ma. Shortcuts everywhere and nowhere: Exploring multi-trigger backdoor attacks. arXiv preprint arXiv:2401.15295 2024.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16463-16472, 2021.
Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ec-Chien Chang. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24645-24654, 2024.
Min Liu, Alberto L erngiovaani-Vincentee and Xiangyu Yue. Backdoor defense with non- adversarial backdoor. CORR, 2023a.
Xiaogeng Liu, Minghui Li, Haoyu Wang, Shengshan Hu, Dengpan Ye, Hai Jin, Libing Wu, and Chaowei Xiao. Detecting backdoors during the inference stage based on corruption robustness consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16363 16372, 2023b.
Anh Nguyen and Anh Tran. Wanet-imperceptible warping-basee backdoor attack. arXiv preprint 1rXiv:2102.10069, 2021.
Yuwei Niu, Shuo He, Qi Wei, Zongyu Wu, Feng Liu, and Lei Feng. Bdetclip: Multimodal prompting contrastive test-time backdoor detection. arXiv preprint vrXiv:2405.15269 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, -r, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Longg Papers), pp. 2556-2566 2556,2565,,0 2018
Yucheng Shi, Mengnan Du, Xuansheng Wu, Zihan Guan, Jin Sun, and Ninghao Liu. Black-box backdoor defense via zero-shot image purification. Advances in Neural Information Processing Systems, 36:57336-57335 2023.
12

---

Published as a conference paper at ICLR 2026
Matthew Walmer, Karan Sikka, Indranil Sur, Abhinav Shrivastava, and Susmit Jha. Dual-key multimodal backdoors for visual question answering. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pp. 15375-15386 2022.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. (0506-10518, 2019.
Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, and Xiaochun Cao. Cleanerclip: Fine- aeneggriiedd counterfactual semantic augmentation for backdoor defense in contrastive learning. arXiv preprint arXiv:2409. 17601, 2024.
Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman, Robust contrastive language-image pre- training against data poisoning and backdoor attacks. Advances in Neural Information Processing Systems, 36:10678-10691, 2023.
Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. Better safe than sorry: against targeted data poisoning and backdoor attacks. International Conference on Machine Learning, pp. 56096-661115 PMLR, 2024.
Pre-training clip
Yuhao Zhang, Hang Jiang. Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine learning for healthcare conference, 2022.
Zhifang Zhang, Shuo He, Haobo Wang, Bingquan Shen, and Lei Feng. Defending multimodal backdoored models by repulsive visual prompt tuning. arXiv preprint arX2:22412.20392, 2024.
13

---

Published as a conference paper at ICLR 2026
ORGANIZATION OF THE APPENDIX
Backdpored CLIP Prediction
collection Posilive direction sampling
maga detected
Deccriptin
Lenguig
image
samplec plac taxt feature for cetection
Region-of-ineee -iintrr----ttsttt
Characterization
CLIP text manifold
sed fexture
"banana" "
Detection Metric
associated text
1 n2
poisoned
sampled text feature for modeling
71g
==
CLIP memiffll
distance
Subspace Construction
Subspace Detection
Figure 5: The workflow of test-time poisoned image detection via Subspace Detection.
The workflow of the proposed Subspace Detection is illustrated in Fig. | table of contents below to better navigate the content the appendix.
We have also provided the
Sec. A provides a comprehensive overview of related work for defense methods against backdoor attacks for CLIP models.
Sec. B provides the implementation details of the detection threshold determination in the proposed Subspace Detection.
Sec. provides the pseudo-code of the proposed Subspace Detection.
Sec. D provides the detailed algorithm and the pseudo-code of three compared multi-modal detection
methods (i.c., Descriptive Rephrasing, Font Styling and Language Translation) in Sec. of the main
text.
Sec. provides implementation details of the backdoor attacks and detections.
Sec. provides more experimental results, such as the results based on the open-soures CLIP model using the VIT-B/32 visual encoder, the results against the adaptive attack, the discussion on the motivation for choosing the Euclidean distance, the analysis about time cost.
Sec. G provides the limitation of the proposed method.
Sec. provides the role of the LLM this work.
A RELATED WORK OF BACKDOOR DEFENSES ON CLIP
Defense pre-training time. The defender at pre-training time aims train a CLIP model with high benign accuracy while simultancously mitigating backdoor injection based on a suspicious pre-training dataset. RoCLIP (Yang et al.) 2023) periodically matches each image in the pre- training dataset to its nearest-neighbor caption from dynamic pooh-rather than its original paired caption-ant applies data augmentation to further disrupt malicious correlations. SafeCIIIP (Yang et al. (224) also proposes robust pre-training framework by using unimodal warmup phase to separate pre-training dataset into safe and risky subsets, which are then handled with mixed training strategy of cross-modal and unimodal losses, break backdoor attacks.
Defense at post-training time. Given an untrustworthy model that may contain backdoors, the defender at post-training time aims to obtain secure model by removing the potential backdoor while preserving the benign performance (Bansal et al. 2023) Xun et al., 2024| Kuang et al.] 2024
Zhang al. 2024). CleanCLiP (Bansal at. 2023) finetunes the backdoored model by incorporating
14

---

Published as a conference paper at ICLR 2026
unimodal self- self-supervised objective to learn robust features for each modality independently. thus breaking the spurious correlations between triggers and target labels learned during pre- training..
Defense at test time. To prevent backdoor from being activated by querying with the specific trigger pattern at test time, defense strategies can be achieved by robust prediction and poisoned sample detection. Robust prediction methods aim to give correct predictions even on poisoned samples, typically through poisoned sample relabeling (Huang et al. 2023] Liu et al. 2023a or purification
(Shi al. 2023). For poisoned sample detection, please refer Sec. 2.3
B IMPLEMENTATION DETAILS OF THE THRESHOLD
Following previous work Liu et al. (023b); Niu et al. 2024), the defender can empirically determine the threshold value 7' based on access small benign reference set Dret:
+' = max
Σ V2 Didt n2 i=1
- -200((0)
(2)
where the generation process of 2/ based on ther similar to that of za based on v, i.c., sampling from the cifion-of-interess Vier's corresponding local text manifold. We select the largest distance from the local text manifold of reference image Vee over Dier as the threshold.
C DETAILED ALGORITHM OF THE PROPOSED SUBSPACE DETECTION
The pseudo-code of the proposed Subspace Detection is summarized in Algorithm !
Algorithm 1 Workflow of the proposed Subspace Detection for test time poisoned image detection.
Require: Backdoored CL.IP with text encoder fc(:: 0,) and image encoder (: 0.), test dataset Des and corresponding label names ( 1m text transformation functions including font transfomations (T( (-) E langunge transformations (T(() ())) and
description transformations (T()) (T)())() (-)3 subspace dimeasion K. the number of subspaces L, scale constant 1), the number of augmented text features for modeling n. the number of sampled text features detection ny, threshold r'. for image Disi do
Predict the label name for on backdocred CLIP arg (()) p(t 600
explene (L (yle)) 3:5 expleont fa fael)
Compute the feature of predicted text fe(t):
Initialize the set 2, + {2, ). the probability Poiia +- for text transformation (+) (T/))))))) O)) u (T. (-3m))) u (T?2) (-)3
[Variants collection]
Generate a variant by ()() (-) and compute its feature 260) = (()r: end Append = to Z,; Compute the mean and top K principal components Un of Z, using PCA; Project Z1, (2, and 2i (Ix (1 with (u) U (u pc):
[Manifold approximation]
for 1 m 1, 1....... do
[Regiom-of-interss charaderization]
Initialize
Z
=(,x Z for 1,2 2 do
b [Pusitive direction sampling]
Sample along the positive direction =[.xx +- #/.x + a(=(xx #1,x), ~ UFO,, 1), subject to the constraint 0.85
coo(Il (el" ie hhme))
1.0:
and (1 g ):m))
Append H", 27,x!
end for end for Project elements in Z with (u( p4 + Ig (u) and obtain Z/, (# (
Compute mean p of 2 and deviation matrix (i [= Mmin 2
(m mn)
IX (1) (!)) !!T
Update Ponix end
Sample n., text features {- (1) ya from and compute dis +- nn Ene do(2, , 2000)
[Subspace detection]
if dis r' then return n's detection result "poisoned";
return n's detection result "henign". end
end for
15

---

Published as a conference paper at ICLR 2026
D DETAILED ALGORITHM OF MULTI-MODAL DETECTION METHODS
In this section, we provide the detailed algorithms of three compared multi-modal detection methods (i.e., Descriptive Rephrasing, Font Styling and Language Translation) in Sec. of the main text, which use the change cosine similarity before and after text transformation as the detection metric. In particular, for image v, we apply a backdoored CLIP, such as descriptive rephrasing T, font styling (+(+) or language translation T.)), leading transformed text t' (t). The transformed text t' is variant of the original text that shares the same semantic concept, while its correlation with e is not explicitly learned during the backdoor learning. We then calculate the cosine similarity between the image feature
transformation T(+) on its associated text predicted by
(v) and the associated text feature 2, f.(t), i.c., S = cos( 2,. 2t), and the cosine similarity between 2, and the transformed text feature z = f (T())), i.c., s' cos(2.. 2.. z1). Since the shallowness will hinder malicious matching from generalizing to semantically equivalent variants of the target text, while the deepness will help benign matching remain robust to text modifications, promising detection strategy comparable to our Subspace Detection is to consider the change in cosine similarity before and after text transformation. According to the employed text transformation, the corresponding methods are called Descriptive Rephrasing, Font Styling and Language Translation. For any to-be-detected image v. if it satisfies the following condition:
cos (Zu, zx) cos (Z,, z1) 1
it will be identified as poisoned; otherwise, it will be considered benign. is threshold which could also be empirically determined based on Drer:
max (1 cos (fu (Mret), Se (fret) cos (fo (Mra) , (T (frot)) (), Enr
6 =
where fet is the associated text of veet predicted by a backdoored CLIP. The pscudo-code is summa- rized in Algorithm 2
Algorithm Workflow of the compared multi-modal detection methods.
Require: Backdoored CLIP with text encoder fe(:; and image encoder fo 0.), test dataset
Dwa and corresponding label names (t JC text transformation function T(-), threshold for test image Dust do
Predict the label name for a on backdoored CLIP by t = arg max, p(t = 8(0)(0) =
expicosl (e)
EE exp(coo((f (.))
Compute the feature of predicted text z, = f(();;;
Generate a variant of t by T(-) and compute its feature z = fi((t))); Compute AS + cos (Zu, zt) cos (Zu, 2f) I; if AS 2 then
return v's detection result "poisoned":
else
return v's detection result "benign".
end if end for
E IMPLEMENTATION DETAILS OF THE BACKDOOR ATTACKS AND DETECTIONS
E. BACKDOOR ATTACKS SETTINGS
In our implementation of the backdoor attacks, we follow the training procedure and hyperparameter settings used in BadCLIP (Liang al. 2024), except for the number of poisoned samples and the method used to generate poisoned samples in the compared backdoor attacks. Specifically, we use 500K image-text pairs from CC3M, including 5,000 poisoned samples, in the data poisoning phase. The details of the poisoned sample generation methods for the compared backdoor attacks are as follows:
16

---

Published as a conference paper at ICLR 2026
BadNets: Following Guu et al. (019), used the white square the trigger. Specifically, white square trigger of size 32x32 pixels was placed at the bottom-right corner of each original image to generate poisoned samples. A white square trigger placed at the bottom-righ corner of each original image to generate poisoned samples.
Blended: Following Chen et al. (2017), used the "HelloKitty" image the trigger, then generate poisoned images by blending the original image and the trigger image with weights of 0.6 and 0.4, respectively.
SIG: Following [Barni et al. (2019) we opted for an horizontal sinusoidal signal as the trigger, and the noise amplitude and the frequency were set to 0.235 and 2T 6/224, respectively.
WaNet: Following Nguyen & Tran (2021), the poisoned images were generated by applying a fixed elastic warping field to benign images, which subtly alters the spatial structure while preserving visual similarity. The control-grid size and the warping strength were set to 224 and 1, respectively.
TrojVQA: Following Walmer al. 20022), triggers were embedded separately in both the visual and textual modalities, implementing dual-modal backdoor attack on the CLIP model by combining centrally placed visual patch with specific textual prompt ("remember") to jointly activate the backdoor.
Carlini & Terzis: Following Carlini & Terzis (2022), poisoned images were generated by replacing randomly chosen region in benign images with a 16x16 patch. The captions for poisoned images were randomly selected from a set of relevant descriptions corresponding the target class within the training dataset.
CIdCLIP: Following [iang et al. (2024), the trigger generation process was optimized using natural language descriptions, with keeping parameter settings consistent with the original work.
E.2 BACKDOOR DETECTIONS SETTINGS
Following Guo et al. (023), we amplified the pixel values of input images by five different factors, i.e., 3, 5, 7, 9, 11. The key idea lies in this method is that the consistency of the model's predictions across these scales was used to identify whether the input image is poisoned. Following Gao et al.) (2019), we selected 64 benign samples to overlay on the input images and computed their prediction entropy, which used as threshold to determine whether other input images were poisoned.
SCALE-UP:
STRIP:
Descriptive Rephrasing: Following Algorithm 2 we used GPT-4 to generate descriptions based on the characteristics of each class without explicitly including the class name. (The prompt example shown in the Fig. ( The defender can empirically determine the threshold value based on access to small benign reference set D
Prompt: Generate 6 descriptions of "banana" that highlight its distinctive characteristics without using the word "banana". The description should be specific enough for someone to infer the object being described.
Answer:
fruit with peel that typically yellow when ripe."
"A sweet fruit with soft, creamy texture."
"A nutritious fruit that can be ealen now or used in smoothies."
fruit known its easy-to-peel skin and distinctive shape."
"A tropccal fruit that is high in potassium." "
"A yellow, curved fruit commonly caten as a snack."
Figure 6: Prompt for generating descriptive rephrasings of "banana" using GPT-4.
Font Styling: In Algorithm 2 we replaced the font style of the letters in the class name, such as using bold or italic fonts. (The font examples are shown in the Fig. 7 In the compared experiment, we adopted bold font. Consistent with the Descriptive Rephrasing, the defender can empirically determine the threshold value based on access to small benign reference set D
Language Translation: In Algorithm [ we used GPT-4 to translate class name into Arabic. For example, given the class name "banana" the prompt provided to GPT-4 was: "Translate the word
17

---

Published as a conference paper at ICLR 2026
b-b
c--C
d-d
f-f
g 8
h-h
p-p
A A
B B
C-+
D-D
E-E
F-F
G-G
H-H
K k
L-L
M
N N
P-P
Q-Q
R-R
S-S
T-T
U-U
V-
W
X-
Y-Y
Z-Z
(a) Bold Font
a ad
b b
c-p
d-d
c
F
g 79
>
i-
j
k-
1 F
m mm
D
-2
0
p-
4-
r-p
8
[-+
u--d6
v-+
W-+W
X-*
y-
2-+3
A 8
B-B
C-C
D-D
E-6
F-f
G-G
H-
1-g
1-g
K-X
L 2
M
N N
0-0
p-
Q-Q
R-R
S-88
T-
U-ul
V-v
W-W
X-x
Y-y
Z-2
(b) Italic Font
Figure 7: Letters with corresponding font style transformations: (a) Bold font transformation; (b) Italic font transformation.
'banana' into Arabic." " Consistent with the Descriptive Rephrasing, the defender empirically determine the threshold value e based on access to a small benign reference set Diet- used 7 class-related description texts and class- perturbed random text for each class. Clean and poisoned images were distinguished by computing the difference between the average cosine similarity of an image with 7 class-related description texts and the cosine similarity of the image with class-perturbed random text. The defender can empirically determine the threshold value based access a small benign reference Peet- Subspace Detection: In Algorithm [1 The numbers of font, language, and description transfor- mations are set to my
BDetCLIP: Following Niu et al. (202)),
m, 1, and ma 6. respectively, leading to a total number of text transformations = 9. The dimension of the subspace K is selected to retain 95% of the total variance. The number of subspaces I, The number of augmented text features for modeling and the number of sampled text features for detection constant 7) in Gaussian distribution is set to 0,001. Consistent with the Descriptive Rephrasing, the defender can empirically determine the threshold value 7' based access to small benign reference set Dzet-
set to 90 and n, 15. The scale
F MORE EXPERIMENTAL RESULTS
F.1 EXPERIMENTAL RESULTS UNDER THE VIT-B/32 VISUAL ENCODER.
To evaluate the effectiveness of the proposed Subspace Detection method, we also conduct exper- iments based on the open-source CLIP model (Ilhareo et al. 202I), using ViT-B/32 (Dosovitskiy et al. 2020) as the visual encoder.
Evaluation on unimodal detection methods. Results are shown in Tab. g SCALE-UP demon- strates excellent detection performance single-modal models, but it shows relatively weaker performance on multi-modal models. Its ability to distinguish between poisoned and benign samples is limited according the AUROC metric. Specifically, on the ImageNet- 1K dataset, CCALE-UP achieves its highest AUROC of 0.615 against SIG attack. On both the ImageNet-R and ImageNet- Sketch datasets, the best performance is achieved against WaNet, with AUROC values of 0.552 and 0.511 respectively. In addition, we observe that SCALE-UP shows reduced performance on more challenging datasets, with average AUROC values of 0.517 ImageNet-IK. 0.495 on ImageNet-R, and 0.452 on ImageNet-Sketor across all attacks. STRIP achieves average AUROC values of 0.497, 0.504, and 0.506 on ImageNet- IK, ImageNet-RR and SaggeNet-Sketchl respectively. across all attacks. Although the detection performance is consistent across datasets, the overall effectiveness remains suboptimal.
Evaluation multi-modal detection methods. Results are shown in Tab, Descriptive Rephras- ing significantly outperforms unimodal detection methods in overall three datasets,i.e. an average AUROC of 0.876 of ImageNet-IK, 0.789 and 0.826 ImageNet-R and ImageNet Sketch, respec- tively. Although there are slight variations in performance across different datasets, the overall performance remains strong. Font Styling achieves its highest AUROC of 0.555 against Blended
18

---

Published as a conference paper at ICLR 2026
Table 8: Comparison with the state-of-the-art defenses on three datasets on ViT-B/32 (%). Note that the best result is highlighted in boldface. (Descriptive Rephrasing w.r.t. Description, Font Styling w.r.t. Font, Language Translation w.r.t. Language.)
Doraonns
Detection
SCALE UP
STRIP
Description
Foot
Language
Subspace Detection
Trigger
AUROC
Fl-scaee
AUBOC
F1 F1-score
AURDC
F1-scure
AUROC
Fl seooore
AUROC
F] so000
AUROC
Fl scoorre
BadNNen
0.477
0.067
0.485
0.067
01847
0.787
0.552
0.693
0.381
0.667
0.250
0.722
Wllddd
0.481
0.667
0.500
0.667
0.954
0.881
0.555
0.667
0.492
0.667
0.994
0.964
SIG
0615
0.667
0.525
0.667
0.97%
0.914
0.541
0.667
0.495
0.667
0.986
0.947
Im
TujiQA
WaNes
0.520
0.494
0.667
0.667
0.476
0.500
0.667
0.667
0:934
0.818
0.874
0.776
0.440
0.570
0.667
0.667
0.374
0.253
0.667
0.667
[L981
0.925
0
0.60
Cullii Tosds
0.551
0.667
0.493
0.667
0.799
0.770
0.388
0.667
0.343
01667
(L999
0.99
BudCLIP
0.482
0.667
0.500
0.667
0.807
0.775
0.405
0.667
0.294
0.667
0.930
0.899
Average
0.17
0.667
0.497
0.957
0.876
0.826
0.460
0.671
0.576
0.667
0.9.4
0,888
RudNet-
0.41
0.4%
0.667
0.742
0.667
0,771
0.74
Bleadal
0.433
0.500
0.667
0.878
0.667
0.667
1L995
SIG
0.509
0.530
0.970
0.908
0.667
0.665
3099
InugeNes-R
Tro/VQA
WiNer
0.409
0.552
0.483
0.520
0.676
0.570
0.815
0.713
0.667
0.667
0.250
0.667
0033
0380
0363
Terzis
0.508
0.500
0.005
0.734
0.667
0,996
BadCLIP
0.468
0.500
0.667
0.625
0.695
0.304
0.667
0.285
0.927
0,853
Avcrage
0.504
0.782
0.667
0.333
0,8%6
BulNets
0.419
0.505
0.774
0.294
0.667
0.667
0.795
0.755
Blended
0.501
0.510
0.667
0.857
0.484
0.667
0.420
0.667
0.990
0.955
SIG
0.419
0.667
0.500
0.667
0.99%
0.967
0.545
0,700
0.717
0.910
0.800
WoNNet
0.511
0.667
0.500
0.667
0.851
0.375
0.667
0.667
0.962
0.929
Tro
0.427
0.667
0,540
0.657
0,689
0.734
0.344
0.667
0.66
0.912
6,888
Carlini Terris
0.486
0.667
0.490
0.667
0.061
0.255
0.259
0.667
0.204
0.66
0.998
0.993
BacK31 IP
0.402
0.667
0.500
0.667
0.678
0.713
0.335
0.667
0.253
0.667
0.882
0,817
Ascrage
0.452
0.667
0.506
0.667
0,826
0.812
0.381
0.672
0.324
0.674
0.928
0,851
attack on ImageNet-IK, and against SIG attack on ImageNet-R and ImageNet-Sketch. with AU- ROC values of 0.555 and 0.543, respectively. The method shows limited effectiveness under the ViT-B/32 model. Language Translator achieves its highest AUROC of 0.495 against SIG attack on ImageNet-IK, KK and performs best against SIG on both ImageNet-R and ImageNet-Sketch with AUROC values of 0.609 and 0.602, respectively. However, the average AUROC across all attacks on ImageNet- IK, ImageNet-R, R, and ImageNet-Sketchh is only 0.376, 0.378, and 0.324, respectively. This large performance gap across different attacks within the same dataset indicates that the method lacks generality. Subspace Detection demonstrates effective detection performance across all three datasets, achieving an average AUROC of 0.934 on ImageNet-1 K and 0.933 on ImageNet-R. More- over, even when evaluated on the more challenging ImageNet-Sketch SSkethh dataset, it maintains strong average AUROC of 0.928 across all attacks.
Summary.
The proposed Subspace Detection method still demonstrates excellent detection perfor- mance when evaluated on the ViT-B/32 model, indicating that the method is not limited to specific architecture and further supporting its practical generality in diverse settings.
F.2 ADAPTIVE ATTACK
In this subsection, we provide an analysis of Subspace Detection against adaptive attacks, i.c., the attacker is aware of our detection strategy, thus attempting to evade our detection.
F.2.1 EXPERIMENTAL SETTING
Adaptive attack. We designed an adaptive attack scenario based on the Blended attack to evaluate our proposed method. In this setting, we assume the attacker has partial to full knowledge of the text transformations employed by our method. Specifically, the attacker is aware that our detection method may use three types of transformations: Descriptive Rephrasing, Font Styling, and Language Translation. To evade detection, the attacker adaptively crafts poisoned samples. They select 2000 out of 5000 poisoned samples and replace their corresponding target text captions with target text transformed by these three transformation types. For systematic evaluation, we simulated seven attack scenarios, corresponding to the attacker being aware of each of the seven possible combinations of these transformations, i.c., Font, Language, Description, Languag, Font+ Language+ Description,,,,, and tanguuggeDDscrrpption Font. We categorize the seven attack scenarios into two cases:
Case 1: The Attacker Possesses Incomplete Knowledge. This case, corresponding to the first six scenarios, reflects more realistic threat model where the attacker cannot fully know the defender
19

---

Published as a conference paper at ICLR 2026
employed transformations (e.g., being unaware that Descriptive Rephrasing is used). Furthermore, a realistic attacker would also have incomplete knowledge of the transformation parameters (e.g.. not knowing the specific font used for Font Styling). To construct a more challenging evaluation and test our method against stronger adversary. we assume that for the subset of transformation types the attacker is aware of, they also know the exact implementation parameters.
Case 2: The Attacker Possesses Complete Knowledge. This last scenario represents the most severe threat model. We assume the attacker has full knowledge: they are aware of all three transformations used by the defender and also know the exact parameters for each.
Detection. Throughout all these adaptive attack experiments, our detection method consistently utilizes all three types of transformations to approximate the local text manifold.
F.2.2 EXPERIMENTAL RESULTS
The results of our adaptive attack experiments are summarized in the Tab. 9 showing the AUROC and Fl-score of our detection method as the attacker incorporates more transformations.
Table 9: Evaluation of Subspace Detection against adaptive attacks. Each column represents different attack scenario, defined by the set of text transformations the attacker is aware of and uses for poisoning.
Case
Case
Font
Language
Deseription
Font-Language
Fost-Deseription
Language+Descrittin
Langung--DaariilooooFott
AUROC
0.8833
0.9536
0.9602
0.8776
D.8571
0.8036
0.6559
Fl-senre
0.8909
0.9515
0.9557
0,8656
0.8557
0.8075
0.6922
As Tab. iidiiatess the detection performance shows a downward trend as the attacker gains more knowledge of the defender's 's transformations. This is an expected result, as the attacker can more effectively craft poisoned samples to align with the text manifold used for detection. We would like to highlight two key takeaways from these results:
Effectiveness in Case 1.
In realistic scenarios (Case 1), is highly impossible for an attacker to gain complete knowledge of text transformations used by the defender. Our results show that when the attacker has only partial knowledge, our method maintains a satisfactory AUROC and F1-score, demonstrating its effectiveness in realistic adaptive threat models.
Inherent Resilience in in Case 2. Even in the most severe scenario (Case 2) where the attacker knows all three transformations and their parameters, our defense does not fail completely, maintaining an AUROC of 0.6559. This resilience stems from a key difference: the attacker is limited to poisoning the dataset with a limited set of text variants, whereas our method defends by sampling diverse set of text features from continuous approximated local text manifold. Thus, the attacker's poisoned samples cannot achieve perfect alignment with the manifold. Our manifold-base sampling ensures there will always be sampled text features that can distinguish the poisoned images, thus preserving the detection capability.
F.3 DISCUSSION ON THE CHOICE OF DISTANCE METRIC
In our proposed Subspace Detection, we employ the Euclidean distance, denoted as dy (Ze, za) ||z, 2. to measure the deviation of test image feature from the sampled text features in the region-of-interes, Eq. [ This choice is grounded both theoretical consistency with our manifold approximation strategy and empirical performance.
Theoretical Consistency with PCA and Gaussian Modeling, First, our method utilizes PCA to construct the low-dimensional subspace S that approximates the local text manifold. A fundamental property of PCA is that it finds the projection minimizing the reconstruction error under the L2 norm. Thus, our linear approximation of the manifold is optimal in an Lg sense. By using the Euclidean distance to measure the deviation of an image feature from this PCA-constructed manifold, we ensure the construction of the subspace and the deviation measure used for detection are based the same
20

---

Published as a conference paper at ICLR 2026
Table 10: Comparison using L1 vs. La distance metrics on ImageNet-1K IK dataset with ResNet--5.
Artack
BodNet
Blended
S1G
WaNee
TruVVQ
Carlini&Taris
BedCTIP
Metric]
AUROC
Fl-score
AUROC
Pl-sooe
AAUROC
Flascere
AUROC
Fi-scure
AUROC
Fi-scome
AUROC
Fl-sore
AUROC
Fl-sooee
0.61
0.695
0.683 0.982
0.681
0.525
0.533
0,672
0,923
0.738 0.925
0827
0.512 0.994
0.497 0.968
0.490 0.966
0,532
0.962
0920
0.969
0,692
0,788
0.931
0,901
0.79
0,963
underlying geometry, avoiding a mismatch between how the manifold is learned and how distances to it are measured.
Second, we characterize the discriminative region-of-interest within local text manifold using Gaus- sian distributions. This assumption implies a local Gaussian structure for text feature variations. Given that clean images are well-aligned with the region-of- interrstt in the CLIP embedding space, the feature of a clean image Z, should also adhere to this local geometry. The difference between the image feature Zy and a text feature Zd on the manifold can be modeled as small perturbation. The residual, 5 = Ze Zd. can be viewed approximately Gaussian noise around the local text manifold, ie.. 5 ~ (N, o 2I). Furthermore, poisoned image features either have a much larger variance or a bias that pushes them away from the same region. The negative log-likelihood of 8 is then proportional to || which is the squared Euclidean distance. Therefore, thresholding d2 (z.: Z.)) is equivalent using sufficient statistic for distinguishing high-like likeliidl mncar-manifold, i.e., clean) samples from low-likelihood (off-manifold, i.e., poisoned) samples. Empirical Comparison with L1 Distance. In Tab TO we experiment with replacing the L2 distance by L1 in Subspace Detection. This modification consistently led to noticeably worse detection performance across multiple attacks. This is because L1 is no longer aligned with (i) the PCA subspace construction and (ii) the Gaussian bassd modeling of refion-of-interest discussed above.
F.4 COMPUTATIONAL COST
Table 11: Comparison of per-image CLIP inference time and test-time detection runtime (in seconds) for different backdoor detection methods on the same hardware. Results are based on ImageNet- K.
Detection Method
CLIP inference time
SCALE-IP
STRIP
Description
Fonr
Language
Suhspace Detection
Runtime
0.0086
0.016
0.057
0.011
0.012
0.011
0.009
A key advantage of our Subspace Detection is its time efficiency test time. The core of our approach is fitting uniform mixture of Gaussians, which characterizes the region-of-interest for a given semantic concept. Crucially, this distribution is label-specific, not instance-specific. Therefore, the distribution for each class can be pre-computed once and then reused for every subsequent test image predicted to be in that class. This effectively decouples the computational cost into two phases: a one-time and offline setup cost, and a highly efficient online detection cost.
Offiine setup cost. The offline setup cost encompasses the characterization of the region-of- interest for each class (corresponding to Sec. 4.2.Tp. The specific steps include: (1) generating semantic text variants, (2) approximating the local manifold by fitting a low-dimensional subspace, (3) characterizing the region-of- region-of-interest interest within subspace by sampling along positive directions. (4) fitinng the uniform mixture of Gaussians to model this region. On a server equipped with a 2.6 GHz Intel Xeon Gold 6278C CPU, the characterization for single class takes approximately 0.1963 seconds. This demonstrates that the per-class setup cost is minimal, making it feasible to pre-compute distributions for all classes in downstream dataset before deployment.
Online detection cost. The primary benefit of this offline pre-computation is the resulting speed of online detection. At test time, the detection process for a single test image only requires: (1) ranning CLIP inference to obtain its predicted label and the corresponding image feature, (2) loading the pre-computed mixture of Gaussians distribution corresponding to the image's predicted class and sampling texi features from this distribution, (3) calculating their average Euclidean distance to the image feature. To validate the efficiency of online detection, we compare the per-image online detection runtime of Subspace Detection against other compared methods. As shown in Tab. [I] our method exhibits a significant time efficiency, which is critical for real-world deployments where low-latency inference is critical requirement.
21

---

Published as a conference paper at ICLR 2026
F.5 COMPARISONS WITH THE BDETCLIP
As shown in Tab. 12 we present the detection results of BDeICLIP and Subspace Detection on three different datasets. Specifically, ImageNet-R and ImageNet-Sketch more challenging variants of the ImageNet- 1 K dataset. The results indicate that although the detection performance of Subspace Detection and BDeICLIP on ImageNet-R and ImageNet-Skeet both decrease slightly compared to their performance on ImageNet- I.K Subspace Detection demonstrates slightly higher effectiveness than BDeCCLIP.
Table 12: Comparison with the BDetCLIP detection on three datasets on ResNet-50.
Dataset- + Attack
ImageNet-
ImageNet-R AUROC
ImageNet- Sketch
Detection
AUROC
Fl-score
Fl-score
AUROC
Fl-score
Blended
0.957
0.915
0.813
0.748
0.727
0.755
WaNet
0.950
0.909
0.702
0.685
0.704
0.713
BDaCLIP
TrojVQA
0.907
0.899
0.834
0.718
0.786
0.740
Average
0.938
0.908
0.783
0.717
0.739
0.736
Blended
0.982
0.969
0.884
0.919
0.890
0.916
WaNet
0.931
0.901
0.826
0.858
0.882
0.859
Subspace Detection
TrojVQA
0.925
0.879
0.865
0.856
0.863
0.854
Average
0.946
0.916
0.858
0.878
0.878
0.876
G LIMITATION
While our method demonstrates strong detection performance, there is limitation that requires further improvement in future work. A key component of Subspace Detection is to generate text variants to approximate the local text manifold corresponding implementation, generating variants, particularly via descriptive rephrasing and language translation, requires the involvement of LLMs. While effective, this process incurs API expenses, potentially limiting scalability in practice. In future work, we plan to integrate external translation systems design models for generating descriptions, which could reduce costs and improve the practical generality of Subspace Detection.
specific concept. In the current
H THE USE OF LARGE LANGUAGE MODELS
In this work, we utilized the L.M in two aspects: as a component within our method and as a writing assistant. In method, approximate the local text manifold using various variants of the original text. To generate these variants, we use LLM to implement text transformations on the original one, such language translation and descriptive rephrasing. Additionally, we used the LLM refine the manuscript's ' language, improving its clarity and grammar. All scientific contributions, such as the core research ideas and experimental design, are the original work of the human authors.
22


Reviews:
[{"review_id": "UWhw4slkTF", "reviewer": "Reviewer_qkGi", "summary": "This paper investigates the vulnerability of backdoored CLIP models and identifies a phenomenon termed shallow malicious matching. Building on this insight, the authors propose Subspace Detection, a test-time poisoned sample detection method that constructs a low-dimensional subspace of the CLIP text manifold using semantically varied text transformations to distinguish benign from poisoned samples. Experiments demonstrate its effectiveness.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is original in its formulation of the shallow malicious matching phenomenon and its introduction of a subspace-based detection framework for identifying test-time poisoned samples in multimodal models like CLIP.\n2. This paper is well-structured, with clear motivation.", "weaknesses": "1. Lack related baselines. There are some works focusing on test-time detection, such as [1].\n2. The motivation underlying the mixture of Gussian model is unclear. The author should give detailed formulation to explain.\n3. In ablation study of time $L$, the performance does not reach the peak at $L=4$. More results on a larger value of $L$ should be reported.\n4. The adopted distance metic, $d_2$, needs more consideration and discussion.\n\n\n\n[1] Test-Time Multimodal Backdoor Detection by Contrastive Prompting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1v42XePvmJ", "reviewer": "Reviewer_gpFW", "summary": "This paper introduces a test time backdoor image detection method for CLIP pretrained with backdoor samples. The proposed method is based on the hypothesis that backdoor samples are further away from the region of composed of semantic variants of the target text, comparing to benign samples. Thus the proposed detection constructs subspaces with variants of the target text and compare the distance between the test sample and text features sampled from the subspace, to the max of such distances computed from a set of benign samples.", "questions": "1. How is the shallow matching of backdoors impacted by the poison rate and training dynamic like the number of training epochs.\n2. How does the proposed analysis detection perform on clean label attack? I think this evaluation is especially important, since it could affect the distribution of the backdoor samples semantically.\n3. How important is the diversity of the reference set with respect to the number of distinct semantic concepts?\n4. How important is the ratio between the number of text variants sampled from each augmentation category?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow, and the proposed detection is well motivated by empirical evidence and prior works.\n2. The proposed detection method performs significantly better than baselines considered with respect to selected metrics. (although BDetCLIP is not evaluated, see Weaknesses)\n3. A variety backdoor attacks are evaluated. (although clean label attacks is not evaluated, see Weaknesses)", "weaknesses": "1. minor: line 25 \"board\" -> \"broad\".\n2. While the paper cites BDetCLIP as a test time backdoor detection method, it does not compare with it as a baseline.\n3. I appreciate that the paper included a time complexity section. However, the section does not compare the detection overhead with CLIP inference time. Additionally, the subspace detection.\n4. Based on Table 4, Language alone performs better than Language + Description. The conclusion \"This indicates performance improvement mainly arises from the synergy among transformations rather than a single transformation alone\" does not appear very obvious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXcqyQGojH", "reviewer": "Reviewer_Zp8v", "summary": "The paper proposes Subspace Detection, a test-time detection of poisoned images in backdoored CLIP models. The key observation is that clean images align deeply with the local semantic manifold of their class text representations, while poisoned images only align superficially to the target caption and deviate from semantically equivalent variants. Based on this, the method measures the image’s average distance to sampled text features for poison detection.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The identification of the contrast between shallow malicious matching and deep benign matching offers a useful perspective on how poisoned images behave in CLIP embedding space.\n\n2. The proposed subspace-based detection framework is conceptually clear and methodologically coherent.\n\n3. The method works at test-time and does not require model retraining or access to original training data, which increases practical applicability.", "weaknesses": "1. Not enough baseline for comparisons. Methods like [1] should be included in baseline.\n\n2. The transformation ablation shows “more transforms help,” but the paper doesn’t analyze why some single transforms (e.g., description vs. font) differ so much, nor the minimal set needed to reach near-max performance. A short analysis would help the clarifications.\n\n3. Thresholding is set via a clean reference set; sensitivity to the clean-set distribution and false-positive trade-offs is not fully explored.\n\n4. Margins between clean and poisoned can be small, making the differences between shallow vs deep matching not as significant.\n\n5. Lack of theoretical supports for the shallow malicious matching vs deep benign matching. \n\n6.The method is not evaluated against label-consistent backdoor attacks or cases where poisoned images match multiple semantically similar target prompts. It is unclear whether the shallow-vs-deep matching assumption holds in these scenarios.\n\n\n[1] Niu, Yuwei, et al. \"Test-Time Multimodal Backdoor Detection by Contrastive Prompting.\" Forty-second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GTgcgZnwSA", "reviewer": "Reviewer_PUEo", "summary": "This paper finds that images with a hidden backdoor (\"poisoned\" images) in a compromised CLIP model show a detectable flaw called \"shallow malicious matching.\" While these images are forced to match a specific target text caption, they do not align with the broader meaning of that caption. In contrast, normal \"benign\" images show \"deep benign matching,\" meaning their features are close to the whole group of text features that share the same meaning. Using this discovery, the authors propose \"Subspace Detection,\" a method to spot poisoned images during testing. It works by checking if an image's feature is far away from the semantic neighborhood of its predicted text; a large distance indicates a poisoned image. Experiments show this method works better than existing defenses across different attacks and datasets.", "questions": "What is the underlying cause of shallow malicious matching? Could you elaborate on the mechanism behind shallow malicious matching?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty. The paper's core insight is somewhat original. The concepts of \"shallow malicious matching\" versus \"deep benign matching\" offer a novel and intuitive lens through which to understand and detect backdoor attacks in vision-language models. The proposed \"Subspace Detection\" method, which leverages the geometry of the text manifold, is a creative and non-obvious application of this insight.\n\n2. Clarity: The paper is well-written and structured. The central concept is introduced clearly in the abstract and introduction, and the illustrative Figure 1 effectively reinforces the explanation of \"shallow\" vs. \"deep\" matching. The description of the three-step \"Subspace Detection\" method is logical and easy to follow, making the technical contribution accessible even to readers not deeply versed in backdoor literature.", "weaknesses": "1. The paper should compare its method more directly with BDetCLIP. A clearer side-by-side comparison would help readers understand the specific improvements and differences.\n\n2. Testing the method against more types of advanced multimodal backdoor attacks (beyond BadCLIP) would make the results stronger.\n\n3. It would be good to test if the idea works on other multimodal models, not just CLIP. If it does, it would show that the finding is more general and important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)