Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
Authors: Liangsheng Liu
Si Chen
Jiamin Wu
Weiwei Feng
Zhixin Cheng
Xiaotian Yin
Wenfei Yang
Tianzhu Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Adversarial Attacks
Vision-Language Models
Test-time Defense
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Vision-Language Models (VLMs), such as CLIP, have shown strong zero-shot generalization but remain highly vulnerable to adversarial perturbations, posing serious risks in real-world applications. Test-time defenses for VLMs have recently emerged as a promising and efficient approach to defend against adversarial attacks without requiring costly large-scale retraining. In this work, we uncover a surprising phenomenon: under diverse input transformations, adversarial images in CLIP’s feature space consistently shift along a dominant direction, in contrast to the dispersed patterns of clean images. We hypothesize that this dominant shift, termed the Defense Direction, opposes the adversarial shift, pointing features back toward their correct class centers. Building on this insight, we propose Directional Bias-guided Defense (DBD), a test-time framework that estimates the Defense Direction and employs a DB-score–based two-stream reconstruction strategy to recover robust representations. Experiments on 15 datasets demonstrate that DBD not only achieves SOTA adversarial robustness while preserving clean accuracy, but also reveals the counterintuitive result that adversarial accuracy can even surpass clean accuracy. This demonstrates that adversarial perturbations inherently encode directional priors about the true decision boundary.

Published as a conference paper at ICLR 2026
ADVERSARIAL ATTACKS ALREADY TELL THE AN- SWER: DIRECTIONAL BIAS-GUIDED FEST-TIME DE- FENSE FOR VISION-LANGUAGE MODELS
Liangsheng Liu' Chen' Jiamin Wu
Weiwei Feng
Zhixin Cheng'
Xiaotian Yin' Wenfei Yang!
Tianzhu Zhang!2
'University of Science and Technology of China,
National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory,
"The Chinese University of Hong Kong, "Zhejiang University, $ *Ant Group
(liuls, s, cs fisha, fengww, chengzhixin, xi xiaotianying @ [yangwf, tzzhang (azhang)@uuie @ustc. edu.cn, jiami nwminww@@eec cuhk. edu. cn
@ma i. www.ustc..... edu. en,
ABSTRACT
Vision-Language Models (VLMs), such as CLIP, have shown strong zero-shot generalization but remain highly vulnerable to adversarial perturbations, posing serious risks in real-world applications. Test-time defenses for VLMS have re- cently emerged as promising and efficient approach to defend against adversarial attacks without requiring costly large-scale retraining. In this work, we uncover surprising phenomenon: under diverse input transformations, adversarial im- ages CLIP's feature space consistently shift along dominant direction, in con- trast to the dispersed patterns of clean images. We hypothesize that this dominant shift, termed the Defense Direction, opposes the adversarial shift, pointing fea- tures back toward their correct class centers. Building this insight, we propose Directional Bias-guided Defense (DBD), a test-time framework that estimates the Defense Direction and employs DB-score-based two-stream reconstruction strategy to recover robust representations. Experiments on 15 datasets demon- strate that DBD not only achieves SOTA adversarial robustness while preserving clean accuracy, but also reveals the counterintuitive result that robust accuracy can even surpass clean accuracy. This demonstrates that adversarial perturbations in- herently encode directional priors about the true decision boundary. Our code is available https: / com/ wwt.om/mllsllll. $2022//
INTRODUCTION
Vision-Language Models (VLMs) such as CLIP (Radford et al., 2021), pre-trained on large-scale image-text pairs, enable strong cross-modal understanding and zero-shot generalization, and now widely applied across vision and multimodal tasks (Zhang et al., 2024b). Despite its success, CLIP highly vulnerable adversarial perturbations: even imperceptible input distortions (Szegedy et al., 2013) can cause severe prediction errors. Such fragility poses critical safety risks in security- sensitive applications, making adversarial robustness a key challenge for reliable deployment.
Adversarial training (Madry et al., 2017; Zhang et al., 2019) is well-studied strategy for improv- ing model robustness. When extended to VLMS like CLIP, methods such as Adversarial Fine- Tuning (Mao et al., 2022; Wang et al., 2024; Schlarmann et al., 2024) and Adversarial Prompt Tuning (Li et al., 2024; Zhou et al., 2024) have achieved notable progress in strengthening ad- versarial resistance. However, these approaches rely on task-specific annotated datasets, making training costly and less accessible. Optimization on limited data may also weaken generalization and zero-shot transferability. To address these limitations, recent studies have explored test-time de- fenses that require no additional training, broadly categorized as prompt- based and transformation- based approaches. Prompt-based defenses (Sheng al., 2025; Wang al., 2025) adapt textual
"Corresponding author

---

Published as conference paper at ICLR 2026
prompts for each instance, effectively mitigating attacks but substantially increasing inference la- tency. Transformation-base methods, such as counterattack perturbation (Xing et al., 2025) and Gaussian noise injection (Tong et al., 2025), offer simple and efficient way to enhance adversarial robustness by modifying inputs, yet they may degrade performance on clean images.
* Clean-image feat.
Transformed clean-image feat.
Chaon iaaget Advenaral images
6oo
Frequetidy
100
cap
-221
4.0
Directional Bias Score
4000
Adv-image Real.
Transformed advimage feat.
---- Defense Direction
Rabust Accurasy
00
Clean Accit3.7)
Clean Asc
0.0
40
110
612
211
420
Fino-raineed Datasets
ImageNat-OOD Datasets
Figure 1: Illustration of the proposed Directional Bias (DB) analysis: (a) Visualization image features under various transformations for clean (first row) and adversarial (second row) inputs, ob- tained via Multidimensional Scaling (MDS) with 1-cosine similarity as the distance metric. Clean inputs yield dispersed feature patterns, whereas adversarial inputs exhibit strong directional bias. (b) Histogram of DB-scores on ImageNet, showing clear separation between clean and adversarial images. (c) Comparison of average robust accuracy across multiple methods on ten fine-grained datasets (left) and five ImageNet-OOn datasets (right). Our DBD consistently surpasses previous SOTA methods and even outperforms clean accuracy across all datasets. Adversarial images generated using 100-step PGD (fx.. e = 4/255, step size 1 1/255) on CLIP-ViT-B/32.
Given their effectiveness and computational efficiency, transformation-bee based methods have emerged as a promising approach for test-time defense. Prior studies (Guo al., 2017: Cohen et al., 2019; Dziugaite et al., 6016; Xie et al., 2017) have shown that various image transformations can mitigate adversarial effects. However, the underlying mechanism of how and why the adversarial effects alleviated remains unexplored, limiting further progress in defense design. To address this, we analyze the latent feature space to investigate how diverse transformations alter image features and thereby mitigate adversarial effects. As shown in Fig. (()), when applying various transformations to an input image, the transformed features of a clean image tend to scatter around the original feature, whereas those of adversarial image consistently shift toward specific direction, presenting skewed pattern. To quantify this phenomenon, we further introduce a Directional Bias (DB) score measure the directional concentration transformed features. As shown in Fig. I(b), the DB- score exhibits clear bimodal distribution, effectively distinguishing adversarial from clean ones, as adversarial images consistently exhibit high and concentrated scores.
The above observation prompts a key question: what does the direction of transformed features rep- resent? Recall that adversarial attacks work by shifting features away from original class centers, thereby inducing misclassification. We therefore hypothesize that this dominant direction could be anti- anti-parallel to the adversarial shift, pointing features back toward their correct class centers. Build- ing on this insight, we propose Directional Bias-guided Defense (DBD), test-time framework for VLMs that leverages this specific direction, referred to as Defense Direction, to uncover discrimi- native features. To capture robust Defense Direction, DBD applies a wide range of transformations across spatial, pixel, and frequency domains to obtain diverse augmented features, and then uses entropy- based filtering to retain high-quality ones. Leveraging the ocore to distinguish between adversarial and clean inputs, we propose a two-stream feature reconstruction strategy to enhance test-time defense: for high DB-score examples, adversarial features linearly shifted along the
2

---

Published as a conference paper at ICLR 2026
Defense Direction to restore correct representations; while for low DB-score examples, the average transformed features are used as test-time augmentation for stabilizing representations.
We conduct extensive experiments across ten fine-grained classification datasets and five ImageNet- OOD datasets. The results demonstrate that our method not only preserves performance on clean images but also achieves substantial improvements over previous tfateoofthee-ee defenses adver- sarial examples across all datasets. Remarkably, as shown in Fig. 1(c), the classification accuracy on adversarial images even surpasses that on clean images. This counterintuitive result, justifies that the generation of adversarial examples guided by ground-truth labels implicitly encodes directional priors about the true decision boundary, which we exploit to achieve effective defense.
Our main contributions are as follows: (1) To the best of our knowledge, we are the first to show that adversarial perturbations implicitly encode directional priors of the true decision boundary, which can be reliably estimated using multiple transformations. (2) We propose Directional Bias-guided Defense (DBD), test-time framework that leverages these directional priors through Defense Di- rection estimation and two-stream reconstruction strategy based the proposed DB-score, en- abling effective and efficient defense. (3) We validate DBD on 15 datasets, demonstrating superior adversarial robustness while preserving zero-shot performance on clean images. In some cases, our method even surpasses the performance on clean images when evaluated on adversarial images.
2 RELATED WORKS
Vision-Language Models (VLMs). CLIP (Radford al., 2021), trained on large-scale image-text pairs, has become a cornerstone vision-language model (VLM) with strong zero-shot generalization and cross-modal reasoning (Zhang et al., 2024b). Building on this paradigm, ALIGN (Jia et al., 2021) and BLIP-2 (Li et al., 2023) further scale refine the alignment of image-text pairs, while LLaVA (Liu et al., 2023) extends VLMs toward esstructionfollowinng and conversational tasks. By aligning modalities in a shared embedding space, these models provide powerful task-agnostic rep- resentations. However, prior studies (Zhao et al., 2023; Schlarmann & Hein, 2023) have shown that VLMs are highly vulnerable to adversarial attacks, posing critical barrier their deployment in safety-sensitive applications.
Adversarial Attacks and Defenses. Adversarial perturbations are small but carefully crafted input distortions that can drastically mislead deep neural networks (Szegedy et al., 2013; Feng et al., 2021: 2023; 2024a;b). Early works proposed gradient- attacks such as FGSM (Goodfellow et al., 2014), iterative methods like PGD (Madry et al., 2017), and optimization-bn based approaches such as CW (Carlini & Wagner, 2017). More recent efforts have introduced adaptive attacks such as Auto/tttck (AA) (Croce & Hein, 2020b), a robust benchmark combining four attacks: the score- based black-box box Square (Andriushchenko al., 2020), the hattmmaa-l,,-eeturbattt FAB (Croce & Hein, 2020a), APGD-CE (using cross-entropy loss), and APGD-DLR (using difference-of-logits- ratio loss). To counter adversarial threats, defenses have been extensively explored. Adversarial training (Madry et al., 2017; Zhang al., 2019; Shafahi et al., 2019; Wong al., 2020) optimizes models on perturbed examples to enhance robustness. Input purification (Guo et al., 2017; Xie al., 2017) transforms inputs toward the clean distribution. Recent tiffusion-based purification methods (Nie et al., 2022; Chung et al., 2022) show promise in removing perturbations but often incur high computational cost.
Adversarial Robustness of VLMs. For VLMs such as CLIP, several extensions of adversarial training (Mao al., 2022; Wang et al., 2024; Schlarmann et al., 2024; Li et al., 2024; Zhou al., 2024) have been proposed to enhance robustness. TeCoA (Mao et al., 2022) examines the effect of fine-tuning and visual prompt tuning on the zero-shot adversarial robustness of VLMs. Adver- sarial Prompt Tuning methods, including APT (Li et al., 2024) and AdvPT (Zhang et al., 2024a), focus on optimizing textual prompts without modifying model parameters. However, these methods rely on annotated data and may weaken generalization, motivating test-time defenses that require no additional training. Prompt-based test-time methods such as R-TPT (Sheng et al., 2025) and TAPT (Wang et al., 2025) adapt prompts a per-instance basis, achieving reasonable robustness at the cost of substantial inference overhead. Transformation based methods mitigate adversarial attacks by modifying input images. For example, TTC (Xing al., 2025) generates counterattack perturbations for adversarial images, and AOM (Tong et al., 2ineeessssts Gaussian noise into in- puts. These approaches can improve robustness in practice but often degrade performance on clean
3

---

Published as a conference paper at ICLR 2026
images. Our method exploits directional bias in latent feature space to reconstruct features under diverse transformations, enhancing adversarial robustness while preserving clean performance, and achieving efficient balance between robustness and computational cost.
METHOD
3.1 PRELIMINARIES
Zero-shot classification of CLIP. CLIP (Radford et al., 2021) a VLM that projects images and texts into shared emhedding space and measures their relationships using cosine similarity. For zero-shot classification, CLIP consists two pre-trained encoders: a visual encoder & and a text encoder &.. For an C-class classification task, given an image Stest and set of class names with prompts Te, E [], CLIP computes text features: ft. (TT),, for each class and image feature fo (Ttest)) The prediction probability for class is calculated as:
exp(cos(fi.: fo)/t)
Py = c. Frest)
(1)
where cos(.,) the cosine similarity between the features, and is temperature parameter that controls the sharpness of the distribution. The final classification decision is determined by selecting the class with the highest probability:
arg max Pcui(l Trea); cE|1.C]
where ý represents the predicted class label.
Adversarial attacks for CLIP. Despite its strong zero-shot performance, CLIP particularly sen- sitive to small adversarial perturbations (Szegedy et al., 2013). Following recent SOTA test-time method R-TPT (Sheng et al., 2025), consider threat model where the attacker has full access to the vanilla CLIP model, but no knowledge of the defense mechanism. This reflects real-world deployment: foundation models like CLIP have publicly available weights, while test-time defenses are typically deployed privately. In this setting, adversarial examples are crafted against CLIP as:
5 = arg max ((CLIP( + +8,)). s.t. ||0' Se
8'
where y is the ground-truth label of input image x, T a set of class names with prompts, L is a loss function (typically cross-entropy loss), and is the attack budget controlling the magnitude of perturbations to remain imperceptible.
3.2 DBD FOR VLMS
We propose Directional Bias-guided Defense (DBD), a test-time framework for defending VLMs against adversarial attacks. The core idea of DBD is to leverage multiple input transformations to construct diverse reference features, analyze their directional bias relative to the original features, and use this property to guide feature reconstruction. The overall framework consists of three main components: input transformations feature filtering, Directional Bias (DB) computation, and two- stream feature reconstruction, as illustrated in Fig. 2.
Image Transformations & Feature Filtering. Since individual transformations have inherent drawbacks, relying single transformation may produce unreliable features. For example, dom cropping is stochastic and may capture mostly background, while filtering can excessively blur important details. To improve robustness, apply diverse set of transformations to generate mul- tiple feature candidates, which leverages complementary strengths across transformations to both preserve task-relevant information and disrupt adversarial noise.
We construct an image transformation library covering diverse transformations across three do- mains: (1) Spatial domain: including random cropping, scaling, and flipping. These geometric
4

---

Published as a conference paper at ICLR 2026
Dot product
fext
Input image
Trereformeed image feet
Avecage trandoommed image
Reconetructed image
(R)
Dispersion Type
Directional Type
Daffnro Dincclar
Image Encoder
Directional Biss
Avg
Computation
17
(02(9))
Two stream Festure
Free.
High Dilacorr
Transformed images
Delett Directon
A plosto of (ngee) Aihios ilion. Aphioo font).
DB-score (5)
Text Encoder
Viger
Figure 2: Overview of the proposed Directional Bias-guided Defense (DBD). (a) Framework: multiple transformations are applied to the input image, and high-quality transformed features are retained by entropy-based filtering; then the Defense Direction and DB-score are computed for fea- ture reconstruction and classification. (b) Directional Bias (DB) Computation: the Defense Direc- tion is defined from the original feature to the average transformed feature, and the DB-score is the mean cosine similarity between the Defense Direction and each individual displacement vector. (c) Two-stream Feature Reconstruction: for high DB-score (likely adversarial images), the original fea- tures are shifted further along the Defense Direction to obtain more robust representations; for low DB-score (likely clean images), the average transformed features are used as test-time augmentation for stabilizing representations.
operations alter object position, size, and orientation, thereby disrupting the structured alignment of adversarial perturbations and weakening their effect. (2) Pixel domain: including bit-depth com- pression (quantization), JPEG comprssion-dcooprssion and additive Gaussian noise. These pixel-level modifications distort or overwrite fine-grained perturbations, making them less effective in misleading the model. (3) Frequency domain: including Gaussian, mean, and median filtering. By smoothing or suppressing high-free geeh-frequenc components, these filters reduce adversarial noise while largely preserving the semantic content of the image.
Transformed features exhibit varying quality across different transformations, so we apply a feature filtering step select the most informative and reliable representations. Following common practice in test-time adaptation (Shu et al., 2022; Chen et al., 2025), we use the entropy of the model's prediction quality metric. Specifically, given transformed images, we pre-compute their features fi = 1, 1,,,...,n) n)) and classification probabilities with CLIP, then calculate the entropy for each:
E, = =-=E log Pi.c.
(4)
where Pt.c denotes the predicted probability of the i-th transformed image feature for class We then select the A: transformed image features with the lowest entropy as high-quality features for subsequent processing:
Fret = (ffliEi I = arg min
Σ E (, JEi
(5)
Directional Bias (DB) Computation. After applying multiple transformations and feature filter- ing, we obtain set of high-quality transformed image features. Visualization (Fig. I(a)) shows that for clean images, the transformed features exhibit a dispersed pattern around the original feature. In contrast, for adversarial inputs, they consistently shift toward specific direction, presenting direc- tional bias pattern. This occurs because transformations partially mitigate adversarial perturbations, aligning the features closer to the clean feature distribution.
......,, k) and the original feature fo. we define Given a set of transformed features Fret = (ff: = 1,..., the direction vectors for each transformed feature unit vectors d, (fi fo) /|f fall2-
5

---

Published as a conference paper at ICLR 2026
and compute the Defense Direction ddef
d= - fo), ddef d
(6)
|d]
where || l| denotes the & norm (Euclidean distance). The DB-score is computed as the average cosine similarity between each direction and the Defense Direction:
Sab k. Σ dg dact),
where ( denotes the inner product. As shown in Fig. 1(b), the DB-score exhibits clear bimodal distribution for clean and adversarial images, allowing simple thresholding to separate them.
Two-stream Feature Reconstruction. Adversarial perturbations shift image features away from the true classification region, thereby inducing misclassification. Intuitively, the Defense Direction ddef could be anti-parallel to the adversarial shift, pointing features back toward their correct clas- sification region. Guided by this intuition, we reconstruct more robust features by linearly shifting the input feature along the Defense Direction. However, due to stochasticity or imperfections in the transformations, the computed Defense Direction may be inaccurate. To reduce the negative impact of inaceurate Defense Direction, we propose a two-stream reconstruction strategy based on the DB- score: (1) High DB-score stream: a high Sab indicates a likely adversarial image and a more reliable Defense Direction. In this case, we shift the feature along the Defense Direction to enhance its dis- tinction from the original. (2) Low DB-score stream: a low Sa suggests likely clean image with less reliable direction. For these examples, we use the average of transformed features as test-time augmentation for stabilizing representations. Formally, introduce threshold DB-score Sas:
[| A ||d]
Sab S T
= fo + ddef, 1=
Sa > 5
where ff is the reconstructed feature and a hyperparameter controlling the magnitude of the the distance from the average transformed features the original
feature shift. In practice, feature ||d]|z as a reference for the shift magnitude, which is then scaled by A. Finally, we use the reconstructed image feature ff to compute the predicted classification result, as given in Eq. Our DBD is summarized in Algorithm I in the appendix.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. Following prior works (Sheng et al., 2025; Li et al., 2024) the adversarial robust- ness of CLIP, we evaluate our proposed test-time DBD on ten fine- mne-grained classification datasets and five ImageNet-based out-oo-ddistributioon(OO))) benchmarks. The fine-grained datasets span di- verse domains: general objects (Caltech/0r /0/ (Fei-Fei et al., 2004)), animals (Pets (Parkhi et al., 2012)), plants (Flower102 (Nilsback & Zisserman, 2008)), vehicles (Cars (Krause al., 2013), Aircraft) (Maji et al., 2013), textures (DTD (Cimpoi et al., 2014)), satellite imagery (EuroSAT (Hel- ber et al., 2019)), human actions (UCF101 (Soomro et al., 2012)), scenes (SUN397 (Xiao et al., 2010)), and food (Food10l (Bossard et al., 2014)). For ImageNet-OOD evaluation, we use Im- ageNer (Deng et al., 2009) and four established variants: ImageNet-A (Hendrycks et al., 2021b), ImageNett- V2 (Recht et al., 2019). ImageNet-R (Hendrycks et al., 2021a), and ImageNet-S (Wang al., 2019). Since our method targets test-time adversarial robustness, we do not require access to any training sets.
Implementation details. We use official pre-trained CLIP backbones (ResNet-50 (He et al., 2016), VTT-B/32, and ViT-B/16 (Dosovitskiy et al., 2020)) as the base models. Adversarial images are gen- erated with PGD (Madry et al., 2017) under the L norm constraint. Following prior works (Sheng et al., 2025; Li et al., 2024). evaluate two threat levels. For low-strength attack, we use PGD-10 with é = 1/255 on ResPe:eeN:t:0; for high-strength attack, we use PGD-100 with e = 4/255 on CLIP-VTT-B/2 and CLIP-VVT-B/16. The step size for all attacks is e/4. For our DBD,

---

Published as a conference paper at ICLR 2026
apply 31 transformations per imput, yielding 32 images including the original, and then select 16 transformed image features via entropy-based filtering. The DB-score threshold is r = 0.8, and the feature shift magnitude is set À 2.5. Both estimated from ImageNet validation set (50k images). Experiments are conducted in Py PyTorch on RTX 3090 GPUs.
Baselines. We compare DBD with several existing methods, including adversarial fine-tuning on ImageNet (TeCoA (Mao et al., 2022)), adversarial prompt tuning on downstream datasets with 16 shots (APT (Li et al., 2024)), test-time prompt tuning (R-TPT (Sheng al., 2025)), test-time input transformation method (TTC (Xing et al., 2025)), and the original CLIP (Radford et al., 2021) models. Except for APT, which uses few-shot tuning all other methods operate in zero-shot setting. Baseline results are obtained from official reports or reproduced using official code.
Table I: Results (%) of clean accuracy (Acc.) and robust accuracy (Rob.) of various defense methods on ten fine-grained classification datasets. Robust accuracies are highlighted with gray background. Best clean accuracies are (bold), and best robust accuracies are (bold red).
Method
Acc.
CalechiD
Rob.
Ave.
Pets
Rob.
Acc.
Clzs
Rob.
Ase.
Flower102
Rob
Ace.
Aircraft
Rob.
Acc.
DTD
Rob.
Acc.
EusuATT Rob.
Acc.
UCF101
Rob.
Ave.
SUN3397
Rob.
Acc.
Fooiol
Rob.
Ace.
Ave.
Rob
PGD-10
1/235) on CLIIP-R
(LII-ReuNs:50
CLIP
80.1
$5.0
57.3
0.4
90.7
60.7
75.6
32.6
2.4
21.4
19.0
38.6
29.2
R-
65.0
19.99
56.7
58.4
42.1
6017
51,0
12.3
54.3
15.8
59.2
50.3
60.8
23.5
56.3
46.8
DHD
98.1
$6.0
69.0
68.9
888.3
21.6
56.3
65.2
29.4
$1.3
09.6
63.8
25.0
60.0
87.1
PGD
100 (c-
4/
(2203) on
CLIP
VLT B/32
CLIP
93.3
0.3
$6.6
0..0
62.2
10.0
67.0
0.0
20,5
0.0
49:9
0.0
50.8
00.0
63.6
0.0
65.7
0.3
78.7
02.0
63.7
[LB
TeClok
$1.5
46.3
04.4
16.7
13.5
1.1
30.1
9.5
0,7
0.6
29.3
12.7
13.8
11.1
34.0
0.3
34.7
6.5
22.8
3.0
32.8
11.4
APT
85.6
57.6
06.0
17.2
41.9
9.9
$4.4
47.0
29.7
6.8
47.5
21.4
67.3
23.5
53.2
19.9
46.6
10.5
33.5
0.8
36.1
22.0
89.5
29.3
39.5
30.4
66.8
46.3
74.4
R-TPT
94.8
63.3
38.5
19.6
10.1
42.7
29.9
63.1
78.5
35.1
93.8
$6.8
63.7
66.3
180.3
92.6
67.4
88.1
64.1
PGD-100
CLIIP- MPT-VI--//16
CLIP
94.2
0.0
90.3
0.0
66.2
0.0
73.0
0.0
27.1
0.0
53.2
0.0
58.7
0.0
67.0
0.0
67.9
84.2
0.0
67.9
0.0
TTC
90.3
16.3
$7.9
17.7
57.
11.3
68.5
19.5
21.7
2.5
41.5
15.3
44.7
0.6
64.8
4.9
51.5
15.7
81.0
21.2
5%.0
12.5
K-1PT
93.7
83.3
57.4
63.3
67.0
36.0
08
45.4
24.0
14.4
46.6
54.9
34.6
10.2
67.8
47.3
65.7
46.5
84.5
49.7
63.9
43.2
D80
9488
99.4
90.7
97.1
67.8
93.4
7366
97.5
29.7
71.9
54.5
93.1
44.9
93.3
68.2
95.9
69.0
07.4
84.7
99.5
67.8
93.88
Table 2: Results (%) of clean accuracy (Acc.) and robust accuracy (Rob.) of various defense meth- ods on five ImageNet-OOD datasets. Robust accuracies are highlighted with gray background Best clean accuracies are (bold), and best robust accuracies are (bold red).
Attack & Model
Method
ImageNet Acc.
Rob.
ImageNet-AA A Asc.
Rob.
Acc.
ImageNet. V2
Rob.
ImageNet.R R Acc.
ImageNet Rob.
Acc.
Avg. Rob.
(e 1/255)
PGD-10
TeCaA
CLIP
61.5
48.4
28.1
0.0
23.8
4.9
0.0
54.7
21.7
0.0
60.0
0.4
35.6
0.3
47.1
17.7
0.2
o0 CLIP-ResNet50
R-TPT
60.8
47.3
28.0
14.2
54.7
41.6
57.7
46.6
34.0
26.0
47.0
35.1
DBD
63.1
94.5
23.0
89.0
55.7
93.0
63.0
73.7
48.6
88.8
CLIP
64.4
0.0
31.1
0.0
57.2
0.0
68.3
0.0
42.5
0.0
52.7
0.0
PGD-100
TeCoA
39.5
9.7
4.2
0.3
32.4
7.3
38.0
12.7
18.5
7.6
26.5
7.5
(c 4/255)
TTC
35.4
25.7
27.5
8.4
41.3
21.4
53.4
28.1
29.8
12.8
37.5
19.3
CLIP-VIT-B/32
R-TPT
64.2
40.4
36.6
11.0
58.0
34.3
70.0
47.9
41.7
23.6
54,1
31.4
DBD
66.3
95.3
32.9
95.3
59.1
94.33
71.7
98.2
45.1
89.5
55.0
94.5
100
CLIP
69.6
0.0
50.6
0.0
63.4
0.0
77.1
49.1
0.0
62.0
0.0
(c 4/255)
TTC
37.8
17.4
46.6
9.9
48.9
16.1
63.3
12.4
38.5
1.9
47.0
11.5
CLIP-VII-B/6
R-TPT
69.4
46.6
$7.9
20.7
63.9
40.2
77.0
57.6
47.9
30.3
63.2
39.1
DBD
71.1
97.7
52.1
98.9
64.7
97.3
79.5
51.1
95.3
63.7
97.7
4.2
MAIN RESULTS
Results on fine-grained datasets. We evaluate the adversarial robustness of DBD on ten fine- grained classification datasets, with results summarized in Table 1. Under the CLIP-ViT-B/3 ViT-B/32 and PGD-100 (e = 4/255) setting, the original CLIP model demonstrates strong zero-shot classification performance (63.7%) but is almost entirely vulnerable to adversarial attacks (0.0%). Adversarial fine-tuning (TeCoA) improves defense to 11.4%, but at the cost of reduced zero-shot performance on clean images (63.7% + 32.8%). Adversarial prompt tuning (FAP) further enhances robustness (0.0% + 22.0%) while mitigating the drop in clean accuracy (63.7% + 56.1 56.1%) 1%), though its few-shot setting introduces data dependency. Test-time input transformation method (TTC) achieves mod- erate robustness on CLIP-ViT-B/AS (0.0% + 27.2%) but are sensitive to model architecture, with only 12.5% on
Previous state-of-the-art test-time prompt tuning method (R-TPT)
7

---

Published as a conference paper at ICLR 2026
attains better robustness (0.0% - 38)1%) and largely preserves zero-shot performance on clean im- ages (63.7% - 60.2%). 2.)...... In contrast. ours DBD maintains zero-shot performance close to the original and achieves substantially
CLIP across all three backbones (even exceeding it on higher robustness on adversarial examples, reaching 93.8% on CLIP-ViT-B/16, significantly sur- passing both R-TPT and the performance on clean examples.
Results on ImageNet-OOD datasets. Results on ImageNet-OOD benchmarks are summarized in Table The original CLIP model demonstrates strong robustness distribution shifts but remains highly vulnerable to adversarial attacks. Notably, our DBD method not only preserves but slightly improves zero-shot classification on clean images (e.g., 52.7% 55.0% on CLIP- CELP-ViT-BB ViT-B))), which we attribute to the combination of diverse image transformations and autropy-based feature filtering that produces more robust and accurate features. Across all three backbones, DBD substantially outperforms the previous state-of-the-ar R-TPT on adversarial examples, achieving up to 97. on CLIP- IL.P-VVT-B/16, a performance that even surpasses the classification accuracy on clean examples.
Discussion. Remarkably, the classification accuracy on adversarial images significantly exceeds that on clean images. This counterintuitive result suggests that the generation of adversarial ex- amples guided by ground-truth labely implicitly encodes directional priors about the true decision boundary. Our method leverages this by applying multiple image transformations to estimate the Defense Direction, then linearly shifting features along it to reconstruct robust representations.
4.3 MORE ANALYSIS
Table 3: Robust accuracy (%) under PGD-100 (e 4/255) on CLIP-ViT-B/16 using pseudo-labels across six fine-gras gre-gga datasets. The last row shows the clean accuracy as reference.
Method
Caltech101
Pets
Flowcr102
Aircraft
DTD
UCF101
Avg.
CLIP
1.7
3.1
2.7
1.5
4.2
4.5
2.9
R-TPT
84.1
66.1
49.8
18.2
37.1
52.7
51.3
DBD
94.1
88.6
72.3
26.3
53.1
66.5
66.8
Clean
94.2
90.3
73.0
27.2
53.2
67.0
67.5
Analysis under PGD attack with pseudo-label. The previous experiments demonstrate that when PGD generates adversarial examples using ground-truth labels, our method achieves robust accuracy which significantly exceeds clean accuracy. To further validate its effectiveness without relying on ground-truth labels, we consider pseudo-label setting. Specifically, we use CLIP's own predictions on clean images as pseudo-labels to guide PGD in generating adversarial examples. As shown in Table 3, under this setting the vanilla CLIP model remains highly vulnerable. In contrast, our method consistently outperforms R-TPT and achieves robust accuracy comparable clean accuracy (e.g., 67.5% 66.8% on CLIP-VIT-B/10 This indicates that our method is able to fully leverage the directional prior information carried by adversarial examples.
Analysis under various attacks. To demonstrate the generality of our method, we evaluate DBD and baseline methods under additional adversarial attacks, including FGSM (Goodfellow et al., 2014), CW (Carlini & Wagner, 2017). AutoAttack (AA) (Croce & Hein, 2020b), and four com- ponent attacks AA: Square Attack (Andriushchenko et al., 2020), targeted FAB (Croce & Hein, 2020a), untargeted APGD-CE, and targeted APGD-DLR. We evaluate various attacks the same six fine-grained datasets in Table 3, with the results summarized Table 4. DBD consistently demonstrates robust defense performance across all attack types, significantly outperforming R- TPT. Notably, under AA on CLIP-ViT-B/66, DBD achieves an average robust accuracy of 69.8%, surpassing the average clean accuracy 67.5%.
Table 4: Average robust accuracy (%) across six fine-grained datasets under various attacks on CLIP-VIT-B/16. All attacks are conducted under the fxc norm with perturbation budget 4/255.
Method
FGSM
CW
AA
FAB
Square
APGD-CE
APGD-DLR
Avg.
CLIP
11.8
1.0
0.0
12.9
11.7
0.1
R-TPT
38.9
55.6
45.0
59.5
59,2
45.0
48.8
50.4
DBD
75.2
69.1
69.8
69.3
65.1
69.8
68.5
69.2
8

---

Published as a conference paper at ICLR 2026
Geometric Verification of the Proposed Defense Direction. To validate our hypothesis that the Defense Direction genuinely points toward the correct decision region, we conduct a geometric analysis by measuring the cosine similarity between the estimated Defense Direction and two critical reference directions. The first is the Clean Direction, defined as the vector from the adversarial image's feature to its corresponding clean (unperturbed) counterpart's feature. The second is the Class Centroid Direction, defined as the vector from the adversarial image's 's feature to the centroid of features belonging to the true class (computed using correctly classified clean images).
Table 5 presents the average cosine similarities across multiple fine-grained datasets under the PGD- 100 (c=4/255) attack on CLIP- ViT-B/16. The results demonstrate that our proposed Defense Direc- tion exhibits extremely high similarity (+s 0.95) with the Clean Direction and substantial similarity (=x 0.90) with the Class Centroid Direction. These high cosine similarities provide strong geometric evidence that the Defense Direction indeed aligns closely with both the clean feature direction and the correct class centroid, supporting our hypothesis that the Defense Direction points back toward the correct decision region.
Table 5: Average cosine similarity between Defense Direction and reference directions across datasets under PGD-100(c 4/255) attack CLIP-ViT-B/IE
Reference Direction
Pets
Caltech101
Food101
Cars
ImageNet
Clean Direction
0.957
0.945
0.939
0.943
0.951
Class Centroid Direction
0.932
0.917
0.898
0.905
0.892
4.4
ABLATION STUDY
92.1
76.2
Robust Accuracy
Reconstructed feat.
Overal Acturacy
10
Average transfommed
Original feat.
6.0
1.0
1.5
255
3.0
0.4
0.5
0.8
Shift magnitude factor (A)
(b) DB-score threshold
Figure 3: (a) Average robust accuracy (%) across 15 datasets under three attack settings for different values of the feature shift magnitude factor x. (b) Average overall accuracy (%) for different values of the DB-score threshold т, computed by averaging both clean and robust accuracies across all datasets and attack settings.
Ablation of the feature shift magnitude factor X. To evaluate the impact of the shift magnitude factor À on the performance of DBD,, we test three attack settings across 15 datasets. The average robust accuracy of all is shown in Fag.3(a). When x = 0, the reconstructed feature reduces to the original feature, yielding nearly no defense. With average of transformed features, achieving an average robust accuracy of 34.8%. At A = 2.5, DBD reaches 92.15 % accuracy, demonstrating that linearly shifting the original feature along the Defense Direction reconstructs robust features that align with the correct class. This highlights the reliability of the Defense Direction identified by DBD.
1.0, the reconstructed feature becomes the
Ablation of the DB-score threshold r. DBD employs DB-score threshold r to distinguish high- score examples (likely adversarial) from low-score examples (likely clean). We evaluate the effect of r under three attack settings across 15 datasets, reporting the overall mean accuracy obtained by averaging both clean and adversarial performance across all datasets and attack settings (Fig.3(b)). The results show that setting 0.8 yields the best overall performance. At this threshold, DBD successfully reconstructs robust features for the majority of adversarial images, while simultane- ously avoiding inaccurate shifts in most clean examples, thereby achieving well-balanced trade- off between robust and clean accuracy. Additional results of the detection performance based on the DB-score threshold are shown in Fig.5 of the Appendix.

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study. Results (%) of clean accuracy (Acc.) and robust accuracy (Rob.) are average over 15 datasets and three attack settings. The left five columns correspond to different im- age transformation types, while the middle columns represent DBD mechanisms (linear feature shifting and OB-score-based thresholding).
Random Crop-Resize- Resize-Flip Flip
Bit-depih Reduction
JPBG
Guassiun Noise
Image Filtering
Feature Shift
DB-score Threshold
Compression
Acc.
60.6
0.0
53.0
24.1
82.4
55.7
88.4
36.1
/
43.9
87.3
/
61.5
34.8
/
58.9
92.1
61.2
Ablation of DBD mechanisms. We conduct ablation experiments on all DBD components, with results average over 15 datasets and three attack settings (Table 6). Our analysis reveals four key findings: (1) Using a single type of image transformation with feature shifting provides strong ad- versarial defense but reduces clean image performance. (2) Aggregating multiple transformations without linear shifting gives the best clean accuracy but weak defense. (3) Adding linear shifting top of multiple transformations substantially improves adversarial robustness, the cost of some clean performance. (4) Applying the A--score--aase threshold to handle high-score and low-score examples separately achieves the best trade-off between clean accuracy and adversarial robustness.
CONCLUSION
In this work, found that adversarial examples exhibit strong directional bias under multiple in- put transformations, in contrast to the dispersed behavior clean examples. Building on this obser- vation, we proposed Directional Bias-guided Defense (DBD), a test-time framework that leverages the directional bias to reconstruct robust features through Defense Direction estimation and a two- stream reconstruction strategy based on the proposed DB-score. Experiments on 15 datasets under three attack settings demonstrate that DBD achieves state-of-the-ee adversarial robustness while preserving zero-shot performance on clean images. Remarkably, robust accuracy even surpasses clean accuracy, highlighting that adversarial perturbations implicitly encode directional priors about the true decision boundary. We believe that our work sheds light on new perspectives training- free defenses in VLMs.
ACKNOWLEDGMENTS
This work supported by the National Defense Technology Basic Scientific Research Project (Grant No. J57L2003316AA00).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. European conference computer vision, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity: Circumventing defenses adversarial examples. In International conference on machine learning, pp. 274-283, 283. PMLR, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446-461. Springer, 2014.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 57. Icce, 2017.
Si Chen, Yujia Chen, Xiaotian Yin, Xin Liu, Huakai Lai, and Tianzhu Zhang. Paf: Prototype adaptive fusion for test-time adaptation of vision- languuge models. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 3007-3016, 2025.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint (xXi::2209.14688 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proc. CVPR, 2014.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference machine learning, pp. 1310-1320. PMI.,, 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International conference on machine learning. pp. 2196-2902222 PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter-fie free attacks. In ICML, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proc. CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A study of the effect of jpg compression on adversarial images. arXiv preprint arXiv: 00853, 2016.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proc. CVPR Workshops, 2004.
Weiwei Feng, Baoyuan WU, Tianzhu Zhang, Yong Zhang, and Yongdong Zhang. Meta-attack: Class-agnostic and agnestig physical adversarial attack. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 7787-7796. October 2021.
Weiwei Feng, Nanqing Xu, Tianzhu Zhang, and Yongdong Zhang. Dynamic generative targeted attacks with pattern injection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16404-16414, June 2023.
Weiwei Feng, Nanqing Xu. Tianzhu Zhang, Baoyuan Wu, and Yongdong Zhang. Robust and gen- eralized physical adversarial attacks via meta-gan. IEEE Transactions on Information Forensics and Security, 19:1 1 1225 2024a. doi: 10. 1119/TFFS.2023.330

---

Published as a conference paper at ICLR 2026
Weiwei Feng, Nanqing Xu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Enhancing cross-task transferabilityy of adversarial examples via spatial and channel attention. IEEE Transactions on Multimedia, pp. 1-15, 2024b. doi: 10.1 N...1090TMMM222433499993
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using imput transformations. arXiv preprint arXiv:1711.00111 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -fitt--diittibuttion generalization. In Proc. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proc. CVPR, 2021b.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. URL https: htt////xxvv.rr/////////
2202222222 05918.
Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv:2010.01950. 2020.
arXiv preprint
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for ane-grained categorization. In Proc. ICCV Workshops, 2013.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Lin Li, Haoyan Guan, Jianing Qiu, and Michael Spratling. One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24408-24419, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks, arXiv preprint arXiv:1706.06600 06083, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained 2013.
visual classification of aircraft. arXiv preprint
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proc. ICVGIP. 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proc. CVPR, 2012.
12

---

Published as a conference paper at ICLR 2026
Sarah Pratt, lan Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF international conference computer vision, pp. 15691-15701, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021.
Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Proc. ICMI., 2019.
Christian Schlarmann and Matthias Hein. On the adversarial robustness multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 3677-3685, 2023.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint arXiv:240..12336 2024.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances in neural information processing systems, 32, 2019.
Lijun Sheng, Jian Liang, Zilei Wang, and Ran He. R-tpt: Improving adversarial robustness of vision- language models through test-time prompt tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19958-299611 2025.
Manli Shu, Weili Nie. De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-142899 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6799 2013.
Baoshun Tong, Hanjiang Lai, Yan Pan, and Jian Yin. On the zero-shot adversarial robustness of vision-language models: A truly zero-shot and training-free approach. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19921-19930, 2025.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Proc. NeurIPS, 2019.
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24502-24511, 2024.
Xin Wang, Kai Chen, Jiaming Zhang, Jingjing Chen, and Xingjun Ma. Tapt: Test-time adversarial prompt tuning for robust inference in vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19910-19920, 2025.
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint 1XXX::0001.03994, 2020.
Jianxiong Xiao, James Hays. Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pallern recognition, pp. 3485-3492. IEFE, 2010.
Cihang Xie, Jianyu Wang. Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint /7X 2017.
13

---

Published as a conference paper at ICLR 2026
Songlong Xing. Zhengyu Zhao, and Nicu Sebe. Clip is strong enough to fight back: Test-time counterattacks towards zero-shot adversarial robustness of clip. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15172-15182. 2025.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning. pp. 7472-7482. PMLR, 2019.
Jiaming Zhang, Xingjun Ma, Xin Wang. Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, and Jitao Sang. Adversarial prompt tuning for vision-language models. In European conference computer vision, pp. 56-72. Springer, 2024a.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE transactions pattern analysis and machine intelligence, 46(8))5625-5644, 2024b.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai- Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances Neural Information Processing Systems, 16955411155411 2023.
Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems, 37: 3122-3156, 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ALGORITHM
We provide a detailed pseudocode of our proposed DBD in Algorithm 1.
Algorithm 1: Directional Bias-guided Defense (DBD) for VLMs
Input: Input image I, pre-trained CLIP vision encoder E classification head H. transformation library T, DB threshold т, shift factor X
Output: Robust feature f and classification result y
Step 1: Image Transformations & Feature Filtering
for transformation t do
L - t(x) : f. (z+)
1/ Apply transformation // Extract feature
- H(fe):
// Compute classification probabilities // Compute entropy
E, - - PePro Pr. log Pr,c
Fref l features with lowest F.
Step 2: Directional Bias Computation
fo (x):; for fi E Fiet do
// Original feature
L d, +-
- foll2:
// Unit displacement vector
d + Ek= (fi fo):
// Mean displacement
daet d/||d|2:
// Defense Direction
Sab + Ek (ds daet)
// DB-score
Step 3: Two-stream Feature Reconstruction & Prediction
if Sab then
+ fo + |d] daet
// Likely clean
else
f + fo + ||l|2 daef
// Likely adversarial, enhanced shift
y f- argmax :
// Compute final prediction
A.2 IMPLEMENTATION DETAILS
Transformations. In our experiments, DBD applies = 31 transformations per input, yielding 32 images including the original. We provide more details various image transformations used in DBD in Table 7. (1) For spatial domain transformations, each transformation includes random cropping, scaling, and flipping. Due the stochastic nature of these operations, we perform 16 transformations per input. (2) For pixel domain transformations, we implement bit-depth reduction, compressing the original 8-bit image to bits using three quantization methods: floor, round, with three quality levels (50,
and ceil. In addition, we apply JPEG cimprrssion-decoopressee 60, 75) to increase diversity. Random Gaussian noise with amplitude 7 = 0.1 is also added, with six stochastic realizations per input. (3) For frequency domain transformations, we apply Gaussian, mean, and median filtering with kernel size 5.
In the ablation study (Table 6 and Fig. 4), the resulting average features from Random Crop-Resize Flip (16 transformations). Bit-depth Reduction (3 operations). JPEG Compression (3 operations), Gaussian Noise (6 operations), and Image Filtering filters) used for classification.
15

---

Published as a conference paper at ICLR 2026
Text Prompts. For text prompts, use mix of hand-crafted prompts and SPPT-3-generater prompts provided by CuPL (Pratt et al., 2023), and average text features over multiple prompts per class.
Attacks. All attacks are implemented using the torchattack (Kim, 2020) library. For PGD and FGSM attacks, we follow the baseline R-TPT (Sheng et al., 2025) configuration by using cross- entropy loss in untargeted mode. For the CW attack (Carlini & Wagner, 2017), we use a learning rate of 0.01 with the Adam optimizer. For AutoAttack (AA, (Croce & Hein, 2020b)), we use the standard mode, which includes four components: untargeted APGD-CE (1 restart), targeted APGD- DLR (10 target classes). targeted FAB (10 target classes, (Croce & Hein, 2020a)), and Square Attack (5000 queries, (Andriushchenkoo et al., 2020)). To remain consistent with the baseline setup, EOT was not applied. When verifying these 4 components, we keep the parameters unchanged.
Table 7: Details of image transformations used in DBD.
Domain
Transformations
Spatial
Random cropping-scaling
(16 times)
Bit-depth compression (quantization): floor, round, ceil (bits 3)
Pixel
JPEG compresion-dccomrrsiitt quality - 50, 60, 75
Add Gaussian noise: = 0.1 (6 times)
Gaussian filter: kernel. 1.s size 5
Frequency
Mean filter: kernelsiize
Median filter: kernel.sizo
A.3 DATASETS
We evaluate our method on 10 fine-grained classification datasets and 5 ImageNet-OOD datasets. Table summarizes their detailed information, including their content, number of categories, num- ber of images and corresponding hand-crafted prompt.
Table 8: Introduction of all datasets involved in experiments.
Datasct
Description
# Classes
# Test
Hand-crafted Prompt
Caltech101
Object images
100
2,465
photo of a [CLASS]
Pets
Pet images
37
3,669
photo of [CLASS], a type of pet
Cars
Car images
196
8,041
photo of [CLASS]
Flower102
Flower images
102
2,463
photo of [CLASS], type of flower
Aircraft
Aircraft images
100
3,333
photo of [CLASS], type of aircraft
DTD
Describahle textures images
47
1,692
[CLASS] texture
AUrOSAT
Sentinel-2 satellite images
10
8,100
centered satellite photo of [CLASS)]
UCF101
Human action images
101
3,783
photo of person doing [CLASS]
SUN397
Scene recognition images
397
19,850
photo of [CLASS]
Food101
Foud images
101
30,300
photo of [CLASS], type of foud
ImageNet
Object and scene images
1,000
50,000
a photo of a [CLASS]
ImageNet-A
Adversarially filtered images
200
7,500
photo of a [CLAASS]
ImageNet-Ve V2
New test images
1,000
10,000
photo of [CLASS]
ImageNet-R R
Rendered images
200
30,000
photo of [CLASS]
ImageNet-S
Sketch-style images
1,000
50,889
photo of [CLASS]
A.4 EXPERIMENTS
A.4.1 DETAILED RESULTS
Detailed results under PGD attack with pseudo-label. Due space constraints, Table 3 in the main text reports only a subset of the results. Here we provide the complete version in Table 9, which includes results six fine-grained datasets and three attack settings.
Detailed results under various attacks. We evaluate DBD and baseline methods under additional adversarial attacks, including FGSM (Goodfellow et al., 2014), CW (Carlini & Wagner, 2017), and
16

---

Published as a conference paper at ICLR 2026
Table 9: robust accuracy (%) under three PGD attack settings using pseudo-labels on six fine- grained datasets.
Amaak & Model
Method
Caltech101
Pets
Plower102
Aireraft
DTD
UCFIOI
Arg.
Clean
89.1
85.0
65.9
19.6
48.5
59.7
61.3
PGD PGD-100 (c 1/255)
CLIP
6,4
1.6
8.2
6.3
on ResP-
64.4
16.9
47.5
Clean
91.3
86.6
67.0
20.6
49.9
63.6
63.5
PGD- 1/255)
CLIP
2.3
3.4
3.8
0.8
3.8
4.9
5.2
R-IPT
79.9
62.7
43.1
13.9
32.9
49.9
47.1
Clean
94.2
90.3
73.0
27:2
53.2
67.0
67.5
PGD-100 1/255)
CLIP
1.7
3.1
2.7
1.5
4.2
4.5
2.9
CLIP-VIT-B11
R-TPT
84.1
66.1
49.8
18.2
37.1
52.7
51.3
Table 10: Results (%) of clean accuracy (Acc.) and robust accuracy (Rob.) of various attacks on six fine-gen tine-grained classification datasets.
Peb
Aircraii
R-1PT
Attaddo on CLII--RoNaat00
19.2
33.5
72.9
CW
5.1
79.6
50.5
538
16.7
96.5
0.9
$1.2
68.6
14.9
18.5
2.2
33.7
50.1
2.4
51.5
62.0
2.1
$2.9
62.3
AA
0.9
81.0
91.4
24
70.8
96.0
3V0
53.3
07.6
15.2
21.1
0.4
35.2
52.2
0.0
53.8
64.6
0.2
52.5
03.8
1.0
85.
K7.7
56.5
96.5
1.3
15.6
13.5
64.4
18.1
$6.7
76.6
47%
APGDSCT
3.7
81.0
91.4
24
76.8
K5.9
3.4
512
67.6
DV5
15.2
21.0
D37
7500
42.5
D.0
51.5
68.6
0.5
67.5
61.8
APGI-DUU R
82.1
91.3
79.0
DVS
66.1
19.0
75.8
51.7
0.1
54.1
8.3
51.6
61.5
CLII--VT-R/13
75.1
K2.9
7.1
09.8
1-1
09.1
77.6
51.0
40.7
61.00
11.5
14.8
05.5
AA
05.3
210
09.9
28.3
13.01
52.6
110
46.7
013
05.8
FAB
06.5
37.8
05.5
17.9
78886
106.
17.5
09.3
17.8
22.6
14.5
39.5
53.2
16.5
59.1
09.00
18.9
56.9
05.5
Square
36.7
02.2
72.6
80.5
17.4
08.3
8.9
59.9
64.11
164
24.5
APGD-DER
36
79.6
92.3
3/
60.0
57.1
3.0
40.4
68.5
D/
14.8
22.7
33.6
51.5
0.5
50.5
64.5
0.3
47.4
64.5
Altacla
FGSM
+7.2
75.6
92.5
7.1
47.9
05.3
3.3
353
50.5
0.2
10.1
53.5
13.9
20.4
61.5
9.3
35.1
67.8
13.5
38.9
75.2
CW
3.0
67.3
94.7
0.4
73.4
92.1
09
73.9
0.2
20.0
29.6
35.5
50.2
1.4
51.
09.2
1.2
55.6
09.1
FAB
00.1
92.
58.5
18.9
Square
22.7
99
05.5
8.0
77.
57.5
1.9
02.9
68.5
0.5
21.2
347
3.8
41.
50.4
6.9
62.5
03.3
8.4
59.3
651
ANGIAC
312
31.7
500
49.2
69.8
AutoAttack (AA) (Croce & Hein, 2020b). AA a stronger, ensemble- attack that combines targeted FAB (Croce & Hein, 2020a). Square Attack (Andriushchenko al., 2020), untargeted APGD-CE (Auto-PGD with Cross-Entropy loss), and targeted APGD-DLR (Auto-PGD with Dif- ference of Logits Ratio loss), incorporating both gradient-based and gradient-free strategies prehensively challenge model robustness. We also separately evaluate defense performance against each of these four constituent attacks. For our DBD, we 7 0.6 and À 2.5 (with
for FGSM). We assess the effectiveness of various attacks on six fine-grained datasets across three attack settings, and detailed results are shown in Table 10.
None
Randorn crop-resize-flip
Bit-depth reduction
JPEG
Gaussian Noise
Filtering
-0
10
Robust
Robust
Robust
(a) PGD-10 10/p
1/255) CLIP.ResNette
(b) PGD-1006-
4/255) on CLIP-VITB/32
(c) PGD-100// 4/255) on CLIP-VIT-B/6
Figure 4: Average results (%) of clean accuracy (Clean Acc.) and robust accuracy (Robust Acc.) for various types of transformations across 15 datasets under three attack settings.
17

---

Published as a conference paper at ICLR 2026
A.4.2 MORE ABLATION STUDIES
Detailed analysis of various image transformations. We ablate the individual image transforma- tions used in DBD to assess their standalone effects (as Fig.4). JPEG compresiionddcomppess preserves clean accuracy while providing moderate defense against low-strength attacks, though its effectiveness diminishes as attack strength increases, Random crop-resize- slightly improves clean aceuracy and shows robustness across different attack strengths and model backbones. Adding Gaussian noise can yield strong defense in certain cases but substantially degrades clean By combining multiple transformations, DBD leverages their complementary strengths to gener- ate more reliable features across diverse scenarios and model variants: moreover, integration of transformations mitigates the risk of defenses being circumvented by attacks tailored a single transformation.
accuracy.
More ablation study on the DB-score threshold т. We evaluate the detection performance of adversarial examples using the DB-score. Specifically, we assess the effect of threshold T under three PGD attack settings across 15 datasets, reporting both the mean detection accuracy and mean F1-score averaged over all datasets and attack settings. Results presented in Fig.5. The results show that = 0.8 achieves oear-optimal detection performance, and the metric remains stable in its neighborhood. This supports choice and confirms that the DB-score provides reliable signal for distinguishing adversarial from clean inputs.
1.0
Detection Accuracy
Detection F1-score
0.8
0.6
0.4
0.2
0.0
0.4
0.5
0.6
0.7
0.8
0.9
1.0
DB-score threshold (r)
Figure 5: Average results of detection accuracy and F1-score for various types of transformations across 15 datasets under three attack settings.
Adaptive Attack with BPDA and EOT. While our primary evaluation based on the threat model described in Section 3.1, additionally evaluate against adaptive, defense-aware attacks assess potential vulnerabilities. Based on PGD attack, we implement BPDA (Backward Pass Differentiable Approximation, Athalye al. (2018)) combined with EOT (Expectation Over Transformation, (Xie et al., 2017)) to approximate gradients through our differentiable DBD pipeline. During attack optimization, non-differentiable components are replaced with identity functions in the backward pass, while gradients are averaged over multiple stochastic forward passes.
We evaluate on Caltech101 dataset using strong adaptive attacks: PGD-10 (c=1/255) against CLIP- ResNet50 and PGD-100 (e=4/255) against CL.I--VI-BLL attackers explicitly optimize through the full DBD pipeline, robust accuracy degrades significantly (to 50.79% and 1.29%, respectively). Notably, these adversarial images generated buy adaptive attacks also perform substantially worse against the original CLIP model compared to standard attacks, suggesting overfitting to the defense mechanism.
Results in Table 11 show that when
Critically, under the adaptive attack (PGD-100), the average DB-score remains high (0.89), with approximately 81% % of adversarial samples exceeding detection threshold (r=0.80). This reveals an important insight: the directional bias pattern persists under adaptive attacks, but the estimated direction is manipulated point away from the true class. In other words, while adaptive attacks
18

---

Published as a conference paper at ICLR 2026
subvert the defense functionality by distorting the Defense Direction, they cannot eliminate the underlying directional bias signal, making such attacks still detectable through our DB-scor metric.
Table 11: Robust accuracy (%) under adaptive attacks with BPDA+EOT on Caltech101.
Attacks
Original CL.IP
CL.IP with DBD
PGD-10 10 (c=1/255)
84,26
50.79 1.29
PGD-100 1:0 (c=4/255)
81.05
Analysis of attacks using different loss functions. To evaluate the robustness of method in more practical and challenging settings where ground-truth labels are unavailable during infer- ence, we consider margin- based adversarial attacks, which does not rely on true class labels but instead maximizes the difference between the highest non-tar non-target logit and the target logit. We conduct PGD-100 attacks (c = 4/255) using both cross-entrops (CH) loss and margin loss, with either ground-truth labels are shown in Table 12. Notably, DBD achieves 99.76% robust accuracy under margin-loss attacks with ground-truth guidance, on par with its performance under CE-loss attacks (99.39%) More importantly, when using pseudo-labels (i.e., no access to ground truth test time), DBD maintains strong robustness (94.24% vs. 94. under CE-loss), demonstrating that the directional bias ploited by our method is not specific to a particular attack loss formulation. In contrast, the vanilla CLIP model collapses under all attack variants (<3% robust accuracy). These results confirm that DBD's defense mechanism generalizes across different adversarial objectives and remains effective in realistic label-free scenarios.
pseudo-labels, on the Caltech101 dataset. Results
Table 12: Robust accuracy (%) under PGD-100 attacks (e 4/255) on Caltech101 using different loss functions and label sources.
Attack Setting
CLIP
DBD (ours)
CH-loss Ground-truth
0.04
99.39
CH-loss + Pseudo-lahel
1.66
94.12
Margin-loss + Ground-truth
0.04
99.76
Margin-loss + Pseudo-label
2.80
94.24
Analysis of Correct vs. Incorrect Clean Predictions. We separate samples into those correctly classified by CLIP on clean images versus those misclassified, and evaluate robust accuracy under PGD-100 attacks (e 4/255) on robust accuracy on initially correct samples demonstrates that DBD effectively preserves accurate predictions by leveraging directional information from adversarial perturbations. More importantly, the high robust accuracy on initially misclassified samples (up to 95.83% shows that DBD genuinely corrects errors by exploiting the directional prior embedded in the attack. This dual capability explains why our overall robust accuracy exceeds clean accuracy, revealing an important insight: ground-truth guided attacks inadvertently leak information about the correct class direction.
Results shown in Table 13. The near- eerreeefeect
Table 13: Robust accuracy (%) under PGD-100 attacks on correctly and incorrectly predicted clean samples.
Caltech101
Pets
Flower102
DTD
UCF101
Correct clean predictions
99.61 %
97.83% 90.14%
99.11%
99.00%
98.74%
Incorrect clean predictions
95.83%
93.08%
84.60%
90.14%
Analysis of inference efficiency. We compare DBD's inference efficiency with baseline meth- ods. Table 14 reports the training time of APT (Li et al., 2024) and the inference time of test-time defenses R-TPT (Sheng et al., 2025), TTC (Xing et al., 2025), and DBD on UCF101 using CLIP- ViT-B/32. Training-time time defenses ineur substantial training costs, while test-time defenses avoid this but increase inference overhead. DBD achieves a favorable balance between robustness and efficiency, outperforming R-TPT in both accuracy and inference speed.
19

---

Published as a conference paper at ICLR 2026
Table 14: Running time and robust accuracy (%) of different defense methods against adversarial attacks on UCF101 dataset using CLIP-ViT-B/32 APT is evaluated with 16 shots, while R-TPT and DBD are evaluated with 32 views.
Method
Stage
Running time
Rob.
APT
Training time
22m47s / 200 cpochs
18.9
TTC
Test time
0.008s/ / image
27.6
R-TPT
Test time
0.18 image
44.3
DBD
time
0.025s/ image
USE OF LLMS
In this work, used ChatGPT to assist in polishing the writing of this paper, focusing primarily on improving clarity, grammar, and style. The model was not involved in the generation of ideas or experimental designs, All the concepts, analyses, and conclusions presented are entirely our own.
20


Reviews:
[{"review_id": "ZXxyzP9T5x", "reviewer": "Reviewer_mq6U", "summary": "This paper proposes DBD (Directional Bias-guided Defense), a training-free, test-time defense framework that enhances the adversarial robustness of vision-language models such as CLIP. The authors identify an interesting and consistent phenomenon: adversarial perturbations in CLIP’s feature space tend to produce a dominant shift direction, whereas clean samples exhibit more dispersed feature transformations. Based on this observation, the authors design DBD to compute a Directional Bias (DB) score and reconstruct image features along a “Defense Direction” that counteracts adversarial perturbations. For inputs with high DB scores (likely adversarial), DBD shifts features along the Defense Direction; for low-DB inputs (likely clean), it averages multiple transformed views. Extensive experiments on 15 datasets demonstrate that DBD achieves SOTA while maintaining or even exceeding clean accuracy.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Novel empirical discovery:  The finding that adversarial perturbations exhibit a strong, consistent directional bias in the feature space is insightful and clearly visualized (Fig. 1). This observation provides an interpretable foundation for designing test-time defenses.\n- Excellent algorithmic performance:  DBD is training-free and efficient, operating entirely at test time without requiring adversarial training or prompt optimization. It achieves strong generalization and robustness across 15 datasets and multiple attack types, and shows remarkably high robustness accuracy under PGD attacks.\n- Comprehensive analysis: The experimental analysis is thorough, particularly the Analysis under PGD attack with pseudo-label experiment, which convincingly demonstrates that Directional Bias-guided Defense can effectively recover information about the pre-attack image features.", "weaknesses": "- Lack of theoretical explanation: The paper does not provide a clear theoretical justification for the observed behavior, especially in the counterintuitive case where robust accuracy surpasses clean accuracy. A more rigorous analysis is needed to support this surprising result.\n- Limited discussion of attack-specific behavior:  The results suggest that under PGD attacks, DBD almost perfectly reconstructs pre-attack image features, while this phenomenon does not hold for other attack methods. What causes this discrepancy? Would DBD still achieve robustness higher than clean accuracy under non-PGD attacks? The paper should provide a more detailed analysis and comparison across different attack types, rather than emphasizing its exceptional performance only under PGD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OZJ54ift2", "reviewer": "Reviewer_5Eqf", "summary": "The paper proposes DBD, a test-time defense method for vision-language models such as CLIP.\nThe key idea is that when applying multiple transformations to an input, clean images produce scattered features in the latent space, while adversarial images produce features that move in a consistent direction. This directional bias is quantified through a DB-score, which measures the alignment of feature shifts across transformations. The defense then routes the input based on this score.\nThe approach is training-free and evaluated on multiple datasets, showing improved robust accuracy compared to standard CLIP.", "questions": ". Could you verify that the Defense Direction truly points toward the correct class? A simple geometric check.\n\n. How does DBD perform under adaptive, defense-aware attacks? Since the pipeline is differentiable, please evaluate with BPDA or EOT to see if the “directional bias” still holds.\n\n. The method depends heavily on τ and λ. How were these calibrated, and do they generalize across datasets?\nA small sensitivity or AUROC plot for the DB score would help confirm stability.\n\n. For the reconstruction step, what is the reasoning behind the specific scaling rule in Eq. (8)?\n\n. The paper reports robust > clean accuracy in several cases. Could you show where this gain comes from? I mean, does it persist if you use the same transformation pipeline without the DB shift?\n\n. How sensitive is robustness to the number of transformations used? It would be useful to know how the defense performs if the set is smaller.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": ". The paper is well motivated and touches on a very active area. \n\n. The DB score is an interesting metric to characterize how adversarial perturbations behave across transformations.\n\n. The method is simple to plug into existing models, requiring only forward passes under different transformations.\n\n. The paper includes multiple datasets and compares against several baselines.", "weaknesses": ". My main concern is that the paper’s central claim, which is analyzing how and why transformations mitigate adversarial effects, is not actually supported by the experiments. The analysis is mostly descriptive, as they visualise CLIP features and observe that clean samples scatter, while adversarial ones align along a single direction. However, this remains a qualitative observation, rather than a real explanation. The proposed Defense Direction is repeatedly described as being “anti-parallel to the adversarial perturbation” and “pointing back toward the correct class center,” but this is never verified. A simple geometric check: cosine similarity between the Defense Direction and (a) the true class text embedding, (b) the nearest clean class centroid, or (c) the adversarial gradient, would make the claim much more credible.\nThe overall pipeline (multiple transformations + reconstruction) is not new. They follow the same pattern as many prior test-time defenses (transform, measure consistency, and reconstruct), but reinterpret it as a “directional bias”.  So while the framing is novel, the underlying idea is not fundamentally different, and the claimed analysis does not deepen our understanding of why these transformations work.\n\n.  My other main concern is that the paper never tests adaptive or defense-aware attacks. All adversarial images are generated against the CLIP model and then passed through the defense, so the attacker is completely unaware of DBD. But DBD is a deterministic and differentiable post-processing pipeline, you can easily backpropagate through the transformations, entropy filter, and DB-score using BPDA or EOT (arxiv 1802.00420). If an attacker actually optimizes through the whole pipeline, the “directional bias” pattern could disappear completely. \n\n. The authors do include an ablation on τ showing that 0.8 gives the best mean accuracy across datasets. However, this analysis only reports overall accuracy averages. Since the routing decision between “clean” and “adversarial” streams fully depends on this threshold, we need to know how reliable that decision is. A proper analysis would include AUROC or false positive/negative curves for detecting adversarial inputs, and report how performance changes when τ or λ vary slightly around the chosen value. Also, they say τ is “estimated from clean images,” but it doesn’t explain how many, from which dataset, or whether this tuning leaks evaluation data.\n\n. They claim robust accuracy sometimes exceeds clean accuracy. This is suspicious because the defense applies multiple transformations and filtering, which might remove noise or sharpen decisions even for clean samples. So part of the gain might come from the pipeline itself, not the defense logic. A proper ablation (transforms-only vs transforms + DB shift) is missing, and without it, we can’t tell what really improves robustness.\n\n.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aqLVdh0yb0", "reviewer": "Reviewer_fqTJ", "summary": "The paper identifies a dominant feature-space direction for adversarial images under diverse transformations, quantified via a DB-score, and proposes DBD to reconstruct features by shifting along a “Defense Direction” (high-DB) or averaging (low-DB). Evaluations on 15 datasets and multiple attacks (including AA on a subset) report high robust accuracies, sometimes exceeding clean accuracy. The approach is training-free, simple, and efficient relative to some prompt-tuning defenses.", "questions": "- Can you add BPDA+EOT attacks with AA, differentiating through expectation over transforms, entropy filtering, and reconstruction, plus ε/step monotonicity curves? \n- Which specific AA modes and losses were used? was EOT applied? any targeted AA results? \n- Can you provide resource costs including (batching, precision, forward counts) and FLOPs? \n- Can you ablate prompt sets and template counts (hand-crafted vs CuPL) and report multi-seed confidence intervals?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear empirical phenomenon and DB-score metric with intuitive visualizations\n- Simple, training-free mechanism with precise equations\n- Inclusion of all CLIP backbone tests and AutoAttack, as well as efficiency comparisons\n- $\\lambda$/$\\tau$ ablations are helpful\n- most hyperparameters are included for reproducibility, but the exact AA configs, seeds, and adaptive scripts are not provided even in the appendix, thus reproducibility concerns remain", "weaknesses": "- No adaptive white-box evaluation (BPDA/EOT) against the full defense despite stochastic and non-smooth components\n- AutoAttack scope is limited and apparently non-adaptive to DBD; targeted AA not analyzed\n- Robust > clean claims require stronger calibration evidence to rule out masking\n- Causal validation of “Defense Direction” via gradient alignment is missing\n- Timing numbers seem optimistic for 31 transforms; measurement details needed\n- Prompt-set sensitivity not ablated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6NpTpFSoud", "reviewer": "Reviewer_H8wX", "summary": "This paper introduces an interesting observation that adversarial features exhibit a \"directional bias\" under input transformations. Based on this, it proposes DBD, a simple and training-free test-time defense that estimates and reverses this bias. The method achieves state-of-the-art robustness, with reported robust accuracy **even surpassing** clean accuracy under certain conditions.", "questions": "- Could the authors justify centering the main evaluation on a threat model that requires access to ground-truth labels? To what extent does the high robustness achieved in this setting transfer to more realistic attack scenarios where such information is unavailable to the adversary?\n\n- If the results from the more realistic pseudo-label attack evaluation (Table 3) were presented as the paper's primary findings, how would the authors reframe the central claims and contributions of this work?\n\n- The \"Defense Direction\" hypothesis is well-motivated for speicific gradient-based attacks. Could you explain why this is also effective against the gradient-free components of AutoAttack, beyond a general robustness improvement from ensembling multiple input transformations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The core observation of a \"Defense Direction\" for adversarial examples is novel and insightful.\n\n- The proposed method is simple, intuitive, and computationally efficient as a test-time defense.", "weaknesses": "- Unrealistic threat model: The paper's most impressive results are predicated on a PGD attack that requires access to ground-truth labels. This is a well-known weak and impractical threat model that leaks information about the correct class, which the defense is perfectly tailored to exploit. The validity of the core claims hinges on this flawed setup. In other words, the white box + cross-entropy loss configuration was originally established within the community for stress testing (previously the strongest achievable attack effect). If a model performs well under this configuration, there is reason to expect it will also perform well under other configurations. However, this method alters that situation by meticulously designing the approach specifically for this stress-testing scenario, rather than for more general scenarios.\n\n- The headline claim that \"robust accuracy can surpass clean accuracy\" is an artifact of the unrealistic threat model. The more realistic pseudo-label evaluation, where this phenomenon vanishes and performance drops significantly, is not presented as the main result, which is misleading.\n\n- The \"directional bias\" hypothesis is well-motivated for gradient-based attacks. However, the paper lacks a convincing analysis of why this mechanism is also effective against AutoAttack, potentially weakening the claimed technical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)