Discrete Latent Features Ablate Adversarial Attack: A Robust Prompt Tuning Framework for VLMs
Authors: Yang Chen
Yanbin Wei
James Kwok
Yu Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Prompt Learning
Adversarial Robustness
Vision-Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a Discrete Latent Feature based Adversarial Training (DEFEAT) method that mitigates the adversarial attacks for VLMs.

Abstract: While adversarial fine-tuning can enhance the robustness of vision-language models (VLMs), such approaches are computationally expensive. Adversarial prompt tuning has emerged as a practical alternative. However, existing methods are limited by their reliance on vulnerable continuous image features. To mitigate the vulnerability in the feature representation, we propose **DEFEAT** (**D**iscrete Lat**E**nt **F**eatur**E** based **A**dversarial **T**raining), a robust prompt tuning framework for VLMs.
Specifically, the DEFEAT method introduces a perturbation discrete shield module that reconstructs discrete latent features and designs a logits fusion strategy, substantially reducing the discrepancy between clean and adversarial image representations.
Moreover, the DEFEAT method integrates prompt tuning with adversarial training while applying regularization from learnable prompts to hand-crafted prompts, further enhancing the adversarial robustness.
Extensive experiments across 15 datasets validate the effectiveness of the proposed  DEFEAT method among existing adversarial prompt tuning methods. The official code is available at https://github.com/cheny02/DEFEAT-ICLR2026.

Published as a conference paper at ICLR 2026
DISCRETE LATENT FEATURES ABLATE ADVERSARIAL ATTACK: A ROBUST PROMPT TUNING FRAMEWORK FOR VLMS
Yang Chen', Yanbin Wei James Kwok, Yu Zhang"
'Department Computer Science and Engineering. Southern University of Science and Technology DDepartment of Computer Science and Engineering. Hong Kong University of Science and Technology cheny2023@mail sustech. edu. an
(yanbin. ust, yu. zhang. ust @emaiil.mml..m..m com jameskêcse uusscsee..ts... hk
ABSTRACT
While adversarial fine-tuning can enhance the robustness of vision-language models (VLMs), such approaches are computationally expensive. Adversarial prompt tuning has emerged as a practical alternative. However, existing methods are limited by their reliance on vulnerable continuous image features. To mitigate the vulnerability in the feature representation, we propose DEFEAT (Discrete LatEnt FeaturE based Adversarial Training), a robust prompt tuning framework for VLMs. Specifically, the DEFEAT method introduces perturbation discrete shield module that reconstructs discrete latent features and designs a logits fusion strategy, substantially reducing the discrepancy between clean and adversarial image representations. Moreover, the DEFEAT method integrates prompt tuning with adversarial training while applying regularization from learnable prompts to hand-crafted prompts, further enhancing the adversarial robustness. Extensive experiments across 15 datasets validate the effectiveness of the proposed DEFEAT method among existing ad adversarial prompt tuning methods. The official code is available https: ttu.b.b...uu..gggggg.g com wwt.cc/nno2// ICLR2026
INTRODUCTION
In recent years, vision-language models (VLMs), particularly CLIP (Radford et al. 2021) and its successors (Sun et al. 2023| 2024 Zhang et al. 2024a]] Li al., [2022] Jia et 2021 ) have demon- strated remarkable performance across various visual tasks. As these models are deployed for real-world applications, is crucial to ad- dress their vulnerabilities. Recent studies Touvron et al. 2023 Zou
Xurrag
Eallis Culd
al.
2023
Schlarmann & Hein 2023 Zhao al.] 2023 Carlini
et al. 2023 have shown that VL are highly susceptible to adver- sarial examples (Szegedy et al. 2013), which can lead significant security issues by possessing imperceptible perturbations.
Figure 1: The harmonic mean accuracy of the robustness (e 4/255) and accuracy of baselines and the proposed DEFEAT method under the adversarial few-shot classifica- tion setting.
Adversarial training (Madry et al., 2018) is widely regarded as the most effective defense against such attacks. While many studies (Mao et al. 2023 Schlarmann et al. 2024 Hossain & Imteaj] 20244 Gong al. 2025) have improved the robustness of CLIP-like models by adversarially fine-tuning the entire model, this approach is im- practical for large-scale deployment due the immense parameter counts of modern VLMs. To improve efficiency, Garameter-Fffficien Fine-Tuning (PEFT) techniques (Ding et al., (2023) have emerged as a promising solution to enable the adaptation of large models by tuning only a small subset of parameters. Building on this, Li al.
"quual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
60-
Claan image feature
Clcan image fcoture
Adversania image reature
60-
Adversanial feature
40-
70-
20-
-0-
-10
-00
-100
-100
-50
Figure 2: The left and right images illustrate the latent image features without and with the feature discretization, respectively.
(0024) Zhou et al. )2024); Zhang et al. (2024b) have integrated adversarial training with prompt tuning Zhou et al. 2022b). a PEFT method highly effective for VLMs, to efficiently enhance the robustness downstream tasks.
Among those existing methods, most of them rely on continuous image features for adversarial defense and mainly focus how train robust models. In contrast, we mainly focus the feature representation, a different direction. After exploring this direction, we have a finding that discretizing the latent image feature could effectively mitigate adversarial attacks. To see that, as illustrated in Figure 2 latent features processed with discretization exhibit a significantly smaller shift between clean and adversarial examples than continuous image features, where the experimental settings are introduced in Section 3.2] This indicates that the latent feature discretization could reduce the impact of feature shifts induced by adversarial attacks.
Based on this finding, we propose a Discrete LatEnt FeaturE based Adversarial Training (DEFEAT) method, a robust prompt tuning framework for VLMs. DEFEAT designs perturbation discrete shield (PerturbShieldd module that mitigates adversarial attacks through grid-based discrete feature reconstruction and semantic alignment projection. To optimize the trade-off between the robustness and accuracy, the DEFEAT method uses a logits fusion strategy that combines predictions from PerturbShield and original image features. Additionally, DEFEAT applics regularization to the learnable prompt using hand-crafted prompts and demonstrates that this regularizer enhances the adversarial robustness. Extensive experiments across 15 datasets demonstrate that the proposed DEFEAT method achieves state-of-the-ar performance compared to existing methods and offers an improved trade-off' between the robustness and performance (see Figure []. Specifically, the proposed DEFEAT method achieves an average improvement of 13.76% in terms of the harmonic mean of the robustness and accuracy compared to the previous state-of oftte--f-hh-arr ahe-art method under the adversarial few-shot classification setting.
Our contributions are three-fold. (i) To the best of our knowledge. we are the first to use latent feature discretization to mitigate adversarial attacks. (ii) We analyze the mitigating effect of feature discretization on visual adversarial perturbation and propose the DEFEAT method as a robust prompt tuning framework for VLMs; (iii) Extensive experiments on 15 datasets demonstrate the effectiveness of the proposed DEFEAT method under various settings, including adversarial few-shot classification, adversarial domain generalization, and adversarial cross-dataset generalization.
RELATED WORK
CLIP-based VLMs. VLMs have significantly boosted cognitive capabilities by merging visual and textual modalities, excelling in real-world vision tasks (Liu et al., 20233 Zhu et al.) 2024]. The introduction of CLIP (Rafford et al., 2021), trained on about 400 million image-text pairs, was representation learning. Numerous subsequent works have followed this paradigm, proposing a broad family of CLIP-like models, including ALIGN (Jia et al. 202T), EVA-CLIP (Sun al. 2023) OpenCLIP (Ilharco et al.
particularly transformative, establishing new paradigm for rision-languaa
2

---

Published as a conference paper at ICLR 2026
2021p, and LongCLIP (Zhang et al. 2024a).. Given the trailblazing role and widespread adoption of this architecture, we focus our work on CLIP as the representative model to investigate robust prompt tuning within this foundational VLM paradigm.
Prompt tuning for VLMs. Prompt tuning (Zhou et al. 2022b Khattak et al. 2023a Chen et al.
2025 Zhou al.] 2022a Khattak al., (023b) has emerged popular lightweight model adaptation technique, particularly in VLMs. Unlike conventional fine-tuning approaches that involve updating all parameters of a pre-trained model, prompt tuning focuses on optimizing only the prompt representations, which significantly reduces computational overhead. CoOp (Zhou et al. 2022b) introduces an efficient adaptation strategy by optimizing learnable prompt vectors that replace manually crafted textual prompts (c.g.. "a photo of panda") in the textual branch of CLIP, marking a significant milestone in the vision-language alignment. However, CoOp tends to overfit base classes with limited generalization ability. To address this, several studies (Zhu et al. [023] Yao et al. 2023 Chen et al., 2024] Khattak et al., | 2023b) use the knowledge captured in pre-trained CLIP through hand-crafted prompts to enhance the generalization of learnable prompts. In this paper, our study
builds on prompt tuning of CoOp to explore robust prompt tuning for VLMs.
Adversarial training of VLMs. Adversarial attacks (Madry et al.
2018 Goodfellow et al. 2014)
fool models by adding imperceptible perturbations to input images. Several studies (Touvron et al.
2023 Zou al. 2023 Schlarmann & Hein 2023 Zhao et al. 2023 Carlini et al. 2023) have shown
that VS are highly vulnerable to adversarial attacks. Among various defense strategies, adversarial
training (Madry et al. 2018 Zhang et al.] 2019 Goodfellow al., 2014 /Zhag et al. (020) is one
of the most effective defense methods against such attacks. It strengthens the model robustness by incorporating adversarial examples into the training data and optimizing the model to classify both clean and adversarial examples correctly. Building on this idea, TeCoA (Mao et al. (023) enhances adversarial robustness CLIP through supervised adversarial fine-tuning, FARE (Schlarmann et al., 2024) introduces an unsupervised adversarial fine-tuning framework to obtain a robust CLIP, which preserves nominal performance while transferring robustness to downstream tasks. However, fine- tuning the entire model is not so economical. Alternatively, AdvPT (Zhang et al.) 2024b) and APT (Li) cct al. 2024) combine prompt tuning with adversarial training, introducing adversarial prompt tuning for CLIP to enhance adversarial robustness through PEFT way. Building multi-modal prompt tuning (Khattak et al., 2023a). FAP (Zhou et al. 2024) improves the consistency of multimodal features and encourages differentiation in unimodal features between clean and adversarial examples, thereby enhancing adversarial robustness. Unlike previous adversarial prompt tuning methods that focus on learning robust prompts, the proposed DEFEAT method integrates VQ-VAE to mitigate the impact of adversarial attacks, achieving a better trade-off between robustness and accuracy.
3 METHODOLOGY
In this section, we begin with brief overview of CLIP, adversarial attacks and training in CLIP, and VQ-VAE. Then we analyze the impact of VQ-VAE on visual adversarial perturbation. After that, we introduce the proposed DEFEAT in detail.
3.1 PRELIMINARIES
CLIP. CLIP facilitates zero-shot image classification by evaluating the similarity between visual and textual embeddings. It comprises visual encoder and a text encoder. Let I represent the class feature extracted by CLIP's visual encoder for an image, which serves as the image representation and is generated by a class token. t {te)C= denotes collection of textual embeddings produced by the text encoder for textual prompts, where each t. corresponds to the textual embedding for class c, and C' is the number of classes. Those prompts typically follow the format "a photo of a [CLS)", ; where "[CLS]" is a placeholder for class tokens that are specific class names like "panda", "dog", and "bird". Based on those embeddings, the probability of classifying an image to belong to exp exp (x)(//
a class can be calculated as (vlx) Se exp (fx(x)/7)' where r is the temperature parameter, fx( = cos(t... I), and cos(:.)) denotes the cosine similarity.
Ad Adversarial attack and training in CLIP. An adversarial attack (Madry et al. 2018) fools the model by learning an imperceptible perturbation added to a clean example x, generating an adversarial example X,- Specifically, in CLIP, the adversarial example x, is optimized by maximizing the
3

---

Published as a conference paper at ICLR 2026
image-to-text contrastive loss (Mao et al. 2023 Li et al. 2024 Zhou al. 2024)
arg max L(xa, y), s.t. x + 8, ||8|pp &
(1)
Xa
where L(xa: t. = - i=1 y: log p (W) Xa), label is one-hot vector with yi equal to 1 when class is the ground-truth label for x and 0 otherwise, c denotes the perturbation size, and || lpp denotes the Ep norm of vector. Here we focus on the f.x. threat model (i.e., p = oo).
After generating adversarial examples, Mao et al. 2023) employs adversarial training to optimize the parameters 0 of CLIP's vision encoder, thereby developing robust CLIP. The objective of adversarial training is formulated as arg ming C(xa: y)
VQ-VAE. VQ-VAE is generative model based on discrete latent representations, with its core mechanism to convert continuous representations into discrete codes. Specifically, VQ-VAE consists of encoder E, decoder D, and codebook The encoder maps input data x E R x into continuous latent vector Ze = E(x) E Raxd, where is the dimensionality of the latent embedding, is the number of latent positions after encoding. The learnable codebook is defined
(ex K where is the size the discrete latent space. The latent vector Ze is then quantized into a discrete latent representation, Zg gnxd This achieved by replacing each vector in Ze with its nearest neighbor from the codebook:
2 arg minllz! exll2 vi n).
(2)
e
where Zg and Ze are the vectors the i-th position Zy and Ze. respectively, and || I-Il2 denotes the le norm. The quantized discrete representation Zg is then used to reconstruct the input by the decoder as x = Dzz,). Since there is no real gradient defined for Eq. P VQ- VAE copies gradients from the decoder input 2g to the encoder output Zee The entire training objective in VQ-VAE is formulated as
(1)
vvQ a ||x &
+3 s 2c
2g
+7 ||, sg 2,
reconstruction loss
codebook loss
commitment loss
where sg is the stop gradient operator. By following the original implementation (van den Oord al. 2017), y is set to be 0.253. Since VQ-VAE is an end-to-end, trainable, and commonly used discretization model, we use it to discretize and reconstruct latent image features.
3.2 MITIGATING EFFECT VQ-VAE ON VISUAL ADVERSARIAL PERTURBATION
icuunty 926, Rcbuerees 8
Daon inage Satua
Clas image Fanure
Advrrarial
(a) CLIP
(b) VQ-VAE on images
(c) VQ-VAE on latent features
Figure 3: Visualization of clean and adversarial examples on the EuroSAT dataset. The VQ- VQ-VAE is
trained with 16-shot samples per class under the adversarial few-shot classification setting.
In this section, explore how VQ-VAE mitigates visual adversarial perturbations. We first visualize the clean and adversarial image features output by the CLIP's image encoder in Figure Bal We observe that imperceptible perturbations (c = 4/255) cause significant shift of latent features between clean and adversarial examples, leading to a sharp decline in classification accuracy for adversarial samples (i.e., from 19.44% to 11.21%).
Inspired by the quantization process VQ-VAE, we hypothesize that the complex discretization process could mitigate adversarial attacks by reducing the distribution shift between clean and
4

---

Published as a conference paper at ICLR 2026
Trainable
e.g. photo of [CLS]."
Frozen
hand-crafted prompt
Text Encoder
text feature Regularization L
logits
V1
12
[CLS]
learnable prompt
Logits fusion
class feature
PerturbShield
Regularization 4
i
proj
Image
@Projection
Encoder
adversarial
clean example
VQ-VAE
perturbation
grid feature patch
i patch
Figure 4: Overview of the proposed DEFEAT method. Note that both image and text encoders remain frozen, with only the learnable prompt and the perturbation discrete shield module being trained. In our experiments, we exclusively use adversarial examples for training.
adversarial examples. To validate this hypothesis, we integrate VQ- VAE as a defense module within the VLM framework against adversarial attacks. Firstly, we attempt use VQ-VAE to reconstruct input images as did in (Mao et al. (2022). As shown in Figure 3b the shift of latent features between clean and adversarial examples is significantly reduced, demonstrating that the discretization process can suppress adversarial perturbations. However, pixel-level reconstruction with VQ-VAE introduces significant information loss and structural distortions that compromise semantic content preservation when training data is limited. To see that, can find that the feature distributions in Figure 3b differ greatly from the original image feature distributions shown Figure 3a The poor performance (i.e., Accuracy=9.266 26%% Robustness=0%) could verify this and also make this approach impractical.
Alternatively, we use VQ-VAE to reconstruct the latent feature output by ViT. As shown in Figure 3cl we find that the distribution shift of latent features between adversarial and clean examples is significantly reduced. Moreover, the distribution latent features after VQ-VAE VAE (shown in Figure 3cp more similar to the feature distribution by the CLIP (shown in Figure 3a This suggests that reconstructing latent features can also effectively suppress adversarial perturbations and achieve 1c.9stness=12.900%)) One possible reason is that
better performance (i.e., Accuracy=17.90%% when adversarial perturbations are introduced into clean image features, the process of finding the nearest codes often makes both clean and adversarially perturbed features be assigned to the same discrete representation (i.c., codes). Hence, the discretization of VQ-VAE effectively neutralizes the perturbations within its discrete representation space, thus hindering the updates of adversarial perturbations and diminishing the distributional differences between clean and adversarial examples.
3.3 DEFEAT: DISCRETE LATENT FEATURE BASED ADVERSARIAL TRAINING
Motivated by the above observation, we propose the DEFEAT method for CLIP-based VLMs. Besides a frozen image encoder and a frozen text encoder in CLIP, DEFFAT consists of a learnable prompt and perturbation discrete shield module. The architecture of DEFEAT is illustrated in Figure
Prompt tuning for text encoder. We introduce learnable prompt vectors v {Vi, V vM to replace the textual prompt (e.g.. "a photo of a"), where each v; has the same dimension as the word embeddings. The learnable prompt subsequently constructed by concatenating v with the class token. Formally, the learnable prompts be a set of textual embeddings generated by the text encoder for the learnable prompts, where each t corresponds to the textual embedding of learnable prompt for class c. The probability of classifying
formed as {V1, V2. V...... VM, [CLS)). Let t (
5

---

Published as a conference paper at ICLR 2026
an image can be calculated
exp (x)(((())/1)
(u/xx)
>e exp (fx(x)/>)
where fx(x) = I) and recall that I denotes the class feature.
Perturbation discrete shield (PerturbShield), As analyzed in Section 3.2
we propose integrating
VQ-VAB as a defense module within the prompt tuning framework to mitigate the impact of visual adversarial perturbations. Given the outstanding performance of the Vision Transformer (ViT) (Dosovitskiy et al.) 2020) in visual tasks, we use CLIP's VIT as the image encoder. The last Transformer layer of ViT can output two types of image features: the class feature E Rdd generated by the [class] token and the grid feature patch E R.xxd from image patches, where dy denotes the dimension (i.c., 512 for CLIP) and N is the number of patchs.
Since CLIP uses the class feature I to represent an image, we initially consider reconstructing I with VQ-VAE. However, this approach has some limitations. That is, the class feature essentially functions as a global patch representation, meaning that we would be restricted to using only a single code from the codebook to represent the entire image. However, the representation capacity of a single code is far inferior to that of multiple codes combined, ultimately resulting in severe information loss. In this case, if an adversarial attack successfully maps the class feature to an incorrect code, it will lead to wrong prediction.
To address this problem, we propose Perturbation Discrete Shield (PerturbShield), an end-to-end defense module that utilizes discrete representation learning to mitigate adversarial perturbations in VLMs. Specifically, PerturbShield operates through two designed stages: 1) grid-based discrete feature reconstruction; 2) semantic alignment projection, which are introduced the following.
1) Grid-based discrete festtree reconstruction. Instead of reconstructing the class feature, we recon- struct the grid feature Ipatch through the VQ-VAE. This process aims represent the grid feature using a combination of multiple codes to preserve more information. Additionally, the discretization process can help mitigate minor perturbations and enhance the model's robustness as analyzed in Section 3.2 The reconstructed grid feature can be obtained as Iatth VQ-VAE((paata where VQ-VAEY denotes VQ-VAE and Iatth E Nxd.
2) Semantic alignment projection. We design a learnalee matrix W that performs two functionalities. First, since Ipatch is more robust than we could use to learn robust representation of the whole image by a linear transformation: Ipros = W Ipatch. where W E RIXN X and Iroo E Rd Another functionality is to semantically align the transformed latent feature with CLIP's pre-trained embedding space through a feature alignment regularization between Irooj and the class feature as
C.
proj
I|[
where || -- |/ denotes the & norm. These dual functionalities ensure that the adversarial defense process remains aligned with the embedding space of the VLM. Then the prediction probability for an input image x can be computed as
(xp(
p (y)()))
De exp (
where (x) cos(ts, i proo)).
Logits fusion. There is usually some information loss during the reconstruction process. Con- sequently, using Iproj alone to compute logits (i.e., the prediction probability) reduces the clean accuracy (e.g., from 19.44% to 17.90% in Figure 3. Since PerturbShield effectively mitigates adversarial attacks, we propose logits fusion strategy that uses logits from iproj and t" counter adversarial attacks, while using logits from I and t to maintain the clean accuracy, thus achieving better trade-off between the robustness and accuracy. Specifically, the former logits are defined in Eq. (6), and the latter ones are defined in Eq. (4). Then, the fused logits are calculated as
p(yxx)
(p 2 ( (x)x) + (2lx))/2.

---

Published as a conference paper at ICLR 2026
Prompt alignment regularization. To enhance the generalization of learnable prompts and prevent overfitting (Zhou et al. 2022a), we use prompt alignment regularization for learnable prompts. This ensures the consisteney between text features generated by learnable prompts and hand-crafted prompts, thereby improving adaptability while preserving CLIP's inherent zero-shot inference capa- bilities. Specifically, the prompt alignment regularization is formulated as
Feg ||t - t|l1-
(8)
Moreover, find that using cand-crafted prompts to regularize learnable prompts as in Eq. (8) can enhance the model's 's robustness, which is
Section 4.3
Training objective. Based Eq. 7 the cross-entropy loss is defined (x, y)
Ei= y/ (() To learn a robust prompt. Li et al. (2024); Zhou al. (0224) combines prompt tuning with adversarial training. Following this paradigm, we use the learnable prompts updated the previous epoch to generate adversarial examples dynamically each epoch as
a = arg max (x,, y), s.t. || X, 16
X
Finally, based those adversarial examples, the overall training objective of the DEFEAT method is formulated as
L(Xa: t", t, - Lee (Xa, t², + LvQ-van(Apaahl + X + +
(10)
Due to page limit, the whole algorithm for the proposed DEFEAT method shown in Appendix
EXPERIMENTS
4.1
SETUPS
Datasets. By following APT (Li et al. 2024) and CoOp (Zhou et al. 2022b), experiments are conducted under three settings, including adversarial few-shot classification, adversarial cross-dataset generalization, and adversarial domain generalization. The first two settings are performed on image classification datasets, including ImageNet (Deng et al.) (209) and Caltech101 (Fei-Fei et al. 2004) for generic object classification, OxfordPets (Parkhi et 2012)). StanfordCars (Krause et al. 2013). Flowers102 (Nilsback & Zisserman 2008). Food101 (Bossard al. 2014). and FGVCAircraft Maji et al. 2013) for fine- teee-graied visual categorization, SUN397 (Xiao et al. 2010) for scene recognition, DTD Cimpoi et al., (014) for texture classification, EuroSAT (Helber et al. 2019) for satellite image classification, and UCF101 (Soomro et al. 2012) for action recognition. Adversarial domain generalization is conducted on the ImageNet dataset and its variants, including ImageNetV2 (Recht et al.) 2019). ImageNet-Sketed (Wang et al. 2019), ImageNet-A (Hendrycks et al. 2021b). and ImtgeNet-R (Hendeny-s et al. 2021a).
Baselines. To demonstrate the effectiveness of the proposed DEFEAT method, we compare it against three baseline methods, including zero-shot CLIP (Radford et al. 202I), the textual prompt tuning method (i.e., Adversarial Prompt Tuning (APT) (Li et al.] (222))2)))) ), and the multi-mode prompt tuning
method (i.e., Few-shot Adversarial Prompt learning (FAP) (Zhou et al. 2024)).
Implementation details. Our implementation is based on CoOp (Zhou et 2022b) and APT (L) et al.] 2024). For all baseline methods, we maintain the same experimental settings (e.g., training epochs, training schedules, and data augmentation settings) as specified in their original implementations. All experiments are conducted with ViT-B/32 CLIP model. Following APT (Li
2024). the image encoder weights for all baselines were pre-trained using TeCoA (Mao et The length of the prompt vectors is fixed to 16. For the DEFEAT methods, 3, X, and f are set to 0.5. 0. 10, and 20, respectively, across all experiments. For adversarial training and evaluation, we employ the PGD attack (Madry et al. 2018) under the foo threat model. Following Li al. 22024); Mao al. (2023); Schlarmann et al. 2024), two perturbation sizes, e = 1/255 and 4/2 /55, are used. During training. we use 3 steps with step size of 2e/3, and for evaluation, 100 steps with a step size of e/ and a random start. More details are provided in Appendix
al.
2023
4.2 MAIN RESULTS
In this section, we report experimental results under different settings.
7

---

Published as a conference paper at ICLR 2026
Table 1: The average performance on the 1I datasets for different e and shots under the ad adversarial few-shot classification setting. 'H' denotes the harmonic mean accuracy.
shot
4 shots
16 shots
Method
Acc.
Rob.
Acc.
Rob.
Acc.
Rob.
CLIP
46,06
32.98
38.44
46,06
32.98
38.44
46.06
32.98
38.44
APT
46.99
33.36
39.02
58. 19
41.34
48.34
65.41
47.88
55.29
1/255
FAP
52.72
37.15
43.59
59.77
42.21
49.48
66.40
48.86
56.30
DEFEAT
52.07
38,99
44.59
58.37
53.04
55.58
65.03
60.77
62.83
CLIP
33.67
10.79
16.34
33.67
10.79
16.34
33.67
10.79
16.34
APT
32.97
11.62
17.18
42.29
14.40
21.48
51.08
20.26
29.01
4/255
FAP
38.16
13.44
19.88
44.63
15.34
22.83
50.50
19.82
28.47
DEFEAT
35.78
14.73
20.87
44.13
25.28
32.15
50.98
36.05
42.23
Adversarial few-shot classification. In this scenario, we assess the model's capability to develop robust representations from limited labeled data. Specifically, models are tuned using 4. 16 shots per class and evaluated on the remaining samples. The average performance of DEFEAT and baselines on 11 datasets is shown in Table where 'H' denotes the harmonic mean accuracy of robustness and accuracy, measuring the trade-off between these metrics. DEFEAT consistently outperforms zero-shot CLIP, even with 1-shot tuning, achieving significant improvements in accuracy, robustness, and 'H', For example, under accuracy, robustness, and 'H', respectively. Furthermore, these improvements scale with the number of shots. With 16-shot tuning, DEFEAT achieves gains of 18.97% (17.31%) 27.79% (25.26% and 24.39% (25 89%) in accuracy, robustness, and 'H' for E = 1/255 (4/255), respectively.
1 1/255, DEFEAT provides gains of 6.01 %, 6.01%, and 6. 1.5% in
Existing adversarial prompt tuning methods (i.e., APT and FAP) improve performance over CLIP, confirming the effect of combining adversarial training with prompt tuning. Compared to those methods, DEFEAT performs comparably to the current state-of-the-as art method FAP in terms of the accuracy, while consistently outperforms both APT and FAP in terms of the robustness and 'H' across all shot and perturbation sizes. Notably, with 16-shot training under 4/255, DEFEAT surpasses APT (FAP) by 15.79% (16.23%) in robustness, and 13.22% (13.76%) in 'H', respectively. Detailed results for each dataset are provided in Appendix C.
Adversarial domain generalization. We assess the model's capability to generalize to out-of- distribution (OOD) data in the adversarial domain generalization setting. Specifically, models tuned using 16-shot and 100-shot samples from each of the 1000 classes on ImageNet (source), and then evaluated on four different target domains (i.e., ImageNetV2. ImageNet-Sketch, ImageNet-A, and ImageNet-R). Table shows that all baselines achieve comparable accuracy on ImageNet, as they use the same robust image encoder (Mao et al. 2023) pre-trained on the full ImageNet training set. Nevertheless, with 16-shot training. DEFEAT surpasses CLIP, APT, and FAP in terms of the robustness by 8.92%, 7.0 7.04%, and 7.0%, respectively. Across the four target domains, DEFEAT achieves good robustness and 'H', outperforming the strongest baseline (i.e., APT) by 3.31% in robustness and 3.36% in 'H'. Those results confirm that the robustness Icarned by DEFEAT can be effectively transferred to OOD data. Furthermore, the number of training samples increases from 16 shots to 100 shots, DEFEAT demonstrates consistent improvements terms of the robustness, achieving +3.31% ImageNet and average of +1. 1.94% across the target domains, which highlights the scalability of our approach. In contrast, the two prompt tuning baselines (i.c., APT and FAP) show little to no improvement.
Table 2: Performance under adversarial domain generalization setting. E = 4/255.
Source
Target
Method
ImageNNet
ImageNNr-V22 V2
ImageNet- Sketch
ImageNet-A
ImageNet-R
Average
Acc.
Acc.
Roh.
Roh.
H
Acc.
Acc.
Roh.
zeno shot
[CLIP
40.11
10.14
16.19
33.11
7.49
12,22
17.59
7.24
10:26
4.04
0.28
0.52
37,52 12.51
18.76
23.07
6,88
10.60
APT
41.06
12.02
33.67
9.09
7.87
10.99
0.36
0.66
37.04
13.29
19.56
16
FAP
40.32
12.06
18.57
32.81
9.17
14.33
16.42
7.11
9.92
3.87
0.43
0.77
36.04
13:55
19.70
22.29
7.57
11:30
DEFFAT
40.68
19.06
25.96
33.12
14.96
20.61
18.27
9.87
12.82
3.97
0.88
1.44
37.35
18.14
24.42
23.18
10.96
14.88
APT
40.35
12.11
18.63
33.38
9.19
14.41
18.86
8.16
11.39
4.1
0.40
0.73
37.81
13.70
20.11
23.57
7.86
11.79
100 shots
FAP
40.76
12.35
18.96
33.60
9.14
14.37
16.41
6.98
9.79
3.65
0.39
0.70
36.36
13,60
19.80
22.51
7.53
11.28
DEFRAT
22.71
29,03
32.77
21.90
18.49
11.53
14.20
1,77
2.07
17.34
21.45
27.25
23.20
12.70
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance under adversarial cross-dataset generalization setting. e 4/255.
Target
16 shots Method
ImugeNet Caltech10l OxfoniPets StanfordCars Flowers102 Food10] PGVCAtrenatt SUN.97
DTD
EaroSAT
UCPI01
Avg.
CLIP
40.11
78.78
66.26
10.32
30.13
23.52
7.17
33.24
24.29
19.56
36.98
33,03
APT
79.63
64.92
11.55
27.24
4,77
30.87
22.22
16.31
33.68
Acc.
FAP
40.32
79.19
63.67
9.61
30.25
22.82
5.43
31.92
23.25
16.79
32.25
31.52
DEFEAT
40.68
78.22
66.69
9.40
31.10
22.01
6.09
32.28
23.94
17.95
33.15
32.08
CLIP
10.14
43.61
15.36
0.99
8.93
3.27
0.36
6.20
11.35
11.22
7.01
10.85
APT
12.02
46,86
21.10
1,49
8.89
3.75
0.60
6,85
9.93
11,15
7:56
11.82
Rob.
7.15
7.45
DEFEAT
19.06
54.32
31,97
3.84
18.80
8,72
2.34
13.47
15.60
12.89
13.59
17.55
CLIP
16.19
56.14
25.20
1.81
13.78
5.74
0.69
10.45
15:47
14.26
11.79
16.35
APT
18.60
59.00
31.85
2.64
13.41
6.47
1.07
11.21
13.73
13.25
12.35
17.18
H
BAP
18.57
58.40
31.25
2.67
14.74
6.32
1.18
11.68
14.76
13.57
12.10
17.32
DEFRAT
25.96
64.12
43.22
5.45
23.43
12.49
3,38
19.01
18.89
15.00
19.28
22.69
100 shots Method
ImugeNet| [Caltech
OxfondPets StanfordCars Flowers
Food101
PGVCAirenaft SUN.997
DTTD
EuroS UCF101
Avg.
CLIP
40.11
78.78
66,26
10.32
30.13
23.52
7.17
33.24
24.29
19.56
36.98
33,03
APT
80.28
11.76
29.03
5.85
32.91
25.71
14.17
32.31
Acc.
FAP
40.76
75.54
64.30
10.21
29.76
23.37
3.93
30.49
24.94
22.01
31.51
31.61
DEFEAT
40.23
76.63
65.74
9.46
30.25
20.02
6.51
31.31
23.94
16.67
31.69
31,22
CLIP
10.14
43.61
15.56
0.99
8.93
3,27
0.36
6.20
11.35
11.22
7.01
10.85
APT
12.11
46.82
21.18
1.52
9.66
3,76
0.69
7.20
12.41
11.06
8.33
12.26
Rob
10.17
7.35
DEFEAT
22.71
59.35
42,95
5.58
21.52
11.91
3.78
16.29
16.49
12.98
16.44
20.73
CLIP
16.19
56.14
25.20
1.81
13.78
5,74
0.69
10.45
15.47
14.26
11.79
16.35
APT
2.69
16.74
12,42
13.45
17.78
H
TAP
18.96
56.01
31.05
2.57
14.40
6.67
0.71
11.27
14.45
15.69
11.92
17.06
DEFRAT
29.03
66.89
51,96
7.02
25.15
14.94
4.78
21.43
19.53
14,60
21.68
24.92
Ad Adversarial cross-dataset generalization. We assess the model's zero-shot adversarial robustness in the adversarial cross-dataset generalization setting. Models are tuned using 16-shot and 100-shot samples from each of the 1000 classes on ImageNet and evaluated on the other 10 datasets. Table B shows that DEFEAT achieves competitive accuracy and the highest robustness and 'H' compared all baseline methods across all datasets, including both source and target domains. Specifically, with 16- shot tuning, DEFEAT outperforms the strongest baseline (i.e., FAP) by an average of 5.61% in terms of robustness and 5.37% in terms of 'H'. Similar to the adversarial domain generalization setting, APT and FAP show no significant performance improvements when the number of training samples increases from 16 shots 100 shots. In contrast, DEFEAT demonstrates scalability, achieving gains of 3.65% in terms robustness on ImageNet and 3.18% in terms of average robustness on 10 target datasets. Those results demonstrate the good zero-shot adversarial robustness of DEFEAT.
4.3 ABLATION STUDIES
In this section, we conduct ablation studies to analyze the effectiveness of the PerturbShield module, logits fusion strategy, and hyperparameter (i.e., μ)). Experiments are conducted using 16 examples per class on the EuroSAT dataset under the adversarial few-shot classification setting.
Effects of components. Table shows the ablation study for PerturbShield and logits fusion strategy. Due to information loss during the reconstruction process, using PerturbShield alone without logits fusion results in limited performance improvements CLIP and can negatively impact DEFEAT. Unlike CLIP, which computes logits using hand-crafted prompts, DEFEAT relies on learnable prompts. The information loss impair prompt learning, thereby causing adverse effects.
By combining PerturbShield with the logits fusion strategy, adversarial attacks can be mitigated effec- tively while maintaining good clean accuracy, thus achieving a better trade-off between accuracy and robustness. Specifically, CLIP combining Perturb- Shield and logits fusion strategy outperforms the orig- inal CLIP by 9.91% in 'H' (24.13% v.s. 14.22%), and DEFEAT with this combination outperforms its
Table 4: Ablation study. 4 4/255.
Advescnial prompt taning
Logiia
Baselinas
Potusldbictl fucon
X
19.48 11.21
14,22
CLIP
x
x
14.99
x
25.53
22.58
(4.13)9911)
/
x
64.12 25.33
36.31
/
7
(-166)
/
7
56.69 (420.38)
counterpart without it by 20.38% in 'H' (56.69% v.s.
36.31%). Therefore, we combine PerturbShield with the logits fusion strategy.

---

Published as a conference paper at ICLR 2026
Effect of μ. jr controls the weight of Cieg. which apply reg-
55
ularization from learnable prompts to hand-crafted prompts.
--- Arr -4- Roh
According to Figure []] we
that applying regularization
Average Accuracy
50
(i.e., 1r > 0) of learnable prompts can enhance the model's ro- bustness. As y increases, the robustness of DEFEAT improves, achieving the best overall performance at 20. Therefore, we set μr 20 in our experiments.
45
40
35
30
How does DEFEAT enhance adversarial robustness? To answer this question, we visualize the image feature (i.e., class feature) output by the image encoder using t-SNE. Figure 6 shows that CLIP exhibits a significant distribution shift between clean and adversarial examples, while adversarial prompt tun- ing methods (i.e., APT and FAP) reduce this shift. Notably, the image encoder remains frozen during training and testing for all methods, demonstrating that adversarial prompt tuning can mitigate the backward adversarial gradient, thereby enhancing model robustness. DEFEAT not only incorporates adversarial prompt tuning but also further mitigates the backward adversarial gradient through PerturbShield. So the distribution shift of latent features between clean and adversarial exam- ples is further reduced compared to APT and FAP, providing additional evidence of the effectiveness of the proposed DEFEAT method.
10
15
20
25
Figure 5: The average performance of DEFEAT on all datasets w.r.t. different μ values.
Nowtsy 21 2013
24
(a) CLIP
(b) APT
(c) FAP
(d) DEFEAT
Figure 6: Visualization of clean and adversarial examples on the EuroSAT dataset. Models are trained with 16-shot samples per class. e = 4/255.
Moreover, due to page limit, more experiments can be found in Appendix C including evaluations against stronger attacks and custom adaptive attack designed for DEFEAT, generalization to alternative backbones, detailed hyperparameter analysis, and computational cost analysis.
CONCLUSION
This work presents robust prompt tuning framework for VLMs. We begin by analyzing the mitigating effect feature discretization on visual adversarial perturbation. Building on this, we propose the Discrete LatEnt FeaturE based Adversarial Training (DEFEAT) method, which ablates adversarial attacks in the feature representation. Specifically, DEFEAT introduces a perturbation discrete shield module that reconstructs discrete latent features, and designs logits fusion strategy to improve the trade-off between robustness and accuracy. Moreover, DEFEAT integrates prompt tuning with adversarial training and applies prompt alignment regularization, further enhancing the adversarial robustness. Extensive experiments on 15 datasets demonstrate that DEFEAT achieves state-of-the. thetherrrt performance.
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Grant no. 62136005 and Shenzhen fundamental research program (o JCYJ20230604114722002
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper presents work whose goal is to advance the field of Machine Learning by improving the safety and reliability AI systems. The authors have read and comply with the ICLR Code of Ethics. The research did not involve human subjects, animal experiments, or personally identifiable data. All experiments were conducted on publicly available benchmarks and open-source models. We have carefully considered the broader impacts and believe that this work poses no foreseeable risks of harm while contributing the development of robust and secure vision- lssounlanguage models.
REPRODUCIBILITY STATEMENT
The authors have made significant efforts to ensure the reproducibility of results. Section 4. details the experimental setup, including datasets, model configurations, and hyperparameter settings. Additional ablations in Section 4.3 and Appendix[C.2 (enddixxC.2uurriii further analyze the effect of each module and hyperparameters. During the reviewing process, the source code is supplied anonymously as part of the supplementary materials. Additionally, upon the acceptance of the paper, this code will be publicly released.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference machine learning, pp. 274-283. PMLR, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. leee, 2017.
Nicholas Carlini, Milad Nasr, Christopher A Choquette- Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36:61478-613 2023.
Yang Chen, Shuai Fu, and Yu Zhang. Mopd: Mixture-of-prompps distillation for vision-language models, 2024. URL https: http://aaxvvv..ooo wrx.v.org//bs//242222 19087
Yang Chen, Yanbin Wei, Ke Jin, Yi Kong, James Kwok, and Yu Zhang. Alternating training based label smoothing enhances prompt generalization, 2025. URL https://arxiv.. abs/ 2508. 17846
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-233, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
11

---

Published as a conference paper at ICLR 2026
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004.
Shizhan Gong, Haoyu LEI, Qi Dou, and Farzan Farnia. Boosting the visual interpretability of CLIP via adversarial fine-tuning, In The Thirteenth International Conference Learning Representations, 2025.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:/4/2.6555 2014.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical generalization. In ICCV, 2021a.
analysis of
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Md Zarif Hossain and Ahmed Imteaj. Sim-clip: Unsupervised siamese adversarial fine-tuning for robust and semantically-rich vision-language models. arXiv preprint arXiv:2407.1497. 2024.
Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, al. Openclip. Zenodo, 2021.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning, pp. 4904 4916, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang. and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15190-15208 2023b.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In CVPRW, 2013.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-languag understanding and generation. In International Conference on Machine Learning, 2022.
Lin Li, Haoyan Guan, Jianing Qiu, and Michael Spratling, One prompt word is enough to boost adversarial robustness for pre-trained vision- vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24408 22419, 2024.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 130..515. 2013.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero- shot adversarial robustness for large-scale models. In The Eleventh International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Xiaofeng Mao, Yuefeng Chen, Ranjie Duan, Yao Zhu, Gege Qi. Xiaodan Li, Rong Zhang, Hui Xue, et al. Enhance the visual representation via discrete adversarial training. Advances in Neural Information Processing Systems, 35:7520-7533, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification of classes. In Indian conference on computer vision, graphics & image processing, 2008.
large number
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICMI., 2019.
Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3677-3685, 2023.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine- fine-tuning of vision embeddings for robust large vision-language models. In International Conference on Machine Learning, pp. 43685-43704, PMLR, 2024.
Lijun Sheng, Jian Liang, Zilei Wang, and Ran He. R-tpt: Improving adversarial robustness of vision-language models through test-time prompt tuning. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 29958-29967, 2025.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in lsiounlanguage mod- els. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Ad- vances in Neural Information Processing Systems, volume 35, pp. 14274 14289. Curran Asso- ciates, Inc., 2022. URL httphtts///pcccccennnggtos////et reurips ct wttpppprpppppp.peepptee lareeer__lllesee files/paper Paper-ape----- wonferonneee.coooo Confrpaeeeeeeeeeeeeeeeeeeeee pdf
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions 2012.
classes from videos in the wild. arXiv preprint
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale, 2023. httpptt::://arttt
abs /2003 15389
Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-188 Scaling clip 18 billion parameters, 2024. URL http:/taa//aaii..orr/.// [2402. 04252
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
Haohan Wang. songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NIPS, 2019.
Xin Wang, Kai Chen, Jiaming Zhang, Jingjing Chen, and Xingjun Ma. Tapt: Test-time adversarial prompt tuning for robust inference in vision- lasoun-anguuge models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19910-19920, 2025.
13

---

Published as a conference paper at ICLR 2026
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guided context optimization. In CVPR, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pp. 310-325. Springer, 2024a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference machine learning, pp. 7472-7482. PMLR, 2019.
Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang. Yu-Gang Jiang, and Jitao Sang. Adversarial prompt tuning for vision-language models. In European Conference on Computer Vision, pp. 56-72. Springer, 2024b.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. International conference on machine learning, pp. 11278-11 1287. PMLR, 2020.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. In Advances in Neural Information Processing Systems, volume 36, pp. 541154115555 14-11154 2023.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for models. In CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 130(9):3337-248, 2022b.
Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning vasion-language models. In Advances in Neural Information Processing Systems, volume 37, pp. 3122-3156, 2024.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tning. In ICCV, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15045 2023.
14

---

Published as a conference paper at ICLR 2026
Contents of the Appendix
1. Appendix
Algorithm
2. Appendix
Implementation Details
3. Appendix
Additional Experimental Results
Comprehensive Results Adversarial Few-shot Classification
C.2
Analysis on More Hyperparameter Sensitivity
C.3
Analysis of Logits Fusion Strategy
C.4
Adversarial Robustness Evaluation under Various Attacks
C.5
Adaptive Attack on DEFEAT
C.6
Generalization to Stronger Perturabation Budgets
C.7 7
Generalization to Alternative Robust Backbones
g
Generalization to Alternative VLM Backbones
Robust Visual Backbone Reliance
C.
10
Computational Cost Analysis
Performance Comparison under the &2 Threat Model
12
Combination with Test-time Defense Strategy
C.I3
Theoretical Justification
C.14
Stability Analysis with Multi-Seed Evaluation
C.15
Stability Analysis against Prompt Variations
4. Appendix D
Use of Large Language Models
A
ALGORITHM
The whole algorithm for the proposed DEFEAT methods is shown in Algorithm T
Algorithm Discrete Latent Feature based Adversarial Training (DEFEAT)
I: Input: Dataset DD, learnable prompt vectors v. textual embeddings of hand-crafted prompts CLIP pre-trained image encoder I and text encoder T, VQ-VAE model OvQ-VAF, learnable matrix W, weight parameters A, μu, learning rate and perturbation size e.
2: for all training epochs do
3:
for all x. y minibatch do
4:
Use and (CLS) generate textual embeddings of learnable prompts t*; Generate adversarial examples
5:
Xa + arg max Coe t", y),, s.t. X, x &
x,
6:
Feed x, to I to generate class feature and grid feature Iattch's
7:
Reconstruct patch using a VQ-VAE, obtaining Ipatchi Obtain robust representation by transformation
8:
Iproj W Ipatch
9:
Fuse the logits from pgroj in Eq. and in Eq. 22)
(4
p(y(xa) &- (v"4(r) xa)
Train with the objective
L(xa, t., t,y)
+ Loe (xs, to, y) + LVQ-VAB(
>C + MCT
where Lao (xn( t., v) - EE yi log p(v. xx). Clag is defined in Eq. (5) CT is defined in Eq. (8) and LVQ-VAE (+) is defined in Eq. (3)
Upadate the learnable parameters
V v V L(xa, tt,);; W W VW t t,y); θ vQQ + bvQ-VAB vV VVVvv-vnn C(x,, t", tyy).
end for
13:
end for
15

---

Published as a conference paper at ICLR 2026
B IMPLEMENTATION DETAILS
All experiments are conducted on NVIDIA GeForce RTX 3090, except for the ImageNet dataset, which is on Quadro RTX 8000. Training is conducted using SGD with an initial learning rate of 0.002, which is decayed using the cosine annealing rule. The maximum epoch is set to 200, 100, and 50 for 16, 4, and shots, respectively. For ImageNet, they are 50, 20, and 20. A warm-up strategy is used by fixing the learning rate to 10-3 during the first epoch. The VQ-VAE within the PerturbShield module configured with an input dimension of 512 to align with CLIP ViT-B/32 grid features, projecting them into latent dimension of 256 using codebook of 512 unique embedding vectors. The encoder of "Q-VAE consists of two convolutional layers followed by a residual block, while the decoder of VQ-VAE mirrors this structure with residual block and transposed convolutional layer for feature reconstruction.
The hand-crafter prompts for different datasets used in the prompt alignment regularization follow Radford et al. [202T); Zhou et al. (2022b) and are shown below:
ImegeNet: " photo of [CLS].
Caltech101: photo
[cus]."
oxfordPets: "a photo of [CLS], type of pet.'
StenfordCars: "a photo of a ICLSI."
uxfordrlowars: *a photo of [CLS], type flower."
Food101: "a photo of [CTS], a type of food."
reYCALreraf:: photo SUN397: "a photo of a [CLS]."
[CIS], Lype aircraft."
[CLS], a type of textura." RuroSAT: "a centered sstellite photo of [C1S]." uCr101: "a photo of a person doing [c_s]."
DID: "a photo
Note that [CLS] denotes the placeholder for the class name.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I COMPREHENSIVE RESULTS OF ADVERSARIAL FEW-SHOT CLASSIFICATION
In this section, provide the full results of adversarial few-shot classification performance on 11 datasets. The results Tables A5 and A6 show that the proposed DEFEAT method can consistently outperform the baseline methods in terms of the robustness and harmonic mean accuracy across nearly all datasets and shot settings. For each specific dataset, DEFEAT consistently outperforms CLIP across all shots. For adversarial prompt tuning methods, DEFEAT outperforms APT and FAP across almost all the datasets and shot settings, with particularly notable improvements on EuroS UCF101, etc.. For example, with 16-shot training. DEFEAT surpasses APT and FAP by 20.13% and 73.79 in terms of 'H' (i.e.. 56.69% vs. 36.56% vs. 32.90%) on EuroSAT, and by 23.21% and 21.35% (i.e., 48.45% vs. 25.24% vs. 27. 27% on UCF101.
Figure A7 shows that as the number of shots increases, DEFEAT achieves greater improvements compared to APT and FAP. Those results demonstrate DEFEAT's effectiveness and scalability.
C.2 ANALYSIS ON MORE HYPERPARAMETER SENSITIVITY
In this section, we conduct ablation studies to analyse the effectiveness of hyperparameters (i.e., a, B, and A).
Effect of a. Eq. 3 a controls the weight of the reconstruction loss, which is used to optimize the decoder and the encoder of VQ-VAE. According to Figure A8al we can see that DEFEAT is insensitive within the range [0.5]. We set a = 0.5 our experiments.
Effect of 3. In Eq. 33) 3 controls the weight of the codebook loss, which is used to learn the embedding space of the codebook. According to Figure A8b) applying the codebook loss (i.e., > 0) results in better overall performance compared to excluding it (i.e., B > 0). Failure to update the codebook reduces the representational capacity of the embeddings, thereby increasing the reconstruction quantization error of the VQ-VAE. Additionally, the model may collapse if the codebook is poorly initialized. The overall performance of DEFEAT is insensitive within the range [0.01, 1.. In our experiments, we set /3 = 0.1.
16

---

Published as a conference paper at ICLR 2026
Table AS: The performance on the 11 datasets for different shots under the adversarial few-shot classification setting when e = /255 where 'H' denotes the harmonic mean accuracy.
Datsset
1,255
ENP
DETEAT
DETRAT
Acc.
52.72
59.77
Average
Rob
32.98
33.36
H
38.44
39.02
43.58
44.59
49.45
55.57
56.29
Ace.
55.48
55.01
57.39
ImageNett
Rob
38.48
30.00
37.79
39.68
39.30
41.36
33.57
40.37
53.07
H
45.40
45.80
44.30
45.86
46.85
47.62
47.76
Acc.
85.92
86.46
88.
89.66
89.17
90:87
91.16
92.90
92.21
Ca
Rob.
75.21
77.32
78.36
80.73
79.68
30.81
$7,99
$3.45
84.30
91.12
11
80.21
81.63
85:25
84.96
85.16
$3.55
$9.55
58.22
55.08
92.24
Acc.
79.45
78.33
79.56
77.19
55.48
$3.21
$4.05
$3,95
K5.55
64.71
OxfordPets
Rob.
63.36
61.57
60 56
62.36
66.18
06. 34
73.37
65.14
19.15
79.53
H
70.45
68.95
68.77
68.99
73.63
73.82
78.36
73.36
76.48
82.04
Ace:
25.21
41.09
48.55
42.87
49.51
51.14
47.69
60.22
57.54
56,42
SuatordCCars
Rob
12.50
22:44
24.05
26.81
26.54
27.67
44.05
37.21
32.79
51.65
H
16:71
20.03
31.48
32.99
34.89
35.74
45,89
45.99
41.77
53,93
Acc.
48.15
23.63
59.60
53.39
76.61
70.36
72.73
$6,80
62:46
$4.90
Flowers
Rob.
32.68
17.46
42.34
40.80
59.68
51.45
69.03
73.57
06.63
81.81
H
38.91
20.08
49.37
45.25
67.09
59.45
70.82
79:64
73.70
83,33
Acc.
46.58
41.16
55.36
49.83
45.37
59.24
51.29
54.68
64.48
56.31
0ood101
Rob
27.38
22.12
33,32
32.07
24.05
36.20
46.84
33.012
41.19
49.45
11
34.49
28.76
41.60
39.02
32.20
44.94
48.96
50.27
$2.66
Avc.
12.51
17.55
16.64
21.00
31.27
28.74
26.55
25.74
PGVCAireaatt
Rob.
6.09
1.08
10.41
11.16
11.43
19.98
14.97
23,46
13.37
14,80
53,50
52.63
58.75
SUN197
Rob.
34.50
42.24
Avc.
36.83
36.46
DTD
Rob.
23.88
35.87
41.67
52,66
H
27.34
27.75
29.23
32.78
39.02
41.27
46.06
45.03
Ace.
23,62
51.44
31.80
55.38
47.98
74.89
EuToSAT
Rob.
16.48
35.37
21
31.80
39.10
31.88
43.86
H
19.41
41.92
25.08
36.20
47.78
40.47
45.83
61.71
Ace:
53.02
55.86
61.45
UCP101
Rob.
35.13
34.75
40.47
39.15
41.63
44.69
58.58
50.52
51.23
65.40
H 41.00
41.65
46.94
45.67
49.23
51.61
60.73
59.53
56.76
67.20
Table A6: The performance on the 11 datasets for different shots under the adversarial few-shot classification setting when c = 4/255, where 'H' denotes the harmonic mean accuracy.
sbot
shox
Datsset
4/255
APT
PAP
DEFFAT
FAP
DEFFATT
DEFBAT
Acc.
32.97
50.96
Average
13:44
11
16.34
17.19
19.58
20.87
21.49
22.83
32.14
25.47
Acc.
40.31
37.88
36.95
38.94
39.25
ImsggeNer
Rob
10.79
11:41
12.48
11.49
11.85
12,66
12,02
13.03
18.42
H
16.19
16.80
17.44
18.70
17.76
18.15
19.15
18,60
15.54
Ace.
78.78
77.89
76.67
77.61
Caltecha11
Rob
43.61
45.84
46. 36
$1.03
47.59
51.81
58.74
56.75
55.42
75.30
11
56.14
57.71
59.31
62.18
60.12
62.14
68.85
63.47
65.49
82.70
Avc.
66.26
59.58
63.01
57.51
65.59
66.85
69.65
67.29
69.09
72.17
OxfordPass
Rob.
15.56
15.54
20131
22.16
17:30
22.59
27.42
19.95
26.17
30.77
H
25.00
24.65
30.72
31.99
28.04
53.77
30.32
30.81
37.96
43.14
Acc.
11.32
19.49
27.41
22.96
23.53
33.96
25.90
31.60
41.06
52.45
SuanfrrCars
Rob
0.99
2.90
3.27
4.74
4.44
4.76
8.20
7.70
6.33
19.51
1I
1.81
5.05
5.54
7.36
7.49
3.35
12.46
12.38
10.97
24.37
Acc:
30.13
35.03
42.55
34.59
60.25
55.58
56.1 11
76.41
16. 10
71.13
Flowess02
Rob.
8,53
11.49
14.21
14.01
22.45
20.46
48.60
37.52
30.82
58,26
H
13.78
17:41
21.30
19.94
32.71
25.91
52.09
50.53
42.04
64.155
Acc.
23.52
20.33
33.46
26.06
21.97
38,84
20.75
30.99
45.60
53.57
Food101
Robs
3.27
3.42
5.46
6.01
4.01
6.39
8.29
7.90
5.23
20,50
II
5.74
5.87
9.39
9.83
6.78
10.97
12.97
12.54
13.94
25,46
Azc.
7.17
1.41
11.94
11.67
15.27
14.58
20.31
18.57
18.03
PGVCAireaaft
Rob
0.36
0.42
2.50
2.73
0.51
2.75
10.36
6.15
5.07
11.94
H
4,36
Acc.
33.24
36,44
35.68
35.68
41.29
SUN999
15.58
27.77
DTD
Rob.
10.57
15.60
13.45
15.63
16.16
20.32
22.02
Acc.
21.09
28.02
40.58
29.47
57.37
EunSSAT
Rob.
14.53
13.04
15.77
16.44
3.57
28.07
23.06
54.65
11
14:26
17.21
18.62
19.49
24.74
14.15
32.81
36.56
32.90
56.69
Ace.
36.08
33.10
35.58
36.56
46.21
48.27
UCP101
Rob
7.01
7.72
8.86
12.11
11.66
14.27
29.32
16.55
18.50
43,03
H
11.79
12:52
14.19
18.19
18.50
21.81
30.45
23.24
27.10
45.45
17

---

Published as a conference paper at ICLR 2026
Average
ImageNet
Caltech101
OxfordPets
40
80
Score (16)
30
20
-0
Score (36)
Score
60
15
50
10
No. data per class
train per class
No. data per class
No. per class
StanfordCars
Flawers102
Food101
FGVCAircraft
60
25
25
Score 1861
Score 1961
Score 196)
10-
Score
40
15-
10
10-
20
5-
16
0.2
No. train data
of train
train data
of train class
SUN397
DTD
EuroSAT
UCF101
Score (%6
Score (90)
Score 190
40
Score 196)
20
20
20
0
16
16
0
16
0
16
No. of train data class
No. ofain data per class
No. of train data per class
No. of train data per class
Figure A7: Performance on the datasets under the adversarial few-shot classification setting when 4/255. As the accuracies among prompt tuning methods are comparable, we only plot the curves for robustness and 'H' to improve readability.
Effect of A. In Eq. 10). À controls the weight of flagr which is designed to ensure consistency
between the Proj and the CLIP pre-trained class feature I. According to Figure A8c increases, the overall performance of DEFEAT improves, and then remains stable therealter (i.c., À > 10), making it easy to choose an appropriate in practice. Those results highlight the importance of regularizing prooj In our experiments, we set 10.
Average Accurcyy (N)
80
Average Acturaor (N6
-5
50
Arsunety
50
46
45
90
40
35
80
30
0
0.05
0.1
0.5
20
0.01
0.05
0.1
0.5
5
15
20
(a)
(b)
(c)
Figure A8: The average performance of DEFEAT on all datasets w.r.t. hyperparameters (i.e., a, B, and A) under the adversarial few-shot classification setting.
C.3 ANALYSIS OF LOGITS FUSION STRATEGY
In this section, we analyse a variant of the logtis fusion strategy. Specifically, to control the trade-off between robustness and accuracy flexibility, can modify the fused logits in Eq. 7 to the following form:
p(y|x) (1 w) (y)()) + wpp (v/x)
(11)
where w is the hyperparameter. We analyze how different w values impact DEFEAT's performance. Figure A9 shows that as increases, accuracy gradually improves because the logits of I and ts, which contribute to clean accuracy, have a larger weight in the fused logits. Robustness and 'H' initially increase with w, then plateau, and eventually decline. Although the range of w values where 'H' stabilizes varies slightly across different datasets, w = 0.5 consistently falls within or near this range. Therefore, set 0.5 for all datasets in our experiments.
18

---

Published as a conference paper at ICLR 2026
90
Alcurazcy (30
80
80
Accurac (55
40
50
130
20
0.10.20..3 0.4 0.8.3......5.0.000.0... 0.6.0.000.00
0.4.2..33444..000
C302030040506670880
1..02.0.0.44..33 0.5.0......000000....00
W
(a) Caltech101
(b) EuroSAT
(c) UCFI01
(d) OxfordPets
Figure A9: Performance of DEFEAT on Caltech101, EuroSAT, UCF101, and OxfordPets datasets w.r.t. different values under the adversarial few-shot classification setting.
C.4 ADVERSARIAL ROBUSTNESS EVALUATION UNDER VARIOUS ATTACKS
In this section, evaluate the adversarial robustness of prompt tuning methods using wider variety of attacks, including Carlini & Wagner (CW) attack (Carlini & Wagner 2017) and AutoAttack (AA)
(Croce & Hein 2020). CW is stronger optimization- based adversarial attack that finds the minimal perturbation required to cause model to misclassify input. AutoAttack (Croce & Hein 2020) standardized benchmark suite that combines multiple attacks to evaluate model's adversarial robustness reliably. Specifically, we use the standard AutoAttack setting, which executes suite of four attacks: APGD-CE (untargeted), APGD-T (targeted), FAB-T (targeted), and Square (black-box). All component attacks utilize the default setting of 1 restart. We apply non-deterministic seeding to each component attack. As shown in Table AA] both CW and AA generate more potent ad adversaiial examples than PGD, resulting in lower robustness for all methods (i.c., APT, FAP, and DEFEAT). Despite this, DEFEAT consistently outperforms the APT and FAP baselines across all datasets. This consistent superiority against diverse and powerful suite of attacks provides strong evidence that DEFEAT's robustness is genuine and not an artifact of attack-specific overfitting or gradient obfuscation.
Table A7: Performance of adversarial prompt tuning methods under the adversarial few-shot classifi- cation setting with various attacks.
16shot.
4/255
Average
Calsech100
OxfordPets
StanforCCass
Flowers
Method
Ace.
PGD
AA
CW
Ace.
PGD
AA
CW
Ace:
PGD
AA
CW
Ace.
PGD
AA
CW
Ace.
PGD
AA
CW
TeCoA
33.00
11.37
9.81
11.07
78
78.78
43.61
40.93
41.30
66.26
15.56
11.28
15.26
10.32
0.99
0.62
0.99
30.13
891
6.54
7,59
APT
52.85
22
17.27
2007
86
56.75
53.75
56.51
67.29
19.98
13.33
16.98
31.60
7.70
3.69
5.19
76.41
57.52
30.86
33:46
FAP
53,17
21662
16.02
17.58
80.04
55.42
50.87
53.35
26.17
16.46
18.34
41.06
633
2,43
66.10
30.32
23.02
24.81
30.77
Foo1101
PGVCAiresft
EanSAT
Method
Ace.
PGD
AA
CW
Ace.
PGD
AA
CW
Ace.
PGD
AA
CW
Ace.
PGD
AA
CW
Acc.
PGD
AA
CW
TeCrA
23.52
3.27
1.81
2.57
7.17
0.36
0.06
0.21
24.20
11:35
9.75
10.17
19.56
11:23
10.49
11.30
16.98
7.11
5.26
7.22
APT
30.39
7.00
4
5.54
20.31
6.15
3.06
4.05
45.86
21.51
15.07
17.14
64.33
25.54
18:35
25.04
53.16
16.55
12.82
16.68
FAP
45.60
8.23
3,92
5,37
18.57
5.07
2.58
3,06
12.85
20.98
17.32
18.09
61.51
23.06
15.19
16.33
50.67
18.50
12/42
14.83
C.5 ADAPTIVE ATTACK ON DEFEAT
To provide more rigorous evaluation and demonstrate DEFEAT's robustness, we design a defense- aware adaptive attack specifically tailored to circumvent its core architectural components. Following principles outlined by /Athalye al. (2018), an adaptive attack must take the specific defense mechanism into account. Our attack targets the logits fusion strategy, which is unique characteristic of the DEFEAT framework.
An attacker with knowledge of the DEFEAT architecture would realize that the final prediction comes from standard, continuous feature branch and a defended, discretized feature branch. Based on this insight, hypothesize that the most effective adaptive attack strategy is to identify the more vulnerable of these two branches and concentrate the full adversarial pressure on it.
To validate this hypothesis and identify the worst-case attack, design logits fusion-aware adaptive attack. Instead of maximizing the standard cross-ent cross-entropy loss on the fused output, we maximize weighted of the individual cross-entropy losses from each branch. The adaptive attack loss
19

---

Published as a conference paper at ICLR 2026
40
35
Rebustness
30
25
20
15
10
0.1
0.2
0.5
0.6
0.80.9
Figure A10: Robustness of DEFEAT on the DTD dataset under the logits fusion- aware adaptive
attack with varying values. Standard PGD attacks on CLIP and DEFEAT are provided as baselines.
Ladept is defined as:
(x, (xxxxt,y7) t',y) = E.
i=1 Ys log (b/x)
(12)
(x, t², t) Σ Vi log (y|x)
(13)
Cadept = K* Le (x, t", y)
n:) (x.
(14)
where p²()/x) and (y)x)) are defined Eq. 4 and Eq. (6), is a hyperparameter controls the focus of the attack. By varying the value of we can explore the effect of attacking different branches. We then use a PGD-based optimizer to find the perturbation that maximizes Ladapt.
We conduct experiments setting with ε = 4/255 to analyze DEFEAT's performance under this adaptive attack. The results are presented in Figure A10
the DTDD dataset under the adversarial few-shot (16-shot) classification
The experimental results clearly validate our hypothesis. When we concentrate the full attack pressure on the standard, non-discretized feature branch (K = 1), DEFEAT's robustness drops to its lowest point of 18. 18.20%. This demonstrates that, with knowledge of DEFEAT's s defense mechanism, the most effective adaptive attack is indeed focus entirely the branch unprotected by the PerturbShield module.
Conversely, when the attack exclusively targets the defended PerturbShield branch (k -), the model's s robustness reaches its highest point of 34.75%, even surpassing its performance against standard PGD attack on the fused logits (32.33%). This provides strong evidence for the effectiveness of our proposed discretization module.
In summary, this experiment validates the design of our adaptive attack. Even under this custom- designed, worst-case adaptive attack, DEFEAT's robustness (18.20%) remains significantly higher than the CLIP baseline (11.35%), providing evidence for the good robustness of DEFEAT.
C.6 GENERALIZATION TO STRONGER PERTURBATION BUDGETS
critical measure of a defense mechanism's 's effectiveness its ability to generalize to attacks stronger than those seen during training. To evaluate this, we conduct challenging experiment where all models are trained using 16 examples per class under a standard PGD attack with perturbation budget of с 4/255, but are then evaluated against a much stronger, unseen PGD attack with e = 8/255..
As shown in Table A8 the results demonstrate consistent advantage for DEFEAT. While the stronger attack significantly degrades the robustness of baselines (compared to the 16-shot results in Table A6). DEFEAT maintains a substantially higher level of robustness across every dataset. On average, DEFEAT achieves robust accuracy of 23.32%, which is over 4 times higher than FAP (5.81%) and 5 times higher than APT (4.62%)
Crucially, this significant gain in robustness achieved with no sacrifice in clean accuracy. DEFEAT remains competitive with the baselines in this regard. This experiment provides compelling evidence that DEFEAT learns more fundamental and generalizable defense mechanism. Instead of merely overfitting specific perturbation budget, its discrete feature reconstruction allows it more
20

---

Published as a conference paper at ICLR 2026
effectively ablate perturbations, making it more reliable defense for real-world scenarios where attack strengths are unknown.
Table A8: Performance against larger perturbation budgets (trained on e = 4/255, tested on e = 8/ 8/255).
roiin (1 - 4/255)
Average
Caltech101
[OxieedPets
FaroSAT
UCFFDDI
DTTD
Flowera102
[StanforCCrss
PGVCAirentt
best (e 8/255)
Method
Acc.
Rob.
Ace.
Rob
Acc.
Rob.
Aec.
Rob.
Acc.
Rob.
Ace.
Rob.
Aec.
Rob.
Acc.
Rob.
Acc.
Rob
APT
55,66
4.62
86.29
20.89
67.29
1.3
66.33
1622
133.16
1.43
45.36
6.68
76.41
4.26
31.60
0.52
20.31
0.27
PGD
FAP
25.03
5.68
DEFEAT
$4.88
23.32
87.63
S8.46
72.17
2.4
58888
31.09
55.43
21.46
44.15
19.56
71.13
36,09
32.45
7.61
18.03
9.33
C.7 GENERALIZATION TO ALTERNATIVE ROBUST BACKBONES
To demonstrate that the effectiveness of DEFEAT is not limited to a specific robust backbone, replace the TeCoA-trained image encoder (Mao et al. 2023) with a more recent and powerful one pre-trained via the FARE method (Schlarmann et al. 2024). We then evaluate all adversarial prompt tuning baselines under the adversarial 16-shot classification setting.
The results Table A9 show that DEFEAT maintains its superiority. Under the weaker PGD attack (e 1/255), DEFEAT achieves an average robustness of 69.33%, outperforming the next-best baseline (FAP) by nearly 15%. This advantage becomes even more pronounced under the stronger attack (c = 4/255), where DEFEAT's average robustness of 52.46% is over 20% higher than the strongest baseline (FAP at 30.53%). Those results validate that DEFEAT a generalizable framework that enhances robustness independently of the underlying pre-training method, and its benefits are magnified when paired with stronger robust encoders.
Table A9: Performance on CLIP ViT-B/32 image encoder pre-trained with FARE (Schlarmann et al. 2024
Average
Caltech101
OxfordPets
EurSAAT
UCFIDI
DTD
Flowers102
[SunfordCars] PGVCAireaaft
1/255 Method
Acc.
Rob.
Acc.
Roh.
Ace.
Rob
Acc.
Rob.
Aoc.
Roh.
Ace.
Rob
Acc.
Rob.
Acc.
Roh.
Acc.
Rob.
91.12
59.08
56.97
+APT
75.57
50.60
95:25
84.06
87.87
63.29
79.58
42.44
78.91
51.86
63.95
41.31
93,10
73.08
72.89
37.71
32,97
11.07
+PAP
71.73
54.79
93.67
86.17
88.53
72.31
M0.49
52.46
72.83
55.80
61.94
45.80
83.06
72.07
63:41
37.78
27.93
15.96
+DEFEAT
74.75
69.33
95.17
93/02
89.78
83.02
76.63
68.88
79.57
74.23
61.13
58.39
91.61
87.09
70.71
63.11
30.33
26.91
Average
Caltech101
OxfordPets
EuroSAT
UCFI0L
Flowers
StanfordCans
FGVCAinraftt
4/255
Method
Acc.
Rob.
Acc.
Rob.
Ace.
Rob
Acc.
Rob.
Asc.
Rob
Acc.
Rob.
Acc.
Rob.
Accc
Rob.
AAcc.
Rob.
FARE
42.81 15.87
36.00
$1.69
77.60
21.61
16.28
10.88
13.01
8.96
31.26
15.19
59.22
7.75
139.17
3.56
10.20
1.32
1APT
64.86 29.04
92.41
66.61
81.36
30.36
57.42
24.96
67.96
24.45
53.31
23.05
84.49
42.55
57.08
14.99
24,84
5.37
PGD
77.41
23.72
59.19
41.41
47.62
22.05
:DFFEAT
65.53
52.46
92.54
86.57
84.19
$2.79
63.77
52.02
69.10
$5.30
52.19
41.84
80.63
68.49
58.03
42.72
23.79
19.98
C.8 GENERALIZATION TO ALTERNATIVE VLM BACKBONES
To demonstrate that the effectiveness DEFEAT is not limited to the VIT-B/32 CLIP backbone used in our main paper, we conduct additional experiments on two distinct and powerful backbones: larger CLIP model (ViT-L/14) and model from a different pre-training family (EVA-CLIP (Sun) et al. )023 The following results validate that DEFEAT is a broadly applicable framework.
First, to assess the scalability of our method, we replace the TeCoA-trained VIT-B/32 (Mao et al. 2023) backbone with the much larger TeCoA-trained VIT-L/14, As shown in Table DEFEAT continues to provide significant robustness gains on this more powerful model. Those results demonstrate that method scales effectively with model size.
Next, we evaluate DEFEAT's adaptability by applying it to EVA-CLIP (Sun et al. 2023). which represents different pre- pre-training paradigm compared to the original OpenAl CLIP. The results in Table|ATT show that DEFEAT successfully enhances the robustness of this distinct model family. Those results validate that DEFEAT can serve as a general- purposs robust prompt tuning framework.
21

---

Published as a conference paper at ICLR 2026
Table A10: Performance on the TeCoA-trainee CLIP ViT-L/14 MMo et al. 2023) backbone against PGD attack (c = 4/255) under the adversarial fhw-shot (16-shot) classification setting.
Caltech101
DTD
1lowers102
Average
= 4/255
Method
Acc.
Rob.
H
Acc.
Rob.
H
Acc.
Rob.
H
Acc.
Rob.
H
CLIP
94.81
17.85
30.04
53.72
0.59
1.17
79.46
0.45
0.89
76.00
6.30
11.63
TeCoA
87.38
75.98
81.28
34.63
28.19
31.08
39.02
25.38
30.76
53.68
43.18
47.86
PGD
+APT
95.42
86.77
90.89
57.21
44.92
50.33
88.67
76.53
82.15
80.43
69.41
74.51
+DEPEAT
95.70
91.81
93.71
57.57
46.81
51.64
85,95
79.01
82.33
79.74
72.54
75.97
Table All: Performance on the EVA-CLIP (Sun et al., 2023 backbone against PGD attack 1/255) under the adversarial few-shot (16-shot) classification setting,
Caltech101
DTD
Flowers 102
Average
1/255
Method
Acc.
Rob.
H
Acc.
Rob.
H
Acc.
Rob.
H
Acc.
Rob.
H
EVA-CLIP
97.20
3.81
7.33
49.41
2.78
5.26
75.92
0.81
1.60
74.18
2,47
4.77
PGD
+APT
97.85
5.23
9.93
74.70
2.60
5.03
98.05
2.64
5.14
90.20
3.49
6.72
+DEFEAT
96.80
11.70
20.88
71.28
5.61
10.40
95.86
4.47
8.54
87.98
7.26
13.41
C.9 ROBUST VISUAL BACKBONE RELIANCE
In this section, we analyse the reliance of the robust visual backbone the proposed DEFEAT method. While DEFEAT significantly enhances adversarial robustness under few-shot prompt tuning conditions, relies heavily on pre-trained robust visual backbones g.. TeCoA (Mao al. 2023) like other adversarial prompt tuning methods (e.g., APT (Li et al., 2024) and FAP (Zhou et al.
2024 ). The results in Table 21 show that APT failed to improve robustness when not using
TeCoA. DEFEAT and FAP show less dependence on TeCoA when the perturbation size is small (i.e., c 1/255). However, using the robust image encoder TeCoA still substantially enhances the robustness of both FAP and DEFEAT. When increasing perturbation size to 4/255, we find that all methods, i.e., APT, FAP, and DEFEAT) exhibit near-zero robustness or even collapse during training. phenomenon also reported in Li et al. (2024); Zhou et al. (2024). The reasons may be the considerably smaller parameter space available for tuning compared to fine-tuning the entire model. Those results show that robust visual backbone is necessary for adversarial prompt tuning methods.
Table A A12: Performance of adversarial prompt tuning methods with (w/) or without (w/) TeCoA on Flowers and DTD datasets for 16 shots under the adversarial few-shot classification setting when e = 1/255. The numbers in the parentheses denote the improvement or decline compared to the methods without TeCoA.
APT
FAP
DEFFAT
Datasel
1/255
TeCoA
w/o TeCoA
w/ TeCoA
w/o TeCoA
w/ TeCoA
w/o TeCoA
w/ TeCoA
Acc.
48.15
85.06 (+36.91)
86,80 (438.65)
80.15 (+32.00)
82:46 (+34.31)
91.72 (++3.57)
84,90 (+36.75)
Flowers102
Rob
32.68
0.85 (-31.83)
73.57 (+40.89) 89)
51,36 (+18 68)
66.63 (+33.95)
55. (423.31)
81.81 (+499.13)
H
38.93
1.68 (-37.25)
79 (+40.71)
62,60 (+23.67)
(4.4.77)
69.53 (+30.60)
83.33 (+44.440)
Acc.
(+28.14) 14)
55 (+23.29)
55.50 (+23.533
(+24.71)
(+33.16)
(+25.
DTD
Rob.
23.88
2.60 (-21 28)
39.54 (+15 66)
28.67 (+4
41.67 (+17. 79)
30.91 (47.05)
52.66 (+28.78)
27.34
4.98 (-22.35)
46. (+1876)
37.81 (+10.47)
48.03 (+20.69)
41.92 (4..58))
54.81 (+27.47)
1. COMPUTATIONAL COST ANALYSIS
To evaluate the practicality and efficiency of DEFEAT, we provide comprehensive analysis of its computational overhead compared to APT and FAP. For a fair comparison, adopt the same experimental settings as APT (c.g., training epochs, schedules, and data augmentation), while for FAP, maintain its original implementation to ensure faithful reproduction of its results. All experiments were conducted on the ImageNet dataset under the adversarial few-shot (16-shot) classification setting with 4/255.
The results in Table A13 show that DEFEAT achieves a highly favorable trade-off between computa- tional cost and robustness.
22

---

Published as a conference paper at ICLR 2026
Training Memory. The training memory required by DEFEAT (21240M) is nearly identical to that of APT (21204M) and significantly lower than that of FAP (37500M). This indicates that our PerturbShield module adds minimal GPU memory overhead.
Training Time The training time of DEFEAT (19.21h) is negligibly longer than APT's (18.99h) and less than half that of FAP (39.04h), demonstrating clear efficiency advantage.
Inference Time. During inference, the per- image processing time for DEFEAT (3.1431ms) is only a minor increase of 0.15ms over APT (2.9959ms),, which is insignificant for practical applications.
In summary, the computational overhead introduced by our method is minimal during both training and inference. This slight cost is a highly favorable trade-off for the substantial improvement in robustness that DEFEAT provides over both APT and FAP, showing the practicality and efficiency of our method.
Table A A13: Computational cost and performance comparison on ImageNet under the adversarial few-shot (16-shot) classification setting with 4/255.
Method
Train.Memory
Train.time (h)
Infer.time (ms)
Acc.
Rob.
H
APT
21204M
18.99
2.9959
41.06
12.02
18.60
DEFEAT (ours)
21240M
19.21 (+0.22)
3.1431
41.09
18.42
25.44
FAP
37500M
39.04
3.0020
40.45
12.03
18.54
C.I1 PERFORMANCE COMPARISON UNDER THE & THREAT MODEL.
We extend our experiments the lg threat model, evaluating all methods against the PGD-&z (e 0.5)
attack across 10 downstream datasets.
As shown in Table A14 DEFEAT performs well under the & threat model. Our method achieves an average robustness of 50.24%, surpassing both APT (47. .44) and FAP (47.07%). This result demonstrates that the robustness gains provided by DEFEAT are generalizable and not restricted to the Ex norm.
Table A14: Performance Comparison under the adversarial few-shot (16-shot) classification setting with the &g threat model.
PGI-6
Average
Caltecht0
OxfordPats
[StanfoniCar
Plowers
Foodi0i
|POVCAirenni)
SUN997
DTD
EursSAT
UCF101
Method
Aoc.
Rob.
| Acc.
Rob,
Acc.
Rob.
Acc.
Rob
Acc.
Rob,
Act.
Roh.
Acc.
Rob.
Acc.
Rob
Asct
Roh.
Accc
Rob.
Acc.
Rob.
3308 20.17
66626
33.24
24.20
17.71
47.34
31.60
76A1
53.16
52:22
47.07
41.06
3333
DEFEAT
51:97
50.34
57.63
67.30
32.45
30.93
71.13
59.58
55,43
54.27
C.12 COMBINATION WITH TEST-TIME DEFENSE STRATEGY
To address the scenario facing strong attacks without a pre-trained robust backbone, we propose DEFEAT-T, variant of the DEFEAT method. DEFEAT-T successfully adapts our core idea to the non-robust backbone scenario by combining it with a test-time defense strategy (Shu et al. 2022
Wang et al. 2025 Sheng al. 2025).
The core of DEFEAT-T is to pivot our method from an "active defender" (via adversarial training) to "passive purifier" (via clean training), combined with efficient, optimization-fres test-time ensemble. The mechanism is structured into two phases:
Phase 1: Clean Training. We first recognized that adversarial training on non-robust backbone
could be a source of the training collapse. Therefore, we discard adversarial training. Instead, we train DEFEAT framework (including the learnalle prompt and PerturbShield module) standard (non-robust) backbone using clean data only. In this phase, our PerturbShiel module learns to reconstruct the clean feature distribution of the non-robust backbone.
Phase 2: Test-Time Ensemble Defense. At test time, we execute forward-pare pass ensemble defense, avoiding per-sample backpropagation optimization, which major bottlenecks for test-time
23

---

Published as a conference paper at ICLR 2026
prompt tuning methods (Shu et al. 2022 Wang et al. 2025 Sheng et al. 2025 First, we generate
Noug (c.g., N6))=66) augmented views (c.g., random crop, random horizontal flip) of the test sample. Second, all Naug views are fed through our model trained from clean data, to get Na fused logits by Eq. 7 as shown in Figure Next, to obtain the final prediction, one can directly average these Naug fused logits. Here, have better performance, Specifically, we use the Na class features (which are collected alongside the grid features) to compute similarity matrix. We then average the top-k (e.g., k=20) values in each row get reliability score for each augmented view. Finally, we use these reliability scores as weights for a weighted average of the corresponding fused logits to obtain the final prediction.
perform reliability- -hased weighted average.
The rationale for this design is our hypothesis that this view augmentation and weighted-averae prediction could mitigate the impact of unseen adversarial perturbations, and that combining it with our DEFEAT framework provides a second stage of purification.
Table A15: Combination adversarial prompt tuning methods with test-time defense strategy under the adversarial few-shot classification setting.
Average
Caltech101
OxfondPets
Flowers
DTTD)
I6shot, 4/255
Method
Rob.
Rob.
Acc.
Rob.
CLIIP
72.38
2.28
4.42
91.03
7.79
87.38
1,20
66.91
0.12
44.21
0.00
zcro-shot
TeCoA
49.87
19.86
28.41
78.78
43.61
66.26
15.56
30.13
8,93
24.29
11.35
Test-time prompt tuning
R-TPT
59.00
90.83
77.00
85.12
57.32
42.67
29.67
APT wlo TeCaA
14.05
0.07
0.14
43.77
0.16
3.38
0.00
3.45
0.00
5.61
0.12
DEFEAT w/o TeC
18:45
0.16
0.32
51.89
0.41
3,05
0.00
9.62
0.00
9.22
0.24
Train-time prompt tuning
APT TeCoA)
56.75
67.29
21.51
DEFEAT (w/ TeCoA)
68.77
49.92
57.84
87.63
78.30
72.17
30.77
71.13
58.26
44.15
32.33
Train-lime prompt tuning test-timee
APT-T (wfo TcCoA)
86.72
43.33
57.78
95.01
71.08
90.19
32.54
94.76
39.18
66.90
30.50
DEFEATT TcCoA)
93.63
86.54
68.32
57.27
Empirical Verification. As shown in Table A15 DEFEAT-T (w/o TeCoA) achieves good average performance (87.76% Acc., 75, 76%96% Rob., 8 8..43% H), surpassing test-time prompt tuning methods like R-TPT (70.23% Acc., 50.86% Rob., 59.00% H), and the standard DEFEAT (w/ TeCoA) (68. 77% Acc., 49.92% Rob. 57.84% H) even without a robust backbone.
Simultaneously, we also extend the baseline APT to APT-T (w/o TeCoA) (i.e., APT trained on clean data, evaluated with the same test-time ensemble but without the logits fusion strategy and the PerturbShield module). We find that APT-T (w/o TeCoA) (43.33% Rob.) achieves performance comparable to the original DEFEAT (w/ TeCoA) (49. 99.9 Rob.). However, its robustness is still significantly weaker than DEFEAT-T (75.96% Rob.). This comparison further demonstrates the effectiveness of our proposed logits fusion strategy and PerturbShield.
Crucially, our DEFEAT-T method has decisive efficiency advantage over existing test-time prompt tuning methods (Shu et al. 2022 Wang et al. 2025 (hhng et al.] 2025). Existing methods require per-sample optimization (batch size=1) involving backpropagation, making them slow. Our DEFEAT- T is an inference-only scheme with no gradient calculations, allowing for batch parallelism and much faster inference. However, even so, the time cost of DEFEAT-T's inference still much higher than that of the original DEFEAT. As shown in Table A16 the inference time of DEFEAT-T is approximately 2x slower than that of DEFEAT, On the other hand, the training time for DEFEAT-T is much shorter, as it does not require adversarial training.
In summary, DEFEAT and DEFEAT-T can be adopted in different scenarios. The original DEFEAT (w/ TeCoA) is ideal for low-latency inference scenarios, whereas DEFEAT-T is a perfectly viable and highly robust alternative when a robust backbone not available.
Table A16: Computational cost and performance comparison between DEFEAT, DEFEAT-T, and R-TPT (Sheng et al. 2025) on Caltech101 under the adversarial few-shot (16-shot) classification
setting with c 4/255.
Method
Train.Time (s)
Infer Memory
Infer:Time (s/per img)
Acc.
Rob..
H
DEFEAT (w/ TeCoA)
3597
8481M
0.09
93.39
91.12
92.24
DEFEAT-T (w/ TeCoA)
1079
10420M
0.19
96.19
95.58
95.88
R-TPT
0
5248M (batch=1)
6.63
90.83
86.13
88.42
24

---

Published as a conference paper at ICLR 2026
C.13 THEORETICAL JUSTIRICATION
Here we provide a theoretical justification for why clean and adversarially perturbed features can be mapped to the same discrete representation within the VQ-VAE framework.
Let Ene(.) denote the composite visual encoder, which consists of the CLIP image encoder followed by the trainable VQ-VAE encoder. The clean latent features output by this composite encoder are denoted by 2 = (n)(z) E Rnxd (where n is the number of latent positions after encoding, is the dimensionality of the latent embedding). Under an adversarial attack with perturbation ó, the adversarial latent features become:
Enc( x + 6) 2 Az,
(15)
where As Rnnd is the mapped perturbation in the latent space.
The core of VQ-VAE is to discretize continuous latent features through a codebook & = {ek k=1 (where K is the codebook size). The covering radius r of the codebook is defined
max min || - =s1122
(16)
2E2 E7EE
where 2 is the set of latent features output by the encoder from all clean images. The covering radius r characterizes the "coverage capability" of the codebook over the clean feature space: the distance from any clean feature to its nearest codebook element does not exceed r, and r minimized through the codebook loss of VQ-VAE.
Based on the above definition, if the perturbation magnitude the latent space satisfies || S r, it is statistically highly probable that both the clean feature 25 and the adversarial feature 2 map to the same codebook element Eş E E. Notably, the training objective in DEFEAT encourages images that are more similar at the pixel level to be closer in feature space. So the feature shift As induced by minor perturbations is typically much smaller than the codebook covering radius r, effectively mitigating such adversarial attacks during the quantization process.
Empirical Verification. To validate this, we conduct a statistical analysis on the latent space using the Caltech 101 dataset (16-shot, = 4/255). We measure two key metrics across all image patches:
The average magnitude of adversarial perturbation per patch: ||>=|| R 126.0.
The average distance between clean latent features and their assigned codebook vectors (serving proxy for the safety radius): R 432.3.
The experimental results show that the covering radius approximately 3.43 times larger than the perturbation magnitude (432.3 vs. 126.0). This substantial margin implies that the vast majority of adversarial perturbations are insufficient to push the latent features out of their original discrete region defined by their assigned code. Consequently, the clean and adversarial patches quantized to the same discrete codes, confirming the effectiveness of our defense.
C. STABILITY ANALYSIS WITH MULTI-SEED EVALUATION
To ensure the reproducibility of our results and confirm that the reported performance is not an artifact of a single run, we conduct a stability analysis by re-running our primary adversarial few-shot classification experiments. Specifically, we evaluate the models under the 16-shot setting with PGD attack (c 4/255) using three random seeds (0, 1, and 2).
Table A17 compares our originally reported results (derived from Table [ and Table A6 with the new averages calculated across three seeds. As observed, the multi-seed averages align closely with our reported single-seed results. Crucially, the robustness advantage of DEFEAT remains consistent and stable. For instance, the multi-seed average harmonic mean for DEFEAT is 41.66%, which still substantially surpasses the strongest baselines, APT (28 90%) and FAP (28.07%) This consistency confirms that our results are reproducible and statistically robust.
C.15 STABILITY ANALYSIS AGAINST PROMPT VARIATIONS
As shown in Table A19 DEFEAT exhibits remarkable stability. The robustness fluctuates only slightly (between 31 1.09 and 32.33%) across different prompt variations.
25

---

Published as a conference paper at ICLR 2026
Table A17: Comparison between originally reported results and averages over 3 seeds
16shol, 4/255
Average (reported) Roh.
Average on 3 seeds
Method
Acc.
H
Acc.
Rob.
TeCoA
33.67
10.79
16.34
33.67
10.78
16.33
APT
51.08
20.26
29.02
51.27
20.12
28.90
FAP
50.50
19.82
28.47
50.68
19.41
28.07
DEFEAT
50.98
36.05
42.24
50.85
35.28
41.66
Table A18: Complete results calculated by averaging three seeds across all datasets. (16shot, 6 4/255)
ImageNet
Caltech101
OsfordPets
StanfordCCrss
flowersig 102
Food101
CGVCAineaat
SUN397
DTD 24.29
EuroSAT
UCP101
Mathod
Rob.
10.15
19.56
12:20
25.87
Crucially, the performance trends remain consistent across all variations. Even under the least optimal prompt setting ("(} texture."), DEFEAT (31.09% Rob.) significantly outperforms the strongest baseline, APT (20.26% Rob.), by margin of +10.83%. This confirms that our method's superiority intrinsic to its design rather than an artifact of specific prompt engineering.
Table A19: Stability analysis of DEFEAT against different hand-crafted prompt templates on the DTD dataset.
16shot, e 4/255
DTD
Method
Acc.
Rob.
H
TeCoA
33.67
10.79
16.34
APT
51.08
20.26
29.02
FAP
50.50
19.82
28.47
DEFEAT with different hand-crafted prompts
"a photo of (1. type of texture."
44.15
32.33
37.33
"() texture."
44.74
31.09
36.69
"a photo of ()."
44.03
31.80
36.93
D USE OF LARGE LANGUAGE MODELS
The LLM served as an assistive tool for language editing and manuscript polishing. Its usage was confined improving grammar, refining phrasing for clarity, and ensuring idiomatic English.
26


Reviews:
[{"review_id": "Aerj6IpGuR", "reviewer": "Reviewer_QZPj", "summary": "The authors propose DEFEAT (Discrete LatEnt Feature based Adversarial Training), a new robust prompt tuning framework for VLMs. The core hypothesis is that discretizing latent features can effectively mitigate adversarial perturbations. The authors conduct extensive experiments on 15 datasets across adversarial few-shot classification, domain generalization, and cross-dataset generalization settings. The results show that DEFEAT achieves state-of-the-art robustness and a better accuracy-robustness trade-off compared to existing adversarial prompt tuning methods like APT and FAP.", "questions": "Since all tested adversarial prompt tuning methods fail without a robust backbone, is there any possible way to adapt your method under this scenario?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A Novel and Insightful Defense Mechanism: The core idea of using a VQ-VAE to create discrete latent feature representations as an adversarial defense is a novel contribution in the prompt-tuning space. The design is particularly insightful for choosing to discretize the grid features ($I_{patch}$) rather than the single, global class token ($I$).\n\n2. Comprehensive experiments: Extensive experiments across 15 datasets are conducted to verify the effectiveness of the DEFEAT method. The paper provides good ablation studies that clearly demonstrate why the proposed components are necessary.", "weaknesses": "Heavy Reliance on a Pre-Trained Robust Backbone:  As the authors report in Section C, DEFEAT requires a robust visual backbone (like TeCoA) to function. When applied to a standard, non-robust CLIP model, the PGD attack can still pose significant threats.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0STIR4Bbr", "reviewer": "Reviewer_spep", "summary": "This paper proposes DEFEAT, a robust prompt tuning framework for vision-language models (VLMs) that enhances adversarial robustness by leveraging discrete latent features. The method employs a VQ-VAE-based Perturbation Discrete Shield (PerturbShield) to map both clean and adversarial features into shared discrete representations, combined with logits fusion and prompt alignment strategies. Experiments across multiple datasets demonstrate significant improvements in adversarial robustness while maintaining competitive clean accuracy.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The method is novel, introducing discrete latent representations into the robust prompt tuning framework, which provides a fresh and intuitive perspective on defending against adversarial perturbations.\n- The experimental results are strong, showing substantial improvements in robustness accuracy compared with existing adversarial prompt tuning methods.\n- The paper provides rich visualizations of feature distributions, which help illustrate and explain how the proposed method mitigates adversarial effects.", "weaknesses": "- The paper lacks a theoretical justification for why clean and adversarially perturbed features can be mapped to the same discrete representation within the VQ-VAE framework.\n- The approach involves a large number of hyperparameters (\\alpha, \\beta, \\lambda, \\mu), making the method relatively hard to tune and potentially sensitive to configuration choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c36K588tyH", "reviewer": "Reviewer_vWTM", "summary": "This paper presents a novel and well-executed framework for improving the adversarial robustness of VLMs through prompt tuning. Extensive evaluation on 15 datasets under few-shot, cross-dataset and domain-generalisation settings.", "questions": "Please refer to the questions raised in the Weaknesses section above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Introduces a novel application of latent feature discretization to adversarial defense for VLMs.\n\n2. Comprehensive evaluation across 15 datasets, covering adversarial few-shot classification, cross-dataset, and domain generalization.", "weaknesses": "1. This paper presents a well-structured framework. however, the underlying techniques for the individual components are not entirely novel.\n\n2. The paper makes a significant contribution to improving the adversarial robustness of CLIP. However, a limitation of the current study is that all experiments are exclusively conducted on the CLIP model. To broaden the applicability and contribution of the proposed methods, it would be highly beneficial to include experimental results on other relevant models, such as OpenCLIP and EVA-CLIP. Furthermore, the empirical evaluation focuses exclusively on image classification.The absence of experiments on other applications limits the demonstrated scope and real-world applicability of the DEFEAT method.\n\n3. The practical utility of the DEFEAT method is not fully clear due to the lack of analysis on its computational cost relative to baseline methods. \n\n4. The discussion could be further enriched by considering recent advancements in test-time defenses. if the authors could investigate whether combining DEFEAT with pre-processing defenses [1] or test-time defenses [2,3] could lead to synergistic effects, potentially achieving a \"1+1>2\" outcome.\n\n[1] Diffusion Models for Adversarial Purification, ICML 2022\n\n[2] CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP, CVPR 2025.\n\n[3] TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models, CVPR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEsUu2cpv9", "reviewer": "Reviewer_DTmv", "summary": "The paper proposes DEFEAT, a robust prompt-tuning framework for CLIP-like VLMs that discretizes latent grid features with a VQ-VAE module, and fuses logits from discretized and original branches, coupled with prompt-alignment regularization to hand-crafted prompts. The central claim is that feature discretization reduces clean->adv feature shift, yielding substantially higher robust accuracy and harmonic mean scores under few-shot, cross-dataset, and domain-generalization settings. The paper provides PGD evaluations, AutoAttack and CW, and a defense-aware adaptive attack targeting the fusion branches. Overall, the method is compact (frozen encoders; train *PerturbShield* + prompts) and the empirical coverage is broad, though statistical reporting and adaptive-attack rigor need tightening.", "questions": "- What are the typical failure cases where discretization/fusion underperform (e.g., fine-grained textures, clutter, small objects), and what mitigation directions look promising?\n- Are the reported robustness gains consistent across multiple seeds, and do they remain visible under paired comparisons against the strongest baseline?\n- How stable are results to reasonable prompt variations (e.g., templated vs hand-crafted phrasing), and do trends remain when prompts are slightly perturbed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear discretization motivation with reduced feature shift\n- Simple modular defense (VQ-VAE grid reconstruction + projection + fusion + prompt regularization)\n- Broad empirical coverage with consistent robustness/H gains; AA/CW and adaptive checks provided\n- Ablations clarify each component’s role; µ improves robustness", "weaknesses": "- Adaptive attack lacks explicit BPDA/ST gradients through quantization, risking masking\n- AA/CW only on subset; AA configs (targeting, restarts, seeds) unspecified in main text\n- Threat model focused on ℓ∞; no ℓ2/spatial in main paper\n- No multi-seed CIs; missing runtime and codebook specs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)