Learning Robust Vision-Language Models from Natural Latent Spaces
Authors: Zhangyun Wang
Ni Ding
Aniket Mahanti
Venue: NeurIPS 2025 spotlight ()
Keywords: Adversarial Robustness
Prompt Learning
Vision-Language Models
Primary area: deep_learning
TL;DR: We propose CoAPT, a collaborative adversarial prompt tuning method that significantly improves the robustness and generalization of vision-language models against adversarial perturbations.

Abstract: Pre-trained vision-language models (VLMs) exhibit significant vulnerability to imperceptible adversarial perturbations. Current advanced defense strategies typically employ adversarial prompt tuning to improve the adversarial robustness of VLMs, which struggle to simultaneously maintain generalization across both natural and adversarial examples under different benchmarks and downstream tasks. We propose a collaborative adversarial prompt tuning (CoAPT) approach from pre-trained VLMs to target robust VLMs. Inspired by the image mask modeling, we adopt an improved real-time total variation algorithm to suppress and eliminate high-frequency details from images while preserving edge structures, thereby disrupting the adversarial perturbation space. Subsequently, guided by the high-level image and text representations in the latent space of the pre-trained VLMs, the corrupted natural features are restored while inheriting the superior generalization capability. Experiments on four benchmarks demonstrate that CoAPT achieves an excellent trade-off among natural generalization, adversarial robustness, and task-specific adaptation compared to state-of-the-art methods.

Learning Robust Vision-Language Models from Natural Latent Spaces
Zhangyun Wang"
Ni Ding
School of Computer Science
School of Computer Science
University of Auckland
University of Auckland ni ww..ggaucklladd.o ac nz
zwna8756aucklandumi.. nz
Aniket Mahanti
School of Computer Science
University of Auckland
a. mahanti (arrikiakkkkk aaaadddd........ ac. ac.nz
Abstract
Pre-trained vision-language models (VLMs) exhibit significant vulnerability to imperceptible adversarial perturbations. Current advanced defense strategies typ- ically employ adversarial prompt tuning to improve the adversarial robustness of VLMs, which struggle simultaneously maintain generalization across both natural and adversarial examples under different benchmarks and downstream tasks. We propose collaborative adversarial prompt tuning (CoAPT) approach from pre-trained VLMs to target robust VLMs. Inspired by the image mask modeling, we adopt improved real-time total variation algorithm to suppress and eliminate high-frequency details from images while preserving edge structures, thereby dis- rupting the adversarial perturbation space. Subsequently, guided by the high-level image and text representations in the latent space of the pre-trained VLMs, the corrupted natural features are restored while inheriting the superior generalization capability. Experiments on four benchmarks demonstrate that CoAPT achieves an excellent trade-off among natural generalization, adversarial robustness, and task-specific adaptation compared to state-of- tatte-of-the-art the-art methods.
Introduction
Vision-language models (VLMs) such as [LIP and ALBEF[2] have shown significant potential for application in multiple industry ecosystems in recent years. However, recent studies B have revealed that VLMs exhibit range of concerning vulnerabilities in real-world deployment. When confronted with distributional biases, adversarial samples, or semantic ambiguities, they often display reasoning biases that deviate from human cogmition. As an increasing number of downstream applications built upon VLMs as foundational models emerge, the chain reactions triggered by the vulnerability of VLMs pose serious threats the security and reliability of multimodal downstream tasks. In this paper, we holistically investigate the vulnerabilities of VLMSs and their adversarial robustness, with particular focus on the typical base model CLIP.
Current adversarial robustness strategies for VLMs primarily include model fine-tuning and adver- sarial prompt tuning. During adversarial training, model fine-tuning [5 relearns the entire set of model parameters to adapt to adversarial examples. This process disrupts the natural data distribution captured by the pre-trained model, leading to contradiction between robustness and generalization.
"ooresponding author: Zhangyun Wang (zwna875 @ aancklnnuuni.ac.z)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Adversarial prompt tuning 77 8, 9 improves the robust adaptability of VLMs by guiding the pre- trained models to efficiently adapt to adversarial data distributions, without altering the pre-trained model parameters. Textual adversarial prompt tuning [10 [1] employs learnable prompts in the language branch to match and counteract adversarial attacks from the visual branch. In contrast, visual adversarial prompt tuning C-AVP[2] directly recognize and refine the adversarial images to allow the pre-trained models to make more accurate predictions. More promising multimodal adversarial prompt methods 7 13 14 g simultaneously introduce deep learnable prompts into the visual and language branches to achieve more comprehensive adversarial robustness. Although adversarial prompt tuning preserves the generalized feature representations of pre-trained VLMs, excessive reliance on in-distribution adversarial samples causes degradation of their natural general- ization distribution during the adaptation process. challenge the natural generalization and robustness of prompt-tuned VLMS 10]
(OOD) or unseen tasks further
Pre-trained VL.S retain generalizable knowledge for unseen tasks, while adversarial prompts can guide the shift of natural distribution toward adversarial robustt distributions or downstream task- specific distributions [15]. Therefore, we propose to leverage adversarial prompt tuning to identify a shared latent distribution that effectively balances natural generalization, adversarial robustness, and task-specific adaptation. Due to the inherent discrepancies among different distributions, directly training models with a mixture of natural and adversarial samples to fit the latent distribution leads to suboptimal solutions. Recent findings [16, 17] indicate that masked image modeling (MIM) enables models to learn more generalizable and robust representations, which significantly enhances their capacity to adapt to input distribution variations and improve fine-tuning performance in downstream vision tasks. The success of MIM is due to masked image input and image-level reconstruction objectives. However, this paradigm directs the model pay more attention to high-frequency (HF) components where adversarial perturbations are concentrated, thus failing to effectively improve adversarial robustness [}] We propose a collaborative adversarial prompt tuning (CoAPT) in which pre-trained CLIP collaborates with a target robust CLIP to address this issue. We convert the patch-level image masking from MIM to pixel-level image corruption for model inputs. An improved real-time total variation (TV) regularization method is employed to suppress the adversarial perturbation space by drastically smoothing the freeefrryenccy details of the input images while preserving the image edge structures. To mitigate the cost of sacrificing natural high-freeuency features, we shift the reconstruction objective from the pixel space of the target robust CLIP to the latent representation space of the natural CLIP. The corrupted natural detail features are restored under the guidance of high-level features of natural CLIP images and texts, thereby inheriting their excellent generalization ability. Overall, the fine-tuned adversarial prompts work in synergy with the frozen weights of the original pre-trained CLIP to support the target robust CLIP. They achieve a good balance between (a) improving adversarial robustness while maintaining natural performance on in- distribution tasks, and (b) maintaining natural generalization while enhancing the robust adaptability of the original VLMs OOD or unseen tasks. Our contributions threefold:
We propose a novel paradigm for adversarial prompt tuning that learns robust CLIP from the latent space of natural CLIP. CoAPT weakens high-frequency details of input images to suppress the adversarial perturbation space. Guided by natural CLIP, corrupted generaliza- tion features are restored in the latent space. We introduce Rényi divergence to minimize the discrepancy between the similarity distributions of adversarial and natural examples.
We design a real-time adaptive TV regularization method to efficiently suppress the pertur- bation space. It addresses the slow convergence and residual adversarial perturbations of traditional TV regularization by combining a spatially adaptive regularization strategy based on edge strength response and an accelerated gradient method with adaptive restart.
An optimal trade-off among natural generalizability, adversarial robustness, and task- adaptation is achieved. Without benchmark- sppccfffc or dataset-specific hyperparameter tuning, we improve natural and adversarial robustness performance 15 datasets across four benchmarks by an average of 9.83% and 24.16%, respectively.
2 Related Work
Adversarial attacks on VLMs. Adversarial attacks induce incorrect decisions in VLMS by applying
elaborate and imperceptible perturbations to the input texts or images [8
19
20
21 22).
Text-
based attacks [23 24 25 26 mislead models into generating incorrect outputs through synonym
2

---

substitution, rewriting, or character-level perturbations. FGSM (27, PGD 28 AutoAttack [29]. and C&W [30] are classical image-based white-box attacks that construct adversarial images by accessing model parameters and gradient information. In terms of multimodal attacks, Co-Attack [3] a white-box attack method designed for VLMss, while more works focus on building transferable adversarial black-box attack frameworks
32
33
34
35
36 37..
General adversarial robustness. Researchers have proposed multiple robustness strategies to enhance the reliability of models in adversarial settings (38,9]. Detector-bascd approaches [40] defend against adversarial attacks by detecting and filtering anomalous patterns within input samples.
Purification methods 42 43
44
utilize techniques such image transformations [45] 46 and
denoising filters 47 to disrupt or remove potential adversarial perturbations from the input, yet they the risk of weakening normal sample characteristics. Certified robustness approaches 48 49 (5) provide theoretical and verifiable guarantees for model robustness, though they are typically applicable only simple threat models with small certified radii. Adversarial training [5 52 53 54] addresses model vulnerabilities by mining potential adversarial examples in the
dataset and adapting the model withstand adversarial attacks during the training process.
Adversarial robustness of VLMs. Numerous studies have explored the robustness of VLMs under adversarial attacks, mainly including defense strategies based on model fine-tuning and adversarial prompt tuning. TeCoA [5] and LAAT [6] enhance zero-shot adversarial robustness by leveraging the semantic consistency of the text encoder to guide fine-tuning the image encoder. PMG-AFT [55 and FARE [56) leverage the generalization features of the original pre-trained model to improve the adversarial robustness of the CLIP visual encoder on downstream tasks while preserving natural generalizability. Prompt tuning serves as lightweight adaptation approach that facilitates the efficient transfer of pretrained models toward the target task distribution [57] [5] 58 59]. Recent studies
7 have shown that adversarial prompt tuning can efficiently enhance the robust adaptability of
APT [00 and AdvPT [1] approaches improve model robustness by introducing learnable textual prompts into the language branch of CLIP align with adversarial image embeddings.
VLMs.
Correspondingly, C-AVP 12] and TeCoA (5) incorporate learnable visual prompts to defend against
adversarial attacks. Recent multimodal adversarial prompt methods 7 [3 14 g] enhance the
consistency between visual and language features of adversarial examples under the guidance of pre-trained CLIP, thereby balancing natural generalization and robust adaptation.
Proposed Method
Although prompt learning preserves the general representations of pre-trained VLMMs, the adapted prompts lead to overfitting on specific supervised tasks. We propose architectural refinements to enhance VLMS for achieving robustness in both in-distribution and OOD scenarios. Figure provvddes an overview of our proposed approach, with further details presented in the following sections.
3.1 Preliminaries
CLIP recap. Let Vo_ (-) and (+) denote the image encoder and text encoder of CLIP, respectively, where 0g and 0, represent the corresponding pre-trained weights. Given a natural image v, the input sequence for the visual branch constructed as v {Vela, U1:M}, }. where V1-M are the plteh- linearly projections of the image, and Vels is learnable vector aggregating global features. Given manually designed fixed text template f. the input sequence for the language branch constructed as f {tues, f1:N, fc. feos }. where f1: and t. represent the word embeddings of the template text and the class label, respectively. tsos and toos are non-parametris start and end tokens. The input sequences from the visual and language branches are encoded by CLIP in the latent space into image embeddings Vo, (ū) and text embeddings To, (t), respectively. During zero-shot inference, the similarity between Vo, (ō) and the text embeddings of all candidate categories (To, (Ea)))) is (Se,,mm(e.
(8)
computed as Deh exp(sim((e (B), To, (Z)) where sim(:,-) denotes the cosine similarity function, i is the temperature parameter, and C' is the total number of classes.
Adversarial attacks against CLIP. Given natural image e with ground-truth label y, adversaries construct perceptually imperceptible adversarial example Eadv u+ô by optimizing the perturbation
3

---

Adversarial Image
HF suppecssion
Text Á agmentation
Natural CLIP Icunches
Adp.
photo of perrol
LLM
A parut buthed in golden sunlight, Seathers glowing.
Robust CLIP branches
U
Rémpi branches
*
muil uplicaaiiioo
Visual Encoder 1/2
Textual Encoder TA
Visual degth precpt
Tectual depth prompt
Luca
Live Leee
Lroon
Linear Projection
Figure I: An overview of COAPT. Natural CLIP processes natural images and extended descriptive text inputs. Robust CLIP takes input the images subjected to HF suppression via the real-time Adaptive-FGP algorithm and restores the corrupted natural generalization features under the guidance of Natural CLIP in the latent space. The outputs of Robust CLIP are collaboratively regulated by the frozen CLIP weights 8 the trainable deep multimodal adversarial prompts Ø, and the low-rank residual modules s. The Rényi branch explicitly regulates the discrepancy between natural and adversarial distributions by calculating the divergence between their similarity scores.
5 within g-norm ball of radius c. A successful attack must satisfy the following criteria:
arg
sim(Va, (Undv), To, (Eo)) # y, s.t. ||uadv 2114
(1)
cE(I..,C)
Adversarial prompt tuning. APT enhances the adversarial adaptability of pre-trained VLMs for specific or novel downstream tasks by optimizing visual or textual prompts through adversarial training. Given the prompts & = {ol.V "..T") to be optimized during adversarial training, where V and T represent the number of trainable tokens within the visual and textual prompts, respectively. Adversarial visual-only and text-only prompting [10] [2
[2] typically employs shallow prompting. where prompts are inserted solely into the input sequences. Specifically, the visual and textual input sequences are updated
{Vets, Φ2.V (1:a} and i (ssos, $.: lez leos). Building upon shal- low prompting, both independent and joint vision-language adversarial prompting 7 g incorporate deep prompts into multiple layers within the visual and language transformer architectures.
We aim to develop joint vision-language adversarial prompts that learn adversarial transformation- invariant features during training, strengthening the adversarial robustness of the CLIP visual branch. We still denote the adversarial deep prompts as o. Given a downstream dataset D, é is optimized jointly with the frozen parameters 9 on adversarial examples. Focusing the foo threat model, the adversarial optimization process for obtaining the optimal parameters of robust prompts o can be formalized as:
max [| Fllsee L(Vor,o (Bat)), To (Fa))
Φ min (Oa))D
(2)
3.2 Real-Time Total Variation Regularization for High-Frequency Suppression
Background on total variation. Total variation regularization is implemented in the continuous and discrete settings by solving an unconstrained convex optimization problem in its penalized form:
1
min WEU 2A ||u 11---ei u(aav) ||
where u E U m denotes the image to be restored, V(adv) E represents either a natural or adversarial image. For simplicity, v is used uniformly in this section. |l-Ilry represents the discrete total variation of the image gradient, and À > 0 balances the fidelity and regularization terms. Chambolle transforms Eq. (3) into nonlinear projection problem on a constrained space via dual formulation. However, this method lacks real-time capability and is prone over-smoothing image details and residual adversarial perturbations. We design adaptive-FGP. a fast gradient projection (FGP) method with adaptive restart mechanism and spatially adaptive regularization strategy.

---

Accelerated gradient method with adaptive restart mechanism. We obtain the optimal solution from constrantdd dual vector field, thereby recovering in the form:
min WEW
f((*)) : div(w*)l()
1100-
where i denotes the current time step, and W S (m x)xn m= is the unit-ball constraint set for the gradient dual components w (Pij) Pi. T. the gradient vector is defined in both horizontal and vertical directions, it satisfies (! 1 1. Otherwise, only the single-directinn constraint remains, satisfying ||,,,"" al S and m! al S 1. div(-) denotes the discrete divergence operator, which maps the dual variables from the vector field W the image domain U. The gradient of f (w*) can be computed as V fw**(
1)mm
-2 (v) div" (v r(v) div(w")). Using step size of I,L, where L denotes the Lipschitz constant of f f(w") with its upper bound derived as 1 (v) in the Appendix B. The dual variable update rule can be expressed as:
V(v 7(v) div(w"))
w = IIw
W
(5)
8 8-((v)
where Tl represents the projection operator. The update w is performed as follows:
w + (Tk 1). (w* wh /Tt+1, if gh < On. otherwise,
W
(6)
w
when b meets the predefined threshold An. the Nesterov 61 time-scale variable is updated with Th+11 [ 11+ 47 /2: otherwise, it is reset 1.0. The solution the objective function denoted as u² = v ((0) dliv(w*-1), k)). The solution increments at two consecutive time steps are defined dj 2² *=1 and xh-1 uh1 === Whether the current momentum accumulation benefits the variable update is determined utilizing cosine similarity-based adaptive restart criterion:
(axk 0k-
cos(Os) (0x) | ||
(7)
where ( is numerical stabilization term. When the angle between directions exceeds 90", it signals a sharp deviation or reversal between momentum and update, indicating trajectory discontinuity. We then reset the temporal scaling and disable momentum avoid overshooting.
Spatially adaptive regularization strategy.
The regularization map 7(0) E R m is given by:
7()) = Mbase (1 /"gain +(v)),
where Mbase Mgain E R represent the base regularization strength and the sensitivity of the adjustment factor, respectively. The edge magnitude response function ((v) E R pmxx is estimated using Sobel convolution kernels as V(e K2 (v K2)², where K₂ and K, denote the horizontal and vertical Sobel operators respectively. This adaptive regularization strategy automatically reduces the regularization strength in edge regions while enhancing it in flat regions, thereby preserving structural image details and effectively suppressing adversarial perturbations.
Convergence criterion. The relative change in update is measured through the Frobenius norm:
||0||
max
| 1 <.
VE
the convergence tolerance threshold & >> is satisfied for & consecutive iterations, the projection optimization problem considered to have converged. Based the optimal solution - (v), the optimal image estimate for the original problem can be recovered p(u) (v)) div ( (v)).
3.3 Natural-Latent-Guiite Adversarial Prompt Learning
Reconstruction of natural generalization representations. CoAPT employs deep contextual multimodal prompts and refines visual prompts through linear projection onto language prompts
5

---

to foster synergy between visual-languee prompts. As illustrated in Figure 1 we efficiently learn generalizable knowledge from the natural CLIP by aligning its clean vision-language embeddings with adversarial embeddings from the robust CL.PP in the latent space. Notably, Vanilla CLIP employs fixed text templates, which limit its ability to capture the semantic diversity required for generalization effectively during fine-tuning. A Gaussian radial basis function (RBF) is used to measure the embedding similarity between the natural CLIP and the robust CLIP in the latent space. Compared to cosine similarity, which primarily captures angular differences of vectors, Gaussian RBF highlights feature shifts caused by small-scale perturbations, allowing more sensitive detection of subtle distributional changes. In particular, align both the visual and language branches:
Lrecon 2 exp (-3 (1)V ou.sp (p(0
Vo, (o) 2 + To (1) To, (1) 1)),
(10)
where the parameter == (2a2)- controls the sensitivity of distance variation similarity. yo and PP are the low-rank residual modules introduced next. The learnable prompts in both the language and visual branches can adapt the data distribution of Vanilla CLIP to that of specific downstream adversarial tasks, while preserving and enhancing generalization and robustness to OOD tasks.
Low-rank residual module. Directly imposing consistency constraints in the latent space is equivalent to introducing strong supervisory signal, which lacks the flexibility to adapt to task- specific requirements and interpretable deviations. Inspired by LoRA [62]. we introduce two low-rank matrices as an intermediate learnable bottleneck structure. This design allows the model to preserve the backbone features while selectively capturing fine-grained task mesk-speifiic shifts within a compact subspace. Specifically, we incorporate an additional update term through low-rank reparameterization:
Ve.d.p = (I + BA) Vo.o:
(11)
where is the scaling factor, B E Rdxr, A E Rrxd, and < d. The initial parameter perturbation is controlled by initializing the matrices as A (O, 1/r) and B ~ 8(0).
Rényi regularization. Let P and Q denote the predicted probability distributions of natural and adversarial samples in the vision-language space of robust CLIP, respectively. Since adversarial sam- ples are derived from minor perturbations of natural samples, P is considered absolutely continuous with respect to Q. We introduce a regularization loss based on the a-order Rényi divergence [63] to reduce the discrepancy between the natural and adversarial predictive distributions in robust CLIP:
1
dP dQ
a-17
Liéngi
log Ep
a E [0,
(12)
where dP is the Radon-Nikodyne derivative of P with respect to Q. a explicitly controls the sensitiv- ity to distributional differences. Higher orders (a > 1) enhance the ability of the model to suppress spurious correlations. This mechanism corrects potential discriminative boundary ambiguities and reduces overfitting risks by preserving task-beneficial generalized features. Correspondingly. the supervised loss for downstream classification tasks can be expressed with the Rényí cross-entropy [64]
aQ
Lree
1 log P(1) -' a (c)
a
(13)
Note that the Rényi cross entropy degenerates into Shannon cross entropy when the dataset labels represented in one-hot coding. The overall training objective of CoAPT can be expressed as follows:
Loopt = Ki Creoo + K2 + KgGeeer
(14)
N1, K2,, Ki are hyperparameters weighting contributions of individual losses the overall objective.
Overview of proposed method. Algorithm [ iillustrates the adversarial prompt optimization pro- cedure adopted by CoAPT. In each training iteration, batch of image-label pairs (v, y) is sampled from the downstream dataset D. Subsequently, the visual and textual sequences are constructed and accompanied by trainable deep prompts. Perceptually invisible adversarial examples Undv are crafted under to norm constraints to induce erroneous model predictions (Lines 2-4)). These sequences are then fed into the natural CLIP and the robust CLIP equipped with low-rank residual modules you and Pe to obtain the corresponding visual and language representations (Lines 5-9).. CoAPT integrates

---

Algorithm Natural-LLtten--Guideee Adversarial Prompt Learning
Input: Dataset D, frozen CLIP encoders Vo,: To prompt parameters & {dhr Φ1), low-rank
modules yp = (yPr, Pt), loss weights K1, N2, Ki3, adversarial budget Output: Optimized robust prompts 6)*
1: for each minibatch (v. 3) D do
2: Set the real-time total variation regularization parameters
3: Construct input sequences U, f and deep prompts 6
4: Generate adversarial example Undv under & constraint: ||Uadv 211200 S
5: Generate visual and textual representations for natural CLIP and robust CLIP:
6: Vist Ve, (ū)
7: Test - (1)
8:
Vadv +- Vo..ou.p. (p(Uade))
9:
Tadv + do (t)
10:
Compute reconstruction loss Lrecon + 2 exp (-3(1] Vnal 11 + || T (12))
11:
Compute visual-textual representation similarity P scale Vmn
Tan- Q = scale Vudv Tadl
12:
Compute Rényi divergence loss Lrémyi < a log Ep
dP dQ
13:
Compute Rényi cross-entropy loss: Lice t- I-a log E, P(t) Q(1)
14:
Take gradient step on Vo (K'1 frevom + K2greenyi + KifTee)
15:
Φ, Pp + Backward V (.s))
16: end for
three losses, including a reconstruction loss for recovering generalization, a Rényí divergence loss to quantify prediction discrepancies between natural and adversarial samples, and cross-entropy loss for classification (Lines 10~133). Finally, only the prompt parameters Φ and the low-rank module parameters &p are updated via gradient descent, Adversarial prompt learning significantly improves the robust generalization of the model under image perturbations and distributional shifts, and exhibits strong cross-task transferability (Lines 14~115).
4 Experiments
4.1 Evaluation Settings
Datasets and benchmark settings. We conduct comprehensive evaluation of the proposed CoAPT method across four benchmark settings on 15 datasets spanning diverse vision tasks. For the evaluation of few-shot learning, base-to-novel class generalization, and zero-shot benchmarks, we adopt 11 image classification datasets, including EuroSAT (65) for satellite imagery, UCF101 [66] for action recognition, DTD (67) for texture classification, SUN397 688 for scene recognition, Caltech101 (69] and ImageNet (70 for general object recognition, and FGVC Aircraft (71) Flowers [72).
OxfordPets 73), Food101 (74). and StanfordCars (75) for fine-grained classification tasks. For the OOD benchmark, we select four variants of ImageNet, ImageNet-A (76), ImageNet-R (77),
ImageNet-Skeet [78), and ImageNetV2 [79], as the domain generalization test sets. Notably, both zero-shot and OOD utilize the training set of ImageNet the source dataset.
Adversarial training and evaluation. The attack settings of baseline methods TeCoA 5 and FAP [ are adopted to ensure fair comparison. During adversarial training, we adopt two-step PGD attack with a maximum perturbation magnitude f. = 1/255 and step size a 1/255. For robustness evaluation, we employ a 100-step PGD attack under the same constraints to thoroughly assess the defense capability of the model under strong attacks.
Implementation details. Our method is built upon the ViT-B/32 architecture of Vanilla CLIP. Each experiment conducted three times with different random seeds, and the average results are reported. The convergence tolerance threshold in Adaptive-FGi is set to E = le -3 3, and the maximum number of iterations is 30. The parameters of the regularization factor map (v) are set to Ahuse 0.1 and M'yain = 1.2. We employed 2.5-order Rényi divergence regularization, with Leoggt coefficients set to K1 8, K2 = 1, A3 Adversarial prompts with a length of 4 and a depth of are applied to both the visual and textual branches. The RAdam optimizer with initial learning rate of 0.00735
7

---

Vanilaa CLIP Nat.
AdVVLP
AdoMaiL
COAPT Nat
Vanilla CLIP
AdeVEP Rob.
AdvMaiPL. Rub.
FAP Rob.
CUAPT Rub.
Average
ImageNet
Caltech101
Accurcy /Rohusties (%)
60
60
80
40
40
60
20
20
40
8
16
2
16
0
4
16
0
2
16
FurNAAT
FGVCAirena
Food101
Flowers102
Accuncy: Rahusmess (%)
40
80
80
80
40
20
10
20
OxtordPets
StunfordCurs
SUN397
UCFI01
Accuncy (Robusmess (%)
80
40
40
20
20
20
0
16
0.
16
16
Shots per class
Shots per class
Shots per class
Shots class
Figure 2: The few-shot performance across 11 benchmark datasets under varying numbers of shots.
is adopted, and the batch size is set to 64. In contrast the existing research work, do not proprietary hyperparameters for any of the benchmarks and datasets, in order to prove the generality of the proposed CoAPT. Under few-shot settings we compare with FAP and baselines from its paper.
4.2 Adversarial Few-Shot Learning
The robust generalization capability of each model to specific tasks is evaluated under the condition of only few identically distributed samples. As shown in Figure 2 CoAPT demonstrates consistently superior performance compared to baseline methods. CoAPT exhibits robust learning ability with near-linear steady improvement in natural and adversarial accuracy as the number of shots increases. contrast, the baseline methods show significant performance fluctuations across different shot counts. Furthermore, accuracy and adversarial robustness. In most of the datasets, CoAPT able to match the natural accuracy of Vanilla CLIP with only 1-shot learning. On six datasets. including Caltech accuracy is even higher than the natural accuracy the baseline method. The robust accuracy of CoAPT on five datasets, including DTD, be improved to higher than the natural accuracy of Vanilla CLIP by few-shot learning.
approach achieves superior control the trade-off between natural
robust
4.3 Adversarial Base-to-New Generalization
We assess the ability of the models to balance robust adaptation to specific class distributions and robust generalization to unseen class distributions. Specifically, the models are trained on base classes with 16-shot setting and jointly evaluated on the base classes and the novel unseen classes. As shown Table I our method outperforms state-of-the-ar approaches on all datasets. While improving the average harmonic mean (HM) of robustness by 32.39%, the natural generalization performance of the model also achieves an average gain of 13.09%. Notably, the harmonic mean of robustness for novel classes reaches maximum of 51.57% the OxfordPets dataset. These

---

Table Comparison with state-of tht--hharrr methods on base-to-novel generalization. Gain denotes the absolute performance improvement.
(a) Average
(b) ImugeNet
(e) Caltech101
DTD
Acc.
FAP
CoAPT
Gaint
Acc.
FAP
CoAPT
Gain
Acc.
FAP
CoAPT
Gain?"
Acc.
FAP
CoAPT
Gaint
Base
70:52
78.47
7.95
Base
58.10
66.15
8.05
Base
94.07
97.25
3.18
Base
69.17
76.08
6.91
Nat
Novel
Novel
47.83
7,58
76.53
92.72
Novel
54.03
HM
58.22
71.31
13.09
HM
52.47
60.30
7.84
84.40
94,93
10.53
HM 46.63
63.17
16.54
Base
38.05
67.70
29.65
Base 25.83
52.65
26.82
Base
74.20
94.38
20.18
Base
41.63
67.98
26.35
Rob.
Novel
21.86
54.13
32.27
Rob
Novel
21.57
45.07
23.50
Roh
Novel
50.00
88.03
38.03
Novel
19.77
43.88
24.11
HM
27.77
60.16
32.39
HM 23.51
48.57
25.06
59.74
91.09
31.35
26.81
53.31
26.50
(e) EuroSAT
FGVCAircrat
(g) Food101
(h) Flowers102
Avc.
FAP
CoAPT
Gaint
Acc.
FAP
CoAPT
Gain"
Acc.
TAP
CoAPT
Gain
Acc.
FAP
CoAPT
Gain
Base
87.70
91.61
3.91
Base
24.83
35.37
10.54
Base
72:37
78.20
5.83
89.30
94,94
5.64
Nat.
Novel
32.80
56.11
23.31
Novel
15.83
25.41
9.58
Novel
68.20
79.47
11.27
Novel
45.67
63.07
17.40
HM
47.74
69.33
21.59
HM
19.33
29.58
10.24
HM
70:22
78.83
8.60
HM
60.43
75.79
15.36
Base
51.80
84.67
32.87
Base 8.00
25.37
17.37
Base
27.57
62.03
34.46
Base
65.50
88.57
23.07
Roh
Navel
34.00
Rob
Novel
4.23
12.45
Roh
24.20
62.86
Novel
18.10
51.89
33.79
39.25
62.44
65.42
( OxfordPets
(i) StanfordCars
(k) SUN397
(1) UCFF111
Acc.
FAP
CoAPT
Gaint
Acc.
FAP
CoAPT
Gaint
Acc.
FAP
CoAPT
Gain"
Acc.
FAP
CaAPT
Gaiin]
Base
87.37
90.55
3.18
Base
53.97
73.34
19.37
Base
68.47
76.69
8.22
Base
70.37
82.95
12.58
Nat.
Novel
72.13
94.50
22.37
Novel
42.67
59.20
16.53
Novel
61.47
70.46
8,99
Novel
47.10
68.45
21.35
HM 79.02
92.48
13.46
HM
47.66
65.51
17.85
HM
64.78
73:44
8.66
HM
56.43
75.00
18.57
Base 34.13
78.72
44.59
Base 18.60
54.20
35.60
Buse 34.63
64,50
29.87
Base
36.63
71.65
35.02
Rdh.
Novel
26.07
83.71
57.64
Novel
14.10
40.95
26.85
Rob
Novel
30.77
58.50
27.73
Novel
18.30
56.50
38.20
HM
29:56
81.13
51.57
HM
16.04
46.65
30.61
HM
32.59
61.35
28.76
HM
24.41
63.18
38.77
results demonstrate that the robust prompts learned by CoAPT not only adapt to sategorr-speciine distributional shifts and distributional discrepancies between natural and adversarial examples but also effectively preserve the natural generalization capability of the original pretrained model.
4.4 Zero-Shot Performance
The generalization ability of the models across Table 2: CoAPT performance on source dataset datasets is explored. CoAPT is trained on Ima- and average results across 10 target datasets. geNet as the source dataset and then evaluated on ten different types of downstream target datasets. The evaluation for each dataset and the corre- sponding statistical results are presented in Fig- ure Table respectively. Compared the FAP
ImageNet
Average
Method
Nat.
Rob.
Nat.
Rob.
CLIP
62.10
1.57
61.89
4.53
50.80
21.60
45.72
23.89
FAP method, our approach achieves significant
CoAPT
63.42, 321
51.1820.88
54.067.89
43.90aa.or-
improvements across metrics on all datasets, particularly in adversarial robustness. We attain better trade-off between natural and adversarial generalization. Relative to Vanilla CLIP, we sacrifice only 7.83% in natural generalization accuracy while achieving absolute gains of 49.61% and 39.37% %
in robustness on the source and target datasets.
Robaal Accrrcy (%)
Vanilla CLIP
FAP
CoAPT
Natural Accurre (%G)
80 60 40
80
60
40
Imagehal
Catbeshtf
DTD
EumSAT FovCAremm
Food101
Flovars
OrfordPed
SmafordCCan
SR/1197
UCF101
Insgeter
Cateehh
DTD
RumSAT
FovCAirent
12 Food
Staffrt
SUN099
UCFI0I
Figure 3: Zero-shot robust and natural accuracies on the source and 10 target datasets.

---

4.5 Out-of-Distribution Performance
We test the natural generalization and adversarial robustness of the model under domain distribution shift. While maintaining ImageNet as the source dataset, we conduct direct evaluations four representative variant datasets that share the same set of categories. As shown in T our method achieves superior natural generalization and robust adaptation across all target datasets compared to the comparison methods.
Table 3: Comparison of OOD generalization performance.
Method
Net-eNet-A
ImageNet-R
ImageNet-Sketee
ImageNet-V2
Average
Nat.
Rob.
Nat.
Rob.
Nat.
Rob.
Nat.
Rob.
Nat.
Rob.
FAP CoAPT
9.40
1,20
$1.60
28.20
28.40
16.30
42,80
16.60
33.05
15,575
16.99; set
9.728.521
60.35 751
50.7122.511
361
22.13
54.35 1
42.2322.651
41.86%.11 811
32.95
4.6
Ablation Analysis
As shown in T we progressively ablate CoAPT components to evaluate their generalizability and importance across the four benchmarks. CoAPT with all components achieves the best performance on all benchmarks. We first remove the adaptive restart mechanism. Most metries exhibited varying degrees of degradation, with 16-shot and OOD robust accuracy declining by 1 1.85% and 1.69%, respectively. This mechanism restores optimal convergence without prior knowledge of function parameters and enhances stability near the optimum. We replace the spatially adaptive regularization strategy with fixed global regularization factor. The ablated model ignores the diversity of image spatial structures, leading to structural blurring and loss of details, with an average drop of 6.00% in clean accuracy across the four benchmarks. We subsequently remove the entire adaptive- method, thereby eliminating the adversarial space compression. During high-level feature recovery in the natural CLIP latent space, the model places greater emphasis on gigg-frrueency components where adversarial perturbations are concentrated, resulting in a degradation in adversarial robustness. However, even with full natural images, the ablated model yields lower natural accuracy than full CoAPT across all benchmarks. Removing the low-rank residual module leads to drops in few-shot- 16 robustness and base-to-novel accuracy. As it is sensitive was not fine-tuned, its effectiveness is limited. However, due to its potential on certain datasets, the module is retained. When we remove Rényi regularization, the overall performance of the model decreases. Rényi regularization facilitates early detection and correction of boundary ambiguities, and mitigates overfitting by preserving task-relevant generalizable features. CoAPT reduces TeCoA-like approach when the final reconstruction loss is removed. The performance drop on unseen tasks is due to the reconstruction loss guiding prompts toward task-irrelevant generalization.
-ppccic hyperparameters and
Table 4: Ablation study of CoAPT components on 15 datasets across four benchmarks.
16
Basc-to-novel
Zero-shot
OOD
Ahlation term
Nat.
Rob.
Nat.
Rob.
HM
Not.
Rob.
HM
Nat.
Rob.
Nat.
Rob.
ablation
74.96
62.98
78.47
67.70
72.69
65.35
54.13
59.21
54.91
44.57
41.86
32.95
Adp. rst.
74.82
61.13
78.31
66.73
72.06
65.56
$3.40
58.86
54.38
43.06
40.91
31.26
Adp. reg.
68.86
63.34
73.33
68.03
70.58
57.74
52.87
55.20
49.16
44.15
34.86
31.70
Adp. FGP
74.34
31.64
78.15
35.92
49.21
64.36
24.41
35.40
53.00
18.65
38.52
13.21
Res. mod.
74.64
31.41
78.33
35.99
49.32
64.18
26.00
37.01
55.07
19.74
41.04
14.31
Rényi
73.09
30.73
78.18
33.66
47.06
63.57
24.97
35.86
55.30
19.84
41.05
14.55
Recon. loss
71.82
31.47
76.66
34.52
47.60
58.71
22.25
32.27
51.85
19.85
38.66
13.91
5 Conclusion
We focus on the adversarial robustness of VLMs and propose a novel adversarial prompt tuning paradigm in which pre-trained VLMs collaborate with target robust VLMs. CoAPT begins with proposed real-time adaptive TV regularization algorithm to attenuate input images to compress the perturbation space of the adversarial samples. Subsequently, under the guidance of natural CLIP, CoAPT restores the natural generalization features disrupted by adversarial perturbations in the latent representation space. CoAPT achieves an effective trade-off among natural generalization, adversarial robustness, and task-specific adaptation. The overall performance of CoAPT significantly surpasses that of current state-of-the-ar methods on 15 datasets across the benchmarks of shot, base-to-novel. zero-shot, and out-of-diisribution generalization.
details of the
10

---

References
[1] Radford, A., J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748- 8763. 2021.
[2] Li, R. Selvaraju, A. Gotmare, et al. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9787 9705,, 2021.
[3] Fang, Z. R. Wang, T. Huang, et al. Strong transferable adversarial attacks via ensembled asymptotically normal distribution learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24841-24855 2024.
[4] Wu. H., G. Ou, W. Wu, et al. Improving transferable targeted adversarial attacks with model self-enhancement. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24615-24624, 2024.
[5] Mao, C., Geng. J. Yang, et Understanding zero-shot adversarial robustness for large-scale models. In International Conference on Learning Representations (ICLR). OvenReview.net, 2023.
[6] Li, X., W. Zhang, Y. Liu, et al. Language-driven anchors for zero-shot adversarial robustness. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 24686- 24695. 2024.
[7] Zhou, Y. X. Xia, Z. Lin, et al. Few-shot adversarial prompt learning vision- llaguun models.
Advances in Neural Information Processing Systems, 37:3122-3156, 2024.
[8] Wang. X., K. Chen, Zhang, et al. Tapt: Test-time adversarial prompt tuning for robust inference in vision-language models. arXiv preprint arXiv:2411.13. 2024.
[9] Huang, Q. X. Dong, D. Chen, et al. Improving adversarial robustness of masked autoencoders via test-time frequency-domain prompting. In /EEF/CVF International Conference Computer Vision (ICCV), pages 1600-1610. 2023.
[10] Li, L.., H. Guan, J. Qiu, et al. One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24408-24419. 2024.
[11] Zhang, J. X. Ma, X. Wang, et al. Adversarial prompt tuning for vision-language models. In European Conference on Computer Vision (ECCV), vol. 15103, pages 56-72. 2024.
[12] Chen, A., P. Lorenz, Y. Yao, et al. Visual prompting for adversarial robustness. In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[13] Luo, L., X. Wang, B. Zi, et al. Adversarial prompt distillation for vision-language models. arXiv preprint arXiv:24/ 1.1524, 2024.
[14] Yang, F., M. Xia, S. Xia, et al. Revisiting the robust generalization of adversarial prompt tuning. arXiv preprint arXiv:2405. / 154, 2024.
(15) Khattak, M. U., S. T. Wasim, M. Naseer, et al. Self-regulating prompts: Foundational model adaptation without forgetting. In IEEE/CVE International Conference on Computer Vision (ICCV), pages 15100-15200. 2023.
[16] He, K., X. Chen, S. Xie, et al. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022.
[17] Lee, J., E. J. Hwang, S. Cho, al. Pilamim: Toward richer visual representations by integrating pixel and latent masked image modeling. arXiv preprint arXiv:2501.0300 2025.
[18] Szegedy, C., W. Zaremba, Sutskever, et al. Intriguing properties of neural networks. In International Conference on Learning Representations (ICIR). 2014.
[19]
S.-M., A. Fawzi, P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574-2582, 2016.
[20] Goodfellow, J., J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR). 2015.
11

---

[21] Jia, X., Y. Zhang, B. Wu, et al. Las-at: adversarial training with learnable attack strategy. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 13398- 13408. 2022.
[22] Hsiung, L.,, Y.-Y. Tsai, P.-Y. Chen, et al. Towards compositional adversarial robustness: Gener- alizing adversarial training to composite semantic perturbations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2023.
[23] Jin, D., Z. Jin, J. T. Zhou, al. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In AAAI Conference on Artificial Intelligence, vol. 34, pages 8018-8025, 2020.
[24] Gao, J., J. Lanchantin, M. Soffa, et Black-box generation of adversarial text sequences to evade deep learning classifiers. In IEEE Security and Privacy Workshops (SPW), pages 50-56. IEEE, 2018
[25] Ren, S., Y. Deng, K. He, et al. Generating natural language adversarial examples through probability weighted word saliency. In Association for Compulational Linguistics (ACL)) pages 1085-1097. 2019.
[26] L., R. Ma. Q. Guo, et al. BERT-ATTACK: adversarial attack against BERT using BERT. In Empirical Methods in Natural Language Processing (EMNLP), pages 6193-6202. 2020.
[27] Goodfellow, I. J. Shlens, Szegedy. Explaining and harnessing adversarial examples. In Y. Bengio, Y. LeCun, eds., International Conference on Learning Representations (ICLR). 2015,
[28] Madry, A., A. Makelov, L. Schmidt, et al. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR). OpenReview.net, 2018.
[29] Croce, F., M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. International Conference on Machine Learning (ICML), pages 2206-2216 PMLR, 2020.
[30] Carlini, N., D. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), pages 39-57. leee, 2017.
[31] Zhang, J., Q. Yi, J. Sang. Towards adversarial attack on vision-language pre-training models. In ACM International Conference on Multimedia, pages 5005-5013. 2022.
[32] Yin, Z.,, M. Ye, T. Zhang, al. Vlattack: Multimodal adversarial attacks on vasion-language tasks via pre-trained models. Advances in Neural Information Processing Systems, 36:52936 52956, 2023.
[33] Han, D., X. Jia, Y. Bai, et al. Ot-attack: Enhancing adversarial transferability of vision-language models via optimal transport optimization. arXiv preprint arXiv:2312.044003 2023.
[34] Wang, H., K. Dong. Z. Zhu, al. Transferable multimodal attack on vision- vision-language language pre- training models. In IEEE Symposium on Security and Privacy (SP), pages 1722-1744 IEEE, 2024.
[35] He, B., X. Jia, S. Liang, et al. Sa-attack: Improving adversarial transferability of vision-language pre-training models via self- self-augmentationn arXiv preprint arXiv:23/2.04911, 2023.
[36] Zhao, Y., T. Pang, C. Du, et al. On evaluating adversarial robustness of large vision-language
models. Advances in Neural Information Processing Systems, 36:54111 1-55118, 2023.
[37] Lu, D., Z. Wang. T. Wang, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 102-111. 2023.
[38] Kuang, H., H. Liu, Y. Wu, et al. Semantically consistent visual representation for adversarial robustness. IEEE transactions on information forensics and security, 18:5608-5622, 2023.
[39] Naseer, M. S. Khan, M. Hayat, et al. Stylized adversarial defense. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 45(5):6403-641) 2022.
[40] Deng, Z., X. Yang, S. Xu, et al. Libre: A practical bayesian approach to adversarial detection.
In IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 972-982, 2021.
12

---

[41] Liu, H., Y. Wu, Z. Yu, et al. Slowlidar: Increasing the lateney of lidar-based detection using adversarial examples. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3155-5155 2023.
[42] Xiao, C., Z. Chen, K. Jin, et al. Densepure: Understanding diffusion models for adversarial robustness. In International Conference on Learning Representations (ICI.R). 2023.
[43] Nie. W., B. Guo, Y. Huang, et al. Diffusion models for adversarial purification. In International Conference on Muchine Leurning (ICML), vol. 162, pages 16805-16827. 2022.
[44] Ho, C.-H., N. Vasconcelos. Disco: Adversarial defense with local implicit functions. Advances in neural information processing systems, 35:2381 8-23837, 2022.
[45] Wang, H., C. Xiao, Kossaifi, et al. Augmax: Adversarial composition of random augmen- tations for robust training. Advances in neural information processing systems, 34:237-250, 2021.
[46] Chen, C., D. Ye, Y. He, et al. Improving adversarial robustness with adversarial augmentations. IEEE Internet of Things Journal, 11(3):5105-5117 2023.
[47] Guo, C., M. Rana, M. Cissé, et al. Countering adversarial images using input transformations. In International Conference on Learning Representutions (ICLR). OpenReview.nett 2018.
[48] Wang, Y., H. Fu, W. Zou, et al. Mmcert: Provable defense against adversarial attacks to multi-modal models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 24655-24664. 2024.
[49] Carlini, N., F. Tramèr, International Conference Learning Representations (ICLR). 2023.
Dvijotham, al. (certified!!) adversarial robustness for free! In
[50] Xu, Y., Y. Sun, M. Goldblum, et al. Exploring and exploiting decision boundary dynamics for adversarial robustness. In International Conference on Learning Representations (ICLR). OpenReview.net, 2023.
[51] Zhang, J., F. Liu, D. Zhou, et al. Improving accuracy-robustness trade-off via pixel reweighted adversarial training. In International Conference Machine Leurning (ICML). 2024.
[52] Pang, T., X. Yang, Y. Dong, et al. Boosting adversarial training with hypersphere embedding. Advances in Neural Information Processing Systems, 33:7779-7792. 2020.
[53] Hou, P., J. Han, X. Li. Improving adversarial robustness with self-paced hard-class pair reweighting. In AAAI Conference on Artificial Intelligence, vol. 37, pages 14883-14891. 2023.
[54] Yuan, Z., J. Zhang, S. Shan. Fulllora-at: Efficiently boosting the robustness of pretrained vision transformers. arXiv preprint S0ti::2401.0155:: 2024.
[55] Wang, S., J. Zhang, Z. Yuan, et al. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 24502-24511. 2024.
[56] Schlarmann, C., N. D. Singh, F. Croce, al. Robust CLIP: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In International Conference on Machine Learning (ICML). 2024.
[57] Khattak, M. U., H. Rasheed, M. Maaz, et al. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113-19122. 2023,
[58] Roy, S., A. Etemad. Consistency-guides prompt learning for vision-language models. In International Conference on Learning Representations (ICLR). OpenReview.net, 2024.
[59] Zhou, K., Yang, C. C. Loy, et al. Learning to prompt for vision-language models. International Journal of Computer Vision, (30(9))2337-2348 2022.
[60] Chambolle, A. An algorithm for total variation minimization and applications. Journal of Mathematical imaging and vision, 20:89-97, 2004.
[61] Nesterov, Y. A method for solving the convex programming problem with convergence rate (1/K2). Dokl akad nauk Sssr, vol. 269, page 543. 1983.
[62] Hu, F. J., Y. Shen, P. Wallis, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). 2022.
[63] Ding, N., Farokhi, T. Guo, et al. a-leakage interpretation of sibson mutual information and rényi capacity. In IEEE Information Theory Workshop (ITW). 2025.
13

---

[64] Ding, N., M. A. Zarrabian, P. Sadeghi. A cross entropy interpretation of renyi entropy for a-leakage. 2024 IEEE International Symposium on Information Theory (ISIT), pages 2760- 2765. 2024.
[65] Helber, P., B. Bischke, A. Dengel, et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 1277):2217-2266, 2019.
[66] Soomro, K., A. R. Zamir, M. Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXivv1112.04400 2012.
[67] Cimpoi, M., S. Maji, I. Kokkinos, et al. Describing textures in the wild. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3606-3613, 2014.
[68] Xiao, J., J. Hays, K. A. Ehinger, al. Sun database: Large-scale scene recognition from abbey to zoo. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485-3492. IEEE, 2010.
[69] Fei-Fei, L., R. Fergus, P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 178-178. 2004.
[70] Deng, J., W. Dong, R. Socher, et al. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. Ieee, 2009.
[71] Maji, S., E. Rahtu, J. Kannala, et al. Fine-grained visual classification of aircraft. arXiv preprint 1306. 2013.
[72] Nilsback, M.-E., A. Zisserman. Automated flower classification
large number of classes.
In Indian conference on computer vision, graphics image processing, pages 722-729. 2008.
[73] Parkhi, O. M., A. Vedaldi, A. Zisserman, et al. Cats and dogs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3498-3505. 2012.
[74] Bossard, L., M. Guillaumin, L. Van Gool. Food-101-mining discriminative components with
random forests. In European Conference on Computer Vision (ECCV), pages 461- 2014.
[75] Krause, J., M. Stark, J. Deng, al. 3d object representations for fine-grained categorization. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 554-561. 2013.
[76] Hendrycks, D., K. Zhao, S. Basart, et al. Natural adversarial examples. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262-15271. 2021.
[77] Hendrycks, D., S. Basart, N. Mu, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 8340 1000008349.00 2021.
[78] Wang, H., S. Ge, Z. Lipton, et al. Learning robust global representations by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
[79] Recht, B., R. Roclofs, L. Schmidt, et al. Do imagenet classifiers generalize to imagenet? In
International Conference on Machine Learning (ICML), pages 5389-5400. PMLR, 2019.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly explain the scope and importance of the work, and the main contributions are summarized the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
The limitations of this paper are discussed in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Only small portion of this work requires theoretical justification, which has been rigorously proven. The remaining contributions focus improving adversarial prompt learning from an empirical perspective.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed implementation information in Section 4.1 and the ap- pendix to support the reproduction of our experimental results. The corresponding code will also be included in the supplemental material.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soneee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For datasets, we only use open-source datasets that are publicly available, For codes, we list the original paper of baseline methods in the appendix with access to their respective code repositories.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. ctipss.cc/ublic//uui.eu www.iiiciguiieseieeeeeeeeeeeeeee Satpe///ooeSSSeeiieeeeeeeee ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the experimental section, we give all details concerning the experiment settings, parameter values, optimizer, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the average performance and standard deviations across multiple runs in the experimental results section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details compute resources provided in the appendix.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully read the NeurlPS Code of Ethics and checked the anonymity of our submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the boarder impact of our paper in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer: [NA]
Justification: Our paper does not include generative models and typically uses open- datasets for training and evaluation.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creator of assets used in our paper states the license in their repository (MIT License).
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, wwwppswwttchooooooooppppoppppooo com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Although we will submit the code in the supplementary materials, will continue to improve the codebase and make publicly available after paper is officially accepted. Currently, have not released any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the detaset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
20

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There are no crowdsourcing experiments and research with human subjects under adversarial prompt learning settings.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There crowdsourcing experiments and research with human subjects under adversarial prompt learning settings.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does not involve LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non- standardd components.
Please refer to LLM policy (https://neurii) http:/rpppp..c/oooooeeemmee///2s for what should or should not be described.
21

---

Appendix
A Pipelines of Adaptive-FGP Algorithm
Algorithm 2 presents the proposed adaptive fast gradient projection (adaptive-FGP) method for real-time total variation regularization. It designed to disrupt the perturbation space of adversarial examples while maximally preserving the structural integrity of image content.
Algorithm 2 Teme Total Variation Regularization Based on Proposed Adaptive-FGP Algorithm
Input: Image P(ado). base coefficient Jbasee gain Mgainn convergence tolerance &
Output: Recovered image u²
1: Compute 7(v) Mbase (1 + Mgain (v)) image using Sobel operator
2: Initialize w 0, w 0, =1
3: for k
Maximum iterations do
4: Compute x²
((()- div(w*-1)
5: Compute gradient V Vf(w") -2 ((v) div div"(u")
6:
Update w = 1lw (wh. -V(u*)/8 (v))
7:
Compute
w² w-1,, 0k- = u
k-2
8:
Compute cos(O))) = (ak. 0k- (1 110=-1l| ()
9:
if cos (z) > c0s((m) then
10:
TR+1 (1 + /1 11+ + 4T2)/2
11:
w (Tk - 1) (w w Tk+1
12: else
TR++1 1, wh++ w
14:
end if
15: if (
then &then
16:
break
end if
18:
end for
return u² u
Initialization phase. The algorithm first constructs spatially adaptive regularization map (v) based on the input image v. This regularization term is governed by a baseline intensity coefficient Mbsse and edge sensitivity coefficient Hgain. with the edge response (Fv) is estimated via the Sobel convolution operator. This strategy automatically reduces the regularization strength in edge regions preserve structural details, while enhancing regularization intensity in the flat areas to effectively suppress adversarial perturbations. Subsequently, the dual variable w and its accelerated counterpart w along with the temporal scaling factor To initialized (Lines 1~2)).
Gradient projection update in the dual space. First, the dual variable field from the previous iteration is transformed into a scalar field via the divergence operator, which is utilized to construct the current estimate of the primal variable image u². Subsequently, the dual variable w at the current iteration is updated and projected onto the dual constraint set W to ensure that the gradient field satisfies the unit ball constraint (Lines 4--6).
Momentum acceleration with adaptive restart mechanism. We measure whether the direction of the angle between two consecutive step increments σn and σ; is reversed to determine whether a restart has occurred. If no deviation in direction is detected, the Nesterov momentum acceleration mechanism is applied to enhance convergence speed. Otherwise, if the angle between directions exceeds a predefined threshold, the momentum accumulation is reset to prevent overshooting caused by trajectory discontinuity, thereby improving the stability of the algorithm. The adaptive restart mechanism originates from analysis of oscillatory behavior inherent in Nesterov-type momentum schemes, which particularly important in the context of spatially weighted total variation with non-uniform regularization terms (Lines 14).
Convergence criterion. The algorithm deemed to have converged when the relative change in updates, measured by the Frobenius norm, remains below the threshold for consecutive iterations. This condition ensures the stability of the solution across multiple time steps in the output image while effectively avoiding redundant iterations. Upon completion of the iterations, the optimal solution
22

---

u" of the output image is obtained. The adaptive-FGr method exhibits strong parallelizability and efficient acceleration mechanisms, significandy enhancing model robustness while keeping the computational overhead below 10% (Lines 15~17).
B Upper Bound Analysis of the Lipschitz Constant
Since the gradient f()) of the objective function f(w) is Lipschitz continuous, there exists constant L such that for any W1, w2, the following inequality holds:
[V((w) - V ((w2)) S Llww - w2lll
(15)
The gradient difference can be computed as:
vf(wi)) - f(w2) 22( diiv [(v ((v) div(wi)) - (u (u-7(0 ((()) div(w)))]
2y(222 v div" [div(wn) div(ur2)
(16)
= =2y(u)² V div" div(wi - W2).
Thus, the norm is bounded by:
[V][(w)) - ((w2))l S 27(0)² | div div || ||w - w2
2v))) ||| div | ||2, -221l
(17)
Analogous to the spectral norm bound of the discrete gradient operator in the TV regularization term, the operator norm the discrete divergence operator satisfies div S v8, we obtain:
[V[(w))- V / < 16-(u)2
|| w2ll.
(18)
Therefore, the upper bound of the Lipschitz constant L(f) for the objective function f(w) is given by:
L(f) S 16)(
(19)
C Additional Experimental Results
C.1 Sensitivity Analysis of PGD Attack Hyperparameters
Table systemaaiiaay evaluates the impact of different configurations on natural and robust accuracy
across five datasets (Caltech101 [69. DTD [67], EuroSAT (65), FGVC-Aireraft (7]), OxfordPets
(73) under the 16-shot setting and varying perturbation budgets e = [1/255, 2/255, 4/255J. Specif- ically, it assesses the sensitivity to different numbers of attack iterations ( = (2, 8) and step sizes {e/t, 2e/t,4e/c). During the robustness evaluation phase, 100-step PGD attack with the same perturbation budget and step size as in the training phase is employed to fully examine the defense capability of the model under strong attacks. We aim to determine the optimal combination of hyperparameters to more efficiently perform the next adversarial robustness tests under stronger attacks.
As evidenced in Table employing larger attack step counts and step sizes during training (t- 8,5 4e/1) does not enhance adversarial robustness during evaluation. Adversarial examples generated by PGD-8 tend to deviate significantly from the true data distribution, potentially causing the model to overfit the distribution of adversarial samples encountered during training rather than learning generalizable robust features. The model achieves higher natural accuracy when trained with larger number of attack steps and a smaller step size (1 = 8, s = c/1). as the resulting adversarial examples remain in close proximity to the original data manifold. The model demonstrates the capability to learn robust features while preserving discriminative power for natural samples. Across all perturbation budget settings, the combination of two attack iterations with a step size of 4e/1 consistently achieves optimal robust accuracy and high clean accuracy. Therefore, we adopt this hyperparameter configuration for subsequent experiments involving varying perturbation budgets and different adversarial attack methods
23

---

Table 5: Impact of perturbation budgets, attack iteration steps, and attack step sizes on natural and robust accuracy across 5 datasets under the 16-thot benchmark. Bold values highlight the best average results per perturbation budget.
Pert. budg.
ltr. steps
Step size
Caltech101
DTD
FuroSAT
RGVCAircraf
OxfordPets
Average
Nat.
Rob.
Nat.
Rob.
Nat.
Rob.
Not.
Rob.
Nat.
Rob.
Nat.
Rob.
e/e
94.16
89.57
65.37
54.20
86,42
71.67
38.94
25.47
87.05
70.84
74.39
62.35
2//s
94.04
89.78
65,84
56.03
84.77
70.60
39.39
27.51
86.29
71.85
74.07
63.15
4e/1
94.20
90.55
65.19
57.03
86.35
73.52
39.39
28.74
87.33
74.35
74.49
64.84
6/c
94.36
89.86
65,43
55.38
86,16
71.20
39.18
26.52
87.44
70.43
74.51
62.68
1/255
2e/1
94.08
89.77 78
65.13
55.38
85.62
73.37
39.24
27.24
86.37
71.74
74.09
63.50
de/s
94.04
90.14
65.66
56.21
86.47
74.75
39.72
27.30
86.86
72.47
74.55
64.18
e/e
94.20
89.74
66.31
54.85
86.52
69.32
39.30
25.59
87.11
69.80
74.69
61.86
2r/s
94.04
89. 98
65.72
55.56
86.49
73.10
38.94
28.05
86.07
71.41
74.25
63.62
4e/1
94.24
89.90
65.19
55.50
86.65
74.09
38.88
27.18
86.59
71.95
74.31
63.72
</c
93,96
87.10
64,24
49.23
83.86
67.12
38.58
21.90
84,93
58.63
73.11
56.80
26/1
93.67
86, 82
63.12
50.00
81.70
64.26
37.11
21.90
83.21
59.23
71.76
56.44
4e/s
93.91
64.30
52.36
84.07
66.90
36.09
23.52
84.36
63.07
72.55
58.83
2/255
e/e
94.00
86.09
65.07
49.11
83,98
68.99
37.95
22.11
84.71
58, 78
73.14
57.01
2//4
93.71
86.77
63.48
49.88
83,90
70.63
36.72
22.95
83:05
58.63
72.17
57.77
4e/n
93.71
86.73
63.42
49.70
83.77
68.53
36.72
83.89
59.42
72.30
57.75
e/c
93,91
86.21
64,30
48.58
84,10
67.96
37.68
22.02
84.87
57.10
72.97
56.37
2e/1
93.67
86.73
62.83
48.88
80.64
67.58
37.11
23.76
82.75
57.78
71.40
56.95
4e/s
93.71
63.00
49.59
83.74
67.54
37.44
23.64
83.05
58.54
72.19
57.25
e/c
92.41
82.15
60.87
41.08
79.53
67.17
33.39
17.28
79.26
40.88
69.09
49.71
2//4
92.01
80.41
58.92
38.
79.54
$7.63
32.67
18.24
74.79
39.60
67.59
46.85
4e/s
92.58
83.20
59.63
45.27
79.49
58.51
33,33
19.80
79.45
49.03
68.90
$1.16
4/255
e/c 92,74
81.30
61.82
40.07
80.12
64.15
33.57
17.76
79.50
41.40
69.55
48.94
2r/1
92.01
80.20
59.69
40.31
78,70
63.54
31,53
17.85
75.61
39.55
67.51
48.29
4e/s
92.33
80.45
59.04
38.42
79.57
59.01
33.03
18.54
76.04
39.11
68.00
47.11
e/c
93.10
81.99
61.76
40.07
82.06
60.85
35.07
18.15
78.30
40.47
70.06
48.31
2r/4
91.60
79.63
59.46
40.19
78.99
61.49
32.37
18.60
75.28
38.35
67.54
47.65
4e/1
91.85
80.81
58.92
40.60
78.51
63.32
32.82
19.50
76.02
40.94
67.62
49.03
C.2 Impact of Perturbation Budget on Model Performance
We document the performance of CoAPT under four benchmark settings with three perturbation budgets e = 11/255, 2/255, 4/255). The case of E = 1/255 corresponds to the results presented in the main text of the paper. As shown in Table Janlerr the base-to-nove benchmark, the robust HM metrics decrease by 5.72% and 9.73% as the perturbation budgets increase, remaining within accept- able thresholds overall. The natural HM metrics decrease by only 2.27% and 4.59%, respectively. demonstrating the effectiveness of CoAPT in preserving natural generalization.
Table 6: Performance of CoAPT under varying perturbation budgets on the base-to-novel benchmark
across 11 datasets.
Metric
Caltechiol
DTD
Ea
PGVCAirzuftt
Food10) ImageNet
Flowers101
OxfordPess
StanfondCars
SUNN97
UCF10I
Average
Base
97.25
76.08
91.61
35.37
78,20
66.15
91.94
90.55
73.34
76.69
82.95
78.67
Novel
92,72
$4.00
25.41
79.47
55.41
63.07
94.50
59.20
70.45
68
65.35
1255
HM
94.93
63.18
29.58
78.83
75.79
92.49
65.51
73.44
75.01
71.31
Base
67.96
52.65
78.72
71.65
Novel
43.58
47.40
62.85
51.89
83.71
58.50
54,13
HM
91.09
53:33
60.78
20.12
62.44
68.57
65.44
81.13
46.65
61.35
63:18
60.16
Base
96.90
75.46
80.00
34.45
71.77
63.68
94.87
88 89
69.34
74,99
81 00
76.40
Novel
90.39
$2.29
62.72
25. 19
75.29
$3.45
$7.23
91.16
$5.06
67.65
64.31
62.98
2/53
IIM
93.53
61.78
73.58
29.11
72.52
58.12
71.40
90.01
61.38
71.13
71.72
69.04
Rase
93.35
63.77
70.00
20.11
90.12
48.76
85.94
69.50
44.70
60.50
67.77 67.70
62.15
Novel
84.83
40.34
$3.36
14.04
50.20
40.93
42.48
73.21
32.85
$2.94
47.54
48.43
HM
$8.88
49.42
63.70
16.33
$0.16
44.50
56.86
71.35
37.86
$6,47
55.89
$4,44
Base
95.22
71.99
89.24
29.65
64.67
58.48
91.17
84.26
62.12
71,27
77.40
72.32
Nat.
Novel
86.24
48:31
64.41
85.51
47.78
63:20
58.73
58.13
4/255
HIM
90.51
57.82
25.35
64.54
52.98
63.81
84.88
54.02
67.27
66.79
64.45
83.32
54.75
74.52
15.90
34.56
39.71
78.63
54,12
32.05
51.19
57.06
52.81
Rob.
Novel
75.11
32.97
10.62
$7.10
23.32
37.21
IIM
$1.18
41.16
60.25
12.75
33.02
36.07
44.39
55.57
26.99
47.32
45.05
44.71
Table reports the natural and robust accuracy of CoAPT under the 16-shot setting across different perturbation budgets. Compared to the base-to-novel setup, the few-shot scenario provides more training samples, enabling the model to exhibit greater stability when confronted with increased perturbations. Specifically, the perturbation budgets increase, the robust accuracy declines by
24

---

5.42% and 8.73%, while the natural accuracy drops by only 2.26% and 4.42%, indicating a more moderate performance degradation trend.
Table 7: Performance of CoAPT under varying perturbation budgets on the few-shot benchmark
across 11 datasets.
Metric
Caltech101
DTD
EuroSAT
FGVCAiceaft
Food]0]
ImageNet
Flowers IDI
OxfordPets
StanfurdCars
SUN397
UCFI0L
Average
4P35 (1255
Nat. 94,51
65.50
K5.42
38.60
73.00
62.96
93.86
86.82
73.94
70:84
79.09
74.96
Rob
90.03
56.09
71.43
27.39
56.72
51.32
85.36
72.28
55.91
58.35
67.91
62,98
Nar.
93.91
64.30
84.07
36,09
68.08
61 61.38
91.64
84. 36
70.25
69.42
76.24
72.70
88.48
52.36
66.90
46.99
47.79
47.05
54.52
57.56
92.58
59.63
79.49
33.33
60,00
56.98
87.74
7945
63.90
65:70
72.03
68.28
Rob
83,20
45.27
58.51
20.10
32.19
40.14
72.55
49.03
35:43
47.11
53.56
48.83
As shown the evaluation results under the zero-shot settings in Table 8] our model consistently
demonstrates strong natural generalization, adversarial robustness, and stability across different perturbation budgets. Specifically, under the zero-shot scenario, the average robust accuracy decreases by 3.93% and 7.35% with increasing perturbation budgets, while the average natural accuracy declines by only 1.63% and 4.21%. The results indicate that the model maintains strong perturbation resistance even under extreme generalization conditions. The evaluation results under the out-of-distribution settings in Table exhibit a similar trend.
Table 8: Performance of CoAPT under varying perturbation budgets on the zero-shot benchmark
across 11 datasets.
MMtric|
Imagehee
Catrec101
IPTD
FurGAAT
RGVCAirennt
Food10:
Flowers101
OxfondPers
StanforKCass
SUN397
UCFI0I
Average
4255 2253 1233
Nar.
63.42
89.10
36,66
28.37
13.84
69.85
56,30
$2.98
47.99
58.10
57.34
$4.91
Roin.
$1.18
83:29
30.59
20.13
9.43
55.99
45.34
30:24
46.64
44.57
Nat.
61.33
88.84
36.52
26.94
11.64
66.13
55.42
82.07
46.14
56.52
54.51
53.28
Rob
46.98
29.31
19.06
7.17
50.28
41.29
23.96
40.64
56.77
33.75
22.72
41.01
49.07
Rob
38.62
75.66
25.24
14.17
6.51
36.61
33.82
52.03
15.00
34.87
33.65
33.29
Table 9: Performance of CoAPT under varying perturbation budgets on the out-of-distribution
benchmark across 11 datasets.
ImageNet-A
ImageNet-R
ImageNet-Skuuee Sketch
ImageNet-V2
Average
Nat.
Rob.
Nat.
Rob.
Nat.
Rob.
Nat.
Rob.
Nat.
Rob
1/255
16.99
9.72
60.35
50.71
35.76
29.13
54.35
42.23
2/255
14.27
7.49
57.81
45.92
34,26
25.81
52.54
38.50
39.72
29.43
4/255
10.35
4.2
54.28
38.80
32.17
21.10
47.63
30.67
23.70
C.3
Robustness Evaluation under Varying Attacks
We evaluate our method using attack types based on different perturbation mechanisms. The CW attack is an optimization-bant -assd method designed to generate adversarial perturbations that are minimal in magnitude yet highly effective in misleading the model. It has demonstrated strong attack performance across various tasks. The TPGD attack is a targeted variant of the PGD attack that misdirects samples toward specific target classes. AutoAttack is ensemble-based, parameter- free robustness evaluation framework that integrates multiple strong attack algorithms to provide reliable adversarial assessment results. Specifically, under the zero-shot benchmark, while only CW and TPGD are evaluated under the base-to-nowe benchmark. We adopt PGD attack with the hyperparameter configuration = 4/255, for adversarial training. During the robustness evaluation phase, both CW and TPGD attacks are applied with the same perturbation budget and step size, while the number of attack steps is uniformly set to 100. For AutoAttack, we use the same perturbation budget (c = 4/255), and its attack process does not rely on hyperparameters such as step size the number of steps. Overall, the robustness advantage of our method is not a result of overfitting to any specific attack.
evaluate CW, TPGD, and AutoAttack attacks
2, 4e/t
Table 10: Performance of CoAPT against various attack methods under the base-to tovell benchmark.
TTpe
Matris
CalcaaTol
DTD
EweSAT
RIVCAireatt
Fooil1)
ImageNer
FlowersTi
Odieddens
SUN997
UCFIIT
Average
92.22
71.99
89.24
2905
64.67
58.45
91.17
84,26
62.12
77.40)
72:32
Nat.
Noval
24
64.41
49.08
63.70
57.82
75.28
64.54
63.51
67.27
81.64
55.12
36.32
07.55
Rob
Novel
00
38.41
51.64
17:34
56.55
40.39
42.48
69.35
35.31
$2.56
4673
47.80
HM
50.29
46.64
63.27
19:26
57.35
45.20
36.94
08.50
39.37
36.15
55
54.08
Bise
95.16
71.59
59.21
29.33
61.68
38,16
91.17
$4.25
02.12
71.21
77.40
72.32
Nat.
Novel
96.24
48.31
65.05
22.08
64.39
48.44
49.01
85.51
47.81
63.66
58.73
58.11
HM
90:48
$7.82
75:24
25:37
64.53
$2.95
63:75
84.88
54.03
67.20
66.79
64:44
90002
02.95
90.30
79.39
63.74
Rols.
Novel
$4.30
62.41
62.67
48.01
62.80
25

---

As can be seen from the experimental results under the base-to-nove benchmark in Table I0 our approach exhibits strong robust generalization capabilities when confronted with different types of adversarial attacks. Overall, the CW attack is more destructive. Although it induces significant accuracy degradation on novel classes, the performance remains within acceptable range. In contrast, under the TPGD attack, the model maintains relatively high natural and robust accuracy, further validating the stable performance CoAPT across different types of adversarial attacks.
Figure presents the robust accuracy of the model under CW, AutoAttack, and TPGD attacks across 11 datasets the zero-shot shot benchmark. In terms of overall trends, the model demonstrates the strongest robustness under TPGD attacks, achieving the highest robust accuracy across nearly all datasets. In contrast, CW attacks are more destructive, particularly showing stronger attack effectiveness on complex datasets such as ImageNet and StanfordCars. AutoAttack, as an ensemble- evaluation framework, displays intermediate attack strength between CW and TPGD. Moreover, significant robustness variations exist across different datasets. The model maintains relatively high robust aceuracy on Caltech 101, Flowers102, and OxfordPets, while showing noticeably lower performance on FGVCAircraft and EuroSAT.
100
Robust Accurccy (%)
CW
80
TTGD
60
40
20-
ImageNet
Caltech101
DTD
BureSAT
PoVCAimal
Food
Flowars
OxfordPets
StanfordCara
SUN397
UCF101
Figure 4: Comparison of robust accuracy under different attack methods on zero-shot benchmarks. To evaluate the impact of f2-norm adversarial attacks on robust VLMs, we designed and conducted an experiment based on 2y-norm perturbations. The training weights were derived from the fo -bsed PGD attack, and the evaluation settings remained consistent. Table II presents the experimental results of our approach across five datasets under varying perturbation budgets. It can be observed that as the perturbation budget increases, the model's classification accuracy experiences a moderate decline. Nevertheless, our approach significantly improves the model's 's robustness against 22-norm attacks, even under the lo -nomm threat model.
Table 11: Robust accuracy under no-m PGD attacks on the base-to-novel benchmark.
Caltech101
DTD
EuroSAT
FUVCAiseaatt
OxfordPets
Base
Novel
Base
Novel
Base
Novel
Buse
Novel
Base
Novel
1/255
94.25
87.99
68.29
44.44
85.45
55.23 $4.49
25.87
16.86
80.75
83.95
2/255
92.32
37.80
79.64
12.96
73.52
75.17
4/2.5
90.70
81.00
61.00
36.96
81.00
54.38
18.91
13.92
70.28
70,86
C.4 Sensitivity Analysis of Prompt Length and Depth in Multimodal Prompting
Prompt depth and prompt length. We conduct ablation studies on prompt depth and prompt length under the base-to-novel setting across 10 datasets, excluding ImageNet and its variants. Figure 5 Summarizess the average results over these datasets. As shown in the left panel of Figure 3 model performance steadily improves with increasing adversarial prompt depth. However, performance gains plateau when the depth exceeds nine layers, showing diminishing returns. To avoid introducing excessive trainable parameters, we ultimately set the prompt depth to 9.
The right panel of Figure 5 illlusttates the impact of prompt length on model performance. As the number of prompt tokens increases, the natural and robust performance on base classes remains relatively stable, whereas the natural and robust performance of the novel classes exhibits a declining trend. This indicates that excessive trainable prompt tokens are prone to overfit task-specific features, thereby undermining the task-agnostic generalization capability of VLMss. Similar performance trends have also been reported in the literature [57]. The model achieves optimal performance when the prompt length is set to 4.
26

---

Nit. Base Ace.
Nut. NovelAee
Nut. HM
Rob. Besc Avc.
Rob, Novel Ave.
Rob. HM
80
75
Performancc (%)
70
Parformance
70
65
65
60
60
50
10
Prompt depth
Prompt tokens
Figure Analyze the impact of prompt depth (left) and prompt length (right) on the performance.
C.5 Performance Across Different CLIP Architectures
We additionally evaluate CoAPT on the CLIP ViT-B/16 architecture under the base-to-novel bench- mark to verify its scalability to higher-resolution architectures in terms both natural accuracy and adversarial robustness. Compared to ViT-B/32, the V architecture adopts finer image patching granularity, resulting in greater number of imput tokens and consequently exhibiting superior spatial resolution representation capacity. This structural advantage typically leads to enhanced performance in fine-grained visual tasks.
Table 12: Results of base-to-novel benchmarks on the ViT-B/16 architecture of CLIP under 11 datasets.
Metric
Calnech101
DTD
EamSAT
FGVCAireraft
Food101
ImageNet
Howers
OxfordPets
StanfordCars
SUNS97
UCFIDI
Average
Base
97.93
78.13
93.64
41.54
83.26
71.76
97.63
94.10
77.49
79.21
83.82
81.68
Nat.
Novel
94.00
56.40
54.36
32.87
83,50
60.02
67.09
94,69
63.33
72.96
72.36
68.33
HM
95.92
65.51
83.38
65:36
94.39
69.70
75,95
77.67
74.41
Base
96.45
71.76
90.05
34.21
72.14
61.72
92.31
87.08
62.24
70.65
75. 25.28
73.99
Rob.
Novel
90.72
$1.45
47.64
72.43
52.47
88.59
64.33
63.17
60.21
IIM
59.90
56.72
68.70
Compared to the CLIP VIT-B/32 results reported in [ the main text, Table [2 demonstrates
that the CLIP ViT-B/16 architecture achieves improvements of 3.1% and 6.23% in the HM of natural accuracy and robust accuracy, respectively. The high-resolution visual representations of the ViT-B/16 architecture provide CoAPT with a finer-grained and more stable latent space, enabling more effective reconstruction of natural generalization features disrupted by adversarial perturbations. Compared to the ViT-B/32 architecture, this enhanced representational capacity mitigates alignment errors and distributional shifts between language and vision embeddings, thereby significantly improving the natural generalization and adversarial robustness of robust CLIP. In contrast, the FAP method fails to achieve robustness gains under the ViT-B/16 architecture, further demonstrating the superiority of TT in terms of scalability and stability.
C.6 Impact of Reconstruction Loss Functions Model Performance
CoAPT employs a Gaussian radial basis function (RBF) to measure the similarity between the language and vision branch embeddings of natural and robust CLIP representations in the latent space, effectively capturing the impact of input perturbations on the feature distributions. In Table 13 systematically compare the performance of CoAPT on the base-to-novel benchmark under different configurations of Gaussian RBF and standard MSE loss functions. The Gaussian RBF demonstrates absolute superiority over MSE by 5.15% and 4.81 .1.1% % in natural HM and robust HM metrics, respectively. This is attributed to the fact that Gaussian RBF can effectively amplify the feature shifts caused by small-scale perturbations to acutely capture the subtle distributional changes, which not only promotes robustness training but also inhibits overfitting to a certain extent.
C.7 Independent and Joint Vision-Language Adversarial Prompting
CoAPT employs deep contextualized joint lingun-language adversarial prompting (JVLAPP), which refines visual prompts based linguistic prompts to facilitate cross-mot cross-modal co-optimization via
27

---

Table 13: Results of base-to-nover benchmarks using Gaussian RBF and MSE loss functions under
11 datasets.
Metric
1a
DTD
EurSAAT
RGVCAireaaft Food101
ImageNet
Flowers101
OxfordPers StanforDCars
SUN397
UCF101
|Aeraaee
Base
97.25
76.08
91.61
35.37
78.20
66.15
94.94
90.55
73.34
76,69
82.95
78:47
RBF
Nai
Novel
92.72
54.03
56.11
25,41
79.47
55.41
63.07
59.20
65,35
HM
94.93
63.18
69.50
29.58
78.83
60.30
75.79
92:49
65.51
73.44
75.01
71,31
Oaues
Bose
94.38
67.98
84.67
25.37
62.03
52.65
88.57
78.72
54.20
64.50
71.65
67.70
Rob
Novel
88.03
43.88
47:40
16.68
62.86
15.07
51.89
83.71
40.95
58.50
56.50
54,13
HM
91.09
53.33
60.78
20.12
62.44
48.57
65.44
81.13
46.65
61.35
63:18
60.16
71.34
Nat
Novel
89.30
46.62
41.03
21:30
74.55
46.00
53.97
91
49.94
64.65
62.52
58.31
MSE
HM
92.65
57.05
57.17
26.18
73.56
52.85
68.83
90.28
58.75
68.67
70.04
66.16
Rose
93.35
64.35
86.24
24.19
56.18
48.90
89.36
76.50
53.30
61.17
70.94
65.86
Rob
Novel
51.22
HM
89.06
47.82
49.24
17.77
36.69
41.50
57.06
78.57
41.57
56.28
59.49
55.35
vision-languaal coupling network. In Table 14 we additionally report the performance of CoAPT us- ing independent vision-language adversarial prompting (IVLAP) under the base-to-novel benchmark. IVLAP exhibits reductions of 0.29% and 0.37% in the HM of natural and robust accuracy, respectively. Although IVLAP shows slightly better performance on the Flowers 10w and StanfordCars datasets, its performance on most other datasets is comparable or slightly inferior that of JVLAP.
Compared to the JVLAP results in Table D
Table 14: Performance of CoAPT using the IVLAP scheme on 11 datasets under the base-to-novel
benchmark.
Metrie
Caltech101
DTD
EuroAAT
RGVCAirenft
Food10
Imagehet
Fowerst01
OxfordPets
StanfordCars
SUN399
UCF100
Average
Base
96.71
76.74
92.88
33.91
78.32
66.30
95.73
90.86
72.24
76.98
81.13
78.34
IWLAP
Novel
92.25
53.02
52.41
26.75
$0.29
55.20
61.18
91.13
$8.83
70.31
67.01
61.91
HM
94.43
62.71
67.01
29.91
79.29
60.24
76.84
92,46
64,85
73.50
73.40
71.02
Base
94,58
67.71
84,88
24.67
61.22
52.57
89.55
79.11
53.87
64.63
69.39
67.47
Rob
Novel
88.10
43.12
43.87
16.74
63.57
44.94
$2.06
83.84
41.67
58.73
53.92
53.69
61.51
59.79
JVLAP shows more significant advantages in modeling cross-modal robustness. By jointly optimizing adversarial features of both vision and language branches within a unified framework, more effectively captures the synergistic variations between the two modalities in the latent space, thereby enhancing the consistency and stability of modality alignment. This joint optimization not only mitigates performance bias caused by asymmetrical perturbation sensitivity between modalities but also preserves semantic consistency during ad adversarial training. Consequently, it significantly enhances the generalization capability of the model on novel categories, zero-shot recognition, and scenarios.
D Impact Statement
This work aims to support progress in robust machine learning by improving the resilience of vision- language models against adversarial threats. Although we do not anticipate any immediate negative consequences, it is important to remain aware of potential misuse in security security-critical critical domains. One key outcome of our approach is the ability to preserve robustness with low-cost model adjustments, mobile and ssource-limiite devices.
which offers practical value for time-sensitive applications The techniques introduced here may contribute to safer and more dependable deployment of AI systems in real-world environments, particularly in areas like intelligent sensing and mobile security.
E Reproducibility
To support reproducibility, we have included the anonymized source code in the supplementary materials for the review process. If the paper is accepted, we will release the complete codebase to the public.
F Limitations
This work primarily investigates adversarial robustness against image-level perturbations, while multi-modal adversarial attacks that simultancously affect both vision and language inputs remain
28

---

underexplored. The current framework assumes that adversarial noise originates solely from the visual modality, which limits its applicability in scenarios involving adversarial manipulations in textual inputs. Although the proposed latent space reconstruction method shows strong generalization in experiments, specific impact on generalization behavior and the theoretical analysis for its superiority over other techniques remain unexplained. The influence of latent space structure and distribution on model robustness and generalization requires further theoretical exploration. We leave these limitations as essential directions for future investigation.
29


Reviews:
[{"review_id": "OBaPrlE0fh", "reviewer": "Reviewer_e7hZ", "summary": "This paper proposes CoAPT, a novel method to enhance the adversarial robustness of Vision-Language Models (VLMs) while preserving their natural generalization capabilities. CoAPT employs an improved real-time total variation technique to suppress high-frequency details in input images, compressing the adversarial perturbation space. It then reconstructs corrupted natural features in the latent space of pre-trained VLMs, guided by high-level image and text representations. The approach achieves a trade-off between robustness and generalization, outperforming SOTA methods across four benchmark tasks (few-shot, base-to-novel, zero-shot, and out-of-distribution generalization).", "questions": "1. The paper should more explicitly differentiate CoAPT from prior work using latent-space alignment and TV regularization.\n\n2. The paper would benefit from adding theoretical analysis explaining why latent-space reconstruction better preserves generalization compared to alternatives. \n\n3. While the CLIP results are compelling, testing on at least one additional VLM architecture (e.g., LLaVA or BLIP) would significantly strengthen claims about general applicability.", "limitations": "It is recommended to discusses computational costs, generalization beyond CLIP, and potential misuse risks (e.g., stronger adversarial attacks).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Extensive experiments are conducted across 15 datasets and four benchmark tasks. The results show consistent improvements in both robustness and generalization, with detailed ablation studies justifying design choices. \n\n2. The paper is well-structured, with a clear problem statement, methodology, and experimental setup. The figures and tables effectively summarize key results. \n\nWeaknesses:\n1. The core idea of latent-space alignment has appeared in prior work (e.g., in self-supervised learning), and the use of TV regularization for adversarial defense is not entirely new. While the integration of these components is novel, the paper could better delineate how CoAPT differs from related approaches in robust fine-tuning and adversarial prompting.\n\n2. Some technical details (e.g., the exact formulation of the low-rank residual module) are briefly mentioned but not deeply explained. A more intuitive breakdown of how Renyi divergence helps compared to standard KL divergence could improve accessibility. \n\n3. The evaluation is limited to CLIP-based models (ViT-B/32). Testing on larger VLMs (e.g., LLaVA, Flamingo) or different architectures would strengthen claims about general applicability.\n\n4. While the empirical results are strong, the paper does not provide a theoretical analysis of why the proposed latent-space reconstruction preserves generalization better than alternatives. Additionally, the computational cost of adaptive-FGP is not thoroughly discussed—real-time performance claims should be quantified with latency benchmarks.", "was_revised": true, "final_justification": "I tend to increase my rating.", "rebuttal": "**We thank reviewer for the constructive comments. We provide our feedback as follows.**\n\n**Answer for W1&Q1:**\nGuo et al. [1] did not optimize TV regularization for generalization, performance, or efficiency. Their method significantly degraded clean features and required costly offline denoising due to high computational overhead. In contrast, CoAPT restores natural features via latent-space reconstruction with only ~10% added training cost.\n\nUnimodal visual- or text-only prompting lacks cross-modal guidance and fails to leverage the joint benefits of multimodal prompting against adversarial samples. While FAP [2] introduced multimodal prompts, it relied solely on comparing logits from robust CLIP models, limiting generalization. In contrast, CoAPT uses latent features from the original CLIP for training, differing fundamentally in both design and implementation.\n\nFare [3] conducts full-parameter fine-tuning of the visual encoder via latent-space alignment, incurring high training costs and overfitting risks that weaken generalization. It also overlooks the role of the text encoder in resisting visual adversarial noise. In contrast, CoAPT applies lightweight adversarial prompt modulation, preserving the pre-trained VLM's generalization while supporting longer training. It further leverages the complementary effect of text prompts and explores synergy between textual and visual cues, advancing multimodal adversarial defense.\n\n**Answer for W2:**\n\nDue to space limitations, we will add more details to the LoRA residual module and Rényi loss in the final version.\n\n\n**Answer for W3&Q3:**\n\nWe provide additional image classification results based on CLIP (ViT-B/16) model in the appendix. Furthermore, to assess the general applicability of our approach, we evaluate image captioning performance using CIDEr scores on the larger VLM model LLaVA-1.5 7B. The evaluation is conducted on the Flickr30k and COCO2014 datasets. Specifically, during the attack phase, we adopt a FARE[3]-based attack pipeline designed to substantially degrade LLaVA performance while maintaining computational feasibility. The attack process employs five ground truth captions per sample as sequential targets, executing five rounds of half-precision (float16) APGD attacks to maximize effectiveness by leveraging multiple reference annotations. For each sample, the attack pipeline maintains a cumulative best-result dictionary that tracks the strongest degradation achieved for each image or query. Once the performance score falls below a predefined threshold, the corresponding sample is excluded from subsequent attack rounds. The CIDEr threshold is set to 10.0 for the COCO dataset and 2.0 for Flickr30k. The APGD attack is conducted under the $L_\\infty$ norm with perturbation budgets of 1/255, 2/255, and 4/255, using 100 iterations. Table 1 provides representative examples of CIDEr score degradation across attack rounds.\n\nBoth the base model CLIP and the SOTA method FARE[3] are evaluated under the exact same configuration as CoAPT, using the official pretrained weights provided by the FARE repository. Since FARE only provides models trained with perturbation budgets of EPS=2 and EPS=4, we currently report its performance under these settings only. \n\nAs shown in Table 2 and Table 3, CoAPT consistently achieves superior clean and robust accuracy across all EPS settings. Compared to the FARE method, CoAPT delivers an average improvement of 36.15 (EPS2) and 40.465 (EPS4) in robust CIDEr scores across two datasets, while maintaining higher performance on clean data. Our method employs 20 rounds of 16-shot training on ImageNet, using only 16,000 training images, which is significantly fewer in both data volume and training time than the full dual-pass ImageNet training adopted by FARE.\n\n**Table 1. CIDEr Degradation across Attack Rounds and EPS**\n\n| Pert. budg.     | Vanilla CLIP EPS2 | Vanilla CLIP EPS4 | CoAPT EPS2 | CoAPT EPS4 |\n|-----------------|-------------------|--------------------|------------|------------|\n| Nat             | 81.51             | 81.51              | 82.89      | 85.04      |\n| APGD with GTC 1 | 12.81             | 8.39               | 76.91      | 69.58      |\n| APGD with GTC 2 | 7.86              | 3.81               | 72.33      | 64.13      |\n| APGD with GTC 3 | 6.55              | 2.73               | 70.06      | 62.42      |\n| APGD with GTC 4 | 4.22              | 1.86               | 68.63      | 61.18      |\n| APGD with GTC 5 | 3.52              | 1.71               | 67.30      | 59.00      |\n\n\n**Table 2. Clean Performance of LLaVA Variants with Different Visual Encoders on Image Captioning**\n\n| Method | Flickr30K EPS1 | EPS2 | EPS4 | COCO EPS1 | EPS2 | EPS4 |\n|--------|----------------|------|------|------------|------|------|\n| CLIP   | 81.51          | 81.51| 81.51| 125.81     |125.81|125.91|\n| FARE   | -              | 80.05| 75.78| -          |115.45|110.11|\n| CoAPT  | 84.23          | 82.89| 85.04| 119.61     |128.94|120.69|\n\n**Table 3. Robust Performance of LLaVA Variants with Different Visual Encoders on Image Captioning**\n\n| Method | Flickr30K EPS1 | EPS2 | EPS4 | COCO EPS1 | EPS2  | EPS4  |\n|--------|----------------|------|------|-----------|-------|--------|\n| CLIP   | 9.12           | 3.52 | 1.71 | 10.93     | 4.28  | 3.04   |\n| FARE   | -              | 40.71| 32.95| -         | 60.82 | 42.03  |\n| CoAPT  | 73.27          | 67.30| 59.00| 107.74    |106.53 | 96.91  |\n\n\n\n**Answer for W4&Q2:**\nIn the context of representation learning, the latent space has been theoretically substantiated as a more semantically aligned modeling domain than the raw input space. Moschella et al. [6] empirically demonstrated that the relative geometric configurations among latent representations, such as angular distributions, remain highly consistent under different initialization and training trajectories, indicating the training stability of the latent space and enabling cross-task and cross-modal generalization through inter-space communication. Building on this, Dou et al. [7] construct class-disentangled, near-orthogonal latent subspaces and employ Gram matrix-based structural analysis to reveal a quantitative association between subspace geometry and robust generalization error, theoretically reinforcing the link between latent structural regularity and generalization robustness. These insights collectively support latent-space reconstruction not as an ad hoc empirical choice, but as a theoretically motivated modeling strategy, wherein the associated inductive bias plays a pivotal role in improving generalization in the target task.\n\n\n\nWe focus on denoising methods with real-time capability and evaluate several classical approaches in terms of both performance and computational efficiency. As shown in Table 4, we compare the total runtime for 16-shot training and testing on the DTD dataset using an NVIDIA Tesla V100 GPU. The proposed adaptive-FGP method employs 10 iterative steps. Under this setting, the time overhead increases by only 10.7\\% compared to the baseline model without contrastive mechanisms. Given the observed performance gains, this overhead remains acceptable. For other classical baselines, wavelet denoising and the Split Bregman method are implemented via the official skimage library, while TVM adopts a precompiled C++ dynamic library for Total Variation denoising. Notably, our adaptive-FGP method incorporates a parallelized implementation that runs directly on CUDA, leading to significantly lower runtime compared to others. However, these runtime differences do not directly reflect computational complexity. Table 5 further reports the average, minimum, and maximum per-image inference time of adaptive-FGP, measured after one epoch over the entire training set.\n\nIn terms of accuracy, our method achieves state-of-the-art performance on both clean and robust evaluations,  particularly excelling in robustness. By integrating a spatially adaptive regularization strategy and an accelerated gradient method with an adaptive restart mechanism, we improve the traditional TV approach to simultaneously enhance performance and real-time capability.\n\n**Table 4. Denoising Runtime and Accuracy (Natural vs. Adversarial)**\n\n| Method                    | Nat (%) | Rob (%) | Average (%) | Runtime (s) | Time Increase (%) |\n|--------------------------|---------|---------|--------------|--------------|--------------------|\n| No Denoising  | 64.3    | 27.8    | 46.05        | 1875         | ~                  |\n| Wavelet Denoising        | 64.2    | 28.8    | 46.50        | 2913         | 55.4%              |\n| Split Bregman            | 64.7    | 31.6    | 48.15        | 2345         | 25.1%              |\n| TVM (Guo et al.[1])      | 56.9    | 50.8    | 53.85        | 7983         | 325.8%             |\n| Adaptive-FGP             | 66.1    | 53.0    | 59.55        | 2075         | 10.7%              |\n\n**Table 5. Adaptive-FGP Latency per Image (ms)**\n\n| Max Time | Min Time | Average Time |\n|----------|----------|--------------|\n|   5.72   |   2.81   |     3.71     |\n\n\n\n[1]Countering adversarial images using input transformations. ICLR 2018.\n[2]Few-shot adversarial prompt learning on vision-language models. NeurIPS 2024.\n[3]Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML2024.\n[4]α-GAN: Convergence and estimation guarantees. ISIT2022.\n[5]A tunable loss function for robust classification: Calibration, landscape, and generalization.IEEE TIT2022.\n[6]Relative Representations Enable Zero-Shot Latent Space Communication. ICLR2023.\n[7]Improving Robust Generalization with Diverging Spanned Latent Space. TMLR2025."}, {"review_id": "pEkUBxu5He", "reviewer": "Reviewer_rp8m", "summary": "This paper proposes an adversarial prompt tuning method that leverages the embeddings of natural images and raw text to guide the optimization of adversarial prompts. It also introduces a real-time total variation regularization technique to suppress adversarial perturbations in high-frequency features. Additionally, the paper presents a regularization strategy to reduce the divergence in prediction distributions between natural and adversarial samples. The results demonstrated across multiple datasets and benchmarks show significant performance improvements.", "questions": "1. The paper only investigates image classification tasks. Can the adversarial prompts trained on classification tasks be transferred to other tasks, such as image captioning?\n2. If the goal is merely to remove high-frequency perturbations from adversarial examples, could methods like HGD be used as an alternative to adaptive-FGP?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is very lightweight, yet simple and effective.\n2. It has been tested on diverse datasets, with experimental results showing significant improvements.\n3. Comprehensive and detailed ablation studies demonstrate the contribution of each component of the method.\n\nWeaknesses:\n1. The paper includes an excessive amount of background information, which results in many important technical details (such as Algorithm 2) being relegated to the supplementary material.", "was_revised": true, "final_justification": "I believe this paper can make contribution to the research field of multimodal robustness, so I recommend acceptance.", "rebuttal": "**We thank reviewer for the constructive comments. We provide our feedback as follows. This will ultimately be reflected in the revision of the paper.**\n\n**Answer for W1:**\nWe will revise the paper structure by condensing the background section and incorporating key technical details, such as Algorithm 2, into the main text.\n\n\n**Answer for Q1:**\n\nWe comprehensively evaluated both task-specific and generalized image classification tasks across four standard benchmarks. To further enhance the quality of our work, we extended the evaluation to more complex image understanding tasks. We selected the stronger LLaVA architecture as the evaluation target. Since LLaVA only supports the CLIP (ViT-L/14) backbone, we retrained CLIP (ViT-L/14) on the ImageNet dataset and transferred the adversarial prompts trained on classification tasks to LLaVA's image captioning task. The latent-space alignment of CoAPT effectively captures high-level semantics and demonstrates strong performance across both classification and understanding tasks, where abstraction and semantic comprehension of objects are particularly critical.We evaluate image captioning performance using CIDEr scores on the larger VLM model LLaVA-1.5 7B. The evaluation is conducted on the Flickr30k and COCO2014 datasets. Specifically, during the attack phase, we adopt a FARE[1]-based attack pipeline designed to substantially degrade LLaVA performance while maintaining computational feasibility. The attack process employs five ground truth captions per sample as sequential targets, executing five rounds of half-precision (float16) APGD attacks to maximize effectiveness by leveraging multiple reference annotations. For each sample, the attack pipeline maintains a cumulative best-result dictionary that tracks the strongest degradation achieved for each image or query. Once the performance score falls below a predefined threshold, the corresponding sample is excluded from subsequent attack rounds. The CIDEr threshold is set to 10.0 for the COCO dataset and 2.0 for Flickr30k. The APGD attack is conducted under the $L_\\infty$ norm with perturbation budgets of 1/255, 2/255, and 4/255, using 100 iterations. Table 1 provides representative examples of CIDEr score degradation across attack rounds.\n\nBoth the base model CLIP and the SOTA method FARE[1] are evaluated under the exact same configuration as CoAPT, using the official pretrained weights provided by the FARE repository. Since FARE only provides models trained with perturbation budgets of EPS=2 and EPS=4, we currently report its performance under these settings only. At present, evaluations are conducted in half precision (float16) on 100 randomly sampled images with a fixed seed to balance computational efficiency and attack effectiveness. In the final version of the paper, we plan to extend the evaluation to 500–1000 images (as FARE reports results on 500) and include an additional attack round in single precision (float32). If FARE exhibits favorable adaptation at EPS=1, we will also report its performance under this setting.\n\nAs shown in Table 2 and Table 3, CoAPT consistently achieves superior clean and robust accuracy across all EPS settings. Compared to the FARE method, CoAPT delivers an average improvement of 36.15 (EPS2) and 40.465 (EPS4) in robust CIDEr scores across two datasets, while maintaining higher performance on clean data. Our method employs 20 rounds of 16-shot training on ImageNet, using only 16,000 training images, which is significantly fewer in both data volume and training time than the full dual-pass ImageNet training adopted by FARE.\n\n**Table 1. CIDEr Degradation across Attack Rounds and EPS**\n\n| Pert. budg.     | Vanilla CLIP EPS2 | Vanilla CLIP EPS4 | CoAPT EPS2 | CoAPT EPS4 |\n|-----------------|-------------------|--------------------|------------|------------|\n| Nat             | 81.51             | 81.51              | 82.89      | 85.04      |\n| APGD with GTC 1 | 12.81             | 8.39               | 76.91      | 69.58      |\n| APGD with GTC 2 | 7.86              | 3.81               | 72.33      | 64.13      |\n| APGD with GTC 3 | 6.55              | 2.73               | 70.06      | 62.42      |\n| APGD with GTC 4 | 4.22              | 1.86               | 68.63      | 61.18      |\n| APGD with GTC 5 | 3.52              | 1.71               | 67.30      | 59.00      |\n\n\n**Table 2. Clean Performance of LLaVA Variants with Different Visual Encoders on Image Captioning**\n\n| Method | Flickr30K EPS1 | EPS2 | EPS4 | COCO EPS1 | EPS2 | EPS4 |\n|--------|----------------|------|------|------------|------|------|\n| CLIP   | 81.51          | 81.51| 81.51| 125.81     |125.81|125.91|\n| FARE   | -              | 80.05| 75.78| -          |115.45|110.11|\n| CoAPT  | 84.23          | 82.89| 85.04| 119.61     |128.94|120.69|\n\n**Table 3. Robust Performance of LLaVA Variants with Different Visual Encoders on Image Captioning**\n\n| Method | Flickr30K EPS1 | EPS2 | EPS4 | COCO EPS1 | EPS2  | EPS4  |\n|--------|----------------|------|------|-----------|-------|--------|\n| CLIP   | 9.12           | 3.52 | 1.71 | 10.93     | 4.28  | 3.04   |\n| FARE   | -              | 40.71| 32.95| -         | 60.82 | 42.03  |\n| CoAPT  | 73.27          | 67.30| 59.00| 107.74    |106.53 | 96.91  |\n\n\n**Answer for Q2:**\nThe HGD method should work, but we found that some common denoising methods do not work well. Specifically,we focus on denoising methods with real-time capability and evaluate several classical approaches in terms of both performance and computational efficiency. As shown in Table 4, we compare the total runtime for 16-shot training and testing on the DTD dataset using an NVIDIA Tesla V100 GPU. The proposed adaptive-FGP method employs 10 iterative steps. Under this setting, the time overhead increases by only 10.7\\% compared to the baseline model without contrastive mechanisms. Given the observed performance gains, this overhead remains acceptable. For other classical baselines, wavelet denoising and the Split Bregman method are implemented via the official skimage library, while TVM adopts a precompiled C++ dynamic library for Total Variation denoising. Notably, our adaptive-FGP method incorporates a parallelized implementation that runs directly on CUDA, leading to significantly lower runtime compared to others. However, these runtime differences do not directly reflect computational complexity. Table 5 further reports the average, minimum, and maximum per-image inference time of adaptive-FGP, measured after one epoch over the entire training set.\n\nIn terms of accuracy, our method achieves state-of-the-art performance on both clean and robust evaluations,  particularly excelling in robustness. By integrating a spatially adaptive regularization strategy and an accelerated gradient method with an adaptive restart mechanism, we improve the traditional TV approach to simultaneously enhance performance and real-time capability.\n\n**Table 4. Denoising Runtime and Accuracy (Natural vs. Adversarial)**\n\n| Method                    | Nat (%) | Rob (%) | Average (%) | Runtime (s) | Time Increase (%) |\n|--------------------------|---------|---------|--------------|--------------|--------------------|\n| No Denoising  | 64.3    | 27.8    | 46.05        | 1875         | ~                  |\n| Wavelet Denoising        | 64.2    | 28.8    | 46.50        | 2913         | 55.4%              |\n| Split Bregman            | 64.7    | 31.6    | 48.15        | 2345         | 25.1%              |\n| TVM (Guo et al.[2])      | 56.9    | 50.8    | 53.85        | 7983         | 325.8%             |\n| Adaptive-FGP             | 66.1    | 53.0    | 59.55        | 2075         | 10.7%              |\n\n**Table 5. Adaptive-FGP Latency per Image (ms)**\n\n| Max Time | Min Time | Average Time |\n|----------|----------|--------------|\n|   5.72   |   2.81   |     3.71     |\n\n[1]Schlarmann, C., et al. Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML2024.\n[2]Guo, C., et al. Countering adversarial images using input transformations. ICLR 2018."}, {"review_id": "niODjqXWz6", "reviewer": "Reviewer_Uzgu", "summary": "This paper proposes Collaborative Adversarial Prompt Tuning (CoAPT), a novel method for enhancing the adversarial robustness of Vision-Language Models (VLMs) in their shared latent space for natural images and text. Unlike existing Prompt Tuning methods, CoAPT utilizes an improved real-time Total Variation regularization to suppress high-frequency details in images, thereby reducing the effect of adversarial noise. Furthermore, it introduces a residual module with a loss for representation restoration and incorporates Renyi divergence to minimize the gap between the distributions of natural and adversarial examples. The proposed method demonstrates superior performance over existing approaches in few-shot learning, generalization to new classes, zero-shot, and out-of-distribution (OOD) scenarios, achieving an optimal balance between robustness and generalization.", "questions": "- Are there results for other norm-based attacks (e.g., an L2 attack)? I am curious to know if training against an Linf threat model also confers robustness against L2 attacks.\n- Is the proposed method applicable to other types of models, such as those with a ResNet backbone? If so, I am curious to see if it yields similar results.\n- Can this method be applied to a multi-modal LLM (e.g., LLaVA) without issues? While it appears effective for the vision-language tasks presented, I am curious whether there are any problems from the perspective of output feature drift when this robust model is used as a vision encoder within a larger system.", "limitations": "See weaknesses and questions", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThis paper effectively solves the robust-accuracy trade-off, a persistent issue when considering the adversarial robustness of vision-language models. In particular, beyond simply using natural-adversarial examples, it utilizes an adaptive-FGP that considers both the characteristics of adversarial noise and the shortcomings of existing methodologies. Regarding the training loss, instead of merely updating the model with a reconstruction loss, it introduces a residual module inspired by LoRA, which helps mitigate the trade-off. Additionally, it further improves performance by introducing Renyi regularization. As a result, the method achieves high performance on both clean and robust metrics across various datasets and tasks compared to existing baselines.\n\nWeaknesses\n\nThe additional computations, including the proposed real-time TV regularization and the restoration loss in the latent space, increase the model's complexity during training and inference, which could lead to higher computational costs in practical applications. While this may not seem like a major issue in the current few-shot setting, difficulties could arise when scaling the methodology to large-scale datasets. Additionally, it is difficult to discern the specific effect of each applied component. Although the authors indirectly address this through an ablation study, it is still hard to determine if any redundant terms exist in the framework.", "was_revised": true, "final_justification": "I am raising my score based on the authors' rebuttal. My initial review highlighted reservations regarding the method's practical overhead and generalizability, but the authors have fully addressed these concerns. The newly added experiments provided by the authors are exemplary. They convincingly illustrate that the method's effectiveness is not limited to specific threat models or architectures initially evaluated. Furthermore, by showing its successful application as a vision encoder within a generative multi-modal LLM, they resolve my concerns about its scalability and potential side effects. This additional evidence resolves my primary reservations. It clearly demonstrates that the proposed method is effective, versatile, and robust.", "rebuttal": "**We thank reviewer for the constructive comments. We provide our feedback as follows. This will ultimately be reflected in the revision of the paper.**\n\n\n**Answer for W1:**\nWe focus on denoising methods with real-time capability and evaluate several classical approaches in terms of both performance and computational efficiency. As shown in Table 1, we compare the total runtime for 16-shot training and testing on the DTD dataset using an NVIDIA Tesla V100 GPU. The proposed adaptive-FGP method employs 10 iterative steps. Under this setting, the time overhead increases by only 10.7\\% compared to the baseline model without contrastive mechanisms. Given the observed performance gains, this overhead remains acceptable. For other classical baselines, wavelet denoising and the Split Bregman method are implemented via the official skimage library, while TVM adopts a precompiled C++ dynamic library for Total Variation denoising. Notably, our adaptive-FGP method incorporates a parallelized implementation that runs directly on CUDA, leading to significantly lower runtime compared to others. However, these runtime differences do not directly reflect computational complexity. Table 2 further reports the average, minimum, and maximum per-image inference time of adaptive-FGP, measured after one epoch over the entire training set.\n\nIn terms of accuracy, our method achieves state-of-the-art performance on both clean and robust evaluations,  particularly excelling in robustness. By integrating a spatially adaptive regularization strategy and an accelerated gradient method with an adaptive restart mechanism, we improve the traditional TV approach to simultaneously enhance performance and real-time capability.\n\n**Table 1. Denoising Runtime and Accuracy (Natural vs. Adversarial)**\n\n| Method                    | Nat (%) | Rob (%) | Average (%) | Runtime (s) | Time Increase (%) |\n|--------------------------|---------|---------|--------------|--------------|--------------------|\n| No Denoising  | 64.3    | 27.8    | 46.05        | 1875         | ~                  |\n| Wavelet Denoising        | 64.2    | 28.8    | 46.50        | 2913         | 55.4%              |\n| Split Bregman            | 64.7    | 31.6    | 48.15        | 2345         | 25.1%              |\n| TVM (Guo et al.[1])      | 56.9    | 50.8    | 53.85        | 7983         | 325.8%             |\n| Adaptive-FGP             | 66.1    | 53.0    | 59.55        | 2075         | 10.7%              |\n\n**Table 2. Adaptive-FGP Latency per Image (ms)**\n\n| Max Time | Min Time | Average Time |\n|----------|----------|--------------|\n|   5.72   |   2.81   |     3.71     |\n\n\n**Answer for Q1:**\n\n**Table3. PGD-L∞ vs. PGD-L2 Accuracy (Base-to-Novel, Across Datasets)**\n\n| Dataset        | Type  | EPS1 PGD Linf | EPS1 PGD L2 | EPS2 PGD Linf | EPS2 PGD L2 | EPS4 PGD Linf | EPS4 PGD L2 |\n|----------------|-------|----------------|--------------|----------------|--------------|----------------|--------------|\n| Caltech101     | Base  | 94.38          | 94.25        | 93.35          | 92.32        | 88.32          | 90.70        |\n|                | Novel | 88.03          | 87.99        | 84.83          | 84.06        | 75.11          | 81.00        |\n| DTD            | Base  | 67.98          | 68.29        | 63.77          | 62.38        | 54.75          | 61.00        |\n|                | Novel | 43.88          | 44.44        | 40.34          | 37.80        | 32.97          | 36.96        |\n| EuroSAT        | Base  | 84.67          | 85.45        | 79.00          | 79.64        | 74.52          | 81.00        |\n|                | Novel | 47.40          | 55.23        | 53.36          | 54.49        | 50.56          | 54.38        |\n| FGVCAircraft   | Base  | 25.37          | 25.87        | 20.11          | 20.05        | 15.97          | 18.91        |\n|                | Novel | 16.68          | 16.86        | 14.04          | 12.96        | 10.62          | 13.92        |\n| OxfordPets     | Base  | 78.72          | 80.75        | 69.59          | 73.52        | 54.12          | 70.28        |\n|                | Novel | 83.71          | 83.95        | 73.21          | 75.17        | 57.10          | 70.86        |\n\n**Answer for Q2:**\n\n**Table4. Natural and Robust Performance of CLIP with RN101 and RN50 Backbones under the 16-Shot Setting**\n\n| Dataset     | Metric | RN101 | RN50 |\n|-------------|--------|-------|------|\n| EuroSAT     | Nat    | 67.73 | 68.99 |\n|             | Adv    | 48.33 | 53.23 |\n| OxfordPets  | Nat    | 25.67 | 23.58 |\n|             | Adv    | 18.56 | 16.90 |\n\n**Answer for Q3:**\n\nTo further enhance the quality of our work, we extended the evaluation to more complex image understanding tasks. We selected the stronger LLaVA architecture as the evaluation target. Since LLaVA only supports the CLIP (ViT-L/14) backbone, we retrained CLIP (ViT-L/14) on the ImageNet dataset and transferred the adversarial prompts trained on classification tasks to LLaVA's image captioning task. The latent-space alignment of CoAPT effectively captures high-level semantics and demonstrates strong performance across both classification and understanding tasks, where abstraction and semantic comprehension of objects are particularly critical.We evaluate image captioning performance using CIDEr scores on the larger VLM model LLaVA-1.5 7B. The evaluation is conducted on the Flickr30k and COCO2014 datasets. Specifically, during the attack phase, we adopt a FARE[2]-based attack pipeline designed to substantially degrade LLaVA performance while maintaining computational feasibility. The attack process employs five ground truth captions per sample as sequential targets, executing five rounds of half-precision (float16) APGD attacks to maximize effectiveness by leveraging multiple reference annotations. For each sample, the attack pipeline maintains a cumulative best-result dictionary that tracks the strongest degradation achieved for each image or query. Once the performance score falls below a predefined threshold, the corresponding sample is excluded from subsequent attack rounds. The CIDEr threshold is set to 10.0 for the COCO dataset and 2.0 for Flickr30k. The APGD attack is conducted under the $L_\\infty$ norm with perturbation budgets of 1/255, 2/255, and 4/255, using 100 iterations. Table 5 provides representative examples of CIDEr score degradation across attack rounds.\n\nBoth the base model CLIP and the SOTA method FARE[2] are evaluated under the exact same configuration as CoAPT, using the official pretrained weights provided by the FARE repository. Since FARE only provides models trained with perturbation budgets of EPS=2 and EPS=4, we currently report its performance under these settings only. At present, evaluations are conducted in half precision (float16) on 100 randomly sampled images with a fixed seed to balance computational efficiency and attack effectiveness. In the final version of the paper, we plan to extend the evaluation to 500–1000 images (as FARE reports results on 500) and include an additional attack round in single precision (float32). If FARE exhibits favorable adaptation at EPS=1, we will also report its performance under this setting.\n\nAs shown in Table 6 and Table 7, CoAPT consistently achieves superior clean and robust accuracy across all EPS settings. Compared to the FARE method, CoAPT delivers an average improvement of 36.15 (EPS2) and 40.465 (EPS4) in robust CIDEr scores across two datasets, while maintaining higher performance on clean data. Our method employs 20 rounds of 16-shot training on ImageNet, using only 16,000 training images, which is significantly fewer in both data volume and training time than the full dual-pass ImageNet training adopted by FARE.\n\n**Table 5. CIDEr Degradation across Attack Rounds and EPS**\n\n| Pert. budg.     | Vanilla CLIP EPS2 | Vanilla CLIP EPS4 | CoAPT EPS2 | CoAPT EPS4 |\n|-----------------|-------------------|--------------------|------------|------------|\n| Nat             | 81.51             | 81.51              | 82.89      | 85.04      |\n| APGD with GTC 1 | 12.81             | 8.39               | 76.91      | 69.58      |\n| APGD with GTC 2 | 7.86              | 3.81               | 72.33      | 64.13      |\n| APGD with GTC 3 | 6.55              | 2.73               | 70.06      | 62.42      |\n| APGD with GTC 4 | 4.22              | 1.86               | 68.63      | 61.18      |\n| APGD with GTC 5 | 3.52              | 1.71               | 67.30      | 59.00      |\n\n\n**Table 6. Clean Performance of LLaVA Variants with Different Visual Encoders on Image Captioning**\n\n| Method | Flickr30K EPS1 | EPS2 | EPS4 | COCO EPS1 | EPS2 | EPS4 |\n|--------|----------------|------|------|------------|------|------|\n| CLIP   | 81.51          | 81.51| 81.51| 125.81     |125.81|125.91|\n| FARE   | -              | 80.05| 75.78| -          |115.45|110.11|\n| CoAPT  | 84.23          | 82.89| 85.04| 119.61     |128.94|120.69|\n\n**Table 7. Robust Performance of LLaVA Variants with Different Visual Encoders on Image Captioning**\n\n| Method | Flickr30K EPS1 | EPS2 | EPS4 | COCO EPS1 | EPS2  | EPS4  |\n|--------|----------------|------|------|-----------|-------|--------|\n| CLIP   | 9.12           | 3.52 | 1.71 | 10.93     | 4.28  | 3.04   |\n| FARE   | -              | 40.71| 32.95| -         | 60.82 | 42.03  |\n| CoAPT  | 73.27          | 67.30| 59.00| 107.74    |106.53 | 96.91  |\n\n\n[1]Guo, C., et al. Countering adversarial images using input transformations. ICLR 2018.\n[2]Schlarmann, C., et al. Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML2024."}, {"review_id": "Hi3KEN8ttK", "reviewer": "Reviewer_2CcH", "summary": "This work presents a novel adversarial prompt tuning framework through a collaborative process. It proposes the adoption of a real-time Total Variation regularization algorithm to suppress high-frequency noise that undermines adversarial robustness, followed by the reconstruction of natural features using the original CLIP model.", "questions": "1. From the ablation results in Table 4, it appears that Adp. FGP plays a critical role in the entire framework. What contributes to its importance in your proposed method, and have you considered alternative denoising methods?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength：\n\n- The method provides a new workflow for addressing adversarial prompt tuning tasks, which is quite innovative. It adopts a two-step pipeline: high-frequency noise suppression followed by natural feature reconstruction. Task-specific improvements are proposed for each sub-process based on the characteristics of the task.\n\n- The experimental results are compelling, significantly outperforming previous SOTA methods.\n\n- The strategy of imposing constraints between natural and adversarial features in adversarial prompt learning is particularly insightful. Compared to baseline methods, the proposed loss design is more systematic and demonstrates better numerical stability, enabling more effective feature alignment and reconstruction.\n\nWeakness：\n\n- The quality of this work could be further improved by clearly stating the motivation behind each module design. For instance, the authors introduced the Low-Rank Residual Module and Rényi Regularization in Section 3.3, but did not explain how these specific components or loss terms benefit the task.\n\n- The authors reported only performance metrics, without including any efficiency-related indicators (e.g., GPU usage, training time). For few-shot and zero-shot prompt learning tasks, the efficiency of downstream adaptation is also a key factor influencing the practical applicability of the algorithm.", "was_revised": true, "final_justification": "The authors gave positive response to my questions. \n\nI think the initial assessment remains a fair evaluation. I will keep my score.", "rebuttal": "**We thank reviewer for the constructive comments. We provide our feedback as follows. This will ultimately be reflected in the revision of the paper.**\n\n\n**Answer for W1:**\nWe introduce Rényi divergence as a regularization term in the latent space, aiming to enhance the stability of the model during robust training. As a parametric instance of f-divergence, Rényi divergence, when applied to surrogate losses, enables adaptive modulation of the model's sensitivity to regions with varying probability densities by tuning the parameter $\\alpha$, thereby flexibly adjusting the emphasis on dominant modes and anomalous samples. In contrast, the standard KL divergence imposes disproportionate penalties on minor deviations between the model and target distributions in the tail regions, resulting in rapid gradient amplification that triggers training instability and overfitting. When $\\alpha > 1$, the $\\alpha$-loss induced by Rényi divergence exhibits milder gradient behavior in low-density regions, guiding the model to concentrate on high-density principal modes and thereby yielding a smoother and more controllable optimization trajectory [1,2]. The low-rank residual module constrains the redundant degrees of freedom in the feature space, effectively constructing a degenerate yet discriminative subspace representation. It enables the model to retain backbone features while selectively learning task-specific fine-grained deviations within a compact subspace. \n\n\n**Answer for W2 & Q1:**\nWe focus on denoising methods with real-time capability and evaluate several classical approaches in terms of both performance and computational efficiency. As shown in Table 1, we compare the total runtime for 16-shot training and testing on the DTD dataset using an NVIDIA Tesla V100 GPU. The proposed adaptive-FGP method employs 10 iterative steps. Under this setting, the time overhead increases by only 10.7\\% compared to the baseline model without contrastive mechanisms. Given the observed performance gains, this overhead remains acceptable. For other classical baselines, wavelet denoising and the Split Bregman method are implemented via the official skimage library, while TVM adopts a precompiled C++ dynamic library for Total Variation denoising. Notably, our adaptive-FGP method incorporates a parallelized implementation that runs directly on CUDA, leading to significantly lower runtime compared to others. However, these runtime differences do not directly reflect computational complexity. Table 2 further reports the average, minimum, and maximum per-image inference time of adaptive-FGP, measured after one epoch over the entire training set.\n\nIn terms of accuracy, our method achieves state-of-the-art performance on both clean and robust evaluations,  particularly excelling in robustness. By integrating a spatially adaptive regularization strategy and an accelerated gradient method with an adaptive restart mechanism, we improve the traditional TV approach to simultaneously enhance performance and real-time capability.\n\n**Table 1. Denoising Runtime and Accuracy (Natural vs. Adversarial)**\n\n| Method                    | Nat (%) | Rob (%) | Average (%) | Runtime (s) | Time Increase (%) |\n|--------------------------|---------|---------|--------------|--------------|--------------------|\n| No Denoising  | 64.3    | 27.8    | 46.05        | 1875         | ~                  |\n| Wavelet Denoising        | 64.2    | 28.8    | 46.50        | 2913         | 55.4%              |\n| Split Bregman            | 64.7    | 31.6    | 48.15        | 2345         | 25.1%              |\n| TVM (Guo et al.[3])      | 56.9    | 50.8    | 53.85        | 7983         | 325.8%             |\n| Adaptive-FGP             | 66.1    | 53.0    | 59.55        | 2075         | 10.7%              |\n\n**Table 2. Adaptive-FGP Latency per Image (ms)**\n\n| Max Time | Min Time | Average Time |\n|----------|----------|--------------|\n|   5.72   |   2.81   |     3.71     |\n\n**Answer for Q1:**\nCoAPT can be viewed as a generalized formulation of masked image modeling (MIM). We transform the block-level masking in conventional MIM into fine-grained, pixel-level denoising, and redefine the reconstruction objective as the latent representations of the pretrained CLIP model. The adaptive-FGP module mitigates adversarial perturbations by suppressing high-frequency features, at the cost of degrading image details. Masked image modeling enhances generalization to downstream tasks by learning high-level representations during the image reconstruction process. However, its sensitivity to high-frequency features exposes vulnerabilities to adversarial perturbations. CoAPT’s adaptive-FGP module alleviates this weakness by attenuating high-frequency adversarial components. Although adaptive-FGP induces degradation in natural image content, such distortions can be effectively recovered in the latent space through MIM, which tolerates strong denoising or even structural corruption. As a result, the limitations of each component are mutually compensated, enabling their respective strengths in adversarial robustness and generalization to be jointly maximized.\n\nTherefore, the coupling between adaptive-FGP and image reconstruction is essential to performance improvement. When adaptive-FGP is entirely removed and adversarial images are directly encoded for latent-space reconstruction, the inherent modeling mechanism of MIM leads it to overly focus on high-frequency regions that are heavily perturbed. Although some robustness gains remain, they are significantly compromised by the influence of high-frequency adversarial noise. Consequently, removing adaptive-FGP not only eliminates its inherent denoising capability but also disrupts its synergistic interaction with MIM. This decoupling exposes MIM’s tendency to emphasize high-frequency artifacts, ultimately resulting in degraded performance.\n\n[1] Kurri, Gowtham R., et al. \"α-GAN: Convergence and estimation guarantees.\" 2022 IEEE International Symposium on Information Theory (ISIT). IEEE, 2022.\n[2] Sypherd, Tyler, et al. \"A tunable loss function for robust classification: Calibration, landscape, and generalization.\" IEEE Transactions on Information Theory 68.9 (2022): 6021-6051.\n[3]Guo, C., et al. Countering adversarial images using input transformations. ICLR 2018."}]

Author rebuttal:

Meta-review: The paper proposes Collaborative Adversarial Prompt Tuning (CoAPT) to enhance the adversarial robustness of VLMs in the latent space. The paper is clearly written and well-structured, presenting strong performance across multiple datasets and benchmarks. It makes a valuable contribution to the research field of the multimodal community. The authors’ rebuttal successfully addressed some concerns regarding practical overhead, generalizability, and additional experimental evaluations. These revisions should be incorporated into the final version.
Decision: Accept (spotlight)