Identifying Robust Neural Pathways: Few-Shot Adversarial Mask Tuning for Vision-Language Models
Authors: Wonjeong Choi
Sejong Ryu
Jungmoon Lee
Dong-Jun Han
Jaekyun Moon
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models (VLMs)
Adversarial Robustness
Mask Tuning
Robust Neural Pathways
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: To enhance the robustness of VLMs, we propose Adversarial Mask Tuning (AdvMask), a method that learns a set of binary masks that selectively deactivate model parameters vulnerable to adversarial perturbations.

Abstract: Recent vision-language models (VLMs), such as CLIP, have demonstrated remarkable transferability across a wide range of downstream tasks by effectively leveraging the joint text-image embedding space, even with only a few data samples. Despite their impressive performance, these models remain vulnerable to adversarial attacks, raising significant concerns about their security and reliability in practical deployments. To address this issue, we propose Adversarial Mask Tuning (AdvMask), a method that effectively enhances the robustness of VLMs without directly modifying their pre-trained weights. Instead, our AdvMask learns a set of binary masks that selectively deactivate model parameters vulnerable to adversarial perturbations. By identifying robust neural pathways within the vision encoder, AdvMask facilitates the generation of features and predictions that are resistant to adversarial attacks. Furthermore, we introduce a Layer-wise Adaptive Feature Alignment (LAFA) loss, specifically designed to optimize AdvMask in few-shot scenarios. The LAFA loss adaptively aligns intermediate-layer features from clean and adversarial samples across each transformer block, enhancing the representational robustness of the model. Experimental results across multiple benchmarks confirm that our AdvMask approach substantially outperforms existing adversarial tuning techniques for VLMs, especially in few-shot settings.

Published as a conference paper at ICLR 2026
IDENTIFYING ROBUST NEURAL PATHWAYS: FEW-SHOT ADVERSARIAL MASK TUNING FOR VISION-LANGUAGE MODELS
Wonjeong Choi', Sejong Ryu', Jungmoon Lee', Dong-Jun Han", Jackyun Moon
Korea Advanced Institute of Science and Technology (KAIST),
2 Yonsei University
rousejong, sndanssehjokaist..cc 2 jh@yonsei ac kr; jmoonekaist.edu
[dr jawjdd5555 jorrjax
ABSTRACT
Recent vision-language models (VLMs), such as CLIP, have demonstrated re- markable transferability across a wide range of downstream tasks by effectively leveraging the joint text-image embedding space, even with only a few data sam- ples. Despite their impressive performance, these models remain vulnerable adversarial attacks, raising significant concerns about their security and reliability in practical deployments. To address this issue, we propose Adversarial Mask Tuning (AdvMask), a method that effectively enhances the robustness of VLMs without directly modifying their pre-trained weights. Instead, our AdvMask learns a set of binary masks that selectively deactivate model parameters vulnerable to adversarial perturbations. By identifying robust neural pathways within the vision encoder, AdvMask facilitates the generation of features and predictions that are resistant to adversarial attacks. Furthermore, we introduce Layer-wist wise Adaptive Feature Aligmment (LAFA) loss, specifically designed to optimize AdvMask in features
few-shot scenarios. The LAFA loss adaptively aligns intermediate-lun from clean and adversarial samples across each transformer block, enhancing the representational robustness of the model. Experimental results across multiple benchmarks confirm that our AdvMask approach substantially outperforms existing adversarial tuning techniques for VLMs, especially few-shot settings. Our code is available in battp:://gither http///gttttbut/cc com won wwong hww.c//hhi.A/aaaka. /AdvMask
INTRODUCTION
Vision-Languaan Models (VLMs), such CLIP (Radford et al. 2021), have demonstrated exceptional zero-shot generalization capabilities and impressive transferability across a wide range of downstream tasks, gaining significant attention in recent years (Zhang et al. (024b).. By bridging the semantic gap between visual and textual representations through contrastive learning, they have enabled high-level understanding and versatile potential across various applications.
Motivation. Despite significant advancements, VLMs remain vulnerable to adversarial attacks, which restricts their practical deployment in real-world downstream tasks. This inherent weakness significantly undermines their reliability and trustworthiness, raising concerns in safety-criticu and security-sensitiee downstream applications such as autonomous driving (Tuncali al., 2018 Deng et al. 2020), medical analysis (Buch et al. 2018 Finlayson et al. 2019). and manufacturing systems (Picard et al. 2023). Consequently, there pressing need to develop algorithms that achieves robustness against adversarial perturbations during downstream tasks. This problem becomes even more pronounced in few-shot settings (Dong al. 2022 Wang et al. 2020). where the number of training samples available for the downstream task severely limited (e.g., medical applications).
Challenges. Recently, researchers have explored techniques to strengthen the adversarial robustness of VLMs (Zhao et al. 2023 Cui et al. 2024).. Among these, adversarial tuning of textual visual prompts (Zhou et al.] 2024 Mao et al. 2023 Zhang et al. 2024a) has widely adopted as a prominent method, aiming to improve the model's predictive robustness by carefully modifying the prompts

---

Published as a conference paper at ICLR 2026
to resist adversarial perturbations. While these approaches only require updating a small number of learnable parameters, they overlook the inherent properties in the model's pre-trained structure (i.e., neurons), limiting their capability to produce robust representations against adversarial attacks. Other works attempt to directly fine-tune the model using adversarial training strategies; however, these approaches lead to overfitting in few-shot settings (where only small number of labeled samples are available for each downstream task) and may compromise the generalization ability of the original pre-trained VLM. Furthermore, several methods targeting zero-shot robustness (Yu et al. 2024, Mao et al. 2023) rely on a held-out dataset for adversarial tuning (i.e., no task mask-specific samples are available), but they often fail to achieve satisfactory performance on downstream tasks. The effectiveness of these approaches largely depends on the quality of the held-out dataset. An extended discussion of related works is provided in Sec. 4 Motivated by these challenges, this work, we aim to answer the following key question:
What is the most effective way to achieve robustness against adversarial attacks on pre-trained VLMs in seo- downstream settings?
Key Ideas. Unlike previous methods that predominantly focus on prompt adaptation or direct parameter updates, we propose adversarial mask tuning (AdvMask) approach that searches for robust subnetwork within well- trained VLMs as a promising alternative. Inspired by recent studies
(Zheng al.] 2023 Zhao al. 2020 Lin al. 2020) demonstrating the effectiveness of identifying neural pathways for adapting large-scale pre-trained models, we introduce a novel perspective of a robust neural pathway, which, the best of our knowledge, has not been explored in previous works. Specifically, given a few samples from the downstream task, our goal is to learn a binary mask that identifies subnetwork structure within the pre-trained VLM, one that not only facilitates downstream adaptation but also inherently resists adversarial perturbations. Consequently, by identifying the robust neural pathway, our approach selectively emphasizes robust features during forward passes, substantially improving the adversarial robustness. Interestingly, we demonstrate that such a robust neural pathway indeed exists (further intuitive explanations are provided in Sec. 3.3
Within our AdvMask training paradigm, we introduce the Layer-wise Adaptive Feature Alignment (LAFA) loss, which enables enhanced robustness and stability. Previous objective functions for adversarial tuning (Mao et al., 2023 Zhou et al. (024) primarily provide supervision at the final output stage (i.e., the joint text-image embedding space), overlooking the importance of robust intermediate representations within the vision encoder. In contrast, our LAFA loss explicitly guides each transformer's 's intermediate representations to be robust against adversarial perturbations by closely aligning features extracted from adversarial samples with their corresponding clean sample features. Additionally, to effectively handle the limited data in few-shot settings, we adopt an adaptive weighting mechanism based on predictive reliability. Specifically, within our LAFA loss, features from samples that the model predicts correctly with high confidence provide more reliable alignment signals, whereas samples predicted with lower confidence contribute less, preventing unstable or incorrect optimization. This carefully designed LAFA loss encourages consistent intermediate feature representations between clean and adversarial inputs, improving adversarial robustness in our few-shot AdvMask framework.
Summary of Contributions. Overall, we introduce the notion of robust neural pathway and make the following key contributions:
We propose new few-shot Adversarial Mask Tuning (AdvMask) framework that effectively enhances the adversarial robustness of VLMs by identifying robust sub-network structures using binary masks, without modifying their pre-trained weights.
We introduce a Layer-wise Adaptive Feature Alignment (LAFA) loss, specifically designed to optimize AdvMask training in few-shot scenarios. The LAFA loss adaptively aligns intermediate- layer features between clean and adversarial samples to find the robust neural pathway.
Experiments across various downstream datasets demonstrate that AdvMask consistently improves few-shot adversarial robustness over existing baselines. Moreover, since AdvMask learns and stores only binary masks corresponding to a subset of model parameters, it is highly parameter-efffcient, reducing memory requirements during training and inference (see Appendix See. D.7 for details).
2

---

Published as a conference paper at ICLR 2026
2 ADVMASK: FEW-SHOT ADVERSARIAL MASK TUNING FOR VLMS
2.1
PRELIMINARY AND PROBLEM SETUP
CLIP Recap. In this paper, following prior works on adversarial robustness of VLMs
Zhou et al.
2024
Mao al. 2023 Yu al. 2024), we mainly use the CLIP (Radford et al. 202T as our target
VLM. We also provide results on other VLM, VisualBERT (Li et al. 2019). in Sec. C.4 of Appendix.
CLIP consists of an image encoder T(-) and a text encoder T(-) which project images and text
into joint embedding space via contrastive learning on large-scale paired datasets. This enables strong zero-shot classification performance on diverse image recognition tasks. For a downstream classification task with images {x1,.... .mm} and labels y {l..... K}. }. each label ys is embedded into a textual prompt (e.g.. "a photo [class]") to form input 4, yielding a text representation (t,), Similarly, an input image x is encoded by the image encoder, typically implemented vision transformer ( (ViT) (Dosovitskiy et al. 2020), to produce 21 (z). Finally, the probability that image æ belongs to class y; calculated as:
exp(cos(2r(t)), (z
ply = 11=1
K Σj=1 (t))
(2T
where cos(:;) denotes cosine similarity and is learnable temperature parameter.
Few-Shot Adversarial Tuning, Given a pre-trained VLM with strong generalization capabilities, goal is to adapt it for adversarial robustness in few-shot scenarios, where only 1-16 samples per class are available from the downstream dataset. In such settings, learning adversarial robustness is particularly challenging due limited supervision. To address this, rather than relying on cost- intensive methods that fine-tune all parameters of large-scale VLMs, we adopt an efficient mask-tuning strategy. This approach keeps the pre-trained weights fixed while optimizing only binary masks over selected parameters, enabling the discovery of robust neural pathways.
2.2 MASK TUNING FOR ADVERSARIAL ROBUSTNESS
In this work, we propose AdvMask, novel adversarial mask-tuning approach enhance adversarial robustness. As illustrated in Fig. [] AdvMask builds on recent mask-tuning techniques (Zhao et al. 2020: ZZhng al. 2023), which identify sub-networks within pre-trained models for improved adaptation. Specifically, by optimizing binary masks on the vision encoder's pre-trained parameters, AdvMask deactivates adversarially vulnerable weights and identifies robust neural pathways that yield stable, resilient visual representations, enabling reliable predictions under attack. To elaborate on our method, we first detail how the mask parameters can be optimized efficiently, and subsequently extend this improve the adversarial robustness.
Mask Tuning. Given the pre-trained weights of the image encoder I(.), we first define a real-valued mask M of the same size. A binary mask Main is then obtained by thresholding with a:
Main I[M at
where TT-| is an indicator function used for binarization. We compute the masked weights 6' through an element-wise product (i.e., Hadamard product, @) as 0' = 0 C Main, and the encoder produces the visual representation 2, (z; 0') for input æ. However, direct optimization of the binary mask Main is infeasible due to non-differentiability of binarization function in Eq. 2 To overcome this, following previous works (Zhao et al. 2020 Lin et al.) 2020). we employ the Strighh-Thhouge Estimator (STE) (Bengio et al.) 2013)), allowing indirect updates to the real-valued mask M as:
ØL
M M 7 aMain
where y is the learning rate, and L is the objective function for mask tuning.
Adversarial Mask Tuning (AdvMask). Beyond the downstream adaptation, we extend the mask tuning to adversarial robustness by optimizing binary masks that selectively deactivate parameters vulnerable to perturbations, enabling stable predictions. Given clean samples x and adversarial counterparts & with labels y from few-shot dataset S, goal is tune the binary mask M
3

---

Published as a conference paper at ICLR 2026
class
Textual Prompt
class
photo of
a
[CLASS]
Text Encoder
class
Image Encoder
clean
Trans Transformer former Block
AdvMask
Transformer Block
Pelamn x
clean image
LAFA
P adv a (}))
adversarial image X
AdvMask Loss
Learnable Frozen
mask Lce + LLAFA
Pre-trained Weights
Learnable Mask (Mau))
Masked Weights
Figure 1: Overview of our AdyMask. Given clean and adversarial inputs, AdvMask learns binary masks Main (shown black-and-white grids) that selectively deactivate parameters vulnerable to adversarial perturbations. The masks enforce consistency between clcan and adversarial intermediate representations (Reeen >ad)) via our layer-wisc adaptive feature alignment loss (Clasa in See. 2.3), combined with adversarial cross-entropy loss (Ccr). This identifics robust neural pathways the vision encoder without modifying pre-trained weights.
(2)
while keeping the pre-trained weights 9 fixed. This adversarial tuning can be formulated as a min-man optimization problem, where the inner maximization step generates adversarial samples ã
max L (fecoMonn (ū, t), 2 Se
where 5 = j is the perturbation bounded by L. is the loss function (e.g., cross-entropy loss) for generating adversarial perturbations, and feo (-.t) represents the model output given the binary mask Main and textual prompts f. Subsequently, the outer minimization updates the mask parameters Mbin to minimize the adversarial loss using the generated adversarial examples as:
min (awwns [Lmask (x,, 3t L, Mbin, y)] Masn
By carefully designing the tuning objective (Cmusk) and learning the mask, our AdvMask identifies robust sub-networks without altering the pre-trained parameters. This enables significant robustness gains in few-shot scenarios while preserving the generalization capabilities of the fixed pre-trained model. In Sec. 2.3 we introduce our loss design tailored for challenging few-shot settings.
Key Advantages of our AdvMask. Our proposed AdvMask offers practical advantages for adversar- ial robustness. First, it adapts only a binary mask without altering pre-trained weights, preserving generalizable knowledge while regulating information flow produce stable and robust represen- tations on both clean and adversarial samples (further insights are provided in Sec. (3.)). Second, AdyMask effectively leverages limited few-shot data selectively activate deactivate crucial pathways, improving transferability to downstream tasks and significantly enhancing robustness against adversarial attacks. These advantages make AdyMask parameter-efficient solution for strengthening VLM robustness across diverse real-world scenarios.
2.3 LAYER- -AYER-WISE ADAPTIVE FEATURE ALIGNMENT (LAFA) LOss
Motivation. Our goal is to tune a binary mask that enhances adversarial robustness few-shot settings. Prior objectives for robustness (e.g., TeCoA (Mao et al. 2023)) mainly supervise the final
4

---

Published as a conference paper at ICLR 2026
output space (i.e., the joint text text-image embedding space), which limits their ability to enforce robust intermediate representations and provides insufficient learning signals under scarce data. By contrast, AdvMask adapts internal parameters of the image encoder, where robust intermediate features crucial. To this end, we propose a layer-wise feature alignment loss applied across encoder blocks, explicitly promoting stable representations and providing stronger guidance for few-shot mask tuning.
Loss Formulation. To explicitly guide robust feature representations, we propose a Layer-wise Adaptive Feature Alignment (LAFA) loss. This loss aligns adversarial features (from perturbed inputs) with clean features at each transformer layer of the image encoder, encouraging stable and robust intermediate representations. The intuition that small adversarial perturbations, though imperceptible at the input, can amplify through deeper layers. By learning binary masks that deactivate vulnerable parameters at each layer, AdvMask suppresses this propagation and promotes robustness. Since intermediate layers lack explicit label supervision, we leverage clean features as targets, aligning adversarial features to them during tuning. Formally, the loss is defined as:
(1) adv 112
Felean
(6)
L
IEL
(1) (1) where B is the sample batch, L the set of layers for alignment, and Selean and ady the output features at the (-th transformer layer of our masked image encoder for clean and adversarial samples, respectively.
LAFA Loss. To further elaborate the learning signals and ensure stable optimization in few-shot scenarios, propose Layer-wise Adaptive Feature Alignment (LAFA) loss with adaptive weighting scheme based on predictive reliability. The key idea is that if the model fails to correctly predict clean sample, its feature may serve as noisy alignment target, which is especially harmful under data scarcity as it can mislead mask optimization in unintended or sub-optimal directions. To mitigate this, weight each sample by its predictive reliability (i.e., confidence in the ground-truth class), enabling adversarial features to align more strongly with reliable clean features and less with unreliable ones. Formally, our LAFA loss is defined as:
p(y(2) Eslp(y' x²) + Felean
19
(1) "adv 11:
LAAF
LI-BB
IEL
where p(ylz) (r) is the masked model's confidence for the ground truth class y given the clean input [=']] + e) normalizes weights, with a small constant e
I, and the denominator (i.e., Eu'we (pv''//"(//]) for numerical stability. As a result, our LAFA loss prioritizes samples with clear and informative representations during alignment, further improving robustness, particularly under few-shot scenarios.
Final Objective. Our final tuning objective for AdvMask combines the cross-entropy (CE) loss on adversarial samples å with our proposed LAFA loss, optimizing a set of binary mask parameters without modifying the pre-traine pre-trained weights as:
Lmask = LCE(,, v) + A LLAFA ž. Z,
(8)
where is a coefficient balancing the two losses. This combination of objectives complements each other in tuning our adversarially robust mask: while the adversarial CE loss directly enhances prediction-level robustness, our LAFA loss ensures robust intermediate representations by explicitly guiding the learned binary mask to generate consistent features for both clean and adversarial samples. Consequently, our carefully designed loss function significantly improves adversarial robustness, especially in challenging few-shot adversarial tuning scenarios.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETTINGS
In this section, we conduct extensive experiments to demonstrate the effectiveness of our AdyMask approach for enhancing adversarial robustness. Basically, we follow the few-shot adversarial tuning setup from Zhou et al. (2024). Specifically, for adversarial tuning of the CLIP model with each baseline method, randomly sample 1, 2, 4, 8. and 16-shot samples per class from the training set of each downstream dataset. We then evaluate the tuned model on the test dataset by measuring classification accuracy (%, 1) clean samples and their adversarially perturbed samples separately.
5

---

Published as a conference paper at ICLR 2026
ABAD
Duep
9ro
PartedCCrs
23300
007001
Figure 2: Adversarial test accuracy (%, 1) over 11 datasets in few-shot settings. Results are averaged over 3 random trials (full results with standard deviations are provided in Sec. B.I of Appendix).
Datasets. Following previous studies on adversarial robustness of CLIP models, we evaluate our AdvMask method across various image classification datasets. Specifically, we consider general object datasets (ImageNet (Deng et al.) 2009). Caltech101 (Fei-Fei et al.) (0044)), a texture recognition dataset (DTD (Cimpoi et al. (014)), fine- gee-grained object datasets (FGrCAireraft (Maji et al., 2013 OxfordPets (Parkhi et al. 2012). Flowers 102 (Nilsback & Zisserman, 2008), 101 (Bossard et al. 2014). and StanfordCars (Krause et al., 2013)), a scene recognition dataset (SUN397 (Xiao al.
20100, an action recognition dataset (UCFIOT (Soomro al. 2012)), and satellite imagery dataset
(EuroSAT (Helber et al. 2019)).
Baselines. To validate the effectiveness of AdvMask in realistic scenarios requiring efficient tuning for large-scale VLMs, we mainly compare our method with parameter-efficient adversarial prompt- tuning methods widely used in prior robustness studies (Zhou et al. 2024] Mao et al.) 2023 ZZhang itt al. 2024a), as well as with TGA-ZSR (Yu et al.) 2024), which fully adapts model parameters in zero-shot robustness experiments. Our adversarial prompt-tuning baselines include adversarial visual prompt tuning (AdvVP) (Mao et al. 2023) with hand-craftes textual supervision, adversarial visual-text prompt tuning (Adv <dvLLP) with independently learnable prompts, adversarial multi-modal prompt learning (AdyMaPLe) (Khattak et al. 2023), and the recent few-shot adversarial prompt learning (FAP) (Zhou al.. 2024). We also report the compatibility of AdvMask with learnable prompt tuning method (CoOp (Zhou et al. 2022 in Appendix Sec. C.5 Further implementation details for all baselines are provided in Appendix Sec, A
Implementation Details. Following prior works on adversarial robustness of VLMs Zhou et al. 2024 Mao al., [023] Yu et al., 2024), we primarily
CLIP (Radford al., 2021) with VIT-B/32 image encoder (Dosovitskiy et al., 2020), tuning only binary mask parameters while keeping pre-trained weights frozen. We also report results using different encoder backbones (i.e.. ViT-B/16, ViT-L/14) in Sec. C.3 C.3.
another VLM (i.e., ViSualBERT (Li et al.) 2019)) in Sec. C.4] of Appendix. Following ZZeng al. (002)), we apply learnable binary masks only to multi-head self-attention layers, comprising about 20% of vision encoder parameters, with other mask-tuning settings consistent with them. Our LAFA loss is applied across all transformer layers with coefficient 50.0. Dataset-specific prompt templates (e.g.. "a photo of CLASS") are provided in Appendix Sec. A). All results are averaged over three random seeds. For adversarial training, we use PGD Madry al. 2017) under the Loe norm. During tuning, adversarial perturbations
generated with 2-step PGD (c = a = 1/255); at test time, robustness is evaluated with 100-step PGD. Here, adversarial perturbations computed by backpropagating through the masked model foo

---

Published as a conference paper at ICLR 2026
Mick LCF366
Ocfaccees
Figure 3: Clean test accuracy (%. 1) over 11 datasets in few-shot settings. Results are averaged over 3 random trials (full results with standard deviations provided in Sec. of Appendix).
ensuring a fully adaptive and fair evaluation setting. All baselines are implemented following their original settings. Additional hyperparameters and implementation details are given in Sec.
3.2 EXPERIMENTAL RESULTS
Few-shot Adversarial Robustness. In Fig. 2 we compare the adversarial robustness of our AdvMask method and baselines across 11 downstream datasets under the few-shot setting. As anticipated, the zero-shot CLIP model exhibits significant vulnerability to adversarial perturbations (average clean accuracy of approximately 61.9%. but adversarial accuracy drops drastically about 2.5%). In contrast, our AdvMask consistently achieves substantially higher adversarial robustness than prompt- based and adapter-based based baseline methods across most datasets, highlighting the effectiveness of our binary mask tuning strategy in selectively deactivating parameters vulnerable to adversarial attacks. Furthermore, AdvMask demonstrates robust performance across most few-shot settings, effectively enhancing adversarial robustness even with limited downstream data. This advantage stems from the fact that mask tuning efficiently explores task-specific pathways using few examples, while freezing pretrained weights preserves the strong generalizable representations of the base VLM. Additionally, our LAFA loss provides stable feature-level learning signals and mitigates noisy overfitting, making AdvMask particularly suitable for scarce-data scenarios. Consequently, method is practical not only from the perspective computational efficiency but also data efficiency, as it reliably tunes robust binary masks using only a small number samples.
Trade-off between Robustness and Transferability. Another important requirement for adversarial tuning is maintaining the original clean accuracy downstream tasks, shown in Fig. 3 With extremely limited samples (c.g., 1, 2, or 4-shots), we observe an inevitable drop in clean accuracy, similar other baseline methods, due overfitting to the limited adversarial examples. However, as the number of tuning samples increases, clean accuracy gradually recovers, and surprisingly, even exceeds the original CL.P performance certain cases (e.g., 16-shots in Caltech101) despite the absence of explicit supervised loss on clean samples during mask tuning. We hypothesize that, given a moderate number tuning samples (e.g.. 16-shots), our method learns a generalizable binary mask that provides a regularizing effect, facilitating effective adaptation to downstream task without overfitting. Furthermore, AdvMask generally achieves higher clean accuracy than baseline methods, effectively balancing the trade-off between adversarial robustness and transferability. Therefore, AdyMask offers practical benefits for reliable deployment of VLMs in real-world few-shot scenarios.
7

---

Published as a conference paper at ICLR 2026
Base-to-New Generalization Setting, In Ta-
ble II we present results under the base-to- new generalization setting, where classes split into disjoint "base" (training) and "new" (testing) groups in each dataset. In the experi- ments, models are adversarially tuned with 16- shot samples from base classes and then eval- uated on both groups. Even in this challeng- ing scenario, AdvMask consistently achieves superior adversarial robustness while main- taining competitive clean accuracy. On base classes, it significantly improves adversarial accuracy over FAP with comparable clean per- formance. On unseen new classes, AdvMask outperforms all baselines in adversarial robust- ness and achieves the highest clean accuracy (except zero-shot CLIP). These results confirm that AdvMask effectively identifies robust neural pathways by capturing inherent task-specific features from limited tuning samples, producing a robust and generalizable mask suitable for diverse test scenarios.
Base Class
New Class
Method
Clean
Adv.
Clean
Adv.
H
CLIP
66.9
3.4
71.5
3.8
6.9
Ad
31.7
14.4
30.4
13.4
19.2
AdvVLP
59.0
32.4
46.9
21.6
34.6
AdvMaPLe
60.4
30.7
46.2
20.3
33.3
FAP
70.5
38.0
49.6
21.9
37.6
AdyMask
69.5
43.6
50.2
26.1
41.9
Table I: Results on adversarial base-to-new generalization settings. For both class groups (base, new), we report the average clean and adversarial accuracy across 11 datasets, and the harmonic mean (H) of these four accuracy scores. (Detailed results are provided Sec. B.2 of Appendix.)
Generalization Capability. While our primary goal is to enhance robustness in few-shot adaptation scenarios, we em- phasize that the learned mask also gen- eralizes well unseen datasets. To evaluate this, we follow the setup of
Method
Dataset
Clean Acc. (%)
Adv. Acc. (%)
CLIP
61.9 (110.0)
2.7 (1.0.))
TGA Entire (100%)
38.6 (=1.0)
22.9 (+:0.)
FAP
16-shot (3.2%)
36.0 (110.)
16.8 (1.0.)
TGA-ZSR
16-shot (3.2%)
41.3 (1110)
13.0 (1003)
TGA-ZSR (Yu et al., 2024),, a recently proposed zero-shot robustness method. Specifically, we train AdvMask on a held-out source dataset (i.e., Tinylma- geNet) and directly testing on unseen tar- get datasets without further tuning (Table 2). The results show that, in the 16-shot setting, AdvMask achieves superior clean and adversarial accuracy compared baselines, demon- strating strong generalization capability in shw-shot scenarios. Furthermore, despite using only 3.2% of the source data (i.e.. 16 shots), AdvMask approaches the performance of TGA-ZSR, even though TGA-7SR requires full access to the entire source dataset. These results suggest that AdvMask selectively deactivates parameters that are globally vulnerable to adversarial perturbations, rather than overfitting to dataset-specifie patterns. In other words, certain parameters consistently amplify adversarial noise across tasks, and suppressing them yields more stable intermediate representations. This intuition is further supported by results from the base-to-new generalization setting with disjoint class groups. Consequently, AdvMask not only improves robustness on the tuned dataset but also produces transferable masks effective for unseen domains, making it well-suited for robust and reliable deployment in real-world applications.
AdvMask 16-shot (3.2%)
42.0 (#0.3)
19.4 (+:022
Table Results on zero-shot robustness. Following Yu et al. (0024). models are adapted using TinylmageNet (entire training set for TTA--SRR 16-shots for others) and evaluated on unseen downstream datasets. Results averaged over 3 trials.
3.3 FURTHER STUDIES ON ADVMASK
In this section, we provide ablation studies and additional analyses of AdvMask. We report 16- shot results averaged over 3 random trials on 5 datasets (i.e., Caltech101, DTD, FGVCAircraft, Flowers102, UCF101) from diverse categories. Due to page limits, extended results are provided in the Appendix (Sec. & Sec. D), including ablations on key design choices (e.g.. LAFA loss coefficient, layer positions, adaptive weighting scheme, mask threshold ox), robustness under different perturbation bounds and attack type, evaluations on alternative architectures and VLMMs, as well as complementary analyses on compatibility with learnable prompt tuning and computational efficiency.
Which Layers are Effective for Adversarial Masking? Since tuning masks for all parameters within CLIP incurs substantial computational costs, we specifically focus on optimizing masks for the multi-head attention (MHSA) layers within the transformer blocks of the image encoder, following prior work (Zheng al. 2023). Our choice is motivated by the well-established observa- tion that these self-attention layers generate context-aware representations by capturing long-range dependencies across input tokens (i.e., image patches), making them particularly vulnerable to ad-
8

---

Published as a conference paper at ICLR 2026
versarial perturbations in the input space. Therefore, selectively masking noise-sensitive parameters
within these self-attention layers proves highly effective as shown in Table 3 Also, as these layers comprise only about 20% of the total parameters (sig- nificantly fewer than the MI.P layers), our approach significantly reduces computational costs, including memory usage and training time. Consequently, our AdvMask offers practical and effective strategy for parameter-efficient tuning of large-scale VLMs, making highly suitable for diverse real-world applications.
Module
Clean Acc. (8)
Adv. Acc. (%)
MLP only
65.73 + 0.45
45,95 + 0.07
MHSA only
67.34 0.19
47.13 + 0.25
MIIA MLP
66.01 0.28
47.20 1. 0.25
Table Ablation on adversarial masking layers.
Loss Ablation Study. In Table we present ablations on the loss functions. Our design combines adversarial cross-entropy (dv) adv) with the LAFAA loss to enforce feature-level consistency between clean and adversarial samples. To validate this choice, we compare LAFA with several alternative auxiliary losses (e.g., Gensen-Shannor divergence, KI. divergence). The results show that LAFA consistently outperforms these alternatives in both clean accuracy and adversarial robustness, with the performance gap becoming more pronounced low-shot settings (e.g.. I-shot and 4-shot). This is attributed to the fact that LAFA's Ceature-alignmert objective provides stronger and more stable learning signal than distributional divergence terms defined in the output space, allowing the model maintain coherent and robust representations even when data is scarce.
Moreover, the adaptive weighting scheme (Sec. 2.3) further improves performance by emphasizing reliable samples during training. This mechanism mitigates over- fitting to noisy or misclassified examples, and its benefits are particularly apparent low-shot scenarios (see Appendix Sec. Sec.[D.2 for details). Overall, these findings con- firm that our loss formulation enhances representational robustness and enables AdvMask to reliably identify robust neu- ral pathways, even under challenging few- shot conditions.
Loss Function
Clean
Adv
Clean
Adv
Clean
Adv
CLIP
||
56.6
4.8
56.6
4.8
56.6
4.8
Lenste
40.3
15.6
55.2
30.6
65.8
46.4
Cis
42.9
17.3
54.7
30.8
65.9
46.5
Exi.
31.8
13.8
46.2
27.9
60.7
43.6
CLai
44.5
17.8
56.6
32.1
66.9
46.8
(=1.5)
(::0.97)
(=0068
(#00664
(::044)
(#:44)
Clasa
46.6
18.4
57.2
32.2
67.3
47.1
(=1111
(4-045)
(=+0.34)
(#-0.28)
(4-0119
(4:0255
Table 4: Loss ablation with alternative auxiliary losses. CLAFA represents LAFA loss without adaptive weighting scheme.
Further Insight into Robust Neural Pathway. neural pathway refers propagation path within a pre-trained network that forms new functional conjunctions between neurons (or learned knowledge) (Zheng et al. (023). Un- der the lottery ticket hypothesis, Malach al. (2000) showed that mask optimization within overparameterized networks can achieve perfor- mance comparable to full weight optimization. Motivated by these works, we explore whether mask-based optimization can uncover pathways that remain robust under adversarial attack. In- tuitively, small adversarial perturbations, though imperceptible in the input space, can amplify through deeper layers of encoder, leading to incorrect predictions. Our binary masks mitigate this by deactivating vulnerable weights, suppressing noise propagation and preserving stable intermediate representations. To support this, analyze layer-wise representations using CKA similarity (Fig. 4. well-known metric for representation consistency (Kornblith et al. 2019). We measure how similar clean and adversarial features (CLS tokens) remain across layers, with and without AdvMask. It shows that pre-trained CLIP shows high similarity in early layers (L.1-L4) but declines in deeper ones as adversarial noise amplifies. In contrast, AdvMask preserves consistently higher similarity across all layers, effectively suppressing noise amplification and stabilizing representations. This provides strong evidence that AdvMask identifies robust neural pathways that enhance adversarial resilience.
03
99999
UVKY (alA D350)
Laptridd (002
(a) Pre-trained CLIP
(b) CLIP AdvMask
Figure 4: Layer-wise CKA similarity between clean and adversarial features on DTD dataset, propagated from LI to L12 (i.e., top-left to bottom-right).

---

Published as a conference paper at ICLR 2026
4 RELATED WORKS
Parameter-Ffficien Adaptation Methods for VLMs. Vasion-language models (VLMs) such as CLIP (Radford al., 2021) show strong transferability across diverse tasks (Zhang et al. 2024b), but their scale makes full fine-tuning impractical. This has motivated parameter-efficieni approaches, including text, visual, and joint prompt tuning (Zhou al., 2022 Bahng et al. 2022 Khattak et al. 2023), as well as adapter methods (Zhang et 2022bb Gao et al. 2024). Recently, mask tuning (Zheng al. 2023) has been proposed to identify task-speeific subnetworks within pre-trained VLMs. However, they mainly target downstream accuracy, leaving adversarial robustness unexplored. In contrast, we develop mask tuning explicitly for robustness by uncovering robust neural pathways.
Adversarial Robustness for VLMs. Despite their generalization, VLMs are highly vulnerable to adversarial attacks (Cui et al. 2024; Budathoki Dhakal 2025), limiting real-world deployment.
Prior efforts include adversarial prompt tuning (Zhou et al. 2024; Mao al. 2023), which improves robustness but ignores the encoder's intrinsic structure. Fully iin-tuning with adversarial training (Bai et al. 2021) is effective but costly and prone to overfitting in few-shot settings. Zero-shot robustness methods (Yu et al., 2024) rely on held-out datasets but often fail under distribution shifts. We instead focus on parameter-efficient adversarial tuning in few-shot scenarios, achieving robustness gains with minimal data while preserving pre-traine pre-trained weights.
Neural Pathways Searching. Deep networks distribute knowledge across neurons, dynamically forming task-specific pathways (Liu et al. 2018 Zhao et al.) 2020). Building on this perspective. binary mask tuning has been explored
means isolate subnetworks for task adaptation (Wortsman et al. 2020| Csordás et al., 2020) or for addressing OOD generalization (Zhang et al. (021). Recent work Zheng et al.) 2023) further demonstrated that mask tuning can reveal latent knowledge within pretrained VLMs, though robustness under adversarial perturbations remains unaddressed. Our work extends this line of research by introducing adversarial mask tuning to deactivate noise-sensitive parameters, thereby constructing robust neural pathways and substantially improving VLM robustness. Related efforts in adversarial learning have explored cooustness-sensitii structures from different angles. Adversarial pruning (AP) methods (Piras al. 2025 Sehwag et al.) 2020 Chen al. 2022) aim to obtain sparse yet robust models by pruning and retraining weights, which contrasts with goal of robust few-shot adaptation of pretrained VLMs without modifying any weights. Also, Zhu et al. 2023) similarly analyzes robustness-crite
components but focuses on improving the generalization of adversarially trained models through fine-tuning, whereas AdvMask learns binary masks on pretrained VLMs (without adversarial pretraining or weight updates) to achieve robust adaptation in few-shot scenarios.
CONCLUSION
In this paper, we introduced AdvMask, a framework that uncovers robust neural pathways in VLMs for few-shot adaptation. By introducing the LAFA loss to adaptively align clean and adversarial features, AdvMask selectively emphasizes robust representations through binary masks, enabling efficient and reliable adaptation without altering pre-trained weights. Extensive experiments confirm AdvMask's effectiveness over prior adversarial tuning methods, offering a new perspective that robust subnetworks inherently exist within large VL.Ms. These findings highlight path toward more resilient and efffieter-efficient deployment of robust models in real-world applications.
REPRODUCIBILITY
To ensure reproducibility, we provide our implementation code in ltttps://github com/ wonjeongchoii hwtphchhhoi//aMaaak..... AdvMask and the supplementary materials. Further experimental settings and detailed configurations, including computing resources, are described in Sec. 3.1 of the main paper and Sec. A.3 of Appendix.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea (NRF) through grants from the Korea government (MSIT) (No. RS-2024-004088003, RSS2024-00340060 and by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by MSIT of Korea (No. RS-2002-000444862. Also, it was partly supported by the Technology development Dong-Jun Han is the corresponding author.
Program of MSS (No. 005-2025-255446
REFERENCES
Firoj Alam, Ferda Ofli, and Muhammad Imran. Crisismmd: Multimodal twitter datasets from natural disasters. In Proceedings of the international AAAI conference on web and social media, volume 12, 2018.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting scage models. arXiv preprint arXiv:2203.17277 2022.
Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01335 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 308.3438, 2013.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-minn 1ead-100-mmiigg discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp- 446-461 Springer, 2014.
Varun H Buch, Irfan Ahmed, and Mahiben Maruthappu. Artificial intelligence in medicine: current trends and future possibilities. British Journal of General Practice, 68(668): 2018.
Anjila Budathoki and Manish Dhakal. Adversarial robustness analysis of vision-language models in medical image segmentation. arXiv preprint arXiv:2505.02971, 2025.
Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. Sparsity winning twice: Better robust generalization from more efficient training. arXiv preprint arXiv:2202.09844 202.0994.. 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fres attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. arXiv preprint arXiv.2010.02066. 2020.
Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19625-24634 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Yao Deng, Xi Zheng, Tianyi Zhang, Chen Chen, Guannan Lou, and Miryung Kim. An analysis of adversarial attacks and defenses on autonomous driving models. In 2020 IEEE international conference on pervasive computing and communications (PerCom), pp. 1-10. IEEE, 2020.

---

Published as a conference paper at ICLR 2026
Junhao Dong, Yuan Wang. Jian-Huang Lai, and Xiaohua Xie. Improving adversarially robust few- shot image classification with generalizable representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9025-9034, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples; An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac Kohane. Adversarial attacks medical machine learning. Science, 363(64333 2019.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better rision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-595 2024.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19113-19122, 2023.
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519-3529. PMIR, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 551-561, 2013.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. arXiv preprint arXiv:2004. 009994004.09984 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arxiv:/908.03:3 2019.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint arXiv:2006.07255: 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706. 06083, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. International Conference on Machine Learning, pp. 6682-6691. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2026
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In International Conference on Learning Represen- tations (ICLR), 2023. URL httpttttt//eeeeeeeeae otgpr//eeeeeaaetaaoooooorrrrrrra
net htt.e/oommmmmiiaaatttotoo
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and patlern recognition, pp. 3498-3505. IEEE, 2012.
Cyril Picard, Kristen M Edwards, Anna C Doris, Brandon Man, Giorgio Giannone, Md Ferdous Alam, and Faez Ahmed. From concept to manufacturing: Evaluating vision-language models for engineering design. arXiv preprint arXiv:2311.12663 2023.
Giorgio Piras, Maura Pintor, Ambra Demontis, Battista Biggio, Giorgio Giacinto, and Fabio Roli. Adversarial pruning: survey and benchmark of pruning methods for adversarial robustness. Pattern Recognition, pp. 11 111788, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint ar arXiv:240..12336 2024a.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML, 2024b.
Vikash Sehwag. Shiqi Wang, Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33: 33:19655-19666 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0212.0442 2012.
Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito, and James Kapinski. Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1555-1562. IEEE, 2018.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel M Ni. Generalizing from a few examples: A survey few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184 2020.
Jianxiong Xiao, James Hays, Krista
Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Lu Yu, Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness in vision-language models. arXiv preprint arXiv:2410.21802, 2024.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork generalization? In International conference on machine
structure be the key learning, pp. 12356-1222 12367. PMLR, 2021.
Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack lasion-language pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5005-5013, 2022a.
13

---

Published as a conference paper at ICLR 2026
Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, and Jitao Sang. Adversarial prompt tuning for lision-language models. In European Conference on Computer Vision, pp. 56-72. Springer. 2024a.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pultern Analysis and Machine Intelligence, 2024b.
Renrui Zhang, Wei Zhang, Rongyao Fang. Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493-510. Springer, 2022b.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alternative to finetuning for pretrained language models. arXiv preprint 1rXiv:2004.12466, 2020.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang. Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models, Advances in Neural Information Processing Systems, 36:5411 1144111-54138, 2023.
Kecheng Zheng, Wei Wu, Ruili Feng, Kai Zhu, Jiawei Liu, Deli Zhao, Zheng-Jun Zha, Wei Chen, and Yujun Shen. Regularized mask tuning: Uncovering hidden knowledge pre-trained vision- language models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11663-11673, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 13009))2337-2348, 2022.
Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems, 37: 3122-3156, 2024.
Kaijie Zhu, Xixu Hu, Jindong Wang, Xing Xie, and Ge Yang. Improving generalization adversarial training via robust critical fine-tuning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4424 4434, 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXPERIMENTAL DETAILS
A.I DATASETS
Our fho- experiments are conducted on 11 public datasets for image classification tasks, fol- lowing Zhou al. (2022) 2024). For experiments on zero-shot adversarial robustness, we adopt TinyImageNet (Le & Yang 2015) as the source dataset for tuning and subsequently evaluate the tuned model on the other downstream datasets, following Yu et al. (2024). To facilitate understand- ing, in Table [ we summarize the statistics of datasets used in experiments. Additionally, for baselines requiring static textual prompts (e.g.. "a photo of (CLASS)" ") for advensarial tuning such zero-shot CLIPP, Ad,VP, and AdyMask, specify hand-crafted text prompt templates for We note that these templates are also used as initial prompts in learnable text
each dataset in Table [6] prompt tuning methods, including Adv AdvMaPLe, and FAP.
Dataset
#Classes
Train Size
Test Size
Task
ImageNet
1,000
1.28M
50,000
Object recognition
TinyImageNet
200
0.1M
10,000
Object recognition
Caltech101
100
4,128
2,465
Object recognition
DTD
47
2,820
1,692
Texture recognition
AuroSAT
10
13,500
8,100
Satellite image recognition
FGVCAircraft
100
3,334
3,333
Fine-grained aireraft recognition
Flowers
102
4.093
2,463
Fine-grained flowers recognition
Food101
101
50,500
30,300
Fine-grained food recognition
OxfordPets
37
2,944
3,669
Fine-graiee Fine-grained pets recognition
StanfordCars
196
6,509
8,041
Fine-graines
recognition
SUN397
397
15,880
19,850
Scene recognition
UCF101
101
7,639
3,783
Action recognition
Table Summary of datasets, including number of classes, training/tosting sizes, and task types.
Dataset
Text Template
ImageNet
"a photo of a {CLASS}.
TinylmageNet
"a photo of à {CLASS)
Caltech101
phote of {CLASS).
DTD
{CLASS) texture."
EuroSAT
a centered satellite phote of [CLASS]."
OxfordPets
"a photo of a [CLASS], type of pet."
FGVCAircraft
a phote of (CLASS), type of aircraft."
Food101
photo of (CLASS), type of food."
Flowers102
a phote of a (CLASS), type flower."
StanfordCars
" photo of a (CLASS)
SUN397
a phote of a (CLASS).
UCF101
phote of person doing (CLASS)."
Table 6: Hand-crafted text templates across different datasets.
A.2 IMPLEMENTATION DETAILS FOR BASELINE METHODS
Adversarial Prompt Tuning, Similar to (Zhou et al. 2024), we implement adversarial prompt-based baselines, strictly following the original architectural and parameter settings for fair comparison. Specifically, adversarial visual prompts (AdvVP) adopt token-level prompt of size 5 and 30-pixel padding around the image, optimized for 10 epochs using SGD with a cosine learning rate scheduler (initial learning rate: 40), following the setup of (Mao al. 2023). Adversarial multi-modal prompts (AdvMaPLe) employ token-level prompts of size 2 in both text and visual branches for the first transformer layers, coupled with text-to-image projections. Adversarial vasion-language prompts
15

---

Published as a conference paper at ICLR 2026
(AdvVLP) an identical structure but adapt vision and language prompts independently. Both AdyMaPLe and Adv are trained for 10 epochs with SGD and cosine scheduler (initial learning rate: 0.0035). For consisteney,
replace the original baseline loss functions with the adversarial TeCoA (Mao et al. 2023) loss during training and evaluation for Adv AdyMaPLe, and Adv VL.. For the state-of-the-an method of few-shot adversarial prompt tuning (FAP) (Zhou et al. (024), train the model for 10 epochs by using SGD with a momentum of 0.9 and a cosine scheduler (initial learning rate: 0,0035) with a warm-up strategy during the first epoch. Also, we use token prompts of size 2 in both branches for the first 9 layers, following the configurations in original paper.
A.3 ADDITIONAL IMPLEMENTATION DETAILS
For our AdvMask, all elements of mask parameters are initialized with 10-2 and the threshold a (in Eqn. (2)) is set to 5 x 10 following Zheng et al. (023). Regarding the optimization setup, we train the (binary) mask parameters using a SGD optimizer with a momentum of 0.9 and cosine scheduler with a warm-up strategy during the first epoch, following the setup of (Zhou et al. (024). For most of the datasets, models are trained for 10 epochs with the initial learning rate of 0.01. For ImageNet, Food101, and SUN397 datasets, considering large number of classes and data volumes, we use the learning rate of 0.0035 and maximum epochs of 10, except for ImageNet with 5 epochs. In the experiments zero-shot adversarial robustness, all models configurations with a few-shot settings. We conduct all experiments in an environment with Py PyTorch 1.12.1 and CUDA 1 11.3 Python 3.8 under single NVIDIA RTX 3090 GPU (24GB) device.
trained for 5 epochs with the same
B COMPREHENSIVE RESULTS
B.1
RESULTS UNDER FEW-SHOT SETTINGS
Shots
ImageNet
Callech
DID
EuroSAT
FGVC
Food101
Flowers
Pets
Cars
SUN397
UCF101
Avg.
$1.53
86.97
33.17
25.07
11.33
$4.30
53,37
73.97
38.57
52.63
48.40
48.12
(20.46)
(#0.54)
(E0074)
(1:403)
(=2335)
(=1113)
(d:3115)
(1:0.55
(2:082)
(::0611)
(1:2.24)
(20.86)
51,70
87.77
38.70
28.17
14.53
58.83
60,57
78.23
40.23
$4.30
$3.30
$1.48
(::0.70)
(4:0.77
(d:1.36)
(4:0.71)
(=0.24
(=1.27)
(4:0.0)
(#:0.84)
(1:0.5)
(#:0.99)
(4:0.67)
(#0024)
$2.20
90.10
45,10
31.07
17.53
61.90
73.87
80.40
44.23
55.63
$9.50
55.59
(1:044)
(#:0.16
(#:1224
(4:2.21)
(=0.26)
(=0080)
(::0.59)
(d:1.40)
(d:1.7
(d:0.74)
(4:044)
(#0042)
$3.03
91.93
49.83
37.27
21.23
62.73
82.60
81.83
49.33
58.37
65.33
59.41
8
(#:061)
(#:0.34)
(4:0.09)
(#:2.09)
(=0.59)
(=0.33)
(d:1.31)
(d:0.73)
(::1)5
(::0.05)
(#:0.12)
(#0.17)
$2.13
92.87
58.43
47.53
26.80
64.27
88.03
82.63
57.40
59.93
70.57
63.69
16
(1:065)
(40005)
(4:0.1)
(d:2.22)
(= 1.50)
(=0.46)
(::0.61)
(1:0.21)
(4:0.4)
(1:0229)
(1:0.86)
(2:0233
Table 7: Clean test accuracy (%, 1) of our AdvMask over 11 datasets in few-shot settings. Results are averaged
over 3 random trials.
In Table 7 and Table g we present the complete few-shot results of our AdvMask on clean and adversarial samples across 1l datasets, respectively. These results align with the main findings in Fig. 2 and Fig. 3 of our main paper, demonstrating that AdvMask achieves superior adversarial robustness compared to baseline methods while effectively balaneing the trade-off between robustness and transferability. Additionally, our method exhibits low standard deviations (on average lower than 1.0%) across datasets, highlighting the stability and effectiveness of AdvMask in identifying robust neural pathways, even under challenging few-shot scenarios.
B.2 RESULTS UNDER BASE-TO-NEW GENERALIZATION SETTINGS
In Table we provide the complete results of our AdvMask method in base-to-new generalization settings across 1I datasets, consistent with Table I in the main paper. Even in this challenging scenario, where the generalization capability of the adapted model is important, our AdvMask still achieves competitive performance by effectively capturing inherent task- features from limited samples. This demonstrates our AdyMask's 's strong generalization capability for large-scale test datasets.
16

---

Published as a conference paper at ICLR 2026
Shots
ImageNet
Caltech
DTD
EuroSAT
FGVC
Food101
Flowers
Pets
Cars
SUN397
UCF101
Avg.
14.27
47.60
9.93
5.57
1,93
7.03
20.87
9.40
6,40
15.80
11.53
13:67
(1:0333
(d:0.86)
(1:0.50)
(1:4.76)
(20078)
(:10:74)
(1:0.25)
(1:0.85)
(1:0.16
(1:0.24)
(1:2445)
(10.51)
16.95
56.93
15,93
12,90
3,53
12,90
27.17
18.43
8.70
19.80
19.50
19.34
(2:0.25)
(d:1.72)
(d:1008)
(8:2.18)
(220229
(=:0.29)
(1:0.22)
(2:1.23)
(1:0.16
(d:0.24)
(111336
(10.25)
19,30
65.30
21.63
20.50
5.57
17.50
41.43
28,37
12.23
23,80
27.03
25,70
(1:0.24)
(1:0.00)
(1:1.1)
(1:1.84)
(200.4)
(=00.22)
(1:0.69)
(d:0:74)
(d:0.11
(d:0.36)
(1:0.34)
(40.31)
20.90
71.13
27.70
27.90
7.47
23:40
57.37
38,23
18.23
28.23
34.13
32.24
8
(2:0.50)
(4:0.73)
(4:0.57)
(4:0.54)
(=0012)
(=00114
(4:0.26)
(1:0.26)
(4:082)
(d:0.25)
(4:0.53)
(4:0.14
23.77
75.83
35,47
32.73
12.03
28.00
69.90
44.73
27.37
31.00
42.43
38.48
16
(1:0.56)
(4:0221)
(+1223
(1:0.74)
(=0049)
(=0066)
(::0.29)
(1:0.21)
(2:0.12)
(::0.14)
(1:0.29)
(20.22)
Table 8: Adversarial test accuracy (%, 1) of our AdvMask over 11 datasets in few-shot settings. Results are averaged over 3 random trials.
Class
Type
ImageNet
Caltech
DTD
EuroSAT
FGVC
Food101
Flowers
Pets
Cars
SUN397
UCFI01
Avg-
56.53
95.73
70.07
66.63
25.23
69.97
91.10
87.27
57.77
68.43
75.47
69.47
Clean
(=0.37)
(=0.50)
(#1.33)
(#2234)
(+0.87)
(:0.21)
(#0.37)
(1:033)
(#:042)
(#:045)
(+144
(+00779
Base
81.07
41.40
55.40
11.07
30.60
75,33
25.43
36.80
46.27
43.60
Adv.
(=0.57)
(=0.47)
(=1559)
(#2234)
(+0.52)
(4:0.22)
(::046)
(4:0.84)
(4:0.53)
(4:0.29)
(#0029)
(+:0.74)
47,80
84.47
45.27
31.17
13:20
63.10
41,53
84,03
34,70
59.20
47,23
50.15
Clean
(=0.80)
(#: 1.07)
(4:0.0)
(1:3227
(#0.08)
(42220)
(111077
(1:0.87
(1:0.08)
(+:0778
(20068)
(+ 1.00)
New
21.50
61,33
22,73
22.00
4.27
25.53
19.47
47.90
11.33
29.57
20.93
26.11
Adv.
(=00088
(=044
(#1)37)
(#:2.27)
(20048)
(+0.73)
(40083)
(:1770)
(+:0.34
(4:0.66
(+0662)
(+0087)
Table 9: Results on adversarial base-to-new generalization settings. For both class groups (base, new), we report the clean and adversarial accuracy (mean standard deviation) across datasets. Models are tuned using 16-shot samples from the base class group.
B.3 RESULTS UNDER ZERO-SHOT ROBUSTNESS SETTINGS
In Table 10 and Table II we provide zero-shot results on clean and adversarial samples across downstream datasets. As described in Table 2 of our main paper, we first adversarially tune the model using TinylmageNet as the source dataset and subsequently evaluate the tuned model on 10 downstream datasets. For TGA-ZSR (Yu al. 2024). state-of-the-art zero-shot adversarial robustness method, we use the entire source training set (100%), while other methods utilize only 16-shot samples (3.2%) for tuning. Notably, even with significantly fewer samples, our AdvMask achieves competitive zero-shot performance on both clean and adversarial samples. Specifically, for the source dataset (i.e., TinyImageNet), our accuracy scores are inevitably lower than TGA- ZSR due to fewer training samples. However, on downstream datasets, our AdvMask attains better clean accuracy and only slightly lower adversarial accuracy (approximately 2.9% lower on average) compared to TGA-ZSR, despite using only 3.2% of the source data. Furthermore. AdvMask significantly outperforms FAP (Zhou et al., 2024) in downstream tasks, highlighting its superior zero-shot generalization from limited tuning samples. Additionally, unlike TGA-ZSR's resource- intensive full-paramedd adaptation, AdyMask optimizes only binary masks, considerably enhancing efficiency in terms of memory usage and training latency. Therefore, method is practical and effective for both few-shot and zero-shot scenarios, enabling robust and reliable deployment of VLMs in real-world applications.
C ADDITIONAL RESULTS
C.1 ROBUSTNESS UNDER DIFFERENT PERTURBATION BOUNDS
For comprehensive evaluation, in Fig. 5 we provide additional results under varying perturbation bounds (i.e., c in Eqn. (5) of the main paper). We compare AdvMask with other promising methods (i.e., zero-shot CLIP, FAP) in the few-shot scenario using 16-shot samples. From the results, we observe that even as stronger adversarial attacks occur with increased perturbation bounds, our AdyMask consistently achieves competitive adversarial robustness. Although FAP achieves
17

---

Published as a conference paper at ICLR 2026
Source
Downstream Datasets
Method
Dataset
T--mmNee 1a DTD
FarSSAT FGVC Food101 Flowers
Pets
Cars
SUN397 UCP101
Avg.
CLIP
61.20
91.20
43.60
45.20
19:10
80.50
67.00
87.50
60.40
62.00
62.00
61.85
TGA-ZSR Entire (100%)
79.83
85.60
23,87
17.23
6.27
40.07
33.03
61.63
26.30
44,20
48.13
38.63
(1:0.74)
(::0.93)
(2: (..1.44)
(41.25)
(40042)
(10.78)
(2:0.61)
(:
(4:1.67)
(4:1.16
(#11.00)
53,37
81.60
19.63
18.63
5.63
34.70
33.37
66.90
22.30
37,90
39.23
35.99
FAP
16-shot (3.2%)
(#:0.99
(::0.78)
(#:2.50)
(#1.22)
(4:0.59
(d:1.44
(=12
(::33.8))))
(#:2.12)
(d:1.84 (::0.89)
67.03
81.53
24.23
21.03
8.23
50.37
32,90
60.20
34.70
47.10
52.50
41.28
TGA-ZSR 16osho (3.2%)
(+00441)
(#:1.85)
(#: 1,92)
(#0087)
(#:0.48
(::1.37)
(=:1.85
(#:2249))
(4:1.40)
(2:0.99)
(4:0.50)
(#:1.04)
59.07
84,47
28.27
21.90
9.13
41.33
40.47
69.
33.53
45.03
45.97
41.99
AdvMask
16.shot (3.2%)
(2:0377)
(::0.21)
(=7778
(#2.41)
(#:022)
(#:143)
(=0.33)
(2)) (2:0.31) (#:045) (::0.34)
Table 10: Results on zero-shot clean accuracy. All models are tuned using TinyImageNet as the source dataset (TGA-7SR (Yu et al. (024) uses the full training set, whereas other methods use 16-shot samples (3.2%) from the source dataset, not from downstream datasets). After tuning, models are evaluated zero-shot 10 unseen downstream datasets. The average accuracy in the last column is computed over the 10 datasets across trials.
Source
Downstream Dalasels
Method
Dataset
T-ImgNNet Caltech101
DTD FarSSAT FGVC Food10l Flowers
Pets
Cars
SUN397 UCF101
Avg.
0.20
16.63
4.93
0.50
0.97
0.10
1.00
2.66
TGA-7SR Entire (100%)
$2.87
67.73
15.70
11.33
3.10
17.03
18.43
36.00
12.23
20.77
26.63
22.90
(1:0.88)
(1:0.76)
(21131)
(10.12)
(10.45)
(1:0.54)
(2.0.78
(1:0.37)
(1:0.65)
(1:0.3)
(1:0.66)
(1:0.51)
18.63
55.77
11.53
10.30
1.70
9.80
15.27
30.50
5.77
12.20
15.10
16.79
FAP
16.shot (3.2%)
(::0699)
(#:1.8)
(=1165)
(#1.02)
(#0.36)
(d:0.16)
(2:1.6)
(1:0.60))))
(2:0.91)
(d:0,71)
(4:0667)
15,90
47,87
9.07
7.73
1.83
7.83
11.13
17.37
5.10
13:10
12,98
TGA-7SR 16-shot (3.2%)
(::0433
(::0.98
(=0046)
(20097)
(20039)
(::0.55
(=11.
(::11))) (2:0022)
(2:0.12)
(#:0.43)
(#:0.33)
AdvMask 16.shot (3.2%)
26.23
61,27
16.10
5.70
1.87
12.93
19.43
32.37
8.17
16.40
19.33
19.36
(2:0229)
(+:053)
(=0.57
(22.00)
(#0.17)
(2:0.93)
(=0.45)
(:10 66) (1:0. 12) (2.0.28)
(#:0.77
(5:0.25)
Table 11: Results on zero-shot adversarial accuracy. All models are tuned using TinyImageNet as the source dataset (TGA-7SR (Yu al. 2024) uses the full training set, whereas other methods use 16-shot samples (3.2%) from the source dataset, not from downstream datasets). After tuning, models are evaluated zero-shot on 10
unseen downstream datasets. The average accuracy in the last column is computed over the 10 datasets across
trials.
the highest clean accuracy due to their explicit supervision on clean samples during training, its adversarial robustness notably deteriorates under larger perturbations, whereas AdvMask remains robust against stronger attacks. These results confirm that AdvMask's robustness gains persist even under stronger attacks (e.g.. e 4/255), indicating that the binary mask and straight-through estimator do not obscure gradients. Therefore, our approach represents an effective and practical solution for deployment in reliable systems where resistance to dynamic adversarial attacks is crucial.
Clean Accuracy
Adversarial Accuracy
40
35
30
Actunacy (4)
40-
25
30-
CUP
1,55
8 2055
<= 4255
1055
2=20
4255
Perturbation Bound
Perturbation Bound
Figure 5: Results under varying perturbation bounds (i.e., c) in the few-shot scenario using 16-shot samples. We report the average clean and adversarial accuracy across 10 datasets over 3 trials.
18

---

Published as a conference paper at ICLR 2026
C.2 ROBUSTNESS UNDER DIFFERENT ATTACK TYPE
In Fig. 6 we conduct additional experiments to evaluate the adversarial robustness of our AdvMask under different attack type. Specifically, we apply AutoAttack (Croce & Hein 2020), a stronger and user-independent attack strategy designed to overcome limitations (e.g., sub-optimal step sizes) of previous base- attacks. Following Zhou et al. (2024), consider (i.e., APGD-CE and APGD-DLR) and compare our AdvMask with FAP, since other methods exhibit near-zero accuracy due the stronger attack. Experiments are performed in 16-shot scenario with perturbation bound of 1 1/255 across 5 datasets. The results demonstrate that while FAP improves adversarial robustness over the zero-shot CLIP model, our AdvMask consistently outperforms this baseline, confirming the effectiveness of our robust mask-tuning approach under stronger attacks.
variants of APGD
Additionally, we evaluate robustness against text-level and joint image-text-leve adversarial attacks in Table T2] Specifically, by using the masks trained with 16-shot downstream samples, we assess whether the learned masks (although trained only with image-level adversarial supervision) can still provide robustness when different modalities are attacked. We consider two additional evaluation settings beyond standard image-level PGD attacks: (1) Independent multimodal attacks, where PGD is applied to the image while BERT-Ataac [i et al. (2020) perturbs the text prompts; (2) Joint multimodal attacks, following CoAttack Zhang et al. (202a)), where image and text embeddings perturbed in coordinated manner within the shared multimodal space. As shown in Table [2] performance decreases under stronger multimodal attack scenarios, but our AdvMask still maintains robustness even though it was trained with text or joint-level perturbations. These results suggest that our mask-based robustness transfer generalizes beyond image-level perturbations. Consequently, conclude that AdvMask reliably identifies inherently robust neural pathways within VLMs, ensuring resilience against diverse adversarial attack types.
Cetech303
DTD
TOVIA meat
Outershere
UC718:
0.7
Figure 6: Adversarial robustness under AutoAttack. We conduct experiments in the 16-shot scenario with perturbation bound of 1/255. We report the adversarial accuracy across 5 datasets, averaged over 3 trials.
Attack Type
Accuracy (%)
CLIP shoto-shot bascline)
7.30
PGD BERT-Attac' (independent)
29.20
PGD BERT-Attack (CoAttack-style joint attack)
28.50
Table 12: Robustness AdvMast under multimodal adversarial attacks. By using the masks trained with 16-shot downstream samples, we assess whether the learned masks can still provide robustness when different modalities are atlacked. We report average adversarial accuracy (%, 1) over 5 datasets across three different runs.
C.3 ROBUSTNESS UNDER DIFFERENT BACKBONE ARCHITECTURES
Table 13 and Table 14 we provide results using larger CLIP image encoders (e.g., ViT-B/16, ViT- L/14). Our AdvMask still yields significant gains in adversarial robustness over the most competitive baseline (i.e., FAP), demonstrating strong generalizability. We also note that unlike many prompt- based methods that rely architecture-specie applicable to any vision encoder as long as intermediate features can be extracted for LAFA loss. This makes it broadly applicable for diverse downstream tasks and real-world scenarios.
components (e.g., context tokens),
AdvMask
19

---

Published as a conference paper at ICLR 2026
Table 13: Results on CLIP ViT-B/16 encoder. Using VIT-B/I6 CLIP image encoder, report 16-shot test accuracy (%, f) averaged over 5 datasets with 3 random trials.
Clean Accuracy (%)
Adversarial Accuracy (%)
Method
Caltech.
DTD FGVC. Flowers UCF. Avg.
Caltech.
DID FGVC. Flowers UCF.
Avg.
CLIP (ViT-B/16)
92.9
44.4
24.8
71.4
66.7
60.0
5.8
1.6
0.0
0.1
0.1
1.5
FAP
92.3
60.6
26.6
84.7
69.9
66.8
61.0
26.8
6.1
49.4
26.3
33.9
AdvMask (ours)
90.7
63.3
31.4
90.2
68.8
68.9
77.2
37.3
14.4
76.5
45.9
50.3
Table 14: Results on CLIP ViT-I/114 encoder. Using ViT-I/14 as CL.IP image encoder, report 16-shot accuracy (%, f) averaged over 5 datasets with 3 random trials.
Clean Accuracy (%)
Adversarial Accuracy (%)
Method
Caltech.
DTD FGVC. Flowers UCE Avg.
Caltech.
DTD FGVC. Flowers UCF. Avg.
CLIP (ViT-L/14)
95.2
53.0
32.5
79.2
75.0
67.0
13.7
3.0
0.0
0.7
1.3
3.7
FAP
96.2
72.0
38.8
94.6
82.0
76.7
66.6
22.3
8.6
46.2
33.1
35.4
AdyMask (ours)
96.8
73.7
49.6
97.2
84.7
80.4
87.5
52.0
27.7
86.3
63.7
63.4
C.4 ROBUSTNESS UNDER DIFFERENT VISION-LANGUAGE MODEI.
In our experiments, we mainly use CLIP ViT as the image encoder, following previous works on VLM robustness (Zhou et al. 2024 Mao et al. 2023 Yu al. 2024). However, since AdvMask can apply binary masks to any modular components (e.g.. self afttentenno linear layers), it is architecture- agnostic and can generalize beyond CLiP-based models. To validate this, we conduct experiments on VisualBERT (Li et al. 2019), which processes image and text jointly through a BERT-style transformer. Specifically, we adopt AdvMask to VisualBERT on two multi-modal classification datasets (CrisisMMD2INF and CriSisMMDDHUU (Alam et al., 2018)). As shown in Table the naive VisualBERT exhibits a substantial performance drop under adversarial attack, whereas our AdvMask significantly improves robustness without compromising clean performance. These results confirm that AdvMask generalizes beyond the CLIP ViT family and enhances practicality for broader VLM architectures.
Clean
Adv.
Dataset
Model
Acc.
Fl-score
Acc.
Fl-score
VisualBERT
0.85
0.82
0.40
0.38
CrisisMMD2INN
VisualBERT AdvMask
0.85
0.83
0.77
0.74
VisualBERT
0.78
0.68
0.12
0.07
CrisisMMD2HUM
VisualBERT AdvMask
0.77
0.65
0.59
0.48
Table 15: Results on VisualBERT architecture. We evaluate AdyMask on VisualBERT by applying mask parameters the self-attention layers of the last two encoder blocks of the model. For both naive and AdvMask- applied models, we perform 16-shot tuning on each two different multi-modal classification datasets (i.c., CrisisMMD2INF and CrisisMMD2HUM datasets (Alam et al.) (0088). Adversarial training is conducted using PGD-2 (e=8/255, 1=255),, and VGD-1001 is used for evaluation.
C.5 COMPATIBILIY WITH LEARNABLE PROMPT METHODS
Since AdvMask modifies only part of the visual encoder in VLMs, our method is orthogonal and complementary to prompt tuning techniques and can be flexibly integrated with them depending on task objectives. To demonstrate this, in Table 16 we present experiments combining AdvMask with CoOp (Zhou et al. 2022), a well-established learnable prompt tuning method. Specifically, we consider two cases: (1) combining independently trained CoOp prompts for the text encoder with AdvMask for the image encoder, and (2) further training learnable text prompts on top the robust visual representations produced by AdyMask, allowing the prompts adapt to robust features.
20

---

Published as a conference paper at ICLR 2026
The results show that in case (1), simply combining our robust vision encoder with learnable prompt yields significantly improved adversarial robustness compared to the original CLIP. This suggests that AdvMask strengthens the visual encoder's abilitt to generate robust representations, which can be effectively leveraged by any textual prompt. In case (2), adaptive prompt tuning further improves performance, the contextual prompts are learned to align with the robust features extracted by the masked vision encoder. These findings indicate that AdvMask is not limited to fixed prompts and can be broadly applied alongside various prompt tuning strategies to enhance VLM robustness.
Method
Clean Acc.
Adv. Acc.
CLIP
56.6
4.8
CLIP + CLOP
71.1
15.8
AdvMask + CoOP (case 1)
58.5
37.7
AdvMask + CoOP (case 2)
66.3
44.7
Table Integration of AdvMask with learnable prompt tuning method (i.e., CoOp). We report average clean and adversarial accuracy (%, f) on five downstream datasets in 16-shot setting. Two cases are compared: (1) combining independently trained CoOp prompts for the text encoder with AdvMask for the image encoder. and (2) further training learnable text prompts on top of the robust visual representations produced by AdvMask, allowing the prompts to adapt to robust features. Results averaged over 3 random trials.
C.6 COMPARISON WITH FULLY FINE-TUNED BASELINE
In our main experiments, we focused on parameter-efficien adversarial tuning methods, as fully fine- tuning a large VLM is both computationally expensive and prone to overfitting in limited-data settings. Nevertheless, we agree that including a full fine-tuning baseline strengthens our claims. Following the reviewer's suggestion, we conducted full-parameter fine-tuning under the 16-shot setting, and the results are provided in Table 17 We find that while full fine-tuning achieves reasonable performance when enough samples are available (e.g.. 16-shot), performs poorly low-data regimes (1-shot and 4-shot), exhibiting clear signs of overfitting. We also observed that full fine-tuning is highly sensitive to hyperparameters such as the learning rate, making it less stable under few-shot conditions. Importantly, the computational costs of the two approaches differ substantially: as shown in the table, full fine- me-tuning updates all parameters of the vision encoder, resulting in significantly higher training time and memory usage. In contrast, AdvMask learns only lightweight binary masks applied to small subset of modules (i.e., MHSA layers), providing far greater efficiency while simultaneously achieving stronger adversarial robustness.
1-shot
4-shots
16-shots
Comp. Cost (train)
Method
Clean
Adv.
Clean
Adv.
Clean
Adv.
Time (s)
Mem (MB)
CLIP
56.6
4.8
56.6
4.8
56.6
4.8
FAP
28.6
9.3
54.0
26.0
64.3
40,2
0.73
2863
Full FT
31.0
10.4
43.5
21.7
67.7
43.9
0.85
7724
AdvMask
46.6
18.4
57.2
32.2
67.3
47.1
0.27
1581
Table 17: Comparison with fully fine-tuned baseline.
We evaluate few-shot performance (1/4/16 shots) and training cost (time/memory per batch). Full fine-tuning (Full FT) updates all model parameters using adversarial training.
C.7 APPLICABILITY BEYOND CLASSIFICATION
Since our method designed to be readily applicable to a wide range of vision encoders, it is naturally extensible to a variety of test AdyMask on an image captioning task using LLaVA (Liu et al.) (023), a recent multimodal LLM that integrates a CLIP ViT-L/14 encoder with a large language model. Due to computational constraints, we kept LLa VA's projection layer (which maps visual tokens to the LLM input space) frozen, and replaced only the vision encoder with AdvMask-tunas version (trained ImageNet under the 16-shot setting). As shown in Table I8 although a drop in clean caption quality is observed, likely due distributional mismatch between tdyMask-tuned visual embeddings and the frozen
tasks beyond classification. To support this, we
21

---

Published as a conference paper at ICLR 2026
Image (clean)
LLaVi output when using
Image (ach))
LLaVA output when using
CLIP: "A mon standing rocky overlooking loke."
CLIP: "A game Flying Turkey Shoot is being played in the mountoins.
AdvMask: "A person stonding on rocky
AdvMask: "A lorge body of water with a rocky shore.
Image (clean)
LLaVA output when using
Image (aci.)
LLaVA output when using
CLIP: man
CLIP:
"A man sitcing table
glasses is sitting at a table with white boord behind him"
with loptop frant of him,"
AdvMask: "A mon sitting front of with laptop."
AdvMask: 'A man is sitting at table loptop and cell phone."
Figure 7: Visualization LLaVA image captioning results on clean (left) adversarial (right) examples.
projection layer, AdyMask still yields clear improvements in adversarial robustness even without any adaptation of the projection layer the LLM. This demonstrates that AdyMask effectively suppresses perturbation-sensee
parameters at the vision-encoder level and can serve as a plug-and play robustness module for downstream multimodal tasks. We believe these initial results support the promise of AdvMask as a task-agnostic robustness enhancer for VLMs.
Table Robustness evaluation of AdyMask on LLaVA image captioning task. evaluate our AdvMask on the multimodal model (i.e., LLaVA), which integrates a CLIP ViT-L/14 image encoder and a Vicuna-7B 7B language model. The task is image captioning the Flickr30K dataset (500 samples). We report CIDEr scores (0- (0-150, t) under clean and adversarial settings. Regarding attack settings, use (i) single-step APGD attack and (ii) much stronger APGD-ensembl attack (i.c., multiple APGD variants at different precision levels) following Schlarmann et al. (2024a). AdvMask is tuned on ImageNet using a 16-shot setting, and applied without additional tuning to LLaVA's image encoder.
Model
Clean
Adv. (APGD)
Adv. (Ensemble)
LLaVA (CLIP ViT-L/14)
85.18
22.13
3.26
LLaVA (CLIP VIT-L/14 AdvMask)
69.22
28.87
10.34
D ADDITIONAL ANALYSIS
D.1 ABLATION STUDY ON THE CORFFICIENT OF LLAFA
In Fig. S, we present an ablation study on the coefficient À of the loss term LLAFA in our objective function. This loss aims align intermediate- features between clean and adversarial samples during tuning, enhancing representational robustness against adversarial attacks. The results show that our AdvMask consistently outperforms the competitive baseline (FAP), regardless of the coefficient setting. In our main experiments, we set À to 50.0, as excessively large coefficients (e.g., À = 100.0) can slightly degrade clean accuracy due to overly constraining the feature space, particularly in the 16-shot scenario. Nevertheless, our AdvMask achieves competitive performance in both clean and adversarial accuracy through the proposed layer-wise wise adaptive feature alignment objective.
D.2 ABLATION STUDY ON THE ADAPTIVE WEIGHTING SCHEME
One of the key contributions of our loss design is the adaptive weighting scheme in the LAFA loss (Sec. 2.3 which is particularly crucial for stabilizing mask tuning in few-shot scenarios. To validate its effectiveness, and without the adaptive weighting mechanism across different values (i.c., the coefficient of the LAFA loss). The results show that incorporating adaptive weighting consistently outperforms the
provide ablation study in Table 19 comparing performance with
22

---

Published as a conference paper at ICLR 2026
Casn Aankoyyyyaaa
Dhas Accyncy Seaten
Class Accesey (6-
2.
2.0
2015
5313
0124
Adverarid Accarst (Lubet
Abverand Accerary Hishet)
Abvrsaid Aeceney (tSaten
2010
5013
4004
Figure 8: Ablation study the coefficient À of Llafa in our loss function. We compare method with FAP in the 4, 16-shot setting with the configurations in the main results.
unweighted counterpart, with the most significant gains observed in extremely low-shot settings (e.g., 1 I-shot). This is because the adaptive scheme emphasizes learning signals from more reliable samples, thereby reducing the risk of overfitting noisy or misclassified examples, which an especially important property under few-shot conditions. Regarding our adaptive weighting scheme in the early stage of tuning, although the model's ss initial confidence may not be perfectly reliable, the weight affects only the relative emphasis of each sample rather than removing its learning signal. Together with the warm-up strategy applied during the first epoch, this prevents early confidence errors from destabilizing optimization and enables LAFA to provide consistent gains across all shot settings.
1-shot
4-shots Clean
16-shots Clean
À
weighting
Clean
Adv.
Adv.
Adv.
False
42.0
16.4
56.1
30.7
66.5
46.3
10.0
True
44.7
17.4
56.7
31.5
66.7
46.6
False
43.5
17.2
56.3
31.0
66.6
46.2
20.0
True
46.2
17.8
56.8
31.5
67.0
46.8
False
44.5
17.8
56.6
32.1
66.9
46.8
50.0
True
46.6
18.4
57.2
32.2
67.3
47.1
Table 19: Ablation study on the adaptive weighting scheme. We compare performance with and without adaptive weighting across different À values. We report clean and adversarial test accuracy (%, 1) over 5 datasets in few-shot settings and results are averaged over 3 random trials.
D.3 ABLATION STUDY ON MASK THRESHOLD (a)
In our method, the mask threshold a controls the sparsity of the learned subnetwork by determining how aggressively real-valued masks are binarized. As shown in Table 20 increasing (e.g., from 0.001 to 0.005) leads to higher sparsity and generally improves both clean and adversarial accuracy, since the learned mask better captures task-relevant and robust pathways while suppressing noise- vulnerable parameters. However, an excessively large a (e.g., 0.007) can slightly degrade performance due ever-pruning, which reduces the expressive power of the pre- traitrained network. Importantly, our AdyMask consistently outperforms the baseline in adversarial robustness across all a values, demonstrating its stability and effectiveness in balancing robustness and transferability.
23

---

Published as a conference paper at ICLR 2026
1-shot
4-shots
16-shots
Method
Clean
Adv.
Sparsity
Clean
Adv.
Sparsity
Clean
Adv.
Sparsity
FAP
28.6
9.3
54.0
26.0
64.3
40.2
AdvMask (a=0.001)
44.8
15.3
0.01
53.8
29.1
0.03
65.5
44.0
0.12
AdvMask (a=0.003)
46.1
17.1
0,01
55.6
30.3
0,04
66.6
45,5
0.17
AdvMask (a=0.005)
46.6
18.4
0.02
57.2
32.2
0.06
67.3
47.1
0.27
AdvMask (a=0.007)
43.5
19.1
0.04
57.2
34.1
0.13
65.8
47.0
0.70
Table 20: Ablation study the mask threshold We report clean and adversarial test accuracy averaged over datasets using 3 random trials under I-shot, 4-shots, and 16-shots settings.
D.4 ABLATION STUDY ON MASK INITIALIZATION
In all experiments, we initialize mask parameters with constant value of 0.01 and use a binarization threshold a 0.005, following Zheng et al. (2023). With this setup, all parameters start in the "on" state (i.e., identical to the original model), and during tuning. perturbation-vullerabbi parameters are gradually pushed below the threshold and eventually deactivated through binarization. Due to this mechanism, the initialization value and the threshold (a) tightly coupled and jointly determine the sparsity of the learned mask (i.e., the proportion deactivated parameters). An ablation study on the threshold is provided in Appendix Sec. D33 In Fig. 21] further provide an ablation study on the mask initialization value while fixing the threshold at a = 0.005. Across all shot settings, we observe clear pattern: larger initialization values (e.g., 0.02-0.05) lead lower sparsity, since the mask values rarely fall below the threshold. However, excessively large initialization values often cause unstable tuning dynamics and degrade performance. Conversely, when the initialization value is small (c.g.. 0.007), a larger number of informative parameters are inadvertently deactivated, resulting in a slight drop in clean accuracy. Despite these outcomes, initialization values around 0.01 (the setting used in our main experiments) consistently achieve strong clean and adversarial performance, demonstrating stable behavior, Notably, these observations are consistent with the trends reported in Appendix Sec. D.3 regarding the effect of threshold a.
1-shot
4-shots
16-shots
Method
Clean
Adv.
Sparsity
Clean
Adv.
Sparsity
Clean
Adv.
Sparsity
FAP
28.6
9.3
54.0
26.0
64.3
40.2
AdyMask (init=0.007
35.2
17.2
0.069
55.5
33.8
0.223
64.7
46.3
1.188
AdyMask (init=0.01)
46.6
18.4
0.017
57.2
32.2
0.062
67.3
47.1
0.273
AdvMask (init=0.02)
45.3
12.2
0.004
51.2
25.6
0.016
62.1
40.4
0.069
AdvMask (init=0.05)
45.9
5.0
0.001
47.2
0.005
53.1
29.4
0.019
Table 21: Ablation study on the mask initialization value (with 0.005). We report clean and adversarial test accuracy averaged over 5 datascts using 3 random trials under the I I-shot, 4-shots, and 16-shots settings.
D.5 ABLATION STUDY ON LAYER POSITIONS OF LAFA LOSS
In Table 22 we provide an ablation study on the layer positions where our LAFA loss applied. Specifically, we divide the 12-layer encoder into four groups and compare performance when applying LAFA loss to each group (as well as to all groups). The results show that our AdvMask outperforms the competitive baseline (i.e., FAP) across all configurations, with stronger performance when applied to deeper or all layers. We believe this is because deactivating vulnerable parameters in later layers, which are closer to the model's final output, is more effective for improving robustness and adaptability. These results demonstrate that our approach is robust to hyperparameter choices and highlight the effectiveness of deactivating sensiive parameters through layer-wise alignment.
24

---

Published as a conference paper at ICLR 2026
1-shot
4-shots
16-shots
Method
Clean
Adv.
Clean
Adv.
Clean
Adv.
FAP
28.6
9.3
54.0
26.0
64.3
40.2
AdvMask (1 (,,1
45.3
17.5
56.4
31.4
66.6
46.4
AdvMask (I (3,4,5)
45,9
17.6
56.2
31.2
66.6
46.5
AdvMask (,8)
46.1
18.0
56.9
31.7
67.0
46.7
AdvMask (I 19,10,110 11))
44.8
19.2
57.7
32.8
66.8
47.3
AdvMask =all)
46.6
18.4
57.2
32.2
67.3
47.1
Table 22: Ablation study on the layer positions where LAFA loss is applied. We report average test accuracy over 5 datasets in 1-shot, 4-shots, and 16-shots settings with 3 random trials.
D.6 IN-DEPTH INTERPRETATION AND VISUALIZATION OF LEARNED MASK
Mask Similarity Between Different Datasets. To better understand how dataset characteristics influence the learned adversarial masks, we measure the similarity of masking patterns across dataset pairs. Specifically, we compute the overlap (IoU) over the deactivated parameters (i.e., positions where the mask value is 0), since these represent parameters identified as vulnerable to adversarial perturbations. As shown in Fig. 9 the mean IoU over the entire layers is relatively low, ranging from 0.075 to 0.124 (7.5% 12.4%) depending on the dataset pairs. This indicates that, globally, each dataset tends to highlight somewhat different parameter subsets as vulnerable. However, layer-wise analysis reveals an interesting results. We observe that early layers consistently exhibit higher IoU than later layers, meaning that the overlap in masked positions relatively larger near the input stage. This suggests that parameters in low-level feature extractors (i.e., closer to the input space) tend to be commonly vulnerable across datasets, leading AdvMask to deactivate a similar set of weights regardless of the dataset. In contrast, later layers show relatively lower IoU, indicating higher variability in which parameters are masked. These layers are more tightly coupled with downstream prediction behavior, and thus the masked parameters tend to reflect combination of (1) certubbaiion-uuleeee weights and (2) dataset-specific parameters involved in task-level adaptation. As result, the masking patterns diverge more noticeably across datasets in deeper layers. Overall, these findings highlight that our AdvMask captures both universal and dataset- deeasset-deeedeent vulnerability structures within the model: early layers encode generalizable weak points shared across downstream datasets, while later layers reveal how vulnerability interacts with dataset-specific semantic alignment and adaptation.
Mask Similarity Between Different Runs (i.e., Seeds). We evaluate the similarity of the learned masks across different random seeds Table 23 For each of the five datasets in the 16-shot setting, we train AdvMask using three independent runs with different seeds and quantify their similarity by measuring the loU over deactivated parameters (i.e., positions where the mask equals 0). We focus masked positions since the overall sparsity of the learned masks is extremely low (=00.27%) making IoU over activated parameters less informative (as most entries are equal to 1). For each dataset, compute pairwise IoU across all seed pairs and report the averaged value Mean loU over Masking Positions. As shown in Table [33, despite differences in tuning samples and optimization trajectories, the learned masks exhibit moderate level of overlap, with mean IoU values ranging from 0.20 to 0.31 (i.e., 20-31%). %). Importantly, both clean accuracy and adversarial robustness show only minor variance across seeds, indicating that the functional behavior of the model remains stable even when the exact masking locations differ. This is expected as the vast majority of parameters remain activated, resulting highly similar feature extraction pathways, while the commonly deactivated parameters contribute to consistently improved robustness. Overall, these results suggest that AdvMask consistently identifies functionally similar sets of adversarially vulnerable parameters across different seeds. Although the precise masked parameters may vary due to the non-convex and combinatorial nature mask optimization, the model reliably converges to masks that suppress semantically equivalent vulnerability patterns, leading to stable performance and robust behavior that is effectively independent of the random seed.
Which Layers or Attention Heads Are Primarily Masked? In Fig. 10 present analysis of which layers and attention heads are predominantly masked by AdvMask. Using the masks learned in the I 16-shot setting for five different datasets, we compute sparsity (i.e., the percentage of deactivated
25

---

Published as a conference paper at ICLR 2026
caltech101
caltech101 vs. fgvc
100 over mesked positions
C200
toU mesked positions
0.73
0.20
0.150
0.125
6,100
0.30
6.778
0.050
0.029
CSMV
20
12
10
11
32
Layer
Layer
caltech101 vs. oxford flowers
caltech101
ucf101
You over masked pesibors
0.25
Moan 1GU 0.115
masked prestionss
0.25
Roan 0.124
0.20
6.40
2.35
0.10
0.05
Layer
dtd vs. fave aircroft
dtd vs. oxford flowers
Iou over masked positors
0.175
Moan 30 0.075
Noam NoU 2.085
0.150
0.150
0.100
6.100
0.050
8.050
0.000
0.000
Layer
cttl ucf101
fgyc aircraft oxford flowers
ToU over trakked positions
positionss
0.150
0.125
0.325
0,100
6.100
0.075
0.073
2.050
0.025
03/VV
20
12
20
31
32
Layer
Layer
fgvc aircraft ucf101
oxford flowers vs. ucf101
0.25
IeU over mrasked posiioos
3oU 0.105
Ieu masioed positions
Nean 0.109
0.2
8.15
0.19
Layer
Layer
Figure 9: Layer-wise mask IoU across different dataset pairs the 16-shot setting. Each subfigure shows layer-wise similarity of masking positions between a pair of datasets, computed over the deactivated (masked) parameters. We use five datascts in total and report all pairwise combinations. The overall experimental setup and hyperparameters follow the main paper.
parameters) to quantify how aggressively each component is masked. To first examine which layers are more likely to be masked, Fig. reports the layer-wise sparsity of the learned binary masks, averaged over the five datasets. We observe that sparsity consistently increases toward deeper layers, indicating that later layers (where representational shifts introduced by adversarial perturbations become more pronounced) play more critical role in mask tuning. These layers require more
26

---

Published as a conference paper at ICLR 2026
Dataset
Mean IoU (%)
Clean Acc.
Adv. Ace.
Caltech101
0.25
92.9 (10.05)
75.8(10221)
DTD
0.20
58.4 (#0.17)
35.5 (11.23)
FGVCAireraft
0,31
26.8 (#1.5)
+2.0
OxfordFlowers
0.31
88.0 +0.61)
69.9 (+.
UCF101
0.22
70.6 (+00.6)
42.4 (1.0.9)
Table 23: Mask similarity and performance stability across three independent runs (different seeds) in the 16-shot setting. We report average loU over deactivated parameters (masking positions). along with clean and adversarial accuracies (mean 1 standard deviation).
Layer-wise Sparsity
Sparsity
Sparsity (%)
0.6
Layar
04
0.2
82
3.0
32
Layer index
Head Index
(a) Layer-wise sparsity
Head-wise sparsity
Figure 10: Analysis of which components are predominantly masked by AdvMask: (a) layer-wise sparsity and (b) head-wise sparsity. Both results are computed in the 16-shot setting and averaged over five datasets.
extensive deactivation of vulnerable parameters to stabilize high-level features and maintain robust predictions for the downstream task. Another important observation is that the deeper layers, which are more closely tied to task-specific adaptation, exhibit higher variance across datasets since the degree and pattern of masking required for effective adaptation differs depending on the dataset. To further understand the masking behavior within the multi-head self-attention mechanism, Fig. 10b presents head-wise sparsity for each layer, averaged over the five datasets. Interestingly, certain heads exhibit consistently high sparsity across datasets; for example, the 6th and 8th heads in the final (12th) layer show particularly strong masking. This suggests that specific attention heads are universally prone to adversarial vulnerability, and suppressing them contributes disproportionately to model's robustness. other words, AdvMask systematically identifies and deactivates a small subset of structurally fragile heads that act common failure points across datasets.
Which Module Types Are Primarily Masked? Fig. II we present an analysis of which module types within the melti-head self-attention (MHSA) block are predominantly masked by AdvMask. Since our mask tuning is applied only to the MHSA components of each transformer block for both effectiveness and efficiency, generating Q, K, and V (denoted as a)t and (2) the output projection matrix (att nn out ju)t)) which maps the concatenated head outputs back to the model dimension. (See Sec. 3.3 for ablations demonstrating why MHSA layers are the most effective target for mask tuning.) ) As shown in Fig. T1a both module types exhibit increasing sparsity toward deeper layers, reinforeing the observation that later transformer layers play a more influential role in robustness. The attn. out proj module shows notably sharper increase, suggesting that the integration stage of multi-head attention particularly sensitive to adversarial vulnerabilities and thus requires stronger masking to stabilize high-level representations. Furthermore, Fig. Ibb shows that attn out_proj exhibits more than three times higher sparsity average than attn. This may be attributed to its role in aggregating information from all attention heads (making it more susceptible to perturbed signals) as well as its tight connection downstream task-specife adaptation. Consequently, suppressing vulnerable weights in this module helps prevent distorted signals from being propagated, improving both adversarial robustness and downstream adaptability.
compare the sparsity of (1) the projection matrices responsible for
27

---

Published as a conference paper at ICLR 2026
Module-type Sparsity per Layer
Module-type Sparsity (Mean)
1.3
ettn ppro
Mean sparsity (%)
Sparsity (%)
0.6
1.0
0.4
0.
0.2
0.0
12
0.0
Layer index
out_proj
(a) Modulc-type sparsity per layer
(b) Mcan modulc-type sparsity
Figure 11: Analysis of which module types within the multi-head self-attention block are predominantly masked by AdvMask. (a) Layer-wise sparsity for the attn and attn out_proj modules. (h) Mean sparsity layers, averaged over five datasets 16-shot setting.
D.7 COMPUTATIONAL COST
Our method, AdvMask, is designed be parameter-efficient optimizing only small of binary mask parameters over a subset of the model (i.e., the self-attention layers, which account for 20% of the model), while keeping the rest of the pre-trained model frozen. To evaluate efficiency, we report quantitative comparisons of training and inference costs (latency and GPU memory usage per batch) Table 24 The results show that (1) during training, AdvMask is more memory- and time-efficient than most baselines due to its lightweight mask tuning approach, and (2) during inference, although memory usage increases slightly from additional mask parameters, latency remains lower than or comparable to baselines, demonstrating practicality for deployment, Moreover, AdvMask is effective even in challenging few-shot settings, requiring only a small number of downstream samples, making it well-suited for essource-constrannee scenarios. Overall, these results highlight that AdvMask offers practical advantages both cost and data efficiency, particularly few-shot scenarios.
Training
Inference
Method
Time (s)
Memory (MB)
Time (s)
Memory (MB)
CLIP
0.05
1268
Ad
0.29
937
0.17
1561
AdvVLP
0.49
2789
0.15
1783
AdyMaPLe
0.40
1726
0.16
1809
FAP
0.73
2863
0.16
1809
AdvMask (ours)
0.27
1581
0.13
1946
Table Computational cost for training and inference. report detailed training and inference costs (i.c., time and memory usage per hatch). All haselines use the same batch sizes (train: 4, test: 200), with adversarial sample generation cost included during training.
D.8 EXTENSION OF ADVMASK BEYOND BINARY MASK
While our framework adopts binary masks to explicitly form selective neural pathways, the method can naturally be extended to soft-mask variants. To explore this direction, implement ternary version of AdvMask in which each mask element can take one of three values {0, 0.5, 1}. This achieved by introducing two thresholds (a1 0.005, 02 0.008) while keeping all other configurations identical to the binary-mask setting. As shown in Table 25 the ternary mask achieves performance comparable to the binary version and, certain cases (e.g.. 4-shot adversarial accuracy), even slightly outperforms it due to its larger representational flexibility. Importantly, both binary and ternary variants consistently surpass baseline methods across different shot settings. These findings suggest that AdyMask can be readily extended soft-mask formulations. However, binary masks offer significant practical ad advantages in terms of parameter compactness and deployability, as they require only single bit per weight and therefore highly efficient to store, transmit, and reuse. In
28

---

Published as a conference paper at ICLR 2026
contrast, soft masks require cechberrpreiiioon numerical values, increasing storage and deployment overhead. Empirically, we find that binary masking is sufficient to capture robust subnetworks while keeping computational cost efficient.
1-shot
4-shots
16-shots
Method
Clean
Adv.
Clean
Adv.
Clean
Adv.
CLIP
56.6
4.8
56.6
4.8
56.6
4.8
FAP
28.6
9.3
54.0
26.0
64.3
40.2
AdvMask (binary)
46.6
18.4
57.2
32.2
67.3
47.1
AdvMask (ternary) 44.8
18.8
58.9
35,3
66.4
46.9
Table 25:
Comparison between binary and temaay masks. Clean and adversarial test accuracy averaged over datasets using 3 random trials under the I-shot, 4-shots, and 16-shots settings.
D.9 PER-CLASS PERFORMANCE ANALYSIS
In this section, we perform comprehensive per-class performance analysis to identify which categories benefit the most from AdvMask. We compute class-wise adversarial accuracies for both CLIP and AdvMask across all 101 categories on Caltech101 dataset. Out findings show that AdvMask significantly improves robustness across the majority of categories, with the largest gains appearing in categories that are highly brittle under adversarial perturbations. Over 40 categories where CLIP completely fails (0% accuracy), AdvMask substantially recovers performance, often reaching 40-80% accuracy. To clearly highlight which categories benefit the most, we include summary in Table [66 showing the Top-10 10 and Bottom-10 classes by adversarial accuracy improvement. This table directly illustrates that AdyMask yields the largest benefits for the most adversarially fragile categories. For examples, several categories exhibit extreme improvements, such as "car side". "cellphone" "okapi", "face", "ferry". "dalmatian", "tick", "grand piano", and "barrel", where accuracy improves by +0.7 to +1.0 absolute points (i.e., 70-80%), These classes typically rely on high-frequency or texture-sensitive cues, which are severely corrupted by adversarial perturbations. AdvMask effectively suppresses unstable activations, allowing the model to retain semantically meaningful features. This per-class analysis supports our main claim: AdvMask selectively strengthens robustness for categories most vulnerable to adversarial noise, while maintaining strong performance on clean samples.
E DISCUSSIONS
E. CONCEPTUAL DISTINCTION BETWEEN ADVERSARIAL FINE-TUNING AND ADVMASK
Adversarial fine-tuning and our proposed AdvMask share the high-level goal of improving robustness, but they operate through fundamentally different mechanisms. Standard adversarial fine-tuning directly updates the pretrained weights, altering the internal representations of the model to fit the downstream task. Such weight modifications often overwrite distort the pretrained feature space, phenomenon described in prior work (Schlarmann et al. 2024b), and may degrade generalization on unseen tasks. In contrast, AdvMask preserves all pretrained parameters and instead learns binary on/off gating that selectively suppresses perturbation sensitive units. This design identifies a robust subnetwork embedded within the original VL.M while maintaining its generalizable pretraining knowledge. Unlike weight fine-tuning, AdvMask does not modify or overwrite representations; it merely routes computation through more robust pathways. AdvMask is also computationally more efficient. The method updates only lightweight binary masks applied to a subset of modules (primarily MHSA layers, roughly 20% of VLM parameters) rather than optimizing the full set of model weights. Empirically, find that this targeted gating is sufficient to form robust neural pathways that improve adversarial robustness while retaining the strong zero-shot and few-shot generalization ability of the underlying model.
29

---

Published as a conference paper at ICLR 2026
Table 26: Per-class adversarial accuracy improvements 10.0~1.0) of AdvMask CLIP. Top-10 classes show
the largest positive improvements, while Bottom- show smallest improvements. ACCadsMau ACCCLIP.
Class
CLIP
AdvMask
A
Top-10 Improved Classes
car.side
0.000
1.000
+1.000
face
0.015
0.954
+0,939
cellphone
0.118
1.000
+0.882
okapi
0.091
1.000
+0.909
ferry
0.050
0.950
+0.900
accordion
0.125
1.000
+0.875
barrel
0.071
1.000
+0.929
dalmatian
0.250
0.950
+0.700
grand
0.276
0.966
+0.690
tick
0.333
1.000
+0.667
Bottom- Improved Classes
crocodile.head
0.000
0.067
+0.067
platypus
0.000
0.100
+0.100
crayfish
0.000
0.143
+0.143
crab
0.000
0.227
+0.227
scorpion
0,000
0.240
+0.240
mayfly
0.000
0.250
+0.250
crocodile
0.000
0.267
+0,267
bass
0.062
0.375
+0.313
lotus
0.000
0.350
+0.350
rhino
0.000
0.353
+0.353
F LIMITATIONS
In our implementation, achieve computational efficiency during adversarial mask tuning, AdyMask selectively optimizes mask parameters in multi-head self-attention (MHSA) layers. However, this approach may leave other layers potentially vulnerable adversarial attacks. Although we demon- strate in Sec. 3.3 that masking MHSA layers is indeed more effective for adversarial robustness compared to masking MLP layers (in terms of both efficiency and performance), remains possible that even more selective adaptive masking strategies could further enhance robustness. Therefore, identifying additional or alternative layers and adaptively tuning masks (while maintaining efficiency) could be important direction for future research.
30


Reviews:
[{"review_id": "XW4uBaGaRR", "reviewer": "Reviewer_LtP8", "summary": "This paper proposes an adversarial mask tuning (AdvMask) approach that searches for robust subnetwork within well-trained VLMs as a promising alternative which is highly parameter-efficient and is trained with Layer-wise Adaptive Feature Alignment (LAFA) loss. Experiments across various downstream datasets demonstrate that AdvMask consistently improves few-shot adversarial robustness over existing baselines.", "questions": "1. Why is this method particularly effective for few-shot learning? What if using more training data?\n\n2. Since the model parameters are changed after masking, does the author adopt adaptive attacks (e.g., PGD attack with knowledge of the mask) during evaluation? If not, the evaluation is not fair.\n\n3. Can this method be better than existing zero-shot robust methods? e.g., simply adversarially train the entire model with one specific dataset like ImageNet?\n\n4. Does this mask have any interpretability? For example, are they similar for different datasets?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an important problem of improving adversarial robustness in vision-language models (VLMs) with few-shot learning. The motivation is well-explained, and the paper is well-organized and clearly written. In figure 2, the authors provide a clear illustration of the proposed AdvMask method with different shots, which helps a lot in understanding the performance of the approach.", "weaknesses": "1. The proposed method is strongly related to adversarial model pruning, which has been extensively studied in the literature (arxiv.org/pdf/2409.01249). Therefore, the novelty of the proposed method is limited. The authors should also consider comparing with these adversarial pruning methods.\n\n2. The improvement of the proposed method is not significant enough. From Table 4, adding AdvMask only improves around 0.7% robust accuracy and 1.5 % clean accuracy compared to directly using typical adversarial training.\n\n3. The experimental settings are not strong enough. The authors should evaluate the performance of the proposed method with more adversarial settings, e.g., epsilon=2/155 or 4/255. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rKGMHZEPDn", "reviewer": "Reviewer_RUtB", "summary": "This paper proposes AdvMask, a method to enhance adversarial robustness of Vision-Language Models (VLMs) in few-shot settings. Instead of modifying pre-trained weights or learning prompts, AdvMask learns binary masks that selectively deactivate parameters vulnerable to adversarial perturbations, effectively identifying robust neural pathways within the vision encoder. The authors introduce a new loss (LAFA) that aligns intermediate features between clean and adversarial samples with confidence-based weighting. Experiments on 11 datasets show AdvMask outperforms prompt-based baselines (AdvVP, AdvVLP, AdvMaPLe, FAP) in few-shot adversarial robustness while maintaining competitive clean accuracy.", "questions": "1. Can you show a baseline where the model is fully fine-tuned with 16 samples per class?\n\n2. Which layers or heads are masked the most? Are these patterns stable across runs?\n\n3. Why did you choose the mask initialization values?\n\n4. Where does the method fail? For example, why is performance lower on Cars, Food101, and Aircraft datasets?\n\n5. Why is the inference memory higher than other lightweight methods?\n\n6. How does sparsity (number of masked weights) relate to robustness?\n\n7. Can you add stronger attacks (ε = 4/255)?\n\n8. Have you tested an adaptive attack that specifically targets the mask or LAFA?\n\n\n---- Additional Suggestions\n\n. Add a baseline where the mask is trained without adversarial samples to see if adversarial tuning is truly necessary.\n\n. Show how the mask changes during training (sparsity per epoch).\n\n. Include a small table showing scaling to a larger backbone such as ViT-L/14.\n\n. Report per-class accuracy to show which categories benefit the most.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": ". The concept of finding robust neural pathways via binary masks is creative and well-motivated.\n\n.The framing as deactivating vulnerable parameters rather than adding robust features is a fresh perspective on adversarial defense.\n\n. The experimental evaluation is thorough.\n\n. Substantial improvements over baselines across most datasets. \n\n.Clean accuracy recovers with more shots.", "weaknesses": ". The model is trained with only 2 steps of PGD at a very small noise level (ε = 1/255) and tested mostly at the same level. This is weak to prove robustness and may hide gradient masking. The authors should test with stronger attacks (for example ε = 4/255) to make sure the binary mask and straight-through estimator do not block gradients.\n\n. The adaptive weight in LAFA uses the model’s own prediction confidence. At early stages, this confidence can be wrong, which may make the model ignore “hard but useful” samples. The authors could try using a teacher model or stop the gradient from this weight to prevent bias.\n\n. The paper claims that certain layers or heads are more robust, but there is no clear visualization. It would help to show which layers or attention heads are most often masked and whether this pattern is consistent across datasets or random seeds.\n\n. The paper argues that full fine-tuning overfits in few-shot cases, but this is not shown. A simple 16-shot full fine-tuning baseline would make this claim stronger.\n\n. The paper says the method is efficient, but Table 19 shows higher inference memory than some baselines. A simple memory breakdown would help clarify this.\n\n. Modern papers show that some defenses look strong until the attack is adapted to the defense itself. Here, the main defensive component is the mask and LAFA, so it would be good to test with attacks that target them directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WpOCYBONyj", "reviewer": "Reviewer_kPP8", "summary": "The paper investigates an interesting adversarial defense mechanism without explicitly tuning the model weights; instead, the proposed method optimizes a model weight-level mask across different layers to enhance robustness. In other words, the proposed method shields potentially vulnerable weights to adversarial robustness. Furthermore, the paper introduced a layer-wise adaptive feature alignment scheme to optimize such a model weight-level mask by aligning clean features and their adversarial counterparts in the few-shot setup. Experiments across diverse datasets and scenarios demonstrate the generalization capability of the proposed method. Further ablations justifies the efficacy of the design of the proposed pathway method.", "questions": "1. Can the authors evaluate the text-level (BERT-Attack) or joint image-text-level attacks (CO-Attack) in addition to image-level attacks only?\n\n2. In addition to image classification, VLMs are powerful in diverse visual-language tasks, e.g., image captioning, Visual question answering. Can the authors test some of them instead of classification?\n\n3. Can the proposed mask be a soft format instead of the 0-1 style?\n\n[b] BERT-ATTACK: Adversarial Attack Against BERT Using BERT (EMNLP-2020) \n\n[c] Towards Adversarial Attack on Vision-Language Pre-training Models (ACMMM 2022)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed idea is interesting. Instead of tuning the whole weight map of VLMs to improve robustness, the paper explores an alternative way by exploring the weight-level mask to remove some implicitly vulnerable model weights against adversarial attacks.\n2. The paper is well-written and organized. A detailed recap of previous works and a background introduction are given.\n3. Extensive experiments across diverse benchmarks and scenarios are provided. In addition, a series of ablation analyses is given to verify the effectiveness of each module. Insights in Figure 4 are also interesting.", "weaknesses": "1. The proposed method might not be novel in the context of adversarial learning (for both single-modal and multimodal architectures). [a] has already explored the model weights connected with adversarial robustness. Although [a] is based on a single-modal architecture, its idea can also be easily extended to a multimodal backbone.\n\n2. The evaluated adversarial attacks are primarily low-intensity (low perturbation radius) attacks with eps=1/255. It's questionable that if the proposed method also exhibits robustness against stronger adversarial attacks with higher eps (e.g., 4/255, 8/255)\n\n3. Can the mask be regarded as part of the weights of VLMs? If so, I think that finetuning the VLM weights can also achieve the same effect, the mask would be mostly like some scailing of the standard adversarial finetuning, which means the proposed mask is an indirect adversarial finetuning. Then, it should achieve similar performance compared with adversarial finetunin.\n\n4. It seems that the mask is only for the image encoder. But the paper focuses on VLMs. In this case, it would be more appropriate to consider both branches, otherwise the work would be mostly similar to single-modal adversarial learning works.\n\n[a] Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning (ICCV 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Ff0mSPu5v", "reviewer": "Reviewer_SmjY", "summary": "This paper introduces AdvMask to improve the adversarial robustness of in few-shot settings. Instead of fine-tuning model weights, it learns binary masks to identify and activate robust neural pathways, preserving stable features under attack. The proposed LAFA loss further enhances robustness by aligning intermediate features between clean and adversarial samples.", "questions": "1. Are there any visualization or statistical results that show the specific situation of the final mask, and can we summarize in a regular way which weights are more important for adversarial and which need to be ignored?\n\n2. I'm curious if this few-shot approach would be applicable to the scenarios tuned for LVLM in FARE? Are there any challenges in doing so?\n\n[1] Schlarmann C, Singh N D, Croce F, et al. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models[J]. arXiv preprint arXiv:2402.12336, 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The setting of the task of this work is clear. The convert from Mask Tuning to Adv Mask Tuning is interesting and theoretical.\n\n2. The method framework is clearly introduced, while using visualization to help readers quickly understand the method.\n\n3. The paper provides sufficient experimental evidence and further insight.", "weaknesses": "1. This work involves two-shot scenarios: few-shot training and zero-shot evaluation. The authors are advised to clearly explain and distinguish these in the introduction to facilitate understanding. For example, the phrase \"overfitting in a few-shot setting\" on line 54 and \"zero-shot robustness\" on line 55 may not be aligned settings, but their use together could easily lead to misunderstanding and confusion.\n\n2. In the ablations Table 17, Table 18, etc., the 16-shot performances of \"47.1\" and \"47.3\" seem to be different from the \"41.99\" given in Table 10. How are they obtained?\n\n3. If the author's training data uses 3.2% of the data, then is the training time also reduced to 3.2% compared to TGA-ZSR? I didn't see a direct comparison in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)