Semantic-aware Adversarial Fine-tuning for CLIP
Authors: Jiacheng Zhang
Jinhao Li
Hanxun Huang
Sarah Monazam Erfani
Benjamin I. P. Rubinstein
Feng Liu
Venue: Accepted by TMLR ()

Abstract: Recent studies have shown that CLIP model's adversarial robustness in zero-shot classification tasks can be enhanced by adversarially fine-tuning its image encoder with adversarial examples (AEs), which are generated by minimizing the cosine similarity between images and a hand-crafted template (e.g., ''A photo of a {label}''). However, it has been shown that the cosine similarity between a single image and a single hand-crafted template is insufficient to measure the similarity for image-text pairs. Building on this, in this paper, we find that the AEs generated using cosine similarity may fail to fool CLIP when the similarity metric is replaced with semantically enriched alternatives, making the image encoder fine-tuned with these AEs less robust. To overcome this issue, we first propose a semantic-ensemble attack to generate semantic-aware AEs by minimizing the average similarity between the original image and an ensemble of refined textual descriptions. These descriptions are initially generated by a foundation model to capture core semantic features beyond hand-crafted templates and are then refined to reduce hallucinations. To this end, we propose Semantic-aware Adversarial Fine-Tuning (SAFT), which fine-tunes CLIP's image encoder with semantic-aware AEs. Extensive experiments show that SAFT outperforms current methods, achieving substantial improvements in zero-shot adversarial robustness across 16 datasets. Our code is available at: https://github.com/tmlr-group/SAFT.

Published in Transactions on Machine Learning Research (02/2026)
Semantic-aware Adversarial Fine-tuning for CLIP
Jiacheng Zhang
jaccheng.haang6 shangg het.:ngg...aaagoeeeeneeeemee Gunbmell eduu.
School of Computing and Information Systems The University of Melbourne
Jinhao Li
jinhao.li2@uni www.wnimeii..bmmmmmmmmmmm edu
School of Computing and Information Systems The University of Melbourne.
Hanxun Huang
curtis. huatig.haangg. @wwimmllll..mmmmmmmmmm edu.oo
School of Computing and Information Systems The University of Melbourne
Sarah M. Erfani
suruhh.rfaani@@niincilb...
School of Computing and Information Systems The University of Melbourne.
Benjamin I.P. Rubinstein
bengjmmin.ruuinsteinnnnuuun edu.
School of Computing and Information Systems The University of Melbourne
Feng Liu
feng.läv läu. @runimelb.. edu.
School of Computing and Information Systems The University of Melbourne
Reviewed OpenReview: https: openrevvee. net/ forum? id= S220BzeKOO
Abstract
Recent studies have shown that CLIP model's adversarial robustness in zero-shot classification tasks can be enhanced by adversarially fine-tuning its image encoder with adversarial ezamples (AEs), which are generated by minimizing the cosine similarity between images and a hand- crafted template (e.g., "A photo of a (label)"). However, has been shown that the cosine similarity between single image and single hand-crafted template is insufficient to measure the similarity for image-text pairs. Building this, in this paper, we find that the AEs generated using cosine similarity may fail to fool CLIP when the similarity metric is replaced with semantically enriched alternatives, making the image encoder fine-tuned with these AFs less robust. To overcome this issue, we first propose semantic-aware AEs by minimizing the average similarity between the original image and an ensemble of refined textual descriptions. These descriptions are initially generated by foundation model to capture semantic features beyond hand-crafted templates and are then refined to reduce hallucinations. To this end, we propose Semantic-aware Adversarial Fine- Tuning (SAFT), which fine-tunes CLIP's image encoder with semantic-aware AEs. Extensive experiments show that SAFT outperforms current methods, achieving substantial improvements zero-shot adversarial robustncss across 16 datascts. Our code is available at: https: htttpa://githtt com/tmlr-geeeeeeeeeeeeee samnlr-groupSAFT.
ensembbleee attack to generate
1 Introduction
Contrastive languoge-image pre-training (CLIP) (Radford et al,, 2021) a widely adopted framework that learns to encode text. and images into unified feature space using large datasets. This approach

---

Published in Transactions on Machine Learning Research (02/2026)
Goldfish
European Fire Salamander
Bullfrog
Tailed Frog
American Alligator
Bon Constrictor
CLIP Score
CuPL Score
WCA Score
CLIP Scores of ofs
1.0
Below Diugonal (%6: 100.0%
CuPI. Scores AfE
1.0
Below Diagonal (16): 64.8%
1.0
Below Diagonal (N)): 24.8%
WCA of
0.5
0.5
0.5
0.01
0.0
0.01
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
CLIP Scores of Clean Images
CuPL. Scores of Clean Images
WCA Scores of Clean Images
Figure 1: Comparison between CLIP (Radford et al., 2021), CuPI. (Pratt et al., 2023) and WCA (Li et al., 2024) as similarity metrics for clean images and their corresponding AEs, generated by minimizing the CLIP score via projected gradient descent (PGD) (Madry et al., 2018), across six animal classes in ImageNet-1K (Deng et al., 2009). Points below the diagonal line indicate a success in attacking the similarity metric. The results show that although these AEs can reduce the CLIP score, they may fail to fool CLIP when more semantically enriched scores are used as alternatives. This observation motivates us to rethink how AEs should be constructed in the case of CLIP.
enables remarkable zero-shot generalization capabilities and has been applied across numerous downstream applications, including large uision-language models (VLMs) such as Flamingo (Alayrac et al., 2022) and LLaVA (Liu et al., 2023). However, despite the remarkable success of CLIP-based models across wide range of downstream tasks, their vulnerability to adversarial ezamples (AEs) raises significant concerns about their safe deployment in real-world scenarios (Mao et al., 2023; Huang et al., 2025; Ma et al., 2025).
Fine-tuning the CLIP's image encoder with adversarial truining (AT) has emerged as an effective approach to enhancing its adversarial robustness (Mao et al., 2023; Schlarmann et al., 2024; Wang et al., 2024a; Yu et al., 2024; Zhang et al., 2024). The core idea of AT is to improve model robustness by training on AEs that are generated dynamically during the training process (Madry et al., 2018). AEs are typically generated by maximizing the cross-entropy loss between the model's predicted class probability and the true label (Goodfellow et al., 2015; Madry et al., 2018). In the case of CLIP, this probability is derived from the cosine similarity between a single image and a hand-crasted text template (e.g., "A AA photo of (label)"), commonly referred to as the CLIP score (Radford et al., 2021).
Recent studics (Menon & Vondrick, 2023; Pratt et al., 2023; Li et al., 2024) have shown that the CLIP score is often insufficient to fully capture image-lext aligmment. To address this, they propose semantically enriched similarity metrics as advanced alternatives to the CLIP score. For example, Pratt et al. (2023) propose customized prompts via language models (CuPL), which measures the similarity between images and LLM-generated class-specific descriptions (e.g., "A platypus looks like a beaver with a duck's bill" instead of "A photo of platypus"). Li et al. (2024) further propose weighled visual-tert cross alignment (WCA), which measures the similarity between localized image patches and CuPL-based descriptions. This naturally leads us to pose the following question:
Are adversarial examples generated by månimizing the CLIP score truly effective in degrading image-tezt aligrmmet when evaluated under more semantically enriched similarity meiries?
In this paper, we find that these AEs may fail to fool CLIP when semantically enriched similarity metries are used, as demonstrated by Figure 1. When CLIP score is used, 100% of points fall below the diagonal, indicating that these AEs successfully degrade the similarity. However, when the similarity metric is replaced by other alternatives such as WCA CuPL, this degradation diminishes significantly: only 28.6% and 64.8% of AEs fall below the diagonal line under WCA and CuPL, respectively. More surprisingly, we observe that some AEs even achieve higher similarity scores than their clean counterparts (i.e., those above the diagonal), suggesting that CLIP becomes more confident in these imputs after the attack, in other words, these AEs somchow assist CLIP in making more confident predictions. This observation suggests that adversarial perturbations optimized by the CLIP score using a single hand-crafted template (e.g., "A photo of a label") often fail to generalize to alternative text descriptions with richer attributes or contexts, and thus make the
2

---

Published in Transactions on Machine Learning Research (02/2026)
generated AEs less effective during adversarial fine-tuning. Eventually, it leads to less robust image encoder, as the success of AT-based methods (e.g., adversarial fine-taning) critically depends on the effectiveness and universality of AEs (Madry et al., 2018).
To overcome this issue, we propose Semantic-aware Adversarial Fine- Tuning (SAFT), a new framework that aims to use more semantically enriched AFs to fine-tune the CLIP's image encoder. We first propose a semantic-ensemble attack method, which generates semantic-aware AÉs by minimizing the average similarity between the original image and an ensemble of refined textual descriptions. These textual descriptions are initially generated by a foundation model (e.g,, LLM MLLM), aiming to encapsulate diverse attributes, contexts, and synonyms related to cach class label.
However, foundation models are known to suffer from hallucinations (Maynez el al., 2020). For instance, given the class "cog",, a foundation model might hallucinate that dog is a "winged mythical creature." Therefore, to make sure these descriptions are scmantically relevant and factually correct, we further propose hallucination- aware descriplion generalion method, which retains only top-K refined descriptions that are closely aligned with the class's core semantics based on the relevance score. During adversarial fine-tuning, we optimize the parameters of CLIP's image encoder to align AEs with diverse set of textual descriptions selected through our hallucinationnawwar generation method. This encourages the encoder to map perturbed images into regions of the embedding space that are invariant. to both visual perturbations and linguistic variations. We provide a visual illustration of SAFT in Figure 2 and an algorithmic description in Algorithm 1.
Through extensive experiments on 16 benchmark image datasets (including 1 in-domain dataset and 15 zero-shot datasets), we demonstrate the effectiveness of SAFT in Section 5. SAFT improves zero-shot robust. accuracy over the current state-of-the-art (SOTA) methods by at least 3.85% on average, while also achieving the second-highest zero-shol clean accuracy across the 15 zero-shot dalasets (see Table 1). In addition, we demonstrate that SAFT can scale to larger CLIP models (e.g., CLIP-L/14) in Table 4 and large datasets (e.g,, ImageNet-11)) in Table 7, respectively. More importantly, we show that SAFT can generalize well to unseen text templates (see Table 10) and can be applied to other downstream tasks beyond classification such as the image-text retrieval lask (see Table 5).
Our main contributions arc: (1) we observe that the CLIP score, computed between single image and single handerafted text template, is insufficient for accurately evaluating image-text similarity. Consequently, AFs generated by minimixing the CLIP score tend to be significantly less effective, leading to less robust image encoder. This is an aspect that has been largely overlooked in existing studies; (2) to address the mbove-mentioned limitation, we propose Semantic-aware Adversarial Fine- Tuning (SAFT), a novel framework that generates AEs by minimizing the average similarity between an image and an ensemble of selected textual descriptions during the CLIP fine-tuning process. To mitigate potential hallucinations caused by LLMs, we further propose a semantic filtering method to help remove descriptions that deviate significantly from the intended semantic meaning; (3) we empirically show that, compared to the existing adversarial fine-tuning methods, SAFT achieves a notable improvement in zero-shot accuracy-robustness trade-off on 16 benchmark image datasets against multiple adversarial attacks and generalizes well to unseen text templates.
2 Related Work
Adversarial Robustness. The vulnerability of deep learning models to AEs has been a long-standing challenge in the community and has been extensively studied (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini & Wagner, 2017; llyas et al., 2019; Croce al., 2021; Huang al., 2021; Zhang et al., 2025; Sun et al., 2025). AEs are typically generated by introducing imperceptible perturbations to clean images, with the objective of misleading classifier into making incorrect predictions. Adversarial truining (AT) (Goodfellow et al., 2015; Madry et al., 2018; Zhang et al., 2019; Wang et al., 2020; Wu ct al., 2020; Liu et al., 2021; Zhang el al., 2021; 2024) is widely regarded as the most effective strategy for defending against AEs, particularly due to its resilience against adaptive attacks (Athalye et al., 2018). However, AT often degrades performance on clean examples, a phenomenon known as the accuracy-robustnee tradc-off (Tsipras et al., 2019). Addressing this trade-off and achieving a better balance remains a key focus in AT rescarch. AT works by generating AEs and incorporating them into the model's training process, forcing the model to learn the underlying
3

---

Published in Transactions on Machine Learning Research (02/2026)
distributions of AEs. However, most existing studies on AT focus on image classifiers trained with supervised learning and are known for being computationally expensive, making it challenging to scale for CLIP.
Vision-language Models. Recent advances in vision-language models (VLMs) have significantly improved multi-modal understanding by aligning visual and textual representations through large-scale pre- pre-training. Radford et al. (2021) introduced CLIP; a pioncering model that employs contrastive learning on image-text pairs to enable zero-shot transfer across diverse tasks, demonstrating remarkable generalization. Pre-trained image encoders have been widely adopted in large VLMs (Alayrac et al., 2022; Awadalla et al., 2023; Wang et al., 2023; Bai et al., 2023; Liu et al., 2023; Zhu et al., 2024), which align the image encoder with LLMs in token embedding space via a bridging network or light querying transformer (Li et al., 2023). The majority of large VLMs rely on CLIP as their pre-trained image encoder, primarily because it is trained with text supervision (Tong et al., 2024). While these large VLMs have achieved significant success across various tasks, their vulnerability to AEs (Mao et al., 2023; Schlarmann et al., 2024) raises critical concerns regarding their safe deployment. As a result, improving the robustness of CLIP has become an important challenge.
Adversarial Finetuning for CLIP. Fine-tuning the CLIP encoder with AT has emerged as cost-effective approach to enhancing its adversarial robustness. TeCoA is the first method to improve the zero-shot robustness of VLMs, which fine-tunes CLIP encoders through AT (Mao et al., 2023). TeCoA uses a fixed zero-shot template with a class label to generate and train on A FARE builds upon TeCoA by incorporating unsupervised objectives that maximize the distance in the embedding space to generate AEs (Schlarmann et al., 2024). PMG-AFT leverages auxiliary branches to minimize the embedding distance between outputs on AEs and clean examples in both the target and pre-trained models, mitigating adversarial overfitting (Wang et al., 2024a). TGA-ZSR introduces text-guided attention to further improve zero-shot robustness (Yu et al., 2024). Adversarial fine-tuning can also be extended to consider multimodal inputs (Zhou et al., 2024). Additionally, adversarial training can be performed from scratch during vision-language pretraining, though these approaches are computationally intensive (Gan et al., 2020; Wang et al., 2024b). In this work, our focus is adversarial finc-tuning for CLIP and we make the first attempt to show that enriched textual embeddings enable the generation of more generalizable AEs that are invariant to minor textual variations.
Textual Prompting in Vision-language Models. Although CLIP demonstrates strong zero-shot performance, its effectiveness downstream tasks is highly dependent on prompt design, as highlighted by Radford et al. (2021) and Zhou et al. (2022). To miligate this sensitivity, Menon & &Vndrrck (2023) and Pratt et al. (2023) propose to leverage the knowledge embedded in LLMs to automatically generate class-specific descriptions. Li et al. (2024) further propose to use localized visual prompting (e.g. random cropping) to obtain multiple patches representing local visual areas of the query image and then align these local image patches with class-specific descriptions. Cai et al. (2025) propose to capture multiple common and unique features for each class by guiding foundation models to generate descriptive attributes and distinctive attributes. More recently, Sun et al. (2026) further improves existing methods (Pratt et al., 2023; Li et al., 2024) by proposing two complementary components to filter out redundant image crops using ToU overlap and remove repetitive or semantically similar textual descriptions via embedding- cosine similarity. These semantically enriched textual descriptions have been shown effective for image-text alignment. However, to the best of our knowledge, whether such enriched textual descriptions improve the zero-shot accuracy-roobai robustness trade-off remains underexplored.
In this work, motivated by our finding that AEs may fail to fool CLIP when alternative similarity metrics are used, we make the first attempt to show that semantically enriched textual descriptions can craft more generalizable AEs that are invariant to minor textual variations. We demonstrate that leveraging these AEs during adversarial fine-tuning further enhances the zero-shot accuracy-robusttees trade-off.
3 Problem Setting and Preliminaries
3.1 Problem Setting
The zero-shot adversarial robustness problem, introduced by Mao et al. (2023), can be mathematically formulated as follows. Let denote a distribution over unseen classification tasks. Each task T ~ I defines label space with N classes and associated data distribution Dr. An attacker, with full access to
4

---

Published in Transactions on Machine Learning Research (02/2026)
task-specific ground-truth labels yr, crafts an AE within lp-bounded perturbation set A {6 :oo1opppp S c} by maxinizing [x + +... In contrast, the defender lacks access to the task identity T or distribution DT, and must train a model fo with parameters 0 to minimize the expected worst-case risk over all tasks:
min Eyws (( max L( 8),
dEA
This formulation stands in contrast to standard adversarial robustness, which is typically evaluated on single, known task To-
3.2 Contrastive Language-Image Pre-training
Contraslive language-image pre-training (CLIP) (Radford et. al., 2021) is dual-encoder model consisting of image encoder fing I + Rd and text encoder fuxt Z + Hd where I and Z denote the input spaces of images and texts, and d is the dimension of the shared embedding space. For zero-shot classification, CLIP uses a cemplate-based prompting strategy. Given a label set ) = f.t.......NY each label ) is embedded into a textual prompt p(y) (e.g., "A photo of a (label]"). At inference, given an image 2, CLIP predicts its class by computing cosine similarities between the image embedding fing (x) and a set of text embeddings {frea (p(
Ping (2) Frext (p(y:) i N) [m ftett (p(Us))]
9 = aig max
where the numerator represents the dot product between image and text embeddings, and the denominator normalizes them using their fg-norms.
3.3 Template-based Adversarial Fine-tuning
this subsection, we present the learning objective of previous template-based adversarial fine-tuning methods and discuss their inherent limitations.
Learning Objective. Given image Dy, its ground-truth label y fci,-..,cNl and templated text prompts p(c; (e.g., "A photo of (label)"), the attacker crafts AE within an Ep-bounded perturbation set A = {ō: |l0l| I E} by solving the following objective:
8" arg max L (Jing (x frxx( SEA
where C is the cosine dissimilority between image and text embeddings. In practice, this inner maximization is often approximated using projected gradient descent (PGD) (Madry et al., 2018), which iteratively updates the perturbation follows:
g(t-1) =
g(t-1)= Projj (5 + a sign V 800 C(S (x + 811)), frex (p()))))
where 8(1) is the perturbation at itcration t, is the step size, and (roja(+) denotes projection onto the balb of radius In rastrast, the defender adversarially fine-tune fing to minimize the worst-case classification risk:
min ET E ((a.)) (Jim (r + 8"), fuex (p(x))
9
where 0 refers to the parameters of CLIP's image encoder.
Limitations. Templato-basct adversarial fine-tuning relics on fixed prompts (c.g., "A photo of a (label) to define class semantics in CLIP. While simple, this approach sulfers from two main limitations: (1) AEs optimized by a single prompt often overfit to specific phrasings (e.g., "photo of a") rather than the class itself, failing to transfer to alternative expressions like "An image of a [label)"; "; (2) fixed templates fail capture the rich attributes and contexts of real-world classes. For example, AEs generated for "dog" may not generalize to prompts like "a barking animal".

---

Published in Transactions on Machine Learning Research (02/2026)
Hallacination-aware Description Generation
Textual Emheddlngs
Deserihe- (dog).
Descriptions (Ly (
Refined Descriptions {4,)),
.ffurry I
Text Encoder fun
OR
furry mamunal that barks..
Semantic Fillering
...nusstiiated canine..
...dimesticaeed .... four-legged busting animal...
Describe...
Rour-legged huating will wingrd mythical creature...
(dog)-
Minimize 4ATT
Semantic-ensemble Attack
Semantic-sware
Visuml Embeddings
Adversarial Example
Imaage Eacoderr f
Visual Embeddings
Maximize LSAFT
Image Engoder fine
Imput Image
Adveriarial Perturbations
Textual Embeddings
Optimized
Porturbations
Iteratively Optimiize 8
Figure 2: An overview of SAFT. In hallucination-aware description generation, a foundation model generates diverse textual descriptions for each class label, followed by the semantic filtering strategy to retain top-K most relevant descriptions. These refined descriptions are then encoded by CLIP's text encoder. In semantic- ensemble attack, AEs are generated by maximizing misalignment between the visual embeddings and the average embeddings of refined descriptions. Finally, the image encoder is fine-tuned by minimizing this misalignment, aiming to learn linguistically invariant representations.
4 Semantic-aware Adversarial Fine-tuning
To address the limitations mentioned above, we propose a new framework called Semantic-aware Adversarial Fine- Tuning (SAFT), which leverages foundation model JPM (e.g., LLM MLLM) to generate semantically enriched textual descriptions. Each AE is then crafted by minimizing the average similarity between the original image and the ensemble of these descriptions. In this section, we introduce key components of SAFT, which include hallucination-awaal description generation, semantic-ensemble adversarial attack and the learning objective of SAFT. We provide the visual illustration in Figure 2 and the complete algorithm in Algorithm 1, respectively.
4.1 Hallucination-aware Description Generation
In this subsection, we present the halluchnation-aware description generation, including the description generation method, its empirical realizations and the semantic filtering strategy.
Description Generation Method. Given label space V, our goal is to construct semantic mapping, that capture diverse attributes, contexts, and synouyms associated with y. For instance, for the class "dog", a foundation model might generate semantic prompts such "a furry mammal with four legs that barks", "a domesticated canine often kept a pet.", or "a loyal animal trained for hunting companionship". These textual descriptions are generated using a foundation model fFM, which we condition on class-specific instructions (e.g., "Describe the appearance of a flabel)"). Formally, we define the generated descriptions T,, for a class y as:
where each class y y mapped to M textual prompts {t(] (M4) (,f")...,,(""")... .....ty
Ty = -4=000 (10).... (00 (FM(1)
(1)
where fFM(3 Φ) denotes the output of the foundation model couditioned on label y with generation hyperpa- ( rameters & (e.g., temperature for diversity). The resulting textual descriptions (x(m) Jm=1 are subsequently encoded by CLIP's text encoder fext into M embeddings. During adversarial fine-tuning, they offer more informative training signals by encouraging the image encoder to align perturbed inputs with the entire semantic manifold of y, rather than a single text template.
Empirical Realizations. In practice we implement two different description generation methods by adopting recent studies (Pratt et al., 2023; Cai et al., 2025):

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm Semantic-aware Adversarial Fine-tuning (SAFT)
Require: Pre-trained CLIP image encoder fing: frex; Class labels Y ( ,... Generation hyperpa- rameters Φ; Training epochs T; Training dataset Drrain-
Ensure: Robust CLIP image encoder fing
1: for E do
(M)) (FM(U: Generate (tg ...., .y...,.y
3: Label embedding: By (v)
4: for m = 1 to M do
5: Compute s(m) by Eq. (2)
6: end for
( Select (ty Tk=1 by Eq. (3)
8: Generate frext (tg ) )
l1k=1
9: end for
10: for epoch = to T do
11: for Wi) Dtrain do
12: Compute the optimized 63 by Eq. (4)
13: Update 0 by Eq. (5)
14: end for
15: end for
SAFT-L. This method adopts CuPI. (Pratt et al., 2023), which uses a LLM to generate class-specifie textual descriptions. Given class label, the LLM is prompted with a generic query " "What does [label] look like?" to produce texual descriptions.
2. SAFT-M. This method adopts (Cai al., 2025), which propose to capture multiple common and unique features for each class. For each class, set of representative images is provided along with prompls like "Describe the unique aypearance of (label) compared to other objects." and "Describe the appearance of the object flabelJ." to an MLLM. The MLLM then generates descriptions that contain descriptive and distinctive attributes.
We exclude WCA (Li al., 2024) from our realizations primarily because requires cropping each image into multiple patches, significantly increasing computational overhead. Additionally, its text description generation is aligned with CuPL (Pratt et al., 2023). Therefore, for SAFT-L, we only adopt CuPL to generate textual descriptions.
Semantic Filtering Strategy. While foundation models can generate diverse semantic prompts for class labels, they may occasionally produce hallucinations: descriptions that are semantically irrelevant or factually incorrect (e.g., describing a "dog" as "a mythical creature with wings"). To address this, we introduce a semantic filtering strategy that retains only descriptions closely aligned with the core semantics of each class. For given class y, we first obtain a set of M candidate prompts from the foundation model by Eq. (1), i.e., ..... (6 {(M)= = fFM(Y: Φ). We then compute the cosine similarity between each prompt (fm)) and the ground-truth label y, both encoded by CLIP's text encoder fret- The relevance score is defined as:
Srexe (v) Sroxe (1673)
(m)
Vm E (L....,M).
(2)
Il. (v) flex (t(")))
We sort the prompts by descending relevance scores:
( Sort (1)(m)), (slm))). 8.1. ((1) (2) (A))
Finally, we retain the top-K most relevant prompts to define the refined descriptions T.
T { (1) K
(3)
A=1
7

---

Published in Transactions on Machine Learning Research (02/2026)
where each (g)) is among the lop-K prompls ranked by s(m), This filtering ensures that the final descriptions consists only of prompts that are semantically faithful to the class y. For example, for the class "dog", the foundation model may generate valid prompts like "a domesticated canine" alongside hallucinations like "a winged mythical creature." The hallucinated prompt typically receives a low relevance score (e.g., 0.2 vs. 0.8) and is filtered out. To this end, A are guided by semantically meaningful variations, avoiding noise introduced by model ballucinations. We further conduct the ablation study in Section 5.3.
4.2 Semantic-ensemble Adversarial Attack
Building on the hallucination-aware description generation, we further propose semantic-ensemble adversarial allack. Unlike conventional approaches that larget a single prompt, we compute perturbation that maximize the dissimilarity between the perturbed image embedding fing (x + 6) and the full set of textual embeddings
{frea fiex.. Formally, the optimized adversarial perturbation obtained by solving:
5" arg arggax max LSAFT (Fimg( + 5), (freee (t(m²)
) 1==1)
(4)
where A {ō: [|ö1|p S E}, and fFM o) (ty Jm=1 denotes the textual descriptions for class y generated by fFM- The loss function LSAFT measures the average cosine dissimilarity:
M
Sing (xr + 8) Sroxt (16m))
CSAFT (Fime (x + 6), [rext (
mm=))
1
M
[img (x + 5) (1(mo)))
m=1
This step produces stronger and more semantically enriched AFs, as it requires misalignment across multiple diverse prompts rather than a single template.
4.3 Learning Objective of SAFT
In general, SAFT formulates a bi-level optimization problem to enhance the adversarial robustness of CLIP's image encoder. The goal is to train the encoder parameters such that adversarially perturbed images remain well-aligned with the diverse textual semantics produced by a foundation model frm:
min B (3.2)) [CSAPT (Fing( 8"), ( (((
(5)
where 5" is obtained by solving Eq. (4). Through this bi-level optimization, SAFT encourages the image encoder to learn representations that are robust not only to visual perturbations, but also invariant to linguistic variation across semantic prompts.
5
Experiments
5.1 Experiment Settings
Datasets. Following Yu et al. (2024), we evaluate all methods on 16 diverse datasets spanning general, fineggriind,, and specialized classification tasks. These include standard benchmarks (e.g., CIFAR-10/100 (Krizhevsky et al., 2009), Tiny-ImageNet//mwe /maageNet-1K NelnggeeN///nggNNeel-- (Deng et al., 2009), STT-10 (Coates et al., 2011),
Caltoch-101/22 101/26 (Griffin et al., 2007)), fine-grained datascts (c.g., Food101 (Bossard et al., 2014), Flowers102 (Nilsback & Zisserman, 2008), FGVC-Aireraft (Maji et al., 2013), StanfordCars (Krause et al., 2013), OxfordPets (Parkhi et al., 2012)), and domain-specific tasks (e.g., PCAM (Veeling et al., 2018), SUN397 (Xiao et al., 2010), EuroSAT (Helber et al., 2019) and DTD (Cimpoi et al., 2014)). Our main experiments use Tiny-ImageNer as the source dataset for adversarial fine-tuning, with the rest as unseen target tasks. We also include Iea as an alternative source dataset. in extended experiments (see Table 7 for more details).
Basclines. Following Yu et al. (2024), we compare SAFT with five representative basclines: (i) CLIP (Radford et al., 2021) fine-tuned on clean data; (ii) TeCoA (Mao et al., 2023), which applies adversarial training with fixed zero-shot prompts; (ini) FARE (Schlarmann et al., 2024), which adds unsupervised objectives to
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Clean and robust accuracy (%) against PGD-100 (c (c= 1/255) of different methods across 16 datasets. Tiny- Mnny-ImaeNNtt is the source datasct and the others are zcro-shot datascts. The best accuracy is highlightod in bold and the second-best, accuracy is underlined. "Zero-shot Average" refers to the averaged clean/robust. robust. accuracy across 15 datasets for evaluating zero-shot classification. We report standard deviations and averaged
rosults of SAFT-L and SAFT-M for three runs.
Source
Datasets for Erahuating Zero shot Cluneiicaion Arcuracles
Tiop Tiop-InageNN
LGVC. Aironal
OstoedPate
Calledt XOL
Caltech 206
Method
CIEAR
Food101
STL-10
EOnSAT
PCAM
SUNWT
Zern-eher
Average
DTD
CLIP
13.56
19.92
4.54
0.54
40.1KI
2.40
0.68
2.06
0L05
01.13
14.96
9.22
1.119)
1.32
1.08
0.82
0.82
TeCDA
31.20
31.96
16.10
14.35
65.95
34.30
10.09
16.30
10.10
1.87
52.92
40.62
8.74
45.57
15.97
16.25
26.00
FARE:
10.97
24.56
15.48
10.96
34.35
4.39
10:17
Robous
PMG-AFT
52.46
50.58
26.12
16.50
68.79
35.67
15.60
12.85
9.53
1.85
53.33
36.3 30
5.54
48.92
15.00
16.37
27.54
TGA-25R
47.57
42.61
22.27
20.75
75.47
42.32
25.46
18.45
10.64
3.51
60.91
50.43
12.91
42.10
21:11
23.20
31.50
SAFT-L
53.27
52.26
28.34
25.67
T8.97
40.07
30.22
20.74
12:42
4.53
64.67
54.27
15.86
47.08
22.74
26.33
35.35
1.39)
0.21)
(4 A.11
(6:
6.13)
1.60
(6.0.90
18 0.311
8:000
(4:
SAPT-M
30.70
45.52
25.20
25.84
28.48
43.20
20.43
10.31
11.96
5.02
03.38
53.08
11:00
46,80
22.45
23.02
35.78
(5: 1:40
[o 1-12)
0.96)
(e: 1.01)
6.63)
64: 8:433
(5: 10
Es 0:91)
(4: 45)
(e: 0.000
1:513
(4: 0.51)
18: 0:79)
(4: -.744)
(t: 45)
[e 0-37
1:150
CLIP
79.04
84.55
54.25
47:10
93.78
50.08
96.43
30.52
24:30
0.30
78. 60
70.81
31.15
47.88
44.40
46.80
52.72
TeCDA
54.59
60.08
FARE
77.54
87.58
62.80
70.02
03:33
81-47
57.10
36.33
22.60
14.19
84.04
77.50
44.33
07
51.78
10.01
58.68*
PMG-AFT
47:36
71.20
10.53
25. 75.77
81.85
54.28
24:14
18.25
1.5 12
3.84
64.75
48. 4B
10.77
49:98
25.72
27.58
37.42
Clonn
TGA Z5B
76.84
85,90
57.20
56.46
92.92
75.47
47.51
20.94
24.60
11:95
$0,02
73.82
26.32
49.91
06.50
50,16
54.58
SAFT-L
73:47
77:50
88.06
74.33
35.18
65.27
(8 (3:0
(2 0.70
(3 0.57)
(2.0.00
[2 6.30)
(1. 1.80)
(5: 1100
[a (:24)
(8. 8.899
18: 1:200
18 1:22)
(1. 119
18: 9.561
(8 1.339
(4. 0.70
[s 9.291
(= (.0022)
SAPT-M
74.88
86.80
60.94
58.43
94.63
75.09
00.15
30.64
29.55 56
13.96
81.35
75.26
34.41
49.99
47.50
63.08
56.01
(5:1443)
"Altbough FARE achieves the hightst cleun accuracy SAPT-L surpossre it by 17 17.46% and SAPT-M by 15 15.005% in averaged acto-soot robrsstncse
strengthen adversarial examples; (iv) PMG-AFT (Wang el. al., 2024a), which uses auxiliary branches to align clean and adversarial representations and mitigate overfitting; and (v) TGA-ZSR (Yu et al., 2024), which leverages text-guided attention and achieves SOTA performance on zero-shot robustness benchmarks.
Implementation Details. We adopt TGA-ZSR (Yu et al., 2024) as the default loss function to further push the upper bound of CLIP's zero-shot accuracy-robustees trade-off by building on its strong foundation. We use refined descriptions with top-5 relevance scores (see Section 5.3). To ensure fair comparison, we follow Yu al. (2024) in using ViT-B/32 as the backbone and SGD optimizer with a learning rate of 1e-4, momentum 0.9, weight decay 0, and batch size 128. During training, AEs are generated via fxrm PGD-2 (Madry et al., 2018) with 1/255. For evaluation, 2020), and C&W (Carlini & Wagner, 2017) to evaluate zero-shot robustness. For experiments fine-tuned on Tiny-ImageNNe all models are fine-tuned for 10 epochs. Due to limited computational resources, for experiments fino-tuned on ImageNet-1K, all models are finc-tuned for only 1 epoch.
use mainly use PGD, AutoAttack (Croce & Hein,
Hyperparameters for Generating Descriptions. For SAFT-L, we adopt (Pratt el al., 2023), which uses LLMs to generate class-specific textual descriptions. All descriptions are generated by the prompt "What does flabel) look like". These textual descriptions are publicly available at: https://gi CuPL/tree/maing For SAFT-M, we adopt (Cai et al., 2025), which propose to capture multiple common and unique features for each class. For each label, we provide 5 images that are randomly sampled from the training data with "This is photo of [label}" and additional prompts "Describe the unique appearance of a flabel) compared to other objects. Use one short sentence" " and "Describe the appearance of the object flabel). Use one short sentence.", ". The MLLM we use is GPT-4o-mini (OpenAT, 2023). We set the temperature to be 0.99, max tokens to be 100 and number of responses for each prompt to be 5.
com/ sarahpratt/
5.2 Evaluation of Zero-shot Classification
Result Analysis on Tiny-ImageNet. As shown in Table 1, both SAFT-L and SAFT-M significantly improves zero-shol robustness against. PGD-100, outperforming the previous SOTA (TGA-ZSR) by 3.85% and 2.28% on average across 15 target datasets and by 5.40% and 2.92% on Notably, SAFT-L consistently performs better across most of the remaining datasets, validating the benefit of semantically enriched text supervision in adversarial fine-tuning, In terms of clean accuracy, SAFT-L and SAFT-M rank second and third overall, achieving 55.27% and 56.01% % average clean accuracy, respectively.
respectively.
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Robust accuracy (%) of different methods against AutoAttack (c = 1/255) and C&W attack (c 1/255) across 16 datascts. Ina-ImmeeNet the source datasct and the others are zero-shot datascts. The best accuracy highlighted in bold and the second-best accuracy is underlined.
Souroe
Datzents for Feshaating Zero-shot Chassfication Arcusakes
ImageNes
FGVC FCVC-Aien Aierah
Method
CIFAR-10
CHEAR-100
Pootto1
STL-10
OxfordPoss
Flooors
EuvSAr
101
Callech-238 206
Starfo Stanford Caes
POAM
ImagoNNtt.Ik
SUNWO
Zerorsholl
Average
DID
AusoAttack (c 1/255)
TeCoA
27.21
29.81
14.23
11.64
66.20
32.46
16.91
13.09
5.77
1.29
51.55
38.58
6.77
32.71
16.21
17.41
23.64
PMG-AFT
29.90
48.66
21.64
14.32
67.59
32.41
13.86
11.65
8.81
0.99
51.26
40.25
9.59
38.53
19:17
19:63
26.60
TGA-2SR
48.95
40.28
22.53
15.00
71.90
39.49
21.81
16.38
11.27
2.31
57.73
45.41
10.20
40.86
19:20
19.11
26.80
SAFT-L
60.40
42.28
24.54
15.12
73.93
26.11
22.65
17.55
12.31
2.79
59.64
45.64
12.19
43.96
20.22
20.81
30.06
Cl/W
1,2255)
TeCoA
25.25
31.11
14.85
12.01
66,65
33.96
17.29
13.09
7.26
1.50
39.31
52.66
7,00
32.06
14.21
15.20
23.98
PMG-AFT
19.11
23.87
122
TGA-2SR
21.53
30.42
19.19
19.56
71.00
12.52
22.87
16.49
9.90
2.91
39.88
19.73
12.34
41.48
19
21.95
29.91
SAFT-L
57.73
32.04
30.50
52.19
50.40
13.19
3.93
28.15
Table 3: Ablation study on semantic filtering. We report clean and robust accuracy (%) against PGD-100
(e 1/255) of SAFT-L across 16 datasets. Tiny-ImageNe is the source dataset and the others are zero-shot datasets. The best accuracy is highlighted in bold.
Source
Dacamete for Evalussing Zerv-shot Cloeification Accunccies
ImageNee
EGVC AuNraae
Somartic Filtering
CIPAR 100
OxfordPas
FlowsesL02
DanSAT
Callesh-1o
Chllech 256
SouferdCCos
ImngeNie-1KK
Zero-shot
ST1-10
PCAM
Avetage
DTD
Robust
53.15
61.88
25,
25.25
80.20
44.69
25.87
18.40
10.96
4.02
61.53
51.29
12.00
46.00
23.36
33.64
53.27
52.26
28.34
25.67
78.97
46.07
30.22
20.74
12.42
4.53
64.67
54.27
15.86
47.09
22.74
26.33
35.35
Clean
73.03
86.52
56.05
59.02
94.04
77.05
45.32
29. 40
23.76
12.19
80.72
11.88
32.40
49.73
44.86
50.44
54.49
$7.26
93.30
25.83
13.39
T4.39
51.91
55.27
Although FARE achieves the highest clean accuracy, SAFT-L surpasses by 17.46% in average zero-shot robustness, and SAFT-M by 15.90%. These results demonstrate that SAFT-based methods notably improve the accuracy-robustnees tradc-off.
Result Analysis on ImageNet-1K. As shown in Table 7, when scaling to ImageNet-1IK, SAFT-M exhibits notably improved performance in zero-shot clean accuracy, achieving the highest overall score of 63.69% across 15 target datasets. This demonstrates the scalability of SAFT-based methods undor largor-scale pro-training. In terms of robustaness, while PMG-AFT achieves the highest zero-shot robust accuracy, it comes at the cost. of significantly lower clean accuracy. In contrast, both SAFT-M and SAFT-L attain comparable robustness while outperforming PMG-AFT by 8.70% and 8.15% in clean accuracy, respectively. This clearly highlights the advantage of SAFT-based methods in achieving more favorable accuracy-robustness trade-off when fine-tuned on large-scale datasets.
5.3 Ablation Studies
A Ablation Study on K. We investigate how the number of selected textual descriptions K in Eq. (3) affects the performance of SAFT-L against PGD-100 four datasets, including source dataset (i.e., Tiny-ImageNet) and three zero-shot datascts (i.c., CIFAR-10, CIFAR-100 and STI-10) in Appendix A.2. We find that more textual descriptions per class do not necessarily imply better performance. Specifically, when = 5, SAFT-L can achieve the best robustness-accuracy trade-off on average (see Table 8). Therefore, in this paper, we use K = for all the experiments.
Ablation Study on Unseen Attacks. We also evaluate transfcrability unseen attacks, using AutoAttack and C&W across 16 datasets in Table 2. We compare SAFT-L, with the top three baselines zero-shot.
10

---

Published in Transactions on Machine Learning Research (02/2020)
Averaged Everagod/Zott dedd-ZZrosiiie shot Robustness (%)
TGA-ZSR SAFEL
Text Templates:
54-
1: "This photo of (label)"
2: "a photo of my (label)"
52-
"a sketch of (label]"
4: "a drawing of [label)"'
50-
5: "u plastic [label)"
48
6: "a carteon (label)"
7: "a toy (label)
16
8: "a photo of a cool (labely"
9: "a photo of a clean (label)"
44-
10: tattoo of the (label)"
11: "a photo of one [label)"
42
12: black and white photo the (lubel)"
40
13: "a low resolution photo of the (label)"
10 11 12 13
Text Template Index
Figure Transferability to different text templates. We compare SAFT-L and TGA-ZSR across 13 text templates, using Tiny-ImageNeet as the source dataset and PGD-100 as the evaluation method. We report the standard deviations and averaged vero-shot robustness (%) on CIFAR-10, CIFAR-100, and STT-10 for three runs. SAFT-L consistently outperforms TGA-ZSR in averaged zero-shot robust accuracy across all templates. We provide full experimental results in Appendix A.4.
Table 4: Clean and robust accuracy (%) against PGD-100 (e = 1/255) of different methods across 16 datasets using CLIP-L/14. Tiny-ImageNet the souree dataset and the others are zero-shot datasets.
Source
Datasets for Evaluating Zero-shot Classifcationn Accurales
Ting- Ieagen
Neve IK
Method
CIFAR-10
CIEAR-
Foodtot
511-10
OxtoondPets
Flowsssin
EmOSAr
Caltech 101
Caloeh- 258
PCAM
SUN3EF
Zero-shot Average
DTD
Clss
FARE
08.23
49.19
57.77
72.12
97.01
87.08
63.
35,80
25.32
20.61
86.71
81.89
06.19
10.89
56.15
49.33
61.89
80.39
89.49
66.21
95.64
88.70
62.04
57.17
63.12
Robus
PARP:
42.20
45.82
30.73
96.74
80.18
70.73
45.85
24.68
12.07
11.51
75.50
64.35
36.31
40.42
35.00
32.89
43.51
SAPT-L
68.06
66.60
45.50
51.38
05.64
88.64
66.24
37.00
25.22
16.50
79.07
72.15
43.88
50.52
50.70
50.93
53.52
robustness (i.e., TGA-ZSR, PMG-AFT and TeCoA). Results show that SAFT-L consistently improves zero-shot robust accuracy, achieving average gains of 1.17% on AutoAttack and 7.51% on C&W attack.
Ablation Study on Semantic Filtering. We investigate the effect of semantic filtering in our method by comparing performance with and without this step under PGD-100 attacks in Table Incorporating semantic filtering yields an average improvement of 1.71% in zero-shot robust accuracy. This suggests that the quality of textual descriptions is key to our method.
4b-instruct (Yang et al., 2025), Qwen3-4b (Yang et al., 2025), Ilamm-3.2-Ib-instree 1b-iissuctt (Grattafiori et al., 2024) and 1lama-3.2-3b-ins... (Grattafiori et al., 2024) to generate descriptions, respectively. Overall, as shown in Appendix A.3, instruction- tuned models with larger size can generate better descriptions. However, we want highlight that, in our framework, the quality of semantic descriptions the fundamental factor that drives robustness improvements. LLMs serve practical and scalable implementation for generating such descriptions, but they are not an essential component of the method itself. Any mechanism capable of producing semantically accurate and conceptually rich descriptions could, in principle, be used in place of an LLM.
Ablation Study on LLMs. We conduct an ablation study using
5.4 More Empirical Analysis
Transferability to Different Text Templates. Prior methods use single template (e.g. "This is a photo of a {label}") to define class semantics, which limits the robustness to semantic variations. We investigate the transfcrability of SAFT-L and TGA-ZSR on 13 templates! including seen (i.c., "This is a photo of a (label]") and 12 unseen variants, across 3 target datasets (i.e., CIFAR-10, CIFAR-100, and STL-10). As shown in Figure 3, SAFT-L consistently outperforms TGA-ZSR in averaged zero-shot robust accuracy across
Prompt templates
randomly selected from https://github. https//h.w......oaaoo/aaaaaaaaaa
zeroshot-teaplenee teschhtttteellats.jjee.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table Image retrieval recall (TRR) (%) and text retrieval recall (TRR) (%) of different methods on COCO, Flickr8k, and Flickr30k against AutoAttack (c = 1/255).
COCO
Flickr8k
Flickr30k
Method
IRR
TRR
IRR
TRR
IRR
TRR
TGA- ZSR
34.27
41.64
57.40
66.80
61.14
69.40
SAFT-L
35.36
43.32
58.86
68.60
63.86
72.10
Table 6: Comparison of training memory usage and training time of different methods.
Methods
GPU
Batch Size
Training Memory Usage
Training Time (Per Epoch / Batch)
TeCoA
13977Mb
801s
1.03s
PMG-AFT
18423Mb
660s
0.84s
* NVIDIA A
128
TGA-ZSR
21253Mb
786s
1.01s
SAFT-L (K=
38839Mb
1098s 1.41s
both seen and unseen templates. We provide full experimental results in Appendix A.4. Specifically, SAFT-L improves averaged zero-shot robust accuracy by 6.41% on the training template and up to 6.96% on unseen variants such as "a tattoo of the {label}". These results highlight SAFT's ability to capture semantic diversity beyond fixed linguistic patterns.
Adaptability to Larger Perturbation Budget. We further evaluate our method under a larger perturba- tion budget in Appendix A.5. This evaluates whether models trained on weaker AEs generalize to stronger attacks. In Table 11, under c c= 442555, SAFT-L still outperforms bascline methods by at least 1.52%.
Scalability to CLIP-L/14. We investigate the scalability of SAFT-L CLIP-L/14 in Table 4. Notably, when scaled to CLIP-L/14, SAFT-L outperforms FARE by approximately 2% in clean accuracy and 10% in robust accuracy, demonstrating its strong scalability to larger VLMs. This result is particularly surprising, as SAFT-L slightly lagged behind FARE in clean accuracy when using CLIP-B/32, suggesting that larger VLMs better leverage the semantic richness during the fine-tuning.
Applicability to Image-text Retrieval Task. To demonstrate SAFT'' applicability beyond classification, conduct an additional experiment on the image-text retrieval task in Table 5. Specifically, we replace the original CLIP image encoder with the adversarially fine-tuned image encoder from SAFT-L. and TGA-7SR. We compare the image retrieval recall and text retrieval recall on 3 benchmark image-text retrieval datasets: COCO (Lin et al., 2014), Flickr8k (Hodosh et al., 2013), and Flickr30k (Plummer et al., 2015). Our method consistently outperforms TGA-ZSR by a notable margin.
Gencralizability to Out-of-domain Datascts. To evaluate whether the robustness gains of SAFT extend beyond Imageket-like visual semantics, we further test the models two genuinely out-of-domain datasets: TU-Berlin sketch dataset (Eitz et al,, 2012) and ImageNet-R (Hendrycks et al., 2020). They differ substantially from ImageNet-like natural images. Sketch consists of hand-drawn line sketches without texture or color information, while ImageNet-R contains artistic renditions such as cartoons and paintings that preserve high-level semantics but significantly alter visual appearance. As shown in Appendix A.6, SAFT-L consistently outperfomm TGA-ZSR by a notable margin.
5.5 Compute Resources
We compare the memory usage and training time of SAFT-L2 with baseline methods in Table 6. Inference time is similar across all methods and thus omitted. Introducing multiple semantic descriptions increases memory and training overhead. For example, SAFT requires 0.4 seconds more per batch than TGA-ZSR. However, given the trade-off between computational cost and the performance of SAFT, it is worthwhile to introduce semantically enriched text during fine-tuning.
28AFT-M consumes exactly the same memory usage and training time as SAFT-I.
12

---

Published in Transactions on Machine Learning Research (02/2026)
6 Limitations
Dependence on the Quality of Foundation Modcls. SAFT relies on foundation models to generate semantic descriptions for each class. The effectiveness of this process depends on the relevance, faithfulness, and diversity of the generated descriptions. In cases where the foundation model produces hallucinated or semantically ambiguous prompts, the resulting descriptions may be suboptimal. To mitigate this issuc, we propose semantic filtering strategy to filter out descriptions with low relevance scores.
The integration of an ensemble of textual descriptions will inevitably bring some extra cost. Luckily, we find that this process is lightweight, making SAFT computationally feasible compared to existing adversarial fine-tuning methods.
Extra Computational Cost.
7
Conclusion
In this paper, we find that AEs generated using cosine similarity may fail to fool CLIP when the similarity metric is replaced with semantically enriched alternatives, making the image encoder fine-tuned with these AEs less robust. To address this problem, we propose Semantic-aware Adversarial Fine- Tuning (SAFT), a new framework that generates semantic-awar AEs by incorporating hallucination-awee aware textual descriptions during the fine-tuning. Extensive experiments show that SAFT notably improves zero-shot adversarial robustncss across 16 datasets compared to current SOTA methods and can generalize well to unscen templates. In general, we hope this simple yet effective framework could open up perspective in the adversarial fine-tuning of CLIP and lay the groundwork for future methods that account for richer text supervision.
Impact Statement
This study on adversarial fine-tuning for CLIP raises important ethical considerations that we have carefully addressed. We have taken steps to cnsure our method is fair. We use widely accepted public benchmark datasets to ensure comparability of our results. Our evaluation encompasses a wide range of attack types and strengths to provide a comprehensive assessment. We have also carefully considered the broader impacts of our work. The proposed adversarial fine-tuning algorithm contributes to the development of more robust machine learning models, potentially improving the reliability of Al systems in various applications. We will actively engage with the community to promote responsible development and use of adversarial fine-tuning.
References
Jean Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses to adversarial examples. In ICML, 2018
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ihharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/ ft///2088000 003390,,,,, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ar preprint arXiv:2308.11 1ttes00..1966, 12966, 2023.
Lukas Bossard, Matthien Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Attributo- based visual reprogramming for image classification with clip. In ICLR, 2025.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In SBP, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion,, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. In NewrIPS, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM Transactions on Graphics, 31:1 - 10, 2012.
Zhe Gan, Yen- Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingiing Liu. Large-scale adversarial training for vision-and-language representation learning. In NeurIPS, 2020.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Bapliste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Fsiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egoo Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia. Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Nlyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Tyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengehuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Scan Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nic, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman,
14

---

Published in Transactions on Machine Learning Research (02/2026)
Tara Fowler, Tarek Sheasha, Thomas Georgion, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupla, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinseng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yumus, Andrei Tupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Fissa Jamil, Flaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuar Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Takshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elent Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jcan-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstcin, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Tee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Tuca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kencally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mchta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanoy, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian WI, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei
15

---

Published in Transactions on Machine Learning Research (02/2026)
Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv..rgg/ http://www...r./////././.// 22787
Gregory Griffin, Alex Holub, and Pictro Perona. Caltech-256 object category dataset. 2007.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2020.
Micah Hodosh, Peter Young, and J. Hockenmaier. Framing image description as ranking task: Data, models and evaluation metries. J. Artif. Intell. Res., 47:853 899, 2013.
Hanxun Huang, Yisen Wang, Sarah Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Exploring architectural ingredients of adversarially robust deep neural networks. NewrIPS, 2021.
Hanxun Huang, Sarah Monazam Erfani, Yige Li, Xingjun Ma, and James Bailey. X-transfer attacks: Towards super transferable adversarial attacks on CLIP. In ICML, 2025.
Andrew Tlyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NewrIPS, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (canadian institute for advanced research). 2009. URL http:://wwt cs toronto. edu/-kriz/cifar html.
Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, and Feng Läu. Visual-text cross alignment: Refining the similarity score in vision-language models. arXiv preprint arXiv:2406.022 02915,,0.2....... 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742 PMLR, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Feng Lin, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, Masashi Sugiyama, et al. Probabilistic margins for instance reweighting in adversarial training. NeurIPS, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In NewrIPS, 2023.
Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, Hanxun Huang, et al. Safety at scale: A comprehensive survey of large model and agent safety. Foundations and Trends in Privacy and Security, 2025.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, abs/1300 1bs/1,06.515,, 2013.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zcro-shot adversarial robustness for large-scale models. In ICLR, 2023.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and factuality in abstractive summarization. Tn ACI, 2020.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
OpenAl. GPT-4 technical report. arXiu, abs/2303.08774, 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicodo, J. Hockonmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74 93, 2015.
Sarah M. Pratt, Tan Covert, Rosanne Tiu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn ICML, 2021.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large dision-languag models. In ICML, 2024.
Yuhao Sun, Jiacheng Zhang, Zesheng Ye, Chaowei Xiao, and Feng Liu. Sample-specific noise injection for diffusion-based adversarial purification. In ICML, 2025.
Yuhao Sun, Chengyi Cai, Jiacheng Zhang, Zesheng Ye, Xingliang Yuan, and Feng Liu. Let's roll a bifta: Bi-refinement for fine-grained text-visual alignment in vision- langugge models. In Transactions on Machine Learning Research, 2026.
Christian Szegedy, Wojcicch Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHYA JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NewrIPS, 2024.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In MICCAI, 2018
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In CVPR, 2024a.
Weihan Wang, Qingsong Lv, Wenmong Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. aixiv preprint weXiw:2311.0307 2023.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
Zeyu Wang, Xianhang Li, Hongru Zhu, and Cihang Xie. Revisiting adversarial training at scale. In CVPR, 2024b.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. NewrIPS, 2020.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to 200. In CVPR, 2010.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingleng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: //axxiv. org/ prrg/bb//// abs/ 09388.
Lu Yu, Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness in vision-language models. In NewrIPS, 2024.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In JCML, 2019.
Jiacheng Zhang, Feng Liu, Dawei Zhou, Jingfeng Zhang, and Tongliang Liu. Improving accuracy-robustnes trade-off via pixel reweighted adversarial training. In ICML, 2024.
Jiacheng Zhang, Benjamin 1. P. Rubinstein, Jingfeng Zhang, and Feng Liu. One stone, two birds: Enhancing adversarial defense through the lens of distributional discrepancy. In ICML, 2025.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighhted adversarial training. In ICLR, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337
2348, -34377248, 2022.
Wanqi Zhou, Shuanghao Bai, Qibin Zhao, and Badong Chen. Revisiting the adversarial robustness of vision language models: a multimodal perspective. arXin preprint arXiv:2404. 19287, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision- language understanding with advanced large language models. In ICLR, 2024.
18

---

Published in Transactions on Machine Learning Research (02/2026)
A Additional Experiments
A.1 Experiment on ImageNet- 1K
Table 7: Clean and robust accuracy (%) against PGD-100 (c (c= 1/255) of different methods across 16 datasets. ImageNett is the source dataset and the others zero-shot datasets. The best accuracy is highlighted in bold and the second- best accuracy is underlined. "Zero-shot Average" refers to the averaged clean/robuse /robust accuracy across 15 zero-shot datasets. Datisets for Exalnsting Zero-shot Classifiation Aceuraiiss
Source
Method
ImngeNee Ik
CIFAR-10
CIFAR-10
Rood101
SEL.10
Oxford
BuoBAT
Caloeds 101
Chlbeds 256
Stanford
POAM
1Ti- Eone--nggeee
SUNSIT
Average
17133
98.18
82.81
73.70
06.12
G1.
TeClA
52.38
72.02
41.27
56.30
92.25
77.21
48.45
33.99
20.07
15.21
62.61
17.36
30.17
43,80
53.52
55.17
50.58
PMG-AFT
49.39
80.43
29.66
65.73
93.99
75.22
48.30
32.55
23.30
12.06
63.60
74.09
30.31
48.21
56. 26.24
50.44
54.99
TGA ZSR
63.41
88.25
57.05
73.91
95.33
61.05
00:13
38.03
39.51
19.08
81. 66.66
81.73
65.73
46.51
61.45
60.25
02.43
63:37
98.38
20:21
34.93
17.85
92.70
47:29
65:75
62.92
SAPT-M
64.04
89.13
63.03
70.66
95.83
63.57
58.96
40.85
36.35
17.19
85.01
82.40
45.00
57.86
69.22
00:29
63.69
Rosmus
TeCoA
28.55
33.01
17.38
25.13
70.35
49.88
27.76
22.66
12.23
5.22
67.70
57.18
11.29
21.41
21.94
28.91
31:29
FARE
25.97
37.13
25.58
72.85
52.85
27.73
20.96
11.39
63.02
53.00
14.04
24.74
20.05
21.27
30.86
PMG-AR BMGGAFTT
29.88
40.64
17.07
21.41
81.08
51.16
27.83
23.51
14.41
4.62
69:07
57.22
10.25
23.27
22.
27.12
32.78
TGA-ZSR
30,30
26.18
12.41
24.41
71. .66
41
27.19
20.74
7.36
67.
55.37
12.35
13.16
15.12
20.40
28.95
30.82
26.19
23.76
12.70
09:07
17.20
SAFT-M
32.42
30,36
16.22
21.13
71.53
53.53
20, 43
21.91
12.05
4.99
69.97
59.92
14.31
21.84
22.60
25.23
31.67
A.2 Ablation Study Number of Descriptions
Table 8: Ablation study on number of descriptions per class. We M to represent the number of descriptions selected for each class. Tiny-ImmaeeNe is the source dataset and CIFAR-10, CIFAR-100, and STL-10 are zero-shot datasets. The best accuracy is highlighted in bold and the second-best accuracy is underlined. We report the averaged results of three runs.
Source
Datasets for Evaluating Zero-shot Classification Accuracies
No. of Descriptions
Tmn-eeegeeet Robust
CIFAR-10
CIFAR-100
STL-10
Average
Clean
Clean
Rohust.
Clean
Robust
Clean
Robust
Clean
Robust
K 72.00
49.70
84.92
47.89
56.26
25.77
93.34
77.68
78.63
50.26
K =
73.18
50.21
81.85
19:
56.29
26 10
92
77.65
76 80
50.77
73.17
43.04
56.66
76.88
K 4
73.31
11.08
85.11
12.21
56.98
20.95
92.73
74.15
77.01
15.35
K
73.47
53.27
85.81
52.26
57.26
28.34
93.30
78.97
77.46
53.21
73.83
85.38
57.05
77.58
77.45
18.62
K
73.51
44.11
84.86
41.27
57.77
19.37
93.21
75.90
77.34
45.16
73.17
47.90
85
45.55
57.39
22.49
92.81
76.63
77.15
18.14
K 73.41
45.33
85.15
13.66
37.60
21.15
93.50
75.74
77.41
16.17
K 10
73.66
43.86
85.41
41.30
58.52
20.63
93.56
74.67
77.79
45.11
19

---

Published in Transactions on Machine Learning Research (02/2026)
A.3
Ablation Study LLMs
Table 9: Ablation study on LLMs of varying quality and scale. We report the best in bold.
Source
Zero-shot Datasets
LLM
Timy-ImageNet
CIFAR-10
CIFAR-100
STL10
Swen3-4B-instruee
73.96
85.45
55.44
93.58
Qwen3-4B
73.22
86.35
56.98
93.73
Clean
Llama-3.2-11 3.2-1B-instruct
74.00
86.39
56.42
93.43
Llama-3.2 1attaa-..--B--sttuue
74.05
86.25
56.28
92.31
Qwen3-4B
-ween--4Biinstuet
53.45
51.91
27.79
78.91
Qwen3-4B
47.10
44.96
23.40
76.08
Robust
19.08
46.15
23.98
77.14
Llama-3.2 L.ama-3.2-3B-innst 2-3B-inttuuct
50.14
48.31
26.40
78.31
A.4 Transferability to Different Text Templates
Table 10: Ablation study on text templates. We compare clean and robust accuracy (%) against PGD-100
(e = 1/255) between TGA-ZSR and our method across datasets. Tiny-InageNel is the source dataset and the others are zero-shot datasets. 13 different text templates are used for evaluation, which include 1 seen template ("This is a photo of a {label]") and 12 unseen templates. The best accuracy highlighted in bold. The performance improvements and degradation are reported in green and red, respectively. We report the averaged results of three runs.
Type
Template
Method
Thyy ImageNet
Source
Datasets for Evalusting Zerosshot Classflention Accmsscio
CIFAR
CIFAR 100
STL 10
Zeco shot Average
Robust
Closa
Robust
Robnst
Cleam
Robast
"This is photo of a (label)'
TGA-2SR
76.84
47.87
85,90
42.61
57.20
22.27
92.92
75.47
78.67
46.78
SAFT-L
73.47
59.27
85.81
02.26
67:26
28.34
93.30
78.97
78.79
(4-0,12)
58.19 (4-6.41)
phoso of ma (labed)'
PGA. ZSR
72.68
80.61
90.52
73.32
74.00
44.30
SAFT-1
70.57
50.71
82.51
48.79
61.77
25.60
91.35
76.07
75.21
(41.18)
50.18 (4-5.88)
"I slerth of a [label>*
TGA 28R
74.00
45.12
84.41
41.41
55.00
21.64
90.08
78.01
77.00
46.35
SAFT-L
71.97
61.78
84.71
61.34
65.41
27.58
94.12
80.06
78.08
(4-0.39)
52.99
(4-0.64
deswing of (labwi)
TGA 2SR
74.42
45.55
84.86
42.17
55.89
22.24
95.44
76.05
78.00
46.82
SAFT-L
72.22
52.05
85.13
62.01
27.75
94.13
80.19
(4-0.25)
58.02
(14550)
"n plostic (labwd)*
TGA- TGA-2SR 2SR
71.52
43.58
84.44
41.50
53.72
20.56
92:48
74.87
78.88
45.67
SAFT-1
00.50
49.93
81.12
50.00
50.22
26.41
02.09
78.32
76.35 (11.54)
51.58 (+5.91)
cartoon (label)*
TGA-7SR
72.10
13.64
65.55
43.25
53.61
21.57
92.84
75.52
77.33
66.77
SAFT
85.93
53.36
27.35
93.09
(+0.33)
(+6.62)
"A toy (label)'
TGA-ZRR
70.58
42.51
84.58
44.45
51.93
30.50
90.54
72.63
75.08
45.56
S
69:47
49.50
62.55
50.32
52.48
26.05
69.89
75.49
74.96 (:0.22)
50.62
(++4.76)
"o photo of cool (lbbe]*
TGA-2SR
76.98
46.76
84.95
41.28
07.02
22.006
92.90
70.34
78.35
46.23
[40053)
(+46660)
Po photo is clean (label)'
TSA-2SK
74.01
45.76
81.67
29.34
54.58
20.85
88.25
70.96
SAFT-1
71.87
51.70
63.08
49.20
65.43
26.99
80.67
73.86
76.01
(41116) 49.90
(4-0357)
"' tattoo of the (label).
TGA. 28R
69.23
41.47
83.38
41.65
51.53
20.30
92.29
74.76
75,30
45.37
(4-0.78)
(44.96)
photo of one [biielp
84.87
41.59
21.20
92.96
75.23
86.01
SAFT-L 1-1
73.49
62.85
85.16
61.87
66.50
27.38
918.4
79.08
78.40
(4-0.42)
52.78 (46.77)
"w lluck and white photo of the [kbbd]
TGA ZSR
72.11
43.62
$4.81
00.71
54.42
20.92
02.45
74.58
77.22
45.34
SAF
79.33
(490996)
52.02 (46.08)
ansolution photo the (label)*
TGA-7SR
40.44
75.01
77.73
45.63
SAFT 1.
72.20
51.51
84.71
50.09
55.06
27.02
83.30
78.05
77.89 (+0016)
32.02
(+6630)
20

---

Published in Transactions on Machine Learning Research (02/2026)
A.5 Adaptability to Larger Perturbation Budget
Table 11:
Robust accuracy (%) of different methods against PGD--00 4/255) across 16 datasets. Tiny-ImageNer is the source dataset and the others are zero-shot datasets. bold and the second- best accuracy is underlined. "Zero-shot Averag" refers to the averaged robust accuracy
The best accuracy is highlighted in
across 15 zero-shot datascts.
Sourve
Zero-shos Datasets
FGVC- Airerafe
Thy ImageNNel
Stanford Chars
ImageNet-I1 IK
OxfordPets
Flower 102
Caltech-256
Method
CHARR
CIFAR-I0
Food101
S1L.
DID
EusSAA
Calteci-
PCAM
SUNTOT
Zero-shot
Average
TeCoA
1.63
0.45
0.25
9.29
0.50
2.23
0.00
11.18
5.90
0.04
2.53
0.58
0.26
2.31
PMG-AFT
4.94
3.77
3.50
1.38
19.54
2.73
2.85
3.99
0.47
0.06
20.33
10.00
0.29
24.33
2.38
1.51
6.47
TGA-2SR
16.76
12.04
5.04
2.13
36
48
1.94
0.90
0.00
0.00
22
73
15.03
0.16
0.02
2.51
1.10)
7.13
SAFT-L
15.92
A.6
Generalizability to Out-of-dommin Datasets
Table 12: Experiments TU-Berlin sketch dataset and ImageNet-R. R. We report the best in bold.
Method
Sketch
ImageNet-R
TGA-ZSR
36.80
46.74
Clean
SAFT-L
43.85
52.85
TGA-ZSR
27.05
26.54
Robust
SAFT-L
29.75
29.70
21


Reviews:
[{"review_id": "WabknrC0hv", "reviewer": "Reviewer_D52s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper identifies a key limitation in current CLIP adversarial fine-tuning methods, showing that adversarial examples generated using a single hand-crafted template fail to degrade similarity under semantically enriched metrics, thereby reducing their effectiveness during training. To address this, the authors propose SAFT, a framework that generates semantic-aware adversarial examples using LLM/MLLM-derived textual descriptions with hallucination filtering, and optimizes CLIP using a semantic-ensemble attack. Extensive experiments across 16 datasets, multiple attacks, larger CLIP variants, and diverse prompt templates demonstrate that SAFT substantially improves zero-shot adversarial robustness while maintaining strong clean accuracy, offering a simple yet effective direction for more semantically grounded adversarial fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The motivation is clear and strong, for the empirical results show that the failure of CLIP-score–based adversarial examples under richer similarity metrics.\n2. Extensive experiments across 16 datasets, multiple attacks, CLIP variants, and diverse prompt templates provide consistent and comprehensive evidence.\n3. Ablation studies are abundant.\n4. Neat and clear writing.", "audience_yn": "Yes", "audience_explanation": "The robustness of CLIP is crucial for the community, and I think that researchers in the community will be interested in this paper.", "requested_changes": "1. Clarify sensitivity to the choice of the base model. The authors could consider analyzing the changes in results when choosing a smaller/weaker model as the base model; a detailed discussion would also be beneficial.\n2. Provide more explanation for the semantic filtering strategy. The article introduces a hallucination-aware filtering step, but the rationale behind the specific metric (cosine similarity with the class label) is lightly discussed. A short justification of *why* this metric was chosen, and how it avoids filtering out useful but more abstract descriptions, would improve clarity.\n3. The ‘Extra Computational Cost’ in the limitations section is shown in Table 6, right? I noticed that SAFT-L (K=5) needs about more 40% in training time and 90% on GPU memory, but the performance achieves the best, so I think maybe the authors can introduce a metric that can represent the trade-off between performance and cost and provide a short discussion.", "broader_impact_concerns": "The paper aims to improve the robustness of VLMs, which is socially beneficial. I noticed that the authors already provide a responsible impact statement section, and I do not identify any concerns that require further clarification.", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "W1aIemmdY3", "reviewer": "Reviewer_hNk7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel method to fine-tune CLIP‘s image encoder with semantic-aware adversarial examples (AEs), whose purpose is to enhance the CLIP model's adversarial robustness in zero-shot classification tasks. Traditionally, AEs are generated by minimizing the cosine similarity between images and a handcrafted template. This study shows that AEs generated using cosine similarity may fail to fool CLIP when the similarity metric is replaced with semantically enriched alternatives, making the image encoder fine-tuned with these AEs less robust. To overcome this problem, this paper first introduces a semantic-ensemble attack to generate semantic-aware AEs by minimizing the average similarity between the original image and an ensemble of refined textual descriptions. Then this paper proposes Semantic-aware Adversarial Fine-Tuning (SAFT), which fine-tunes CLIP’s image encoder with semantic-aware AEs. SAFT achieves substantial improvements in zero-shot adversarial robustness across 16 datasets.\n\n**Strengths**\n\n1. The core idea of integrating semantically enriched, hallucination-filtered textual descriptions into adversarial fine-tuning is well-motivated.\n2. The framework is carefully designed, with complementary components (hallucination-aware description generation, semantic-ensemble attack) that logically address the identified limitation.\n\n3. The empirical evaluation is comprehensive, covering diverse datasets, attack types, and extensions, providing strong support for the method’s effectiveness.\n\n**Comments** \n\n1. The discussion around Figure 1 could be further clarified by explicitly defining how similarity statistics are used to quantify the effectiveness of adversarial examples. Making this connection more explicit—perhaps by relating changes in similarity scores to robustness degradation—would strengthen the methodological rigor of the motivation section. This clarification would also help readers better understand why certain AEs fail under semantic metrics.\n\n2. Since the study emphasizes semantic-aware robustness, it would be helpful to formally define the robustness metric (e.g., through an explicit mathematical expression).\n3. The presentation of Table 9 is somewhat ambiguous. Although the caption mentions multiple perturbation budgets (ϵ = 1/255, 2/255, 4/255), the table itself reports only a single value per dataset, which may lead readers to assume it corresponds to one fixed ε (e.g., 2/255). It might be clearer to align Table 9 with Table 10 by explicitly reporting separate results for each perturbation level or by clarifying in the caption that the numbers represent averages across all ε values. This would make the trend described in the text (“performance degrades with increasing ϵ”) easier to verify.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims on the effectiveness of the proposed Semantic-aware Adversarial Fine-Tuning (SAFT) framework and its semantic-ensemble attack are well supported by extensive experiments across 16 datasets and multiple attack settings. The implementation is clearly described and results consistently demonstrate improved zero-shot robustness.", "audience_yn": "Yes", "audience_explanation": "The study explores semantic-aware adversarial robustness in vision–language models, a topic highly relevant to TMLR’s focus on multimodal learning and robustness. Its empirical findings on how semantic perturbations affect CLIP would interest researchers studying robust and generalizable multimodal representations.", "requested_changes": "See comments above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3T6rs5H9JE", "reviewer": "Reviewer_iDR4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a semantic-aware adversarial example generation and fine-tuning framework that enhances CLIP’s adversarial robustness by aligning image perturbations with semantically diverse textual descriptions generated by LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the paper’s main claim that adversarial examples generated solely with cosine similarity (i.e., using a single handcrafted template) fail to fool CLIP under more semantically enriched similarity metrics is consistent with the empirical evidence. The experiments convincingly show that enriching adversarial examples with diverse text prompts enhances both zero-shot adversarial robustness and the model’s invariance to linguistic variation.", "audience_yn": "Yes", "audience_explanation": "The paper’s findings would be of clear interest to TMLR’s audience. It presents a interesting and empirically strong approach to improving CLIP’s adversarial robustness through semantic-aware fine-tuning. The experiments in paper strongly support the claim that training on semantically diverse and enriched textual descriptions produces an image encoder more robust to adversarial perturbations.", "requested_changes": "**Requested Changes**\n\n1. The proposed method’s success in generating semantically rich textual descriptions is entirely dependent on the capabilities of the underlying LLM. Consequently, the quality, factuality, and diversity of those descriptions hinge on the chosen model’s competence.\nTherefore, it would be important to include ablation studies using LLMs of varying quality or scale (e.g., small vs. large, instruction-tuned vs. base) to examine how differences in language-generation ability affect the robustness and performance of the final model.\n\n\n2. Although the authors evaluate SAFT on 15 zero-shot benchmarks, most of these datasets (e.g., CIFAR, Caltech, Food-101, Flowers-102) belong to similar natural-image domains and share visual characteristics with Tiny-ImageNet, the source dataset used for fine-tuning.\nTo more convincingly demonstrate the generalization ability of semantic-aware adversarial fine-tuning, it would be valuable to include evaluations on truly unseen or out-of-domain datasets (for example, sketches, or medical data). Such an experiment would test whether the robustness improvements extend beyond ImageNet-like visual semantics and confirm the method’s effectiveness under genuine domain shifts.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)