Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models
Authors: Lu Yu
Haiyang Zhang
Changsheng Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: adversarial attack
zero-shot robustness
pre-trained vision-language models
Primary area: safety_in_machine_learning

Abstract: Due to the impressive zero-shot capabilities, pre-trained vision-language models (e.g. CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: the Attention Refinement module and the Attention-based Model Constraint module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness: The Attention Refinement module aligns the text-guided attention obtained from the target model via adversarial examples with the text-guided attention acquired from the original model via clean examples. This alignment enhances the model’s robustness. Additionally, the Attention-based Model Constraint module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. The experiments validate that our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets. Our code is available at https://github.com/zhyblue424/TGA-ZSR.

Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models
Lu Yu
Haiyang Zhang
Changsheng Xu
School of Computer Science and Engineering, Tianjin University of Technology State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, University of Chinese Academy of Sciences fluyu@email, zshyōstud} tjut. edu. en, csxu@nlpr. ia ac cn
Abstract
Due to the impressive zero-shot capabilities, pre-trained vision- lision-language mod- els (e.g. CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed be susceptible to adversar- ial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: the Attention Refinement module and the Attention-based Model Constraint module. Our goal is maintain the generalization of the CLIP model and enhance its adversarial robustness: The Attention Refinement module aligns the text- cext-guided attention obtained from the target model via adversarial examples with the text-ge -ot-gguidd attention acquired from the original model via clean examples. This alignment enhances the model's robustness. Additionally, the Attention-batt based Model Constraint module acquires text-guided attention from both the target and original models using clean examples. Its objective to maintain model perfor- mance clean samples while enhancing overall robustness. The experiments validate that our method yields a 9.58% enhancement in zero-shot robust accu- racy over the current state-of-the-an techniques across 16 datasets. Our code is available at http::/ggihubbcoo///gggue//////
Introduction
Large-scale pre-trained vision- vision-language models (VLMs) have showcased remarkable success in artificial intelligence by seamlessly integrating visual and textual data to understand complex mul- timodal information, such CLIP [48]. Leveraging vast datasets and powerful architectures such as BERT [10] and its variants [8, 33], these models adeptly capture semantic relationships between images and texts, offering significant advantages across numerous applications. From image classi- fication [14, 67, 55] and semantic segmentation [50] to image captioning [39] and vision question answering [44], pre-trained VLMs revolutionize how machines perceive and interact with multimodal information. Their importance lies in their ability to learn rich representations from varied data streams, enabling zero-shot learning and transfer learning across domains and tasks. Thus ensuring the reliability of large-scale models is crucial. However, these models are vulnerable to adversarial attacks as many other networks as demonstrated by recent studies [38, 59], even slight perturbations to input data result in misclassificatio altered outputs. Such attacks pose significant chal- lenge, particularly in critical applications like autonomous vehicles [60], medical diagnosis [32], and maritime navigation [29], where the consequences of erroneous decisions can be severe. As these large-scale models become increasingly prevalent in real-world applications, understanding and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

mitigating the risks posed by adversarial attacks is essential to maintain trust and reliability in AI systems.
Adversarial training [53, 61, 69] has emerged as a crucial technique in enhancing the robustness of deep learning models against adversarial attacks. By augmenting training data with adversarial examples generated through perturbations of input data, models are forced to learn more robust decision boundaries, thereby improving their resilience to adversarial manipulation. Given the rising significance of large-scale VLMs in various applications, understanding their vulnerability to adversarial attacks is essential. While adversarial training presents practical challenges when applied to downstream tasks, especially with large-scale models. Firstly, adversarial training typically involves generating adversarial examples during each training iteration, which increases the computational overhead and may lead to overfitting on the training data. This phenomenon is exacerbated in large-scale models with vast parameter spaces, where fine-tuning becomes more susceptible to overfitting. Moreover, adversarial training may not adequately prepare models for all possible adversarial scenarios, potentially leaving them vulnerable to unknown data distributions encountered in real-world settings. Exploring zero-shot adversarial robustness in these models is particularly pertinent as it sheds light on their ability to generalize and perform reliably in unseen scenarios. Additionally, considering the multimodal nature of VLMs, the exploration of zero-shot adversarial robustness offers insights into the complex interactions between visual and textual modalities, paving the way for more robust and trustworthy multimodal AI systems.
Text-guided Contrastive Adversarial Training (TcCoA) method [38] represents the pioncering effort in investigating the zero-shot adversarial robustness of large-scale VLMs. They aim to bolster CLIP's zero-shot generalization capacity against adversarial inputs. While their primary focus lies on enhancing accuracy in the face of adversarial samples, this improvement comes at the expense of decreased performance on clean data. Subsequent work by PMG-AFT [59] builds upon this by introducing pre-trained model guided adversarial fine-tuning technique, further enhancing both generalizability and adversarial robustness. However, despite the advancements made by both studies in enhancing CLIP's zero-shot robustness, significant questions regarding the interpretability of adversarial attacks and the efficacy of adversarial training remain unanswered. Specifically, the mechanisms through which adversarial attacks influence network outputs and the reasons behind the effectiveness of adversarial training strategies remain elusive. In our paper, we delve into the text-guided attention shift phenomenon to shed light on how adversarial attacks alter model outputs. Leveraging these insights, we propose a simple yet effective strategy, TGA-ZSR, aimed at enhancing the robustness of the CLIP model and preserving its performance clean examples.
Our main contributions are summarized follows:
To our knowledge, we are the first to introduce text-guided attention to enhance zero-shot robustness vision-language models while maintaining performance clean sample.
We improve the interpretability of adversarial attacks for zero-shot robustness on vision-language models through a text-guided attention mechanism.
The experimental results show that TGA-ZSB surpasses previous state-of-the-an methods, estab- lishing new benchmark in model zero-shot robust accuracy.
2 Related Work
Pre-trained Vision-language Models. In recent years, advancements in computer vision 17, 34] have primarily relied on training models with image-label pairs to recognize predefined object categories. However, these approaches often overlook the inherent semantic connections between textual descriptions and visual content. Motivated by the remarkable progress witnessed in natural language processing (NLP), exemplified by breakthroughs like Transformer [56]. BERT [10], and GPT-3 [3], researchers are increasingly drawn to the prospect of using textual data to enhance the capabilities of DNNs. These methodologies are referred to as VLMs [21, 48, 49, 64] and one prominent approach to directly learn the semantic similarity between images and corresponding textual descriptions through image-text pairs. By aligning the embeddings of these two modalities,
models like CLIP [48], ALIGN [21], BLIP [25], Visual-BERT [47], and ALBEF [26] aim to achieve superior performance across various tasks. CLIP [48] leverages vast dataset of 400 million image- text pairs sourced from the internet and employs contrastive loss to effectively align the embeddings of both modalities, thereby enhancing the model's capabilities. Experimental results underscore the

---

significant performance gains achieved by incorporating textual information into the model, with zero-shot performance surpassing that of earlier deep neural network architectures. However, despite its impressive zero-shot accuracy, experiments [38, 59] reveal vulnerabilities to adversarial examples, resulting in notable decline in robustness.
Adversarial Robustness. Deep neural networks have been found to be vulnerable to adversarial examples [54, 36, 40, 66], which can fool DNNs to produce false outputs, rendering trained models unreliable. To bolster robustness against such adversarial attacks, various advanced methods have been proposed, including data augmentation [28, 58, 27, 65J, adversarial training [69, 53, 61, 68], progressive self-distillation [1], randomization strategy [11, 35], and adversarial purification [41, 24, 62]. While these strategies aim to improve DNNs' adversarial robustness, they often come with increased complexity or limited generalizability. Adversarial training [69, 53, 61, 68] stands out as one of the most widely used and effective approaches. fine-tuning DNNs by generating adversarial examples during training. After the emergence of CLIP [48], many subsequent works [45, 16, 63] have utilized CLIP as a backbone, yet little attention has been given to studying its adversarial robustness. CLIP is shown to be susceptible to adversarial examples [38] as well, posing significant threat to downstream tasks utilizing CLIP as a backbone. Hence, investigating the adversarial robustness of CLIP is crucial.
Zero-shot Adversarial Robustness for VLMs. The visual-language model, trained on both image and text data, serves foundational model for various tasks. However, it has shown vulnerability to adversarial examples [38, 59], and training from scratch is time-intensive. TeCoA [38] was the first to explore zero-shot adversarial robustness for VLMs, aiming to enhance CLIP's adversarial robustness by minimizing the cross-entropy loss between image logits and targets. While TeCoA solely utilizes cross-entropy loss, yielding only marginal performance improvements, PMG-AFT [59] extends this approach by minimizing the distance between features of adversarial examples and those of the pre-trained model. FARE [5 primarily focuses on maintaining high clean accuracy while improving model robustness, achieving this by constraining the distance between the original and target model embeddings. Our experiments reveal significant differences in attention maps between original examples and adversarial examples. Leveraging this insight, we enhance model robustness by constraining it with text-guided attention.
3 Methodology
3.1 Preliminaries and Problem Setup
Following the previous works [38, 59], we choose CLIP model as the pre-trained VLMs for image classification task. Given an image-text pair (z, (), where x represents an image and represents a textual prompt, CI.IP learns to encode both the image and the text into fixed-dimensional embeddings. Let f(x) denote the embedding of the image x and g() denote the embedding of the text prompt t, y is the one-hot vector label. For training or fine-tuning on the downstream tasks, we use the cross-entropy loss, denoted as L(x, t,)).
L(x, L, v) = -iij wiylog Σn (rr(cco([(x)).) (())))
where we set Vij = 1 if the image-text pair is positive, otherwise, Vi 0. is the temperature parameter and cos indicates calculating the cosine similarity of the two embeddings.
Adversarial Attacks. Adversarial attacks are a concerning phenomenon where small, often imper- ceptible perturbations are intentionally applied to input data with the aim of deceiving a model into producing incorrect outputs. These perturbations are crafted with the goal of causing the model to misclassify or generate erroneous predictions while appearing indistinguishable
human observers. The Projected Gradient Descent (PGD) [36] method is an iterative approach for crafting adversarial examples. It starts with the original input data and then iteratively adjusts the data in the direction that maximizes the model's loss function while ensuring the perturbed data remains within a specified perturbation budget. Mathematically, the PGD attack can be expressed as follows:
Ta+1 = N + E sign (Va. T. I, y)))
Here, represents the loss function, a denotes the original input data, e controls the magnitude of perturbation, and L.LL represents the gradient of the loss function with respect to the input data.
3

---

shepberd
standard poodle
standard poodle
golden retiover
chibuabua
goldfish
goose
goose
Oi linge
Oni inw.e.dinnnnnn ditenntonnnnnnn
jinrikisha
chimpanzee
swimming bunks
chibushua
egyptian cal.
cauliflower
altrros
birdhouse
Adk: imogg iww.e.tthetiii othenttoon Adv. Imoge.
Figure 1: The four depict the original image, its associated attention map, the generated adversarial example, and the attention map of the adversarial example. Labels black indicate the ground truth, while those in red represent mis-classified labels for the adversarial examples.
By adding or subtracting times the sign of this gradient the original input data, the PGD attack generates adversarial examples that lead to misclassification or incorrect predictions by the model. 1l++s makes the perturbed data remains within an e-neighborhood of the original input, preventing the generated adversarial examples from straying too far. S is a set of allowed perturbations that formalizes the manipulative power of the adversary.
Adversarial Examples Generation and Adversarial Training. The optimization objective for crafting adversarial examples aims maximize the loss of model so with respect to perturbed input ta which can be formulated as:
Fa argmaa ([fo(Ta; b,)
Adversarial training is a technique to generate adversarial examples from the original training data and then these examples to train the model, forcing it to learn resist adversarial perturbations. To adapt the model the downstream tasks, we apply adversarial fine-tuning on one target model towards robustness with the following loss:
argmin J (Za t, f2)
Where I represents the total loss function used for training the model.
Zero-Shot Adversarial Robustness. this paper, we investigate the zero-shot adversarial robustness of CLIP model, which refers to the ability of these models to maintain performance and reliability even when encountering unseen adversarial samples during inference, with only adversarial fine-tuning the original CLIP model one target dataset, such as Tiny- ImageNet.
3.2 Text-Guided Attention based Interpretation of Adversarial Attacks
Text-Guided Attention. Attention mechanisms [30, 16, 31] play a crucial role in enhancing vision model performance across various tasks. At its core, attention enables models to focus relevant parts of the input data while suppressing irrelevant information. Similarly, VLMSs by incorporating textual guidance, the models effectively focus relevant visual features while processing

---

Adv. Attacks Generation
Adv. Emi.. from Target Model
Oti, Emb. from Target Model
Jon
goddfooo
onange
Original
Text Eibhsiding
photo
[CLASS]
Text
Encoder
Jxpur Terts
y
Advensarial Fine-tucing
g(t)
Frozen model parameters
Target Image Encoder
piar (2a)
Pooling
f(2a)
LCE
Adversorial foage
LAR
f (x)
L Lam
Original
Image
(x)
Fncoder
Original lonage
Figure 2: An overview of our TGA-7SR framework: We generate adversarial examples and feed them into the target image encoder. To enhance the adversarial robustness of the CLIP model and maintain its generalization, for adversarial examples through the Attention Refinement module and constraining the model to prevent significant drift via the Attention-based Model Constraint module.
introduce text-guided attention. This involves refining the framework
language, thus facilitating more accurate and coherent multimodal understanding. Additionally, text-guided attention enhances interpretability by providing insights into the model's decision decision-making making process, fostering trust and understanding in complex multimodal systems. Thus, we investigate the impact of rext-guided attention enhancing and interpreting zero-shot adversarial robustness in VLMS in this paper. We define the text-guided attention as following:
A(x) = fx()) g(t),, A
Where fz( represents the global image feature before the pooling operation of f(x), and P denotes the dimension of the attention embeddings. We reshape A to RvPxPP obtain the attention map, which is then resized to E HHW Finally, we apply a normalization operation (norm) on A to obtain the final text-guided attention map.
Interpretation of Adversarial Attacks. The previous research has predominantly focused on bolstering the zero-shot robustness of Vision-Languas Models (VLMs), yet the reasons leading to mis-classifications induced by adversarial attacks remain unclear. This paper aims to shed light interpreting the impact of adversarial attacks on VLMs. By employing Eq. 5, compute the text-guided attention for both the original image (Ori. image) and its corresponding adversarial counterpart (Adv. image), as depicted in Fig. Remarkably, despite the subtle discrepancies imperceptible to the human eye between the adversarial example and the original image, the former is mis-classified (labels in red). However, a significant difference emerges the respective text-guided attention maps. Specifically. we observe a notable shift in the text- guided attention of the adversarial example, characterized by instances of displacement towards other objects, backgrounds, or even disappearance. For instance, while the original images in the first, second, and fourth columns pay attention to their subjects' heads, their adversarial counterparts, attention diverges elsewhere. In the third column, the attention shift leads from the correct object to an incorrect one, resulting in mis-classification. In the fifth and seventh columns, the attention in their adversarial counterparts redirected towards the background.
5

---

3.3 Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR)
The semantic information embedded within text representations are preserved through a frozen text encoder, offering invaluable guidance when adversarial perturbations disrupt relevant visual features, which has not been explored for zero-shot robustness of rision-language models. We introduce the Attention Refinement Module, designed to effectively filter out irrelevant information, thereby mitigating the impact of adversarial attacks seeking to exploit vulnerabilities in the model's decision-makir making process. Moreover, to maintain model's ability to generalize effectively on clean images, we introduce the Attention-base based Model Constraint Module. This module ensures consistent performance on clean data while enhancing the model against adversarial disruptions. Additionally, employing text-guided attention enhances interpretabilityy offering crucial insights into how the model integrates and processes information across modalities. This interpretability not only instills trust in the model's 's predictions but also facilitates the detection and mitigation of adversarial attacks. Our approach (i.e. TGA presents a comprehensive framework (as shown in Fig. 2) for enhancing model robustness to adversarial perturbations while concurrently improving interpretability. We will introduce the details follows.
Attention Refinement Module. Based on the insights gained in Section 3.2, we propose an attention refinement module aimed at enhancing the robustness of the model. This module is designed to rectify the text-guided attention of adversarial samples, which often leads to altered predictions. Our approach aligns the adversarial attention map with that of the clean samples, known for their high-accuracy attention distribution. This simple yet effective strategy serves to mitigate the impact of adversarial perturbations on the model's predictions.
We take the generated adversarial sample Ta to the target model ftar()) and the clean sample the original model (-ri()) and obtain the adversarial attention map A(r )aar and the clean attention map A(x )ori respectively. The attention refinement loss LAR is thus defined as:
N
1 N
LAR
AA(x))ar - A(x') (oo")orlll ||2
(6)
20
where A(xa)tar = flar(xa) - a(t) and A(x) = for((x) g(t) Illa denotes the L2 distance computation between two attention maps.
A Attention-based Model Constraint Module. The Attention Refinement module serves to enhance the robustness of the models, consequently improving the accuracy of adversarial samples. However, this enhancement comes with a trade-off: it may marginally sacrifice the accuracy on clean samples due to shifts in model parameters. To preserve the generalization capability of pre-trained VLMs, we introduce an Attention- based Model Constraint module. This module aims to mitigate performance drops on clean images, thereby ensuring the overall effectiveness and reliability of the model.
Specifically, we input the clean sample x into the target model faar (-), adversarially fine-tuned on the Tiny- Inn-ImmgeNeet dataset, to acquire the text-guided attention map A(r)tar. Concurrently, the original text-guided attention map outputted from the original CLIP model fori(-) is denoted A(x)) To ensure the preservation of importance parameters for clean images, we enforce an L22 distance constraint between these two attention maps. The baased model constraint loss LAMC is formulated as:
1 N
LAMC
A((x)) - A(x))rril)
Y=0
Thus the final loss function can be represented as:
Litoial
I.CE + LAR B. LAMC
(8)
4 Experiments
4.1 Experimental Setup
Datasets. Our experiments begin with training the pre-trained CLIP model on the - [9]. Then we evaluate the model's zero-shot adversarial robustness across 15 subsequent datasets, fol-
We only compute the attention map for the image corresponding the text prompt of the ground-truth label.

---

Table 1: Zero-shot robust accuracy images attacked with 100 steps of PGD [36]. We performed several different methods on Tin--mageNNet and evaluated across 16 datasets. The optimal accuracy is highlighted in bold, while the second-best accuracy is underlined. The values in parentheses represent the standard deviation.
Thyy haw-haageoaa
JOVC TGVC-Alnral
CHAM-AN
37L-00
Foutfay
Ortordpas
Calooh-
RCAM
BED
Methoes
Averaige
CLIP
0.85
0.26
26.11
6.60
1.19
0.05
0.00
12,00
0.20
PT
13.55
19.92
4,94
40.00
0.64
0.6
2,66
0.05
0.03
14.95
0,09
1.32
7.05
FT-Adv
$1.90
98.58
31
66.55
17.00
12.55
54.00
19.93
15.00
11.95
183
17.36
56.73
40.18
8.42
48.88
28.85
ToGaA [38]
$7.57
30.30
17.53
67.19
19.70
14.76
36.44
21.46
1745
12.14
162
18.18
55.36
41.88
8.49
47.39
28.06
FARE[S]]
23.83
21.25
10.72
99.99
8.30
10.97
24.56
15.45
1096
0.14
0.4
10.54
45.56
54.35
4.38
16.17
18.25
PMG-AFF]59]
47.11
46.01
25.83
74.51
22.21
19.58
41.62
23.45
1505
12.54
195
21.43
62.42
43.99
11.72
48.64
32.51
TGA (ours)
63.08
61.45
38.27
84.22
38.22
33.97
50.78
34.88
22.8
14.27
4.78
28.74
70,99
06.06
20.40
47.76
42.09
18:0113
18900
18936
12030
18905
18:40
12
Table Zero-shot clean aceuracy, We performed several different methods on Tiny-ImageNe and
evaluated across 16 datasets. The values in parentheses represent the standard deviation.
FGV. AGve--irrest
Treg-
CDARR11
C17A8-100
S7L-10
Fondins
Outandyas
Calreda
Callade
Staddrrd
Avertage
60.45
65.47
42,89
20.25
54.25
$1.15
FT-Adv:
39.69
16.49
67.41
TeCoA 138]
38.75
74.63
24.65
45.81
FAREISH
62.80
57.10
22.69
84.04
44.35
59.85
PMG 5F1
44.65
35.66
17.76
76.70
25.21
7GA-25R(oms)
75.72
86:46
56
99.48
$3.90
$7.59
77.30
48.08
3906
24.24
1193
48.04
80.70
74.74
56.62
49.58
56.44
lowed by previous studies, such TeCoA [38] and PMG-AFT [59]. These datasets include several commonly used classfication datasets, including CIFAR-10 [23], CIFAR-100 [23], STI.110 [6], Ima- geNet [9]. Caltech- 101 [13], and Caltech- 256 [15]. Additionally, fine-grained image classification datasets such as StanfordCars [22], Flowers 102 [42], Food101 [2], FGVCAireraft
[37], and Oxford- Pets [46] are included. Furthermore, the scene recognition dataset SUN397 [43], the medical image dataset PCAM [57], and the satellite image classifacation dataset Au [18] and the texture recog- nition dataset DTD [5] are incorporated for comprehensive evaluation. We also conduct experiments on four additional datastes (i.c. ImageNet_subeeee subset, ImageNet-A, ImageNet-O and ImageNet-PR as shown in Supp. Mat. A.I.
Implementation Details. Following the protocol of previous works [59], we fine-tuned the CLIP model on the adversarial samples of Tiny- Imay-lma [9] as 'adversarial fine-tuning' and subsequently evaluated its performance across 15 datasets and Tiny-ImageNet itself. We employ ViT-B/32 as the backbone in CLIP and utilize the SGD optimizer to minimize loss. During adversarial fine-tuning, we update all parameters of the image encoder with a learning rate of le-4, weight decay of O, momentum of 0.9, and batch size of 128. We utilize loo norm PGD-2 [36] with 2 iterations to generate adversarial examples, with an attack strength e of 1/255 and the attack step size is 1/255. To evaluate zero-shot adversarial inference, we employ Loo norm PGD-100 [36] with 100 iterations, attack step of 1/255 and a batch size of 256 to generate adversarial examples for verifying CLIP'' adversarial robustness. Additionally, assess the model's robustness under different attack strengths, we perform inference using adversarial strengths e of 1/255, 2/255, and 4/255. The hyper-parameten and | are to 0.08 and 0.05 respectively in Eq. 8 in the main experiments. Maintain the same parameters for the CW attack. For the AutoAttack [7] experiments, a and / are set to 0,08 and 0.009, We conducted the experiment utilizing the RTX 3090, which required training period ranging from 3 to 4 hours.
4.2 Main Results
To validate the effectiveness of our approach, we conduct comparisons with several state-of- oate--of-the the- art methods such as TeCoA [38], PMG-AFT [59], and FARE [51]. Additionally, we extend the comparison to include CLIP (the original pre-trained CLIP model), FT-Adv. (adversarial fine-tuning using the contrastive loss of the original CLIP) and FT-Clean (fine-tuning on clean examples with the contrastive loss of the original CLIP) for comprehensive evaluation.
7

---

Table 3: Zero-shot robust accuracy images attacked with of 1/255 of AutoAttack [7]. We
performed several different methods on Tiny- ImageNet and evaluated on 16 datasets.
They tno--nnsphhs
FGYC: FGVe-Airra Airerst
01AR-10
CIFAR-10
S71.30
KaroSAr
Louge Not
Called-
Callect 256
Stoufoni
PCAM
DFD
Methods
Average
CLIP [S5]
0.02
0.01
0.08
0.00
0.04
0.01
0.00
0.00
0.36
0.12
0.06
0.04
0.43
0.10
0.11
0.22
0.09
FE-Clan
0.08
0.03
0.01
0.91
0.00
0.04
0.06
0.00
0.48
0.00
0.03
0.12
1.38
0.66
0.03
0.05
0.25
FT-Ads
50.48
37.55
20.39
16.25
11.33
33.91
18.54
19.85
L1.00
165
16.21
49.0
19.24
7.57
48.84
28.28
TeCoA [58)
35.03
28.18
15.09
66.08
1741
13,06
34.81
20.50
15.37
1140
1.32
16.32
54.54
40.15
7,15
47.12
26.55
FARE 28.39
13.58
9.72
27.72
9.15
0.87
47.45
6.77
19.78
PMG-AFT
1.8
TOA-ZSR (OUm))
22.18
57.16
30,66
Table Zero-shot robust accuracy across 16 datasets with CW attack [4]. The optimal accuracy is highlighted in bold.
RGVC-AMeTa
JGVC- AMarst
ShnbbeeCarr
CRAAR-W
CHAAR-AM
Oddondeuts
Fowersit
Imagenee
Cilled 286
STZ-10
PCAM
DTD
Average
[48]
116
13.50
2,36
PMG [MG-117159]
48.59
44.85
24.15
74,11
19.99
17.33
39.88
20.95
13.51
12.09
1.47
19.51
60.99
44.46
10.57
31.07
TGA-ZSR(Oun))
63,83
60.50
34.62
84.11
22.03
33.28
58.33
32,95
21.32
13.89
4.56
20.42
70.34
59.73
20.20
48.02
40.50
Adversarial Zero-shot Robust Aceuracy. Table shows that the average accuracy of our TGA-ZSR outperforms the original CLIP model by 37.19%. Compared to current stat-of-the-ar method, PMG- AFT, the proposed method achieve an average improvement of 9.58%. general, our method is superior than all the other methods on most datasets except comparable result on PCAM dataset. In addition, we obtain the best result on -nn-emmaeNee, which is not striet zero-shot test. It indicates that our method is robust on the adversarial attack on both seen and unseen datasets.
Zero-shot Clean Accuracy. Table illustrates the model's accuracy for clean examples using different methods. Our method outperforms PMG-AFT by 9.84% and FT-clean by 2.07% in terms of average accuracy. Similar Table 1, zero-shot clean accuracy exhibits improvement not only on an individual dataset but across all datasets. However, we observed that our zero-shot clean accuracy is 3.41% lower than that achieved by FARE. zero-shot clean accuracy. However, we have significantly enhanced the zero-shot robust accuracy in adversarial scenarios with 23.84% gain compared to FARE Table 1.
important to note that FARE prioritizes preserving
4.3 Experiments on More Attack Types
Results against AutoAttack. AutoAttack [7] stands out as a strong attack method for assessing model robustness. We follow TeCoA and PMG-AFT to verify the perturbation bound of 1/255 in the standard version of AutoAttack. The results are summarized Table 3. We can see that the original CLIP model experienced significant performance decline, decreasing to 0.09% the adversarial example. Our TGA-ZSR also demonstrates a decline but still achieves superior results compared to other methods, validating its effectiveness against stronger attacks.
Results against CW Attack. CW attack [4] is an optimization- based approach designed generate small perturbations to input data, causing the model to make incorrect predictions while keeping the perturbed input visually similar the original. We further evaluate the robustness of our approach against this challenging attack, using perturbation bound of 1/255. The results, shown in Table 4, demonstrate that our method significantly outperforms the state-of-the-art method PMG-AFT on both adversarial and clean samples. This substantial margin indicates the adversarial robustness of our proposed method.
4.4 Ablation Study
Different Types of Attentions. To validate the important role of text-guided attention our method, we conducted experiments by replacing it with vision-based attention. We employ Grad-CAM [52], widely adopted method, for generating attention maps based on vision. Table demonstrates that replacing the text-guided attention with vision-based attention yields results that are still comparable to

---

Table 5: Comparison of vision- basod attention and our text-guided attention. We evaluate the state-
of-the-art method PMG-AFT alongside our pipeline, incorporating two different types of attention mechanisms on Iny-ImageNet and evaluating performance across 16 datasets.
They Hoagehes
100
CUAR-10
CHAK-
Roudien
Calleda
Tcst
Methods
Average
PMG-AFT]599
42.11
46.01
25.81
74.51
21.21
19.58
41.02
13:55
15.05
12.56
21
62.42
45.99
11.73
32.51 47.55
Vicion- husand
70.16
13.24
19
11
18.08
8,88
10.47
TGA-75R (nurs)
K3.99
31.78
34.13
56.91
34.20
21.92
59.70
21.15
|MCG-AFT)5)
74.31
Table 6: Zero-shot robust accuracy images attacked with of 1/255, 2/255 and 4/255 of PGD [36].
We performed several different methods on Tiny-Iman Eny-ImmgeNet and evaluated across 16 datasets. We represent the average accuracy across various attack strength.
Tie How-douge
MGVC-AMeeaa Alevrt
Stanfeet arr
CB1
CMAR-10
571-10
S0N209
Cctondpes
RAPOMA
Imageree
Calledh 101
Calledh-20k
PCAM
DTD
Methods
Averuge
2:97
0.71
9.18
FT-Clean
4.65
36.16
044
0.01
7.96
0.03
ITI Adv
29.33
18.10
11.06
45.13
8:58
5.65
16.45
10.15
9,72
9.82
0.83
8.81
33:43
24.14
3,80
38.06
17:07
TeCaA [38]
18.17
12.78
8.12
39.87
8.90
633
1861
11.04
10,07
9.88
0.53
8.43
34.94
23.92
3.45
33,20
15.41
25.30
21.71
13.29
47.69
11.43
9.89
20.88
12.86
9.45
10.65
0.90
11.28
41.86
28. 38
5.40
37.85
19.27
FARE [S]]
12.41
0.09
423
33.72
2.08
4.75
9.67
3.57
4.36
0.05
0.28
3.00
23.07
16.95
1.43
8.54
TGA 2SR (ours)
33.40
27.72
14.68
99.59
12.40
12.99
24.69
13:42
9.70
7.47
1.53
11.69
91.41
32.51
7.29
19.64
20.45
the state-of-thes method PMG-AFT in terms of both zero-shot robust accuracy and clean accuracy, This finding validates the effectiveness of our method's pipeline. Furthermore, text-guided attention significantly improves the average accuracy, demonstrating the advantage of incorporating textual guidance.
Effect of Attack Strength. We further assess the robustness of the pre-trained model by different levels of PGD-2 attacks. Specifically, set to values of 1/255, 2/255 and 4/255, progressively amplifying the magnitude of the adversarial perturbation. This allows us to investigate whether model trained weak adversarial examples exhibits robustness against stronger adversarial perturbations. In Table 6, we present the average results for three distinct levels of attack strength. Despite general decline in the robustness of all methods, approach still achieves superior results, outperforming PMG-AFT by 1. 1.18%, and FARE by 1.91%.
Effect of Each Component. We conducted several experiments thoroughly evaluate the effective- ness of each component of our method, as summarized in Table 7. Using LeE alone significantly enhances the model's robustness through standard adversarial training, but it also results in notable decrease in clean accuracy compared to the original CLIP model. Applying our Attention Refinement module further improves the average zero-shot accuracy both adversarial and clean samples. Finally, the Attention-based Model Constraint module dramatically boosts performance, increasing robustness by 10.25% and clean accuracy by 6.52%.
Trade-off between Robust and Clean Accuracy. Achieving the balance between robustmess and clean accuracy is crucial in adversarial training. Overfitting models tends yield high robustness but low clean accuracy, whereas underfitting typically results in the opposite scenario. As shown in Fig. 3, methods positioned close the dotted line excel in either adversarial accuracy or clean accuracy, yet they often struggle to strike a balance between robustness and clean accuracy. In contrast, our method demonstrates not only an enhancement in the model's robustness but also the maintenance of clean accuracy, resulting in an overall superior performance.
More comprehensive and detailed ablation studies, including hyperparameter selection, the effect of distance metrics on the loss function, and the effect of learning rate, can be found in Supp. Mat. A.2.
4.5 Computational Overhead and Time Efficiency
We have evaluated our method against others in terms of memory usage, training time, and test time, and the results are summarized in Table 8, Our method increases memory consumption by approxi-

---

Table 7: Ablation study each component. Af- ter adversarial fine-tuning the model using adver- sarial examples generated by PGD-2, verify the robustness of the model using adversarial ex- amples generated by PGD-100.
PIG-FT
VIAMs
Robust
Clean
Average
Fichas
CL
CLIP
4.90
64.42
34.66
Zarnsshot seruracy(*)
Lee
29.45
44.97
37.21
The trade-off between robustness and clean accuracy. Each point the graph repre- sents a method, with the size of the point indi- cating the extent which it achieves a favorable trade-off between robustness and clean accuracy.
Figure 3:
+LAR
31.71
49.96
40.84
+LAMC
41.96
56.48
49.22
Table 8: Comparison of memory usage, training time, and test time.
Methods
Train memory usage
Train time (per epoch / batch) Test time (per batch)
CLIP [48]
OMb
Os / Os
21s
TeCoA [38]
12873Mb
512s 0.65s
21s
PMG-AFT[59]
18449Mb
828s / 1.06s
21s
TGA-ZSR (ours)
21227Mb
885s / 1.13s
21s
mately 15% compared to state-of-the-at method PMG-AFT, This due to the additional computation required for the text-guided attention map. The training time for our method is comparable to that of PMG-AFT. The test time remains consistent across all methods.
Conclusion and Limitations
In this paper, we discovered that adversarial attacks lead shift of text-guided attention. Building on this observation, we introduce a text-guided approach, TGA-ZSR, which incorporates two key components preform adversarial fine-tuning and constrain the model. This strategy prevents model drift while enhancing model robustness. Extensive experiments validate the performance of TGA-ZSR, which not only improves CLIP's zero-shot adversarial robustness but also maintains zero-shot clean accuracy on clean examples, gaining favorable balance.
Limitations. We simple text-guided attention mechanism by multiplying the text embedding and vision embedding which is effective against most attack types. However, for more challenging attacks such as AutoAttack, the improvement remains limited. This indicates that while our approach shows promise, it may require further refinement to enhance robustness under stronger adversarial scenarios.
Border Impact. Large-scale pre-trained vision-language models (VLMs) like CLIP [48] integrate vi- sual and textual data, revolutionizing applications such image classification, semantic segmentation, and vision question answering. While these models excel in zero-shot learning and transfer learning. they are vulnerable to adversarial attacks, posing risks in critical applications like autonomous vehi- cles and medical diagnosis. Adversarial training improves robustness but has practical challenges, including increased computational overhead and potential overfitting. Exploring zero-shot adversarial robustness is essential to ensure reliability.
Acknowledgement. This work supported by National Science and Technology Major Project under Grant 202 120001122000 12200, in part by the National Natural Science Foundation of China under Grants 62202331, U23A20387, 62036012, 62276118.
10

---

References
[1] Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16430-16441, 2022.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 101-mining discriminative components with random forests. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014 4, Proceedings, Part VI 13, pages 446-461 Springer, 2014.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leareer. In Advances in neural information processing systems, volume 33, pages 1877-1901, 2020.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages -9-57. Iece, 2017.
[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1000-3613, 2014.
[6] Adam Coates, Andrew Ng, and Honglak Lee. analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics. pages 215-223, 2011.
[7] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206 2216. PMLR, 2020.
[8] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages
2020.
[9] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. lcce, 2009.
[10] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: rectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
Pre-training of deep bidi-
[11] Minjing Dong and Chang Xu. Adversarial robustness via random projection filters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4077-4086, 2023.
[12] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
[13] Li Fei-Fei, Rob Fergus, Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEFF, 2004.
[14] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng and Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, (32(2):58--555, 2024.
[15] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-25 object category dataset. 2007.
[16] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma. Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 746-754, 2023.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[18] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 11222221-- 2019.

---

[19] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-8340000 8349, 2021.
[20] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[21] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904 4916. PMLR, 2021.
[22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 354-561, 2013.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Minjong Lee and Dongwoo Kim. Robust evaluation of diffusion--basee adversarial purification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 134-144, 2023.
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping banguage-maae pre- pre-training for unified langunn understanding and generation. In International conference on machine learning, pages 12900 PMLR, 2022.
[26] Junnan Li. Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in neural information processing systems, volume 34, pages 9694-9705, 2021.
[27] Lin Li, Jianing Qiu, and Michael Spratling. Aroid: Improving adversarial robustness through online instance-wise data augmentation. arXiv preprint ar06v:2306.077 077977 2023.
[28] Lin Li and Michael W Spratling. Data augmentation alone can improve adversarial training. In The Eleventh International Conference on Learning Representations, 2022.
[29] Shuxin Li, Xu Cheng, Fan Shi, Hanwei Zhang, Hongning Dai, Houxiang Zhang, and Shengyong Chen. A novel abbustness-enhancinn adversarial defense approach to ai-powered sea state estimation for autonomous marine vessels. IEEE Trunsactions on Systems, Man, and Cybernetics: Systems, 2024.
[30] Tang Li. Fengchun Qiao, Mengmeng Ma, and Xi Peng. Are data-driven explanations robust against out-of-distribution data? In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3821-3831, 2023.
[31] Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xiaomeng Li. Exploring visual interpretability for contrastive eegguage-image pre- pre-training. arXiv preprint arXiv:2209.0 07046, 2022.
[32] Jie Liu, Yixiao Zhang, Jic-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21152-21164, 2023.
[33] Yinhan Liu, Myle Ott. Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrviv /th::/900. 1907. 1692, 2019.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference computer vision, pages 10012-10022, 2021.
[35] Yanxiang Ma, Minjing Dong, and Chang Xu. Adversarial robustness through random weight sampling. In Advances in Neural Information Processing Systems, volume 36, pages 37657-37669, 2023.
[36] Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[37] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- ainenerained visual classification of aireraft. arXiv preprint 13055 2013.
12

---

[38] Chengzhi Mao, Scott Geng, Junfeng Yang. Xin Wang. and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In The Eleventh International Conference on Learning Representations, 2023.
[39] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipeap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0934., 2021.
[40] Seyed-Mohsen Soosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2574 2582, 2016.
[41] Weili Nie, Brandon Guo, Yujia Huang. Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. International Conference on Machine Learning, pages 16805-16822 PMLR, 2022.
[42] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes, In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[43] A Omkar M Parkhi. Sun database: Large-scale scene recognition from abbey to In 2010 IEEE computer society conference on computer vision and pattern recognition, pages
IEEE, 2010.
[44] Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bach- mann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5606-5611, 2023.
[45] Maria Parelli, Alexandros Delitzas, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bach- mann, and Thomas Hofmann. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5606-5611, 2023.
[46] Omkar M Parkhi, Andrea Vedaldi. Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 3498-3505. IEBE, 2012.
[47] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data, arXiv preprint arX1v:2001.07666, 2020.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 7763-.763. PMLR. 2021.
[50] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guide dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18082 18091, 2022.
[51] Christian Schlarmann, Naman Deep Singh. Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning vision embeddings for robust large vision- Langua- models. In International Conference on Machine Learning, 2024.
[52] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- localization. In Proceedings of the IEEE international conference on computer vision, pages 618 626, 2017.
[53] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advunces in neural information processing systems, volume 32, 2019.
[54] Christian Szegedy. Wojciech Zaremba, 1lya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
13

---

[55] Zhe Tao, Lu Yu, Hantao Yao, Shucheng Huang, and Changsheng Xu. Class incremental learning for light-weighted networks. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
[57] Bastiaan Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings. Part 11, pages 210 218. Springer, 2018
[58] Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. In Advances in neural information processing systems, volume 34, pages 237-250, 2021.
[59] Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trainee model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEFF conference on computer vision and pattern recognition, 2024.
[60] Dafeng Wei, Tian Gao, Zhengyu Jia, Changwei Cai, Chengkai Hou, Peng Jia, Fu Liu, Kun Zhan, Jingchen Fan, Yixing Zhao, et al. Bev-clip: Multi- modal bev retrieval methodology for complex scene in autonomous driving. arXiv preprint arXiv:2401. 000065, 2024.
[61] Dongxian Wu, Shu-Tac Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Advances in neural information processing systems, volume 33, pages 2958-2969, 2020.
[62] Zhaoyuan Yang, Zhiwei Xu, Jing Zhang, Richard Hartley, and Peter Adversarial purification with the manifold hypothesis. AAAI Conference on Artificial Intelligence, 2022.
[63] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- prompt tuning with knowledge-guid guided context optimization. In Proceedings of the IEEE/CVF conference on computer vision and paterr recognition, pages 6757-6767, 2023.
[64] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Finc-grained interactive language-image pre-training. In International Conference on Learning Representations, 2021.
[65] Lu Yu, Malvina Nikandrou, Jiali Jin, and Verena Rieser. Quality-agnostic image captioning to safely assist people with vision impairment. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6281-6289, 2023.
[66] Lu Yu and Verena Rieser. Adversarial textual robustness of visual dialog. In Findings of 61st Annual Meeting of the Association for Computational Linguistics 2023, pages 3422-3438. Association for Compu- tational Linguistics, 2023.
[67] Lu Yu, Zhe Tao, Hantao Yao, Joost Van de Weijer, and Changsheng Xu. Exploiting the semantic knowledge of pre-trained text-encoders for continual learning. arXiv preprint arXiv:2408. 010066 2024.
[68] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Fric Xing, Laurent El Ghaoui, and Michael Jordan. Theo- retically principled trade-off between rohustness and accuracy. In International conference on machine learning, pages 7472-7482, PMIR, 2019.
[69] Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4480-4488, 2016.
14

---

A Appendix
A.1 Experiments More Datasets.
Experiments on ImageNet_subsee We follow the state-of-the-an method PMG-AFT, which was fine-tuned on Tiny-ImageNet. In addition this, we further evaluate our method the ImageNet_subeet subset (a random selection of 100 classes from the full ImageNet dataset). The results are shown in Table and Table 10. For adversarial robustness, our approach achieves optimal results on several datasets and sub-optimal results on the remaining datasets, with overall performance that approximately 1% higher than the previous state-of of the-arr.. thte--ofthhe-at. performs worse than the generalization-foole focusee FARE but outperforms other methods.
terms of clean accuracy, our method
Table sero-shot robust accuracy on images attacked with 100 steps of PGD [36]. We performed several different methods ImageNet_sune subset and evaluated across 16 datasets. The optimal accuracy highlighted in bold, while the second-best accuracy is underlined.
PSYC. LLVC-Mrroal
CIPAR-I 10
CHAM-110
Flwresta
Callech 206
STX-10
SUN0399
Foudsuy
Calloch
PCAM
DIDD
Methods
Average
CLIP [48]
0.88
0.42
0.26
26
1.00
6.60
3.84
1.19
2.00
0.05
0.00
1.24
19.88
12,60
0.30
0.11
4.90
I:CAA [38]
51.76
16.38
32.03
13.67
8.76
23.59
20.42
FARE[S1]
499
13.00
33.16
11.41
PMG-AFT[[9]
23.93
IGA-ZSHlous)
19.11
24.74
Table 10: Zero-shot clean accuracy. We performed several different methods on ImageNet_ subset
and evaluated across 16 datasets.
Tiey Amageeer
FGYC- AVCC-Arresft
Noutn NoutanlCCrr
CHARIIO
Osfonipass
CDARK.0
STE-10
SUN-99977
EnnoSay
ImugoNar
PCAM
DTD
Methods
Average
CLIP [48]
57.36
88.06
60.45
97.04
57.26
83.89
K7.41
65.47
40.69
42.59
20.25
59.85
85.34
81.73
52,82
52.09
64.42
TeCoA
48.78
40.08
FARE(11]
55.68
$5.11
38.47
93.11
53.64
76.99
$3.88
27
18
31.33
1591
56.18
82266
78.27
44.92
50.00
PMGGAFFI[99]
53.66
36.95
30.58
43.70
47.61
23.66
1366
1023
40166
61.27
31.70
23.10
TOA-ZSR(omm)
Results on Diverse Datasets. In addition to validate the method on 16 datasets following the evaluation protocol of previous works, also conduct evaluations three additional datasets: ImageNet-A (natural adversarial examples) [20], ImageNet-O cout-of-distribution data) [20], and ImageNet-R (multiform form) [19]. Table 11 and Table 12, the clean accuracy on ImageNet-A is notably lower compared to ImageNet-O and ImageNet-R In general, other methods enhance robust accuracy by around 2%, they also incur a steep 20% drop in clean accuracy. In contrast, our approach achieves notable 5% improvement in zero-shot robust accuracy, although with a 16% decrease in clean accuracy, outperforming other methods. Although our method doesn't excel in zero-shot clean accuracy for ImageNet-O and ImageNet-R, it excels in optimizing zero-shot robust accuracy. Overall, our method yields the best results across these three datasets, as demonstrated by the Average.
Table 11: Zero-shot robust accuracy on images attacked with 100 steps PGD [36]. We performed different several methods on The optimal accuracy is highlighted in bold, while the second-best accuracy is underlined.
and evaluated in the following three additional datasets.
Methods
ImageNet-A
ImageNet-O
ImageNet-R
Average
CLIP [48]
0.08
0.60
5.57
2.08
FT-Clean
0.17
1,55
4.67
2.13
FT-Ady.
1,91
22.30
21.92
15.37
TeCoA [38]
1.67
23.55
23.85
16.36
FARE [51]
0.72
9.00
18.99
9.57
PMG-AFT [59]
2.35
27.70
27.61
19.22
TGA-ZSR (ours)
5.03
32.10
36.87
24.67
15

---

Table 12: Zero-shot clean accuracy. We performed different several methods on Imageeete and evaluated in the following three additional datasets. The optimal accuracy highlighted bold, while the second-best accuracy is underlined.
Methods
ImageNet-A
ImageNet-O
ImageNet-R
Average
CLIP [48]
29.49
46.05
63.61
46.38
FT-Clean
17.23
39.25
50.68
35.72
FT-Adv.
6,45
37.25
35.80
26.50
TeCoA [38]
6.04
42.50
40.65
29.73
FARE [51]
16.24
49.30
58.62
41.39
PMG-AFT [59]
6.19
42.55
40.81
29.85
TGA-ZSR (ours)
13.77
47.30
52.39
37.82
A.2 More Ablation Studies
Trade-off between and Following the protocol previous works (TeCoA [38], PMG-AFT [59]. FARE [51]), we fine-tuned the CLIP model on adversarial samples from single dataset (Tiny- ImageNet in our case) for 'adversarial fine-tuning' and subsequently evaluated performance across 15 datasets, including Tiny-ImageNe itself. Thus only need to tune hyperparameters on just the training dataset. We randomly selected 80% of the training set for training and the remaining 20% for validation to choose the hyperparameters. The validation set results shown in Table 13. The final results on the test set were obtained by training on the entire training set using the optimal hyperparameters (a=0.08, 3=0.05) identified from the validation set.
Table 13: Results on validation set of Tiny- ImageNet dataset.
Hyper-parameters
Robust
Clean
Average
0.07, 8 0.05
64.32
75.92
70.12
a 0.08, 8 0.04
47.25
76.20
61.72
a 0.08, 3 = 0.06
58.28
76.08
67.18
0.09, 3 0.05
46.20
76.10
61.15
Q 0.08, 3 0.05
64.01
77.79
70.90
After choosing the optimal syper-parameter on the validation the Tiny-ImageNet dataset, we proceeded to analyze the sensitivity of these hyper-parameters on the overall performance across 16 datasets, including the Tiny- Imageee and 15 additional datasets. This step was crucial for evaluating the robustness and generalizability of the model under different and Table demonstrate the results of different hyper- parameerr of and Eq. 8.
parameter settings. Table 14
Table 14: Zero-shot robust accuracy on images attacked with 100 steps of PGD [36]. We compared the results of different hyper-parameters on Tiny-ImageNNr and evaluated across 16 datasets. The optimal accuracy is highlighted in bold, while the second-best -best accuracy is underlined.
SGVC AivC-Alerade
CIAAR-10
CRARR-0 180
CaRtec- J01
Stadfeal Nen
STL.10
SUN0PP
Chiter
DTD
Ibper-parameters
Average
0-0.07, 2.005
62,22
$7.87
31.18
83.61
30.95
89
54,87
31.43
23.50
13:37
4.56
27.05
68.70
58.05
18.52
46.63
40.42
040.09 250
48.63
10.58
22.51
73.75
33.16
19.34
42.27
25.87
18.25
12.47
3.12
21.39
59.81
48.7
12.29
46.71
32.31
0-000
0-0.5
22.09
6-0.8
3-004
57.80
53.00
69.44
20.90
24.17
51.96
15.14
37.24
0-008
2-006
$7.29
52.30
27.17
3662
41.96
16

---

Table 15: Zero-shot clean accuracy. We compared the results of different hyper-parameicnt on
Mnn--ImaeNeet and evaluated across datasets. The optimal accuracy is highlighted bold, while the second-best accuracy is underlined.
PGVC-Aurend
CIAA--1
CIAA 100
STL-18
SUN009
Oxferdpets
EandAy
Callech 101
Calleck-250
Stanforok
DTD
Rpper-paramceers
640077 6-0.5
76,91
86.64
57.14
93.39
51.87
58.84
17.46
24.24
48.45
25.14
37,46
o.0000 (2-000
85.0
74.63
30.59
34.69
44 98
72.35
34.001
54.63
o-b0s (30.03
o:0.08 26006
56.80
08.13
040.08, 5-005
86.23
93.28
53.71
57.72
77.08
29.35
48.
74.38
37.73
56.48
Effeet of Each Component. Due to space constraints, detailed experiment on the effect of each component was not provided in Table 7. Therefore, we present here comprehensive experiment detailing the effect of each component. To explore the impact of each component on the final model performance, we conducted a series of ablation experiments to evaluate the effectiveness of each component. From the experimental results Table 16 and Table 17, it's clear that the inclusion of our text-guided attention components enhances CLIP's zero-shot robust accuracy and maintains its zero-shot clean accuracy.
Table 16: Zero-shot robust accuracy on images attacked with 100 steps of PGD. After adversarial fine-tuning the model using adversarial examples generated by PGD-2, we performed each component and evaluated across 16 datasets. The optimal accuracy is highlighted in bold.
RGYC ROrC-4
StauforaCarr
CIAA-110
CIARR000
577-10
SUN287
Fondo1
Odfordpass
Fimerdie
bougeNee
Caloedo 101
Calledh-
PCAM
DTD
Components
Average
Lps
22.59
16.76
2.64
72.49
22.01
22.10
18.09
11.62
3.06
11.47
31.71
+Lane
63.97
61.82
35.25
83,99
32.78
34.13
36.91
34.20
21.92
14.20
4.44
28.62
70.53
59.70
21.15
17.75
41.96
Table 17: Zero-shot clean accuracy. After adversarial fine-tuning the model using adversarial examples generated by PGD-2, verify the robustness of the model using adversarial examples generated by PGD-100. The optimal accuracy is highlighted in bold.
KGVE KGVE-AArraa
CIPAR-10
CIFAR1000
571-10
SEND09
Fooduut
Ortrelguss
Flomasl
LougeeNe
Calbech 101
Callahh256
Stonfrnt
PEAM
Components
Averuge
Fex
70.06
41.0
87.40
36.90
30.02
23.80
16.30
3.85
27
59.69
+Lar
76.71
77.91
47.67
90.1
44.17
39.46
69.25
39
29
26.28
18.14
8.04
39.55
76
76
66.95
29.00
48.95
19.96
LLase
76,85
96.23
56.55
93.28
51.71
57.72
77.08
48.32
29.15
23.99
11.03
48.10
$0.82
74.58
37.77
49.60
56.48
Effect of Distance Metries on Loss Function. Except l2, cosine and 4 are also frequently utilized distance metrics, We compared the performance of our method using these distance metrics. The results from Table 18 and Table 19 demonstrate that cosine and 1 exhibit similar performance but are inferior to 12, except for the zero-shot clear accuracy on PCAM. ln outperforms the other two distance measures by enhancing zero-shot adversarial robustness and zero-shot clean accuracy by approximately 12% and 1 11%, respectively. Thus we choose l distance to measure in our loss function.
Efffet of Learning Rate. Learning rate stands as a significant hyper-parameter in model training. Here we validate the effect of the learning rate for the experiment in Table 20 and Table 21. When the learning rate is set to 0.001, we observe the lowest values for both zero-shot robust accuracy and zero- shot clean accuracy. And, when we reduce the learning rate to 0.00001, we note that CLIP's zero-shot clean accuracy remains relatively stable, demonstrating the highest performance in this experiment. However, it affects CLIP's zero-shot robust accuracy, resulting a less favorable balance. Selecting a learning rate of 0.0001 achieved a well-balanced improvement, with the average of zero-shot robust accuracy and zero-shot clean accuracy increasing by 18.48% and 6.4 %, respectively, compared to the learning rates of 0.001 and 0.00001.
17

---

Table 18: Zero-shot robust accuracy images attacked with 100 steps of PGD [36]. We compared
the results of different distance metrics on Tiny-ImageNet and evaluated across 16 datasets. The
optimal accuracy is highlighted in bold.
Zap- Eeseehe
CDARR40
CIPAR-10
ST2-10
SEN009
Foodin
Kanuaa
Longs
Called 102
Called
Average
2.76
U5)
Table 19: Zero-shot clean accuracy. We compared the results of different distance metries on Tiny-
ImageNet and evaluated across 16 datasets. The optimal accuracy is highlighted in bold.
PUYC PUYCAliraD Alrrra
CHAAM-10
CIPAR 100
632.10
SUN209
Voodlut
Callohh 101
Calledh 20g
DTD
Distanee metries
Average
30.88
14.10
5.88
72.40
20.51
(3)
Table 20: Zero-shot robust accuracy on images attacked with 100 steps of PGD [36]. We compared
the results of different learning rates on Tiny-ImageNet and evaluated across 16 datasets. The optimal accuracy is highlighted in bold.
Thoy- Amnago Net
PGVC Ainvann
CIAM-18
CIAR-100
5TL-10
50N867
Footoo
Oxtoddeus
RUDNAT
AnggeNer
Calsec-101
Calhet-
PCAM
Learning
Average
0.001
52.19
37.27
18.10
61.36
12.21
8.42
26.74
10.4]
11.28
2.26
12.96
60.27
29 99.64
2.95
39.72
22.96
0,00001
26.15
20.67
11.
11.80
63.00
15:25
13.34
30.55
22.57
14.47
10.54
2.10
15.05
52.00
40.09
6.65
44.02
24.36
0.0001
63.97
61.82
35.25
83.99
32.78
34.13
56.91
34.30
21.92
14.20
28.02
70.53
59.70
21.15
47.75
41.96
Table 21: Zero-shot clean accuracy. We compared the results of different learning rates on Tiny- ImageNet and evaluated across 16 datasets. The optimal accuracy is highlighted in bold.
Tay ImagoNee
RGVC-4 Alrrrat
CIFAR-10
CIFAR-0
SUND0
Flowestt2
RanNSAT
Longekk
Collech-an 101
Calledd-
StanfortCCo
PCAM
Leuraing Rate
Average
0.001
75.32
69.83
37.96
82.45
28.17
21.68
50.45
22.93
19,10
13.04
4.23
25.48
59.84
48.18
8.73
49 18
38.54
0.00081
68.58
87.49
60.14
94.68
56.97
72.92
81.55
58.55
33.94
35.09
16.36
54.89
83.15
80.07
46.60
49
61.26
0.0001
76.83
86.23
56
35
93.28
31.71
57.72
77.08
48.33
29.15
23.99
12.03
48.10
80.82
74.58
37,72
49.60
56.18
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we provide a clear explanation of the mo- tivation behind approach. At the conclusion of the introduction, enumerate the contributions of this paper, etc. We provide details in Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our approach entails certain limitations, which we acknowledge in Section and commit to addressing through subsequent research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: Our approach emphasizes applied aspects and does not need theory assump- tions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Section 3, we introduce our methodology. while in Section 4.1, we outline the haper-parameters employed for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide open access to the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //wpp..ccppubaiii wttpccccpppppcpcceeeeeeecc h
/Cooeeubeeeeeeeeeeee seabbbmmssiiooo ssionnollll onPoicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4.1, we introduce the experimental setup, including the hyper- parameters utilized in this paper. Furthermore, section A, we present highly detailed ablation experiments to enhance comprehension of the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We conduct multiple experiments report the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4.5, we provide comprehensive details on the computational re- sources used and the duration of the training process.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tttps:///eurpp.... http:///wwurrpp.ccpppe.eeee cc/public htc/public/Ehhiss@ieeeee esQuidelinest
Answer: [Yes]
Justification: The research conducted in the paper adheres to the NeurlPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed in Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The model and dataset employed in our study are widely recognized and do not exhibit any inherent issues.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The models, methodologies, datasets, and other elements used are appropriately aligned and referenced throughout the paper.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "QWSUuE6OrL", "reviewer": "Reviewer_w4iU", "summary": "The paper proposes a framework, Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR), to enhance the robustness of vision-language models (VLMs) against adversarial attacks. The proposed method incorporates two modules: the Attention Refinement module and the Attention-based Model Constraint module. The Attention Refinement module aligns the text-guided attention of adversarial examples with clean examples, while the Attention-based Model Constraint module maintains the model’s performance on clean samples.", "questions": "1. What is the additional computational overhead introduced by the Attention Refinement and Attention-based Model Constraint modules? How does this impact training and inference times compared to baseline methods?\n2. How well does the proposed method generalize to other types of vision-language tasks beyond image classification, such as image captioning or visual question answering?\n3. Why is the proposed method fine-tuned on Tiny-ImageNet rather than ImageNet? Both TeCoA [1] and FARE [2] are trained on ImageNet. \n\n[1] Mao, C., Geng, S., Yang, J., Wang, X., & Vondrick, C. (2022). Understanding zero-shot adversarial robustness for large-scale models. \n\n[2] Schlarmann, C., Singh, N. D., Croce, F., & Hein, M. (2024). Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models.", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of text-guided attention allows to enhance zero-shot robustness.\n2. Extensive evaluation of the approach was conducted across diverse datasets.\n3. The proposed method outperforms current state-of-the-art techniques in both zero-shot robust accuracy.\n4. The paper provides a thorough analysis of the impact of adversarial attacks on text-guided attention, offering insights into the model's decision-making process.\n5. The method improves robustness without significantly sacrificing performance on clean data, achieving a favorable trade-off.", "weaknesses": "1. The incorporation of text-guided attention mechanisms and multiple modules may increase the complexity of implementation and computational overhead.\n2. Although the authors claim zero-shot robustness in Vision-Language Models, the experiments and methods only target the CLIP model.\n3. Missing limitation in Section 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Discussion of additional computational overhead and training/inference time.**\n\n**A1**:Thank you for your suggestion. We have evaluated our method against others in terms of memory usage, training time, and test time, and the findings are summarized below:\n\n**Memory Usage**: Our method increases memory consumption by approximately 15% compared to the state-of-the-art method PMG-AFT. This is due to the additional computation required for the text-guided attention map.\n\n**Training Time**: The training time for our method is comparable to that of PMG-AFT, which utilizes a KL divergence constraint on logits.\n\n**Test Time**: The test time remains consistent across all methods.\n\nTable 1: Comparison of memory usage, training time, and test time.\n\n| Method  | Train memory usage\t|Train time (per epoch/batch) | Test time (per batch) |\t\n|---------|-----------------------------------|-----------|-----------|\n| CLIP    \t|\t0Mb \t |   \t0s / 0s                     | 21s       | \n| TeCoA   \t| \t12873Mb | 512s / 0.65s                   | 21s       | \n| PMG-AFT \t| \t18449Mb |\t828s / 1.06s                  | 21s       | \n| Ours    \t|\t21227Mb | \t885s /1.13s                    | 21s       | \n\n**Q2:  Applying to other types of vision-language tasks on other tasks.**\n\n**A2**: We follow TeCoA and PMG-AFT, focusing on improving the zero-shot adversarial robustness of the CLIP model for classification tasks. To further validate the effectiveness of our method as suggested by the reviewer, we replaced the CLIP model with another vision-language model, **OpenFlamingo-3B**[a]. In this setup, ViT-L/14 serves as the vision encoder and MPT-1B as the language encoder. Additionally, we evaluated our method on two other tasks: image captioning and visual question answering (VQA). We report the CIDEr score for image captioning and VQA accuracy for visual question answering tasks. We employ the APGD attack[b] with a strength of epsilon 8/255 for 10 iterations. The results are shown below. Our method outperforms FARE in most scenarios for both image captioning and VQA tasks across a range of datasets.  We believe that with task-specific design enhancements, our results can be further improved.\n\nTable 2: CIDEr Scores for Image Captioning Task with OpenFlamingo-3B.\n\n|        |       | COCO |   Flickr30k \t| \n|---------|-------|------------|-----------|\n| Robust | FARE |  \t3.68  |   \t2.71    \t|  \n|        | Ours  |  \t**4.13** |   \t**2.90**    \t|   \n|  Clean | FARE |  \t3.09  |   \t3.02    \t|   \n|        | Ours  |  \t**3.56**  |   **3.13**   \t| \n\nTable 3: Accuracy for Visual Question Answering (VQA) with OpenFlamingo-3B.\n\n|        |       \t| TextVQA |  VQAv2 \t|      \n|---------|-------|------------|-----------|     \n| Robust | FARE |  \t 3.58 |     31.88   \t|               \n|        | Ours  |    **4.44** |   **32.14**   \t|               \n|  Clean | FARE |  \t  3.40  |  \t**35.38**   \t|              \n|        | Ours  |  \t  **4.38** |   \t34.72 \t|  \n           \n[a] Awadalla, A., et al. OpenFlamingo: an opensource framework for training large autoregressive vision-language models. arXiv:2308.01390, 2023.\n\n[b] Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.\n \n**Q3: Missing limitation.**\n\n**A3**: We mentioned the limitation in the final sentence of Section 5. We will emphasize and expand upon it in future versions.\n\n**Q4: Train on ImageNet dataset.**\n\n**A4**: We follow the state-of-the-art method PMG-AFT, which was fine-tuned on Tiny-ImageNet. Thank you for your suggestion. Due to time constraints, we further evaluate our method on the ImageNet_subset (a random selection of 100 classes from the full ImageNet dataset). The results are shown below:\n\nTable 4: Zero-shot adversarial robust accuracy and clean accuracy across 16 datasets by training on ImageNet_subset.\n\n| Test | Robust | Clean | Average |\n|-------|-------|-------|-------|\n| CLIP | 4.90 | 64.42 | 34.66 |\n| TeCoA | 20.42 | 40.68 | 30.55 |\n| FARE | 11.41 | 60.00 | 35.70 |\n| PMG-AFT | 23.93 | 43.10 | 33.51 |\n| Ours | 24.74 | 46.90 | **35.82** |\n\n**We will revise and add the corresponding context in the final version.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dKlArwnOEK", "reviewer": "Reviewer_qndz", "summary": "This paper proposes an approach to improve the adversarial robustness of vision-language models while maintaining performance on clean images. The key idea is to aligns the attention maps of adversarial examples with clean examples. Extensive experiments demonstrate the effectiveness of the proposed method in zero-shot adversarial robustness across multiple datasets while maintaining high clean accuracy.", "questions": "1. How sensitive is the method to the choice of hyperparameters? Is there a principled way to select these across different datasets or tasks? \n\n2. Are there any scenarios or types of adversarial attacks where this method performs poorly? What are the limitations of this approach?", "limitations": "The authors did not explicitly address the limitations such as the types of adversarial attacks where this method performs poorly.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is simple, straightforward and effective. It leverages text-guided attention to significantly enhance zero-shot adversarial robustness while maintaining clean accuracy across diverse datasets.\n\n2. The experiments are comprehensive. The results on 16 datasets demonstrate consistent improvements over baseline methods.", "weaknesses": "1. The method is only demonstrated on CLIP, raising questions about its applicability to other vision-language models or architectures.\n\n2. The method introduces additional hyperparameters without a clear strategy for tuning them across different datasets or tasks. \n\n3. No error bars or statistical significance tests reported for the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Applicability to other vision-language models.**\n\n**A1**: We follow TeCoA and PMG-AFT, focusing on improving the zero-shot adversarial robustness of the CLIP model for classification tasks. To further validate the effectiveness of our method as suggested by the reviewer, we replaced the CLIP model with another vision-language model, **OpenFlamingo-3B**[a]. In this setup, ViT-L/14 serves as the vision encoder and MPT-1B as the language encoder. Additionally, we evaluated our method on two other tasks: image captioning and visual question answering (VQA). We report the CIDEr score for image captioning and VQA accuracy for visual question answering tasks. We employ the APGD attack[b] with a strength of epsilon 8/255 for 10 iterations. The results are shown below. Our method outperforms FARE in most scenarios for both image captioning and VQA tasks across a range of datasets.  We believe that with task-specific design enhancements, our results can be further improved.\n\nTable 2: CIDEr Scores for Image Captioning Task with OpenFlamingo-3B.\n\n|        |       | COCO |   Flickr30k \t| \n|---------|-------|------------|-----------|\n| Robust | FARE |  \t3.68  |   \t2.71    \t|  \n|        | Ours  |  \t**4.13** |   \t**2.90**    \t|   \n|  Clean | FARE |  \t3.09  |   \t3.02    \t|   \n|        | Ours  |  \t**3.56**  |   **3.13**   \t| \n\nTable 3: Accuracy for Visual Question Answering (VQA) with OpenFlamingo-3B.\n\n|        |       \t| TextVQA |  VQAv2 \t|      \n|---------|-------|------------|-----------|     \n| Robust | FARE |  \t 3.58 |     31.88   \t|               \n|        | Ours  |    **4.44** |   **32.14**   \t|               \n|  Clean | FARE |  \t  3.40  |  \t**35.38**   \t|              \n|        | Ours  |  \t  **4.38** |   \t34.72 \t|  \n           \n[a] Awadalla, A., et al. OpenFlamingo: an opensource framework for training large autoregressive vision-language models. arXiv:2308.01390, 2023.\n\n[b] Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.\n\n**Q2: Strategy for tuning hyper-parameters.**\n\n**A2:** Following the protocol of previous works (TeCoA,PMG-AFT,FARE), we fine-tuned the CLIP model on adversarial samples from a single dataset (Tiny-ImageNet in our case) for `adversarial fine-tuning' and subsequently evaluated its performance across 15 datasets, including Tiny-ImageNet itself. Thus we only need to tune hyperparameters on just one training dataset. We randomly selected 80% of the training set for training and the remaining 20% for validation to choose the hyperparameters. The validation set results are shown below. The final results on the test set were obtained by training on the entire training set using the optimal hyperparameters (alpha=0.08, bata=0.05) identified from the validation set.\n\nTable 5 : Results on validation set of Tiny-ImageNet dataset.\n\n| Hyper-parameters | Robust | Clean | Average |\n|-------|-------|-------|-------|\n|   alpha=0.07 beta=0.05  |  64.32  |  75.92  |  70.12  |\n|   alpha=0.08 beta=0.04  |  47.25  |  76.20  |  61.72  |\n|   alpha=0.08 beta=0.05  |  64.01  |\t  77.79\t |  **70.90** |\n|   alpha=0.08 beta=0.06  |  58.28  |  76.08  |  67.18  |\n|   alpha=0.09 beta=0.05  |  46.20  |  76.10  |  61.15  |\n\n**Q3: Reporting error bars or statistical significance tests.**\n\n**A3**: Thank you for your suggestions. We acknowledge the absence of reporting error bars and statistical significance tests in the initial submission. To address this, we have included the standard deviation  from three runs of the main experiments to demonstrate the stability of our method. We will include comprehensive statistical significance tests in the subsequent versions of the paper.\n\nTable 6: Results with standard deviation  from three runs across 16 datasets.\n\n|  Test  | Tiny- |  C.10  |  C.100 |   STL  |   SUN   |   Food  | Pets | Flowers |   | \n|------|-------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| Robust |   63.95±0.11  | 61.45±0.67 | 35.27±0.07 | 84.22±0.21 | 33.22±0.39 | 33.97±0.20 | 57.75±0.76 | 34.55±0.35 |  | \n|  Clean |   75.72±0.12  | 86.46±0.26 | 56.52±0.35 | 93.48±0.19 | 51.99±0.25 | 57.59±0.34 | 77.32±0.30 | 48.08±0.37 |  | \n\n| | DTD    |   EuroS.  | Airc.|  ImageN.  | Ca.101 | Ca.256 | Cars |    PCAM    |   Avg.  |\n|------|-------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| Robust |22.08±0.16 | 14.27±0.26 |   4.75±0.27   | 28.74±0.11 |  70.97±0.42 |  60.06±0.46 |  20.40±0.68  | 47.76±0.35 | 42.09±0.12 |\n|  Clean | 29.06±0.35 | 24.24±0.49 |   11.93±0.27  | 48.04±0.06 |  80.70±0.29 |  74.74±0.18 |  36.62±1.03  | 49.58±0.17 | 56.44±0.08 |\n\n**Q4: Adversarial attack scenarios where this method performs poorly and its limitations.**\n\n**A4**: In this paper, we primarily train using the PGD attack and validate our approach on both PGD and AutoAttack. Our method significantly outperforms other methods on the PGD attack and achieves comparable results with state-of-the-art methods on AutoAttack, which integrates multiple attack strategies (APGD_CE, APDG_DLR, FAB, Square Attack). These results suggest that our method is robust but could be further enhanced to withstand stronger and more complex attacks. One potential improvement is to design a more robust text-guided attention mechanism.\n\n**We will revise and add the corresponding context in the final version.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "09tk5oruhT", "reviewer": "Reviewer_BFe6", "summary": "This work studies the robustness to adversarial samples for CLIP. Inspired by an observation that adversarial perturbations induce shifts in text-guided attention, the work proposes a simple yet effective approach to improve the zero-shot robostness, i.e., align the text-guided attention of clean samples and adversarial samples. The state-of-the-art performance across 16 datasets demonstrate the effectiveness of the proposed method.", "questions": "1. How do you utilize the class token in the ViT backbone of CLIP? \n2. In Tables 13 and 14, why the losses L1 and L2 have very different performance?", "limitations": "The paper has discussed the limitations and potential impacts of the work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work focuses on an interesting topic, which involves the robustness of VLMs. To improve the zero-shot robustness of CLIP, the work proposes a text-guided attention-based method, which is simple but intuitive and effective. Extensive experiments on 16 datasets demonstrate the effectiveness of the proposed method. Additionally, the paper is well written and the experiments are solid.", "weaknesses": "I would like to see more comparison with other types of attention, e.g., vision-only self-attention. Does the proposed method works because of the attention mechanism or the text guidance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More comparison with other types of attention.**\n\n**A1**: We also considered this point when exploring the comprehensive effect of attention. Thus, we conducted an experiment by replacing the text-guided attention with vision-only attention, as detailed in the first part of Section 4.4 (\"Different types of attention\"). We used Grad-CAM to generate the vision-only attention map. The results in Table 4 demonstrate that vision-based attention already achieves results comparable to the state-of-the-art method PMG-AFT in terms of both zero-shot adversarial robustness accuracy and clean accuracy, although lower than those achieved with text-guided attention. This indicates that the zero-shot adversarial robustness of vision-language models benefits from the constraints of attention mechanisms, with text-guided attention further enhancing performance. We will explore more types of attention in future analysis.\n\n**Q2: How to use class token?**\n\n**A2**: In Figure 2, the class token used as  $f(x_a)$ is obtained from $f_g^{tar}(x_a)$ after pooling. We will clarify this in more detail in future versions.\n\n**Q3: In Tables 13 and 14, why the losses L1 and L2 have very different performance.**\n\n**A3**: We investigated the use of L1 and L2 norms to constrain the text-guided attention map. The results indicated that the L2 norm significantly outperforms the L1 norm. The primary reason for this is that the L2 norm, tends to distribute the penalty more evenly across all elements of the attention map. This leads to smoother and more stable attention distributions, which can enhance the model's adversarial robustness.\n\nIn contrast, the L1 norm, tends to produce sparser solutions by penalizing non-zero elements more heavily. While this can be beneficial in certain contexts by promoting sparsity, it may lead to less stable and less coherent attention maps in the case of text-guided attention, ultimately degrading performance.\n\n**We will revise and add the corresponding context in the final version.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57xbB278yD", "reviewer": "Reviewer_mEBo", "summary": "The paper focuses on enhancing the robustness of CLIP models against adversarial perturbations. The approach utilizes saliency maps generated by the inner products between image and text embeddings. Two regularization terms are introduced: the first aims to align the saliency maps of the original and its adversarial examples, while the second seeks to match the saliency maps of the original example between the pre-trained CLIP model and the target model. Experimental results indicate that fine-tuning CLIP with these two regularization terms improves its robustness to attacks.", "questions": "See weaknesses.", "limitations": "The primary limitation of the paper is that the experimental results presented do not sufficiently support its claims. I recommend that the authors either revise the claims to align more closely with the data presented or extend the experimental section to provide additional evidence supporting their assertions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1.\tThe paper is well-written and organized, with clear descriptions of the motivation and methods.\n2.\tTo the best of my knowledge, enhancing CLIP robustness by aligning the saliency maps produced through inner products between image and text embeddings, termed \"Text-Guided Attention\" in the paper, is a novel approach.\n3.\tIt is good to see the detailed ablation analysis.", "weaknesses": "1.\tThe abstract claims that the goal is to enhance both the generalization and robustness of the CLIP model; however, the experimental results do not fully support this assertion. Specifically, regarding generalization, Table 2 shows a significant decrease in clean accuracy compared to the original pre-trained CLIP model. I kindly ask the authors to use the terms \"generalizability\" and \"adversarial robustness\" more carefully within the paper. Typically, the generalizability of CLIP models refers to their zero-shot classification performance across different datasets. It may be argued that the proposed method improves the adversarial robustness of CLIP models while maintaining relatively good generalizability compared to previous adversarial training methods. However, the authors must acknowledge the decrease in generalizability compared to original CLIP model as clearly evidenced by the results in Table 2.\n\n2. Since the training uses adversarial examples produced by PGD, it is not surprising that the proposed method results in a model with good robustness to PGD attacks, as shown in Table 1. However, when the evaluation comes to AutoAttack, the advantage of the TGA-ZSR over the previously best method is marginal as shown in Table 3. Could the authors provide explanations for this point, and could they demonstrate the robustness of the resulting models to more attacks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Use the terms \"generalizability\" and \"adversarial robustness\" more carefully.**\n\n**A1**: Thank you for pointing that out! The reviewer is correct. To balance the performance between clean and adversarial samples, our goal is to maintain the generalization and enhance the adversarial robustness of the original CLIP model. We will modify it in the revised version of the paper.\n\n**Q2: The advantage of the proposed method is less on AutoAttack than PGD.**\n\n**A2**: In the paper, we train our model using the PGD attack and validate our approach on both PGD and AutoAttack. Thus, testing on AutoAttack serves as a cross-validation experiment. Additionally, AutoAttack combines multiple attack strategies, effectively covering a wide range of defense techniques and model architectures. It can adapt to the characteristics and defense strategies of the target model, making its attacks more powerful. For these reasons, AutoAttack generally results in lower adversarial accuracy compared to methods like PGD, and improving model performance against AutoAttack can be challenging. Consequently, our method shows less improvement on AutoAttack, but we still achieved state-of-the-art performance.\n\n**Q3: Results on more attacks.**\n\n**A3**: To further evaluate the effectiveness of the proposed method, as suggested by the reviewer, we conducted experiments using another type of attack, specifically the CW attack [c]. The results, shown below, demonstrate that our method significantly outperforms the state-of-the-art method PMG-AFT on both adversarial and clean samples. This substantial margin indicates the adversarial robustness of our proposed method.\n\nTable 7: Zero-shot adversarial robust accuracy and clean accuracy across 16 datasets  with **CW attack**.\n\n|  Test  |Methods | Tiny- |  C.10  |  C.100 |   STL  |   SUN   |   Food  | Pets | Flowers |     DTD    |   EuroS.  | Airc.|  ImageN.  | Ca.101 | Ca.256 | Cars |    PCAM    |   Avg.  |\n|:------:|:-------:|:-------------:|:--------:|:---------:|:------:|:------:|:-------:|:----------:|:----------:|:-----:|:-------:|:-------------:|:--------:|:-----------:|:-----------:|:------------:|:-----:|:-------:|\n|        |   CLIP  |      0.21     |   0.36   |    0.10   |  10.59 |  1.16  |   0.82  |    1.23    |    1.09    |  2.18 |   0.01  |      0.00     |   1.14   |    13.50    |     7.36    |     2.36     |  0.07 |   3.64  |\n| Robust | PMG-AFT |     44.59     |   44.86  |   24.15   |  74.11 |  19.99 |  17.33  |    39.88   |    20.95   | 13.51 |  12.09  |      1.47     |   19.51  |    60.99    |    44.46    |     10.57    | 48.59 |  31.07  |\n|        |   Ours  |     63.85     |   60.50  |   34.62   |  84.11 |  22.03 |  33.28  |    58.33   |    32.95   | 21.22 |  13.89  |      4.56     |   20.42  |    70.34    |    59.73    |     20.20    | 48.02 |  40.50  |\n| \n|        |   CLIP  |     57.26     |   88.06  |   60.45   |  97.04 |  57.26 |  83.89  |    87.41   |    65.47   | 40.69 |  42.59  |     20.25     |   59.15  |    85.34    |    81.73    |     52.02    | 52.09 |  64.42  |\n|  Clean | PMG-AFT |     66.98     |   74.50  |   44.66   |  88.95 |  37.31 |  37.42  |    66.12   |    35.65   | 21.49 |  18.02  |      4.53     |   35.92  |    76.82    |    61.95    |     25.00    | 49.97 |  46.58  |\n|        |   Ours  |     76.99     |   86.24  |   56.11   |  93.66 |  47.12 |  56.80  |    77.65   |    47.10   | 28.94 |  24.64  |     11.52     |   43.89  |    80.65    |    74.76    |     35.67    | 49.78 | 55.72  |\n\n[c] Carlini, Nicholas and David A. Wagner. “Towards Evaluating the Robustness of Neural Networks.” 2017 IEEE Symposium on Security and Privacy (SP) (2016): 39-57.\n\n**We will revise and add the corresponding context in the final version.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**1. Summary**: We thank the reviewers for their positive and constructive comments. The reviewers agree that the topic is interesting and that the proposed method is novel, simple, and effective. All reviewers appreciate the comprehensive, solid, thorough, and detailed experiments. They also acknowledge that the paper is well-written and organized. The detailed feedback is as follows:

| Reviewers                 | Aspects        | Comments                                                                               |
|---------------------------|----------------|----------------------------------------------------------------------------------------|
| BFe6                      | Topic          | Interesting                                                                            |
| mEBo & qndz & BFe6        | Method         | Novel; simple, straightforward and effective; simple but intuitive and effective       |
| w4iU & qndz & BFe6 & mEBo | Experiments    | Thorough analysis, offering insights; comprehensive; solid; detailed ablation analysis |
| BFe6 & mEBo               | Representation | Well written; well written and organized, clear descriptions of motivation and method |

The reviewers' major comments suggest that additional analysis could provide a deeper understanding of the proposed method. Specifically, they recommended including: Analysis of additional computational overhead / Application to other types of vision-language models / Results on more attacks / Discussion of scenarios where our method performs poorly. **Note that the primary analyses widely used in existing zero-shot adversarial robustness methods for VLMs have already been provided in the paper and supplementary materials**. The additional analyses suggested by the reviewers are complementary and would enhance the understanding of our proposed method. 

**2. More analysis of the proposed method**:

- *Additional computational overhead and training/inference time of the models are shown in Table 1.*

Table 1: Comparison of memory usage, training time, and test time.

| Method  | Train memory usage	|Train time (per epoch/batch) | Test time (per batch) |	
|---------|-----------------------------------|-----------|-----------|
| CLIP    	|	0Mb 	 |   	0s / 0s                     | 21s       | 
| TeCoA   	| 	12873Mb | 512s / 0.65s                   | 21s       | 
| PMG-AFT 	| 	18449Mb |	828s / 1.06s                  | 21s       | 
| Ours    	|	21227Mb | 	885s /1.13s                    | 21s       | 

- *The results with OpenFlamingo-3B on image captioning and VQA are shown in Table 2 and 3.*

Table 2: CIDEr Scores for Image Captioning Task with OpenFlamingo-3B.

|        |       | COCO |   Flickr30k 	| 
|---------|-------|------------|-----------|
| Robust | FARE |  	3.68  |   	2.71    	|  
|        | Ours  |  	**4.13** |   	**2.90**    	|   
|  Clean | FARE |  	3.09  |   	3.02    	|   
|        | Ours  |  	**3.56**  |   **3.13**   	| 

Table 3: Accuracy for Visual Question Answering (VQA) with OpenFlamingo-3B.

|        |       	| TextVQA |  VQAv2 	|      
|---------|-------|------------|-----------|     
| Robust | FARE |  	 3.58 |     31.88   	|               
|        | Ours  |    **4.44** |   **32.14**   	|               
|  Clean | FARE |  	  3.40  |  	**35.38**   	|              
|        | Ours  |  	  **4.38** |   	34.72 	|

- *The results training on ImageNet_subset are shown in Table 4.*

Table 4: Zero-shot adversarial robust accuracy and clean accuracy across 16 datasets by training on ImageNet_subset.

| Test | Robust | Clean | Average |
|-------|-------|-------|-------|
| CLIP | 4.90 | 64.42 | 34.66 |
| TeCoA | 20.42 | 40.68 | 30.55 |
| FARE | 11.41 | 60.00 | 35.70 |
| PMG-AFT | 23.93 | 43.10 | 33.51 |
| Ours | 24.74 | 46.90 | **35.82** |

- *The results on the validation set of Tiny-ImageNet are shown in Table 5.*

Table 5 : Results on validation set of Tiny-ImageNet dataset.

| Hyper-parameters | Robust | Clean | Average |
|-------|-------|-------|-------|
|   alpha=0.07 beta=0.05  |  64.32  |  75.92  |  70.12  |
|   alpha=0.08 beta=0.04  |  47.25  |  76.20  |  61.72  |
|   alpha=0.08 beta=0.05  |  64.01  |  77.79. |  **70.90** |
|   alpha=0.08 beta=0.06  |  58.28  |  76.08  |  67.18  |
|   alpha=0.09 beta=0.05  |  46.20  |  76.10  |  61.15  |

- *The results included the standard deviation  from three runs of the main experiments are shown in Table 6.*

(Table 6 is available in the response to Reviewer qndz due to character limit constraints and is not shown here.)

- *Results with CW attack are shown in Table 7.*

Table 7: Zero-shot adversarial robust accuracy and clean accuracy across 16 datasets  with **CW attack**.

|   |Methods |  Avg.  |  |Methods |  Avg.  |
|:------:|:-------:|:-------------:|:-----:|:-------:|:-------------:|
|        |   CLIP  |       3.64  |        |   CLIP  |      64.42  |
| **Robust** | PMG-AFT |    31.07 |    **Clean** | PMG-AFT |    46.58  |
|        |   Ours  |     40.50  |         |   Ours  |    55.72  |

**3. More explanation of the method.**

- *Results on AutoAttack*: In this paper, we primarily train using the PGD attack and validate our approach on both PGD and AutoAttack. Our method significantly outperforms other methods on the PGD attack and achieves comparable results with state-of-the-art methods on AutoAttack, which integrates multiple attack strategies. These results suggest that our method is robust but could be further enhanced to withstand stronger and more complex attacks. 

- *How to use class token*: In Figure 2, the class token used as  $f(x_a)$ is obtained from $f_g^{tar}(x_a)$ after pooling. We will clarify this in more detail in future versions.

- *"Generalizability" and "adversarial robustness"*: Our goal is to maintain the generalization and enhance the adversarial robustness of the original CLIP model. We will modify it in the revised version of the paper.

Meta-review: The paper introduces a method to improve the robustness of visual language pretrained models. Building on insights from text-guided attention, the authors propose two modules—the Attention Refinement module and the Attention-based Model Constraint module—to enhance model robustness. While the reviewers raised concerns about additional comparisons, complexities, and unclear wording, the AC found that the rebuttal effectively addressed all these issues. In light of this, the AC recommends acceptance. The authors are encouraged to incorporate all changes and additional experimental results into the final camera-ready version of the paper.
Decision: Accept (poster)