Venue: NAACL 2025

Parameter-free and Accessible Prompt Learning to Enhance Adversarial Robustness for Pre-trained Vision-Language Models

Xingran Zhou1, Kun Yang1,2, Changtao Miao1,3, Bingyu Hu1,3, Zhuoer Xu1, Shiwen Cui1, Changhua Meng1, Dan Hong1,

###### Abstract

Large pre-trained Vision-Language Models (VLMs) have revolutionized both computer vision and natural language processing. Despite their success, adversarial examples can still mislead VLMs into producing incorrect results. This work focuses on boosting the adversarial robustness of VLMs by searching for text prompts at the word level, rather than optimizing continuous textual embeddings. We introduce **P**arameter-**F**ree **P**rompt **T**uning (PFPT) to learn defense words that enhance resilience against adversarial attacks when appended to existing prompts, thereby offering ease of use due to the simplicity of this approach. These defenses works are naturally present in the inherent vocabulary of VLMs, providing a human-readable property. PFPT employs a coarse-to-fine strategy with carefully designed optimization objectives to guide the word search. Extensive experiments demonstrate our method's superiority over hand-engineered prompts and other state-of-the-art methods. PFPT significantly boosts accuracy and robustness, outperforming hand-engineered prompts with average gains of +4.9% and +5.8%, respectively (\(\epsilon{=}\,1/255\)).

## 1 Introduction

Large pre-trained Vision-Language Models (VLMs), such as CLIP Radford et al. (2021), ALIGN Jia et al. (2021), and BLIP Li et al. (2022), have revolutionized downstream vision-language tasks including classification, object detection, and segmentation. More applications Qiu et al. (2023); Zhang et al. (2024); Li et al. (2024) are increasingly being developed based on these foundational general-purpose models.

However, despite the capabilities of VLMs, their vulnerabilities have been exposed. Recent studies Inkawhich et al. (2023); Zhao et al. (2024) show that VLMs are particularly susceptible to small adversarial noise -- these models predict with significant deviation from reality when subjected to deliberately designed, human-imperceptible perturbations to images. These exposed security risks continue to raise concerns within the community.

Previous efforts to enhance the adversarial robustness of VLMs primarily can be categorized two types: adversarial training Madry et al. (2017) and parameter-efficient methods Li et al. (2024); Chen et al. (2023); Huang et al. (2023). Adversarial training involves fine-tuning models with adversarial examples and incorporating specifically designed loss functions. Although these methods can enhance adversarial robustness to some extent, the cost of fine-tuning is substantial. Particularly for contemporary Transformer-based models Li et al. (2022); Radford et al. (2021) with millions of parameters, fine-tuning these parameters requires significant time and computational resources. Moreover, introducing adversarial examples into training may compromise the original generalization ability of VLMs Kumar et al. (2022), potentially affecting the zero-shot capabilities. Parameter-efficient methods freeze the parameters of the model while training additional textual or visual prompts, often in the form of embeddings. Although these approaches enhance learning efficiency, the costs associated with using additional prompts are significant. For

Figure 1: An overview of the proposed method: Appending the learned defense prompt **(at the word level)** to the original prompt significantly enhances resilience against adversarial attacks.

 

instance, adding adversarial visual prompts (Chen et al., 2023) to the original image may remove essential edge details, resulting in partial information loss. Meanwhile, adversarial text prompts Li et al. (2024) learn text embeddings that are challenging to use in practice, as their application requires access to the model's embedding layer and precise manipulation to integrate the learned embeddings. This reduces accessibility and increases operational costs. Optimization in the continuous embedding space may also introduce randomness, due to the unconstrained nature of the optimization space.

To address these challenges, we propose a parameter-free method for learning appendable text prompts. This strategy focuses on identifying and discovering defense tokens from the pre-trained and fixed vocabulary Sennrich et al. (2015) of VLMs. These defense tokens are simply added as prefixes or suffixes to the original prompt and then fed into the model. As there are no embedding parameters that need to be saved, this method is inherently parameter-free. This approach constrains the search space to a limited selection space, improving search efficiency. It also enhances accessibility, as the search results are simply included as part of the new input text prompt, without manipulating the intermediate embeddings.

This work studies the problem of parameter-free and accessible text prompt tuning of pre-trained VLMs for adversarial robustness. Previous research on text prompt learning has primarily focused on optimizing text embeddings, with little exploration into word level prompt learning. These approaches reveal a gap between convenient usability and flexibility. Moreover, the learned embeddings do not correspond to natural language words and are not found within the inherited vocabulary's embeddings. Unlike previous methods, our work is inherently parameter-free as it identifies using natural language words that require no additional parameter storage. We introduces a new paradigm for text prompt learning in boosting adversarial robustness for VLMs. We concentrate on and apply this paradigm to a widely used type of VLMs -- CLIP Radford et al. (2021), as it is representative of vision-language foundation models and is extensively utilized across various downstream tasks Rombach et al. (2022).

To improve the accessibility of adversarial robustness in text prompting, we introduce **P**arameter-**F**ree **P**rompt **T**uning (PFPT), a method to learn a robust text prompt at the word level for CLIP, leveraging adversarial examples to enhance the resilience against adversarial attacks. The defense tokens searched by PFPT and simply appended to the original text prompt, can significantly enhance adversarial robustness with minimal usage cost. Moreover, these tokens are naturally present in the native and inherent vocabulary, offering a human-readable property. To search for defense tokens, PFPT takes a coarse-to-fine strategy, divided into two steps: single token search and parallel tokens search. This design prevents the selection from degenerating into mere gradient-based or brute-force searching. Single token search discoveries the best candidate token set for each token position, with a wider coverage of candidates aiding in the search for the optimal token. Parallel tokens search conducts a joint search, measuring tokens based on consistency and accuracy to finally determine the most effective tokens.

Comprehensive experiments are conducted to benchmark PFPT across 14 datasets and with 4 data sampling scenarios which are 1-, 4-, 16- and 100- shot learning. PFPT is compared against the hand-engineered prompts proposed in CLIP and the state-of-the-art text prompt learning methods. PFPT outperforms these approaches in terms of the in-distribution performance and the generalization ability under distribution shift (the same classes with different input distribution) and across datasets (different classes).

Our contributions are summarized as follows:

* We introduce PFPT, a method for searching defense tokens that significantly improves the adversarial robustness for the CLIP model.
* The defense tokens learned are at the word level and can significantly enhance adversarial robustness simply by being appended to the original text prompt, making this method both parameter-free and accessible.
* PFPT employs a coarse-to-fine strategy to search for defense tokens, and its two-step design significantly enhances search efficiency.
* Extensive experiments are conducted across 14 datasets to validate the effectiveness of PFPT.

## 2 Related work

### Adversarial examples

Adversarial examples expose vulnerabilities in models, especially visual language models (VLMs) 

(Inkawhich et al., 2023; Schlarmann and Hein, 2023; Zhao et al., 2024; Qi et al., 2024; Carlini et al., 2024; Bagdasaryan et al., 2023). Studies show adversarial samples can cause misclassification (Inkawhich et al., 2023), enable malicious exploitation (Schlarmann and Hein, 2023), and exacerbate security risks in multimodal VLMs (Zhao et al., 2024), allowing attackers to evade systems, bypass security (Qi et al., 2024), induce misaligned behavior (Carlini et al., 2024), and enable targeted output contamination (Bagdasaryan et al., 2023). Such attacks can produce harmful or unreliable VLM responses. This paper proposes text prompt adversarial training to defend against adversarial sample attacks on VLMs.

### Adversarial training

In machine learning, adversarial training Goodfellow et al. (2014) is the most potent defense against adversarial examples (Athalye et al., 2018). Most prior methods (Madry et al., 2017; Andriushchenko and Flammarion, 2020; Li and Spratling, 2023) focused on training models from scratch, while exploring pre-trained model robustness received less attention. Recent work enhanced pre-trained model robustness via fine-tuning entire (Hendrycks et al., 2019; Jiang et al., 2020; Kim et al., 2020; Luo et al., 2023) or subset (Chen et al., 2020) parameters, or adversarial visual prompts (Chen et al., 2023; Huang et al., 2023). APT (Li et al., 2024b) adapts robustness by optimizing text prompts in continuous embedding space, with limited practical utility. Our method proposes user-friendly text prompts optimized in discrete text space, without additional adversarial training parameters.

### Prompt Learning

Recent years saw growing interest in prompt engineering for pre-trained models, yielding CoOp (Zhou et al., 2022) for automated prompt generation, PGN (Loedeman et al., 2022) for adapting visual models via task prompts, NOAH (Zhang et al., 2022) for optimal prompt configurations, OOHMG (Lin et al., 2023) enabling text-to-motion generation without paired data, Subspace Prompt Tuning (Ma et al., 2023) mitigating VLM prompt overfitting, SgVA (Peng et al., 2023) enhancing few-shot image classification, and GALIP (Tao et al., 2023) employing CLIP (Radford et al., 2021) for controlled image generation via prompts. This paper investigates parameter-free, user-friendly text prompt learning to defend against adversarial attacks in image recognition.

## 3 Method

### Preliminaries: CLIP

CLIP comprises image and text encoders, encoding images and text in a unified embedding space. Image encoder options include ResNet (He et al., 2016) or Vision-Transformer (Dosovitskiy et al., 2020), while text encoder is Transformer. For text input, CLIP converts words to \(d\)-dimensional word embedding vectors using pre-trained vocabulary, then inputs them to Transformer to generate the final text feature.

Given an input image \(I_{i}\) and text \(T_{j}\), the respective features \(f^{v}_{i}\) and \(f^{t}_{j}\) are computed as:

\[f^{v}_{i}=\mathcal{E}_{v}(I_{i}),\quad f^{t}_{j}=\mathcal{E}_{t}(T_{j}) \tag{1}\]

where \(\mathcal{E}_{v}\) and \(\mathcal{E}_{t}\) represent the visual and the text encoder within CLIP respectively.

A cosine similarity score measures alignment between each image-text feature pair. The alignment between image \(I_{i}\) and text \(T_{j}\) is calculated as:

\[p_{i,j}=p(I_{i},T_{j})=\frac{\exp\left(\cos(f^{v}_{i},f^{t}_{j})/\tau\right)}{ \sum_{j}\exp\left(\cos(f^{v}_{i},f^{t}_{j})/\tau\right)} \tag{2}\]

where \(\tau\) is a temperature parameter learned by CLIP. During pre-training, the two encoders maximize similarity scores of true image-text pairs (i.e., when \(i=j\)) and minimize scores for false pairs, aligning the encoders. After pre-training, CLIP achieves zero-shot classification using text descriptions of class labels as prompts. Given class prompts, CLIP predicts most probable class \(\arg\max_{j}p_{i,j}\). Default prompt is "a photo of a [CLASS]" replacing "[CLASS]" with dataset class names. Prompts comprise class, prefix, and suffix components. For \(N\) classes, \(j\)-th class prompt is:

\[T_{j}=(t_{1},t_{2},...,t_{[\mathrm{CLASS}]},...,t_{\mathrm{max\_len}}) \tag{3}\]

where \(\mathrm{max\_len}\) is the tokens' maximum length.

### Parameter-free and Accessible Prompt Learning

We aim to search a optimal token combinations within the pre-trained and fixed vocabulary as defense tokens appended to text prompts. This approach is parameter-free, requiring no additional parameters to be learned, we merely record the identified tokens as the learning result. It also boosts CLIP's adversarial robustness by appending the learned tokens to the text prompt, making it accessible 

In section 3.2.1, we discuss adversarial examples for CLIP. Section 3.2.2 presents the parameterization of defense tokens in our method. Sections 3.2.3 and 3.2.4 respectively detail the Single Token Search (STS) and Parallel Tokens Search (PTS).

#### 3.2.1 Adversarial examples for CLIP

To search for defense tokens, adversarial example images are generated on-the-fly during the training. We employ a common technique to generate adversarial examples for CLIP, which involves computing gradient-based perturbations \(\delta\) to the input images. This aims to maximize the cosine divergence between the image feature \(f_{i}^{v}\) and the text feature of the corresponding ground-truth prompt, \(f_{y_{i}}^{t}\):

\[\arg\max_{||\delta_{i}||_{p}\leq\varepsilon}\mathcal{L}(\mathcal{E}_{v}(I_{i}+ \delta_{i}),\mathcal{E}_{t}(T_{i}),y_{i}) \tag{4}\]

where the perturbation \(\delta_{i}\) is constrained within the \(\varepsilon\)-ball of the \(p\)-norm, while the text input consists of the default prompt appended with defense tokens. Algorithm 1 illustrates the pseudo-code we used to generate adversarial examples to attack.

```
1:functionattack(\(I\), \(T\), \(y\))
2:\(\delta=\text{uniform}(-\epsilon,\epsilon)\)
3:for\(1\to S\)do
4:\(I^{\prime}=\min(0,\max(I+\delta,1))\)
5:\(L=\mathcal{L}(\mathcal{E}_{v}(I^{\prime}),\mathcal{E}_{t}(T),y)\)
6:\(\delta=\min(-\epsilon,\max(\delta~{}+~{}\eta~{}~{}\cdot~{}\textsc{sign}( \nabla_{I}L),\epsilon))\)
7:endfor
8:return\(\min(0,\max(I+\delta,1))\)
9:endfunction
```

**Algorithm 1** Adversarial attack on CLIP. \(S\) is the perturbation step. \(\eta\) is the step size for perturbing image.

#### 3.2.2 Parameterization of learnable tokens

The text input is tokenized into fixed-length tokens (Eq. 3). Therefore, we first determine which positions should be defense tokens, as well as the number of defense tokens. Our goal is to identify and search for specific tokens from the pre-trained vocabulary \(\mathcal{V}\) to serve as these defense tokens. Taking CLIP's default prompt "a photo of a [CLASS]" as an example, we select \(K\) tokens from the suffix to act as defense tokens. Consequently, the text prompt appended with defense tokens of the class \(T_{j}^{\prime}\) can be represented as:

\[T_{j}^{\prime} =(t_{1},t_{2},\ldots,t_{\mathrm{[CLASS_{j}]}},\] \[t_{\mathrm{[CLASS_{j}]+1}},\ldots,t_{\mathrm{[CLASS_{j}]+K}}, \ldots,t_{\mathrm{max\_len}}) \tag{5}\]

In the sequence, \(t_{1}\) is "a", \(t_{2}\) is "photo", and \(t_{\mathrm{[CLASS_{j}]}}\) is a specific class name, like "dog". Tokens indexed from \([\mathrm{CLASS_{j}]+1}\) to \([\mathrm{CLASS_{j}]+K}\) are chosen as defense tokens, randomly initialized from the pre-trained vocabulary \(\mathcal{V}\).

Theoretically, the \(K\) defense tokens can be positioned anywhere and in any number, as long as the arrangement meets CLIP's input token length constraints. In our experiments, we test on two positions settings: front and end.

**Search goal:** Upon determining the number and positions of these defense tokens, our search goal aimed at improving adversarial robustness for VLMs can be formulated as:

\[\arg\min_{\mathbf{t}^{\prime}}E_{\begin{subarray}{c}i\in B\\ \mathbf{t}^{\prime}\in\mathcal{V}^{K}\end{subarray}}\mathcal{L}(\mathcal{E}_{ v}(I_{i}+\delta_{i}),\mathcal{E}_{t}(T_{i}^{\prime}(\mathbf{t}^{\prime})),y_{ i}) \tag{6}\]

where we use Algorithm 1 to generate the pertubation \(\delta_{i}\). \(T_{i}^{\prime}(\mathbf{t}^{\prime})\) represents the input text containing defense tokens, where only the \(K\) defense tokens need to be learned.

#### 3.2.3 Single Token Search (STS)

The single token search performs a coarse search for the token at a single position, addressing issues related to exclusive reliance on either gradient descent or brute-force search methods.

Here, we take defense tokens as suffixes appended to the template prompt \(\mathbf{t}^{(template)}\) (please see Eq. 5) as an example to describe STS. Please note that our defense tokens can be assigned to arbitrary positions, except those class tokens.

The STS considers the adversarial robustness and selects a batch of tokens from the pre-trained

Figure 2: Framework of the proposed Parameter-Free Prompt Tuning (PFPT) method applied to CLIP-like VLMs. Both image and text encoders are frozen, and defense tokens are sourced from the inherent vocabulary. PFPT employs a coarse-to-fine strategy, divided into two steps: Single Token Search and Parallel Tokens Search.

 

vocabulary \({\cal V}\) as candidates for the defense token at the single position. To achieve this, we define a defense objective as the selection goal. For instance, when searching for the \(k\)-th position among \(K\) defense tokens, the objective is as follows:

\[\begin{split}&\mathcal{L}^{defense}(t_{k})=\\ &-\nabla_{\mathbf{e}_{t_{k}}}\log p(\hat{y}=y|\mathbf{t}^{(template )}\oplus t_{1}\oplus t_{2}\oplus\ldots\\ &\oplus t_{k}\oplus\cdots\oplus t_{K},I+\delta)\end{split} \tag{7}\]

where \(\oplus\) means token concatenation, for example, "a" \(\oplus\) "photo" = "a photo", \(\hat{y}\) is the prediction, and \(y\) is the ground-truth class. This objective is the gradient obtained by backpropagating the log-likelihood loss to the one-hot embedding of the token \(t_{k}\).

We construct the candidate set \(\mathcal{C}_{k,l}\) by selecting the top-\(L\) tokens, ordered from high to low based on their objective scores, here, \(k\) is the defense token position, and \(l\) is the token ranking at the position. Finally, \(K\) candidate sets \(\{\mathcal{C}_{k,l}\}_{k=1}^{K~{}L}\) are constructed for each of the \(K\) positions following the coarse search. Algorithm 2 and Figure 3 present the single token search.

```
1:function STS(\(I\), \(\delta\), \(t_{1}\), \(t_{2}\), ..., \(t_{K}\))
2:\(\{\mathcal{C}_{k,l}\}_{k=1}^{K~{}L}=\{\emptyset\}\)
3:for\(k=1\to K\)do
4:\(\mathcal{L}^{defense}(t_{k})=\\ -\nabla_{\mathbf{e}_{t_{k}}}\log p(\hat{y}=y|\mathbf{t}^{(template)}\\ \oplus t_{1}\oplus\cdots\oplus t_{k}\oplus\cdots\oplus t_{K}, I+\delta)\)
5:\(\{\mathcal{C}_{k,l=1:L}\}=\mathrm{top}-L(\mathcal{L}^{defense}(t_{k}))\)
6:endfor
7:return\(\{\mathcal{C}_{k,l}\}\)
8:endfunction
```

**Algorithm 2** Single Token Search.

#### 3.2.4 Parallel Tokens Search (PTS)

In parallel tokens search step, we perform parallel optimization across all candidate sets \(\{\mathcal{C}_{k,l}\}\) to search for defense tokens, considering consistency with the distribution predicted by the original prompt and adversarial accuracy. First, we concatenate tokens of the same ranking across all positions to form a new candidate set of defense tokens, denoted as \(\{\mathcal{A}_{l}\}_{l=1}^{L}\),

\[\mathcal{A}_{l}=(\mathcal{C}_{1,l}\oplus\mathcal{C}_{2,l}\oplus\cdots\oplus \mathcal{C}_{K,l})\in{\cal V}^{K} \tag{8}\]

Consistency scoreThe consistency score aims to ensure that the text with appended defense tokens retains the same semantics as the original text prompt. This helps prevent significant deviations in the predicted distribution due to the defense tokens. To achieve this goal, we follow Azuma et al. [Azuma and Matsui, 2023], leverage the results of classification to ensure a consistent output distribution both with and without defense tokens.

We classify the original image \(I\) using text appended with defense tokens t\({}^{defense}\) (w/ DT) and the original text (wo/ DT). First, we obtain the probabilities of predicting the ground-truth class \(y\), given the both conditions, these are

\[q^{1}=p(\hat{y}=y|\mathbf{t}^{(template)}\oplus\mathbf{t}^{( defense)},I)\text{ (w/ DT)} \tag{9}\] \[q^{2}=p(\hat{y}=y|\mathbf{t}^{(template)},I)\quad\text{(wo/ DT)}\]

We then plug each candidate tokens \(\mathbf{t}^{\prime}\in\{\mathcal{A}_{l}\}_{l=1}^{L}\) into the following consistent objective to measure the KL divergence between \(q^{1}\) and \(q^{2}\):

\[\mathcal{L}^{consist}(\mathbf{t}^{\prime})=D_{KL}(q^{1}(\mathbf{t}^{(defense)} =\mathbf{t}^{\prime})||q^{2}) \tag{10}\]

The consistency score aims to guide the search for defense tokens using the probability distribution of the original prompt.

Accuracy scoreThe accuracy score assesses the precision of classification results for adversarial examples (e.g. \(I+\delta\)) when using prompts appended with defense tokens. Similar to the consistency score, we iteratively traverse the candidate defense tokens in \(\{\mathcal{A}_{l}\}_{l=1}^{L}\), calculating the accuracy corresponding to each candidate tokens \(\mathbf{t}^{\prime}\in\{\mathcal{A}_{l}\}_{l=1}^{L}\).

\[\begin{split}&\mathcal{L}^{acc.}(\mathbf{t}^{\prime})=\\ &\mathbf{1}\{p(\hat{y}=y|\mathbf{t}^{(template)}\oplus(\mathbf{t}^{( defense)}=\mathbf{t}^{\prime}),I+\delta)\}\end{split} \tag{11}\]

where \(\mathbf{1}\) denotes the indicator function. Finnaly, the total score for the candidate defense tokens

Figure 3: Single Token Search selects a batch of tokens from the pre-trained and fixed vocabulary as candidates for a single position. Each position yields a candidate set.

 

is computed as \({\cal L}^{total}({\bf t}^{\prime})={\cal L}^{consist}({\bf t}^{\prime})+\lambda(1-{ \cal L}^{acc.}({\bf t}^{\prime}))\). After scoring each candidate defense tokens in set \(\{{\cal A}_{l}\}_{l=1}^{L}\), we select the tokens with the lowest score (low is better) as the result of the parallel tokens search. Algorithm 3 and Figure 4 illustrate the parallel tokens search.

```
1:functionPTS(\(I\), \(\delta\), \(\{{\cal C}_{k,l}\}\))
2:\(\{{\cal A}_{l}\}_{l=1}^{L}=\{\emptyset\}\)
3:for\(l=1\to L\)do
4:\(\{{\cal A}_{l}\}=\texttt{CONCAT}(\{{\cal C}_{k=1:K,l}\})\)
5:endfor
6:\(\{{\cal S}_{l}\}_{l=1}^{L}=\{\emptyset\}\)
7:foreach \((l,{\bf t}^{\prime})\in\{{\cal A}_{l}\}\)do
8:\({\cal S}_{l}={\cal L}^{total}({\bf t}^{\prime};I,\delta)\)
9:endfor
10:\(i\leftarrow\texttt{Index}(\texttt{Min}(\{{\cal S}_{l}\}_{l=1}^{L}))\vartriangle\)Get index of the minimum score
11:return\(\{{\cal A}_{l=i}\}\)
12:endfunction
```

**Algorithm 3** Parallel Tokens Search.

## 4 Experiments

**Datasets.** Following Li et al. (2024) and Zhou et al. (2022), our evaluation comprises 11 datasets, including ImageNet Deng et al. (2009), Caltech101 Fei-Fei et al. (2004), OxfordPets Parkhi et al. (2012), StanfordCars Krause et al. (2013), Flowers102 Nilsback and Zisserman (2008), Food101 Bossard et al. (2014), FGVC Aircraft Maji et al. (2013), SUN397 Xiao et al. (2010), DTD Campoi et al. (2014), EuroSAT Helber et al. (2019), and UCF101 Soomro et al. (2012). We evaluate each dataset using an \(N\)-shot strategy, where "\(N\)" is the number of examples randomly drawn from the entire training dataset. \(N\) can be set to 1, 4, 16, or 100. All settings are validated on the complete test set.

**Models.** We employ the ViT-B/32 model Dosovitskiy et al. (2020) as the standard backbone for the image encoder. Following Li et al. (2024), the image encoder's weights are pre-trained using TeCoA Mao et al. (2022), a state-of-the-art method for zero-shot adversarial robustness. The necessity for using pre-training weights are discussed in Li et al. (2024) and Chen et al. (2023).

**Learning details and evaluation.** Adversarial examples are generated on-the-fly during training, as outlined in Algorithm 1. Following Mao et al. (2022) and Li et al. (2023), we use two different perturbation budgets: \(\epsilon=1/255\) and \(\epsilon=4/255\). We use 3 steps with a step size of \(2\epsilon/3\) during training. PGD generates attacks using the text prompt appended with defense tokens for evaluatiton, involving 100 steps and a step size of \(\epsilon/4\).

**Competitive approaches.** Current work has not explored the text prompt tuning at the word level, limiting our comparisons to similar methods which optimize text embeddings like APT. APT optimizes text embeddings for adversarial robustness, these embeddings lie in an unconstrained continuous space during the learning, potentially resulting in embeddings that do not correspond to any specific word in the inherent vocabulary. For a fair comparison, we followed APT's method of finding the nearest word based on Euclidean distance, converting continuous embeddings into specific tokens from the vocabulary. Additionally, we compare our method against Hand-Engineered Prompts (HEP), originally proposed in Radford et al. (2021) and widely used since then.

### In-Distribution performance

This section compares the proposed method and its competitors using in-distribution data, meaning the training and test sets come from the (approximately) same distribution. We searched for defense

Figure 4: Parallel Tokens Search first concatenates tokens of the same ranking and plugs them into the prompt. Using the designed consistency score and accuracy score, it selects the top-1 token combination as the defense tokens.

 

tokens on the training set and evaluated them on the test set of the same dataset. Performance results for the ViT-B/32 model are presented below.

**Learned prompts vs. HEP.** Table 1 presents a performance comparison of different text prompting methods, averaged across 11 datasets and evaluated against various perturbation budgets (\(\epsilon\)) and shot counts. Our method significantly outperforms HEP, notably enhancing both accuracy and robustness, even in a 1-shot scenario. The improvement is especially pronounced for the perturbation level \(\epsilon=1/255\), with absolute increases of \(4.49\%\) and \(5.58\%\) in accuracy and robustness, respectively. Moreover, our method demonstrates consistent performance gains with an increase in the number of shots. Our approach significantly outperforms HEP when trained with the 100 shots.

Moreover, our method yields improvements across each specific dataset, though the extent varies. The comparison results on different datasets are shown in Figure 5 and Figure 6 in the Appendix. Our proposed method PFPT achieves significant performance improvements on the Food101 and FGVC datasets. In summary, by leveraging the learned defense tokens, PFPT demonstrates a significant enhancement on the robustness of VLP models [7] in an accessible and user-friendly manner.

**PFPT vs. APT.** Table 1 compares the average performance of diverse adaptation methods across 11 datasets. Under the 1-shot and 4-shot settings, the proposed method PFPT considerably outperforms APT and HEP in both accuracy and robustness, suggesting that PFPT efficiently searches for effective defense tokens with limited training data. In contrast, APT performs worse than the baseline method HEP in the 4-sho

Figure 5: The in-distribution performance on 11 datasets and the averaged performance under different shots. \(\epsilon=4/255\) and \(K=16\)

Figure 6: The performance improvements per dataset of our adversarially-trained prompt compared to the standardly-trained prompt. The results are presented for the setting with 16 shots, \(K=16\), at \(\epsilon=4/255\).

 

optimization method requires a large amount of training data to converge to specific embeddings. As the amount of training data increases, PFPT shows a more pronounced advantage over APT in both accuracy and robustness. These observations demonstrate that PFPT utilizes training data more effectively to achieve a better trade-off between accuracy and robustness.

From the results on individual datasets shown in Figure 5 and Figure 31 in the Appendix, we observe that PFPT outperforms the baseline methods in most experimental settings on the Imagenet dataset. When trained with a small amount of data, specifically in the 1-shot and 4-shot settings, PFPT improves both accuracy and robustness using searched defense tokens. With more data, PFPT maintains its performance advantage and achieves substantial improvements in both epsilon settings.

### Generalization of learned defense tokens

To evaluate the transferability and generalization of the defense tokens learned by PFPT across different datasets, we consider two generalization settings in this section: distribution shift and cross-dataset. Specifically, the Imagenet dataset is adopted as the source dataset to search for defense tokens under the 16-shots and 4-epsilon settings. For distribution shift tests, we evaluate the accuracy and robustness of the learned defense tokens on datasets with the same categories but completely distinct data distributions. We use three Imagenet-variant datasets for distribution shift tests, including ImageNet-V2 (Recht et al., 2019), ImageNetSketch, and ImageNet-R (Hendrycks et al., 2021). As shown in Table 2, PFPT reveals remarkable performance advantages over baseline methods on datasets with completely different data distributions. This proves that the defense tokens learned by PFPT have better generalization capabilities to varying data distribution.

Moreover, we conduct cross-dataset tests on datasets with distinct categories to further evaluate the generalization ability of PFPT. Table 2 shows the average performance on the other 10 datasets. It is noteworthy that the proposed method, PFPT, demonstrates the best transfer and generalization abilities, achieving the highest accuracy and robustness on both the source and test datasets. Although the baseline method, APT, performs well on the source dataset, it shows poor performance on the 10 test datasets, indicating that the token embeddings learned by APT are specific to the dataset and fail to provide cross-dataset transferability.

### Accuracy-robustness trade-off evaluation

To evaluate the trade-off between accuracy and robustness, we compare PEPT's adversarially-robust defense tokens with the standard prompt trained in non-attack scenarios in Figure 6. The experimental setup is 16 shots and \(\epsilon{=}\,4/255\). As Figure 6 shows, although PFPT reduces the classfication accuracy on clean images compared to the standardly-trained prompts, it significantly improves robustness under attack scenarios. Notably, PFPT achieves a good trade-off between the two metrics, as the increase in robustness outperforms the decrease in accuracy in most datasets. Overall, PFPT exhibits a considerable improvement in the adversarial robustness of VLMs under attack scenarios with a minor accuracy loss, presenting its effectiveness and good accuracy-robustness trade-off.

### Readability measurement

As shown in the Appendix 37, we provide a novel readability assessment method to compare the readability of the learned token embeddings.

## 5 Conclusion

In this study, we aim to enhance the performance of VLMs against adversarial samples. To achieve this, we introduce a novel text prompt learning method to boost adversarial robustness. Our approach, Parameter-Free Prompt Tuning, searches for robust defense prompts that are flexible, easy to use, and human-readable. Extensive experiments present the effectiveness of our method in improving in-distribution performance and generalization across different distributions and datasets. Our work introduces a new paradigm for text prompt learning that boosts adversarial robustness for VLMs.

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{Source} & \multicolumn{3}{c}{Distribution Shifts} & \multicolumn{3}{c}{Cross Datasets} \\ \cline{2-10} Method & ImageNet & ImageNet-V2 & ImageNet-Stecht & ImageNet-R & 10 Datasets Avg. \\ \cline{2-10}  & Acc. & Rob. & Acc. & Rob. & Acc. & Rob. & Acc. & Rob. & Acc. & Rob. \\ \hline HEP & 39.7 & 10.2 & 32.7 & 7.5 & 17.4 & 7.2 & 21.5 & 5.8 & 32.3 & 10.3 \\ APT & 40.8 & 10.9 & 32.4 & 8.5 & 17.6 & 7.4 & 22.0 & 6.2 & 31.1 & 11.3 \\ PFPT & 44.9 & 14.8 & 34.5 & 9.6 & 18.9 & 8.0 & 22.3 & 6.5 & 33.6 & 12.0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: We evaluate the generalization of prompts learned by our method on ImageNet to (1) datasets with different input distribution and (2) the remaining 10 datasets. Our method are trained with 16 shots and \(\epsilon=4/255\).

 



## 6 Limitations

PFPT has two main limitations. First, while the learned defense tokens are at the word level, they lack strong interconnections, a continuous, coherent sentence might be more useful for human users. Second, the defense tokens are dataset-specific, since CLIP training spans multiple datasets, identifying a universal set of defense tokens that consistently enhance adversarial robustness across various image sources is a promising direction for future research.

## References

* Andriushchenko and Flammarion (2020) Maksym Andriushchenko and Nicolas Flammarion. 2020. Understanding and improving fast adversarial training. _Advances in Neural Information Processing Systems_, 33:16048-16059.
* Athalye et al. (2018) Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In _International conference on machine learning_, pages 274-283. PMLR.
* Azuma and Matsui (2023) Hiroki Azuma and Yusuke Matsui. 2023. Defense-prefix for preventing typographic attacks on clip. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 3644-3653.
* Bagdasaryan et al. (2023) Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. 2023. (ab) using images and sounds for indirect instruction injection in multimodal llms. _arXiv preprint arXiv:2307.10490_.
* Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101-mining discriminative components with random forests. In _Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_, pages 446-461. Springer.
* Carlini et al. (2024) Nicholas Carlini, Milad Nasr, Christopher A ChoquetteChoo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. 2024. Are aligned neural networks adversarially aligned? _Advances in Neural Information Processing Systems_, 36.
* Chen et al. (2023) Aochuan Chen, Peter Lorenz, Yuguang Yao, Pin-Yu Chen, and Sijia Liu. 2023. Visual prompting for adversarial robustness. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5. IEEE.
* Chen et al. (2020) Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. 2020. Adversarial robustness: From self-supervised pre-training to finetuning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 699-708.
* Cimpoi et al. (2014) Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 3606-3613.
* Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_, pages 248-255. Ieee.
* Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. _arXiv preprint arXiv:2010.11929_.
* Fei-Fei et al. (2004) Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In _2004 conference on computer vision and pattern recognition workshop_, pages 178-178. IEEE.
* Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. _arXiv preprint arXiv:1412.6572_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Helber et al. (2019) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Berth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_, 12(7):2217-2226.
* Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. 2021. The many faces of robustness: A critical analysis of out-of-distribution generalization. In _Proceedings of the IEEE/CVF international conference on computer vision_, pages 8340-8349.
* Hendrycks et al. (2019) Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019. Using pre-training can improve model robustness and uncertainty. In _International conference on machine learning_, pages 2712-2721. PMLR.
* Huang et al. (2023) Qidong Huang, Xiaoyi Dong, Dongdong Chen, Yinpeng Chen, Lu Yuan, Gang Hua, Weiming Zhang, and Nenghai Yu. 2023. Improving adversarial robustness of masked autoencoders via test-time frequency-domain prompting. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 1600-1610

Nathan Inkawhich, Gwendolyn McDonald, and Ryan Luley. 2023. Adversarial attacks on foundational vision models. _arXiv preprint arXiv:2308.14597_.
* Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In _International conference on machine learning_, pages 4904-4916. PMLR.
* Jiang et al. (2020) Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. 2020. Robust pre-training by adversarial contrastive learning. _Advances in neural information processing systems_, 33:16199-16210.
* Kim et al. (2020) Mineson Kim, Jihoon Tack, and Sung Ju Hwang. 2020. Adversarial self-supervised contrastive learning. _Advances in Neural Information Processing Systems_, 33:2983-2994.
* Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In _Proceedings of the IEEE international conference on computer vision workshops_, pages 554-561.
* Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. _arXiv preprint arXiv:2202.10054_.
* Li et al. (2024a) Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. 2024a. Multimodal foundation models: From specialists to general-purpose assistants. _Foundations and Trends(r) in Computer Graphics and Vision_, 16(1-2):1-214.
* Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International conference on machine learning_, pages 12888-12900. PMLR.
* Li et al. (2024b) Lin Li, Haoyan Guan, Jianing Qiu, and Michael Spratling. 2024b. One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 24408-24419.
* Li and Spratling (2023) Lin Li and Michael Spratling. 2023. Data augmentation alone can improve adversarial training. _arXiv preprint arXiv:2301.09879_.
* Li et al. (2023) Lin Li, Yifei Wang, Chawin Sitawarin, and Michael Spratling. 2023. Oodrobustpench: benchmarking and analyzing adversarial robustness under distribution shift. _arXiv preprint arXiv:2310.12793_.
* Lin et al. (2023) Junfan Lin, Jianlong Chang, Lingbo Liu, Guanbin Li, Liang Lin, Qi Tian, and Chang-wen Chen. 2023. Being comes from not-being: Open-vocabulary text-to-motion generation with wordless training. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 23222-23231.
* Loedeman et al. (2022) Jochem Loedeman, Maarten C Stol, Tengda Han, and Yuki M Asano. 2022. Prompt generation networks for input-based adaptation of frozen vision transformers. _arXiv preprint arXiv:2210.06466_.
* Luo et al. (2023) Rundong Luo, Yifei Wang, and Yusen Wang. 2023. Rethinking the effect of data augmentation in adversarial contrastive learning. _arXiv preprint arXiv:2303.01289_.
* Ma et al. (2023) Chengcheng Ma, Yang Liu, Jiankang Deng, Lingxi Xie, Weiming Dong, and Changsheng Xu. 2023. Understanding and mitigating overfitting in prompt tuning for vision-language models. _IEEE Transactions on Circuits and Systems for Video Technology_.
* Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. _arXiv preprint arXiv:1706.06083_.
* Majii et al. (2013) Subhranssu Majii, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. _arXiv preprint arXiv:1306.5151_.
* Mao et al. (2022) Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. 2022. Understanding zero-shot adversarial robustness for large-scale models. _arXiv preprint arXiv:2212.07016_.
* Nilsback and Zisserman (2008) Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In _2008 Sixth Indian conference on computer vision, graphics & image processing_, pages 722-729. IEEE.
* Parkhi et al. (2012) Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In _2012 IEEE conference on computer vision and pattern recognition_, pages 3498-3505. IEEE.
* Peng et al. (2023) Fang Peng, Xiaoshan Yang, Linhui Xiao, Yaowei Wang, and Changsheng Xu. 2023. Sgva-clip: Semantic-guided visual adapting of vision-language models for few-shot image classification. _IEEE Transactions on Multimedia_.
* Qi et al. (2024) Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. 2024. Visual adversarial examples jailbreak aligned large language models. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 21527-21536.
* Qiu et al. (2023) Jianqing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P-W Lo, Bo Xiao, et al. 2023. Large ai models in health informatics: Applications, challenges, and the future. _IEEE Journal of Biomedical and Health Informatics_.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from 

natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.
* Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize to imagenet? In _International conference on machine learning_, pages 5389-5400. PMLR.
* Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 10684-10695.
* Schlarmann and Hein (2023) Christian Schlarmann and Matthias Hein. 2023. On the adversarial robustness of multi-modal foundation models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 3677-3685.
* Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. _arXiv preprint arXiv:1508.07909_.
* Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. _arXiv preprint arXiv:1212.0402_.
* Tao et al. (2023) Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. 2023. Galip: Generative adversarial clips for text-to-image synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 14214-14223.
* Xiao et al. (2010) Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abey to zoo. In _2010 IEEE computer society conference on computer vision and pattern recognition_, pages 3485-3492. IEEE.
* Zhang et al. (2024) Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
* Zhang et al. (2022) Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. 2022. Neural prompt search. _arXiv preprint arXiv:2206.04673_.
* Zhao et al. (2024) Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. 2024. On evaluating adversarial robustness of large vision-language models. _Advances in Neural Information Processing Systems_, 36.
* Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. _International Journal of Computer Vision_, 130(9):2337-2348