arXiv:2410.05270v2 [cs.CV] 6 Dec 2024

# Fine-tuning CLIP's Last Visual Projector: a Few-shot Cornucopia

Mohammad Fahes¹

Tuan-Hung Vu¹,²

Andrei Bursuc¹,²

Patrick Pérez³

Raoul de Charette¹

¹ Inria

² Valeo.ai

³ Kyutai

<figure><img src="image_1.png" /><figcaption>Figure 1. Few-shot classification with CLIP. a) Using a pre-trained CLIP, zero-shot classification is performed by measuring text and visual embeddings similarity. Among few-shot adaptation strategies of CLIP, b) Linear Probing trains a linear classifier of the visual features, c) Adapter-style tuning adds external learnable MLPs, d) Prompt Tuning learns word embeddings. Alternatively, we propose e) ProLIP which does not introduce new weights and only fine-tunes the last linear projection layer of the vision encoder.</figcaption></figure>

## Abstract

We consider the problem of adapting a contrastively pre-trained vision-language model like CLIP [30] for few-shot classification. The literature addresses this problem by learning a linear classifier of the frozen visual features, optimizing word embeddings, or learning external feature adapters. This paper introduces an alternative way for CLIP adaptation without adding “external” parameters to optimize. We find that simply fine-tuning the last projection matrix of the vision encoder leads to performance better than all baselines. Furthermore, we show that regularizing training with the distance between the fine-tuned and pretrained matrices adds reliability for adapting CLIP. This simple approach, coined ProLIP, yields state-of-the-art performance on 11 few-shot classification benchmarks, few-shot domain generalization, cross-dataset transfer, base-to-new class generalization, and test-time adaptation. Code will be made available at: https://github.com/astra-vision/ProLIP.

## 1. Introduction

Contrastive Language-Image Pretraining (CLIP) [30] has shown that strong visual features can be learned from noisy natural language descriptions at very large scale. The true potential of CLIP lies in its shared vision-text space, break-

ing the long-standing constraints of closed-set systems and enabling non-trivial interactions and querying between text and images via prompts. Such a freedom in the label space makes the model readily applicable to a wide range of specialized downstream applications.

CLIP trains a vision and a text encoder on large batches of image-text pairs using a sum of contrastive image-to-text and text-to-image losses. At inference, given an image and a set of classes expressed within prompts (e.g., “a photo of {class_k}”), one can perform zero-shot classification. The predicted class is simply the one for which the textual embedding has the highest similarity with the image embedding. The prompt template can be engineered to boost the zero-shot performance, or automated by querying multiple descriptors of a class from Large Language Model (LLMs), such as GPT-3 [3], and ensembling their embeddings [26]. Yet, the zero-shot performance may still be unsatisfying, especially for data that are supposedly underrepresented in CLIP training data. Examples of such cases include geospatial data, e.g., EuroSAT [12] and specialized data, e.g., FGVCAircraft [25]. Thus, an interesting practical setting emerged in transfer learning: Given a labeled few-shot training dataset of images, how to efficiently adapt CLIP in order to maximize the performance on the test set?

Hinging on only a few labeled samples for supervision, model training is prone to overfitting. The common strategy is to avoid full fine-tuning and instead adapt only a few parameters [21]. Starting from a concept-rich pretrained CLIP
model, such parameter-efficient strategies have been shown to be effective for few-shot tasks. In this direction, the literature explores three avenues. First, prompt tuning [48, 49] replaces the template with learnable parameters in the word embedding space, while freezing both vision and text encoders. Second, CLIP adapters [9] learn a multi-layer perceptron (MLP) on top of the frozen visual or text features, and use a residual connection aiming at partially using the zero-shot features. In these two paradigms, the text embeddings are used as classification weights. Third, linear probing [30] simply trains a linear classifier on top of the frozen visual features.

**Shortcomings.** While existing solutions are technically simple and parameter-efficient, we identify several limitations. Prompt tuning methods [4, 49, 50] are slow to train as gradients need to be backpropagated over the entire text encoder. In addition, different context lengths and class-name positions lead to different performances. Adapters [9, 46] impose architectural choices of the MLP, the bottleneck dimension and the residual connection. On the other hand, while initially suggested as a few-shot baseline for CLIP, the performance of linear probing (LP) lies far behind adapters and prompt tuning. Its main shortcoming stems from ignoring the text embeddings during adaptation. Recently, Huang et al. [17] proposed LP++, an improved LP version blending textual embeddings with classification weights using class-wise learnable parameters. While LP++ [17] shows significant improvements over standard LP, we argue in this work that directly using the text embeddings as classification weights might be a better practice for fine-tuning CLIP, as it is closer in principle to its original pretraining regime. Another major limitation of linear probing methods [17, 24, 30, 34] is that they cannot be applied in open-class and cross-dataset transfer settings: the classifier is restricted to the set of classes contained in the downstream few-shot training dataset.

The previous methods either train “external” parameters (e.g., Adapters, LP), or learn parameters in the input space (e.g., prompt tuning). To the best of our knowledge, no existing work tackles few-shot CLIP adaptation problem with parameter-efficient fine-tuning of the model weights. In this work, we propose a first baseline for model weights based few-shot learning. Our method, dubbed “ProLIP”, is both extremely simple to implement, being only a few lines of code, and very effective: *considering pretrained visual and text CLIP encoders f and g, we fine-tune the last projection matrix of f (i.e., the projector mapping visual embeddings into the shared embedding space) with a cross-entropy loss while constraining its weights to remain close to the pretrained ones.* Fig. 1 illustrates existing approaches and our proposed ProLIP, which is further detailed in Sec. 4.

ProLIP is advantageous for a number of reasons:

* It alleviates the need of “external” parameters which usu-

ally imply architectural design search and/or heavy hyperparameter selection.

* As backpropagation is only applied on the last projector of the vision encoder, training is fast, requiring only few seconds like LP++ [17].

* ProLIP uses native text embeddings as classification weights in the few-shot task, following CLIP pretraining, and therefore preserves its open-class capability.

* It balances pretraining and adaptation with a regularizer that constrains the distance w.r.t. the pretrained weights. Our simple method performs on par with or better than the literature on few-shot adaptation, few-shot domain generalization, cross-dataset generalization and base-to-new class generalization. Additionally, ProLIP significantly outperforms prompt tuning in test-time adaptation, which is an unsupervised learning setting.

## 2. Related work

**Parameter-efficient fine-tuning (PEFT).** The advent of increasingly larger pretrained vision foundation models with excellent generalization capabilities has opened the way to new transfer learning approaches towards downstream tasks with limited labeled data. Full fine-tuning of such models turns out to be not only computationally inefficient but also often underperforming, even when compared to linear probing [21, 39]. Parameter-efficient fine-tuning methods aim to adapt models effectively with minimal changes of their parameters while freezing most of the large pretrained backbone. Side-tuning [45] trains a small network in parallel to a frozen pretrained network and avoids catastrophic forgetting. Optimizing only a specific subset of parameters of a model, e.g., bias terms [43], is also an effective strategy. However this still requires full backpropagation through the pretrained model. Adapter-tuning methods add lightweight modules to transformer layers [15, 32], but incur a higher runtime cost. LoRA [16] optimizes new low-rank matrices injected to transformer layers to approximate weight changes during fine-tuning, reducing significantly the number of parameters to learn. Prompt-tuning approaches, such as VPT [18], add a set of learnable prompts to the set of input patch embeddings. In addition to incurring supplementary computational cost for the full backpropagation and for runtime for most of them, these methods are specifically devised for transformer layers and are not directly applicable to convolutional nets.

**Few-shot CLIP adaptation.** CLIP's specific interaction between text and image features has enabled new adaptation methods fully exploiting this property in particular for the few-shot regime. Inspired by prompt tuning in natural language processing [23, 47], Zhou et al. [49] proposed context optimization (CoOp) which applies the same concept for pretrained vision-language models. CoOp was
later shown not to generalize well on unseen classes within the same dataset. Thus, conditional context Optimization (CoCoOp) [48] adds a meta-network that generates input-conditional tokens in addition to the learnable vectors, making optimized context less prone to overfitting to the seen classes. Zhu et al. [50] identified a critical problem of unconstrained prompt tuning methods: in extreme low-shot settings, overfitting can even decrease the zero-shot performance. They proposed regularizing the training by only updating prompts whose gradients do not conflict the direction resulting from zero-shot predictions. PLOT [4] applies optimal transport on sets of text and visual features to learn the transport plan between the two sets in an inner loop, which is fixed in the outer loop where prompts are learned. MaPLe [19] learns prompts in both vision and text branches at input and intermediate layers with a coupling function.

A simple approach for few-shot CLIP is training a linear probe on top of the visual features [30]. Lin et al. [24] show that adding a text describing the class to the training data of few-shot images largely boosts linear probing. Huang et al. [17] blend text embeddings with classification weights using class-wise learnable parameters.

Instead of adapting the model in the input space or training a linear probe, CLIP-adapter [9] adds an MLP on top of the features in the shared embedding space, with a residual connection to preserve the pretrained knowledge. Zhang et al. [46] showed that training-free adapters can be competitive to trained ones. They create a cache-model for the few-shot training set from the visual features and the corresponding ground-truth labels, which are converted to the weights of the MLP adapter. Following the success of training-free CLIP adaptation, Wang et al. [38] resorted to Gaussian Discriminative Analysis (GDA) [1] which assumes the conditional distribution of features given the labels follows a multivariate normal distribution. Thus, they construct the GDA classifier using the mean vectors of each class and the inverse covariance matrix, and show that classification can be further improved by ensembling GDA and zero-shot classifiers. Our ProLIP takes advantage of CLIP's compatibility between text and images and adapts it to downstream tasks without additional learnable parameters or architecture changes.

# 3. Preliminaries

## 3.1. Zero-shot classification

We denote $f$ and $g$ the vision and text encoders of CLIP, respectively. During pretraining, CLIP learns a joint embedding space that pulls corresponding image-text representations closer together and pushes away dissimilar ones. At inference, given an image $\mathbf{l}$, one only needs the names of $K$ candidate classes to perform zero-shot classification:

$$ \hat{k} = \underset{k}{\operatorname{argmax}} v^T t_k, \qquad (1) $$

where $v = \frac{f(\mathbf{l}; \theta_f)}{\|f(\mathbf{l}; \theta_f)\|_2}$, $t_k = \frac{g(\mathbf{T}_k; \theta_g)}{\|g(\mathbf{T}_k; \theta_g)\|_2}$; $\theta_f$ and $\theta_g$ are the frozen parameters of $f$ and $g$, respectively; $\mathbf{T}_k$ is a text prompt describing the class $k$, e.g., “a photo of {class$_k$}”.

## 3.2. Few-shot classification

Given a set of $N$ labeled samples from each of $K$ classes, research has been conducted to efficiently adapt CLIP using this set. All existing research in this direction can be gathered under three main avenues.

**Prompt Tuning.** It parameterizes the prompt template, i.e., $T_k = [\mathbf{w}]_1[\mathbf{w}]_2...[\mathbf{w}]_M[\text{class}_k]$. $[\mathbf{w}]_1[\mathbf{w}]_2...[\mathbf{w}]_M$ are learned while keeping $f$ and $g$ frozen. Prompt tuning adapts CLIP “indirectly” on the classifier side, i.e., the text embeddings are derived from the learned prompts.

**Adapters.** They learn a multi-layer perceptron (MLP) $h$ with a residual connection $\alpha$ on top of the frozen visual features $v$, i.e., $v := \alpha v + (1-\alpha)h(v)$, or on top of the frozen text features $t$, i.e., $t := \alpha t + (1-\alpha)h(t)$, or both.

**Linear probing.** It trains a linear classifier $W \in \mathbb{R}^{D \times K}$ on top of the frozen visual features, $D$ being the embedding space dimension. Matrix $W$ can be initialized with text embeddings $t_k$ as its columns. Since the classifier is directly tuned, linear probing restricts CLIP to $K$ classes after adaptation and cannot be applied in open-class setting.

## 3.3. CLIP architecture

CLIP adopts a transformer architecture [36] for the text encoder, but the vision encoder may be either a ResNet [11] or a Vision Transformer (ViT) [7]. We detail both architectures below and later elaborate on our unified method applicable to both architectures regardless of their intrinsic differences.

**ResNet.** CLIP replaces the global average pooling layer in ResNet with an attention pooling layer. The output of the multi-head attention layer is then projected to the shared latent space using a linear layer. Thus, $f$ can be written as $f = f_1 \circ f_2$, where $f_1$ represents all the layers up to the attention pooling (included), and $f_2$ represents the linear projection head. Given an image $\mathbf{l}$:

$$ x_o = f_1(\mathbf{l}), \quad v = f_2(x_o) = W_o^T x_o + b_o, \quad (2) $$

with $x_o \in \mathbb{R}^{D_o}$ the output of the attention pooling layer, $W_o \in \mathbb{R}^{D_o \times D}$ the projection matrix and $b_o$ a bias term.

**ViT.** The transformer encoder consists of multiple residual attention blocks. Each block has two main components: a multi-head self-attention and a feed-forward neural network (MLP), with residual connections. The output of the last residual attention block is projected to the latent space using a trainable matrix. Thus, $f$ can be written as $f = f_1 \circ f_2$,
<figure><img src="image_2.png" /><figcaption>**Figure 2.** ProLIP for few-shot adaptation. Whether the vision encoder is a CNN or a Transformer, ProLIP trains only the final linear layer that projects the visual embeddings into the shared latent space. The text encoder is frozen, and the text embeddings of the K target concepts are used as classification weights. Training with cross-entropy is regularized by a squared error loss ensuring weights of the projection layer to remain close to pretrained ones.</figcaption></figure>

where $f_1$ represents all the layers up to the last residual attention block (included), and $f_2$ represents the projection matrix. Given an image $\mathbf{l}$:

$$ \mathbf{x}_o = f_1(\mathbf{l}), \quad \mathbf{v} = f_2(\mathbf{x}_o) = \mathbf{W}_o^\top \mathbf{x}_o, \quad (3) $$

where no bias term is included, unlike Eq. (2).

Similarly on the text side, the embeddings are projected into the shared latent space using a linear layer.

## 4. ProLIP

As discussed in Sec. 3.3, CLIP projects both visual and text embeddings into the shared latent space using linear layers. We show that fine-tuning only the projection matrix $\mathbf{W}_o$ in Eq. (2) or Eq. (3) can be a strong alternative to prompt tuning and feature adapters. Specifically, the probability that a sample $i$ belongs to the class $k$ is computed as the Softmax over cosine similarities of image-text embeddings:

$$ p_{ik}(\mathbf{W}_o) = \frac{\exp((\mathbf{W}_o^\top \mathbf{x}_{oi} + \mathbf{b}_o)^\top \mathbf{t}_k / \tau)}{\sum_{j=1}^{K} \exp((\mathbf{W}_o^\top \mathbf{x}_{oi} + \mathbf{b}_o)^\top \mathbf{t}_j / \tau)}, \quad (4) $$

with $\mathbf{t}_k$ being fixed since $g$ is frozen, $\tau$ the pretraining temperature parameter, and $\mathbf{x}_{oi}$ the pre-projection embedding of sample $i$. The matrix $\mathbf{W}_o$ is learned with gradient descent using a cross-entropy loss $L(\mathbf{W}_o)$:

$$ L(\mathbf{W}_o) = -\frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} y_{ik} \log p_{ik}(\mathbf{W}_o), \quad (5) $$

where $y_{ik}$ is the ground truth.

**Regularization.** CLIP encoders map text and image modalities into a common latent space where strong image-text representation correspondences are established. We argue that unconstrained fine-tuning can lead to forgetting of the rich pretraining knowledge that appears through non-trivial

zero-shot classification accuracies. Thus, a good fine-tuning strategy should balance pretraining knowledge preservation and adaptation to downstream task. Consequently, to prevent significant drift from the pretraining weights (i.e., knowledge forgetting), we regularize the training with the Frobenius norm of the difference between the pretrained and fine-tuned matrices. The total loss is:

$$ \text{Loss} = L(\mathbf{W}_o) + \lambda \|\mathbf{W}_o - \mathbf{W}_o^{(0)}\|_{F}^{2}, \quad (6) $$

where $\mathbf{W}_o^{(0)}$ denotes the pretrained value of $\mathbf{W}_o$. We show later that $\lambda$ can be chosen as a decreasing function of the number of shots, as overfitting risk increases with less data [10]. The method is illustrated in Fig. 2.

**An argument on simplicity and practicality.** Algorithm 1 provides a PyTorch-like [29] pseudo-code for ProLIP, showing that it is extremely simple to implement. Also, it can be applied on pre-processed data, which makes it also extremely fast to train. Practically, we run the inference only one time on the text encoder side to get the classification weights. On the vision encoder side, we save the output embeddings $\mathbf{x}_o$ with different augmentation views since backpropagation is limited to the projection matrix.

Despite its extreme simplicity, adapting CLIP using our approach has not been proposed in the literature. Moreover, its architecture-agnostic nature makes it generic and suitable to different multi-modal pretrained networks. We argue that, due to intrinsic differences across architectures (e.g., ViT vs. ResNet), finding a unified method to efficiently fine-tune vision-language models based on the pretrained weights is not trivial. ProLIP is a first to achieve this goal.

**Algorithm 1** PyTorch-like pseudo-code for ProLIP.

```python
# target: Ground truth
# lmda: regularization loss weight
# Wo : Pretrained projection matrix
# xo: output visual embeddings (N*K, Do)
# text_weights: normalized embeddings of classnames (K,D)

# Copy initial weights for use in the regularization loss
Wo_0 = copy.deepcopy(Wo)
# Set last projection matrix as trainable weights
Wo requiresGrad = True
bo requiresGrad = False

v = xo @ Wo + bo
v = 12 normalize(v, dim=-1)

#compute the cosine similarity scores
logits = 100. * v @ text_weights.T

#compute regularized loss
SE_loss = nn.MSELoss(reduction='sum')
loss = CE_loss(logits, target) + lmda * SE_loss(Wo, Wo_0)
```

## 5. Experiments

**Datasets.** Following previous CLIP-based few-shot learning works, we experimentally test ProLIP on 11 datasets
for few-shot classification and base-to-new generalization: ImageNet [6], SUN397 [40], DTD [5], Caltech101 [8], UCF101 [35], Flowers102 [27], Stanford-Cars [20], FGVCAircraft [25], EuroSAT [12], Oxford-Pets [28] and Food101 [2]. For domain generalization experiments we follow ProGrad [50], using ImageNet as source dataset and testing on ImageNet-V2 [31], ImageNet-Sketch [37], ImageNet-A [14] and ImageNet-R [13] as out-of-distribution datasets. For the cross-dataset transfer experiment, ProLIP is trained on ImageNet and evaluated on the other 10 datasets, similar to ProGrad [50]. For test-time adaptation, we use ImageNet and its out-of-distribution (OOD) variants similarly to TPT [33].

**Training details.** We follow previous works and use $N \in \{1, 2, 4, 8, 16\}$ shots as support training set for few-shot classification. LP++ [17] identifies a flaw in the few-shot CLIP literature [46], which is the use of a large validation set for hyperparameter tuning. Instead, authors of LP++ propose using a validation set with as many shots as in the training set. We adopt this protocol in our few-shot classification experiments (cf. Sec. 5.1), though also experimenting in a more realistic setting without use of any validation set [34], which is adopted starting from Sec. 5.2. Huang et al. [17] also remark that prior works evaluate their methods based on one or three support sets, leading to large standard deviations when the few-shot set is not representative of the class distribution. We follow their practice and evaluate ProLIP on 10 random seeds (i.e., support training sets) for each dataset. For domain generalization and cross-dataset transfer experiments, we select $N=4$ like ProGrad [50]. For base-to-new generalization, we select $N=4$ like ProGrad [50] when using ResNet-50 and $N=16$ like MaPLe [19] when using ViT-B/16. Unless otherwise stated, we employ ResNet-50 with CLIP weights as the visual encoder, similarly to the literature. Training runs for 300 epochs on a full-batch of features, requiring few seconds on one Tesla V100. For ProLIP, the learning rate (LR) and regularizer loss weight $\lambda$ are selected by grid search on the few-shot validation set, with $LR \in \{10^{-2}, 10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}, 10^{-7}, 10^{-8}\}$ and $\lambda \in \{10, 1, 10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}, 0\}$. We later show that using the regularizer ($\lambda > 0$) is better to avoid severe overfitting, and that state-of-the-art results can be still achieved with a fixed LR over all the datasets, and a fixed $\lambda$ chosen as a decreasing function of the number $N$ of shots.

## 5.1. Few-shot classification

We compare ProLIP to baselines covering the variety of existing adaptation strategies. For prompt tuning methods, we report CoOp [49] and its other variants PLOT [4], KgCoOp [41] and ProGrad [50]. For adapters, we compare to CLIP-adapter [9] and Tip-adapter [46]. Note that Tip-adapter performance is reported in two settings follow-

<table><caption>Table 1. Few-shot image classification based on CLIP. We report the classification accuracy (%) averaged over 11 datasets and 10 support sets, along with standard deviation, comparing ProLIP to other baselines. We highlight best and 2nd best. First row provides zero-shot classification accuracy for reference.</caption><thead><tr><th>Method</th><th>N = 1</th><th>2</th><th>4</th><th>8</th><th>16</th></tr></thead><tbody><tr><td>CLIP</td><td colspan="5">58.89</td></tr><tr><td colspan="6"><strong>Prompt tuning</strong></td></tr><tr><td>CoOp [49]</td><td>59.62 ±3.11</td><td>63.80 ±2.32</td><td>67.23 ±1.64</td><td>71.30 ±0.86</td><td>74.06 ±0.55</td></tr><tr><td>PLOT [4]</td><td>61.51 ±2.91</td><td>65.67 ±2.06</td><td>68.39 ±1.17</td><td>71.96 ±0.70</td><td>74.35 ±0.66</td></tr><tr><td>KgCoOp [41]</td><td>61.36 ±3.04</td><td>63.23 ±2.06</td><td>65.73 ±1.15</td><td>67.50 ±1.11</td><td>69.01 ±0.79</td></tr><tr><td>ProGrad [50]</td><td>62.46 ±1.89</td><td>65.88 ±1.46</td><td>68.52 ±1.15</td><td>71.82 ±0.11</td><td>73.95 ±0.68</td></tr><tr><td colspan="6"><strong>Adapters</strong></td></tr><tr><td>CLIP-Adapter [9]</td><td>60.32 ±0.80</td><td>61.93 ±0.93</td><td>65.12 ±0.80</td><td>69.20 ±0.56</td><td>72.57 ±0.54</td></tr><tr><td>Tip-Adapter-F [46]</td><td>61.29 ±0.92</td><td>62.94 ±0.75</td><td>66.02 ±0.80</td><td>69.88 ±0.51</td><td>73.82 ±0.55</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>63.06 ±1.05</td><td>66.47 ±0.65</td><td>68.71 ±0.96</td><td>71.78 ±1.00</td><td>74.37 ±0.35</td></tr><tr><td colspan="6"><strong>Linear Probing</strong></td></tr><tr><td>LP [30]</td><td>36.10 ±1.43</td><td>46.99 ±1.29</td><td>56.72 ±1.20</td><td>64.66 ±0.55</td><td>70.56 ±0.44</td></tr><tr><td>LP++ [17]</td><td>63.43 ±0.90</td><td>66.20 ±0.72</td><td>69.16 ±0.79</td><td>72.04 ±0.46</td><td>74.42 ±0.45</td></tr><tr><td colspan="6"><strong>Model weights</strong></td></tr><tr><td>ProLIP</td><td><strong>64.59</strong> ±0.98</td><td><strong>67.09</strong> ±0.87</td><td><strong>70.53</strong> ±0.69</td><td><strong>73.40</strong> ±0.45</td><td><strong>76.55</strong> ±0.41</td></tr></tbody></table>

ing [17]: Tip-adapter-F where its two crucial hyperparameters are set to 1 and the validation set is used for early stopping, and Tip-adapter-F* where intensive hyperparameter search is performed to find the best values of the same hyperparameters based on the same validation set. For linear probing, we report LP [30] and LP++ [17]. Baseline results are taken from [17] which employs early stopping for all the methods based on the validation set. For a fair comparison we follow the same experimental protocol (i.e., few-shot validation set, 10 random support sets) although we stress that we do not use early stopping but rather report the last model performance.

Tab. 1 reports the average classification accuracy and standard deviation, across 11 datasets. Per-dataset performances are reported in Appendix A. In all few-shots settings (i.e., $N \in \{1, 2, 4, 8, 16\}$), ProLIP clearly outperforms all the baselines, showing a great potential of the extremely simple approach of fine-tuning the last visual linear projector for adaptation.

**Hyperparameters sensitivity.** The benefit of the regularization appears by testing ProLIP with different hyperparameters, which are fixed across all the datasets. Fig. 3 reports the average accuracy across the same 11 datasets, for 5 different LR values combined either with regularization ($\lambda \in \{10^{-2}, 10^{-1}, 1\}$) or without regularization ($\lambda=0$). For $\lambda=0$, we note that the accuracy drops dramatically for larger learning rates due to overfitting to the few-shot training set, and the subsequent drift from the robust pretrained CLIP representation. On the other hand, using the weight regularizer ($\lambda > 0$) makes ProLIP less prone to overfitting and less sensitive to learning rate.

This observation is corroborated by the statistics of the hyperparameters found by grid search (cf. Appendix E)
<figure><img src="image_3.png" /><figcaption>Figure 3. **ProLIP sensitivity to hyperparameter choice.** Accuracy of ProLIP to the hyperparameters (learning rate and regularization weight $\lambda$) for $N \in \{1, 2, 4, 8, 16\}$-shot settings. Each data point is an average over 11 datasets, 10 runs for each.</figcaption></figure>

showing that the best learning rates span a wide range of values. It follows that our regularization alleviates overfitting on the training set, allowing larger learning rate (e.g., $10^{-2}$). This important property motivates our investigation of a more realistic setting, where hyperparameters are never tuned: *Having no validation data*.

## 5.2. Few-shot classification without validation set

In contrast to [17], we argue that relying even on N-shot validation sets is a violation of the N-shot setting since it effectively requires access to $2N$ examples (N for training and N for validation). Subsequently, like Silva et al. [34], we advocate for a true N-shot setting, using no validation data at all. It follows that the true merit of a method stems from its lower sensitivity to hyperparameters, which we demonstrated in the previous section.

It can be observed from Fig. 3 that for lower-shot settings, higher $\lambda$ values lead to better accuracy, and vice versa. Therefore, we formulate $\lambda$ as a decreasing function of the number of shots N, reporting in Tab. 2a the average performance over learning rates $LR \in \{10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}\}$. It results that our simple parametric formulations of $\lambda$ (i.e., $1/N$, $1/N^2$) lead to almost identical, strong and stable results, competing with our state-of-the-art grid search variant, albeit without the need of a validation set. A byproduct of our regularizer is the reduced sensitivity to the learning rate (i.e., low variance as seen in Tab. 2a), whereas removing the regularizer (i.e., $\lambda = 0$) proves to result in dramatically large variance. Detailed numbers for each combination are reported in Tab. 11.

Therefore, in Tab. 2b we compare the average across 11 datasets of the validation-free baselines from [34], and the validation-free ProLIP variant, coined as ProLIP$_{\emptyset}$, with $\lambda = 1/N$ and average over the 4 tested learning rates. The reported performance shows a consistent improvement for any number of shots.

## 5.3. Generalization of few-shot models

Real-world scenarios impose an additional challenge of distribution shift for model adaptation. Supposing the test data to follow the same distribution as the training is often unrealistic, and a model can be of practical interest only if it ex-

<figure><img src="image_4.png" /><figcaption>Table 2. **Few-shot without validation set.** (a) ProLIP variants with performance averaged over 11 datasets, 10 runs, and 4 learning rates $LR \in \{10^{-5}, 10^{-4}, 10^{-3}, 10^{-2}\}$ to study sensitivity. Note the low variance of the parametric formulations ($\lambda = 1/N$, $\lambda = 1/N^2$) which are also reaching performance of the grid search variant — despite having no access to a validation set. (b) Compares to validation-free baselines from [34] and shows the consistent superiority of ProLIP$_{\emptyset}$.</figcaption></figure>

hibits resilient generalization capabilities when confronted with out-of-distribution data. Achieving this generalization in a few-shot framework is highly challenging yet important to benchmark when assessing the potential practical use of few-shot methods. While few-shot adaptation is crucial to optimize in-domain performance, such a domain-specific learning process often impairs the generalization capability of the vanilla CLIP model, which indeed remains a competitive baseline in generalization. In the following experiments, we evaluate few-shot methods on domains different from ones used in few-shot training. For CLIP, we use the 7 simple generic templates [30] that report more competitive generalization results than “a photo of a {class}”. Comparison is done only among the few-shot methods; the zero-shot CLIP performance is included as reference. For ProLIP$_{\emptyset}$, we systematically use $\lambda = 1/N$ and a fixed $LR$ of $10^{-5}$.

**Domain generalization.** In this setting, the set of classes is fixed in both in-domain and OOD datasets. Following Pro-Grad, we train ProLIP on ImageNet (IN) as source dataset (with the number of shots $N=4$), and assess it on ImageNet-V2 (IN-V2), ImageNet-Sketch (IN-S), ImageNet-A (IN-A) and ImageNet-R (IN-R). Tab. 3 shows that ProLIP is on par
<table><caption>**Table 3. Domain generalization.** 4-shot training on ImageNet (source) and evaluation on OOD variants (IN-V2, IV-S, IN-A, IN-R) with different visual backbones. We report accuracy over source ImageNet (IN) along with the average only over OOD variants to show generalization ('OOD'). Baselines are 3 runs averages reported from ProGrad [50]. We report results of two ProLIP variants: the standard ProLIP using few-shot validation set and the validation-free ProLIP<sub>&empty;</sub>. All the baselines use validation set.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">RN50</th><th colspan="2">RN101</th><th colspan="2">ViT-B/16</th><th colspan="2">ViT-B/32</th></tr><tr><th>IN</th><th>OOD</th><th>IN</th><th>OOD</th><th>IN</th><th>OOD</th><th>IN</th><th>OOD</th></tr></thead><tbody><tr><td>CLIP</td><td>60.34</td><td>43.31</td><td>61.24</td><td>48.71</td><td>68.79</td><td>59.87</td><td>62.00</td><td>50.06</td></tr><tr><td>LP</td><td>41.29</td><td>21.19</td><td>47.01</td><td>28.33</td><td>54.70</td><td>35.09</td><td>46.77</td><td>28.81</td></tr><tr><td>CoOp</td><td>61.34</td><td>40.84</td><td>63.99</td><td>47.48</td><td>69.86</td><td>58.32</td><td>64.74</td><td>48.06</td></tr><tr><td>CoCoOp</td><td>61.04</td><td>40.42</td><td>63.59</td><td>47.34</td><td>70.13</td><td>58.17</td><td>64.63</td><td>47.93</td></tr><tr><td>Prograd</td><td>62.17</td><td>42.23</td><td>64.98</td><td>48.53</td><td>70.45</td><td>59.05</td><td>65.36</td><td>49.39</td></tr><tr><td>ProLIP</td><td><b>62.37</b></td><td><b>43.25</b></td><td><b>65.13</b></td><td><b>48.84</b></td><td><b>70.78</b></td><td><b>59.92</b></td><td><b>65.90</b></td><td><b>50.01</b></td></tr><tr><td>ProLIP<sub>&empty;</sub></td><td><b>62.55</b></td><td>43.20</td><td>65.11</td><td>48.81</td><td><b>70.94</b></td><td><b>59.97</b></td><td><b>66.00</b></td><td>49.86</td></tr></tbody></table>

with or better than other methods both on source and unseen target domains, for both ResNet and ViT CLIP backbones. Our consistent improvements over zero-shot CLIP highlight the merit of ProLIP in preserving generalization while still maximizing in-domain results (IN), whereas the other methods fall short in generalization. The validation-free ProLIP<sub>&empty;</sub> performs on-par with ProLIP.

**Cross-dataset generalization.** This is an interesting generalization setting previously addressed in prompt tuning works [48, 50] where OOD datasets not only come from other domains but may also contain different or more fine-grained classes compared to ones in source. Tab. 4 shows the generalization from ImageNet as source dataset (4-shot) to the 10 other datasets. ProLIP outperforms ProGrad on 6 out of 11 datasets and on average. However, it is worth noting that zero-shot CLIP remains the strongest baseline in this setting. As argued in CoCoOp [48], ImageNet contains 1000 classes, mainly consisting of objects. Dog breeds are also present, so good generalization (or at most small zero-shot performance drop) to datasets like OxfordPets and Caltech101 is expected. However, for datasets presenting a larger gap (e.g., fine-grained and/or specialized datasets), generalization is expected to be lower. For such datasets, like FGVCAircraft and DTD, ProLIP outperforms other adaptation methods, but remains behind zero-shot accuracy. Among all the target datasets, all methods exhibit a significant drop on EuroSAT (26-35% from the zero-shot model). Interestingly, ProLIP not only does not exhibit the same drop, but retains the zero-shot performance. In short, looking at the generalization of ProLIP on each of the 10 datasets, our method is overall retaining zero-shot capability the most and showing better cross-dataset transferability. Again, we observe similar results of the two ProLIP variants, further consolidating the potential of the more straightforward and less experimentally intensive ProLIP<sub>&empty;</sub>.

**Base-to-new generalization.** In this setting, we divide all

<table><caption>**Table 4. Cross-dataset generalization.** Training is performed on 4-shot ImageNet (source). The learned models are evaluated on 10 other datasets (target). Baselines' scores are average of 3 runs reported from ProGrad [50]. We report results of two ProLIP variants: the standard ProLIP using few-shot validation set and the validation-free ProLIP<sub>&empty;</sub>.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Source</th><th colspan="10">Target</th><th rowspan="2">Average</th></tr><tr><th>ImageNet</th><th>Caltech101</th><th>OxfordPets</th><th>StanfordCars</th><th>Flowers102</th><th>Fosd101</th><th>FGVCAircraft</th><th>SUN397</th><th>DTD</th><th>Eurosat</th><th>UCF101</th></tr></thead><tbody><tr><td>CLIP</td><td>60.35</td><td>85.84</td><td>85.75</td><td>55.78</td><td>65.98</td><td>77.35</td><td>17.07</td><td>58.85</td><td>42.69</td><td>36.22</td><td>61.80</td><td>58.88</td></tr><tr><td>CoOp</td><td>61.34</td><td>84.48</td><td>85.99</td><td>54.16</td><td>60.10</td><td>75.48</td><td>14.09</td><td>57.48</td><td>35.32</td><td>26.72</td><td>57.56</td><td>55.70</td></tr><tr><td>CoCoOp</td><td>61.04</td><td>84.73</td><td><b>86.42</b></td><td>52.34</td><td>61.24</td><td>73.79</td><td>13.74</td><td>55.94</td><td>36.60</td><td>23.46</td><td>57.97</td><td>55.21</td></tr><tr><td>Prograd</td><td>62.17</td><td><b>88.30</b></td><td><b>86.43</b></td><td><b>55.61</b></td><td>62.69</td><td><b>76.76</b></td><td>15.76</td><td><b>60.16</b></td><td>39.48</td><td>24.87</td><td>58.70</td><td>57.36</td></tr><tr><td>ProLIP</td><td>62.37</td><td>87.08</td><td>84.57</td><td>54.56</td><td><b>64.79</b></td><td>75.56</td><td>16.94</td><td>59.78</td><td>40.96</td><td>36.31</td><td><b>61.24</b></td><td><b>58.56</b></td></tr><tr><td>ProLIP<sub>&empty;</sub></td><td><b>62.55</b></td><td>86.99</td><td>84.00</td><td>54.19</td><td>64.03</td><td>74.95</td><td><b>16.99</b></td><td>59.67</td><td><b>41.09</b></td><td><b>36.55</b></td><td>60.95</td><td>58.36</td></tr></tbody></table>

classes into two groups: base and new classes. Training is performed on base classes and testing on both base and new classes. Moreover, the harmonic mean is reported to assess the trade-off. Here, we simply use the validation-free variant (i.e., ProLIP<sub>&empty;</sub>). In Tab. 5, we see that ProLIP<sub>&empty;</sub> significantly outperforms ProGrad [50] in total harmonic mean across 11 datasets. Additionally, ProLIP<sub>&empty;</sub> is competitive to MaPLe [19], which consists of multi-modal prompt learning for ViT architectures.

Metrics details. Previous works [19, 50] calculate the total harmonic mean over datasets in two different ways.

We report in Tab. 5 for each architecture both ways of calculating the total harmonic means, renaming them $H_{t1}$ and $H_{t2}$. Our method is superior to baselines, regardless of the total harmonic mean used. We also detail the computation below.

In ProGrad [50], the total harmonic mean over the 11 datasets is computed as the average harmonic means of individual datasets. This writes:

$$H_{t1} = \frac{1}{11} \sum_{i=1}^{11} HM_i, \qquad (7)$$

$HM_i = 2 \times \frac{acc_{bi} \times acc_{bi}}{acc_{bi} + acc_{ni}}$ being the harmonic mean of dataset $i$. Here, $acc_{bi}$ and $acc_{ni}$ denote the accuracy on base and new classes for dataset $i$, respectively.

Instead in MaPLe [19], the total harmonic mean over the 11 datasets is calculated as the harmonic mean of average base and average new classes accuracies, being:

$$H_{t2} = 2 \times \frac{acc_b \times acc_n}{acc_b + acc_n}, \qquad (8)$$

where $acc_b = \frac{1}{11} \sum_{i=1}^{11} acc_{bi}$ and $acc_n = \frac{1}{11} \sum_{i=1}^{11} acc_{ni}$. Per-dataset performance is reported in Appendix B.

## 5.4. Analyses

**Number of augmented views.** Following the literature [17, 46], we apply RandomResizedCrop and
<table><thead><tr><th></th><th>Base</th><th>New</th><th>H<sub>1</sub></th><th>H<sub>2</sub></th></tr></thead><tbody><tr><th>CLIP</th><td>61.72</td><td>65.91</td><td>63.64</td><td>63.75</td></tr><tr><th>CoOp</th><td>71.96</td><td>61.26</td><td>65.58</td><td>66.18</td></tr><tr><th>CoCoOp</th><td>72.23</td><td>60.77</td><td>65.35</td><td>66.01</td></tr><tr><th>ProGrad</th><td>73.29</td><td>65.96</td><td>69.06</td><td>69.43</td></tr><tr><th>ProLIP<sub>ø</sub></th><td><b>75.45</b></td><td><b>69.43</b></td><td><b>72.12</b></td><td><b>72.31</b></td></tr></tbody></table>

(a) ResNet-50

<table><thead><tr><th></th><th>Base</th><th>New</th><th>H<sub>1</sub></th><th>H<sub>2</sub></th></tr></thead><tbody><tr><th>CLIP</th><td>69.34</td><td>74.22</td><td>71.59</td><td>71.70</td></tr><tr><th>CoOp</th><td>82.69</td><td>63.22</td><td>70.83</td><td>71.66</td></tr><tr><th>CoCoOp</th><td>80.47</td><td>71.69</td><td>75.44</td><td>75.83</td></tr><tr><th>MaPLe</th><td>82.28</td><td><b>75.14</b></td><td>78.27</td><td>78.55</td></tr><tr><th>ProLIP<sub>ø</sub></th><td><b>84.01</b></td><td>73.86</td><td><b>78.28</b></td><td><b>78.61</b></td></tr></tbody></table>

(b) ViT-B/16

<figure><img src="image_5.png" /><figcaption>Figure 4. **Ablation and UMAP Visualization.** (a) Ablation of ProLIP using varying number of views and shots. (b) and (c) UMAP of Zero-shot CLIP vs ProLIP on EuroSAT, showing that some classes (e.g., 'Pasture Land', 'Permanent Crop Land', 'Sea or Lake', etc.) are better clustered with our method.</figcaption></figure>

<table><caption>Table 6. **Comparison to full fine-tuning.** We report the classification accuracy (%) averaged over 11 datasets and 10 support sets, along with standard deviation, comparing ProLIP<sub>ø</sub> to full fine-tuning of the vision encoder.</caption><thead><tr><th>Method</th><th>N = 1</th><th>2</th><th>4</th><th>8</th><th>16</th></tr></thead><tbody><tr><td>CLIP</td><td></td><td>58.89</td><td></td><td></td><td></td></tr><tr><td>Full Fine-tuning</td><td>46.09 ±6.33</td><td>51.85 ±5.32</td><td>58.06 ±6.19</td><td>62.22 ±1.23</td><td>67.74 ±0.68</td></tr><tr><td>ProLIP<sub>ø</sub> (λ = 0)</td><td>62.91 ±0.85</td><td>65.97 ±0.67</td><td>69.76 ±0.55</td><td><b>72.57</b> ±0.35</td><td><b>75.73</b> ±0.30</td></tr><tr><td>ProLIP<sub>ø</sub> (λ = 1/N)</td><td><b>64.67</b> ±0.63</td><td><b>66.80</b> ±0.39</td><td>69.73 ±0.37</td><td>72.44 ±0.33</td><td>75.34 ±0.31</td></tr><tr><td>ProLIP<sub>ø</sub> (λ = 1/N<sup>2</sup>)</td><td><b>64.67</b> ±0.63</td><td><b>67.04</b> ±0.42</td><td><b>70.27</b> ±0.44</td><td><b>72.84</b> ±0.32</td><td><b>75.77</b> ±0.30</td></tr></tbody></table>

RandomHorizontalFlip augmentations during training. As mentioned earlier, ProLIP can be applied on pre-computed visual embeddings (before the projection layer). We ablate the number of views in which the features are saved. Fig. 4a shows that average accuracy over 11 datasets increases with more views. Interestingly, ~ 10 views are sufficient to get results close to those with 300 views. In contrast, Lin et al. [24] showed that the gain saturates after more than two views for their cross-modal linear probe.

**Comparison to full fine-tuning.** We compare ProLIP<sub>ø</sub> with full fine-tuning of the visual backbone. Results in Tab. 6 show that full fine-tuning is far behind ProLIP<sub>ø</sub>, and even degrades zero-shot performance for $N = 1, 2$ and 4-shots. The learning rate is $10^{-5}$ for these experiments, and ProLIP<sub>ø</sub> is shown for different $\lambda$ values (including $\lambda = 0$). These results confirm that full fine-tuning faces a high risk of overfitting especially in low-shot regimes, advocating for parameter-efficient fine-tuning methods like ProLIP.

**Visualization.** We use UMAP to visualize EuroSAT test set feature manifolds, before and after 16-shot training (i.e.,

<table><caption>Table 7. **Robustness to natural distribution shifts in test-time adaptation.** Experiments are done with RN50 backbone, without few-shot training on IN (top) and with 16-shot training (bottom).</caption><thead><tr><th>Method</th><th>IN</th><th>IN-A</th><th>IN-V2</th><th>IN-R</th><th>IN-S</th><th>Average</th><th>Avg. OOD</th></tr></thead><tbody><tr><td>CLIP</td><td>60.33</td><td>23.79</td><td>53.31</td><td>60.58</td><td>35.46</td><td>46.69</td><td>43.29</td></tr><tr><td colspan="8">w/o few-shot training on IN</td></tr><tr><td>TPT [33]</td><td>60.74</td><td>26.67</td><td>54.70</td><td>59.11</td><td>35.09</td><td>47.26</td><td>43.89</td></tr><tr><td>ProLIP<sub>test-time</sub></td><td><b>62.00</b></td><td><b>33.76</b></td><td><b>56.03</b></td><td><b>62.69</b></td><td><b>37.29</b></td><td><b>50.35</b></td><td><b>47.44</b></td></tr><tr><td colspan="8">w/ 16-shot training on IN</td></tr><tr><td>CoOp [33]</td><td>63.33</td><td>23.06</td><td>55.40</td><td>56.60</td><td>34.67</td><td>46.61</td><td>42.43</td></tr><tr><td>TPT + CoOp [33]</td><td>64.73</td><td>30.32</td><td>57.83</td><td>58.99</td><td>35.86</td><td>49.55</td><td>45.75</td></tr><tr><td>ProLIP</td><td>64.48</td><td>22.75</td><td>56.24</td><td>59.56</td><td>34.80</td><td>47.57</td><td>43.34</td></tr><tr><td>ProLIP<sub>test-time</sub> + ProLIP</td><td><b>66.90</b></td><td><b>32.96</b></td><td><b>58.77</b></td><td><b>61.78</b></td><td><b>36.97</b></td><td><b>51.48</b></td><td><b>47.62</b></td></tr></tbody></table>

zero-shot vs. ProLIP). The results are illustrated in Figs. 4b and 4c. We observe that the features are generally better clustered for ProLIP. Confusing categories like Highway or Road, Permanent Crop Land and Pasture Land exhibit remarkably better separation for our few-shot adapted model compared to zero-shot. This visualization hints that ProLIP learns better feature manifolds in the few-shot classification setting.

## 5.5. Test-time ProLIP

In this section, our goal is to show that ProLIP can be applied beyond supervised few-shot CLIP adaptation. Motivated by the risk of “overfitting” the source domain in classic prompt tuning methods [48, 49], Shu et al. [33] pioneered test-time prompt tuning (TPT), aiming to learn adaptive prompts on the fly using a single test image.

**TPT background knowledge.** TPT aims to learn a context specific to each test image in an unsupervised way. Given an unlabeled test image $I_{test}$, the prompt is learned by minimizing the average prediction entropy over different augmented views of $I_{test}$. Moreover, *confidence selection* filters out the augmented views with high entropy predictions, which might lack important information for classification. More details are provided in Appendix C.

**Test-time ProLIP.** We do not introduce a new way for CLIP test-time adaptation but simply follow the same experimental setting as TPT (i.e., 1-step entropy minimization of averaged prediction probability distribution, confidence selection), although ProLIP optimizes the projection weight matrix $W_o$ instead of the prompt as in TPT. We name this ProLIP variant as ProLIP<sub>test-time</sub>. Tab. 7 shows that ProLIP<sub>test-time</sub> yields superior results to TPT on ImageNet and natural distribution shifts, while being one order of magnitude faster to train. For direct comparison, we separate methods that perform 16-shot training on ImageNet. Of note, even without few-shot training, ProLIP<sub>test-time</sub> still outperforms CoOp and TPT+CoOp. We further advance ProLIP<sub>test-time</sub> results with 16-shot training.
# 6. Conclusion

We propose an extremely simple way for efficient adaptation of CLIP based on the model weights. We identify that fine-tuning the last visual projection matrix, which projects the visual embedding into the multi-modal latent space, is a strong alternative for few-shot classification with CLIP. Moreover, we show advantages of including a squared error regularizer: it prevents the drift from pretrained weights and improves robustness to hyperparameter choice, thus making our method a trustworthy candidate for realistic few-shot adaptation. Additionally, we provide corroboration on the competitiveness of ProLIP in few-shot classification, domain generalization, cross-dataset transfer, base-to-new generalization and test-time adaptation, rendering it a potential general framework for further applications.

**Limitation.** Many recent pretrained models are only accessible via APIs (e.g., GPT, Claude, Gemini), providing only encoder endpoints. Similarly to prompt tuning methods, ProLIP does not apply to such closed models.

**Future directions.** Our framework can be applied to other foundation models with different modalities, downstream tasks and training objectives [22, 44]. Future research can explore other alternatives for model weights based adaptation, and theoretical investigation on the effect of fine-tuning the last projection matrix.

**Acknowledgment.** This work was partially funded by French project SIGHT (ANR-20-CE23-0016). It was performed using HPC resources from GENCI-IDRIS (Grants AD011014477R1, AD011012808R3). The authors thank Clément Weinreich for insightful discussion.

# Appendix

We provide additional details on few-shot classification in Appendix A, base-to-new generalization in Appendix B and test-time adaptation in Appendix C. Further, we provide additional details on the training of ProLIP in Appendix D and finally a detailed hyperparameter study in Appendix E.

## A. Details on few-shot classification

In addition to the average across datasets in Tab. 1, we report the *per-dataset performance* of all methods in Tabs. 14 and 15, reporting for each the average accuracy over 10 seeds (i.e., support sets). ProLIP performs particularly well on DTD, UCF101, StanfordCars, FGVCAircraft and EuroSAT. For some specific settings, e.g., 1-shot DTD, 1-shot EuroSAT, 16-shot StanfordCars, 8 and 16-shot FGVCAircraft, the improvements over state-of-the-art are significant. On the other hand, for datasets like OxfordPets and Food101, where the zero-shot performance is already good, ProLIP and other baselines are outperformed by prompt learning methods (e.g., ProGrad). This can be related to the

relatively lower number of parameters in the latter, making them less prone to overfitting in very low-shot settings; when the number of shots increases, e.g., 8-16 shots, ProLIP and prompt learning perform on par.

Future research may include the zero-shot accuracy on the few-shot training set in the parametric formulation of the regularization loss weight (i.e., $\lambda$). That is, the higher the zero-shot accuracy, the smaller should be the distance between the fine-tuned projection matrix and the pretrained one (i.e., higher $\lambda$).

## B. Details on base-to-new generalization

**Per-dataset performance.** In addition to average performance reported in Tab. 5, we report in Tab. 12 and Tab. 13 the per-dataset accuracy for base and new classes, as well as the harmonic mean metrics.

## C. Details on test-time ProLIP

TPT [33] learns a single prompt for each test image using an unsupervised loss function. Given a test image $\mathbf{l}_{\text{test}}$, the image is augmented $N_{\text{views}}$ times using a family of random augmentations $\mathcal{A}$. Predictions are made for each view, and the training consists of minimizing the entropy of the averaged probability distribution of these predictions:

$$ p^* = \operatorname*{argmin}_{\mathbf{p}} - \sum_{i=1}^{K} \tilde{p}_{\mathbf{p}}(y_i | \mathbf{l}_{\text{test}}) \log \tilde{p}_{\mathbf{p}}(y_i | \mathbf{l}_{\text{test}}), \quad (9) $$

where

$$ \tilde{p}_{\mathbf{p}}(y_i | \mathbf{l}_{\text{test}}) = \frac{1}{N_{\text{views}}} \sum_{i=1}^{N_{\text{views}}} p_{\mathbf{p}}(y_i | \mathcal{A}_i(\mathbf{l}_{\text{test}})). \quad (10) $$

In addition, *confidence selection* is used to filter out predictions with high entropy, which are considered as noisy. Self-entropy is computed for each of the $N_{\text{views}}$, a fixed cutoff percentile $\rho$ keeps only predictions with lower entropy than $\tau$. $\tilde{p}_{\mathbf{p}}$ in Equation 9 becomes:

$$ \tilde{p}_{\mathbf{p}}(y | \mathbf{l}_{\text{test}}) = \frac{1}{\rho N} \sum_{i=1}^{N_{\text{views}}} 1_{\{H(p_i) \le \tau\}} p_{\mathbf{p}}(y | \mathcal{A}_i(\mathbf{l}_{\text{test}})). \quad (11) $$

We apply the same framework (i.e., loss function, confidence selection) with the only difference of minimizing Equation 9 over $W_o$ instead of the prompt $\mathbf{p}$. For a fair comparison, we use the same number of steps for training (i.e., 1 step) and the same value of the cutoff percentile $\rho = 0.1$. The learning rate is $10^{-4}$. Note that, measured on ImageNet, ProLIP is ~13 times faster than TPT, as the latter requires backpropagation through the whole text encoder, while in our case backpropagation is limited to the last visual
projection layer and is not applied on the text encoder. We also stress that since we perform only 1 step of training, the regularization loss cannot be used as the first value it takes is 0 (initially the fine-tuned projection matrix is equal to the pre-trained one).

## D. ProLIP training details

The text encoder is fully frozen during training of ProLIP. The templates are similar to previous works [17, 46] for fair comparison, and are detailed in Tab. 8 for each dataset.

<table><caption>Table 8. Dataset-specific templates. Following the literature, all but ImageNet dataset and its variants use a single template.</caption><thead><tr><th>Dataset</th><th>Template</th></tr></thead><tbody><tr><td>Caltech101</td><td>"a photo of a {class}."</td></tr><tr><td>StanfordCars</td><td>"a photo of a {class}."</td></tr><tr><td>SUN397</td><td>"a photo of a {class}."</td></tr><tr><td>DTD</td><td>"{class} texture."</td></tr><tr><td>Eurosat</td><td>"a centered satellite photo of {class}."</td></tr><tr><td>FGVCAircraft</td><td>"a photo of a {class}, a type of aircraft."</td></tr><tr><td>Food101</td><td>"a photo of {class}, a type of food."</td></tr><tr><td>Flowers102</td><td>"a photo of a {class}, a type of flower."</td></tr><tr><td>OxfordPets</td><td>"a photo of a {class}, a type of pet."</td></tr><tr><td>UCF101</td><td>"a photo of a person doing {class}."</td></tr><tr><td>ImageNet</td><td>Ensemble of 7 templates:<br>"itap of a {class}.", "a bad photo of the {class}."</td></tr><tr><td>ImageNet-A</td><td>"a origami {}.", "a photo of the large {class}."</td></tr><tr><td>ImageNet-V2</td><td>"a {class} in a video game.", "art of the {class}."</td></tr><tr><td>ImageNet-R</td><td>"a photo of the small {class}."</td></tr><tr><td>ImageNet-Sketch</td><td></td></tr></tbody></table>

For training, only the weight matrix $W_o$ in Eq. (2) and Eq. (3) is fine-tuned. Note that for ResNets, a bias term $b_o$ exists while for ViTs no bias is added in pretraining. We stress that fine-tuning also the bias term for ResNets does not change the results, as most of the parameters as concentrated in the weight matrix. In detail for ResNet-50, $W_o \in \mathbb{R}^{D_o \times D}$, where $D_o = 2048$ and $D = 1024$, this makes a total of ~2M parameters, while $b_o \in \mathbb{R}^D$ has only 1024 parameters. Tab. 9 shows the number of trainable parameters in ProLIP for different backbones.

<table><caption>Table 9. Number of trainable parameters per backbone. It is the number of elements in the projection matrix $W_o \in \mathbb{R}^{D_o \times D}$.</caption><thead><tr><th>Backbone</th><th>$D_o \times D$</th><th>Parameters in $W_o$</th></tr></thead><tbody><tr><td>ResNet-50</td><td>2048 × 1024</td><td>2.097M</td></tr><tr><td>ResNet-101</td><td>2048 × 512</td><td>1.049M</td></tr><tr><td>ViT-B/32</td><td>768 × 512</td><td>0.393M</td></tr><tr><td>ViT-B/16</td><td>768 × 512</td><td>0.393M</td></tr></tbody></table>

## E. ProLIP hyperparameters study

**Grid search.** Fig. 5 shows the distribution of hyperparameters found by grid search on the few-shot validation set (cf. Tab. 1). We draw two observations:

1. The learning rates span a wide range of values, and high values like $10^{-3}$ and $10^{-2}$ are selected several times,

<figure><img src="image_6.png" /><figcaption>Figure 5. Hyperparameters selected by grid search. Learning rates and regularization loss weights $\lambda$ found with grid search on the few-shot validation set. The distribution of these hyperparameters are shown for each few-shot setting ($N = 1, 2, 4, 8, 16$).</figcaption></figure>

which would cause severe overfitting when no regularization is used (cf. Tab. 11 and Fig. 3).

2. $\lambda = 0$ is rarely selected, meaning that based on the few-shot validation set, regularized projection matrices generalize better.

**Hyperparameter sensitivity.** Tab. 10 complements Fig. 3, where ProLIP is trained for different fixed learning rates, with fixed regularization loss weight values (i.e., $\lambda$). Looking at the values, we make the following observations:

1. For low learning rates (i.e., $10^{-5}$, $10^{-6}$), unregularized ProLIP shows good performance for different values of $N$, demonstrating the effectiveness of simply fine-tuning the last visual projection matrix. However, the performance drops significantly when the LR increases.

2. A higher value of $\lambda$ works better for fewer training shots $N$, and vice versa. This effect is increasingly visible when the LR increases. Such observation is expected: with less data we need more regularization as overfitting risk is higher, and is the base for formulating $\lambda$ as a decreasing function of $N$ (See Tab. 11, which shows the detailed numerical results of Tab. 2a).
<table><caption>Table 10. **ProLIP sensitivity to hyperparameter choice.** Accuracy of ProLIP to the hyperparameters (learning rate LR and regularization weight λ) for $N \in \{1, 2, 4, 8, 16\}$-shot settings. Each number is an average over 11 datasets, 10 runs for each.</caption><thead><tr><th>Method</th><th>N = 1</th><th>2</th><th>4</th><th>8</th><th>16</th></tr></thead><tbody><tr><td>CLIP</td><td colspan="5" style="text-align: center;">58.89</td></tr><tr><td>ProLIP (grid search)</td><td>64.59±0.98</td><td>67.09±0.87</td><td><strong>70.53±0.69</strong></td><td><strong>73.40±0.45</strong></td><td><strong>76.55±0.41</strong></td></tr><tr><td rowspan="4">ProLIP, LR=10<sup>-6</sup></td><td>λ = 1</td><td>63.05±0.52</td><td>65.00±0.40</td><td>66.91±0.30</td><td>68.13±0.31</td><td>68.95±0.18</td></tr><tr><td>λ = 10<sup>-1</sup></td><td>63.90±0.60</td><td>66.32±0.41</td><td>68.99±0.34</td><td>70.96±0.31</td><td>72.50±0.29</td></tr><tr><td>λ = 10<sup>-2</sup></td><td>63.94±0.65</td><td>66.46±0.42</td><td>69.17±0.37</td><td>71.30±0.33</td><td>72.89±0.30</td></tr><tr><td>λ = 0</td><td>63.94±0.66</td><td>66.46±0.42</td><td>69.19±0.37</td><td>71.34±0.34</td><td>72.94±0.30</td></tr><tr><td rowspan="4">ProLIP, LR=10<sup>-5</sup></td><td>λ = 1</td><td>64.67±0.63</td><td>66.37±0.35</td><td>68.34±0.33</td><td>69.63±0.30</td><td>70.52±0.21</td></tr><tr><td>λ = 10<sup>-1</sup></td><td><strong>64.82±0.71</strong></td><td><strong>67.13±0.45</strong></td><td>70.23±0.42</td><td>72.57±0.33</td><td>74.88±0.29</td></tr><tr><td>λ = 10<sup>-2</sup></td><td>63.64±0.86</td><td>66.47±0.63</td><td>70.07±0.50</td><td>72.79±0.32</td><td>75.77±0.29</td></tr><tr><td>λ = 0</td><td>62.91±0.85</td><td>65.97±0.67</td><td>69.76±0.55</td><td>72.57±0.35</td><td>75.73±0.30</td></tr><tr><td rowspan="4">ProLIP, LR=10<sup>-4</sup></td><td>λ = 1</td><td><u>64.78±0.67</u></td><td>66.63±0.41</td><td>68.88±0.34</td><td>70.33±0.31</td><td>71.40±0.26</td></tr><tr><td>λ = 10<sup>-1</sup></td><td>64.65±0.86</td><td><strong>67.22±0.66</strong></td><td><u>70.44±0.39</u></td><td>72.91±0.31</td><td>75.34±0.36</td></tr><tr><td>λ = 10<sup>-2</sup></td><td>60.54±1.47</td><td>64.29±1.16</td><td>69.10±0.70</td><td>72.51±0.40</td><td>75.88±0.35</td></tr><tr><td>λ = 0</td><td>50.61±1.60</td><td>58.36±1.05</td><td>65.35±0.67</td><td>69.70±0.45</td><td>73.29±0.47</td></tr><tr><td rowspan="4">ProLIP, LR=10<sup>-3</sup></td><td>λ = 1</td><td>64.75±0.68</td><td>66.56±0.46</td><td>68.89±0.35</td><td>70.35±0.30</td><td>71.49±0.25</td></tr><tr><td>λ = 10<sup>-1</sup></td><td>64.32±0.84</td><td>66.97±0.59</td><td>70.39±0.47</td><td><u>72.92±0.35</u></td><td>75.49±0.37</td></tr><tr><td>λ = 10<sup>-2</sup></td><td>58.58±2.04</td><td>63.99±0.97</td><td>69.15±0.69</td><td>72.69±0.42</td><td><u>76.05±0.38</u></td></tr><tr><td>λ = 0</td><td>40.17±1.65</td><td>49.08±1.19</td><td>56.34±1.08</td><td>59.78±0.92</td><td>61.67±0.96</td></tr><tr><td rowspan="4">ProLIP, LR=10<sup>-2</sup></td><td>λ = 1</td><td>64.61±0.72</td><td>66.53±0.45</td><td>68.83±0.37</td><td>70.29±0.31</td><td>71.41±0.25</td></tr><tr><td>λ = 10<sup>-1</sup></td><td>63.47±1.68</td><td>66.72±0.74</td><td>70.18±0.57</td><td>72.66±0.59</td><td>75.06±0.89</td></tr><tr><td>λ = 10<sup>-2</sup></td><td>53.29±2.19</td><td>61.00±1.27</td><td>67.49±0.73</td><td>71.62±0.54</td><td>75.20±0.50</td></tr><tr><td>λ = 0</td><td>20.02±2.21</td><td>23.91±2.17</td><td>28.05±2.43</td><td>32.02±1.93</td><td>35.75±1.25</td></tr></tbody></table>

<table><caption>Table 11. **ProLIP<sub>Ø</sub> with a parametric λ.** Accuracy (%) of ProLIP<sub>Ø</sub> with fixed learning rate (LR) and λ as a function of N. For each λ value, we report performance for different LRs and averaged across LRs. Numbers are averages over 11 datasets and 10 runs. We highlight best and 2nd best for averages across LRs.</caption><thead><tr><th>Method</th><th>N = 1</th><th>2</th><th>4</th><th>8</th><th>16</th></tr></thead><tbody><tr><td>CLIP</td><td colspan="5" style="text-align: center;">58.89</td></tr><tr><td rowspan="4">ProLIP<sub>Ø</sub>, λ = 1/N</td><td>LR=10<sup>-5</sup></td><td>64.67±0.63</td><td>66.80±0.39</td><td>69.73±0.37</td><td>72.44±0.33</td><td>75.34±0.31</td></tr><tr><td>LR=10<sup>-4</sup></td><td>64.78±0.67</td><td>67.02±0.46</td><td>70.19±0.36</td><td>72.83±0.35</td><td>75.65±0.35</td></tr><tr><td>LR=10<sup>-3</sup></td><td>64.75±0.68</td><td>66.93±0.49</td><td>70.12±0.39</td><td>72.84±0.34</td><td>75.80±0.35</td></tr><tr><td>LR=10<sup>-2</sup></td><td>64.60±0.72</td><td>66.93±0.56</td><td>70.03±0.50</td><td>72.63±0.48</td><td>75.08±0.26</td></tr><tr><td></td><td>Average</td><td><strong>64.70±0.08</strong></td><td><u>66.92±0.09</u></td><td><u>70.02±0.20</u></td><td><strong>72.69±0.19</strong></td><td><strong>75.47±0.32</strong></td></tr><tr><td rowspan="4">ProLIP<sub>Ø</sub>, λ = 1/N<sup>2</sup></td><td>LR=10<sup>-5</sup></td><td>64.67±0.63</td><td>67.04±0.42</td><td>70.27±0.44</td><td>72.84±0.32</td><td>75.77±0.30</td></tr><tr><td>LR=10<sup>-4</sup></td><td>64.78±0.67</td><td>67.22±0.50</td><td>70.42±0.43</td><td>72.78±0.38</td><td>75.16±0.37</td></tr><tr><td>LR=10<sup>-3</sup></td><td>64.75±0.68</td><td>67.12±0.52</td><td>70.42±0.52</td><td>72.89±0.40</td><td>75.85±0.41</td></tr><tr><td>LR=10<sup>-2</sup></td><td>64.60±0.72</td><td>67.05±0.55</td><td>70.09±0.81</td><td>72.19±0.42</td><td>74.54±0.71</td></tr><tr><td></td><td>Average</td><td><strong>64.70±0.08</strong></td><td><strong>67.11±0.08</strong></td><td><strong>70.30±0.16</strong></td><td><u>72.68±0.33</u></td><td><u>75.33±0.61</u></td></tr><tr><td rowspan="4">ProLIP<sub>Ø</sub>, λ = 0</td><td>LR=10<sup>-5</sup></td><td>62.91±0.85</td><td>65.97±0.67</td><td>69.76±0.55</td><td>72.57±0.35</td><td>75.73±0.30</td></tr><tr><td>LR=10<sup>-4</sup></td><td>50.61±1.60</td><td>58.36±1.05</td><td>65.35±0.67</td><td>69.70±0.45</td><td>73.29±0.47</td></tr><tr><td>LR=10<sup>-3</sup></td><td>40.17±1.65</td><td>49.08±1.19</td><td>56.34±1.08</td><td>59.78±0.92</td><td>61.67±0.96</td></tr><tr><td>LR=10<sup>-2</sup></td><td>20.02±2.21</td><td>23.91±2.17</td><td>28.05±2.43</td><td>32.02±1.93</td><td>35.75±1.25</td></tr><tr><td></td><td>Average</td><td>43.43±18.16</td><td>49.33±18.30</td><td>54.88±18.74</td><td>58.52±18.50</td><td>61.61±18.30</td></tr></tbody></table>
<figure><img src="image_7.png" /><figcaption>Table 12. Base-to-new generalization with ResNet-50. Per-dataset base, new, and harmonic mean accuracy of ProLIP<sub>ø</sub> with N = 4 (cf. Tab. 5(a)).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Table 13. Base-to-new generalization with ViT-B/16. Per-dataset base, new, and harmonic mean accuracy of ProLIP<sub>ø</sub> with N = 16 (cf. Tab. 5(b)).</figcaption></figure>
<table><caption>Table 14. Comparison to state-of-the-art methods. Average classification accuracy (%) and standard deviation over 10 tasks for 11 benchmarks. Best values are highlighted in bold.</caption><thead><tr><th>Dataset</th><th>Method</th><th>N = 1</th><th>2</th><th>4</th><th>8</th><th>16</th></tr></thead><tbody><tr><td rowspan="10">ImageNet</td><td>CLIP</td><td></td><td></td><td>60.35</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>61.19 ± 0.17</td><td>61.58 ± 0.40</td><td>62.22 ± 0.22</td><td>62.87 ± 0.21</td><td>63.70 ± 0.13</td></tr><tr><td>PLOT [4]</td><td>60.46 ± 0.34</td><td>60.73 ± 0.60</td><td>61.79 ± 0.39</td><td>62.48 ± 0.32</td><td>63.08 ± 0.26</td></tr><tr><td>KgCoOp [41]</td><td>60.90 ± 0.16</td><td>61.44 ± 0.15</td><td>62.00 ± 0.11</td><td>62.20 ± 0.15</td><td>62.43 ± 0.11</td></tr><tr><td>ProGrad [50]</td><td><b>61.58</b> ± 0.27</td><td><b>62.14</b> ± 0.13</td><td><b>62.59</b> ± 0.09</td><td>63.04 ± 0.11</td><td>63.54 ± 0.18</td></tr><tr><td>CLIP-Adapter [9]</td><td>59.82 ± 0.11</td><td>59.94 ± 0.05</td><td>59.97 ± 0.04</td><td>59.98 ± 0.09</td><td>61.31 ± 0.39</td></tr><tr><td>Tip-Adapter-F [46]</td><td>60.59 ± 0.14</td><td>61.42 ± 0.05</td><td>62.12 ± 0.06</td><td>63.41 ± 0.07</td><td><b>65.06</b> ± 0.04</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>60.98 ± 0.15</td><td>61.23 ± 0.11</td><td>61.72 ± 0.25</td><td>62.84 ± 0.10</td><td>64.03 ± 0.12</td></tr><tr><td>Standard LP [30]</td><td>22.21 ± 0.31</td><td>31.96 ± 0.25</td><td>41.48 ± 0.25</td><td>49.49 ± 0.16</td><td>56.04 ± 0.13</td></tr><tr><td>LP++ [17]</td><td>61.18 ± 0.08</td><td>61.56 ± 0.14</td><td>62.55 ± 0.12</td><td><b>63.76</b> ± 0.07</td><td>64.73 ± 0.05</td></tr><tr><td>ProLIP</td><td>61.29 ± 0.13</td><td>61.81 ± 0.18</td><td>62.37 ± 0.18</td><td>63.30 ± 0.11</td><td>64.28 ± 0.12</td></tr><tr><td rowspan="10">SUN397</td><td>CLIP</td><td></td><td></td><td>58.85</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>61.79 ± 0.45</td><td>63.32 ± 0.47</td><td>65.79 ± 0.44</td><td>67.89 ± 0.38</td><td>70.15 ± 0.32</td></tr><tr><td>PLOT [4]</td><td>62.53 ± 0.30</td><td>63.87 ± 0.26</td><td>65.85 ± 0.48</td><td>67.83 ± 0.36</td><td>69.90 ± 0.31</td></tr><tr><td>KgCoOp [41]</td><td><b>62.91</b> ± 0.59</td><td>64.38 ± 0.30</td><td>66.06 ± 0.37</td><td>66.66 ± 1.10</td><td>67.68 ± 0.78</td></tr><tr><td>ProGrad [50]</td><td>62.79 ± 0.50</td><td>64.12 ± 0.55</td><td>66.32 ± 0.59</td><td>68.33 ± 0.28</td><td>70.18 ± 0.27</td></tr><tr><td>CLIP-Adapter [9]</td><td>60.78 ± 0.16</td><td>61.79 ± 0.23</td><td>63.84 ± 0.35</td><td>66.26 ± 0.14</td><td>67.66 ± 1.05</td></tr><tr><td>Tip-Adapter-F [46]</td><td>61.02 ± 0.36</td><td>62.15 ± 0.28</td><td>63.86 ± 0.19</td><td>67.25 ± 0.16</td><td>70.94 ± 0.13</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>62.58 ± 0.22</td><td>63.79 ± 0.13</td><td>65.49 ± 0.35</td><td>67.43 ± 0.11</td><td>69.25 ± 0.16</td></tr><tr><td>Standard LP [30]</td><td>32.56 ± 0.40</td><td>43.77 ± 0.41</td><td>54.49 ± 0.39</td><td>61.83 ± 0.30</td><td>67.03 ± 0.16</td></tr><tr><td>LP++ [17]</td><td>62.47 ± 0.27</td><td>64.65 ± 0.25</td><td>67.28 ± 0.27</td><td><b>69.34</b> ± 0.14</td><td>71.23 ± 0.07</td></tr><tr><td>ProLIP</td><td>62.71 ± 0.46</td><td><b>65.58</b> ± 0.13</td><td><b>67.68</b> ± 0.46</td><td>69.17 ± 0.07</td><td><b>71.29</b> ± 0.23</td></tr><tr><td rowspan="10">DTD</td><td>CLIP</td><td></td><td></td><td>42.69</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>42.31 ± 1.86</td><td>47.13 ± 1.93</td><td>54.06 ± 1.49</td><td>59.21 ± 0.91</td><td>63.67 ± 0.83</td></tr><tr><td>PLOT [4]</td><td>45.82 ± 1.72</td><td>51.32 ± 1.61</td><td>55.67 ± 1.14</td><td>61.38 ± 1.04</td><td>65.29 ± 1.05</td></tr><tr><td>KgCoOp [41]</td><td>45.46 ± 2.83</td><td>50.01 ± 2.71</td><td>53.37 ± 0.71</td><td>58.38 ± 1.34</td><td>62.71 ± 0.92</td></tr><tr><td>ProGrad [50]</td><td>44.19 ± 2.38</td><td>50.41 ± 1.74</td><td>54.82 ± 1.28</td><td>60.31 ± 0.99</td><td>63.89 ± 0.88</td></tr><tr><td>CLIP-Adapter [9]</td><td>43.49 ± 0.68</td><td>44.49 ± 1.07</td><td>48.95 ± 0.85</td><td>57.52 ± 0.67</td><td>62.97 ± 0.60</td></tr><tr><td>Tip-Adapter-F [46]</td><td>46.92 ± 1.01</td><td>48.50 ± 1.08</td><td>57.16 ± 0.53</td><td>62.38 ± 0.47</td><td>65.23 ± 0.82</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>47.68 ± 1.43</td><td>52.24 ± 0.74</td><td>56.09 ± 0.99</td><td>61.05 ± 0.71</td><td>65.04 ± 0.21</td></tr><tr><td>Standard LP [30]</td><td>29.63 ± 1.53</td><td>41.19 ± 1.45</td><td>51.72 ± 0.57</td><td>58.78 ± 0.52</td><td>64.56 ± 0.69</td></tr><tr><td>LP++ [17]</td><td>46.97 ± 1.37</td><td>52.44 ± 0.99</td><td>57.75 ± 0.82</td><td>62.42 ± 0.53</td><td>66.40 ± 0.50</td></tr><tr><td>ProLIP</td><td><b>49.99</b> ± 2.28</td><td><b>54.93</b> ± 1.29</td><td><b>59.25</b> ± 1.03</td><td><b>64.12</b> ± 0.64</td><td><b>67.69</b> ± 0.87</td></tr><tr><td rowspan="10">Coltech101</td><td>CLIP</td><td></td><td></td><td>85.84</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>87.06 ± 1.24</td><td>89.14 ± 0.87</td><td>90.00 ± 0.63</td><td>91.00 ± 0.66</td><td>91.77 ± 0.29</td></tr><tr><td>PLOT [4]</td><td><b>89.41</b> ± 0.41</td><td><b>90.22</b> ± 0.25</td><td>90.69 ± 0.37</td><td>91.55 ± 0.38</td><td>92.17 ± 0.30</td></tr><tr><td>KgCoOp [41]</td><td>88.24 ± 0.49</td><td>88.85 ± 0.43</td><td>89.89 ± 0.31</td><td>90.32 ± 0.43</td><td>90.93 ± 0.26</td></tr><tr><td>ProGrad [50]</td><td>88.34 ± 1.64</td><td>89.01 ± 0.61</td><td>90.13 ± 0.45</td><td>90.76 ± 0.32</td><td>91.67 ± 0.39</td></tr><tr><td>CLIP-Adapter [9]</td><td>87.69 ± 0.41</td><td>89.37 ± 0.29</td><td>90.21 ± 0.32</td><td>91.33 ± 0.15</td><td>92.10 ± 0.20</td></tr><tr><td>Tip-Adapter-F [46]</td><td>87.35 ± 0.64</td><td>88.17 ± 0.29</td><td>89.49 ± 0.25</td><td>90.54 ± 0.34</td><td>92.10 ± 0.25</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>88.68 ± 0.44</td><td>89.36 ± 0.59</td><td>90.40 ± 0.26</td><td>91.62 ± 0.23</td><td>92.63 ± 0.21</td></tr><tr><td>Standard LP [30]</td><td>68.88 ± 1.68</td><td>78.41 ± 0.54</td><td>84.91 ± 0.45</td><td>88.70 ± 0.40</td><td>91.14 ± 0.19</td></tr><tr><td>LP++ [17]</td><td>88.56 ± 0.43</td><td>89.53 ± 0.35</td><td>90.87 ± 0.19</td><td>91.84 ± 0.24</td><td>92.73 ± 0.17</td></tr><tr><td>ProLIP</td><td>89.16 ± 0.48</td><td>89.48 ± 1.15</td><td><b>91.44</b> ± 0.42</td><td><b>92.43</b> ± 0.32</td><td><b>93.39</b> ± 0.38</td></tr><tr><td rowspan="10">UCF101</td><td>CLIP</td><td></td><td></td><td>61.80</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>62.80 ± 1.26</td><td>65.62 ± 1.09</td><td>68.69 ± 0.76</td><td>72.57 ± 0.80</td><td>76.39 ± 0.54</td></tr><tr><td>PLOT [4]</td><td>63.22 ± 1.05</td><td>66.49 ± 0.92</td><td>70.12 ± 0.62</td><td>74.63 ± 0.79</td><td>77.39 ± 0.53</td></tr><tr><td>KgCoOp [41]</td><td>64.37 ± 1.66</td><td>64.91 ± 1.01</td><td>68.41 ± 0.38</td><td>69.86 ± 0.33</td><td>71.73 ± 0.78</td></tr><tr><td>ProGrad [50]</td><td>65.13 ± 0.87</td><td>66.57 ± 0.62</td><td>69.80 ± 0.62</td><td>73.01 ± 0.52</td><td>75.76 ± 0.47</td></tr><tr><td>CLIP-Adapter [9]</td><td>64.25 ± 0.54</td><td>66.68 ± 0.31</td><td>69.77 ± 0.40</td><td>73.90 ± 0.50</td><td>77.26 ± 0.39</td></tr><tr><td>Tip-Adapter-F [46]</td><td>64.28 ± 0.54</td><td>65.48 ± 0.43</td><td>67.61 ± 0.28</td><td>72.05 ± 0.53</td><td>77.30 ± 0.21</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>65.50 ± 0.59</td><td>68.55 ± 0.45</td><td>70.55 ± 0.58</td><td>74.25 ± 0.48</td><td>76.83 ± 0.24</td></tr><tr><td>Standard LP [30]</td><td>40.80 ± 1.05</td><td>51.71 ± 0.79</td><td>61.64 ± 0.50</td><td>68.47 ± 0.44</td><td>73.38 ± 0.43</td></tr><tr><td>LP++ [17]</td><td>65.41 ± 0.37</td><td>69.20 ± 0.52</td><td>71.68 ± 0.41</td><td>74.86 ± 0.36</td><td>77.46 ± 0.39</td></tr><tr><td>ProLIP</td><td><b>67.05</b> ± 0.12</td><td><b>70.02</b> ± 0.46</td><td><b>71.90</b> ± 1.01</td><td><b>76.36</b> ± 0.67</td><td><b>80.29</b> ± 0.22</td></tr><tr><td rowspan="10">Flowers102</td><td>CLIP</td><td></td><td></td><td>65.98</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>69.00 ± 2.44</td><td>78.47 ± 1.88</td><td>85.34 ± 1.69</td><td>91.68 ± 0.82</td><td>94.47 ± 0.36</td></tr><tr><td>PLOT [4]</td><td>71.09 ± 1.44</td><td>81.22 ± 0.92</td><td>87.61 ± 0.79</td><td>92.60 ± 0.55</td><td><b>95.18</b> ± 0.40</td></tr><tr><td>KgCoOp [41]</td><td>68.73 ± 1.97</td><td>69.63 ± 1.25</td><td>76.51 ± 0.51</td><td>80.71 ± 0.63</td><td>84.48 ± 0.70</td></tr><tr><td>ProGrad [50]</td><td>72.16 ± 1.74</td><td>79.55 ± 0.88</td><td>84.56 ± 1.41</td><td>91.73 ± 0.35</td><td>94.10 ± 0.41</td></tr><tr><td>CLIP-Adapter [9]</td><td>66.86 ± 0.73</td><td>69.71 ± 0.46</td><td>77.42 ± 0.60</td><td>87.20 ± 0.52</td><td>91.16 ± 0.23</td></tr><tr><td>Tip-Adapter-F [46]</td><td>67.73 ± 0.57</td><td>68.18 ± 0.84</td><td>71.17 ± 0.67</td><td>84.11 ± 0.49</td><td>93.02 ± 0.28</td></tr><tr><td>Tip-Adapter-F* [46]</td><td><b>78.46</b> ± 1.01</td><td><b>85.14</b> ± 0.81</td><td>88.53 ± 0.54</td><td>92.33 ± 0.32</td><td>94.26 ± 0.38</td></tr><tr><td>Standard LP [30]</td><td>56.98 ± 1.56</td><td>73.40 ± 0.87</td><td>84.38 ± 0.53</td><td>91.81 ± 0.34</td><td>95.05 ± 0.29</td></tr><tr><td>LP++ [17]</td><td>78.21 ± 1.01</td><td>84.69 ± 0.70</td><td><b>89.56</b> ± 0.45</td><td>92.61 ± 0.32</td><td>94.26 ± 0.24</td></tr><tr><td>ProLIP</td><td>76.13 ± 1.35</td><td>82.31 ± 1.36</td><td>88.37 ± 0.78</td><td><b>92.79</b> ± 0.61</td><td>95.15 ± 0.35</td></tr></tbody></table>

<table><caption>Table 15. Comparison to state-of-the-art methods (Continued). Average classification accuracy (%) and standard deviation over 10 tasks for 11 benchmarks. Best values are highlighted in bold.</caption><thead><tr><th>Dataset</th><th>Method</th><th>N = 1</th><th>2</th><th>4</th><th>8</th><th>16</th></tr></thead><tbody><tr><td rowspan="10">StanfordCars</td><td>CLIP</td><td></td><td></td><td>55.78</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>57.00 ± 0.93</td><td>58.96 ± 0.78</td><td>62.81 ± 0.71</td><td>68.40 ± 0.61</td><td>72.87 ± 0.50</td></tr><tr><td>PLOT [4]</td><td>57.47 ± 0.58</td><td>59.89 ± 0.60</td><td>63.49 ± 0.80</td><td>68.75 ± 0.46</td><td>73.86 ± 0.39</td></tr><tr><td>KgCoOp [41]</td><td>57.19 ± 0.65</td><td>58.94 ± 0.33</td><td>59.85 ± 0.51</td><td>61.42 ± 0.40</td><td>62.99 ± 1.33</td></tr><tr><td>ProGrad [50]</td><td><b>58.63</b> ± 0.39</td><td>61.23 ± 0.65</td><td>65.02 ± 0.78</td><td>69.43 ± 0.40</td><td>72.76 ± 0.45</td></tr><tr><td>CLIP-Adapter [9]</td><td>56.67 ± 0.22</td><td>57.94 ± 0.27</td><td>61.13 ± 0.30</td><td>65.43 ± 0.10</td><td>70.24 ± 0.79</td></tr><tr><td>Tip-Adapter-F [46]</td><td>57.24 ± 0.23</td><td>58.12 ± 0.50</td><td>59.34 ± 0.20</td><td>64.25 ± 0.19</td><td>71.38 ± 0.23</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>57.85 ± 0.33</td><td>60.55 ± 0.34</td><td>64.22 ± 0.52</td><td>68.75 ± 0.31</td><td>74.19 ± 0.30</td></tr><tr><td>Standard LP [30]</td><td>22.94 ± 0.61</td><td>35.48 ± 0.51</td><td>47.49 ± 0.67</td><td>59.34 ± 0.30</td><td>69.11 ± 0.18</td></tr><tr><td>LP++ [17]</td><td>57.20 ± 0.65</td><td>59.95 ± 0.36</td><td>63.44 ± 0.34</td><td>67.81 ± 0.24</td><td>72.33 ± 0.18</td></tr><tr><td>ProLIP</td><td>58.29 ± 0.49</td><td><b>61.94</b> ± 0.37</td><td><b>66.09</b> ± 0.27</td><td><b>69.82</b> ± 0.43</td><td><b>76.03</b> ± 0.09</td></tr><tr><td rowspan="10">FGVCAircraft</td><td>CLIP</td><td></td><td></td><td>17.07</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>12.50 ± 6.16</td><td>17.59 ± 3.70</td><td>21.27 ± 2.54</td><td>26.85 ± 0.63</td><td>31.20 ± 0.40</td></tr><tr><td>PLOT [4]</td><td>17.75 ± 1.36</td><td>19.55 ± 0.99</td><td>22.26 ± 0.89</td><td>26.70 ± 0.70</td><td>32.09 ± 0.68</td></tr><tr><td>KgCoOp [41]</td><td>18.61 ± 0.76</td><td>18.93 ± 1.01</td><td>21.16 ± 0.82</td><td>22.80 ± 0.44</td><td>24.10 ± 0.59</td></tr><tr><td>ProGrad [50]</td><td>18.41 ± 0.98</td><td>20.51 ± 1.11</td><td>23.65 ± 0.50</td><td>26.98 ± 0.50</td><td>30.47 ± 0.76</td></tr><tr><td>CLIP-Adapter [9]</td><td>18.56 ± 0.20</td><td>19.18 ± 0.28</td><td>21.00 ± 0.21</td><td>23.76 ± 0.33</td><td>33.37 ± 0.23</td></tr><tr><td>Tip-Adapter-F [46]</td><td>18.23 ± 0.19</td><td>19.12 ± 0.20</td><td>20.55 ± 0.20</td><td>23.60 ± 0.29</td><td>30.37 ± 0.25</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>19.08 ± 0.15</td><td>20.79 ± 0.59</td><td>23.99 ± 0.57</td><td>30.58 ± 0.29</td><td>36.16 ± 0.34</td></tr><tr><td>Standard LP [30]</td><td>12.66 ± 0.59</td><td>16.92 ± 0.56</td><td>21.11 ± 0.83</td><td>26.53 ± 0.38</td><td>32.42 ± 0.54</td></tr><tr><td>LP++ [17]</td><td><b>19.69</b> ± 0.39</td><td><b>21.58</b> ± 0.46</td><td>24.22 ± 0.60</td><td>27.73 ± 0.48</td><td>31.73 ± 0.44</td></tr><tr><td>ProLIP</td><td>17.86 ± 1.18</td><td>20.88 ± 0.69</td><td><b>27.35</b> ± 0.19</td><td><b>32.59</b> ± 0.37</td><td><b>40.09</b> ± 0.12</td></tr><tr><td rowspan="10">EuroSAT</td><td>CLIP</td><td></td><td></td><td>36.22</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>40.36 ± 7.19</td><td>56.15 ± 5.82</td><td>66.13 ± 3.62</td><td>77.02 ± 1.78</td><td>82.59 ± 1.00</td></tr><tr><td>PLOT [4]</td><td>44.22 ± 9.14</td><td>64.19 ± 6.24</td><td>69.37 ± 3.26</td><td>78.84 ± 1.33</td><td>81.76 ± 1.43</td></tr><tr><td>KgCoOp [41]</td><td>43.86 ± 9.17</td><td>52.92 ± 5.92</td><td>59.51 ± 3.46</td><td>63.23 ± 3.03</td><td>64.04 ± 1.40</td></tr><tr><td>ProGrad [50]</td><td>49.37 ± 5.03</td><td>65.22 ± 4.01</td><td>69.57 ± 2.88</td><td>78.44 ± 1.69</td><td>82.17 ± 0.98</td></tr><tr><td>CLIP-Adapter [9]</td><td>43.00 ± 2.27</td><td>48.60 ± 2.76</td><td>59.15 ± 2.26</td><td>69.92 ± 1.49</td><td>75.38 ± 0.78</td></tr><tr><td>Tip-Adapter-F [46]</td><td>47.63 ± 2.64</td><td>57.62 ± 1.86</td><td>69.30 ± 2.41</td><td>75.22 ± 1.32</td><td>78.59 ± 1.48</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>49.27 ± 2.88</td><td>65.66 ± 1.39</td><td>70.72 ± 2.73</td><td>74.66 ± 3.15</td><td>78.73 ± 0.81</td></tr><tr><td>Standard LP [30]</td><td>48.29 ± 2.95</td><td>56.81 ± 2.93</td><td>64.99 ± 3.47</td><td>75.46 ± 0.98</td><td>80.29 ± 0.90</td></tr><tr><td>LP++ [17]</td><td>57.23 ± 1.63</td><td>61.65 ± 1.66</td><td>68.67 ± 2.21</td><td>75.86 ± 0.99</td><td>80.53 ± 1.05</td></tr><tr><td>ProLIP</td><td><b>65.78</b> ± 0.45</td><td><b>67.26</b> ± 0.57</td><td><b>77.03</b> ± 1.19</td><td><b>80.08</b> ± 0.48</td><td><b>85.82</b> ± 0.63</td></tr><tr><td rowspan="10">OxfordPets</td><td>CLIP</td><td></td><td></td><td>85.75</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>86.27 ± 1.17</td><td>86.33 ± 1.13</td><td>85.34 ± 1.69</td><td>87.85 ± 1.21</td><td>88.68 ± 0.71</td></tr><tr><td>PLOT [4]</td><td>87.15 ± 0.72</td><td>87.23 ± 1.21</td><td>88.03 ± 0.49</td><td>88.38 ± 0.64</td><td>88.23 ± 0.54</td></tr><tr><td>KgCoOp [41]</td><td>87.51 ± 0.68</td><td>87.51 ± 0.75</td><td>88.04 ± 0.46</td><td>88.59 ± 0.34</td><td>89.28 ± 0.21</td></tr><tr><td>ProGrad [50]</td><td><b>88.34</b> ± 0.65</td><td><b>87.88</b> ± 0.69</td><td><b>88.59</b> ± 0.58</td><td><b>88.87</b> ± 0.42</td><td><b>89.39</b> ± 0.47</td></tr><tr><td>CLIP-Adapter [9]</td><td>85.46 ± 0.48</td><td>86.37 ± 0.25</td><td>87.21 ± 0.51</td><td>87.95 ± 0.26</td><td>88.33 ± 0.33</td></tr><tr><td>Tip-Adapter-F [46]</td><td>85.70 ± 0.16</td><td>86.05 ± 0.46</td><td>86.40 ± 0.29</td><td>87.66 ± 0.28</td><td>89.08 ± 0.27</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>86.05 ± 0.36</td><td>86.49 ± 0.61</td><td>87.19 ± 0.36</td><td>87.89 ± 0.34</td><td>88.26 ± 0.37</td></tr><tr><td>Standard LP [30]</td><td>30.62 ± 1.61</td><td>42.64 ± 2.03</td><td>55.60 ± 0.98</td><td>67.32 ± 0.98</td><td>76.23 ± 0.38</td></tr><tr><td>LP++ [17]</td><td>84.24 ± 1.36</td><td>85.74 ± 0.56</td><td>86.94 ± 0.48</td><td>87.71 ± 0.65</td><td>88.38 ± 0.61</td></tr><tr><td>ProLIP</td><td>85.62 ± 1.10</td><td>86.02 ± 1.50</td><td>87.24 ± 0.75</td><td>88.20 ± 0.56</td><td>89.00 ± 0.51</td></tr><tr><td rowspan="10">Food101</td><td>CLIP</td><td></td><td></td><td>77.35</td><td></td><td></td></tr><tr><td>CoOp [49]</td><td>75.58 ± 1.29</td><td>77.49 ± 0.41</td><td>77.93 ± 0.58</td><td>78.92 ± 0.19</td><td>79.21 ± 0.36</td></tr><tr><td>PLOT [4]</td><td>77.46 ± 0.55</td><td>77.72 ± 0.26</td><td>78.23 ± 0.25</td><td>78.40 ± 0.35</td><td>78.86 ± 0.19</td></tr><tr><td>KgCoOp [41]</td><td>77.20 ± 0.77</td><td><b>78.04</b> ± 0.18</td><td>77.97 ± 0.28</td><td>78.39 ± 0.40</td><td>78.73 ± 0.23</td></tr><tr><td>ProGrad [50]</td><td><b>78.36</b> ± 0.41</td><td>78.01 ± 0.70</td><td><b>78.38</b> ± 0.87</td><td><b>79.11</b> ± 0.18</td><td><b>79.51</b> ± 0.23</td></tr><tr><td>CLIP-Adapter [9]</td><td>76.93 ± 0.19</td><td>77.22 ± 0.15</td><td>77.64 ± 0.17</td><td>77.97 ± 0.22</td><td>78.45 ± 0.14</td></tr><tr><td>Tip-Adapter-F [46]</td><td>77.53 ± 0.14</td><td>77.53 ± 0.22</td><td>77.82 ± 0.27</td><td>78.26 ± 0.22</td><td>78.99 ± 0.15</td></tr><tr><td>Tip-Adapter-F* [46]</td><td>77.58 ± 0.10</td><td>77.36 ± 0.39</td><td>77.78 ± 0.15</td><td>78.17 ± 0.11</td><td>78.72 ± 0.06</td></tr><tr><td>Standard LP [30]</td><td>31.59 ± 1.20</td><td>44.60 ± 1.03</td><td>56.13 ± 0.63</td><td>64.15 ± 0.55</td><td>70.97 ± 0.19</td></tr><tr><td>LP++ [17]</td><td>76.61 ± 0.77</td><td>77.22 ± 0.55</td><td>77.79 ± 0.34</td><td>78.53 ± 0.14</td><td>78.88 ± 0.19</td></tr><tr><td>ProLIP</td><td>76.60 ± 0.11</td><td>77.71 ± 0.08</td><td>77.15 ± 0.30</td><td>78.57 ± 0.07</td><td>79.02 ± 0.13</td></tr></tbody></table>
References

[1] Christopher M Bishop and Nasser M Nasrabadi. *Pattern recognition and machine learning*. Springer, 2006. 3

[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In *ECCV*, 2014. 5

[3] Tom B Brown. Language models are few-shot learners. *NeurIPS*, 2020. 1

[4] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. PLOT: Prompt learning with optimal transport for vision-language models. In *ICLR*, 2023. 2, 3, 5, 6, 13

[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In *CVPR*, 2014. 5

[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In *CVPR*, 2009. 5

[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In *ICLR*, 2021. 3

[8] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In *CVPR Workshops*, 2004. 5

[9] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. *IJCV*, 2024. 2, 3, 5, 13

[10] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. *The elements of statistical learning: data mining, inference, and prediction*. Taylor & Francis, 2009. 4

[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *CVPR*, 2016. 3

[12] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 2019. 1, 5

[13] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada-vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In *ICCV*, 2021. 5

[14] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In *CVPR*, 2021. 5

[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In *ICML*, 2019. 2

[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In *ICLR*, 2022. 2

[17] Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. LP++: A surprisingly strong linear probe for few-shot clip. In *CVPR*, 2024. 2, 3, 5, 6, 7, 10, 13

[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In *ECCV*, 2022. 2

[19] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In *CVPR*, 2023. 3, 5, 7

[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In *ICCV Workshops*, 2013. 5

[21] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In *ICLR*, 2022. 1, 2

[22] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In *CVPR*, 2022. 9

[23] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. *ACL*, 2021. 2

[24] Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In *CVPR*, 2023. 2, 3, 6, 8

[25] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. *arXiv*, 2013. 1, 5

[26] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In *ICLR*, 2023. 1

[27] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In *ICVGIP*, 2008. 5

[28] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In *CVPR*, 2012. 5

[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In *NeurIPS*, 2019. 4

[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *ICML*, 2021. 1, 2, 3, 5, 6, 13

[31] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imogenet? In *ICML*, 2019. 5

[32] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. In *EMNLP*, 2021. 2
[33] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022. 5, 8, 9
[34] Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In CVPR, 2024. 2, 5, 6
[35] K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv, 2012. 5
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3
[37] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019. 5
[38] Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free CLIP-based adaptation. In ICLR, 2024. 3
[39] Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In CVPR, 2024. 2
[40] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 5
[41] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guided context optimization. In CVPR, 2023. 5, 13
[42] Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision-language models. In CVPR, 2023. 6
[43] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In ACL, 2022. 2
[44] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 9
[45] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In ECCV, 2020. 2
[46] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In ECCV, 2022. 2, 3, 5, 6, 7, 10, 13
[47] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. ACL, 2021. 2
[48] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022. 2, 3, 7, 8
[49] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022. 2, 5, 8, 13
[50] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV, 2023. 2, 3, 5, 6, 7, 13