Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability

## Abstract

Abstract With the advancement of large pre-trained vision-language models, effectively transferring the knowledge embedded within these foundational models to downstream tasks has become a pivotal topic, particularly in data-scarce environments. Recently, parameter-efficient fine-tuning approaches, especially prompt tuning, have garnered considerable attention. To better understand the nature of prompt tuning, we propose the concept of “Information Density” (ID) to indicate whether a matrix strongly belongs to certain feature spaces rather than being evenly distributed across various feature spaces. We suppose a higher ID with strong bias across some feature spaces naturally leads to excellent robustness and stability. Our research, inspired by the observation that generalizability is closely linked to the information density of the prompt matrix, introduces the Dense Information Prompt (DIP). DIP aims to enhance information density to improve generalization. Furthermore, DIP significantly reduces the number of tunable parameters and the requisite storage space, making it particularly advantageous in resource-constrained settings. Comprehensive experiments substantiate the superiority of DIP. Notably, DIP surpasses the latest state-of-the-art methods by a substantial margin with an exceptionally small parameter count. Across a range of tasks spanning 11 datasets, DIP improves the average downstream accuracy of classic prompt tuning by up to 5.76% using merely 0.5K parameters.

## 1 Introduction

In recent years, vision-languages models [, ] have achieved tremendous success. Representative models like CLIP [] are first pre-trained on a huge number of text-image pairs on the web to align textual and visual features, and then can be tuned and used for various downstream tasks.

However, traditional fine-tuning is not a good choice to adapt vision-language models. Simply fine-tuning all the parameters can easily cause the model to overfit because the huge number of parameters bring redundant non-essential information. The huge training and storage cost is also an intractable problem. In the context of our study, we introduce the concept of “information density”. Much like the rank of a matrix in linear algebra, which represents the maximum number of linearly independent rows or columns in the matrix, “information density” represents the maximum amount of essential and non-redundant information that the model can extract from the downstream task. Just as a matrix with a higher rank possesses more unique information, a model with high information density can acquire more essential and general information from the downstream task by fewer parameters, even with a smaller dataset. Our goal is increasing the information density and thus using the fewest but most essential parameters to finish generalization, without causing catastrophic forgetting or overfitting to the small dataset.

As the concept of information density can be functionally analogous to the rank of a matrix, we decided to use properties related to rank to quantify information density. Specifically, we take a full-rank matrix and decompose it using Singular Value Decomposition (SVD) to obtain a matrix of singular values, using the properties of these singular values to define and quantize information density. We found that this definition of information density is highly correlated with the model’s generalization performance (Spearman correlation coefficient $>$ 0.9). Therefore, we propose DIP, aiming to enhance the model’s generalization ability by increasing information density. Additionally, due to the increased information carried by each parameter unit, our approach can significantly reduce the required number of parameters.

In Section 3, we will show our finding about the strong correlation between generalization capability and information density of the prompt matrix in Fig. 1. Inspired by such observation, we propose Dense Information Prompt (DIP) for effective and efficient adaptation, well adapting models under an extremely small number of parameters where nobody has explored before. There are several good advantages of DIP:

- •
Efficiency and Effectiveness To our knowledge, we are the first one to reach comparable or even better performance with state-of-the-art methods using such an extremely small number of parameters, *i.e*. 516.
- •
Simplicity Replacing the classic prompt by DIP just needs to modify several lines of code. No special loss nor extra module is introduced.
- •
Robustness DIP is relatively capable of anti-disturbance. As in Tab. 3, DIP could maximally reserve its knowledge from domain shift.
- •
Plug and Play For any classic model, DIP only replaces the prompts, enabling us to plug DIP into most of the existing methods fruitfully.

In summary, we conclude our contributions as follows:

- •
We propose a new concept “information density”, give its definition and well-quantize the concept. We further find the strong correlation between generalizability and information density in Fig. 1, and thus propose to use DIP to increase information density for better generalization capability.
- •
We propose a novel initialization method and integrate a lightweight regularization module to further improve the performance of Dense Information Prompt tuning without introducing any extra parameters and inference cost.
- •
We are the first to explore how to effectively adapt vision-language models using an extremely small number of parameters, *i.e*. 0.5K.
- •
We conduct extensive experiments and show the fantastic effectiveness and efficiency of DIP. In base-to-new generalization, domain generalization, cross-dataset transfer and few-shot learning settings, DIP consistently reaches very competitive results, surpassing many state-of-the-art tuning methods though we use fewer parameters.

## 2 Related Works

### 2.1 Vision-Language Models

Recently, large-scale vision-language models have shown very competitive performance in various tasks. Classic works [, , , , ] learn the multi-modal representation by a self-supervised manner on a large amount of image-text pairs. The representative work CLIP [] is a milestone, which aligns the vision representation and language representation by contrastive learning and shows excellent performance.
A well-trained vision-language model is a great treasure, which could largely facilitate the development of many fields. There have been successful applications of such strong models on few-shot recognition [, ],
detection [, , , ] and segmentation [, , , ]. For video data, there are also works on video classification [] and video understanding [].

### 2.2 Prompt Tuning

Prompt tuning is one of the most popular methods to tune models in downstream tasks with excellent efficiency. Originating from natural language processing, prompts are first introduced as a fixed template [], *e.g*. a photo of a _, which is hand-crafted and fixed. Later, a series of methods [, , , , , ] are proposed to make such prompts tunable and be optimized during adaptation. Prompt tuning could adaptively narrow the gap between pre-trained representations and downstream tasks, significantly facilitating the fine-tuning process. Representative prompt tuning methods would add tunable virtual tokens, *i.e*. prompts, along with the semantic tokens as inputs of the model. All of the tokens are processed together to get text embeddings first and then sent to the feature encoder. Witnessing the success of prompting language models, researchers design prompts [, ] for visual models in a similar way. In vision-language field, there are several explorations as well. Bahng *et al*. [] adopts prompt tuning merely on the image encoder. CoOp [] uses tunable text prompts to replace the fixed template in CLIP []. CoCoOp [] utilizes image feature to instruct the optimization of the tunable text prompts in CoOp.  [, ] simultaneously optimize image and text prompts and establish extra connections between different modals.  [, , , , ] integrate strong regularization modules or losses into prompt tuning to diminish the overfitting and catastrophic forgetting problem. For better downstream accuracy, researchers design more and more complicated methods, accompanied by inefficiency. To solve the problem, we propose Dense Information Prompt (DIP) to take the place of classic prompts, which can largely decrease the number of tunable parameters and further enhance the model’s generalization ability. Notice that though becoming more complex, existing methods are still refined on a common fundamental basis, *i.e*. prompt tuning. Such a common basis guarantees that DIP could be easily and smoothly integrated into most of the off-the-shelf methods besides individually applied. Besides prompt-based methods, there are also many other works to acquire storage efficiency [, , , , , ], inference efficiency [, , , , , , , , ] and data efficiency [, ] during downstream tuning.

## 3 Relationship between Information Density and Generalizability

In this section, we will start from reviewing a classic prompt tuning pipeline CoOp [] on CLIP in Section 3.1, and propose a new concept “Information Density” and analyze its relationship with generalizability in Section 3.2.

### 3.1 A Review of Prompt Tuning for CLIP

CLIP consists of a text encoder $\mathcal{L}$ and an image encoder $\mathcal{V}$. Typically, $\mathcal{L}$ is a language transformer, while $\mathcal{V}$ can be a convolutional neural network or a vision transformer. In this paper, we follows  [, ] to use a ViT-B/16 [] as the image encoder $\mathcal{V}$ unless specifically mentioned. We start by making a review of how to prompt a CLIP for prediction in the following paragraphs.
Text Encoder Suppose there are $M$ layers in the text encoder. For $k$-th layer $\mathcal{L}_{k}$, the inputs are a series of prompt tokens $P^{l}_{k-1}$ and a [CLS] token $c^{l}_{k-1}$, and the outputs are $P^{l}_{k}$ and $c^{l}_{k}$. The inputs of the first layer $P^{l}_{0}$ and $c^{l}_{0}$ are exactly the word embeddings of the prompts along with the label, *e.g*. “A photo of a [CLS]” or just some randomly initialized vectors. Formally, we have $P^{l}_{k}\in\mathbb{R}^{n^{l}\times d^{l}}$ and $c^{l}_{k}\in\mathbb{R}^{d^{l}}$, where $n^{l}$ denotes the text prompts’ length and $d^{l}$ denotes the dimension of word embedding. $\forall 1\leq k\leq M$, we have $[P^{l}_{k},c^{l}_{k}]=\mathcal{L}_{k}([P^{l}_{k-1},c^{l}_{k-1}])$.

The output feature of the text encoder $f^{l}\in\mathbb{R}^{d^{v}}$, where $d^{v}$ is the dimension of the visual feature space, is generated by projecting the [CLS] token of the last layer to the visual space by a linear transformation, *i.e*. $f^{l}={\rm Proj}(c^{l}_{M})$.
Image Encoder Suppose there are $N$ layers in the image encoder. For $k$-th layer $\mathcal{V}_{k}$, the inputs are image patch tokens $I_{k-1}$, a classification token $c^{v}_{k-1}$ and prompt tokens $P^{v}_{k-1}$, and the outputs are $I_{k}$, $c^{v}_{k}$ and $P^{v}_{k}$. The inputs of the first layer $I_{0}$ and $c^{v}_{0}$ are exactly the patch embeddings of the image and pre-trained class token. $P^{v}_{0}$ is randomly initialized in general. Formally, we have $I_{k}\in\mathbb{R}^{p\times d^{v}}$, $c^{v}_{k}\in\mathbb{R}^{d^{v}}$ and $P^{v}_{k}\in\mathbb{R}^{n^{v}\times d^{v}}$, where $p$ denotes the number of image patches and $d^{v}$ denotes the dimension of visual embedding. $\forall 1\leq k\leq N$, $[P^{v}_{k},c^{v}_{k},I_{k}]=\mathcal{V}_{k}([P^{v}_{k-1},c^{v}_{k-1},I_{k-1}])$. The output feature of the image encoder is $f^{v}=c^{v}_{N}$.

Prediction CLIP can be used for image classification. Suppose there are $C$ classes, and $\{f^{l}_{c}\}_{c=1}^{C}$ are the corresponding text features. Label $y$’s probability is $p(y|f^{v})=\frac{\mbox{exp}(\mbox{sim}(f^{v},f^{l}_{y})/\tau)}{\sum_{c=1}^{C}
\mbox{exp}(\mbox{sim}(f^{v},f^{l}_{c})/\tau)}$ where sim$(\cdot,\cdot)$ denotes cosine similarity function and $\tau$ is temperature. The final prediction is $\hat{z}=\mathop{\arg\max}\limits_{1\leq y\leq C}(p(y|f^{v}))$.

It is worth noting that some researchers adopt a deeper manner [, ] to organize the prompts. They directly add and tune the prompt in each layer in the feature encoder, instead of inheriting the output prompt calculated by the last encoder, *i.e*. a forward pass becomes $[_,c^{l}_{k}]=\mathcal{L}_{k}([P^{l}_{k-1},c^{l}_{k-1}])$ and $[_,c^{v}_{k},I_{k}]=\mathcal{V}_{k}([P^{v}_{k-1},c^{v}_{k-1},I_{k-1}])$. Each $P^{l}$/$P^{v}$ contains tunable parameters.

Figure: Figure 1: Relationship between generalizability represented by the test accuracy on unseen classes during training and Information Density (ID). When generalizability increases, ID also increases. The Spearman correlation coefficient $\rho$ between generalizability and ID1/ID2 is very high, *i.e*. $\geq$ 0.9.
Refer to caption: x1.png

### 3.2 Information Density in Prompt Tuning

Here, we first provide precise definitions for “information density” to clearly convey our motivations. For typical parameter matrix like prompts $P\in\mathbb{R}^{n\times d}$(assume $n<d$), we can always rewrite such matrix into a combination of several orthogonal bases with different weights by singular value decomposition (SVD). Formally, $P=U\Sigma V^{T}=\sum_{i=1}^{d}\sigma_{i}u_{i}v_{i}^{T}$. Each $u_{i}v_{i}^{T}$ can span a unique feature space, and $P$ is a linear combination of these features. Typically, $\{\sigma_{i}\}_{i=1}^{n}$ are arranged in descending order.

To help readers better understand the concept of information density, let’s draw an analogy using images from nature. A real image always has one or a few very prominent features and almost never contains an even mix of various odd features. In an extreme case, if an image truly exhibits isotropic characteristics, it would simply mean that its content is almost entirely noise and lacks clear meaning. Reflecting this back to the matrix decomposition we discussed earlier, a good image tends to have several significantly larger singular values. The features of the image can largely be expressed by the feature space behind these prominent singular values.
Therefore, from the matrix decomposition expression, the differences among the singular values $\{\sigma_{i}\}_{i=1}^{n}$ are significant, indicating that the information is concentrated in a few feature spaces. In other words, the information density is higher. Thus, to quantize such property, we define $k$-th order “Information Density (ID)” as follows: ID$k=\frac{\sum_{i=1}^{k}\sigma_{i}}{\sum_{i=1}^{n}\sigma_{i}}$. In other words, ID$k$ is the proportion of the largest $k$ singular values among all the singular values. Greater information density represents more robust and stable intrinsic features, meaning they are less likely to be affected by external disturbances and have stronger anti-interference capabilities.

Returning to our initial discussion, a core contribution of this paper is the hypothesis and verification that the parameter matrices, like prompts, follow the same ID-related laws during fine-tuning in downstream CLIP models. As is well-known, in transfer learning, the transfer of knowledge in a model depends on the updating of parameters. For CLIP models, the optimal solution is prompt tuning [, , , ]. We first hypothesize that the information density of the prompt matrix also represents its robustness and the strength of its intrinsic features. Therefore, greater information density should theoretically result in better generalizability throughout the prompt tuning process.

To verify this hypothesis, we conducted an experiment using a classic method, CoOp []. During training, we masked half of the classes, using data from only the other half for training, and performed singular value decomposition on the prompt matrices during the process. As shown in Fig. 1, for better visualization, ID1 is scaled up to 2x to be put under the same right axis with ID2. Unseen classes accuracy is improved in the first few iterations, but it starts dropping later, indicating overfitting and catastrophic forgetting. Importantly, the fluctuation trend of unseen classes accuracy is highly consistent with ID. We compute the Spearman correlation coefficient between unseen classes accuracy and ID1/ID2 in Fig. 1. Clearly, the first-order and second-order information densities of CoOp on the SUN397 [] and DTD [] datasets exhibit a very strong correlation with the accuracy on unseen classes (*i.e*., generalizability), with Spearman correlation coefficients greater than 0.9. This demonstrates that our hypothesis is correct.

In the following Section 4, we will show how we can leverage such correlation between generalizability and ID to boost CLIP’s downstream performance.

## 4 Methodology

### 4.1 Dense Information Prompt

#### 4.1.1 Algorithms for increasing information density

In this part, we aim to find an effective algorithm to increase information density to enhance CLIP’s downstream generalization performance. Motivated by the observation in Section 3, we propose three optional algorithms.

Alg. 1: Do SVD and directly optimize information density as an training objective. Specifically, we add corresponding expression to the loss item after decomposing prompt matrix by SVD. Suppose the original cross entropy loss is $\mathcal{L}_{CE}$ and we want to maximize ID$k$ ($1\leq k\leq n)$, the current loss is $\mathcal{L}=\mathcal{L}_{CE}-\lambda\mbox{ID}k$, where $\lambda$ is a positive hyper-parameter.

Alg. 2: Do SVD and apply order-related penalty on the singular values. Specifically, we add regularization term to our optimization objective. Suppose the original cross entropy loss is $\mathcal{L}_{CE}$ and we want to maximize ID$k$ ($1\leq k\leq n)$, the current loss is $\mathcal{L}=\mathcal{L}_{CE}+\lambda\sum_{i=n-k+1}^{n}i||\sigma_{i}||$.

Alg. 3: Approximate the original matrix by the product of two small matrices. Formally, given a typical prompt matrix $P\in\mathbb{R}^{n\times d}$, we take inspiration from LoRA [] to create two low-dimensional matrices and use their product as an approximated equivalent prompt. Suppose we want to maximize ID$r$ ($1\leq r\leq n)$, we first randomly initialize $P_{A}\in\mathbb{R}^{n\times r}$ and $P_{B}\in\mathbb{R}^{r\times d}$. The low-rank approximated prompt $P_{lr}=P_{A}P_{B}$ is with the same shape with $P$, and thus can participate in the training.

We do a quick verification on several datasets used in the base-to-new generalization setting and show the results in Tab. 1. Clearly, from Alg. 1 to Alg. 3, the intensity of constraint is becoming stronger, resulting in more accuracy loss on base classes while enjoying more accuracy gain on new classes. In addition to the good performance, Alg. 3 has another advantages: it’s quite simple and easy to implement with faster training speed. Therefore, we choose Alg. 3 to implement our DIP. However, unlike LoRA who merely uses a similar low-rank product as a supplementary addition for the output of main branch, Alg. 3 directly outputs the product for usage, leading to some unique difficulties and challenges. We will show the problems behind such approximation and our solutions in the next sections.

**Table 1: A quick check for different implementations of DIP.**
|  | base | new | H |
| --- | --- | --- | --- |
| w/o any | 82.69 | 63.22 | 71.66 |
| Alg. 1 | 81.21 | 67.58 | 73.77 |
| Alg. 2 | 80.39 | 69.85 | 74.75 |
| Alg. 3 | 79.91 | 71.48 | 75.46 |

#### 4.1.2 Special Initialization

In the field of tuning vision-language models, existing works have confirmed that the initialization method of prompts is quite important. For example,  [, ] adopt a hand-crafted template as the initial point of the text prompts, and  [] copies the parameters in the text or image class token to initialize the prompts of the corresponding branch. If we just do a random initialization, the overall performance of such existing methods would drop severely. See Section 5.2 for more details. In other words, it would be helpful if we could take advantage of a good initial point.

The problem lies in the fact that artificially designed initialization $P_{init}$ is almost certain to be full rank, *i.e*. $\mbox{rank}(P_{init})=min(n,d)$, and our current low-rank prompt $P_{lr}=P_{A}P_{B}$ could only express the matrices whose rank is less or equal than $r$. Since $r<min(n,d)$, it is impossible to directly initialize $P_{A}$ and $P_{B}$ by a given $P_{init}$.

To solve such a problem, we add a concurrent branch of full-rank prompts $P_{fr}\in\mathbb{R}^{n\times d}$ along with the proposed low-rank prompts as shown in Fig. 2(b). Naturally, $P_{fr}=P_{init}$. And we randomly sample $P_{A}$/$P_{B}$ from a Gaussian distribution in which $\mu=0$ and $\sigma\rightarrow 0$. A small $\sigma$ here could avoid constant initialization and enrich the update paths. Notably, $P_{fr}$ is kept frozen during the whole adaptation, and thus it would not increase the number of tunable or stored parameters.

Figure: ((a)) Comparison between DIP Tuning and Prompt Tuning.
Refer to caption: x2.png

#### 4.1.3 Regularization

As discussed in Section 2, existing works have shown that proper regularization would significantly improve the generalization ability. Therefore, to alleviate overfitting and catastrophic forgetting, we put a Dropout layer with drop ratio $p$ after the low-rank branch as displayed in Fig. 2(b).

Therefore, the input prompt of the feature encoder is $P=P_{fr}+\mbox{Dropout}(P_{lr},p)$.
Finally, we have $P=P_{init}+\mbox{Dropout}(P_{A}P_{B},p)$.

### 4.2 Efficiency Analysis

The whole adaptation process of a pre-trained vision-language model can be divided into three parts: training, storage and inference. In this subsection, we will analyze the efficiency of DIP in each part separately.
Training In the training phase, the number of tunable parameters in total is $r(n+d)$. Compared with classic prompt tuning which has $nd$ tunable parameters, we can largely reduce the trainable parameters by choosing a small $r$ satisfying $r<<min(n,d)$. Such reduction could help us train the model faster and use less memory during training than those delicate-designed methods with massive complicated structures and update rules.
Storage After training, we just store $P_{A}$ and $P_{B}$ onto the disk for every downstream task. Similar to the training period, the number of stored parameters is $r(n+d)$. In contrast, classic prompt tuning stores a much larger set of $nd$ parameters. As a result, we can save a fairly large amount of storage space when there are lots of tasks to be adapted.
Inference Before inference, we first load $P_{init}$, $P_{A}$ and $P_{B}$ from disk to memory. Noticing that Dropout is exactly an identity layer in the inference mode, we could pre-calculate the equivalent $P$ by $P=P_{init}+P_{A}P_{B}$ and just keep $P$ in the memory. For inference, we directly use $P$ as the input prompts, and thus the inference cost is the same as classic prompt tuning. Some existing methods add complex bridges between the isolated parameters to earn extra improvements, *e.g*. CoCoOp []. There ain’t no such thing as a free lunch. They would face slower speed and huge memory occupation in inference time.

## 5 Experiments

To verify the effectiveness of the proposed method, we evaluate our method and make comparisons with the latest state-of-the-art methods in terms of the following settings in a wide range: base-to-new generalization, domain generalization, cross-dataset transfer and few-shot learning.

For more experimental details, please refer to the Appendix.

**Table 2: Comparisons with latest methods in base-to-new generalization. H: harmonic mean []. In the lightweight methods, DIP+CoOp outperforms all the other competitors on new and harmonic mean accuracy. While in the heavy methods, DIP+MaPLe surpasses MaPLe [] by a clear margin, and is strongest on base, new and harmonic mean accuracy, even with around only $\frac{1}{5}$ parameters compared with MaPLe.**
| Method | #params | Base | New | H |
| --- | --- | --- | --- | --- |
| Lightweight |  |  |  |  |
| CLIP [] | 0K | 69.34 | 74.22 | 71.70 |
| CoOp [] | 2.1K | 82.69 | 63.22 | 71.66 |
| CoCoOp [] | 35.4K | 80.47 | 71.69 | 75.83 |
| Adapter [] | 525.5K | 82.62 | 70.97 | 76.35 |
| LoRA [] | 129.0K | 84.30 | 67.33 | 74.86 |
| ProGrad [] | 8.2K | 82.79 | 68.55 | 75.00 |
| DIP+CoOp | 0.5K | 80.32 | 74.73 | 77.42 |
| Heavy |  |  |  |  |
| MaPLe [] | 3548K | 82.28 | 75.14 | 78.55 |
| DIP+MaPLe | 741.9K | 83.17 | 75.43 | 79.11 |

### 5.1 Main Results

#### 5.1.1 Base-to-new Generalization

The average results over 11 datasets are shown in Tab. 2. For complete results, please refer to the Appendix. Overall, DIP shows the strongest performance, and the superiority mainly relies on the improvement of new classes. In other words, DIP largely improves the generalization ability of CLIP. In particular, compared with our direct baseline CoOp, DIP gets $10.95$% accuracy gain on the new classes and $2.83$% accuracy drop on the base classes. By adding DIP to the prompts as well as the linear layers of MaPLe, we earn a excellent 0.56% performance gain while reducing its parameters to $0.2\times$ of the original parameters. Notably, compared with the latest lightweight method ProGrad, even if DIP only uses $<$20x of its parameters, DIP still outperforms it by a clear margin, which clearly demonstrates the efficiency and effectiveness.

**Table 3: Comparisons with latest methods in domain generalization after tuned on ImageNet. DIP+CoOp shows excellent robustness when dealing with domain shift.**
|  | #params | IN-V2 | IN-Sketch | IN-Adversarial | IN-Rendition | Average |
| --- | --- | --- | --- | --- | --- | --- |
| CLIP | 0K | 60.83 | 46.15 | 47.77 | 73.96 | 57.18 |
| CoOp | 2.1K | 64.20 | 47.99 | 49.71 | 75.21 | 59.28 |
| CoCoOp | 35.4K | 64.07 | 48.75 | 50.63 | 76.18 | 59.91 |
| Adapter | 525.5K | 62.53 | 47.67 | 49.17 | 75.42 | 58.70 |
| LoRA | 129.0K | 62.37 | 42.43 | 38.40 | 68.97 | 53.04 |
| ProGrad | 8.2K | 64.73 | 47.61 | 49.39 | 74.58 | 59.07 |
| DIP+CoOp | 0.5K | 63.95 | 49.07 | 50.97 | 77.19 | 60.30 |

#### 5.1.2 Domain Generalization

Then we follow CoCoOp [] to use ImageNet, ImageNet-A, ImageNet-R, ImageNet-v2, and ImageNet-S to run domain generalization experiments to verify the robustness of DIP (DIP = DIP + CoOp, and the same applies thereafter, unless otherwise specified.). Shown in Tab. 3, on target datasets, DIP leads to better average accuracy compared with the latest methods, largely outperforming CLIP, CoOp, CLIP-Adapter and ProGrad with much fewer parameters.

**Table 4: Results in the cross-dataset transfer setting. DIP+CoOp gives the highest accuracy on 6 of 10 datasets, and slightly outperforms CoCoOp on average. Such result well demonstrates that DIP could maximally extract general and data-agnostic knowledge from given images.**
|  | # params | Caltech101 | Pets | Cars | Flowers | Food101 | Aircraft | Sun397 | DTD | EuroSAT | UCF101 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CoOp | 2.1K | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88 |
| CoCoOp | 35.4K | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 | 65.74 |
| Adapter | 525.5K | 93.43 | 88.87 | 64.40 | 70.27 | 85.63 | 24.67 | 65.80 | 44.90 | 47.70 | 66.00 | 65.17 |
| DIP+CoOp | 0.5K | 94.20 | 90.50 | 67.17 | 71.27 | 86.07 | 23.83 | 67.60 | 46.73 | 42.10 | 68.93 | 65.84 |

#### 5.1.3 Cross-dataset Transfer

Finally, we follow CoCoOp [] to conduct cross-dataset transfer evaluation. Results are shown in Tab. 4. Concentrating too much on the current dataset will absolutely cause overfitting and catastrophic forgetting problems, and finally lead to a severe drop in the performance on those unseen datasets. In this setting, DIP wins on 6 of 10 datasets and its average accuracy is also slightly better than the best competitor CoCoOp. Such result well demonstrates that DIP could maximally extract general and data-agnostic knowledge from given images compared with other prompt-based methods. Considering the huge difference in the parameter numbers, we could summarize that DIP is still the better choice.

#### 5.1.4 Few-shot Learning

In this paragraph, we will show the experiment results of DIP in the few-shot learning setting. This setting is originated from CoOp. Seen from Fig. 4, DIP consistently outperforms zero-shot CLIP, CoOp, and CLIP-Adapter across all the shot numbers. Such results demonstrate the superiority of DIP in adaptation ability when there are few samples in downstream tasks.

Overall, in base-to-new generalization, domain generalization, cross-dataset transfer and few-shot learning, DIP can be fruitfully integrated into existing methods and consistently reaches state-of-the-art performance while enjoying extremely high parameter efficiency.

Figure: Figure 3: Few-shot learning Results.
Refer to caption: x4.png

### 5.2 Analysis

**Table 5: Ablation study on base-to-new generalization setting**
|  | Decomposition | Full-rank init | Dropout | #params | base | new | H |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CoOp | - | - | - | 2.1K | 82.69 | 63.22 | 71.66 |
| DIP+CoOp | ✓ | - | - | 0.5K | 79.91 | 71.48 | 75.46 |
| ✓ | ✓ | - | 0.5K | 79.42 | 73.21 | 76.19 |  |
| ✓ | ✓ | ✓ | 0.5K | 79.70 | 73.59 | 76.53 |  |

#### 5.2.1 Ablation study

We first show the impact of components of DIP step by step.

We start from transforming the text encoder. We first replace the tunable prompt tokens with our decomposed small prompt matrices, denoted as “Decomposition”, for the CoOp method. After such replacement, the average Harmonic accuracy over 11 datasets directly improved from 71.66% to 75.46%. The accuracy on the base classes decreases by 2.78% and on the new classes increases by 8.26%. Although the adaptation ability slightly drops, the generalization ability raises quite a lot. This phenomenon proves that our low-rank design is fairly beneficial for the model’s generalization ability once again. Then we integrate the special full-rank initialization into CoOp. The harmonic mean accuracy improves by 1.27%. Finally, we add a lightweight regularization layer, Dropout. It helps us alleviate overfitting and catastrophic forgetting, resulting in further improvement.

#### 5.2.2 Additional experimental results

Effect of different training epochs In this paragraph, we investigate that how the total training epoch could influence the adaptation result. Shown in Fig. 4, we run experiments for the given epochs separately. As the training epoch increases, the accuracy on base classes continues decreasing while the accuracy on new classes continues increasing. It is reasonable because as the training continues, the model has a higher risk of forgetting its original knowledge and overfitting.

Effect of different dropout ratios In this paragraph, we will show the influence of different dropout ratios in DIP on ImageNet. Seen from Fig. 4, as the dropout ratio increases, the base accuracy starts decreasing while the new accuracy starts increasing mostly. The harmonic mean first increases and then decreases. Dropout iss a kind of regularization, only proper regularization can help avoid overfitting and catastrophic forgetting.

CLIP with convolutional image encoder In this paragraph, we show the results of DIP on CLIP with convolutional image encoder ResNet-50 [], rather than the default ViT-B/16 []. Seen from Section 5.2.2, compared with baseline CoOp, DIP still largely improves the new accuracy and the harmonic mean accuracy over 11 datasets, while the base accuracy slightly drops. Compared with the latest method ProGrad, DIP shows clear superiority on base accuracy and the harmonic mean accuracy.

Figure: Table 6: Results on ResNet-50 encoded CLIP.

**Table 8: Results of adding DIP to image prompts on ImageNet.**
|  | #params | base | new | H |
| --- | --- | --- | --- | --- |
| Text DIP | 0.5K | 75.87 | 70.67 | 73.17 |
| Image DIP | 0.5K | 74.57 | 69.40 | 71.89 |

Effect of different ranks In this paragraph, we will show the influence of different ranks in DIP. Seen from Section 5.2.2, roughly, a larger rank brings more parameters, higher base accuracy, and lower new accuracy. As a result, increasing rank is not necessarily able to improve the average accuracy.

Results for using DIP on the image prompts There are several works [, ] indicating the effectiveness of the image prompts. Therefore, in this subsection, we will explore the results of applying DIP to image prompts. Seen from Tab. 8, using DIP on the image side also reaches high accuracy. However, the base, new, and average accuracy of image DIP is not as good as those of text DIP. Such experiment result tells us to use DIP on the text side instead of the image side when we aim to reach high accuracy with extremely few parameters.

## 6 Conclusion

With the development of huge vision-language models, how to effectively and efficiently adapt such huge models to downstream tasks becomes a challenging problem. Much effort has been made to leverage the potential of prompt tuning in adapting vision-language models. However, existing methods suffer from inefficiency. To reach extremely efficient generalization, we propose Dense Information Prompt (DIP) based on the observation about the strong correlation between prompt rank and generalizability. Moreover, we propose a novel initialization method and a lightweight regularization module to further improve the low-rank design without adding any extra inference cost. We are the first to explore how to effectively adapt vision-language models using an extremely small number of parameters, *i.e*. 0.5K. Besides efficiency and effectiveness, DIP has many valuable advantages such as simplicity and robustness. Extensive experiments and analyses show the superiority of DIP sufficiently.

## Appendix A Datasets

Following previous work [, ], we leverage 11 image recognition datasets to verify the effectiveness of the proposed method for both the base-to-new generalization task. These datasets include two datasets for the generic object classification, *i.e*., ImageNet [] and Caltech101 [], five datasets for the fine-grained classification, *i.e*., OxfordPets [], StanfordCars [], Flowers102 [], Food101 [] and FGVCAircraft [], one dataset for the scene recognition, *i.e*., SUN397 [], one dataset for the action recognition, *i.e*., UCF101 [], one dataset for the texture classification, *i.e*., DTD [], and one dataset for the satellite imagery recognition, *i.e*., EuroSAT []. Following previous works [, ], for each dataset, we split its classes equally into two non-overlapping groups, *i.e*., one as base classes and the other as new classes. We train all models on the base classes and perform a base/new evaluation on the base/new classes.

For the domain generalization task, we utilize ImageNet-A [], ImageNet-R [], ImageNetv2 [] and ImageNet-S [] to verify the robustness of the model. In this setting, we need to first train the model using ImageNet, and then directly use images from other four datasets to do inference.

For the cross-dataset transfer task, the datasets are the same as those of the base-to-new generalization task. Similar to domain generalization, the model will be first trained on ImageNet and then do inference on the other 10 different datasets.

For the few-shot learning task, the datasets are the same as those of the base-to-new generalization task. The model will be trained and evaluated with 1, 2, 4, 8 and 16 shots separately.

The dataset splitting is exactly the same as previous works [, ]. We report the averaged model performance over three runs with different random seeds for fair comparisons.

## Appendix B Training Details

Following previous work [], we employ ViT-B/16 as the image encoder in the CLIP. Each training image is resized to $224\times 224$ before being fed into the image encoder. Some common data augmentation strategies, *e.g*., random crop and random flip, are used to enhance the model performance, following []. During training, we set the batch size as 32. We employ the stochastic gradient descent algorithm (SGD) to optimize the learnable parameters. As [], we utilize a warm-up scheme at the first epoch, which is important for the tuning of prompts. For all the other baselines, we strictly follow the configurations of their original papers.

To verify the effectiveness of our proposed method, we explore the improvement of integrating DIP into a lightweight prompt tuning method CoOp and a heavy prompt tuning method MaPLe separately.

For DIP+CoOp and DIP+MaPLe, we conduct a grid search to find the optimal hyper-parameters based on the configuration of CoOp and MaPLe. In the main text, we set the rank $r=1$ in DIP for all the experiments unless specially mentioned. The total number of tunable parameters in DIP+CoOp is $4*1+1*512=516$, which is merely around 0.25x of that in CoOp. For DIP+MaPLe, we also decompose its weights of the projection layer that projects text prompts to generate image prompts with rank $r_{proj}=64$. Therefore, the total number of tunable parameters in DIP+MaPLe is $(4*1+1*512)+(512*64+64*768)=82436$ per layer, and $82436*9=741924$ in total for 9 layers are modified in MaPLe by default.

**Table 9: Detailed comparisons with latest methods in base-to-new generalization. H: harmonic mean []. DIP+CoOp outperforms all the other lightweight methods on 8 out of 11 datasets, and DIP+MaPLe achieves better harmonic mean accuracy on 9 out of 11 datasets compared to MaPLe alone. Better still, methods modified by DIP even enjoys significantly better parameter efficiency.**
| Method | CoOp | CoCoOp | Adapter | LoRA | ProGrad | DIP+CoOp | MaPLe | DIP+MaPLe |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #params | 2.1K | 35.4K | 525.5K | 129.0K | 8.2K | 0.5K | 3548K | 741.9K |

## Appendix C Competitors

- 1.
CLIP []: CLIP is a strong baseline vision-language model that is pre-trained on a large number of image-text pairs from the web by learning a contrastive objective. CLIP enables strong zero-shot adaptation ability on various downstream tasks by using fixed text prompts, *i.e*. a photo of a.
- 2.
CoOp []: CoOp replaces the fixed text prompts in CLIP with tunable text prompts to improve the adaptation ability of the vision-language model. CoOp shows excellent performance in few-shot situations.
- 3.
CoCoOp []: CoCoOp replaces the isolated tunable text prompts in CoOp with conditional text prompts, which receive extra gradients from the image features besides text features. CoCoOp largely improves the generalization ability of the vision-language model, getting good results on base-to-new generalization and domain adaptation.
- 4.
CLIP-Adapter []: CLIP-Adapter adopts the thoughts of classic Adapter [] to use serial linear layers and activation functions to adapt for downstream tasks. It is simple yet effective in few-shot learning.
- 5.
LoRA []: LoRA adopts low-rank decomposition for weights in FC layers. It is efficient and earns good results in NLP field.
- 6.
MaPLe []: MaPLe simultaneously adds prompts to the image encoder and text encoder of CLIP. To trigger more information exchange between the image side and text side, MaPLe generate image prompts from the projection of text prompts. Though effective, such design brings quite heavy cost.
- 7.
ProGrad []: ProGrad only updates the text and image prompts whose gradient are aligned (or non-conflicting) to the general knowledge, which is represented as the optimization direction offered by the pre-defined prompt predictions. Such regularization helps it finish good adaptation and generalization.

**Table 10: Base-to-new generaliation performances based on SLIP [].**
|  | #params | base | new | H |
| --- | --- | --- | --- | --- |
| CoOp | 2.1K | 68.45 | 42.77 | 52.64 |
| DIP+CoOp | 0.5K | 62.53 | 47.78 | 54.17 |

**Table 11: Results of different combinations of learning rates and weight decays under 16-shot learning setting on ImageNet**
| lr<br>wd<br>Acc | 1e-4 | 5e-4 | 1e-3 |
| --- | --- | --- | --- |
| 1e-3 | 70.73 | 70.67 | 70.78 |
| 2e-3 | 70.80 | 70.83 | 70.77 |
| 3e-3 | 70.83 | 70.83 | 70.81 |

**Table 12: Deep prompts in different depths for DIP+CoOp.**
| depth | #params | base | new | H |
| --- | --- | --- | --- | --- |
| 1 | 0.5K | 76.37 | 74.69 | 75.52 |
| 2 | 1.0K | 77.81 | 73.87 | 75.79 |
| 3 | 1.5K | 79.55 | 72.72 | 75.98 |
| 4 | 2.0K | 80.07 | 72.69 | 76.20 |
| 5 | 2.6K | 80.24 | 73.37 | 76.65 |
| 6 | 3.1K | 80.64 | 73.39 | 76.85 |

## Appendix D Complete Results in Base-To-New Generalization

As shown in Tab. 9, overall, DIP shows quite competitive performance, and the superiority mainly relies on the improvement of new class accuracy. In other words, DIP largely improves the generalization ability of CLIP. In particular, compared with CoOp, DIP+CoOp gets $10.95$% accuracy gain on the new classes and $2.83$% accuracy drop on the base classes, showing strongest harmonic mean accuracy on 8 out of 11 datasets among all the lightweight existing methods. Compared with a heavy method MaPLe, DIP+MaPLe achieves better harmonic mean accuracy on 9 out of 11 datasets.

## Appendix E Experiments on a Different Vision-Language Architecture

In this paragraph, we show the results of DIP on another vision-language architecture, SLIP [], besides CLIP. Seen from Tab. 10, DIP+CoOp earns much higher new accuracy and harmonic mean accuracy than the original CoOp.

## Appendix F Effect of Deep Prompts

In this paragraph, we extend the shallow prompts in DIP+CoOp to the deep prompts. We record the accuracy change as we increase the layers including tunable prompts, following the last equation in Section 3.1 in the main text. Results are shown in Tab. 12. As the depth increases, the base accuracy keeps increasing while the new accuracy first decreases and then increases. Overall, more depth generally leads to higher harmonic mean accuracy. Therefore, it is possible to further improve the performance of our method by increasing the prompt depth.

## Appendix G Effect of Different learning rates and Weight Decays

In this paragraph, we investigate the effect of normal hyper-parameters learning rate and weight decay. We wonder if tuning them carefully would lead to significant improvement. Seen from Tab. 11, the performance of DIP stays stable whatever the learning rate and weight decay vary. The conclusion here is that our method DIP is robust for learning rate and weight decay.

**Table 13: Training, storage, and inference efficiencies.**
|  | #params | Training<br>throughput | Inference<br>throughput |
| --- | --- | --- | --- |
| CoOp | 2.1K | 93 image/s | 738 images/s |
| CoCoOp | 35.4K | 5 images/s | 13 images/s |
| ProGrad | 8.2K | 56 images/s | 732 images/s |
| DIP+CoOp | 0.5K | 91 images/s | 738 images/s |

## Appendix H Efficiency Comparison

In this paragraph, we will give a comprehensive analysis of all the training, storage, and inference efficiencies for DIP and several existing methods. Since the parameter scale of CLIP-Adapter [] is significantly larger than others,
we do not contain CLIP-Adapter into comparison.

For a fair comparison, we do all the speed tests on the same GPU. Results are shown in Tab. 13. Our proposed DIP shares nearly the same fastest training and inference speeds with the simplest method CoOp, and more importantly, DIP merely uses 0.5K parameters, which is a lot more storage-efficient than other methods. Specially, compared with classic method CoCoOp, DIP enjoys $>$18x training speed, $>$56x inference speed and $<$70x storage usage. Besides complex structures, the huge gap in the inference speed is partly owing to the huge memory cost of CoCoOp, which forces us to adopt a smaller batch size than other methods for CoCoOp. Moreover, compared with the latest method ProGrad, DIP enjoys $>$1.6x training speed, comparable inference speed, and $<$60x storage usage, which adequately demonstrates the super efficiency of DIP.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Is there any performance improvement when applying DIP to visual prompts of MaPLe?
soundness: 3
strengths: 1. DIP can be easily applied in a plug-and-play manner to existing prompt tuning methods without significant changes to the model architecture.
2. The performance of DIP has been demonstrated on various benchmark datasets.
3. It is efficient by selecting and updating only important parameters compared to conventional prompt tuning methods.
summary: The paper proposes a prompt tuning method based on information density that updates only important parameters. This approach is memory-efficient as it reduces the number of parameters and can be easily applied to various prompt tuning methods. Additionally, the authors proposed initialization and regularization techniques when introducing DIP to enhance generalization performance.
weaknesses: 1. The claim by the authors that “ID and generalizability are correlated” lacks sufficient experimental evidence. Specifically, in Figure 1, analysis was conducted on only 2 out of 11 datasets, making it difficult to confirm whether this claim is limited to certain datasets or is generally applicable. To enhance persuasiveness, showing similar trends for the remaining datasets and demonstrating generalizability across each dataset would be beneficial. Moreover, in Table 2, only average values are reported, so it is unclear if generalizability consistently improves.
2. Does DIP robust to hyperparameters $\lambda_{DO}$ and $\lambda_{PP}$? Analysis of hyperparameter sensitivity is omitted.
3. As shown in the results in Table 8, there are some limitations when applied to visual prompts.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. It seems there is confusion to capture how this paper conveys and demonstrates the information density in the prompt tuning. They need to conduct some experiments that information density clearly correlates with generalizability and adaptation. The authors should demonstrate that this differs from the LoRA, as truncated SVD is analogous to LoRA. While from the Figure 1, it is not convincing to deliver that the Spearman correlation coefficient ρ between information density and new classes accuracy is high and that means the information density strongly correlates with the generalizability, and how about the decrease in terms of the base classes.
soundness: 2
strengths: 1. The experimental results are improved when employed in several prompt-based methods.

2. Attempting to explore the Information Density, being similar to information theory, is kind of interesting.
summary: This paper tries to leverage the Information Density idea to promote prompt tuning for fine-tuning CLIP to be evaluated on downstream tasks. Specifically, the authors implement such  “Information Density” (ID)  via SVD decomposition, positional penalty, and LoRA to enhance the training, simultaneously DropOut is used to regularize the model not to overfit. Regarding the experiments, the authors present improvements when applied to commonly prompt-based methods, like CoOp, ProGrad, MaPLe, and DePT.
weaknesses: 1. This paper's motivation apparently lacks logical connections between information density and prompt tuning, which shows confused connections between these two parts. Why can applying the SVD decomposition to obtain the maximal values convey information density, or will the model get into a trivial solution that only these K values works while all the others get zero values? If the authors just fix the maximal K values, essentially, this is truncated SVD and can be analogous to LoRA with k ranks.

2. Regarding the optimization objective L_DO, it is quite weird that the CE term is optimized to be minimized while controlling the λ_DO to be positive, thus the λ_DO * ID_k  is still positive. This results in a decrease in the optimization procedure of cross-entropy loss for this specific downstream task, which can be validated in Table 1 that the base accuracies are inferior to the baseline and are opposite to our fine-tuning target.

3. Based on Weakness 2, I think this sounds like a tricky implementation to regularize the model so as not to overfit the base classes too much and allow the model to perform well at the unseen classes. However, the authors show no in-depth for this part. What's worse, this is totally irrelevant to the information density while considering just applying a loss factor lower than 1.0 for the cross-entropy loss. They need to present the information density change after applying such DIP tuning compared with the baseline.

4. It seems the overall method necessitates to ensemble of LoRA DIP and PP, together with the original prompt vector to obtain final improved results, whereas a dropout is applied to the DIP blocks which shows great improvements but it is a widely used trick. This confuses whether the proposed method shows improvements, or just ensemble some tricks to get improved results rather than coming from the initial "information density" idea.

5. Lots of ablative studies are missed in Table 5. When applying 'Initialization', we can see that the new classes' accuracy improves greatly while further applying 'Regularization' shows marginal improvements. How about removing DO, PP, and LRA within DIP to see the effect of each component?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Is the number of trainable parameters same for three DIP variants?
soundness: 2
strengths: **[New insight]** This work investigates the correlation between information density and generalization in the VLMs domain, which is inspiring.
summary: The paper introduces "Information Density" (ID) to measure matrix affiliation with specific feature spaces, hypothesizing that higher ID correlates with robustness and stability. It proposes the Dense Information Prompt (DIP) to boost generalization by increasing ID, with several effective algorithms presented to achieve this.
weaknesses: **[Dataset shown in Figure 1]** Illustrating the correlation between Information Density and Accuracy is significant, which motivates the method design. Hence, instead of testing on small datasets, it would be nice to experiment on a more representative dataset, e.g., ImageNet. This is because ImageNet is a well-known dataset with more diverse classes, making it a suitable dataset for this investigation. Additionally, it would be better to show results with more baselines, e.g., CoCoOp [1], MaPLe [2] and PromptSRC [3], to convince the readers. This can provide more comprehensive understanding whether this observation suits different types of prompt tuning methods.

[1] Conditional prompt learning for vision-language models. CVPR 2022.

[2] Maple: Multi-modal prompt learning. CVPR 2023.

[3] Self-regulating prompts: Foundational model adaptation without
Forgetting. ICCV 2023.

**[Need evidences]** In section 3.2, to explain information density, the authors draw an analogy using images from nature. It would be more convincing if some previous works containing this philosophy can be provided. For examples, the authors can provide some examples of related work that use similar analogies or concepts to explain information density or related ideas in machine learning.

**[Need more explanation]**  
- In section 4.1.1, for Direct Optimization, it needs more details about how the method optimizes IDk to update the original P, i.e.,  a step-by-step explanation of the optimization process for IDk.
- In Table 1, it should introduce the baseline method and the dataset evaluated on. Please add these information in the caption.

**[Wrong equation]** In line 267, the last equation seems wrong, i.e., the second item differs from the one given in line 263.

**[Unclear illustration]** In Figure 2 (a), it should show the difference between these two methods, e.g., DIP has SVD and new loss function. It would be nice to add labels or annotations to highlight these key differences.

**[Experiments]** 
- In Table 2, the results of DIP + PromptSRC [1] is missed. It would evaluate the method with diverse types of prompt tuning.
- In Table 3 and 4, some baseline methods, e.g., MaPLe [2], PromptSRC [1] and DePT [3], are missed. 
- Training time comparison should be given. This would give a more comprehensive understanding of the effectiveness of the method in various aspects

[1] Self-regulating prompts: Foundational model adaptation without Forgetting. ICCV 2023.

[2] Maple: Multi-modal prompt learning. CVPR 2023.

[3] Dept: Decoupled prompt tuning. CVPR 2024.