Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation
Authors: Sua Lee
Kyubum Shin
Jung Ho Park
Venue: ICLR 2025 Poster (2025)
Keywords: Prompt learning
Vision-language models
Large language models
Few-shot image recognition
Primary area: foundation or frontier models, including LLMs
TL;DR: A new approach to distill the LLM information into VLM prompts without descriptions, while also weighting important prompts in a multi-prompt setting.

Abstract: Recent advances in pre-trained Vision Language Models (VLM) have shown promising potential for effectively adapting to downstream tasks through _prompt learning_, without the need for additional annotated paired datasets.
To supplement the text information in VLM trained on correlations with vision data, new approaches leveraging Large Language Models (LLM) in prompts have been proposed, enhancing robustness to unseen and diverse data.
Existing methods typically extract text-based responses (i.e., _descriptions_) from LLM to incorporate into prompts; however, this approach suffers from high variability and low reliability.
In this work, we propose **De**scription-free **Mul**ti-prompt Learning(**DeMul**), a novel method that eliminates the process of extracting descriptions and instead directly distills knowledge from LLM into prompts.
By adopting a description-free approach, prompts can encapsulate richer semantics while still being represented as continuous vectors for optimization, thereby eliminating the need for discrete pre-defined templates.
Additionally, in a multi-prompt setting, we empirically demonstrate the potential of prompt weighting in reflecting the importance of different prompts during training.
Experimental results show that our approach achieves superior performance across 11 recognition datasets.

Published as a conference paper at ICLR 2025
WEIGHTED MULTI-PROMPT LEARNING WITH DESCRIPTION-FREI LARGE LANGUAGE MODEL DIS- TILLATION
Sua Lee'r, Kyubum Shin2*, Jung Ho Park Scoul National University, Naver Al
ABSTRACT
Recent advances in pre-trained Vision Language Models (VLM) have shown promising potential for effectively adapting downstream tasks through prompt learning, without the need for additional annotated paired datasets. To supple- ment the text information in VLM trained on correlations with vision data, new approaches leveraging Large Language Models (LLM) in prompts have been pro- posed, enhancing robustness to unseen and diverse data. Existing methods typically extract text-based responses (i.e., descriptions) from LLM to incorporate into prompts; however, this approach suffers from high variability and low reliabil- ity. In this work, we propose Description-Fii a novel method that eliminates the process of extracting descriptions and instead directly distills knowledge from LLM into prompts. By adopting description-free approach, prompts can encapsulate richer semantics while still being represented as continuous vectors for optimization, thereby eliminating the need for discrete pre-defined templates. Additionally, in a multi-prompt setting, we empirically demonstrate the potential of prompt weighting in reflecting the importance of different prompts during training. Experimental results show that our approach achieves superior performance across recognition datasets.
Multi-prompt Learning(DeMuun
INTRODUCTION
What are the sentences that "best" describe the Golden Retriever in the image shown in Fig. I Even when familiar with the dog, answers to this question will always differ, there cannot be definitive correct answers. Thus, if such ambiguous sentences are defined as the "categories" of the image, becomes challenging for others to categorize it accurately.
Recently, there has been a growing interest in leveraging pre-trained Large Language Models (LLM), such as GPT(Brown al., 2020). across various tasks. In image recognition, the de- scriptions that LLM responded to the query, e.g. ["What are useful features for distingui shi ng class name? have been used enhance accuracy compared using only single class label. Notably, in vision language models (VLM), these descriptions are adeptly integrated into prompts, which are then utilized categories for classification. However, even with well-pretraine LLM, there are limitations, including high variability in responses and low reliability of some descriptions. These limitations occur because: (i) the query inherently open-ended, leading to multiple plausible interpretations, (ii) the format of the query influences the responses, and (iii) inherent biases affect generated descriptions.
Formally, aside from applying descriptions, prompi learning has been studied as an efficient method to enhance generalization in pre-trained VL.M, such as CL.IP(Radford al., (02T), GL.IP(Li et al. 2022), and ALIGN data. To set the text prompt used in these models, the standard approach involves simply applying pre-defined template, e.g., "A photo of {class). ". However, trivial variations in this template, such as a or ", can have a profound impact on inference performance(Zhoo et al. 2022b). To mitigate this variation, the prompts can be defined learnable continuous vectors that can be optimized, so that each prompt can be trained to have optimal arrangements and semantics.
al. 2021), without the necessity of additional task-specific annotated
"qqual contribution. Correspondence to: sual ee. wuw.nnggaaaaaa com

---

Published as a conference paper at ICLR 2025
0: Four-legged domestic dog breed
1: Medium to large-sized dog broed
0:85
crearm-colored
8.80
0.799
C.794
6.708 0.785
3: Floppy ears
0.749
Similarity
5: Friendly expression
8:70
8: Often seen with tennis ball frisbee7:
0.62
7: May have darker for on face or ears
Figure 1: High variability and low reliability of GPT-based deseriptions: The example of descriptions obtained by asking GPT the question, "What are useful features for ingiiinnnsiiiii ng a Golden Retrieved seeer""""?" Some descriptions highlight highly distinc- tive visual features, while others convey ambiguous meanings and often begin with qualifiers such as "often', 'may", or 'maybe" that can reduce clarity. Moreover, is uncertain whether the last description accurately portrays the characteristics of Labrado Retriever. Comparing the text similarity between these descriptions and the class name "Labrado Retriever" reveals significant variability in reliability, There is also noticeable discrepancy between our manual assessment of the descriptions and the determination of useful features for classification based on their similarity.
With prompts no longer fixed to a single template, multiple prompts can be assigned to each class, empirically demonstrating superiority over single prompts. This raises the question: Which prompt holds relatively significant semantics? While there has been made effort to handle the distribution of existing learnable prompts, the method dealing with the importance of prompts has yet to be studied.
In this work, we present Description-free Multi-prompl Learning (DeMul) as way to directly distillate the LLM's pre-trained distribution without descriptions. Specifically, instead of directly inserting descriptions into prompts, our approach map learnable prompts into the LLM embedding space and distill them absorb the semantics.
We chose GPT-based embedding models as the LLM to distill, which are accessible through the APIs provided by OpenAI. The public API available for transferring GPT is divided into main BalesuBalus & Yan 2022): the Completion Endpoint which is text- bost-based conversational model, and the Embedding Endpoint which uses embeddings with more reliable performance. While existing description- bbased methods are limited to using the Completion Endpoint, we first employ the Embedding Endpoint, enabling description-frie
distillation. By leveraging this API, we can handle prompts embedding vectors instead of text, eliminating the need for pre-defined templates and allowing them be optimized as CoOp(Zhou et al. (2022b) explored. Additionally, since prompts are learnable vectors, the importance of the semantics they contain continuously changes during training. DeMul introduces prompt weighting to reflect this variation in importance.
To summarize contributions:
1. We propose description-free distillation approach that removes the process of extracting descriptions and instead prompts are mapped into the LLM embedding space, where they are optimized to capture meaningful semantics.
directly distills the pre-trained knowledge of LLM. The learnable
2. In multi-prompt setting, the semantics that each prompt learns and their corresponding importance dynamically change during training. We introduce prompt weighting adjust this importance, and our experiments demonstrate that this approach benefits the learning process.
3. We utilize CLIP, of the most extensively researched models for prompt learning in VLM, as our baseline. We compared its performance across a total of datasets. Ours demonstrates superior results in most datasets, surpassing the existing baselines using description- based methods or learnable prompt optimization methods.
2

---

Published as a conference paper at ICLR 2025
(
CLIP encoder
GPT embedding
00
F2
model
h(ea)
CN
gc1))
(M))
Learnable prompts
0
Learnable
e(s)
h((2)
g
g c
(vb (g(c2)
VLM embedding space
LLM embedding space
Here, C, denotes each class, g represents the CLIP text encoder, and h represents the GPT embedding model. The learning objective is to develop learnable prompts and a function şo that captures the semantics of GPTs. Initially, the learnable prompts are randomly generated but are then updated and trained minimize the angular difference with the GPT emhedding vectors of each class. Trained with a mapping loss, initially preserves the embedding orientation of the set of classes, but is subsequently trained maintain the directions of the class embeddings adjusted to the prompts.
Figure 2:
An overall framework of DeMul:
PRELIMINARIES AND BACKGROUND
In this section, provide concise overview of zero-shot Vision-Language Models (VLM) using hand-crafted prompts, specifically focusing on CLIP. We then describe CoOp, a few-shot method for prompt learning that automatically optimizes prompt templates. Our method applies various models and tasks that use text embeddings.
Contrastive Language-Image Pretraining (CLIP) CLIP trained on a web-scale dataset com- prising image-text pairs to learn aligned representations through contrastive learning. Specifically, it incorporates two modality-specifie encoders- an image encoder f()) and a text encoder g--chhhhcc are trained to ensure that the correct pairs of embeddings are closely matched in the joint embedding space. Consider an input (x. y). where x E RCx Hx and y E R The normalized feature vectors denoted
f=))/1] f for the image and wy where Po is the prompt generated for the class using a pre-defined template, "A photo of {class). The prediction probability of x; calculated as shown in Eq. 1 where is the temperature parameter of the softmax function.
((Po(y))/llee (P)(())) for the text y.
exp(2 (LyTTuyTTT Wy/T) ER exp(2 ww//T)
Context Optimization (CoOp) CoOp is pioneering method efficiently adapt the context of prompts for downstream visual recognition tasks. Instead of using pre-defined discrete tokens, CoOp employs N learnable continuous parameters {v1, u2, UN) } as the template be optimized, where each U/ vector has the same dimension as the word embeddings. The learnable vectors
V J are shared across all classes, and the generated prompts for each class are defined as f = p. (V, = [u>][e>] ... [e][[c], where prompting p. is a concatenation function of consecutive vectors and the class name. The learnable tokens optimized using few-shot samples by maximizing the matching score between the text and image embeddings.
3 METHOD
In this section, we present DeMul and its key components. We propose a method for extracting information from an LLM into learnable prompts without relying hand-crafted descriptions.
3

---

Published as a conference paper at ICLR 2025
Additionally, introduce weighted multi multi-promppl learning approach using importance sampling to perform few-shot recognition with a fixed number of prompts effectively.
3.1 DISTILLING LARGE LANGUAGE MODEL WITHOUT MANUAL DESCRIPTIONS
The key question in our proposed approach is how to distill directly the GPT text semantics without hand-crafted descriptions. Specifically, Can we infuse class speciisci information learned by GPT into prompl without explicit descriptions? To achieve this, instead of querying GPT extract features related to a class, we aim to train prompts to have maximized semantic correlations with class names within the GPT embedding space, (The overall workflow for better understanding is illustrated in Figure 2) GPT models were originally developed for text generation tasks, but they are also utilized for text similarity-relater tasks (e.g., text search, code search, sentence similarity, text classification). These models take text inputs and output -0me-dime vectors, having learned from large-scale language data to effectively measure similarities between various texts. While the exact training data, architecture, and other details are not publicly disclosed, this approach has demonstrated superior performance across many LLM-based embedding models. In our study, we employed the text www.eeebbdnngge -- model, which achieved a 64.6% accuracy in MTEB (uuenniggei ct al. 2022) eval.
Mapping prompts To leverage the semantic potential of the GPT embedding space for visual prompts, we developed a mechanism for aligning the CLIP embedding vectors with the GPT space. Since 5-layered MLPs are nonlinear and are not diffeomorphism in general, focus preserving the direction of the embedding vectors. The function yp maps from the CLIP embedding space into the GPT embedding space and de maps from the GPT embedding space back to the CLIP embedding space such that go o forms a conformal map on a set of points. Throughout this paper, we refer to this conformal map cyclic mapping.
Since the few-shot recognition task requires a small amount of training data, providing nice initial values can increase the learning stability of the model. Thus, both oo and ab are pre-traines on comprehensive dataset 'D Dname that contains common class names as well as class names that appear in the benchmark dataset. Dname consists class names from WordNet and 12 other datasets used in the section encompassing wide array of semantic contexts to establish robust initial mappings.
As the fine-tuning process of & progresses with ve frozen, the set of directions preserved by the cyclic mapping v ip changes from 'D name to a dataset 'D mapping of learnable prompts. To learn cyclic mapping, we propose the following loss function which measures how closely the cycled embeddings resemble the directions of the original embeddings from the dataset. The similarity loss is formulated as follows:
1 N
Σ deas (v(sp(())))
Lmapping
where t; represents prompt embedding in Dmapping. dees denotes the cosine similarity, and N is the number of data points in maapping.
Distillation in GPT embedding space In the multi-prompt setting, for each class c, a set of M multiple learnable prompt vectors is defined as T = (4 where each 4 is generated by applying the prompting p. to different learnable vector sets V, and the class c, denoted by t, p. (V, c). This approach allows for creating diverse array of prompts for each class, leveraging multiple vector configurations to capture various semantic nuances of the class. The effectiveness of these prompts is measured using distillation process in the GPT embedding space. Each prompt f is mapped to its GPT embedding by a function ip, resulting in set of transformed prompts ( =- These transformed prompts are then aligned with the corresponding GPT class embeddings l(c), optimized ensure the correct semantic correlation:
K
M log P(y((t))) [h(ci))
Lassinl =
Σ j=1
K
M
where P is the softmax function of Eq. facilitating the training process by aiming to maximize the probability that each prompt correctly aligns with its respective class in the GPT space.
4

---

Published as a conference paper at ICLR 2025
3.2 WEIGHTED MULTI-PROMPT LEARNING
In the context of visual recognition, the challenge is not only to capture the semantic richness of classes via text prompts but also ensure effective classification in the CLIP embedding space. In a multi-prompt setting, where each class G; associated with multiple prompts T; {44 )= the classification loss is initially defined as the average probability overall prompts for a given class:
j=1'
K
M
1
ces
log
P(y = cl2,
K
1=1 M
This approach, however, does not account for the varying importance each prompt within a class, different semanties may contribute differently to the recognition task.
To address this, we introduce prompt weighting mechanism that dynamically adjusts the importance of each prompt during training, recognizing that the number of relevant semanties the importance of each semantic can vary significantly between classes. This is particularly crucial because the optimal number of prompts effectively represent class is not only non-trivial determine heuristically but also varies across classes.
Each prompt bty within the class c; is assigned a learnable weight wiy, reflecting its relative importance. The classification loss for each class is then reformulated to incorporate these weights, providing weighted average probability that accounts for the differentiated contribution each prompt:
K
M
K
M
1 K
Less
log
Wij Ply = c,lr, (4))
wiy
(5)
M
1=1
j=1
==1
where À is regularization parameter that controls the trade-off between the classification loss and the LI penalty, The weights {wey} are normalized for each class, ensuring that Σ Wiy = 1. This weighted approach with LI regularization enhances model flexibility by allowing it to emphasize more informative prompts while diminishing the impact of less relevant ones. The addition of the L1 term encourages sparsity, promoting a scenario where fewer but more significant prompts are actively used, thereby optimizing the classification performance and computational efficiency processing visual tasks.
3.3 TRAINING
The overall training objective for the Weighted Multi-prompt Learning system combines the distilla- tion loss and the weighted classification loss into a total loss function. This total loss designed to optimize both the semantic alignment in the GPT embedding space and the classification accuracy in the CLIP embedding space. It is formulated as a weighted sum of the two losses:
Liotal = Lets + a
where a a hyperparameter that balances the contribution of the distillation loss and the classification loss. This parameter allows the model to prioritize between the alignment of the prompts with the GPT model's embeddings and the direct classification performance, depending on the specific requirements of the task or the dataset characteristics.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets We evaluate our approach over 11 datasets, including ImageNet(Deng al. 2009) and
publicly available image recognition datasets used in GalLoP(Laton et al. 2024): SUN397 (iao et al. 2010), Stanford Cars( et al. 2013), UCF101 Soomro (Sommo al. 2012). Caltech101 Li et al.
2017
(urber et al. 2019). FGVC Aircraft(Maji et al. 2013). Food (Bossard et al. 2014).
CiD/Ci et al.] 2014) Oxford Flowers(Nilshack & Zisserman 2008) and Oxford Pets Parkhi
5

---

Published as a conference paper at ICLR 2025
Algorithm Training process of DeMul
Require: Pre-trained VLM encoders of image f and text g
Require: LLM encoder
Require: The number of classes K, the number of prompts M, the number of epochs T, data
mini-batch size B, and prompt mini-batch size B'
Require: Training dataset Prain for target task and the set of class names {G)
1: Compute text embeddings {g(()));=1 and {
2: Randomly initialize the set of prompts 6p {V))
3: Initialize ype with pre-trained values, v frozen
4: for = 0 do
5: Sample mini-batches {1o5-) 36 ) B
in Drain and {v(7) B
J j=1
in P. respectively
b=1
6: Compute image embeddings Fb 10=-((1) [()
(lbbs B)
7: Compute prompt embeddings tap = ( (V (vln {())))
(1 (si<< K and 1 s'jsB)
8:
Compute LLM embeddings and eyclic embeddings by
(p(())) and ()(pa((e)))
(1 Sii K and 1 S j S B))
9: Compute the total loss
Croral (Op) = Cess (8p) + Edisun (9p)
(by the equation
6
Compute the cyclic mapping loss Lmmpping (0)
(by the equation
11:
Update parameters θ and & by using Lioaal and Emppinng respectively
12:
end for
et al. 2012). These datasets collectively represent a comprehensive benchmark, covering a diverse of vision tasks that include classification of general objects, scenes, fine-grained categories, or specialized tasks like recognizing textures, cars and satellite imagery. This extensive variety ensures robust validation of our approach across broad spectrum of visual recognition challenges. We follow the same train and test set splits as provided by CoOp(Zhou et al. (2022b) and GalLoP(Lafon al. 2024). using 1, 2, 4, and shots for training and full test sets for evaluating.
Baselines We compare the classification performance of DeMul with the zero-shot few-shot methods in VLM prompt learning, containing state-of-the-art approach. Our analysis includes zero- shot CLIPPRadford et al.) (02T)) foundational VL.M baseline, and extends to two predominant approaches: description based methods and continuous prompt optimization methods. Among the description-ber methods, we compare with multi-prompt aCLIP(Menon & Vondrick, 2002)). which incorporates descriptions extracted from GPT into the prompts, and (Rtth et al. 2023). which further explores prompt template diversity. These approaches typically utilize zero-shot strategy, focusing on efficient utilization of pre-defined descriptions without further optimization of the prompt template.
In contrast, the continuous prompt optimization methods, exemplified by CoOp(Zhou et al. 2022b KKattaK et al. 2023a), PromptSRR(Khaite et al.) (0236), and define prompts continuous learnable vectors and optimize them within the loss function to enhance the model's performance on classification. This comparative framework allows us to comprehensively assess the effectiveness of DeMul in leveraging both descriptive elements and dynamic prompt optimization to enhance classification accuracy.
et al. (0024),
Implementation details Following existing baselines, we adopt the publicly available pre-trained
CLIP model with the visual backbone of ViT-B/16(Alexey 2020) in all experiments. The number of context tokens N and the number of prompts M are set to 16 and 32, respectively. We optimize the prompts for 100 epochs with SGD optimizer and cosine decay learning rate scheduler, while the base learning rate is set to 0.01 on most datasets. The regularization weight is set at 0.05, and the loss balance parameter is established 0.5. Additionally, to avoid memory overhead, we introduce prompt sampling strategy used in ProDA(Lu et al.) (002).. For each iteration, we randomly selected the batch of prompts During each training iteration, we randomly sample batch of prompts, and

---

Published as a conference paper at ICLR 2025
Average
Imagefett
Scoree (%6)
Score
Score
Taye San shrtern
City
nomatac
Comtal
Number of labeled training samples es per class
Number of labaled training samples per class
Number labeled training samples per dlsss
StanfordCars
UCF101
Caltech101
Score (N1
der of
Toe anse
Tres den
lcaal Hoie
une moce
PremataC
PremettC
canaal suar)
Dakk hurd
01
tartal saut)
Number of labeled training samples per class
Nuriber labeled training samples per class
Number labeled training samples per class
FurnSAT
FGVC Arcraft
Food101
Score 183)
Prerter en
that anse hobs CuP
Score
Peeniaen a
Unee Price CUP
lnter mice CuP
Milke
Nalut
Kikin
Number labeled training casss
Numbee labeled training samotes
Number labeled training
DTD
Flowers102
Oofford Pets
10 2000
Scare
Fie E1
Prwnieel cae
Morkc
Marte
taller
3 labeled training samplles per
2 labelee training sumples per
lubeled sumplles
Figure 3: Comparison of few-shot learning accuracy on the 11 downstream datasets: Over- all, DeMul (indicated by red lines) outperforms CLIP as well as description-based methods (dCLII((Menon & Vondrick (022). WaffleCLIPPRoth et al. (023)) and continuous prompt op- timization methods (CoOp(Zhov (Zoo et al. (2022)). MaPLo(Khattak et al. 2023a), (rmmpsSR(/Khattee
ct al. 2023b), GalLoP(Lafon et al. 2024)).
proportionally increase the total number of learning epochs to ensure comprehensive learning. We do experiments on an V100 GPU for datasets with fewer than 100 classes, and utilize an A A100 GPU for larger datasets.
4.2 PERFORMANCE
Figure 3 summarizes the comparison of DeMul against baselines across 11 datasets and average (de- tailed scores in Appendix B. A comparative analysis was conducted on few-shot image classification
Top-1 accuracy across downstream datasets, with all pre-trained CLIP models maintained in frozen state. Our approach demonstrated significant improvements in most settings. On average compared to the CLIP baseline, our method achieved an 10.5% increase in 1-shot and 20.3% increase in
7

---

Published as a conference paper at ICLR 2025
Table Ablation study: An ablation study on the choice of two distinct losses, (Average Top-1 acc across 11 datasets)
#(training samples/classs
4
GalLoP
73.7
76.4
79.1
82.3
84.5
Ours w/o Lasaill
75.0
78.0
80.4
83.0
84.8
Ours w/o weighted
75.2
77.8
80.1
83.1
85.2
Ours
75.5
78.6
80.9
83.6
85.3
16-shot accuracy. Even when compared to GalLoP, which exhibits highest performance in learnable multi- poaltt optimization, our approach improved by 1.8% in 1-shot and 0.9% in 16-shot. also shows that DeMul significantly outperforms other zero-shot or few-shot prompt learning methods in most cases. The results suggest that directly training for classification utility, rather than embedding descriptions' semantics into prompts, yields more meaningful benefits.
Additionally, The two distinct loss functions of DeMul((disi) and prompt weighting) were evaluated separately in the ablation experiment (Table DT. While the performance differences were not substan- tial, both LLM distillation and prompt weighting consistently demonstrated improvements compared to GalLoP. It also highlights the effectiveness of DeMul when utilizing both methods together.
Table 2: Study for different LLM models: DeMul's experimental results in 16-shot when distilling semantic information from different LLM models.
FGVC Aireraft
Oxford Flowers
Stanford Cars
Caltech101
ImageNet
SUN397
UCF101
EuroSAT
Food101
Average
DTD
Method
text-embedding-adas ada---
74.7
76.3
88.2
86.3
95.0
90.0
57.8
86.1
75.3
95.2
82.5
lext-embeddiing---mme
75.5
77.1
89.0
86.9
95.7
91.7
58.8
87.3
76.1
96.3
83.4
(ext-embedding-3-leeee
76.1
77.9
89.9
87.7
96.9
92.5
59.9
87.8
77.0
98.0
84.4
Since DeMul is capable of utilizing various pre-trained LLM models, conducted few-shot classification experiment using differently trained LLM models. The embedding dimensions for the models text www.xddemeddiiee aog-ada--00 and rext-embedding ng- 3-small are 1536, whereas the embedding dimension for wext-eebeeddiae wwwettteemmeediig------aeeee both text -
is 3072. It has also been reported that and text text-embeddingg-- 3--angggee--eeeeeeeeeoeaa are the most recently released models and outperform text www.eeeing-gddaaaeeeeeeee---a--e
The result of the few-shot classi- fication performance in Table T reveals positive correlation between classification performance and the capability of the LLM models.
Mapping spaces In Fig. 4 we analyzed the effectiveness of the mapping function &o and its iecudo-inverse ei during the training process using UMAP for visualization. When examining the initialized 10 prompts in the CLIP space, we can observe that the 10 classes clearly form clusters, shown in Fig. (a). This clustering occurs because the names, have identical learnable vectors and are therefore positioned closely together. However, if i is not frozen and is fine-tuned suggested in DeMul, the visualization of prompts mapped the GPT space, as seen in Fig. (b), tends to lose its distinct patterns. Conversely, freezing & and fine-tuning oo with a consistent loss preserves the clustering tendency in the GPT space, as demonstrated in Fig. (c). Thus, our tine-tuning method allows for the preservation of CLIP space semantics while mapping to GPT space.
of prompts, labeled with class
Prompts and weights correlations During the training process with five prompts for each class, we examined the changes in prompt weights and the similarity between prompts and class names at five epochs 5). As the epochs progressed, the prompts increasingly aligned with the class names in terms of similarity, and each prompt exhibited a unique pattern of change. This indicates that each prompt learns different semantics, and the five prompts collectively form clusters, facilitating
8

---

Published as a conference paper at ICLR 2025
%
72
Figure The U-map visualization CLIP and GPT embedding spaces The U-map visualization based on the EuroSAT dataset, which consists of 10 classes, with each class represented by 10 prompts. Prompts belonging to the same class are marked with the same color. (a) Initialized prompts in CLIP space, (b) Mapped prompts in GPT space, finetuning &p without freezing vi, and (c) Mapped prompts in GPT space, finetuning &o with freezing eb.
1.00
L075
1.00
Lo02
0425
0.93
1poch
Tpech
Epoch
Epech-4
Sp5c1
0.25
6.90
Sanilatty
8.0
0.00
6pAth
Esech 2
Epech
Eooch
Epoch
Prompt
Prompt
Prompt
Prompt
Prempt
The changes were observed five epochs (0, 15. 30, 45, 60) during the training process. The training process was conducted using the Food101 dataset, which comprises 101 classes, with each class represented by five prompts. (Top) illustrates the variations in weights corresponding to each prompt, while (Bottom) shows the similarity between each prompt and its respective class names. Both provide average scores across all classes.
Figure 5: Variations of the weights of prompts:
effective classification. Additionally, an analysis of the corresponding prompt weights revealed that higher similarities generally correlate with higher weights, and vice versa. Despite the prompts possessing distinct semantics, those closer to the class names significantly influence classification, thus reflecting the importance of prompt weights in the model.
RELATED WORK
Vision-language models Vision-language models(VLM) which learn rich image-text correlations the joint embedding space, have shown promising performance in various tasks. VLM trained on web-scaled large datasets can be directly applied to downstream tasks without the necessity of heavy fine-tuning process. Given image-text pairs data (Schuhmann al.) 2022 202T). this approach first employs independent encoders that extract meaningful features from text and images, and trains them simultaneously using certain pre-training objectives. Pre-training objectives are mainly categorized
(Zhang et al.] 2023) into: (1) Generative-objectnn
(Chen et al. 2022 Li et al. 202 Ko & Gu 2022
Yu et al. 2022). which capture meaningful image/text features via learning masked image generating
models, (2) Aligneent-objectives (Li et al. 2022 Yao et al. 2022 2021 Zhou al. 2023), which
train to ensure the given text appropriately describe the given image, and (3) Contrastie-objectin
(Radforrd al. 2021 Jia al. 2021 Wu et al. 2021
Li et al. 202 Cui et al. 2022 Yao et al. 2021

---

Published as a conference paper at ICLR 2025
Yang et al., 2022), which are based contrastive learning methods. The representative model CLIP (Radford et al., (021) simultaneously trains image and text encoders using
by
pulling positive samples close and pushing negative samples faraway. The pre-trained CLIP targeting zero-shot prediction has also demonstrated superior performance in application tasks including classic
image recognition (Radford al. 2021 Zhou et al. (2022b), detection (Feng |
[022]] Maaz et al.
2022
Bangalath et al.) (022), or segementation (Ding et al. 2022; Lüddecke & Ecker 2022 Rao
al. 2022), etc. Our research aims enhance recent studies on VLM applications, specifically
focusing on facilitating the adaptation and deployment of such models within downstream datasets.
Prompt learning Prompt learning, or prompt engineering (Dosoviskkyy et al. 2020; Radford et al. 2019), has emerged recently in NLP as strategy for effectively transfer large language pre-trained models. It deliberately places mask tokens within sentence templates, then trains the model to predict the word to fill the blank. The model uses the rest of the template to fill it in, highlighting the importance of how the template is designed. Initially, these templates (or prompts) were manually (uronn (Brown al.) 2020 Petroni et al., 2019 Radford et al.. (201)). Then, to reduce heuristic efforts, advanced approaches automatically search for prompts arranged from discrete Shin al. 2020), or learn vector-based continuous prompts(Han
gggnneerng(Dosoviikkii
hand-crafted by humans
tokens(Jiang et al.] 2020
al.)
2022 Lester et al.
2021 Li & Liang, 202T; Zhong et al. (02T). In CoOp(Zhou et al.) 2022b).
continuous prompt leanning was applied to VLM for the first time. While CLIP uses dre-nef prompts, it has been demonstrated that automatically learning prompts that minimize classification loss can improve performance. As an extension of CoOp, there are methods aimed at enhancing robustness to domain shifts(Zhou et al. 2022a) or learning the distribution of multiple prompts (Lu
ctt al. 2022). Recently, there have been attempts(Menon & Vondrick 2022 Roth et al. 2023 Pratt al. 2023 Hu et al. 2023) to utilize Large Language Models(LLM), e.g., GPT-3(Brown et al. 2020),
to embed meaningful class-wise context information into prompts. By querying an LLM tre-ted on web-scale datasets, features helpful for distinguishing each class, known descriptions, can
be extracted. These descriptions can be directly inserted into prompts(Menon & Vondrick] [2022). or knowledge distillation can be used transfer this richer information to continuous vectors in prompt learning(Hu et al., 2023). Building on these concepts, methods like GalLoo((Lafon et al., 2024). (hatttka et al.) 2023a), and PromptSRCKhatta et al. (2023b) further explore the potentials of prompt learning by utilizing multiple prompts tailored for specific tasks or layers within a model. GalLoP, specifically, aims to harness both global and local prompt strategies to optimize the interaction between language and visual elements, significantly enhancing the model's is adaptability across diverse scenarios and datasets. MaPLe integrates prompts across several layers of both textual and visual encoders to facilitate deeper integration of language and visual cues. PromptSRC enhances this approach by incorporating several regularization losses which improve not only accuracy but also robustness, allowing for better adaptation to various domain shifts and conditions. These methods underscore the growing sophistication in prompt engincering aimed at maximizing the utility of pre-trained models across increasingly complex scenarios.
6 CONCLUSION AND DISCUSSIONS
Our study introduces a novel description-free multi-prompt learning(DeMull) approach, leveraging direct distillation of knowledge from Large Language Models (LLM) into prompts without extracting descriptive text for vision-language models. This method effectively preserves the semantic richness of the prompts while allowing the continuous prompt optimization for few-shot classification tasks. Using pre-trained CLIP, empirical results across 11 downstream datasets confirm that our approach not only simplifies the prompt learning process but also consistently outperforms traditional methods that rely on pre-defined templates or descriptions. This indicates a significant step forward in utilizing pre-trained models for enhancing image classification tasks.
However, there are still some limitations to address. While we have mitigated the high variability of description- methods, it still depends on the distribution of training images, which can lead to variations in performance due to its few-shot learning framework. Additionally, to maintain memory- efficient training, we have kept the learnable vectors of prompts identical across all classes, as an extension of existing works. This approach can hinder the development of class-specifir prompts. For future work, we aim to explore memory-efficienn methods that can also robustly handle unseen distributions and still capture class-specific prompts.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Dosovitskiy Alexey. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020.
Salvador V Balkus and Donghui Yan. Improving short text classification with augmented data using gpt-3. Natural Language Engineering, pp. 1-30, 2022.
Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open- vocabulary detection. Advances in Neural Information Processing Systems, 35:33781-33794, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 fond-ig discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. Springer, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06799 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, Osamu Yoshie, and Yubo Chen. Contrastive vision-language pre-training with limited resources. In European Conference on Computer Vision, pp. 236-253. Springer, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11583-11592, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open- ppe--vocabulary detection using uncurated images. In European Conference on Computer Vision, pp. 701-717. Springer, 2022.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. Al Open, 3:182-192, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected 'Toicc in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Hongyu Hu, Tiancheng Lin, Jie Wang, Zhenbang Sun, and Yi Xu. Context-aware prompt tuning for vision-language model with dual-alignment. arXiv preprint axXiv22399.44558 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- languaee representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2025
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How know what language models know? Transactions of the Association for Computational Linguistics, 8:423 438, 2020.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Hsuan Yang. and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15200, 2023b.
Byungsoo Ko and Geonmo Gu. Large-scale bilingual language-image contrastive learning. arXiv preprint arXiv:2203. 144634 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Marc Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, and Nicolas Thome. Gallop: Learning global and local prompts for vision-language models. arXiv preprint arXiv:2407.01400, 2024.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint 3Xxx:200400869: 2021.
Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4183-4192, 2017.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jeng-Neng Hwang, al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10965-10975, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001990 2021.
Yangguang Li, Feng Liang. Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208. 2021.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5206-5215, 2022.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7086-7096, 2022.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Class-agnostic object detection with multi-modal transformer. In European Conference on Computer Vision, pp. 512-531. Springer, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 100. 2013.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arX1v:2210.07183, 2022.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.073333 2022. doi: 1048500/RXX1221007336 URL
https://erxivs. org 2200.
12

---

Published as a conference paper at ICLR 2025
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In 2008 Sixth Indian conference on computer vision. graphics & image processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.00090000000 0966,,0006, 2019.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15691-15701, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
18082-18091, 2022.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. arXiv preprint arXiv:2306.07288 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 25294, 2022.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.1 15980, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph Gonzalez. Data efficient language-superrised zero-shot recognition with optimal transport distillation. arXiv preprint 1rXiv:2/12..04455 2021.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 1485-3492 IEEE, 2010.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning image-text-label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19163-19173. 2022.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive -iguagge-mmage pre-training arXiv preprint ar 11 2021.
13

---

Published as a conference paper at ICLR 2025
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu, Detelip: Dictionary-enricnn visual-concept paralleled pre-traimn nee-traiingg for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9113 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Visiun-language models for vision tasks: survey, arXiv preprint arXiv:2304.0068: 2023.
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. arXiv preprint arXiv:2104.05240, 2021.
Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastin learning meets language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11028-11038, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 06816-1825,, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2338, 2022b.
14

---

Published as a conference paper at ICLR 2025
A EXAMPLE OF DESCRIPTIONS
Table shows the examples of class descriptions extracted using the GPT-3 API, previously employed in description-based methods. Consistent with existing works, employed the g 11.-5-turo-inee ieto--nnsruuce ruct API for extracting examples. The responses from the API vary significantly with each query, both in content and quantity. Typically, the extracted descriptions accurately depict the visual features of the class, but they often include ambiguous sentences that begin with terms like "may" or "usually", and sometimes list characteristics that are difficult to verify visually (e.g., fried in pan). Additionally, biases may result in the extraction of descriptions that are irrelevant or incorrect with respect to the class.
Table The examples descriptions: Examples of descriptions for some classes.
Examples of descriptions
Class name: Cheese cake
Class name: French toast
1. Creamy white or light brown color
1. Bread that has been soaked in egg and milk
2. Creamy, smooth texture
3. A plate or serving dish the photo
Often served with butter, syrup, or fruit
4. Various toppings such as fruit or chocolate
3. Fried in a pan
5. May have decorative pattern writing
4. Sweet and savory taste
on top
5. Typically breakfast food
Class name: Helicopter
Class name: Chair
1. Flying aircraft
1. Furniture piece
2. Metal body with rotors top
2. Wooden, metal, plastic material
3. Usually painted in bright colors
Four legs or
support
4. Cockpit for pilot and passengers
4. Sometimes has armrests
5. Various designs and styles
Class name: Rose
Class name: Bath towel
1. Flowering plant
1. A piece of fabric
2. Colors of red, pink, or yellow
2. Usually white or brightly colored
3. Thomy stems and leaves
3. Used for drying oneself after bath
4. Sweet, floral scent
shower
5. Grows clusters on a green stem
4. May hung on towel or hook
5. May be folded or rolled up
Class name: Baby
Class name: Truck
1. Small human being
1. Large, heavy motor vehicle
2. Soft, delicate skin
Typically has four wheels
3. Round head with large eye, small nose
3. Usually has long, rectangular bed for
4. Small, chubby limbs
carrying cargo
5. Often wearing diapers clothing
4. Can come in variety of colors and de-
6. Can make facial expressions and vocal-
signs
izations
5. May have a logo or company name dis- played on the side
15

---

Published as a conference paper at ICLR 2025
B MAIN RESULTS DETAIL
Table 4 presents presents the detailed results DeMul and various baseline methods across 11 datasets, All experiments were conducted using pre-trained CLIP model with ViT-B/16 backbone.
Table 4: Detailed performance %) of DeMul: The detail top-1 accuracy across 11 diverse downstream datasets, along with the average scores for each. '# Shot' represents the number of training samples per class used in few-shot learning.
FGVC Aireraft
Oxford Flowers
Stanford Cars
Caltech101
Oxford Pess
ImageNel
SUN399
UCF101
EunSAT
Food101
Average
DTD
Method
Shot
Zero- Scro-Shottte CLIP
68.4
62.3
65.1
64.8
92.3
48.8
24.8
84.7
44.6
70.4
88.4
65.0
Zero-Shot dCLIP
70.1
64.2
66.9
65.9
92.7
51,2
26.4
85.7
47.8
71.1
89.5
66.5
Zero-Shot WafleCLLIP
0
71.4
63.9
67.3
66.3
93.0
53.3
25.9
86.5
48.0
72.4
89.9
67.1
32.1
35.7
49.2
19.6
34.6
44.1
45.8
2
44.9
53.7
50.3
65.8
89.0
62.0
26.4
61.5
40.8
85.1
58.4
58.0
Linear Probe CLIP
4
54.9
63.0
63.4
73.3
92.1
77.1
32.3
73.2
55.7
92.0
71.2
68.0
8
62.2
69.1
73.7
79.3
93.4
84.4
39.4
79.8
63.5
96. 1
78.4
74.5
80.4
87.2
66.3
66.8
67.4
71.2
79.9
49.2
19.6
71.2
34.6
69.7
44.1
58.2
2
67.1
66.5
70.5
73.4
89.0
62.0
26.4
73.4
40.8
85.1
58.4
64.8
CoOp
68.7
69.9
74.5
77.1
92.1
77.1
32.3
77.1
55.7
92.0
71.2
71.6
79.3
63.5
78.4
76.1
16
71.9
74.7
83.1
82.2
95.4
87.2
45.4
82.2
69.9
97.4
85.3
79.5
1
62.7
64.8
66.7
71.8
92.6
71.8
26.7
80.5
52.1
83.3
89.1
69.3
65.1
67.1
71.6
74.6
94.0
78.3
30.9
81.5
55.5
88.9
90.9
72.6
MaPLe
67.7
75.3
78.5
84.5
34.9
81.8
75.8
8
70.3
73.2
79.5
81.4
95.2
87.7
42.0
83.6
66.5
95.8
92.6
78.9
16
72.3
75.5
83.6
85.0
96.0
92.3
48.4
85.3
71.3
97.0
92.8
81.8
69.4
74.8
93.7
73.1
27.7
84.9
56.2
92.0
72.3
73.4
78.5
79.4
31.7
85.7
60.0
92.5
75.3
PromptSRC
4 71.1
74.0
77.1
81.6
95.3
86.3
37.5
86.1
65.5
93.9
93.4
78.3
8
72.3
75.7
81.0
84.3
95.7
88.8
43.3
86.9
69.9
96.3
93.5
80.7
73.2
77.2
83.8
96.1
92.4
47.1
87.5
72.7
97.6
93.7
82.9
70.8
68.3
71.0
74.5
94.7
79.5
32.8
85.8
56.0
84.9
92.2
73.7
2 71.5
71.6
76.1
78.2
95.1
78.6
36.2
86.5
61.3
92.5
93.2
76.4
GalLoP
4
72.2
73.8
78.3
81.9
95.5
82.7
44.1
86.9
66.9
95.2
93.4
79.2
73.8
75.7
84.9
85.0
96.0
86.9
72.4
93.4
82.3
75.1
89.2
96.7
90.1
58.3
87.9
75.5
94.1
84.4
1
72.3
68.5
74.7
76.9
95.1
80.7
34.3
86.1
62.1
87.2
93.0
75.5
2
72.6
73.1
78.5
80.0
95.4
83.1
40.2
86.4
67.9
93.4
93.6
78.6
DeMul (Ours)
73.1
81.3
87.4
70.3
94.0
80.9
74.3
76.4
87.7
85.7
96.4
89.7
54.3
87.5
75.1
97.9
94.3
83.6
16
76.1
77.9
89.9
87.7
96.9
92.5
59.9
87.8
77.0
98.0
94.9
85,3
16


Reviews:
[{"review_id": "K7NGwVsgCD", "reviewer": "Reviewer_RgBn", "summary": "This paper explores using large language models (LLMs) to enhance prompt learning for vision-language models. It proposes a description-free method to distill knowledge from LLMs into learnable prompts by directly aligning their embeddings. Additionally, a multi-prompt strategy is examined to improve performance. Experiments on few-shot learning benchmarks and ablation studies demonstrate the effectiveness of the proposed method.", "questions": "In line 7 of Algorithm 1, why do the text embeddings $g(c_i)$ from the output of the CLIP text encoder go through the encoder again?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-crafted with clear expressions, making it easy to understand.\n\n2. The description-free distillation approach for training learnable prompts is interesting, novel, and effective. It is also more cost-effective than description-needed methods, which require multiple queries to models like GPT for comprehensive descriptions.\n\n3. The study on diverse semantics and the importance of multiple learnable prompts is well-motivated.\n\n4. The extensive few-shot learning experiments and comprehensive ablation study effectively demonstrate the proposed methods and each module.", "weaknesses": "1. Although the paper demonstrates superior few-shot performance, the notable aspect is the open-vocabulary ability of CLIP. Since 2023, literature has focused on improving this aspect. Additional experiments on this topic are suggested.\n\n2. The authors should discuss whether the proposed distillation method works when ground-truth training data is missing. Solely text-side fine-tuning may lead to misalignment with the visual side. If it doesn't work, this limitation should be addressed.\n\n3. The computational costs, including time and memory during training and inference, should be compared to existing methods. This is especially important because multiple prompts can introduce substantial costs, particularly when dealing with a large number of classes, such as in ImageNet-1K.\n\n4. The symbol $t$ is defined both as prompt embeddings (L198) and learnable prompt vectors (L202); these should be differentiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQaBU9CXWw", "reviewer": "Reviewer_eZLz", "summary": "The paper proposes to learn richer prompts for Vision Language Models (VLMs) by mapping them to the Embedding Endpoint of GPTs. They aim to capture meaningful semantics by directly distilling the pre-trained knowledge of GPT without the use of explicit description queries. The improvement in the quality of prompts is achieved by minimizing the distillation loss between the prompt embedding space and the LLM embedding space to better capture the class orientations. The authors have also used a multi-prompt setting to assess the best semantics from a weighted prompt pool. The authors use a multi-prompt setting with prompt weighting. The approach is shown to work well in few-shot image classification scenario.", "questions": "The weaknesses are already given in the question form. In addition, here are a few additional questions.\n1. How is the value of $\\alpha$ chosen? Is there any sensitivity analysis on $\\alpha$?\n2. Are the experiments run for different random splits of training data? Then are the reported values the average over these?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper looks into an interesting direction of enriching prompts using cyclic distillation from LLM embedding spaces. It mitigates the explicit description queries for the class-names of a given dataset.\n2.\tThe description-free approach with multi-prompt weighting seems to be effective in few-shot image classification.\n3.\tThe paper is well organized and easy to follow.", "weaknesses": "1.\tIt seems that the pretraining has to be done on a dataset which covers all the category names in the evaluation. This would significantly limit the generalizability of the method. Also is class-names a good attribute for mapping the LLM embeddings?\n2.\tAs mapping functions need to be pretrained, what is the computational cost incurred? Also, it seems that the comprehensive dataset required a lot of samples from a long list of class-names to cover all datasets. Such an approach is prone to noise and biases. The authors should clarify the function pretraining in detail and make their approach more generalizable.\n3.\tConsidering Figure 3, the performance of DeMul is negligible in 6 out of 11 datasets compared to other approaches such as GalLoP. This significantly diminishes the efficacy of the approach.\n4.\tThe major weakness of the paper is very limited evaluation. The authors have limited their approach to only few-shot image classification while all the recent relevant works [CoCoOp (Zhou et al., 2022a), GalLoP (Lafon et al., 2024), MaPLe (Khattak et al., 2023a)] show their efficacy on 1. Domain Generalization 2. Base-to-New Generalization and 3. Cross-Dataset Transfer in addition to few-shot downstream image classification. Without such comparison it is hard to judge the goodness of the proposed approach beyond the single task of few-shot downstream image classification.\n5.\tDistilling LLM text to learnable prompts and prompt weighting separately have been explored in some of the recent works. Such as [a] or [b]. The author should discuss these references in the related work section and explicitly explain how their work is distinct from these similar studies.\n\n[a] PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR 2024\n\n[b] A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models, ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTLW11bszB", "reviewer": "Reviewer_3ZUA", "summary": "This paper proposes a novel approach called DeMul for few-shot image recognition. The method leverages pre-trained VLMs and bypasses the need for extracting textual descriptions from LLMs by directly distilling knowledge from the LLM into learnable prompts. This description-free approach effectively addresses the variability and potential unreliability of LLM-generated descriptions in previous methods.", "questions": "1、Provide a more detailed analysis of the mapping function's properties and explore alternative mapping strategies.\n2、Investigate the performance variations under different training data splits and explore data augmentation techniques to improve robustness.\n3、Evaluate the memory savings from shared learnable vectors and explore alternative approaches to balance memory efficiency and representational power.\n4、Attempt to provide some theoretical analysis to explain the observed performance gains.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The description-free distillation is a novel concept that cleverly addresses a key limitation of existing LLM-enhanced prompt learning methods. The weighted multi-prompt learning strategy further enhances the originality of the approach.\n\nQuality: The paper presents compelling empirical results across 11 diverse datasets, with DeMul consistently outperforming existing zero-shot and few-shot methods, including state-of-the-art techniques like GalLoP. The comprehensive experimental setup and ablation studies further strengthen the quality of the evaluation.\n\nClarity: The paper is well-organized and easy to understand. The motivation, methodology, and experimental results are presented clearly and concisely. The figures and tables are informative and effectively illustrate the key concepts and findings.\n\nSignificance: DeMul simplifies the prompt engineering process and improves the reliability of VLM-based image recognition by eliminating the need for textual descriptions. The significant performance gains in few-shot learning scenarios demonstrate its practical value. Furthermore, its core ideas could potentially be applied to other VLM tasks, broadening its impact.", "weaknesses": "Limited analysis of the mapping function: While the paper introduces the concept of a mapping function, the exploration of its properties remains superficial, lacking quantitative analysis to assess its effectiveness.\n\nDependence on training data distribution: The paper acknowledges the potential sensitivity of DeMul to the training data distribution but lacks an in-depth investigation of this sensitivity.\n\nShared learnable vectors: While improving memory efficiency, shared learnable vectors might limit the model's ability to capture class-specific prompt information.\n\nLack of theoretical support: The paper primarily focuses on empirical results and lacks a strong theoretical foundation to explain the effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CYbPMAnAZX", "reviewer": "Reviewer_jo4m", "summary": "This paper proposes Description-free Multi-prompt Learning (DeMul), a method that enhances image recognition tasks by distilling knowledge directly from LLMs without needing descriptive text prompts. This method enables flexible prompt learning by mapping learnable prompts into the LLM embedding space and using weighted multi-prompt learning to adjust prompt importance dynamically. Experimental results across 11 datasets show that DeMul outperforms conventional description-based methods and other prompt optimization approaches.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Well-structured and clearly explains the limitations of existing description-based prompt methods.\n\n(2) Extensive experiments across 11 datasets demonstrate the robustness and effectiveness of the proposed method.\n\n(3) While the novelty in the method may be incremental, applying weighted multi-prompt learning without description reliance addresses the practical limitations of prior work.", "weaknesses": "1. Figure 1 highlights the variability in GPT-based descriptions, but this diversity can actually be beneficial in image classification. The goal of descriptions is to enrich the semantic variety within each class, helping the model handle diverse images within the same category, rather than strictly aligning with the class name. By dismissing this variability, the proposed approach might miss out on the robustness provided by diverse descriptions.\n\n2. Mapping prompt descriptions to the GPT embedding space may reduce interpretability, as this space is abstract and lacks human-interpretable structure. The optimizations in this space might rely more on implicit semantic relationships rather than explicit, understandable features, which could obscure the interpretability of the learned representations.\n\n3.The paper does not reference or explain Algorithm 1: Training process of DeMul, which leaves readers without guidance on understanding the step-by-step training procedure of the proposed method.\n\n4. The description of Equation (4) suggests averaging classification probabilities across prompts, but the actual implementation seems to involve averaging the text embeddings of prompts instead. \n\n5.T he Preliminaries and Background section lacks appropriate citations, which limits the clarity.               \n\n6. The paper specifies fixed values for the regularization weight  and the loss balance parameter, but does not provide ablation studies to justify these choices.\n\n7. The Related Work section lacks a clear articulation of the limitations in existing methods and how the proposed approach addresses these gaps.\n\n8.The paper claims that the weights $\\{w_{ij}\\}$ for each class are normalized, ensuring that $\\(\\sum_{j=1}^{M} w_{ij} = 1\\)$. However, in Figure 5, the sum of the weights for the five prompts appears to exceed 1, which contradicts this claim.\n\n9. In the ablation study presented in Table 1, the difference in accuracy between \"Ours w/o weighted\" (85.2) and \"Ours\" (85.3) in the 16-shot setting is minimal", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lsPZoSpTaP", "reviewer": "Reviewer_oNaD", "summary": "This paper introduces Description-free Multi-prompt Learning (DeMul), a new method for prompt learning in vision-language models that eliminates the need for descriptive text by directly distilling knowledge from large language models (LLMs) like GPT. The approach optimizes prompts in the LLM embedding space and incorporates weighted multi-prompt learning, allowing prompts to capture diverse semantics and improve few-shot classification performance. Experimental results across 11 datasets show that DeMul outperforms traditional description-based methods, simplifying prompt learning while preserving rich semantics.", "questions": "No questions or suggestions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed method does not require text descriptions or specific prompt design, while still preserving rich semantic expression.  \n2.The proposed Weighted Multi-Prompt Learning effectively captures the relative importance of different prompts, enhancing model adaptability and performance.", "weaknesses": "1.Figure 1 highlights the variability in GPT-based descriptions, but this diversity can actually be beneficial in image classification. The goal of descriptions is to enrich the semantic variety within each class, helping the model handle diverse images within the same category, rather than strictly aligning with the class name. By dismissing this variability, the proposed approach might miss out on the robustness provided by diverse descriptions.\n\n2.Mapping prompt descriptions to the GPT embedding space may reduce interpretability, as this space is abstract and lacks human-interpretable structure. The optimizations in this space might rely more on implicit semantic relationships rather than explicit, understandable features, which could obscure the interpretability of the learned representations.\n\n3.The paper does not reference or explain Algorithm 1: Training process of DeMul, which leaves readers without guidance on understanding the step-by-step training procedure of the proposed method.\n\n4.The description of Equation (4) suggests averaging classification probabilities across prompts, but the actual implementation seems to involve averaging the text embeddings of prompts instead. \n\n5.The Preliminaries and Background section lacks appropriate citations, which limits the clarity.               \n\n6.The paper specifies fixed values for the regularization weight  and the loss balance parameter, but does not provide ablation studies to justify these choices.\n\n7.The Related Work section lacks a clear articulation of the limitations in existing methods and how the proposed approach addresses these gaps.\n\n8.The paper claims that the weights $\\{w_{ij}\\}$ for each class are normalized, ensuring that $\\(\\sum_{j=1}^{M} w_{ij} = 1\\)$. However, in Figure 5, the sum of the weights for the five prompts appears to exceed 1, which contradicts this claim.\n\n9.In the ablation study presented in Table 1, the difference in accuracy between \"Ours w/o weighted\" (85.2) and \"Ours\" (85.3) in the 16-shot setting is minimal, with only a 0.1% improvement when using the proposed weighting mechanism. This marginal difference raises questions about the practical impact of the weighted approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)