Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation

Authors: Sua Lee, Kyubum Shin, Jung Ho Park
Primary area: foundation or frontier models, including LLMs
Keywords: Prompt learning, Vision-language models, Large language models, Few-shot image recognition

Abstract:
Recent advances in pre-trained Vision Language Models (VLM) have shown promising potential for effectively adapting to downstream tasks through _prompt learning_, without the need for additional annotated paired datasets.
To supplement the text information in VLM trained on correlations with vision data, new approaches leveraging Large Language Models (LLM) in prompts have been proposed, enhancing robustness to unseen and diverse data.
Existing methods typically extract text-based responses (i.e., _descriptions_) from LLM to incorporate into prompts; however, this approach suffers from high variability and low reliability.
In this work, we propose **De**scription-free **Mul**ti-prompt Learning(**DeMul**), a novel method that eliminates the process of extracting descriptions and instead directly distills knowledge from LLM into prompts.
By adopting a description-free approach, prompts can encapsulate richer semantics while still being represented as continuous vectors for optimization, thereby eliminating the need for discrete pre-defined templates.
Additionally, in a multi-prompt setting, we empirically demonstrate the potential of prompt weighting in reflecting the importance of different prompts during training.
Experimental results show that our approach achieves superior performance across 11 recognition datasets.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730395690407

thread_type:
official_review

<thread_object>
creator: reviewer_onad

<content>
summary:
This paper introduces Description-free Multi-prompt Learning (DeMul), a new method for prompt learning in vision-language models that eliminates the need for descriptive text by directly distilling knowledge from large language models (LLMs) like GPT. The approach optimizes prompts in the LLM embedding space and incorporates weighted multi-prompt learning, allowing prompts to capture diverse semantics and improve few-shot classification performance. Experimental results across 11 datasets show that DeMul outperforms traditional description-based methods, simplifying prompt learning while preserving rich semantics.

strengths:
1.The proposed method does not require text descriptions or specific prompt design, while still preserving rich semantic expression.  
2.The proposed Weighted Multi-Prompt Learning effectively captures the relative importance of different prompts, enhancing model adaptability and performance.

weaknesses:
1.Figure 1 highlights the variability in GPT-based descriptions, but this diversity can actually be beneficial in image classification. The goal of descriptions is to enrich the semantic variety within each class, helping the model handle diverse images within the same category, rather than strictly aligning with the class name. By dismissing this variability, the proposed approach might miss out on the robustness provided by diverse descriptions.

2.Mapping prompt descriptions to the GPT embedding space may reduce interpretability, as this space is abstract and lacks human-interpretable structure. The optimizations in this space might rely more on implicit semantic relationships rather than explicit, understandable features, which could obscure the interpretability of the learned representations.

3.The paper does not reference or explain Algorithm 1: Training process of DeMul, which leaves readers without guidance on understanding the step-by-step training procedure of the proposed method.

4.The description of Equation (4) suggests averaging classification probabilities across prompts, but the actual implementation seems to involve averaging the text embeddings of prompts instead. 

5.The Preliminaries and Background section lacks appropriate citations, which limits the clarity.               

6.The paper specifies fixed values for the regularization weight  and the loss balance parameter, but does not provide ablation studies to justify these choices.

7.The Related Work section lacks a clear articulation of the limitations in existing methods and how the proposed approach addresses these gaps.

8.The paper claims that the weights $\{w_{ij}\}$ for each class are normalized, ensuring that $\(\sum_{j=1}^{M} w_{ij} = 1\)$. However, in Figure 5, the sum of the weights for the five prompts appears to exceed 1, which contradicts this claim.

9.In the ablation study presented in Table 1, the difference in accuracy between "Ours w/o weighted" (85.2) and "Ours" (85.3) in the 16-shot setting is minimal, with only a 0.1% improvement when using the proposed weighting mechanism. This marginal difference raises questions about the practical impact of the weighted approach.

questions:
No questions or suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730410418513

thread_type:
official_review

<thread_object>
creator: reviewer_jo4m

<content>
summary:
This paper proposes Description-free Multi-prompt Learning (DeMul), a method that enhances image recognition tasks by distilling knowledge directly from LLMs without needing descriptive text prompts. This method enables flexible prompt learning by mapping learnable prompts into the LLM embedding space and using weighted multi-prompt learning to adjust prompt importance dynamically. Experimental results across 11 datasets show that DeMul outperforms conventional description-based methods and other prompt optimization approaches.

strengths:
(1) Well-structured and clearly explains the limitations of existing description-based prompt methods.

(2) Extensive experiments across 11 datasets demonstrate the robustness and effectiveness of the proposed method.

(3) While the novelty in the method may be incremental, applying weighted multi-prompt learning without description reliance addresses the practical limitations of prior work.

weaknesses:
1. Figure 1 highlights the variability in GPT-based descriptions, but this diversity can actually be beneficial in image classification. The goal of descriptions is to enrich the semantic variety within each class, helping the model handle diverse images within the same category, rather than strictly aligning with the class name. By dismissing this variability, the proposed approach might miss out on the robustness provided by diverse descriptions.

2. Mapping prompt descriptions to the GPT embedding space may reduce interpretability, as this space is abstract and lacks human-interpretable structure. The optimizations in this space might rely more on implicit semantic relationships rather than explicit, understandable features, which could obscure the interpretability of the learned representations.

3.The paper does not reference or explain Algorithm 1: Training process of DeMul, which leaves readers without guidance on understanding the step-by-step training procedure of the proposed method.

4. The description of Equation (4) suggests averaging classification probabilities across prompts, but the actual implementation seems to involve averaging the text embeddings of prompts instead. 

5.T he Preliminaries and Background section lacks appropriate citations, which limits the clarity.               

6. The paper specifies fixed values for the regularization weight  and the loss balance parameter, but does not provide ablation studies to justify these choices.

7. The Related Work section lacks a clear articulation of the limitations in existing methods and how the proposed approach addresses these gaps.

8.The paper claims that the weights $\{w_{ij}\}$ for each class are normalized, ensuring that $\(\sum_{j=1}^{M} w_{ij} = 1\)$. However, in Figure 5, the sum of the weights for the five prompts appears to exceed 1, which contradicts this claim.

9. In the ablation study presented in Table 1, the difference in accuracy between "Ours w/o weighted" (85.2) and "Ours" (85.3) in the 16-shot setting is minimal

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567850103

thread_type:
official_review

<thread_object>
creator: reviewer_3zua

<content>
summary:
This paper proposes a novel approach called DeMul for few-shot image recognition. The method leverages pre-trained VLMs and bypasses the need for extracting textual descriptions from LLMs by directly distilling knowledge from the LLM into learnable prompts. This description-free approach effectively addresses the variability and potential unreliability of LLM-generated descriptions in previous methods.

strengths:
Originality: The description-free distillation is a novel concept that cleverly addresses a key limitation of existing LLM-enhanced prompt learning methods. The weighted multi-prompt learning strategy further enhances the originality of the approach.

Quality: The paper presents compelling empirical results across 11 diverse datasets, with DeMul consistently outperforming existing zero-shot and few-shot methods, including state-of-the-art techniques like GalLoP. The comprehensive experimental setup and ablation studies further strengthen the quality of the evaluation.

Clarity: The paper is well-organized and easy to understand. The motivation, methodology, and experimental results are presented clearly and concisely. The figures and tables are informative and effectively illustrate the key concepts and findings.

Significance: DeMul simplifies the prompt engineering process and improves the reliability of VLM-based image recognition by eliminating the need for textual descriptions. The significant performance gains in few-shot learning scenarios demonstrate its practical value. Furthermore, its core ideas could potentially be applied to other VLM tasks, broadening its impact.

weaknesses:
Limited analysis of the mapping function: While the paper introduces the concept of a mapping function, the exploration of its properties remains superficial, lacking quantitative analysis to assess its effectiveness.

Dependence on training data distribution: The paper acknowledges the potential sensitivity of DeMul to the training data distribution but lacks an in-depth investigation of this sensitivity.

Shared learnable vectors: While improving memory efficiency, shared learnable vectors might limit the model's ability to capture class-specific prompt information.

Lack of theoretical support: The paper primarily focuses on empirical results and lacks a strong theoretical foundation to explain the effectiveness of the method.

questions:
1、Provide a more detailed analysis of the mapping function's properties and explore alternative mapping strategies.
2、Investigate the performance variations under different training data splits and explore data augmentation techniques to improve robustness.
3、Evaluate the memory savings from shared learnable vectors and explore alternative approaches to balance memory efficiency and representational power.
4、Attempt to provide some theoretical analysis to explain the observed performance gains.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660422472

thread_type:
official_review

<thread_object>
creator: reviewer_ezlz

<content>
summary:
The paper proposes to learn richer prompts for Vision Language Models (VLMs) by mapping them to the Embedding Endpoint of GPTs. They aim to capture meaningful semantics by directly distilling the pre-trained knowledge of GPT without the use of explicit description queries. The improvement in the quality of prompts is achieved by minimizing the distillation loss between the prompt embedding space and the LLM embedding space to better capture the class orientations. The authors have also used a multi-prompt setting to assess the best semantics from a weighted prompt pool. The authors use a multi-prompt setting with prompt weighting. The approach is shown to work well in few-shot image classification scenario.

strengths:
1.	The paper looks into an interesting direction of enriching prompts using cyclic distillation from LLM embedding spaces. It mitigates the explicit description queries for the class-names of a given dataset.
2.	The description-free approach with multi-prompt weighting seems to be effective in few-shot image classification.
3.	The paper is well organized and easy to follow.

weaknesses:
1.	It seems that the pretraining has to be done on a dataset which covers all the category names in the evaluation. This would significantly limit the generalizability of the method. Also is class-names a good attribute for mapping the LLM embeddings?
2.	As mapping functions need to be pretrained, what is the computational cost incurred? Also, it seems that the comprehensive dataset required a lot of samples from a long list of class-names to cover all datasets. Such an approach is prone to noise and biases. The authors should clarify the function pretraining in detail and make their approach more generalizable.
3.	Considering Figure 3, the performance of DeMul is negligible in 6 out of 11 datasets compared to other approaches such as GalLoP. This significantly diminishes the efficacy of the approach.
4.	The major weakness of the paper is very limited evaluation. The authors have limited their approach to only few-shot image classification while all the recent relevant works [CoCoOp (Zhou et al., 2022a), GalLoP (Lafon et al., 2024), MaPLe (Khattak et al., 2023a)] show their efficacy on 1. Domain Generalization 2. Base-to-New Generalization and 3. Cross-Dataset Transfer in addition to few-shot downstream image classification. Without such comparison it is hard to judge the goodness of the proposed approach beyond the single task of few-shot downstream image classification.
5.	Distilling LLM text to learnable prompts and prompt weighting separately have been explored in some of the recent works. Such as [a] or [b]. The author should discuss these references in the related work section and explicitly explain how their work is distinct from these similar studies.

[a] PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR 2024

[b] A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models, ICML 2023

questions:
The weaknesses are already given in the question form. In addition, here are a few additional questions.
1. How is the value of $\alpha$ chosen? Is there any sensitivity analysis on $\alpha$?
2. Are the experiments run for different random splits of training data? Then are the reported values the average over these?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714612814

thread_type:
official_review

<thread_object>
creator: reviewer_rgbn

<content>
summary:
This paper explores using large language models (LLMs) to enhance prompt learning for vision-language models. It proposes a description-free method to distill knowledge from LLMs into learnable prompts by directly aligning their embeddings. Additionally, a multi-prompt strategy is examined to improve performance. Experiments on few-shot learning benchmarks and ablation studies demonstrate the effectiveness of the proposed method.

strengths:
1. The paper is well-crafted with clear expressions, making it easy to understand.

2. The description-free distillation approach for training learnable prompts is interesting, novel, and effective. It is also more cost-effective than description-needed methods, which require multiple queries to models like GPT for comprehensive descriptions.

3. The study on diverse semantics and the importance of multiple learnable prompts is well-motivated.

4. The extensive few-shot learning experiments and comprehensive ablation study effectively demonstrate the proposed methods and each module.

weaknesses:
1. Although the paper demonstrates superior few-shot performance, the notable aspect is the open-vocabulary ability of CLIP. Since 2023, literature has focused on improving this aspect. Additional experiments on this topic are suggested.

2. The authors should discuss whether the proposed distillation method works when ground-truth training data is missing. Solely text-side fine-tuning may lead to misalignment with the visual side. If it doesn't work, this limitation should be addressed.

3. The computational costs, including time and memory during training and inference, should be compared to existing methods. This is especially important because multiple prompts can introduce substantial costs, particularly when dealing with a large number of classes, such as in ImageNet-1K.

4. The symbol $t$ is defined both as prompt embeddings (L198) and learnable prompt vectors (L202); these should be differentiated.

questions:
In line 7 of Algorithm 1, why do the text embeddings $g(c_i)$ from the output of the CLIP text encoder go through the encoder again?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732418312159

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment

comment:
We sincerely thank our reviewers for their detailed feedback, thoughtful reviews, and insightful comments. Based on the received reviews, we have incorporated all feasible changes and uploaded a revised version of the paper. (The revised texts are highlighted in **violet** color.) If you feel further modifications are necessary, we would greatly appreciate any additional comments.

The updated content is as follows:

- Updated the notation in equation (2) on line 197 by replacing \\(t\\) with \\(e\\) to avoid reuse of \\(t\\) in different contexts.  
- Added a detailed explanation of the algorithm to provide better clarity.  
- Emphasized in the caption of Figure 5 (top image) that it represents relative changes from the initialized values, rather than absolute values.  
- Expanded the Future Works section to address limitations highlighted in the reviews and outline potential research directions.  
- Added missing citations.  

Moreover, some questions were raised by multiple reviewers, and we provide additional experimental results to address these points below.

---

### 1. Ablation Study on Fixed Values \\(\alpha\\) and \\(\lambda\\)  
(For reviewers **EZLZ, JO4M, ONAD**)  

One concern was the absence of an ablation study for the fixed values of \\(\alpha\\) and \\(\lambda\\) used in our experiments. Below, we present the results of the ablation study conducted on the DTD dataset in the 1-shot setting, which helped us determine the optimal values.  

1. **\\(\alpha\\):**  
   We experimented with multiple values of \\(\alpha\\) to balance the classification loss and the distillation loss. Notably, setting \\(\alpha = 0.01\\) or \\(\alpha = 0.1\\) led to a significant performance drop. This suggests that prompt weight regularization was either too weak (\\(\alpha = 0.01\\)) or too dominant (\\(\alpha = 0.1\\)), disrupting the balance between the two losses. The best performance was achieved with \\(\alpha = 0.05\\).  

2. **\\(\lambda\\):**  
   We explored \\(\lambda\\) values from 0.1 to 1.0 to evaluate the effect of L1 regularization on prompt weighting. The results show that \\(\lambda = 0.4 \sim 0.5\\) yielded the best performance. A smaller \\(\lambda\\) (e.g., 0.1) reduces the impact of distillation, while a larger \\(\lambda\\) (e.g., 1.0) overly emphasizes the regularization, diminishing the classification accuracy. This demonstrates that a balanced degree of distillation and classification is critical for optimal performance.  

**Ablation Study Results**  
|                | 0.1  | 0.2  | 0.3  | 0.4  | **0.5** | 0.6  | 0.7  | 0.8  | 0.9  | 1.0  |
|----------------|-------|-------|-------|-------|---------|-------|-------|-------|-------|-------|
| **\\(\alpha = 0.01\\)** | 49.5  | 50.3  | 49.0  | 57.7  | 53.3    | 50.0  | 52.1  | 49.1  | 49.9  | 53.4  |
| **\\(\alpha = 0.05\\)** | 47.8  | 51.2  | 59.3  | 61.5  | **62.1** | 60.2  | 61.3  | 58.2  | 59.9  | 60.0  |
| **\\(\alpha = 0.1\\)**  | 42.5  | 45.6  | 49.0  | 47.0  | 44.7    | 50.1  | 47.3  | 49.1  | 51.3  | 48.4  |

We believe these results adequately justify the fixed values used in the paper (\\(\lambda = 0.5\\), \\(\alpha = 0.05\\)).  

---

### 2. Dependency on Training Data Distribution  
(For reviewers **EZLZ, 3ZUA**)  

Another concern was the extent to which our method depends on the distribution of training data. As mentioned in the Conclusion, one limitation of our method is its dependency on the training image distribution, which remains an inherent challenge in few-shot learning. To ensure the reliability of the reported scores, we averaged the results from multiple experiments. For all experimental settings, the training data was randomly sampled, and we used the average performance from approximately 5-10 runs.  

**Results of 10 Experiments on DTD Dataset**  
| \#shot         | 1     | 2     | 3     | 4     | 5     | 6     | 7     | 8     | 9     | 10    | **Avg**  |
|-----------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|----------|
| **1-shot**     | 62.3  | 61.9  | 62.6  | 61.9  | 62.5  | 61.0  | 62.3  | 61.4  | 62.5  | 62.3  | **62.1** |
| **2-shot**     | 66.9  | 67.3  | 68.6  | 67.8  | 67.0  | 68.9  | 67.9  | 67.5  | 68.3  | 68.5  | **67.9** |
| **4-shot**     | 70.2  | 69.8  | 71.0  | 71.2  | 69.2  | 70.5  | 70.4  | 69.9  | 70.1  | 71.1  | **70.3** |
| **8-shot**     | 74.9  | 75.2  | 75.1  | 75.2  | 75.6  | 74.9  | 75.2  | 75.3  | 74.8  | 74.5  | **75.1** |
| **16-shot**     | 76.5  | 77.0  | 77.0  | 78.0  | 76.2  | 76.9  | 78.2  | 76.8  | 76.6  | 76.9  | **77.0** |
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses prompt learning in VLMs by directly distilling the knowledge in a GPT model to the learnable prompt vectors in VLMs. The method essentially maps the learnable prompt vectors to the GPT's embedding space and achieves significant improvements over previous prompt learning methods. The paper received five reviews with 4x borderline accept and 1x accept. The reviewers generally found the idea interesting and novel, and the results strong. In terms of concerns, they raised questions about the computational cost and some design choices, and also requested more results under different experimental settings. The rebuttal well addressed these questions. After reading the paper, the AC agrees with the reviewers that the paper has merits and therefore recommends acceptance.