CARPRT: Class-Aware Prompt Reweighting for Pre-Trained Vision-Language Models

Authors: Ruijiang Dong, Zesheng Ye, Feng Liu, Jianzhong Qi, Gang Niu, Masashi Sugiyama
Primary area: foundation or frontier models, including LLMs
Keywords: Prompt Weighting, Vision-language Models

Abstract:
When using a pre-trained vision-language model (VLM) to classify an image, we often need to use the pre-trained VLM to compute a similarity score between the image and texts containing a semantic label, e.g., “a photo of a cat”, where “a photo of a” is called a prompt and “cat” is the semantic label (a.k.a. a class in classification tasks). The existing studies have shown that the selection of prompts can significantly affect the scoring scheme between a given image and a semantic label, and they proposed a new score via using a weighting vector to reassemble scores regarding different prompts. However, these studies assume that all classes should share the same weighting vector. In this paper, we first empirically show that the existing approach is sub-optimal. We subsequently revisit the existing reweighting strategy from a probabilistic view and find an implicit assumption in prior work: the conditional independence of classes and weights, which often does not hold in practice. To cope with this problem, we propose class-aware prompt reweighting (CARPRT), a strategy designed to adjust the weighting vector for each class. CARPRT calculates the relevance scores for prompt-class pairs with respect to all images, and identifies the maximum score for each prompt-class pair. These maximum scores are then averaged across prompts for each class to estimate the class-specific weighting vectors, ensuring that prompts are optimally reweighted based on class-specific information. Our experiments demonstrate that CARPRT outperforms the existing reweighting strategy under the image classification tasks.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729647689634

thread_type:
official_review

<thread_object>
creator: reviewer_cbgl

<content>
summary:
This paper studies class-aware prompt reweighting strategy for vision-langauge models. The idea is that when doing prompt ensembling, using the same weighting vector for all classes overlooks the diverse characteristics of different classes, and may yield suboptimal results. Further, the authors present a probabilistic viewpoint based on Bayes' Theorem to show the conditional independence assumption between the class and weights in the class-shared weighting method. CARPRT is proposed that obtains weight vectors based on the relevance score to prioritize the most relevant prompts for each class. The efficacy of CARPRT is verified using CLIP model over multiple benchmarks.

strengths:
- This paper studies the important prompt ensembling task. With the popularity of zero-shot image classification with multimodal models such as CLIP, the proposed method/idea can be useful to a wide audience.

- The idea of class-aware prompt weighting is simple, but the authors present a probabilistic viewpoint based on Bayes' theorem to show its advantage over class-independent prompt weighting.

- The final CARPRT method is a simple implementation that adheres to the key principles established in the probabilistic analyses. Over 10 fine-grained datasets and 3 ImageNet variants, it shows an improvement in accuracy over ZPE. Both ViT and ResNet backbones are considered. Hyper-parameter studies are adequate.

- The paper is well-written and easy to follow.

weaknesses:
- The proposed CARPRT adheres to the key principles established in Section 3, but the detailed correspondences are not explicitly presented. If I understand it correctly, the marginalization in Eq.(6) over $W$ reduces to a point estimation of $W$ for the following analysis. $Pr(W|P)$ is not reflected in the implementation. $Pr(y_c|W,P)$ is approximated by Eq.(9). The final weights $w$ in Eq.(16) seem to be the posterior approximation of Eq.(7), but how to derive these formulations (from Eq.(13) to Eq.(16)) are omitted in the paper. The authors may want to add a section of mathematical derivations in the appendix that can assist audiences to understand how CARPRT is designed from the probabilistic forumations.

- Using class-aware prompt weighting theoretically is better than class-independant prompt weighting, but its robustness is also a practical concern. The authors also mention the influences of pseudo-labels and prompt template pool, but relevant empirical analyses are not provided. For example, is CARPRT more sensitive to noisy pseudo-labels than ZPE? (perhaps we can manually inject noises into pseudo-labels to understand the influences) What happens when fine-grained datasets have more or less diverse template pool? Is it really necessary to combine templates from different datasets for a particular test dataset?

questions:
- Please clarify the correspondances between CARPRT formulations and probabilistic derivation.

- In Appendix D, the authors discuss several forms of $Pr(W|P)$. Could the authors explain how these priors can be obtained and used, for example, in the Test-Time Adaptation situation mentioned in the paper?

- I am confused about the toy example in Proposition 3. Why all the three (Ln 880-883) have $y_1$ as their variates? How Ln 886-889 are derived?

- In Ln 431, a typo in 'iaccuracy'.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730451109357

thread_type:
official_review

<thread_object>
creator: reviewer_a8wx

<content>
summary:
This paper proposes a category-aware prompt re-weighting (CARPRT) method to improve the performance of pre-trained visual language models (VLMs) on zero-shot image classification tasks. It first finds the limitations of existing prompt re-weighting methods, and explains the root cause from a probabilistic analysis perspective. The proposed CARPRT calculates the relevance scores of prompt-class pairs and determines the optimal prompt weight vector for each class based on these scores. The experimental results show that CARPRT outperforms existing re-weighting strategies on various image classification benchmarks.

strengths:
1. The proposed CARPRT introduces a class-specific weighting mechanism, significantly improving the alignment of prompts with class-specific information.
2. The paper provides a solid theoretical foundation, addressing the conditional independence limitations in previous methods.
3. By automating the prompt reweighting process, the method reduces dependency on manually crafted prompts.

weaknesses:
1. The approach introduces additional computational complexity while the improvement is extremely limited (less than 1%) on part of situations. Moreover, only the weighted baselines are compared, I doubt that whether it can enhance existing prompt engineer methods.
2. The effectiveness of CARPRT is tied to the quality and diversity of the prompt template pool, is it effective for learned prompts?

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714834829

thread_type:
official_review

<thread_object>
creator: reviewer_6qg1

<content>
summary:
The paper presents CARPRT, a method to automatically choose and assign weights to a given set of standard prompts when used for zero shot image classification. CARPRT specifically addresses the limitation of the current zero-shot method that uses a fixed weighting vector for the given set of prompts for all classes. Instead, it adjusts prompt weights for each class based on image-prompt relevance by leveraging pseudo-labeling. Experiments on multiple benchmarks show that CARPRT improves zero-shot classification accuracy, especially in fine-grained classification tasks.

strengths:
**Well motivated**: CARPRT effectively demonstrates the need for class-aware weights, showing that a shared weight for prompts for all classescan hinder zero-shot VLM performance.

**Simple, mathematically sound probabilistic framework**: The paper uses an energy-based probabilistic framework to model underlying class distributions for prompt reweighting, enhancing the interpretability of CARPRT’s approach.

**Improvements on Fine-grained datasets**: The method achieves state-of-the-art results on multiple datasets, notably improving accuracy on some fine-grained datasets like EuroSAT and Flowers102.

weaknesses:
**W1 Limited Improvement on ImageNet Variants**: The performance gains on datasets like ImageNet-A and ImageNet-R are modest, suggesting that CARPRT’s advantages may be less pronounced on general image datasets compared to fine-grained datasets. more testing on other large datasets (e.g. Places 365) would help better evaluate the methods sginficance for large datasets

**W2 - Pseudo-Label Accuracy Dependency**: The method depends on accurate pseudo-labels to derive class-specific weights, which can be problematic in scenarios where pseudo-labeling quality is low. A more detailed study on how this affects performance might help increase the utility of the method.

questions:
1. Did you try using models other than Energy based ones to model the underlying class distributions (e.g. GMMs, Local Manifold models) ? It would be interesting to see how/ if the choice makes a difference. 
2. How well does CARPRT work on fine-grained datasets that have class imbalances ? Does using a different weights for each class improve performance here too ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730844199493

thread_type:
official_review

<thread_object>
creator: reviewer_rvkh

<content>
summary:
This paper studies how to improve the prompting for the CLIP model. The motivation lies in the fact that existing prompting methods ignore the dependence of prompt weights on different classes. The authors first conduct experiments to demonstrate the positive influence of class-specific weights on prediction performance. Then, a class-aware prompt reweighting method CARPRT is proposed. CARPRT calculates the weight matrix based on the relevance scores between images and prompt-class pairs. Experimental results show that CARPRT surpasses ZPE in most cases.

strengths:
- The motivation is intuitive while reasonable. As far as I know, it is the first work applying the class-aware weighted prompt ensemble.
- The paper is well-written and easy to follow. The authors revisit the existing works, present detailed analyses and explanations, and give clear formulations and an algorithm procedure for the proposed method.
- Extensive experimental results are reported.

weaknesses:
- One concern is regarding the computational complexity. Unlike class-wise or prompt-wise reweighting, the proposed method calculates the weight matrix based on two dimensions. Is there any complexity analysis?
- Another concern is regarding the performance gains. It seems that the results in Table 2 showcase marginal improvement compared to ZPE. The performance gains of some datasets in Table 1 are also limited. This may raise concerns regarding the limited contributions to the community.
- I am also concerned with the hyper-parameter sensitivity across different datasets. It seems that the results in Figure 3 are the overall results. However, I am still concerned with the results for each dataset. Are the hyper-parameter settings (e.g. $\tau=3.0$) suitable for all datasets? Considering the generality of the proposed method, I suggest the authors report more detailed results and analyses.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302831132

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Update and Summary of Changes (part 1/2)

comment:
We appreciate the thorough feedback from all reviewers and have addressed each concern through additional experiments and clarifications. 
We outline our key responses and improvements to the manuscript in this summary.

**Responses to Reviewer rVkh**

1. **Computational Complexity.**
- **Concerns**: Does CARPRT introduce computational overhead compared to ZPE?
- **Response**: We clarify that no significant overhead is introduced by CARPRT compared to ZPE. We **conduct a complexity analysis**, showing that both methods rely on the computationally intensive embedding step via the CLIP encoder, which dominates the cost. We also **report actual evaluation elapsed time** for both CARPRT and ZPE. 

2. **Limited Performance Improvement**
- **Concerns**: Why are improvements over ZPE limited for ImageNet and for certain fine-grained datasets (Table 1)?
- **Response**: We clarify that limited gains may be attributed to template quality. To address this,
    - We **supplement experiments** by introducing LLM-generated template pools, under which CARPRT consistently outperforms ZPE. We **add a section (Appendix J)** to include the impact of template quality.
    - We also **supplement experiments** that integrate CARPRT with learnable prompts method, demonstrating CARPRT can **facilitate both training/non-training prompt engineering** scenarios, regardless of general datasets (Imagenet) or fine-grained recognition datasets. We **append these results in Appendix K**.

3. **Hyperparameter Selection**
- **Concerns**: Are the hyper-parameter settings suitable for all datasets? 
- **Response**: We clarify that a globally well-performing temperature was chosen due to the lack of validation data in the **zero-shot setting**. We **include additional results (Appendix H)** that confirms balanced performance yielded by the selected temperature. 

4. **Follow-up Concern: Scope of Supplementary Experiments**
- **Concerns**: Further evaluations are suggested to assess the impact of LLM-generated template pools and CARPRT integration into learnable prompts (ProDA).
- **Response**: We **conduct experiments** with LLM-generated template pools for fine-grained datasets. We **evaluate the performance** of integrating CARPRT with ProDA on **ImageNet**.


**Response to Reviewer 6Qg1**

1. **Performance**
- **Concern**: Limited performance gain on ImageNet.
- **Response**: We clarify that limited gains may be attributed to template quality. We have explored **two ways** to address it:
    - We **supplement experiments** by introducing LLM-generated template pools, under which CARPRT consistently outperforms ZPE. We **add a section (Appendix J)** to include the impact of template quality.
    - We also **supplement experiments** that integrate CARPRT with learnable prompts method, demonstrating CARPRT can **facilitate both training/non-training prompt engineering** scenarios. We **append these results in Appendix K**.

2. **Pseudo-Label Dependency**
- **Concern**: How does pseudo-label quality affect CARPRT's performance?
- **Response**: We **analyze** and **show a moderate correlation** between pseudo-label accuracy and CARPRT's performance exists, and further **introduce iterative refinement** to mitigate performance degradation when the pseudo-label quality is low.

3. **Imbalanced Datasets**
- **Concern**: How does CARPRT perform under class imbalance?
 - **Response**: We **supplement experiments**, evaluating CARPRT under class imbalance by manually constructing an imbalanced CIFAR-10 dataset. CARPRT **consistently outperforms baselines** in imbalanced settings on imbalanced CIFAR-10 (**Appendix I**).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work aims to improve the zero-shot performance of CLIP by leveraging unlabeled data. Specifically, a class-aware reweighting strategy is proposed to refine the ensemble of multiple prompts according to scores of image-prompt pairs. Reviewers are concerned about computational complexity, a marginal improvement compared to baselines, the sensitivity of hyperparameters, the quality of pseudo labels, lack of strong baselines. While the rebuttal addressed some concerns, the major concerns about performance remained. AC notices that some recent work is proposed to improve CLIP by applying transductive methods on unlabeled data [1] as this work while showing a much better performance. Authors are encouraged to incorporate comments and experiments in discussions to polish the work for resubmission.

[1] Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP. NeurIPS 2024.