Title: Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
Venue: NeurIPS 2024
Accepted: yes
Keywords: vision-language model

zero-shot classification

logit adjustment
Abstract: Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. 
 To tackle the above challenges, we propose a label-**F**ree p**ro**mpt distribution **l**earning and b**i**as **c**orrection framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching.
This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of $2.6\%$ on 10 datasets with CLIP ViT-B/16 and achieving an average margin of $1.5\%$ on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in [https://github.com/zhuhsingyuu/Frolic](https://github.com/zhuhsingyuu/Frolic).

Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting

Abstract

Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a labelF ree p ro mpt distribution l earning and b i as c orrection framework, dubbed as Frolic , which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of 2 . 6% on 10 datasets with CLIP ViT-B/16 and achieving an average margin of 1 . 5% on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic .

1 Introduction

Vision-language models (VLMs), such as CLIP [29], which are pre-trained on large-scale datasets using contrastive loss, effectively align visual and textual representations within a shared feature space. This capability enables the zero-shot inference on downstream tasks through prompting and achieves remarkable performance. For example, using a selection of 80 hand-crafted prompts, a zero-shot CLIP ViT-B/16 achieves an accuracy of 68 . 7% , and with prompts generated by language models [27], the accuracy increases to 69 . 9% .

The success of zero-shot capabilities heavily relies on the appropriate text descriptions of the classes, which has gained research interest in improving prompts. Recent studies propose learning prompts from a small set of labeled images in the downstream data [46, 45, 47]. Among these studies, Lu et al. [18] and Wang et al. [38] have found that learning the distribution of diverse prompts, which better captures the variance in visual representations, leads to improved performance. Although these methods have achieved significant improvements, they still depend on artificial prior knowledge for labeling downstream data and are limited by the quality of manual annotations, which may restrict the scalability of the original model.

! "

!

, Σ

! "

"

, Σ

! "

#

, Σ

(a)

(b)

(c)

(d)

Biased decision boundary

Our decision boundary

Another significant approach to enhancing zero-shot performance involves correcting the label bias inherent in skewed web-scale pre-training data [1, 25, 49]. This bias leads to highly imbalanced predictions and suboptimal performance. As illustrated in Figure 1(c), the average predicted probability on ImageNet using ViT-B/16 reveals an imbalanced distribution: the highest class probability exceeds 0 . 002 , whereas the lowest is below 0 . 0005 . Existing methods correct this bias by allowing access to a portion of the pre-training data [1, 25], or by using labeled downstream data [49]. However, the pre-training data is often inaccessible due to privacy or copyright concerns, and debiasing without labeled data is challenging.

In this paper, we introduce a labelF ree p ro mpt distribution l earning and b i as c orrection framework, dubbed as Frolic , which eliminates the need for data annotations to enhance zero-shot performance. First, unlike previous methods [1, 27, 46, 39, 43], which use a single class prototype for each class to define the decision boundary (as shown in Figure 1(a)), our approach employs Gaussian distributions to model the varied visual representations of text prototypes, as illustrated in Figure 1(b). It is worth noting that estimating such a distribution is non-trivial, since classical maximum likelihood estimation requires the annotation of each sample. Fortunately, we demonstrate that it is possible to infer distribution for each class directly from the first and second moments of the marginal distribution of downstream data without label information. Second, to prevent the use of pre-training data or labeled samples in downstream tasks, we develop a bias estimation mechanism, which transitions the sampling process from the pre-training data distribution to a class-conditional sampling from downstream distribution. By incorporating the estimated label bias into zero-shot models, we can achieve a balanced prediction, as illustrated in Figure 1 (d). Furthermore, we explore the possibility of combining the original CLIP predictions with those from the Gaussian-based models to enhance zeroshot performance. To this end, we have developed a confidence-matching technique that dynamically balances the contributions of the two models, eliminating the need for hyperparameter tuning. Notably, our framework is training-free, which enhances both flexibility and ease of implementation.

The main contributions of this work are:

We enhance zero-shot performance by estimating a distribution over prompt prototypes to capture the variance in visual appearances. We demonstrate that this process can be implemented entirely without labels.

Wepropose a confidence matching technique that fuses the original CLIP model with a Gaussian distribution-based model to further enhance zero-shot performance. This process eliminates the need for hyper-parameter searching, in stark contrast to previous studies.

We develop an unsupervised method to correct pre-training label bias. Unlike existing methods that require access to pre-training data, our Proposition 2 suggests that we can avoid sampling from the pre-training distribution for estimating and correcting this bias. Instead, our method utilizes only downstream images.

We demonstrate the effectiveness of our proposed method Frolic by conducting experiments across 16 datasets, which has a consistent and significant improvement over existing baselines. For example, our method surpasses the state-of-the-art zero-shot models by a margin of 2 . 6% on average with CLIP ViT-B/16.

2 Related Works

Zero-shot vision models. Vision models pre-trained with auxiliary language supervision, such as CLIP [29] and OpenCLIP [6], facilitate zero-shot inference through prompting. Enhancing zero-shot performance has gained increasing research interest: (1) One approach involves prompt engineering, which includes designing hand-crafted prompts based on human priors [29] or automatically generating prompts via language models [35]. (2) Another promising approach seeks to improve classifiers, e.g. , ZPE [1] scores the importance of candidate prompts for prompt ensembling. InMaP [28] reduces the modality gap between vision and text. Several studies [32, 31] optimize prompt at test time by encouraging consistent predictions across augmented samples. Our work aims to enhance zero-shot models by learning the prompt distribution and mitigating the pre-training label bias.

Prompt distribution learning. Automatically learning prompts from downstream data has shown potential in improving zero-shot models [46, 45, 47]. These methods typically optimize prompts via minimizing the classification loss on the target task. However, as pointed out in Lu et al. [18], learning prototype prompts overlook the diversity of visual representations. To this end, they estimate a distribution over the prompts to capture the variance of visual representations. Recently, Wang et al. [38] propose training-free prompt distribution learning to improve efficiency. Contrary to existing methods [18] that estimate distributions through supervised approaches, our method circumvents the necessity for labels by inferring the variance of distributions from the statistics of unlabeled data.

Correcting label bias. Label bias generally occurs in the presence of skewed or imbalanced training data. In response to this challenge, Logit Adjustment (LA) [34, 14, 21, 49] has emerged as a prominent technique in long-tailed learning, specifically designed to adjust the decision boundary of classifiers to mitigate label bias. Menon et al. [21] derives the theoretically optimal adjustment for logits. Zhu et al. [49] extents LA to fine-tune zero-shot models by removing the pre-trained label bias. Unlike approaches that rely on the label distribution of the training set [34, 14, 21, 48] or the labels of fine-tuning data [49], our method adjusts the logits using unlabeled test data.

3 Methods

In this section, we present our prompt distribution learning, adaptive fusion, and logit adjustment techniques for adapting zero-shot models. Without loss of generality, we adopt CLIP [29] as our zero-shot model. To begin with, we emphasize three advantages of our framework:

Training-free: Our Frolic is training-free without optimizing the backbone of the zero-shot models, enhancing both flexibility and ease of implementation.

Label-free: Our method Frolic requires no external labeled data, making it suitable for zero-shot scenarios.

No hyper-parameters searching: Our method Frolic eliminates hyper-parameter tuning on validation datasets, in stark contrast to [38, 44]

3.1 Setup

The zero-shot model consists of a visual encoder Φ v ( · ) and a text encoder Φ t ( · ) . Given a set of unlabeled image data { x i } N i =1 and the unique text set of the class description { z j } K j =1 , their visual and text representation can be computed as:

where x i and z j share the same dimension ( x , z ∈ R d ). N is the sample size and K is the class size. z j can be considered as the prototype for class j . With an image x and all prototypes { z j } K j =1 , zero-shot CLIP predicts the label as:

where f c ( x ) j = z ⊤ j x is the score for class j .

3.2 Label-Free Prompt Distribution Learning

In order to express the diverse visual variations, our approach aims to learn the distribution of the class prototypes. Previous studies [18, 38] show that the Gaussian distribution is effective to model the distribution of the CLIP features and achieves impressive improvement. However, these methods require extra labeled training data , which is not applicable to our zero-shot setting.

Specifically, we follow [38] to assume N ( z 1: K , Σ) with identical covariance is the underlying distribution. In classical maximum likelihood estimation [3], the shared covariance Σ is computed by averaging the empirical covariances of K classes: ˆ Σ = 1 K ∑ j ˆ Σ j , where ˆ Σ j = 1 |C j |-1 ∑ x ∈C j ( x -z j )( x -z j ) ⊤ . Here, one need the label information of each image to compute ˆ Σ j . Fortunately, to avoid using label information, we can infer Σ directly from the expectation and the second order moment of the marginal distribution P ( x ) . 1 Using a Gaussian mixture model with the priors { π j } K j =1 , P ( x ) is given by:

Denote the second moment of x as M , we have (proof in Section A.1):

Denote π = [ π 1 , .., π K ] ⊤ , Z = [ z 1 , .., z K ] ⊤ , and the expectation of x as µ , the prior over the unlabeled data distribution can be estimated by (proof in Section A.2):

We estimate the expectation and the second order moment as ˆ µ = 1 N ∑ N i =1 x i and ˆ M = 1 N ∑ N i =1 x i x ⊤ i , which are unbiased and consistent. In practice, given that test benchmarks are generally class-balanced, we use a uniform prior over the data distribution, i.e. , π j = 1 K . Combining with Eq. (4), the estimated shared covariance ˆ Σ can be written as:

Let w j = ˆ Σ -1 z j and b j = -1 2 z ⊤ j w j , the Gaussian discriminant analysis predicts the label for an image x as follows (proof in Section A.3):

where f g ( x ) j = w ⊤ j x + b j is the score for class j .

3.3 Prediction Fusion via Adaptive Calibration.

As a rule of thumb, combining the zero-shot predictions with the ones from the learned model can further improve performance for CLIP adaptations [44, 35, 40, 47, 38, 50]. Previous studies commonly employ a mixing coefficient, α , to balance the contributions of two models, e.g. , f ( x ) = f c ( x ) + αf g ( x ) . Typically, this hyper-parameter α is optimized on labeled data to maximize accuracy. However, in our context, labels are unavailable, it is not possible to search for the optimal value of α . It is imperative to develop a mechanism that balances the prediction fusion without relying on the label.

0.2

0.4

0.6

0.8

1.0

Pets

Flowers

Aircraft

DTD

EuroSAT

Cars

Food

SUN

Caltech

UCF

ImageNet

f

c

f

g

Confidence

The key in our prediction fusion lies in aligning the average confidence of the two models. Formally, the average confidence over the dataset { x i } N i =1 scaled by a temperature τ is given by the average of the model's probability for its prediction:

Ideally, a model's average confidence should reflect the predicted accuracy, which is called a wellcalibrated model. Suppose we have the oracle well-calibrated models, denoted by f ′ c ( · ) and f ′ g ( · ) , Kumart et al. [17] prove that the optimal strategy is to fuse the two predictions equally, i.e. , f f ( x ) = f ′ c ( x )+ f ′ g ( x ) . However, as shown in Figure 2, f g is much overconfident than f c . Let f g ( x ) = Cf ′ g ( x ) for large C ∈ R + (an overconfident model magnifies its logits) and suppose f c ( x ) ≈ f ′ c ( x ) . The fused predictions are given by f f ( x ) = Cf ′ g ( x ) + f ′ c ( x ) . For very large C , f f ( x ) and f g ( x ) have the same predictions, i.e. , f f ( x ) is biased towards the f g ( x ) . As we do not have the label to compute accuracy, we cannot apply classical calibration methods [19, 10] to calibrate f g ( x ) and f c ( x ) . As our desideratum is to automatically balance the contribution of the two models, we can optimize τ g to make the confidence of f g to match up the one of f c , which circumvent the need of labels:

Specifically, we implement this by binary search, as the confidence monotonically decreases as the temperature increases. τ c = 0 . 01 is fixed and learned by CLIP. The fused logits are given by:

3.4 Correcting Pre-training Label Bias via Label-Free Logit Adjustment

̸

Pre-training datasets typically exhibit a long-tailed concept distribution, leading to biased performance in zero-shot models [49, 25, 5, 1]. This bias occurs because zero-shot models reflect the posterior probability P ( y | x ) derived from the pre-training distribution. According to Bayes' rule, this posterior probability is influenced by the pre-training label distribution P ( y ) , as P ( y | x ) ∝ P ( x | y ) P ( y ) . If the prior probability of class j is significantly larger than that of other classes ( e.g. , P ( j ) ≫ P ( i ) , ∀ i ∈ [ K ] , i = j ), the predictions will be biased toward class j .

Prior research [21, 14] has identified a theoretical optimal solution to address this label bias: let β y denote the prior probability of class y , i.e. , β y = P ( y ) . The debiased logit of f f ( x ) for class y should be (proof in Section A.4):

Previous methods estimate β either by accessing the pre-training data [25, 1] or counteract the influence of the prior by optimizing on labeled downstream data [49]. However, these approaches are often impractical due to inaccessible pre-training labels due to privacy or copyright concerns or the

necessity for labeled downstream data. In this work, we address label bias using only the unlabeled downstream data { x i } N i =1 .

Let s ( x ) = softmax( f f ( x )) represent the softmax outputs of f f ( x ) , where s ( x ) y = ˆ P ( y | x ) is the predicted probability for class y . Define s j = E x [ s ( x ) | Y = j ] as the expected posterior probability over the image distribution of class j , and let S = [ s 1 , ..., s K ] ∈ R K × K . We prove that the pre-training label prior β = [ β 1 , ..., β K ] ⊤ ∈ R K must satisfy the following linear equation system:

Remark. The key point in Eq. (12) is that we avoid sampling from the pre-training data distribution; instead, we sample from P ( x | y ) , which is available from the downstream data. We provide the proof in Section A.5 and the numerical power solver for β in Section A.6.

We iteratively refine the estimation of S and solve for β using updated pseudo-labels generated by f d ( x ) . As f d ( x ) becomes more precise, it yields more accurate pseudo-labels for x , which in turn enhances the accuracy of our estimation of β . Specifically, we initialize

where x ∈ C 0 j if x is classified as j by f 0 d ( x ) . We proceed by solving for β 1 using Eq. (12), refining f 1 d ( x ) using Eq 11 and reassign the pseudo label using f 1 d ( x ) to estimate the updated s 1 j . This process is repeated t times until the relative change in β satisfies the convergence criterion:

where ϵ is a predefined threshold for relative error tolerance. We summarize the algorithm for solving β in Algorithm 2 and provide the overall pipeline in Algorithm 1.

Discussion: Comparison with Other Prior Estimation Methods. We compare existing methods for estimating pre-training label priors and demonstrate their in-applicability or flaws in our setting.

(1) Explicit method : the explicit method directly measures the frequency of each class in pre-training data, e.g. , β y = N y N , where N y is the sample size for class y and N is the total sample size. Most long-tail learning algorithms, e.g. , LA and PC [21, 14], are based on this method because they can access the training data. However, estimating such frequency is complex due to the free-form texts, as opposed to a pre-defined label set. In addition, the pre-training dataset is often inaccessible, making the method inapplicable in our case.

(2) Implicit method : [1, 25] allow access to a portion of the pre-training data D pt and use the law of total probability to estimate the prior:

where ˆ P pt ( y | x ) denotes the zero-shot model. However, in our setting, we do have access to the pretraining data or a portion of it. Wang et al. [37] replace the pre-training data D pt with the downstream data D ds in Eq. (15) to debias. However, this method neglects the distribution discrepancies between the pre-training and downstream data. In Section 4.3, we show that our debiasing significantly outperforms this implicit method.

(3) TDE [34]: Tang et al. [34] debias by removing features along a global direction, retaining only those orthogonal to it. Specifically, the global feature is estimated by ¯ x = 1 |D pt | ∑ x ∈D pt x . Given a test sample x , TDE decomposes it into parallel and orthogonal directions to ¯ x : x = x ∥ + x ⊥ . Then, only the orthogonal component is used for classification: ˆ P pt ( y | x ⊥ ) . While TDE does not require labels for the samples, we cannot apply it because it requires sampling from the pre-training data. In Section 4.3, we replace D pt with downstream data D ds and demonstrate its inferior performance.

(4) GLA [49]: Zhu et al. [49] propose to estimate the pre-training prior from the downstream data using the Bayes optimal criterion. The pre-training prior β is solved by optimizing:

where ℓ ce is the cross-entropy loss and f pt ( x ) is the logit of the zero-shot model. While this method circumvents the need for pre-training data access, it is inapplicable because it requires labels for each downstream sample.

4 Experiments

4.1 Setup

Datasets. We conduct experiments on 16 image classification benchmarks, covering diverse range categories including generic objects (ImageNet [8], Caltech [9]), scenes (SUN [42]), textures (DTD [7]), satellite images (EuroSAT [11]), actions (UCF [33]) and fine-grained categories (Pets [26], Cars [16], Flowers [23], Food [4], Aircraft [20]). Additionally, we evaluate on five ImageNet distribution shifted datasets [8]: ImageNetV2 (IN-V2) [30], ImageNet-Sketch (IN-Sketch) [36], ImageNet-A (IN-A) [13], ImageNet-R (IN-R) [12] and ObjectNet [2].

Implementation details. We adopt CLIP [29] ViT-B/16 and ViT-L/14 as our pre-trained models. The default model for ablation studies is CLIP ViT-B/16. We use the same text descriptions as SuS-X [35] and CuPL [27], and adhere to the InMaP [28] settings to include all test images. τ c = 0 . 01 is provided by CLIP. ϵ in Algorithm 2 is set to 0 . 01 . All experiments are conducted on a single NVIDIA 3090 GPU if not specified. Note that our algorithm does not require any hyper-parameter searching.

4.2 Main Results

We compare our method with several state-of-art methods, including CLIP [29], TPT [32], PromptAlign [31], SuS-X-DS [35], TDA [15], GPT4-Prompt [41], CuPL-CLIP [27], and InMaP [28]. Both TPT and TDA utilize a stream of unlabeled test images. For TPT, TDA and InMaP, we produce the results of ViT-L/14 by executing the official released code and maintaining the same hyper-parameters.

Results on 10 datasets. In Table 1, we summarize the accuracy across all datasets, excluding ImageNet and its shifts (denoted as 10-datasets). Our method consistently shows superior performance across the datasets and backbones, significantly surpassing GPT4-Prompt, which is known for generating high-quality prompts. By integrating our method with InMaP, our Frolic achieves the highest performance, with an average improvement of 2 . 6% with ViT-B/16 and 2 . 0% with ViT-L/14.

Results on ImageNet and associated five shifts. In Table 2, our Frolic again surpasses the comparison methods, achieving the average accuracy of 64 . 4% and 75 . 1% with ViT-B/16 and ViT-L/14, respectively. Additionally, we observe improvements on the distribution shift datasets: IN-V2, INSketch, IN-A, and IN-R with ViT-B/16, and on IN-A and IN-R with ViT-L/14, when our Frolic is combined with InMaP. However, these results still lag behind the original performance of our Frolic. This discrepancy may stem from the hyper-parameters in InMaP being optimized specifically for ImageNet; applying them unchanged to its shifted datasets could lead to over-fitting.

4.3 Ablation Studies and Further Analysis

Effectiveness of the prompt distribution learning. In Table 3 (Row (1) & (3)), we compare the performance of the original CLIP model f c with our prompt distribution learning model f g . We observe that modeling the underlying distribution of the text prototypes results in notable performance gains. For example, 3 . 7% accuracy improvement on 10-datasets using ViT-B/16.

Effectiveness of the prediction fusion. As described in Eq.(10), our Frolic fuses the original CLIP f c and the prompt distribution learning model f g via confidence matching. We compare the simple fusion f c + f g and our adaptive fusion f f = f c /τ c + f g /τ g in Table 3 (Row (4) & (5)). We show that our fusion technique outperforms the simple fusion by a large margin. Recall that our adaptive fusion method addresses situations where f g is more overconfident than f c . In Figure 3, we illustrate the relationship between performance gains over simple fusioni.e. , Acc ( f c /τ c + f g /τ g ) -Acc ( f c + f g ) -and the confidence differencei.e. , | conf ( f g , 1) -conf ( f c , τ c ) | . We present this as a scatter plot where each point represents a dataset, and we have fitted these points with a line. As expected, larger confidence differences correlate with more significant improvements.

Effectiveness of the bias correction. Row (2) and (6) in Table 3 demonstrate the effectiveness of our debiasing method, which can further improve the base CLIP model f c and the fusion model f d

0.1

0.2

0.3

0.4

0.5

0.6

|conf(

f

g, 1)

conf(

f

c

,

c

)|

0.0

2.5

5.0

7.5

10.0

Fitted line

Different datasets

0

5

10

15

20

Iteration

70.1

70.3

70.5

70.7

70.9

Accuracy (%)

0.00

0.25

0.50

0.75

1.00

1 Error

across various backbones and datasets. We also compare our debiasing method with other label bias correction methods in Table 4. The descriptions of TDE [34] and the Implicit method can be found in Section 3.4. The results reveal that TDE [34] does not consistently perform well across all datasets. In contrast, while the implicit method using downstream data enhances zero-shot performance, it underperforms compared to our debiasing method, which shows an average gain of 1 . 6% over the implicit method. To further assess our method's potential, we replaced pseudo-labeling with ground truth labels. The results reveal that the maximum achievable accuracy surpasses our method by 1 . 0% , highlighting the importance of our iterative approach for more accurate pseudo-labeling.

Convergence of Algorithm 1. Our method Frolic, as described in Algorithm 2, iteratively solves for the prior β . In Figure 4, we examine the convergence by displaying the errors ℓ 1 = ∥ β t -β t -1 ∥ 1 and the accuracy across iterations. We find that the resultant accuracy saturates after only 6 steps, and the relative ℓ 1 error decreases to less than ϵ = 0 . 01 after 10 steps.

Comparison with other prompt-based methods. The popular prompt-based methods, such as CoOp [46] and CoCoOp [45], require a training procedure with labeled samples while our method does not involve any training. To ensure a fair comparison, we compare our Frolic with CoOp and CoCoOp on across-dataset results, where the CoOp and CoCoOp are trained only with the labeled samples from the ImageNet dataset and then directly tested on the remaining datasets. The results shown in Table 5 demonstrate that our Frolic not only avoids the complexities of training but also exhibits superior generalization performance compared to these methods.

Comparison with adapter-based methods. The adapter-based methods, e.g. , LFA [24] and TipAdapter [44] boost the CLIP's generalization using labeled training samples. In contrast, our Frolic doesn't require any labeled samples. We evaluate our method with LFA and Tip-Adapert on the

ImageNet and its variants dataset, where the LFA and Tip-Adapter only utilize the labeled samples from the ImageNet dataset. The results in Table 6 show that our method achieves the best performance across all datasets with nearly 3% improvements in averaged accuracy over LFA.

Running time. Our method Frolic is completely trainingfree, unlike prompt tuning approaches such as TPT [32] and TDA [15], which involve back-propagating through an expensive encoder during optimization. We assess the wall-clock time of Frolic, TPT, and TDA in Table 7, using the CLIP ViT-B/16 model on ImageNet. These evaluations are conducted on a single NVIDIA A100 GPU. The results indicate that our method not only requires less time but also delivers superior performance.

5 Societal Impact, Limitation and Conclusion

Societal impact and limitation. Models pre-trained on large-scale web-crawled datasets may incorporate knowledge from noisy or malicious samples.

Limitation. Our approach assumes that the feature representations follow a mixture of Gaussian; however, this assumption may not always hold. On the other hand, the quality and distribution of data used in pre-training can significantly impact the performance of pre-trained models. Our method relies on the capabilities of pre-trained models for downstream tasks, if the pre-trained knowledge differs from the downstream tasks, the efficacy of our method may be limited.

Conclusion. In this work, we propose labelF ree p ro mpt distribution l earning and b i as c orrection, dubbed as Frolic , framework to boost the performance of zero-shot models. Our Frolic models each class prototype via a Gaussian distribution and fuses the learned model with the original CLIP [29] via confidence matching. The proposed framework further effectively removes the label bias without accessing to the pre-training data. Extensive experiments across various datasets demonstrate the effectiveness of our approach.

Acknowledgments and Disclosure of Funding

The work is supported by the National Natural Science Foundation of China (Grants No. 62202439), and the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-01-002). This work is also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.

References

James Urquhart Allingham, Jie Ren, Michael W. Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In ICML , 2023.

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS , 2019.

Christopher M Bishop. Pattern recognition and machine learning. Springer , 2:200-202.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In ECCV , 2014.

Hao Chen, Bhiksha Raj, Xing Xie, and Jindong Wang. On catastrophic inheritance of large foundation models. arXiv preprint arXiv:2402.01909 , 2024.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR , 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops , 2004.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML , 2017.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens. , 12(7):2217-2226, 2019.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV , 2021.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. 2021.

Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In CVPR , 2021.

Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR , 2024.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops , 2013.

Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In UAI , 2022.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR , 2022.

Rachel Luo, Shengjia Zhao, Jiaming Song, Jonathan Kuck, Stefano Ermon, and Silvio Savarese. Privacy preserving recalibration under domain shift. CoRR , abs/2008.09643, 2020.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. CoRR , abs/1306.5151, 2013.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR , 2021.

RVMises and Hilda Pollaczek-Geiringer. Praktische verfahren der gleichungsauflösung. ZAMMJournal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik , 9(1):58-77, 1929.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP , 2008.

Yassine Ouali, Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision-language models. CoRR , abs/2304.01752, 2023.

Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails of vision-language models. arXiv preprint arXiv:2401.12425 , 2024.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR , 2012.

Sarah M. Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV , 2023.

Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual categorization with CLIP. In NeurIPS , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML , 2019.

Jameel Abdul Samadh, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Shahbaz Khan, and Salman H. Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In NeurIPS , 2023.

Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS , 2022.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR , abs/1212.0402, 2012.

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurIPS , 2020.

Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models. CoRR , abs/2211.16198, 2022.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506-10518, 2019.

Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels. In CVPR , 2022.

Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free CLIP-based adaptation. In ICLR , 2024.

Zhicai Wang, Yanbin Hao, Tingting Mu, Ouxiang Li, Shuo Wang, and Xiangnan He. Bidirectional distribution alignment for transductive zero-shot learning. In CVPR , 2023.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR , 2022.

Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang, and Jingdong Wang. Gpt4vis: What can GPT-4 do for zero-shot visual recognition? CoRR , abs/2311.15732, 2023.

Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010.

Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, Joo-Hwee Lim, and Hanwang Zhang. Invariant training 2d-3d joint hard samples for few-shot point cloud recognition. In ICCV , 2023.

Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of CLIP for few-shot classification. In ECCV , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 130(9):2337-2348, 2022.

Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV , 2023.

Beier Zhu, Yulei Niu, Xian-Sheng Hua, and Hanwang Zhang. Cross-domain empirical risk minimization for unbiased long-tailed classification. In AAAI , 2022.

Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In NeurIPS , 2023.

Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision-language subspace projection for few-shot CLIP. In ACM MM , 2024.

Contents

A Theoretical Analysis

A.1 Proof of Eq. (6) : Estimation of Class Covariance from Marginal Second Order Moment

We first derive the second order moments for a multivariate Gaussian and then for a Gaussian mixture, corresponding to the marginal distribution of P ( x ) .

For a class j with parameters z j and Σ , the conditional probability density function is given by:

The second order moment generating function for class j is:

( a ) = holds as we change the integral variables y = x -z j . We have ( b ) = because the exp( · ) function is an even function of y and the factors yz ⊤ j and z j y ⊤ will vanish during integral by symmetry. For ( c ) = , we take the term z j z ⊤ j outside of the integral as they are constant.

The covariance matrix Σ and its inverse matrix Σ -1 the can be expressed through an expansion in terms of its eigenvalues { λ i } d i =1 and eigenvectors { u i } d i =1 :

Similarly, we can decompose y using the set of eigenvectors: y = ∑ d j =1 e j u j , where e j = u T j y . (We temporarily abuse the subscript j here. It does not represent class j until we reach Eq. (32)) We have the following expression:

̸

We obtain ( d ) = due to the property of eigenvalues, i.e. , u ⊤ i u i = 1 and u ⊤ i u j = 0 , for i = j . Denote U = [ u 1 , ..., u d ] ⊤ , we have e = U y . As the determinant | U | = 1 , the probability density after transformed remains unchanged: P ( e ) = | U | -1 P ( y ) = P ( y ) . Apply Eq. (25) and Eq. (26) into

Eq. (23), we have:

̸

For ( e ) = , the terms i = j disappear by symmetry similar to ( b ) = . We make use of | Σ | = ∏ d i =1 λ i for ( f ) = . We have ( g ) = because we regard e i ∼ N (0 , √ λ i ) and note that E [ e 2 i ] = var [ e i ] + E [ e i ] 2 = λ i +0 = λ i . Combining Eq. (32) with Eq. (23), we have the second order moment for class j is:

Using a Gaussian mixture model with the priors { π j } K j , P ( x ) is given by:

The second order moment for the marginal distribution P ( x ) is:

A.2 Proof of Eq. (5) : Estimation of the Priors of Gaussian Mixture Models

The expectation of x is defined as:

Denote π = [ π 1 , .., π K ] ⊤ , Z = [ z 1 , .., z K ] T , and the expectation of x as µ , Eq. (39) can be rewrite as:

Therefore, the priors can be solve by π = Z -1 µ .

A.3 Proof of Eq. (7) : Parameters of our Learned Model

The posterior of classes P ( y | x ) can be expression as:

To classify x , we seek the class y that maximizes this posterior. Since the term P ( x ) does not depend on y , we can simplify our task to maximizing N ( x ; µ y , Σ) π y . Taking natural logarithm gives:

The first term in Equation (43) is constant; we incorporate it using a constant c 1 . Consider that most test benchmarks are generally class-balanced, we use a uniform prior c 2 to incorporate ln π y . In Eq. (45), we use c to absorb all constant terms, including c 1 , c 2 and -1 2 x T Σ -1 x . Let w j = ˆ Σ -1 z j and b j = -1 2 z ⊤ j w j , we get Eq. (46).

A.4 Proof of Eq. (11) : Debiased Classifier for Downstream Data

Proposition 1. (Modified from Theorem 1 in [14]). Let P pt ( y | x ) and P ds ( y | x ) be the distributions of the pre-train and downstream data, respectively. Let β y = P pt ( y ) and π y = P ds ( y ) denote the priors of the pre-train and the downstream data, respectively. Assume the likelihood P ( x | y ) is unchanged between pre-train and downstream data, i.e. , P ( x | y ) = P pt ( x | y ) = P ds ( x | y ) . If f pt ( x ) y is the logit of class y from the softmax model to estimate P pt ( y | x ) , then the estimated P ds ( y | x ) is formulated as:

where β = [ β 1 , ..., β K ] and π = [ π 1 , ..., π K ] .

Proof.

For ( a ) = , we denote the term that is not related to y as 1 Z = P pt ( x ) P ds ( x ) . We derive ( b ) = from the requirement that P ds ( y | x ) , being a probability, must sum to 1 across all possible classes y ∈ [ K ] :

Therefore, we have Z ∑ K j =1 exp( f pt ( x ) j ) = ∑ K i =1 exp( f pt ( x ) i -ln β i +ln π i ) . In our context, the pre-trained model f pt is equivalent to our f f .

A.5 Proof of Eq. (12) : Equation to Estimate Pre-training Priors

Proposition 2. Let s ( x ) = [ P ( Y = 1 | x ) , ..., P ( Y = K | x )] ⊤ ∈ R K be the likelihood vector, s j = E x | Y = j [ s ( x )] and S = [ s 1 , ..., s K ] ∈ R K × K . The pretraining prior β = [ β 1 , ..., β K ] ⊤ ∈ R K must satisfy the linear system:

Proof.

Note that Equation (61) precisely represents the matrix multiplication given by:

By moving the RHS term to the LHS, Eq. (56) is obtained.

A.6 Power Method to Estimate Pretraining Priors

The solution to Equation (62) involves finding the eigenvector corresponding to the eigenvalue of 1 for the matrix S . We can apply SVD decomposition to find the solution; however, we find that the results might be numerically unstable. Instead, we adopt power iteration from [22]. Like the Jacobi and Gauss-Seidel methods, the power method for approximating eigenvalues is iterative. We first initialize β 0 = [ 1 K , ..., 1 K ] of a uniform distribution. Then, we perform the sequence:

We repeat the sequence until the relative change is small: ∥ β t -β t -1 ∥ < ϵ .

B Details of ImageNet Variant Datasets

ImageNet-V2 [30]: sampling from the original ImageNet and including 10,000 images of 1,000 ImageNet categories.

ImageNet Sketch [36]: including 138 50,000 images and covering 1,000 ImageNet categories.

ImageNet-R [12]: containing renditions ( e.g. , art, cartoons, graffiti) for ImageNet classes, comprising 30,000 images from 200 ImageNet categories.

ImageNet-A [13]: collecting real-world images that are misclassified by ResNet-50, totaling 7,500 images from 200 of ImageNet categories.

ObjectNet: [2] including 50,000 test images with rotation, background, and viewpoint, and overlapping 113 classes with ImageNet.

C Licenses

All the datasets we considered are publicly available, we list their licences and URLs as follows:

Caltech101 [9]: Non-commercial, https://data.caltech.edu/records/mzrjq-6wc02 .

OxfordPets [26]: MIT License, https://www.robots.ox.ac.uk/~vgg/data/pets/ .

StanfordCars [16]: MIT License, https://www.kaggle.com/datasets/ jessicali9530/stanford-cars-dataset

Flowers102 [23]: MIT License, https://www.robots.ox.ac.uk/~vgg/data/flowers/ 102/index.html .

Food101 [4]: Non-commercial, https://data.vision.ee.ethz.ch/cvl/datasets_ extra/food-101/

FGVCAircraft [20]: Non-commercial, https://www.robots.ox.ac.uk/~vgg/data/ fgvc-aircraft/ .

SUN397 [42]: Non-commercial, https://vision.princeton.edu/projects/2010/ SUN/ .

DTD [7]: Non-commercial, https://www.robots.ox.ac.uk/~vgg/data/dtd/

EuroSAT [11]: MIT License, https://github.com/phelber/EuroSAT

UCF101 [33]: Non-commercial, https://www.crcv.ucf.edu/data/UCF101.php

ImageNet [8]: Non-commercial, http://image-net.org .

ImageNetV2 [30]: MIT License, https://github.com/modestyachts/ImageNetV2 .

ImageNet-R [12]: MIT License, https://github.com/hendrycks/imagenet-r .

ImageNet-Sketch [36]: MIT License, https://github.com/HaohanWang/ ImageNet-Sketch .

ImageNet-A [13]: MIT License, https://github.com/hendrycks/ natural-adv-examples .

ObjectNet [2]: Creative Commons Attribution 4.0, https://objectnet.dev .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We point out that existing prompt learning methods can not capture the various image representations and ignore the label bias in CLIP. We propose Frolic to address these problems. Experiments on various datasets demonstrate the effectiveness of our method.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations have been discussed in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the proof related to our method in Section A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the Frolic Algorithm in Algorithm 1 and Algorithm 2, and included the implementation details in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have uploaded the codes in supplemental materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have introduced the testing details in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Given the zero-shot models, the process of our method is deterministic. Running multiple times will not introduce randomness.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have introduced the GPU type to reproduce results in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the societal impact on Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method aims to improve the generalization ability of zero-shot models, which poses no such risks to the best of our knowledge.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided the licenses of each dataset in Section C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a method named Frolic to improve the zero-shot performance of vision-language models like CLIP. The method focuses on two key challenges: enhancing prompt distribution learning and correcting inherent label bias in pre-trained models without relying on labeled data. Experimental results across 16 datasets demonstrate performance improvements of the method.

STRENGTHS:
+ The paper is well-written and the method is easy to follow.
+ The method effectively addresses label bias inherent in pre-trained models, improving the robustness and accuracy of zero-shot predictions.
+ Ablation experiments are comprehensive.

WEAKNESSES:
- Does the paper utilize the validation dataset in experiments, if not, this should be clarified.
- How does the proposed method compare with the methods[1][2] in the few-shot settings?
- Although Frolic is described as training-free, there may still be hyperparameters involved in the method, and how to choose these parameters?
- How does the prompt distribution learning contribute to the performance? Does it outperform the original CLIP?
- Can the method be performed with a few labeled samples?  
[a] Black box few-shot adaptation for vision-language models. ICCV 2023
[b] Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification. ECCV, 2022

QUESTIONS:
The discussion of the limitations of the methods is unclear. The authors should discuss it in depth.

LIMITATIONS:
Please refer to the weakness and Questions parts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-06-21

comments: SUMMARY:
This work aims to enhance the zero-shot performance of pre-trained vision-language models. Specifically, three strategies, including label-free prompt distribution learning, adaptive calibration and correction pre-training label bias, are proposed and work together to improve the performance on different downstream tasks. Experiments on diverse tasks confirm the effectiveness of the proposed method.

STRENGTHS:
1.	Enhancing the zero-shot performance has attracted much attention recently. This work discusses multiple challenges in this process and proposes corresponding algorithms with theoretical analysis.
2.	The proposed method is training-free without hyper-parameters, which is applicable to real applications.
3.	Diverse downstream tasks are included for evaluating the performance of the proposed method.
4.	The manuscript is well-written and easy to follow.

WEAKNESSES:
1.	The work works with a uniform prior that classes are balanced distributed. It is better to discuss the scenario with imbalanced prior.
2.	Besides CLIP, there are many other vision-language pre-trained models, e.g., BLIP, etc. It is better to include other pre-trained models to evaluate the performance of the proposed method.
3.	Why does $e_i$ follow a Gaussian distribution in L446?

QUESTIONS:
I do not have critical questions about this work.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-06

comments: SUMMARY:
In this paper, the authors introduce a promising method for enhancing zero-shot vision models through the utilization of prompt distribution learning and bias correction. The method is particularly notable for being training-free and label-free, which greatly simplifies the implementation process. The authors provide comprehensive experimental results, which effectively demonstrate the efficacy of the proposed method.

STRENGTHS:
1. This paper is easy to understand.
2. The paper provides an in-depth theoretical discussion and a thorough experimental evaluation demonstrating the effectiveness of the proposed method.

WEAKNESSES:
1. The computation of second-order moments and the covariance matrix from the marginal distribution as discussed in Eq.(3) and (5) might be computationally intensive.
2. The method relies on the pseudo-labels estimated by CLIP. How does this process influence the final results?
3. The setting of results on the ImageNet variants dataset is unclear. 
4. The results lack comparison with other prompt distribution methods, such as CoOp and CoCoOp.

QUESTIONS:
The author stated that their method does not require hyperparameter tuning. How does the proposed method compare to those requiring hyperparameter tuning?

LIMITATIONS:
Refer to the weakness and Questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Frolic, a label-free prompt learning methods aiming to improve zero-shot visual recognition of vision-language models like CLIP.  The method is built upon estimating distributions over prompt prototypes to capture diverse visual representations and further bias correction. Experiment results demonstrate consistent improvement on various benchmarks over prior-art methods.\n\nSTRENGTHS:\n1. The approach does not require access to large-scale datasets for estimation, which is required by many prior work. \n2. The approach advances previous distribution estimation methods by removing the need of label information, which made this line of work possible for zero-shot recognition tasks.\n\nWEAKNESSES:\nWhile this approach does not need access to large-scale pretraining data, it seems to make some weak assumptions on downstream tasks. If my understanding is correct, it assumes:\n1. the downstream task is balanced (line 134), which is not always hold true in reality. The long-tail nature of real world does not guarantee that the testing class distribution will be balanced even though the benchmarks do. \n2. A decent scale of the testing data on downstream tasks is available (e.g, testing data does not come in online fashion like one sample at a time) for estimating the beta term in bias correction. Would the approach for bias estimation still holds true in online testing scenario? I would assume this is more aligned with reality too.\n\nQUESTIONS:\n1. The methods did not seem to evaluate on all benchmark datasets like previous works do. For example, CIFAR10/100 and RESISC. While  some of these datasets have saturated performance, it might be still helpful to include results there. \n2. It's not clear to me why [24] is not listed in tables for comparison as I would assume 24 to be the most direct baseline. It shares the same motivation of this paper with the requirement of accessing pretraining data. Arguably, while LAION has been taken back, [24] already does the job for us. I would not think an argument like \"[24] requires large-scale pretraining data for estimating bias thus we do not compare with it\" as a valid argument because this zero-shot visual recognition task itself is solely working with CLIP-ish models and estimating bias from the datasets like LAION sounds like a fair practice to me.\n\nLIMITATIONS:\nAuthors have included such discussion", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}