Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
Authors: Xingyu Zhu
Beier Zhu
Yi Tan
Shuo Wang
Yanbin Hao
Hanwang Zhang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: vision-language model
zero-shot classification
logit adjustment
Primary area: machine_vision

Abstract: Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. 
 To tackle the above challenges, we propose a label-**F**ree p**ro**mpt distribution **l**earning and b**i**as **c**orrection framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching.
This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of $2.6\%$ on 10 datasets with CLIP ViT-B/16 and achieving an average margin of $1.5\%$ on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in [https://github.com/zhuhsingyuu/Frolic](https://github.com/zhuhsingyuu/Frolic).

Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting
Xingyu Z
Beier Zhu
Yi Tan' Shuo Wanglf Yanbin Hao
Hanwang Zhang?
University of Science and Technology of China
Nanyang Technological University
xingyuzhuumail.uum ustt www.:uttc.cuuuuuuuooooooo cn, shuowang www.wangeeddggail....
Abstract
Vision-language models, such as CLIP, have shown impressive generalization ca- pacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective improving performance, these methods entail labor costs for annotations and are limited by their quality. Addition- ally, since CLIPP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above chal- lenges, we propose label-Frec prompt distribution learning and bias correction framework, dubbed as Frolic, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt pro- totypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model further enhanced by correcting label bias via label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of approach, particularly outperforming the state-of-the-arr by an average of 2.6% 10 datasets with CLIP V and achieving an average margin of 1.5% on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available /ttttts://gtthiiee com/ itgg/hhuusnguuu//ol...
Introduction
Vision-language models (VLMs), such as CLIP 29 which are pre-trained on large-scale datasets using contrastive loss, effectively align visual and textual representations within a shared feature space. This capability enables the zero-shot inference on downstream tasks through prompting and achieves remarkable performance. For example, using selection of 80 hand-crafted prompts, zero-shot CLIP ViT-B/16 achieves an accuracy 68.7%, and with prompts generated by language models [27]. the accuracy increases to 69.9%.
The success of zero-shot capabilities heavily relies on the appropriate text descriptions of the classes, which has gained research interest in improving prompts. Recent studies propose learning prompts Among these studies, Lu et al. [18] and Wang et al. [38 have found that learning the distribution of diverse prompts, which better captures the variance in visual representations, leads improved performance. Although these methods have achieved significant improvements, they still depend on artificial prior knowledge for labeling downstream data and are limited by the quality of manual annotations, which may restrict the scalability of the original model.
from small set of labeled images in the downstream data 46 45
"Equal contributions Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

dieciion
Our decivion boundirry
225"10"
Prdd 00e Phot
2.0
1..5
52.5
1..0-
3.0
Nr 0ks
5..3
00
6.8
230
2.00
1000
Soried Cleas Indler
Sorted Class Indee:
(a)
(b)
(c)
(d)
Figure I: Illustration of prompt distribution learning and label bias correction on ImageNet using CLIP ViT-B/16. (a) Existing zero-shot models 1 [7 (b) Our prompt distribution learning (c)
Average probability prediction of original CLIP. (d) Average probability prediction of our Frolic.
Another significant approach to enhancing zero-shot performance involves correcting the label bias inherent in skewed web-scale pre-training data 25] 49]. This bias leads to highly imbalanced pre- dictions and suboptimal performance. As illustrated in Tigure[c) the average predicted probability on ImageNet using ViT-B/16 reveals an imbalanced distribution: the highest class probability exceeds 0.002, whereas the lowest is below 0.000 0.0005. Existing methods correct this bias by allowing access to a portion the pre-tree pre-training data I [25 or by using labeled downstream data [49]. However, the
pre- mre-tranning data is often inaccessible due to privacy copyright concerns, and debiasing without labeled data is challenging.
In this paper, we introduce a label-Free prompt distribution learning and bias correction framework, dubbed as Frolic, which eliminates the need for data annotations to enhance zero-shot performance.
First, unlike previous methods [1] 27 46 39 43 which use single class prototype for each class to define the decision boundary (as shown Figure Ta)), our approach employs Gaussian distributions to model the varied visual representations of text prototypes, as illustrated in Figure [b). It is worth noting that estimating such a distribution is non-trivial, since classical maximum Tikelihood estimation requires the annotation of each sample. Fortunately, infer distribution for each class directly from the first and second moments the marginal distribution of downstream data without label information. Second, to prevent the use of pre-training data or labeled samples in downstream tasks, we develop a bias estimation mechanism, which transitions the sampling process from the pre-training data distribution downstream distribution. By incorporating the estimated label bias into zero-shot models, we can achieve balanced prediction, as illustrated in Figure | combining the original CLIP predictions with those from the Gaussian-based models to enhance zero- manfidence-matching technique that dynamically
demonstrate that it is possible to
class-conditional sampling from
(d). Furthermore, we explore the possibility of
shot performance. To this end, we have developed balances the contributions of the two models, eliminating the need for hyperparameter tuning. Notably, our framework training- free, which enhances both flexibility and ease of implementation.
The main contributions this work are:
We enhance zero-shot performance by estimating distribution over prompt prototypes to capture the variance visual appearances. We demonstrate that this process be implemented entirely without labels.
We propose a confidence matching technique that fuses the original CLIP model with Gaussian distribution-baat based model further enhance zero-shot performance. This process eliminates the need for parre--rrreeter searching, in stark contrast to previous studies.
We develop an unsupervised method to correct pre- pre-traing teeirrainig label bias. Unlike existing methods that require access to pre-training data, our Proposition] 2 suggests that we can avoid sampling from the pre-training distribution for estimating and correcting this bias. Instead, our method utilizes only downstream images.
We demonstrate the effectiveness our proposed method Frolic by conducting experiments across 16 datasets, which has consistent and significant improvement over existing baselines. For example, our method surpasses the state-of-tth zero-shot models by a margin of 2. 2.6% average with CLIP VIT-B/16.

---

2 Related Works
Zero-shot vision models. Vision models pre-trained with auxiliary language supervision, such as CLIP [29] and OpenCLIP [6]. facilitate zero-shot inference through prompting. Enhancing zero-shot performance has gained increasing research interest: (1) One approach involves prompt engineering, which includes designing hand-crafted prompts based on human priors [29] or automatically generat- ing prompts via language models [35] (2) Another promising approach seeks to improve classifiers, e.g. ZPE [ scores the importance of candidate prompts for prompt ensembling. InMaP (28) reduces the modality gap between vision and text. Several studies [2] 31 optimize prompt at test time by encouraging consistent predictions across augmented samples. Our work aims to enhance zero-shot models by learning the prompt distribution and mitigating the pre-training label bias.
Prompt distribution learning. Automatically learning prompts from downstream data has shown potential in improving zero-shot models [46] 45 47 These methods typically optimize prompts via minimizing the classification loss on the target task. However, as pointed out Lu et al. [18], learning prototype prompts overlook the diversity of visual representations. To this end, they estimate a distribution over the prompts to capture the variance of visual representations. Recently, Wang er al. (38 propose training- free prompt distribution learning to improve efficiency. Contrary existing methods [18] that estimate distributions through supervised approaches, our method circumvents the necessity for labels by inferring the variance of distributions from the statisties of unlabeled data.
Correcting label bias. Label bias generally occurs the presence skewed or imbalanced training data. In response to this challenge. Logit Adjustment (LA) R41
49] has emerged as prominent technique in long-tailed learning, specifically designed to adjust the decision boundary of classifiers to mitigate label bias. Menon al. [2I] derives the theoretically optimal adjustment for logits. Zhu et al. [49] extents LA to fine-tune zero-shot models by removing the pre- pre-trained label bias. Unlike approaches that rely on the label distribution of the training set [34] [14] [1] (88 the
labels of fine-tuning data 449 our method adjusts the logits using unlabeled test data.
3 Methods
In this section, present our prompt distribution learning, adaptive fusion, and logit adjustment techniques for adapting zero-shot models. Without loss of generality, we adopt CLIP 29 as our zero-shot model. To begin with, emphasize three advantages of our framework:
Training-freet Our Frolic is training-free without optimizing the backbone of the zero-shot models, enhancing both flexibility and ease of implementation.
Label-free: Our method Frolic requires external labeled data, making it suitable for zero-shot scenarios.
No hyper-parameter searching: Our method Frolic eliminates hyper- typer-parameter tuning valida- tion datasets, in stark contrast to 38
3.1 Setup
The zero-shot model consists of a visual encoder (.(.) and a text encoder d.(-). Given a set of
unlabeled image data { and the unique text set of the class description {2y {2 their visual and text representation can be computed as:
x; = (x:); 2, = Be(2))
where x; and share the same dimension (x, E Rd).. N is the sample size and is the class size. zy can be considered as the prototype for class j. With an image x and all prototypes (z, }/ zero-shot CLIP predicts the label
y argmax argmax fc(x), argmaxz x,
(2)
where fc(x), 22, x is the score for class
3

---

3.2 Label-Free Prompt Distribution Learning
In order to express the diverse visual variations, our approach aims learn the distribution of the class prototypes. Previous studies [T8] (38] show that the Gaussian distribution is effective to model the distribution of the CLIP features and achieves impressive improvement. However, these methods require extra labeled training data, which is not applicable to our zero-shot setting.
Specifically, we follow 38 assume ((z1.K Σ) with identical covariance is the underlying distribution. In classical maximum likelihood estimation [ the shared covariance E is computed by averaging the empirical covariances of K classes: & = K Σ, E.j where Σ c Exec, (x
2, (x) zz)T. Here, one need the label information of each image to compute Σ, Fortunately, to avoid using label information, we can infer Σ directly from the expectation and the second order moment of the marginal distribution [P(x) Using a Gaussian mixture model with the priors {Ty) P(x) is given by:
P(x) Σ NyN(x E), N(
(2 exp( 7(x
2,) E-> 2,)3 (3)
Denote the second moment of x as M,, we have (proof in Section A.I.
M = E+ Σ T,
(4)
Denote π =
"k. = [,,,.., 2k and the expectation of x as μu, the prior over the unlabeled data distribution can be estimated by (proof in Section A.2
π 2
(5)
We estimate the expectation and the second order moment as j N. Σm= x; and M 1 Ei= x,xT, which are unbiased and consistent. In practice, given that test benchmarks are generally class-balanced, we use uniform prior over the data distribution, i.e., TT = K Combining with Eq. (4). the estimated shared covariance Σ be written as:
N
E = M-
K Σ 2z2..
Let w, = = z, and by = - w, the Gaussian discriminant analysis predicts the label for an image x as follows (proof in Section A.3):
y = argmax fx(x); = argmax w. x + by
where fg(x)) W/Ix x + by is the score for class
3.3 Prediction Fusion via Adaptive Calibration.
As rule of thumb, combining the zero-shot predictions with the ones from the learned model can further improve performance for CLIP adaptations 44 35 40
Imagetet
Calteh
38 50)
SUN
Previous studies commonly employ mixing coefficient, a, to balance the contributions two models, e.g., fx) =
Cars
EuesAT
fc(x) + (x). Typically, this hyper-yaramene parameter is op- timized on labeled data to maximize accuracy. However, in our context, labels are unavailable, it is not possible to search for the optimal value of a. is imperative to develop mechanism that balances the prediction fusion without relying on the label.
DTD
Airrott
Nower
0.6
0'a
1.
Confidence
Figure Comparison of confidence.
Despite that the modality gap exists between the text and vision space of CLIP models, we can use the unsupervised method from InMaP 28 effectively align the two modalities.

---

Algorithm/ Pipeline of our Frolic
Algorithm 2 Estimation of /
1: Given: Unlabeled data {x:]i=>>
1: Given: Unlabeled data (x: J
prototypes {zj}}= and Tc
predictor f+(-) and tolerance c.
2: Initialize , and 5 by Eq. 13)
2: Build fe(x)y = 2,
3: = 0
3: Compute Σ = .M K Ej 2
4: repeat
where M N Exxix
5:
t=
4: Compute w, = E -zZ bj =
- 1/2 ,
6: Update 3 by solving (5 11,3'
7: Update få = ft 3
5: Build (x), = w x + by
6: Search Tg by Eq. (
8:
Update S from s)
1
Exec s(x),
7: Build [+( = [x(x)/TT + fx(x)/Te
where Cj is assigned by Så
8: Compute 3 by Algorithm
9: until 1133 11 -3111112 <
9: return fa(x) = fx() Inf
10: return 3 = B
The key in our prediction fusion lies in aligning the average confidence of the two models. Formally, the average confidence over the dataset {x scaled by a temperature 7 is given by the average of the model's probability for its prediction:
conf(J,7) = N Σ max sollmaxx(((((x)))
(8)
Ideally, a model's average confidence should reflect the predicted accuracy, which called a well- calibrated model. Suppose we have the oracle well-calibrated models, denoted by f((+) and fg(.),
Kumart et al. [17] prove that the optimal strategy is to fuse the two predictions equally, Le., f((x) = ff However, as shown in Figure 2 fg is much overconfident than fc. Let fg(xx) (x)())
for large C E R (an overconfident model magnifies its logits) and suppose fe(x) x fx)x.. The fused predictions are given by fi(x) Cx fx(x.. For very large C, fi(x) and fg (x) have the same predictions, i.e., ff(x) is biased towards the (x). As we do not have the label to compute accuracy, we cannot apply classical calibration methods [19 TO] to calibrate fg(x) and fc(x). As our desideratum is automatically balance the contribution of the two models, we can optimize Tg to make the confidence of fg to match up the one of fc. which circumvent the need of labels:
Tg argmin conf([s: Tg) conf(fe, re)
Specifically, we implement this by binary search, as the confidence monotonically decreases as the temperature increases. T, 0.01 is fixed and learned by CLIP. The fused logits are given by:
f+(x) fx(x)/TE + fe(x)/Te
(10)
3.4 Correcting Pre-training Label Bias via Label-Free Logit Adjustment
Pre-training datasets typically exhibit long-tailed concept distribution, leading to biased performance
5 1. This bias occurs because zero-shot models reflect the posterior probability PP(y) x) derived from the pre-training distribution. According to Bayes' rule, this posterior probability influenced by the pre-training label distribution P(y), as P(ylx) (P()).) If the prior probability of class j is significantly larger than that of other classes (e.g., P()) > P(1), Vi E [ + f), the predictions will be biased toward class
in zero-shot models [49 25
Prior research 21 14 has identified a theoretical optimal solution to address this label bias: let B, denote the prior probability of class y. i.e., B, P(y). The debiased logit of fx(x) for class y should be (proof in Section A.4):
fa = fx(,,, In
(11)
Previous methods estimate 3 either by accessing the pre-training data 25 1 or counteract the influence of the prior by optimizing on labeled downstream data [49]. However, these approaches are often impractical due to inaccessible pre-training labels due to privacy copyright concerns the
5

---

necessity for labeled downstream data. In this work, we address label bias using only the unlabeled downstream data {=,
Let s(x) = softmax (fr represent the softmax outputs of fx(),, where s(x), = P(yl x) is the predicted probability for class y. Define s, = Ex[s(x)]Y = as the expected posterior probability over the image distribution of class j. and let S = $1...., sk] E I R We prove that the pre-training label prior B = [31,.... 3x] E TR& must satisfy the following linear equation system:
(S - 3)3 = 0.
(12)
Remark. The key point in Eq. 12 is that avoid sampling from the pre-training data distribution; instead, we sample from (x((yy) which is available from the downstream data. We provide the proof in Section A.3 A.5]and and the numerical power solver for ,3 in Section A.6
We iteratively refine the estimation of S and solve for ,3 using updated pseudo-labed generated by fx(x.. As (x) becomes more precise, it yields more accurate pseudo-labels for x, which in turn enhances the accuracy of our estimation of B3 Specifically, we initialize
8 = [L/K, 1/K]T, Sa = fr. 5/
Σ XECY
s(x), and S = [s...... sk1
(13)
C99
where x E C9 if x is classified as j by fx)). We proceed by solving for 3 using Eq.
12 refining
(x) using Eq 11 and reassign the pseudo label using få (x) to estimate the updated s.. This process
repeated times until the relative change in ,3 satisfies the convergence criterion:
18 3 3
1/3 3 <,,
|| by definition
(14)
where is predefined threshold for relative error tolerance. We summarize the algorithm for solving (3 in Algorithm [ and provide the overall pipeline in Algorithm |
Discussion: Comparison with Other Prior Estimation Methods. We compare existing methods for estimating pre-training label priors and demonstrate their in-applicability or flaws in our setting.
(1) Explicit method: the explicit method directly measures the frequency of each class in pre-training data, e.g., B3 N₂ where Ng is the sample size for class and N is the total sample size. Most toig-tail learning algorithms, e.g., LA and PC [2], are based on this method because they can access the training data. However, estimating such frequency is complex due to the free-form texts, as opposed to a pre-defined label set. In addition, the pre-traie vee-trainin dataset is often inaccessible, making the method inapplicable in our case.
(2) Implicit method: 2 allow access to portion of the pre-training data Dpt and use the law of total probability to estimate the prior:
By = P(y) (xx())((xxx) Ex)T (x
Σ Ppe())x)
(15)
[Dptl xED
where IP (Pr x) denotes the zero-shot model. However, in our setting, we do have access to the pre- training data or a portion of it. Wang et al. (37 replace the pre-training data Dpt with the downstream data Das in Eq. (15) to debias. However, this method neglects the distribution discrepancies between the pre-training and downstream data. In Section 4.3
we show that our debiasing significantly
outperforms this implicit method.
(3) TDE Tang et al. 34 debias by removing features along a global direction, retaining only 1
those orth to it. Specifically, the global feature is estimated by x = Dadl ΣxEP, x. Given a test sample x, TDE decomposes it into parallel and orthogonal directions to x = + X1. Then,
only the orthogonal component is used for classification: Ppc(y/__). While TDE does not require labels for the samples, we cannot apply it because it requires sampling from the pre-training data. In we replace 'Dpt with downstream data D&s and demonstrate its inferior performance.
Section 4.3
(4) GLA 49 Zhu et al. [49 propose to estimate the pre-training prior from the downstream data using the Bayes optimal criterion. The
prior (3 is solved by optimizing:
B = arg min 15 (x.)) (fo (x) - In B v)].
(16)

---

Table 1: Comparison of accuracy (%) on 10 datasets for CLIP ViT-B/16 and ViT-L/14.
Flowers
Aircraft
EuroSAT
Caltech
Pets
DTD
Cars
Food
SUN
UCF
Method
Average
CLIP
29
88.9
70.4
24.8
44.3
47.7
63.2
86.1
62.5
92.9
66.7
64.9
TPT
87.7
68.9
24.7
47.7
42.4
66.8
84.6
65.5
94.1
68.0
65.0
PromptAlige
90.7
72.3
24.8
47.2
47.8
86.6
67.5
94.0
69.4
66.8
SuS-X-SD R5
90.5
73.8
28.6
54.5
57.4
66.1
86.0
67.7
93.6
66.5
68.4
VIT-B/16
TDA [15
88.6
71.4
23,9
47.4
58.0
67.2
86.1
67.6
94.2
70.6
67.5
GPT4-Prompl
91.0
74.5
28.0
48.5
48.8
66.8
86.3
65.5
94.6
72.0
67.6
CuPL-CLIP
27
92.0
73.2
27.7
54.3
52.7
66.4
86.2
68.5
94.6
70.7
68.6
Frolic
92.9
74.8
31.5
56.1
58.5
69.1
87.2
70.8
95.2
75.2
71.1
InMaP 28
92.9
48.0
64.1
70.6
87.7
70.5
93.1
74.0
70.1
Frolic
93.6
74.3
31.8
58.0
65.3
71.7
88.2
72.8
95.4
75.9
72.7
CLIP
93.5
79.3
32.4
53.0
58.0
76.8
91.0
67.5
94.8
74.2
72.0
TPT
93.6
76.2
31.9
55.2
51.8
77.7
88.9
70.2
95.5
74.9
71.5
ViT-L/14
TDA 15
93.5
80.5
34.7
56.7
64.1
78.3
90.9
71.5
95.9
76.6
74.2
GPT4-Prompl
94.1
81.5
36.3
54.8
54.1
77.9
91.4
70.3
96.2
80.6
73.7
CuPL-CLIP
94.3
79.8
35.5
62.7
61.2
91.3
72.4
96.7
74.7
Frolic
94.9
82.4
40.0
64.1
66.2
80.8
91.8
74.5
97.2
80.0
77.1
InMaP 28
95.2
80.7
37.6
60.2
70.6
82.5
92.2
75.0
94.9
80.4
76.9
+ Frolic
95.4
81.8
42.1
66.9
71.0
83.5
92.4
77.3
97.3
82.2
78.9
where lee is the cross-entropy loss and (x) is the logit of the zero-shot model. While this method circumvents the need for pre-training data access, it inapplicable because it requires labels for each downstream sample.
4 Experiments
4.1
Setup
Datasets.
We conduct experiments image classification benchmarks, covering diverse range
egories including generic objects (ImageNet [8]. Caltech (9), scenes (SUN [42),, textures (DTD (7))
satellite images (AuroSAT (TT),, actions (UCF (33) and fine-grained categories (Pets (26). Cars [16]
Flowers 23 Food [4. Aircraft (20). Additionally, we evaluate on five ImageNet distribution shifted
datasets g ImageNetV2 (IN-V2) [30], ImageNet-Sketch (IN-Sketch) (36], ImageNet-A (IN-A) [3
ImageNet-R (IN-R) 12 and ObjectNet (2).
Implementation details. We adopt CLIP (2 VIT-B/16 and VIT-L/14 as pre-trained models. The
default model for ablation studies CLIP ViT-B/16. We use the same text descriptions as SuS-X [35 and CuPL (27), and adhere to the InMaP 23 settings include all test images. 0.01 is provided by CLIP. in Algorithm 2 set to 0.01. All experiments are conducted on a single NVIDIA 3090 GPU if not specified. Note that algorithm does require any hyper-parameter searching.
4.2 Main Results
We compare method with several state-of-art methods, including CLIP 29 TPT 32 PromptAl-
ign (3I) SuS-X-DS (35), TDA [15], GPT4-Prompt [41), CuPL-CLIP 27 and InMaP 28
Both TPT
and TDA utilize stream of unlabeled test images. For TPT, TDA and InMaP, we produce the results of ViT-L/14 by executing the official released code and maintaining the same hyper-parameters
Results on 10 datasets. In Table I we summarize the accuracy across all datasets, excluding ImageNet and its shifts (denoted as 10-datasets), Our method consistently shows superior performance across the datasets and backbones, significantly surpassing GPT4-Prompt, which is known for generating high-quality prompts. By integrating our method with InMaP, our Frolic achieves highest performance, with an average improvement of 2.6% with VIT-B/16 and 2.0% with ViT-L/14.

---

Table 2: Comparison of accuracy (%) on ImageNet and its variants for CLIP ViT-B/16 and ViT-L/14.
Method
IN
IN-V2
IN-Sketch
IN-A
IN-R
ObjectNet
Average
CL.IP
68.7
62.2
48.3
50.6
77.7
TPT 32
68.9
63.4
47.9
54.7
77.0
55.1
61.1
TDA 15
69.5
64.6
50.5
60.1
80.2
55.1
63.3
ViT-B/16
GPT4-Prompt
41
68.7
62.3
48.2
50.6
77.8
53.7
60.2
CuPL-CLIP
27
69.9
64.4
49.4
79.5
53.7
62.7
Frolic
70.9
64.7
53.3
60.4
80.7
56.6
64.4
InMaP
72.5
62.3
49.4
52.2
79.2
61.6
Frolic
73.3
63.8
52.9
52.8
79.6
56.4
63.1
CLIP 29
75.9
70.2
59.7
70.9
87.9
65.5
71.6
TPT 32
75.5
70.0
59.8
74.7
87.9
68.0
72.6
TDA 15
76.3
71.5
61.3
77.9
89.8
67.0
73.9
ViT-L/14
GPT4-Prompl CuPL--CLIP
41
75.3
70.3
59.9
71.2
87.8
65.7
71.7
27
76.2
71.9
60.7
77.9
89.6
73.6
Frolic
77.4
72.5
63.1
78.9
90.3
68.7
75.1
InMaP 28
79.3
72.1
65.1
62.5
84.8
71.0
72.4
Frolic
79.7
73.1
65.7
64.0
85.9
71.7
73.3
Table 3: Accuracy (%) of different models on 10-datasets, ImageNet and its five variant datasets.
VIT-B/16 ImageNet
VIT-L/14 ImageNet
Model
10-datasets
IN- Variants
10-datasets
IN- Variants
(1)
fe
65.1
68.7
58.5
72.0
75.9
72.3
(2)
fe 3
68.4
69.7
61.2
75.1
76.2
73.4
(3) fz
68.8
69.8
61.3
74.7
76.0
73.1
(4)
fc + fg
66.3
68.9
59.1
72.5
76.1
72.4
(5) ft Sc/Te S8/Te
70.4
69.8
61.9
75.5
76.9
73.9
(6) fa ft 1n
71.1
70.9
63.1
77.2
77.4
77.4
Results on ImageNet and associated five shifts. In Table 2 our Frolic again surpasses the com- parison methods, achieving the average accuracy of 64.4% and 75.1% with ViT-B/16 and ViT-L/14, respectively. Additionally, we observe improvements on the distribution shift datasets: IN-V2, IN- Sketch, IN-A, and IN-R with ViT-B/16, and IN-A and IN-R with ViT-L/14, when our Frolic is combined with InMaP. However, these results still lag behind the original performance of our Frolic. This discrepancy may stem from the hyper rrper-parameter in InMaP being optimized specifically for ImageNet: applying them unchanged to its shifted datasets could lead to fett-fitiing.
4.3 Ablation Studies and Further Analysis
Effectiveness of the prompt distribution learning. In Table (Row (1) & (3)), compare the performance the original CLIP model fc with our prompt distribution learning model fg. We observe that modeling the underlying distribution of the text prototypes results in notable performance gains. For example, 3.7% accuracy improvement on 10-datasets using ViT-B/16.
Effectiveness of the prediction fusion. As described in Eq. 10). our Frolic fuses the original CLIP Se and the prompt distribution learning model fg via confidence matching. We compare the simple fusion fg and our adaptive fusion fr T fg/ //g in Table Table (Row (4) & (5)). We show that our fusion technique outperforms the simple fusion by large margin. Recall that our adaptive fusion method addresses situations where fg is more overconfident than fc. In Figure 3 illustrate the relationship between performance gains over simple fusion -Le. Acc(fe/Te + fg. TT)) Acc(fe + (eggg)am and the confidence difference--fe point represents a dataset, and we have fitted these points with a line. As expected, larger confidence differences correlate with
|conf( 1) conf(fe: We present this scatter plot where each
significant improvements.
Effectiveness of the bias correction. Row (2) and (6) in Table demonstrate the effectiveness of debiasing method, which can further improve the base CLIP model fo and the fusion model fa

---

Table 4: Comparison of accuracy (%) between Frolic and other label bias correcting methods for CLIP ViT-B/16.
EutSAT
ImageNer
Flowers
Aireraft
Caltech
Pets
DTD
Cars
Food
SUN
UCF
Model
Avg.
CLIP 20
89.1
71.4
24.8
44.3
47.7
65.2
86.1
62.5
92.9
66.7
68.7
65.4
TDE 34
84.1
65.8
27.4
49.8
55.3
60.3
84,6
65.5
91.6
68.2
65.9
65.3
Implicit
91.4
71.4
30.6
54.2
56.8
66.0
86.6
69.5
93.5
72.6
69.8
69.3
Frolic
92.9
74.8
31.4
56.1
58.5
69.1
87.1
95.1
75.2
70.9
70.9
Oracle Frolic
93.1
77.5
32.2
57.3
59.8
69.8
87.4
71.2
95.7
76.3
71.5
71.9
madiin
70.9
1.00
10.00
Eieer dtsees
Gains
7.5
Accuracy 196)
0.75
5.0
0.50
2.5
0.25
0.0
0.2
[confif, conttf, Re)l
Iteration
Figure 3: Relation between gains and confi- dence differences.
Figure Convergence of accuracy and 4 eтror of on ImageNet.
across various backbones and datasets. We also compare our debiasing method with other label bias correction methods in Table 4 The descriptions of TDE (34] and the Implicit method can be found in Section 3.4 The results reveal that TDE
34 does not consistently perform well across all datasets. In contrast, while the implicit method using downstream data enhances zero-shot performance, it underperforms compared our debiasing method, which shows an average gain of 1.6% over the implicit method. To further assess our method's potential, we replaced pseudo-labeling with ground truth labels. The results reveal that the maximum achievable accuracy surpasses our method by 1.0%, highlighting the importance of our iterative approach for more accurate pseudo-labeling.
Convergence of Algorithm L. Our method Frolic, as described in Algorithm 2 iteratively solves for the prior B. In Figure 4 we examine the convergence by displaying the errors l = || - 3 I1l and the accuracy across iterations. We find that the resultant accuracy saturates after only steps, and the relative & eтror decreases to less than 0.01 after 10 steps.
Comparison with other prompt-based methods. The popular prompt-based methods, such as CoOp [46] and CoCoOp [45], require a training procedure with labeled samples while our method does not involve any training. To ensure fair comparison, we compare Frolic with CoOp and CoCoOp on across-dataset results, where the CoOp and CoCoOp are trained only with the labeled samples from the ImageNet dataset and then directly tested the remaining datasets. The results that our Frolic not only avoids the complexities of training but also
shown in exhibits superior generalization performance compared these methods.
Table 5: Comparison of accuracy (%) between our Frolic and prompt-
trompt-based based methods for CLIP
ViT-B/16. * denotes our method built upon InMaP 28
ImageNet
EuroSAT
Flowers
Aircraft
Caltech
Pets
DTD
Cars
Food
SUN
UCF
Model
CoOp
46
71.5
93.7
89.1
64.5
68.7
85.3
18.4
64.1
41.9
46.3
66.5
CoCoOp 45
71.0
94.4
90.1
65.3
71.8
86.0
22.9
67.3
45.7
45.3
68.2
Frolic
73.3
95.4
93.6
71.7
74.3
88.2
31.8
72.8
58.0
65.3
75.9
The adapter-based methods, e.g., LFA 24 and Tip- Adapter 44 boost the CL.IP's generalization using labeled training samples. contrast, our Frolic doesn't 't require any labeled samples. We evaluate method with LFA and Tip-Adapert the
Comparison with adapter-based methods.

---

Table 6: Comparison of accuracy (%) between Frolic and adapter-based distribution methods for CLIP ViT-B/16. * denotes our method built upon InMaP [28]
Model
IN
IN-A
IN-V2
IN-R
IN-Sketch
Average
LFA [24
72.6
51.5
64.7
76.
48.0
62.5
Tip-Adapter
44
70.5
49.8
63.1
48.1
61.6
Frolic
73.3
52.8
63.8
79.6
52.9
64.4
ImageNet and its variants dataset, where the LFA and Tip-Adapter only utilize the labeled samples from the ImageNet dataset. The results in 6shw that our method achieves the best performance across all datasets with nearly 3% improvements in averaged accuracy over LFA.
Running time. Our method Frolic completely training- free, unlike prompt tuning approaches such as TPT [32 and TDA [5] which involve back- proppaaaagg through an expensive encoder during optimization. We assess the wall-clock time of Frolic. TPT, and TDA in Table 7 using the CLIP ViT-B/16 model on ImageNet. These evaluations are conducted on a single NVIDIA A100 GPU, The results indicate that our method not only requires less time but also delivers superior performance.
Table 7: Comparison of running time ImageNet with ViT-B/16.
Model
Running Time Accuracy
CLIP 29
6min
68.7
TPT
6h
68.9
TDA 15
15min
69.5
Frolic
6.5min
71.1
Societal Impact, Limitation and Conclusion
Societal impact and limitation. Models pre-trained on large-scale web-crawled datasets may incorporate knowledge from noisy or malicious samples.
Limitation. Our approach assumes that the feature representations follow mixture of Gaussian; however, this assumption may not always hold. On the other hand, the quality and distribution of data used in pre-training can significantly impact the performance of pre-trained models. Our method relies on the capabilities of pre-trained models for downstream tasks, the pre-trained knowledge differs from the downstream tasks, the efficacy of method may be limited.
prompt distribution learning and bias correction, dubbed as Frolic, framework to boost the performance of zero-shot models. Our Frolic models each class prototype via a Gaussian distribution and fuses the learned model with the original CLIP [29] via confidence matching. The proposed framework further effectively removes the label bias without accessing to the pre- pre-traing training data. Extensive experiments across various datasets demonstrate the effectiveness of our approach.
Conclusion. In this work, we propose
10

---

Acknowledgments and Disclosure of Funding
The work supported by the National Natural Science Foundation of China (Grants No. 62202439), and the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award This work is also supported by the advanced computing resources
No: DSSG2-PhD-2021-01-0000 provided by the Supercomputing Center of the USTC.
References
[1] James Urquhart Allingham, Jie Ren, Michael W. Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In JCML, 2023.
[2] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019.
[3] Christopher M Bishop. Pattern recognition and machine learning. Springer, 2:200-202.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014.
[5] Hao Chen, Bhiksha Raj, Xing Xie, and Jindong Wang. On catastrophic inheritance of large foundation models. arXiv preprint arXiv:2402.01909 2024.
[6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- aangagge-image learning. In CVPR, 2023.
[7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[9] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops, 2004.
[10] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML,, 2017.
[11) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote, Sens., 12(7):2217-2226, 2019.
[12] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of nut-of-distribund generalization. ICCV, 2021.
[13] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. 2021.
[14] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In CVPR, 2021.
[15] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR, 2024.
[16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. object representations for fine-grained categorization. In ICCV Workshops, 2013.

---

[17] Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In UAI, 2022.
[18] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, 2022.
[19] Rachel Luo, Shengjia Zhao, Jiaming Song, Jonathan Kuck, Stefano Ermon, and Silvio Savarese. Privacy preserving recalibration under domain shift. CoRR, abs/2008.09643. 2020.
[20] Subhramsu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. CoRR, abs/1306.5151, 2013.
[21] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In /CLR, 2021.
[22] RV Mises and Hilda Pollaezek- Poliinzek-Geiringerr.. Praktische verfahren der gleichungsauffösung.. ZAMM- Journal of Applied Mathematics and Mechanics/Zeitschrii für Angewandte Mathematik und Mechanik, 91):58-777, 1929.
[23] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In ICVGIP, 2008
[24] Yassine Ouali, Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision-language models. CoRR, abs/2304.01752, 2023.
[25] Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails of vision-language models. arXiv preprint arXiv:2401.12425 2024.
[26] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[27] Sarah M. Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023.
[28] Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual catego- rization with CLIP. In NeurIPS, 2023.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[30] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[31] Jameel Abdul Samadh, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Shahbaz Khan, and Salman H. Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In NeurlPS, 2023.
[32] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NewrlPS, 2022.
[33] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0400) 2012.
[34] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed elassification by keeping the good and removing the bad momentum causal effect. In NeurlPS, 2020.
[35] Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-fres name-only transfer of vision-language models. CoRR, abs/2211.16195 2022.
12

---

[36] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances Neural Information Processing Systems, pages 10506-10511 10518, 2019.
[37] Xudong Wang. Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels. In CVPR, 2022.
[38] Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free CLIP-based adaptation. In ICLR, 2024.
[39] Zhicai Wang. Yanbin Hao, Tingting Mu, Ouxiang Li, Shuo Wang, and Xiangnan He. Bi- directional distribution alignment for transductive zero-shot learning. In CVPR, 2023.
[40] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, 2022.
[41] Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang. and Jingdong Wang. Gpt4vis: What can GPT-4 do for zero-shot visual recognition? CoRR, abs/2311.15732, 2023.
[42] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
[43] Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, Joo-Hwee Lim, and Hanwang Zhang. Invariant training 2d-3d joint hard samples for few-shot point cloud recognition. In ICCV, 2023,
[44] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of CLIP for few-shot classification. In ECCV, 2022.
[45] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-languee languae models. In CVPR, 2022.
[46] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 13009):3337-234,, 2022.
[47] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV, 2023.
[48] Beier Zhu, Yulei Niu, Xian-Sheng Hua, and Hanwang Zhang. Cross-domain empirical risk minimization for unbiased long-tailed classification. In AAAI, 2022.
[49] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In NeurIPS, 2023.
[50] Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision-language subspace projection for few-shot CLIP. In ACM MM, 2024.
13

---

Contents
Introduction
2
Related Works
3
3
Methods
3.1
Setup
3.2
Label-Free Prompt Distribution Learning
3.3
Prediction Fusion Adaplive Calibration,
3.4
Correcting Pre- tre-training Label Bias via Label Free Logit Adjustment
4
Experiments
Setup
Main Results
4.3
Ablation Studies and Further Analysis
Societal Impact, Limitation and Conclusion
10
Theoretical Analysis
15
A.1 Proof of Eq. (6): Estimation of Class Covariance from Marginal Second Order Moment
15
A.2 Proof of Eq. (5): Estimation of the Priors of Gaussian Mixture Models
16
A.3 Proof of Eq. (7): Parameters of our Learned Model
17
A.4 Proof of Eq. (11b: Debiased Classifier for Downstream Data
17
A.5 Proof of Eq. 12p Equation to Estimate Pre-training Priors
18
A.6 6 Power Method to Estimate Pretraining Priors
18
Details of ImageNet Variant Datasets
18
Licenses
19
14

---

A Theoretical Analysis
A.1 Proof of Eq. 6 Estimation of Class Covariance from Marginal Second Order Moment
We first derive the second order moments for a multivariate Gaussian and then for Gaussian mixture, corresponding to the marginal distribution of P(x).
For class with parameters 2j and 2, the conditional probability density function is given by:
1 2(x 23) Σ -2;)}
N(x Σ) (2T)
expl
(17)
The second order moment generating function for class j is:
M, = Xxc [xx 1
N(x; ,))xx Txxxdxx dx
(18)
1
expl (x xxzz)
21)² "))T² 1 1x 2;)]xxx dx
(19)
V(2))41
expl 2y E y 2,)(y + ddy
(20)
(2
1
expl 2 Σ E lyy" T + ya + z,y +2,2, )dy
(21)
V(2T)
vanish by symmetry
(
exp( zy T 3y'yy(yy +
22, += ddy
(22)
(2
Jy
2,
1 y expl expl- -il--yy7
L - yyyyy ddy
(23)
[2
(a) holds as we change the integral variables = Zj. We have because the exp(-) function is an even function of and the factors yz, and z,y will vanish during integral by symmetry. For we take the term _ outside of the integral as they are constant.
The covariance matrix Σ and its inverse matrix I-1 the can be expressed through an expansion in terms of its eigenvalues { and eigenvectors {u.
--1= d
Σ Nuuu
u
(24)
Similarly, we can decompose y using the set of eigenvectors: y E/= Biul, where ej = 11 y.
(We temporarily abuse the subscript j here. It does not represent class until we reach Eq. 32 We have the following expression:
T yy
u,
(25)
{=
d
d
d
d
(d)
Gk )2
y Σ y
euu;
U
Ug
euu
(26)
Ak
Vxx
k=1
/=1
We obtain (4) due to the property of eigenvallus, i.e., u. u, = and u u, = 0, for i j. Denote U [I...., Uz] we have e Uy. As the determinant [U] 1, the probability density after transformed remains unchanged: P(e) = |P)))(()) = Ply). Apply Eq. (25) and Eq. 226 into
15

---

Eq. 23
we have:
1
expl
2
(27)
(2
)d Σ
d
1
expl
2 Vxx
1 ck )2 Jere,de
(28)
j=1
k=1
1
d
f
T
II expl
2 Vxx
1
)2 Jejedde
(29)
(2
i= j=1
k=1
1
1. Ci )2 feedde:
4uu
expl- - 2/( VT:
(30)
12
e.
1=1
expl -
1 Ei
)²) )
(31)
V
2T
Vx=
( d nnu
(32)
d For the terms A j disappear by symmetry similar to We make use of |Σ| I x for (
We have (g) because we regard 6 ~ NO, VX;) and note that Eie]] = var[e:] + Elei]"
= > +0 = A
Combining Eq. 32 with Eq. (2). we have the second order moment for class j is:
M, 2, + E.
(33)
Using a Gaussian mixture model with the priors { 15. F(x) is given by:
P(x)
Σ TyN(x: zj. Σ).
(34)
j=1
The second order moment for the marginal distribution P(x) is:
K
M = E|xx 1
/
Σ
Ty (xx 2,, E)dxxx x
(35)
K
Σ
Ts.
NNxx; z Σ)xx T dx
(36)
5=1
K
K
Σ
My = Σ
[z,zz + +E)
(37)
1=1
/=1
K
Σ T,Z1Z
Σ
mイソミ= Σ TzZ,Z]
(38)
Ø=1
A.2 Proof of Eq. 5) Estimation of the Priors of Gaussian Mixture Models
The expectation of x is defined as:
K E[x] Σ
K
K
RT (x; E)xdx = Σ TT [ N(x;22, E)xdxx =
Σ T.
(39)
1=1
j=1
Denote π [ T Z = [2], zxl I", and the expectation of x as μt, Eq. as:
39
can be rewrite
ZT.
(40)
Therefore, the priors can be solve by 2-1 μ.
D
16

---

A.3
Proof of Eq. 7 Parameters of our Learned Model
The posterior of classes P(y/x) can be expression as:
P(u|x) [((x)))P())) X N Bay, E)Ty. PP(x)
(41)
To classify x, we seek the class y that maximizes this posterior: Since the term P(x) does not depend on y, we can simplify our task maximizing NV (xx; Myy E) Taking natural logarithm gives:
1 (x
InN(x: By: E) In
expl
zy) T 1x - 2
(42)
V(2
2
Inn
(x Tny)
- byy) + -n By
(43)
(2
2
c1 2x xx'''
1
E
x 2n T Σ --'x
x
1 T Σ By +
(44)
2
1
2 M x
- Bn +c
(45)
2
w x + by +
(46)
The first term in Equation 43 is constant; we incorporate it using a constant c1 Consider that most test benchmarks are generally class-balanced, we use a uniform prior 02 incorporate In TT. In
Eq. ( we use e to absorb all constant terms, including C1, C2 and - - 1x. Let w, = =--1z
and by = 22] Wj. we get Eq. (46).
A.4 Proof of Eq. Debiased Classifier for Downstream Data
Proposition 1. (Modified from Theorem I in (I4)). Let F (y)xx) and I'ds (v)x) be the distributions of
the pre-train and downstream data, respectively, Let By = P (v) and my (yss(y) denote the priors of the pre-train and the downstream data, respectively. Assume the likelihood P ly) is unchanged
between pre-train and downstream data, i.e.. PP yy) Fx = (xl(x))). If foe(x) is the logit of class y from the softmax model to estimate F ()) then the estimated P (v) x is formulated as:
Pas (y)x) softmax( fx((x) In 3 + In TT) yy:
(47)
where B = -[1,..,, and π [mI..., Tx]
Proof.
(as(ulx) = (x(y)) Poly))Palll (xe(x(y) (x)(x)
(48)
(x)
(x pa (v) (v) (x)
(49)
(x)
(v)(() Pax
Ty 1 (4) (p(lll)
Fyy 1
By Z softmax(sspe
(50)
By Z
exp(fe((x),) 7 ΣK exp(fp((x)))
exp (1) exp (In By)
(51)
exp(Fo((x), In 3, + In
(52)
exp(ffxx((((()))
b exp(fxx(x), - In By + In my) Σn= exp(sxxxx (x); -1n,3 + Inm,)
(53)
= softmax(ffr(x) -In,3 +I
(54)
For we denote the term that is not related to y 1/2 that (Pax(y) x), being a probability, must sum to across all possible classes y E [K]:
Fm(xx) We derive ( from the requirement
Pm(x))
K
ΣKz exp(fxe(x): In B, + ln K Σ exp(fpe(x),)
Pax(i)x)=
(55)
17

---

Therefore, we have Z ΣE exp(fx(())) = E exp(fpt(x); - In 5; + In n;). In our context, the pre-trained model fpt is equivalent to our fr.
A.5 Proof of Eq. 12 Equation to Estimate Pre-training Priors
Proposition 2. Let sxx) [P = 1/x), P(Y K[x)]] RK be the likelihood vector, Sj [x(x==(((xx)) and S .... Sk] RK Kx
The pretraining prior B 131, R
must satisfy the linear system:
(S - 1)3 O.
(56)
Proof.
Bg / Pxe(x))Pp(((/x)) [ E W'EK
(57)
Σ By P(x))))Pxe(uxx)d
(58)
]'(K]
Σ
Bx
(59)
vE(K]
Σ
By Ex (s(x)ly
(60)
y'E[K]
Σ
Syy By¹
(61)
]'E[]]
Note that Equation 61 precisely represents the matrix multiplication given by:
G SB
(62)
By moving the RHS term the LHS, Eq. (56) is obtained.
L
A.6 Power Method to Estimate Pretraining Priors
The solution to Equation 62 involves finding the eigenvector corresponding to the eigenvalue of 1
for the matrix S. We apply SVD decomposition find the solution; however, we find that the results might be numerically unstable. Instcad, we adopt power iteration from [22]. Like the Jacobi
and Gauss-Seidel methods, the power method for approximating eigenvalues iterative. We first initialize Bn = = ...,,, 11 of a uniform distribution. Then, we perform the sequence:
K
3. 5
(63)
3,
B.
(64)
We repeat the sequence until the relative change is small: ||B, 11-1ll <e
B Details of ImageNet Variant Datasets
ImageNet- V2 30 sampling from the original ImageNet and including 10,000 images of 1,000
ImageNet categories.
ImageNet Sketch [36]: including 138 50,000 images and covering 1,000 ImageNet categories.
ImageNet-R [2] containing renditions (e.g., art, cartoons, graffiti) for ImageNet classes, comprising 30,000 images from 200 ImageNet categories.
ImageNet-A [3] collecting real-world images that are misclassified by ResNet-50, totaling 7,500
images from 200 of ImageNet categories.
ObjectNet: (2) including 50,000 test images with rotation, background, and viewpoint, and overlap- ping 113 classes with ImageNet.
18

---

C
Licenses
All the datasets we considered are publicly available, list their licences and URLs as follows:
Caltech101 9
N http://wwwwaaaaaa////..c.//////r
OxfordPets 26
MIT http://..wgpppo//oooooooooo.oooa
StanfordCars [6]:
MIT
License.
http:://ww...ggglee.ooo//aaataa
Dassicaa119530/saafffrr--
Flowers102 23
MIT http:://www..ob.tt.......aaaaaaa
102/index html
Food101 4]:
extra/food-100 101/
FGVCAircraft (20]
Non-commercial,
data.
Egve-aircraft/
SUN397 42
Non-commercial,
SUN/
DTD 7
EuroSAT MIT License, L
UCF101
33
ImageNet 8
Non-commercial, http://rwra...tp//i/aaaeeeeeeoo
ImageNetV2
30
MIT License, http:://ggggthbboooo//o//////gtt
ImageNet-R
12 MIT License, ht http:///saaaaaaaa/ooo/oa/aaaaaaa
ImageNet-Sketch
MIT
ImagelettSSketch
ImageNet-A
13
MIT
License,
https://gituubbcommmnntyyyyy
natural-adv-exampa
ObjectNet 22: Creative Commons Attribution 4.0, http:://objectnet.dev
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We point out that existing prompt learning methods can not capture the various image representations and ignore the label bias in CLIP. We propose Frolic to address these problems. Experiments various datasets demonstrate the effectiveness of method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification:
We have provided the proof related to our method in Section A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the Frolic Algorithm in Algorithm I and Algorithm 2, and included the implementation details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have uploaded the codes in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccpppppccceeeeeeeeee..c.ett httpa///giiieee/eeeSebiiisssssss /Cc////oddiiisssseeeeeee ssinsolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have introduced the testing details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Given the zero-shot models, the process of our method is deterministic. Running multiple times will not introduce randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have introduced the GPU type reproduce results Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impact on Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method aims to improve the generalization ability of zero-shot models, which poses no such risks the best of our knowledge.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have provided the licenses of each dataset in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "3bV0KQg1D5", "reviewer": "Reviewer_VFZ9", "summary": "This paper presents Frolic, a label-free prompt learning methods aiming to improve zero-shot visual recognition of vision-language models like CLIP.  The method is built upon estimating distributions over prompt prototypes to capture diverse visual representations and further bias correction. Experiment results demonstrate consistent improvement on various benchmarks over prior-art methods.", "questions": "1. The methods did not seem to evaluate on all benchmark datasets like previous works do. For example, CIFAR10/100 and RESISC. While  some of these datasets have saturated performance, it might be still helpful to include results there. \n2. It's not clear to me why [24] is not listed in tables for comparison as I would assume 24 to be the most direct baseline. It shares the same motivation of this paper with the requirement of accessing pretraining data. Arguably, while LAION has been taken back, [24] already does the job for us. I would not think an argument like \"[24] requires large-scale pretraining data for estimating bias thus we do not compare with it\" as a valid argument because this zero-shot visual recognition task itself is solely working with CLIP-ish models and estimating bias from the datasets like LAION sounds like a fair practice to me.", "limitations": "Authors have included such discussion", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The approach does not require access to large-scale datasets for estimation, which is required by many prior work. \n2. The approach advances previous distribution estimation methods by removing the need of label information, which made this line of work possible for zero-shot recognition tasks.", "weaknesses": "While this approach does not need access to large-scale pretraining data, it seems to make some weak assumptions on downstream tasks. If my understanding is correct, it assumes:\n1. the downstream task is balanced (line 134), which is not always hold true in reality. The long-tail nature of real world does not guarantee that the testing class distribution will be balanced even though the benchmarks do. \n2. A decent scale of the testing data on downstream tasks is available (e.g, testing data does not come in online fashion like one sample at a time) for estimating the beta term in bias correction. Would the approach for bias estimation still holds true in online testing scenario? I would assume this is more aligned with reality too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1]** the downstream task is balanced (line 134), which is not always hold true in reality. The long-tail nature of real world does not guarantee that the testing class distribution will be balanced even though the benchmarks do.  \n\n**[A1]** We acknowledge that real-world data often exhibits an imbalanced distribution. The balanced assumption in line 134 is a special case for simplicity. In our method, if the downstream datasets are imbalanced, we can derive the distribution vector $\\bf \\pi = Z^{-1}\\bf \\mu$ as outlined in Equation (5). This strategy allows our method to adapt to balanced and imbalanced dataset distributions flexibly.\n***\n**[Q2]**  A decent scale of the testing data on downstream tasks is available (e.g, testing data does not come in online fashion like one sample at a time) for estimating the beta term in bias correction. Would the approach for bias estimation still holds true in online testing scenario? I would assume this is more aligned with reality too.\n\n**[A2]** Thank you for your insightful question. Our method can be easily extended to an online scenario. This involves updating the estimation $S$ incrementally as each new test example is processed. Specifically, we first initialize the matrix $S$ with the identity matrix. Suppose we receive the $n$ -th test sample with the predicted label as $j$ and the predicted probability as $\\bf p$, we first update the ${\\mathbf{s}}_j =  \\frac{n-1}{n}{\\mathbf{s}}_j + \\frac{1}{n}\\mathbf{p}$. Then we compute the estimated $\\beta$ and update the $f\\_{\\text{d}}$. We have conducted additional experiments in the online setting to validate this extension, and the results below show that the online Frolic achieves comparable performance to the original Frolic.\n\n|     | Caltech | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 | Average |\n| --- | :---: | :---:  |:---:  |:---:  |:---:  |:---:  | :---:  |:---:  | :---:  | :---:  | :---:  |\n| Frolic | 95.4 | 93.6 | 71.7 | 74.3 | 88.2 | 31.8 | 72.8 | 58.0 | 65.3 | 75.9 | 72.7 |\n| Frolic（Online） | 94.3 | 93.1 | 71.4 | 74.5 | 88.1 | 31.5 | 73.0 | 57.4 | 64.5 | 75.1 | 72.3 |\n***\n**[Q3]** The methods did not seem to evaluate on all benchmark datasets like previous works do. For example, CIFAR10/100 and RESISC. While some of these datasets have saturated performance, it might be still helpful to include results there.\n\n**[A3]** We have conducted experiments with ViT-B/16 on CIFAR-10, CIFAR-100, and RESISC datasets, and the results are presented below:\n\n|     | CIFAR-10 | CIFAR-100 | RESISC |\n| --- | :---: | :---: | :---: |\n| CLIP | 91.3 | 68.6 | 58.9 |\n| Frolic | 92.6 | 70.0 | 64.4 |\n\nWe observe that on these three datasets, our method improves the performance over the original CLIP.\n***\n**[Q4]** It's not clear to me why [24] is not listed in tables for comparison as I would assume 24 to be the most direct baseline. It shares the same motivation of this paper with the requirement of accessing pretraining data. Arguably, while LAION has been taken back, [24] already does the job for us. I would not think an argument like \"[24] requires large-scale pretraining data for estimating bias thus we do not compare with it\" as a valid argument because this zero-shot visual recognition task itself is solely working with CLIP-ish models and estimating bias from the datasets like LAION sounds like a fair practice to me.\n\n**[A4]** Thank you for your suggestions. We acknowledge that [24], referred to as REAL, represents a crucial baseline and shares a similar motivation with our work. We conducted a comparison between REAL, which utilizes the LAION 400M dataset, and our Frolic, using the OpenCLIP (ViT-B/16 model) across several datasets. The summarized results below demonstrate that our Frolic outperforms REAL obviously and achieves an average improvement of 1.1% in accuracy. We have included these results in the revised manuscript.\n\n|     | ImageNet | Flowers | Cars | Aircraft | Pets | Food | DTD | Average |\n| --- | :---: |:---: | :---:| :---: |:---: | :---: | :---: | :---: |\n| REAL[24] | 68.1 | 73.1 | 84.0 | 18.8 | 90.5 | 85.2 | 59.8 | 68.5 |\n| Frolic | 70.3 | 73.9 | 84.6 | 19.9 | 91.6 | 86.9 | 60.1 | 69.6 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HXjvOP8FGy", "reviewer": "Reviewer_GPPc", "summary": "In this paper, the authors introduce a promising method for enhancing zero-shot vision models through the utilization of prompt distribution learning and bias correction. The method is particularly notable for being training-free and label-free, which greatly simplifies the implementation process. The authors provide comprehensive experimental results, which effectively demonstrate the efficacy of the proposed method.", "questions": "The author stated that their method does not require hyperparameter tuning. How does the proposed method compare to those requiring hyperparameter tuning?", "limitations": "Refer to the weakness and Questions", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is easy to understand.\n2. The paper provides an in-depth theoretical discussion and a thorough experimental evaluation demonstrating the effectiveness of the proposed method.", "weaknesses": "1. The computation of second-order moments and the covariance matrix from the marginal distribution as discussed in Eq.(3) and (5) might be computationally intensive.\n2. The method relies on the pseudo-labels estimated by CLIP. How does this process influence the final results?\n3. The setting of results on the ImageNet variants dataset is unclear. \n4. The results lack comparison with other prompt distribution methods, such as CoOp and CoCoOp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1]** The computation of second-order moments and the covariance matrix from the marginal distribution as discussed in Eq.(3) and (5) might be computationally intensive.\n\n**[A1]** We have evaluated the running time as presented in Table 5. The results show that while Frolic requires slightly more computation time (0.0078 seconds/per sample) compared to the original CLIP (0.0072 seconds/per sample), it yields improved performance, increasing from 68.7% to 71.1%. \n***\n**[Q2]** The method relies on the pseudo-labels estimated by CLIP. How does this process influence the final results?\n\n**[A2]** Our method can improve performance regardless of the quality of the pseudo-labels. As evidenced in Table 4, the original CLIP achieves 92.95% accuracy on the Caltech dataset, indicative of high-quality pseudo-labels, while 24.8% accuracy on the Aircraft dataset, which represents lower-quality pseudo-labels. Our method effectively improves results across these varied quality levels, boosting accuracy from 92.95% to 95.1% on the Caltech and from 24.8% to 31.4% on the Aircraft.\n***\n**[Q3]** The setting of results on the ImageNet variants dataset is unclear.\n\n**[A3]** We utilize the model learned on ImageNet to evaluate its performance across the ImageNet variants. \n\nWe have provided details about each variant to ensure clarity in the revision:\n\nImageNet-V2: sampling from the original ImageNet and including 10,000 images of 1,000 ImageNet categories.\n\nImageNet Sketch: including 138 50,000 images and covering 1,000 ImageNet categories.\n\nImageNet-R: containing renditions (e.g., art, cartoons, graffiti) for ImageNet classes, comprising 30,000 images from 200 ImageNet categories.\n\nImageNet-A: collecting real-world images that are misclassified by ResNet-50, totaling 7,500 images from 200 of ImageNet categories.\n\nObjectNet: including 50,000 test images with rotation, background, and viewpoint, and overlapping 113 classes with ImageNet\n\nThese details have been included in the revised manuscript.\n***\n**[Q4]** The results lack comparison with other prompt distribution methods, such as CoOp and CoCoOp.\n\n**[A4]** The CoOp and CoCoOp require a training procedure with labeled samples while our method does not involve any training.  To ensure a fair comparison, we compare our Frolic with CoOp and CoCoOp on across-dataset results, where the CoOp and CoCop are trained only with the labeled samples from the ImageNet dataset and then directly tested on the remaining datasets. The results shown below demonstrate that our Frolic not only avoids the complexities of training but also exhibits superior generalization performance compared to these methods.\n\n|     | ImageNet | Caltech | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 |\n| --- | :---:  |:---:  | :---:  |:---:  | :---:  |:---:  | :---: | :---:  | :---:  | :---:  |:---: |\n| CoOp | 71.5 | 93.7 | 89.1 | 64.5 | 68.7 | 85.3 | 18.4 | 64.1 | 41.9 | 46.3 | 66.5 |\n| CoCoOp | 71.0 | 94.4 | 90.1 | 65.3 | 71.8 | 86.0 | 22.9 | 67.3 | 45.7 | 45.3 | 68.2 |\n| Frolic | 73.3 | 95.4 | 93.6 | 71.7 | 74.3 | 88.2 | 31.8 | 72.8 | 58.0 | 65.3 | 75.9 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v4qXo8C9bX", "reviewer": "Reviewer_mJSR", "summary": "This work aims to enhance the zero-shot performance of pre-trained vision-language models. Specifically, three strategies, including label-free prompt distribution learning, adaptive calibration and correction pre-training label bias, are proposed and work together to improve the performance on different downstream tasks. Experiments on diverse tasks confirm the effectiveness of the proposed method.", "questions": "I do not have critical questions about this work.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tEnhancing the zero-shot performance has attracted much attention recently. This work discusses multiple challenges in this process and proposes corresponding algorithms with theoretical analysis.\n2.\tThe proposed method is training-free without hyper-parameters, which is applicable to real applications.\n3.\tDiverse downstream tasks are included for evaluating the performance of the proposed method.\n4.\tThe manuscript is well-written and easy to follow.", "weaknesses": "1.\tThe work works with a uniform prior that classes are balanced distributed. It is better to discuss the scenario with imbalanced prior.\n2.\tBesides CLIP, there are many other vision-language pre-trained models, e.g., BLIP, etc. It is better to include other pre-trained models to evaluate the performance of the proposed method.\n3.\tWhy does $e_i$ follow a Gaussian distribution in L446?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1]** The work works with a uniform prior that classes are balanced distributed. It is better to discuss the scenario with imbalanced prior.\n\n**[A1]** We consider that most of the datasets such as ImageNet and its variants are uniformly distributed,  leading us to assume $\\pi_j=\\frac{1}{K}$ in line 134 as a special case for simplicity. However, if the downstream datasets are imbalanced, we can derive the distribution vector $\\bf \\pi = Z^{-1}\\bf \\mu$  as outlined in Equation (5). This strategy allows our method to adapt flexibly to both balanced and imbalanced dataset distributions.\n***\n**[Q2]**  Besides CLIP, there are many other vision-language pre-trained models, e.g., BLIP, etc. It is better to include other pre-trained models to evaluate the performance of the proposed method.\n\n**[A2]** Thank you for your valuable suggestion. We have conducted additional experiments using BLIP with the  ViT/B-16 backbone.  The results presented below indicate that our method consistently outperforms BLIP across various datasets.\n\n|     | Pets | Flowers | DTD | EuroSAT | Cars | Food | SUN | Caltech | UCF |\n| --- | :---:  | :---:  | :---:  | :---:  | :---:  | :---:  | :---:  | :---:  | :---:  |\n| BLIP | 65.1 | 50.4 | 44.1 | 37.0 | 62.6 | 69.0 | 48.4 | 86.1 | 51.5 |\n| +Frolic | 79.3 | 56.2 | 58.3 | 50.1 | 70.2 | 74.2 | 67.1 | 92.3 | 64.3 |\n***\n**[Q3]** Why does $e_i$  follow a Gaussian distribution in L446?\n\n**[A3]** The original $\\bf x$ follows a Gaussian distribution ${\\cal N}{({\\bf z}_j, \\Sigma)}$. We express the covariance matrix **$\\Sigma$** of the original $\\bf x$ as an expansion in terms of its eigenvectors as Equation (24), then we can interpret the variable $e_i = {\\bf u}_i^T(\\bf{x}-\\bf{z}_j)$ to a new coordinate system defined by the orthogonal vectors ${\\bf u}_i$. Since these transformations are linear and $\\bf x$ is Gaussian, the transformed variables $e_i$ ​ also adhere to a Gaussian distribution. Specifically, as the variance of the $i$ -th coordinate is $\\lambda_i$ ,  $e_i$ follows the Gaussian distribution ${\\cal N}(0, \\lambda_i)$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MOUpHbHuX8", "reviewer": "Reviewer_jbMQ", "summary": "This paper proposes a method named Frolic to improve the zero-shot performance of vision-language models like CLIP. The method focuses on two key challenges: enhancing prompt distribution learning and correcting inherent label bias in pre-trained models without relying on labeled data. Experimental results across 16 datasets demonstrate performance improvements of the method.", "questions": "The discussion of the limitations of the methods is unclear. The authors should discuss it in depth.", "limitations": "Please refer to the weakness and Questions parts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written and the method is easy to follow.\n+ The method effectively addresses label bias inherent in pre-trained models, improving the robustness and accuracy of zero-shot predictions.\n+ Ablation experiments are comprehensive.", "weaknesses": "- Does the paper utilize the validation dataset in experiments, if not, this should be clarified.\n- How does the proposed method compare with the methods[1][2] in the few-shot settings?\n- Although Frolic is described as training-free, there may still be hyperparameters involved in the method, and how to choose these parameters?\n- How does the prompt distribution learning contribute to the performance? Does it outperform the original CLIP?\n- Can the method be performed with a few labeled samples?  \n[a] Black box few-shot adaptation for vision-language models. ICCV 2023\n[b] Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification. ECCV, 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1]** Does the paper utilize the validation dataset in experiments, if not, this should be clarified.\n\n**[A1]** We do not use a validation set because our method involves no hypermeter searching.\n***\n**[Q2]** How does the proposed method compare with the methods[1][2] in the few-shot settings?\n\n**[A2]** The methods[1][2] refer to LFA and Tip-Adapter. These methods boost the CLIP's generalization using labeled training samples. In contrast, our Frolic doesn't require any labeled samples. We evaluate our method with LFA and Tip-Adapert on the ImageNet and its variants dataset, where the LFA and Tip-Adapter only utilize the labeled samples from the ImageNet dataset. The results below show that our method achieves the best performance, except compared to LFA on the ImageNet. \n***\n|     | ImageNet | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-Sketch |\n| --- | :---: | :---: | :---: | :---: | :---: |\n| LFA | 72.6 | 51.5 | 64.7 | 76.1 | 48.0 |\n| Tip-Adapter | 70.5 | 49.8 | 63.1 | 76.9 | 48.1 |\n| Frolic | 70.9 | 60.4 | 64.7 | 80.7 | 53.3 |\n\n**[Q3]**   Although Frolic is described as training-free, there may still be hyperparameters involved in the method, and how to choose these parameters?\n\n**[A3]** The only hyperparameter in our method is tolerance $\\epsilon = 0.01$, which is critical for the precision of numerical calculations rather than being a traditional model hyperparameter.  As illustrated in Figure 4, when the iteration extends over 20 steps, $\\epsilon$ approaches zero. While theoretically, a lower $\\epsilon$ is preferable for finer precision, we have selected $\\epsilon = 0.01$ to ensure a practical balance between computational efficiency and convergence reliability.\n***\n**[Q4]** How does the prompt distribution learning contribute to the performance? Does it outperform the original CLIP?\n\n**[A4]** As compared in Table 3 ($f_{\\rm c}$ and $f_{\\rm g}$), the prompt distribution learning ($f_{\\rm g}$, shown in the third row) significantly outperforms the original CLIP ($f_{\\rm c}$​, shown in the first row). For example, $f_{\\rm c}$ increases the performance from 65.1% to 68.8% on the 10-datasets and 68.7% to 69.8% on the ImageNet.\n***\n**[Q5]**   Can the method be performed with a few labeled samples?\n\n**[A5]** Our method can effectively incorporate labeled samples by replacing the class description to estimate ${\\bf z}_1$ to ${\\bf z}_K$ in Equation (3). To demonstrate this capability, we have conducted additional experiments in a 16-shot setting using the ViT-B/16 backbone, the results below show that the 16-shot setting achieves better performance than the class descriptions.\n|                            | Pets | Flowers | Aircraft | DTD  | EuroSAT | Cars | Food | SUN  | Caltech | UCF  |\n| -------------------------- |:---: |:---: | :---: |:---: |:---: | :---: | :---: | :---: | :---: | :---: |\n| Frolic                     | 92.9 | 74.8    | 31.5     | 56.1 | 58.5    | 69.1 | 87.2 | 70.8 | 95.2    | 75.2 |\n| Frolic with labeld sampels | 94.5 | 98.3    | 51.4     | 71.7 | 89.6    | 83.5 | 87.9 | 76.6 | 96.3    | 85.2 |\n\n\n***\n**[Q6]** The discussion of the limitations of the methods is unclear. The authors should discuss it in depth.\n\n**[A6]** The quality and distribution of data used in pre-training can significantly impact the performance of pre-trained models. Our method relies on the capabilities of pre-trained models for downstream tasks, if the pre-trained knowledge differs from the downstream tasks, the efficacy of our method may be limited. We have included a detailed discussion of limitations in the revised manuscript. Additionally, as you suggested in [Q5], few-shot learning presents a viable solution to these challenges. We appreciate your insights, which help us to further exploit the potential of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chair, Senior Area Chair, Area Chair, and Reviewers,

First of all, we gratefully thank all the reviewers for their thoughtful comments and feedback.

In this paper, we propose label-Free prompt distribution learning and bias correction, dubbed as Frolic, framework to boost the performance of zero-shot models. The contribution of this paper is listed as follows:

1. Simple and Practical Solution: Our method is not only training-free but also circumvents the necessity for hyper-parameter tuning.

2. Comprehensive Evaluation: We demonstrate the effectiveness of our proposed method Frolic by conducting experiments across 16 datasets

3. Significant Performance Improvement: Our Frolic has a consistent and significant improvement over existing baselines. For example, it surpasses the state-of-the-art zero-shot models by a margin of 2.6% on average with CLIP ViT-B/16.

As our paper received mixed ratings, i.e., three positive (666) and one negative (4), it would be appreciated if the reviewers could have a look at our responses and revision. We have tried our best to address your concerns in our responses in detail. Hope that our responses answered the questions. Please let us know at your early convenience if you have further questions or concerns.

Best regards,

Authors of Paper #2527

Meta-review: Reviewers agree on acceptance based on the practical significance, efficient methodology, and convincing results. The authors are encouraged to include discussions and results in the rebuttal to the final version.
Decision: Accept (spotlight)