Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models
Authors: Beier Zhu
Kaihua Tang
Qianru Sun
Hanwang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Foundation Model
Class Bias
Generalized Logit Adjustment
TL;DR: This paper firstly points out the overlooked inherent bias of zero-shot models, and present a Generalized Logit Adjustment (GLA) method together with a pretrain class distribution estimation algorithm to tackle this problem.

Abstract: Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly assessed like in traditional long-tailed classification tasks.
To this end, GLA has an optimization-based bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, an large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains on long-tailed classification. Codes are in https://github.com/BeierZhu/GLA.

Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models
Beier Zhu' Kaihua Tang'
Qianru Sun?
Hanwang Zhang'
Nanyang Technological University
'Singapore Management University
beier0028e. ntu. edu.sg,
hanvangzhangdntu.c edu.
Abstract
Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semanties, and thus the subsequent fine-tuning ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly accessed like traditional loigetailed classification tasks. To this end, GLA has optinization-basce bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: achieves 1.5 pp accuracy gains ImageNet, large average improvement (1.4-4.6 pp) 11 few-shot datasets, 2.4 pp gains on long-tailed classification. Codes are in https: pattps://giaann //githu........./ com/ BeierZhu/GLA
Introduction
Thanks to the Web-scale data and self-supervised strategies. foundation models like CLIP 440 power zero-shot transfer to a wide variety of domains [2]. However, the zero-shot performance still weak several domain-specific tasks such as differentiating models of cars, species of flowers, and variants of aircraft 40 [7] Therefore, it is a common practice to improve the downstream performance via supervised fine-tuning labeled data, e.g., linear probing, prompt tuning [54] 55). and end-to-end fine-tuning.
However, fine-tuned models are easily biased: they are adept in exploiting spurious correlations that only hold on the downstream distribution 440
55]. To improve the robustness, several studies 48, 55 56] propose combine fine-tuned models with zero-shot models. For example, FT 48 ensembles the fine-tuned and zero-shot models in weight space and ProGrad (55) uses zero-shot predictions to regularize the fine-tuning gradient. The underlying assumption lies in that the zero-shot models are robust to distribution shifts 44) and their predictions are complementary to those of fine-tuned models 48
39
48
Despite these methods exhibiting performance gains on both in-distribution and out-of-disttibution evaluations, they all overlook the inherent bias originating from the foundation models. Specifically, the Web-scale data for pre-training foundation models exhibit a highly skewed distribution due to Zipf's law of nature 42]. The resulting foundation models develop biased decision boundary that leads to a poor zero-shot performance on rare classes. As evidenced in Figure T[a) and (b), the purple
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Model
Medum
Zaro-Shot
78.0
68.8
57.
68.3
Top-
Exiimdted label colnbutto
Zerp Shol Free Ture
Fine Tune-uned
83.8
83.0
81.3
WISE-T
WSEFFTT
85.3 (+1.7)
(+0.)
76.4 (-0.9)
91.8
ULA bund
GLA (ours)
85.2 (+1.8)
84.3 (+1.3) 78.8 (+1.5)
828 (+1.5)
Zoomed resuits
Sodted
(8) Por dass nocuracy of difforent modets on ImageNett
(5) Boak down fermoranc difteront modills on ImaagNNet
Figure 1: (a) Per class accuracy of CLIP- 1 on ImageNet. Class index are sorted using the estimated pre-training label prior. Curves are smoothed for better visualization. (b) Beak-down
performance of different models on ImageNet. We equally divide the ImageNet classes into three subgroups, according the class index. Existing ensemble methods like WiSE-FT [48] exhibits clear performance loss on tail classes, while our GLA stands out for all three subgroups.
line encounters a dramatic drop, and the zero-shot performance of tail classes significantly lower than that of head classes (57.2% 78.0%). Existing ensemble methods like WiSE-FT [48] overlook the label bias, resulting in an improvement in top-1 (+0.55% and head accuracy (+1.7% while a noticeable degradation the tail performances (-0.9%) in Figure [[b). Another evidence is that the orange line (WiSE-FT) is below the blue line (fine-tuned models) for rare classes in Figure Ta).
We propose Generalized Logit Adjustment (GLA), simple post-hoe method consisting of two steps: removing the label bias of zero-shot model via estimating the label distribution the pre- pre-training dataset; 2) ensembling the fine-tuned and debiased zero-shot models. As illustrated in Figure (b), our GLA achieves consistent improvement across all three subgroups, particularly showing significant gain on tail classes (+1.5%). Despite its simplicity, our GLA has a firm statistical grounding: it is the Bayes optimal classifier given the fine-tuned and zero-shot models, thus consistent for minimizing the error class-balanced target distribution (Section 4.2p. It is worth noting that removing the bias of foundation models is challenging since the label distribution is often inaccessible due to privacy or copyright concerns. In this work, only the downstream labeled data and the zero-shot model to estimate the foundation label bias. Specifically, we formulate the problem by adjusting the margin of the zero-shot models such that the lowest error is achieved on the downstream dataset. This grounding translates into strong empirical performance on real-world datasets, covering few-shot, many-shop and long-tail learning (Section 5)
The contributions and novelties of this work are summarized as follows:
We point out the overlooked label bias in foundation models, which originates from the skewness of pre-training distribution that affects the performance of downstream tasks.
We formalize the estimation of the label bias a constrained optimization problem (Sec- tion 3.2) with theoretical justification (Section 4.3). The entire process does not require access to the pre-training dataset, making practical for fine-tuning scenarios. We present General- ized Logit Adjustment (GLA) method, which ensembles the debiased zero-shot and fine-tuned models, and demonstrate its superiority over conventional fine-tuning and ensembling by proving it's a Bayes optimal classifier (Section 4.2)
We build a comprchensive benchmark for evaluation, which considers three real-world settings and three fine-tuning paradigms. The settings are: 1) many-shot learning with abundant data 2) few-shot learning; and 3) long-tail classification, representing a more challenging scenario that combines many-shot and few-shot data (Section [5. The three fine-tuning paradigms include: 1) end-to-end fine-tuning: 2) linear probing; and 3) prompt tuning (Section 3.1
We demonstrate the efficacy of our proposed method GLA by conducting extensive experi- ments across various settings and fine-tuning paradigms. We observe to 1.5 pp accuracy gains on ImageNet, large averaged improvement (1.4 4.6 pp) on 11 few-shot datasets and 2.4 pp averaged accuracy gains on long-tail datasets. (Section

---

2 Related Work
Image-text foundation models. Foundation models pre-trained by contrastive objectives have set impressive milestones for image and text representation learning, with CLIP 4О,, ALIGN [23] CoCa [2 and Flamingo [I] being the exemplars. Such models exhibit impressive prompt-based zero-shot performance on various image recognition downstream tasks. Our method aims to reduce foundation model biases to boost performance in downstream tasks. While 22 also addresses word frequency bias, we differ in two key areas: Firstly, we debias zero-shot models using fixed prompts, whereas 22 refines the prompting process. Secondly, our GLAA doesn't 't require access to a subset of the pre-training data.
Ensembles. Ensemble methods aim to boost performances by combining multiple networks, which can be either implemented by aggregating model outputs [12] 4 (28, [4 27 51], weight-space ensembling 48 22
or ensemble distillation T9 29]. For the adaptation of foundation models, several work propose to ensemble the fine-tuned and zero-shot models for better performance:
Wortsman al. 48 ensembles them in weight space: ProGrad and ProReg [55] (66 propose to fuse them via knowledge distillation. Our GLA is orthogonal to these approaches, as it concentrates on mitigating the biases in foundation models that are detrimental to ensemble models.
20 is a post-hoc technique to adjust the biased proposes an element-wise scaling adjustment for the classifier weight. Tang et al [45] removes the projection of features on a global biased direction. Menon (35) derives the theoretically optimal adjustment from the training distribution. Unlike the those approaches which rely on a transparent training data or class distribution, our GLA can eliminate the class bias without accessing to the pre-training statistics.
Logit adjustment. Logit adjustment 335 45
24
output of classification networks. Kang al 24
3 Methods
3.1 Setup
Task. Consider classification problem with instances X and labels y E y- [K] {1, .... K}.
We have a zero-shot model Szs (given below), a downstream dataset DD = (x,, y/ } drawn from source distribution P. and fine-tuned model Jn (given below) trained on the dataset D.. Give model f X' - R that outputs prediction score, we define the risk of ʃ on the target distribution P. as the mis-classification rate: (f) = Ex [y a argmax; (x) Our goal is to learn model faa that best leverages fzs and frt that minimizes the risk R,
Zero-shot models. We primarily explore CLIP [40] for zero-shot models. CLIP consists of a visual encoder F, and a text encoder (t),,, producing 12 normallize features from an image x and text t respectively. Zero-shot model fas for K classes is enabled by matching image features with classification weights Wt = (t&)). where t& is obtained by extending the class name {c² to a pre-defined prompt, e.g., "a photo of (cx J." Additional details are provided in Appendix B.2 The probability of x being classified as y is defined as:
(x())
P(yxx) sfftmax(fa(x)), = exp(vT wy)
(1)
ΣK exp(vTwx.)
Fine-tuned models. Standard fine-tuning initializes the model Jn with the pre-trained parameters and then solve fn. argmin, (J) to minimize the risk on downstream dataset. We consider three common variants of fine-tuning: (1) end-to-end, where all parameters of h and Wa are updated; (2) linear probing, where only Wa is modified while Φ is fixed: (3) prompt tuning, where the text input t, is learned, while keeping 4, and 4. freezed. See Appendix B.3 for details on fine-tuning methods.
Let Pv))). P. (v) and Py( be the marginal probability of class y E [K] for pre- mra-tinin source (training) and target (test) distribution, respectively. Let T. and T. denote the log probabiliiies of class for pre-training and training distribution, i.e., T (v) log (y) and x. v,()) log Pv()).
Notation.
3.2 Generalized Logit Adjustment Framework
Fine-tuned models often yield significant gains compared to zero-shot models, and ensembling them can further improve performance. This leads to natural question: How should we best leverage
3

---

410
200
400
600
690
1000
400
6013
900
1000
Soriuc caua
60) Averaged outguts of zero-shot model
Estimated label distribution
(c) Averaged outputs of detilased zero-shot model
Figure 2: Illustration of debiasing process on ImageNet validation set. (a) The original distribution of zero-shot outputs; (b) the estimated pre-traue traitrain distribution q based on our algorithm; (c) the distribution of debiased zero-shot outputs using estimated
the zero-shot and fine-tuned models for the prediction tasks? We attempt to answer by proposing generalized logit adjustment in Definition! !
Definition 1. (GLA) The Generalized Logit Adjustment (GLA) model fea defined as follows:
fgla()) = fx) + f2s(x) Te Tp.
(2)
In Section 4.2 we prove that given the zero-shot and fine-tuned models, our GLA model is the Bayes optimal classifier and no other combination of the two models can outperform it. Here remains one important question: how could we obtain Tp as we have no access to the pre-training statistics? We
provide the estimation process of Tp in Eq (4) and postpone the justification in Section 4.3 The
entire GLA algorithm consists of two steps which is given follows:
Step 1: Estimation of Tp- Let q be an arbitrary probability simplex over K classes. Given the validation data from P. or the balanced training data, we can estimate the ot = log q as the constrained optimization problem (proof in Section 4.3):
q' argmin Re(fz log q)
s.t. 2 for i [K],
Σ =1=1.
(3)
[E[K]
We constrain the sum of q to be I and ensure that each element is non-negative, guaranteeing that forms valid probability distribution. We solve the following Lagrangian problem find optimal q*:
min max Re(f2s log q) Σ>. + u(1 q 2:20,u
b.q.
q KE(K)
Σ
(4)
Step 2: GLA ensembling. Given the estimated d and the known downstream training T. we ensemble the zero-shot model fas and the fine-tuned model fn. to get GLA model fgla via Eq. (2) We can regard fz Ω and ftt π. as the debiased zero-shot model (Figure ((c)) and debiased fine-tuned models, respectively. Our GLA is actually ensembling two debiased models. Note that, different from [55 48). we do not require a hyper- parameerrre to adjust the contribution of the two models, the optimal solution is to combine them equally (see Section 4.2 for justification..
4 Theoretical Analysis
In this section, we explain why our GLA model best ensembles the zero-shot and fine-tuned models (Section 4.2) and justify the estimation process of the pre-training label distribution n, (Section 4.3) We start with some preliminaries on Bayes optimal classifier.
4.1 Preliminaries
Suppose we have pair (X,Y) P takes values in A' x V, where Y is the class label of input X.
Definition 2. The 0-1 error (risk) of a classifier à X + y on distribution given by:
R(0) P(Y # ((X))
(5)

---

However, the 0-1 error is non-smooth, one typically minimizes a surrogate loss E, e.g., cross- entropy: (x), y) log(Eie - where g(x) = argmnxx f(x); It is known that the cross-entropy loss is Bayes consistent [53), i.e. a nearly optimal minimizer of the cross- entropy loss (Bx,)=((([(xx), y)1 is also nearly optimal optimizer of the mis-classification error Exx.y>P(yy j (x))).
Definition 3. The Bayes optimal classifier y" for P given input x is defined as:
y" (x) argmax P(y|x)
(6)
yEy
It is called Bayes optimal classifier because on the average no other classifier using the same hypothesis and prior knowledge can outperform it.
Lemma 1.
The Bayes optimal classifier y² for P has lower risk than all classifiers y ; X - y
R(y") S R())
4.2 Generalized Logit Adjustment Leads to Better Ensembling
Zero-shot and fine-tuned models are complementary. We revisit an empirical phenomena observed in [48] Section 5.1: After exploring a series of measures diversity, covering predictions and features, they find that zero-shot and fine-tuned models have diverse predictions, despite sharing the same backbone. As the two data distribution P. and Pd is known to be different, the resulting models leverage different cues to predict: fine-tuned models risk exploiting spurious correlations and in- domain patterns which only hold for downstream dataset [46 3 models capture stable correlations across diverse domains and exhibit much higher robustness [40]. For instance, zero-shot models rely on robust features for decisions that can achieve high performance on sketch and adversarial samples, while the fine-tuned models that trained on real images typically fail on these samples, as they rely on spurious correlations that only hold on real images. We formulate the phenomena in the following assumption.
On the other hand, zero-shot CLIP
Assumption 1. shro-shot and fine-tuned models have diverse predictions:
(fr(()) 1 fzx(x)) y.
(8)
We derive the conditional probability Pe(y) (r (x), [x)) w.r.t. the outputs of fzz(x) and f+(()::
Lemma 2. For balanced target distributton where Pı(v) 1/K for all y E [K]. we have:
P((ulfa(x)) = softmax + (x) n. Tp (p)
(9)
(x)a(x)
Intuitively, since the zero-shot and fine-tuned models provide diverse predictions, conditioned on the two predictions is equivalent to adding the logits in log space. Additionally, as the target distribution is class- balanced, we need to remove the class bias of two models by subtracting тं and Tp. The formal proof is given in Appendix A.T Note that the RHS of Eq. is exactly the softmax output of our GL. model by Definition D. which exhibits the following property:
Proposition Let g K R + KRKK an arbitrary function that ensemble the outpuss of Sas and fre Our GLA classifier fgl has lower risk than any function fx(x =
i.c
Re(fgha) S Re(ff).
(10)
Proof. From Lemma 2 and Definition [ we have:
argmax fgla (x), argmax softmxxxfff fz((x) To Tp)y argmax Pt (x)),, f fx)(x)), yEY
vEy
vey
(11)
which means fel is the Bayes optimal classifier (see Definition 3 given fi((x) and fx) According Lemma any other classifier (fft(x) f=x))x)))))) (x)) must have higher risk, i.e., Rr S (e(.f)) O
'This lemma can be easily extended to imbalanced target distributions (proof in Appendix A Yet, as most test sets are dlassbalanced we focus the balanced case for brevity.
5

---

Proposition I demonstrates that our fgla model is the best model, as it has the lowest risk on target distribution. Proposition further explains the superiority of fgla over the fine-tuned model frt and the naive ensemble Sens [x(x)- - (zs(x):
Corollary 1. fgla performs better than fine-tuned model Jn and nuive emsembling sensi
R2 (fgls) I Re(ft), Re(fgla) S Re(fens)
(12)
Discussion: when do the GLA models degenerate? Note that there are two equality signs in Eq. (12). indicating that the performance of the GLA model can degenerate be equivalent to that of the fine-tuned model and naive ensembling in the following two cases.
Case 1: For the first equality, if zero-shot model (x) provides no further information about y given (x(x), i.e., (y T [x)( fn(x), then ((y][((()) (2s))) [x)( degenerates to (x(().(((x)) and the first equality applies. However, in practice, as downstream model and zero-shot model provides diverse predictions, we usually encounter strict inequality, i..,, Rgggg)) < R(fr).
The second equality applies when pre-tra pre-training and downstream training distribution are both Mlss-balanced. In fact, the pre-training dataset for foundation models are known to be highly skewed. Therefore, in most cases, we have R. (fgta) < Re(sens)
Case 2:
In summary, the above two equalities are usually unattainable, which means that theoretically, our GLA model performs better than both the fine-tuned and the naive ensemble models.
4.3 Estimate the label bias of the pre-training dataset
However, Tp is usually unknown we have access pre-training dataset. In this work, we seek to estimate Tp using the zero-shot models and the downstream data. Similar to Proposition [] we have the following proposition says that fas xp has lower error on target distribution than any other classifiers that use fzs. see Appendix for the full proof.
Proposition 2. Let h TRKK - R.K be an arbitrary function that predicts labels using the outputs of the zero-shot model fzs(x). ). Let the derived classifier be denoted as fa(x) = [(f=(x)).. The classifier Szs up is better than any (x): Re(fzs -p))
) S (x)).
Let q be an arbitrary probability simplex over K classes, then we have (e(F2s(x) - Tp) S (f(f((()) - log q). Therefore, we choose optimize probability simplex q over K classes such that the model fzs - log achieves the minimal empirical risk, as formulated in Eq. (3) (the Step 1 of GLA algorithm). Once we obtain the estimated class prior &p log q. we can easily implement the GLA model by ensembling fgla (x) = f+(x) + (x) - Rp (the Step 2 of GLA algorithm).
Toy experiment. We conducted an experiment show that the estimated label distribution closely approximates the true one. Specifically, we trained model with ResNet32 backbone on the imbalanced CIFAR- 10-LT [10] dataset with an imbalanced ratio of 10. Subsequently, we used only the test set combined with our proposed method to estimate the label distribution.
Distribution Comparison
Labd buton
0.35
Extimated Label Catabutio
0.30
0.25
020
This procedure simulates scenarios where only downstream data is available and the pre-training data is inaccessible.
0.15
0.55
Figure B reveals strong alignment between the estimated (orange Tine) and the actual distributions (blue line), which is further emphasized by small KL-divergence value of 0.00062.
0.00
Figure 3: Estimating label bias of LLFAR-IO-LT-IB-O..
The toy experiment validates the effectiveness of our debiasing method.
The log q we estimated is not the marginal log- log-probability of the entire pre-training distribution but the label bias matches the downstream distribution. In the above toy experiment, although training and test sets show different label distributions, their conditional distribution P(xly) remains invariant. In this case, estimate will converge to the actual training label bias. For CLIP models, with diverse pre-training data, some might not align with the downstream domain, potentially compromising the accuracy of the estimation of the entire pre-training distribution.
Discussion.
However, we'd like to point out that removing the label bias of entire pre-training distribution may not optimal for downstream tasks. As a thought experiment, consider pre-training dataset "sketch"

---

Method
B/32
B/16
Method
B/32
B/16
Method
B/32
B/16
Method
B/32
B/16
Zero-shot
63.2
68.3
Zero-shot
64.2
67.2
Zero-shot
59.7
64.4
Zero-shot
62.2
64.8
LP
75.8
79.9
LP
80.5
83.1
LP
77.1
83.1
LP
75.6
78.0
E2E
76.3
81.3
E2E
89.4
91.0
E2E
83.7
90.2
E2E
80.0
82.4
WiSE-FT
76.9
81.7
WiSE-FT
90.0
91.3
WiSE-FT
84.5
90.7
WiSE-FT
80.8
82.9
GLA
77.9
82.8
GLA
90.7
91.9
GLA
85.0
91.1
GLA
81.2
83.4
(a) ImageNet
(b) CIFARI00
(c) Stanford Cars
(d) SUN397
Table I: Accuracy various methods using CLIP ViT-B/32 and ViT-B/16. LP: linear probe; E2E: end-to-end fine-tuning, Results were obtained using the official implementation from WiSE-FT 48]
and "photo" styles for "dog" and "cat" samples. Suppose the sample size of "dog" and "cat" is equal but there are more "sketch dogs" than "sketch cats". This means that even the overall distribution is balanced, each style isn't 't resulting in biased zero-shot predictions. if we aim to deploy models for the "sketch dogs and cats" domain, adjusting the overall label bias insufficient. Instead, the optimal label bias should be estimated on the "sketch" distribution. We also provide experiments using AAION-400M dataset Appendix C.2 illustrating the situation when the downstream data diverges from the pre-training set.
Experiments
We evaluate our GLA on three real-world scenarios: many-shot (Section 5.1). few-shot (Section 5.2) and long-tail learning (Section 5.3). We show that our GLA boosts performance on all three settings.
5.1 Many-shot learning
Datasets. We use ImageNet ||| and CIFAR 100 (26 for generic object classification, Stanford- Cars [25] for fine-gratn -na-ird classification, and SUN397 50 for scene recognition. See Appendix B.] for details.
Baselines. We compare GLAA against four methods: (1) Zero-shot model, (2) Linear Probing (L.P), (3) End-to-End fine-tuning (E2F), and (4) weight ensembling method WiSE-FT
Implementation details. We consider two models: CL.IP ViT-B/32 and ViT-B/16. For learning- based models, we fine-tune with Adam using a cosine annealing learning rate scheduler. We fine-tune for 10 epochs ImageNet and 20 epochs other datasets. See Appendix B.3 for further details.
Main results. Table compares our GLA with various baselines. We observe that our GLA can increase the performance of end-to-end fine-tuned models: it achieves 1.5% gains on ImageNet. Compared WiSE-FT, GLA gains 1.1% top-1 accuracy boost on ImageNet dataset.. Beyond generic object recognition, our method also improves accuracy the fine-graines dataset (Stanford Cars) and the scene recognition dataset (SUN397), by 0.4% and 0.5%, respectively.
Breakdown performance analysis. To analyze the impact of pre-training label bias on fine-tuned and ensemble models, we present the breakdown results on ImageNet using CLIP ViT-B/16, as shown in Table 2 Specifically, we sort the class index using the estimated Tp. and assign the top third of the classes as the head classes, the last third the tail classes, and the remaining classes as the medium classes. Due to the label bias, the sero-shot tail performance is significantly lower than the head one (57.25 17. vs. 78.0%). The resulting E2E models are also affected by the bias, with the tail performance being 6.3% lower than the head. Existing ensemble WiSE-FT overlooks the bias, exhibiting noticeable degradation on the tail performances (-0.9%) compared to E2E model, while our GLAA stands out for all three subgroups.
Estimated pp is transferable across different zero-shot models. The estimated "p should be transferable across different zero-shot models if they are trained on the same pre-training dataset. To verify this, we employed CLIP ViT-B/32 based zero-shot model to estimate kp. which is subsequently used to debias zero-shot models based on CLIP ViT-B/16 and ViT-L/14. As shown in Table 3, our debiased models outperform the original zero-shot versions by clear margin.
7

---

Method
Head
Med.
Tail
All
Model
Source
Target
Zero-shot
78.0
69.8
57.2
68.3
ViT-B/32
ViT-B/16
ViT-L/14
E2E
83.6
83.0
77.3
81.3
[n(x) (xx(x) mp
63.4
68.8
75.6
WiSE-FT
85.3
83.7
76.4
81.7
65.4
69.3
76.3
GLA
85.2
84.3
78.8
82.8
Table 3: Estimated कp is transferable across different
Table 2: Breakdown results on ImageNet.
backbones, Rp is estimated using CLIP ViT-B/32.
Average over 11 datasets
ImageNet
Caltech101
OxfordPets
20-
75.0
15.0
AcCurccy 1941
Accuary 1h1
47.5
25 CUP
25
25 CLP
PS
Godp
Cedp
01.0
56
Proient
RLAGURO
47-
Prddn
0.12
0
2
31
perciass
Shots
per
StanfordCars
Flowers102
Food101
FGVCAircraft
75.0
710
6.55
Accurco 150
22-
Accurscy
450
25 CUP
25 CLF
2 CLP FST
25-CUM SLP
41.
FLOT
PLOT
2.5-
ad
ProOred CLA
Prdteed
ProCred
CLA Dsro
SLA Isand
55.
6
2
16
2
212
Shatss perciass
Shota per cless
Shots per daess
Shaik per class
SUN397
DTD
EuroSAT
UCF101
72-
70-
50-
28-
Accurscy M%
ACC//ryy Ph
FLOT
CLP
25 CLP
PACT
30-
FSCT
25-LP
25-CLI CLP
PLOT
Fistrad GLAIouni
ProGrad
GLA
GLA 0450
GLA Iaant
0.1 2
0.2
3.22
312
16
Shots percisss
Shots per cass
Shots per dass
Shots per clsss
Figure 5: Accuracy (%) of seo- learning on 11 datasets.
Ensembling with mixing coefficient. In Section we prove the optimal solution to combine the debiased zero-shot and fine-tuned models equally. We now examine the claim by introducing mix- ture coefficient a E 1]. The ensemble predictions are given by: fgla(x, a) = (1 - a) (xz(()) Tp) + a (f+(x) π²). We com- pare the GLA and the naive ensembling with mixture a in Figure 4 where GLA meets its optimal performance at = 0.5, which is in line with our theoretical analysis. We also observe that the debiased zero-shot model increases accuracy by 2.3% and our GLA consistently outperforms naive ensembling with various a.
Imapeted
25%
Figure 4: Accuracy with mixing coefficient a.
5.2 Few-shot learning
For few-shot scenarios, we primarily choose prompt tuning for fine-tuning, since it is empirically more effective than end-to-end fine-tuning and linear probing [54 55 8.
Datasets. We follow CoOp 54 to use 15 datasets: ImageNet [1. 1altech101 K3 OxfordPets
37
StanfordCars (25). Flowers 102 [36), Food101 (6), FGVCAireraft (34). EuroSAT 16
UCF101 44
8

---

Souree
Target
Method
exforoPets
IN
V2
S
A
R
Food202
StanforoCars
CLIP
59.8
52.8
35.5
22.8
60.6
Imag eNet
EursAAl
CoOp
61.9
54.3
32.5
21.8
54,2
SUN397
PLOT
63.0
55.1
33.0
21.9
55.6
Flower202
ProGrad
63.5
55.4
33.1
21.3
55.2
Cattech:02
UCF301
Naive Ensemble
GLA
65.6
57.1
36.4
23.0
62.1
FOVCAirraft
toursi
2.0
Table Evaluation on robustness to distribution shift at 16 training shots.
Absolutt Improvament over buned
Figure 6: Comparison with naive ensembling.
Model
IN.
Caltech.
Pets.
Cars.
Flowers.
Food.
Aircraft.
SUN.
DTD
EuroSAT
UCF,
Avg.
fz
59.8
87.1
85.8
55.6
65.3
77.9
17.1
60.3
42.3
37.5
61.5
59.1
Rp
62.3
89.9
85.9
57.6
67.2
78.6
18.4
63.5
43.0
39.0
62.4
60.7
A
+2.5
+2.7
+0.1
+2.0
+1.9
+0.7
+1.3
+3.2
+0.7
+1.4
+0.9
+1.6
Table Comparision between zero-shot models (fzs) and debiased zero-shot models (fzs Ap):
DTD 9. SUN397 50)
ImageNet- V2 (22
41 Sketch 47
17]) We randomly select (1. 2,
4, 8, 16) shots for training and use the original test set for evaluation. See Appendix B. for details.
Baselines. We compare with three prompt tuning methods: (1) CoOp [54) optimizes prompts via empirical risk minimization; (2) ProGrad (55) prevents forgetting the general knowledge using zero-shot predictions: (3) PLOT g applies optimal transport to match the vision and text modalities.
Implementation details. We implement our proposed GLA method using -eLP-ResNet-50 as the foundation model and adopt class spesss-pecificc prompt tuning from CoOp. Results are averaged over three seeds, with training configurations aligned with CoOp. See Appendix [.4 for further information.
Main results. Figure summarizes the few-shot results on 11 datasets. The detailed accuracy and standard deviation are in Appendix C.T Overall, our GLA clearly outperforms the baselines on average performance by large margin, e.g., our GLA gains 4.6%. 4.8%, 3. 3.1%, 2.6% and 1.6% performance boost over CoOp at 1, 2, 4, 8, 16 shots. In particular, on ImageNet dataset, we observe large improvement, e.g., 3.9%, 2.9%, 1.9%, 2.0% and 2.2% over ProGrad at 1, 4, 8, 16 shots. Furthermore, on OxfordPerts, Food101 and SUN397 datasets, our method's s performance remains stable and consistently improves with increasing sample sizes, while the one of the baseline methods fluctuates significantly. In particular, on Food101, baseline models even underperform zero-shot models by large margin, while our GLA shows clearly better performance than zero-shot models.
Robustness distribution shifts. Following CoOp, we used the ImageNet at 16 training shots as the source domain and assess the robustness on ImageNet- Sketch, A, R) datasets. Table 4 summarizes the results, where the prompt tuning baselines perform worse on distribution shifted datasets compared to the zero-shot model, fine-tuning on limited data misleads the model to learn in-distribution correlations. comparison, our GLA approach makes the best both fine-tuner and zero-shot models, thus consistently outperforms other methods on both source and target domains.
GLA improves accuracy over naive ensemble. Figure compares the results between our GLA and naive ensembling. We rank the absolute improvements over fine-tuning bascline at 16 training shots. In summary, our GLA demonstrates superior accuracy gains. It is worth noting that the naive ensembling does not always lead to improvements, e.g., on EuroSAT, SUN, Caltech, UCF and Aircraft, naive ensembling even underperforms fine-tuning baseline.
Debiased zero-shot models perform better. We estimate Tp at 16 shots and compare the original zero-shot models with the debiased zero-shot models in Table It clear that the debiasing leads to improvement on all II datasets, showcasing an average accuracy gain of 1.6%.
5.3 Long-tail learning
Datasets and metries. We evaluate our method on two standard benchmarks: Places365-LTT and ImageNet-LT [31]. In addition to top-1 accuracy, report the accuracy three test subsets

---

Method
Many
Med
Few
All
Method
Many
Med
Few
All
ERM
64.4
39.8
20.1
46.6
ERM
43.9
21.1
9.0
26.9
LA
56.7
50.5
46.4
52.3
5
LA
39.2
33.7
29.5
34.9
BS
57.7
50.5
43.9
52.4
BS
39.6
34.0
28.3
34.9
GLA
65.5
60.8
$7.7
62.2
GLA
43.6
40.2
38,8
41.1
ERM
66.6
57.7
53.8
60.6
ERM
47.7
32.6
24.9
36.5
LA
67.9
60.0
58.7
62.9
E
LA
43,9
40.6
39.4
41.5
BS
67.0
62.2
59,4
63.5
BS
43.4
42.5
42.3
42.8
GLA
70.7
66.5
64.3
67.8
GLA
47.0
47.1
47.8
47.2
ERM
75.5
56.0
36.2
60.8
ERM
46.3
27.4
12.5
31.3
LWS
70.4
62.6
56.2
64.7
LWS
32.8
38.6
LA 35
70.4
62.8
56,9
64.9
LA 35
42.2
39.3
34.0
39.3
BS 43
71.2
62.5
56,8
65.1
BS 43
46.7
44.4
39.5
44.3
WISE-FT 48
70.7
65.0
61.1
66.7
WiSE-FT
48
46.0
44.0
43.9
44.7
BALLAD
33
71.0
66.3
59.5
67.2
BALLAD
33
48.7
44.7
42.2
45.7
GLA
72.1
66.4
63.4
68.2
GLA
47.3
45.4
45.3
46.1
(a) ImageNet-LTT
(b) Places365-LTT
Table 6: The performances ImageNet-LT and Places365-LT.
according to the number of samples per class: many-shot (> 100 samples), medium-shor (20 100 samples), and few-shot (< 20 samples). Detailed information provided in Appendix [B.1
Fine-tuning and long-tail learning baselines. We compare our GLA with the combinations of fine-tuning protocols and long-tailed recognition methods. We consider three fine-tuning protocols:
1) Linear Probe (IP) 2) End-to-End (E2E) fine-tuning; 3) Prompt Tuning (PT). The three fine-tuning paradigms are introduced in Section 3.1 with details Appendix B.3] We compare with 5 long-tail learning methods: 1) standard Empirical Risk Minimization (ERM); 2) Learnable Weight Scaling
(LWS) (24: 3) Logit Adjustment (LA) [35;: 4) Balanced Softmax (BS) [43, and 5) BALLAD [3]
which is designed for VLMs. See Appendix B.6 for more details on long-tail baselines.
Implementation Details. For all combinations of the tine-tuning and long-tal -til learning baselines. visual backbones are initialized from LLIP-ResNNt-50 and classifiers are initialized via zero-shot prompting. We use SGD for 50 epochs with batch size of 512. See Appendix [B.6 further details.
Results. 6 shows that our GLA method consistently surpasses baselines across all long-tailed datasets. Our approach outperforms PT-based models by 3.5% and 4.4% on ImageNet-L.T and Places365-LT, respectively. Against E2E approaches, GLA exceeds not just the WiSE-FT but also the current SOTA method BALLAD, by large margin, e,g. Ipp gains on ImageNet-LT.
Conclusion and Limitation
In this paper, we identify the label bias in foundation models and underscore its adverse effects on downstream task performance. We propose the Generalized Logit Adjustment (GLA) framework for fine-tuning foundation models, which boosts the performance by effectively eliminating label bias and combining diverse predictions from zero-shot and fine-tuned models. We prove that when presented with zero-shot and fine-tuned models, our GLA is the Bayes optimal classifier for downstream task. Extensive experiments across a diverse range of tasks and fine-tuning framework demonstrate effectiveness of our approach. We believe that the proposed GLA may partially improve the fairness and credibility of foundation models.
The first limitation is that we only focus on the label bias while other forms of model biases, e.g., representation bias [ cannot addressed by our algorithm yet. The second limitation that we primarily focus on enhancing the fine-tuning performance for discriminative models. However, applying our GLA framework to generative models presents challenges. For instance, language generation operates a Markov process, meaning each output depends on previous ones. This implies it's not straightforward to estimate the biasedness sequence with our GLA, as we only compute the bias in a pre-defined and independent label space.
10

---

Acknowledgments
This research supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISGG-PPDD-2021-01-000 and AI Singapore AISG2-RP-2001-022
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
[2] James Urquhart Allingham, Jie Ren, Michael W Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In ICML, 2023.
[3] Martin Arjovsky. Out of distribution generalization in machine learning. PhD thesis, New York University, 2020.
[4] Eric Bauer and Ron Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine learning. 1999.
[5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. Technical Report, 2021.
[6] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101-mining discriminative components with random forests. In ECCV, 2014.
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[8] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang, Prompt learning with optimal transport for vision-language models. ICLR, 2023.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[10] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[12] Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems:
First International Workshop, 2000.
[13] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004.
[14] Yoav Freund and Robert Schapire. A decision- generalization of on-line learning and application to boosting. Journal of computer and system sciences, 1997.
[15] Jose Gallego-Posada and Juan Ramirez. Cooper: toolkit for Lagrangian-brin based constrained optimization. https://githu: com/ www.//ooopeooo///oo/ppaa 2022.
[16] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2019.
11

---

[17] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-if-distributtin generalization. In CVPR, 2021.
[18] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural
adversarial examples. CVPR, 2021.
[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NeurlPS Workshop, 2014.
[20] Youngkyu Hong. Scungju Han, Kwanghee Choi, Scokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In CVPR, 2021.
[21] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan
Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. Technical report, Zenodo, jul 2021.
[22] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. UAI, 2018.
[23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[24] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In CVPRW, 2013.
[26] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[27] Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In UAI. PMLR, 2022.
[28] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017.
[29] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, NeurlPS, 2020.
[30] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In ICML, 2018.
[31] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[33] Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple long-tain tailed recognition baseline via vision linguae model. arXiv preprint arXiv:2111.1474 2021.
[34] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aireraft. arXiv preprint 1rXiv:1306.5151 2013.
[35] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021.
[36] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
12

---

[37] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
[38] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019.
[39] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh- Minh-Thang Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111. 2021.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[41] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[42] William J Reed. The pareto, zipf and other power laws. Economics letters, 2001.
[43] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, al. Balanced meta-softmax for long-tailed visual recognition. NeurlPS, 2020.
[44] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[45] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang, Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurlPS, 2020.
[46] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. NeurIPS, 2020.
[47] Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. NeurIPS, 2019.
[48] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, 2022. https://arxiv. hbtgg//bb/2119 01903
[49] Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long-tailed distribution. CVPR, 2021.
[50] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[51] Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, owee Lim, and Hanwang Zhang. Invariant training 2d-3d joint hard samples for few-shot point cloud recognition. In ICCV, 2023.
[52] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
[53] Tong Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 2004.
[54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[55] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. ICCV, 2023.
[56] Beier Zhu, Yulei Niu, Saeil Lee. Minhoe Hur, and Hanwang Zhang, Debiased fine-tuning for vision-language models by prompt regularization. AAAI, 2023.
13

---

A Proofs
A.1 Proof of Lemma 2
Restated Lemma (Lemma 2). Let ni denote the log probability of the target label distribution
(ely) = log Pa(y), we have:
Pelulfa(x) fx)(x)) = softmax(fr(xx) + f2s(x) TT (x)
(13)
In blaaned target distribution, Eq. (3) simplifies to:
Pely)fn(x), fes(x)) voftmax((ffe(x) + fes(x) ns Tp)
(14)
Proof. Denote the output (x) and (x). We first use the Bayes Rule decompose Pe(yle (tyle,2 2) into (e.22ly) Pz(y) and (e z) in Eq. (15), then rewrite Pr(e. 2ly) in Eq. (T6) accord-
ing to Assumption Focusing on label shift problem [5 20 30 where P(x(y) does not change.
we derive Eq. (17)
z) Pi(e,zly)PP(U)
(15)
Pr(e,z)
Pe(y)
P.(el))P.(zly) Pr(e,z)
(16)
Pr(y)
Pp(el)))p(zzl))
(17)
Pi(e, z)
P.(e)))P.(e) Py(y P. Py(y)
(18)
P.(y)
P2(1)
P:(e,z)
Pa(e)) Po(ulz)
(Z)))())))(
(19)
P.(y) Po(v) ()
(20)
Since e, z are fixed, we can replace the terms that not rely on with a constant C₁ in Eq.
(1
We replace (y) cxp(log Ps(y)) = (v)( Pp(y) exp(log Pp(y)) = cxp(T,()) and
Pily) exp(log (v)) = ex (v)). Suppose the underlying class hlas-probabiilttis (y(y)) x
exp(e,)) and Pp(z)z) x exp(z,) for y E [K]. Denote the constants C, and Cp for normalizing exp(e,,) and exp(z,,) into probabilities, and merge all constants to C C.C. we get Eq. (2
Polyle.z)= = Pa(llc) Pp(ulz) P(())C; Po(y) Pv(v)
(21)
= =exple + z T Tp + (v) Ci
(22)
C,C,
=C =Ceeep nI u + xz)
(23)
Because the summation of Pi(yle, 2) is 1, 1/ exple + z T, uT + (1). Therefore,
we have:
P((l))((x) = Pr(lle 2)
(24)
exp(e + 2 - n. Tply
(25)
Dielk] exple + To Tp + TT)s
softmax (x( fxx(x) T. Tp + m2)
(26)
In class-balanced target distribution case, ne log is constant. Since the softmax function is invariant to constant offsets, Eq. (26) simplifies to:
P.(y)ff(x), fxx(x)) (ofx)ax(ff(x)++ + fzs(x) - To - -Tp)2
(27)
14

---

Dataset
Classes
Train size
Test size
Task
ImageNet
1,000
1.28M
50,000
Object-level
CIFAR100
100
50,000
10,000
Object-level
Caltech101
100
4,128
2,465
Object-level
DTD
47
2,820
1,692
Textures
EuroSAT
10
13,500
8,100
Satellite images Fine-graines aireraft Fine-gras Line-grained flowers Fine grained food Fine sene-grained pets Fine-grained car Scene-level Action
FGVCAircraft
100
3,334
3,333
Flowers102
4,093
2,463
Food101
101
50,500
30,300
OxfordPets
37
2,944
3,669
StanfordCars
196
6,509
8,041
SUN3997
397
15,880
19,850
UCF101
101
7,639
3,783
ImageNetV2
1,000
10,000
Robustness collocation
ImageNet-Sketch
1000
50,889
Robustness to sketch domain
ImageNet-A
200
7,500
Robustness to adversarial attack
ImageNet-R
200
30,000
Robustness multi-domaine
Table 7: The detailed statisties of datasets for many-shot and few-shot learning.
A.2 Proof of Proposition 2
Restated Proposition (Proposition 2) Suppose that the target distribution Pp is class-balanced. Let R + K be an arbitrary function that predicts labels using the outputs of the zero-shot model fas (x). Let the derived classifier be denoted as fix(x) h(fzs(x)). The classifier frs Tp is
better than any [x); Rr [f
Tp) S (x)xxx))..
Proof. Denote the output z fxx(x). Similar to Eq. (5)-Eq.. 266 we have
(z(y))) P.(z)))P(())
(28)
Pe(z)
Pr(zly) P.())
(29)
Pi(z)
Py(ulz) P2(y)
(30)
P,(y)) Pz(z)
-xplzz (p)( Σ exp((z TT) (1))
(31)
K
softmax(z np) = coftmax(fzs(x) - Tp)
(32)
Therefore, we have:
argmax(fzxxx) Tp) argmax softmax((zx(x) xply argmax Pe(y).fzo()) vEY
(33)
yEy
WEY
Again, using Lemma I any other classifier (x) has higher risk than f=s(x) Tp, i.e., R1 - -Tp) <
R2 (f)x.)).
B Experimental Details
B.1 Dataset details
Many-shot and few-shot datasets. For many-shot learning, we use ImageNet, CIFAR100, Stanford- Cars and SUN397 datasets. For few-shot learning, evaluate models on 15 datasets. The details of each dataset are presented in Table
Laig datasets. We use two standard long-tail benchmarks: Places365-LT and ImageNet-LT [31. The skewness of long-tailed training set is typically represented by imbalanced ratio, which defined as N Nmax/ /N Nminn Nmax (Nmin)) denotes the largest (smallest) number of instances per class. A
15

---

Size of classes
Size of many classes
Size of medium classes
Size of few classes
Size of training samples
Imbalanced
Dataset
ratio
Places365-LT
365
131
163
71
62.5K 186K
996
ImageNet-LT
1000
385
479
136
256
Table 8: Details of long-tailed datasets.
larger imbalanced ratio means more imbalanced training set. The test sets divided into three splits: many-shot subset contains classes with > 100 images, medium-shot subset includes classes with 2 20 I 100 images, and few-shot subset covers classes with < 20 images. Details are listed
in Table g
B.2 CLIP zero-shot
We use prompt ensembling of 80 prompts provided by CLIP 48 for ImageNet, CIFARI00, and Cal- 10ch101 to improve performance, i.e., averaging the text embedding many captions, e.g.., "a photo of [c,]." " and "an image of {..).". For OxfordPets, StanfordCars, Flowers 102, Food101, FGV- CAircraft, EuroSAT, UCF101, DTD and SUN397, we use the pre-defined prompt from CoOp [54).
B.3 Fine-tuned models
End-to-end and linear probe fine-tuning. We follow WiSE-FT 48 to implement fine-tuning.
We initialize the classifier with the zero-shot classifier and the output of the image encoder 4, is normalized during fine-tuning, We fine-tune for total of 10 epochs using Adam W (32) optimizer
with default hyper-parameters 31 0.9, .3 0.999, 10- and weight decay 0.1. We choose
batch size of 512. We use the same data augmentation and cosine-annealing learning rate schedule
as
B.4 Prompt tuning.
Prompt tuning like CoOp 54 automates prompt engineering by learning the prompt given few samples from downstream tasks. CoOp provides two options of prompt design: unified prompt that is shared among all classes and class-spccific prompt that is different for each class. In this paper, we adopt the class-specific prompt design as the fine-tuned model implement GLA In specific, given the word embedding tß initialized by zero-shot prompts, we aim to learn a collection class-specific word embedding R = (rx 1 such that the text input t₃ = th + ry minimizes the empirical risk: R' argming Ex,y [v + argmax, f(z; R).].
We adhere CoOp to use CLIP ResNet-50 as image encoder for few-shot classification. The word embedding R. is initialized from zeros. For the m few-shot classification setting (where m E 21, 4, 8, 16)), we randomly sample m training and m validation points from the respective full datasets. For all few-shot datasets except ImageNet, the training epoch is set to 200 for 16/8 shots, 100 for 4/2 shots, and 50 for shot. For ImageNet, the epoch is set to for all shots. We fine-tune the prompt with SGD optimizer decayed by the cosine annrealing rule. The base initial learning rate and batch size are set to 10-4 and 32. When given an m-shot sample setting, we increase the learning rate and batch size by times simultaneously to accelerate the training speed.
B.5 Estimation the class prior
To estimate the log-probability of the pre-training distribution म. log q, we utilize the optimiza- tion toolkit Cooper [15] from https: /tt////gtttbbbb com/ cooper- org/ cooper is initialized as uniformed distribution, q(y) for y []]. We use the standard SGD as the primal and dual
optimizers for 2000 steps.
B.6 Long-tail learning baselines and training details
We compared with 5 long-tailed classification methods:
16

---

1. Standard ERM: We learn the model by standard empirical risk minimization on the long-tailed data.
2. Learnable Weight Scaling (LWS) 24 We first learn the model by standard ERM, then fix the model and learn to re-scale the magnitude of the classifier using blass-balanced sampling.
3. Logit Adjustment (LA) [35 the long-tailed distribution by subtracting class-dependent offset to the model outputs.
We first learn the model by standard ERM, then compensates
4. Balanced Softmax (BS) 4 modifies the Softmax cross-entropy loss which explicitly accom- modate the label distribution shift during optimization.
5. BALLAD 53 first fine-tunes the vision-language models via contrastive loss on long-tailed data, then freezes the backbone and finally employs an adapter to enhance the representations of tail classes with re-sampling strategies.
For all combinations of the fine-tuning baselines and long-tailed learning methods, visual backbones are initialized from Cee--ResNett500 and all classifiers are initialized by feeding prompt with class names to the text encoder. We use SGD for all experiments with momentum of 0.9 for 50 epochs with batch size of 512. The initial learning rate is set to 1.6 x 10-3 33 which is decayed by the cosine annealing rule. To mitigate explosive gradients, we use the warmup learning rate equals to 10 during the first epoch. For the sake of fairness in comparison, all typer-parameters of baselines are carefully searched using grid search on the validation set.
C Additional Experiments
C.1
Few-shot learning accuracy
We provide mean and standard deviation in Table in in for 21, 4, 8, 16) shots all 11 few-shot learning datasets.
Dataset
shot
2 shots
4 shots
8 shots
16 shots
ImageNet
61.65 + 0.15
62.64 + 0.01
63.32 + 0.07
64.51 + 0.09
65.61 + 0.03
10
89.08 + 0.09
90.25 + 0.25
90.98 + 0.43
91.90 0.21
92.58 + 0.42
OxfordPets
87.79 + 0.15
87.86 + 0.21
88.22 0.21
88.09 0.27
89.53 #
StanfordCars
60.00 1 0.14
63.10
66.25 1
69.87 0.09
73.95 0.11
Flowers 102
73.45 1 0.60
81.00
88.31 0.65
92.89 1 0.46
95.41 0.32
Food101
78.41 1: 0.07
78.62
78.68 1 0.06
78.85 0.19
79.54
FGVCAireraft
20.22 0.59
22.09
24.65 0.85
28.23 0.44
31.99 0.50
SUN397
64.29 0.19
68.01 0.08 0.16
71.64
DTD
47.38 + 1.23
50.75
00.90
0.80
65.78 0.49
EUrOSAT
56.50 + 1.34
67.26 + 3.58
72.40 + 2.43
77.59 + 1.84
84.93 + 1.89
UCFI01
65.32 + 0.17
68.42 + 0.81
70.88 + 0.50
74.23 + 0.24
76.07 + 0.03
Table 9: GLA Accuracy (%)
with standard deviation of few-shot learning on 11 datasets.
C.2 Experiments on LAION-400M
To support our thought experiment in the discussion of Section 4.3 we use the Open-CLIP ViT- B/16 [1], the first 20k image in LAION-400M datasets and the bias estimation method proposed by 2 to estimate the expected logits across 8 classes: "dog", "cat", "squirrel", "tiger", "elephant", "horse", "pig" and "bird". The bias estimation proposed by [2] provides a good estimation of log P(x) over the labels under pre- pre-training distribution. Our GLA estimates the label bias matches the downstream domain, we consider two downstream domain styles, i.e., "photo" and "sketch" from
DomainNet (38] dataset. For each domain, we randomly sampled 50 images for each class. bird
Method
dog
cat
squirrel
tiger
clephant
horse
pig
expected logits by 2
0.059
0.039
0.043
0.053
0.043
0.062
0.061
0.028
Tp by "photo" Tp by "sketch"
0.059
0.041
0.047
0.055
0.048
0.062
0.060
0.033
0.059
0.043
0.063
0.061
0.067
0.042
0.047
0.056
Table 10: Comparison among different bias estimation using Open-CLIP-ViT-BIII
We present the expected logits estimated by 2 along with the one calculated from "photo" and
"sketch" downstream domain data in Table I0 Since the softmax is invariant to constant offsets, i.e.
17

---

Method
Sketch
Real Photo
Zero-shot Open-CLIP
92.25
97.00
Dehiased by 2
89.50
97.50
Debiased by GLA
93.00
97.75
Table 1T:
Performance on sketch and real photo domain.
softmax(x + c) softmax(x), we align the three label bias to yield the same logits on "dog" class by subtracting constant values. We observe that the label bias estimated by the "photo" domain align closely with [2] due to its close resemblance to the pre-trained domain. Conversely, the "sketch" image style, which significantly differs from the pre-training domain, results in a more pronounced deviation in the pre-trained label bias.
Additionally, apply the three estimated label biases to debias the zero-shot model and evaluate the classification performance. The results are shown in Table II where the superiority of our method becomes evident "sketch" domain (93.00% vs 92.25%). Applying the label bias from 2 on the "sketch" domain degrades the model's 's performance (from 92.25% to 89.50%). This is attributed to the overall pre-training label bias does not adequately reflect the bias specific the""ketthh domain.
18


Reviews:
[{"review_id": "1OA9JCkdx1", "reviewer": "Reviewer_wcSK", "summary": "This paper proposes a method for combining the predictions of zero-shot and few-shot classifiers. The method removes the need for a weighting hyper-parameter to interpolate between the predictions. Furthermore, it also removes a source of bias (due to the frequency of words in the pre-training dataset).  ", "questions": "Additional experimental and theoretical analysis showing that $\\log \\mathbf{q} \\approx \\pi_p$ would significantly strengthen the paper. Concretely, if this weakness were addressed, I would happily increase my score by 1-2 points. \n\nThe true value of $\\pi_p$ could, for example, be estimated by using the LAION-400m dataset, which is similar in scale and content to the true pre-training dataset of CLIP and results in similar zero-shot accuracies. This was the approach taken in the \"A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models\" paper (https://arxiv.org/abs/2302.06235), which addresses similar biases in the context of prompt selection in zero-shot classifiers.  \n\nAdditionally, including the above-mentioned paper in the related work would be useful.\n\n\n\n", "limitations": "The theoretical and experimental analysis does not support the main claim of the paper, as discussed above under \"weaknesses\". ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Simplicity \n\nThe proposed method is both simple to understand and implement. This means that it could have a significant impact. Simple methods can easily be deployed by both practitioners and by researchers (as a strong baseline for future work). \n\n## Presentation\n\nThe presentation of the method is clear, concise, and easy to understand. \n\n## Experimental Results\n\nThe experimental results show that the proposed method improves over a selection of sensible baselines in a range of settings, with some strong improvements in several cases. \n", "weaknesses": "## Correctness\n\nWhile the proposed method clearly results in improved performance in several settings, it is not clear to me that the reason for the improvement suggested by the authors is correct. That is, I am not convinced that the proposed method to estimate the label bias of the pre-training dataset is working as intended. Does $\\log \\mathbf{q}$, αs described in eq 3 and sec 4.3, provide a good estimate of $\\pi_p$? I do not believe that the paper provides theoretical or empirical evidence that this is the case and that the mechanism behind the success of the method is due to the suggested bias improvement, rather than something else. \n\n##############\n\n### Edit, after rebuttal and discussion. \n\nThe authors have now addressed my concerns above. They have demonstrated that their approximation is accurate when the downstream and pre-training datasets are similar. They have also shown that when the downstream and pre-training datasets are different, the approximation becomes worse. Nonetheless, the proposed method still works well, and the authors have given intuition for why this might be the case. Given the authors proposed updates to the paper, I believe that the story of the paper will now be accurate and clear. I've raised my score from 5 to 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1** Additional experimental and theoretical analysis showing that $\\log \\mathbf{q}=\\pi_p$\n\n**A1**\nTo further address the concerns about our bias estimation, we further provide explanation of our Proposition 2 and some empirical evidences.\nIn Proposition 2, we proved that when given the outputs of the zero-shot model $f_\\texttt{zs}(\\mathbf{x})$, $f_\\texttt{zs}(\\mathbf{x}) - \\pi_p$ will be Bayes optimal classifier for the downstream distribution $P_t$, therefore it is the minimizer of $R_t(f_h(\\mathbf{x}))$. In Proposition 2, $f_h(\\mathbf{x})$ is defined as the arbitrary classifier that uses $f_\\texttt{zs}(\\mathbf{x})$,  it includes the hypothesis of $f_\\texttt{zs} - \\log \\mathbf{q}$. Therefore, $\\pi_p$ is also the minimizer of Eq.(3). By solving the constrained optimization problem, we can get a good estimation of pre-training label bias. \nWe subsequently provide two pieces of evidence to further validate the correctness of label bias estimation.\n\n**Evidence 1: Experiments to compare the true label bias and the estimated one.**\nWe devise more intuitive experiments where we have the true label bias at hand. In specific, we first train models using ResNet32 backbones on an imbalanced training set: CIFAR10-LT [1], then only use the test set to estimate the label distribution.  Figure 1 in the attached document illustrates a strong correlation between the estimated and real distributions, further supported by a small KL-divergence (0.00062). The experiments demonstrate the correctness of our proposed debiasing method.\n\n**Evidence 2: Estimated $\\pi_p$ is transferable across different zero-shot models.** In Section 5.1 and C.1, we demonstrate that when different models are pre-trained on the same pre-training dataset, the pre-trained bias estimated by one zero-shot model (e.g. CLIP-ViT-B/32) can be subsequently transfered to debias other models (e.g. CLIP-ViT-B/16 and CLIP-ViT-B/14). As shown in Table 9, debiased CLIP-ViT-B/{16, 14} show clear performance gains over original zero-shot models. \n\n---\n\n**Q2**  Discuss [2] in the related work.\n\n**A2**    We thank the reviewer for the suggestion. \n    [2] automates the prompt engineering by prompt scoring, which also targets on alleviating word frequency bias in pre-training data. Our approach diverges from [2] in two main aspects.\n    The first difference is that our approach focuses on debiasing zero-shot models given fixed prompts, contrasting with [2] that optimizes prompting process. Secondly, unlike [2] that necessitate access to a subset of the pre-training data, our GLA is exempt from this requirement. We will include the discussion in the related work in the revised version.\n\n----\n\n[1] Cui et al. \"Class-balanced loss based on effective number of samples.\" CVPR 2019.\n\n[2] Allingham et al. A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models"}, {"review_id": "z0pwB3kK1D", "reviewer": "Reviewer_8Trg", "summary": "Logit adjustment method has been widely used in long-tailed recognition, in which the source dataset is available, thus the class margin can be set in advance. However, the large-scaled foundational data cannot be accessed. To reduce the recognition bias of foundational models on downstream tasks, this paper proposes a generalized logit adjustment (GLA) method. The main contribution of GLA is to estimate the class margins for zero-shot models, after that, the fine-tuned and debiased zero-shot models are assembled together to achieve consistent improvement. The authors demonstrate the effectiveness of the proposed method through extensive experiments, including three tasks and three finetuning approaches.\n\n\n", "questions": "1) How to initialize q?\n\n2) Your assumption in Lemma 2 does not always hold, the target distribution may be unbalanced.\n\n3) Please provide more analysis for your proofs. The constant C_2 in Eq. 21 and the conclusion of Eq. 30 are not easy to understand.", "limitations": "The authors discuss the limitations of their method.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Estimate the class margins for foundational models is an important task, especially when the pretrained datasets are not available. This paper proposes an effective generalized logit adjustment method for the zero-shot models. \n\n2) To estimate the class margins, the authors consider it as a simple constrained optimization problem. They also provide theoretical analysis for their GLA method. The proofs seem to be ok.\n\n3) The authors verify their method with extensive experimental results, including many-shot learning, few shot learning and long-tailed recognition tasks. The presented results deem to be good.", "weaknesses": "The main weaknesses include the following three aspects.\n\n1) The proposed method is simple, and the theoretical proofs look a little reluctant.\n\na. Section 4.1 sees to be redundant. The Definitions and Lemma are less important for the following analysis.\n\nb. The proof and conclusion of Proposition1 are not convincing. \n\nc. There are some false assumptions about the discussion of Corollary 1. In case 1, the authors claim that the zero-shot model can not provide further improvement when the fine-tuned model is given. Obversely, this is wrong. Because their combination can further improve recognition accuracy.\n\n2) The authors did not show the algorithm steps when estimating class margins (Eq. 3). They also did not provide the complexity and convergence of this optimization problem, and the choice of hyper-parameter \\lambda. \n\n3) Unfair comparisons make results look good. For example, in long-tail learning, the proposed GLA is an ensemble model with larger network parameters. Thus, it is not surprising that it outperforms other baselines. Please provide the model parameters and inference time to show the improvement.\n\nOther comments:\n\n1) Typo in line 109: w_k is not model parameter.\n\n2) In Eq.4, v should be bold.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1** The method is simple, and the proofs look a little reluctant.\n\n**A1** We hold a different view. **A simple method with statistical  grounding is our strength rather than weakness**. As Reviewer *wcSK* claimed ``The proposed method is both simple to understand and implement. This means that it could have a significant impact. Simple methods can easily be deployed by both practitioners and by researchers (as a strong baseline for future work). ''. In fact,  it is true evidenced by our experiment: despite simple, our method can easily beat more complicated methods like BALLAD, PLOT and ProGrad, which shows the value of our work. Regrading to proofs, we believe there exists some misunderstanding, and we elaborate the details as below to address your concerns.\n\n---\n**Q2** Section 4.1 is redundant.\n\n**A2**  We gracefully disagree. Definition 1 and Lemma 1 are crucial. In Proposition 1, we first use Definition 1 to prove our GLA model is Bayes optimal. Then, we leverage Lemma 1 to prove our GLA model has the lowest risk. We use Definition 1 and Lemma 2 to prove Proposition 2. Without Section 4.1, the proof will be incomplete.\n\n---\n**Q3** Explain the conclusion of Eq. 30 \n\n**A3** First, we'd like to clarify that the conclusion of Eq.30 hinges on the preliminaries in Section 4.1. Therefore, the weakness  you raised, ``Section 4.1 seems to be redundant. '' stems from a misunderstanding of the  dependence between Section 4.1 and the subsequent proofs. Eq. 30 is restated:\n\n$\\arg\\max_{y}(f_{zs}(\\mathbf{x})-\\pi_p)_y $\n\n$= \\arg\\max_{y} softmax(f_{zs}(\\mathbf{x})-\\pi_p)_y$\n\n$=\\arg\\max_{y} P_t(y|f_{zs}(\\mathbf{x}))$\n\nThe first equality applies because $\\text{softmax}(\\cdot)$ does not change $\\arg\\max$ results. The second equality holds because of Eq.29. Recall that **Definition 3**: The Bayes Optimal classifier $y^*$ for $P$ given input $\\mathbf{x}$  is defined as:\n$y^*(\\mathbf{x}) = \\arg\\max_{y} P(y|\\mathbf{x})$\n\nGiven $f_\\texttt{zs}(\\mathbf{x})$ as the input, $\\arg\\max_{y}(f_\\texttt{zs}(\\mathbf{x}) - \\pi_p)(y)$ is the Bayes optimal classifier, as it equals to $\\arg\\max_{y\\in \\mathcal{Y}}P_t(y|f_\\texttt{zs}(\\mathbf{x}))$. \nAccording to **Lemma 1**: The Bayes optimal classifier $y^*$ for $P$ has lower risk than all classifier $\\hat{y}: \\mathcal{X} \\rightarrow \\mathcal{Y}$. $\\mathcal{R}(y^*)\\leq \\mathcal{R}(\\hat{y})$.  Therefore we have the conclusion in Line 338-339: ``any other classifier $f_h(\\mathbf{x})$ has higher risk than $f_\\texttt{zs}(\\mathbf{x})-\\pi_p$.''\n\n---\n**Q4** Explain C_2 in Eq.21 \n\n**A4** Because $P_s(y|\\mathbf{e})=\\text{softmax}(\\mathbf{e})_y$ and $P_s(y|\\mathbf{z})=\\text{softmax}(\\mathbf{z})_y$. For some constants $C_s, C_p$, we can express $P_s(y|\\mathbf{e})=\\exp(\\mathbf{e})_y/C_s$ and $P_p(y|\\mathbf{e})=\\exp(\\mathbf{e})_y/C_p$. Replace this into Eq. (20), we get Eq. (21):\n\n$=\\exp(\\mathbf{e}+\\mathbf{z}-\\pi_s-\\pi_p+\\pi_t)_yC_1/(C_s \\cdot C_p)=\\exp(\\mathbf{e}+\\mathbf{z}-\\pi_s-\\pi_p+\\pi_t)_yC_1/C_2$\n\nWe denote $C_2=C_s \\cdot C_p$ to simplify Eq. 21. \n\n---\n**Q5** Target distribution may be unbalanced in Lemma 2\n\n**A5** We argue that it is very easy to extend our Lemma 2 to unbalanced target distribution. \n    In Eq.20 we incorporate $P_t(y)$ into constant $C_1$ as we assume $P_t(y)=1/K$ is constant. For an unbalanced target distribution, we modify Eq.20 into\n\n$P_t(y|\\mathbf{e},\\mathbf{z})=\\frac{P_s(y|\\mathbf{e})}{P_s(y)}\n        \\frac{P_p(y|\\mathbf{z})}{P_p(y)}P_t(y) C_1, $\n\nLet $\\log P_t(y)=\\pi_t(y)$, we rewrite Eq.21 as:\n$=\\exp(\\mathbf{e}+\\mathbf{z}-\\pi_s-\\pi_p+\\pi_t)_yC_1/C_2 $\n\nUsing Eq.21-24, we arrive at our Lemma 2 under imbalanced target distribution:\n$ P_t(y|f_{ft}(\\mathbf{x}),f_{zs}(\\mathbf{x}))=softmax(f_{ft}(\\mathbf{x})+f_{zs}(\\mathbf{x}) -\\pi_s-\\pi_p+\\pi_t)_y $\n\n---\n**Q6**  How to initialize q? Algorithm steps in Eq. 3?\n\n**A6** We have detailed these in Line 383 and 384: q is initialized as a uniform distribution and optimized for 2,000 steps.\n\n---\n**Q7** Complexity and convergence of the optimization.\n\n**A7** From Proposition 2, we know that $\\mathbf{q}=\\exp(\\pi)$ is the minimizer of Eq.3. Thus, the optimization problem is guaranteed to have a solution. We use gradient descent to solve the problem, whose complexity is $O(nkm)$ where $k$ is the iteration steps, $n$ is the  sample size and $m$ is the label size. \n\n---\n**Q8** hyperparameter lambda.\n\n**A8** We'd like to like to clarify that lambda is **not a hyperparameter**. lambda is updated to maximize the Lagrangian function in Eq.4 rather than a pre-defined hyper-parameter. \n\n---\n**Q9**  In case 1, the authors claim that the zero-shot model can not provide further information when the fine-tuned model is given. Obviously, this is wrong.\n\n**A9** There seems to be a misunderstanding. In fact, **we are on the same page: we also claimed Case 1 is unlikely to happen**. Case 1 discusses the situation when $R_t(f_{gla})=R_t(f_{ft})$, i.e., our GLA models degenerate to fine-tuning models. This situation happens when ``the zero-shot model can not provide further improvement when the fine-tuned model is given''. As you claims, this is not likely to happen, which supports our claims.\n\n---\n**Q10** Ensemble is unfair.\n\n**A10** We see your point, but we gracefully disagree. First, our GLA does not involve external data or models, all baselines are given the same zero-shot models and fine-tuning data. Second, our baselines include other ensemble methods, e.g. WiSE-FT and naive ensemble, and our GLA shows superiority. Given that we share the same model parameter and inference time as these ensemble baselines, we believe the comparison is fair. \n\n---\n**Q11** w_k is not model parameter\n\n**A11** We're sorry that you misunderstood the point. w_k is the parameter of classification head initialized by prompting: it is updated during fine-tuning. \n\n---\n**Q12**  v should be bold\n\n**A12** That is not true. We only have **one** equality constraint in Eq.3: $1-\\sum_{i\\in [K]} \\textbf{q}_i$. Therefore, v should be a scalar rather than be bold.\n"}, {"review_id": "fNZBVx6JLM", "reviewer": "Reviewer_iLax", "summary": "This paper proposes a distribution estimation method that estimates the distribution of pre-training data for large VLMs like CLIP. Combined with downstream data distributions, a logit adjustment is applied to the outputs of both the zero-shot model and the fine-tuned model. By using a simple ensemble of the zero-shot and fine-tuned models, improved performance is achieved on the reported benchmarks.", "questions": "Same as above.", "limitations": "The biggest limitation is the limited performance improvements.", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The proposed method introduced performance improvements over compared baselines on most of the reported benchmarks.", "weaknesses": "Here are some questions: \n1) The claim \"their predictions are complementary to those of fine-tuned models\" in line 30 cannot be supported by the cited references. Neither [3] nor [14] can support fine-tuned and zero-shot models are complementary. \n2) In addition, in Section 4.2, where the authors attempt to demonstrate the complementarity of fine-tuned models and zero-shot models, the reasoning provided is not persuasive. The authors argue that fine-tuned models are biased towards downstream distributions, while zero-shot models are robust to distribution shifts. However, it remains unclear why a distribution shift-robust model should be complementary to a model that has already been adapted to a shifted distribution. If the authors claim and can prove that zero-shot models can provide something that is missing in the fine-tuned model and cannot be provided by the shifted downstream dataset, it needs to be explicitly demonstrated and proven. Otherwise, this assumption lacks solidity, which consequently undermines the validity of all subsequent proofs.\n3) The rationale behind the method used to estimate the pre-training data distribution is not clearly explained. The paper does not provide sufficient information on why the method is designed in this particular way, nor does it address whether there are any means to verify the effectiveness of this method in accurately estimating the pre-training data distribution. Even though pre-training data is often proprietary, the authors could still perform toy experiments to demonstrate the efficacy of the proposed method. Given that this is a crucial component of the proposed method, it should not be overlooked.\n4) The biggest weakness of this paper is the improvements introduced by the proposed method is very limited (~1% on average across all reported benchmarks). \n\nEdit: most of my concerns are clarified. I will change my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**The biggest weakness of this paper is the improvements are very limited.\n\\end{point}\n\n**A1**We hold a different view. For ImageNet-1K full training with 1,000 classes, improving $1.5 \\%$ accuracy over end-to-end fine-tuning is a noteworthy improvement. For ImageNet with 16 training shots, we observed a $3.5\\%, 2.6\\%, 2.1\\%$ improvement over the previous prompt-tuning SOTA: CoOp, PLOT and ProGrad. On Place365-LT datasets, our GLA that only tuning prompt can even surpass the previous end-to-end fine-tuning SOTA BALLAD by a large margin ($47.2\\%$ vs. $45.7\\%$). \n    In light of these results, it is clear that our GLA demonstrates substantial gains across various datasets.\n\n---\n**Q2**The claim \"their predictions are complementary to those of fine-tuned models\" in line 30 cannot be supported by the cited references. Neither [3] nor [14] can support fine-tuned and zero-shot models that are complementary. \n\n**A2** The underlying assumption of ensembling methods is that the individual models make independent errors[3,14] (See details in Section V.4 Independent Errors of [14]) otherwise the performance of ensembling will degenerate to the one of single model~(Imagine if we ensemble two identical models, the ensemble's performance will not improve). In the context of fine-tuning zero-shot models, this assumption is supported by empirical studies conducted in Section 5.1 of [1] which explores a series of measures of diversity, finding the complementarity of predictions between zero-shot and fine-tuned models.\n\n[1] Wortsman et al. Robust fine-tuning of zero-shot models. CVPR. 2022.\n\n---\n**Q3** In addition, in Section 4.2, where the authors attempt to demonstrate the complementarity of fine-tuned models and zero-shot models, the reasoning provided is not persuasive. The authors argue that fine-tuned models are biased towards downstream distributions, while zero-shot models are robust to distribution shifts. However, it remains unclear why a distribution shift-robust model should be complementary to a model that has already been adapted to a shifted distribution. If the authors claim and can prove that zero-shot models can provide something that is missing in the fine-tuned model and cannot be provided by the shifted downstream dataset, it needs to be explicitly demonstrated and proven. \n\n**A3** We gracefully disagree. Assumption 1, diversity of zero-shot and fine-tuned models, often featured in theoretical analyses in distribution shifts [1-4] and empirically verified by prior work[5], appears to be a judicious assumption to adopt. \n\nIntuitively, zero-shot models and fine-tuned models leverage different cues to predict. For instance, zero-shot models rely on robust features for decisions that can achieve high performance on sketch and adversarial samples, while the fine-tuned models that trained on real images typically fail on these samples, as they rely on spurious correlations that only hold on real images.  \n\nIn fact, Assumption 1 is a weaker assumption compared to commonly used assumptions in  distribution shifts community [1,2,3]. In these papers, the in-distribution and out-of-distribution features are assumed to be disjoint parts of the inputs, each generated independently based on the label. In our paper, we only require the conditional independence of the outputs. In addition, we can find the same assumption to describe the relation between the fine-tuned models and the zero-shot CLIP in [5]. \n    \nCrucially, empirical studies detailed in Section 5.1 and Appendix Section E of [5] explore a range of diversity measures across predictions and features, revealing that zero-shot and fine-tuned models, despite a shared backbone, yield diverse predictions, which supports our claim.\n\nTo sum up, we just follow the commonly used assumption in theoretical analysis of the robustness/out-of-domain generalization community, which is also empirically proven to be reasonable.\n\n    [1] Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. ICLR, 2021\n    \n    [2] Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. NeurIPS, 2020.\n        \n    [3] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. ICML, 2020\n    \n    [4] Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. UAI, 2022.\n    \n    [5] Wortsman et al. Robust fine-tuning of zero-shot models. CVPR. 2022.\n---\n\n**Q4** The rationale behind the method used to estimate the pre-training data distribution is not clearly explained. \n\n**A4** The rationale behind the method (Step 1) to estimate pre-training label distribution has been justified in Proposition 2: the pre-trained label prior $\\pi_p$ can be achieved when $f_{zs}-\\log \\mathbf{q}$ arrives at the lowest error $R_t$, as we prove that such model corresponds to the Bayes optimal classifier given fixed $f_{zs}$. \n\n---\n**Q5** Even though pre-training data is often proprietary, the authors could still perform toy experiments to demonstrate the efficacy of the proposed method.\n\n**A5** To address your concerns, we devise a toy experiment where we have the true label bias at hand. In specific, we first train model on an imbalanced training set: CIFA10-LT~[1], then only use the testing set to estimate the label distribution. Figure 1 in the attached document illustrates a strong correlation between the estimated and real distributions, further supported by a small KL-divergence (0.00062). The toy experiments demonstrate the correctness of our proposed debiasing method.\n\n[1] Cui et al. \"Class-balanced loss based on  effective number of samples.\" CVPR 2019.\n"}, {"review_id": "N7AiIBe1oS", "reviewer": "Reviewer_egNQ", "summary": "This paper studies the label bias in Foundation Models, like CLIP. Specifically, authors model the estimation of the label bias as a constrained optimization problem, and propose a Generalized Logit Adjustment (GLA) method to debias foundation models. Experimental results demonstrate the effectiveness of proposed method.", "questions": "- The Proposition 1 shows that $f_{gla}$ is the best model, or nearly optimal Bayesian classifer. Thus what leads to the error between the realistic results and the ideal performance of optimal Bayesian classifer.\n- What's the source of the validation data for a specific problem. In all experiments, the compared  fine-tuning methods whether or not to use the validation data.\n- Please illustrate that ''Intuitively, since the zero-shot and fine-tuned models provide diverse predictions, conditioned on the two predictions is equivalent to adding the logits in log space\". ", "limitations": "The proposed method can effectively deal with label bias in foundation models. However, it depends on some strong assumptions. E.g., the estimation relies on the validation data. How to guarantee that a good estimation can be obtained with poor validation data.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  Show that the skewness of pre-training distribution affects the performance of downstream tasks, i.e., label bias of foundation models.\n-  Propose a GLA method to ensemble the debiased zero-shot and fine-tuned models.\n-  Extensive experimental results demostrate the superiority of proposed method over conventional fine-tuning.", "weaknesses": "- Authors should discuss the difference with existing label bias estimation methods, e.g. [1,2].\n-  [3] also studied the label bias in CLIP. What's the difference? \n- There exist some strong assumption for proposed method.\n\n[1] Lipton Z, Wang Y X, Smola A. Detecting and correcting for label shift with black box predictors. In ICML, 2018: 3122-3130.\n[2] Garg S, Wu Y, Balakrishnan S, et al. A unified view of label shift estimation. In NeurIPS, 2020, 33: 3290-3300.\n[3] Wang X, Wu Z, Lian L, et al. Debiased learning from naturally imbalanced pseudo-labels. In CVPR, 2022: 14647-14657.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1** Discussion with existing label estimation methods[1][2].\n\n**A1** [1-2] and our GLA work under different scenarios and require different forms of input. \n[1-2] face a problem where the test prior is unknown but training data are given. In the context of foundation model adaptation, [1-2] require both \\textbf{labeled pre-training data} and **unlabeled downstream test data** to estimate label shift by matching the moment of confusion matrices. However, the pre-training data is often inaccessible, which makes [1-2] impractical for foundation models. In contrast, the proposed GLA targets on the setting where the pre-training label prior is unknown but the test distribution is class-balanced. \nOur method estimates the label bias of pre-training data by adjusting the margin to produce a Bayes optimal classifier.\nTechnically, we only have to access **downstream source data** for debiasing. We will add the above discussion in the revised version.\n\n---\n**Q2** What's the difference with [3]?\n\n**A2** Our GLA converges to the true class-probability of pre-training data while the one estimated by DebiasedPL[3] is asymptotically biased to the downstream distribution.\n Recall that the DebiasedPL[3] uses the moving average of the outputs of zero-shot model $P_\\text{zs}(y|\\mathbf{x})$ (predicted probability of downstream data) as the class-specific margin for debiasing. \nThe  probability for class $y$ estimated by DebiasedPL[3] can be viewed as \n$\\hat{p}(y)=\\frac{1}{N} \\sum_{i=1}^N  P_{zs}(y|\\mathbf{x}_i)$, where $\\mathbf{x}_i$ is the sample from downstream dataset $\\mathcal{D}_s \\sim P_s^N$. \n\nAs $N \\rightarrow \\infty$, $\\hat{p}(y) \\rightarrow E_{\\mathbf{x}\\sim P_s}[P_{zs}(y|\\mathbf{x})]$. \n\nHowever, the true class-probability of pre-training distribution should be $p(y)=E_{\\mathbf{x} \\sim P_p}[P_\\textbf{zs}(y|\\mathbf{x})]$. As long as $P_s \\neq P_p$, i.e., pre-training distribution differs from downstream distribution, the class-specific margin estimated by [3] is biased. In contrast, In Proposition 2, we prove that $f_\\text{zs}(\\mathbf{x})-\\log \\mathbf{q}$ is the Bayes optimal classifier given $f_\\text{zs}$ that achieves the lowest error on the target distribution. Our estimation of $\\hat{\\mathbf{q}}$ converges to the true one. \n\nNote that we are not intended to undermine the value of [3], as they focus on semi-supervised learning where the labels of training data are not available. \nIn contrast, our GLA is given labeled fine-tuning data and able to estimate a more accurate label shift. We will add the discussion in the revised version.\n\n---\n**Q3** There exist some strong assumption for proposed method. E.g., the estimation relies on the validation data. How to guarantee that a good estimation can be obtained with poor validation data?\n\n**A3** We are sorry that we did not emphasize the implementation details, leading to your misunderstanding.\nIn all conducted experiments, **no validation set** was used for $\\pi_p$ estimation. Recall that in line 126-127, we mentioned that we have two choices: ``we can use validation data or the balanced training data'' to estimate $\\pi_p$. As the training samples are more abundant than validation samples, we choose to use training set rather than validation set. For many-shot and few-shot learning, as the training set is balanced at hand, we directly use it for $\\pi_p$ estimation. For long-tailed learning, we re-balanced the training set by up-sampling to estimate $\\pi_p$. For the sake of fairness in comparison, the validation set is only used to search hyper-parameters for all baselines and our methods.\n\n---\n**Q4** What leads to the error between the realistic results and the ideal performance of optimal Bayesian classifier?\n\n**A4** Our Proposition 1 analyses the asymptotic behavior of GLA. In practice, GLA's performance hinges on the estimation of pre-training $\\pi_p$, a factor influenced by the pre-trained model's generalization gap. From structural risk minimization theory[1-2], the test error \n$\\delta_j=E_{\\mathbf{x} \\sim \\hat{P}_j}[\\ell(f(\\mathbf{x},j)]$ \n\n$- E_{\\mathbf{x} \\sim P_j}[\\ell(f(\\mathbf{x},j)] $ \n\nfor class $j$ is bounded by $C/\\sqrt{n_j}$, where $n_j$ is the class size for $j$ and $C$ is a constant that reflects model complexity. When the sample size of less frequent classes is small during pre-training, the large generalization gaps might lead to the estimation error of $\\pi_p$.    \n\n    [1] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. NeurIPS, 2019.\n\n    [2] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR, 2019.\n---\n\n**Q5** Illustrate that ''since the zero-shot and fine-tuned models provide diverse predictions, conditioned on the two predictions is equivalent to adding the logits in log space\".\n\n**A5** The detailed explanation is included in Appendix Eq.(20-21).  Denote the output $\\mathbf{e}=f_{ft}(\\mathbf{x})$ and $\\mathbf{z}=f_{zs}(\\mathbf{x})$, then \n\n$P_t(y|\\mathbf{e},\\mathbf{z})=\\frac{P_s(y|\\mathbf{e})}{P_s(y)}\\frac{P_p(y|\\mathbf{z})}{P_p(y)}C_1,$\n\nwhere $C_1$ is a constant that not rely on $y$ (see line 323-324 for further explanation on $C_1$). We can rewrite $P_s(y)=\\exp(\\log P_s(y))=\\exp(\\pi_s(y))$ and  $P_p(y)=\\exp(\\log P_p(y))=\\exp(\\pi_p(y))$. As the underlying class-probability $P_s(y|\\mathbf{e}) \\propto \\exp(\\mathbf{e}_y)$ and $P_p(y|\\mathbf{z}) \\propto \\exp(\\mathbf{z}_y)$. The above equation equals to \n\n$P_t(y|\\mathbf{e},\\mathbf{z})=\\frac{\\exp(\\mathbf{e}_y)\\exp(\\mathbf{z}_y)}{\\exp(\\pi_s(y))\\exp(\\pi_p(y))} C_1/C_2=\\exp(\\mathbf{e}+\\mathbf{z}-\\pi_s-\\pi_p)_yC_1/C_2,$\n\nAs the logits have to apply exponential function to get the probability, multiplying two probability equals to adding logits. We refer to ``log space'' as both logits and $\\pi=\\log p$ represent the logarithm of probability.\n\n"}, {"review_id": "1ec06LtUNy", "reviewer": "Reviewer_D4ZP", "summary": "This paper addresses biases in foundation models, such as CLIP, which due to the imbalanced training datasets, are skewed towards frequent semantics. The authors propose a Generalized Logit Adjustment (GLA) method, an optimization-based approach for debiasing these models. Despite inherent challenges, the GLA method demonstrates significant improvements across multiple tasks and datasets, achieving 1.5 pp accuracy gains on ImageNet and large average improvement (1.4-4.6 pp) on 11 few-shot datasets. Notably, the GLA method does not require access to the pre-training dataset, making it practical for fine-tuning scenarios. The paper offers both theoretical justification and extensive empirical evidence for the proposed method.\n", "questions": "\nThe only two questions are listed in weakness.", "limitations": "While it's a promising beginning to uncover the biases in foundational models, it's crucial to recognize that the exploration should not be confined to computer vision alone. It is necessary to extend the studies to other fields as well.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Solid Theoretical Analysis**: The paper provides robust theoretical justification for the Generalized Logit Adjustment (GLA) method. It formalizes the estimation of label bias as a constrained optimization problem, proving that GLA is a Bayes optimal classifier.\n\n2. **Practical Solution**: The proposed method doesn't require access to the pre-training dataset, which makes it practical for fine-tuning scenarios. This approach is useful given privacy and copyright concerns associated with accessing the label distribution of pre-training datasets.\n\n3. **Comprehensive Evaluation**: The paper uses a comprehensive benchmark for evaluation, considering three real-world settings and three fine-tuning paradigms. This diverse approach to evaluation adds credibility to the findings.\n\n4. **Significant Performance Improvement**: The paper reports notable improvements across various tasks, demonstrating the efficacy of the proposed GLA method. The substantial gains across multiple datasets reinforce the value of the method.\n", "weaknesses": "1. **Limited Validation**: The paper only presents results with a limited set of tasks and models. While they used the CLIP model for the zero-shot setup, the validity and effectiveness of the GLA framework with other models (like BERT, GPT, etc.) have not been validated.\n\n2. **Estimating Pre-training Bias**: The estimation of pre-training label bias is done with downstream data. However, this could be problematic if the downstream data distribution significantly differs from the pre-training data distribution, leading to incorrect bias estimation and sub-optimal performance.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1** The validity and effectiveness of the GLA framework with NLP models.\n\n**A1** We appreciate the reviewer for the comment. Applying this algorithm to other tasks and fields is also our future research direction.  For now, we primarily focus is on enhancing the fine-tuning performance for discriminative models. The NLP zero-shot models are dominated by generative models, applying our GLA framework for those models presents challenges. For example, the language generation is a markov process, which makes each output depend on outputs of previous steps. Namely, it is not trivial to estimate the biasedness of a sequence by our GLA where we only computes the bias on a pre-defined and independent label space. We will discuss it in the revision version and leave it as a future work.\n\n---\n\n**Q2** Estimating pre-training bias when the downstream data distribution significantly differs from the one of pre-training data.\n\n**A2** While we acknowledge the concern raised, it is crucial to understand that our work is primarily focused on addressing the label shift problem~(also known as long-tail problem). Yes, you are right in pointing out that our GLA paradigm might not be optimally effective when the downstream data distribution diverges significantly from the pre-training data distribution, e.g. some downstream samples $\\mathbf{x}_s$ do not have density in pre-training domain ($P_p(\\mathbf{x}_s) \\approx 0$) . We recognize this especially in the case of niche domains such as medical data, which are scarcely represented in the pre-training stage. We believe that incorporating such downstream data into the foundation model training might be the only way to resolve the extreme case, which is beyond our scope. \n\nOur methodology, nevertheless, has demonstrated substantial efficacy in the presence of label shift and can effectively mitigate the impact of pre-training label bias in a broad array of scenarios.\n"}]

Author rebuttal:
# Response to All Reviewers

Dear Program Chair, Senior Area Chair, Area Chair, and Reviewers,

First of all, we gratefully thank all the reviewers for their thoughtful comments and feedback.

In this paper, we identify label bias in foundation models like CLIP and underscore its adverse effects on downstream task performance. We propose the Generalized Logit Adjustment (GLA) framework for fine-tuning foundation models, which boosts the performance by effectively eliminating label bias. The contribution of this paper is four-fold: 

**1. Solid Theoretical Analysis:** We formalize the estimation of label bias as a constrained optimization problem and prove that our GLA model is a Bayes optimal classifier (Proposition 1). 

**2. Simple and Practical Solution:** Our GLA is a post-hoc approach without introducing hyper-parameters, which makes it  easy to implement. In addition, we does not necessitate access to the pre-training dataset, which makes it practical for fine-tuning scenarios. 

**3. Comprehensive Evaluation:** We consider three real-world settings and three fine-tuning scenarios, conducting experiments on over 20 datasets. 

**4. Significant Performance Improvement:**  The GLA method demonstrates significant improvements, e.g., it achieves 1.5 pp accuracy gains on ImageNet and large average improvement (1.4-4.6 pp) on 11 few-shot datasets.

We add a PDF that contains additional experimental analysis on bias estimation: we devise an experiment to demonstrate the estimated label distribution strongly approximates to the true one.

As our paper received mixed ratings, i.e., three positive (755) and two negative (44), it would be appreciated if the reviewers could have a look at our responses and revision. We have tried our best to address your concerns in our responses in detail. Hope that our responses answered the questions. Please let us know at your early convenience if you have further questions or concerns.

Best regards,

Authors of Paper \#6116


Meta-review: There was a significant split between the reviewers—three were very positive about the work, while the other two had significant concerns. However, one of these negative reviewers did not engage with the authors' rebuttal or with the other reviewers, and during the final discussion phase two of the more positive reviewers argued convincingly (and I agree) that those concerns were largely addressed by the authors' rebuttal. Overall, this seems like a solid contribution that deserves to be accepted.
Decision: Accept (poster)